Dave Gettleman Vs. The Nerds

The following is a guest post by @sunset_shazz.

New York Giants General Manager Dave Gettleman is a true football man (#TrueFootballMan). He has no time for nerds who sit behind their keyboards. Though some may be concerned about drafting a running back second overall, he is not. From his recent presser after picking Saquon Barkley:

I think a lot of that’s nonsense. I think it’s someone who had this idea and got into the analytics of it and did all these running backs and went through their – whatever. Hey, Jonathan Stewart is in his 10th year and he’s hardly lost anything.

Gettleman appears to believe that the case against using a top 10 pick on a running back rests on perceived longevity. He is misapprehended.

Ben Baldwin, an economist (and Seahawks fan) who makes his living sitting behind his keyboard, summarized the case against using a premium pick on a running back in an excellent post at Field Gulls – do read the whole thing. I’m going to expand on two subsets of his argument: that rookie first round running back contracts are bad values, and bad risks.

The objective of the first round of the NFL draft is to sign an above-average player at a below-average contract for 4 years, with an embedded team option in year 5. Article 7 of the 2011 Collective Bargaining Agreement specifies a rookie wage scale that varies based on the draft pick used to select the player. Importantly, after the 2011 CBA was implemented, the player’s position doesn’t matter: a player picked 2nd overall is paid the same money over 4 years, whether he is a quarterback, running back or long-snapper. Moreover, the market has reached an equilibrium where first round contracts are fully guaranteed. A quick survey of contracts at overthecap.com shows that position value for post-rookie contracts varies significantly in today’s NFL. As a result, the “rookie contract discount” varies dramatically by position. For example, a QB drafted with the 2nd overall pick in 2018 would be the 25th highest paid QB in the league (by average annual compensation) and would have the 15th highest guaranteed money. An RB selected 2nd overall would immediately become the 4th highest paid player at his position, with the highest guaranteed money – all before taking a single professional snap.

The chart above shows selected positions, with their leaguewide positional salary rank plotted against overall draft number (all data courtesy overthecap.com). Running back is the clear outlier – a top 10 pick is automatically among the highest paid RBs in the league.

Here is the same plot for guaranteed money (the rookie contract is compared to the veterans’ amount guaranteed on their current contracts):

Though the 2011 CBA’s wage scale typically serves as a price ceiling for rookies, with running backs drafted in the first round, it serves as a de facto price floor. In terms of guaranteed money, the three highest RB contracts in the league are Barkley (2018, drafted 2nd overall), Leonard Fournette (2017, 4th) and Ezekiel Elliott (2016, 4th). Here are the top 10 picks in this year’s draft, with their annual compensation and total guarantee compared to their league peers by position group (players in top 10 highlighted in red):

The New York Giants have thus expended the #2 draft pick (a considerable use of capital) for the privilege of paying Barkley the #4 annual salary and #1 guarantee at his position. They are paying (through the nose) not just once, but twice!

But perhaps Gettleman is merely acting upon justified conviction. If Barkley is a generational player, surely he’s worth it?

As Ben Baldwin notes in his piece, 1st round running backs have a high bust rate relative to other positions. Data scientist Dr. Sean J. Taylor sent me the following plot, exploring this idea further:

This plot evaluates the set of running backs drafted (or undrafted) between 2009 and 2014. The x-axis represents draft position. The y-axis is the player’s Wins Above Replacement (nflWAR) over the ensuing 4 years. NflWAR is a statistic developed by Yurko, Ventura and Horowitz of Carnegie Mellon University which uses multinomial logistic regression to isolate the contribution of individual players to NFL wins. NflWAR represents a novel effort to advance beyond Approximate Value (AV), and deserves wider recognition.

I draw 3 conclusions from the scatterplot above:

1.     Taking a running back early is risky (bottom left quadrant);

2.     It is possible to find success at running back in later rounds;

3.     Running backs don’t really matter very much (note the Y-axis scale – over 4 years, you get at best 1.5 extra wins from a running back, and typically 0.25 extra win; quarterbacks are approximately 4x more important).

The risk of a bust is even more acute with highly drafted running backs, because the financial commitment to the player is so much higher, relative to other players at the same position. A bust at QB taken at 2nd overall saddles you with the salary of a bottom quartile starter. A bust at RB at the same pick saddles you with a top 4 salary.

But there is no good alternative to taking such a risk, right? Don’t you need to take risks at RB in order to win championships? Ezekiel Elliott, Leonard Fournette and Todd Gurley (taken 4th, 7th and 10th overall, respectively) are commonly cited as evidence of risks that have paid off.

One recurrent theme here at MoK is the application of insights from behavioral science to football. Our past posts have relied heavily on the work of Gary Becker, Daniel Kahneman & Amos Tversky, William F. Sharpe, Kahneman and Tversky again and Joseph Henrich. Today’s post is dedicated to 1990 Nobel Laureate Harry Markowitz who demonstrated that a portfolio of individually risky assets can collectively carry less risk than any of its underlying constituents, even when adjusted for its prospective return.

View fullsize

The above chart shows the three commonly cited high pick successes, and the RB-by-committee groups of the two Super Bowl teams. The “Draft Capital” column dispenses with the archaic Jimmy Johnson scale, instead using Dr. Michael Lopez’s blended draft curve which improves on prior efforts by not only paying attention to expected/modal outcomes, but also giving weight to the probability of drafting a superstar (i.e. the right tail of the distribution). PHI and NE expended between 1/4 and 1/7 the draft resources for their running backs as JAX, DAL and LAR. Though PHI and NE paid relatively high 2017 cap numbers, they locked up minimal resources over the long term (i.e. they could cut bait in 2018). The “gty” column shows guarantees over the entire contracts of those players (Sproles’ and Blount’s initial guarantees for PHI, Gillislee’s, Burkhead’s and White’s for NE).

The advantage of the portfolio approach is: you can be wrong, and still have success. Donnel Pumphrey is not good at football and Darren Sproles was lost for the season. Gillislee, Burkhead and White did not cover themselves in glory in 2017. The portfolio approach diversifies you against injury, suspension or disappointing play. Yet, each portfolio achieved similar yards/attempt and total yards as the 3 high draft picks, for less overall guarantee / draft capital. As a team, NE and PHI ranked 1st and 8th in offensive DVOA, respectively (the Eagles won the Super Bowl). Also, note that NE’s total 2017 expenditure, while high, was less than Le’Veon Bell’s cap number. JAX additionally paid $6MM in 2017 for Chris Ivory, who offered minimal return for this expenditure. As Harry Markowitz showed, the portfolio approach offers something vanishingly rare in economics: a free lunch. A properly constructed portfolio lowers risk, without sacrificing expected return. (Though running backs are risky, they are independently risky. Idiosyncratic risk is diversifiable.)

In summary, Dave Gettleman in his press conference constructed a straw man. The case for positional value does not rest on running back longevity. Instead the TL;DR argument is as follows:

• Using a high draft pick on a running back is a bad bet. At best, you expend draft capital in order to pay a guaranteed contract at a market equivalent price for a good player. At worst, you overpay twice: in draft capital and guaranteed salary for a bad player.
• Drafting a running back is risky.
• By assembling a portfolio of RBs, one can achieve similar performance to drafting a star, while diversifying risk, and saving draft / guarantee capital to deploy elsewhere.
• Your mother was right about eggs and baskets.

The above argument relies upon the prior work of a number of individuals who sit behind keyboards, all of whom have advanced degrees in a quantitative field such as economics, and none of whom have played a snap of professional football. Gettleman, a #TrueFootballMan, will confidently dismiss this argument, regardless of its merit, due to its provenance. Eagles fans should pray he never gets fired, and lives forever.

Nick Foles Is The Playoff GOAT

The following is a guest post by @sunset_shazz.

Nick Foles is a high-variance quarterback. His performance ricochets from abysmal to sublime with such frequency that he made me re-adjust my chart axis, twice. And yet: including the 2013 loss to the Saints (in which he engineered a comeback from a 13-point deficit and left the field with the lead) his postseason play has been consistently excellent. There have been 93 quarterbacks since the 1970 merger who have played at least 4 playoff games. Of these, Foles ranks 1st in completion percentage and 2nd in Adjusted Net Yards / Attempt (ANY/A).

Obviously, this is not statistically dispositive. Nothing about playoff analysis is. Mark Messier and Reggie Jackson’s playoff performances comprised a mere fraction of their total careers, yet their knack for elevating their game on the biggest stage is what made them memorable. One way to think about the playoffs: there is a tide in the affairs of men, which, taken at the flood, leads on to fortune. As I will show, Foles has taken the tide at the flood in historic fashion.

Note, from the chart above, that the fewer games played, the greater variance in ANY/A between individual players. But what about each player’s game-by-game variance? I measured the standard deviation of each player’s game ANY/A, and scaled this by his mean ANY/A, thus constructing a coefficient of variation.

Of all 93 QBs in the sample, Foles has been the 4th most consistent (i.e. has the 4th lowest variation). Moreover, he has the lowest variation of the 16 QBs who have only played 4 games.

Perhaps Foles has benefitted from playing in a QB-friendly era? I compared each QB’s game ANY/A to the league average for the year in which that game was played. One can then plot mean Relative ANY/A against the coefficient of variation:

Foles has the 5th highest Relative ANY/A in addition to having the 4th lowest variation. One way to think about the above graph is to imagine an “efficient frontier” on the upper left quadrant. When considering similar efficient frontiers in the context of financial economics, Nobel Laureate William F. Sharpe constructed a “Sharpe ratio” which compares a fund manager’s relative return (e.g. versus an index) to the standard deviation of the fund’s return.

I similarly devised a playoff QB Sharpe Ratio, which is each QB’s mean Relative ANY/A divided by the standard deviation of his game ANY/A. Think of it as one number which captures both efficiency and consistency of play. The following table shows the top 10 playoff QB Sharpe Ratios since the merger:

All 10 of these quarterbacks played in a Super Bowl, and all but two of them were champions. Only Bengals starter Ken Anderson and Bills backup Frank Reich did not win the season’s final game. (Reich, of course, will receive a ring as Offensive Coordinator of the 2017 Super Bowl champions.)

By this metric, Foles will have to settle for second place out of 93 playoff QBs. The Raiders’ Ken Stabler, who played in 13 playoff games between the 1971 and 1979 seasons, passed for 3.08 ANY/A above average (3rd) and had the 8th lowest coefficient of variation in the sample. Combining efficiency and consistency, he is the greatest playoff quarterback of all time. Here are the rankings of some other notable QBs, and Eli Manning:

Obviously, I’m not suggesting Foles is better than any of those quarterbacks (except Eli; he’s indisputably better than Eli, it’s not even close). However, in the inherently limited sample that consists of the playoffs, Foles has performed at a historically great level, in terms of both efficiency and consistency. Also, he can catch.

Clearing Up The Coaching Confusion

The following is a guest post by @sunset_shazz.

Should the Carolina Panthers have fired Head Coach Ron Rivera or traded QB Cam Newton the day after they lost the Super Bowl? Scott Kacsmar at FiveThirtyEight argues they should have done either or both. Do read the whole piece; the argument is presented as follows:

• In NFL history, only 4 coaches have won their first Super Bowls after 5 seasons on the job with the same team;
• No team has ever started the same quarterback under the same head coach for more than 5 years and seen that duo win its first championship.

Having examined the history of prior first time Super Bowl winners, FiveThirtyEight infers that these characteristics are conducive to winning championships. The study’s conclusion: “If championship success doesn’t come within five years, things tend to get stale, and someone eventually has to move on from their position of power.”

Can you spot the flaw in this reasoning?

How about if I used the same exact logic, using a more emotionally salient characteristic:

• In NFL history, only 4 minority head coaches have won Super Bowls. Therefore you shouldn’t hire minority head coaches. [1]

Does that framing device make the flaw in reasoning clearer?

FiveThirtyEight’s study suffers from the confusion of the inverse, a statistical fallacy that undergraduates are commonly taught to avoid. One of the best recent treatments of this problem was a brilliant piece by Katherine Hobson on the lab-testing startup Theranos  (also, funnily enough, at FiveThirtyEight). Chapter 8 of Nate Silver’s excellent The Signal and the Noise provides a lucid discussion on this topic, in the context of Bayes’s theorem.

Here is the issue: the fraction of Super Bowl winners that possess a certain characteristic, by itself, tells you nothing about the probability that those who possess that characteristic will win a Super Bowl. A better way to estimate the latter would be to go back and examine the historical success rate of coaches who possess the characteristic you’d like to study.

I compiled every season coached since the 1970 merger, then excluded the seasons after a coach has won his first Super Bowl.  Coaches who were tenured 5 years or fewer with their teams won 24 first Super Bowls in 1009 opportunities, for a success rate of 2.38%. Coaches tenured 6 or more years won 4 first Super Bowls in 176 opportunities, a 2.27% success rate. Using a technique previously used in the Duck Bias study, I applied the cumulative distribution function of the binomial distribution to test whether the success rates were different, to a statistically significant degree. The P value of 0.592 indicates no statistically significant difference.[2]

However, Super Bowl success is a noisy, sparse data set, due to the very small sample size. An alternative measure of coaching success which enjoys the advantage of more data is the frequency with which a coach makes the playoffs. I compiled the playoff rate for every coach in the dataset, and compared this with the base rate of success for that year.[3] The data shows that coaches with longer tenure are actually ­­more likely (47.7%) to make the playoffs than shorter tenured coaches (31.1%) and the base rate (38.0%); both of these differences are statistically significant.

Obviously, this data doesn’t tell you anything about causation. There is likely a survivorship bias / selection effect: those coaches who are kept by their team after 5 years without a championship are likely of higher quality than average, which is probably why their subsequent success rate is higher.

For Coach-QB pairings, 28 Super Bowls were won in 1137 opportunities for the short tenured pairs, a 2.46% success rate. There were only 48 seasons where a Coach-QB pairing lasted more than 5 years without having won a Super Bowl. The zero success rate is, statistically speaking, the effect of randomness, rather than a measured effect. In terms of playoff success rate, once again the longer tenured coaches had a higher success rate, though this effect was not found to be statistically significant.

The data is pretty clear: you shouldn’t fire your coach, or your QB, just because he has not won a Super Bowl after an arbitrary number of years. The only reason short tenured coaches seem to have been historically more successful is they vastly outnumber the long tenured ones. FiveThirtyEight’s model was fooled by the fallacy of the inverse.

But given that we’re in the midst of a coaching carousel accompanied by a Rooney Rule kerfuffle: what about the reductio ad absurdum argument I cheekily proffered above? What does the data say about minority head coaches?

I used Wikipedia’s Rooney Rule page to code every minority head coach in the dataset, presented below. This data comprises every coaching season between 1970 and 2016, including all seasons for coaches who won multiple Super Bowls.

Minority coaches won Super Bowls in 3.31% of their opportunities, which is statistically indistinguishable from the base rate of success of 3.22% (note that minorities have disproportionately coached in more recent years, after the league has expanded, which lowers the base rate of championship success). Interestingly, minority coaches made the playoffs 58 times in 121 opportunities (47.9%) which is 11.6 more times than one would expect given the base rate, a difference that is statistically significant (p=0.02). This is a noteworthy result: historically, the presence of a minority head coach is associated with a 25% greater rate of making the playoffs.

Again, one shouldn’t make causal inference claims from historic data. I’m not arguing that minorities are inherently better coaches. In this situation, there is no survivorship bias. Might there be a selection effect? The late Nobel Laureate Gary Becker argued in 1957 that employment discrimination (racial or otherwise) is inefficient. Not only does the victim of discrimination bear a cost, but so does the discriminating employer (through lower productivity per unit labor cost). Axiomatically, to the extent that some employers exhibit an unfounded bias, an employer who doesn’t discriminate can capture a portion (but not all) of the foregone surplus. Moreover, selecting from a pool of employee candidates who are the victims of racial discrimination will yield supernormal productivity. Becker’s theory of discrimination is one plausible explanation for the effect shown by the data.

What does this mean for NFL teams today? The data is unequivocal that Ron Rivera shouldn’t be fired solely because he hasn’t yet won a Super Bowl. The data also shows that Al Davis, who got many things wrong, got a few things very right. An NFL team should examine the pool of minority head coach candidates very carefully, and should strongly consider hiring from this pool.

Not merely because it’s the right thing to do, but because the data suggests it helps you Just Win, Baby.

[1] To be clear, this is not argued by FiveThirtyEight. I employed this reductio ad absurdum to permit the reader to more easily intuit the confusion of the inverse.

[2] The P-value is the probability that, conditional on the null hypothesis being correct (i.e. no effect), one would observe the data in question by chance. Though subject to recent debate, the conventional standard for social science is to reject P-values greater than 0.05.

[3] The base rate of success is # of playoff teams / # of total teams in the league, both of which have changed over time. I accounted for the league’s expansion of teams in 1976, 1995, 1999 and 2002, as well as the evolution of the playoff format from 8 to 12 teams in 1978 and 1990, and the 16 team playoff that occurred during the strike-shortened 1982 season.

Can't Run On Us

The following is a guest post by @sunset_shazz.

As the inimitable Jimmy Kempski recently explained, the Eagles’ defensive game plan is rather simple:

1. Stop the run.
2. Make the opposing offense one-dimensional.
3. Get after the quarterback.

Indeed, Eagles opponents do appear to give up on the run – this year the defense has faced the fewest rushing attempts per game. One caveat: they also have the second-highest point-differential. As everyone knows, you face fewer run attempts when holding a lead.

Are the Eagles’ opponents giving up on the run because they’ve fallen behind? Or are the Eagles facing fewer rush attempts due to their stout run defense, irrespective of the scoreboard?

I examined Game Script data compiled by Chase Stuart. Game Script is basically the average score margin over a total game. As a stylized example, let’s say Team A returns the opening kickoff for a touchdown, kicks the extra point, then neither team scores for the rest of the game. Team A’s Game Script in this simplified example would be +7 (the average lead held the entire game); Team B’s Game Script would be -7. A higher game script is often associated with more rushing by the team who’s leading (because you run when you win, not win when you run) and more passing by the opposing team (which has a negative game script).

The plot below shows each team’s average Game Script on the X-axis and average Pass/Run Ratio on the Y axis (all data through weeks 1-10). The regression line represents the expected Pass/Run Ratio, given Game Script, computed from the 146 games that were played through week nine. [1] A team that is toward the right (LAR, PHI) has enjoyed a higher average lead, and a team toward the top (SFO) has a higher pass/run ratio.

View fullsize

By comparing each team’s pass-run ratio to what one would theoretically expect given game situation (denoted by the regression line above), one may construct a “Pass Heavy Index”:

View fullsize

This year, Bill Belichick has been 10.9% more likely to call a pass, given game situation, than average. With Mitchell Trubisky behind center, John Fox is 15.5% less likely to call a pass, given game situation, than average. Despite almost being run out of town after a week 2 game in which his Pass Heavy Index was +27%, Pederson is basically in the middle of the pack.

What about the defense? One may similarly plot each team’s opponent’s average pass/run ratio against the opponent’s average game script:

View fullsize

Did you notice the outlier on the upper left? One may also compute each team’s opponents’ Pass Heavy Index:

View fullsize

The table above shows that Eagles opponents are not only passing more than any other team’s opponents (70.1% of the time), but that they are the most pass heavy adjusted for game situation. The evidence supports Kempski’s thesis: Eagles opponents this year have become one dimensional. One way to look at it is that opponents have a healthy respect for the Birds’ run defense. Another way to view it: they think they can attack the secondary. Somebody please inform the Green Goblin that he’s being disrespected.

View fullsize

Stick Figure GIF reprinted with permission, courtesy Jimmy Kempski

Alternatively, perhaps this is an artifact of sampling bias – maybe the Eagles just happen to have faced teams who pass a lot (like Arizona).

View fullsize

Looking at it game-by-game, 6 out of 9 teams the Eagles played chose to pass more than they typically do, adjusted for game situation. There were three exceptions: the Cardinals 76.7% pass ratio, though 8.3% higher than expected, was a tad (0.5%) less pass heavy than Bruce Arians’ typical game; this was due to an extreme game script driven by a three touchdown first quarter by Carson Wentz. In the most recent two weeks, the Niners and Broncos both continued to run more than would be expected, despite falling behind by two scores in game script. Could this portend a change in opponent strategy, perhaps due to the absence of LB Jordan Hicks, whose season ended in the first series of Week 7? Or was this merely due to the injury to Joe Staley for the Niners  and the move to Brock Osweiler for the Broncos? The next few weeks should be interesting.

The analysis presented above demonstrates that Eagles opponents are 10% less likely to run the ball than average, given the game situation. If opponents indeed are choosing to attack the Eagles’ passing defense, they are picking a different, though still potent, poison. The Eagles passing defense is ranked 7th in ANY/A allowed, and ranked 8th in defensive passing DVOA.

View fullsize

Interestingly, the Rams’ and Jaguars’ average lead has been similar to the Eagles’, though their opponents are running more than typical, given such a deficit. Those teams are ranked 2nd and 1st against the pass, respectively, in DVOA, and are ranked 15th and 30th against the run. Though opponents of each team are falling behind during games at a similar rate, they are choosing to attack the Eagles differently, given the relative strengths of their defensive units.

Thanks to Eagles fan Noah Becker and MoK Editor-in-Chief Brian Solomon for discussion leading to this post. 

[1] The Y-intercept indicates the neutral pass/run ratio, 57.8%, which also mathematically corresponds to the league average pass-run ratio.

The Kids Are Alright

The following is a guest post by @sunset_shazz.

Carson Wentz’s start to the 2017 season has garnered national plaudits for his stewardship of the Eagles’ league-leading offense. But it being 2017, there lurks a coterie of skeptics who claim his underlying ability is “horrendous” like Blake Bortles or merely pedestrian like Andy Dalton. Even more emphatically, poor Jared Goff was confidently pronounced a bust after one season.

Is it fair to judge a quarterback solely on his rookie year? What about after the first nine weeks of his second season in the league? And how might one systematically evaluate a developing quarterback, relative to historical data?

Let us consider some advanced metrics that are used to evaluate quarterbacks:

• Adjusted Net Yards / Attempt (ANY/A) was developed by the great Chase Stuart, and accounts for sack yards, while providing a bonus for touchdowns and a penalty for interceptions. Both Stuart and Topher Doll have shown that ANY/A predicts wins. Danny Tuccitto has brilliantly used confirmatory factor analysis to show that ANY/A is a stable indicator of QB quality.
• Defense-adjusted Value over Average (DVOA), the brainchild of Aaron Schatz at Football Outsiders, is a success-based, opponent-adjusted per-play efficiency metric intended to both correlate with non-opponent adjusted wins (descriptive) and to predict future opponent-adjusted wins.
• Defense-adjusted Yards above Replacement (DYAR) uses similar success-rate inputs to DVOA, in order to compute an aggregate value for a player (combining volume and efficiency).
• Total QBR is ESPN Stats & Information’s proprietary efficiency metric that combines both passing and running contributions, adjusted for game situation, with charting to assign responsibility to a quarterback’s receivers and blockers.

Through nine weeks, the 2017 sophomore class is playing at an extraordinarily high level, as measured by each of these advanced stats:

View fullsize

Please note that nothing herein intends to argue for any of these quarterbacks to the detriment of the others. Though the data presented above is insufficiently precise to draw ordinal rankings, it is unequivocal:

Wentz is good. Goff is good. Prescott is good. All three of these things can simultaneously be true, pace internet trolls.

Some epistemic humility is in order: the first-nine-week sample size is obviously noisy, with varying degrees of luck, opponent quality, team injuries, coaching quality and supporting casts influencing the statistical performance of each QB. Danny Tuccitto warns us that ANY/A stabilizes at 326 dropbacks, and even at that sample size, 50% of the observation represents randomness/luck. Nonetheless, the broad takeaway should be that each sophomore QB has thus far performed at a top-quartile level, judged by a variety of different metrics. Is this good? And how confident can we be that such performance will continue?

Recently, Chase Stuart noted that three sophomores from the same class have not played this well since at least the NFL-AFL merger. Though ANY/A is less context-specific than the other measures, it has the advantage of being transparent and easy to calculate, permitting historical analysis. Stuart compared the first 8 weeks of 2017 for Goff, Prescott and Wentz to full seasons of prior 2nd year QBs. Comparing partial to full seasons isn’t quite neutral, due to the disparity in number of games sampled; we should expect some mean reversion of our reference QBs as sample size increases. Using pro-football-reference’s excellent query engine, I examined the first 9 weeks for each sophomore quarterback from 1999 through 2017. Historical comparisons need to be adjusted for era, due to the enormous change in average NFL passing efficiency over time. To account for this, I divided each quarterback’s ANY/A by the league average for that year. [1]

Top ANY/A vs Average since 1999, sophomore QBs, weeks 1-9

View fullsize

The 76 QB sample set in this study is itself a product of survivorship bias: only those QBs who were successful enough to throw 100 passes in the first 9 weeks of their second year in the league are included. On the other side of the distribution, successful QBs who rode the pine for their first few years (like Aaron Rodgers, Tony Romo or Philip Rivers) are not in this sample. The average age of the sample is 24, similar to our reference QBs.

The three 2017 sophomores are, as Stuart observed, performing extraordinarily well relative to their peer set (all are in the top quartile of the sample). Relative to their era, they are passing with greater efficiency than Tom Brady, Drew Brees, Matt Ryan or Andrew Luck did in their second seasons.

You will also note that the top ranked sophomore QBs include many future hits (Big Ben, Kurt Warner, P. Manning) and a few notable misses (Nick Foles, Derek Anderson). The last column I included is the Career Approximate Value (CAV), which is a (very) rough method developed by Doug Drinen that puts a single number on a player’s total career, encompassing both longevity and performance.

Below, I plotted log Career Approximate Value against ANY/A relative to league average for the first 9 weeks for second year QBs from 1999-2015 (I excluded QBs from 2016-2017 because recent QBs have not yet had sufficient time to accumulate CAV points).

View fullsize

The positive relationship shown above indicates that the first 9 weeks of a sophomore season predicts 37% of a QB’s future CAV. Do note that the correlation is sensitive to a few outliers. The odious Ryan Leaf and Akili Smith are on the bottom left, whereas Foles and Anderson are on the bottom right. I don’t want to ascribe an illusion of precision to this rough analysis – don’t fixate on the exact R-squared number, or the model coefficients. Both sample size and the extremely imprecise nature of CAV make me hesitant to draw definitive conclusions from the data. What is interesting to me is that the same plot using a QB’s full rookie season yields an R-squared of 0.224 – in other words, the first 9 weeks of a QB’s sophomore season tells you roughly 70% more about his future career than his entire rookie season does. Extending this analysis to full seasons since 1970, the R-squared is 0.083 and 0.2348 for rookie and sophomore years, respectively (n=155 & 204). My interpretation of this data: though rookie and second year passing efficiency predict only a small fraction of a quarterback’s career value, the sophomore year deserves 2.8x as much weight as the rookie year, in terms of confidence about predictive power. Rookie performance, in particular, is extremely noisy. One would have been wise to heavily discount Troy Aikman, Donovan McNabb and Terry Bradshaw’s dreadful rookie seasons. Rams fans should take note.

Relatedly, I didn’t find any predictive power when measuring the degree of era-adjusted-ANY/A improvement from rookie to sophomore season. This echoes Vincent Verhei’s study of second year improvement using DVOA. In hypothesis testing, a negative result can be an interesting result.

Quantitative analysis is not the only tool in an NFL researcher’s kit. Film study (though not my sphere of competence) is also valuable. Though Nick Foles had a magical sophomore season, the film showed reason for concern, as my friend Derek Sarley noted. I don’t personally see similar issues with Wentz – both his pre-snap adjustments and post-snap play appear to pass the “eye test”. No, he’s not perfect. Yes, he has flaws he needs to address. But so do all second year quarterbacks.

Moreover, our penchant for treating quarterbacks as static vessels of talent/ability shortchanges the importance of coaching and development. The installation of a new coaching regime in Los Angeles appears to be an interesting natural experiment, in terms of Goff’s maturation. Similarly, we can view Ezekiel Elliott’s probable(?) suspension as an instrumental variable when evaluating Prescott.

All inductive statements are, by their very nature, revisable. We don’t know the future; we can only use informed judgment to hazard a prediction. The false-positive rate for the top 20 QBs in table 2 above is 25% by my count [2], so let’s take that as the “base rate” of failure for the 2016 Sophomore QBs. It is therefore reasonable to expect that two – perhaps all three – of the 2016 sophomores will enjoy successful careers as NFL starters.

Finally, in these impatient times, let us remind ourselves that transcendent quarterbacks do not emerge, fully formed, from the forehead of Zeus. Each of these young, relatively inexperienced quarterbacks is playing the most technically and cognitively demanding position in sports at a very high level. Adjusted for experience and era, their achievements are even more astounding. The evidence suggests that the future of quarterback play is bright. Football fans, rejoice.

Thanks to Eagles fan / Data Scientist Sean J. Taylor for his insightful discussion on methodology. Any errors are mine alone.

[1] PFR’s partial season engine shows results from 1999 onward. Full season results go back before the merger, and also generate an era-adjusted ANY/A+ which uses a “Z-score” methodology, expressed in standard deviations above or below the population mean. My method is less sophisticated, though nonetheless robust.

[2] I excluded the reference QBs, as well as Marcus Mariota.

@sunset_shazz is an Eagles fan who lives in Marin County, California. He previously wrote about 4th down decisions.

Can The Eagles Defense Carry The Offense?

Last week, I placed the blame for the Eagles' 0-2 start squarely at the feet of the offense. Despite this week's win over the Jets (yay!), Chip Kelly's favorite unit continued to perform woefully.

Yes, Ryan Mathews, subbing in for DeMarco Murray, rushed for 108 yards on 25 carries. But his numbers hide the fact that there were still far too many negative or no gain rushes (10 total). Between Mathews and Darren Sproles, the rushing game averaged just 3.47 yards per carry, a heady improvement from the prior weeks, but nothing to get excited about. Everyone on the offensive line, including Kelce, Peters, and Johnson, are struggling to run block consistently.

Meanwhile, despite being sacked just once, Sam Bradford played another horrible game. His final numbers were 14 for 28 (50% completion rate) for just 118 yards. One touchdown and zero turnovers, if you want to focus on that. But his accuracy was poor, his ball placement terrible, and he has zero guts to throw the ball downfield, one of the only things that might give the running game enough room to breathe. See more at ChipWagon, as always.

All of which leads to...

The #Eagles offense is regressing. 2013: 6.3 yards per play 2014: 5.6 yards per play 2015: 4.5 yards per play

— Phil Bicking (@p_Red) September 28, 2015

Really bad. So the question is, until the time the offense recovers (assuming that's in the cards?), can the Eagles defense carry the team? And on that front, at least, there are some promising signs. To the stats!

Through three games, the Eagles have given up the 5th-fewest points per drive (1.42), the 9th-fewest yards per drive (26.9) and the 10th-fewest plays per drive (5.6). The run defense has been fantastic—allowing a league-low of 3.1 yards per rush and zero rushing touchdowns. Most important, they've forced turnovers on one out of every five opposing possessions. To keep it going, we'll need more Walter Thurmond ball-hawking and disruption at the line of scrimmage (Brandon Bair sighting FTW).

The worrisome note is that the Eagles haven't gotten consistent pressure on quarterbacks. The defense ranks 8th-worst in sack rate so far. Overall, this isn't terribly surprising. The two-gapping scheme means players like Fletcher Cox are responsible for the run first, then the pass second. Neither Connor Barwin nor Brandon Graham are consistent man-beaters on the outside. And now the top two middle linebackers are hurt. Billy Davis needs to mix in some good blitz concepts to get pressure if this defense is going to sustain itself.

Three games is also quite a small sample size, especially when about half of that time the Eagles faced Brandon Weeden and Ryan Fitzpatrick. Luckily, due to a combination of easy schedule and injuries, the Eagles aren't scheduled to face many top quarterbacks down the stretch. The next three games match up against Kirk Cousins, Luke McCown, and Eli Manning, with Cam Newton as the pre-bye week chaser.

Still, that we're having this conversation at all is a good sign. The defense was atrocious two years ago, and improved to average or slightly above last year. If it can take another leap, that's a huge deal for this team—especially if the offense can't turn it around.

Will The Eagles Get Better?

If you watched the putrid Thursday Night Football game between the Giants and the Redskins last night, there was some solace for Philly fans. Mainly that everyone else in this division is pretty much just as #terrible as the Eagles.

That does leave room for optimism, despite two of the worst Eagles games in recent memory, strung back to back at the start of this season. The truth is that as bad as the Eagles have been, they still can turn things around. The question is whether their awful play so far is the result of fundamental, uncorrectable problems or something more fixable.

The answer to that question is in the eyes of the beholder. The offensive run blocking has been so bad that it's the worst (through two game sample size, of course) of any offensive line I can find in Football Outsiders' history. Their Adjusted Line Yards (aka roughly the yardage that the line is responsible for) is just 1.11 yards. To put that in perspective, last year's poor line was ranked 29th in ALY with 3.52 yards. Ryan over at ChipWagon can give you both the execution and scheme reasons for such failure. 

On one hand, maybe this is a sign that with a historically-bad offensive line and questionable play calling, the Eagles will never get their run game together. On the other hand, we can probably expect the Eagles to revert back at least somewhat to the mean, making the run offense moderately usable at some point. (Optimism!)

Ditto for Sam Bradford. This is a quarterback who has endured very little pressure from defenses (just 23% of the time through two games). The offensive line, for all of its run blocking woes, has allowed just one sack, good for the 3rd-best Adjusted Sack Rate in the NFL of 1.5%. Yet Bradford continues to be unable to throw the ball down the field.

From @PFF, Bradford has attempted just 5 passes (5.6% of att.) of 20+ yards. 0 completions, 2 drops.

— EaglesRewind (@EaglesRewind) September 25, 2015

Bradford has averaged just 6.3 yards per attempt so far, 28th best in the league. To go with that, he's thrown 2 touchdowns against 4 interceptions and has looked out of sync with his receivers the whole time. Maybe Bradford is just bad, and Chip Kelly bet on the wrong quarterback. Maybe Mark Sanchez (barf) is actually the better choice right now. Or maybe Bradford hasn't played much in two years and is still adjusting to a new offense, new personnel, and being hit. Maybe he'll get better.

The defense, meanwhile, has shown promising signs (Malcolm Jenkins, Walter Thurmond, Bennie Logan) and worrisome ones (Byron Maxwell). The stats are slightly above average through the first two games against two potentially decent offenses. The defense certainly isn't good enough to carry an offense this bad, but it isn't the reason the Eagles have lost, either. More tests to come, though.

This weekend's game against the Jets is a must win. You don't go 0-3 and make the playoffs in this league. According to the chart below from FiveThirtyEight, that would leave them with just a 2% chance. But if they go 1-2, that's 25% and a tie for second place in a historically weak division. Then just maybe, there's room for hope...

Ranking The 2013 NFL Kickers (Or Why Alex Henery Doesn't Deserve Your Support)

Everyone knows that NFL kickers are getting better. The numbers make this obvious. We saw an all-time high for field goal accuracy in 2013: over 86%. And that's despite many more long attempts. Field goal tries from over 50 yards have increased 57% in the last decade.

When ranking kickers, we have to take into account the new landscape. For an example, just look close to home. After his first three seasons, Alex Henery is tied for fourth all time in overall field goal accuracy among players with at least 80 attempts. But if you check out the top 20 on that list, 17 of them played in the last two years. This is like the late 1990s in baseball -- everyone's breaking records.

More to the point, we can't quote field goal accuracy without taking distance into account. As Jimmy Kempski noted, Henery doesn't look nearly as good when you compare him to his peers. Among full time kickers since 2011, Henery is tied for the fewest attempts over 50 yards (5). He also has the shortest "long field goal" (51). If everyone else is taking longer attempts and making them, Henery's value is significantly lower than his overall accuracy would suggest.

Last offseason, I took a deep dive into Henery's value, comparing him to other kickers in 2012. By generating an estimated value for every kick, we could plot how much each kicker actually contributed to his team as compared to the league standard. The results put Henery just a hair above average, with the noted caveat of having not a single make over 50 yards.

This year I wanted to go deeper. To start, I pulled all 2013 field goal attempts and plotted them out. While upgrading from last year's 5-yard buckets required a little smoothing, the granular data this year should be more exact:

View fullsize

Cool, right? As you can see, kicks from in close are automatic and everything up to about 32 yards out is a 90% success rate or higher. But perhaps even more impressive, kickers in 2013 maintained over 80% accuracy up to 45 yards out -- and 70% all the way to 55 yard attempts. The data gets meager from there, but the graph is proof of how great NFL kickers have become.

Anyway, with this data, it becomes fairly easy to generate an estimated point value for every field goal attempt. For example, a 35 yard try is worth 2.63 points while a 50 yarder is worth 2.1 points. If a kicker makes a kick from that distance, he adds points to his team above the expected value. Missing, he loses his team the expected points. (The only major caveat here is weather conditions. Certainly a field goal into strong wind would have a lower expected value than one in a dome.)

This year, we'll also add kickoff analysis. I spent some time playing with large data sets of kickoffs and ultimately was disappointed by the many problems in the NFL's play by play data. So at least until Football Outsiders releases its always-awesome game charting this summer, I can't work with the type of granular data on field goals above. The next best thing is Pro Football Focus' kickoff stats. They track distance, touchbacks, and average yard line start. I adjusted that data to eliminate the effect of onside kicks, but there are other variables (including weather) which still exist outside the scope of the investigation.

For kickoffs, the goal is to reach a similar expected value score. With help from Derek Sarley, we calculated a quick and dirty correlation between touchbacks and starting drive position of .67, suggesting that kickers have about two-thirds of the responsibility on kickoffs. Using that, we can multiply each kicker's total drive start yardage difference from the league average by .67, then by the approximately 0.06 expected points each extra yard is worth. Again, it's not a perfect score, but it will give us a good estimate.

Got all that? Click on the table below to enlarge the full results:

View fullsize

Hopefully the table isn't too confusing. Let's break it down. The left hand side is kickoff data, and you can see adjusted kickoff distance and drive start. From there you can calculate points generated in relation to the average kicker (KICKOFF PTS). On the right side are field goals, first the basic data then the estimated values based on the calculations above, leading to the FG PTS above or below average. Finally, TOTAL PTS combines the two scores and ranks all kickers (including those who only kickoff or attempt field goals).

The best kickers of 2013 did well on both metrics. Dan Bailey of the Cowboys, for example, helped force opponents to start their drives 1.9 yards further back than average. Over the course of the season, that was worth just over a touchdown in field position. On field goals, he made over 93% of his attempts, when the average kicker would have only made 84.5%. That was another 8 points he generated for Dallas, bringing Bailey's total points over average to a league-leading 15.1. If we can translate that using Brian Burke's data at Advanced NFL Stats, Bailey was worth about as much in expected points added as the 25th-ranked quarterback.

On the other end of the spectrum, Henery ranked third-worst in the NFL last year. He lost the Eagles approximately 10.8 points, split about equally between kickoffs and field goals. Henery was near the bottom of the pack in kickoff distance and average drive start, losing 1.6 yards per kickoff. Meanwhile, his 80% field goal accuracy should have been better, given he took mostly shorter kicks. The league average accuracy for his attempts would have been 85.8%.

Though Henery has some ardent supporters, it's tough to find any silver lining for him in these numbers. Only two kickers in this sample had worse results: a historically-bad rookie and a veteran who was fired in December. Maybe Henery can improve this offseason and post a better effort in 2014. But I wouldn't count on it. At the very least, the Eagles must bring in competition for what is clearly one of their weakest positions.