Dave Gettleman Vs. The Nerds

The following is a guest post by @sunset_shazz.

New York Giants General Manager Dave Gettleman is a true football man (#TrueFootballMan). He has no time for nerds who sit behind their keyboards. Though some may be concerned about drafting a running back second overall, he is not. From his recent presser after picking Saquon Barkley:

I think a lot of that’s nonsense. I think it’s someone who had this idea and got into the analytics of it and did all these running backs and went through their – whatever. Hey, Jonathan Stewart is in his 10th year and he’s hardly lost anything.

Gettleman appears to believe that the case against using a top 10 pick on a running back rests on perceived longevity. He is misapprehended.

Ben Baldwin, an economist (and Seahawks fan) who makes his living sitting behind his keyboard, summarized the case against using a premium pick on a running back in an excellent post at Field Gulls – do read the whole thing. I’m going to expand on two subsets of his argument: that rookie first round running back contracts are bad values, and bad risks.

The objective of the first round of the NFL draft is to sign an above-average player at a below-average contract for 4 years, with an embedded team option in year 5. Article 7 of the 2011 Collective Bargaining Agreement specifies a rookie wage scale that varies based on the draft pick used to select the player. Importantly, after the 2011 CBA was implemented, the player’s position doesn’t matter: a player picked 2nd overall is paid the same money over 4 years, whether he is a quarterback, running back or long-snapper. Moreover, the market has reached an equilibrium where first round contracts are fully guaranteed. A quick survey of contracts at overthecap.com shows that position value for post-rookie contracts varies significantly in today’s NFL. As a result, the “rookie contract discount” varies dramatically by position. For example, a QB drafted with the 2nd overall pick in 2018 would be the 25th highest paid QB in the league (by average annual compensation) and would have the 15th highest guaranteed money. An RB selected 2nd overall would immediately become the 4th highest paid player at his position, with the highest guaranteed money – all before taking a single professional snap.

The chart above shows selected positions, with their leaguewide positional salary rank plotted against overall draft number (all data courtesy overthecap.com). Running back is the clear outlier – a top 10 pick is automatically among the highest paid RBs in the league.

Here is the same plot for guaranteed money (the rookie contract is compared to the veterans’ amount guaranteed on their current contracts):

Though the 2011 CBA’s wage scale typically serves as a price ceiling for rookies, with running backs drafted in the first round, it serves as a de facto price floor. In terms of guaranteed money, the three highest RB contracts in the league are Barkley (2018, drafted 2nd overall), Leonard Fournette (2017, 4th) and Ezekiel Elliott (2016, 4th). Here are the top 10 picks in this year’s draft, with their annual compensation and total guarantee compared to their league peers by position group (players in top 10 highlighted in red):

The New York Giants have thus expended the #2 draft pick (a considerable use of capital) for the privilege of paying Barkley the #4 annual salary and #1 guarantee at his position. They are paying (through the nose) not just once, but twice!

But perhaps Gettleman is merely acting upon justified conviction. If Barkley is a generational player, surely he’s worth it?

As Ben Baldwin notes in his piece, 1st round running backs have a high bust rate relative to other positions. Data scientist Dr. Sean J. Taylor sent me the following plot, exploring this idea further:

This plot evaluates the set of running backs drafted (or undrafted) between 2009 and 2014. The x-axis represents draft position. The y-axis is the player’s Wins Above Replacement (nflWAR) over the ensuing 4 years. NflWAR is a statistic developed by Yurko, Ventura and Horowitz of Carnegie Mellon University which uses multinomial logistic regression to isolate the contribution of individual players to NFL wins. NflWAR represents a novel effort to advance beyond Approximate Value (AV), and deserves wider recognition.

I draw 3 conclusions from the scatterplot above:

1.     Taking a running back early is risky (bottom left quadrant);

2.     It is possible to find success at running back in later rounds;

3.     Running backs don’t really matter very much (note the Y-axis scale – over 4 years, you get at best 1.5 extra wins from a running back, and typically 0.25 extra win; quarterbacks are approximately 4x more important).

The risk of a bust is even more acute with highly drafted running backs, because the financial commitment to the player is so much higher, relative to other players at the same position. A bust at QB taken at 2nd overall saddles you with the salary of a bottom quartile starter. A bust at RB at the same pick saddles you with a top 4 salary.

But there is no good alternative to taking such a risk, right? Don’t you need to take risks at RB in order to win championships? Ezekiel Elliott, Leonard Fournette and Todd Gurley (taken 4th, 7th and 10th overall, respectively) are commonly cited as evidence of risks that have paid off.

One recurrent theme here at MoK is the application of insights from behavioral science to football. Our past posts have relied heavily on the work of Gary Becker, Daniel Kahneman & Amos Tversky, William F. Sharpe, Kahneman and Tversky again and Joseph Henrich. Today’s post is dedicated to 1990 Nobel Laureate Harry Markowitz who demonstrated that a portfolio of individually risky assets can collectively carry less risk than any of its underlying constituents, even when adjusted for its prospective return.

View fullsize

The above chart shows the three commonly cited high pick successes, and the RB-by-committee groups of the two Super Bowl teams. The “Draft Capital” column dispenses with the archaic Jimmy Johnson scale, instead using Dr. Michael Lopez’s blended draft curve which improves on prior efforts by not only paying attention to expected/modal outcomes, but also giving weight to the probability of drafting a superstar (i.e. the right tail of the distribution). PHI and NE expended between 1/4 and 1/7 the draft resources for their running backs as JAX, DAL and LAR. Though PHI and NE paid relatively high 2017 cap numbers, they locked up minimal resources over the long term (i.e. they could cut bait in 2018). The “gty” column shows guarantees over the entire contracts of those players (Sproles’ and Blount’s initial guarantees for PHI, Gillislee’s, Burkhead’s and White’s for NE).

The advantage of the portfolio approach is: you can be wrong, and still have success. Donnel Pumphrey is not good at football and Darren Sproles was lost for the season. Gillislee, Burkhead and White did not cover themselves in glory in 2017. The portfolio approach diversifies you against injury, suspension or disappointing play. Yet, each portfolio achieved similar yards/attempt and total yards as the 3 high draft picks, for less overall guarantee / draft capital. As a team, NE and PHI ranked 1st and 8th in offensive DVOA, respectively (the Eagles won the Super Bowl). Also, note that NE’s total 2017 expenditure, while high, was less than Le’Veon Bell’s cap number. JAX additionally paid $6MM in 2017 for Chris Ivory, who offered minimal return for this expenditure. As Harry Markowitz showed, the portfolio approach offers something vanishingly rare in economics: a free lunch. A properly constructed portfolio lowers risk, without sacrificing expected return. (Though running backs are risky, they are independently risky. Idiosyncratic risk is diversifiable.)

In summary, Dave Gettleman in his press conference constructed a straw man. The case for positional value does not rest on running back longevity. Instead the TL;DR argument is as follows:

• Using a high draft pick on a running back is a bad bet. At best, you expend draft capital in order to pay a guaranteed contract at a market equivalent price for a good player. At worst, you overpay twice: in draft capital and guaranteed salary for a bad player.
• Drafting a running back is risky.
• By assembling a portfolio of RBs, one can achieve similar performance to drafting a star, while diversifying risk, and saving draft / guarantee capital to deploy elsewhere.
• Your mother was right about eggs and baskets.

The above argument relies upon the prior work of a number of individuals who sit behind keyboards, all of whom have advanced degrees in a quantitative field such as economics, and none of whom have played a snap of professional football. Gettleman, a #TrueFootballMan, will confidently dismiss this argument, regardless of its merit, due to its provenance. Eagles fans should pray he never gets fired, and lives forever.

Nick Foles Is The Playoff GOAT

The following is a guest post by @sunset_shazz.

Nick Foles is a high-variance quarterback. His performance ricochets from abysmal to sublime with such frequency that he made me re-adjust my chart axis, twice. And yet: including the 2013 loss to the Saints (in which he engineered a comeback from a 13-point deficit and left the field with the lead) his postseason play has been consistently excellent. There have been 93 quarterbacks since the 1970 merger who have played at least 4 playoff games. Of these, Foles ranks 1st in completion percentage and 2nd in Adjusted Net Yards / Attempt (ANY/A).

Obviously, this is not statistically dispositive. Nothing about playoff analysis is. Mark Messier and Reggie Jackson’s playoff performances comprised a mere fraction of their total careers, yet their knack for elevating their game on the biggest stage is what made them memorable. One way to think about the playoffs: there is a tide in the affairs of men, which, taken at the flood, leads on to fortune. As I will show, Foles has taken the tide at the flood in historic fashion.

Note, from the chart above, that the fewer games played, the greater variance in ANY/A between individual players. But what about each player’s game-by-game variance? I measured the standard deviation of each player’s game ANY/A, and scaled this by his mean ANY/A, thus constructing a coefficient of variation.

Of all 93 QBs in the sample, Foles has been the 4th most consistent (i.e. has the 4th lowest variation). Moreover, he has the lowest variation of the 16 QBs who have only played 4 games.

Perhaps Foles has benefitted from playing in a QB-friendly era? I compared each QB’s game ANY/A to the league average for the year in which that game was played. One can then plot mean Relative ANY/A against the coefficient of variation:

Foles has the 5th highest Relative ANY/A in addition to having the 4th lowest variation. One way to think about the above graph is to imagine an “efficient frontier” on the upper left quadrant. When considering similar efficient frontiers in the context of financial economics, Nobel Laureate William F. Sharpe constructed a “Sharpe ratio” which compares a fund manager’s relative return (e.g. versus an index) to the standard deviation of the fund’s return.

I similarly devised a playoff QB Sharpe Ratio, which is each QB’s mean Relative ANY/A divided by the standard deviation of his game ANY/A. Think of it as one number which captures both efficiency and consistency of play. The following table shows the top 10 playoff QB Sharpe Ratios since the merger:

All 10 of these quarterbacks played in a Super Bowl, and all but two of them were champions. Only Bengals starter Ken Anderson and Bills backup Frank Reich did not win the season’s final game. (Reich, of course, will receive a ring as Offensive Coordinator of the 2017 Super Bowl champions.)

By this metric, Foles will have to settle for second place out of 93 playoff QBs. The Raiders’ Ken Stabler, who played in 13 playoff games between the 1971 and 1979 seasons, passed for 3.08 ANY/A above average (3rd) and had the 8th lowest coefficient of variation in the sample. Combining efficiency and consistency, he is the greatest playoff quarterback of all time. Here are the rankings of some other notable QBs, and Eli Manning:

Obviously, I’m not suggesting Foles is better than any of those quarterbacks (except Eli; he’s indisputably better than Eli, it’s not even close). However, in the inherently limited sample that consists of the playoffs, Foles has performed at a historically great level, in terms of both efficiency and consistency. Also, he can catch.

The Kids Are Alright

The following is a guest post by @sunset_shazz.

Carson Wentz’s start to the 2017 season has garnered national plaudits for his stewardship of the Eagles’ league-leading offense. But it being 2017, there lurks a coterie of skeptics who claim his underlying ability is “horrendous” like Blake Bortles or merely pedestrian like Andy Dalton. Even more emphatically, poor Jared Goff was confidently pronounced a bust after one season.

Is it fair to judge a quarterback solely on his rookie year? What about after the first nine weeks of his second season in the league? And how might one systematically evaluate a developing quarterback, relative to historical data?

Let us consider some advanced metrics that are used to evaluate quarterbacks:

• Adjusted Net Yards / Attempt (ANY/A) was developed by the great Chase Stuart, and accounts for sack yards, while providing a bonus for touchdowns and a penalty for interceptions. Both Stuart and Topher Doll have shown that ANY/A predicts wins. Danny Tuccitto has brilliantly used confirmatory factor analysis to show that ANY/A is a stable indicator of QB quality.
• Defense-adjusted Value over Average (DVOA), the brainchild of Aaron Schatz at Football Outsiders, is a success-based, opponent-adjusted per-play efficiency metric intended to both correlate with non-opponent adjusted wins (descriptive) and to predict future opponent-adjusted wins.
• Defense-adjusted Yards above Replacement (DYAR) uses similar success-rate inputs to DVOA, in order to compute an aggregate value for a player (combining volume and efficiency).
• Total QBR is ESPN Stats & Information’s proprietary efficiency metric that combines both passing and running contributions, adjusted for game situation, with charting to assign responsibility to a quarterback’s receivers and blockers.

Through nine weeks, the 2017 sophomore class is playing at an extraordinarily high level, as measured by each of these advanced stats:

View fullsize

Please note that nothing herein intends to argue for any of these quarterbacks to the detriment of the others. Though the data presented above is insufficiently precise to draw ordinal rankings, it is unequivocal:

Wentz is good. Goff is good. Prescott is good. All three of these things can simultaneously be true, pace internet trolls.

Some epistemic humility is in order: the first-nine-week sample size is obviously noisy, with varying degrees of luck, opponent quality, team injuries, coaching quality and supporting casts influencing the statistical performance of each QB. Danny Tuccitto warns us that ANY/A stabilizes at 326 dropbacks, and even at that sample size, 50% of the observation represents randomness/luck. Nonetheless, the broad takeaway should be that each sophomore QB has thus far performed at a top-quartile level, judged by a variety of different metrics. Is this good? And how confident can we be that such performance will continue?

Recently, Chase Stuart noted that three sophomores from the same class have not played this well since at least the NFL-AFL merger. Though ANY/A is less context-specific than the other measures, it has the advantage of being transparent and easy to calculate, permitting historical analysis. Stuart compared the first 8 weeks of 2017 for Goff, Prescott and Wentz to full seasons of prior 2nd year QBs. Comparing partial to full seasons isn’t quite neutral, due to the disparity in number of games sampled; we should expect some mean reversion of our reference QBs as sample size increases. Using pro-football-reference’s excellent query engine, I examined the first 9 weeks for each sophomore quarterback from 1999 through 2017. Historical comparisons need to be adjusted for era, due to the enormous change in average NFL passing efficiency over time. To account for this, I divided each quarterback’s ANY/A by the league average for that year. [1]

Top ANY/A vs Average since 1999, sophomore QBs, weeks 1-9

View fullsize

The 76 QB sample set in this study is itself a product of survivorship bias: only those QBs who were successful enough to throw 100 passes in the first 9 weeks of their second year in the league are included. On the other side of the distribution, successful QBs who rode the pine for their first few years (like Aaron Rodgers, Tony Romo or Philip Rivers) are not in this sample. The average age of the sample is 24, similar to our reference QBs.

The three 2017 sophomores are, as Stuart observed, performing extraordinarily well relative to their peer set (all are in the top quartile of the sample). Relative to their era, they are passing with greater efficiency than Tom Brady, Drew Brees, Matt Ryan or Andrew Luck did in their second seasons.

You will also note that the top ranked sophomore QBs include many future hits (Big Ben, Kurt Warner, P. Manning) and a few notable misses (Nick Foles, Derek Anderson). The last column I included is the Career Approximate Value (CAV), which is a (very) rough method developed by Doug Drinen that puts a single number on a player’s total career, encompassing both longevity and performance.

Below, I plotted log Career Approximate Value against ANY/A relative to league average for the first 9 weeks for second year QBs from 1999-2015 (I excluded QBs from 2016-2017 because recent QBs have not yet had sufficient time to accumulate CAV points).

View fullsize

The positive relationship shown above indicates that the first 9 weeks of a sophomore season predicts 37% of a QB’s future CAV. Do note that the correlation is sensitive to a few outliers. The odious Ryan Leaf and Akili Smith are on the bottom left, whereas Foles and Anderson are on the bottom right. I don’t want to ascribe an illusion of precision to this rough analysis – don’t fixate on the exact R-squared number, or the model coefficients. Both sample size and the extremely imprecise nature of CAV make me hesitant to draw definitive conclusions from the data. What is interesting to me is that the same plot using a QB’s full rookie season yields an R-squared of 0.224 – in other words, the first 9 weeks of a QB’s sophomore season tells you roughly 70% more about his future career than his entire rookie season does. Extending this analysis to full seasons since 1970, the R-squared is 0.083 and 0.2348 for rookie and sophomore years, respectively (n=155 & 204). My interpretation of this data: though rookie and second year passing efficiency predict only a small fraction of a quarterback’s career value, the sophomore year deserves 2.8x as much weight as the rookie year, in terms of confidence about predictive power. Rookie performance, in particular, is extremely noisy. One would have been wise to heavily discount Troy Aikman, Donovan McNabb and Terry Bradshaw’s dreadful rookie seasons. Rams fans should take note.

Relatedly, I didn’t find any predictive power when measuring the degree of era-adjusted-ANY/A improvement from rookie to sophomore season. This echoes Vincent Verhei’s study of second year improvement using DVOA. In hypothesis testing, a negative result can be an interesting result.

Quantitative analysis is not the only tool in an NFL researcher’s kit. Film study (though not my sphere of competence) is also valuable. Though Nick Foles had a magical sophomore season, the film showed reason for concern, as my friend Derek Sarley noted. I don’t personally see similar issues with Wentz – both his pre-snap adjustments and post-snap play appear to pass the “eye test”. No, he’s not perfect. Yes, he has flaws he needs to address. But so do all second year quarterbacks.

Moreover, our penchant for treating quarterbacks as static vessels of talent/ability shortchanges the importance of coaching and development. The installation of a new coaching regime in Los Angeles appears to be an interesting natural experiment, in terms of Goff’s maturation. Similarly, we can view Ezekiel Elliott’s probable(?) suspension as an instrumental variable when evaluating Prescott.

All inductive statements are, by their very nature, revisable. We don’t know the future; we can only use informed judgment to hazard a prediction. The false-positive rate for the top 20 QBs in table 2 above is 25% by my count [2], so let’s take that as the “base rate” of failure for the 2016 Sophomore QBs. It is therefore reasonable to expect that two – perhaps all three – of the 2016 sophomores will enjoy successful careers as NFL starters.

Finally, in these impatient times, let us remind ourselves that transcendent quarterbacks do not emerge, fully formed, from the forehead of Zeus. Each of these young, relatively inexperienced quarterbacks is playing the most technically and cognitively demanding position in sports at a very high level. Adjusted for experience and era, their achievements are even more astounding. The evidence suggests that the future of quarterback play is bright. Football fans, rejoice.

Thanks to Eagles fan / Data Scientist Sean J. Taylor for his insightful discussion on methodology. Any errors are mine alone.

[1] PFR’s partial season engine shows results from 1999 onward. Full season results go back before the merger, and also generate an era-adjusted ANY/A+ which uses a “Z-score” methodology, expressed in standard deviations above or below the population mean. My method is less sophisticated, though nonetheless robust.

[2] I excluded the reference QBs, as well as Marcus Mariota.

@sunset_shazz is an Eagles fan who lives in Marin County, California. He previously wrote about 4th down decisions.

Time To Clear The Air

The following is a guest post by @sunset_shazz.

This is a wonderful time to be an Eagles fan. Jim Schwartz’s Attack Nine defense is rapidly exorcizing the ghost of Juan Castillo. Doug Pederson has rejuvenated an offense that had become stale and predictable under Chip Kelly. And, of course, rookie quarterback Carson Wentz is turning heads across the league, not to mention in the oval office.

Eagles fans, unexpectedly blessed with success, look to the poet Browning to give voice to their collective sentiment:

The lark's on the wing; 
The snail's on the thorn: 
God's in His heaven— 
All's right with the world!

But wait. From his perch at the indispensable Football Outsiders, Scott Kacsmar has some discomfiting news: both Wentz and Cowboys rookie QB Dak Prescott are mere dink and dunkers, with lower than average air yards per attempt (defined as the average distance a football is thrown beyond the line of scrimmage). A low score on this metric is undesirable, in Kacsmar’s view.

The inimitable Jimmy Kempski responded to Kacsmar’s initial claim with a sardonic video rewind post, prompting Kacsmar, in an entertainingly vitriolic rant, to frame this argument as a contest between enlightened, statistically rigorous analysts on one side and straw-manning “numbers are for nerds” egg avatars on the other.

I don’t believe that view is correct.

As Brian Burke has explained:

A statistic that both correlates with winning and correlates with itself would be a reliable predictor of future wins.

First, you want your in-sample measure to have some predictive power in estimating out-of-sample future wins, because, hello, you play to win the game. Second, you want a metric to have some degree of statistical persistence over time, in order to be confident you are measuring a signal (in this case, an attribute of quality quarterbacking) rather than mere noise.

Regarding the latter, Kacsmar notes that in 2015, the correlation between air yards in the first three weeks of the year and the air yards for the entire season was 0.80. Well, that doesn’t seem quite fair, does it? After all, what we really care about is the correlation between the first 3 weeks of the season and the ensuing 14 weeks. Using his dataset, and using the Spearman rank correlation estimator rather than a standard Pearson estimator, which in this case would be considered less robust, I found that the correlation between the first 3 weeks and ensuing 14 weeks last year was 0.60. That’s pretty good, as far as football statistics go. However, do note that within a season a number of other factors surrounding the quarterback are, for the most part, held relatively constant: coaching scheme, strength of running game, defensive strength, etc.

When Chase Stuart examined the persistence of the Air Yards metric from year to year, he found that between 2006 and 2012 for 100 qualifying QBs the correlation between Year N and Year N+1 for Air Yards was 0.34. Both completion percentage and yards/attempt were “stickier” with N to N+1 correlations of 0.51.

Kacsmar, in his FO piece, assembles a smaller dataset (than Stuart, above) which he judges to be salient:

I gathered that yearly data on 21 quarterbacks with at least four years of starting experience, all of whom are still active starters this year except for the retired Peyton Manning. The following table shows their average air yards by year for the period of 2006 to 2015.

The first rule of Analytics Club is to plot your data, so I plotted Kacsmar’s data into a time series chart, in order to visualize the range and variability of the attribute, segregated by quarterback, over time:

Taking Kacsmar’s dataset (which, it is important to note, uses 21 quarterbacks who have experienced some career longevity rather than Stuart’s more comprehensive analysis of 100 QBs), and running a similar autocorrelative N to N+1 analysis, I found that the year-to-year correlation was 0.40. My friend, real-life data scientist Dr. Sean J. Taylor, was generous enough to both replicate my work and provide me with a scatterplot, complete with line of best fit and confidence interval shading:

Chart courtesy Sean J. Taylor

The autocorrelation statistic, the scatterplot and time series visuals each show the same thing: we are measuring mostly noise, with a faintly detectable QB signal. The attributes I mentioned before—scheme, effectiveness of the running game, defensive efficiency which affects game script—are all likely to change the calculus of decision-making with regard to throwing shallow or deep.

In fact, Kacsmar himself gives us a good reason to doubt the validity of Air Yards in capturing an attribute of QB quality: it doesn’t improve as a player gains more experience. Quarterbacks, like all athletes, typically experience an age curve, reflecting both athletic maturation and decline, as well as the steep learning curve imposed by formidable NFL defenses. Chase Stuart has shown that the age curve for NFL quarterbacks is pronounced. The absence of an “age/experience curve” for Air Yards is yet another red flag.

Air Yards doesn’t appear to measure a persistent quarterback attribute over time, particularly when compared with a conventional statistic such as completion percentage or advanced statistics such as Adjusted Net Yards / Attempt (ANY/A, for which Danny Tuccitto brilliantly used confirmatory factor analysis to verify its validity) or Defensive Yards Above Replacement (DYAR, rigorously developed and tested by Aaron Schatz).

But does it predict wins?

My general model of the production function of football is as follows: runs and passes are inputs; completions and first downs are intermediate goods; points are outputs. Success rate metrics such as Defensive-Adjusted Value Over Average (DVOA), DYAR, and ANY/A are all measures of intermediate goods which are of interest to the analyst because they tend to reliably convert to points. And as Chip reminds us, if you (f__king) score points you are more likely to win. 

Chart courtesy Sean J. Taylor

The scatterplot above shows the relationship between a QB’s average air yards over a season and the points scored by his team over that season. There is no statistically significant relationship between the two measures. Contrast this with ANY/A, which correlates 0.55 with wins. Or DYAR & DVOA, whose parameters were specified in order to predict future wins.

Kacsmar has been careful to note that he isn’t an advocate of maximizing Air Yards; he thinks middle is best. He elaborates in his FO piece:

Generally, air yards are a stat where you don't want to rank at the bottom, because that is where many ineffective passers dwell, including Blaine Gabbert. That preference for short throws often extends to crucial downs, which is why these quarterbacks tend to do poorly in ALEX and attacking the sticks. However, it is not preferable to rank at the very top in air yards either, because that is how "screw it, I'm going deep" players such as Michael Vick, Tim Tebow, Vince Young and Rex Grossman have earned their reputation as inefficient passers.

His claim, if I have understood it correctly, is that quarterbacks at the tails of the distribution are less likely to be successful in future. Our scatterplot above doesn’t show any relationship between the middle of the distribution and success, measured by points scored. But could Kacsmar’s anecdotal observation that “middle is best” be a mere artifact of sampling? If successful quarterbacks have longer careers, the law of large numbers dictates that they will, by mere virtue of larger samples, be less prone to the extremes in Air Yards. Taking a separate dataset evaluating quarterback air yards between 1992 and 2012, and plotting those against passes thrown, one arrives at the following:

You can see that the more passes a given quarterback throws, the less variance he exhibits with respect to his peer cohort. This needs to be examined further, in my view. I admit that I am not familiar with the nuances surrounding various measures of air yards (various observers have different estimates), but a longer, broader dataset would be desirable to plot air yards versus attempts. We don’t want to fall prey to the famous Bill and Melinda Gates Foundation misstep where it was initially claimed that small schools are consistently among the best performing schools, when it was merely the case that small schools experience more variance than larger schools, and therefore disproportionately comprise the tails of the distribution.

Here is the plot of the fourth-grade math scores versus number of students in the school:

The prior two sections showed that Air Yards as a measure is neither statistically persistent nor predictive of success, in terms of points scored. I did mention some alternative, robust metrics, two of which are generated by Football Outsiders. As of Week 3, FO has not applied opponent adjustments to their measures. On a raw Value Over Average and Yards Above Replacement measure, these young QBs have performed in the top quartile over the first 3 games.

Looking merely in the rearview mirror, without making any judgments about the future, they appear to have performed well.

Another measure I have mentioned, Adjusted Net Yards / Attempt (the “adjustment” gives a bonus for touchdowns and a penalty for interceptions, and the “net” deducts sack yards) is a persistent, predictive measure. With a hat tip to the excellent Derek Sarley, I prefer to plot this against completion %, to show both efficiency and consistency of per-play execution (weeks 1-3, minimum 46 attempts):

Once again, the rookies have played impressively: Wentz and Prescott are in the top quartile (4th and 8th, respectively) in ANY/A and the 2nd quartile (13th and 10th, respectively) in completion %.

As Bill Barnwell has noted, the statistics from 3 games tell us very little about how a QB will play in the future. A very small sample size disadvantages a purely statistical analysis; the comparative advantage shifts towards the film analyst. Ideally, one would combine both, but in this case, the stats aren’t meaningfully more robust than mere anecdotes. This is why I disagree with Kacsmar’s adversarial Michael Lewis-style “stats versus scouts” framing; the NFL stats on these two rookies don’t really tell you anything dispositive yet. From a purely Bayesian perspective, the eye test is just as likely as a mere three weeks of quantitative data to meaningfully update one’s priors. I have not yet enjoyed the privilege of watching Prescott, but I’ve seen every Wentz throw; moreover, I’ve seen astute film analysts such as Greg Cosell, Fran Duffy, Jimmy Kempski and Ryan from ChipWagon break his film down. Lastly, as Brent from EaglesRewind notes, one’s priors should be heavily influenced by draft position, which was the NFL auction market’s initial “revealed preference” view of value.

As for me, I’m on the Wentz Wagon. Dan McQuade reasons persuasively that Eagles fans should enjoy this run, because life is fleeting. Memento mori, football fans.

TL;DR:

• The early results from the credible advanced statistics, meaning those that tend to be both persistent and predictive, are that Wentz and Prescott have played well in their first three games.
• Looking at the numbers alone, a three game stretch is insufficient to give us high confidence that such success will continue in future. 
• The Air Yards statistic is neither persistent nor predictive, and reflects the aesthetic tastes of one particular writer, rather than a desirable quarterback attribute.

Thanks to Sean J. Taylor for his methodological insight and scatterplot work. Any errors are mine alone.

@sunset_shazz is a Philadelphia Eagles fan who lives in Marin County, California. He previously wrote about Chip Kelly's Oregon bias and other topics, and contributed to the 2015 Eagles Almanac.

There Are No Shortcuts To The Super Bowl

Last March, Jeff Lurie told reporters he was was tired of waiting for the Eagles to be great. He’d seen sustained success, he’d been to a Super Bowl, and he’d watched his franchise post a solid 20-13 record (including a playoff loss) over the prior two seasons. It wasn’t enough.

“I’ve lived through a lot of division championships, a lot of playoff appearances, a lot of final four appearances, but our goal is we want to deliver a Super Bowl,” he said at the time. “And sometimes maybe I’m influenced by the notion of it’s very difficult to get from good to great, and you’ve got to take some serious looks at yourself when you want to try to make that step. It’s a gamble to go from good to great because you can go from good to mediocre with changes, but I decided it was important enough…”

On Wednesday, after firing Chip Kelly before the final game of the season, Lurie didn’t walk away from his words earlier in the year.

“I said, with Chip’s vision, it was an opportunity that he wanted to lead the way, to try to go from good to great,” he said. “In fact, I remember saying to all of you, there’s dangers in that, in terms of having two 10-6 seasons in a row, and when making significant changes, you can easily achieve mediocrity. I think it would be a shame not to try, but… that is the danger when you take a risk.”

I hope Lurie learned more than that, because his "strategy" was little more than a desperate hope. He handed over all power to Kelly, a moderately successful coach with zero experience running the intricacies of a NFL organization. 

The road to the Super Bowl is not paved with such gambles, with an outsider making a bunch of questionable bets that luckily pay off. You don’t win by cutting talent and building #culture. You don’t win with a rigid set of measurables that dictate player acquisition. You don’t win by ignoring critical positions and spending excessive guaranteed money on less important ones. You don’t win by overpaying for a subpar quarterback coming off two major knee injuries, or turning over half the roster in one offseason. That’s how you end up 7-9 in one of the worst divisions ever.

There are no shortcuts.

Remember Andy Reid’s binder? Reid came to Philadelphia with a detailed multi-year plan of how to build an organization. He arrived in 1999, overhauled the roster, brought in an impressive group of experienced coaches (including 6 eventual head coaches), drafted a quarterback in the first round, and went 5-11. In 2000 the team and young quarterback improved, overachieving to reach the playoffs. By 2001 they were one of the best teams in the conference. In 2002 and 2003 they missed the Super Bowl by inches. In 2004 they came minutes away from the trophy itself.

Lurie said he doesn’t want the middle steps, he just wants the Super Bowl, and he was willing to gamble to get there. But those middle steps are important. That’s how you build a champion—not overnight, but consistently, step by step.

The best teams in the NFL have the best talent. The best teams in the NFL have a top quarterback they drafted and groomed. The best teams in the NFL have smart, experienced coaches who adjust their schemes to the players they have. The best teams in the NFL have a front office structure that empowers multiple voices and balances scouting with analytics and financial understanding.

Going into 2016, the Eagles need to avoid the quick fix, or the allure of competing for the playoffs in year one of a new regime. That won't set the team up for long term success. (Plus, they'll likely be in the running anyway unless the putrid NFC East changes significantly in a year.)

The blueprint needs to be for a Super Bowl contender in 2018. Let's lay out what that looks like...

Front Office: Howie Roseman has a mixed reputation among fans and league sources, but he can succeed in the Joe Banner role. He has experience on the personnel side to pair with a firm grasp of the league's economics. Roseman does need to find a qualified general manager-type to run player personnel, someone less washed-up than Tom Donahoe, with more experience than Ed Marynowitz, who's not obvious idiot Ryan Grigson.

Coaching: A NFL team is a crazy thing to manage, and any head coach needs to be able to bring in the right people and command respect across the organization. He doesn't have to be a brilliant innovator on the cutting edge, but he does need to be flexible enough to adapt his schemes and techniques to get the most out of his players in each situation. The single most important skill set within that is the ability to find and develop a franchise quarterback, which is what makes a candidate like Adam Gase so attractive.

Quarterback: You cannot be a consistent Super Bowl contender until you have a quarterback. As such, you should exhaust every possible avenue to get one, especially the draft. I stand by my recipe for QB hunting laid out four years ago, on the eve of Andy Reid's firing:

Draft a quarterback early and late. Sign somebody in free agency. Trade for a promising backup. Rinse and repeat. You're never going to be able to compete for the Super Bowl until you find your one franchise guy. Might as well cycle through as many potentials as you can until you do. The financial cost of doing so is less than the opportunity cost of sitting pat with one player, [Bradford], who is statistically unlikely to ever become an elite quarterback.

In the Eagles' case, that means avoiding any multi-year guarantee to a still-unproven quantity like Sam Bradford, and perhaps letting him go entirely depending on the cost.

Roster Building: Outside of quarterback, which takes priority over everything else, the offensive line is next on the list of must-haves. Though I don't know many of the names, Jimmy Kempski's mock draft would thrill me based on the selection of two quarterbacks and three offensive linemen. Lane Johnson and Jason Kelce are the only long term building blocks you can count on there. Meanwhile, don't waste resources on the rest of the offensive skill positions, none of which will matter much until the offensive line and quarterback are fixed.

The Eagles defense needs more talent across the board, but it also likely needs a scheme that better takes advantage of the players in house. Kelly wanted a two-gapping 3-4 system, but it would be nice to see what Fletcher Cox and company could do in a one-gapping 4-3 instead.

Overall: Both the organization and its fans need patience. We were spoiled by Chip's quick turnaround, but where did that leave us? Let's plan for sustainability.

Read more: How The NFL Chewed Chip Kelly Up And Spit Him Back Out

Coach Chip Strikes Back, Or Something

It's amazing what a win over Dallas will do for your spirits. The Eagles went into the bye week at 3-4, reeling from a loss to Carolina that wasn't particularly close. Post-bye, post-victory over a hated rival, and they're 4-4, with the wind at their back, a rejuvenated team.

All it took was a few fourth down tries, a revived running game, and a kicker that finally made a clutch field goal longer than 50 yards. Suddenly it's "the biggest win of the Chip Kelly era" (which is really saying something about the Chip Kelly era, unfortunately).

Sam Bradford is fixed. He's "continuing to flip the script on his legion of detractors". Apparently, the main issue was footwork, with QB coach Ryan Day noticing a problem three weeks back. "The results have been evident," writes the Inquirer.

Have they?

Via XKCD

The funny thing about the win on Sunday night is that on paper, not a lot looks different from the losses earlier in the season. If you're going to give the Eagles credit for any improvement, it has to be in the run game, where they've reeled off four straight games with more than 150 rushing yards. Kelly also seems to be mixing up the play calling a bit better. 

But Bradford suddenly proving he's the guy of the future? This team, suddenly a legit playoff contender? Seems like wishful thinking.

Take Bradford on Sunday. His stats were solid: 25 for 36, 69% completion rate, 295 yards, 8.2 yards per attempt, 1 TD, 0 INTs. But the funny thing about those numbers is how much they were influenced by one great overtime drive. In regulation, his stats weren't quite so good: 20 for 31, 65% completion rate, 239 yards, 7.7 yards per attempt, 0 TD, 0 INT. That's arguably worse than Matt Cassel, who completed a higher percentage of passes for more yards and 3 TDs (plus one bad INT).

The point is not that Bradford's game-winning drive has no value. But just remember that he was one Caleb Sturgis screw up or one heads-tails controversy from being the same just-good-enough-to-lose Bradford we've seen all season. If Jordan Hicks tears his pectoral muscle before grabbing that pick six, the Eagles don't make it to overtime and there is no "Sam Bradford has fixed it" narrative worth telling.

Nor, really, is there one for the Eagles, who had two weeks to prepare and narrowly avoided falling to a Cowboys team that lost 5 straight games coming in and had posted just 91 passing yards against Seattle a week earlier. This is a 4-4 team that has played a bunch of bad teams in a bad division. Luckily, they play more going forward: their next three opponents are a combined 7-17.

Maybe this group has finally turned a corner, and will cruise to 7-4 before heading to New England. I hope so. But don't be surprised if this turnaround is a mirage, and inconsistency dooms the Eagles to underperform down the stretch. That would simply be par for the course this season.

How The NFL Chewed Chip Kelly Up And Spit Him Out

When Chip Kelly entered the NFL in 2013, many people derided his offense as a "college" or "gimmick" system, with its unconventional no huddle, quarterback read option, plus other aspects of the spread. Kelly pioneered packaged play concepts (those run-pass hybrids) and play call communication techniques like giant signs and coaches doing weird hand signals. Anyone who thought that wouldn't work was wrong. His offense rocked the league in year one, putting up big numbers and carrying a terrible defense to the playoffs.

Many Eagles fans, including myself, enjoyed a victory lap. We taunted the idiotic commentators who questioned Chip's revolution and basked in the glory of his high octane offense. He looked like perhaps the best coach in the NFL, someone we could count on for a tactical advantage for years to come.

But the dirty truth is that winning in the NFL is a result of many, many things, and scheme isn't that high on the list. Actually, scheme may be the least important—because it's the one anybody can copy or subvert. The offensive elements that Chip brought to the league are already being adopted widely. Bill Belichick and Tom Brady borrowed one-word play calls and no huddle from Chip years ago, and both concepts are now widely practiced by teams as old school as the New York Giants. The read option is overtaking the classic I-formation as the trusted standby for numerous teams with mobile quarterbacks. Packaged plays are everywhere. Even his prized sports science may soon be table stakes.

Meanwhile, Kelly has had to make compromises on his core tenets. As it turns out, quarterbacks are not fungible at the NFL level. You can't go from Dennis Dixon to Jeremiah Masoli to Darron Thomas, winning all the way. Chip has struggled to fit indecisive, immobile QBs like Nick Foles and Mark Sanchez and Sam Bradford into his offense, its purest form diluted by the slow feet and damning inconsistencies of each signal caller.

And NFL defensive coordinators are getting paid too. Gone is Chip's surprise factor. These days, the Eagles offense is more likely to be surprised by the defense than the other way around. (Oddly, Chip's offense seems to have shrunk rather than grown over the years.) Recently, for example, defensive line stunts designed by smart defensive coaches like Rob Marinelli and Dan Quinn have crippled Kelly's vaunted interior run game. 

Overall, the scheme playing field is being leveled. And as that advantage fades, other factors become much more important—talent being number one. Acquiring, developing, and retaining talent is 90% of what makes a good football team in the long term. Yes, the less-talented team can luck into a win for any one game. And luck and scheme and hard work can overcome the odds over the course of a season. But if you want to win consistently in a competitive league, you need more.

Just because you're a brilliant engineer doesn't make you a good salesman. Just because you can write doesn't mean you can add. Just because you're a star performer doesn't mean you can manage others. Yet in football, teams have a habit of assuming that their coach can do much more than coach.

Kelly is a smart coach, with a novel and mostly effective scheme. But he is a woeful manager of personnel. I wrote the offseason about the cascade effect of his poor decisions, and the list was long then. But it's getting longer. Sam Bradford looks like a timid check down artist. Byron Maxwell is doing a great Bradley Fletcher impression for $22 million guaranteed. The Evan Mathis cut was questionable from the start, but is appalling now with the current state of the offensive line—a unit whose problems have canceled out any positive effect of a shiny new stable of (expensive) running backs.

In college, Chip didn't have the constraints of the salary cap or the draft or free agency. He was never the best recruiter, but he did fine; from 2010-2012, his recruiting classes ranked top 15 in the country. (At most, Oregon played 3 games a season against opponents who recruited at that level, and went about .500 in those match-ups.) Every year he added another 20-25 new players to a scholarship roster of 85, a game day pool 60% larger than the NFL. If a recruit didn't work out, there was always someone else to take his place, and none of them ever got too old and slow or cost too much money.

Under those conditions, the scheme was just icing on the cake. Kelly maximized the talent advantage he already enjoyed, and thus he won most of the time. But in the NFL, talent margins are too thin to ever make scheme a permanent advantage. Defenders who weren't smart enough or fast enough to disrupt the Oregon offense never made it to the NFL, and the best players are rationed out to professional teams, keeping the league in a constant tumble of parity. Plus, coaches are, for the most part, smarter. They won't sit back and let you call the same three run plays over and over. Oh, and you actually need starting-caliber guards to execute any scheme.

Hopefully Kelly can make some rapid adjustments this week. He needs to add new wrinkles to his play calling and consider benching one or more of his under-performing linemen. He needs to prove that Bradford was worth the major investment the Eagles made for him. The team is only 0-2, and the rest of the NFC East is putrid. There's plenty of time to come back—if he starts right now.

Still, I worry that the NFL has caught up to Chip, and Chip hasn't caught up to the NFL. Opponents are all over his early scheme innovations. But Kelly never figured out the intricacies of running all the aspects of a professional football team, the job he aspired to and finally wrested from Howie Roseman this offseason. The NFL is a lot harder than it looks on Sunday, and maybe we're finally seeing that.

Five Myths of the 2015 Eagles Offseason

1. The offensive line won't miss Evan Mathis

Outside of Allen Barbre pushing Ryan Mathews into the end zone, the Eagles got some incredibly poor guard play last night. (See Ryan at Chipwagon for all the ugly.) Even Chip gave up on his signature inside zone runs because he didn't trust the inside blocking. Thank your lucky stars that the injury to Jason Peters (who looks pretty much the same as he did at the end of last year) wasn't serious.

Maybe because every time he called an inside run the Falcons threw Philly's guards to the side and blew up the play? https://t.co/ljs64qepqg

— Chris B. Brown (@smartfootball) September 15, 2015

2. 11 new starters won't take time to adjust

Including the guards, the Eagles replaced about half of their offensive and defensive starters this offseason. And despite their sterling offseason work (which was little more than a couple quarters of work against vanilla schemes), this team needs time to gel. Sam Bradford was out of sync with every wide receiver and tight end not named Jordan Matthews. The defense frequently looked confused presnap. While the second half was more promising, this looks like a team that may not put it all together in the first few weeks.

3. Byron Maxwell is a #1 shutdown corner, worth $22 million guaranteed.

@gonzoCSN pic.twitter.com/5PeckWCPAo

— Joseph Kornik (@joekornik) September 15, 2015

The Eagles paid Maxwell like he was a top five NFL cornerback. Hard to look at his performance last night and not think Cary Williams could have done just as poorly for a fraction of the cost. Julio Jones is one of the best receivers in the league, so any corner is going to struggle--but you don't get the the Super Bowl without getting by a few great wideouts. Oh, and old man Roddy White made it look easy against Maxwell too.

4. Nolan Carroll is a starting cornerback.

Chip and company praised Carroll to high heavens this offseason. Now we see he's the same player who could couldn't beat Bradley Fletcher last year. And if he's this bad, what about rookie and supposed instant contributor Eric Rowe? While we're at it, isn't everyone glad that the Eagles traded Brandon Boykin? It frees up playing time for future Pro Bowler Chris Maragos, who completely lost Roddy White in the end zone.

5. DeMeco Ryans isn't washed up.

He's done. 

Two Good Things: Walter Thurmond and Kiko Alonso

One handed. Diving backwards. In the end zone. Kiko Alonso, you gotta be kidding with this INT. #PHIvsATL http://t.co/U9ei4PenNl

— NFL (@NFL) September 14, 2015

Neither played a perfect game, but they were difference makers on a defense that generated little pass rush and couldn't cover. This looks like a boom-or-bust unit that can mostly stop the run but has the same problems on the back end as last year. If that's the case, they'll need turnovers to make up the difference (looking at you, Malcolm Jenkins -- spend some time with the JUGS machine).

One Wait-And-See: Sam Bradford

Bradford was on fire most of the second half, raising his final completion rate (69%). Overall, you'll take that line from him, and the progress he showed in his first regular season game in 23 months. But I'll be interested to see in the All-22 whether he had opportunities deep and didn't take them. You want quarterbacks to take what the defense gives them, but one major concern with Bradford is whether he's willing to throw it long. He ended on Monday night with just 6.46 yards per passing attempt, basically his career average.

Overall Thoughts: no, the sky isn't falling. It's one last minute road loss against a team that could be better than we thought. But if you were predicting Super Bowl for this Eagles group, consider that this first game was remarkably similar to the opener last year--when the team needed a few lucky breaks to win 10 games, and still didn't make the playoffs. There's a long way to go.