When Will NFL Coaches Stop Acting Like Sheep?

The following is a post by @sunset_shazz.

We have previously discussed in these pages the principal-agent problem, and how idiot principals (aka NFL owners) are to blame for suboptimal decision-making by agents (aka GMs and coaches). In an entertaining Twitter thread, some analytics nerds were discussing the use of RPOs and the implications for run/pass decision making. A turning point in the thread:

Brian Burke believes that coaches are optimizing for success rate. They want to move the chains and string together first downs to form drives. If you get nearly 5 yards per rush on average there’s no way you’ll convince a coach to pass.

— Josh Hermsmeyer (@friscojosh) May 18, 2019

Josh is citing this piece of descriptive analysis from Brian Burke back in 2014, where Burke showed (from a positive, as opposed to normative standpoint) how coaches tend to maximize the success rate of each individual play, which doesn’t necessarily map to game-level success (“just win, baby”).

I replied with my time-worn take that this suboptimal decision-making is the fault of the principals, who have set up bad incentive structures. Sean Domnick asks an excellent question: why did the coaches choose this particular strategy? It’s not as if the owners incentives are particularly clear in favoring per-play success rate:

Yeah, right now I'm just curious about why incentives lead to success rate optimization compared to other metrics.

— Sean Domnick (@sean_domnick) May 18, 2019

The hypothesis below is drawn from social science, and is speculative.

Decision theory is a very young field. My Decision Sciences professor in the nineties came from Xerox PARC and leaned heavily on Kahneman, Slovic and Tversky. But prior to formalized decision theory, Friedrich Hayek and Edmund Burke posited mechanisms by which traditional decision-making can evolve to be “good enough” over time. Conventional wisdom improves through an iterative, blind process. In the decision space, though traditional decision-making may be suboptimal, it can still be good enough to have adaptive success. Joseph Henrich has, using modern data collection methods, formalized how norms and traditions may evolve to be broadly efficient. Bill Walsh did not have access to modern data, software or computing power. But his analysis was good enough, and was dominant enough, to change how coaches stretch the field and emphasize short passes.

Yes, per-play success rate is suboptimal. But it is an evolutionarily stable strategy in that it is dominant, provided that nobody has implemented a better strategy. Moreover, to the extent that norm violators are punished (cf. Henrich) herding behavior is a rational response to the threat of being fired. Thus, a “good enough” strategy such as per-play-success-rate maximization will perpetuate because everybody is doing it.

1)     Maximizing success rate, though suboptimal for maximizing wins, has the advantage of being an easy, tractable heuristic that dominates over other, worse, decision rules.

2)     Over time, principals and agents will coalesce around this (arbitrary, though minimally successful) decision rule, which becomes self-perpetuating.

Is this equilibrium stable? It is until it isn’t. Just as Roger Bannister showed the world what is possible, it takes only one coach to show that sound evidence-based decision-making will dominate the conventional wisdom. We are at a moment in time when internet nerds are mining data to show the traditionalists there is a better way. My tentative prediction is that NFL coaching heuristics will change for the better, just as they have in the NBA and MLB.

Numbers Are No Substitute For Trust

The following is a guest post by @sunset_shazz.

The news of Cleveland head of personnel Sashi Brown’s dismissal was met with uncharacteristic emotion by the customarily sober Aaron Schatz. Evidently, some quarters of the analytics community regard Brown, his colleague Paul DePodesta and the rest of the Browns front office team as one of their own. Brown’s strategy consisted of trading high draft picks for more (albeit lower ranked) picks. The opportunity cost of this strategy was to pass on both Deshaun Watson and Carson Wentz.

Talent evaluation is hard. You will be wrong more often than you will be right. I won’t fault Brown for misevaluating quarterbacks, just as I don’t fault 32 teams for repeatedly passing on Tom Brady (even the Patriots passed six times before using their 7th highest pick on him). There is data that suggests the market for talent evaluation is efficient and that no team has a sustainable edge.

But it doesn’t follow that just because you aren’t able to out-evaluate your peers, you should always trade down. Higher draft picks have higher success rates. The idea that lower picks might be undervalued dates from a classic 2005 paper by Cade Massey and newly-minted Nobel Laureate Richard Thaler. Massey-Thaler observed that NFL teams were overvaluing high picks relative to “the surplus value of drafted players, that is the value they provide to the teams less the compensation they are paid.” (Emphasis in the original.)

Two things have changed since 2005. First, the 2011 Collective Bargaining Agreement repriced the rookie wage scale, increasing the “surplus value” of higher picks relative to lower picks. [1] Secondly, the market learned and absorbed the Massey-Thaler result. We know from other domains that market inefficiencies almost always disappear as soon as they are found. Sashi Brown’s claim that talent markets are efficient – yet the market for picks is systematically inefficient – is an extraordinary claim, which demands extraordinary evidence. More likely, the market has moved from partial to general equilibrium. Football has no farm system; the strategy of hoarding picks is constrained by the 63-man roster and practice squad. Successful teams with quietly robust analytics departments (e.g. the Eagles and Patriots) trade both up and down, depending on the situation; constrained optimization is complicated. Moreover, as Brian argued in 2011, in order to fill the most critical position on an NFL team, the data shows you need to pick early.

But the real reason that Jimmy Haslam absolutely had to fire Sashi Brown is simple: his staff failed to execute a trade which was negotiated in good faith. Whether it exists under a buttonwood tree in lower Manhattan or in a coffee house in eighteenth century London, all markets are governed by rules, traditions and norms. As an example, the market for trading picks, which often occurs while teams are on the draft clock, is governed by trust and verbal agreements (not legal documents).

As Joseph Henrich notes in his brilliant anthropological survey The Secret of Our Success, pro-social norms are enforced in small communities through punishment, such as ostracism. In any community of traders, walking away from a duly negotiated agreement is a flagrant violation, a taboo. I will never forget how at the beginning of my career the head of our firm reacted to a similar situation: “We will never do business with those scumbags again, and we will make sure everybody knows how they have behaved.”

Punishing norm violators is not irrational. A trust-based community of traders is a fragile equilibrium. Erosion of trust can cause permanent, deadweight losses; thus pro-social norm enforcement evolves over time, and is Pareto efficient.

If Haslam had not fired Brown, some or all of the other 31 teams would have punished the norm violator by refusing to trade with Cleveland. This is both individually rational (trading with a deadbeat counterparty is risky) and collectively rational (pro-social norms promote gains from trade). Brown’s reputation as a counterparty was permanently impaired, thus compromising his ability to discharge his duties, and leaving Haslam with no choice but to clean house.

[1] Brian Burke found continued persistence of a second day surplus value anomaly, though it’s still early in terms of data, and as he notes, the replication model is sensitive to key methodological assumptions. His results cover the player market, but not the draft pick market.

Ranking The 2013 NFL Kickers (Or Why Alex Henery Doesn't Deserve Your Support)

Everyone knows that NFL kickers are getting better. The numbers make this obvious. We saw an all-time high for field goal accuracy in 2013: over 86%. And that's despite many more long attempts. Field goal tries from over 50 yards have increased 57% in the last decade.

When ranking kickers, we have to take into account the new landscape. For an example, just look close to home. After his first three seasons, Alex Henery is tied for fourth all time in overall field goal accuracy among players with at least 80 attempts. But if you check out the top 20 on that list, 17 of them played in the last two years. This is like the late 1990s in baseball -- everyone's breaking records.

More to the point, we can't quote field goal accuracy without taking distance into account. As Jimmy Kempski noted, Henery doesn't look nearly as good when you compare him to his peers. Among full time kickers since 2011, Henery is tied for the fewest attempts over 50 yards (5). He also has the shortest "long field goal" (51). If everyone else is taking longer attempts and making them, Henery's value is significantly lower than his overall accuracy would suggest.

Last offseason, I took a deep dive into Henery's value, comparing him to other kickers in 2012. By generating an estimated value for every kick, we could plot how much each kicker actually contributed to his team as compared to the league standard. The results put Henery just a hair above average, with the noted caveat of having not a single make over 50 yards.

This year I wanted to go deeper. To start, I pulled all 2013 field goal attempts and plotted them out. While upgrading from last year's 5-yard buckets required a little smoothing, the granular data this year should be more exact:

View fullsize

Cool, right? As you can see, kicks from in close are automatic and everything up to about 32 yards out is a 90% success rate or higher. But perhaps even more impressive, kickers in 2013 maintained over 80% accuracy up to 45 yards out -- and 70% all the way to 55 yard attempts. The data gets meager from there, but the graph is proof of how great NFL kickers have become.

Anyway, with this data, it becomes fairly easy to generate an estimated point value for every field goal attempt. For example, a 35 yard try is worth 2.63 points while a 50 yarder is worth 2.1 points. If a kicker makes a kick from that distance, he adds points to his team above the expected value. Missing, he loses his team the expected points. (The only major caveat here is weather conditions. Certainly a field goal into strong wind would have a lower expected value than one in a dome.)

This year, we'll also add kickoff analysis. I spent some time playing with large data sets of kickoffs and ultimately was disappointed by the many problems in the NFL's play by play data. So at least until Football Outsiders releases its always-awesome game charting this summer, I can't work with the type of granular data on field goals above. The next best thing is Pro Football Focus' kickoff stats. They track distance, touchbacks, and average yard line start. I adjusted that data to eliminate the effect of onside kicks, but there are other variables (including weather) which still exist outside the scope of the investigation.

For kickoffs, the goal is to reach a similar expected value score. With help from Derek Sarley, we calculated a quick and dirty correlation between touchbacks and starting drive position of .67, suggesting that kickers have about two-thirds of the responsibility on kickoffs. Using that, we can multiply each kicker's total drive start yardage difference from the league average by .67, then by the approximately 0.06 expected points each extra yard is worth. Again, it's not a perfect score, but it will give us a good estimate.

Got all that? Click on the table below to enlarge the full results:

View fullsize

Hopefully the table isn't too confusing. Let's break it down. The left hand side is kickoff data, and you can see adjusted kickoff distance and drive start. From there you can calculate points generated in relation to the average kicker (KICKOFF PTS). On the right side are field goals, first the basic data then the estimated values based on the calculations above, leading to the FG PTS above or below average. Finally, TOTAL PTS combines the two scores and ranks all kickers (including those who only kickoff or attempt field goals).

The best kickers of 2013 did well on both metrics. Dan Bailey of the Cowboys, for example, helped force opponents to start their drives 1.9 yards further back than average. Over the course of the season, that was worth just over a touchdown in field position. On field goals, he made over 93% of his attempts, when the average kicker would have only made 84.5%. That was another 8 points he generated for Dallas, bringing Bailey's total points over average to a league-leading 15.1. If we can translate that using Brian Burke's data at Advanced NFL Stats, Bailey was worth about as much in expected points added as the 25th-ranked quarterback.

On the other end of the spectrum, Henery ranked third-worst in the NFL last year. He lost the Eagles approximately 10.8 points, split about equally between kickoffs and field goals. Henery was near the bottom of the pack in kickoff distance and average drive start, losing 1.6 yards per kickoff. Meanwhile, his 80% field goal accuracy should have been better, given he took mostly shorter kicks. The league average accuracy for his attempts would have been 85.8%.

Though Henery has some ardent supporters, it's tough to find any silver lining for him in these numbers. Only two kickers in this sample had worse results: a historically-bad rookie and a veteran who was fired in December. Maybe Henery can improve this offseason and post a better effort in 2014. But I wouldn't count on it. At the very least, the Eagles must bring in competition for what is clearly one of their weakest positions.

How Chip Kelly Uses The Run To Post Near-Record 3rd Down Efficiency

Here's a stat you probably didn't know: this year's Eagles offense is on pace to register one of the smallest three-and-out percentages in recent history. Through six games, Football Outsiders clocks the Eagles as having three-and-outs on only 10.8% of all drives. If they can keep that pace up, it would tie them for second-best among all teams over the last 16 years (the entire period FO has accounted for this stat).

There are a number of things that help Chip Kelly's offense avoid the dreaded three-and-out. Part is just being tremendously efficient across the board. The historical team the Eagles are tied with right now is the 2001 St. Louis Rams, an offensive juggernaut that rode to a 14-2 record and Super Bowl appearance. Like those Rams, the 2013 Eagles rank second overall in offensive DVOA, with both passing and rushing offenses in the top 5. When you register a full 1.2 yards more per play than the average NFL offense, you're going to have fewer dead drives.

But that's not all. The number two ranked team in three-and-out percentage is the Carolina Panthers, and their offense is middling. There's another key to Kelly's success in this area, and it's relatively simple: he runs the ball more on third down and short.

On third down with 1 or 2 yards to go, NFL teams run just slightly more than they pass this year: 52%. This is odd, because running is significantly more effective. 66% of all runs from this distance convert a first down, while only 51% of passes do. Such odds would be even more favorable for the Eagles offense. The offensive line has looked good so far (although only 3.81 adjusted line yards), but more importantly LeSean McCoy is a beast. Kelly clearly knows this. He has called run plays in those situations 84% of the time, far more than any other team -- and has an 81% success rate to show for them.

The tendency extends to third downs from 3 to 5 yards out as well. For most NFL offenses, this is a straight passing down. A quarter of teams have yet to call a run in this situation, and the overall rush rate is a measly 10%. This seems to be a rational decision for most teams. At this run-pass ratio, the success rate for both has hit an equilibrium of about 47%.

Kelly, however, has called runs on 35% of these plays. That percentage is exceeded by only one other team: the Panthers, who we mentioned above. Both teams have succeeded with this strategy. The Panthers, with Cam Newton and a couple of successful backs, have been successful on 60% of their runs. The Eagles have converted a first down on over 70% of such plays.

This isn't new territory. Brian Burke wrote about how NFL teams should run more on third and short back in 2008:

On 3rd and 1, offenses scored an average of 2.38 points if they ran, and an average of 2.24 points if they passed. The difference of 0.14 points is remarkably close to the 0.12-point theoretical estimate calculated above.

The differences are even bigger for 3rd and 2 and 3rd and 3 situations. The advantage of running is 0.45 and 0.31 expected points respectively—convincing evidence that offenses should be running more often on 3rd and short.

With the numbers -- and an elite back -- to support him, Kelly's play calling has been spot on. The Eagles have converted 74% of all third downs from 1-5 yards. That's the highest rate in the league, and is a key factor keeping their up tempo drives rolling.

Chip Stew: Changing the Way the NFL Thinks About Going for Two

For the latest entry in our Chip Stew series, let's look at the coach's aggressive approach to the game. Not only does he routinely go for it on fourth down (a move that's heavily backed by statistics), but Kelly also tries for two point-conversions far more than other coaches. Here's Michael David Smith:

In five of Oregon’s 13 games last season, the Ducks tried for a two-point conversion after scoring their first touchdown. Three of those five two-point tries were successful, and two were unsuccessful. When the Ducks didn’t go for two, they often sent out their field goal/PAT unit and had them line up in a trick-play formation, shifting into the standard kick formation only after seeing if the defense appeared ready to stop the play they planned to run.

Kelly’s approach to two-point conversions — trying them after taking a 6-0 lead, rather than only trying them late in the game when the “chart” says they should — is unheard of in the NFL. According to Aaron Schatz of FootballOutsiders.com, the last time a team tried for a two-point conversion in the first quarter (other than on bad snaps or fumbled holds when the team initially planned to kick the extra point) was in 1998. And even that play was done in an attempt to chase points by a team that had just scored a touchdown to narrow the deficit from 14-6 to 14-12 and wanted to tie the game with a two-point conversion.

The math behind attempting a two-point conversation is more questionable than going for it on fourth down. Brian Burke analyzed it a couple years ago. His numbers showed a potential advantage, especially if you run the ball (or execute a quarterback run), but it's not clear cut. Kelly seems to have confidence it can work, at least in college. I can't wait to see if he will bring that attitude to the NFL.

Won't Be Outsiders Forever

Great Sports Illustrated profile by Albert Chen of some of the brightest stats minds (and good guys) in the NFL. Check it out:

In recent years NFL teams started to get smarter. M.B.A. graduates from elite universities began infiltrating football front offices as they had baseball's in the mid-2000s, but the brain power was devoted to salary-cap management and personnel decisions. There was one area in which teams remained stuck in flat-earth thinking: game strategy.Burke believed this would change—that in an information age in which advanced stats had the power to predict a presidential election to the decimal point, new math would be impossible for the Football Men to ignore. He believed the revolution would reach the field, that the game was "reaching a tipping point where one coach would buy into the analytics approach. And if that coach were successful, there would be an avalanche."