I found this slightly interesting but not very insightful. The numbers people still have a really long way to go. I think he does a solid job of explaining the flaws of the stats we generally use, but the suggested alternatives are unproven, subjective metrics not dissimilar to PFF's clickbait grading system, and of course the author had a hand in developing many of them. And of course, the "better/smarter fan" bit in the opening remains as eye rolling as when Bernstein does it.
https://theathletic.com/1452369/2019/12 ... -analysis/Football is a complicated sport, and not everything can or should be quantified. But because there are so many players, teams and games, inevitably there will be times we rely on numbers to make comparisons. Which NFL teams are best at running the ball? Which pass-rushers have been the most effective this season? Numbers can provide a shortcut to watching all 256 NFL games when certain types of questions are asked.
There is a place in the sport for careful, numbers-based analysis, and if we’re going to use numbers to talk about football, there are good ways and bad ways to approach it. Following these rules will make you a better football analyst or fan.
These are the 10 commandments of numbers-based football analysis.
1. Thou shalt not use sacks or pressures to measure offensive line playExample: “The Titans have the league’s highest sack rate, therefore they need help on the offensive line.”
Why it’s misleading: As documented by PFF’s Eric Eager, a quarterback’s time in the pocket is quite stable from year to year (meaning that some quarterbacks tend to get rid of the ball more quickly than others), and quarterbacks thus largely control their own sack and pressure rates. A sack can happen only if the quarterback has not yet thrown the ball, and the quarterback is ultimately responsible for when he throws. Drew Brees is well known for getting the ball out quickly, and as a result, he rarely takes sacks. The same was true for Peyton Manning.
My favorite analogy is from Jason Lisk, who likens our unenlightened understanding of sacks as similar to the decades-ago thinking about walks in baseball, noting that Bill James once wrote that walks were viewed as “something that the pitcher did; the batter was just the guy who was standing there when he did it.” Of course, that has turned out to be wrong: Batters have a great deal of control over how many walks they take. The same is true for quarterbacks and sacks. Sacks tell us more about the play of quarterbacks than about offensive lines.
What to use instead: With player-tracking data, ESPN has developed a measure of pass protection intended to be independent of quarterback play, called Pass Block Win Rate (PBWR). PBWR simply measures the percentage of snaps where the pass protection holds up for at least 2.5 seconds after the snap, meaning that offensive lines aren’t penalized when their quarterbacks hold onto the ball. Going back to the Titans above, they rank No. 9 in PBWR, suggesting that their high sack rate is a product of their quarterbacks. The message here isn’t that offensive lines aren’t important, but rather that sack rates are a poor way to measure pass protection.
2. Thou shalt not use sacks in isolation to measure the effectiveness of a pass-rusherExample: “Jadeveon Clowney has only three sacks in 2019, therefore he is not a good pass-rusher.”
Why it’s misleading: Setting aside the control that quarterbacks have over sack rates as mentioned above (i.e., defenses that play against quarterbacks who take a lot of sacks are going to have higher sack totals), sacks are such rare-frequency events that they are not a great measure of the skill of a pass-rusher. If a pass-rusher consistently applies pressure to a quarterback without getting home, chances are that his sack rate will rebound. Citing Eager again, pressures are a better predictor of how many sacks a player will have than how many sacks he’s had in the past, meaning pressures are a better indicator of how often a pass-rusher is winning.
What to use instead: Similar to the above, ESPN publishes Pass Rush Win Rate (PRWR) for defensive players, and Clowney ranks No. 7 among edge rushers in 2019 (the numbers in this section are updated through Week 14; Clowney missed Sunday’s game in Carolina with an injury). Pro Football Reference now includes charted pressures, and Clowney’s 30 pressures rank No. 17. Clowney may not be quite up there with the league’s elite, but his three sacks paint a misleading picture of his performance as a pass-rusher in 2019.
3. Thou shalt not use TD-INT ratio to measure quarterback playExample: “Aaron Rodgers has a league-leading TD-INT ratio of 24-2, therefore he has been among the league’s best quarterbacks in 2019.”
Why it’s misleading: First, quarterbacks throw a touchdown or interception on, at most, 10 percent of plays. TD-INT ratio tells us nothing about what happened on the other 90 percent of plays. How often do a quarterback’s passes go for a first down? How many sacks does he take? Do the quarterback’s touchdowns come from 50 yards away or at the goal line? TD-INT ratio tells us nothing about these important questions. Second, the number of touchdowns a player throws is, in part, a reflection of how often his team passes close to the goal line. For example, Rodgers has thrown 16 passes inside the opponent’s 5-yard line (with seven touchdowns), compared with Dak Prescott’s seven passes (four touchdowns). But that doesn’t tell us much about the quarterbacks, only that the Cowboys are more likely to call a run play at the opponent’s goal line than the Packers are. And third, many interceptions are thrown when quarterbacks are trailing late in games and have very little impact on who wins. Avoiding interceptions at all costs in these scenarios is probably not even a good thing.
What to use instead: Expected Points Added (EPA) per play quantifies the value of every play rather than just the touchdowns and interceptions. ESPN’s QBR is based on this measure. Returning to Rodgers, he is currently No. 11 in EPA per play and No. 17 in QBR. There’s no easy way to look up player-level EPA for a season aside from checking my Twitter feed, but given the proliferation of numbers-based football analysis, we’ll probably get there soon. Player-level EPA per play from a given game can be found on Josh Hermsmeyer’s Air Yards site. Finally, Next Gen Stats’ Completion +/-, which measures how many more passes a quarterback completed than expected given where he threw his passes, is also worth keeping an eye on.
4. Thou shalt not use passer rating to measure quarterback playExample: “Dak Prescott’s passer rating of 99.3 is a middling No. 12, therefore he’s had an average season.”
Why it’s misleading: In addition to the problems with counting touchdowns and interceptions presented above, passer rating overweights completion percentage and omits sacks, fumbles and quarterback rush attempts. As Football Perspective’s Chase Stuart documents, “for predictive purposes, two of the four inputs in passer rating (completion percentage and interception percentage) are meaningless.” The flaws with passer rating are obvious when individual quarterbacks are examined this season. Lamar Jackson adding more than 1,000 yards on the ground? Ignored by passer rating. Daniel Jones losing eight fumbles on sacks? Ignored by passer rating. Teddy Bridgewater completing 68 percent of his passes by throwing the shortest passes of any player in the league? Overvalued by passer rating.
What to use instead: As discussed above, EPA per play, QBR and Completion +/- are all useful measures. Prescott has been top six in EPA per play (No. 5), QBR (No. 3), and Completion +/- (No. 6). If you want a simple passing statistic that can be obtained from a box score, just use yards per pass attempt (Y/A) or net yards per pass attempt after deducting sack yardage (NY/A). But even these simple measures are problematic because they ignore rushing.
5. Thou shalt not use a running back’s production to measure running back playExample: “Ezekiel Elliott topping the charts in rushing yards per game year over year shows that he is an elite running back.”
Why it’s misleading: How many rushing yards a player accrues is primarily a function of opportunity: The players with the most yards are going to be the ones with the most carries. However, even when looking at per-carry measures like EPA per carry or success rate, we have a lot of evidence that the results of a running play are primarily dictated by factors outside the running back’s control, such as the number of defenders in the box and the blocking of the offensive line. In other words, running back efficiency is not a direct measure of running back play.
What to use instead: Just assume that all running backs are equally good at running the ball and you won’t often be led astray. Elliott, in particular, is fine. And please don’t use rushing yards per carry.
6. Thou shalt not use a team’s win-loss record as a measure of team strengthExample: “The Seahawks’ 11-3 record proves they are one of the top teams in the league and validates Pete Carroll’s conservative approach to football.”
Why it’s misleading: A team’s win-loss record is heavily influenced by how they perform in close games, and every piece of evidence we have suggests that performance in close games is random — that is, teams do not possess some special skill that allows them to consistently win close games. For example, ESPN’s Bill Barnwell found that teams whose record exceeded their predicted record (based on point differential) by two to three wins in a given season saw an average 2.5-win decrease the following season. Winning close games helps in the standings but is not an indication of a good team.
What to use instead: The point differential column in NFL standings is much more informative about team strength than win-loss record. If one wanted to use a record-based approach, record in non-one-score games is a better measurement of team strength than overall win-loss record. For example, while the Seahawks and 49ers are both 11-3, the 49ers are 8-0 in games decided by eight points or more, while the Seahawks are 2-2. Other good options for measuring team strength are a team’s EPA per play, DVOA, or 538’s Elo.
7. Thou shalt not cite a team’s record when it reaches a certain number of rush attempts to suggest that running the ball is importantExample: “The Titans are 13-2 since 2016 when Derrick Henry has 18 or more carries, so they need to make an effort to feed him carries early in a game.”
Why it’s misleading: This is a variant of the Run To Win fallacy, originally debunked by Football Outsiders back in 2003 and which I previously discussed here. To repeat what I wrote there: “The problem with citing the correlation between rush attempts and wins as a reason to prioritize the run game is that the relationship exists only for teams that have already built a lead. This advice gives no actionable information about the hard part: getting the lead in the first place. The correlation between rush attempts and wins is just as useless as the correlation between quarterback kneels and wins. Both contain the same information: Teams that have a lead late in the game tend to end up winning. Closely related is Brian Burke’s so-called “passing paradox”: worse passing teams tend to pass more because they’re likely to fall behind in games and then pass at a high rate to try to catch up, leading to a misleading inverse relationship between pass attempts and wins.
What to use instead: Just don’t. Unless your team has a rushing threat at quarterback like Lamar Jackson, using a numbers-based argument to justify running the ball at a high rate is probably going to be a losing battle. It’s time to move on.
8. Thou shalt not use a team’s rush attempts per game to gauge its run-heavinessExample: “The Patriots being top 10 in rush attempts per game demonstrates their commitment to running the football.”
Why it’s misleading: As discussed above, how often a team runs the ball reflects, in part, game state. A team that often has a big lead in games will run the ball a lot to run out the clock, and the Patriots have been ahead a lot this season. Rather than rush attempts or pass attempts per game, what is more meaningful is how often a team runs or passes the ball when the game is close.
What to use instead: Run percentage on early downs in close games, excluding the final two minutes of halves (for an example, see the figure in Mike Sando’s piece on the Seahawks’ run-heaviness over time). The reason for setting aside third downs and the final two minutes of halves when measuring run-heaviness is that run-pass decisions then are usually dictated by the situation. For third downs, yards to go is the primary driver of whether a team passes or not. On third-and-long, the vast majority of plays are passes. In the final two minutes of halves, clock considerations will largely dictate run-pass choices. Finally, omitting non-close situations is more informative because teams that are trailing big will tend to pass more and teams that are ahead will tend to run more. By looking at early-down, neutral-game-script situations, we can get a better measurement of a team’s underlying preferences for rush frequency. Situational run rates can be accessed on Warren Sharp’s stats site. Going back to the Patriots, they are the eighth-most pass-heavy team in these early-down neutral situations, indicating that they are a pass-heavy team, not run-heavy.
9. Thou shalt not use a team’s unit-level yards per game as a measurement of effectivenessExample: “The Packers are No. 17 in rushing yards per game, therefore they have an average rushing offense.”
Why it’s misleading: Rushing and passing yards per game are reflections of a team’s preferences (how often it wants to run) and the game states it finds itself in (how often it is leading). Because the Packers are a pass-heavy team in early-down, neutral situations, they are only No. 17 in rush attempts per game despite being 11-3 and leading in a lot of games. Their rushing yards per game is a function of how often they run the ball, not how good they are at it.
What to use instead: The previously mentioned EPA per play is a good option for measuring the per-play effectiveness of a given unit. The Packers rushing offense is No. 7 in EPA/play and No. 10 in the percentage of plays with positive EPA. Football Outsiders’ DVOA is a similar measure that adjusts for the defense faced and game situation and had the Packers rushing offense at No. 4 through Week 14.
10. Thou shalt exercise extreme caution when presenting small-sample splitsExample: “Seattle’s defense forced 14 turnovers in its first nine games without Quandre Diggs, and then 10 in its first three games with Diggs, meaning the Seahawks defense has been transformed for good.”
Why it’s misleading: There is a lot of randomness in football. Sometimes, a team can go through a sudden change due to a direct cause. Perhaps that was the case with the Seahawks and Diggs. But a lot of times, splits happen. The Falcons defense got lit up all season before having a dominant two-week stretch against the Saints and Panthers. Did a switch flip? Were they fixed? Probably not, as the following week they allowed 35 points to Tampa Bay. The Seahawks defense would see a three-game stretch of great performances followed up by Jared Goff and Sean McVay running circles around them.
What to use instead: If one is presenting a split, always include sample sizes. For example, if you read something about how a team performs in a certain personnel package or against a certain type of coverage and no sample sizes are presented, it’s probably safe to ignore. And second, when splits are presented, make it clear that many splits are driven by the randomness inherent to football rather than a specific cause and that it’s hard to tell the difference in the moment. Always ask whether a split is meaningful and likely to continue. If not, treat it as an interesting factoid.
Wrapping upMistakes in numbers-based football analysis can broadly be grouped into three categories. First, looking at the wrong statistics (Commandments 1-6). When new and better measures become available, there’s little reason to keep using the old ones. For example, sacks shouldn’t be used in isolation to assess the play of pass-rushers because they are low-frequency events, and we can now also look at pressures. For statistics like passer rating, it’s time to let old things die.
The second prevalent mistake is using statistics that are strongly affected by game state, such as rush attempts or rushing yards per game (Commandments 7-9). This results in misleading numbers that measure whether a team is winning but not what actually led to winning in the first place (a mistake that is made all over the place, especially this time of year). Finally, there’s reading too much into randomness in small samples (Commandment 10).
Our understanding of the game is evolving. My recommendations for “What to use instead” in each of the above are based on an understanding of what the evidence shows at this point, so it’s possible that some of the recommendations won’t age well. But I’m confident that each is better than what has been traditionally used. As Billy Beane said in “Moneyball,” adapt or die.
Login
Register
FAQ
Search
