So now we're gonna give you an extended example of people analytics at work and follow that with some detailed issues involved in many analytics projects. So this example comes from my work with the national football league. As a result of some research I did a few years ago, I've been pulled in to work with some professional teams on their evaluation of players. And especially their evaluation of college players. One team in particular asked me, who is good at evaluating college players? Which team should they be paying attention to, which team should they be copying?...

So, now what we want to know is, do teams tend to have relatively successful picks? Do they tend to have relatively unsuccessful? Or is it just randomly distributed around what would be expected? That's where we're going when we're trying to determine is there skill here? Or is there luck. So, here's an example of one of the most famous draft classes in national football league history. The Pittsburg Steelers, famous draft class of 1974, they drafted four hall of fame players, Jack Lambert, Lynn Swan, John Stallworth, Mike Webster. All these guys were hugely successful, not just for a year, but for a career. Now, if the draft, if the Center Field draft involves skill. This is a team. They drafted these players because they're especially good at their job. What would you expect to happen the next year in 1975? So, assume you've got the same scouts, you've got the same general manager. They select all these great players in '74. If that's the result of skill, what would you expect to happen in '75. Or, what would you expect to happen in 73? Let's look at 73. In 73, their 2nd round pick never played a game. Their 3rd and 4th round picks were average at best. And yet, that was just one year before they had this Hall of Fame class. What about 75? What about the next year? In 75 it was even worse. Not a single player drafted started. Out of 21 picks at this enormous draft, and not a single player drafted started. And picks in each of the top six rounds. They only played a total of 24 games for the team. So, what do we think about a process where you draft one of the best classes ever, probably the best class ever, in 74. And yet the 73 draft was completely average. And the 75 draft was actually tragically bad by any measure. What does that say about how much skill is involved in this? How much credit should we give them for that Hall of Fame class in 74? That's the idea, and that's a very general idea that's maybe the biggest lesson in the performance evaluation is. The question is, does it persist? Skill persists. Chance doesn't persist. And if the challenge is to parse skill from chance, the single most important test is persistence. Do you see it across periods? Do you see it over time? Do you see the positive performance measures persist? That's what we're gonna do here.

We're gonna take all the years that we observed teams draft and we're gonna code them up in just the way we described. We're going to, evaluate does a player do better or worse than expectations. And for that year we're going to add up all of those deviations or the positive and the negative, do they add up to zero, whatever. We're going to evaluate every team and each year of the draft. And we're going to rank the league, one to 32 within a year on how they performed in a draft. And then we're gonna ask what happens to next year. So for all the teams that did best in a year, what happens to them in the next year's draft? And we're gonna do for the teams that rate 16th in a given year, what happens in the next years draft? That's gonna give us a test of persistence. If this is a skill based task, those teams that do well in the draft in a given year will do well the following year. Those teams that do poorly will do poorly. If it's completely chance, how a team performs in one year will have nothing to do with how they perform the next year. There'll be no correlation between the two. And if it's a mix of skill and chance, you'll see something in between. That the teams that do well one year will tend to do better the next year and the teams that do poorly will tend to do poorly but they'll regress to the mean. So that's the test. We'll find out what happens. This is what we find. I've shown all the teams, over all the years in our study here in gray. But then I've highlighted the bunch that were rated in number one, from the top in green. And then in the middle in blue, and the bottom in red. And what do we see? What do we see here about the relation between how a team does in one year and the following year? At a high level, we see essentially no relation. Consider, for example, the teams at the very top in green. These are teams that, in a given season, were the single top performing team in drafting players. What happens to them the next year? Well, one team was again the top-performing team. But, one team was also the worst-performing team. And you can see that there's a full spread, that from that number one position, they went to every other position from one to thirty-two, there was no predictive quality. About their first year performance in the next year. And conversely, the same at the bottom. You can look at the teams in red, which is 28th or so, and in one case, 28th was again 28th, almost perfect persistence. But, in the other cases, they drifted up, some were kinda middling the next year, and some were actually quite good in the following year. So, this tells us overall the correlation is slightly negative. Not different from zero but slightly negative. Essentially, zero, there's no correlation between how a team drafts at one year and how they draft the next year. And when there's no correlation, we can be sure, that means the differences that we observe are not the product of skill, the differences we observe are the product of chance.

So, this is one performance measure, it's a starts, you can do this for other performance measures, this is how much a player receives in compensation, when he reaches free agency, you can use any number of performance measure and you get the same result. Most of the deviation goes away the next year which means that most of the deviations, most of the differences between teams are the result of chance and not skill. So, different performance stats, different player career stage. You can norm for additional factors like a player position. You can evaluate not the team level performance, but you actually look at the person in charge, whether it's a general manager or an owner, and you can track that individual's performance over his career, and again you don't see persistence. The vast majority of the variation is purely a product of chance.

We asked people how much draft outcome is completely due to random chance versus draft outcomes are completely due to drafting skill? Where on that continuum, skill to chance, do you believe draft outcomes fall? And we asked this of NFL fans and we screened them, for actually following the NFL and what do you see, you see people, they don't think it's all skill. But the vast majority of people believe that it is on the skill side of the continuum. Almost nobody says that it's chance-related. By far the most are two-thirds, three-quarters, skill. And these are folks that actually follow NFL, follow the draft. They know that there's a little chance involved, but they greatly underestimate the amount of chance that's involved.

I'm a big dumb shitlib baby

The future holds the possibility to be great or terrible, and since it has not yet occurred it remains simultaneously both.

Every aspect of sports has more luck than most are willing to admit, especially when it comes to winning/championships.

Drafting at the top has a lot of luck. Drafting for depth takes talent, whether that's the GM and scouting or the instructors through the minors excelling at getting players to buy in and execute. What the Cardinals do, as the Twins and Braves did before them, in having a constant pipeline of above-average everyday talent speaks more to drafting/developing aptitude than picking the right guy out of the top three or whatever.

To be clear, the argument is if luck is 100% involved or just a big factor. I say luck is a big factor but there's an obvious skill/talent gap between the best and worst GMs. Vegan says it's 100% luck like bingo. Let's not misrepresent the argument.

I'm a big dumb shitlib baby

To be clear, the argument is if luck is 100% involved or just a big factor. I say luck is a big factor but there's an obvious skill/talent gap between the best and worst GMs. Vegan says it's 100% luck like bingo. Let's not misrepresent the argument.

This is going to reach a head pretty soon.

To be clear, the argument is if luck is 100% involved or just a big factor. I say luck is a big factor but there's an obvious skill/talent gap between the best and worst GMs. Vegan says it's 100% luck like bingo. Let's not misrepresent the argument.

Let's look at Steph Curry for example. David Kahn drafted 2 point guards ahead of him. Johnny Flynn. Ricky Rubio. Curry easily better than both. Now you can say that G.S. got lucky to draft him where they did or could say that Kahn was incredibly stupid for passing on him twice. Does a Buford West or Presti make that mistake if they are in the same position? Don't think so.

To be clear, the argument is if luck is 100% involved or just a big factor. I say luck is a big factor but there's an obvious skill/talent gap between the best and worst GMs. Vegan says it's 100% luck like bingo. Let's not misrepresent the argument.

Yeah it isn't 100% luck. Otherwise, why not just draft random DIII players and hope they are all diamonds in the rough.

What if GS had Thabeet or Flynn ranked higher on their draft board than Curry? They settle for Curry once their top two options are gone. In that case they got lucky.

This is going to reach a head pretty soon.

But that isn't an appropriate analogy. If you pick at the 3, 4, 8, and 12 spots in consecutive years, and the guys you had as your top choice went off the board each year, before your turn and ended up sucking, and the guys you ended up drafting (all of whom were not your first choice) ended up being perennial pro-bowlers, are you an uber talented GM or just lucky? And being lucky doesn't necessarily mean you aren't talented.

To be clear, the argument is if luck is 100% involved or just a big factor. I say luck is a big factor but there's an obvious skill/talent gap between the best and worst GMs. Vegan says it's 100% luck like bingo. Let's not misrepresent the argument.

On behalf of sane, logical, and reasonable people everywhere...I'd like to apologize for this thread's existence.

We asked people how much draft outcome is completely due to random chance versus draft outcomes are completely due to drafting skill? Where on that continuum, skill to chance, do you believe draft outcomes fall? And we asked this of NFL fans and we screened them, for actually following the NFL and what do you see, you see people, they don't think it's all skill. But the vast majority of people believe that it is on the skill side of the continuum. Almost nobody says that it's chance-related. By far the most are two-thirds, three-quarters, skill. And these are folks that actually follow NFL, follow the draft. They know that there's a little chance involved, but they greatly underestimate the amount of chance that's involved.

Nobody here is underestimating luck or chance or whatever. You're simply overestimating it. That's it. It's not 100% luck and it's asinine to suggest it is.

I'm pretty confident we can call the Front office of the New England Patriots more skilled than lucky.

FFS I'm a Sox fan and can even admit Theo Epstein is more skilled than lucky.

Carry on with your data....skill > luck

I do think guys are underestimating chance, otherwise you wouldn't be as strident as you are here about the level of skill involved.

I'm pretty confident we can call the Front office of the New England Patriots more skilled than lucky.

FFS I'm a Sox fan and can even admit Theo Epstein is more skilled than lucky.

Carry on with your data....skill > luck

But that isn't an appropriate analogy. If you pick at the 3, 4, 8, and 12 spots in consecutive years, and the guys you had as your top choice went off the board each year, before your turn and ended up sucking, and the guys you ended up drafting (all of whom were not your first choice) ended up being perennial pro-bowlers, are you an uber talented GM or just lucky? And being lucky doesn't necessarily mean you aren't talented.

That isn't an appropriate analogy because if you are telling me you are picking your next best available, then you are applying some skill in choosing who to pick by the very nature of having ranked the players. If it was 100% luck who became good or not, you could draw names out of a hat. Even someone who picks up SI's top 50 players in the draft column and uses it to pick players is now applying some sort of skill to their drafting.

If that is the question, I think it is not 100% luck; your pick may be different due to team needs/style than an overall consensus would point to. However, I think it is a safe argument that in the Luck + Skill = 100% equation that Luck > Skill.

Yes I would be. Any time I see an argument this ridiculous I'm probably going to go hard against it.

This is a simple argument that I think you are trying to complicate for no reason. Let's start over. Is drafting 100% luck or is there at least 1% skill when it comes to drafting? If it's the latter, I don't even care to continue this argument. I just want you to admit how ridiculous your 100% argument is.

If a player drafted by another team is a bust but proves to be a skilled player in the hands of Patriot coaches, does the original drafter get credit for identifying the talent?

How are you defining skill though? The entire structure of a draft in any league is based on chance - everything from where you pick to who is available to draft. Krause had two high lottery picks in 2001 that he used on Curry and Chandler. He left the team a few years later. If he had two high lottery picks in 2003 he might have his own statue right next to MJ's.

This is going to reach a head pretty soon.

How are you defining skill though? The entire structure of a draft in any league is based on chance - everything from where you pick to who is available to draft. Krause had two high lottery picks in 2001 that he used on Curry and Chandler. He left the team a few years later. If he had two high lottery picks in 2003 he might have his own statue right next to MJ's.

Seacrest vibe

A skilled GM with a nose for talent probably drafts Gasol over those two in that draft.

A skilled GM doesn't draft Darko over Melo or overrate Tayshaun Prince.

Skilled GM doesn't trade Kobe Bryant for Vlade Divac. Those are but a few examples.