After two teams play n games against each other, what is the probability that the team that won the majority of those games is actually the more skilled team?
This assumes that we have some way to measure how skilled a team is going into the series. One measure, although far from perfect, could be the team’s standings at the end of a season.
Imagine each team has two sources of ‘points’. One source is skill, and the other source is luck. The team with the most ‘points’ wins the series. The ‘points’ from skill are fixed: a good team has more ‘skill points’ than a bad team. But the ‘points’ from luck are variable: on any given day one team could have more ‘luck points’ than the other. And the fewer the number of games that two teams play against each other, the larger the role that luck plays in determining who will win the majority of those games.
So we can ask a couple of questions. How many more ‘luck points’ does a less skilled team need in order to beat a more skilled team? And what are the odds that the less skilled team will get the required ‘luck points’?
I crunched the numbers (see supplementary document) to come up with some answers. I’m not sure how sound the analysis is, but I have reason to believe it’s reasonable. I should note that the purpose of this analysis is not to create an accurate model of win probabilities; the purpose is to show how we can use what we know about the role of randomness in a league to address the first question in this post (as it turns out, the results seem reasonable when compared to actual statistics).
The playoffs in the major sports leagues are short. In the NFL each round is 1 game, in the NBA and NHL they are 7 games, and in the MLB the first round is 5 games. The purpose of a playoff round is to determine which is the better team. But can you do that in so few games?
In their first round of playoffs, the top seed team plays a team ranked lower than them. How much lower depends on the league. Let’s assume that the team’s seed tells us how skilled they are. So what are the chances that after a playoff series the team that won the majority of the games is actually the more skilled (top seeded) team and not just a lucky underdog? Here are the results – based on the above ‘skill points’ and ‘luck points’ analysis – for the different leagues given their playoff formats and possible rankings of the lower seed.
So in the NBA an underdog will rarely get lucky enough to win a first round playoff series, and in the MLB, an underdog will often get lucky enough to win a first round playoff series. And it turns out that top seeded NBA teams do win around 90% of the time, and top seeded MLB teams do win around 50% of the time in their first playoff series. This analysis suggests that we might be able to attribute these upsets to how big of a role randomness/luck plays in each of the sports.
So if we wanted to be sure that the more skilled team would win their first round of the playoffs, how many games would we have to play? How many games are needed to lower the chances of a lucky underdog upsetting the top seed to 10%?
So to be pretty confident that the team that deserves to win isn’t a victim of a lucky underdog, an MLB playoff series should be 10 times longer – same with an NFL playoff series – and an NBA series is about the right length.
Playoff series in the MLB and NFL tell us almost as much about which team was the luckiest as it does about which team is better.
If we are relying on playoff series to tell us which team is actually the best (most skilled), we need to make NHL, MLB, and NFL playoff series much longer.
In the next post, I’ll wrap things up. In the meantime, check out this supplementary material for this post. I wouldn’t blame you for being curious, even skeptical, about how I came up with the numbers/plots in this post. But the accompanying document should clear things up. Also, if you want to try changing variables in the code to see how these plots change, download the notebook and give it a try.