How To Fix The Top Ten

I had regular complaints about the U SPORTS Top Ten rankings last season in women's hockey. Teams that beat teams ranked higher would rarely move, teams that were making serious waves within their respective conferences rarely got the respect they deserved, and there appeared to be biases towards perennially strong teams regardless of their win-loss record. Today, U SPORTS announced that they are changing the method to determine the top ten teams for all sports governed by U SPORTS.

Perhaps the most damning statement about the previous method of determining the top ten teams in each sport was delivered by Ken Saint-Eloy, U SPORTS Senior Manager of Marketing and Communications, in the press release today. He said,

"There wasn't a clear and objective answer when we were challenged by our audience about why a team moved up or down, or didn't make the Top 10 after a series of wins. We were not able to provide simple answers on who were the voters, and/or what criteria they were voting on. Even in the same sport, for example, women's soccer was not following the same process as men's soccer. Given the situation and in the interest of making things more visible and structured following the rebranding of the organization, we mapped out the rules and process for every single sport and looked at options of objective, stats-based solutions for these Top 10 rankings. Because you want your Top 10 to create conversations about the teams, the games played the past weekend or the coming matchups, not about how consistent or reliable the Top 10s are, especially if they play a role in the seeding of the national championships."

Game, set, and match on why fans and broadcasters were disappointed with the top-ten rankings over the last few seasons. It became glaringly apparent that there was a significant voter bias - whether it was coaches or media - towards Manitoba, Alberta, and UBC in women's hockey last season despite Saskatchewan's incredible play over the last half of the season.

Saskatchewan wasn't ranked in the U SPORTS Top Ten until the week of February 27 through March 6 - one full week after they eliminated the UBC Thunderbirds from the Canada West Playoffs. What makes that harder to believe is that Saskatchewan finished ahead of UBC in the regular-season standings, were neck-and-neck all season with Manitoba, Alberta, and UBC during the regular season, and Saskatchewan finished the season 4-0-0 against the Thunderbirds in head-to-head play in the regular season. Despite none of us having a vote, there were many of us up in the Wayne Fleming Arena pressbox who fully believed there was something flawed with the system when Saskatchewan continually missed out on the Top Ten rankings.

Today, U SPORTS announced that all Top Ten rankings will be determined by the Elo Rating System. The Elo Rating System originally was created by Arpad Elo as a way for calculating the relative skill levels of players in zero-sum games. As an example of a large-scale use, FIFA uses it to determine the world rankings for women's soccer and is implementing it for men's soccer following the most recent World Cup. Games such as chess and Scrabble use it for head-to-head meetings of its player, and video game tournaments often use these rankings for its participants.

Before we go any further, I should explain how the Elo Rating System works in a game like chess before we get too deep into this. Every chess game has a winner and a loser unless a stalemate occurs, but stalemates are rarely seen at the elite level. Because of this each player has a number of points that they have accumulated through the playing of games. Winning and losing games adds and subtracts points, resulting in specific ratings for players based on their accumulated point totals. The difference between these ratings of the winner and loser determines the total number of points gained or lost after a game. This is the key point of the Elo Rating System as heavily-favored players won't see their ratings increase very much with wins over lower-ranked opponents, but upsets have a much bigger effect on the ratings.

In a series of games between a high-rated player and a low-rated player, the high-rated player is expected to win more than he or she loses. If the high-rated player wins, few rating points will be taken from the low-rated player due to the expected outcome being reached. As stated above, if the lower-rated player upsets the higher-rated player, many rating points will be transferred from the higher-rated player to the lower-rated player because of the unexpected outcome. In short, players that rise above and maintain that level of play will find themselves moving up the ratings system. Players that were thought to be strong who underperform will fall down the ratings. Makes sense, right?

Mario Kovacevic, a former player for the University of Toronto Varsity Blues men's soccer team and assistant coach with the York Lions, made a few adjustments to the Elo Rating System calculations for each sport through his RankR company. As written in the article, "[e]ach system is modified and tailored for the specific sport in question, and it brings in historical data from past seasons to help compare teams that don’t directly play each other, such as those in different conferences." In short, the new system should be able to tell us whether Sasakatchewan is better than Queen's, Montreal, or St. Francis-Xavier next season and moving forward.

"I was playing soccer for U of T and also doing a master's of engineering, and between being a grad student and playing for a U SPORTS soccer team, I was interested in the rankings," Kovacevic said in the release. "It was a lot to do with votes and you couldn't really drill down to the bottom and figure out 'Okay, what does my team have to do to get better than the team ahead of us?' And furthermore, if you were outside of that Top 10, you really didn't have an idea if your team was 40th in the country or 11th in the country. So between those two main things, I just started playing around with different rankings systems. From there, I put together a really simple rating system, and I sent it to some of the coaches and they were really impressed."

Unfortunately for women's hockey, here's the catch:

Women’s hockey is one of the sports that isn't fully adopting the statistical system yet, and that's because its Top 10 rankings are used for the purposes of national championship seeding. That means any change to that procedure can't be made without altering the official playing regulations, which require consultation with the sport's technical subcommittee. But these rankings will still be calculated for women's hockey too and sent to the voters each week, giving them some statistical information they can consult if they wish. And the plan for women's hockey, along with cross country, men's and women's basketball and women's volleyball is to eventually move towards this system, thereby standardizing all U SPORTS offerings, with swimming, track and field, wrestling and men's volleyball having been calculated statistically in the past.

So while the rankings will be calculated, the actual usage of the new Elo Rating System won't be factored in due to the national championship seeding. It will still be available, but the question will be whether or not the voters use this information, particularly when it comes to seeding teams from across the four conferences appropriately.

What I do know is that this new ratings tool will certainly be good for some debate if the voters' rankings differ greatly from the Kovacevic ratings. This will be something to keep an eye on as women's hockey was a hotly-contested topic regarding the final seedings for the national championship last season.

Until next time, keep your sticks on the ice!