Matchmaker InsightsLast updated 1 year ago
Published on 2015-02-22 19:46:46
A few months ago Hi-Rez adjusted the way their queues worked, ditching their set-time queues in the hopes of increasing match quality. There were certainly some growing pains as the new system was implemented. Some refused to play ranked, while others tried, but couldn’t get into a game. Many a Reddit thread was made bemoaning the new matchmaker. Over time, Hi-Rez made incremental improvements, and we’ve come a long way since those first days after the switch.
But how good is matchmaking now? With the help of Smite.guru’s database, I’ve collected and analysed around 12,500 League Conquest matches from this past weekend in order to answer this very question. I created a few graphs that I hope will paint an accurate picture of the state of matchmaking, but before we get to that, let’s talk a little about how matchmaking works and how we might evaluate it.
Matchmaking and Elo
Smite uses a variation of the Elo rating system in order to rate its players on their ability to play the game. I do not want to get into all the minute details of how Elo works, but essentially it is a single number which represents a player’s current skill at the game. A player with a high Elo is expected to be better than a player with a low Elo. As it was originally developed for chess, a two player game, some question whether Elo is an appropriate rating system for a team game, but for the purpose of this analysis we’ll assume that Elo is in fact an accurate portrayal of a player’s capability.
How does this relate to matchmaking? Well, for the most enjoyable and fair playing experience, you generally want games which have players of similar skill level on both teams. It can be a very frustrating experience being matched with players either much more or much less skilled than yourself. A diamond-level player might get bored if she consistently found herself crushing through low-level competition. In the same vein, a less skilled player might become discouraged dying to a professional player over and over. So if we assume that Elo is a good measure of ability, then a good matchmaking system will put players with similar Elo ratings into the same matches. This is how we’ll evaluate Smite’s matchmaking system, we will look at the closeness of Elo ratings in League Conquest matches.
There is a second job of a matchmaking system though, and that is to get players into a match in a reasonable amount of time. No one wants to sit around for an hour waiting for nine other players. Unfortunately, we do not have any queue time data, so we will not be evaluating the matchmaker on this aspect today.
Who is Playing?
First, let’s take a look at the playerbase of Smite. From the histogram above we can see that the vast majority of players in matches have between 1000 and 2000 Elo. In fact, quite a few players were between 1250 and 1750. This makes sense as players start league at 1500 Elo, so an average player should also have around 1500. We see a nice, nearly normal distribution, with a slight falling off on the high side.
One thing to note is that this is a count of players that got into matches, so it could be slightly skewed to the middle as players with lots of similar Elo might be able to find more matches more quickly.
Measuring the Closeness of Elo: Standard Deviation
The best way to measure the closeness of data is to find its standard deviation. In this case, the standard deviation tells us roughly how far the average player’s Elo is from the average Elo of the match. The smaller the standard deviation, the closer together the Elos are to each other. As mentioned earlier, we want the matchmaker to put players with similar Elo together for a better playing experience, so we expect the standard deviation of the Elos to be low.
The histogram above tells us that almost all matches have a standard deviation of less than 250 Elo. Is this number good? Not really. Most matches hover around 200, which means that some of the players are at least 400 Elo apart from each other. As a point of reference, the average standard deviation of all matches Elos was 204. As we’ll see in the next section, this translates into a lot of matches with large differences in player Elo.
Measuring the Closeness of Elo: Max Difference
I also wanted to see how often matches featured large differences in Elo. To do this, I found the difference between the maximum and minimum Elo in each match, then made a histogram of the data. As you can see, the results are perhaps a little disheartening. There are multiple matches that feature Elo gaps greater than 500. Even differences of a 1000 or more are fairly common. Considering that the vast majority of the playerbase is between 1000 and 2000 Elo, the commonness of matches with Elos that span this range is somewhat disappointing.
The Matchmaker’s Effect on Winning
Last, we should look at whether the matchmaker has any effect on our ability to win a match. Are you getting into matches that are basically unwinnable through no fault of your own? The short answer is probably not. I looked at several factors that could lead to an unfair advantage for one team, and only one of them seemed to be at all predictive of match outcome.
The team with the higher average Elo won 51% of its matches, which is hardly an undeniable advantage. Teams with the highest Elo player had the same winrate (51%), suggesting that just having the best player doesn’t ensure victory. There’s also the idea that teams with similarly-skilled players will do better than teams with large skill gaps. However, this idea is not supported by the data, as teams with a smaller standard deviation of Elos only won 49.7% of their matches.
There was only one attribute that seemed to have a noticeable effect on match outcome: an imbalance of qualifier players. In matches with a different number of qualifiers on each team, the team with fewer qualifiers won 65% of the time. This is somewhat expected, as qualifiers are usually newer to the conquest scene, and so will tend to perform below average. In addition, they haven’t played enough matches for their Elo to accurately reflect their skill level. This is a natural consequence of using an Elo system and is to be expected, especially with the large influx of new League Conquest players that season 2 has seen.
So what can we say about the state of matchmaking in League Conquest? While the standard deviation and max difference graphs showed some rather poor results, I think the real problem is a matter of population size. Even with the large increase in the League Conquest playerbase going into season 2, there were still only around 23,000 unique players across two regions that played over the course of two days. If we’re going to have higher-quality and more fair matches, we’ll need to encourage more people to play Leagues.
You’ve been reading jhuns, tune in next time for more mind-blowing stats!