Predicting the Patch

Predicting the Patch: Do Winrate, Pickrate, or Hot Topics in the Overwatch Forums Predict Future Balance Changes?

Hi everyone, Sine’s Stats back with another question: can we predict future Overwatch balance changes with publicly available data? Overwatch 2 and it’s associated changes–whether it be the 5v5 format, role passives, nerfing CC, or comments on specific heroes–have caused a big uproar in the community. It’s one of the few topics I’ve seen permeate into places like r/OverwatchUniversity–a community that usually focuses on improvement over balance complaints–but it’s certainly not the first time forum-based communities have commented en masse about game balance. Today, I’m going to use data from Overbuff, Patch Notes, and the Overwatch Forums to see whether a user keeping up-to-date on these platforms might potentially be more informed about upcoming balance changes than one who doesn’t.

TL;DR - Main Takeaways

This blurb ended up pretty length, so here are the main takeaways of my findings!

Overwatch Forums

• It’s not your imagination, the Official Forums are a hotbed for change/buff/nerf posts.
• The most (in)famous heroes on the Official Forums lately are Genji, Mercy, McCree, Sombra, and Rein.
• For some reason, the Forums really love Mercy (and Junkrat?) and they really hate Doomfist.
• Calls for nerfs/buffs align really well with the various metas we’ve seen- Sigma/Ball, McCree, etc.
• Some heroes are simultaneously called for nerfs/buffs, usually because they’re weak but hated (Genji, Hanzo, Reaper) or begrudgingly necessary (Rein)

Relationship with Balance Changes

• Winrates are predictive of balance changes in a nonlinear way
• Being a tank or healer is predictive of getting rebalanced during this study period
• Number of Replies to a post and calls for nerfs are marginally predictive of getting rebalanced, but their predictive power goes away after you include winrates and pickrates. Forums topics do not significantly affect balance changes!

The Forums

Unlike r/OverwatchUniversity, r/CompetitiveOverwatch, or r/Overwatch, the Overwatch General Forums seem extremely concentrated on the balance of the game. When I visit the official forums, I get the sense that the vast majority of posts are about whether game updates will be or have been good or completely destructive of the game. This might be a way to connect with other players, develop and change one’s opinions, but do these constant discussion efforts effect change? Is my intuition even right, that there are a lot of comments about balance and hero changes, or am I just prone to noticing those posts? I decided to dig a little deeper.

The Data

To check it out, I scraped Overwatch Forum General Discussion post titles, number of views, and number of replies from 11-01-2020 to 5-28-2021. I wound up with a dataset of 30317 unique posts in that timespan, which spans just about 8 Overwatch retail patches.

The Hot Topics

The first thing I was interested in is what are people talking about?. To get a sense of the “hot topics” in the Overwatch Forums General Discussion, I generated a Word Cloud of the most mentioned words in post titles that appeared in the timespan of my data, shown in Figure 1. Word Clouds are a graphical display of the most frequently mentioned words, where relative frequency is indicated by the size of the words. Right away, we can see that the most mentioned words are Overwatch, game, OW2, tank, need, change, nerf, and buff. It looks like intuition is right, that game updates are the hottest topics in the forums.

Figure 1. WordCloud of most common word mentions in the Overwatch Forum titles from 11-01-2020 to 5-28-2021.	Figure 2. WordCloud of most common hero mentions in the Overwatch Forum titles from 11-01-2020 to 5-28-2021.

But since we don’t have any data on OW2 or the upcoming tank changes, let’s concentrate on hero-specific changes that have been made. To get a sense of the hot-topic heroes, we can also take a look at the most frequently mentioned heroes in forum post titles (shown in Figure 2). We can see right away that the most (in)famous heroes are Genji, Mercy, McCree, Sombra, Sigma, Reaper and Rein. The frequency of hero mentions might tell us about, for example, the forum demographic or the state of a hero. But to assess which it is, we need to dig a little more into how these heroes are regarded when they’re mentioned.

Hero Sentiment

As a first pass, we’re going to check whether post title is more likely to be positive or negative in nature when a hero is mentioned. We evaluate this with a metric called sentiment.

Calculating the Sentiment

To calculate the sentiment, we’re going to use Python’s Natural Language Processing Toolkit (NLTK). This package uses a model called VADER to predict the sentiment of social media text as a number ranging from -1 (entirely negative) to 1 (entirely positive). I’ll leave it up to you guys to explore how exactly the model is trained ;).

Some Examples

However, there are some difficulties with this model. For example, sentiment predictions are often quite bad at predicting sarcasm, generalizing to very field specific lingo (buff/nerf would likely not be given a score), and the sentiment could easily capture positive or negative attitudes about non-balance features (e.g. skins). To get a better sense of this, let me show you some hero-specific examples:

Top Positive Examples	Score	Top Negative Examples	Score
To me, no hard counter heroes like Widow is a great s…	.934	KIll, kill, kIll Genji	-.944
The best way to help Genji without buff: show DAMAGE VALUES	.926	Dead Junkrat should STAY DEAD!	-.892
Super Genji, Super Hog and now Super Bap	.914	Rein suicide charge resulting in 1 kill = POTG	- .881
Mccree TOP MOST picked hero in all ranks and 2nd most picked hero in GM	.913	Dva mech kill is the worst balance change since brig	-.869
The Best Sym Buff that will make her super fun	.908	Ball is so unbelievably broken i hate comp now	-.837

Here, we see the good and the bad of our sentiment measure. Firstly, the negative sentiments look pretty good. All of the statements are clearly expressing negative feelings about or aspects of the hero mentioned. However, the positive sentiments are a mixed bag. The top two and the fifth look reasonable, positive ideas associated with particular heroes, but the third is clearly a misinterpretation of the word “super”, and the fourth is attempting to convey something negative (i.e. overabundance) about McCree. These sentiment predictions aren’t going to be perfect, but the hope is that more often not we’ll get the appropriate association.

Sentiment Distribution

To get a sense of what our sentiment measure looks like overall, let’s take a look at the distributions of sentiment across every hero that was mentioned in our forum data during the time period using box plots (shown in Figure 3). Very high level interpretation of the box plots: the location of the box shows where most of the sentiment is concentrated, and the spread of the box (and “lines” coming out of the box) shows how variable the sentiment is.

Figure 3. Box plots of Overwatch of Forum title sentiment mentioning a specific hero from 11-01-2020 to 5-28-2021.

There’s no doubt that our sentiment is a noisy measure, but there are a few things that stand out. Firstly, mercy has an overwhelmingly positive sentiment, and there is very little variation. It seems like the forums generally like mercy. This might be because there are a lot of healers or mercy mains that use the forums, or it might be because she tends to have had the most positive changes associated with her (e.g. balance, skins, etc.) during the time frame. The second most positively regarded hero is somewhat surprisingly Junkrat, though there seem to be less observations overall about him. For example, there are no outliers (individual dots) and the distribution is relatively tight. Finally, the most obviously negative sentiment is associated with Doomfist. This can come from folks disliking Doomfist, disliking that he is supposedly under- or over-powered, etc. Whatever it is, it’s negative.

There is certainly more curating one could do of the sentiment, but let’s save that for another time. For now we’ll just try and find out if there is some indication of a relationship between this sentiment measure and future patches.

Calls for Nerfs and Buffs

As stated above, though it often agrees with intuition, there are multiple difficulties associated with our sentiment measure. If our interest is how forum discussion predicts future patches, could instead use a simpler approach getting at the heart of the matter: how often is a hero name mentioned in the same title as the words “buff” or “nerf”?.

Figure 4 and Figure 5 show Word Clouds of hero names appearing in the same title as the words “buff” and “nerf”, respectively, from 11-01-2020 to 5-28-2021. The top heroes in the “buff” category are Genji, Orisa, Reaper, Moira, Dva, and Sombra. The top heroes in the “nerf” category are McCree, Ball, Sigma, Mercy, and Reaper. s

Figure 4. Word Cloud of most common heros mentioned in Overwatch Forums titles along with the word “buff” from 11-01-2020 to 5-28-2021.	Figure 5. Word Cloud of most common heros mentioned in Overwatch Forums titles along with the word “nerf” from 11-01-2020 to 5-28-2021.

These track extremely well with the time period- the sigma-ball tank meta and now the McCree meta, so not altogether suprising, but there are some curiousities. For example, how is Mercy often being cited in unison with nerfs while having a largely positive sentiment? Or, how is reaper in the top heroes mentioned for both “buffs” and “nerfs”? Since the first question might have in large part to do with cosmetics (which is outside of our scope of interest), let’s focus on the second.

Figure 6. Frequencies of Overwatch Forum titles mentioning “buff” or “nerf” along with specific hero name from 11-01-2020 to 5-28-2021.

Figure 6 shows the frequencies of post titles mentioning “buff” or “nerf” side-by-side and by hero name in our time period. Again, the calls for “buffs” and “nerfs” track well with the metas that have appeared, but some heroes seem more hotly contested than others. The most contested (i.e. most even splits between calls for buffs and nerfs) are Hanzo, Rein, Reaper, and Genji. Of course, these points of contention can come from various sources, like the feeling of “fairness” of play for the user versus the opponent (e.g. Genji, Hanzo), or a somewhat miserable play experience on both sides (e.g. Rein, Reaper). Ultimately, while there are multiple reasons for it, this is clearly capturing different variation than sentiment, so let’s include it in our analysis as well.

Overbuff Metrics

Of course, it’s possible that forum activity is a noisy reflection of simple, objective, and accessible metrics associated with hero balance. To test this, I scrape Overbuff Winrates and Pickrates for each hero and each elo over the 6 month period 10-18-2020 to 4-18-2020. Overbuff does have the potential for sample selection issues, particularly after the option of profile privitization (an issue we’ll dig into later), it’s the best a lot of us have public access to.

A Note on Winrates

We’ve all heard the anecdotes. Winrates go up, until heroes start getting double-picked, and then they start to increase. When heroes become to strong, they are often mirrored, forcing their winrates to zero. To account for this (and somewhat test for it), instead of using just winrate we’ll be using a quadratic winrate term in what follows.

Predicting the Patch

The Data

To answer whether forum activity, win rate and pick rate can predict future balance changes, I’ll integrate the above two data sources (Forum data and Overbuff data) with information from 7 balance patches (10/29, 11/17, 11/19, 12/10, 1/28, 2/18, 3/11). The final dataset will be a balance patch by hero dataset consisting of the following aggregate explanatory metrics: total views, total replies, average sentiment, total buff posts, total nerf posts, average win rate (by elo), and average pick rate (by elo), aggregated over the time window after the prior balance patch and before the next balance patch.

Our dependent variable will be a 0/1 variable on whether the hero recieved a change in the next two balance patches. Due to the relatively low sample size (\(32 \times 7\)), I do not attempt to distinguish positive from negative balance changes for now.

The Method

Given the variables listed above, I’ll conduct three seperate logistic regressions. First, regressing a future balance change on win rates and pick rates by elo. Second, regressing a future balance change on forum metrics. Finally, I’ll regress a future balance change on the combination of the two data sources. This last regression will give us an idea of if forum metrics can provide additional predictive validity over and above just win rates or pick rates.

Predicting Changes with Pickrate and Winrate

First up is the following model

\[P(\text{Future Balance Change} | \text{Aggregate WR/PR Data}) = g(\beta_0 + \sum_{\text{elos}} (\beta_1^{\text{elo}}WR_{\text{elo}} + \beta_2^{\text{elo}}PR_{\text{elo}}) + \varepsilon)\]

where \(g\) handles the fact that our outcome is a \(0/1\) variable. Then we’ll use some post-processing to get the average effects of each of our variables on the probability of a future balance change.

vars	mean	lower	upper
Avg. WR	-0.0324868	-0.7346612	0.6696876
Avg. WR2	0.3435003	-0.0079146	0.6949151
Avg. PR	1.5870687	-1.5057089	4.6798462
Bronze WR	0.0584433	-0.1230573	0.2399439
Bronze WR2	-0.0482074	-0.1327486	0.0363339
Bronze PR	-0.3625222	-0.9543633	0.2293189
Silv. WR	0.0111743	-0.2302079	0.2525566
Silv. WR2	-0.1321387	-0.2905082	0.0262308
Silv. PR	-0.0848494	-1.6619060	1.4922071
Gold WR	-0.1799870	-0.5659191	0.2059451
Gold WR2	-0.1567373	-0.3787129	0.0652384
Gold PR	-0.6589242	-2.9354520	1.6176036
Plat WR	0.0328308	-0.3418315	0.4074931
Plat WR2	-0.0918695	-0.2220211	0.0382822
Plat PR	-0.5003072	-1.9199234	0.9193089
Diam. WR	0.0808547	-0.2201565	0.3818659
Diam. WR2	0.0938898	-0.0813438	0.2691235
Diam PR	-0.3624998	-1.5584335	0.8334339
Mast WR	0.1046181	-0.0590011	0.2682374
Mast WR2	-0.0882321	-0.2118841	0.0354198
Mast PR	-0.0758558	-0.4601865	0.3084748
Healer	0.2380775	-0.0228934	0.4990485
Tank	0.3319556	0.1528026	0.5111085

Here we see a significant nonlinear relationship between winrates and future balance changes. Moral of the story: don’t listen to people telling you winrates don’t matter, they just aren’t as easily interpreted!

Predicting Changes with Forum Data

Second we have

\[P(\text{Future Balance Change} | \text{Forum Data}) = g(\beta_0 + \beta_1 \text{Sentiment} + \beta_2\text{Buffs} + \beta_3\text{Nerfs} + \beta_4\text{Views} + \beta_4\text{Replies} + \varepsilon)\]

where again \(g\) handles the fact that our outcome is a \(0/1\) variable. Then we’ll use some post-processing to get the average effects of each of our variables on the probability of a future balance change.

vars	mean	lower	upper
Replies	-0.0019925	-0.3442328	0.3402478
Replies2	0.0301375	-0.0372320	0.0975071
Views	-0.0057345	-0.2609346	0.2494657
Sentiment	-0.0052091	-0.0790481	0.0686300
“Buff”	0.0212186	-0.0511389	0.0935761
“Nerf”	-0.0415746	-0.2026093	0.1194601
“Nerf”2	0.0238220	-0.0187653	0.0664094
Healer	-0.0140269	-0.1897619	0.1617082
Tank	0.2600947	0.0663256	0.4538638

Here we see marginally significant effects of the number of topic replies and the number of posts calling for nerfs on future balance changes, but they are not statistically significant (at 95% confidence).

Predicting Changes with Everything

Lastly we have

\[P(\text{Future Balance Change} | \text{All Data}) = g(\beta_0 + \vec{\beta}_1 \vec{F} + \vec{\beta}_2 \vec{PR} + \vec{\beta}_3 \vec{WR} +\varepsilon)\]

where again \(g\) handles the fact that our outcome is a \(0/1\) variable, \(F\) is all of our forum variables and \(WR/PR\) are the winrates/pickrates for all of our elos.

vars	mean	lower	upper
Replies	0.2883689	-0.1637780	0.7405158
Replies2	-0.0351596	-0.1307498	0.0604305
Views	0.0430104	-0.2199064	0.3059272
Sentiment	0.0079231	-0.0683564	0.0842027
“Buff”	-0.0172377	-0.1052137	0.0707383
“Nerf”	-0.0684674	-0.2108674	0.0739326
“Nerf”2	0.0088853	-0.0370518	0.0548225
Avg. WR	0.3529789	-0.4127360	1.1186939
Avg. WR2	0.4278046	0.0292879	0.8263213
Avg. PR	0.1311060	-3.1246112	3.3868232
Bronze WR	0.0176470	-0.1566686	0.1919626
Bronze WR2	-0.1029205	-0.2191266	0.0132855
Bronze PR	-0.6465270	-1.3228893	0.0298353
Silv. WR	0.0140570	-0.2223770	0.2504910
Silv. WR2	-0.1708302	-0.3434631	0.0018027
Silv. PR	1.5287636	-0.4620864	3.5196135
Gold WR	-0.3015441	-0.7502623	0.1471742
Gold WR2	-0.1409125	-0.3465935	0.0647684
Gold PR	-1.0689632	-3.2627248	1.1247983
Plat WR	-0.2272664	-0.6224207	0.1678880
Plat WR2	-0.0122763	-0.1301495	0.1055970
Plat PR	-0.7988460	-1.9888325	0.3911406
Diam. WR	0.1054151	-0.2004750	0.4113052
Diam. WR2	0.0210498	-0.1522999	0.1943996
Diam PR	0.0494570	-1.0371827	1.1360967
Mast WR	0.0369549	-0.1313352	0.2052450
Mast WR2	-0.1255782	-0.2703998	0.0192433
Mast PR	0.1548599	-0.2757008	0.5854206
Healer	0.3576286	0.0976392	0.6176181
Tank	0.2122334	-0.0034956	0.4279624

Finally, while we see significance in winrates and pickrates by elo, any marginal significance of forum characteristics is captured by objective and more easily interpreted variables. Don’t waste your time worrying about the Overwatch Forums mob!

Conclusions

Phew, that turned into quite the report. Let’s go through a short summary of the findings. Firstly, it’s not your imagination, the Official Forums are a hotbed for change/buff/nerf posts. The most (in)famous heroes on the Official Forums lately are Genji, Mercy, McCree, Sombra, and Rein. Moreover, for some reason the Forums really love Mercy (and Junkrat?) and they really hate Doomfist. While calls for nerfs/buffs align really well with the various metas we’ve seen- Sigma/Ball, McCree, etc, some heroes are simultaneously called for nerfs/buffs, usually because they’re weak but hated (Genji, Hanzo, Reaper) or begrudgingly necessary (Rein).

In terms of balance, I find evidence that winrates are predictive of balance changes in a nonlinear way. Moreover, being a tank or healer is predictive of getting rebalanced during this study period. Finally, while number of Replies to a post and calls for nerfs are marginally predictive of getting rebalanced, their predictive power goes away after you include winrates and pickrates. Forums topics do not significantly affect balance changes if you just look at the facts!

Until Next Time!