Distribution on Drop Frequencies

When analyzing the frequency of each of the nine types of item values, it does appear as if the regular drops are occurring more often for lower-valued items and the scavenger drops are outperforming on middle-valued items (as would be expected). However at least visually, the differences do not appear to be as stark as the cost of the two drops would suggest. The premium drops are adding some value to its users, but it does not look like it’s a major one. Furthermore, the regular drops are actually slightly outperforming the scavenger drops on all three of the highest value items (albeit with relatively small samples compared to the other more frequently occurring items).

Expected Frequencies vs. Observed Frequencies for Drop Types

The difference really stands out when looking at the expected frequencies of each item versus their observed frequency in the game. Generally speaking, the scavenger items appear to be occurring close to expectation across all nine items. However, there is clearly a disproportionate amount of rare and ultra items occurring on regular drops, which are coming at the expense of fewer of the lower priced common and uncommon items.

To actually quantify this, we can see that the 1,000 and 2,500 valued regular drops are occurring at 5.76% and 22.33% respectively, which are 11.9% and 6.5% under their respective expectations. The fewer number of occurrences for these lower-valued items are very clearly going towards the 10,000 and 25,000 valued regular drops, which are performing 10.5% and 8.3% over their respective expectations.

Conclusion

Ultimately it seems not as if there are any internal issues with the scavenger drops, but rather that they just look like they’re not of value to the players because they aren’t that much more likely to produce mid-valued items than the regular drops. On one hand, the players’ complaints are invalid because the scavenger drops are performing according to their expected frequencies (which were designed to increase their likelihood of better drops). On the other hand, the game is dropping too many 10,000 and 25,000 items on the regular drops, which is causing the scavenger drops to seem like a poor value by comparison (why purchase the upgrade if the free items are helping them progress at a similar pace?). In that sense, the players’ complaints are valid, but they are directed at the wrong facet of the game. The problem appears to be with the regular drops, not the scavenger drops.

---

Total Drops Performed by Users in Sample

In total, there were over 142,000 unique users in the sample. The users were grouped into segments based on their total number of item drops (both regular and scavenger) during the three day period. The distribution is charted below.

As we can see, the vast majority of users (96.9% to be exact) performed 1-3 drops in total. Another 2.7% of users performed exactly 4 drops. That means that only 0.4% (or approximately 1 out of every 250 users) performed more than 4 drops over the sample. So it stands within reason that achieving more than 4 loot drops is either some sort of a pain point in the game’s progression or is an indication that the drops are not actually all that useful because players are just willingly not taking them.

The Rational Game Assumption (RGA)

In determining if the loot drops are useful for game play, we would have to assume what I am calling “The Rational Game Assumption” (RGA for short). Logically speaking, if we were to assume that the drops are essential to a player’s progress through the game, then we would also assume that players with fewer numbers of total drops would stand to gain the most from a singular loot drop. The reason being that we would assume a positive correlation between total drops taken throughout the game with total coins collected. Under the RGA, players with fewer numbers of drops taken would typically have smaller coin balances than players who had taken many drops, most likely because they are just not as far along in the game. Therefore, players with fewer drops taken would be more prone to variation in their coin balance from a single loot drop.

The percentage change of every loot drop recorded against the players’ coin balances at the time of the drop were calculated. This was then plotted against the same player segments created earlier based on their total drops taken throughout the sample. When we look at the density function of each loot drop’s percentage increase to a user’s coin balance, we see there are extremely similar functions for all user groups. Players with four or fewer total drops are generally gaining about the same from a single loot drop as players with dozens and even hundreds of total loot drops. It seems that regardless of how many drops a player has taken throughout the game, each successive loot drop is usually only adding between 0 - 5% to that player’s coin balance. This would be a violation of the RGA that the players with fewer drops would have smaller coin balances, which would be one indicator that the loot drop function may not be too important to players.

One way to look at seeing if the percentage increases on drops invalidating the RGA is true would be to look at the actual coin balances of the players at the time of their drop transactions. The following graph is a box plot of the coin balance of the player at the time of each loot drop recorded (regardless of size of the drop or type), broken down by the player segments. Under the RGA where we assume that the loot drops are important to the player’s progression in their coin balance throughout the game, this graph should generally be sloping upwards (i.e. more loot drops taken throughout the game equating to higher coin balances later in the game).

However, notice that the chart shows that there’s almost no pattern with the coin balances at each segment whatsoever. The distribution starts to slope upward from users with 1 total drop to 2 total drops to 3, but then for no real reason comes back down for users with 4 drops and continues sloping down to the segments with users who have taken 5-10 and 11-20 total drops. The only segment that seemingly shows any type of discernible pattern of total drops meaning higher coin balances are those in the 101-1000 drop group, by basis of its higher median coin balance and larger upper bound on its bar. However, more than half the users in that segment (those displayed in the box under the median bar) are still making loot drops with the same coin balances as users in the 1 - 3 loot drop segments. So it’s difficult to even call anything in this segment a pattern demonstrating that higher coin balances correlating to more total loot drops taken. Instead it seems that players are collecting coins at similar rates regardless of how many drops they have taken prior.

Conclusion

There was no pattern found in this data that suggests that the RGA is validated. In other words, the loot drops do not in any way appear to be essential towards game progression. Most of the time the players taking loot drops are only adding less than 5 percent to their coin balance per drop taken. These small gains do not seem to entice players enough to want to take more drops, as the vast majority of all players have taken 1 - 3 total drops in the game yet are still achieving median coin balances akin to players who have taken up to 100 total drops. This indicates that players are seemingly able to find enough coins throughout the game to where the loot drops, even just the regular ones, don’t seem as important.

---

Client Version 573

The first was that the reading for the subtype of each drop taken had 13 missing values in the entire data set. While this is an extremely small percentage, it does affect the data quality even just a little bit. However, a further inspection of the 13 rows with the missing subtypes shows a clear pattern.

It is clear that all 13 missing subtypes are coming from gamers playing on Client Version 573. Furthermore, there were no other loot drops recorded playing on 573 other than these 13 transactions. All other client versions always recorded a drop as either regular or scavenger. So developers will need to inspect why users on 573 are not able to record the type of drop they are taking in the game.

The Drop Glitch?

Earlier in this report, it was noted that over 99% of all users in the sample performed 4 drops or less. While many users did progress past that level, one user in particular stood out amongst all the rest. Some users had performed dozens, a few had performed in the hundreds, but then there was one user who performed 7,081 drops in the 3-day sample.

In the charts below, the left chart is the number of drops performed by all users not including the outlier 7,000 drop player. The chart on the right shows the drops performed by all users, including the 7,000 drop user (called “the anomaly” for short). Notice on the left chart that the majority of users clustered around 1-4 drops with a few trickling up towards 100 and one outlier around 660. However when that same distribution (the points in blue) was looked at compared to the 7,000 drop anomaly (the big red point on the chart), they all kind of cluster together. Even the 660 drop user looks kind of normal by comparison. Clearly this is an indication of some sort of major outlier.

At first glance, there does not appear to be any noticeable reason as to why this one user was able to perform so many drops. As far as I can tell, the two most likely reasons would either be:

1. One player has found a glitch in the game that allows him to exploit the loot drop system.

2. There is an issue with the data where it is actually multiple players (probably thousands of players) accidentally being read as playing on the same device ID.

It seems unlikely that only one person out of over 140,000 users would identify such a huge glitch, and then perhaps made more unlikely that one person would also choose not to share that information with anybody else. For that reason, I would tend to lean in the direction of a data quality issue that may perhaps be causing a bunch of players to be read under the same ID. That may even be somewhat validated upon closer inspection of the anomaly’s ID.

The anomaly ID recorded at least one drop on 13 different client versions. Furthermore, we can also see that the ID not only had drops performed on many different clients but that the drops were being performed across all three days in the sample as well. It seems unlikely that one person was playing the game on 13 different clients consistently across three days. That seems like a far more likely explanation that there is some sort of data issue where multiple users are being read onto the same device ID, and I would recommend further examination by the programming team.

---