Task 1

First I made the density plots only using players with at least 80 responses. I did this both when keeping the (new_prob / old_prob) and not.

First is without (new_prob / old_prob) in the ratio calculation:

p1 <- readPNG("comparing_all_and_frequent_players_without_old_prob.png", native = TRUE, info = TRUE)
grid.raster(p1)

Now the plost with (new_prob / old_prob) in the ratio calculation:

p2 <- readPNG("Comparing_full_and_filtered_RMSE_old_prob.png", native = TRUE, info = TRUE)
grid.raster(p2)

Looks like in both cases the two plots are almost identical, but I have double checked and the plots on the right are definitely based on only those players who have above 80 responses. In both cases this brought down the number of rows in the main data from 4 million to around 3 million.

Task 2

Next I checked the accuracy of responses of players who are in the top bin for the prediction (above .975). I did this both with (new_prob / old_prob) and without, but the results were very similar so I’m only including one the second case.

main_df <- read.csv("main_df.csv")


task_2_df_1_without_old_prob<- main_df %>% dplyr::mutate(reponse_accuracy = ifelse(outcome == 1, 1, 0)) %>% 
  group_by(Player_id) %>% 
  mutate(max_pred = max(thbe_p_bin)) %>% 
  filter(max_pred == 21) %>% 
  summarise(mean_resp_acc = mean(reponse_accuracy))


mean(task_2_df_1_without_old_prob$mean_resp_acc)
## [1] 0.6924005

So the mean response accuracy of the players in the top bin is around .7, which is maybe not as high as we thought it would be.

task_2_df_2_without_old_prob <- main_df %>% 
  group_by(Player_id) %>% 
  mutate(max_pred = max(thbe_p_bin)) %>% 
  filter(max_pred == 21) %>% 
  mutate(item_beta = current_item_pv/80)

mean(task_2_df_2_without_old_prob$item_beta)
## [1] 0.1952999

I also calculated the average item configuration for items that are matched to the players in the top bin. We see that average is quite low at .2.

But then I was also interested in looking at players that have consistently very high green balls and what kind of items they get. So I selected players that have above 76 green balls for at least 200 games.

high_player_ids <- main_df %>% group_by(Player_id) %>% 
  count(above_76 = current_player_pv > 76) %>% 
  filter(above_76 == TRUE & n > 200) 
  
main_df %>%  filter(Player_id %in% high_player_ids$Player_id & current_player_pv > 76) %>% 
  summarise(mean(current_item_pv))
##   mean(current_item_pv)
## 1              26.65836
26/80
## [1] 0.325

The average green balls of items that are matched to players above 76 green balls is 26, which divided by 80 is .325. That is definitely lower then it should be, but it makes sense here considering the density plot.

Task 3

Finally I plotted the variances of items, both with (new_prob / old_prob) and without.

First is with:

p3 <- readPNG("item_variance_plot_with_old_prob.png", native = TRUE, info = TRUE)
grid.raster(p3)

And without:

p3 <- readPNG("Item_variances_without_old_prob.png", native = TRUE, info = TRUE)
grid.raster(p3)

The variance increase up to the first 500,000 games as item configurations get adjust but then look pretty stable in both cases. So perhaps this is not enough to explain the skewness of the density plot?