3
Descriptives
How many games:
Note – one 6 p thick game excluded by fiat – the assigned speaker seemed to either really, really not be fluent or not understand the game. They do not provide descriptions and only barely answer when the listeners play 20 questions.
Aimed for 40 each.
Partial means it didn’t go all the rounds, reduced means it has fewer people than it started out with. (Note we also started games even if they were incomplete, so there might be more classification adventures there. )
## # A tibble: 4 × 4
## # Groups: name [4]
## name complete partial reduced
## <chr> <int> <int> <int>
## 1 2_thick 39 3 NA
## 2 2_thin 35 3 NA
## 3 6_thick 17 2 21
## 4 6_thin 25 NA 19Of rounds included
How long full games took
Note that this is inaccurate for games where some players kept the tab open but didn’t respond (but we don’t have an easy way to distinguish “exited” from “inactive, but tab open” in this record).
## # A tibble: 4 × 7
## name games min_time `25th_time` median_time `75th_time` max_time
## <chr> <int> <dbl> <dbl> <dbl> <dbl> <dbl>
## 1 2_thick 39 19 26 33 41 68
## 2 2_thin 35 15 31 40 56 87
## 3 6_thick 38 28 51 58 66 89
## 4 6_thin 44 40 54 63 79 108Exit survey
Most people think they understood directions
## # A tibble: 8 × 3
## # Groups: name [4]
## name correctness n
## <chr> <chr> <int>
## 1 2_thick yes 82
## 2 2_thin yes 73
## 3 2_thin <NA> 1
## 4 6_thick yes 217
## 5 6_thick <NA> 4
## 6 6_thin no 1
## 7 6_thin yes 224
## 8 6_thin <NA> 6Did you think you were playing with humans? Mostly yes.
## # A tibble: 4 × 4
## # Groups: name [4]
## name no yes no_answer
## <chr> <dbl> <dbl> <dbl>
## 1 2_thick 0.134 0.854 0.0122
## 2 2_thin 0.0811 0.892 0.0270
## 3 6_thick 0.0995 0.887 0.0136
## 4 6_thin 0.0693 0.909 0.0216people mostly thought they worked well together
## # A tibble: 4 × 7
## # Groups: name [4]
## name stronglyAgree agree neutral disagree stronglyDisagree `NA`
## <chr> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
## 1 2_thick 0.756 0.195 0.0366 0 0.0122 0
## 2 2_thin 0.649 0.311 0.0135 0.0135 0 0.0135
## 3 6_thick 0.511 0.330 0.104 0.0271 0.00905 0.0181
## 4 6_thin 0.442 0.463 0.0649 0.0130 0.00433 0.0130we also have text answers about whether the chat was useful, time was sufficient, pay was fair, and general feedback which I guess I should read sometime.
Trial internal structure
speaker language
Amount of speaker language (total, not cleaned) before selection
you can kind of see the block structure where there’s discontinuities at 12, etc.
how much said before listener selected
How much did the speaker say before the first listener selected?
this is relatively similar across games/sizes, although there’s some order statistics stuff going on.
how much of the time did the speaker say more things between the n-1 and nth selection?
How much feedback do listeners give (per trial)?
Note, most listeners say nothing most of the time, but we’re not showing all the zeros.
Said before that listener selecting (no chit-chat)
These are faceted by game size and how many listeners had selected so far.
note decline in dot density over trials.
Said after selecting (no chit-chat)
Note that we expect more opportunity for pre-selection talk when there are fewer selections, and more post-selection when there are more selections (b/c of number of listeners eligible).
Amount of listener talking declines, mostly because things turn into more zeros. As expected, in 2p games, there’s little talk post selection (because they’ve clicked it, so it should be ~impossible). In 6p games there is some. This is probably mostly listener-listener as some help explain to laggards.
Sbert results
Across game divergence
Within game divergence
Within game similarity
CAMP graphs
Models
From pre-reg
We will do graphical visualizations for accuracy, time taken and utterance reduction.
Our primary analysis will be a Bayesian mixed model for utterance reduction in the words written by the speaker on each trial: words ~ blockchannelgroup_size + (blockchannelgroup_size|tangram)+ (1|tangram*group)+(block|group)
We will also run a logistic Bayesian mixed model on listener accuracy: listener_accurate ~ blockchannelgroup_size + (blockchannelgroup_size|tangram)+ (1|tangram*group)+(block|group) Additionally, we will analyse the effect of the language. Using SBERT embeddings we will embed the concatenation of everything the speaker said in a trial. We will then take pairwise cosine distances of these to look at the following effects. : (divergence across games) For the same condition & block & tangram, distance between utterances from different games. (divergence within games) For the same condition & block & game, distance between utterances for different tangrams. (convergence within games) For the same condition & game & tangram, distance between utterances from different blocks. We plan to look at the similarities for block 1 with all later blocks; block 6 with all earlier blocks; and block N with block N+1. (Note: these SBERT analyses were done as exploratory analyses on the earlier collected conditions.) Additionally, exclusive to the thin channel parts of this condition, we will analyse the distribution of emoji’s produced as a function of block and its relation to accuracy and speaker utterance length.
Reduction of words
rerun for longer with fuller mixed effects
## Family: gaussian
## Links: mu = identity; sigma = identity
## Formula: words ~ repNum * channel * gameSize + (repNum * channel * gameSize | tangram) + (1 | tangram:gameId) + (repNum | gameId)
## Data: chat_mod (Number of observations: 11285)
## Draws: 4 chains, each with iter = 2000; warmup = 1000; thin = 1;
## total post-warmup draws = 4000
##
## Group-Level Effects:
## ~gameId (Number of levels: 164)
## Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS
## sd(Intercept) 8.80 0.51 7.89 9.85 1.00 695
## sd(repNum) 2.07 0.12 1.84 2.32 1.01 789
## cor(Intercept,repNum) -0.90 0.02 -0.93 -0.86 1.00 907
## Tail_ESS
## sd(Intercept) 1314
## sd(repNum) 1353
## cor(Intercept,repNum) 1603
##
## ~tangram (Number of levels: 12)
## Estimate Est.Error
## sd(Intercept) 2.22 0.49
## sd(repNum) 0.36 0.09
## sd(channelthin) 0.37 0.29
## sd(gameSize6) 1.26 0.38
## sd(repNum:channelthin) 0.17 0.09
## sd(repNum:gameSize6) 0.13 0.09
## sd(channelthin:gameSize6) 0.55 0.42
## sd(repNum:channelthin:gameSize6) 0.35 0.15
## cor(Intercept,repNum) -0.64 0.19
## cor(Intercept,channelthin) -0.04 0.32
## cor(repNum,channelthin) -0.02 0.32
## cor(Intercept,gameSize6) 0.49 0.23
## cor(repNum,gameSize6) -0.45 0.24
## cor(channelthin,gameSize6) 0.07 0.32
## cor(Intercept,repNum:channelthin) 0.36 0.30
## cor(repNum,repNum:channelthin) -0.39 0.30
## cor(channelthin,repNum:channelthin) -0.07 0.33
## cor(gameSize6,repNum:channelthin) 0.32 0.29
## cor(Intercept,repNum:gameSize6) -0.15 0.32
## cor(repNum,repNum:gameSize6) 0.10 0.32
## cor(channelthin,repNum:gameSize6) -0.01 0.33
## cor(gameSize6,repNum:gameSize6) -0.27 0.33
## cor(repNum:channelthin,repNum:gameSize6) -0.08 0.33
## cor(Intercept,channelthin:gameSize6) -0.07 0.32
## cor(repNum,channelthin:gameSize6) 0.08 0.32
## cor(channelthin,channelthin:gameSize6) -0.05 0.34
## cor(gameSize6,channelthin:gameSize6) -0.07 0.32
## cor(repNum:channelthin,channelthin:gameSize6) -0.04 0.33
## cor(repNum:gameSize6,channelthin:gameSize6) 0.02 0.34
## cor(Intercept,repNum:channelthin:gameSize6) 0.33 0.25
## cor(repNum,repNum:channelthin:gameSize6) -0.33 0.26
## cor(channelthin,repNum:channelthin:gameSize6) -0.09 0.33
## cor(gameSize6,repNum:channelthin:gameSize6) 0.08 0.28
## cor(repNum:channelthin,repNum:channelthin:gameSize6) 0.00 0.31
## cor(repNum:gameSize6,repNum:channelthin:gameSize6) -0.09 0.33
## cor(channelthin:gameSize6,repNum:channelthin:gameSize6) -0.26 0.34
## l-95% CI u-95% CI Rhat
## sd(Intercept) 1.48 3.42 1.00
## sd(repNum) 0.22 0.56 1.00
## sd(channelthin) 0.01 1.08 1.00
## sd(gameSize6) 0.64 2.13 1.00
## sd(repNum:channelthin) 0.01 0.37 1.01
## sd(repNum:gameSize6) 0.01 0.36 1.00
## sd(channelthin:gameSize6) 0.02 1.58 1.00
## sd(repNum:channelthin:gameSize6) 0.07 0.66 1.01
## cor(Intercept,repNum) -0.91 -0.19 1.00
## cor(Intercept,channelthin) -0.63 0.58 1.00
## cor(repNum,channelthin) -0.62 0.60 1.00
## cor(Intercept,gameSize6) -0.01 0.85 1.00
## cor(repNum,gameSize6) -0.84 0.07 1.00
## cor(channelthin,gameSize6) -0.57 0.66 1.00
## cor(Intercept,repNum:channelthin) -0.33 0.83 1.00
## cor(repNum,repNum:channelthin) -0.84 0.29 1.00
## cor(channelthin,repNum:channelthin) -0.68 0.56 1.00
## cor(gameSize6,repNum:channelthin) -0.34 0.80 1.00
## cor(Intercept,repNum:gameSize6) -0.72 0.52 1.00
## cor(repNum,repNum:gameSize6) -0.54 0.68 1.00
## cor(channelthin,repNum:gameSize6) -0.66 0.62 1.00
## cor(gameSize6,repNum:gameSize6) -0.81 0.45 1.00
## cor(repNum:channelthin,repNum:gameSize6) -0.67 0.55 1.00
## cor(Intercept,channelthin:gameSize6) -0.67 0.56 1.00
## cor(repNum,channelthin:gameSize6) -0.56 0.68 1.00
## cor(channelthin,channelthin:gameSize6) -0.66 0.59 1.00
## cor(gameSize6,channelthin:gameSize6) -0.67 0.59 1.00
## cor(repNum:channelthin,channelthin:gameSize6) -0.64 0.58 1.00
## cor(repNum:gameSize6,channelthin:gameSize6) -0.62 0.65 1.00
## cor(Intercept,repNum:channelthin:gameSize6) -0.23 0.76 1.00
## cor(repNum,repNum:channelthin:gameSize6) -0.77 0.22 1.00
## cor(channelthin,repNum:channelthin:gameSize6) -0.67 0.56 1.00
## cor(gameSize6,repNum:channelthin:gameSize6) -0.47 0.61 1.00
## cor(repNum:channelthin,repNum:channelthin:gameSize6) -0.56 0.61 1.00
## cor(repNum:gameSize6,repNum:channelthin:gameSize6) -0.69 0.55 1.00
## cor(channelthin:gameSize6,repNum:channelthin:gameSize6) -0.81 0.47 1.00
## Bulk_ESS Tail_ESS
## sd(Intercept) 1586 2154
## sd(repNum) 1990 2976
## sd(channelthin) 1614 2299
## sd(gameSize6) 2193 2724
## sd(repNum:channelthin) 1534 1354
## sd(repNum:gameSize6) 1753 1758
## sd(channelthin:gameSize6) 1737 2543
## sd(repNum:channelthin:gameSize6) 1088 908
## cor(Intercept,repNum) 2290 3116
## cor(Intercept,channelthin) 6081 2951
## cor(repNum,channelthin) 6200 3424
## cor(Intercept,gameSize6) 3211 3273
## cor(repNum,gameSize6) 3431 3266
## cor(channelthin,gameSize6) 2687 3358
## cor(Intercept,repNum:channelthin) 3347 2905
## cor(repNum,repNum:channelthin) 2761 3060
## cor(channelthin,repNum:channelthin) 3889 3500
## cor(gameSize6,repNum:channelthin) 3256 2721
## cor(Intercept,repNum:gameSize6) 5451 2686
## cor(repNum,repNum:gameSize6) 4612 2932
## cor(channelthin,repNum:gameSize6) 3844 3246
## cor(gameSize6,repNum:gameSize6) 3423 2571
## cor(repNum:channelthin,repNum:gameSize6) 3680 3379
## cor(Intercept,channelthin:gameSize6) 5125 3061
## cor(repNum,channelthin:gameSize6) 5314 3283
## cor(channelthin,channelthin:gameSize6) 3935 3415
## cor(gameSize6,channelthin:gameSize6) 3746 3426
## cor(repNum:channelthin,channelthin:gameSize6) 3370 3398
## cor(repNum:gameSize6,channelthin:gameSize6) 2883 3569
## cor(Intercept,repNum:channelthin:gameSize6) 3651 2124
## cor(repNum,repNum:channelthin:gameSize6) 3302 2921
## cor(channelthin,repNum:channelthin:gameSize6) 2593 3277
## cor(gameSize6,repNum:channelthin:gameSize6) 3542 3390
## cor(repNum:channelthin,repNum:channelthin:gameSize6) 2806 3279
## cor(repNum:gameSize6,repNum:channelthin:gameSize6) 2915 3540
## cor(channelthin:gameSize6,repNum:channelthin:gameSize6) 2104 3008
##
## ~tangram:gameId (Number of levels: 1968)
## Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## sd(Intercept) 2.59 0.10 2.40 2.78 1.00 1762 2315
##
## Population-Level Effects:
## Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS
## Intercept 15.22 1.57 12.10 18.29 1.02 278
## repNum -2.33 0.36 -3.03 -1.64 1.01 279
## channelthin 0.56 2.00 -3.26 4.61 1.01 320
## gameSize6 7.15 1.85 3.36 10.67 1.01 329
## repNum:channelthin 0.33 0.48 -0.62 1.27 1.01 325
## repNum:gameSize6 -1.15 0.46 -2.06 -0.24 1.01 333
## channelthin:gameSize6 -2.14 2.57 -7.05 2.99 1.01 403
## repNum:channelthin:gameSize6 0.65 0.65 -0.66 1.91 1.00 444
## Tail_ESS
## Intercept 724
## repNum 866
## channelthin 535
## gameSize6 546
## repNum:channelthin 688
## repNum:gameSize6 567
## channelthin:gameSize6 735
## repNum:channelthin:gameSize6 755
##
## Family Specific Parameters:
## Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## sigma 6.61 0.05 6.52 6.71 1.00 4160 3078
##
## Draws were sampled using sampling(NUTS). For each parameter, Bulk_ESS
## and Tail_ESS are effective sample size measures, and Rhat is the potential
## scale reduction factor on split chains (at convergence, Rhat = 1).Accuracy model
## Family: bernoulli
## Links: mu = logit
## Formula: correct.num ~ repNum * channel * gameSize + (repNum * channel * gameSize | tangram) + (1 | tangram:gameId) + (repNum | gameId)
## Data: acc_data (Number of observations: 29356)
## Draws: 4 chains, each with iter = 2000; warmup = 1000; thin = 1;
## total post-warmup draws = 4000
##
## Group-Level Effects:
## ~gameId (Number of levels: 164)
## Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS
## sd(Intercept) 0.45 0.04 0.37 0.54 1.00 1960
## sd(repNum) 0.25 0.02 0.21 0.29 1.00 1451
## cor(Intercept,repNum) 0.14 0.13 -0.10 0.39 1.01 635
## Tail_ESS
## sd(Intercept) 2739
## sd(repNum) 2379
## cor(Intercept,repNum) 1176
##
## ~tangram (Number of levels: 12)
## Estimate Est.Error
## sd(Intercept) 0.43 0.05
## sd(repNum) 0.03 0.02
## sd(channelthin) 0.09 0.06
## sd(gameSize6) 0.06 0.04
## sd(repNum:channelthin) 0.04 0.03
## sd(repNum:gameSize6) 0.03 0.02
## sd(channelthin:gameSize6) 0.08 0.05
## sd(repNum:channelthin:gameSize6) 0.05 0.03
## cor(Intercept,repNum) -0.21 0.30
## cor(Intercept,channelthin) 0.28 0.29
## cor(repNum,channelthin) -0.05 0.33
## cor(Intercept,gameSize6) 0.04 0.32
## cor(repNum,gameSize6) -0.02 0.34
## cor(channelthin,gameSize6) 0.04 0.33
## cor(Intercept,repNum:channelthin) 0.15 0.28
## cor(repNum,repNum:channelthin) -0.07 0.34
## cor(channelthin,repNum:channelthin) 0.00 0.32
## cor(gameSize6,repNum:channelthin) 0.05 0.33
## cor(Intercept,repNum:gameSize6) -0.18 0.30
## cor(repNum,repNum:gameSize6) -0.04 0.33
## cor(channelthin,repNum:gameSize6) -0.03 0.33
## cor(gameSize6,repNum:gameSize6) -0.04 0.34
## cor(repNum:channelthin,repNum:gameSize6) 0.01 0.32
## cor(Intercept,channelthin:gameSize6) 0.18 0.31
## cor(repNum,channelthin:gameSize6) -0.03 0.33
## cor(channelthin,channelthin:gameSize6) 0.00 0.33
## cor(gameSize6,channelthin:gameSize6) -0.03 0.34
## cor(repNum:channelthin,channelthin:gameSize6) -0.03 0.33
## cor(repNum:gameSize6,channelthin:gameSize6) -0.05 0.33
## cor(Intercept,repNum:channelthin:gameSize6) 0.23 0.27
## cor(repNum,repNum:channelthin:gameSize6) -0.06 0.33
## cor(channelthin,repNum:channelthin:gameSize6) -0.01 0.33
## cor(gameSize6,repNum:channelthin:gameSize6) -0.01 0.33
## cor(repNum:channelthin,repNum:channelthin:gameSize6) -0.03 0.33
## cor(repNum:gameSize6,repNum:channelthin:gameSize6) -0.10 0.34
## cor(channelthin:gameSize6,repNum:channelthin:gameSize6) -0.06 0.33
## l-95% CI u-95% CI Rhat
## sd(Intercept) 0.34 0.53 1.00
## sd(repNum) 0.00 0.08 1.00
## sd(channelthin) 0.01 0.22 1.00
## sd(gameSize6) 0.00 0.17 1.00
## sd(repNum:channelthin) 0.00 0.10 1.00
## sd(repNum:gameSize6) 0.00 0.08 1.00
## sd(channelthin:gameSize6) 0.01 0.21 1.00
## sd(repNum:channelthin:gameSize6) 0.00 0.12 1.00
## cor(Intercept,repNum) -0.72 0.42 1.00
## cor(Intercept,channelthin) -0.37 0.76 1.00
## cor(repNum,channelthin) -0.65 0.59 1.00
## cor(Intercept,gameSize6) -0.58 0.64 1.00
## cor(repNum,gameSize6) -0.65 0.63 1.00
## cor(channelthin,gameSize6) -0.62 0.66 1.00
## cor(Intercept,repNum:channelthin) -0.44 0.65 1.00
## cor(repNum,repNum:channelthin) -0.68 0.60 1.00
## cor(channelthin,repNum:channelthin) -0.61 0.62 1.00
## cor(gameSize6,repNum:channelthin) -0.58 0.65 1.00
## cor(Intercept,repNum:gameSize6) -0.70 0.46 1.00
## cor(repNum,repNum:gameSize6) -0.65 0.58 1.00
## cor(channelthin,repNum:gameSize6) -0.65 0.61 1.00
## cor(gameSize6,repNum:gameSize6) -0.66 0.62 1.00
## cor(repNum:channelthin,repNum:gameSize6) -0.60 0.62 1.00
## cor(Intercept,channelthin:gameSize6) -0.47 0.71 1.00
## cor(repNum,channelthin:gameSize6) -0.65 0.62 1.00
## cor(channelthin,channelthin:gameSize6) -0.63 0.63 1.00
## cor(gameSize6,channelthin:gameSize6) -0.66 0.61 1.00
## cor(repNum:channelthin,channelthin:gameSize6) -0.65 0.61 1.00
## cor(repNum:gameSize6,channelthin:gameSize6) -0.66 0.60 1.00
## cor(Intercept,repNum:channelthin:gameSize6) -0.36 0.69 1.00
## cor(repNum,repNum:channelthin:gameSize6) -0.66 0.58 1.00
## cor(channelthin,repNum:channelthin:gameSize6) -0.62 0.61 1.00
## cor(gameSize6,repNum:channelthin:gameSize6) -0.65 0.62 1.00
## cor(repNum:channelthin,repNum:channelthin:gameSize6) -0.63 0.64 1.00
## cor(repNum:gameSize6,repNum:channelthin:gameSize6) -0.71 0.57 1.00
## cor(channelthin:gameSize6,repNum:channelthin:gameSize6) -0.66 0.57 1.00
## Bulk_ESS Tail_ESS
## sd(Intercept) 2768 2530
## sd(repNum) 1843 2202
## sd(channelthin) 1601 1449
## sd(gameSize6) 3132 2570
## sd(repNum:channelthin) 1827 1802
## sd(repNum:gameSize6) 1945 2270
## sd(channelthin:gameSize6) 2071 2276
## sd(repNum:channelthin:gameSize6) 1430 1709
## cor(Intercept,repNum) 4031 2607
## cor(Intercept,channelthin) 3482 2702
## cor(repNum,channelthin) 4365 3243
## cor(Intercept,gameSize6) 6329 3067
## cor(repNum,gameSize6) 4908 3231
## cor(channelthin,gameSize6) 4039 3146
## cor(Intercept,repNum:channelthin) 3849 2846
## cor(repNum,repNum:channelthin) 4195 3161
## cor(channelthin,repNum:channelthin) 3292 3143
## cor(gameSize6,repNum:channelthin) 3398 3362
## cor(Intercept,repNum:gameSize6) 3983 2785
## cor(repNum,repNum:gameSize6) 4911 3338
## cor(channelthin,repNum:gameSize6) 3763 2904
## cor(gameSize6,repNum:gameSize6) 3434 3265
## cor(repNum:channelthin,repNum:gameSize6) 3441 3227
## cor(Intercept,channelthin:gameSize6) 4329 3090
## cor(repNum,channelthin:gameSize6) 4683 3022
## cor(channelthin,channelthin:gameSize6) 3851 3155
## cor(gameSize6,channelthin:gameSize6) 3526 3325
## cor(repNum:channelthin,channelthin:gameSize6) 3133 3056
## cor(repNum:gameSize6,channelthin:gameSize6) 3098 3242
## cor(Intercept,repNum:channelthin:gameSize6) 3764 2768
## cor(repNum,repNum:channelthin:gameSize6) 3339 3268
## cor(channelthin,repNum:channelthin:gameSize6) 3195 3357
## cor(gameSize6,repNum:channelthin:gameSize6) 2814 3444
## cor(repNum:channelthin,repNum:channelthin:gameSize6) 2776 3153
## cor(repNum:gameSize6,repNum:channelthin:gameSize6) 2949 3474
## cor(channelthin:gameSize6,repNum:channelthin:gameSize6) 2711 3438
##
## ~tangram:gameId (Number of levels: 1968)
## Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## sd(Intercept) 0.70 0.03 0.64 0.77 1.00 1692 2761
##
## Population-Level Effects:
## Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS
## Intercept 1.79 0.17 1.45 2.13 1.00 1653
## repNum 0.55 0.06 0.42 0.67 1.00 1658
## channelthin -0.34 0.17 -0.66 -0.01 1.00 2133
## gameSize6 -0.44 0.15 -0.73 -0.16 1.00 2207
## repNum:channelthin -0.09 0.09 -0.26 0.07 1.00 1722
## repNum:gameSize6 -0.17 0.08 -0.33 -0.02 1.00 1315
## channelthin:gameSize6 0.22 0.19 -0.16 0.60 1.00 2243
## repNum:channelthin:gameSize6 -0.02 0.10 -0.23 0.18 1.00 1384
## Tail_ESS
## Intercept 2551
## repNum 2475
## channelthin 2722
## gameSize6 2163
## repNum:channelthin 2356
## repNum:gameSize6 2071
## channelthin:gameSize6 2750
## repNum:channelthin:gameSize6 2104
##
## Draws were sampled using sampling(NUTS). For each parameter, Bulk_ESS
## and Tail_ESS are effective sample size measures, and Rhat is the potential
## scale reduction factor on split chains (at convergence, Rhat = 1).Sbert models
## Family: gaussian
## Links: mu = identity; sigma = identity
## Formula: sim ~ repNum * channel * gameSize + (1 | tangram)
## Data: game_divergence (Number of observations: 217188)
## Draws: 4 chains, each with iter = 2000; warmup = 1000; thin = 1;
## total post-warmup draws = 4000
##
## Group-Level Effects:
## ~tangram (Number of levels: 12)
## Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## sd(Intercept) 0.07 0.01 0.05 0.10 1.01 509 606
##
## Population-Level Effects:
## Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS
## Intercept 0.41 0.02 0.37 0.45 1.01 421
## repNum -0.02 0.00 -0.02 -0.02 1.00 2997
## channelthin 0.01 0.00 0.01 0.02 1.00 1981
## gameSize6 0.05 0.00 0.05 0.05 1.00 2375
## repNum:channelthin 0.00 0.00 0.00 0.00 1.00 2356
## repNum:gameSize6 -0.01 0.00 -0.01 -0.01 1.00 2648
## channelthin:gameSize6 -0.03 0.00 -0.04 -0.02 1.00 1910
## repNum:channelthin:gameSize6 0.02 0.00 0.02 0.02 1.00 2244
## Tail_ESS
## Intercept 493
## repNum 3086
## channelthin 2192
## gameSize6 2333
## repNum:channelthin 3064
## repNum:gameSize6 2757
## channelthin:gameSize6 1924
## repNum:channelthin:gameSize6 2501
##
## Family Specific Parameters:
## Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## sigma 0.19 0.00 0.19 0.19 1.00 3130 2200
##
## Draws were sampled using sampling(NUTS). For each parameter, Bulk_ESS
## and Tail_ESS are effective sample size measures, and Rhat is the potential
## scale reduction factor on split chains (at convergence, Rhat = 1).## Family: gaussian
## Links: mu = identity; sigma = identity
## Formula: sim ~ repNum * channel * gameSize + (1 | gameId)
## Data: tangram_distinctive (Number of observations: 62106)
## Draws: 4 chains, each with iter = 2000; warmup = 1000; thin = 1;
## total post-warmup draws = 4000
##
## Group-Level Effects:
## ~gameId (Number of levels: 164)
## Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## sd(Intercept) 0.09 0.01 0.08 0.10 1.05 113 230
##
## Population-Level Effects:
## Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS
## Intercept 0.38 0.01 0.35 0.41 1.06 47
## repNum -0.03 0.00 -0.03 -0.03 1.00 4235
## channelthin 0.04 0.02 -0.01 0.07 1.06 64
## gameSize6 0.07 0.02 0.03 0.11 1.10 31
## repNum:channelthin -0.00 0.00 -0.00 0.00 1.00 4317
## repNum:gameSize6 -0.01 0.00 -0.01 -0.01 1.00 4361
## channelthin:gameSize6 -0.06 0.03 -0.10 -0.00 1.03 72
## repNum:channelthin:gameSize6 0.01 0.00 0.01 0.01 1.00 4252
## Tail_ESS
## Intercept 107
## repNum 3380
## channelthin 109
## gameSize6 48
## repNum:channelthin 3372
## repNum:gameSize6 3649
## channelthin:gameSize6 153
## repNum:channelthin:gameSize6 3243
##
## Family Specific Parameters:
## Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## sigma 0.18 0.00 0.18 0.18 1.00 4366 3085
##
## Draws were sampled using sampling(NUTS). For each parameter, Bulk_ESS
## and Tail_ESS are effective sample size measures, and Rhat is the potential
## scale reduction factor on split chains (at convergence, Rhat = 1).## Family: gaussian
## Links: mu = identity; sigma = identity
## Formula: sim ~ earlier * channel * gameSize * samespeaker
## Data: to_last (Number of observations: 9252)
## Draws: 4 chains, each with iter = 2000; warmup = 1000; thin = 1;
## total post-warmup draws = 4000
##
## Population-Level Effects:
## Estimate Est.Error
## Intercept 0.53 0.10
## earlier 0.06 0.09
## channelthin -0.01 0.10
## gameSize6 -0.06 0.09
## samespeakersame_speaker 0.06 0.10
## earlier:channelthin -0.01 0.09
## earlier:gameSize6 -0.01 0.09
## channelthin:gameSize6 0.04 0.09
## earlier:samespeakersame_speaker 0.02 0.09
## channelthin:samespeakersame_speaker 0.01 0.10
## gameSize6:samespeakersame_speaker -0.01 0.09
## earlier:channelthin:gameSize6 -0.02 0.09
## earlier:channelthin:samespeakersame_speaker -0.01 0.09
## earlier:gameSize6:samespeakersame_speaker 0.01 0.09
## channelthin:gameSize6:samespeakersame_speaker 0.02 0.10
## earlier:channelthin:gameSize6:samespeakersame_speaker -0.00 0.09
## l-95% CI u-95% CI Rhat
## Intercept 0.34 0.71 1.01
## earlier -0.11 0.25 1.01
## channelthin -0.19 0.18 1.01
## gameSize6 -0.24 0.12 1.00
## samespeakersame_speaker -0.13 0.25 1.01
## earlier:channelthin -0.19 0.17 1.01
## earlier:gameSize6 -0.19 0.17 1.00
## channelthin:gameSize6 -0.14 0.23 1.00
## earlier:samespeakersame_speaker -0.17 0.19 1.01
## channelthin:samespeakersame_speaker -0.19 0.20 1.01
## gameSize6:samespeakersame_speaker -0.20 0.17 1.01
## earlier:channelthin:gameSize6 -0.19 0.17 1.01
## earlier:channelthin:samespeakersame_speaker -0.19 0.17 1.01
## earlier:gameSize6:samespeakersame_speaker -0.16 0.20 1.00
## channelthin:gameSize6:samespeakersame_speaker -0.17 0.21 1.00
## earlier:channelthin:gameSize6:samespeakersame_speaker -0.19 0.17 1.00
## Bulk_ESS Tail_ESS
## Intercept 1077 1334
## earlier 875 1057
## channelthin 1083 1322
## gameSize6 1054 1161
## samespeakersame_speaker 1073 1279
## earlier:channelthin 874 1072
## earlier:gameSize6 887 1059
## channelthin:gameSize6 1057 1270
## earlier:samespeakersame_speaker 875 1024
## channelthin:samespeakersame_speaker 1110 1335
## gameSize6:samespeakersame_speaker 1068 1227
## earlier:channelthin:gameSize6 886 1051
## earlier:channelthin:samespeakersame_speaker 882 1057
## earlier:gameSize6:samespeakersame_speaker 890 1058
## channelthin:gameSize6:samespeakersame_speaker 1116 1333
## earlier:channelthin:gameSize6:samespeakersame_speaker 898 1118
##
## Family Specific Parameters:
## Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## sigma 0.22 0.00 0.22 0.22 1.00 2562 2109
##
## Draws were sampled using sampling(NUTS). For each parameter, Bulk_ESS
## and Tail_ESS are effective sample size measures, and Rhat is the potential
## scale reduction factor on split chains (at convergence, Rhat = 1).tSNE
PoS
Sloppy first attempt – we group by trial & speaker, summarize & pct-ize
ToDo the naive word count and the PoS sum are not always equal (off in both direction sometimes) – some punctuation normalization may help?
Following Robert’s analysis. First graph is raw-averages, second is grand-average (first do pcts for each speaker-trial, then do grand avg)
Diff between them shows that where there is more reduction, there is more noun-yness.
Emoji analysis
What fraction of listener-trials produce any emoji? (Grouped by game, but measured per listener)
Of emoji that are used, what’s the distribution?
A few people sent many copies of the same emoji over the course of a single trial which could sway things (see above graph), so we check by binarizing by did that listener produce that emoji on that trial and then summarizing.
Throw all the models at this
Note, we’re being quick and dirty here with lmer.
log words
##
## Call:
## lm(formula = log_words ~ repNum * channel * gameSize, data = chat_mod)
##
## Residuals:
## Min 1Q Median 3Q Max
## -2.94570 -0.46080 0.08428 0.55429 2.82290
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 2.50986 0.02619 95.825 < 2e-16 ***
## repNum -0.25794 0.00874 -29.513 < 2e-16 ***
## channelthin 0.04163 0.03832 1.086 0.2774
## gameSize6 0.43584 0.03775 11.546 < 2e-16 ***
## repNum:channelthin 0.05267 0.01274 4.135 3.58e-05 ***
## repNum:gameSize6 -0.03958 0.01254 -3.157 0.0016 **
## channelthin:gameSize6 -0.13153 0.05356 -2.456 0.0141 *
## repNum:channelthin:gameSize6 0.05263 0.01773 2.968 0.0030 **
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 0.8035 on 11277 degrees of freedom
## Multiple R-squared: 0.2416, Adjusted R-squared: 0.2411
## F-statistic: 513.2 on 7 and 11277 DF, p-value: < 2.2e-16log block
Reduction
##
## Call:
## lm(formula = words ~ log_block * channel * gameSize, data = chat_mod)
##
## Residuals:
## Min 1Q Median 3Q Max
## -23.665 -5.642 -1.910 3.354 101.599
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 16.5757 0.3510 47.220 < 2e-16 ***
## log_block -6.5671 0.2829 -23.212 < 2e-16 ***
## channelthin 0.4013 0.5120 0.784 0.4332
## gameSize6 8.0894 0.5086 15.904 < 2e-16 ***
## log_block:channelthin 0.8041 0.4116 1.953 0.0508 .
## log_block:gameSize6 -3.4921 0.4079 -8.562 < 2e-16 ***
## channelthin:gameSize6 -3.5472 0.7199 -4.928 8.45e-07 ***
## log_block:channelthin:gameSize6 2.7758 0.5761 4.818 1.47e-06 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 9.264 on 11277 degrees of freedom
## Multiple R-squared: 0.2196, Adjusted R-squared: 0.2192
## F-statistic: 453.5 on 7 and 11277 DF, p-value: < 2.2e-16Why not try log-log ?
##
## Call:
## lm(formula = log_words ~ log_block * channel * gameSize, data = chat_mod)
##
## Residuals:
## Min 1Q Median 3Q Max
## -3.1208 -0.4759 0.0837 0.5546 2.6942
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 2.67583 0.03048 87.781 < 2e-16 ***
## log_block -0.74058 0.02457 -30.144 < 2e-16 ***
## channelthin -0.01161 0.04446 -0.261 0.79410
## gameSize6 0.44497 0.04417 10.074 < 2e-16 ***
## log_block:channelthin 0.16735 0.03574 4.682 2.88e-06 ***
## log_block:gameSize6 -0.09510 0.03542 -2.685 0.00726 **
## channelthin:gameSize6 -0.16344 0.06251 -2.615 0.00895 **
## log_block:channelthin:gameSize6 0.14828 0.05003 2.964 0.00304 **
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 0.8045 on 11277 degrees of freedom
## Multiple R-squared: 0.2397, Adjusted R-squared: 0.2393
## F-statistic: 508 on 7 and 11277 DF, p-value: < 2.2e-16Accuracy
Note that this accuracy is of players who clicked something!
##
## Call:
## glm(formula = correct.num ~ log_block * channel * gameSize, family = "binomial",
## data = acc_data)
##
## Deviance Residuals:
## Min 1Q Median 3Q Max
## -2.5469 0.4152 0.5574 0.6414 0.8008
##
## Coefficients:
## Estimate Std. Error z value Pr(>|z|)
## (Intercept) 1.4332 0.1038 13.812 < 2e-16 ***
## log_block 0.9881 0.1040 9.504 < 2e-16 ***
## channelthin -0.3491 0.1426 -2.448 0.014378 *
## gameSize6 -0.3778 0.1134 -3.331 0.000864 ***
## log_block:channelthin -0.1656 0.1384 -1.196 0.231589
## log_block:gameSize6 -0.3804 0.1116 -3.407 0.000657 ***
## channelthin:gameSize6 0.2665 0.1556 1.713 0.086756 .
## log_block:channelthin:gameSize6 0.0103 0.1487 0.069 0.944804
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## (Dispersion parameter for binomial family taken to be 1)
##
## Null deviance: 26173 on 29355 degrees of freedom
## Residual deviance: 25396 on 29348 degrees of freedom
## AIC: 25412
##
## Number of Fisher Scoring iterations: 5