Natural Stories model analyses
Prep
Maze Prep
Maze GAMs
#no hierarchical, has freq x len interaction
formula_interact <- rt ~ s(surprisal, bs="cr", k=20)+
ti(freq, bs="cr") +
ti(len, bs="cr")+
ti(freq,len, bs="cr")+
s(prev_surp, bs="cr", k=20)+
ti(prev_freq, bs="cr")+
ti(prev_len, bs="cr")+
ti(prev_freq, prev_len, bs="cr")
# no hierarchical, NO freq x len interaction
formula_no_interact <- rt ~ s(surprisal, bs="cr", k=20)+
ti(freq, bs="cr") +
ti(len, bs="cr")+
s(prev_surp, bs="cr", k=20)+
ti(prev_freq, bs="cr")+
ti(prev_len, bs="cr")All of this is on by-item mean data.
## Analysis of Deviance Table
##
## Model 1: rt ~ s(surprisal, bs = "cr", k = 20) + ti(freq, bs = "cr") +
## ti(len, bs = "cr") + ti(freq, len, bs = "cr") + s(prev_surp,
## bs = "cr", k = 20) + ti(prev_freq, bs = "cr") + ti(prev_len,
## bs = "cr") + ti(prev_freq, prev_len, bs = "cr")
## Model 2: rt ~ s(surprisal, bs = "cr", k = 20) + ti(freq, bs = "cr") +
## ti(len, bs = "cr") + s(prev_surp, bs = "cr", k = 20) + ti(prev_freq,
## bs = "cr") + ti(prev_len, bs = "cr")
## Resid. Df Resid. Dev Df Deviance Pr(>Chi)
## 1 6324.8 355096280
## 2 6337.0 357451934 -12.278 -2355654 4.008e-05 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1## Analysis of Deviance Table
##
## Model 1: rt ~ s(surprisal, bs = "cr", k = 20) + ti(freq, bs = "cr") +
## ti(len, bs = "cr") + ti(freq, len, bs = "cr") + s(prev_surp,
## bs = "cr", k = 20) + ti(prev_freq, bs = "cr") + ti(prev_len,
## bs = "cr") + ti(prev_freq, prev_len, bs = "cr")
## Model 2: rt ~ s(surprisal, bs = "cr", k = 20) + ti(freq, bs = "cr") +
## ti(len, bs = "cr") + s(prev_surp, bs = "cr", k = 20) + ti(prev_freq,
## bs = "cr") + ti(prev_len, bs = "cr")
## Resid. Df Resid. Dev Df Deviance Pr(>Chi)
## 1 6328.9 326754835
## 2 6339.6 328514957 -10.785 -1760122 0.0002991 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1## Analysis of Deviance Table
##
## Model 1: rt ~ s(surprisal, bs = "cr", k = 20) + ti(freq, bs = "cr") +
## ti(len, bs = "cr") + ti(freq, len, bs = "cr") + s(prev_surp,
## bs = "cr", k = 20) + ti(prev_freq, bs = "cr") + ti(prev_len,
## bs = "cr") + ti(prev_freq, prev_len, bs = "cr")
## Model 2: rt ~ s(surprisal, bs = "cr", k = 20) + ti(freq, bs = "cr") +
## ti(len, bs = "cr") + s(prev_surp, bs = "cr", k = 20) + ti(prev_freq,
## bs = "cr") + ti(prev_len, bs = "cr")
## Resid. Df Resid. Dev Df Deviance Pr(>Chi)
## 1 6326.2 338211260
## 2 6340.0 340297053 -13.794 -2085793 0.0003155 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1## Analysis of Deviance Table
##
## Model 1: rt ~ s(surprisal, bs = "cr", k = 20) + ti(freq, bs = "cr") +
## ti(len, bs = "cr") + ti(freq, len, bs = "cr") + s(prev_surp,
## bs = "cr", k = 20) + ti(prev_freq, bs = "cr") + ti(prev_len,
## bs = "cr") + ti(prev_freq, prev_len, bs = "cr")
## Model 2: rt ~ s(surprisal, bs = "cr", k = 20) + ti(freq, bs = "cr") +
## ti(len, bs = "cr") + s(prev_surp, bs = "cr", k = 20) + ti(prev_freq,
## bs = "cr") + ti(prev_len, bs = "cr")
## Resid. Df Resid. Dev Df Deviance Pr(>Chi)
## 1 6324.7 316288721
## 2 6335.7 318041282 -10.989 -1752561 0.0002371 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1Plots
For the length: frequency interaction, it looks like the effect is ~ 0 for most of the space where there’s actually data, and it only goes off the rails in the region where there isn’t data.
Maze LMs
Maze Model comparison
I’m not sure if this is what we want, but these are the P-values from anova’s comparison the value of adding one set of surprisal predictors to a model that already has another. (So smaller values mean that the column has a lot of predictive utility beyond the row label).
I’m not sure what the best presentational format is – or whether we want F values or p values or what. I also not sure how to arrange the table (should I transpose?) or label it so which one is better is clear.
This is all on meaned data with no mixed effects and predictors of freq x length and surprisal(s) for word and previous word.
| Model | over 5-gram | over GRNN | over TXL | over GPT-2 | Log Lik | r_squared |
|---|---|---|---|---|---|---|
| 5-gram | 2 (p=0.153) | 3 (p=0.035) | 0 (p=0.611) | -43817 | 0.16 | |
| GRNN | 287 (p<0.001) | 113 (p<0.001) | 13 (p<0.001) | -43544 | 0.23 | |
| TXL | 174 (p<0.001) | 5 (p=0.006) | 2 (p=0.137) | -43650 | 0.2 | |
| GPT-2 | 394 (p<0.001) | 113 (p<0.001) | 213 (p<0.001) | -43445 | 0.25 |
Maze LM with freq**2 effects
Based on GAM viz, we try models with freq**2 effects on item mean data.
(Tried this with orthagonal polynomials, but I get ridiculous predictions where both the effects for 1st and 2nd order are huge with huge std error, so probably cancelling out?).
Instead, square and then center raw frequency. This also finds the effects, and seems a little absurd, since it’s finding big effects, in opposite directions for squared and not squared.
##
## Call:
## lm(formula = str_c(no_surp_sq, ngram), data = item_mean_data)
##
## Residuals:
## Min 1Q Median 3Q Max
## -620.35 -141.86 -44.84 80.92 2889.72
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 857.07279 13.69310 62.592 < 2e-16 ***
## freq_sq -1.08506 0.23604 -4.597 4.37e-06 ***
## freq_center 37.91433 9.03805 4.195 2.77e-05 ***
## length_center 23.86528 2.35851 10.119 < 2e-16 ***
## past_freq_sq 0.82069 0.24232 3.387 0.000711 ***
## past_freq_center -27.80680 9.15120 -3.039 0.002387 **
## past_length_center 2.31919 2.37505 0.976 0.328862
## ngram_center 10.73085 1.13183 9.481 < 2e-16 ***
## past_ngram_center 0.31597 1.13842 0.278 0.781363
## freq_center:length_center -3.57009 0.83090 -4.297 1.76e-05 ***
## past_freq_center:past_length_center -0.07921 0.85895 -0.092 0.926530
## length_center:ngram_center -2.40370 0.52020 -4.621 3.90e-06 ***
## past_length_center:past_ngram_center -0.14831 0.53503 -0.277 0.781637
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 237.6 on 6344 degrees of freedom
## Multiple R-squared: 0.1637, Adjusted R-squared: 0.1621
## F-statistic: 103.5 on 12 and 6344 DF, p-value: < 2.2e-16##
## Call:
## lm(formula = str_c(no_surp_sq, grnn), data = item_mean_data)
##
## Residuals:
## Min 1Q Median 3Q Max
## -577.02 -133.10 -38.38 77.91 2813.95
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 835.9723 12.8025 65.297 < 2e-16 ***
## freq_sq -1.3090 0.2248 -5.824 6.03e-09 ***
## freq_center 53.6625 8.5115 6.305 3.08e-10 ***
## length_center 20.0227 2.2328 8.967 < 2e-16 ***
## past_freq_sq 0.3833 0.2301 1.666 0.0958 .
## past_freq_center -11.9293 8.5866 -1.389 0.1648
## past_length_center 0.1059 2.2354 0.047 0.9622
## grnn_center 24.9862 0.9533 26.210 < 2e-16 ***
## past_grnn_center 1.6513 0.9056 1.823 0.0683 .
## freq_center:length_center -2.6904 0.6528 -4.122 3.81e-05 ***
## past_freq_center:past_length_center -1.4445 0.6642 -2.175 0.0297 *
## length_center:grnn_center -1.9313 0.4262 -4.531 5.97e-06 ***
## past_length_center:past_grnn_center -0.6520 0.4072 -1.601 0.1094
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 227.5 on 6344 degrees of freedom
## Multiple R-squared: 0.2333, Adjusted R-squared: 0.2318
## F-statistic: 160.8 on 12 and 6344 DF, p-value: < 2.2e-16##
## Call:
## lm(formula = str_c(no_surp_sq, txl), data = item_mean_data)
##
## Residuals:
## Min 1Q Median 3Q Max
## -576.99 -138.75 -42.69 83.67 2946.83
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 851.4766 13.0041 65.478 < 2e-16 ***
## freq_sq -1.1686 0.2286 -5.111 3.29e-07 ***
## freq_center 46.0778 8.6597 5.321 1.07e-07 ***
## length_center 22.8865 2.2674 10.094 < 2e-16 ***
## past_freq_sq 0.5817 0.2339 2.487 0.01291 *
## past_freq_center -20.1397 8.7374 -2.305 0.02120 *
## past_length_center 1.4238 2.2735 0.626 0.53117
## txl_center 19.5499 0.9265 21.100 < 2e-16 ***
## past_txl_center -0.2331 0.8934 -0.261 0.79415
## freq_center:length_center -1.8950 0.6681 -2.836 0.00458 **
## past_freq_center:past_length_center -0.9455 0.6787 -1.393 0.16361
## length_center:txl_center -1.2301 0.4180 -2.943 0.00326 **
## past_length_center:past_txl_center -0.4410 0.4048 -1.090 0.27591
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 231.7 on 6344 degrees of freedom
## Multiple R-squared: 0.2051, Adjusted R-squared: 0.2036
## F-statistic: 136.4 on 12 and 6344 DF, p-value: < 2.2e-16##
## Call:
## lm(formula = str_c(no_surp_sq, gpt), data = item_mean_data)
##
## Residuals:
## Min 1Q Median 3Q Max
## -579.89 -131.08 -37.58 81.06 2900.52
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 834.9056 12.5860 66.336 < 2e-16 ***
## freq_sq -1.3257 0.2217 -5.979 2.37e-09 ***
## freq_center 50.1884 8.3601 6.003 2.04e-09 ***
## length_center 17.1176 2.2083 7.752 1.05e-14 ***
## past_freq_sq 0.2914 0.2270 1.283 0.199387
## past_freq_center -9.2724 8.4420 -1.098 0.272089
## past_length_center -1.2015 2.2142 -0.543 0.587411
## gpt_center 27.3653 0.9216 29.694 < 2e-16 ***
## past_gpt_center 3.5142 0.8975 3.916 9.11e-05 ***
## freq_center:length_center -2.1035 0.6128 -3.433 0.000601 ***
## past_freq_center:past_length_center -1.4636 0.6210 -2.357 0.018455 *
## length_center:gpt_center -1.3979 0.4056 -3.446 0.000572 ***
## past_length_center:past_gpt_center -0.6488 0.3928 -1.652 0.098647 .
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 224.2 on 6344 degrees of freedom
## Multiple R-squared: 0.2556, Adjusted R-squared: 0.2542
## F-statistic: 181.6 on 12 and 6344 DF, p-value: < 2.2e-16##
## Call:
## lm(formula = str_c(no_surp_poly, ngram), data = item_mean_data)
##
## Residuals:
## Min 1Q Median 3Q Max
## -620.35 -141.86 -44.84 80.92 2889.72
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 8.677e+02 6.367e+00 136.278 < 2e-16 ***
## poly(freq_center, 2)1 -1.211e+03 5.149e+02 -2.352 0.018691 *
## poly(freq_center, 2)2 -1.371e+03 2.983e+02 -4.597 4.37e-06 ***
## length_center 2.387e+01 2.359e+00 10.119 < 2e-16 ***
## poly(past_freq_center, 2)1 1.098e+03 5.067e+02 2.167 0.030246 *
## poly(past_freq_center, 2)2 1.024e+03 3.022e+02 3.387 0.000711 ***
## past_length_center 2.319e+00 2.375e+00 0.976 0.328862
## ngram_center 1.073e+01 1.132e+00 9.481 < 2e-16 ***
## past_ngram_center 3.160e-01 1.138e+00 0.278 0.781363
## length_center:freq_center -3.570e+00 8.309e-01 -4.297 1.76e-05 ***
## past_length_center:past_freq_center -7.921e-02 8.589e-01 -0.092 0.926530
## length_center:ngram_center -2.404e+00 5.202e-01 -4.621 3.90e-06 ***
## past_length_center:past_ngram_center -1.483e-01 5.350e-01 -0.277 0.781637
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 237.6 on 6344 degrees of freedom
## Multiple R-squared: 0.1637, Adjusted R-squared: 0.1621
## F-statistic: 103.5 on 12 and 6344 DF, p-value: < 2.2e-16##
## Call:
## lm(formula = str_c(no_surp_poly, grnn), data = item_mean_data)
##
## Residuals:
## Min 1Q Median 3Q Max
## -577.02 -133.10 -38.38 77.91 2813.95
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 869.9051 6.0545 143.678 < 2e-16 ***
## poly(freq_center, 2)1 1106.0700 418.6693 2.642 0.00827 **
## poly(freq_center, 2)2 -1654.2059 284.0420 -5.824 6.03e-09 ***
## length_center 20.0227 2.2328 8.967 < 2e-16 ***
## poly(past_freq_center, 2)1 848.0635 412.9727 2.054 0.04006 *
## poly(past_freq_center, 2)2 478.0679 286.9693 1.666 0.09578 .
## past_length_center 0.1059 2.2354 0.047 0.96222
## grnn_center 24.9862 0.9533 26.210 < 2e-16 ***
## past_grnn_center 1.6513 0.9056 1.823 0.06829 .
## length_center:freq_center -2.6904 0.6528 -4.122 3.81e-05 ***
## past_length_center:past_freq_center -1.4445 0.6642 -2.175 0.02968 *
## length_center:grnn_center -1.9313 0.4262 -4.531 5.97e-06 ***
## past_length_center:past_grnn_center -0.6520 0.4072 -1.601 0.10937
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 227.5 on 6344 degrees of freedom
## Multiple R-squared: 0.2333, Adjusted R-squared: 0.2318
## F-statistic: 160.8 on 12 and 6344 DF, p-value: < 2.2e-16##
## Call:
## lm(formula = str_c(no_surp_poly, txl), data = item_mean_data)
##
## Residuals:
## Min 1Q Median 3Q Max
## -576.99 -138.75 -42.69 83.67 2946.83
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 873.8193 6.1630 141.784 < 2e-16 ***
## poly(freq_center, 2)1 395.0382 431.9890 0.914 0.36051
## poly(freq_center, 2)2 -1476.7718 288.9162 -5.111 3.29e-07 ***
## length_center 22.8865 2.2674 10.094 < 2e-16 ***
## poly(past_freq_center, 2)1 642.6468 429.3309 1.497 0.13448
## poly(past_freq_center, 2)2 725.5358 291.7433 2.487 0.01291 *
## past_length_center 1.4238 2.2735 0.626 0.53117
## txl_center 19.5499 0.9265 21.100 < 2e-16 ***
## past_txl_center -0.2331 0.8934 -0.261 0.79415
## length_center:freq_center -1.8950 0.6681 -2.836 0.00458 **
## past_length_center:past_freq_center -0.9455 0.6787 -1.393 0.16361
## length_center:txl_center -1.2301 0.4180 -2.943 0.00326 **
## past_length_center:past_txl_center -0.4410 0.4048 -1.090 0.27591
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 231.7 on 6344 degrees of freedom
## Multiple R-squared: 0.2051, Adjusted R-squared: 0.2036
## F-statistic: 136.4 on 12 and 6344 DF, p-value: < 2.2e-16##
## Call:
## lm(formula = str_c(no_surp_poly, gpt), data = item_mean_data)
##
## Residuals:
## Min 1Q Median 3Q Max
## -579.89 -131.08 -37.58 81.06 2900.52
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 867.9353 5.9667 145.463 < 2e-16 ***
## poly(freq_center, 2)1 -227.1907 383.1912 -0.593 0.553275
## poly(freq_center, 2)2 -1675.3331 280.2226 -5.979 2.37e-09 ***
## length_center 17.1176 2.2083 7.752 1.05e-14 ***
## poly(past_freq_center, 2)1 580.2372 380.8405 1.524 0.127666
## poly(past_freq_center, 2)2 363.4322 283.1718 1.283 0.199387
## past_length_center -1.2015 2.2142 -0.543 0.587411
## gpt_center 27.3653 0.9216 29.694 < 2e-16 ***
## past_gpt_center 3.5142 0.8975 3.916 9.11e-05 ***
## length_center:freq_center -2.1035 0.6128 -3.433 0.000601 ***
## past_length_center:past_freq_center -1.4636 0.6210 -2.357 0.018455 *
## length_center:gpt_center -1.3979 0.4056 -3.446 0.000572 ***
## past_length_center:past_gpt_center -0.6488 0.3928 -1.652 0.098647 .
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 224.2 on 6344 degrees of freedom
## Multiple R-squared: 0.2556, Adjusted R-squared: 0.2542
## F-statistic: 181.6 on 12 and 6344 DF, p-value: < 2.2e-16SPR Prep
SPR GAMs
#no hierarchical, has freq x len interaction
spr_formula_interact <- rt ~ s(surp, bs="cr", k=20)+
ti(length_center, bs="cr")+
ti(freq_center,bs="cr")+
ti(freq_center,length_center, bs="cr")+
s(past_surp, bs="cr", k=20)+
ti(past_freq_center, bs="cr")+
ti(past_length_center, bs="cr")+
ti(past_freq_center, past_length_center, bs="cr")+
s(past2_surp, bs="cr", k=20)+
ti(past2_freq_center, bs="cr")+
ti(past2_length_center, bs="cr")+
ti(past2_freq_center, past2_length_center, bs="cr")+
s(past3_surp, bs="cr", k=20)+
ti(past3_freq_center, bs="cr")+
ti(past3_length_center, bs="cr")+
ti(past3_freq_center, past3_length_center, bs="cr")All of this is on by-item mean data.
SPR LMs
## Linear mixed model fit by REML ['lmerMod']
## Formula: rt ~ surp_center * length_center + length_center * freq_center +
## past_surp_center * past_length_center + past_length_center *
## past_freq_center + past2_surp_center * past2_length_center +
## past2_length_center * past2_freq_center + past3_surp_center *
## past3_length_center + past3_length_center * past3_freq_center +
## (1 | Word_ID)
## Data: d
##
## REML criterion at convergence: 103479.7
##
## Scaled residuals:
## Min 1Q Median 3Q Max
## -4.0157 -0.4678 -0.1733 0.2520 25.9155
##
## Random effects:
## Groups Name Variance Std.Dev.
## Word_ID (Intercept) 2722 52.17
## Residual 14156 118.98
## Number of obs: 8242, groups: Word_ID, 4123
##
## Fixed effects:
## Estimate Std. Error t value
## (Intercept) 359.5187 3.5379 101.620
## surp_center 1.9159 0.5829 3.287
## length_center 2.1500 1.2006 1.791
## freq_center 2.3728 0.8163 2.907
## past_surp_center 1.5188 0.5781 2.627
## past_length_center -0.2172 1.1883 -0.183
## past_freq_center 1.7774 0.8271 2.149
## past2_surp_center 1.1163 0.5749 1.942
## past2_length_center 2.2575 1.1979 1.885
## past2_freq_center 2.0352 0.8246 2.468
## past3_surp_center 0.5297 0.5919 0.895
## past3_length_center 0.4446 1.2038 0.369
## past3_freq_center 0.8798 0.8209 1.072
## surp_center:length_center -0.1924 0.2744 -0.701
## length_center:freq_center -0.4802 0.3867 -1.242
## past_surp_center:past_length_center -0.1461 0.2709 -0.539
## past_length_center:past_freq_center -0.3246 0.3858 -0.841
## past2_surp_center:past2_length_center -0.3198 0.2706 -1.182
## past2_length_center:past2_freq_center -0.5462 0.3854 -1.417
## past3_surp_center:past3_length_center -0.4291 0.2833 -1.515
## past3_length_center:past3_freq_center -0.5775 0.3927 -1.471## Linear mixed model fit by REML ['lmerMod']
## Formula: rt ~ surp_center * length_center + length_center * freq_center +
## past_surp_center * past_length_center + past_length_center *
## past_freq_center + past2_surp_center * past2_length_center +
## past2_length_center * past2_freq_center + past3_surp_center *
## past3_length_center + past3_length_center * past3_freq_center +
## (1 | Word_ID)
## Data: d
##
## REML criterion at convergence: 103463.1
##
## Scaled residuals:
## Min 1Q Median 3Q Max
## -3.9620 -0.4653 -0.1724 0.2471 25.9692
##
## Random effects:
## Groups Name Variance Std.Dev.
## Word_ID (Intercept) 2676 51.73
## Residual 14156 118.98
## Number of obs: 8242, groups: Word_ID, 4123
##
## Fixed effects:
## Estimate Std. Error t value
## (Intercept) 363.53925 3.44129 105.640
## surp_center 2.62457 0.50650 5.182
## length_center 2.14866 1.18160 1.818
## freq_center 2.43173 0.68069 3.572
## past_surp_center 1.65655 0.50286 3.294
## past_length_center -0.29636 1.16576 -0.254
## past_freq_center 1.38108 0.69664 1.982
## past2_surp_center 0.71623 0.50378 1.422
## past2_length_center 2.13372 1.17169 1.821
## past2_freq_center 1.27925 0.69730 1.835
## past3_surp_center -0.50376 0.49606 -1.016
## past3_length_center 0.48990 1.17596 0.417
## past3_freq_center 0.02131 0.68779 0.031
## surp_center:length_center -0.44342 0.22953 -1.932
## length_center:freq_center -0.57042 0.30071 -1.897
## past_surp_center:past_length_center -0.24032 0.22751 -1.056
## past_length_center:past_freq_center -0.37320 0.29943 -1.246
## past2_surp_center:past2_length_center -0.44804 0.23120 -1.938
## past2_length_center:past2_freq_center -0.56168 0.30050 -1.869
## past3_surp_center:past3_length_center -0.22918 0.22308 -1.027
## past3_length_center:past3_freq_center -0.34285 0.30781 -1.114## Linear mixed model fit by REML ['lmerMod']
## Formula: rt ~ surp_center * length_center + length_center * freq_center +
## past_surp_center * past_length_center + past_length_center *
## past_freq_center + past2_surp_center * past2_length_center +
## past2_length_center * past2_freq_center + past3_surp_center *
## past3_length_center + past3_length_center * past3_freq_center +
## (1 | Word_ID)
## Data: d
##
## REML criterion at convergence: 103490.1
##
## Scaled residuals:
## Min 1Q Median 3Q Max
## -4.0304 -0.4640 -0.1720 0.2508 25.9009
##
## Random effects:
## Groups Name Variance Std.Dev.
## Word_ID (Intercept) 2739 52.33
## Residual 14156 118.98
## Number of obs: 8242, groups: Word_ID, 4123
##
## Fixed effects:
## Estimate Std. Error t value
## (Intercept) 363.80523 3.45081 105.426
## surp_center 1.66038 0.48658 3.412
## length_center 2.33305 1.18323 1.972
## freq_center 1.95207 0.69242 2.819
## past_surp_center 1.26134 0.48356 2.608
## past_length_center -0.05669 1.16798 -0.049
## past_freq_center 1.25686 0.70712 1.777
## past2_surp_center 0.52483 0.48157 1.090
## past2_length_center 2.19668 1.17385 1.871
## past2_freq_center 1.21807 0.71093 1.713
## past3_surp_center -0.59755 0.47791 -1.250
## past3_length_center 0.38996 1.18304 0.330
## past3_freq_center -0.10020 0.70382 -0.142
## surp_center:length_center -0.07813 0.22197 -0.352
## length_center:freq_center -0.29545 0.30387 -0.972
## past_surp_center:past_length_center 0.12527 0.22236 0.563
## past_length_center:past_freq_center -0.06776 0.30212 -0.224
## past2_surp_center:past2_length_center -0.13051 0.22096 -0.591
## past2_length_center:past2_freq_center -0.32754 0.30215 -1.084
## past3_surp_center:past3_length_center -0.19992 0.21584 -0.926
## past3_length_center:past3_freq_center -0.36037 0.30728 -1.173## Linear mixed model fit by REML ['lmerMod']
## Formula: rt ~ surp_center * length_center + length_center * freq_center +
## past_surp_center * past_length_center + past_length_center *
## past_freq_center + past2_surp_center * past2_length_center +
## past2_length_center * past2_freq_center + past3_surp_center *
## past3_length_center + past3_length_center * past3_freq_center +
## (1 | Word_ID)
## Data: d
##
## REML criterion at convergence: 103490.1
##
## Scaled residuals:
## Min 1Q Median 3Q Max
## -4.0304 -0.4640 -0.1720 0.2508 25.9009
##
## Random effects:
## Groups Name Variance Std.Dev.
## Word_ID (Intercept) 2739 52.33
## Residual 14156 118.98
## Number of obs: 8242, groups: Word_ID, 4123
##
## Fixed effects:
## Estimate Std. Error t value
## (Intercept) 363.80523 3.45081 105.426
## surp_center 1.66038 0.48658 3.412
## length_center 2.33305 1.18323 1.972
## freq_center 1.95207 0.69242 2.819
## past_surp_center 1.26134 0.48356 2.608
## past_length_center -0.05669 1.16798 -0.049
## past_freq_center 1.25686 0.70712 1.777
## past2_surp_center 0.52483 0.48157 1.090
## past2_length_center 2.19668 1.17385 1.871
## past2_freq_center 1.21807 0.71093 1.713
## past3_surp_center -0.59755 0.47791 -1.250
## past3_length_center 0.38996 1.18304 0.330
## past3_freq_center -0.10020 0.70382 -0.142
## surp_center:length_center -0.07813 0.22197 -0.352
## length_center:freq_center -0.29545 0.30387 -0.972
## past_surp_center:past_length_center 0.12527 0.22236 0.563
## past_length_center:past_freq_center -0.06776 0.30212 -0.224
## past2_surp_center:past2_length_center -0.13051 0.22096 -0.591
## past2_length_center:past2_freq_center -0.32754 0.30215 -1.084
## past3_surp_center:past3_length_center -0.19992 0.21584 -0.926
## past3_length_center:past3_freq_center -0.36037 0.30728 -1.173Summaries of LMER
These have some hierarchical effects
SPR Model comparison
I’m not sure if this is what we want, but these are the P-values from anova’s comparison the value of adding one set of surprisal predictors to a model that already has another. (So smaller values mean that the column has a lot of predictive utility beyond the row label).
I’m not sure what the best presentational format is – or whether we want F values or p values or what. I also not sure how to arrange the table (should I transpose?) or label it so which one is better is clear.
This is all on meaned data with no mixed effects and predictors of freq x length and surprisal(s) for word and previous word.
| Model | over 5-gram | over GRNN | over TXL | over GPT-2 | Log Lik | r_squared |
|---|---|---|---|---|---|---|
| 5-gram | 3 (p=0.032) | 4 (p=0.001) | 3 (p=0.033) | -51798 | 0.01 | |
| GRNN | 7 (p<0.001) | 6 (p<0.001) | 2 (p=0.153) | -51790 | 0.01 | |
| TXL | 3 (p=0.010) | 0 (p=0.910) | 1 (p=0.462) | -51801 | 0.01 | |
| GPT-2 | 10 (p<0.001) | 5 (p<0.001) | 10 (p<0.001) | -51783 | 0.01 |
SPR LM with freq**2 effects
##
## Call:
## lm(formula = str_c(no_surp_poly, ngram), data = item_mean_spr)
##
## Residuals:
## Min 1Q Median 3Q Max
## -216.4 -66.0 -25.9 34.6 3578.2
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 367.1462 4.1196 89.122 < 2e-16 ***
## poly(freq_center, 2)1 928.1652 285.7340 3.248 0.001165 **
## poly(freq_center, 2)2 193.7760 165.2863 1.172 0.241084
## length_center 2.4822 1.1521 2.154 0.031232 *
## poly(past_freq_center, 2)1 608.1251 285.2777 2.132 0.033061 *
## poly(past_freq_center, 2)2 -95.6702 168.0492 -0.569 0.569169
## past_length_center -0.3613 1.1527 -0.313 0.753942
## poly(past2_freq_center, 2)1 700.4677 279.7548 2.504 0.012304 *
## poly(past2_freq_center, 2)2 -21.4001 166.8474 -0.128 0.897945
## past2_length_center 2.1724 1.1567 1.878 0.060401 .
## poly(past3_freq_center, 2)1 390.1674 276.0361 1.413 0.157557
## poly(past3_freq_center, 2)2 377.5809 163.5949 2.308 0.021023 *
## past3_length_center 1.1277 1.1562 0.975 0.329441
## ngram_center 1.8645 0.5478 3.403 0.000669 ***
## past_ngram_center 1.5255 0.5442 2.803 0.005068 **
## past2_ngram_center 0.9544 0.5412 1.764 0.077842 .
## past3_ngram_center 0.3895 0.5532 0.704 0.481349
## length_center:freq_center -0.2682 0.4088 -0.656 0.511714
## past_length_center:past_freq_center -0.4285 0.4104 -1.044 0.296523
## past2_length_center:past2_freq_center -0.5729 0.4103 -1.396 0.162627
## past3_length_center:past3_freq_center -0.1331 0.4129 -0.322 0.747190
## length_center:ngram_center -0.1840 0.2557 -0.720 0.471757
## past_length_center:past_ngram_center -0.1667 0.2525 -0.660 0.509012
## past2_length_center:past2_ngram_center -0.2955 0.2527 -1.170 0.242171
## past3_length_center:past3_ngram_center -0.3648 0.2642 -1.381 0.167412
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 129.9 on 8217 degrees of freedom
## Multiple R-squared: 0.008587, Adjusted R-squared: 0.005691
## F-statistic: 2.965 on 24 and 8217 DF, p-value: 1.564e-06##
## Call:
## lm(formula = str_c(no_surp_poly, grnn), data = item_mean_spr)
##
## Residuals:
## Min 1Q Median 3Q Max
## -220.4 -66.3 -25.3 34.0 3577.1
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 370.2825 4.0457 91.525 < 2e-16 ***
## poly(freq_center, 2)1 962.6813 241.3175 3.989 6.69e-05 ***
## poly(freq_center, 2)2 205.6680 164.0520 1.254 0.21000
## length_center 2.4908 1.1384 2.188 0.02869 *
## poly(past_freq_center, 2)1 449.6060 243.5749 1.846 0.06495 .
## poly(past_freq_center, 2)2 -115.6865 166.5073 -0.695 0.48721
## past_length_center -0.5109 1.1353 -0.450 0.65273
## poly(past2_freq_center, 2)1 452.1801 238.9742 1.892 0.05850 .
## poly(past2_freq_center, 2)2 -13.0991 165.4909 -0.079 0.93691
## past2_length_center 2.0607 1.1371 1.812 0.07000 .
## poly(past3_freq_center, 2)1 120.9526 233.8813 0.517 0.60506
## poly(past3_freq_center, 2)2 443.8665 161.9194 2.741 0.00613 **
## past3_length_center 1.3283 1.1339 1.171 0.24145
## grnn_center 2.6041 0.4747 5.485 4.25e-08 ***
## past_grnn_center 1.6619 0.4718 3.522 0.00043 ***
## past2_grnn_center 0.5771 0.4732 1.219 0.22270
## past3_grnn_center -0.6274 0.4629 -1.355 0.17541
## length_center:freq_center -0.3422 0.3322 -1.030 0.30295
## past_length_center:past_freq_center -0.4962 0.3343 -1.484 0.13779
## past2_length_center:past2_freq_center -0.5984 0.3345 -1.789 0.07370 .
## past3_length_center:past3_freq_center 0.1319 0.3372 0.391 0.69565
## length_center:grnn_center -0.4291 0.2135 -2.010 0.04448 *
## past_length_center:past_grnn_center -0.2678 0.2117 -1.265 0.20598
## past2_length_center:past2_grnn_center -0.4313 0.2153 -2.003 0.04521 *
## past3_length_center:past3_grnn_center -0.1945 0.2080 -0.935 0.34973
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 129.7 on 8217 degrees of freedom
## Multiple R-squared: 0.01158, Adjusted R-squared: 0.008692
## F-statistic: 4.011 on 24 and 8217 DF, p-value: 1.53e-10##
## Call:
## lm(formula = str_c(no_surp_poly, txl), data = item_mean_spr)
##
## Residuals:
## Min 1Q Median 3Q Max
## -225.3 -66.0 -26.2 34.4 3577.3
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 370.98916 4.05176 91.563 < 2e-16 ***
## poly(freq_center, 2)1 782.47068 244.55802 3.200 0.001382 **
## poly(freq_center, 2)2 229.66844 164.39971 1.397 0.162448
## length_center 2.69760 1.13778 2.371 0.017766 *
## poly(past_freq_center, 2)1 422.95187 246.78238 1.714 0.086591 .
## poly(past_freq_center, 2)2 -42.50768 166.34717 -0.256 0.798316
## past_length_center -0.12982 1.13544 -0.114 0.908972
## poly(past2_freq_center, 2)1 439.03848 243.37107 1.804 0.071270 .
## poly(past2_freq_center, 2)2 37.08715 165.14534 0.225 0.822317
## past2_length_center 2.20808 1.13619 1.943 0.052002 .
## poly(past3_freq_center, 2)1 83.15345 238.93080 0.348 0.727832
## poly(past3_freq_center, 2)2 460.26339 161.96758 2.842 0.004498 **
## past3_length_center 1.24474 1.13706 1.095 0.273679
## txl_center 1.61127 0.45484 3.542 0.000399 ***
## past_txl_center 1.23850 0.45177 2.741 0.006130 **
## past2_txl_center 0.38057 0.45087 0.844 0.398648
## past3_txl_center -0.69589 0.44491 -1.564 0.117829
## length_center:freq_center -0.05280 0.33353 -0.158 0.874212
## past_length_center:past_freq_center -0.11241 0.33543 -0.335 0.737540
## past2_length_center:past2_freq_center -0.30642 0.33471 -0.916 0.359955
## past3_length_center:past3_freq_center 0.14128 0.33632 0.420 0.674433
## length_center:txl_center -0.06686 0.20612 -0.324 0.745658
## past_length_center:past_txl_center 0.10374 0.20663 0.502 0.615654
## past2_length_center:past2_txl_center -0.11301 0.20545 -0.550 0.582296
## past3_length_center:past3_txl_center -0.16271 0.20084 -0.810 0.417861
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 129.9 on 8217 degrees of freedom
## Multiple R-squared: 0.007983, Adjusted R-squared: 0.005086
## F-statistic: 2.755 on 24 and 8217 DF, p-value: 8.824e-06##
## Call:
## lm(formula = str_c(no_surp_poly, gpt), data = item_mean_spr)
##
## Residuals:
## Min 1Q Median 3Q Max
## -224.0 -65.9 -25.7 34.6 3580.9
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 369.3487 4.0478 91.246 < 2e-16 ***
## poly(freq_center, 2)1 787.2679 225.1586 3.497 0.000474 ***
## poly(freq_center, 2)2 181.3571 164.3237 1.104 0.269774
## length_center 2.1029 1.1409 1.843 0.065332 .
## poly(past_freq_center, 2)1 295.9560 228.1958 1.297 0.194689
## poly(past_freq_center, 2)2 -108.4075 166.6901 -0.650 0.515482
## past_length_center -0.7037 1.1399 -0.617 0.537033
## poly(past2_freq_center, 2)1 520.1060 224.5531 2.316 0.020573 *
## poly(past2_freq_center, 2)2 -51.6043 165.6383 -0.312 0.755392
## past2_length_center 1.6410 1.1420 1.437 0.150781
## poly(past3_freq_center, 2)1 135.2172 219.3713 0.616 0.537657
## poly(past3_freq_center, 2)2 426.2727 162.0943 2.630 0.008560 **
## past3_length_center 0.9408 1.1385 0.826 0.408594
## gpt_center 2.7110 0.4647 5.834 5.61e-09 ***
## past_gpt_center 1.6778 0.4621 3.631 0.000284 ***
## past2_gpt_center 1.4885 0.4645 3.204 0.001360 **
## past3_gpt_center -0.4596 0.4624 -0.994 0.320253
## length_center:freq_center -0.2129 0.3180 -0.670 0.503118
## past_length_center:past_freq_center -0.3810 0.3185 -1.196 0.231554
## past2_length_center:past2_freq_center -0.5789 0.3180 -1.821 0.068672 .
## past3_length_center:past3_freq_center 0.1772 0.3186 0.556 0.578030
## length_center:gpt_center -0.2619 0.2060 -1.271 0.203780
## past_length_center:past_gpt_center -0.1245 0.2019 -0.617 0.537207
## past2_length_center:past2_gpt_center -0.4113 0.2026 -2.030 0.042394 *
## past3_length_center:past3_gpt_center -0.0901 0.2007 -0.449 0.653525
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 129.6 on 8217 degrees of freedom
## Multiple R-squared: 0.01251, Adjusted R-squared: 0.00963
## F-statistic: 4.339 on 24 and 8217 DF, p-value: 7.204e-12Maze and SPR scatter
Abnormally low and high times were excluded, then for each word, an average was taken across subjects. For sentence level, these averages were then summed across the sentence. (Because of A-maze sentence discontinuation and general outlier handling, summing the average is better than averaging the sum.)
We want to compare w/i Maze, w/i SPR and between Maze and SPR. Maze is the smaller dataset (63 subjects), so we split half for each story.
For SPR, we subsample the 165 subjects down to 63 (to match the number of subjects in Maze), and again split half for each story.
For Maze-SPR comparison, we compare each of the Maze half-SPR half and then average the correlations across these 4 pairs.
SPR-SPR
## [1] 0.1368166
## [1] 0.7795845
## [1] 0.2341913Maze-Maze
## [1] 0.4378548
## [1] 0.9503789
## [1] 0.3606054Maze-SPR
For maze-spr comparisons we take the 4 pairs (maze1 with spr1; maze1 with spr2, etc) We overplot all the combinations (not sure this is right), and take the average of the 4 correlations.
## [1] 0.1353904
## [1] 0.8765725
## [1] 0.2489745