Pre-registered analysis, with real data
Richard D. Morey
02/07/2018
This document includes the initial, pre-registered analysis of the ACE effect data. The meta-analytic analysis is performed by simply including the lab as a random effect.
This initial pre-registration is done on blinded data.
This version of the document has been modified to perform the pre-registered analyses. Several tweaks were needed, including:
- Uncommenting lines that performed the response time cleaning.
- Properly restricting the response time analyses to correct trials.
- Incorporating information about left-handed participants in order to exclude them.
- Rearranging the document – putting the LME analyses first – so that some quantities computed from the LMEs could be added to graphs.
In addition, code has been included to (optionally) perform some additional cleaning that was determined to be needed after looking at the data. A small proportion of participants have very fast movement times. We feared this indicates that these participants have adopted an idiosyncratic response strategy, such as responding with two hands; otherwise, we see no way that the movement times could be as fast as they are. Thus, we adopted the following rule: if 20% or more of a participant’s response times would be removed under the pre-registered response time filtering rules, then that participant is eliminated from the analysis.
This step in the filtering can be applied by uncommenting the line containing mean(valid.rt) > .2
Data loading
Data cleaning
Details of the pre-registered data cleaning can be found in the pre-registration documents (initial and addendum).
Mixed effects models, all labs
Monolingual
Accuracy
There appears to be a very small, albeit detectable, effect on Accuracy. Overall accuracy is very high, but there does seem to be a slight advantage for trials of a particular cue direction.
lmer_obj = lme4::glmer(Accuracy ~ CueDirection*SentenceDirection + (1|ptid) + (1|lab) + (1|ItemNumber) + (1|Counterbalance.List),
data = ace_filtered_mono, family = "binomial")## Warning in checkConv(attr(opt, "derivs"), opt$par, ctrl = control
## $checkConv, : Model failed to converge with max|grad| = 0.00765728 (tol =
## 0.001, component 1)summary(lmer_obj)## Generalized linear mixed model fit by maximum likelihood (Laplace
## Approximation) [glmerMod]
## Family: binomial ( logit )
## Formula: Accuracy ~ CueDirection * SentenceDirection + (1 | ptid) + (1 |
## lab) + (1 | ItemNumber) + (1 | Counterbalance.List)
## Data: ace_filtered_mono
##
## AIC BIC logLik deviance df.resid
## 1247.2 1313.0 -615.6 1231.2 27567
##
## Scaled residuals:
## Min 1Q Median 3Q Max
## -9.3895 0.0073 0.0117 0.0119 0.4377
##
## Random effects:
## Groups Name Variance Std.Dev.
## ptid (Intercept) 1.854e+01 4.305e+00
## ItemNumber (Intercept) 4.152e-07 6.444e-04
## lab (Intercept) 4.810e-09 6.935e-05
## Counterbalance.List (Intercept) 1.343e-07 3.664e-04
## Number of obs: 27575, groups:
## ptid, 900; ItemNumber, 37; lab, 12; Counterbalance.List, 4
##
## Fixed effects:
## Estimate Std. Error z value Pr(>|z|)
## (Intercept) 9.83549 0.66652 14.757 < 2e-16 ***
## CueDirection -1.02531 0.32324 -3.172 0.00151 **
## SentenceDirection -0.06366 0.37826 -0.168 0.86636
## CueDirection:SentenceDirection 0.08846 0.45191 0.196 0.84481
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Correlation of Fixed Effects:
## (Intr) CDrctn SntncD
## CueDirectin -0.372
## SentncDrctn -0.294 0.605
## CDrctn:SntD 0.252 -0.716 -0.838
## convergence code: 0
## Model failed to converge with max|grad| = 0.00765728 (tol = 0.001, component 1)This can be seen in the crosstabs:
with(ace_filtered_mono, table(CueDirection, Accuracy))## Accuracy
## CueDirection 0 1
## 0 29 13725
## 1 72 13749The odds of being correct increase by about 2.5 when CueDirection is 0. In light of the high accuracy, this does not seem problematic.
Lift-off times
lmer_obj_lo_mono = lme4::lmer(log(LiftOffLatency) ~ CueDirection*SentenceDirection + (1|ptid) + (1|lab) + (1|ItemNumber) + (1|Counterbalance.List),
data = ace_filtered_mono %>% filter(Accuracy == 1))
summary(lmer_obj_lo_mono)## Linear mixed model fit by REML ['lmerMod']
## Formula: log(LiftOffLatency) ~ CueDirection * SentenceDirection + (1 |
## ptid) + (1 | lab) + (1 | ItemNumber) + (1 | Counterbalance.List)
## Data: ace_filtered_mono %>% filter(Accuracy == 1)
##
## REML criterion at convergence: -41172.7
##
## Scaled residuals:
## Min 1Q Median 3Q Max
## -6.8976 -0.5802 -0.1161 0.4159 8.1244
##
## Random effects:
## Groups Name Variance Std.Dev.
## ptid (Intercept) 5.086e-03 0.071313
## ItemNumber (Intercept) 1.135e-02 0.106517
## lab (Intercept) 6.591e-05 0.008118
## Counterbalance.List (Intercept) 0.000e+00 0.000000
## Residual 1.188e-02 0.108983
## Number of obs: 27474, groups:
## ptid, 900; ItemNumber, 37; lab, 12; Counterbalance.List, 4
##
## Fixed effects:
## Estimate Std. Error t value
## (Intercept) 7.560575 0.017885 422.7
## CueDirection 0.003514 0.001865 1.9
## SentenceDirection 0.007074 0.001867 3.8
## CueDirection:SentenceDirection -0.001225 0.002640 -0.5
##
## Correlation of Fixed Effects:
## (Intr) CDrctn SntncD
## CueDirectin -0.052
## SentncDrctn -0.052 0.503
## CDrctn:SntD 0.037 -0.708 -0.708confint(lmer_obj_lo_mono, method = "Wald")## 2.5 % 97.5 %
## .sig01 NA NA
## .sig02 NA NA
## .sig03 NA NA
## .sig04 NA NA
## .sigma NA NA
## (Intercept) 7.525521420 7.595627663
## CueDirection -0.000142266 0.007169913
## SentenceDirection 0.003415046 0.010732459
## CueDirection:SentenceDirection -0.006399898 0.003950471Move times
lmer_obj_mt_mono = lme4::lmer(log(MoveTime) ~ CueDirection*SentenceDirection + (1|ptid) + (1|lab) + (1|ItemNumber) + (1|Counterbalance.List),
data = ace_filtered_mono %>% filter(Accuracy == 1))
summary(lmer_obj_mt_mono)## Linear mixed model fit by REML ['lmerMod']
## Formula: log(MoveTime) ~ CueDirection * SentenceDirection + (1 | ptid) +
## (1 | lab) + (1 | ItemNumber) + (1 | Counterbalance.List)
## Data: ace_filtered_mono %>% filter(Accuracy == 1)
##
## REML criterion at convergence: -10786.9
##
## Scaled residuals:
## Min 1Q Median 3Q Max
## -6.9926 -0.5671 -0.0211 0.5433 11.2283
##
## Random effects:
## Groups Name Variance Std.Dev.
## ptid (Intercept) 0.0612485 0.24748
## ItemNumber (Intercept) 0.0001017 0.01009
## lab (Intercept) 0.0114219 0.10687
## Counterbalance.List (Intercept) 0.0000000 0.00000
## Residual 0.0345468 0.18587
## Number of obs: 27474, groups:
## ptid, 900; ItemNumber, 37; lab, 12; Counterbalance.List, 4
##
## Fixed effects:
## Estimate Std. Error t value
## (Intercept) 5.838544 0.032121 181.76
## CueDirection -0.076265 0.003182 -23.97
## SentenceDirection -0.002805 0.003184 -0.88
## CueDirection:SentenceDirection -0.002209 0.004504 -0.49
##
## Correlation of Fixed Effects:
## (Intr) CDrctn SntncD
## CueDirectin -0.050
## SentncDrctn -0.050 0.503
## CDrctn:SntD 0.035 -0.708 -0.708confint(lmer_obj_mt_mono, method = "Wald")## 2.5 % 97.5 %
## .sig01 NA NA
## .sig02 NA NA
## .sig03 NA NA
## .sig04 NA NA
## .sigma NA NA
## (Intercept) 5.775587195 5.901500944
## CueDirection -0.082501613 -0.070028911
## SentenceDirection -0.009045623 0.003435517
## CueDirection:SentenceDirection -0.011037264 0.006618794Bilingual
Accuracy
lmer_obj = lme4::glmer(Accuracy ~ CueDirection*SentenceDirection + (1|ptid) + (1|lab) + (1|ItemNumber) + (1|Counterbalance.List),
data = ace_filtered_bi, family = "binomial")## Warning in checkConv(attr(opt, "derivs"), opt$par, ctrl = control
## $checkConv, : unable to evaluate scaled gradient## Warning in checkConv(attr(opt, "derivs"), opt$par, ctrl = control
## $checkConv, : Model failed to converge: degenerate Hessian with 1 negative
## eigenvaluessummary(lmer_obj)## Generalized linear mixed model fit by maximum likelihood (Laplace
## Approximation) [glmerMod]
## Family: binomial ( logit )
## Formula: Accuracy ~ CueDirection * SentenceDirection + (1 | ptid) + (1 |
## lab) + (1 | ItemNumber) + (1 | Counterbalance.List)
## Data: ace_filtered_bi
##
## AIC BIC logLik deviance df.resid
## 349.3 405.8 -166.7 333.3 8614
##
## Scaled residuals:
## Min 1Q Median 3Q Max
## -32.965 0.038 0.047 0.065 0.092
##
## Random effects:
## Groups Name Variance Std.Dev.
## ptid (Intercept) 7.180e-08 0.000268
## ItemNumber (Intercept) 2.021e-01 0.449560
## lab (Intercept) 0.000e+00 0.000000
## Counterbalance.List (Intercept) 0.000e+00 0.000000
## Number of obs: 8622, groups:
## ptid, 375; ItemNumber, 32; lab, 6; Counterbalance.List, 4
##
## Fixed effects:
## Estimate Std. Error z value Pr(>|z|)
## (Intercept) 7.0829 0.7318 9.679 <2e-16 ***
## CueDirection -1.7991 0.7646 -2.353 0.0186 *
## SentenceDirection -1.1120 0.8172 -1.361 0.1736
## CueDirection:SentenceDirection 1.9805 0.9760 2.029 0.0424 *
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Correlation of Fixed Effects:
## (Intr) CDrctn SntncD
## CueDirectin -0.893
## SentncDrctn -0.838 0.801
## CDrctn:SntD 0.702 -0.783 -0.837
## convergence code: 0
## unable to evaluate scaled gradient
## Model failed to converge: degenerate Hessian with 1 negative eigenvaluesCrosstabs:
with(ace_filtered_bi, table(CueDirection, Accuracy))## Accuracy
## CueDirection 0 1
## 0 8 4320
## 1 17 4277Lift-off times
lmer_obj_lo_bi = lme4::lmer(log(LiftOffLatency) ~ CueDirection*SentenceDirection + (1|ptid) + (1|lab) + (1|ItemNumber) + (1|Counterbalance.List),
data = ace_filtered_bi %>% filter(Accuracy == 1))
summary(lmer_obj_lo_bi)## Linear mixed model fit by REML ['lmerMod']
## Formula: log(LiftOffLatency) ~ CueDirection * SentenceDirection + (1 |
## ptid) + (1 | lab) + (1 | ItemNumber) + (1 | Counterbalance.List)
## Data: ace_filtered_bi %>% filter(Accuracy == 1)
##
## REML criterion at convergence: -10062.5
##
## Scaled residuals:
## Min 1Q Median 3Q Max
## -3.2112 -0.5987 -0.1423 0.4040 6.6724
##
## Random effects:
## Groups Name Variance Std.Dev.
## ptid (Intercept) 0.0074426 0.08627
## ItemNumber (Intercept) 0.0102353 0.10117
## lab (Intercept) 0.0001347 0.01161
## Counterbalance.List (Intercept) 0.0003460 0.01860
## Residual 0.0159561 0.12632
## Number of obs: 8597, groups:
## ptid, 375; ItemNumber, 32; lab, 6; Counterbalance.List, 4
##
## Fixed effects:
## Estimate Std. Error t value
## (Intercept) 7.602455 0.021486 353.8
## CueDirection -0.003453 0.003874 -0.9
## SentenceDirection 0.020360 0.003877 5.3
## CueDirection:SentenceDirection 0.001503 0.005484 0.3
##
## Correlation of Fixed Effects:
## (Intr) CDrctn SntncD
## CueDirectin -0.090
## SentncDrctn -0.089 0.495
## CDrctn:SntD 0.063 -0.703 -0.703confint(lmer_obj_lo_bi, method = "Wald")## 2.5 % 97.5 %
## .sig01 NA NA
## .sig02 NA NA
## .sig03 NA NA
## .sig04 NA NA
## .sigma NA NA
## (Intercept) 7.560343840 7.644565725
## CueDirection -0.011046230 0.004140343
## SentenceDirection 0.012760760 0.027958524
## CueDirection:SentenceDirection -0.009244081 0.012250933Move times
lmer_obj_mt_bi = lme4::lmer(log(MoveTime) ~ CueDirection*SentenceDirection + (1|ptid) + (1|lab) + (1|ItemNumber) + (1|Counterbalance.List),
data = ace_filtered_bi %>% filter(Accuracy == 1))
summary(lmer_obj_mt_bi)## Linear mixed model fit by REML ['lmerMod']
## Formula: log(MoveTime) ~ CueDirection * SentenceDirection + (1 | ptid) +
## (1 | lab) + (1 | ItemNumber) + (1 | Counterbalance.List)
## Data: ace_filtered_bi %>% filter(Accuracy == 1)
##
## REML criterion at convergence: -2205.7
##
## Scaled residuals:
## Min 1Q Median 3Q Max
## -6.6161 -0.5645 -0.0402 0.5032 9.3056
##
## Random effects:
## Groups Name Variance Std.Dev.
## ptid (Intercept) 6.442e-02 2.538e-01
## ItemNumber (Intercept) 6.796e-08 2.607e-04
## lab (Intercept) 4.865e-03 6.975e-02
## Counterbalance.List (Intercept) 3.147e-15 5.610e-08
## Residual 3.843e-02 1.960e-01
## Number of obs: 8597, groups:
## ptid, 375; ItemNumber, 32; lab, 6; Counterbalance.List, 4
##
## Fixed effects:
## Estimate Std. Error t value
## (Intercept) 5.759157 0.032438 177.54
## CueDirection -0.101308 0.006007 -16.87
## SentenceDirection -0.005691 0.006013 -0.95
## CueDirection:SentenceDirection 0.008842 0.008500 1.04
##
## Correlation of Fixed Effects:
## (Intr) CDrctn SntncD
## CueDirectin -0.092
## SentncDrctn -0.092 0.494
## CDrctn:SntD 0.064 -0.703 -0.703confint(lmer_obj_mt_bi, method = "Wald")## 2.5 % 97.5 %
## .sig01 NA NA
## .sig02 NA NA
## .sig03 NA NA
## .sig04 NA NA
## .sigma NA NA
## (Intercept) 5.695579816 5.82273390
## CueDirection -0.113080133 -0.08953498
## SentenceDirection -0.017476200 0.00609423
## CueDirection:SentenceDirection -0.007816363 0.02550112Visualizations of the ACE effect
Lift-off times
Move times
Mixed effects models, individual labs
Accuracy
The accuracy check analysis is performed below, for all labs at once (with lab as a random factor).
Lift off times
# Do analyses by lab
x = ace_filtered %>%
filter(Accuracy == 1) %>%
split(.$lab) %>%
map( ~ lme4::lmer(log(LiftOffLatency) ~ CueDirection*SentenceDirection + (1|ptid) + (1|ItemNumber) + (1|Counterbalance.List),
data = .)
) %>%
map(my_lmer_summary)# %>% flatten_dfr()
# There is a bug in flatten_dfr, so we can't use it
## Add overall fit to data frame
x[["ALL (mono)"]] = my_lmer_summary(lmer_obj_lo_mono)
x[["ALL (bi)"]] = my_lmer_summary(lmer_obj_lo_bi)
Move times
# Do analyses by lab
x = ace_filtered %>%
filter(Accuracy == 1) %>%
split(.$lab) %>%
map( ~ lme4::lmer(log(MoveTime) ~ CueDirection*SentenceDirection + (1|ptid) + (1|ItemNumber) + (1|Counterbalance.List),
data = .)
) %>%
map(my_lmer_summary)# %>% flatten_dfr()
# There is a bug in flatten_dfr, so we can't use it
## Add overall fit to data frame
x[["ALL (mono)"]] = my_lmer_summary(lmer_obj_mt_mono)
x[["ALL (bi)"]] = my_lmer_summary(lmer_obj_mt_bi)