Pre-registered analysis, with real data
Richard D. Morey
02/07/2018
This document includes the initial, pre-registered analysis of the ACE effect data. The meta-analytic analysis is performed by simply including the lab as a random effect.
This initial pre-registration is done on blinded data.
This version of the document has been modified to perform the pre-registered analyses. Several tweaks were needed, including:
- Uncommenting lines that performed the response time cleaning.
- Properly restricting the response time analyses to correct trials.
- Incorporating information about left-handed participants in order to exclude them.
- Rearranging the document – putting the LME analyses first – so that some quantities computed from the LMEs could be added to graphs.
- Make sure that variables that should be factors are factors (although this won’t effect the results).
In addition, code has been included to (optionally) perform some additional cleaning that was determined to be needed after looking at the data. A small proportion of participants have very fast movement times. We feared this indicates that these participants have adopted an idiosyncratic response strategy, such as responding with two hands; otherwise, we see no way that the movement times could be as fast as they are. Thus, we adopted the following rule: if 20% or more of a participant’s response times would be removed under the pre-registered response time filtering rules, then that participant is eliminated from the analysis.
This step in the filtering can be applied by uncommenting the line containing mean(valid.rt) > .2
Data loading
Data cleaning
Details of the pre-registered data cleaning can be found in the pre-registration documents (initial and addendum).
Mixed effects models, all labs
Monolingual
Accuracy
There appears to be a very small, albeit detectable, effect on Accuracy. Overall accuracy is very high, but there does seem to be a slight advantage for trials of a particular cue direction.
lmer_obj = lme4::glmer(Accuracy ~ CueDirection*SentenceDirection + (1|ptid) + (1|lab) + (1|ItemNumber) + (1|Counterbalance.List),
data = ace_filtered_mono, family = "binomial")
summary(lmer_obj)## Generalized linear mixed model fit by maximum likelihood (Laplace
## Approximation) [glmerMod]
## Family: binomial ( logit )
## Formula: Accuracy ~ CueDirection * SentenceDirection + (1 | ptid) + (1 |
## lab) + (1 | ItemNumber) + (1 | Counterbalance.List)
## Data: ace_filtered_mono
##
## AIC BIC logLik deviance df.resid
## 1247.2 1313.0 -615.6 1231.2 27567
##
## Scaled residuals:
## Min 1Q Median 3Q Max
## -9.3925 0.0073 0.0117 0.0119 0.4378
##
## Random effects:
## Groups Name Variance Std.Dev.
## ptid (Intercept) 18.54 4.306
## ItemNumber (Intercept) 0.00 0.000
## lab (Intercept) 0.00 0.000
## Counterbalance.List (Intercept) 0.00 0.000
## Number of obs: 27575, groups:
## ptid, 900; ItemNumber, 37; lab, 12; Counterbalance.List, 4
##
## Fixed effects:
## Estimate Std. Error z value Pr(>|z|)
## (Intercept) 9.83633 0.66728 14.741 <2e-16 ***
## CueDirection1 -1.02639 0.32332 -3.175 0.0015 **
## SentenceDirection1 -0.06466 0.37832 -0.171 0.8643
## CueDirection1:SentenceDirection1 0.08985 0.45197 0.199 0.8424
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Correlation of Fixed Effects:
## (Intr) CDrct1 SntnD1
## CueDirectn1 -0.373
## SntncDrctn1 -0.295 0.606
## CDrctn1:SD1 0.252 -0.716 -0.838This can be seen in the crosstabs:
with(ace_filtered_mono, table(CueDirection, Accuracy))## Accuracy
## CueDirection 0 1
## 0 29 13725
## 1 72 13749The odds of being correct increase by about 2.5 when CueDirection is 0. In light of the high accuracy, this does not seem problematic.
Lift-off times
lmer_obj_lo_mono = lme4::lmer(LiftOffLatency ~ CueDirection*SentenceDirection + (1|ptid) + (1|lab) + (1|ItemNumber) + (1|Counterbalance.List),
data = ace_filtered_mono %>% filter(Accuracy == 1))
summary(lmer_obj_lo_mono)## Linear mixed model fit by REML ['lmerMod']
## Formula: LiftOffLatency ~ CueDirection * SentenceDirection + (1 | ptid) +
## (1 | lab) + (1 | ItemNumber) + (1 | Counterbalance.List)
## Data: ace_filtered_mono %>% filter(Accuracy == 1)
##
## REML criterion at convergence: 381019.3
##
## Scaled residuals:
## Min 1Q Median 3Q Max
## -5.0453 -0.5423 -0.1406 0.3310 11.5406
##
## Random effects:
## Groups Name Variance Std.Dev.
## ptid (Intercept) 21903 148.00
## ItemNumber (Intercept) 44087 209.97
## lab (Intercept) 247 15.72
## Counterbalance.List (Intercept) 0 0.00
## Residual 56348 237.38
## Number of obs: 27474, groups:
## ptid, 900; ItemNumber, 37; lab, 12; Counterbalance.List, 4
##
## Fixed effects:
## Estimate Std. Error t value
## (Intercept) 1951.967 35.301 55.30
## CueDirection1 6.697 4.063 1.65
## SentenceDirection1 8.847 4.066 2.18
## CueDirection1:SentenceDirection1 -2.863 5.751 -0.50
##
## Correlation of Fixed Effects:
## (Intr) CDrct1 SntnD1
## CueDirectn1 -0.058
## SntncDrctn1 -0.058 0.503
## CDrctn1:SD1 0.041 -0.708 -0.708confint(lmer_obj_lo_mono, method = "Wald")## 2.5 % 97.5 %
## .sig01 NA NA
## .sig02 NA NA
## .sig03 NA NA
## .sig04 NA NA
## .sigma NA NA
## (Intercept) 1882.7789335 2021.154561
## CueDirection1 -1.2661030 14.660222
## SentenceDirection1 0.8782574 16.815944
## CueDirection1:SentenceDirection1 -14.1350407 8.408453Move times
lmer_obj_mt_mono = lme4::lmer(MoveTime ~ CueDirection*SentenceDirection + (1|ptid) + (1|lab) + (1|ItemNumber) + (1|Counterbalance.List),
data = ace_filtered_mono %>% filter(Accuracy == 1))
summary(lmer_obj_mt_mono)## Linear mixed model fit by REML ['lmerMod']
## Formula: MoveTime ~ CueDirection * SentenceDirection + (1 | ptid) + (1 |
## lab) + (1 | ItemNumber) + (1 | Counterbalance.List)
## Data: ace_filtered_mono %>% filter(Accuracy == 1)
##
## REML criterion at convergence: 314883.4
##
## Scaled residuals:
## Min 1Q Median 3Q Max
## -5.4502 -0.5349 -0.0837 0.4148 22.5134
##
## Random effects:
## Groups Name Variance Std.Dev.
## ptid (Intercept) 7052.17 83.977
## ItemNumber (Intercept) 10.92 3.305
## lab (Intercept) 1331.15 36.485
## Counterbalance.List (Intercept) 0.00 0.000
## Residual 4899.85 69.999
## Number of obs: 27474, groups:
## ptid, 900; ItemNumber, 37; lab, 12; Counterbalance.List, 4
##
## Fixed effects:
## Estimate Std. Error t value
## (Intercept) 362.3911 10.9659 33.05
## CueDirection1 -27.1717 1.1983 -22.68
## SentenceDirection1 -0.8938 1.1991 -0.75
## CueDirection1:SentenceDirection1 -0.6444 1.6963 -0.38
##
## Correlation of Fixed Effects:
## (Intr) CDrct1 SntnD1
## CueDirectn1 -0.055
## SntncDrctn1 -0.055 0.503
## CDrctn1:SD1 0.039 -0.708 -0.708confint(lmer_obj_mt_mono, method = "Wald")## 2.5 % 97.5 %
## .sig01 NA NA
## .sig02 NA NA
## .sig03 NA NA
## .sig04 NA NA
## .sigma NA NA
## (Intercept) 340.898259 383.883983
## CueDirection1 -29.520305 -24.823083
## SentenceDirection1 -3.243934 1.456418
## CueDirection1:SentenceDirection1 -3.969019 2.680219Bilingual
Accuracy
lmer_obj = lme4::glmer(Accuracy ~ CueDirection*SentenceDirection + (1|ptid) + (1|lab) + (1|ItemNumber) + (1|Counterbalance.List),
data = ace_filtered_bi, family = "binomial")## Warning in checkConv(attr(opt, "derivs"), opt$par, ctrl = control
## $checkConv, : unable to evaluate scaled gradient## Warning in checkConv(attr(opt, "derivs"), opt$par, ctrl = control
## $checkConv, : Model failed to converge: degenerate Hessian with 1 negative
## eigenvaluessummary(lmer_obj)## Generalized linear mixed model fit by maximum likelihood (Laplace
## Approximation) [glmerMod]
## Family: binomial ( logit )
## Formula: Accuracy ~ CueDirection * SentenceDirection + (1 | ptid) + (1 |
## lab) + (1 | ItemNumber) + (1 | Counterbalance.List)
## Data: ace_filtered_bi
##
## AIC BIC logLik deviance df.resid
## 349.3 405.8 -166.7 333.3 8614
##
## Scaled residuals:
## Min 1Q Median 3Q Max
## -32.965 0.038 0.047 0.065 0.092
##
## Random effects:
## Groups Name Variance Std.Dev.
## ptid (Intercept) 7.467e-08 2.733e-04
## ItemNumber (Intercept) 2.021e-01 4.496e-01
## lab (Intercept) 0.000e+00 0.000e+00
## Counterbalance.List (Intercept) 1.015e-10 1.007e-05
## Number of obs: 8622, groups:
## ptid, 375; ItemNumber, 32; lab, 6; Counterbalance.List, 4
##
## Fixed effects:
## Estimate Std. Error z value Pr(>|z|)
## (Intercept) 7.0829 0.7317 9.679 <2e-16 ***
## CueDirection1 -1.7991 0.7645 -2.353 0.0186 *
## SentenceDirection1 -1.1120 0.8172 -1.361 0.1736
## CueDirection1:SentenceDirection1 1.9805 0.9759 2.029 0.0424 *
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Correlation of Fixed Effects:
## (Intr) CDrct1 SntnD1
## CueDirectn1 -0.893
## SntncDrctn1 -0.838 0.801
## CDrctn1:SD1 0.702 -0.783 -0.837
## convergence code: 0
## unable to evaluate scaled gradient
## Model failed to converge: degenerate Hessian with 1 negative eigenvaluesCrosstabs:
with(ace_filtered_bi, table(CueDirection, Accuracy))## Accuracy
## CueDirection 0 1
## 0 8 4320
## 1 17 4277Lift-off times
lmer_obj_lo_bi = lme4::lmer(LiftOffLatency ~ CueDirection*SentenceDirection + (1|ptid) + (1|lab) + (1|ItemNumber) + (1|Counterbalance.List),
data = ace_filtered_bi %>% filter(Accuracy == 1))
summary(lmer_obj_lo_bi)## Linear mixed model fit by REML ['lmerMod']
## Formula: LiftOffLatency ~ CueDirection * SentenceDirection + (1 | ptid) +
## (1 | lab) + (1 | ItemNumber) + (1 | Counterbalance.List)
## Data: ace_filtered_bi %>% filter(Accuracy == 1)
##
## REML criterion at convergence: 123115.5
##
## Scaled residuals:
## Min 1Q Median 3Q Max
## -2.9292 -0.5467 -0.1657 0.3052 8.9875
##
## Random effects:
## Groups Name Variance Std.Dev.
## ptid (Intercept) 35894 189.46
## ItemNumber (Intercept) 44288 210.45
## lab (Intercept) 654 25.57
## Counterbalance.List (Intercept) 1404 37.47
## Residual 86328 293.82
## Number of obs: 8597, groups:
## ptid, 375; ItemNumber, 32; lab, 6; Counterbalance.List, 4
##
## Fixed effects:
## Estimate Std. Error t value
## (Intercept) 2043.991 44.811 45.61
## CueDirection1 -8.532 9.011 -0.95
## SentenceDirection1 38.777 9.018 4.30
## CueDirection1:SentenceDirection1 4.115 12.754 0.32
##
## Correlation of Fixed Effects:
## (Intr) CDrct1 SntnD1
## CueDirectn1 -0.100
## SntncDrctn1 -0.099 0.495
## CDrctn1:SD1 0.070 -0.704 -0.703confint(lmer_obj_lo_bi, method = "Wald")## 2.5 % 97.5 %
## .sig01 NA NA
## .sig02 NA NA
## .sig03 NA NA
## .sig04 NA NA
## .sigma NA NA
## (Intercept) 1956.16206 2131.819664
## CueDirection1 -26.19288 9.129618
## SentenceDirection1 21.10247 56.450582
## CueDirection1:SentenceDirection1 -20.88255 29.113091Move times
lmer_obj_mt_bi = lme4::lmer(MoveTime ~ CueDirection*SentenceDirection + (1|ptid) + (1|lab) + (1|ItemNumber) + (1|Counterbalance.List),
data = ace_filtered_bi %>% filter(Accuracy == 1))
summary(lmer_obj_mt_bi)## Linear mixed model fit by REML ['lmerMod']
## Formula: MoveTime ~ CueDirection * SentenceDirection + (1 | ptid) + (1 |
## lab) + (1 | ItemNumber) + (1 | Counterbalance.List)
## Data: ace_filtered_bi %>% filter(Accuracy == 1)
##
## REML criterion at convergence: 99414.5
##
## Scaled residuals:
## Min 1Q Median 3Q Max
## -4.1612 -0.5028 -0.0942 0.3492 15.4277
##
## Random effects:
## Groups Name Variance Std.Dev.
## ptid (Intercept) 6.529e+03 8.080e+01
## ItemNumber (Intercept) 0.000e+00 0.000e+00
## lab (Intercept) 4.468e+02 2.114e+01
## Counterbalance.List (Intercept) 3.113e-10 1.764e-05
## Residual 5.326e+03 7.298e+01
## Number of obs: 8597, groups:
## ptid, 375; ItemNumber, 32; lab, 6; Counterbalance.List, 4
##
## Fixed effects:
## Estimate Std. Error t value
## (Intercept) 335.923 9.973 33.68
## CueDirection1 -33.888 2.236 -15.16
## SentenceDirection1 -0.966 2.238 -0.43
## CueDirection1:SentenceDirection1 1.750 3.164 0.55
##
## Correlation of Fixed Effects:
## (Intr) CDrct1 SntnD1
## CueDirectn1 -0.111
## SntncDrctn1 -0.111 0.494
## CDrctn1:SD1 0.078 -0.703 -0.703confint(lmer_obj_mt_bi, method = "Wald")## 2.5 % 97.5 %
## .sig01 NA NA
## .sig02 NA NA
## .sig03 NA NA
## .sig04 NA NA
## .sigma NA NA
## (Intercept) 316.376938 355.468573
## CueDirection1 -38.270562 -29.506215
## SentenceDirection1 -5.352865 3.420800
## CueDirection1:SentenceDirection1 -4.451143 7.951044Visualizations of the ACE effect
Lift-off times
Move times
Mixed effects models, individual labs
Accuracy
The accuracy check analysis is performed above, for all labs at once (with lab as a random factor).
Lift off times
# Do analyses by lab
x = ace_filtered %>%
filter(Accuracy == 1) %>%
split(.$lab) %>%
map( ~ lme4::lmer(LiftOffLatency ~ CueDirection*SentenceDirection + (1|ptid) + (1|ItemNumber) + (1|Counterbalance.List),
data = .)
) %>%
map(my_lmer_summary)# %>% flatten_dfr()
# There is a bug in flatten_dfr, so we can't use it
## Add overall fit to data frame
x[["ALL (mono)"]] = my_lmer_summary(lmer_obj_lo_mono)
x[["ALL (bi)"]] = my_lmer_summary(lmer_obj_lo_bi)
Move times
# Do analyses by lab
x = ace_filtered %>%
filter(Accuracy == 1) %>%
split(.$lab) %>%
map( ~ lme4::lmer(MoveTime ~ CueDirection*SentenceDirection + (1|ptid) + (1|ItemNumber) + (1|Counterbalance.List),
data = .)
) %>%
map(my_lmer_summary)# %>% flatten_dfr()
# There is a bug in flatten_dfr, so we can't use it
## Add overall fit to data frame
x[["ALL (mono)"]] = my_lmer_summary(lmer_obj_mt_mono)
x[["ALL (bi)"]] = my_lmer_summary(lmer_obj_mt_bi)