ME MA exploration for paper
Molly Lewis
2017-11-25
Summary of coded data
## study_ID short_cite expt_num
## merriman1989:17 Merriman et al. (1989) :17 1 :108
## frank2016 :12 Frank et al. (2016) :12 1-2: 1
## gollek2016 :10 Gollek & Doherty (2016) :10 2 : 25
## jarvis2004 : 9 Jarvis et al. (2004) : 9 2a : 4
## markman2003 : 8 Markman, Wasow, & Hansen (2003): 8 2b : 2
## merriman1991: 8 Merriman & Schuster (1991) : 8 3 : 15
## (Other) :93 (Other) :93 4 : 2
## response_mode dependent_measure native_lang
## behavior :131 looking_time_change: 10 American English:42
## eye-tracking: 26 target_selection :147 English :32
## British English :12
## Canadian English: 5
## German Germany : 5
## (Other) :21
## NA's :40
## infant_type n_1
## typical :135 Min. : 5.00
## bilingual : 10 1st Qu.:15.00
## ASD : 4 Median :16.00
## specific language imparement (SLI) : 3 Mean :19.06
## trilingual : 2 3rd Qu.:22.00
## deaf/hard-of-hearing preschoolers w: 1 Max. :72.00
## (Other) : 2
## mean_age_1 x_1 x_2 SD_1
## Min. : 445.2 Min. :-0.0300 Min. :0.0000 Min. :0.02196
## 1st Qu.: 730.5 1st Qu.: 0.5308 1st Qu.:0.5000 1st Qu.:0.16606
## Median : 974.0 Median : 0.6738 Median :0.5000 Median :0.23000
## Mean :1133.3 Mean : 0.6538 Mean :0.4518 Mean :0.27439
## 3rd Qu.:1461.0 3rd Qu.: 0.8295 3rd Qu.:0.5000 3rd Qu.:0.29866
## Max. :3926.4 Max. : 0.9860 Max. :0.5000 Max. :2.08000
## NA's :15 NA's :16 NA's :25
## t d d_var object_stimulus
## Min. :-3.140 Min. :-0.9467 Min. :0.02801 digital :35
## 1st Qu.: 1.555 1st Qu.: 0.2436 1st Qu.:0.10614 objects :76
## Median : 3.645 Median : 1.0790 Median :0.13219 objects/paper: 1
## Mean : 5.113 Mean : 1.4551 Mean :0.17196 paper :45
## 3rd Qu.: 5.168 3rd Qu.: 1.8525 3rd Qu.:0.20219
## Max. :31.700 Max. :11.4100 Max. :0.86701
## NA's :119 NA's :93 NA's :104
## N_AFC mean_comprehension_vocab mean_production_vocab
## N_AFC-2:138 Min. :156.0 Min. : 35.0
## N_AFC-3: 11 1st Qu.:226.8 1st Qu.: 76.0
## N_AFC-4: 6 Median :295.7 Median :158.9
## N_AFC-5: 2 Mean :292.2 Mean :191.7
## 3rd Qu.:357.0 3rd Qu.:225.8
## Max. :449.0 Max. :534.0
## NA's :136 NA's :128
## N_langs d_notes
## Mode:logical dissertation : 4
## NA's:157 looking and pointing - I report pointing : 3
## sd imputed based on 3-4 greek bilinguals (from t): 3
## ASD: high-functioning : 1
## ASD: high-risk : 1
## (Other) : 4
## NA's :141
## infant_type2 ME_trial_type data_source
## typical :135 FN:136 figure : 13
## bilingual : 10 NN: 21 paper :107
## ASD : 4 paper - some calculations: 7
## SLI : 3 paper - some imputation : 4
## trilingual : 2 paper/author : 26
## deaf/hard-of-hearing: 1 raw data from paper : 0
## (Other) : 2
## lab_group d_calc d_var_calc mean_age
## merriman:48 Min. :-0.9467 Min. :0.01435 Min. : 14.63
## framkm :12 1st Qu.: 0.2887 1st Qu.:0.06383 1st Qu.: 24.00
## gollek :10 Median : 0.8734 Median :0.08558 Median : 32.00
## markman :10 Mean : 1.3293 Mean :0.22861 Mean : 37.23
## estis : 7 3rd Qu.: 1.8500 3rd Qu.:0.17323 3rd Qu.: 48.00
## horst : 6 Max. : 9.3462 Max. :2.97835 Max. :129.00
## (Other) :64
## year
## Min. :1988
## 1st Qu.:1998
## Median :2004
## Mean :2005
## 3rd Qu.:2014
## Max. :2017
## | infant_type2 | n |
|---|---|
| ASD | 4 |
| bilingual | 10 |
| deaf/hard-of-hearing | 1 |
| DS | 1 |
| late_talkers | 1 |
| SLI | 3 |
| trilingual | 2 |
| typical | 135 |
| ME_trial_type | n |
|---|---|
| FN | 136 |
| NN | 21 |
There are 157 conditions. There are 43 in the sample. There are 28 lab groups in the sample (lab group is defined as the senior author (common across studies), or the last author).
| !is.na(mean_production_vocab) | infant_type | n |
|---|---|---|
| FALSE | ASD | 3 |
| FALSE | bilingual | 8 |
| FALSE | late_talkers | 1 |
| FALSE | specific language imparement (SLI) | 3 |
| FALSE | trilingual | 1 |
| FALSE | typical | 112 |
| TRUE | ASD | 1 |
| TRUE | bilingual | 2 |
| TRUE | deaf/hard-of-hearing preschoolers w | 1 |
| TRUE | DS | 1 |
| TRUE | trilingual | 1 |
| TRUE | typical | 23 |
| !is.na(mean_comprehension_vocab) | n |
|---|---|
| FALSE | 136 |
| TRUE | 21 |
Forest plot
Fail-safe-N
| dataset | fsn_string |
|---|---|
| mutex | 20713 |
Funnel plot
With age and response mode and trial type moderators 
Eggers test
| dataset | egg.random.z | egg.random.p |
|---|---|---|
| mutex | 17.31427 | 0 |
Evidence for skew. Lots of heterogenity.
P-curve
Stouffer test
| pp.measure | Z.pp | p.Z.pp | sig |
|---|---|---|---|
| ppr.full | -27.16405 | 0 | TRUE |
Strong evidence for left skew (no phacking).
MA models
Simple (no moderators)
##
## Multivariate Meta-Analysis Model (k = 157; method: REML)
##
## logLik Deviance AIC BIC AICc
## -437.5768 875.1535 879.1535 885.2532 879.2319
##
## Variance Components:
##
## estim sqrt nlvls fixed factor
## sigma^2 0.6353 0.7970 44 no short_cite
##
## Test for Heterogeneity:
## Q(df = 156) = 1379.1191, p-val < .0001
##
## Model Results:
##
## estimate se zval pval ci.lb ci.ub
## 1.0576 0.1256 8.4235 <.0001 0.8115 1.3037 ***
##
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1ME_trial type
##
## Multivariate Meta-Analysis Model (k = 157; method: REML)
##
## logLik Deviance AIC BIC AICc
## -427.1495 854.2990 860.2990 869.4292 860.4579
##
## Variance Components:
##
## estim sqrt nlvls fixed factor
## sigma^2 0.8893 0.9430 44 no short_cite
##
## Test for Residual Heterogeneity:
## QE(df = 155) = 1371.6365, p-val < .0001
##
## Test of Moderators (coefficient(s) 2):
## QM(df = 1) = 21.5920, p-val < .0001
##
## Model Results:
##
## estimate se zval pval ci.lb ci.ub
## intrcpt 1.2065 0.1498 8.0569 <.0001 0.9130 1.5000 ***
## ME_trial_typeNN -0.7107 0.1529 -4.6467 <.0001 -1.0105 -0.4109 ***
##
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1Age
##
## Multivariate Meta-Analysis Model (k = 157; method: REML)
##
## logLik Deviance AIC BIC AICc
## -356.4074 712.8148 720.8148 732.9626 721.0833
##
## Variance Components:
##
## estim sqrt nlvls fixed factor
## sigma^2 0.5954 0.7716 44 no short_cite
##
## Test for Residual Heterogeneity:
## QE(df = 154) = 1121.3202, p-val < .0001
##
## Test of Moderators (coefficient(s) 2:3):
## QM(df = 2) = 160.4053, p-val < .0001
##
## Model Results:
##
## estimate se zval pval ci.lb ci.ub
## intrcpt -0.1419 0.1664 -0.8528 0.3938 -0.4681 0.1842
## mean_age 0.0369 0.0031 11.9606 <.0001 0.0309 0.0430 ***
## ME_trial_typeNN -0.9187 0.1499 -6.1285 <.0001 -1.2125 -0.6249 ***
##
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1Age is reliable, controling for ME_trial_type
Vocab
(we don’t have production scores for any NN trials)
##
## Multivariate Meta-Analysis Model (k = 29; method: REML)
##
## logLik Deviance AIC BIC AICc
## -27.8365 55.6730 61.6730 65.5605 62.7165
##
## Variance Components:
##
## estim sqrt nlvls fixed factor
## sigma^2 0.4073 0.6382 11 no short_cite
##
## Test for Residual Heterogeneity:
## QE(df = 27) = 131.2620, p-val < .0001
##
## Test of Moderators (coefficient(s) 2):
## QM(df = 1) = 8.6147, p-val = 0.0033
##
## Model Results:
##
## estimate se zval pval ci.lb ci.ub
## intrcpt 0.4909 0.2619 1.8744 0.0609 -0.0224 1.0041
## mean_production_vocab 0.0024 0.0008 2.9351 0.0033 0.0008 0.0039
##
## intrcpt .
## mean_production_vocab **
##
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1Vocab is reliable
Age + Vocab
Plots
Correlation between age and vocab:
| estimate | statistic | p.value | parameter | conf.low | conf.high | method | alternative |
|---|---|---|---|---|---|---|---|
| 0.3512851 | 1.94958 | 0.0616829 | 27 | -0.0174698 | 0.6359177 | Pearson’s product-moment correlation | two.sided |
Age and vocab are weakly correlated, not quite significant. Note the heterskadastisky!
Age and vocab vs. effect size
Typical only
Models
##
## Multivariate Meta-Analysis Model (k = 29; method: REML)
##
## logLik Deviance AIC BIC AICc
## -30.6257 61.2515 67.2515 71.1390 68.2949
##
## Variance Components:
##
## estim sqrt nlvls fixed factor
## sigma^2 0.6174 0.7857 11 no short_cite
##
## Test for Residual Heterogeneity:
## QE(df = 27) = 153.7200, p-val < .0001
##
## Test of Moderators (coefficient(s) 2):
## QM(df = 1) = 1.1347, p-val = 0.2868
##
## Model Results:
##
## estimate se zval pval ci.lb ci.ub
## intrcpt 0.5485 0.4807 1.1410 0.2539 -0.3937 1.4907
## mean_age 0.0155 0.0146 1.0652 0.2868 -0.0130 0.0440
##
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1##
## Multivariate Meta-Analysis Model (k = 29; method: REML)
##
## logLik Deviance AIC BIC AICc
## -27.8365 55.6730 61.6730 65.5605 62.7165
##
## Variance Components:
##
## estim sqrt nlvls fixed factor
## sigma^2 0.4073 0.6382 11 no short_cite
##
## Test for Residual Heterogeneity:
## QE(df = 27) = 131.2620, p-val < .0001
##
## Test of Moderators (coefficient(s) 2):
## QM(df = 1) = 8.6147, p-val = 0.0033
##
## Model Results:
##
## estimate se zval pval ci.lb ci.ub
## intrcpt 0.4909 0.2619 1.8744 0.0609 -0.0224 1.0041
## mean_production_vocab 0.0024 0.0008 2.9351 0.0033 0.0008 0.0039
##
## intrcpt .
## mean_production_vocab **
##
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1##
## Multivariate Meta-Analysis Model (k = 29; method: REML)
##
## logLik Deviance AIC BIC AICc
## -27.0414 54.0828 62.0828 67.1151 63.9875
##
## Variance Components:
##
## estim sqrt nlvls fixed factor
## sigma^2 0.3819 0.6180 11 no short_cite
##
## Test for Residual Heterogeneity:
## QE(df = 26) = 114.7773, p-val < .0001
##
## Test of Moderators (coefficient(s) 2:3):
## QM(df = 2) = 9.4131, p-val = 0.0090
##
## Model Results:
##
## estimate se zval pval ci.lb ci.ub
## intrcpt 0.2672 0.4170 0.6406 0.5218 -0.5502 1.0846
## mean_age 0.0088 0.0134 0.6549 0.5125 -0.0175 0.0351
## mean_production_vocab 0.0022 0.0008 2.6903 0.0071 0.0006 0.0039
##
## intrcpt
## mean_age
## mean_production_vocab **
##
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1Typical only:
##
## Multivariate Meta-Analysis Model (k = 23; method: REML)
##
## logLik Deviance AIC BIC AICc
## -17.1061 34.2122 42.2122 46.1951 44.8789
##
## Variance Components:
##
## estim sqrt nlvls fixed factor
## sigma^2 0.5170 0.7190 8 no short_cite
##
## Test for Residual Heterogeneity:
## QE(df = 20) = 86.1817, p-val < .0001
##
## Test of Moderators (coefficient(s) 2:3):
## QM(df = 2) = 10.2650, p-val = 0.0059
##
## Model Results:
##
## estimate se zval pval ci.lb ci.ub
## intrcpt -0.7825 0.7066 -1.1076 0.2681 -2.1674 0.6023
## mean_age 0.0705 0.0329 2.1435 0.0321 0.0060 0.1350
## mean_production_vocab 0.0004 0.0012 0.3149 0.7528 -0.0019 0.0027
##
## intrcpt
## mean_age *
## mean_production_vocab
##
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1With set of conditions with complete production and age, age is not a reliable predictor, vocab is. In model with both, vocab is reliable. But for subset of typical participants, age but not vocab is predictive.
Misc MA summary stats (for typical only)
Some evidence for a relationship between effect size and sample size: smaller ES, bigger ns (residualizing out age and method)
Sample size and effect size
ME trial type
Controling for age, huge effect of ME_trial_type 
Stimulus type
No effect
Lab group
Not interpretable (small sample sizes)
Time
Effect gets bigger over time, residulaizing out method and age 
Mega developmental plot
