t-Tests and ANOVAs from Summary Statistics
Functions and Rationale
To assess some papers I’ve been looking at, I had to make functions for t-tests and ANOVAs from summary statistics. I found the t-test one on statexchange.
library(pacman); p_load(rpsychi, lavaan, pwr)
sum.t.test <- function(m1, m2, s1, s2, n1, n2, m0 = 0, equal.variance = F){
if(equal.variance == F){
se <- sqrt((s1^2/n1) + (s2^2/n2))
df <- ((s1^2/n1 + s2^2/n2)^2)/((s1^2/n1)^2/(n1-1) + (s2^2/n2)^2/(n2-1))
}else{
se <- sqrt((1/n1 + 1/n2) * ((n1-1)*s1^2 + (n2-1)*s2^2)/(n1+n2-2))
df <- n1+n2-2}
t <- (m1-m2-m0)/se
dat <- c(m1-m2, se, t, 2*pt(-abs(t),df))
names(dat) <- c("Mean Difference", "Standard Error", "t", "p-Value")
return(dat)}
sum.anova <- function(m1, m2, s1, s2, n1, n2){
library(rpsychi)
rest <- ind.oneway.second(c(m1, m2),
c(s1, s2),
c(n1, n2))
FP <- pf(rest$anova.table$F[1], rest$anova.table$df[1], rest$anova.table$df[3], lower.tail = F)
dat <- list(rest$anova.table, FP)
names(dat) <- c("ANOVA Table", "p-Value")
return(dat)}
CRITR <- function(n, alpha = .05) {
df <- n - 2; CRITT <- qt(alpha/2, df, lower.tail = F)
CRITR <- sqrt((CRITT^2)/((CRITT^2) + df ))
return(CRITR)}sum.t.test(0.29, 0.35, 0.09, 0.16, 20, 22) #Virtually identical results to https://www.usablestats.com/calcs/2samplet&summary=1## Mean Difference Standard Error t p-Value
## -0.06000000 0.03960601 -1.51492144 0.13912858sum.anova(0.29, 0.35, 0.09, 0.16, 20, 22) #Virtually identical results to https://statpages.info/anova1sm.html## $`ANOVA Table`
## SS df MS F
## Between (A) 0.038 1 0.038 2.182
## Within 0.692 40 0.017
## Total 0.729 41
##
## $`p-Value`
## [1] 0.1472736Empirical Examples
Otero et al. (2003) reported a bunch of ANOVAs of low- versus high-risk children in their Table 2. They said they only reported significant (p < 0.0001), but I cannot replicate their results, so I am wondering what went unspecified, if there were transformations used without providing the data, or if their ANOVA was specified very differently than it seemed. Their results for Table 2 are presented below.
cat("Four-Year-Olds", "\n")## Four-Year-Oldssum.anova(0.29, 0.35, 0.09, 0.16, 20, 22)$'p-Value' < 0.0001## [1] FALSEsum.anova(0.28, 0.38, 0.12, 0.16, 20, 22)$'p-Value' < 0.0001## [1] FALSEsum.anova(0.29, 0.39, 0.10, 0.17, 20, 22)$'p-Value' < 0.0001## [1] FALSEsum.anova(0.30, 0.38, 0.11, 0.14, 20, 22)$'p-Value' < 0.0001## [1] FALSEsum.anova(0.29, 0.43, 0.13, 0.21, 20, 22)$'p-Value' < 0.0001## [1] FALSEsum.anova(0.28, 0.36, 0.10, 0.16, 20, 22)$'p-Value' < 0.0001## [1] FALSEsum.anova(0.34, 0.41, 0.13, 0.19, 20, 22)$'p-Value' < 0.0001## [1] FALSEsum.anova(0.32, 0.38, 0.11, 0.18, 20, 22)$'p-Value' < 0.0001## [1] FALSEsum.anova(0.28, 0.33, 0.10, 0.12, 20, 22)$'p-Value' < 0.0001## [1] FALSEsum.anova(0.30, 0.37, 0.11, 0.18, 20, 22)$'p-Value' < 0.0001## [1] FALSEsum.anova(0.34, 0.39, 0.14, 0.17, 20, 22)$'p-Value' < 0.0001## [1] FALSEsum.anova(0.34, 0.38, 0.13, 0.14, 20, 22)$'p-Value' < 0.0001## [1] FALSEsum.anova(0.32, 0.40, 0.11, 0.16, 20, 22)$'p-Value' < 0.0001## [1] FALSEsum.anova(0.32, 0.39, 0.14, 0.18, 20, 22)$'p-Value' < 0.0001## [1] FALSEsum.anova(0.22, 0.16, 0.10, 0.09, 20, 22)$'p-Value' < 0.0001## [1] FALSEsum.anova(0.23, 0.14, 0.09, 0.08, 20, 22)$'p-Value' < 0.0001## [1] FALSEsum.anova(0.28, 0.18, 0.11, 0.09, 20, 22)$'p-Value' < 0.0001## [1] FALSEsum.anova(0.27, 0.19, 0.13, 0.01, 20, 22)$'p-Value' < 0.0001## [1] FALSEsum.anova(0.29, 0.20, 0.12, 0.11, 20, 22)$'p-Value' < 0.0001## [1] FALSEsum.anova(0.29, 0.23, 0.19, 0.10, 20, 22)$'p-Value' < 0.0001## [1] FALSEsum.anova(0.20, 0.16, 0.08, 0.01, 20, 22)$'p-Value' < 0.0001## [1] FALSEsum.anova(0.23, 0.12, 0.08, 0.08, 20, 22)$'p-Value' < 0.0001## [1] TRUEsum.anova(0.28, 0.14, 0.10, 0.07, 20, 22)$'p-Value' < 0.0001## [1] TRUEsum.anova(0.31, 0.21, 0.11, 0.10, 20, 22)$'p-Value' < 0.0001## [1] FALSEsum.anova(0.24, 0.17, 0.13, 0.10, 20, 22)$'p-Value' < 0.0001## [1] FALSEsum.anova(0.27, 0.19, 0.11, 0.05, 20, 22)$'p-Value' < 0.0001## [1] FALSEcat("Five-Year-Olds", "\n")## Five-Year-Oldssum.anova(0.24, 0.32, 0.10, 0.15, 20, 22)$'p-Value' < 0.0001## [1] FALSEsum.anova(0.23, 0.30, 0.09, 0.12, 20, 22)$'p-Value' < 0.0001## [1] FALSEsum.anova(0.27, 0.36, 0.12, 0.12, 20, 22)$'p-Value' < 0.0001## [1] FALSEsum.anova(0.29, 0.34, 0.13, 0.13, 20, 22)$'p-Value' < 0.0001## [1] FALSEsum.anova(0.28, 0.21, 0.13, 0.16, 20, 22)$'p-Value' < 0.0001## [1] FALSEsum.anova(0.29, 0.21, 0.12, 0.14, 20, 22)$'p-Value' < 0.0001## [1] FALSEsum.anova(0.26, 0.18, 0.12, 0.12, 20, 22)$'p-Value' < 0.0001## [1] FALSEsum.anova(0.25, 0.16, 0.09, 0.10, 20, 22)$'p-Value' < 0.0001## [1] FALSEsum.anova(0.32, 0.27, 0.10, 0.11, 20, 22)$'p-Value' < 0.0001## [1] FALSEsum.anova(0.32, 0.27, 0.13, 0.14, 20, 22)$'p-Value' < 0.0001## [1] FALSEsum.anova(0.35, 0.26, 0.14, 0.12, 20, 22)$'p-Value' < 0.0001## [1] FALSEsum.anova(0.36, 0.26, 0.12, 0.16, 20, 22)$'p-Value' < 0.0001## [1] FALSEcat("Six-Year-Olds", "\n")## Six-Year-Oldssum.anova(0.20, 0.25, 0.08, 0.11, 20, 22)$'p-Value' < 0.0001## [1] FALSEsum.anova(0.25, 0.30, 0.10, 0.10, 20, 22)$'p-Value' < 0.0001## [1] FALSEsum.anova(0.29, 0.32, 0.10, 0.14, 20, 22)$'p-Value' < 0.0001## [1] FALSEsum.anova(0.41, 0.32, 0.18, 0.16, 20, 22)$'p-Value' < 0.0001## [1] FALSEsum.anova(0.40, 0.31, 0.14, 0.15, 20, 22)$'p-Value' < 0.0001## [1] FALSEOnly two tests out of all of those actually met the criteria Otero et al. (2003) said they used. I’m not sure what was going on.
Random Additional Analyses
Unrelated to the above, Cantiani et al. (2019) fit a saturated mediation model and then provided the fits, which were obviously perfect. I do not know why they thought that would be meaningful. I also cannot tell, for the life of me, why they would have only run the mediation model with left-central EEG gamma power as opposed to other regions. With n = 62, the critical r is 0.25 with \(\alpha\) = 0.05 and the mean significant r for a true correlation of 0.10 is 0.30 at this sample size, so it was probably a power issue.
CanMat <- '
1
0.27 1
0.10 0.31 1
0.23 0.35 0.68 1'
Names = list("SES", "LCEEG", "VOC", "MLU")
Can.cor = getCov(CanMat, names = Names)
CanMod <- '
VOC ~ SES + LCEEG
MLU ~ SES + LCEEG
LCEEG ~ SES
VOC ~~ MLU'
CanFit <- sem(CanMod, sample.cov = Can.cor, sample.nobs = 62)
p_load(semPlot)
semPaths(CanFit, "model", "std", title = F, residuals = F, pastel = T, mar = c(3, 1, 3, 1))
summary(CanFit, stand = T, fit = T)## lavaan 0.6-9 ended normally after 21 iterations
##
## Estimator ML
## Optimization method NLMINB
## Number of model parameters 9
##
## Number of observations 62
##
## Model Test User Model:
##
## Test statistic 0.000
## Degrees of freedom 0
##
## Model Test Baseline Model:
##
## Test statistic 54.045
## Degrees of freedom 6
## P-value 0.000
##
## User Model versus Baseline Model:
##
## Comparative Fit Index (CFI) 1.000
## Tucker-Lewis Index (TLI) 1.000
##
## Loglikelihood and Information Criteria:
##
## Loglikelihood user model (H0) -235.388
## Loglikelihood unrestricted model (H1) -235.388
##
## Akaike (AIC) 488.776
## Bayesian (BIC) 507.920
## Sample-size adjusted Bayesian (BIC) 479.603
##
## Root Mean Square Error of Approximation:
##
## RMSEA 0.000
## 90 Percent confidence interval - lower 0.000
## 90 Percent confidence interval - upper 0.000
## P-value RMSEA <= 0.05 NA
##
## Standardized Root Mean Square Residual:
##
## SRMR 0.000
##
## Parameter Estimates:
##
## Standard errors Standard
## Information Expected
## Information saturated (h1) model Structured
##
## Regressions:
## Estimate Std.Err z-value P(>|z|) Std.lv Std.all
## VOC ~
## SES 0.018 0.125 0.140 0.888 0.018 0.018
## LCEEG 0.305 0.125 2.435 0.015 0.305 0.305
## MLU ~
## SES 0.146 0.122 1.196 0.232 0.146 0.146
## LCEEG 0.311 0.122 2.542 0.011 0.311 0.311
## LCEEG ~
## SES 0.270 0.122 2.208 0.027 0.270 0.270
##
## Covariances:
## Estimate Std.Err z-value P(>|z|) Std.lv Std.all
## .VOC ~~
## .MLU 0.560 0.131 4.275 0.000 0.560 0.646
##
## Variances:
## Estimate Std.Err z-value P(>|z|) Std.lv Std.all
## .VOC 0.889 0.160 5.568 0.000 0.889 0.904
## .MLU 0.844 0.152 5.568 0.000 0.844 0.858
## .LCEEG 0.912 0.164 5.568 0.000 0.912 0.927All marginal.
What about some power analyses for EEG-SES studies? Critical r is calculated with the CRITR() function, and the critical F value is calculated with qf(), where df1 is between (k - 1) and df2 is within (N - k). Power analyses are done for small effect sizes, since those are reasonable, whereas medium or large effects are simply not, unless trivial, like intraband correlations or related construct correlations. Power analyses here assume balance, but that is not usually the case, which means the power estimates are overestimates. When testing for ANOVA power, the n is per-group, so I rounded up, to give studies the benefit of the doubt.
cohen.ES(test = "r", size = c("small")); cohen.ES(test = "anov", size = c("small"))##
## Conventional effect size from Cohen (1982)
##
## test = r
## size = small
## effect.size = 0.1##
## Conventional effect size from Cohen (1982)
##
## test = anov
## size = small
## effect.size = 0.1cat("Otero (1994)", "\n"); pwr.anova.test(k = 2, n = 25, f = 0.1, sig.level = 0.05); qf(p = 0.05, df1 = 1, df2 = 48, lower.tail = F)## Otero (1994)##
## Balanced one-way analysis of variance power calculation
##
## k = 2
## n = 25
## f = 0.1
## sig.level = 0.05
## power = 0.1065814
##
## NOTE: n is number in each group## [1] 4.042652cat("Otero et al. (2003)", "\n"); pwr.anova.test(k = 2, n = 21, f = 0.1, sig.level = 0.05); qf(p = 0.05, df1 = 1, df2 = 40, lower.tail = F) #Otero (1997) used the same sample, but only with four-year-olds.## Otero et al. (2003)##
## Balanced one-way analysis of variance power calculation
##
## k = 2
## n = 21
## f = 0.1
## sig.level = 0.05
## power = 0.09696293
##
## NOTE: n is number in each group## [1] 4.084746cat("McLaughlin et al. (2010)", "\n"); pwr.anova.test(k = 4, n = 34, f = 0.1, sig.level = 0.05); qf(p = 0.05, df1 = 3, df2 = 131, lower.tail = F)## McLaughlin et al. (2010)##
## Balanced one-way analysis of variance power calculation
##
## k = 4
## n = 34
## f = 0.1
## sig.level = 0.05
## power = 0.1392751
##
## NOTE: n is number in each group## [1] 2.673748cat("Tomalski et al. (2013)", "\n"); pwr.anova.test(k = 2, n = 23, f = 0.1, sig.level = 0.05); qf(p = 0.05, df1 = 1, df2 = 43, lower.tail = F)## Tomalski et al. (2013)##
## Balanced one-way analysis of variance power calculation
##
## k = 2
## n = 23
## f = 0.1
## sig.level = 0.05
## power = 0.1017651
##
## NOTE: n is number in each group## [1] 4.067047cat("Pierce et al. (2019)", "\n"); pwr.r.test(n = 69, r = 0.1, sig.level = 0.05); CRITR(69)## Pierce et al. (2019)##
## approximate correlation power calculation (arctangh transformation)
##
## n = 69
## r = 0.1
## sig.level = 0.05
## power = 0.1296632
## alternative = two.sided## [1] 0.2369092cat("Cantiani et al. (2019)", "\n"); pwr.r.test(n = 81, r = 0.1, sig.level = 0.05); CRITR(81); pwr.anova.test(k = 2, n = 41, f = 0.1, sig.level = 0.05); qf(p = 0.05, df1 = 1, df2 = 78, lower.tail = F)## Cantiani et al. (2019)##
## approximate correlation power calculation (arctangh transformation)
##
## n = 81
## r = 0.1
## sig.level = 0.05
## power = 0.1444763
## alternative = two.sided## [1] 0.2185305##
## Balanced one-way analysis of variance power calculation
##
## k = 2
## n = 41
## f = 0.1
## sig.level = 0.05
## power = 0.1455326
##
## NOTE: n is number in each group## [1] 3.963472cat("Debnath et al. (2019)", "\n"); pwr.anova.test(k = 3, n = 46, f = 0.1, sig.level = 0.05); pwr.r.test(n = 45, r = 0.1, sig.level = 0.05); qf(p = 0.05, df1 = 2, df2 = 135, lower.tail = F); CRITR(45)## Debnath et al. (2019)##
## Balanced one-way analysis of variance power calculation
##
## k = 3
## n = 46
## f = 0.1
## sig.level = 0.05
## power = 0.1644236
##
## NOTE: n is number in each group##
## approximate correlation power calculation (arctangh transformation)
##
## n = 45
## r = 0.1
## sig.level = 0.05
## power = 0.1002777
## alternative = two.sided## [1] 3.063204## [1] 0.2939552cat("Troller-Renfree et al. (2020)", "\n"); pwr.r.test(n = 79, r = 0.1, sig.level = 0.05); CRITR(79)## Troller-Renfree et al. (2020)##
## approximate correlation power calculation (arctangh transformation)
##
## n = 79
## r = 0.1
## sig.level = 0.05
## power = 0.1420028
## alternative = two.sided## [1] 0.2212982cat("Brito et al. (2020)", "\n"); pwr.r.test(n = 60, r = 0.1, sig.level = 0.05); CRITR(60)## Brito et al. (2020)##
## approximate correlation power calculation (arctangh transformation)
##
## n = 60
## r = 0.1
## sig.level = 0.05
## power = 0.1186028
## alternative = two.sided## [1] 0.2542043cat("Jensen et al. (2021)", "\n"); pwr.r.test(n = 187, r = 0.1, sig.level = 0.05); CRITR(187)## Jensen et al. (2021)##
## approximate correlation power calculation (arctangh transformation)
##
## n = 187
## r = 0.1
## sig.level = 0.05
## power = 0.2760175
## alternative = two.sided## [1] 0.1435461What are the mean power levels and N’s for Troller-Renfree et al.’s cited EEG-cognition and EEG-SES studies? I am using the upper-bound sample sizes.
#Cognition
mean(c(0.12, 0.14, 0.06, 0.01, 0.12, 0.20, 0.12, 0.14, 0.16)) #Power## [1] 0.1188889mean(c(63, 81, 17, 13, 64, 129, 79, 90)) #N's## [1] 67#SES
mean(c(0.11, 0.10, 0.14, 0.10, 0.13, 0.15, 0.16, 0.14, 0.12, 0.28))## [1] 0.143mean(c(50, 42, 135, 45, 70, 81, 138, 79, 60, 187)) #N's## [1] 88.7cat("Cognition", "\n")## Cognitionpwr.r.test(n = 67, sig.level = 0.05, power = 0.8)##
## approximate correlation power calculation (arctangh transformation)
##
## n = 67
## r = 0.3345273
## sig.level = 0.05
## power = 0.8
## alternative = two.sidedpwr.r.test(n = 67, sig.level = 0.05, power = 0.5)##
## approximate correlation power calculation (arctangh transformation)
##
## n = 67
## r = 0.2387044
## sig.level = 0.05
## power = 0.5
## alternative = two.sidedpwr.r.test(n = 67, sig.level = 0.05, power = 0.3)##
## approximate correlation power calculation (arctangh transformation)
##
## n = 67
## r = 0.1763768
## sig.level = 0.05
## power = 0.3
## alternative = two.sidedpwr.r.test(n = 67, sig.level = 0.05, power = 0.12)##
## approximate correlation power calculation (arctangh transformation)
##
## n = 67
## r = 0.09538439
## sig.level = 0.05
## power = 0.12
## alternative = two.sidedcat("SES", "\n")## SESpwr.r.test(n = 89, sig.level = 0.05, power = 0.8)##
## approximate correlation power calculation (arctangh transformation)
##
## n = 89
## r = 0.2918731
## sig.level = 0.05
## power = 0.8
## alternative = two.sidedpwr.r.test(n = 89, sig.level = 0.05, power = 0.5)##
## approximate correlation power calculation (arctangh transformation)
##
## n = 89
## r = 0.2072472
## sig.level = 0.05
## power = 0.5
## alternative = two.sidedpwr.r.test(n = 89, sig.level = 0.05, power = 0.3)##
## approximate correlation power calculation (arctangh transformation)
##
## n = 89
## r = 0.1527874
## sig.level = 0.05
## power = 0.3
## alternative = two.sidedpwr.r.test(n = 89, sig.level = 0.05, power = 0.14)##
## approximate correlation power calculation (arctangh transformation)
##
## n = 89
## r = 0.09307399
## sig.level = 0.05
## power = 0.14
## alternative = two.sidedcat("Cognition", "\n")## CognitionMeanSigR(N = 67, P = 0.05, R = 0.33, Reps = 1000, seed = 1)## [1] 0.3635437MeanSigR(N = 67, P = 0.05, R = 0.24, Reps = 1000, seed = 1)## [1] 0.3299675MeanSigR(N = 67, P = 0.05, R = 0.18, Reps = 1000, seed = 1)## [1] 0.3130114MeanSigR(N = 67, P = 0.05, R = 0.09, Reps = 1000, seed = 1)## [1] 0.301092cat("SES", "\n")## SESMeanSigR(N = 89, P = 0.05, R = 0.29, Reps = 1000, seed = 1)## [1] 0.324105MeanSigR(N = 89, P = 0.05, R = 0.21, Reps = 1000, seed = 1)## [1] 0.2862015MeanSigR(N = 89, P = 0.05, R = 0.15, Reps = 1000, seed = 1)## [1] 0.2684268MeanSigR(N = 89, P = 0.05, R = 0.09, Reps = 1000, seed = 1)## [1] 0.2599198Is there any possibility that Troller-Renfree et al.’s results could be necsesarily transitive? We need to convert their effect sizes to the point-biserial to check that possibility. Here are two functions to do that exactly (not like Ruscio, 2008, which deals with near approximations).
rpb2d <- function(r, n1, n2){
m = n1 + n2 - 2
h = m/n1 + m/n2
d <- (r*sqrt(h))/sqrt(1-r^2)
return(d)}
d2rpb <- function(d, n1, n2){
m = n1 + n2 - 2
h = m/n1 + m/n2
rpb <- (d*sqrt(d^2+h))/(d^2+h)
return(rpb)}To confirm the functions work:
d2rpb(0.3, 500, 500)## [1] 0.1484857rpb2d(d2rpb(0.3, 500, 500), 500, 500)## [1] 0.3Using Troller-Renfree et al.’s largest effect size, lets change it to \(r_{pb}\).
d2rpb(0.26, 184, 251)## [1] 0.1276911For Troller-Renfree et al., their largest effect was consistent with a correlation of 0.13. To have transitivity, you need \(r^2_{xy} + r^2_{yz}\) to be above 1. Since we have the putatively causal effect of cash transfers on EEG power, we are now looking for the putatively causal effect of EEG power on an outcome like cognitive ability. Since \(0.13^2 = 0.0169\), to have necessary transitivity, the EEG-outcome relationship has to be r =
sqrt(1-0.0169)## [1] 0.991514That is decidedly not the case.
Using Gelman & Carlin’s (2014) code, how exaggerated is Troller-Renfree et al.’s effect? I will use their two significant effects.
retrodesign <- function(A, s, alpha=.05, df=Inf, n.sims=10000){
z <- qt(1-alpha/2, df)
p.hi <- 1 - pt(z-A/s, df)
p.lo <- pt(-z-A/s, df)
power <- p.hi + p.lo
typeS <- p.lo/power
estimate <- A + s*rt(n.sims,df)
significant <- abs(estimate) > s*z
exaggeration <- mean(abs(estimate)[significant])/A
return(list(power=power, typeS=typeS, exaggeration=exaggeration))}First, the standard errors.
SE1 <- 0.26/qnorm(1 - 0.02)
SE2 <- 0.23/qnorm(1 - 0.04) Then, the power, exaggeration ratios, etc., assuming a ‘true’ effect size of 0.1, which is more consistent with large studies, if still probably too large.
set.seed(1)
cat("Beta Power", "\n")## Beta Powerretrodesign(0.1, SE1)## $power
## [1] 0.1239693
##
## $typeS
## [1] 0.02404603
##
## $exaggeration
## [1] 3.087927cat("Gamma Power", "\n")## Gamma Powerretrodesign(0.1, SE2)## $power
## [1] 0.1185566
##
## $typeS
## [1] 0.02743788
##
## $exaggeration
## [1] 3.225009And then, how large of a sample would we need to get an effect without exaggeration? This means, to a first approximation, a power of 1, which for simplicity, I will set to 0.99. Through simple algebra, we can derive the SEs, SDs, and other quantities needed for this.
set.seed(1)
retrodesign(0.1, 0.023)## $power
## [1] 0.9915267
##
## $typeS
## [1] 1.427333e-10
##
## $exaggeration
## [1] 1.004845SD1 <- (sqrt(435)*SE1*435)/435 #SD1
SD2 <- (sqrt(435)*SE2*435)/435 #SD2To get an SE of 0.023, we would need sample sizes of
SD1^2/0.023^2## [1] 13179.1SD2^2/0.023^2## [1] 14192.95But what if we just wanted 80% power?
retrodesign(0.1, 0.035)## $power
## [1] 0.815189
##
## $typeS
## [1] 8.9338e-07
##
## $exaggeration
## [1] 1.110359ESS1 <- SD1^2/0.035^2; ESS1## [1] 5691.217ESS2 <- SD2^2/0.035^2; ESS2## [1] 6129.038The ratio of the sample sizes needed to get 80% power to those achieved were
ESS1/435## [1] 13.08326ESS2/435## [1] 14.08974How large would our samples need to be with smaller SDs?
SD1TF <- sqrt(SD1^2 * 0.75) #Three-fourths
SD1H <- sqrt(SD1^2 * 0.50) #Half
SD1Q <- sqrt(SD1^2 * 0.25) #Quarter
SD2TF <- sqrt(SD2^2 * 0.75)
SD2H <- sqrt(SD2^2 * 0.50)
SD2Q <- sqrt(SD2^2 * 0.25)
SD1TF^2/0.023^2; SD1TF^2/0.035^2## [1] 9884.321## [1] 4268.413SD1H^2/0.023^2; SD1H^2/0.035^2## [1] 6589.548## [1] 2845.609SD1Q^2/0.023^2; SD1Q^2/0.035^2## [1] 3294.774## [1] 1422.804SD2TF^2/0.023^2; SD2TF^2/0.035^2## [1] 10644.71## [1] 4596.778SD2H^2/0.023^2; SD2H^2/0.035^2## [1] 7096.476## [1] 3064.519SD2Q^2/0.023^2; SD2Q^2/0.035^2## [1] 3548.238## [1] 1532.259The issues with Troller-Renfree et al.’s paper were probably not sign-related, but instead, magnitude-related. The lack of a sign issue is worrying, since the largest validity study they cited suggested SES was negatively related to beta and gamma EEG power, whereas they found positive effects. The issue of magnitude may seem impossible to solve, given that anything short of perfect power yields some exaggeration. With very large samples, the degree of it is so small that it is likely inconsequential; moreover, our ability to precisely discover the ‘true’ effect size is improved as N grows, so what may be intractable with a true ES of 0.10 may not be if it turns out to be 0.15 because scaling is rapid.
References
Otero, G. A., F. B. Pliego-Rivero, T. Fernandez, and J. Ricardo. 2003. “EEG Development in Children with Sociocultural Disadvantages: A Follow-up Study.” Clinical Neurophysiology: Official Journal of the International Federation of Clinical Neurophysiology 114(10):1918–25. doi: 10.1016/s1388-2457(03)00173-1.
Cantiani, Chiara, Caterina Piazza, Giulia Mornati, Massimo Molteni, and Valentina Riva. 2019. “Oscillatory Gamma Activity Mediates the Pathway from Socioeconomic Status to Language Acquisition in Infancy.” Infant Behavior and Development 57:101384. doi: 10.1016/j.infbeh.2019.101384.
Tomalski, Przemyslaw, Derek G. Moore, Helena Ribeiro, Emma L. Axelsson, Elizabeth Murphy, Annette Karmiloff-Smith, Mark H. Johnson, and Elena Kushnerenko. 2013. “Socioeconomic Status and Functional Brain Development - Associations in Early Infancy.” Developmental Science 16(5):676–87. doi: 10.1111/desc.12079.
McLaughlin, Katie A., Nathan A. Fox, Charles H. Zeanah, Margaret A. Sheridan, Peter Marshall, and Charles A. Nelson. 2010. “Delayed Maturation in Brain Electrical Activity Partially Explains the Association Between Early Environmental Deprivation and Symptoms of Attention-Deficit/Hyperactivity Disorder.” Biological Psychiatry 68(4):329–36. doi: 10.1016/j.biopsych.2010.04.005.
Ruscio, John. 2008. “A Probability-Based Measure of Effect Size: Robustness to Base Rates and Other Factors.” Psychological Methods 13(1):19–30. doi: 10.1037/1082-989X.13.1.19.
Gelman, Andrew, and John Carlin. 2014. “Beyond Power Calculations: Assessing Type S (Sign) and Type M (Magnitude) Errors.” Perspectives on Psychological Science 9(6):641–51. doi: 10.1177/1745691614551642.