0.0.0.0.1 Missing data
## 
## FALSE  TRUE 
##   341     2
## 
## FALSE  TRUE 
##   197   149

1 Mood

1.1 Meta-analysis

k = number of studies; sqrt in “Variance components” = tau, the standard deviation of true effects; estimate in “Model results” = naive MA estimate

## 
## Number of outcomes:   23 
## Number of clusters:   17 
## Outcomes per cluster: 1-2 (mean: 1.35, median: 1)
## 
## Model Results:
## 
## estimate      se    tval    pval   ci.lb   ci.ub     
##   0.1585  0.0299  5.3081  <.0001  0.0952  0.2218  ***
## 
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

1.1.0.1 95% prediction interval

As an approximation, in 95% of cases the true effect in a new published study can be expected to fall between 0.095 and 0.222. Note that these are non-adjusted estimates. An unbiased newly conducted study will likely fall in an interval centered around PET-PEESE estimate with a similar CI width of 0.127.

1.1.1 Heterogeneity

1.1.1.1 Total heterogeneity - tau

The sum of the two variance components is equal to 0 . That can be interpreted as the total amount of heterogeneity in the true effects.

1.1.1.2 \(I^2\)

\(I^2\) represents the ratio of true heterogeneity to total variance across the observed effect estimates.

Here, total relative heterogeneity was

0 %. Separate estimates of between- and within-cluster heterogeneity were 0, 0 %, respectively.

Jackson’s approach to \(I^2\) yields a relative heterogeneity estimate of 0 %.

1.1.1.3 Proportion of significant results

## [1] 0.04

1.1.1.4 Intra-class correlation of underlying true effects

## [1] 0.426

1.1.2 Contour enhanced funnel plot

Correlation between the ES and precision

## [1] 0.3359684

1.1.3 Forest plot

1.2 Small-study effects correction

1.2.1 3-parameter selection model

Bias-corrected estimate, note especially the CI (conf.low, conf.high).

method term variable value
1 3PSM b0 estimate 0.207
4 3PSM b0 p.value 0.120
5 3PSM b0 conf.low -0.054
6 3PSM b0 conf.high 0.467

1.2.2 PET-PEESE

Estimated effect size of an infinitely precise study. Using 3PSM as the conditional estimator instead of PET. If the PET-PEESE estimate is negative, the effect can be regarded 0. pval = p-value testing H0 that the effect is zero. cil.lb and ci.ub are upper and lower bound of the CI.
PET estimate -0.097
se 0.093
zval -1.041
pval 0.314
ci.lb -0.295
ci.ub 0.101

1.2.2.1 PET-PEESE plot

y-axis intercept represents the estimated bias-corrected ES.

1.2.2.2 p-uniform

Additional bias-corrected estimate. Because it’s far less precise than PET-PEESE, when the n of studies is small, look just at the CI width and p-value.

## 
## Method: P
## 
## Effect size estimation p-uniform
## 
##        est     ci.lb     ci.ub       L.0      pval      ksig
##      0.812   -2.1562    2.2095   -0.9105    0.1813         1
## 
## ===
## 
## Publication bias test p-uniform
## 
##       L.pb      pval
##    -0.7731    0.7803
## 
## ===
## 
## Fixed-effect meta-analysis
## 
##     est.fe     se.fe   zval.fe pval.fe  ci.lb.fe  ci.ub.fe     Qstat
##     0.1585    0.0449    3.5315   <.001    0.0705    0.2465   12.1921
##      Qpval
##     0.9533

2 Overall effect (excluding mood)

2.1 Evidential value

2.1.0.1 Permutation p-curve

P-Curve analysis combines the half and full p-curve to make inferences about evidential value. In particular, if the half p-curve test is right-skewed (halfp) with p<.05 or both the half and full test (fullp) are right-skewed with p < .1, then p-curve analysis indicates the presence of evidential value. Similarly, p-curve analysis indicates that evidential value is inadequate or absent if the 33% power test is p < .05 for the full p-curve (fullp33) or both the half p-curve (halfp33) and binomial 33% power test (binomp33) are p < .1.

ksig = average number of effects associated with p < .05; khalf = average number of effects associated with p < .025;…z = average z-values; power.est = average estimated statistical power of the studies (with lower bound and upper bound)

NOTE: For now, all the p-curve permutations are based just on 100 sets of draws, because they are quite computationally intensive.

vars n mean sd se
ksig 1 1000 137.125 2.587 0.082
khalf 2 1000 83.566 2.770 0.088
fullz 3 1000 -9.684 0.636 0.020
fullp 4 1000 0.000 0.000 0.000
fullz33 5 1000 2.130 0.567 0.018
fullp33 6 1000 0.968 0.042 0.001
halfz 7 1000 -11.652 0.787 0.025
halfp 8 1000 0.000 0.000 0.000
halfz33 9 1000 14.352 0.535 0.017
halfp33 10 1000 1.000 0.000 0.000
binomp 11 1000 0.010 0.012 0.000
binomp33 12 1000 0.013 0.014 0.000
power.ci.lb 13 1000 0.366 0.037 0.001
power.est 14 1000 0.475 0.037 0.001
power.ci.up 15 1000 0.579 0.034 0.001

2.2 Meta-analysis

k = number of studies; sqrt in “Variance components” = tau, the standard deviation of true effects; estimate in “Model results” = naive MA estimate

## 
## Number of outcomes:   174 
## Number of clusters:   119 
## Outcomes per cluster: 1-5 (mean: 1.46, median: 1)
## 
## Model Results:
## 
## estimate      se     tval    pval   ci.lb   ci.ub     
##   0.4454  0.0318  14.0119  <.0001  0.3824  0.5083  ***
## 
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

2.2.0.1 95% prediction interval

As an approximation, in 95% of cases the true effect in a new published study can be expected to fall between -0.155 and 1.046. Note that these are non-adjusted estimates. An unbiased newly conducted study will likely fall in an interval centered around PET-PEESE estimate with a similar CI width of 1.201.

2.2.1 Heterogeneity

2.2.1.1 Total heterogeneity - tau

The sum of the two variance components is equal to 0.302 . That can be interpreted as the total amount of heterogeneity in the true effects.

2.2.1.2 \(I^2\)

\(I^2\) represents the ratio of true heterogeneity to total variance across the observed effect estimates.

Here, total relative heterogeneity was 87.72 %. Separate estimates of between- and within-cluster heterogeneity were 51.53, 36.19 %, respectively.

Jackson’s approach to \(I^2\) yields a relative heterogeneity estimate of 99.94 %.

2.2.1.3 Proportion of significant results

## [1] 0.69

2.2.1.4 Intra-class correlation of underlying true effects

## [1] 0.59

2.2.2 Contour enhanced funnel plot

Correlation between the ES and precision

## [1] 0.583815

2.2.3 Forest plot

2.3 Small-study effects correction

2.3.1 3-parameter selection model

Bias-corrected estimate, note especially the CI (conf.low, conf.high).

method term variable value
1 3PSM b0 estimate 0.217
4 3PSM b0 p.value 0.000
5 3PSM b0 conf.low 0.133
6 3PSM b0 conf.high 0.301

2.3.2 PET-PEESE

Estimated effect size of an infinitely precise study. Using 3PSM as the conditional estimator instead of PET. If the PET-PEESE estimate is negative, the effect can be regarded 0. pval = p-value testing H0 that the effect is zero. cil.lb and ci.ub are upper and lower bound of the CI.
PEESE estimate 0.129
se 0.036
zval 3.597
pval 0.000
ci.lb 0.059
ci.ub 0.200

2.3.2.1 PET-PEESE plot

y-axis intercept represents the estimated bias-corrected ES.

2.3.2.2 p-uniform

Additional bias-corrected estimate. Because it’s far less precise than PET-PEESE, when the n of studies is small, look just at the CI width and p-value. Leaving out study id# 211 because p-uniform won’t converge due to huge variance.

## 
## Method: P
## 
## Effect size estimation p-uniform
## 
##        est     ci.lb     ci.ub       L.0    pval      ksig
##     0.2702    0.1759    0.3598    -4.435   <.001       113
## 
## ===
## 
## Publication bias test p-uniform
## 
##       L.pb      pval
##    -0.1013    0.5403
## 
## ===
## 
## Fixed-effect meta-analysis
## 
##     est.fe     se.fe   zval.fe pval.fe  ci.lb.fe  ci.ub.fe     Qstat
##      0.266    0.0121   21.9559   <.001    0.2423    0.2898  712.2357
##    Qpval
##    <.001

2.3.2.3 Power based on PEESE and 3PSM parameter estimates

## [1] "Power to detect PEESE estimate = 11.59%"
## [1] "Power to detect 3PSM estimate = 24.13%"

How many studies had more than 50% power to detect the overall PEESE estimate?

## 
## FALSE  TRUE 
##   163    11

3 Effect type: compensatory vs priming

3.1 Evidential value for effect type = compensatory

3.1.0.1 Permutation p-curve

P-Curve analysis combines the half and full p-curve to make inferences about evidential value. In particular, if the half p-curve test is right-skewed (halfp) with p<.05 or both the half and full test (fullp) are right-skewed with p < .1, then p-curve analysis indicates the presence of evidential value. Similarly, p-curve analysis indicates that evidential value is inadequate or absent if the 33% power test is p < .05 for the full p-curve (fullp33) or both the half p-curve (halfp33) and binomial 33% power test (binomp33) are p < .1.

ksig = average number of effects associated with p < .05; khalf = average number of effects associated with p < .025;…z = average z-values; power.est = average estimated statistical power of the studies (with lower bound and upper bound)

vars n mean sd se
ksig 1 1000 37.233 0.853 0.027
khalf 2 1000 22.920 1.032 0.033
fullz 3 1000 -3.900 0.196 0.006
fullp 4 1000 0.000 0.000 0.000
fullz33 5 1000 0.002 0.179 0.006
fullp33 6 1000 0.501 0.070 0.002
halfz 7 1000 -4.339 0.274 0.009
halfp 8 1000 0.000 0.000 0.000
halfz33 9 1000 6.659 0.060 0.002
halfp33 10 1000 1.000 0.000 0.000
binomp 11 1000 0.115 0.054 0.002
binomp33 12 1000 0.150 0.072 0.002
power.ci.lb 13 1000 0.150 0.016 0.001
power.est 14 1000 0.335 0.023 0.001
power.ci.up 15 1000 0.550 0.022 0.001

3.2 Evidential value for effect type = priming

3.2.0.1 Permutation p-curve

vars n mean sd se
ksig 1 1000 96.684 2.217 0.070
khalf 2 1000 58.774 2.336 0.074
fullz 3 1000 -8.671 0.772 0.024
fullp 4 1000 0.000 0.000 0.000
fullz33 5 1000 2.333 0.686 0.022
fullp33 6 1000 0.973 0.044 0.001
halfz 7 1000 -10.689 0.934 0.030
halfp 8 1000 0.000 0.000 0.000
halfz33 9 1000 12.544 0.660 0.021
halfp33 10 1000 1.000 0.000 0.000
binomp 11 1000 0.030 0.027 0.001
binomp33 12 1000 0.031 0.029 0.001
power.ci.lb 13 1000 0.388 0.054 0.002
power.est 14 1000 0.515 0.052 0.002
power.ci.up 15 1000 0.633 0.046 0.001

3.3 Meta-analysis

3.3.0.1 Compensatory vs. priming effect type

k = number of studies; sqrt in “Variance components” = tau, the standard deviation of true effects; estimate in “Model results” = naive MA estimate

meta k estimate stderror tau
Compensatory 41 0.319 0.049 0.207
Priming 127 0.484 0.040 0.322

3.3.0.2 Wald test p-value

Testing the difference in the uncorrected MA estimates between effect types

## [1] 0.009080724

3.3.0.3 95% prediction interval for compensatory effect

As an approximation, in 95% of cases the true effect in a new published study can be expected to fall between -0.116 and 0.753. Note that these are non-adjusted estimates. An unbiased newly conducted study will likely fall in an interval centered around PET-PEESE estimate with a similar CI width of 0.869.

3.3.0.4 95% prediction interval for priming effect

As an approximation, in 95% of cases the true effect in a new published study can be expected to fall between -0.16 and 1.129. An unbiased newly conducted study will likely fall in an interval centered around PET-PEESE estimate with a similar CI width of 1.289.

3.3.0.5 Relative heterogeneity

3.3.0.6 \(I^2\) for Compensatory

\(I^2\) represents the ratio of true heterogeneity to total variance across the observed effect estimates.

Here, total relative heterogeneity was

73.06 %. Separate estimates of between- and within-cluster heterogeneity were 73.06, 0 %, respectively.

Jackson’s approach to \(I^2\) yields a relative heterogeneity estimate of 86.23 %.

3.3.0.7 \(I^2\) for Priming

Here, total relative heterogeneity was

85.92 %. Separate estimates of between- and within-cluster heterogeneity were 46.48, 39.44 %, respectively.

Jackson’s approach to \(I^2\) yields a relative heterogeneity estimate of 99.96 %.

3.3.0.8 Intra-class correlation of underlying true effects

For compensatory

## [1] 1

For priming

## [1] 0.54

3.3.1 Contour enhanced funnel plot

Correlation between the ES and precision

For compensatory

## [1] 0.704878

For priming

## [1] 0.5685539

3.3.2 Forest plot

3.4 Small-study effects correction for compensatory effect

3.4.1 3-parameter selection model for compensatory effect

Bias-corrected estimate, note especially the CI (conf.low, conf.high).

method term variable value
1 3PSM b0 estimate 0.144
4 3PSM b0 p.value 0.007
5 3PSM b0 conf.low 0.039
6 3PSM b0 conf.high 0.249

3.4.2 PET-PEESE for compensatory effect

Estimated effect size of an infinitely precise study. Using 3PSM as the conditional estimator instead of PET. If the PET-PEESE estimate is negative, the effect can be regarded 0. pval = p-value testing H0 that the effect is zero. cil.lb and ci.ub are upper and lower bound of the CI.
PEESE estimate 0.063
se 0.054
zval 1.180
pval 0.238
ci.lb -0.042
ci.ub 0.169

PET-PEESE plot for compensatory effect

y-axis intercept represents the estimated bias-corrected ES.

3.4.2.1 p-uniform for compensatory effect

Additional bias-corrected estimate. Because it’s far less precise than PET-PEESE, when the n of studies is small, look just at the CI width and p-value.

## 
## Method: P
## 
## Effect size estimation p-uniform
## 
##        est     ci.lb     ci.ub       L.0      pval      ksig
##     0.0437   -0.4603    0.2599   -0.2227    0.4119        24
## 
## ===
## 
## Publication bias test p-uniform
## 
##       L.pb      pval
##      1.275    0.1012
## 
## ===
## 
## Fixed-effect meta-analysis
## 
##     est.fe     se.fe   zval.fe pval.fe  ci.lb.fe  ci.ub.fe     Qstat
##     0.2228    0.0182   12.2244   <.001    0.1871    0.2586  117.2793
##    Qpval
##    <.001

3.5 Small-study effects correction for priming effect

3.5.1 3-parameter selection model for priming effect

Bias-corrected estimate, note especially the CI (conf.low, conf.high).

method term variable value
1 3PSM b0 estimate 0.234
4 3PSM b0 p.value 0.000
5 3PSM b0 conf.low 0.125
6 3PSM b0 conf.high 0.343

3.5.2 PET-PEESE for priming effect

Estimated effect size of an infinitely precise study. Using 3PSM as the conditional estimator instead of PET. If the PET-PEESE estimate is negative, the effect can be regarded 0. pval = p-value testing H0 that the effect is zero. cil.lb and ci.ub are upper and lower bound of the CI.
PEESE estimate 0.157
se 0.047
zval 3.340
pval 0.001
ci.lb 0.065
ci.ub 0.250

PET-PEESE plot for priming effect

y-axis intercept represents the estimated bias-corrected ES.

3.5.2.1 p-uniform for priming effect

Additional bias-corrected estimate. Because it’s far less precise than PET-PEESE, when the n of studies is small, look just at the CI width and p-value. Leaving out study id# 211 because p-uniform won’t converge due to huge variance.

## 
## Method: P
## 
## Effect size estimation p-uniform
## 
##        est     ci.lb     ci.ub       L.0    pval      ksig
##     0.3225    0.2019    0.4318   -4.5552   <.001        85
## 
## ===
## 
## Publication bias test p-uniform
## 
##       L.pb      pval
##    -0.4735    0.6821
## 
## ===
## 
## Fixed-effect meta-analysis
## 
##     est.fe     se.fe   zval.fe pval.fe  ci.lb.fe  ci.ub.fe     Qstat
##     0.2948    0.0164   17.9748   <.001    0.2626    0.3269  572.7005
##    Qpval
##    <.001

4 Physical manipulation

4.1 Evidential value for non-physical manipulation

4.1.0.1 Permutation p-curve

P-Curve analysis combines the half and full p-curve to make inferences about evidential value. In particular, if the half p-curve test is right-skewed (halfp) with p<.05 or both the half and full test (fullp) are right-skewed with p < .1, then p-curve analysis indicates the presence of evidential value. Similarly, p-curve analysis indicates that evidential value is inadequate or absent if the 33% power test is p < .05 for the full p-curve (fullp33) or both the half p-curve (halfp33) and binomial 33% power test (binomp33) are p < .1.

ksig = average number of effects associated with p < .05; khalf = average number of effects associated with p < .025;…z = average z-values; power.est = average estimated statistical power of the studies (with lower bound and upper bound)

vars n mean sd se
ksig 1 1000 70.313 1.749 0.055
khalf 2 1000 43.974 1.815 0.057
fullz 3 1000 -8.755 0.684 0.022
fullp 4 1000 0.000 0.000 0.000
fullz33 5 1000 3.078 0.602 0.019
fullp33 6 1000 0.996 0.008 0.000
halfz 7 1000 -10.538 0.941 0.030
halfp 8 1000 0.000 0.000 0.000
halfz33 9 1000 11.983 0.651 0.021
halfp33 10 1000 1.000 0.000 0.000
binomp 11 1000 0.028 0.019 0.001
binomp33 12 1000 0.087 0.043 0.001
power.ci.lb 13 1000 0.467 0.056 0.002
power.est 14 1000 0.611 0.050 0.002
power.ci.up 15 1000 0.730 0.040 0.001

4.2 Evidential value for physical manipulation

4.2.0.1 Permutation p-curve

P-Curve analysis combines the half and full p-curve to make inferences about evidential value. In particular, if the half p-curve test is right-skewed (halfp) with p<.05 or both the half and full test (fullp) are right-skewed with p < .1, then p-curve analysis indicates the presence of evidential value. Similarly, p-curve analysis indicates that evidential value is inadequate or absent if the 33% power test is p < .05 for the full p-curve (fullp33) or both the half p-curve (halfp33) and binomial 33% power test (binomp33) are p < .1.

ksig = average number of effects associated with p < .05; khalf = average number of effects associated with p < .025;…z = average z-values; power.est = average estimated statistical power of the studies (with lower bound and upper bound)

vars n mean sd se
ksig 1 1000 68.257 1.905 0.060
khalf 2 1000 40.183 2.161 0.068
fullz 3 1000 -4.763 0.559 0.018
fullp 4 1000 0.000 0.000 0.000
fullz33 5 1000 -0.275 0.503 0.016
fullp33 6 1000 0.403 0.177 0.006
halfz 7 1000 -5.639 0.597 0.019
halfp 8 1000 0.000 0.000 0.000
halfz33 9 1000 8.152 0.373 0.012
halfp33 10 1000 1.000 0.000 0.000
binomp 11 1000 0.112 0.086 0.003
binomp33 12 1000 0.038 0.043 0.001
power.ci.lb 13 1000 0.176 0.036 0.001
power.est 14 1000 0.309 0.045 0.001
power.ci.up 15 1000 0.462 0.046 0.001

4.3 Meta-analysis

4.3.0.1 Physical vs. non-physical manipulation

k = number of studies; sqrt in “Variance components” = tau, the standard deviation of true effects; estimate in “Model results” = naive MA estimate

meta k estimate stderror tau
(Physical manipulation = N) 90 0.381 0.038 0.268
(Physical manipulation = Y) 81 0.509 0.057 0.347

4.3.0.2 95% prediction interval for non-physical manipulation

As an approximation, in 95% of cases the true effect in a new published study can be expected to fall between -0.159 and 0.922. Note that these are non-adjusted estimates. An unbiased newly conducted study will likely fall in an interval centered around PET-PEESE estimate with a similar CI width of 1.081.

4.3.0.3 95% prediction interval for physical manipulation

As an approximation, in 95% of cases the true effect in a new published study can be expected to fall between -0.196 and 1.215. An unbiased newly conducted study will likely fall in an interval centered around PET-PEESE estimate with a similar CI width of 1.411.

4.3.0.4 Wald test p-value

Testing the difference in the uncorrected MA estimates between physical and non-physical manipulation

## [1] 0.06176207

4.3.0.5 Relative heterogeneity

4.3.0.6 \(I^2\) for non-physical manipulation

\(I^2\) represents the ratio of true heterogeneity to total variance across the observed effect estimates.

Here, total relative heterogeneity was

89.23 %. Separate estimates of between- and within-cluster heterogeneity were 48.51, 40.72 %, respectively.

Jackson’s approach to \(I^2\) yields a relative heterogeneity estimate of 99.96 %.

4.3.0.7 \(I^2\) for physical manipulation

\(I^2\) represents the ratio of true heterogeneity to total variance across the observed effect estimates.

Here, total relative heterogeneity was

70.39 %. Separate estimates of between- and within-cluster heterogeneity were 70.39, 0 %, respectively.

Jackson’s approach to \(I^2\) yields a relative heterogeneity estimate of 81.16 %.

4.3.0.8 Intra-class correlation of underlying true effects

For physical manipulation == N

## [1] 0.54

For physical manipulation == Y

## [1] 1

4.3.1 Contour enhanced funnel plot

Correlation between the ES and precision

For physical manipulation == N

## [1] 0.6089888

For physical manipulation == Y

## [1] 0.5679012

4.3.2 Forest plot

4.4 Small-study effects correction for non-physical manipulation

4.4.1 3-parameter selection model for non-physical manipulation

Bias-corrected estimate, note especially the CI (conf.low, conf.high).

method term variable value
1 3PSM b0 estimate 0.221
4 3PSM b0 p.value 0.000
5 3PSM b0 conf.low 0.118
6 3PSM b0 conf.high 0.324

4.4.2 PET-PEESE for non-physical manipulation

Estimated effect size of an infinitely precise study. Using 3PSM as the conditional estimator instead of PET. If the PET-PEESE estimate is negative, the effect can be regarded 0. pval = p-value testing H0 that the effect is zero. cil.lb and ci.ub are upper and lower bound of the CI.
PEESE estimate 0.153
se 0.039
zval 3.891
pval 0.000
ci.lb 0.076
ci.ub 0.230

PET-PEESE plot for non-physical manipulation

y-axis intercept represents the estimated bias-corrected ES.

4.4.2.1 p-uniform for non-physical manipulation

Additional bias-corrected estimate. Because it’s far less precise than PET-PEESE, when the n of studies is small, look just at the CI width and p-value. Leaving out study id# 211 because p-uniform won’t converge due to huge variance.

## 
## Method: P
## 
## Effect size estimation p-uniform
## 
##        est     ci.lb     ci.ub       L.0      pval      ksig
##     0.2248    0.0761     0.317   -2.7295    0.0032        55
## 
## ===
## 
## Publication bias test p-uniform
## 
##       L.pb      pval
##     0.3583    0.3601
## 
## ===
## 
## Fixed-effect meta-analysis
## 
##     est.fe     se.fe   zval.fe pval.fe  ci.lb.fe  ci.ub.fe     Qstat
##     0.2424    0.0139   17.3917   <.001     0.215    0.2697  331.1839
##    Qpval
##    <.001

4.5 Small-study effects correction for physical manipulation

4.5.1 3-parameter selection model for physical manipulation

Bias-corrected estimate, note especially the CI (conf.low, conf.high).

method term variable value
1 3PSM b0 estimate 0.205
4 3PSM b0 p.value 0.005
5 3PSM b0 conf.low 0.063
6 3PSM b0 conf.high 0.347

4.5.2 PET-PEESE for physical manipulation

Estimated effect size of an infinitely precise study. Using 3PSM as the conditional estimator instead of PET. If the PET-PEESE estimate is negative, the effect can be regarded 0. pval = p-value testing H0 that the effect is zero. cil.lb and ci.ub are upper and lower bound of the CI.
PEESE estimate 0.026
se 0.079
zval 0.321
pval 0.748
ci.lb -0.130
ci.ub 0.181

PET-PEESE plot for physical manipulation

y-axis intercept represents the estimated bias-corrected ES.

4.5.2.1 p-uniform for physical manipulation

Additional bias-corrected estimate. Because it’s far less precise than PET-PEESE, when the n of studies is small, look just at the CI width and p-value.

## 
## Method: P
## 
## Effect size estimation p-uniform
## 
##        est     ci.lb     ci.ub       L.0    pval      ksig
##     0.3331    0.1587    0.4739   -3.3377   <.001        57
## 
## ===
## 
## Publication bias test p-uniform
## 
##       L.pb      pval
##     0.0529    0.4789
## 
## ===
## 
## Fixed-effect meta-analysis
## 
##     est.fe     se.fe   zval.fe pval.fe  ci.lb.fe  ci.ub.fe     Qstat
##     0.3372    0.0249     13.56   <.001    0.2885     0.386  363.4729
##    Qpval
##    <.001

5 Method effects

5.1 Evidential value

5.1.1 Permutation p-curve for Visual.Verbal.Temperature.Prime.

P-Curve analysis combines the half and full p-curve to make inferences about evidential value. In particular, if the half p-curve test is right-skewed (halfp) with p<.05 or both the half and full test (fullp) are right-skewed with p < .1, then p-curve analysis indicates the presence of evidential value. Similarly, p-curve analysis indicates that evidential value is inadequate or absent if the 33% power test is p < .05 for the full p-curve (fullp33) or both the half p-curve (halfp33) and binomial 33% power test (binomp33) are p < .1.

ksig = average number of effects associated with p < .05; khalf = average number of effects associated with p < .025;…z = average z-values; power.est = average estimated statistical power of the studies (with lower bound and upper bound)

vars n mean sd se
ksig 1 1000 18.589 0.854 0.027
khalf 2 1000 11.071 0.707 0.022
fullz 3 1000 -3.165 0.097 0.003
fullp 4 1000 0.001 0.000 0.000
fullz33 5 1000 0.498 0.119 0.004
fullp33 6 1000 0.690 0.042 0.001
halfz 7 1000 -4.194 0.321 0.010
halfp 8 1000 0.000 0.000 0.000
halfz33 9 1000 5.263 0.046 0.001
halfp33 10 1000 1.000 0.000 0.000
binomp 11 1000 0.282 0.067 0.002
binomp33 12 1000 0.212 0.057 0.002
power.ci.lb 13 1000 0.157 0.013 0.000
power.est 14 1000 0.423 0.023 0.001
power.ci.up 15 1000 0.692 0.021 0.001

5.1.2 Permutation p-curve for Outside.Temperature.

vars n mean sd se
ksig 1 1000 11.482 0.500 0.016
khalf 2 1000 7.482 0.500 0.016
fullz 3 1000 -5.434 1.262 0.040
fullp 4 1000 0.000 0.000 0.000
fullz33 5 1000 2.933 1.143 0.036
fullp33 6 1000 0.974 0.051 0.002
halfz 7 1000 -6.540 1.638 0.052
halfp 8 1000 0.000 0.000 0.000
halfz33 9 1000 6.175 1.260 0.040
halfp33 10 1000 1.000 0.000 0.000
binomp 11 1000 0.236 0.040 0.001
binomp33 12 1000 0.434 0.037 0.001
power.ci.lb 13 1000 0.564 0.194 0.006
power.est 14 1000 0.798 0.125 0.004
power.ci.up 15 1000 0.927 0.053 0.002

5.1.3 Permutation p-curve for Temperature.Estimate.

vars n mean sd se
ksig 1 1000 20.000 0.000 0.000
khalf 2 1000 12.000 0.000 0.000
fullz 3 1000 -2.272 0.029 0.001
fullp 4 1000 0.012 0.001 0.000
fullz33 5 1000 -0.401 0.025 0.001
fullp33 6 1000 0.344 0.009 0.000
halfz 7 1000 -2.327 0.070 0.002
halfp 8 1000 0.010 0.002 0.000
halfz33 9 1000 4.549 0.019 0.001
halfp33 10 1000 1.000 0.000 0.000
binomp 11 1000 0.252 0.000 0.000
binomp33 12 1000 0.204 0.000 0.000
power.ci.lb 13 1000 0.076 0.002 0.000
power.est 14 1000 0.265 0.005 0.000
power.ci.up 15 1000 0.552 0.004 0.000

5.1.4 Permutation p-curve for Subjective.Warmth.Judgment

vars n mean sd se
ksig 1 1000 5.473 0.500 0.016
khalf 2 1000 3.473 0.500 0.016
fullz 3 1000 -4.583 0.102 0.003
fullp 4 1000 0.000 0.000 0.000
fullz33 5 1000 2.570 0.138 0.004
fullp33 6 1000 0.995 0.002 0.000
halfz 7 1000 -5.572 0.479 0.015
halfp 8 1000 0.000 0.000 0.000
halfz33 9 1000 4.939 0.179 0.006
halfp33 10 1000 1.000 0.000 0.000
binomp 11 1000 0.426 0.078 0.002
binomp33 12 1000 0.499 0.055 0.002
power.ci.lb 13 1000 0.669 0.050 0.002
power.est 14 1000 0.921 0.020 0.001
power.ci.up 15 1000 0.986 0.004 0.000

5.2 Meta-analysis for different methods

k = number of studies; sqrt in “Variance components” = tau, the standard deviation of true effects; estimate in “Model results” = naive MA estimate

meta k estimate stderror tau I2
Visual.Verbal.Temperature.Prime. 17 0.407 0.034 0.000 0.00
Outside.Temperature. 14 0.443 0.123 0.392 95.51
Temperature.Estimate. 23 0.465 0.072 0.265 71.95
Subjective.Warmth.Judgment 8 0.111 0.083 0.209 86.39

5.2.1 Contour enhanced funnel plots

5.2.2 Forest plots

5.3 Small-study effects correction for different methods

Bias-corrected estimate, note especially the CI (conf.low, conf.high).

method term variable value
Visual.Verbal.Temperature.Prime..1 3PSM b0 estimate 0.243
Visual.Verbal.Temperature.Prime..4 3PSM b0 p.value 0.000
Visual.Verbal.Temperature.Prime..5 3PSM b0 conf.low 0.188
Visual.Verbal.Temperature.Prime..6 3PSM b0 conf.high 0.298
Outside.Temperature..1 3PSM b0 estimate 0.438
Outside.Temperature..4 3PSM b0 p.value 0.009
Outside.Temperature..5 3PSM b0 conf.low 0.111
Outside.Temperature..6 3PSM b0 conf.high 0.765
Temperature.Estimate..1 3PSM b0 estimate 0.122
Temperature.Estimate..4 3PSM b0 p.value 0.269
Temperature.Estimate..5 3PSM b0 conf.low -0.094
Temperature.Estimate..6 3PSM b0 conf.high 0.338
Subjective.Warmth.Judgment.1 3PSM b0 estimate 0.230
Subjective.Warmth.Judgment.4 3PSM b0 p.value 0.166
Subjective.Warmth.Judgment.5 3PSM b0 conf.low -0.096
Subjective.Warmth.Judgment.6 3PSM b0 conf.high 0.556

5.3.1 PET-PEESE

Estimated effect size of an infinitely precise study. Using 3PSM as the conditional estimator instead of PET. If the PET-PEESE estimate is negative, the effect can be regarded 0. pval = p-value testing H0 that the effect is zero. cil.lb and ci.ub are upper and lower bound of the CI.

y-axis intercept represents the estimated bias-corrected ES.

PET-PEESE estimate se zval pval ci.lb ci.ub
Visual.Verbal.Temperature.Prime. 0.265 0.077 3.425 0.001 0.113 0.416
Outside.Temperature. 0.120 0.150 0.799 0.424 -0.174 0.414
Temperature.Estimate. -0.150 0.088 -1.700 0.105 -0.334 0.034
Subjective.Warmth.Judgment 0.024 0.273 0.087 0.934 -0.644 0.691

6 Category effects

6.1 Evidential value

6.1.1 Permutation p-curve for Category..Emotion

P-Curve analysis combines the half and full p-curve to make inferences about evidential value. In particular, if the half p-curve test is right-skewed (halfp) with p<.05 or both the half and full test (fullp) are right-skewed with p < .1, then p-curve analysis indicates the presence of evidential value. Similarly, p-curve analysis indicates that evidential value is inadequate or absent if the 33% power test is p < .05 for the full p-curve (fullp33) or both the half p-curve (halfp33) and binomial 33% power test (binomp33) are p < .1.

ksig = average number of effects associated with p < .05; khalf = average number of effects associated with p < .025;…z = average z-values; power.est = average estimated statistical power of the studies (with lower bound and upper bound)

vars n mean sd se
ksig 1 1000 27.476 0.875 0.028
khalf 2 1000 16.157 0.998 0.032
fullz 3 1000 -4.676 0.327 0.010
fullp 4 1000 0.000 0.000 0.000
fullz33 5 1000 1.178 0.300 0.009
fullp33 6 1000 0.870 0.061 0.002
halfz 7 1000 -6.147 0.326 0.010
halfp 8 1000 0.000 0.000 0.000
halfz33 9 1000 6.942 0.097 0.003
halfp33 10 1000 1.000 0.000 0.000
binomp 11 1000 0.246 0.114 0.004
binomp33 12 1000 0.135 0.085 0.003
power.ci.lb 13 1000 0.267 0.042 0.001
power.est 14 1000 0.513 0.044 0.001
power.ci.up 15 1000 0.721 0.031 0.001

6.1.2 Permutation p-curve for Category…Interpersonal

vars n mean sd se
ksig 1 1000 60.983 1.235 0.039
khalf 2 1000 36.161 1.752 0.055
fullz 3 1000 -6.391 0.619 0.020
fullp 4 1000 0.000 0.000 0.000
fullz33 5 1000 1.412 0.516 0.016
fullp33 6 1000 0.895 0.087 0.003
halfz 7 1000 -7.826 0.758 0.024
halfp 8 1000 0.000 0.000 0.000
halfz33 9 1000 9.822 0.527 0.017
halfp33 10 1000 1.000 0.000 0.000
binomp 11 1000 0.108 0.069 0.002
binomp33 12 1000 0.048 0.038 0.001
power.ci.lb 13 1000 0.312 0.051 0.002
power.est 14 1000 0.478 0.053 0.002
power.ci.up 15 1000 0.632 0.046 0.001

6.1.3 Permutation p-curve for Category..Person.Perception

vars n mean sd se
ksig 1 1000 21.454 1.291 0.041
khalf 2 1000 13.988 1.453 0.046
fullz 3 1000 -4.875 0.746 0.024
fullp 4 1000 0.000 0.000 0.000
fullz33 5 1000 1.879 0.678 0.021
fullp33 6 1000 0.939 0.078 0.002
halfz 7 1000 -5.897 0.676 0.021
halfp 8 1000 0.000 0.000 0.000
halfz33 9 1000 6.693 0.419 0.013
halfp33 10 1000 1.000 0.000 0.000
binomp 11 1000 0.147 0.120 0.004
binomp33 12 1000 0.372 0.195 0.006
power.ci.lb 13 1000 0.371 0.104 0.003
power.est 14 1000 0.619 0.091 0.003
power.ci.up 15 1000 0.807 0.057 0.002

6.1.4 Permutation p-curve for Category..Group.Processes

vars n mean sd se
ksig 1 1000 11.191 0.675 0.021
khalf 2 1000 8.431 0.795 0.025
fullz 3 1000 -6.808 1.134 0.036
fullp 4 1000 0.000 0.000 0.000
fullz33 5 1000 4.047 0.919 0.029
fullp33 6 1000 0.999 0.004 0.000
halfz 7 1000 -7.156 1.371 0.043
halfp 8 1000 0.000 0.000 0.000
halfz33 9 1000 6.737 0.994 0.031
halfp33 10 1000 1.000 0.000 0.000
binomp 11 1000 0.090 0.044 0.001
binomp33 12 1000 0.724 0.096 0.003
power.ci.lb 13 1000 0.787 0.143 0.005
power.est 14 1000 0.928 0.068 0.002
power.ci.up 15 1000 0.977 0.022 0.001

6.1.5 Permutation p-curve for Category..Self.Regulation

vars n mean sd se
ksig 1 1000 46.895 1.181 0.037
khalf 2 1000 27.904 1.399 0.044
fullz 3 1000 -3.682 0.232 0.007
fullp 4 1000 0.000 0.000 0.000
fullz33 5 1000 -0.607 0.211 0.007
fullp33 6 1000 0.276 0.070 0.002
halfz 7 1000 -3.523 0.342 0.011
halfp 8 1000 0.000 0.001 0.000
halfz33 9 1000 6.527 0.083 0.003
halfp33 10 1000 1.000 0.000 0.000
binomp 11 1000 0.137 0.074 0.002
binomp33 12 1000 0.071 0.048 0.002
power.ci.lb 13 1000 0.121 0.014 0.000
power.est 14 1000 0.266 0.024 0.001
power.ci.up 15 1000 0.456 0.025 0.001

6.1.6 Permutation p-curve for Category..Cognitive.Processes

vars n mean sd se
ksig 1 1000 28.729 1.411 0.045
khalf 2 1000 16.450 1.499 0.047
fullz 3 1000 -4.635 0.676 0.021
fullp 4 1000 0.000 0.000 0.000
fullz33 5 1000 1.190 0.610 0.019
fullp33 6 1000 0.844 0.132 0.004
halfz 7 1000 -6.084 0.516 0.016
halfp 8 1000 0.000 0.000 0.000
halfz33 9 1000 6.728 0.410 0.013
halfp33 10 1000 1.000 0.000 0.000
binomp 11 1000 0.298 0.157 0.005
binomp33 12 1000 0.111 0.093 0.003
power.ci.lb 13 1000 0.276 0.077 0.002
power.est 14 1000 0.496 0.081 0.003
power.ci.up 15 1000 0.697 0.061 0.002

6.1.7 Permutation p-curve for Category..Economic.Decision.Making

vars n mean sd se
ksig 1 1000 36.866 1.685 0.053
khalf 2 1000 22.362 1.692 0.054
fullz 3 1000 -3.455 0.592 0.019
fullp 4 1000 0.001 0.003 0.000
fullz33 5 1000 -0.314 0.522 0.016
fullp33 6 1000 0.391 0.180 0.006
halfz 7 1000 -3.423 0.712 0.023
halfp 8 1000 0.003 0.007 0.000
halfz33 9 1000 5.506 0.461 0.015
halfp33 10 1000 1.000 0.000 0.000
binomp 11 1000 0.152 0.100 0.003
binomp33 12 1000 0.145 0.104 0.003
power.ci.lb 13 1000 0.135 0.041 0.001
power.est 14 1000 0.296 0.062 0.002
power.ci.up 15 1000 0.499 0.062 0.002

6.2 Meta-analysis for effect categories

k = number of studies; sqrt in “Variance components” = tau, the standard deviation of true effects; estimate in “Model results” = naive MA estimate

meta k estimate stderror tau I2
Category..Emotion 26 0.315 0.051 0.200 74.18
Category…Interpersonal 75 0.423 0.053 0.362 82.88
Category..Person.Perception 31 0.471 0.088 0.342 83.23
Category..Group.Processes 18 0.554 0.070 0.186 43.90
Category..Self.Regulation 41 0.346 0.055 0.242 76.10
Category..Cognitive.Processes 29 0.499 0.054 0.154 32.76
Category..Economic.Decision.Making 60 0.439 0.058 0.287 85.05

6.2.1 Contour enhanced funnel plots

6.2.1.1 Forest plots

6.3 Small-study effects correction for effect category

6.3.1 3-parameter selection model

Bias-corrected estimate, note especially the CI (conf.low, conf.high).

method term variable value
Category..Emotion.1 3PSM b0 estimate 0.046
Category..Emotion.4 3PSM b0 p.value 0.528
Category..Emotion.5 3PSM b0 conf.low -0.096
Category..Emotion.6 3PSM b0 conf.high 0.188
Category…Interpersonal.1 3PSM b0 estimate 0.212
Category…Interpersonal.4 3PSM b0 p.value 0.005
Category…Interpersonal.5 3PSM b0 conf.low 0.064
Category…Interpersonal.6 3PSM b0 conf.high 0.358
Category..Person.Perception.1 3PSM b0 estimate 0.346
Category..Person.Perception.4 3PSM b0 p.value 0.001
Category..Person.Perception.5 3PSM b0 conf.low 0.138
Category..Person.Perception.6 3PSM b0 conf.high 0.555
Category..Group.Processes.1 3PSM b0 estimate 0.391
Category..Group.Processes.4 3PSM b0 p.value 0.000
Category..Group.Processes.5 3PSM b0 conf.low 0.206
Category..Group.Processes.6 3PSM b0 conf.high 0.576
Category..Self.Regulation.1 3PSM b0 estimate 0.178
Category..Self.Regulation.4 3PSM b0 p.value 0.006
Category..Self.Regulation.5 3PSM b0 conf.low 0.051
Category..Self.Regulation.6 3PSM b0 conf.high 0.304
Category..Cognitive.Processes.1 3PSM b0 estimate 0.294
Category..Cognitive.Processes.4 3PSM b0 p.value 0.000
Category..Cognitive.Processes.5 3PSM b0 conf.low 0.129
Category..Cognitive.Processes.6 3PSM b0 conf.high 0.459
Category..Economic.Decision.Making.1 3PSM b0 estimate 0.202
Category..Economic.Decision.Making.4 3PSM b0 p.value 0.002
Category..Economic.Decision.Making.5 3PSM b0 conf.low 0.078
Category..Economic.Decision.Making.6 3PSM b0 conf.high 0.326

6.3.2 PET-PEESE

Estimated effect size of an infinitely precise study. Using 3PSM as the conditional estimator instead of PET. If the PET-PEESE estimate is negative, the effect can be regarded 0. pval = p-value testing H0 that the effect is zero. cil.lb and ci.ub are upper and lower bound of the CI.

y-axis intercept represents the estimated bias-corrected ES.

PET-PEESE estimate se zval pval ci.lb ci.ub
Category..Emotion -0.061 0.122 -0.502 0.621 -0.316 0.193
Category…Interpersonal 0.076 0.063 1.214 0.225 -0.047 0.200
Category..Person.Perception 0.154 0.077 1.996 0.046 0.003 0.304
Category..Group.Processes 0.344 0.095 3.621 0.000 0.158 0.530
Category..Self.Regulation 0.060 0.052 1.153 0.249 -0.042 0.163
Category..Cognitive.Processes 0.217 0.085 2.552 0.011 0.050 0.384
Category..Economic.Decision.Making 0.037 0.041 0.903 0.367 -0.043 0.117

7 Meta-regression

7.1 Moderation by citations and IF

7.1.0.1 Overall effect moderated by citations and IF

## 
## Number of outcomes:   132 
## Number of clusters:   92 
## Outcomes per cluster: 1-5 (mean: 1.43, median: 1)
## 
## Test of Moderators (coefficient(s) 2:4): 
## F(df1 = 3, df2 = 88) = 3.7952, p-val = 0.0131
## 
## Model Results:
## 
##                                        estimate      se     tval    pval
## intrcpt                                  0.4589  0.0316  14.5258  <.0001
## scale(Publication.Year)                  0.0193  0.0315   0.6130  0.5414
## scale(Citations.March.1.2016..GS.)       0.0903  0.0313   2.8807  0.0050
## scale(H5.Index.GS.Journal.March.2016)   -0.1037  0.0384  -2.7016  0.0083
##                                          ci.lb    ci.ub     
## intrcpt                                 0.3961   0.5217  ***
## scale(Publication.Year)                -0.0433   0.0820     
## scale(Citations.March.1.2016..GS.)      0.0280   0.1526   **
## scale(H5.Index.GS.Journal.March.2016)  -0.1800  -0.0274   **
## 
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

7.2 Moderation by lattitude ??added code scale(mod)

7.2.0.1 Overall effect moderated by lattitude

## 
## Number of outcomes:   136 
## Number of clusters:   92 
## Outcomes per cluster: 1-5 (mean: 1.48, median: 1)
## 
## Test of Moderators (coefficient(s) 2): 
## F(df1 = 1, df2 = 90) = 0.3049, p-val = 0.5822
## 
## Model Results:
## 
##                                                 estimate      se     tval
## intrcpt                                           0.5018  0.0366  13.6943
## scale(Latitude.University..proxy.for.climate.)    0.0186  0.0336   0.5522
##                                                   pval    ci.lb   ci.ub
## intrcpt                                         <.0001   0.4290  0.5746
## scale(Latitude.University..proxy.for.climate.)  0.5822  -0.0482  0.0853
##                                                    
## intrcpt                                         ***
## scale(Latitude.University..proxy.for.climate.)     
## 
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

7.2.0.2 Priming effects moderated by lattitude

## 
## Number of outcomes:   101 
## Number of clusters:   70 
## Outcomes per cluster: 1-5 (mean: 1.44, median: 1)
## 
## Test of Moderators (coefficient(s) 2): 
## F(df1 = 1, df2 = 68) = 0.2842, p-val = 0.5957
## 
## Model Results:
## 
##                                                 estimate      se     tval
## intrcpt                                           0.5691  0.0398  14.2962
## scale(Latitude.University..proxy.for.climate.)   -0.0155  0.0290  -0.5331
##                                                   pval    ci.lb   ci.ub
## intrcpt                                         <.0001   0.4897  0.6485
## scale(Latitude.University..proxy.for.climate.)  0.5957  -0.0734  0.0425
##                                                    
## intrcpt                                         ***
## scale(Latitude.University..proxy.for.climate.)     
## 
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

7.2.0.3 Compensatory effects moderated by lattitude

## 
## Number of outcomes:   30 
## Number of clusters:   21 
## Outcomes per cluster: 1-4 (mean: 1.43, median: 1)
## 
## Test of Moderators (coefficient(s) 2): 
## F(df1 = 1, df2 = 19) = 4.0102, p-val = 0.0597
## 
## Model Results:
## 
##                                                 estimate      se    tval
## intrcpt                                           0.3074  0.0620  4.9609
## scale(Latitude.University..proxy.for.climate.)    0.1276  0.0637  2.0025
##                                                   pval    ci.lb   ci.ub
## intrcpt                                         <.0001   0.1777  0.4371
## scale(Latitude.University..proxy.for.climate.)  0.0597  -0.0058  0.2609
##                                                    
## intrcpt                                         ***
## scale(Latitude.University..proxy.for.climate.)    .
## 
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

7.2.0.4 Mood effects moderated by lattitude

## 
## Number of outcomes:   20 
## Number of clusters:   14 
## Outcomes per cluster: 1-2 (mean: 1.43, median: 1)
## 
## Test of Moderators (coefficient(s) 2): 
## F(df1 = 1, df2 = 12) = 0.3976, p-val = 0.5401
## 
## Model Results:
## 
##                                                 estimate      se     tval
## intrcpt                                           0.2083  0.0297   7.0198
## scale(Latitude.University..proxy.for.climate.)   -0.0160  0.0255  -0.6306
##                                                   pval    ci.lb   ci.ub
## intrcpt                                         <.0001   0.1436  0.2729
## scale(Latitude.University..proxy.for.climate.)  0.5401  -0.0715  0.0394
##                                                    
## intrcpt                                         ***
## scale(Latitude.University..proxy.for.climate.)     
## 
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

7.3 Moderation by gender ??added code

7.3.0.1 Overall effect moderated by gender

## 
## Number of outcomes:   122 
## Number of clusters:   88 
## Outcomes per cluster: 1-5 (mean: 1.39, median: 1)
## 
## Test of Moderators (coefficient(s) 2): 
## F(df1 = 1, df2 = 86) = 9.6126, p-val = 0.0026
## 
## Model Results:
## 
##                      estimate      se     tval    pval   ci.lb   ci.ub
## intrcpt                0.4755  0.0337  14.1217  <.0001  0.4086  0.5425
## scale(gender.ratio)    0.1103  0.0356   3.1004  0.0026  0.0396  0.1810
##                         
## intrcpt              ***
## scale(gender.ratio)   **
## 
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

7.3.0.2 Priming effects moderated by gender

## 
## Number of outcomes:   82 
## Number of clusters:   61 
## Outcomes per cluster: 1-4 (mean: 1.34, median: 1)
## 
## Test of Moderators (coefficient(s) 2): 
## F(df1 = 1, df2 = 59) = 13.9214, p-val = 0.0004
## 
## Model Results:
## 
##                      estimate      se     tval    pval   ci.lb   ci.ub
## intrcpt                0.4918  0.0412  11.9368  <.0001  0.4093  0.5742
## scale(gender.ratio)    0.1554  0.0416   3.7311  0.0004  0.0720  0.2387
##                         
## intrcpt              ***
## scale(gender.ratio)  ***
## 
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

7.3.0.3 Compensatory effects moderated by gender

## 
## Number of outcomes:   34 
## Number of clusters:   25 
## Outcomes per cluster: 1-4 (mean: 1.36, median: 1)
## 
## Test of Moderators (coefficient(s) 2): 
## F(df1 = 1, df2 = 23) = 0.0312, p-val = 0.8613
## 
## Model Results:
## 
##                      estimate      se     tval    pval    ci.lb   ci.ub
## intrcpt                0.3850  0.0480   8.0231  <.0001   0.2857  0.4843
## scale(gender.ratio)   -0.0082  0.0462  -0.1767  0.8613  -0.1038  0.0874
##                         
## intrcpt              ***
## scale(gender.ratio)     
## 
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

7.4 Overall effect moderated by published

7.4.1 Evidential value

7.4.1.1 Permutation p-curve for Unpublished

P-Curve analysis combines the half and full p-curve to make inferences about evidential value. In particular, if the half p-curve test is right-skewed (halfp) with p<.05 or both the half and full test (fullp) are right-skewed with p < .1, then p-curve analysis indicates the presence of evidential value. Similarly, p-curve analysis indicates that evidential value is inadequate or absent if the 33% power test is p < .05 for the full p-curve (fullp33) or both the half p-curve (halfp33) and binomial 33% power test (binomp33) are p < .1.

ksig = average number of effects associated with p < .05; khalf = average number of effects associated with p < .025;…z = average z-values; power.est = average estimated statistical power of the studies (with lower bound and upper bound)

vars n mean sd se
ksig 1 1000 10.748 0.857 0.027
khalf 2 1000 8.756 0.862 0.027
fullz 3 1000 -3.702 0.615 0.019
fullp 4 1000 0.001 0.002 0.000
fullz33 5 1000 1.451 0.561 0.018
fullp33 6 1000 0.896 0.102 0.003
halfz 7 1000 -2.686 0.702 0.022
halfp 8 1000 0.015 0.025 0.001
halfz33 9 1000 4.067 0.410 0.013
halfp33 10 1000 1.000 0.000 0.000
binomp 11 1000 0.049 0.040 0.001
binomp33 12 1000 0.844 0.108 0.003
power.ci.lb 13 1000 0.300 0.108 0.003
power.est 14 1000 0.627 0.103 0.003
power.ci.up 15 1000 0.856 0.053 0.002

7.4.2 Evidential value

7.4.2.1 Permutation p-curve for Published

P-Curve analysis combines the half and full p-curve to make inferences about evidential value. In particular, if the half p-curve test is right-skewed (halfp) with p<.05 or both the half and full test (fullp) are right-skewed with p < .1, then p-curve analysis indicates the presence of evidential value. Similarly, p-curve analysis indicates that evidential value is inadequate or absent if the 33% power test is p < .05 for the full p-curve (fullp33) or both the half p-curve (halfp33) and binomial 33% power test (binomp33) are p < .1.

ksig = average number of effects associated with p < .05; khalf = average number of effects associated with p < .025;…z = average z-values; power.est = average estimated statistical power of the studies (with lower bound and upper bound)

vars n mean sd se
ksig 1 1000 76.469 1.590 0.050
khalf 2 1000 48.385 1.931 0.061
fullz 3 1000 -8.292 0.719 0.023
fullp 4 1000 0.000 0.000 0.000
fullz33 5 1000 2.492 0.626 0.020
fullp33 6 1000 0.983 0.026 0.001
halfz 7 1000 -9.921 0.896 0.028
halfp 8 1000 0.000 0.000 0.000
halfz33 9 1000 11.476 0.638 0.020
halfp33 10 1000 1.000 0.000 0.000
binomp 11 1000 0.019 0.018 0.001
binomp33 12 1000 0.113 0.072 0.002
power.ci.lb 13 1000 0.411 0.056 0.002
power.est 14 1000 0.554 0.052 0.002
power.ci.up 15 1000 0.680 0.044 0.001
estimate stderror meta tau
0.339 0.074 Unpublished 0.296
0.464 0.035 Published 0.297

7.4.2.2 \(I^2\) for unpublished

Here, total relative heterogeneity was 77.04 %. Separate estimates of between- and within-cluster heterogeneity were 71.66, 5.39 %, respectively.

7.4.2.3 \(I^2\) for published studies

Here, total relative heterogeneity was 87.12 %. Separate estimates of between- and within-cluster heterogeneity were 48.06, 39.06 %, respectively.

7.4.2.4 Wald test p-value

Testing the difference in the uncorrected MA estimates between published and unpublished

## [1] 0.126761

7.4.3 Contour enhanced funnel plots

7.4.4 Small-study effects correction for unpublished effects

7.4.4.1 3-parameter selection model for unpublished effects

Bias-corrected estimate, note especially the CI (conf.low, conf.high).

method term variable value
1 3PSM b0 estimate 0.234
4 3PSM b0 p.value 0.012
5 3PSM b0 conf.low 0.052
6 3PSM b0 conf.high 0.416

7.4.4.2 PET-PEESE for unpublished effects

Estimated effect size of an infinitely precise study. Using 3PSM as the conditional estimator instead of PET. If the PET-PEESE estimate is negative, the effect can be regarded 0. pval = p-value testing H0 that the effect is zero. cil.lb and ci.ub are upper and lower bound of the CI.
PEESE estimate 0.037
se 0.079
zval 0.463
pval 0.644
ci.lb -0.119
ci.ub 0.192

y-axis intercept represents the estimated bias-corrected ES.

7.4.5 Small-study effects correction for published effects

7.4.5.1 3-parameter selection model for published effects

Bias-corrected estimate, note especially the CI (conf.low, conf.high).

method term variable value
1 3PSM b0 estimate 0.209
4 3PSM b0 p.value 0.000
5 3PSM b0 conf.low 0.113
6 3PSM b0 conf.high 0.305

7.4.5.2 PET-PEESE for published effects
PEESE estimate 0.152
se 0.040
zval 3.775
pval 0.000
ci.lb 0.073
ci.ub 0.231

7.5 Randomization

Overall effect moderated by the presence of randomization.

estimate stderror meta tau
0.410 0.071 Observational 0.345
0.466 0.036 Randomized 0.290

COMMENT: Observational studies seem to give somewhat higher ESs but not significantly so. Note especially huge heterogeneity of observational studies. That causes the huge SE of MA estimate, primarily producing ns difference btw those two sets.

7.5.0.1 Wald test p-value

Testing the difference in the uncorrected MA estimates between effects from observational and randomized studies.

## [1] 0.4820557

7.5.0.2 Contour enhanced funnel plots

7.5.1 Small-study effects correction for observational studies

7.5.1.1 3-parameter selection model for observational studies

Bias-corrected estimate, note especially the CI (conf.low, conf.high).

method term variable value
1 3PSM b0 estimate 0.218
4 3PSM b0 p.value 0.057
5 3PSM b0 conf.low -0.006
6 3PSM b0 conf.high 0.442

7.5.1.2 PET-PEESE for observational studies

Estimated effect size of an infinitely precise study. Using 3PSM as the conditional estimator instead of PET. If the PET-PEESE estimate is negative, the effect can be regarded 0. pval = p-value testing H0 that the effect is zero. cil.lb and ci.ub are upper and lower bound of the CI.
PET estimate -0.094
se 0.094
zval -1.000
pval 0.327
ci.lb -0.288
ci.ub 0.100

y-axis intercept represents the estimated bias-corrected ES.

7.5.2 Small-study effects correction for randomized studies

7.5.2.1 3-parameter selection model for randomized studies

Bias-corrected estimate, note especially the CI (conf.low, conf.high).

method term variable value
1 3PSM b0 estimate 0.208
4 3PSM b0 p.value 0.000
5 3PSM b0 conf.low 0.117
6 3PSM b0 conf.high 0.300

7.5.2.2 PET-PEESE for randomized studies
PEESE estimate 0.114
se 0.040
zval 2.851
pval 0.004
ci.lb 0.036
ci.ub 0.193

7.6 Year of Publication

Linear mixed-effects model. Taking into effect clustering of ESs due to originating from the same study. Using square root of variance to make the distribution normal.

Estimate Std. Error df t value Pr(>|t|)
(Intercept) -0.119 0.097 88.946 -1.234 0.220
scale(H5.Index.GS.Journal.March.2016) -0.169 0.101 88.853 -1.664 0.100
scale(Publication.Year) -0.210 0.094 88.964 -2.244 0.027

Comment: all the variables were centered for easier interpretation of model coefficients. See the negative beta for Publication Year. The higher the publication year, the lower the variance (better precision), controlling for H5.

Thus, practices regarding the precision of studies (mainly due to N) seem to have improved throughout last years.

7.6.0.1 Scatterplot year <-> precision

Size of the points indicate the H5 index (the bigger the higher) of the journal that the ES is published in.

7.7 Citations

Linear mixed-effects model. Taking into effect clustering of ESs due to originating from the same study. Using square root of variance to make the distribution normal.

Estimate Std. Error df t value Pr(>|t|)
(Intercept) -0.166 0.093 87.939 -1.781 0.078
scale(Publication.Year) 0.033 0.118 88.034 0.284 0.777
scale(H5.Index.GS.Journal.March.2016) -0.358 0.113 87.694 -3.161 0.002
scale(Citations.March.1.2016..GS.) 0.355 0.111 88.042 3.187 0.002

7.7.0.1 Scatterplot precision <-> citations

The relationship between precision (sqrt of variance) and number of citations.

7.7.0.2 Scatterplot precision <-> journal H5

Linear mixed-effects model. Taking into effect clustering of ESs due to originating from the same study. Using square root of variance to make the distribution normal.

Estimate Std. Error df t value Pr(>|t|)
(Intercept) -0.096 0.098 89.995 -0.978 0.331
scale(H5.Index.GS.Journal.March.2016) -0.129 0.102 89.910 -1.266 0.209

The relationship between precision (sqrt of variance) and H5 index of the journal.

7.8 Decline effect

Linear mixed-effects model. Taking into effect clustering of ESs due to originating from the same study.

Estimate Std. Error df t value Pr(>|t|)
(Intercept) 0.008 0.084 84.478 0.100 0.920
scale(sqrt(g.var.calc)) 0.245 0.086 86.269 2.843 0.006
scale(Publication.Year) 0.161 0.087 97.746 1.858 0.066

7.9 Citation bias

Linear mixed-effects model. Taking into effect clustering of ESs due to originating from the same study.

Estimate Std. Error df t value Pr(>|t|)
(Intercept) 0.025 0.082 111.614 0.302 0.763
scale(sqrt(g.var.calc)) 0.220 0.082 117.606 2.665 0.009
scale(Citations.March.1.2016..GS.) -0.086 0.079 145.676 -1.087 0.279