This is the supplementary analytic output for the paper Stress regulation via being in nature and social support in adults: a meta-analysis
It reports detailed results for all models reported in the paper. The analytic R script by which this html report was generated can be found on the project’s OSF page at: https://osf.io/6wpav/
Brief information about the methods used in the analysis:
RMA results with model-based SEs k = number of studies; sqrt in “Variance components” = tau, the standard deviation of true effects; estimate in “Model results” = naive MA estimate
RVE SEs with Satterthwaite small-sample correction Estimate based on a multilevel RE model with constant sampling correlation model (CHE - correlated hierarchical effects - working model) (Pustejovsky & Tipton, 2020; https://osf.io/preprints/metaarxiv/vyfcj/). Interpretation of naive-meta-analysis should be based on these estimates.
Prediction interval Shows the expected range of true effects in similar studies. As an approximation, in 95% of cases the true effect in a new published study can be expected to fall between PI LB and PI UB. Note that these are non-adjusted estimates. An unbiased newly conducted study will more likely fall in an interval centered around bias-adjusted estimate with a wider CI width.
Heterogeneity Tau can be interpreted as the total amount of heterogeneity in the true effects. I^2$ represents the ratio of true heterogeneity to total variance across the observed effect estimates. Estimates calculated by two approaches are reported. This is followed by separate estimates of between- and within-cluster heterogeneity and estimated intra-class correlation of underlying true effects.
Proportion of significant results What proportion of effects were statistically at the alpha level of .05.
ES-precision correlation Kendalls’s correlation between the ES and precision.
4/3PSM Applies a permutation-based, step-function 4-parameter selection model (one-tailed p-value steps = c(.025, .5, 1)). Falls back to 3-parameter selection model if at least one of the three p-value intervals contains less than 5 p-values. For this meta-analysis, we applied 3-parameter selection model by default as there were only 11 independent effects in the opposite direction overall (6%), causing the estimates to be unstable across iterations. pvalue = p-value testing H0 that the effect is zero. ciLB and ciUB are lower and upper bound of the CI. k = number of studies. steps = 3 means that the 4PSM was applied, 2 means that the 3PSM was applied. We also ran two sensitivity analyses of the selection model, the Vevea & Woods (2005) step function model with a priori defined selection weights and the Robust Bayesian Meta-analysis model employing the model-averaging approach (Bartoš & Maier, 2020).
PET-PEESE Estimated effect size of an infinitely precise study. Using 4/3PSM as the conditional estimator instead of PET (can be changed to PET). If the PET-PEESE estimate is in the opposite direction, the effect can be regarded nil. By default (can be changed to PET), the function employs a modified sample-size based estimator (see https://www.jepusto.com/pet-peese-performance/). It also uses the same RVE sandwich-type based estimator in a CHE (correlated hierarchical effects) working model with the identical random effects structure as the primary (naive) meta-analytic model.
We report results for both, PET and PEESE, with the first reported one being the primary (based on the conditional estimator).
WAAP-WLS The combined WAAP-WLS estimator (weighted average of the adequately powered - weighted least squares) tries to identify studies that are adequately powered to detect the meta-analytic effect. If there is less than two such studies, the method falls back to the WLS estimator (Stanley & Doucouliagos, 2015). If there are at least two adequately powered studies, WAAP returns a WLS estimate based on effects from only those studies.
type = 1: WAAP estimate, 2: WLS estimate. kAdequate = number of adequately powered studies
p-uniform P-uniform* is a selection model conceptually similar to p-curve. It makes use of the fact that p-values follow a uniform distribution at the true effect size while it includes also nonsignificant effect sizes. Permutation-based version of p-uniform method, the so-called p-uniform* (van Aert, van Assen, 2021).
p-curve Permutation-based p-curve method. Output should be self-explanatory. For more info see p-curve.com
Power for detecting SESOI and bias-corrected parameter estimates Estimates of the statistical power for detecting a smallest effect sizes of interest equal to .20, .50, and .70 in SD units (Cohen’s d). A sort of a thought experiment, we also assumed that population true values equal the bias-corrected estimates (4/3PSM or PET-PEESE) and computed power for those.
Handling of dependencies in bias-correction methods To handle dependencies among the effects, the 4PSM, p-curve, p-uniform are implemented using a permutation-based procedure, randomly selecting only one focal effect (i.e., excluding those which were not coded as being focal) from a single study and iterating nIterations times. Lastly, the procedure selects the result with the median value of the ES estimate (4PSM, p-uniform) or median z-score of the full p-curve (p-curve).
## from to
## 1993 2021
## [1] 54
## [1] 16
## [1] 15
## [1] 52.5
## [1] 1697
## Mean SD
## 49.90571 32.49089
## [1] 29.54122
## $`RMA results with model-based SEs`
##
## Multivariate Meta-Analysis Model (k = 54; method: REML)
##
## Variance Components:
##
## estim sqrt nlvls fixed factor
## sigma^2.1 0.1128 0.3358 16 no study
## sigma^2.2 0.0853 0.2920 54 no study/result
##
## Test for Heterogeneity:
## Q(df = 53) = 361.3608, p-val < .0001
##
## Model Results:
##
## estimate se zval pval ci.lb ci.ub
## -0.4211 0.1084 -3.8848 0.0001 -0.6335 -0.2086 ***
##
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
##
## $`RVE SEs with Satterthwaite small-sample correction`
## $`RVE SEs with Satterthwaite small-sample correction`$test
## Coef. Estimate SE t-stat d.f. p-val (Satt) Sig.
## 1 intrcpt -0.421 0.109 -3.86 14.3 0.00166 **
##
## $`RVE SEs with Satterthwaite small-sample correction`$CIs
## Coef Estimate SE d.f. Lower 95% CI Upper 95% CI
## 1 intrcpt -0.421 0.109 14.3 -0.654 -0.188
##
##
## $`Prediction interval`
## 95% PI LB 95% PI UB
## -1.397 0.555
##
## $Heterogeneity
## Tau I^2
## 0.4450121 85.7479248
## Jackson's I^2 Between-cluster heterogeneity
## 94.6700000 48.8300000
## Within-cluster heterogeneity ICC
## 36.9200000 0.5700000
##
## $`Proportion of significant results`
## [1] 0.4444444
##
## $`Publication bias`
## $`Publication bias`$`ES-precision correlation`
## [1] -0.2634521
##
## $`Publication bias`$`4/3PSM`
## est se zvalue pvalue ciLB ciUB k steps
## -0.598 0.215 -2.776 0.006 -1.020 -0.176 16.000 2.000
##
## $`Publication bias`$`Vevea & Woods SM`
## est se zvalue pvalue ciLB ciUB k steps
## moderateSelection -0.307 0.123 -2.500 0.012 -0.548 -0.066 16 9
## severeSelection -0.181 0.131 -1.378 0.168 -0.437 0.076 16 9
## extremeSelection -0.009 0.150 -0.058 0.954 -0.303 0.286 16 9
##
## $`Publication bias`$`Robust BMA`
## Model-averaged estimates:
## Mean Median 0.025 0.975
## mu -0.423 -0.421 -0.573 -0.283
## tau 0.434 0.429 0.340 0.547
## omega[0,0.025] 1.000 1.000 1.000 1.000
## omega[0.025,0.05] 0.991 1.000 0.856 1.000
## omega[0.05,0.5] 0.967 1.000 0.511 1.000
## omega[0.5,0.95] 0.964 1.000 0.471 1.000
## omega[0.95,0.975] 0.965 1.000 0.471 1.000
## omega[0.975,1] 0.967 1.000 0.471 1.000
## PET 0.017 0.000 0.000 0.279
## PEESE 0.014 0.000 0.000 0.000
## (Estimated publication weights omega correspond to one-sided p-values.)
##
## $`Publication bias`$`PET-PEESE`
## PET estimate se zvalue pvalue ciLB
## -0.546 0.336 -1.628 0.126 -1.266
## ciUB PEESE estimate se zvalue pvalue
## 0.173 -0.437 0.198 -2.209 0.044
## ciLB ciUB
## -0.861 -0.013
##
## $`Publication bias`$`WAAP-WLS`
## method term estimate std.error statistic p.value conf.low
## 1 WAAP-WLS b0 -0.5931473 0.0742671 0.0742671 0.0000000001175175 -0.7421082
## conf.high type kAdequate
## 1 -0.4441863 2 0
##
## $`Publication bias`$`p-uniform*`
## est ciLB ciUB pvalue
## -0.6797468091 -1.0265737657 -0.3180098564 0.0002537374
##
## $`Publication bias`$`p-curve`
## P-curve analysis
## -----------------------
## - Total number of provided studies: k = 16
## - Total number of p<0.05 studies included into the analysis: k = 4 (25%)
## - Total number of studies with p<0.025: k = 3 (18.75%)
##
## Results
## -----------------------
## pBinomial zFull pFull zHalf pHalf
## Right-skewness test 0.312 -6.681 0 -7.962 0
## Flatness test 0.740 5.691 1 7.767 1
## Note: p-values of 0 or 1 correspond to p<0.001 and p>0.999, respectively.
## Power Estimate: 99% (99%-99%)
##
## Evidential value
## -----------------------
## - Evidential value present: yes
## - Evidential value absent/inadequate: no
##
##
## $`Power for detecting SESOI and bias-corrected parameter estimates`
## Median power for detecting a SESOI of d = .20
## 0.174
## Median power for detecting a SESOI of d = .50
## 0.718
## Median power for detecting a SESOI of d = .70
## 0.944
## Median power for detecting PET-PEESE estimate
## 0.791
## Median power for detecting 4/3PSM estimate
## 0.858
Using the sqrt(2/n) and 2/n terms instead of SE and var for PET and PEESE, respectively since modified sample-size based estimator was implemented (see https://www.jepusto.com/pet-peese-performance/).
## $Nature
## $Nature$`Proportion of females`
## [1] 50
##
## $Nature$`Model results`
## Coef. Estimate SE t-stat p-val (z) Sig.
## 1 percFemale -0.00609 0.00162 -3.77 <0.001 ***
##
##
## $Social
## $Social$`Proportion of females`
## [1] 43.45
##
## $Social$`Model results`
## Coef. Estimate SE t-stat p-val (z) Sig.
## 1 percFemale -0.00159 0.00105 -1.5 0.133
## $Nature
## $Nature[[1]]
##
## 1 2
## 10 44
##
## $Nature$`Model results`
## Coef. Estimate SE t-stat p-val (z) Sig.
## 1 factor(comparisonGroupType)1 0.150 0.0251 5.98 <0.001 ***
## 2 factor(comparisonGroupType)2 -0.547 0.1011 -5.41 <0.001 ***
##
## $Nature$`RVE Wald test`
## test Fstat df_num df_denom p_val sig
## HTZ 44.8 1 1.48 0.0438 *
##
##
## $Social
## $Social[[1]]
##
## 0 1
## 16 2
##
## $Social$`Model results`
## Coef. Estimate SE t-stat p-val (z) Sig.
## 1 factor(comparisonGroupType)0 -0.172 0.0496 -3.46 <0.001 ***
## 2 factor(comparisonGroupType)1 0.127 0.0905 1.40 0.16
##
## $Social$`RVE Wald test`
## test Fstat df_num df_denom p_val sig
## HTZ 8.39 1 1.24 0.172
## $Nature
## $Nature[[1]]
##
## 1 2
## 29 24
##
## $Nature$`Model results`
## Coef. Estimate SE t-stat p-val (z) Sig.
## 1 factor(populationType)1 -0.357 0.137 -2.61 0.00897 **
## 2 factor(populationType)2 -0.506 0.166 -3.05 0.00226 **
##
## $Nature$`RVE Wald test`
## test Fstat df_num df_denom p_val sig
## HTZ 0.479 1 10.6 0.504
##
##
## $Social
## $Social[[1]]
##
## 1 2 3
## 6 6 6
##
## $Social$`Model results`
## Coef. Estimate SE t-stat p-val (z) Sig.
## 1 factor(populationType)1 -0.25094 0.0712 -3.5223 <0.001 ***
## 2 factor(populationType)2 -0.13547 0.0613 -2.2091 0.0272 *
## 3 factor(populationType)3 -0.00389 0.1172 -0.0332 0.9735
##
## $Social$`RVE Wald test`
## test Fstat df_num df_denom p_val sig
## HTZ 1.53 2 5.96 0.292
## [[1]]
##
## 1 2 3 4
## 23 6 18 7
##
## $`Model results`
##
## Multivariate Meta-Analysis Model (k = 54; method: REML)
##
## Variance Components:
##
## estim sqrt nlvls fixed factor
## sigma^2.1 0.1410 0.3755 16 no study
## sigma^2.2 0.0870 0.2949 54 no study/result
##
## Test for Residual Heterogeneity:
## QE(df = 50) = 347.4456, p-val < .0001
##
## Test of Moderators (coefficients 1:4):
## QM(df = 4) = 13.5188, p-val = 0.0090
##
## Model Results:
##
## estimate se zval pval ci.lb ci.ub
## factor(typeOfExposure)1 -0.4093 0.1759 -2.3271 0.0200 -0.7540 -0.0646 *
## factor(typeOfExposure)2 -0.4339 0.3031 -1.4313 0.1523 -1.0279 0.1602
## factor(typeOfExposure)3 -0.5594 0.2610 -2.1434 0.0321 -1.0709 -0.0479 *
## factor(typeOfExposure)4 -0.3072 0.2541 -1.2087 0.2268 -0.8052 0.1909
##
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
##
## $`RVE Wald test`
## test Fstat df_num df_denom p_val sig
## HTZ 0.503 3 4.23 0.699
Only for Being in nature, since there were 0 inconsistent means or SDs for Social support studies.
## $Nature
## $Nature$`Count of GRIM/GRIMMER inconsistencies`
##
## FALSE TRUE
## 15 10
##
## $Nature$`Model results`
## Coef Estimate SE d.f.
## 1 factor(as.logical(inconsistenciesCountGRIMMER))FALSE -0.557 0.157 Inf
## 2 factor(as.logical(inconsistenciesCountGRIMMER))TRUE -0.715 0.168 Inf
## Lower 95% CI Upper 95% CI
## 1 -0.865 -0.249
## 2 -1.044 -0.387
##
## $Nature$`RVE Wald test`
## test Fstat df_num df_denom p_val sig
## HTZ 0.94 1 2.18 0.427
## $Nature
## $Nature$RoB
##
## FALSE TRUE
## 42 12
##
## $Nature$`Model results`
## Coef Estimate SE d.f.
## 1 factor(robOverall > acceptableRiskOfBias)FALSE -0.367 0.120 Inf
## 2 factor(robOverall > acceptableRiskOfBias)TRUE -0.597 0.261 Inf
## Lower 95% CI Upper 95% CI
## 1 -0.602 -0.1322
## 2 -1.110 -0.0846
##
## $Nature$`RVE Wald test`
## test Fstat df_num df_denom p_val sig
## HTZ 0.638 1 3.34 0.477
##
##
## $Social
## $Social$RoB
##
## FALSE TRUE
## 1 17
##
## $Social$`Model results`
## Coef Estimate SE d.f.
## 1 factor(robOverall > acceptableRiskOfBias)FALSE 0.237 0.0000 Inf
## 2 factor(robOverall > acceptableRiskOfBias)TRUE -0.157 0.0486 Inf
## Lower 95% CI Upper 95% CI
## 1 0.237 0.237
## 2 -0.253 -0.062
##
## $Social$`RVE Wald test`
## test Fstat df_num df_denom p_val sig
## HTZ 65.7 1 10.8 <0.001 ***
## $`Model results`
## Coef Estimate SE d.f. Lower 95% CI Upper 95% CI
## 1 factor(strategy)1 -0.701 0.203 Inf -1.099 -0.3032
## 2 factor(strategy)2 -0.158 0.102 Inf -0.358 0.0406
##
## $`RVE Wald test`
## test Fstat df_num df_denom p_val sig
## HTZ 5.39 1 10.5 0.0415 *
Controlling for design-related factors that are prognostic w.r.t. the effect sizes (i.e., might vary across moderator categories)
## $`Model results`
## Coef Estimate SE d.f. Lower 95% CI Upper 95% CI
## 1 factor(strategy)1 -0.73444 1.944 Inf -4.545 3.076
## 2 factor(strategy)2 -1.16232 2.025 Inf -5.131 2.806
## 3 researchDesign 0.12006 0.317 Inf -0.500 0.741
## 4 populationType 0.00769 0.246 Inf -0.474 0.489
## 5 comparisonGroupType -0.37602 0.440 Inf -1.238 0.486
## 6 published -0.08972 0.349 Inf -0.773 0.594
## 7 robOverall 0.23530 0.317 Inf -0.386 0.856
##
## $`RVE Wald test`
## test Fstat df_num df_denom p_val sig
## HTZ 0.263 1 5.15 0.629
How many results were analyzed
## [1] 112
How many papers reported results in APA format
## [1] 8
How many statcheck errors
##
## FALSE TRUE
## 0.96428571 0.03571429
What proportion of statcheck errors affected the decision
## TRUE
## 0.5
How many papers contained statcheck errors
## [1] 0.5
The below reported meta-regressions are all implemented as a multivariate RVE-based models using the CHE working model (Pustejovsky & Tipton, 2020; https://osf.io/preprints/metaarxiv/vyfcj/). Testing of contrasts is carried out using a robust Wald-type test testing the equality of estimates across levels of the moderator.
## $Nature
## $Nature[[1]]
##
## 0 1
## 8 46
##
## $Nature$`Model results`
## Coef Estimate SE d.f. Lower 95% CI Upper 95% CI
## 1 factor(published)0 -0.0333 0.212 Inf -0.449 0.383
## 2 factor(published)1 -0.4796 0.114 Inf -0.702 -0.257
##
## $Nature$`RVE Wald test`
## test Fstat df_num df_denom p_val sig
## HTZ 3.44 1 1.32 0.266
##
##
## $Social
## $Social[[1]]
##
## 0 1
## 7 11
##
## $Social$`Model results`
## Coef Estimate SE d.f. Lower 95% CI Upper 95% CI
## 1 factor(published)0 -0.3150 0.0268 Inf -0.367 -0.2625
## 2 factor(published)1 -0.0887 0.0547 Inf -0.196 0.0185
##
## $Social$`RVE Wald test`
## test Fstat df_num df_denom p_val sig
## HTZ 13.8 1 3.33 0.0284 *
## $Nature
## $Nature[[1]]
##
## 1 2 3
## 9 4 41
##
## $Nature$`Model results`
## Coef Estimate SE d.f. Lower 95% CI
## 1 factor(researchDesign == 1)FALSE -0.46 0.1435 Inf -0.741
## 2 factor(researchDesign == 1)TRUE -0.31 0.0984 Inf -0.503
## Upper 95% CI
## 1 -0.178
## 2 -0.117
##
## $Nature$`RVE Wald test`
## test Fstat df_num df_denom p_val sig
## HTZ 0.74 1 6.76 0.419
##
##
## $Social
## $Social[[1]]
##
## 1 3 4
## 1 1 16
##
## $Social$`Model results`
## Coef Estimate SE d.f. Lower 95% CI
## 1 factor(researchDesign == 1)FALSE -0.157 0.0486 Inf -0.253
## 2 factor(researchDesign == 1)TRUE 0.237 0.0000 Inf 0.237
## Upper 95% CI
## 1 -0.062
## 2 0.237
##
## $Social$`RVE Wald test`
## test Fstat df_num df_denom p_val sig
## HTZ 65.7 1 10.8 <0.001 ***
## $Nature
## $Nature$`Number of included effects per category`
##
## 1 2
## 48 6
##
## $Nature$`Model results`
## Coef Estimate SE d.f. Lower 95% CI Upper 95% CI
## 1 as.factor(stressAffective)1 -0.421 0.109 Inf -0.634 -0.2078
## 2 as.factor(stressAffective)2 -0.429 0.169 Inf -0.760 -0.0969
##
## $Nature$`RVE Wald test`
## test Fstat df_num df_denom p_val sig
## HTZ 0.00305 1 5.05 0.958
##
##
## $Social
## $Social$`Number of included effects per category`
##
## 1 2
## 14 4
##
## $Social$`Model results`
## Coef Estimate SE d.f. Lower 95% CI Upper 95% CI
## 1 as.factor(stressAffective)1 -0.109 0.0531 Inf -0.214 -0.00534
## 2 as.factor(stressAffective)2 -0.262 0.0712 Inf -0.402 -0.12254
##
## $Social$`RVE Wald test`
## test Fstat df_num df_denom p_val sig
## HTZ 4.3 1 2.65 0.142
## $Nature
## $Nature$`Number of included effects per category`
##
## 1 3
## 28 20
##
## $Nature$`Model results`
## Coef Estimate SE d.f. Lower 95% CI Upper 95% CI
## 1 as.factor(stressCompRecoded)1 -0.488 0.156 Inf -0.794 -0.1821
## 2 as.factor(stressCompRecoded)3 -0.306 0.149 Inf -0.599 -0.0138
##
## $Nature$`RVE Wald test`
## test Fstat df_num df_denom p_val sig
## HTZ 0.609 1 5.43 0.468
##
##
## $Social
## $Social$`Number of included effects per category`
##
## 1 3
## 13 1
##
## $Social$`Model results`
## Coef Estimate SE d.f. Lower 95% CI Upper 95% CI
## 1 as.factor(stressCompRecoded)1 -0.1184 0.0575 Inf -0.2311 -0.00567
## 2 as.factor(stressCompRecoded)3 0.0527 0.0000 Inf 0.0527 0.05270
##
## $Social$`RVE Wald test`
## test Fstat df_num df_denom p_val sig
## HTZ 8.85 1 9.63 0.0145 *
Social support