Reproducibility Report for Mechanisms of Meditation: Investigating the Components and Covariates of a Single Session of Meditation by Taylor E. Bondi (2021, Doctoral Dissertation)
Author
Quan Tang (qutang@ucsd.edu)
Published
December 7, 2025
Introduction
This dissertation, titled “The Mechanism of Meditation: A Study on the Components and Covariates of a Single Meditation Session”, mainly explores whether a single brief meditation practice can enhance well-being (operationalized as reduction of state stress and anxiety), and further examines the underlying mechanisms (expectations and thought valence) and situational factors. The author designed four experiments, comparing meditation condition with control condition (using ANCOVA), different components of the meditation, testing placebo and labeling effects, and conducting exploratory analysis at the end. Her research provides clear and rigorous analysis on how meditation reduce state stress and anxiety, and what role thought valence plays in this process. Her result suggests that meditation is good at dealing with negative thoughts, which can contribute to well-being. The most interesting thing is she failed to replicate the result by herself in the second experiment. The conclusion is that even a single-session meditation can bring about a considerable improvement in well-being. Moreover, expectations and thought valence play an important role in this process. The research results of the first chapter show that a 20-minute mindfulness meditation can significantly reduce state stress and state anxiety, with better effects than the control group. At the same time, meditation is more effective on dealing with negative thought than relaxation exercises.
** Apart from all the preparatory analyses, what I aim to reproduce is the outcome related to the meditation mechanism. That are, the interaction effects between thought valence and conditions on state stress and anxiety reduction. **
During the initial stage of examining the effects of meditation, I will incorporate three 2 (condition: meditation vs. control, between-subjects) × 2 (time: pre-test vs. post-test, within-subjects) two-way mixed ANOVAs with state stress, state anxiety, and trait anxiety as dependent variables, respectively. The purpose is to prevent the loss of information from compositing the pre-test and post-test data.
In addition, apart from using Analysis of Covariance (ANCOVA), I also plan to attempt using linear mixed model (LMM) to separately incorporate the pre-test and post-test (instead of compositing them into variable of difference) to examine the effect of mindfulness meditation, as well as the role of thought valence within this process.
Links
Project Repository (on Github): https://github.com/qutang123/bondi2021.git
Original paper (as hosted in your repo): https://github.com/qutang123/bondi2021/blob/7e24871f1073cf9826759e85d40d99d2692410ac/original_paper/original_paper.pdf
Link of Final Project Proposal: https://github.com/qutang123/bondi2021/blob/aaaf443a7b4cd80e19587de0a8e3772621f149b3/writeup/Reproducibility%20Report.qmd
Description of the Steps Required to Reproduce the Results
Data Generation
The data will be generated based on R, using the rtmvnorm function from R package tmvtnorm. This function generates random numbers from the truncated multivariate normal distribution with mean, and covariance matrix sigma (calculated by SD and correlation), lower and upper truncation points with rejection sampling. The features of the data will be obtained from the original data, including ranges, means, standard deviations and correlation coefficients provided in the original article and data by the author. All the variables are assumed to follow approximately normal distribution. The variables simulated will include the composite scores of:
state stress (both pre- and post-test, continuous variable, normal distribution);
state anxiety (both pre- and post-test, continuous variable, normal distribution);
trait anxiety (both pre- and post-test, continuous variable, normal distribution);
thought valence (post-test only, continuous variable, normal distribution);
conditions (categorical variable, binomial distribution), based on the means, SDs, ranges, and correlations at participant-level.
A more detailed analysis plan is as follows:
1. Using the original dataset provided by the author, the ranges, means, standard deviations and zero-order correlation coefficients (pre-test and post-test correlation coefficients of the dependent variable were included) were calculated for both the meditation condition and the control condition.
2. Generate the data for the meditation condition and the control condition separately.
3. The generated data for each variable is a composite data at the variable level. In other words, I will not generate the item data used for calculating each variable (e.g., the scores of each item in a scale).
Regarding the preparation of data, there will be some differences compared to the original literature. In the original article, the cleaning and elimination of data were mainly based on whether the participants completed the tasks, whether they passed the attention check, whether they passed the subjective engagement check, and whether they passed the subjective validity test. Since this is a simulation study, I will not deliberately create a dataset with missing values. Attention checks, engagement levels, and subjective validity were not included in the analysis either, so these data will not be generated. The different criterion for evaluating the validity of data generation in this simulate research is whether all the data fall within the range of values of the original data. If the data of any variable fails to meet this standard, the entire dataset will be regenerated without elimination, to ensure that the sample size is consistent with the original literature.
Data Analysis
The data analysis will rely on R. The full pipeline is:
1. Data preparation for reduction of state stress, state anxiety, and trait anxiety (post-test score minus the pre-test score, then normalizing the reduction scores;
2. Descriptive statistics (visualized) for all variables by conditions;
3. Zero-order correlation analysis for all variables;
4. Three one-way ANOVAs (for effects of meditation) with conditions (meditation vs. control, between-subjects) as independent variable, the reduction of state stress, state anxiety, and trait anxiety as dependent variables, respectively;
5. Three two-way mixed ANOVAs (for effects of meditation) with conditions (meditation vs. control, between-subjects) and test points (pre-test vs. post-test, within-subjects) as independent variables, the reduction of state stress, state anxiety, and trait anxiety as dependent variables, respectively;
6. Three ANCOVAs (for the role of thought valence in meditation) with conditions and thought valence as independent variables, the reduction of state stress, state anxiety, and trait anxiety as dependent variables respectively, mainly focusing on the interactions between conditions and thought valence;
7. Three LMMs (Linear Mixed Model, also called Multilevel Model; for the role of thought valence in meditation with the data of both pre- and post-test) with condition, thought valence, and time as independent variables, state stress, state anxiety, and trait anxiety as dependent variables respectively, and subject ID as random effect, without random slopes, mainly focusing on the three-way interactions between conditions, thought valence, and time;
8. Independent sample t-tests on all variables except for condition to compare the difference of the original data and the generated data.
The 6th steps is the core analysis of reproduction and the 5th & 7th step are the core exploration of this project.
Differences from Original Study
Since a normal distribution was assumed, this may differ from the original study.
There may be some differences between the generated data and the actual data.
The version of R used for the original data analysis is different from those currently in use.
Note. The additional ANOVA and LMM Analysis does not involve data generation and does not affect the analysis results.
If the significance of the interaction of LMM do not match those of ANCOVA, there are three possible explanations. First, the individual differences among the participants may have led to significant unexpected interactions. LMM estimates individual differences using random effects, while ANCOVA treats them as errors. Secondly, the spherical assumption is not valid, resulting in an underestimation of the standard error. ANCOVA assumes that the residuals at each time point are independent and have equal variances, while LMM allows for correlations between time points. Third, The operation of subtracting the pre-test score from the post-test score reduces the amount of error information, resulting in a significant effect.
Reliability and Validity
In my simulation study, the key construct are mindfulness, state stress, state anxiety, trait anxiety and thought valence from the original literature. The mindfulness was manipulated by meditation, while other variables were measured by self-reported survey. The author of the original paper used State-Trait Anxiety Inventory (STAI; Spielberger et al., 1983) to measure the state anxiety and trait anxiety, which has been used in previous studies, and has passed numerous reliability and validity tests. For state stress, she used a 3-items self-developed questionnaire. For two measurements above, I expected to see the Cronbach’s alpha, but she didn’t report it. What’s more, since the measurement of state stress was newly created by the original author, I believe it would be more appropriate if such a report of reliability and validity were included in the thesis. However, they were not mentioned. For thought valence, she use a single-item scale to measure it, and there aren’t any report about its reliability. But she use another question to test the confidence of the answer from the participants for validity test, and I think it is a kind of reliability, which makes the measurement errors smaller.
Project Progress Check 1
Measure of Success
Please describe the outcome measure for the success or failure of your reproduction and how this outcome will be computed.
Success criterion is when I input the original data, I should obtain the same results as the original article (reproduction success), and when I input the simulated data, the significance should consistent with that of the original article (simulation success).
For the 2 core results of ANCOVAs (for the reduction of state stress and state anxiety), I should get significant interaction between condition and thought valence. More specifically, for the control condition, the more positive the thought valence, the greater the reduction in state stress and state anxiety. For the meditation condition, there should be no significant difference in the reduction of state stress and state anxiety across different levels of thought valence. From a theoretical perspective, this indicates that mindfulness can block the effect of thought valence on state stress and state anxiety.
The key indicators for comparing simulation and reproduction are the significance of the condition × thought valence interactions in ANCOVAs. Meanwhile, the original literature also reported the condition × thought valence interaction effect sizes in partial eta square and their 95% confidence intervals. Therefore, whether the confidence interval of the simulation effect size overlaps with the confidence interval reported in the original literature is also one of the references.
Pipeline Progress
Earlier in this report, you described the steps necessary to reproduce the key result(s) of this study. Please describe your progress on each of these steps (e.g., data preprocessing, model fitting, model evaluation).
For the descriptive statistics and correlation analysis, which are employed for the data simulation, I hope the results I obtain can be as close as possible to the original literature. Therefore, generating the data might require several attempts.
For ANCOVAs and LMMs, I hope they can detect the conditions × thought valence interactions on the reduction of state stress and state anxiety but not on trait anxiety, and conditions × thought valence × time interactions on state stress and state anxiety but not on trait anxiety. In fact, all of this also depends on the aforementioned data generation.
As of now, all data generation, data analysis, and exploratory analysis have been completed. The code used for data analysis completely reproduces the results presented in the article using the original data. In the generated data, both of the two interaction effects of the target mentioned above were significant, and their effect sizes were similar to those reported in the original literature.
Results
Power Analysis
I conducted a post hoc power analysis on the original data for both state stress and state anxiety. Trait anxiety was not included in post hoc power analysis, as its interaction was not significant in the original study.
library(pwr)# ---- Sample ----k <-2# number of level of IVcov_num <-1# number of CoVn <-207# sample sizealpha <-0.05# significant level# ---- Effect Size ----eta2_SS =0.04eta2_SA =0.02f2_SS = eta2_SS / (1- eta2_SS)f2_SA = eta2_SA / (1- eta2_SA)# ---- df ----u <- k -1# numerator df (effect)v <- n - k - cov_num -1# denominator df (error)# ---- Power Analysis ----pwr_result_SS <-pwr.f2.test(u = u, v = v, f2 = f2_SS, sig.level = alpha)pwr_result_SA <-pwr.f2.test(u = u, v = v, f2 = f2_SA, sig.level = alpha)pwr_result_SS
Multiple regression power calculation
u = 1
v = 203
f2 = 0.04166667
sig.level = 0.05
power = 0.8286382
pwr_result_SA
Multiple regression power calculation
u = 1
v = 203
f2 = 0.02040816
sig.level = 0.05
power = 0.5302261
The results found that the original study had good statistical power for ANCOVA on state stress, power = 0.8286382, but not had a enough statistical power for ANCOVA on state anxiety, power = 0.5302261. The results also shows that the ANCOVAs on state stress and anxiety in simulated data also had good statistical power, whose power = 0.8286382 for both state stress and anxiety.
Data Preparation
Data preparation following the analysis plan. Prepare all the R packages.
# Library loadinglibrary(tidyverse)
Warning: package 'ggplot2' was built under R version 4.5.2
── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ──
✔ dplyr 1.1.4 ✔ readr 2.1.5
✔ forcats 1.0.1 ✔ stringr 1.5.2
✔ ggplot2 4.0.1 ✔ tibble 3.3.0
✔ lubridate 1.9.4 ✔ tidyr 1.3.1
✔ purrr 1.1.0
── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
✖ dplyr::filter() masks stats::filter()
✖ dplyr::lag() masks stats::lag()
ℹ Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errors
Loading required package: carData
Attaching package: 'car'
The following object is masked from 'package:DescTools':
Recode
The following object is masked from 'package:dplyr':
recode
The following object is masked from 'package:purrr':
some
Loading required package: mvtnorm
Attaching package: 'mvtnorm'
The following object is masked from 'package:effectsize':
standardize
Loading required package: Matrix
Attaching package: 'Matrix'
The following objects are masked from 'package:tidyr':
expand, pack, unpack
Loading required package: stats4
Loading required package: gmm
Loading required package: sandwich
library(psych)
Attaching package: 'psych'
The following object is masked from 'package:car':
logit
The following objects are masked from 'package:DescTools':
AUC, ICC, SD
The following object is masked from 'package:effectsize':
phi
The following objects are masked from 'package:ggplot2':
%+%, alpha
library(afex)
Loading required package: lme4
Warning: package 'lme4' was built under R version 4.5.2
Attaching package: 'lme4'
The following object is masked from 'package:gmm':
checkConv
************
Welcome to afex. For support visit: http://afex.singmann.science/
- Functions for ANOVAs: aov_car(), aov_ez(), and aov_4()
- Methods for calculating p-values with mixed(): 'S', 'KR', 'LRT', and 'PB'
- 'afex_aov' and 'mixed' objects can be passed to emmeans() for follow-up tests
- Get and set global package options with: afex_options()
- Set sum-to-zero contrasts globally: set_sum_contrasts()
- For example analyses see: browseVignettes("afex")
************
Attaching package: 'afex'
The following object is masked from 'package:lme4':
lmer
library(lme4)library(lmerTest)
Attaching package: 'lmerTest'
The following object is masked from 'package:lme4':
lmer
The following object is masked from 'package:stats':
step
library(emmeans)
Welcome to emmeans.
Caution: You lose important information if you filter this package's results.
See '? untidy'
library(effects)
lattice theme set by effectsTheme()
See ?effectsTheme for details.
library(boot)
Attaching package: 'boot'
The following object is masked from 'package:psych':
logit
The following object is masked from 'package:car':
logit
library(sjstats)
Attaching package: 'sjstats'
The following object is masked from 'package:psych':
phi
The following objects are masked from 'package:effectsize':
cohens_f, cramers_v, phi
Read the original data. I will not disclose the original data, so please refer to the .html file for the result of this step.
# Loading dataRAW.DATA =read.csv('OriginalData.csv')# Data preparation# Raw sample: n = 525DATA =subset(RAW.DATA, Finished==1&# excluded: 12 Attention.Checks >=6&# excluded: 26 Engagement >=10&# excluded: 48 Validity.survey ==1&# excluded: 54 Validity.audio ==1) # excluded: 178# Final sample: n = 207#### Descriptive stats####table(DATA$Gender)
Female Male
168 39
table(DATA$Race)
Asian Black or African American
106 5
Hispanic or Latino Middle Eastern or North African
40 3
Mixed Prefer not to say
14 4
White/Caucasian
35
table(DATA$Year)
5th or more Freshman Junior Senior Sophomore
4 23 95 58 27
summary(DATA$Age)
Min. 1st Qu. Median Mean 3rd Qu. Max.
18.00 20.00 20.00 20.71 21.00 39.00
sd(DATA$Age)
[1] 2.683123
At this stage, I carried out the most basic cleaning of the original data and saved it in a.csv file. Once again, due to copyright concerns, we will not disclose the original data.
DATA$expect.m[is.na(DATA$expect.m)] =0DATA$expect.c[is.na(DATA$expect.c)] =0DATA$Expectation = DATA$expect.c + DATA$expect.m # I want not normalized data first# So I will normalize it afterDATA$PRE.SS = DATA$pre.stress + (11-DATA$pre.relax) + (2*(6-DATA$pre.feel))DATA$POST.SS = DATA$post.stress + (11-DATA$post.relax) + (2*(6-DATA$post.feel))## Standardized Diff Scores (indicate % change pre to post due to intervention)#State StressDATA$SS = (DATA$POST.SS - DATA$PRE.SS) /30#State AnxietyDATA$SA = (DATA$POST.STAI.S - DATA$PRE.STAI.S) /80#Trait AnxietyDATA$TA = (DATA$POST.STAI.T - DATA$PRE.STAI.T) /80## export datafileFDATA =data.frame(Con = DATA$INT,Tho = DATA$Thoughts,PRE.SA = DATA$PRE.STAI.S,PRE.TA = DATA$PRE.STAI.T,PRE.SS = DATA$PRE.SS,POS.SA = DATA$POST.STAI.S,POS.TA = DATA$POST.STAI.T,POS.SS = DATA$POST.SS)write.table(FDATA, file="FinalData.csv",sep=",",row.names=F)
Descriptive Statistics for Original Data
In this step, I calculated the descriptive statistics (mean, SD, range, skewness, and kurtosis) of the original data and visualize them using histograms and violin plots. The following variables are in sequence: thought valence, pre-test state anxiety, pre-test trait anxiety, pre-test state stress, post-test state anxiety, post-test trait anxiety, and post-test state stress.
ggplot(FDATA, aes(x = Con, y = Tho, fill = Con)) +geom_violin(trim =FALSE, alpha =0.2, color ="black") +geom_boxplot(width =0.1, outlier.shape =NA, alpha =0.3) +geom_jitter(aes(color = Con),width =0.15, height =0, alpha =0.3, size =1.8) +labs(title ="Distribution of Thought Valence by Conditions",x ="Condition",y ="Thought Valence" ) +theme_minimal() +theme(plot.title =element_text(hjust =0.5) )
ggplot(FDATA, aes(x = PRE.SA, fill = Con)) +geom_histogram(position ="identity", alpha =0.5, bins =20, color ="black") +labs(title ="Distribution of Score by Conditions",x ="State Anxiety in Pretest",y ="Count" ) +theme_minimal() +theme(plot.title =element_text(hjust =0.5) )
ggplot(FDATA, aes(x = Con, y = PRE.SA, fill = Con)) +geom_violin(trim =FALSE, alpha =0.2, color ="black") +geom_boxplot(width =0.1, outlier.shape =NA, alpha =0.3) +geom_jitter(aes(color = Con),width =0.15, height =0, alpha =0.3, size =1.8) +labs(title ="Distribution of State Anxiety in Pretest by Conditions",x ="Condition",y ="State Anxiety in Pretest" ) +theme_minimal() +theme(plot.title =element_text(hjust =0.5) )
ggplot(FDATA, aes(x = PRE.TA, fill = Con)) +geom_histogram(position ="identity", alpha =0.5, bins =20, color ="black") +labs(title ="Distribution of Score by Conditions",x ="Trait Anxiety in Pretest",y ="Count" ) +theme_minimal() +theme(plot.title =element_text(hjust =0.5) )
ggplot(FDATA, aes(x = Con, y = PRE.TA, fill = Con)) +geom_violin(trim =FALSE, alpha =0.2, color ="black") +geom_boxplot(width =0.1, outlier.shape =NA, alpha =0.3) +geom_jitter(aes(color = Con),width =0.15, height =0, alpha =0.3, size =1.8) +labs(title ="Distribution of Trait Anxiety in Pretest by Conditions",x ="Condition",y ="Trait Anxiety in Pretest" ) +theme_minimal() +theme(plot.title =element_text(hjust =0.5) )
ggplot(FDATA, aes(x = PRE.SS, fill = Con)) +geom_histogram(position ="identity", alpha =0.5, bins =20, color ="black") +labs(title ="Distribution of Score by Conditions",x ="State Stress in Pretest",y ="Count" ) +theme_minimal() +theme(plot.title =element_text(hjust =0.5) )
ggplot(FDATA, aes(x = Con, y = PRE.SS, fill = Con)) +geom_violin(trim =FALSE, alpha =0.2, color ="black") +geom_boxplot(width =0.1, outlier.shape =NA, alpha =0.3) +geom_jitter(aes(color = Con),width =0.15, height =0, alpha =0.3, size =1.8) +labs(title ="Distribution of State Stress in Pretest by Conditions",x ="Condition",y ="State Stress in Pretest" ) +theme_minimal() +theme(plot.title =element_text(hjust =0.5) )
ggplot(FDATA, aes(x = POS.SA, fill = Con)) +geom_histogram(position ="identity", alpha =0.5, bins =20, color ="black") +labs(title ="Distribution of Score by Conditions",x ="State Anxiety in Post Test",y ="Count" ) +theme_minimal() +theme(plot.title =element_text(hjust =0.5) )
ggplot(FDATA, aes(x = Con, y = POS.SA, fill = Con)) +geom_violin(trim =FALSE, alpha =0.2, color ="black") +geom_boxplot(width =0.1, outlier.shape =NA, alpha =0.3) +geom_jitter(aes(color = Con),width =0.15, height =0, alpha =0.3, size =1.8) +labs(title ="Distribution of State Anxiety in Post Test by Conditions",x ="Condition",y ="State Anxiety in Post Test" ) +theme_minimal() +theme(plot.title =element_text(hjust =0.5) )
ggplot(FDATA, aes(x = POS.TA, fill = Con)) +geom_histogram(position ="identity", alpha =0.5, bins =20, color ="black") +labs(title ="Distribution of Score by Conditions",x ="Trait Anxiety in Post Test",y ="Count" ) +theme_minimal() +theme(plot.title =element_text(hjust =0.5) )
ggplot(FDATA, aes(x = Con, y = POS.TA, fill = Con)) +geom_violin(trim =FALSE, alpha =0.2, color ="black") +geom_boxplot(width =0.1, outlier.shape =NA, alpha =0.3) +geom_jitter(aes(color = Con),width =0.15, height =0, alpha =0.3, size =1.8) +labs(title ="Distribution of Trait Anxiety in Post Test by Conditions",x ="Condition",y ="Trait Anxiety in Post Test" ) +theme_minimal() +theme(plot.title =element_text(hjust =0.5) )
ggplot(FDATA, aes(x = POS.SS, fill = Con)) +geom_histogram(position ="identity", alpha =0.5, bins =20, color ="black") +labs(title ="Distribution of Score by Conditions",x ="State Stress in Post Test",y ="Count" ) +theme_minimal() +theme(plot.title =element_text(hjust =0.5) )
ggplot(FDATA, aes(x = Con, y = POS.SS, fill = Con)) +geom_violin(trim =FALSE, alpha =0.2, color ="black") +geom_boxplot(width =0.1, outlier.shape =NA, alpha =0.3) +geom_jitter(aes(color = Con),width =0.15, height =0, alpha =0.3, size =1.8) +labs(title ="Distribution of State Stress in Post Test by Conditions",x ="Condition",y ="State Stress in Post Test" ) +theme_minimal() +theme(plot.title =element_text(hjust =0.5) )
Here, I observe that almost all variables approximately follow a normal distribution, except for the post-test state anxiety. I believe that this might be because the effect of mindfulness meditation causes the distribution of state anxiety to exhibit a floor effect.
Correlation Analyses
Next, I calculated the correlation matrices for all seven variables in both the meditation condition and the control condition separately.
In this section, I have stored the means, standard deviations, ranges and correlation coefficients in their respective variables, ready to proceed with data generation. The correlation matrices were transformed into a covariance matrix here.
At this step, I generate data for the meditation condition and the control condition respectively, based on the already calculated means, standard deviations, ranges (i.e., maximum and minimum values), and correlation coefficients (which have been converted into covariances). The “rtmvnorm” function generates random numbers from the truncated multivariate normal distribution with mean and covariance matrices sigma (calculated by SDs and correlations), lower and upper truncation points with rejection sampling.
ggplot(sim_data, aes(x = Con, y = Tho, fill = Con)) +geom_violin(trim =FALSE, alpha =0.2, color ="black") +geom_boxplot(width =0.1, outlier.shape =NA, alpha =0.3) +geom_jitter(aes(color = Con),width =0.15, height =0, alpha =0.3, size =1.8) +labs(title ="Distribution of Thought Valence by Conditions",x ="Condition",y ="Thought Valence" ) +theme_minimal() +theme(plot.title =element_text(hjust =0.5) )
ggplot(sim_data, aes(x = PRE.SA, fill = Con)) +geom_histogram(position ="identity", alpha =0.5, bins =20, color ="black") +labs(title ="Distribution of Score by Conditions",x ="State Anxiety in Pretest",y ="Count" ) +theme_minimal() +theme(plot.title =element_text(hjust =0.5) )
ggplot(sim_data, aes(x = Con, y = PRE.SA, fill = Con)) +geom_violin(trim =FALSE, alpha =0.2, color ="black") +geom_boxplot(width =0.1, outlier.shape =NA, alpha =0.3) +geom_jitter(aes(color = Con),width =0.15, height =0, alpha =0.3, size =1.8) +labs(title ="Distribution of State Anxiety in Pretest by Conditions",x ="Condition",y ="State Anxiety in Pretest" ) +theme_minimal() +theme(plot.title =element_text(hjust =0.5) )
ggplot(sim_data, aes(x = PRE.TA, fill = Con)) +geom_histogram(position ="identity", alpha =0.5, bins =20, color ="black") +labs(title ="Distribution of Score by Conditions",x ="Trait Anxiety in Pretest",y ="Count" ) +theme_minimal() +theme(plot.title =element_text(hjust =0.5) )
ggplot(sim_data, aes(x = Con, y = PRE.TA, fill = Con)) +geom_violin(trim =FALSE, alpha =0.2, color ="black") +geom_boxplot(width =0.1, outlier.shape =NA, alpha =0.3) +geom_jitter(aes(color = Con),width =0.15, height =0, alpha =0.3, size =1.8) +labs(title ="Distribution of Trait Anxiety in Pretest by Conditions",x ="Condition",y ="Trait Anxiety in Pretest" ) +theme_minimal() +theme(plot.title =element_text(hjust =0.5) )
ggplot(sim_data, aes(x = PRE.SS, fill = Con)) +geom_histogram(position ="identity", alpha =0.5, bins =20, color ="black") +labs(title ="Distribution of Score by Conditions",x ="State Stress in Pretest",y ="Count" ) +theme_minimal() +theme(plot.title =element_text(hjust =0.5) )
ggplot(sim_data, aes(x = Con, y = PRE.SS, fill = Con)) +geom_violin(trim =FALSE, alpha =0.2, color ="black") +geom_boxplot(width =0.1, outlier.shape =NA, alpha =0.3) +geom_jitter(aes(color = Con),width =0.15, height =0, alpha =0.3, size =1.8) +labs(title ="Distribution of State Stress in Pretest by Conditions",x ="Condition",y ="State Stress in Pretest" ) +theme_minimal() +theme(plot.title =element_text(hjust =0.5) )
ggplot(sim_data, aes(x = POS.SA, fill = Con)) +geom_histogram(position ="identity", alpha =0.5, bins =20, color ="black") +labs(title ="Distribution of Score by Conditions",x ="State Anxiety in Post Test",y ="Count" ) +theme_minimal() +theme(plot.title =element_text(hjust =0.5) )
ggplot(sim_data, aes(x = Con, y = POS.SA, fill = Con)) +geom_violin(trim =FALSE, alpha =0.2, color ="black") +geom_boxplot(width =0.1, outlier.shape =NA, alpha =0.3) +geom_jitter(aes(color = Con),width =0.15, height =0, alpha =0.3, size =1.8) +labs(title ="Distribution of State Anxiety in Post Test by Conditions",x ="Condition",y ="State Anxiety in Post Test" ) +theme_minimal() +theme(plot.title =element_text(hjust =0.5) )
ggplot(sim_data, aes(x = POS.TA, fill = Con)) +geom_histogram(position ="identity", alpha =0.5, bins =20, color ="black") +labs(title ="Distribution of Score by Conditions",x ="Trait Anxiety in Post Test",y ="Count" ) +theme_minimal() +theme(plot.title =element_text(hjust =0.5) )
ggplot(sim_data, aes(x = Con, y = POS.TA, fill = Con)) +geom_violin(trim =FALSE, alpha =0.2, color ="black") +geom_boxplot(width =0.1, outlier.shape =NA, alpha =0.3) +geom_jitter(aes(color = Con),width =0.15, height =0, alpha =0.3, size =1.8) +labs(title ="Distribution of Trait Anxiety in Post Test by Conditions",x ="Condition",y ="Trait Anxiety in Post Test" ) +theme_minimal() +theme(plot.title =element_text(hjust =0.5) )
ggplot(sim_data, aes(x = POS.SS, fill = Con)) +geom_histogram(position ="identity", alpha =0.5, bins =20, color ="black") +labs(title ="Distribution of Score by Conditions",x ="State Stress in Post Test",y ="Count" ) +theme_minimal() +theme(plot.title =element_text(hjust =0.5) )
ggplot(sim_data, aes(x = Con, y = POS.SS, fill = Con)) +geom_violin(trim =FALSE, alpha =0.2, color ="black") +geom_boxplot(width =0.1, outlier.shape =NA, alpha =0.3) +geom_jitter(aes(color = Con),width =0.15, height =0, alpha =0.3, size =1.8) +labs(title ="Distribution of State Stress in Post Test by Conditions",x ="Condition",y ="State Stress in Post Test" ) +theme_minimal() +theme(plot.title =element_text(hjust =0.5) )
Pre-Analysis Data Preparation for Simulated Data
In this section, I calculated the reduction of state stress, state anxiety, and trait anxiety, using simulated data (pre-test minus post-test), and then normalized them for ANCOVAs.
# data preparation## normalization# State Stresssim_data$SS = (sim_data$POS.SS - sim_data$PRE.SS) /30# State Anxietysim_data$SA = (sim_data$POS.SA - sim_data$PRE.SA) /80# Trait Anxietysim_data$TA = (sim_data$POS.TA - sim_data$PRE.TA) /80
Descriptive Statistics for Composited Simulated Data
Here I calculated the descriptive statistics including means, SDs, SEs, ranges, skewnesses, and kurtoses for composited simulated data.
# A tibble: 2 × 8
Con min max mean sd se skewness kurtosis
<chr> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
1 CON -0.433 0.1 -0.143 0.116 0.0121 -0.280 2.68
2 MED -0.6 0.133 -0.227 0.148 0.0138 0.00258 2.96
S_SA
# A tibble: 2 × 8
Con min max mean sd se skewness kurtosis
<chr> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
1 CON -0.5 0.275 -0.0851 0.143 0.0149 -0.0672 2.96
2 MED -0.55 0.175 -0.160 0.133 0.0124 -0.255 3.35
S_TA
# A tibble: 2 × 8
Con min max mean sd se skewness kurtosis
<chr> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
1 CON -0.162 0.075 -0.0480 0.0466 0.00486 -0.168 2.84
2 MED -0.2 0.0875 -0.0604 0.0635 0.00592 0.304 2.70
These 3 one-way ANOVAs were used to compare the difference of the reduction of state stress, state anxiety and trait anxiety between meditation condition and control condition using the simulated data.
# one-way ANOVAoptions(contrasts =c('contr.sum', 'contr.poly'))ano.ss =lm(SS ~ Con, data = sim_data)ano.sa =lm(SA ~ Con, data = sim_data)ano.ta =lm(TA ~ Con, data = sim_data)anova(ano.ss)
Analysis of Variance Table
Response: SS
Df Sum Sq Mean Sq F value Pr(>F)
Con 1 0.3599 0.35989 19.946 1.315e-05 ***
Residuals 205 3.6989 0.01804
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
anova(ano.sa)
Analysis of Variance Table
Response: SA
Df Sum Sq Mean Sq F value Pr(>F)
Con 1 0.2846 0.284591 15.062 0.0001403 ***
Residuals 205 3.8735 0.018895
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
anova(ano.ta)
Analysis of Variance Table
Response: TA
Df Sum Sq Mean Sq F value Pr(>F)
Con 1 0.00795 0.0079514 2.4789 0.1169
Residuals 205 0.65756 0.0032076
cohens_d(SS ~ Con, data = sim_data)
Cohen's d | 95% CI
------------------------
0.62 | [0.34, 0.90]
- Estimated using pooled SD.
cohens_d(SA ~ Con, data = sim_data)
Cohen's d | 95% CI
------------------------
0.54 | [0.26, 0.82]
- Estimated using pooled SD.
cohens_d(TA ~ Con, data = sim_data)
Cohen's d | 95% CI
-------------------------
0.22 | [-0.06, 0.49]
- Estimated using pooled SD.
The results showed that there were significant differences of the reduction of state stress (F(1,205) = 19.946, p < 0.001, Cohen’s d = 0.62, 95% CI = [0.34, 0.90]) and anxiety (F(1,205) = 15.062, p < 0.001, Cohen’s d = 0.54, 95% CI = [0.26, 0.82]) between two conditions, which indicate that compared with the control condition, participants in meditation condition experienced more state stress and anxiety reduction. Such difference cannot be found regarding the reduction of trait anxiety (F(1,205) = 2.4789, p = 0.1169, Cohen’s d = 0.22, 95% CI = [-0.06, 0.49]). These findings in simulated data are partially aligning with the original study.
Core of the Reproduction: The ANCOVAs for Simulated Data
In this section, following the original study, I conducted three type III ANCOVAs respectively on the reduction of state stress, state anxiety, and trait anxiety as the dependent variables, with condition and thought valence as the independent variables for simulated data.
# ANCOVA for simulated dataR_SS <-lm(SS ~ Con * Tho, data = sim_data)R_SA <-lm(SA ~ Con * Tho, data = sim_data)R_TA <-lm(TA ~ Con * Tho, data = sim_data)Anova(R_SS, type='III')
Anova Table (Type III tests)
Response: SS
Sum Sq Df F value Pr(>F)
(Intercept) 0.0828 1 4.7835 0.0298759 *
Con 0.2264 1 13.0758 0.0003774 ***
Tho 0.0767 1 4.4269 0.0366075 *
Con:Tho 0.1331 1 7.6880 0.0060766 **
Residuals 3.5150 203
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
ANOVA results using SS as the dependent variable
Predictor SS df MS F p partial_eta2 CI_95_partial_eta2
(Intercept) 0.08 1 0.08 4.78 .030
Con 0.23 1 0.23 13.08 .000 .06 [.01, .13]
Tho 0.08 1 0.08 4.43 .037 .02 [.00, .07]
Con x Tho 0.13 1 0.13 7.69 .006 .04 [.00, .10]
Error 3.51 203 0.02
Note: Values in square brackets indicate the bounds of the 95% confidence interval for partial eta-squared
apa.aov.table(R_SA, conf.level = .95, type='III')
ANOVA results using SA as the dependent variable
Predictor SS df MS F p partial_eta2 CI_95_partial_eta2
(Intercept) 0.00 1 0.00 0.06 .813
Con 0.23 1 0.23 13.01 .000 .06 [.01, .13]
Tho 0.17 1 0.17 9.67 .002 .05 [.01, .11]
Con x Tho 0.15 1 0.15 8.45 .004 .04 [.00, .10]
Error 3.59 203 0.02
Note: Values in square brackets indicate the bounds of the 95% confidence interval for partial eta-squared
apa.aov.table(R_TA, conf.level = .95, type='III')
ANOVA results using TA as the dependent variable
Predictor SS df MS F p partial_eta2 CI_95_partial_eta2
(Intercept) 0.02 1 0.02 5.99 .015
Con 0.00 1 0.00 0.01 .928 .00 [.00, .01]
Tho 0.00 1 0.00 0.15 .696 .00 [.00, .02]
Con x Tho 0.00 1 0.00 0.05 .818 .00 [.00, .02]
Error 0.66 203 0.00
Note: Values in square brackets indicate the bounds of the 95% confidence interval for partial eta-squared
For the reduction of state stress, I identified significant condition × thought valence interaction (F(1,203) = 7.69, p = 0.006, partial_eta2 = 0.04, 95% CI = [.00, .10]), main effects of condition (F(1,203) = 13.08, p < 0.001, partial_eta2 = 0.06, 95% CI = [.01, .13]) and thought valence (F(1,203) = 4.43, p = 0.037, partial_eta2 = 0.02, 95% CI = [.00, .07]). The significant interaction indicates that, compared to the control condition, the participants in the meditation condition still maintained a relatively high level of state stress reduction when their thoughts were more negative.
For the reduction of state anxiety, I found that there were significant condition × thought valence interaction (F(1,203) = 8.45, p = 0.004, partial_eta2 = 0.04, 95% CI = [.00, .10]), main effects of condition (F(1,203) = 13.01, p < 0.001, partial_eta2 = 0.06, 95% CI = [.01, .13]) and thought valence (F(1,203) = 9.67, p = 0.002, partial_eta2 = 0.05, 95% CI = [.01, .11]). The significant interaction indicates that, compared to the control condition, the participants in the meditation condition still maintained a relatively high level of state anxiety reduction when their thoughts were more negative.
For the reduction of trait anxiety, there was no significant main effects of condition (F(1,203) = 0.01, p = 0.928, partial_eta2 = 0.00, 95% CI = [.00, .01]) and thought valence (F(1,203) = 0.15, p = 0.696, partial_eta2 = 0.00, 95% CI = [.00, .02]), as well as interaction between condition and thought valence (F(1,203) = 0.05, p = 0.818, partial_eta2 = 0.00, 95% CI = [.00, .02]). The non-significant interaction effect indicates that, at different levels of thought valence, there were no significant difference in the reduction of trait anxiety among participants in the meditation and control conditions. This result is also reasonable, because a single-session meditation practice is indeed not sufficient to bring about a significant enough effect to change people’s anxiety traits.
Success Check (Reproduce Findings in Original Paper)
Independent Sample t-Tests
These independent sample t-tests were used to compare the differences between original data and simulated data for each variable.
The results show that there are significant differences between two datasets for state stress (t(217.949) = -2.327, p = 0.021, Cohen’s d = -0.307, 95% CI = [-0.567, -0.047]) and anxiety (t(220.868) = -2.775, p = 0.006, Cohen’s d = -0.366, 95% CI = [-0.626, -0.105]) in pre-test under meditation condition as well as thought valence (t(180.733) = 2.045, p = 0.042, Cohen’s d = 0.301, 95% CI = [0.010, 0.592]) under control condition.
Descriptive Statistics for Composited Original Data
This is the reproduction of the descriptive statistics of the original study using the composited and normalized variables from original data.
# A tibble: 2 × 8
Con min max mean sd se skewness kurtosis
<chr> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
1 CON -0.467 0.333 -0.140 0.156 0.0163 0.191 3.39
2 MED -0.633 0.333 -0.201 0.160 0.0149 -0.193 3.96
S_SA_F
# A tibble: 2 × 8
Con min max mean sd se skewness kurtosis
<chr> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
1 CON -0.412 0.075 -0.103 0.113 0.0117 -0.848 3.09
2 MED -0.55 0.212 -0.134 0.128 0.0120 -0.744 4.24
S_TA_F
# A tibble: 2 × 8
Con min max mean sd se skewness kurtosis
<chr> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
1 CON -0.325 0.025 -0.0409 0.0485 0.00506 -2.63 14.5
2 MED -0.425 0.05 -0.0452 0.0666 0.00621 -2.54 13.5
Covariance Homogeneity Test Based on the Bootstrap Method
I employed a covariance homogeneity test based on the bootstrap method with 5000-time sampling to examine whether there were differences between the covariance matrices of the original data and the simulated data. For each bootstrapping sample, I computed the covariance matrices and constructed 95% percentile confidence intervals for every covariance element. I then examined whether each corresponding covariance in the simulated dataset fell within these bootstrap intervals.
Across all covariances of each bootstrapping sample, 66.1% of the simulated covariances fell within the 95% bootstrap confidence intervals derived from the original data. This indicates that the simulated data replicated the empirical covariance structure moderately well, while still showing noticeable deviations in a subset of covariance elements. As a sanity check, 100% of the original covariance elements fell within their own bootstrap intervals, confirming that the bootstrap procedure behaved as expected (i.e., consistent with the nominal 95% coverage).
One-Way ANOVAs for Original Data
These are the reproduction of the one-way ANOVAs in original study which aimed to identify the differences of the reduction of state stress, state anxiety, and trait anxiety between meditation and control condition. The results from the original literature will not be reported again in this project.
# one-way ANOVAsano.ss_F =lm(SS ~ Con, data = FDATA)ano.sa_F =lm(SA ~ Con, data = FDATA)ano.ta_F =lm(TA ~ Con, data = FDATA)anova(ano.ss_F)
Analysis of Variance Table
Response: SS
Df Sum Sq Mean Sq F value Pr(>F)
Con 1 0.1939 0.193908 7.7216 0.005964 **
Residuals 205 5.1481 0.025113
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
anova(ano.sa_F)
Analysis of Variance Table
Response: SA
Df Sum Sq Mean Sq F value Pr(>F)
Con 1 0.04888 0.048877 3.3089 0.07037 .
Residuals 205 3.02817 0.014772
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
anova(ano.ta_F)
Analysis of Variance Table
Response: TA
Df Sum Sq Mean Sq F value Pr(>F)
Con 1 0.00095 0.0009541 0.2716 0.6028
Residuals 205 0.72021 0.0035132
cohens_d(SS ~ Con, data = FDATA)
Cohen's d | 95% CI
------------------------
0.39 | [0.11, 0.66]
- Estimated using pooled SD.
cohens_d(SA ~ Con, data = FDATA)
Cohen's d | 95% CI
-------------------------
0.25 | [-0.02, 0.53]
- Estimated using pooled SD.
cohens_d(TA ~ Con, data = FDATA)
Cohen's d | 95% CI
-------------------------
0.07 | [-0.20, 0.35]
- Estimated using pooled SD.
The results from simulated data were partially aligning with the results in original study. Here, the difference in original study was that I found significant different on reduction of state anxiety in simulated data, but the difference in the original data was moderately significant (p = 0.07037). What’s more, the effect sizes in the simulated data are higher than the original data.
Core of the Original Study: The ANCOVAs for Original Data
In this section, I also reproduced the three type III ANCOVAs respectively on the reduction of state stress, state anxiety, and trait anxiety as the dependent variables, with condition and thought valence as the independent variables in original study, which aimed to have a clear comparation on the simulated and original effects.
# ANCOVAsR_SS_F <-lm(SS ~ Con * Tho, data = FDATA)R_SA_F <-lm(SA ~ Con * Tho, data = FDATA)R_TA_F <-lm(TA ~ Con * Tho, data = FDATA)Anova(R_SS_F, type='III')
Anova Table (Type III tests)
Response: SS
Sum Sq Df F value Pr(>F)
(Intercept) 0.0087 1 0.3689 0.5442865
Con 0.3058 1 12.9538 0.0004012 ***
Tho 0.2066 1 8.7506 0.0034624 **
Con:Tho 0.2131 1 9.0281 0.0029934 **
Residuals 4.7924 203
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
ANOVA results using SS as the dependent variable
Predictor SS df MS F p partial_eta2 CI_95_partial_eta2
(Intercept) 0.01 1 0.01 0.37 .544
Con 0.31 1 0.31 12.95 .000 .06 [.01, .13]
Tho 0.21 1 0.21 8.75 .003 .04 [.00, .11]
Con x Tho 0.21 1 0.21 9.03 .003 .04 [.01, .11]
Error 4.79 203 0.02
Note: Values in square brackets indicate the bounds of the 95% confidence interval for partial eta-squared
ANOVA results using SA as the dependent variable
Predictor SS df MS F p partial_eta2 CI_95_partial_eta2
(Intercept) 0.09 1 0.09 5.85 .016
Con 0.10 1 0.10 6.74 .010 .03 [.00, .09]
Tho 0.01 1 0.01 0.47 .494 .00 [.00, .03]
Con x Tho 0.07 1 0.07 5.02 .026 .02 [.00, .08]
Error 2.95 203 0.01
Note: Values in square brackets indicate the bounds of the 95% confidence interval for partial eta-squared
ANOVA results using TA as the dependent variable
Predictor SS df MS F p partial_eta2 CI_95_partial_eta2
(Intercept) 0.01 1 0.01 3.24 .073
Con 0.00 1 0.00 0.01 .943 .00 [.00, .01]
Tho 0.00 1 0.00 0.26 .609 .00 [.00, .03]
Con x Tho 0.00 1 0.00 0.04 .842 .00 [.00, .02]
Error 0.72 203 0.00
Note: Values in square brackets indicate the bounds of the 95% confidence interval for partial eta-squared
Comparing the original and simulated results, I found that the reproduction is partially success. Basically I reproduced all the significant/non-significant effects, especially the condition × thought valence two-way interactions on the reduction of state stress and anxiety instead of trait anxiety, but except for the significant main effect of thought valence on state anxiety, which was not significant in the original study. In addition, the effect sizes of the simulated data are slight larger than the original study.
Exploratory Analyses
Any follow-up analyses desired (not required).
Preparation for Two-Way Mixed ANOVAs for Simulated Data
To conduct two-way mixed ANOVAs, I transformed the data into long format.
In order to preserve the information lost by the calculation of the difference in the post-test and pre-test of the dependent variables, I conducted a further exploration on the one-way ANOVAs mentioned above. I conducted 3 two-way mixed ANOVAs including condition (between-subjects IV) and time (within-subjects IV), with state stress, state anxiety, and trait anxiety as DV respectively, using simulated data.
eta_TA <-eta_squared(aov_TA, partial =TRUE, ci =0.95)eta_TA
# Effect Size for ANOVA (Type III)
Parameter | Eta2 (partial) | 95% CI
-----------------------------------------
Con | 1.20e-03 | [0.00, 1.00]
Time | 0.48 | [0.40, 1.00]
Con:Time | 0.01 | [0.00, 1.00]
- One-sided CIs: upper bound fixed at [1.00].
ggplot(long_sim_data, aes(x = Time, y = l.SS, color = Con, group = Con)) +stat_summary(fun = mean, geom ="line", linewidth =1.2) +stat_summary(fun = mean, geom ="point", size =3) +stat_summary(fun.data = mean_cl_normal, geom ="errorbar", width =0.1) +geom_line(aes(group = ID, color = Con), alpha =0.1, linewidth =0.6) +geom_point(aes(group = ID, color = Con), width =0.1, height =0, alpha =0.2, size =1.8) +labs(title ="Two-way Mixed ANOVA: State Stress across Time by Condition",x ="Time",y ="Mean State Stress (95% CI)" ) +theme_minimal(base_size =13) +theme(plot.title =element_text(hjust =0.5) )
Warning in geom_point(aes(group = ID, color = Con), width = 0.1, height = 0, :
Ignoring unknown parameters: `width` and `height`
ggplot(long_sim_data, aes(x = Time, y = l.SA, color = Con, group = Con)) +stat_summary(fun = mean, geom ="line", linewidth =1.2) +stat_summary(fun = mean, geom ="point", size =3) +stat_summary(fun.data = mean_cl_normal, geom ="errorbar", width =0.1) +geom_line(aes(group = ID, color = Con), alpha =0.1, linewidth =0.6) +geom_point(aes(group = ID, color = Con), width =0.1, height =0, alpha =0.2, size =1.8) +labs(title ="Two-way Mixed ANOVA: State Anxiety across Time by Condition",x ="Time",y ="Mean State Anxiety (95% CI)" ) +theme_minimal(base_size =13) +theme(plot.title =element_text(hjust =0.5) )
Warning in geom_point(aes(group = ID, color = Con), width = 0.1, height = 0, :
Ignoring unknown parameters: `width` and `height`
ggplot(long_sim_data, aes(x = Time, y = l.TA, color = Con, group = Con)) +stat_summary(fun = mean, geom ="line", linewidth =1.2) +stat_summary(fun = mean, geom ="point", size =3) +stat_summary(fun.data = mean_cl_normal, geom ="errorbar", width =0.1) +geom_line(aes(group = ID, color = Con), alpha =0.1, linewidth =0.6) +geom_point(aes(group = ID, color = Con), width =0.1, height =0, alpha =0.2, size =1.8) +labs(title ="Two-way Mixed ANOVA: Trait Anxiety across Time by Condition",x ="Time",y ="Mean Trait Anxiety (95% CI)" ) +theme_minimal(base_size =13) +theme(plot.title =element_text(hjust =0.5) )
Warning in geom_point(aes(group = ID, color = Con), width = 0.1, height = 0, :
Ignoring unknown parameters: `width` and `height`
For the state stress, I identified significant condition × time interaction (F(1,205) = 19.95, p < 0.001, partial_eta2 = 0.09, 95% CI = [.04, 1.00]) and main effects of time (F(1,205) = 386.57, p < 0.001, partial_eta2 = 0.65, 95% CI = [0.59, 1.00]), but non-significant main effect of condition (F(1,205) = 0.26, p = 0.611, partial_eta2 = 0.001, 95% CI = [.00, 1.00]). The significant interaction indicates that, compared to the control condition, the participants under the meditation condition experienced a greater reduction of state stress.
For the state anxiety, I identified significant condition × time interaction (F(1,205) = 15.06, p < 0.001, partial_eta2 = 0.07, 95% CI = [.02, 1.00]) and main effects of time (F(1,205) = 162.01, p < 0.001, partial_eta2 = 0.44, 95% CI = [0.36, 1.00]), but non-significant main effect of condition (F(1,205) = 1.17, p = 0.280, partial_eta2 = 0.005, 95% CI = [.00, 1.00]). The significant interaction indicates that, compared to the control condition, the participants under the meditation condition experienced a greater reduction of state anxiety.
For the trait anxiety, there was no significant main effects of condition (F(1,205) = 0.25, p = 0.620, partial_eta2 = 0.001, 95% CI = [.00, 1.00]) and interaction between condition and time (F(1,205) = 2.48, p = 0.117, partial_eta2 = 0.001, 95% CI = [.00, 1.00]), but I found significant main effect of time (F(1,205) = 187.22, p < 0.001, partial_eta2 = 0.48, 95% CI = [.40, 1.00]). The non-significant interaction effect indicates that the changes in trait anxiety brought about by meditation are not significantly different from those resulting from the control condition. The significant main effect of time indicates that there is indeed a significant difference in trait anxiety between the pre-test and the post-test. But I believe that this difference maybe due to the sequential effect.
LMMs (for Simulated Data) with Simple Slopes Analyses
In this section, I estimated three LMMs, using state stress, state anxiety, and trait anxiety as the dependent variables. Condition, thought valence, and time were entered as independent variables. The goal was to examine whether mindfulness meditation moderates the associations between thought valence and these outcome variables, while preserving pre- and post-test information without combining the difference score. Because the full three-way LMMs included lots of effects, and to maintain clarity and conciseness in the report, I focus exclusively on the core significant three-way interaction that are the aims of this project.
# Multilevel model (LMMs for simulated data) with simple slope analysislmm_SS_N <-lmer(l.SS ~1+ (1| ID), data = long_sim_data)summary(lmm_SS_N)
Linear mixed model fit by REML. t-tests use Satterthwaite's method [
lmerModLmerTest]
Formula: l.SS ~ 1 + (1 | ID)
Data: long_sim_data
REML criterion at convergence: 2517.5
Scaled residuals:
Min 1Q Median 3Q Max
-2.14798 -0.77100 -0.02748 0.72238 2.32362
Random effects:
Groups Name Variance Std.Dev.
ID (Intercept) 0.6668 0.8166
Residual 24.9614 4.9961
Number of obs: 414, groups: ID, 207
Fixed effects:
Estimate Std. Error df t value Pr(>|t|)
(Intercept) 15.198 0.252 206.000 60.3 <2e-16 ***
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
anova(lmm_SS_N)
Type III Analysis of Variance Table with Satterthwaite's method
Sum Sq Mean Sq NumDF DenDF F value Pr(>F)
Con = CON:
Time Tho emmean SE df lower.CL upper.CL
PRE 4.78 48.8 1.010 227 46.8 50.8
POS 4.78 45.0 1.010 227 43.0 47.0
Con = MED:
Time Tho emmean SE df lower.CL upper.CL
PRE 4.78 50.6 0.896 227 48.8 52.4
POS 4.78 45.8 0.896 227 44.0 47.6
Degrees-of-freedom method: kenward-roger
Confidence level used: 0.95
joint_tests(lmm_TA, by ="Con")
Con = CON:
model term df1 df2 F.ratio p.value
Tho 1 203 7.050 0.0086
Time 1 203 64.358 <.0001
Tho:Time 1 203 0.011 0.9154
Con = MED:
model term df1 df2 F.ratio p.value
Tho 1 203 8.116 0.0048
Time 1 203 127.112 <.0001
Tho:Time 1 203 0.227 0.6343
emm_plot_TA <-emmeans( lmm_TA, ~ Time | Con | Tho,at =list(Tho = Tho_grid$Tho))emm_df_TA <-as.data.frame(emm_plot_TA)emm_df_TA$Tho_level <-factor(Tho_grid$Tho_level[match(emm_df_TA$Tho, Tho_grid$Tho)],levels =c("-1 SD", "Mean", "+1 SD"))ggplot(emm_df_TA, aes(x = Time, y = emmean, color = Tho_level, group = Tho_level)) +geom_line(linewidth =1.2) +geom_point(size =3) +geom_errorbar(aes(ymin = lower.CL, ymax = upper.CL), width =0.1) +facet_wrap(~ Con) +labs(title ="Interaction of Time × Thought Valence at Each Condition",x ="Time",y ="Trait Anxiety",color ="Thought valence" ) +theme_minimal(base_size =14)
Above all, I estimated 3 null models containing only random intercepts for each of state stress, state anxiety, and trait anxiety to compute the ICCs. The results indicated that for state stress, ICC = 0.026, 95% CI = [0.001, 0.148]; for state anxiety, ICC = 0.048, 95% CI = [0.002, 0.174]; and for trait anxiety, ICC = 0.803, 95% CI = [0.755, 0.844].
For the LMM on state stress, I identified significant condition × thought valence × time three-way interaction (F(1,203) = 7.688, p = 0.006, β = 0.08, 95% CI = [0.02, 0.13]), thought valence × time two-way interaction, condition × time two-way interaction, and main effects of time and thought valence.The significant three-way interaction and the results of the simple slope analysis indicate that under control condition, the more positive the participants’ thought valence were, the greater the reduction in state stress was, while under the meditation condition, the effect of participants’ thought valence on the reduction of state stress was not significant. Mindfulness meditation indeed blocks the association between thought valence and the reduction of state stress.
For the LMM on state anxiety, I identified significant condition × thought valence × time three-way interaction (F(1,203) = 8.448, p = 0.004, β = 0.10, 95% CI = [0.03, 0.17]), thought valence × time two-way interaction, condition × time two-way interaction, condition × thought valence two-way interaction, and main effects of condition and thought valence. The significant three-way interaction and the results of the simple slope analysis indicate that under control condition, the more positive the participants’ thought valence were, the greater the reduction in state anxiety was, while under the meditation condition, the effect of participants’ thought valence on the reduction of state anxiety was not significant. Mindfulness meditation indeed blocks the association between thought valence and the reduction of state anxiety.
For the LMM on trait anxiety, I only identified significant main effects of time and thought valence.The non-significant three-way interaction indicates that the conditions did not moderate the relationship between thought valence and trait anxiety. A single session of mindfulness meditation is not sufficient to effectively change the trait of anxiety.
Preparation for Two-Way Mixed ANOVAs for Original Data
To conduct two-way mixed ANOVAs, I transformed the data into long format.
In order to preserve the information lost by the calculation of the difference in the post-test and pre-test of the dependent variables, I conducted a further exploration on the one-way ANOVAs mentioned above. I conducted 3 two-way mixed ANOVAs including condition (between-subjects IV) and time (within-subjects IV), with state stress, state anxiety, and trait anxiety as DV respectively, using original data.
# two-way mixed ANOVA (for original data)long_sim_data_F$Con <-factor(long_sim_data_F$Con)long_sim_data_F$Time <-factor(long_sim_data_F$Time, levels =c("PRE", "POS"))# SSaov_SS_F <-aov_car( l.SS ~ Con * Time +Error(ID/Time),data = long_sim_data_F,type =3)aov_SS_F
eta_TA_F <-eta_squared(aov_TA_F, partial =TRUE, ci =0.95)eta_TA_F
# Effect Size for ANOVA (Type III)
Parameter | Eta2 (partial) | 95% CI
-----------------------------------------
Con | 1.88e-03 | [0.00, 1.00]
Time | 0.34 | [0.26, 1.00]
Con:Time | 1.32e-03 | [0.00, 1.00]
- One-sided CIs: upper bound fixed at [1.00].
ggplot(long_sim_data_F, aes(x = Time, y = l.SS, color = Con, group = Con)) +stat_summary(fun = mean, geom ="line", linewidth =1.2) +stat_summary(fun = mean, geom ="point", size =3) +stat_summary(fun.data = mean_cl_normal, geom ="errorbar", width =0.1) +geom_line(aes(group = ID, color = Con), alpha =0.1, linewidth =0.6) +geom_point(aes(group = ID, color = Con), width =0.1, height =0, alpha =0.2, size =1.8) +labs(title ="Two-way Mixed ANOVA: State Stress across Time by Condition",x ="Time",y ="Mean State Stress (95% CI)" ) +theme_minimal(base_size =13) +theme(plot.title =element_text(hjust =0.5) )
Warning in geom_point(aes(group = ID, color = Con), width = 0.1, height = 0, :
Ignoring unknown parameters: `width` and `height`
ggplot(long_sim_data_F, aes(x = Time, y = l.SA, color = Con, group = Con)) +stat_summary(fun = mean, geom ="line", linewidth =1.2) +stat_summary(fun = mean, geom ="point", size =3) +stat_summary(fun.data = mean_cl_normal, geom ="errorbar", width =0.1) +geom_line(aes(group = ID, color = Con), alpha =0.1, linewidth =0.6) +geom_point(aes(group = ID, color = Con), width =0.1, height =0, alpha =0.2, size =1.8) +labs(title ="Two-way Mixed ANOVA: State Anxiety across Time by Condition",x ="Time",y ="Mean State Anxiety (95% CI)" ) +theme_minimal(base_size =13) +theme(plot.title =element_text(hjust =0.5) )
Warning in geom_point(aes(group = ID, color = Con), width = 0.1, height = 0, :
Ignoring unknown parameters: `width` and `height`
ggplot(long_sim_data_F, aes(x = Time, y = l.TA, color = Con, group = Con)) +stat_summary(fun = mean, geom ="line", linewidth =1.2) +stat_summary(fun = mean, geom ="point", size =3) +stat_summary(fun.data = mean_cl_normal, geom ="errorbar", width =0.1) +geom_line(aes(group = ID, color = Con), alpha =0.1, linewidth =0.6) +geom_point(aes(group = ID, color = Con), width =0.1, height =0, alpha =0.2, size =1.8) +labs(title ="Two-way Mixed ANOVA: Trait Anxiety across Time by Condition",x ="Time",y ="Mean Trait Anxiety (95% CI)" ) +theme_minimal(base_size =13) +theme(plot.title =element_text(hjust =0.5) )
Warning in geom_point(aes(group = ID, color = Con), width = 0.1, height = 0, :
Ignoring unknown parameters: `width` and `height`
For the state stress, I identified significant condition × time interaction (F(1,205) = 7.72, p = 0.006, partial_eta2 = 0.04, 95% CI = [.01, 1.00]) and main effects of time (F(1,205) = 237.09, p < 0.001, partial_eta2 = 0.54, 95% CI = [0.46, 1.00]), but non-significant main effect of condition (F(1,205) = 0.06, p = 0.804, partial_eta2 = 0.0003, 95% CI = [.00, 1.00]). The significant interaction indicates that, compared to the control condition, the participants under the meditation condition experienced a greater reduction of state stress.
For the state anxiety, I identified marginally significant condition × time interaction (F(1,205) = 3.31, p = 0.070, partial_eta2 = 0.02, 95% CI = [.00, 1.00]) and significant main effects of time (F(1,205) = 194.19, p < 0.001, partial_eta2 = 0.49, 95% CI = [0.41, 1.00]), but non-significant main effect of condition (F(1,205) = 0.03, p = 0.855, partial_eta2 = 0.0001, 95% CI = [.00, 1.00]). The marginally significant interaction shows a trend that, compared to the control condition, the participants under the meditation condition experienced a greater reduction of state anxiety.
For the trait anxiety, there was no significant main effects of condition (F(1,205) = 0.39, p = 0.535, partial_eta2 = 0.001, 95% CI = [.00, 1.00]) and interaction between condition and time (F(1,205) = 0.27, p = 0.603, partial_eta2 = 0.001, 95% CI = [.00, 1.00]), but I found significant main effect of time (F(1,205) = 107.88, p < 0.001, partial_eta2 = 0.34, 95% CI = [.26, 1.00]). The non-significant interaction effect indicates that the changes in trait anxiety brought about by meditation are not significantly different from those resulting from the control condition. The significant main effect of time indicates that there is indeed a significant difference in trait anxiety between the pre-test and the post-test. But I believe that this difference maybe due to the sequential effect.
The results of the two-way ANOVAs for both the original data and the simulated data were consistent, except that the two-way interaction on state anxiety was significant in the simulated data, while it was only marginally significant in the original data. Moreover, similar to the results of one-way ANOVAs, the effect sizes in two-way ANOVAs are slightly larger.
LMMs (for Original Data) with Simple Slopes Analyses
Just like the analyses conducted on the simulated data, I also performed LMMs and corresponding simple slopes analyses on the original data. In fact, this is precisely the result that I was most eager to see. Specifically, the three LMMs were respectively estimated with the state stress, state anxiety, and trait anxiety as dependent variables, and condition, time, and thought valence as independent variables. Random intercepts were included but no random slopes were included for the analyses. The aim was to discover the effect of the full model that incorporates within-subjects effects in the original data. Again, because the full three-way LMMs included lots of effects, and to maintain clarity and conciseness in the report, I focus exclusively on the core significant three-way interaction that are the aims of this project.
# Multilevel model (LMMs for original data)lmm_SS_F_N <-lmer(l.SS ~1+ (1| ID), data = long_sim_data_F)summary(lmm_SS_F_N)
Linear mixed model fit by REML. t-tests use Satterthwaite's method [
lmerModLmerTest]
Formula: l.SS ~ 1 + (1 | ID)
Data: long_sim_data_F
REML criterion at convergence: 2571.9
Scaled residuals:
Min 1Q Median 3Q Max
-1.86967 -0.72535 -0.07856 0.76485 2.52750
Random effects:
Groups Name Variance Std.Dev.
ID (Intercept) 4.298 2.073
Residual 25.249 5.025
Number of obs: 414, groups: ID, 207
Fixed effects:
Estimate Std. Error df t value Pr(>|t|)
(Intercept) 14.2101 0.2859 206.0000 49.7 <2e-16 ***
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
anova(lmm_SS_F_N)
Type III Analysis of Variance Table with Satterthwaite's method
Sum Sq Mean Sq NumDF DenDF F value Pr(>F)
Con = CON:
Time Tho emmean SE df lower.CL upper.CL
PRE 4.88 46.9 1.100 225 44.8 49.1
POS 4.88 43.7 1.100 225 41.5 45.8
Con = MED:
Time Tho emmean SE df lower.CL upper.CL
PRE 4.88 47.6 0.981 225 45.7 49.5
POS 4.88 44.0 0.981 225 42.0 45.9
Degrees-of-freedom method: kenward-roger
Confidence level used: 0.95
joint_tests(lmm_TA_F, by ="Con")
Con = CON:
model term df1 df2 F.ratio p.value
Tho 1 203 12.814 0.0004
Time 1 203 43.035 <.0001
Tho:Time 1 203 0.042 0.8385
Con = MED:
model term df1 df2 F.ratio p.value
Tho 1 203 11.657 0.0008
Time 1 203 66.683 <.0001
Tho:Time 1 203 0.309 0.5789
# plottingemm_plot_TA_F <-emmeans( lmm_TA_F, ~ Time | Con | Tho,at =list(Tho = Tho_grid_F$Tho))emm_df_TA_F <-as.data.frame(emm_plot_TA_F)emm_df_TA_F$Tho_level <-factor(Tho_grid_F$Tho_level[match(emm_df_TA_F$Tho, Tho_grid_F$Tho)],levels =c("-1 SD", "Mean", "+1 SD"))ggplot(emm_df_TA_F, aes(x = Time, y = emmean, color = Tho_level, group = Tho_level)) +geom_line(linewidth =1.2) +geom_point(size =3) +geom_errorbar(aes(ymin = lower.CL, ymax = upper.CL), width =0.1) +facet_wrap(~ Con) +labs(title ="Interaction of Time × Thought Valence at Each Condition",x ="Time",y ="Trait Anxiety",color ="Thought valence" ) +theme_minimal(base_size =14)
Following the analysis procedure above, I estimated 3 null models containing only random intercepts for each of state stress, state anxiety, and trait anxiety to compute the ICCs. The results indicated that for state stress, ICC = 0.145, 95% CI = [0.050, 0.275]; for state anxiety, ICC = 0.400, 95% CI = [0.285, 0.499]; and for trait anxiety, ICC = 0.862, 95% CI = [0.832, 0.889].
For the LMM on state stress, I identified significant condition × thought valence × time three-way interaction (F(1,203) = 9.0281, p = 0.003, β = 0.09, 95% CI = [0.03, 0.15]), thought valence × time two-way interaction, condition × time two-way interaction, and main effects of thought valence. There were also marginal significant condition × thought valence two-way interaction and main effect of condition. The significant three-way interaction and the results of the simple slope analysis indicate that under control condition, the more positive the participants’ thought valence were, the greater the reduction in state stress was, while under the meditation condition, the effect of participants’ thought valence on the reduction of state stress was not significant. Mindfulness meditation indeed blocks the association between thought valence and the reduction of state stress.
For the LMM on state anxiety, I identified significant condition × thought valence × time three-way interaction (F(1,203) = 5.0246, p = 0.026, β = 0.06, 95% CI = [0.01, 0.12]), condition × time two-way interaction, and main effects of time and thought valence. Although the simple slope analysis on thought valence × time interactions of control condition was marginal significant, the significant three-way interaction still indicates that under control condition, there was a trend that the more positive the participants’ thought valence were, the greater the reduction in state anxiety was, while under the meditation condition, the effect of participants’ thought valence on the reduction of state anxiety was not significant. Mindfulness meditation indeed blocks the association between thought valence and the reduction of state anxiety. However, this effect size may be relatively small, and it needs to be further confirmed in future studies with a larger sample size.
For the LMM on trait anxiety, I only identified significant main effect of thought valence and marginal significant main effect of time.The non-significant three-way interaction indicates that the conditions did not moderate the relationship between thought valence and trait anxiety. A single session of mindfulness meditation is not sufficient to effectively change the trait of anxiety.
Discussion
Summary of Reproduction Attempt
Open the discussion section with a paragraph summarizing the primary result from the key analysis and assess whether you successfully reproduced it, partially reproduced it, or failed to reproduce it.
In this simulation study, I partially successfully reproduced the research results of the original literature. As mentioned earlier, this study has two key findings, namely the reproduced results (condition × thought valence two-way interaction) and the exploratory results (condition × time × thought valence three-way interaction).
To examine the two-way interaction between condition and thought valence, we employed two ANCOVAs on state stress and state anxiety, with condition and thought valence as independent variables. I also conducted ANCOVA on trait anxiety just for exploration.
For the ANCOVA on state stress in simulated data, I found a two-way interaction effect that has almost the same amounts of effect size as the original literature. And in accordance with the original literature, the main effects of conditions and thought valence were both significant. The effect size of the main effect of the condition was approximately consistent with the original literature, while the effect size of the main effect of the thought value was 0.02 smaller than that in the original literature.
For the ANCOVA on state anxiety in simulated data, I also found a two-way interaction effect，but the effect size in simulation study was 0.02 larger than in original literature. I reproduced the main effect of condition, with the effect being 0.03 larger than those in original study. I failed to reproduce the non-significant main effect of thought valence in original study.
For the ANCOVA on trait anxiety in simulated data, all main effects and interaction are not significant, which is consistent with the hypothesis in original literature.
For the exploration analyses, to examine the condition × time × thought valence three-way interaction, we employed three LMMs on state stress and state anxiety. I also conducted LMM on trait anxiety just for exploration. Moreover, the results of LMM are basically consistent with those of ANCOVA.
For the LMM on state stress in simulated data, I found a condition × time × thought valence three-way interaction that support the hypothesis in original study. I also found two-way interactions of condition × time and thought valence × time, which are consistent with the main effect in ANCOVA. Main effects of time and thought valence were also significant.
For the LMM on state anxiety in simulated data, I also found a condition × time × thought valence three-way interaction that support the hypothesis in original study. Two-way interactions of condition × time and thought valence × time were found which are consistent with the main effect in ANCOVA. I was surprised by the significant interaction of condition × thought valence. I think that is the information that not showing in the results of ANCOVA because of the composition of pre- and post-scores Main effects of condition, time, and thought valence were also significant.
For the LMM on trait anxiety, all interaction are not significant, which is consist with the hypothesis in original literature.
In summary, I believe that this simulation reproduction study partially successfully reproduced the results of the original study, except that it failed to reproduce the non-significant main effect of thought valence in the ANCOVA for state anxiety.
Commentary
Add open-ended commentary (if any) reflecting (a) insights from follow-up exploratory analysis of the dataset, (b) assessment of the meaning of the successful or unsuccessful reproducibility attempt - e.g., for a failure to reproduce the original findings, are the differences between original and present analyses ones that definitely, plausibly, or are unlikely to have been moderators of the result, and (c) discussion of any objections or challenges raised by the current and original authors about the reproducibility attempt (if you contacted them). None of these need to be long.
In this reproduction study, I conducted two kinds of exploratory analyses, namely two-way ANOVAs and LMMs.
Two-way ANOVAs were used to compare the effect of mindfulness meditation with the control condition, without subtracting the scores of dependent variables (state stress, state anxiety, and trait anxiety for validation check) from the post-test from the pre-test scores, both in original data and simulated data. Specifically, I used three 2 condition (between-subjects variable) × 2 time (within-subject variables) two-way mixed ANOVAs and tried to detected the two-way interactions.
In original data, I found significant interaction on state stress, marginally significant interaction on state anxiety, and non-significant interaction on trait anxiety. At same time, the main effects of time were all significant across three dependent variables. These results show that it’s true that two conditions have different effects on changing state stress and state anxiety, although there are significant overall differences between pre- and post-test.
In addition, I identified the marginally significant interaction on state anxiety, which might because the floor effect. In the original data, the post-test state anxiety showed a positive skewed trend. This led me to believe that it might be due to the fact that the level of state anxiety before the treatment was relatively low, and it dropped to the floor level after the treatment. The floor effect also provides evidence for why the interaction was marginally significant.
Moreover, the significant main effect of time on trait anxiety indicates that the overall trait anxiety decreased. I suppose that it might because the repeated measuring same scale or participants guessed that there should be some differences. This significant main effect of time make the composition more reasonable, because there should be some differences between pre- and post-scores, but they are not caused by the manipulation. What’s more, the insignificant interaction on trait anxiety can provide evidence that the effect size of mindfulness meditation on traits is to small to be detected.
The results of the two-way ANOVAs in the simulated data are basically consistent with those of the original data. However, the two-way interaction on state anxiety was significant in the simulated data, while it was only marginally significant in the original data. Moreover, similar to the results of one-way ANOVAs, the effect sizes in two-way ANOVAs are slightly larger.
I conducted LMMs also for same reason. I’d like to figure out what would happen if I take the within-subjects effects between pre- and post-scores into account. Fortunately, in original data, the significance of condition × time × thought valence three-way interactions of LMMs were consistent with those condition × thought valence two-way interactions of ANCOVA. This indicates that even when the within-subject effects are taken into account, the relationship between mindfulness meditation and the reduction of thought valence, state stress and anxiety remains robust.
It is worth noting that when estimating the LMMs, I found that the results of the original data and the simulated data were roughly similar, but there were still some minor differences. Firstly, I found that compared to the original data, the individual differences in the simulated data were smaller. The ICCs estimated by the null model were all lower than those of the original data. Secondly, in the simple slope analysis of the three-way interaction of state anxiety, the two-way interaction in control condition in the original data was significant, while the interaction in the simulated data was completely significant. This is in line with the larger effect size observed in the simulation data. Thirdly, apart from the consistency of the significance of the three-way interactions, the significance of the other main effects and interactions varies slightly between the two datasets. In the LMM estimation of state stress, the simulated data has an additional time main effect compared to the original data. In the LMM estimation of state anxiety, the simulated data had less main effect of time compared to the original data, but had more two-way interactions of thought valence × time, condition × thought valence, and the main effect of condition. In the LMM estimation of trait anxiety, the simulated data had a main effect of time, which was not present in the original data. These inconsistencies indicate that, after fully decomposing the effects, although the original data and the simulated data are similar, there are still minor differences.
Regarding the significant differences between the original data and the simulated data found in t-test, it shows that there are some differences of the distribution characteristics between two datasets. In terms of the situation where the mean, standard deviation, correlation coefficient in the generated data do not exactly match the original data, I believe that from the perspective of the bias–variance tradeoff, imposing numerous fixed parameters in data generation reduces model flexibility and increases systematic bias, making the simulated data less able to approximate the true underlying structure. Likewise, the principle of maximum entropy indicates that adding constraints forces the distribution away from the most natural, high-entropy solution, thereby reducing stability and accuracy.
So I think, although the original data and the simulated data are roughly similar, and similar target effects have been found from them, the reason why they are not completely consistent might be because I imposed too many constraints. Specifically, I have limited at least 112 parameters, including the 7 variables’ sample sizes, means, maximum values, minimum values, as well as their variances and covariances under the two conditions. The excessive fixed parameters have led to a relatively wide range of values for each parameter.
Another possible reason for the deviation is that in the multivariate truncated distribution, conducting rejection sampling may cause a bias in the mean due to the truncation of the multivariate distribution.
However, all these limitations and stages are designed to achieve a more accurate simulation of the original data. Therefore, a slight deviation in parameters might be the inevitable cost to be paid.
Questions
1. What assumptions does your generative model make about participant behavior, and which aspects of the real experiment does it simplify or omit?
The assumptions in simulation model of this project regarding human behavior are that the overall distribution characteristics of thought valence, pre-test state stress, pre-test state anxiety, pre-test trait anxiety, post-test state stress, post-test state anxiety, and post-test trait anxiety are consistent with the original data, and all follow the multivariate truncated normal distribution.
It must be admitted that the distribution of the original data is not entirely normal. Considering that a relatively large sample was used in the original article, according to the Central Limit Theorem, the data should asymptotically follow a normal distribution. For example, Regarding the post-test anxiety of the original data, it can be observed that the distribution of this data may not be strictly normal, but rather skewed to the right. I have considered this matter and believe that it might be due to the experimental operation that the floor effect occurred, causing the mean value of anxiety after the intervention to gradually approach zero.
In addition, using the mean and standard deviation of the original data as the parameters of the population is actually not completely precise. After all, the mean and standard deviation of the sample are, and they are only estimates of the population parameters. The parameters of the population distribution are likely to fall within their 95% confidence intervals, but the possibility of complete consistency is still relatively small.
2. Show how your simulated dataset flows through your planned analysis pipeline. Does this recover any mismatches or bugs?
My analysis process is as follows: first, descriptive statistical analysis and correlation analysis were conducted on the original data; then, based on the obtained descriptive statistics and correlation coefficients, simulated data was generated; sequentially, descriptive, statistical, correlation, one-way ANOVAs, and two-way ANCOVAs were carried out on the simulated data. To further verify the results, I contducted independent sample t-test and bootstrap covariance homogeneity test to compare the original and simulated data, and repeated descriptive statistical analysis, correlation analysis, one-way ANOVAs, and two-way ANCOVAs with original data. Finally, to step further, I explored the two datasets with two-way ANOVAs and LMMs estimation respectively.
To be honest, when I was trying to reproduce the results of the two-way ANCOVAs using the original data, I encountered a great difficulty. I just couldn’t replicate the results in the article until I noticed the code “options(contrasts = c(‘contr.sum’, ‘contr.poly’))” from the author’s code book. By coding categorical independent variables using sum-to-zero contrasts, the intercept in ANCOVA represents the grand mean, and the effects of each factor level reflect symmetric deviations from that mean, thereby yielding more stable and interpretable main effects. It was not until I incorporated it into my code that I finally obtained the original research results. Base on this experience, I realized that using sum-to-zero contrasts is not required for ANCOVA, but it is often recommended in experimental psychology because it yields interpretable main effects, symmetric factor comparisons, and a grand-mean-centered intercept.
3. Identify one insight your simulation gave you about the strengths or limitations of the original experimental design.
I think the biggest problem in the original research experimental design is that it asks participants to recall the average thought valence within a 20-minute period. I believe that whether in the context of mindfulness meditation or under the controlled conditions (listening to a lecture on relaxation exercises), having participants recall for 20 minutes is likely to introduce bias in the average thought valence. For the participants in the mindfulness meditation condition, they were instructed to observe their thoughts when they arose, but not to be attached to them. This may result in unclear memories of the thoughts. However, if participants under the control condition focused on the lecture, then it becomes even more difficult for them to notice their own thoughts. This made me want to break the 20-minute intervention into several smaller segments, making the recall process easier. This is also something I hope to achieve in future research in the next question.
4. How would this simulation help you design a follow-up experiment with a similar paradigm?
There is no doubt that reproducing the analysis process of this study has been of great help to me, and I also thought a lot during this process. The research I have planned is to make some minor modifications based on this study. Specifically, the experimental design adopted in this study was a pre-test - 20-minute intervention - post-test design. Although this study achieved significant interactions, I believe that asking the participants to recall their thought valence within 20 minutes might cause some confounds. I plan to conduct a study where the 20-minute intervention will be divided into three 5-minute segments, and sampling will be conducted four times (T0-T3). This design can reduce the confound caused by memories and also increase the statistical power. In addition, in this study, the authors regarded thought valence as a moderator in the relationship between meditation and wel-being. But I was wondering if it was possible that mindfulness meditation affects people’s thought valence, thereby influencing their wel-being. In other words, it’s a model that thought valence mediates the relationship between mindfulness meditation and well-being. In this study, I have observed a marginally significant two-way interaction on state anxiety. This indicates that the statistical power is insufficient. However, to estimate the mediating effect, a greater statistical power is still required, and this is precisely what is suitable for my repeated measurement design.
