Abstract & Background

Engagement strategies such as retrieval practice, feedback and visual aids can enhance learning in traditional settings, but little is known about their impact on virtual learning. The current study assesses efficacy of engagement strategies integrated into a virtual stress management intervention. Preliminary analyses indicate that these engagement strategies facilitated participant comprehension. Effects were magnified in a subset of high-neuroticism participants. Findings suggest that engagement strategies warrant consideration for inclusion in online pedagogy.

Background

  • Online learning on the rise, requiring study of engagement strategies

Learning Engagement Strategies

  • Retrieval practice and feedback can enhance content-based learning (Putnam, Nestojko, & Roediger, 2017).
    • Retrieval practice (testing effect): repeated quizzing strengthens semantic network connections and enhances understanding of material → stronger long-term learning (Kornell, 2014; Roediger & Karpicke, 2006)
    • Feedback: explains why the answer is correct and magnifies the effects of retrieval practice (Butler, Godbole, & Marsh, 2013; Kornell, Klein, & Rawson, 2015)
  • Visuals enhance understanding because they are processed very quickly, faster than text (Thorpe, Fize, & Marlot, 1996)
  • Less is known about the combined effects of the different engagement strategies when used online and outside classroom settings

Therapeutic Interventions

  • Online therapy is effective and can be more feasible and accessible
  • Transition to college is stressful and stress management interventions can help, and these can be fully online (e.g., Titov et al., 2008)
  • The current study fills a gap by examining the impact of common traditional learning strategies in an online program

Hypotheses

  • Holistic effect of retrieval practice plus feedback and visuals will result in the greatest reduction in cognitive distortions, the greatest increase in perceived control over stressful events, and the highest scores on the final comprehension test
    • Exploratory analyses for effects of personality traits on intervention and cognitive distortions (particularly neuroticism)

Method

The present study examined whether engagement strategies enhanced learning of the material of a 1-session online stress intervention.

Participants

  • 100 introductory-psychology student participants
  • Class Year & Age: First year (27); Sophomore (54); Junior (14); Senior (5); Age (19.05 [0.8])
  • Race/Ethnicity: White (60); Black (3); Asian (21); Latinx (14); Middle Eastern (2)
  • Gender: Man (21); Woman (78); Specified Gender (1-nonbinary)

Total # of participants100 + 26 for N = 60 subset

Measures

Baseline only

  • Big Five Personality Inventory (a measure of five facets of personality)
    • I am someone who… can be tense
  • Cognitive Flexibility Inventory (a measure of ability to produce alternate explanations)
    • I consider multiple options before making a decision. (13 to 78, high scores mean more flexibility)

Post-intervention only

  • Final comprehension test (a measure of how well the material was learned and understood)
    • Which of the following statements is the most accurate about stress’s role in people’s life? (Moderate stress can motivate you to complete tasks that need to get done.)
  • Demographics questionnaire (class year, age, race/ethnicity, gender)

Baseline and post-intervention

  • Inventory of Cognitive Distortions (a measure of maladaptive thoughts)
    • I attempt to achieve perfection in all areas of my life. (0-34, higher indicates more cognitive distortions
  • Perceived Control over Stressful Events scale (a measure of perceived control the fictional character has over an event) (8-32, higher numbers mean more perceived control)
    • There isn’t much Alex can do to keep the event from affecting them.

Procedure & Study Design

100 participants completed the Baseline measures before being randomly assigned to one of three intervention conditions the experimental group (N = 34), the visual control group (N = 33), and the dialogue control group (N = 33).

Intervention summary

The stress intervention described a counseling session for a fictional college student who experiences stress and goes to counseling. Participants read dialogue between the student and the counselor that emphasized fundamental aspects of cognitive-behavioral interventions for stress. The text describing the intervention was identical for all 3 conditions, but other features varied. The experimental group included visual aids to supplement key ideas in the text as well as several engagement strategies: quiz-style questions and feedback interspersed throughout the “session” that tested comprehension of prior therapy content. The visual control group also had visual aids but no engagement strategies. The dialogue control group had neither engagement strategies nor visual aids. Instead of engagement strategies, participants in both control groups (visual control and dialogue control) read content summaries (i.e., the restudy condition in traditional retrieval practice studies) interspersed throughout. After completing the intervention, all participants completed the post-test measures and were debriefed.

About the variables

A summary of the variables and their information including scoring

Pre-test measures

ICDbaseline: summed inventory of cognitive distortions baseline (range 0-34, higher=stronger endorsement of distortions experienced)

CFIbaseline: summed cognitive flexibility inventory

PCSEbaselineraw: raw summed score of perceived control over stressful events baseline (range 8-32, higher scores=more control)

PCSEbaseline: adjusted summed score for baseline (range of 0-24, higher scores=more control)

Post-test measures

Ftscore: final test score without penalty for wrong answers–each question is worth one point, broken up across correct number of answers (range of 0-8 with higher scores indicating better performance)

Ftpenalty: final test score with a partial penalty incorporated for incorrect answers (range of 0-8 with higher scores indicating better performance)

ICDposttest: summed inventory of cognitive distortions posttest (range 0-34, higher=stronger endorsement of distortions experienced)

ICDdiff: baseline-posttest (0=no change, negative numbers=worsening distortions, positive numbers=better distortions)

ICDcount: count of posttest “this sounds a lot like me” responses (range of 0-17, higher=more endorsed distortions)

ICDhml: split into High, Medium, and Low cognitive distortions counts (0-1=low [0], 2-6=medium [1], 7-17=high [2])

PCSEposttestraw: raw summed score of perceived control over stressful events posttest (range 8-32, higher scores=more control)

PCSEpost: adjusted summed score for posttest (range of 0-24, higher scores=more control)

PCSEdiff: difference between posttest and baseline [posttest-baseline] (0=no change, negative numbers=worsening perceived control posttest, positive numbers=better perceived control posttest)

Big Five Personality Inventory

Extroversion: Big 5 extroversion (higher numbers=more extroverted)

Agreeableness: Big 5 agreeableness (higher numbers=more agreeableness)

ConscientiousnessR: Big 5 conscientiousness reverse coded (higher numbers=lower conscientiousness)

NeuroticismR: Big 5 neuroticism reverse coded (higher numbers=lower neuroticism)

Openness: Big 5 openness (higher numbers=more openness)

Demographics

classyr: year in school (1=first-year, 2=sophomore, 3=junior, 4=senior)

age: text entry of age in years

gender: gender identified (1=man, 2=woman, 3=specified)

gendertext: text specified gender if gender=3

race: race identified (1=white, 2=Black, 3=Asian, 4=Latinx, 5=specified)

racetext: text specified race if race=5, both happened to specify Middle Eastern

Results

The results of this pilot study indicate that the intervention impacted predictions of future stress. Those in the experimental group expected the fictional character to be more successful in controlling future stress than the dialogue control group, with the visual control group’s perceived control falling between.

Preparing the data

PCSE Means (SD) Table
BaselineMean BaselineSD PostMean PostSD
17.93 4.11 19.73 3.66
PCSE Means (SD) High Neuroticism Subset Table
BaselineMean BaselineSD PostMean PostSD
18.02 4.16 19.67 3.78

Perceived Control over Stressful Events (PCSE) Analyses for all participants

  • A 3(group: experimental, visual, dialogue) x 2(time: baseline & post-intervention) mixed factorial analysis of variance (ANOVA) on perceived control found main effects of group (p < .001, η²p = 0.16) and time (p < .001, η²p = 0.08).

  • These results indicate that overall perceived control increased from baseline to post-intervention for all conditions as well as differentially across condition. The effect sizes were moderate for the main effect of time and large for the main effect of group.

  • As predicted, the dialogue control group had the smallest changes in perceived control (non-significant contrasts) across the three conditions. In contrast to my prediction that the experimental group would experience larger increases than the visual control group, the biggest changes were in the visual control group.

PCSE Effect Sizes (Partial Eta Squared
  Df Sum Sq Mean Sq F value Pr(>F)
group 2 179.3 89.67 4.458 0.01405
Residuals 97 1951 20.11 NA NA
PCSE 1 162 162 18.27 4.488e-05
group:PCSE 2 7.848 3.924 0.4425 0.6437
Residuals 97 860.2 8.868 NA NA
PCSE Main Effects of Group
Group Parameter Eta2_partial CI CI_low CI_high
subject group 0.0841857 0.95 0.0035868 0.1953577
subject:PCSE PCSE 0.1584892 0.95 0.0481100 0.2912841
subject:PCSE group:PCSE 0.0090420 0.95 0.0000000 0.0633754
PCSE Post-Hoc Differences
group emmean SE df lower.CL upper.CL
Experimental 20.07124 0.5474774 97 18.98464 21.15783
MediaControl 18.63362 0.5501991 97 17.54163 19.72562
DialogueControl 17.78514 0.5501991 97 16.69315 18.87713
PCSE Paired Post-Hoc Differences
PCSE group emmean SE df lower.CL upper.CL
PCSEbaseline Experimental 19.07124 0.6558480 167.9703 17.77647 20.36600
PCSEpost Experimental 21.07124 0.6558480 167.9703 19.77647 22.36600
PCSEbaseline MediaControl 17.55787 0.6611171 168.9336 16.25275 18.86298
PCSEpost MediaControl 19.70938 0.6611171 168.9336 18.40427 21.01450
PCSEbaseline DialogueControl 17.16393 0.6611171 168.9336 15.85881 18.46904
PCSEpost DialogueControl 18.40635 0.6611171 168.9336 17.10124 19.71147
contrast group estimate SE df t.ratio p.value
PCSEbaseline - PCSEpost Experimental -2.000000 0.7222327 97 -2.769191 0.0067336
PCSEbaseline - PCSEpost MediaControl -2.151515 0.7330940 97 -2.934842 0.0041660
PCSEbaseline - PCSEpost DialogueControl -1.242424 0.7330940 97 -1.694768 0.0933276

PCSE Analyses for Neuroticism Subsets

  • In a subset (N = 60 from additional targeted recruitment) of participants with higher neuroticism, I conducted an exploratory 3(group) x 2(time) mixed factorial ANOVA and compared it to an ANOVA on participants with low neuroticism (N = 54). In the high neuroticism subset, there was a main effect of time (p < .001, η²p = 0.12) and group (p = .025, η²p = 0.29), while there were no main effects or interactions in the low neuroticism subset (All p’s > .05).

  • These results indicate that the intervention across conditions was more effective for participants with high neuroticism than participants with low neuroticism. Specifically, it seems as though the participants with high neuroticism drove the effects of group and time on all participants, as the high neuroticism subset had larger effect sizes.

  • Experimental and visual control groups experienced significant increases in perceived control while the dialogue control group did not in high neuroticism subset post-hocs.

Analysis of Variance for a high neuroticism subset

PCSE High Neuroticism Effect Sizes
  Df Sum Sq Mean Sq F value Pr(>F)
group 2 146.2 73.11 3.342 0.03867
Residuals 120 2625 21.87 NA NA
PCSE 1 169.2 169.2 18.96 2.825e-05
group:PCSE 2 11.2 5.602 0.6279 0.5355
Residuals 120 1071 8.922 NA NA
PCSE Subset Main Effects of Group
Group Parameter Eta2_partial CI CI_low CI_high
subject group 0.0527684 0.95 0.0000000 0.1400053
subject:PCSE PCSE 0.1364505 0.95 0.0425839 0.2533936
subject:PCSE group:PCSE 0.0103568 0.95 0.0000000 0.0609434
PCSE Post-Hoc
group emmean SE df lower.CL upper.CL
Experimental 19.95233 0.5355792 120 18.89192 21.01273
MediaControl 18.50326 0.5182067 120 17.47724 19.52927
DialogueControl 18.08100 0.5055401 120 17.08007 19.08194
PCSE Paired Post-Hocs
PCSE group emmean SE df lower.CL upper.CL
PCSEbaseline Experimental 18.95233 0.6465672 210.7834 17.67776 20.22689
PCSEpost Experimental 20.95233 0.6465672 210.7834 19.67776 22.22689
PCSEbaseline MediaControl 17.50326 0.6142813 203.5284 16.29209 18.71443
PCSEpost MediaControl 19.50326 0.6142813 203.5284 18.29209 20.71443
PCSEbaseline DialogueControl 17.51850 0.5903448 197.0812 16.35430 18.68271
PCSEpost DialogueControl 18.64350 0.5903448 197.0812 17.47930 19.80771
contrast group estimate SE df t.ratio p.value
PCSEbaseline - PCSEpost Experimental -2.000 0.7244420 120 -2.760746 0.0066732
PCSEbaseline - PCSEpost MediaControl -2.000 0.6597071 120 -3.031649 0.0029806
PCSEbaseline - PCSEpost DialogueControl -1.125 0.6097088 120 -1.845143 0.0674822

Big 5 Personality and Cognitive Distortions

  • I conducted a planned multiple linear regression examining the predictive abilities of the Big 5 personality facets on baseline cognitive distortions. Lower neuroticism (reverse scored: β = -0.38, p < .001), extroversion (β = -0.23, p = 0.02), low conscientiousness (reverse scored: β = -0.23, p = 0.012), and low openness (β = 0.25, p = 0.005) predicted lower cognitive distortions, model R2adj = .264). Agreeableness did not significantly predict baseline cognitive distortions. These results suggest that low neuroticism, high extroversion, low openness, and low conscientiousness may serve as “protection” against cognitive distortions.
  Estimate Std. Error t value Pr(>|t|)
(Intercept) 36.03 7.744 4.653 1.068e-05
Extroversion -0.2873 0.1217 -2.362 0.02026
NeuroticismR -0.5127 0.1279 -4.01 0.0001216
Agreeableness 0.01396 0.1296 0.1077 0.9144
ConscientiousnessR -0.5649 0.2404 -2.35 0.02088
Openness 0.3146 0.1104 2.849 0.005388
Fitting linear model: ICDbaseline ~ Extroversion + NeuroticismR + Agreeableness + ConscientiousnessR + Openness
Observations Residual Std. Error \(R^2\) Adjusted \(R^2\)
100 6.31 0.294 0.2564
  Estimate Std. Error t value Pr(>|t|)
(Intercept) 36.15 7.624 4.742 7.435e-06
Extroversion -0.2867 0.1209 -2.371 0.01974
NeuroticismR -0.5103 0.1252 -4.076 9.516e-05
ConscientiousnessR -0.5537 0.2157 -2.567 0.01182
Openness 0.3153 0.1096 2.876 0.004971
Fitting linear model: ICDbaseline ~ Extroversion + NeuroticismR + ConscientiousnessR + Openness
Observations Residual Std. Error \(R^2\) Adjusted \(R^2\)
100 6.277 0.2939 0.2642
Fitting linear model: ICDbaseline ~ Extroversion + NeuroticismR + ConscientiousnessR + Openness (continued below)
  Estimate Standardized Std. Error t value
(Intercept) 36.15 0 7.624 4.742
Extroversion -0.2867 -0.2267 0.1209 -2.371
NeuroticismR -0.5103 -0.3826 0.1252 -4.076
ConscientiousnessR -0.5537 -0.2265 0.2157 -2.567
Openness 0.3153 0.2528 0.1096 2.876
  Pr(>|t|)
(Intercept) 7.435e-06
Extroversion 0.01974
NeuroticismR 9.516e-05
ConscientiousnessR 0.01182
Openness 0.004971

Final Comprehension Test

  • There were no differences between groups on the final comprehension test.
Analysis of Variance Table
  Df Sum Sq Mean Sq F value Pr(>F)
group 1 0.01424 0.01424 0.01576 0.9004
Residuals 98 88.55 0.9036 NA NA
Kruskal-Wallis rank sum test: Ftscore by group ## Cognitive Flexibility and Perceived Control over Stressful Events (baseline)
Test statistic df P value
0.00268 2 0.9987 

## 
     ##  Pearson's product-moment correlation
     ## 
     ## data:  CFIbaseline and PCSEbaseline
     ## t = 0.70445, df = 98, p-value = 0.4828
     ## alternative hypothesis: true correlation is not equal to 0
     ## 95 percent confidence interval:
     ##  -0.1272108  0.2637222
     ## sample estimates:
     ##        cor 
     ## 0.07098086
  • Interestingly, perceived control over stressful events and cognitive flexibility do not seem to be correlated, so they must be measuring different constructs.

ICD

  • 2(baseline and post) x 3(group: experimental, visual, dialogue) mixed ANOVA on Inventory of Cognitive distortions, main effect of baseline–post-intervention only. The main effect of group is not statistically significant, but there is a main effect of pre-post with a modest partial eta squared effect size (0.18). Here I reject the null, though my hypotheses were only somewhat confirmed (that actual vs anticipated cognitive distortions would become reduced but may possible demand effects; no difference by group).

ICD Effect Sizes
  Df Sum Sq Mean Sq F value Pr(>F)
group 2 565 282.5 2.534 0.0846
Residuals 97 10815 111.5 NA NA
ICD 1 108 108 21.14 1.286e-05
group:ICD 2 1.797 0.8984 0.1758 0.839
Residuals1 97 495.7 5.11 NA NA
ICD Baseline-Post Differences
Group Parameter Eta2_partial CI CI_low CI_high
subject group 0.0496484 0.95 0.0000000 0.1456095
subject:ICD ICD 0.1789704 0.95 0.0616203 0.3132258
subject:ICD group:ICD 0.0036120 0.95 0.0000000 0.0402571
ICD emmean SE df lower.CL upper.CL
ICDbaseline 16.19133 0.7635662 105.8742 14.67747 17.70520
ICDposttest 14.71867 0.7635662 105.8742 13.20480 16.23253

Conclusion

  • Overall improvements in perceived control over stress for the fictional character, and the experimental group tended to predict higher perceived control than the dialogue control group. Contrary to predictions, the experimental group did not have larger changes in perceived control over the visual control group, though both did have larger improvements in perceived control than the dialogue group.

  • There was an anticipated decrease in cognitive distortions overall from baseline to post-intervention measures. There was no main effect of group, which means that all the groups changed from baseline to post-intervention at the same rate.

  • Personality factors predicted cognitive distortions and specifically, in a high neuroticism subset, participants predicted a greater amount of perceived control.

  • Results are promising for both the utility of engagement strategies in online treatment and of a brief, single session online stress management intervention.

References

Butler, A. C., Godbole, N., & Marsh, E. J. (2013). Explanation feedback is better than correct answer feedback for promoting transfer of learning. Journal of Educational Psychology, 105(2), 290-298. https://doi.org/10.1037/a0031026

Kornell, N. (2014). Attempting to answer a meaningful question enhances subsequent learning even when feedback is delayed. Journal of Experimental Psychology: Learning, Memory, and Cognition, 40(1), 106-114. https://doi.org/10.1037/a0033699

Kornell, N., Klein, P. J., & Rawson, K. A. (2015). Retrieval attempts enhance learning, but retrieval success (versus failure) does not matter. Journal of Experimental Psychology: Learning, Memory, and Cognition, 41(1), 283-294. https://doi.org/10.1037/a0037850

Putnam, A. L., Nestojko, J. F., & Roediger, H. L. III. (2017). Improving student learning: Two strategies to make it stick. In J. C. Horvath, J. M. Lodge, & J. Hattie (Eds.), From the laboratory to the classroom: Translating science of learning for teachers (p. 94–121). Routledge/Taylor & Francis Group.

Roediger, H. L., & Karpicke, J. D. (2006). Test-enhanced learning. Psychological Science, 17(3), 249–255. https://doi.org/10.1111/j.1467-9280.2006.01693.x

Thorpe, S., Fize, D., & Marlot, C. (1996). Speed of processing in the human visual system. Nature, 381, 520-522.

Titov, N., Andrews, G., Choi, I., Schwencke, G., & Mahoney, A. (2008). Shyness 3: Randomized controlled trial of guided versus unguided Internet-based CBT for social phobia. Australian and New Zealand Journal of Psychiatry, 42, 1030-1040.