Momentary outcomes question (RQ #3)

How do in-the-moment experiences of youth (i.e., challenge, relevance, learning, and affect) cultivate situational interest and engagement in STEM activities?

Broadly, these in-the-moment experiences predict situational interest and engagement, though the subsequent models may be of greater interest to us (and our audience).

Correlations for situational experiences (interest and engagement)

rowname overall_engagement interest challenge relevance learning positive_affect
overall_engagement
interest .69
challenge .30 .28
relevance .65 .61 .39
learning .68 .56 .30 .65
positive_affect .65 .56 .27 .52 .48

Null models for situational experiences (interest and engagement; with CLASS composite)

m30 <- lmer(interest ~ 1 +
                 overall_pre_interest + 
                 (1|program_ID) + (1|participant_ID) + (1|beep_ID_new),
             data = df)

sjPlot::sjt.lmer(m30, p.kr = F, show.re.var = F, show.ci = F, show.se = T)
## Warning in checkMatrixPackageVersion(): Package version inconsistency detected.
## TMB was built with Matrix version 1.2.10
## Current Matrix version is 1.2.11
## Please re-install 'TMB' from source or restore original 'Matrix' package
    interest
    B std. Error p
Fixed Parts
(Intercept)   2.44 0.19 <.001
overall_pre_interest   0.14 0.06 .018
Random Parts
Nbeep_ID_new   248
Nparticipant_ID   181
Nprogram_ID   9
ICCbeep_ID_new   0.040
ICCparticipant_ID   0.328
ICCprogram_ID   0.017
Observations   2738
R2 / Ω02   .448 / .438

The ICC for the null model for interest shows high levels of unexplained participant-level variability and small amounts of momentary and program-level variability.

m30i <- lmer(overall_engagement ~ 1 + 
                  overall_pre_interest + 
                  (1|program_ID) + (1|participant_ID) + (1|beep_ID_new),
              data = df)

sjPlot::sjt.lmer(m30i, p.kr = F, show.re.var = F, show.ci = F, show.se = T)
    overall_engagement
    B std. Error p
Fixed Parts
(Intercept)   2.50 0.16 <.001
overall_pre_interest   0.11 0.05 .030
Random Parts
Nbeep_ID_new   248
Nparticipant_ID   181
Nprogram_ID   9
ICCbeep_ID_new   0.034
ICCparticipant_ID   0.429
ICCprogram_ID   0.000
Observations   2738
R2 / Ω02   .523 / .518

The ICC for the null for model engagement shows high levels of unexplained participant-level variability, small amounts of momentary variability, and no program-level variability.

Models for situational experiences (interest and engagement; with CLASS composite)

Predicting interest (sans challenge and relevance):

m3a <- lmer(interest ~ 1 +
                gender +
                classroom_versus_field_enrichment +
                CLASS_comp + 
                youth_activity_rc +
                overall_pre_interest +
                (1|program_ID) + (1|participant_ID) + (1|beep_ID_new),
            data = df)

sjPlot::sjt.lmer(m3a, p.kr = F, show.re.var = F, show.ci = F, show.se = T)
    interest
    B std. Error p
Fixed Parts
(Intercept)   2.22 0.22 <.001
gender (M)   0.11 0.10 .283
classroom_versus_field_enrichment   0.05 0.06 .425
CLASS_comp   0.03 0.03 .177
youth_activity_rc (Basic Skills Activity)   -0.07 0.07 .303
youth_activity_rc (Creating Product)   0.04 0.07 .628
youth_activity_rc (Field Trip Speaker)   0.11 0.13 .367
youth_activity_rc (Lab Activity)   0.00 0.12 .987
youth_activity_rc (Program Staff Led)   -0.04 0.08 .554
overall_pre_interest   0.14 0.06 .016
Random Parts
Nbeep_ID_new   228
Nparticipant_ID   180
Nprogram_ID   9
ICCbeep_ID_new   0.036
ICCparticipant_ID   0.325
ICCprogram_ID   0.023
Observations   2483
R2 / Ω02   .452 / .442

Predicting interest (with challenge and relevance):

m3b <- lmer(interest ~ 1 +
                challenge + relevance +
                gender +
                classroom_versus_field_enrichment +
                CLASS_comp + 
                youth_activity_rc +
                overall_pre_interest +
                (1|program_ID) + (1|participant_ID) + (1|beep_ID_new),
            data = df)

sjPlot::sjt.lmer(m3b, p.kr = F, show.re.var = F, show.ci = F, show.se = T)
    interest
    B std. Error p
Fixed Parts
(Intercept)   0.96 0.15 <.001
challenge   0.05 0.02 .007
relevance   0.60 0.02 <.001
gender (M)   -0.04 0.06 .453
classroom_versus_field_enrichment   0.07 0.05 .167
CLASS_comp   0.02 0.02 .464
youth_activity_rc (Basic Skills Activity)   -0.13 0.06 .030
youth_activity_rc (Creating Product)   -0.08 0.07 .248
youth_activity_rc (Field Trip Speaker)   -0.01 0.11 .959
youth_activity_rc (Lab Activity)   -0.02 0.11 .854
youth_activity_rc (Program Staff Led)   -0.10 0.07 .133
overall_pre_interest   0.07 0.04 .045
Random Parts
Nbeep_ID_new   228
Nparticipant_ID   180
Nprogram_ID   9
ICCbeep_ID_new   0.049
ICCparticipant_ID   0.131
ICCprogram_ID   0.018
Observations   2483
R2 / Ω02   .543 / .539

This model suggests that relevance and learing predict momentary interest; and that basic skills activities are less interesting to learners. Much of the individual-level variability has been explained, though not very much has been explained at the momentary and program levels.

Predicting overall engagement (sans challenge and relevance):

m4a <- lmer(overall_engagement ~ 1 + 
                gender + 
                classroom_versus_field_enrichment +
                CLASS_comp + 
                youth_activity_rc +
                overall_pre_interest + 
                (1|program_ID) + (1|participant_ID) + (1|beep_ID_new),
            data = df)

sjPlot::sjt.lmer(m4a, p.kr = F, show.re.var = F, show.ci = F, show.se = T)
    overall_engagement
    B std. Error p
Fixed Parts
(Intercept)   2.25 0.19 <.001
gender (M)   0.05 0.09 .622
classroom_versus_field_enrichment   0.09 0.04 .046
CLASS_comp   0.03 0.02 .118
youth_activity_rc (Basic Skills Activity)   0.09 0.05 .079
youth_activity_rc (Creating Product)   0.07 0.06 .233
youth_activity_rc (Field Trip Speaker)   0.15 0.09 .097
youth_activity_rc (Lab Activity)   0.10 0.09 .283
youth_activity_rc (Program Staff Led)   -0.02 0.06 .687
overall_pre_interest   0.12 0.05 .023
Random Parts
Nbeep_ID_new   228
Nparticipant_ID   180
Nprogram_ID   9
ICCbeep_ID_new   0.023
ICCparticipant_ID   0.426
ICCprogram_ID   0.015
Observations   2483
R2 / Ω02   .523 / .517

Predicting overall engagement (sans challenge and relevance):

m4b <- lmer(overall_engagement ~ 1 + 
                challenge + relevance +
                gender + 
                classroom_versus_field_enrichment +
                CLASS_comp + 
                youth_activity_rc +
                overall_pre_interest + 
                (1|program_ID) + (1|participant_ID) + (1|beep_ID_new),
            data = df)

sjPlot::sjt.lmer(m4b, p.kr = F, show.re.var = F, show.ci = F, show.se = T)
    overall_engagement
    B std. Error p
Fixed Parts
(Intercept)   1.05 0.13 <.001
challenge   0.06 0.01 <.001
relevance   0.53 0.02 <.001
gender (M)   -0.09 0.06 .118
classroom_versus_field_enrichment   0.10 0.04 .007
CLASS_comp   0.01 0.02 .381
youth_activity_rc (Basic Skills Activity)   0.03 0.04 .516
youth_activity_rc (Creating Product)   -0.03 0.05 .490
youth_activity_rc (Field Trip Speaker)   0.05 0.08 .512
youth_activity_rc (Lab Activity)   0.07 0.08 .378
youth_activity_rc (Program Staff Led)   -0.07 0.05 .139
overall_pre_interest   0.07 0.03 .038
Random Parts
Nbeep_ID_new   228
Nparticipant_ID   180
Nprogram_ID   9
ICCbeep_ID_new   0.042
ICCparticipant_ID   0.258
ICCprogram_ID   0.012
Observations   2483
R2 / Ω02   .653 / .651

This model suggests that releavence and learning predict momentary engagement; and that challenge predicts momentary engagement, though with a very small effect. Classroom settings are associated with higher engagement. Some of the individual-level variability has been explained, though not very much has been explained at the momentary and program levels.