Momentary outcomes question

This is our guiding research question.

How do in-the-moment experiences of youth (i.e., challenge and relevance) and other momentary factors cultivate situational interest and engagement in STEM activities?

Overall engagement now includes hard working, concentrating, enjoy, and interest (this is also the case for the longer-term outcomes study).

1. Models predicting interest

    interest
    B std. Error p
Fixed Parts
(Intercept)   0.76 0.23 <.001
challenge   0.04 0.02 .022
relevance   0.60 0.02 <.001
gender (M)   -0.05 0.06 .391
classroom_versus_field_enrichment   0.07 0.05 .181
CLASS_comp   -0.00 0.02 .910
youth_activity_rc (Basic Skills Activity)   -0.12 0.06 .053
youth_activity_rc (Creating Product)   -0.10 0.07 .136
youth_activity_rc (Field Trip Speaker)   -0.01 0.11 .949
youth_activity_rc (Lab Activity)   -0.02 0.11 .838
youth_activity_rc (Program Staff Led)   -0.10 0.07 .159
overall_pre_interest   0.07 0.04 .047
youth_development_overall   0.01 0.01 .095
prop_attend   0.23 0.20 .262
Random Parts
Nbeep_ID_new   227
Nparticipant_ID   176
Nprogram_ID   9
ICCbeep_ID_new   0.048
ICCparticipant_ID   0.132
ICCprogram_ID   0.021
Observations   2428
R2 / Ω02   .545 / .542

This model suggests that:

  • relevance and challenge both predict interest, though relevance is much larger in terms of the magnitude of the effect
  • all of the activities have negative coefficients; that’s weird
  • pre-interest seems to be an important covariate
  • the youth development measure from PQA has a very small coefficient

1a

What if we remove youth activities?

    interest
    B std. Error p
Fixed Parts
(Intercept)   0.77 0.22 <.001
challenge   0.04 0.02 .025
relevance   0.60 0.02 <.001
gender (M)   -0.05 0.06 .432
classroom_versus_field_enrichment   0.06 0.05 .235
CLASS_comp   -0.02 0.02 .302
overall_pre_interest   0.07 0.04 .043
youth_development_overall   0.02 0.01 .074
prop_attend   0.23 0.20 .249
Random Parts
Nbeep_ID_new   230
Nparticipant_ID   176
Nprogram_ID   9
ICCbeep_ID_new   0.048
ICCparticipant_ID   0.131
ICCprogram_ID   0.025
Observations   2457
R2 / Ω02   .545 / .541

This model shows basically a similar story.

1c

What if we remove challenge and relevance?

    interest
    B std. Error p
Fixed Parts
(Intercept)   1.96 0.36 <.001
gender (M)   0.10 0.10 .347
classroom_versus_field_enrichment   0.05 0.06 .438
CLASS_comp   0.02 0.03 .518
youth_activity_rc (Basic Skills Activity)   -0.05 0.07 .422
youth_activity_rc (Creating Product)   0.02 0.08 .841
youth_activity_rc (Field Trip Speaker)   0.12 0.13 .354
youth_activity_rc (Lab Activity)   0.00 0.12 .983
youth_activity_rc (Program Staff Led)   -0.03 0.08 .660
overall_pre_interest   0.14 0.06 .020
youth_development_overall   0.01 0.01 .219
prop_attend   0.31 0.34 .368
Random Parts
Nbeep_ID_new   227
Nparticipant_ID   176
Nprogram_ID   9
ICCbeep_ID_new   0.035
ICCparticipant_ID   0.325
ICCprogram_ID   0.027
Observations   2428
R2 / Ω02   .453 / .444

These activities seem to make more sense now. What if we remove the random beep effect?

1d

    interest
    B std. Error p
Fixed Parts
(Intercept)   0.75 0.22 <.001
gender (M)   -0.05 0.06 .398
challenge   0.04 0.02 .013
relevance   0.60 0.02 <.001
classroom_versus_field_enrichment   0.09 0.04 .036
CLASS_comp   0.00 0.02 .827
youth_activity_rc (Basic Skills Activity)   -0.12 0.05 .017
youth_activity_rc (Creating Product)   -0.13 0.06 .023
youth_activity_rc (Field Trip Speaker)   -0.00 0.08 .954
youth_activity_rc (Lab Activity)   -0.03 0.09 .750
youth_activity_rc (Program Staff Led)   -0.09 0.05 .091
overall_pre_interest   0.07 0.04 .054
youth_development_overall   0.01 0.01 .096
prop_attend   0.22 0.20 .271
Random Parts
Nparticipant_ID   176
Nprogram_ID   9
ICCparticipant_ID   0.129
ICCprogram_ID   0.027
Observations   2428
R2 / Ω02   .499 / .498

Not a huge difference. What if we remove some of the covariates / controls but otherwise run model 1?

1e

    interest
    B std. Error p
Fixed Parts
(Intercept)   0.99 0.14 <.001
gender (M)   -0.05 0.06 .415
challenge   0.06 0.02 .002
relevance   0.59 0.02 <.001
classroom_versus_field_enrichment   0.07 0.05 .201
youth_activity_rc (Basic Skills Activity)   -0.11 0.06 .049
youth_activity_rc (Creating Product)   -0.05 0.06 .430
youth_activity_rc (Field Trip Speaker)   0.01 0.11 .960
youth_activity_rc (Lab Activity)   0.00 0.11 .982
youth_activity_rc (Program Staff Led)   -0.07 0.06 .316
overall_pre_interest   0.08 0.04 .028
Random Parts
Nbeep_ID_new   235
Nparticipant_ID   180
Nprogram_ID   9
ICCbeep_ID_new   0.050
ICCparticipant_ID   0.128
ICCprogram_ID   0.019
Observations   2583
R2 / Ω02   .541 / .537

Basic skills have a negative relationship with interest; otherwise, not large changes. What if we remove challenge and relevance?

1f

    interest
    B std. Error p
Fixed Parts
(Intercept)   2.32 0.21 <.001
gender (M)   0.11 0.10 .281
classroom_versus_field_enrichment   0.04 0.06 .506
youth_activity_rc (Basic Skills Activity)   -0.03 0.06 .630
youth_activity_rc (Creating Product)   0.09 0.07 .152
youth_activity_rc (Field Trip Speaker)   0.13 0.13 .294
youth_activity_rc (Lab Activity)   0.05 0.12 .672
youth_activity_rc (Program Staff Led)   0.01 0.07 .890
overall_pre_interest   0.15 0.06 .012
Random Parts
Nbeep_ID_new   235
Nparticipant_ID   180
Nprogram_ID   9
ICCbeep_ID_new   0.040
ICCparticipant_ID   0.324
ICCprogram_ID   0.022
Observations   2583
R2 / Ω02   .452 / .442

Not a big difference, again. And if we remove the momentary effect?

1g

    interest
    B std. Error p
Fixed Parts
(Intercept)   2.83 0.08 <.001
youth_activity_rc (Basic Skills Activity)   -0.00 0.06 .974
youth_activity_rc (Creating Product)   0.13 0.06 .037
youth_activity_rc (Field Trip Speaker)   0.13 0.12 .262
youth_activity_rc (Lab Activity)   0.05 0.12 .677
youth_activity_rc (Program Staff Led)   0.04 0.07 .610
Random Parts
Nbeep_ID_new   235
Nparticipant_ID   203
Nprogram_ID   9
ICCbeep_ID_new   0.036
ICCparticipant_ID   0.325
ICCprogram_ID   0.021
Observations   2826
R2 / Ω02   .450 / .441

Let’s take a look at engagement as the outcome.

2

Same as model 1, but with engagement as the outcome.

    df$overall_engagement
    B std. Error p
Fixed Parts
(Intercept)   0.89 0.20 <.001
challenge   0.05 0.01 <.001
relevance   0.54 0.02 <.001
gender (M)   -0.07 0.05 .169
classroom_versus_field_enrichment   0.10 0.04 .010
CLASS_comp   0.01 0.02 .781
youth_activity_rc (Basic Skills Activity)   -0.00 0.04 .965
youth_activity_rc (Creating Product)   -0.06 0.05 .255
youth_activity_rc (Field Trip Speaker)   0.04 0.08 .614
youth_activity_rc (Lab Activity)   0.05 0.08 .528
youth_activity_rc (Program Staff Led)   -0.07 0.05 .172
overall_pre_interest   0.08 0.03 .020
youth_development_overall   0.01 0.01 .371
prop_attend   0.17 0.18 .344
Random Parts
Nbeep_ID_new   227
Nparticipant_ID   176
Nprogram_ID   9
ICCbeep_ID_new   0.054
ICCparticipant_ID   0.253
ICCprogram_ID   0.020
Observations   2428
R2 / Ω02   .687 / .685

This model seems to tell us:

  • Challenge and relevance demonstrate a similar pattern as for interest (as we would expect)
  • Classroom experiences seem to be more engaging
  • Pre-interest is an important covariate
  • Activities have more sensible relations with engagement than with interest; though the coefficients are not significant.

2a

What if we remove challenge and relevance?

    df$overall_engagement
    B std. Error p
Fixed Parts
(Intercept)   2.00 0.32 <.001
gender (M)   0.06 0.09 .529
classroom_versus_field_enrichment   0.08 0.05 .065
CLASS_comp   0.02 0.02 .252
youth_activity_rc (Basic Skills Activity)   0.06 0.05 .256
youth_activity_rc (Creating Product)   0.05 0.06 .429
youth_activity_rc (Field Trip Speaker)   0.15 0.10 .113
youth_activity_rc (Lab Activity)   0.08 0.09 .397
youth_activity_rc (Program Staff Led)   -0.01 0.06 .857
overall_pre_interest   0.13 0.06 .016
youth_development_overall   0.00 0.01 .682
prop_attend   0.27 0.31 .386
Random Parts
Nbeep_ID_new   227
Nparticipant_ID   176
Nprogram_ID   9
ICCbeep_ID_new   0.031
ICCparticipant_ID   0.439
ICCprogram_ID   0.025
Observations   2428
R2 / Ω02   .550 / .545

About the same.

2b

What if we remove the random beep effect from model 2?

    df$overall_engagement
    B std. Error p
Fixed Parts
(Intercept)   0.90 0.20 <.001
challenge   0.05 0.01 <.001
relevance   0.54 0.02 <.001
gender (M)   -0.07 0.05 .188
classroom_versus_field_enrichment   0.12 0.03 <.001
CLASS_comp   0.00 0.01 .736
youth_activity_rc (Basic Skills Activity)   -0.00 0.03 .918
youth_activity_rc (Creating Product)   -0.08 0.04 .038
youth_activity_rc (Field Trip Speaker)   0.04 0.06 .436
youth_activity_rc (Lab Activity)   0.04 0.06 .485
youth_activity_rc (Program Staff Led)   -0.06 0.04 .092
overall_pre_interest   0.08 0.03 .022
youth_development_overall   0.00 0.00 .358
prop_attend   0.17 0.18 .349
Random Parts
Nparticipant_ID   176
Nprogram_ID   9
ICCparticipant_ID   0.249
ICCprogram_ID   0.030
Observations   2428
R2 / Ω02   .647 / .647

Some of the coefficients for momentary predicts change a bit (creating product is significantly less engaging than not focused); program staff led demonstrates a similar pattern.

2c

Lets look at the same model but with challenge and relevance removed.

    df$overall_engagement
    B std. Error p
Fixed Parts
(Intercept)   1.98 0.32 <.001
gender (M)   0.07 0.09 .473
classroom_versus_field_enrichment   0.10 0.04 .006
CLASS_comp   0.03 0.02 .096
youth_activity_rc (Basic Skills Activity)   0.06 0.04 .123
youth_activity_rc (Creating Product)   0.02 0.05 .602
youth_activity_rc (Field Trip Speaker)   0.16 0.07 .020
youth_activity_rc (Lab Activity)   0.08 0.07 .279
youth_activity_rc (Program Staff Led)   -0.00 0.04 .930
overall_pre_interest   0.14 0.06 .016
youth_development_overall   0.00 0.01 .735
prop_attend   0.27 0.31 .388
Random Parts
Nparticipant_ID   176
Nprogram_ID   9
ICCparticipant_ID   0.435
ICCprogram_ID   0.033
Observations   2428
R2 / Ω02   .504 / .501

These seem a bit more sensible; field trip speaker is associated with higher engagement; otherwise, there are similar patterns.

We may be having too many predictors. What can we remove?

2d

Let’s consider removing some of the highly not significant predictors: CLASS composite, youth development from PQA, maybe prop attend from model 2.

    df$overall_engagement
    B std. Error p
Fixed Parts
(Intercept)   0.89 0.20 <.001
challenge   0.05 0.01 <.001
relevance   0.54 0.02 <.001
gender (M)   -0.07 0.05 .169
classroom_versus_field_enrichment   0.10 0.04 .010
CLASS_comp   0.01 0.02 .781
youth_activity_rc (Basic Skills Activity)   -0.00 0.04 .965
youth_activity_rc (Creating Product)   -0.06 0.05 .255
youth_activity_rc (Field Trip Speaker)   0.04 0.08 .614
youth_activity_rc (Lab Activity)   0.05 0.08 .528
youth_activity_rc (Program Staff Led)   -0.07 0.05 .172
overall_pre_interest   0.08 0.03 .020
youth_development_overall   0.01 0.01 .371
prop_attend   0.17 0.18 .344
Random Parts
Nbeep_ID_new   227
Nparticipant_ID   176
Nprogram_ID   9
ICCbeep_ID_new   0.054
ICCparticipant_ID   0.253
ICCprogram_ID   0.020
Observations   2428
R2 / Ω02   .687 / .685

These seem to change a bit, but not much. What if we remove challenge and relevance?

2e

    df$overall_engagement
    B std. Error p
Fixed Parts
(Intercept)   2.33 0.18 <.001
gender (M)   0.06 0.09 .516
classroom_versus_field_enrichment   0.07 0.04 .095
youth_activity_rc (Basic Skills Activity)   0.08 0.05 .070
youth_activity_rc (Creating Product)   0.11 0.05 .021
youth_activity_rc (Field Trip Speaker)   0.16 0.09 .093
youth_activity_rc (Lab Activity)   0.12 0.09 .167
youth_activity_rc (Program Staff Led)   0.01 0.05 .862
overall_pre_interest   0.13 0.05 .017
Random Parts
Nbeep_ID_new   235
Nparticipant_ID   180
Nprogram_ID   9
ICCbeep_ID_new   0.032
ICCparticipant_ID   0.440
ICCprogram_ID   0.018
Observations   2583
R2 / Ω02   .546 / .541

If we remove challenge and relevance, creating product has a positive relation, and field trip speaker is approaching significance.