Longer-term outcomes question

This is our guiding research question.

RQ: Are situational (momentary) challenge and relevance associated with (a) individual (sustained) interest in STEM and future goals and aspirations related to STEM?

1. Predicting post-interest on the basis of challenge, relevance, and engagement, activity codes, and controls

Importance: Engagement, relevance, and challenge should positively predict post-interest. Engaging activities (lab, creating) should positively predict post-interest.

    overall_post_interest
    B std. Error p
Fixed Parts
(Intercept)   1.18 0.45 .009
pred_challenge   -0.14 0.10 .152
pred_relevance   -0.35 0.16 .024
pred_engagement   0.92 0.17 <.001
gender (M)   0.21 0.11 .053
scale(lab)   0.22 0.08 .005
scale(psl)   -0.04 0.10 .668
scale(basic)   0.03 0.10 .781
scale(create)   0.15 0.09 .103
scale(overall_pre_interest, scale = F)   0.47 0.07 <.001
prop_attend   0.51 0.42 .222
Random Parts
Nprogram_ID   9
ICCprogram_ID   0.000
Observations   138
R2 / Ω02   .582 / .582

Interpretation: When predicting post interest, we see that engagement, proportion of time spent in lab, and pre-interest are significant. We also see that gender is moderately (<.1) significant. Of note is that relevance coefficient is negative.

2. Same model, but without engagement

Importance: Relevance, and challenge should positively predict post-interest. Engaging activities (lab, creating) should positively predict post-interest.

    overall_post_interest
    B std. Error p
Fixed Parts
(Intercept)   2.03 0.47 <.001
pred_challenge   -0.21 0.11 .059
pred_relevance   0.31 0.11 .003
gender (M)   0.16 0.12 .173
scale(lab)   0.19 0.09 .030
scale(psl)   -0.05 0.11 .625
scale(basic)   -0.04 0.11 .741
scale(create)   0.07 0.10 .508
scale(overall_pre_interest, scale = F)   0.51 0.08 <.001
prop_attend   0.77 0.46 .094
Random Parts
Nprogram_ID   9
ICCprogram_ID   0.000
Observations   138
R2 / Ω02   .485 / .485

Interpretation: When predicting post interest, we see that relevance, proportion of time spent in lab, and pre-interest are significant. We also see that challenge, and attendance rate are moderately (<.1) significant. When we took out engagement, the coefficient of relevance became positive.

2A

Running model without relevance because of negative challenge.

    overall_post_interest
    B std. Error p
Fixed Parts
(Intercept)   2.39 0.47 <.001
pred_challenge   -0.03 0.09 .718
gender (M)   0.22 0.12 .069
scale(lab)   0.18 0.09 .040
scale(psl)   -0.04 0.11 .730
scale(basic)   -0.04 0.12 .702
scale(create)   0.07 0.11 .481
scale(overall_pre_interest, scale = F)   0.53 0.08 <.001
prop_attend   0.81 0.47 .086
Random Parts
Nprogram_ID   9
ICCprogram_ID   0.000
Observations   138
R2 / Ω02   .450 / .450

Interpretation: Noting that challenge remains negative alone.

2i

We are not very sure which to prefer; the model with engagement fits better:

## Data: mod_df
## Models:
## m4ai: overall_post_interest ~ pred_challenge + pred_relevance + gender + 
## m4ai:     scale(lab) + scale(psl) + scale(basic) + scale(create) + 
## m4ai:     scale(overall_pre_interest, scale = F) + prop_attend + (1 | 
## m4ai:     program_ID)
## m4a: overall_post_interest ~ pred_challenge + pred_relevance + pred_engagement + 
## m4a:     gender + scale(lab) + scale(psl) + scale(basic) + scale(create) + 
## m4a:     scale(overall_pre_interest, scale = F) + prop_attend + (1 | 
## m4a:     program_ID)
##      Df    AIC    BIC  logLik deviance  Chisq Chi Df Pr(>Chisq)    
## m4ai 12 295.01 330.14 -135.50   271.01                             
## m4a  13 268.43 306.48 -121.22   242.43 28.579      1  8.996e-08 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

We may want to look at proportion of time spent in class versus out of class.

2ii

Here are the correlations between the activity codes:

rowname fts lab create not_focused basic psl
fts -.14 .15 -.16 -.28 -.15
lab -.14 -.47 .39 -.27 .07
create .15 -.47 -.06 -.45 -.65
not_focused -.16 .39 -.06 -.72 -.65
basic -.28 -.27 -.45 -.72 .79
psl -.15 .07 -.65 -.65 .79

2B

Does removing activity codes change anything?
    overall_post_interest
    B std. Error p
Fixed Parts
(Intercept)   2.14 0.48 <.001
pred_challenge   -0.19 0.11 .083
pred_relevance   0.31 0.11 .004
gender (M)   0.17 0.12 .172
scale(overall_pre_interest, scale = F)   0.57 0.07 <.001
prop_attend   0.60 0.46 .198
Random Parts
Nprogram_ID   9
ICCprogram_ID   0.041
Observations   138
R2 / Ω02   .469 / .469

Interpretation: Noting that ICC increases when taking out activities.

3. Same, for future goals and plans

Importance: Relevance, and challenge should positively predict post-future goals and plans. Engaging activities (lab, creating) should positively predict post-future goals and plans.

    post_future_goals_plans
    B std. Error p
Fixed Parts
(Intercept)   1.52 0.57 .007
pred_challenge   -0.11 0.13 .420
pred_relevance   0.37 0.13 .006
gender (M)   0.38 0.15 .010
scale(lab)   0.04 0.10 .680
scale(psl)   0.06 0.13 .650
scale(basic)   0.01 0.14 .913
scale(create)   0.04 0.12 .750
prop_attend   0.45 0.54 .404
scale(pre_future_goals_plans, scale = F)   0.32 0.09 <.001
Random Parts
Nprogram_ID   9
ICCprogram_ID   0.000
Observations   130
R2 / Ω02   .317 / .317

Interpretation: When predicting future goals and plans, we see that relevance, gender, and pre future goals and plans are significant.

Reliabilities for new post-survey measures (need to think about pre-measures)

4. Adding in at program, teacher staff, and peer relations.

Importance: Relevance, and challenge should positively predict post-interest. Engaging activities (lab, creating) should positively predict post-interest. Would expect agency, teacher support, and peer relations to positively predict post-interest.

The only one that predicts on its own is agency, so we took out teacher support and peer relations.

The only thing missing is some measure of quality; should we add PQA?

    overall_post_interest
    B std. Error p
Fixed Parts
(Intercept)   1.58 0.50 .002
pred_challenge   -0.19 0.11 .075
pred_relevance   0.25 0.11 .023
post_agency   0.21 0.09 .025
gender (M)   0.16 0.12 .170
scale(lab)   0.22 0.09 .010
scale(psl)   -0.04 0.11 .724
scale(basic)   -0.02 0.11 .882
scale(create)   0.11 0.10 .302
scale(overall_pre_interest, scale = F)   0.51 0.07 <.001
prop_attend   0.83 0.45 .065
Random Parts
Nprogram_ID   9
ICCprogram_ID   0.000
Observations   138
R2 / Ω02   .505 / .505

Interpretation: When predicting post interest, we see that relevance, agency, proportion of time spent in lab, and pre interest are significant. We also see that challenge, and attendance rate are moderately (<.1) significant.

4i

    overall_post_interest
    B std. Error p
Fixed Parts
(Intercept)   1.38 0.57 .015
pred_challenge   -0.18 0.11 .109
pred_relevance   0.24 0.11 .036
post_agency   0.18 0.11 .118
post_feelings_of_staff   0.01 0.11 .948
post_peer_relations   0.11 0.14 .419
gender (M)   0.14 0.13 .253
scale(lab)   0.21 0.09 .017
scale(psl)   -0.01 0.11 .904
scale(basic)   -0.04 0.12 .749
scale(create)   0.12 0.11 .280
scale(overall_pre_interest, scale = F)   0.50 0.08 <.001
prop_attend   0.81 0.46 .077
Random Parts
Nprogram_ID   9
ICCprogram_ID   0.000
Observations   137
R2 / Ω02   .504 / .504