Abstract from proposal

Out-of-school time (OST) STEM programs have started focusing on STEM content due to the lack of individuals choosing to focus on STEM careers. Although adaptive outcomes have been found while youth are attending OST STEM programs, little research has focused on outcomes after the program has ended. Often, the goal of OST STEM programs is to increase youth’s long-term interest and competence in STEM. Therefore, examining youth interest and competence in STEM after spending time in the program is key. Framing the study around Emergent Motivation Theory (Csikszentmihalyi, 1990), we use a profile-oriented approach to investigate the relationship between youths’ momentary engagement and their interest and perceived competence at program completion. Research questions include: 1) What momentary profiles emerge? 2) What profiles are predictive of interest and perceived competence after attending an OST STEM program?

Data were collected from 203 youth in nine OST programs, each lasting four weeks. Through an Experience Sampling Method (ESM) approach, youth were signaled through mobile phones, yielding 2,463 total ESM responses. Youth were asked to complete a survey before and after the program. A two-step cluster analysis was used to identity momentary profiles. Multilevel Modeling was used to account for the nesting of momentary responses within the nine OST STEM programs.

Four profiles were selected on the basis of fit indices, cross-validation, and concerns of parsimony and interpretability in light of theoretical accounts (See Figure 1). The profiles represent both the conditions of engagement (as indicated by students’ perceptions of challenge and skill) and the experience of engagement (as indicated by behavioral, cognitive, and affective dimensions). For subsequent analyses, the profile assignments were aggregated at the individual-level as proportions, so that each student had a value between 0 and 1 representing the proportion of their responses in each of the profiles. Results of multilevel modeling with Universally low specified as the comparison group, suggest that the proportion of youth’s responses in Skilled but not challenged and Fully engaged predicted positive and statistically significant changes in interest and competence. However, the magnitude of the effect for the relationship between Skilled but not challenged and change in interest (B = 1.01) and competence (B = 0.75) was much stronger than Fully engaged, with more modest relations with interest (B = 0.76) and competence (B = 0.55).

This study provides evidence that in the context of OST STEM programs, not only engagement, but also its conditions - perceptions of challenge and skill - foster changes in youth’s interest and competence. Although spending more time in profiles with high levels of challenge, skill, and engagement was associated with positive, significant changes in interest and competence, these changes were smaller than those in the profile with more modest levels of engagement, high levels of skill, and low levels of challenge, pointing to potentially unique features of the OST context. The use of ESM in combination with a person-oriented approach helped to provide a richer and more complete understanding of outcomes from engaging experiences and their conditions.

Background

This document outlines attempts to model relations between the engagement of youth in summer STEM programs (in-the-moment, measured using the experience sampling method) and how their engagement relates to their post-program interest in STEM.

Model 0

This model uses Maximum Likelihood (ML) estimation. Note that this model does not account for the error in the predictions for engagement when relating repeated measures engagement and post interest. The first (model 0a) does not include any covariates while the second (model 0b) adds gender, under-represented minority (URM) status, and pre-interest.

Model 0a

## Linear mixed model fit by REML ['lmerMod']
## Formula: rm_engagement ~ 1 + (1 | participant_ID) + (1 | program_ID)
##    Data: d
## 
## REML criterion at convergence: 6298.8
## 
## Scaled residuals: 
##     Min      1Q  Median      3Q     Max 
## -4.4043 -0.5274  0.1040  0.5904  3.6900 
## 
## Random effects:
##  Groups         Name        Variance Std.Dev.
##  participant_ID (Intercept) 0.3244   0.5695  
##  program_ID     (Intercept) 0.0130   0.1140  
##  Residual                   0.4124   0.6422  
## Number of obs: 2970, groups:  participant_ID, 203; program_ID, 9
## 
## Fixed effects:
##             Estimate Std. Error t value
## (Intercept)   2.8566     0.0573   49.85
## Linear mixed model fit by REML ['lmerMod']
## Formula: post_interest ~ 1 + rm_engagement_BLUP + (1 | program_ID)
##    Data: d_for_m1
## 
## REML criterion at convergence: 389.5
## 
## Scaled residuals: 
##      Min       1Q   Median       3Q      Max 
## -3.14505 -0.58832  0.07935  0.68551  2.28150 
## 
## Random effects:
##  Groups     Name        Variance Std.Dev.
##  program_ID (Intercept) 0.2028   0.4503  
##  Residual               0.5999   0.7745  
## Number of obs: 159, groups:  program_ID, 9
## 
## Fixed effects:
##                    Estimate Std. Error t value
## (Intercept)          3.0744     0.1629  18.869
## rm_engagement_BLUP   0.4818     0.1160   4.152
## 
## Correlation of Fixed Effects:
##             (Intr)
## rm_ngg_BLUP -0.012
##               Effect   Rsq upper.CL lower.CL
## 1              Model 0.089    0.186    0.024
## 2 rm_engagement_BLUP 0.089    0.186    0.024
## correlation between BLUP and interest 0.2983287
## Linear mixed model fit by REML ['lmerMod']
## Formula: rm_engagement ~ 1 + (1 | participant_ID) + (1 | program_ID)
##    Data: d
## 
## REML criterion at convergence: 6298.8
## 
## Scaled residuals: 
##     Min      1Q  Median      3Q     Max 
## -4.4043 -0.5274  0.1040  0.5904  3.6900 
## 
## Random effects:
##  Groups         Name        Variance Std.Dev.
##  participant_ID (Intercept) 0.3244   0.5695  
##  program_ID     (Intercept) 0.0130   0.1140  
##  Residual                   0.4124   0.6422  
## Number of obs: 2970, groups:  participant_ID, 203; program_ID, 9
## 
## Fixed effects:
##             Estimate Std. Error t value
## (Intercept)   2.8566     0.0573   49.85
## Linear mixed model fit by REML ['lmerMod']
## Formula: post_interest ~ 1 + rm_engagement_BLUP + gender_female + urm +  
##     pre_interest + (1 | program_ID)
##    Data: d_for_m2
## 
## REML criterion at convergence: 307
## 
## Scaled residuals: 
##     Min      1Q  Median      3Q     Max 
## -4.2017 -0.4728  0.0722  0.5639  2.7381 
## 
## Random effects:
##  Groups     Name        Variance Std.Dev.
##  program_ID (Intercept) 0.0249   0.1578  
##  Residual               0.4681   0.6842  
## Number of obs: 141, groups:  program_ID, 9
## 
## Fixed effects:
##                    Estimate Std. Error t value
## (Intercept)          1.6870     0.2782   6.064
## rm_engagement_BLUP   0.4669     0.1096   4.261
## gender_female       -0.0933     0.1208  -0.772
## urm                 -0.2056     0.1630  -1.262
## pre_interest         0.5309     0.0731   7.263
## 
## Correlation of Fixed Effects:
##             (Intr) r__BLU gndr_f urm   
## rm_ngg_BLUP  0.056                     
## gender_feml -0.210  0.158              
## urm         -0.458 -0.028 -0.118       
## pre_interst -0.809 -0.106  0.047 -0.005
##               Effect   Rsq upper.CL lower.CL
## 1              Model 0.392    0.510    0.288
## 5       pre_interest 0.296    0.415    0.184
## 2 rm_engagement_BLUP 0.112    0.221    0.034
## 4                urm 0.011    0.071    0.000
## 3      gender_female 0.004    0.052    0.000
## correlation between BLUP and interest 0.334664

Model 1

These models use Markov Chain Monte Carlo (MCMC) estimation (via the MCMCglmm package). The first (model 1a) does not include any covariates while the second (model 1b) adds gender, under-represented minority (URM) status, and pre-interest.

Model 1a

This is a bivariate model predicting repeated measures engagement (rm_engagement) and post interest (post_interest). Two random effects are used, one for the participant random effect, and one for the program random effect. The correlation among the two outcomes is examined and appears to take on a positive, non-zero value with a 95% credible interval (slightly different from a confidence interval) that does not include zero.

## 
##  Iterations = 3001:12991
##  Thinning interval  = 10
##  Sample size  = 1000 
## 
##  DIC: 10984.07 
## 
##  G-structure:  ~us(trait):participant_ID
## 
##                                                      post.mean l-95% CI
## traitrm_engagement:traitrm_engagement.participant_ID    0.3332  0.25850
## traitpost_interest:traitrm_engagement.participant_ID    0.1244  0.05416
## traitrm_engagement:traitpost_interest.participant_ID    0.1244  0.05416
## traitpost_interest:traitpost_interest.participant_ID    0.7240  0.58466
##                                                      u-95% CI eff.samp
## traitrm_engagement:traitrm_engagement.participant_ID   0.4037    749.5
## traitpost_interest:traitrm_engagement.participant_ID   0.2030   1000.0
## traitrm_engagement:traitpost_interest.participant_ID   0.2030   1000.0
## traitpost_interest:traitpost_interest.participant_ID   0.8707   1000.0
## 
##                ~us(trait):program_ID
## 
##                                                  post.mean   l-95% CI
## traitrm_engagement:traitrm_engagement.program_ID   0.02650  1.438e-10
## traitpost_interest:traitrm_engagement.program_ID   0.01483 -1.660e-01
## traitrm_engagement:traitpost_interest.program_ID   0.01483 -1.660e-01
## traitpost_interest:traitpost_interest.program_ID   1.91005  5.552e-01
##                                                  u-95% CI eff.samp
## traitrm_engagement:traitrm_engagement.program_ID   0.0853    154.8
## traitpost_interest:traitrm_engagement.program_ID   0.1991    865.2
## traitrm_engagement:traitpost_interest.program_ID   0.1991    865.2
## traitpost_interest:traitpost_interest.program_ID   4.0654   1000.0
## 
##  R-structure:  ~idh(trait):units
## 
##                          post.mean l-95% CI u-95% CI eff.samp
## traitrm_engagement.units    0.4124   0.3906   0.4344     1000
## traitpost_interest.units    0.0001   0.0001   0.0001        0
## 
##  Location effects: cbind(rm_engagement, post_interest) ~ trait 
## 
##                    post.mean l-95% CI u-95% CI eff.samp  pMCMC    
## (Intercept)           2.8585   2.7323   3.0195     1964 <0.001 ***
## traitpost_interest    0.1797  -0.6851   1.0355     1216  0.662    
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

##      var1 
## 0.2480669

Model 2a

This model is identical to model 1a, except it adds fixed effects for a) pre-interest (the same measure as for post-interest, but given to youth before, rather than after).

## 
##  Iterations = 3001:12991
##  Thinning interval  = 10
##  Sample size  = 1000 
## 
##  DIC: 9872.584 
## 
##  G-structure:  ~us(trait):participant_ID
## 
##                                                      post.mean l-95% CI
## traitrm_engagement:traitrm_engagement.participant_ID    0.3403  0.26686
## traitpost_interest:traitrm_engagement.participant_ID    0.1261  0.05747
## traitrm_engagement:traitpost_interest.participant_ID    0.1261  0.05747
## traitpost_interest:traitpost_interest.participant_ID    0.6076  0.47429
##                                                      u-95% CI eff.samp
## traitrm_engagement:traitrm_engagement.participant_ID   0.4155    902.7
## traitpost_interest:traitrm_engagement.participant_ID   0.2023   1000.0
## traitrm_engagement:traitpost_interest.participant_ID   0.2023   1000.0
## traitpost_interest:traitpost_interest.participant_ID   0.7392   1000.0
## 
##                ~us(trait):program_ID
## 
##                                                  post.mean   l-95% CI
## traitrm_engagement:traitrm_engagement.program_ID  0.028194  1.416e-11
## traitpost_interest:traitrm_engagement.program_ID -0.006809 -1.967e-01
## traitrm_engagement:traitpost_interest.program_ID -0.006809 -1.967e-01
## traitpost_interest:traitpost_interest.program_ID  2.091860  5.544e-01
##                                                  u-95% CI eff.samp
## traitrm_engagement:traitrm_engagement.program_ID  0.09944    214.1
## traitpost_interest:traitrm_engagement.program_ID  0.22412   1000.0
## traitrm_engagement:traitpost_interest.program_ID  0.22412   1000.0
## traitpost_interest:traitpost_interest.program_ID  4.52577   1000.0
## 
##  R-structure:  ~idh(trait):units
## 
##                          post.mean l-95% CI u-95% CI eff.samp
## traitrm_engagement.units    0.4091   0.3867   0.4319     1099
## traitpost_interest.units    0.0001   0.0001   0.0001        0
## 
##  Location effects: cbind(rm_engagement, post_interest) ~ trait + gender_female + urm + pre_interest 
## 
##                    post.mean l-95% CI u-95% CI eff.samp  pMCMC    
## (Intercept)          2.37254  1.97588  2.74802    909.4 <0.001 ***
## traitpost_interest   0.21143 -0.73525  1.23497   1000.0  0.642    
## gender_female       -0.09336 -0.25609  0.05576   1248.5  0.236    
## urm                 -0.06380 -0.30531  0.14722   1000.0  0.576    
## pre_interest         0.19042  0.08965  0.28672    836.5 <0.001 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

##      var1 
## 0.2806292

Summary

Questions and future directions

To calculate these correlations, I calculated the partial R^2 (i.e., the change in R-squared when adding the predictor) for the BLUPs for engagement, which I think helps to compare the results with the two approaches. However, I’m not sure this is correct. In the lme4 models, we used the BLUPs in a separate linear model and obtained a regression coefficient relating repeated measures engagement to post interest, whereas in the MCMCglmm models we modeled the correlation of the random effects for repeated measures engagement and interest. One way to compare them is to use correlations instead of a linear model to examine the relations between the predictions (BLUPs) for repeated measures engagement and interest, but this does not allow us to account for covariates. To address this, could a partial correlation be examined (possibly using the r2glmm package or a manual approach)? Or, is there another way to transform the beta coefficient into a partial correlation coefficient? Or, is there a way to interpret the correlation between repeated measures engagement and post interest in the MCMCglmm models as a regression coefficient?