Notes

Just a short note on how to interpret R-squared values

“For linear mixed models, an r-squared approximation by computing the correlation between the fitted and observed values, as suggested by Byrnes (2008), is returned as well as a simplified version of the Omega-squared value (1 - (residual variance / response variance), Xu (2003), Nakagawa, Schielzeth 2013), unless n is specified.”

Only output is displayed

To make it easier to interpret the results, only output is printed and displayed, though there are still processing steps going on (and the headers for the processing steps are still included). Some descriptive statistics and correlation tables are also not included to simplify interpretation, but these (and the processing steps) can be quickly added back in.

0. Loading, setting up

0A. Loading packages

0B. Loading data

0C. Joining

Note that parent and future_goals_plans had to be processed first for merging later on; can change.

## Joining, by = "participant_ID"

0D. Processing CLASS video data

OE. Further processing

df$participant_ID <- as.factor(df$participant_ID)
df$program_ID <- as.factor(df$program_ID)
df$beep_ID <- as.factor(df$beep_ID)
df$beep_ID_new <- as.factor(df$beep_ID_new)

# Recode problem solving, off task, student presentation, and showing video as other

df$youth_activity_rc <- ifelse(df$youth_activity == "Off Task", "Not Focused", df$youth_activity)

df$youth_activity_rc <- ifelse(df$youth_activity_rc == "Student Presentation" | df$youth_activity_rc == "Problem Solving", "Creating Product", df$youth_activity_rc)

df$youth_activity_rc <- ifelse(df$youth_activity_rc == "Showing Video", "Program Staff Led", df$youth_activity_rc)

df$youth_activity_rc <- as.factor(df$youth_activity_rc)

df$youth_activity_rc <- forcats::fct_relevel(df$youth_activity_rc, "Not Focused")

df$relevance <- jmRtools::composite_mean_maker(df, use_outside, future_goals, important)

# need to move up
video$youth_activity_rc <- ifelse(video$youth_activity == "Off Task", "Not Focused", video$youth_activity)

video$youth_activity_rc <- ifelse(video$youth_activity_rc == "Student Presentation" | video$youth_activity_rc == "Problem Solving", "Creating Product", video$youth_activity_rc)
video$youth_activity_rc <- ifelse(video$youth_activity_rc == "Showing Video", "Program Staff Led", video$youth_activity_rc)

ncol(df)

## [1] 161

0F: CLASS correlations

df %>% 
    select(CLASS_comp, CLASS_EmotionalSupportEncouragement, CLASS_InstructionalSupport, CLASS_STEMConceptualDevelopment, CLASS_ActivityLeaderEnthusiasm, CLASS_Autonomy,
           challenge, relevance, learning, positive_affect) %>%  
    correlate() %>% 
    shave() %>% 
    fashion() %>%
    knitr::kable()

0G. Processing demographics

## # A tibble: 6 x 2
##          race     n
##         <chr> <int>
## 1       Asian    14
## 2       Black    72
## 3    Hispanic    97
## 4 Multiracial     6
## 5       White    13
## 6        <NA>     3

1. QUESTION 1

How are instructional practices in STEM summer programs related to perceptions of challenge, relevance, learning, and affect for participating youth?

Findings seem to suggest specific youth activities impacts upon challenge, relevance, and learning (we are considering not including affect for now). Females, generally, experience lower levels of these outcomes, and the overall quality composite, measured through the CLASS variable, seems to be an important control variable to include.

1A. Null models

Note that our normal display presently doesn’t work if there are no predictor variables: it should be fixed in a bit, but for now, we’re just printing the ICCs.

m0i <- lmer(challenge ~ 1 + 
                (1|program_ID) + (1|participant_ID) + (1|beep_ID_new),
            data = df)

sjstats::icc(m0i)

## Linear mixed model
##  Family: gaussian (identity)
## Formula: challenge ~ 1 + (1 | program_ID) + (1 | participant_ID) + (1 | beep_ID_new)
## 
##      ICC (beep_ID_new): 0.066428
##   ICC (participant_ID): 0.371089
##       ICC (program_ID): 0.034016

m0ii <- lmer(relevance ~ 1 + 
                 (1|program_ID) + (1|participant_ID) + (1|beep_ID_new),
             data = df)

sjstats::icc(m0ii)

## Linear mixed model
##  Family: gaussian (identity)
## Formula: relevance ~ 1 + (1 | program_ID) + (1 | participant_ID) + (1 | beep_ID_new)
## 
##      ICC (beep_ID_new): 0.019134
##   ICC (participant_ID): 0.515087
##       ICC (program_ID): 0.009082

m0iii <- lmer(learning ~ 1 + 
                  (1|program_ID) + (1|participant_ID) + (1|beep_ID_new),
              data = df)

sjstats::icc(m0iii)

## Linear mixed model
##  Family: gaussian (identity)
## Formula: learning ~ 1 + (1 | program_ID) + (1 | participant_ID) + (1 | beep_ID_new)
## 
##      ICC (beep_ID_new): 0.023511
##   ICC (participant_ID): 0.349134
##       ICC (program_ID): 0.000000

# df$positive_affect <- jmRtools::composite_mean_maker(df,
#                                                      happy, excited)

1B. Models for youth activity (with CLASS composite)

m1i <- lmer(challenge ~ 1 + 
                youth_activity_rc +
                gender_female + 
                CLASS_comp +
                (1|program_ID) + (1|participant_ID) + (1|beep_ID_new),
            data = df)

sjPlot::sjt.lmer(m1i, p.kr = F, show.re.var = F, show.ci = F, show.se = T)

	challenge
	B	std. Error	p
Fixed Parts
(Intercept)	2.21	0.14	<.001
youth_activity_rc (Basic Skills Activity)	0.06	0.07	.399
youth_activity_rc (Creating Product)	0.29	0.08	<.001
youth_activity_rc (Field Trip Speaker)	-0.12	0.13	.359
youth_activity_rc (Lab Activity)	0.15	0.13	.245
youth_activity_rc (Program Staff Led)	-0.13	0.08	.112
gender_female	-0.25	0.11	.016
CLASS_comp	0.04	0.03	.155
Random Parts
N_{beep_ID_new}	228
N_{participant_ID}	201
N_{program_ID}	9
ICC_{beep_ID_new}	0.046
ICC_{participant_ID}	0.376
ICC_{program_ID}	0.032
Observations	2693
R² / Ω₀²	.527 / .520

m1ii <- lmer(relevance ~ 1 + 
                 youth_activity_rc +
                 gender_female + 
                 CLASS_comp +
                 (1|program_ID) + (1|participant_ID) + (1|beep_ID_new),
             data = df)

sjPlot::sjt.lmer(m1ii, p.kr = F, show.re.var = F, show.ci = F, show.se = T)

	relevance
	B	std. Error	p
Fixed Parts
(Intercept)	2.49	0.10	<.001
youth_activity_rc (Basic Skills Activity)	0.12	0.04	.004
youth_activity_rc (Creating Product)	0.18	0.05	<.001
youth_activity_rc (Field Trip Speaker)	0.27	0.07	<.001
youth_activity_rc (Lab Activity)	0.10	0.08	.183
youth_activity_rc (Program Staff Led)	0.11	0.05	.015
gender_female	-0.22	0.10	.032
CLASS_comp	0.03	0.02	.081
Random Parts
N_{beep_ID_new}	228
N_{participant_ID}	201
N_{program_ID}	9
ICC_{beep_ID_new}	0.006
ICC_{participant_ID}	0.518
ICC_{program_ID}	0.017
Observations	2693
R² / Ω₀²	.589 / .586

m1iii <- lmer(learning ~ 1 + 
                  youth_activity_rc +
                  gender_female + 
                  CLASS_comp +
                  (1|program_ID) + (1|participant_ID) + (1|beep_ID_new),
              data = df)

sjPlot::sjt.lmer(m1iii, p.kr = F, show.re.var = F, show.ci = F, show.se = T)

	learning
	B	std. Error	p
Fixed Parts
(Intercept)	2.53	0.10	<.001
youth_activity_rc (Basic Skills Activity)	0.17	0.05	.002
youth_activity_rc (Creating Product)	0.04	0.06	.499
youth_activity_rc (Field Trip Speaker)	0.06	0.10	.502
youth_activity_rc (Lab Activity)	0.09	0.10	.347
youth_activity_rc (Program Staff Led)	0.04	0.06	.490
gender_female	-0.06	0.10	.549
CLASS_comp	0.06	0.02	.005
Random Parts
N_{beep_ID_new}	228
N_{participant_ID}	201
N_{program_ID}	9
ICC_{beep_ID_new}	0.008
ICC_{participant_ID}	0.355
ICC_{program_ID}	0.006
Observations	2692
R² / Ω₀²	.427 / .421

df$positive_affect <- jmRtools::composite_mean_maker(df,
                                                     happy, excited)

# m1iv <- lmer(positive_affect ~ 1 + 
#                  youth_activity_rc +
#                  gender_female + 
#                  CLASS_comp +
#                  (1|program_ID) + (1|participant_ID) + (1|beep_ID_new),
#              na.action = "na.omit",
#              data = df)
# 
# sjPlot::sjt.lmer(m1iv, p.kr = F, show.re.var = F, show.ci = F, show.se = T)

2. QUESTION 2

How is the relationship between STEM practices and perceptions of challenge, relevance, learning, and affect moderated by: (a) youth characteristics at program entry (L2); (b) activity leader perceptions of effective instructional practice (L3 – limited power); (c) the type of activity(L1 – how different from STEM practices?); and (d) classroom versus field-based settings? STEM practices = CLASS items, value statements, etc. stuff from video

In order to examine how youth characteristics at program entry, particularly their pre-program interest, we examined the fixed effect impact of interest and whether there is pre-interest variability at the youth level. Broadly, interest does not seem to be an important predictor overall as a fixed effect, and, predictably, there is not substantial variability at the youth level. Thus there does not seem to be the potential for significant pre-interest by youth activity interactions. We can consider interactions for classroom versus field settings, which we have not yet considered, and activity leader perceptions of effective instructional practice, which are not in these models.

2A. Descriptives and correlations for youth activities and ESM variables

df$youth_activity_rc_fac <- as.factor(df$youth_activity_rc)
dc <- as.tibble(psych::dummy.code(df$youth_activity_rc_fac))
df_ss <- bind_cols(df, dc)

df_ss %>% 
    select(challenge, relevance, learning, positive_affect, overall_pre_interest,
           `Not Focused`, `Basic Skills Activity`, `Creating Product`, 
           `Field Trip Speaker`, `Lab Activity`, `Program Staff Led`) %>% 
    correlate() %>% 
    shave() %>% 
    fashion() %>% 
    knitr::kable()

2B. Models for youth activity (with CLASS composite)

df_d <- df[!is.na(df$youth_activity_rc), ]

m1i <- lmer(challenge ~ 1 +
                youth_activity_rc +
                gender_female +
                CLASS_comp +
                overall_pre_interest +
                classroom_versus_field_enrichment +
                
                (1|program_ID) + (1|participant_ID) + (1|beep_ID_new),
            data = df_d)

sjPlot::sjt.lmer(m1i, p.kr = F, show.re.var = T, show.ci = F, show.se = T)

	challenge
	B	std. Error	p
Fixed Parts
(Intercept)	2.20	0.26	<.001
youth_activity_rc (Basic Skills Activity)	0.06	0.07	.440
youth_activity_rc (Creating Product)	0.26	0.08	.001
youth_activity_rc (Field Trip Speaker)	-0.12	0.14	.399
youth_activity_rc (Lab Activity)	0.11	0.14	.415
youth_activity_rc (Program Staff Led)	-0.15	0.08	.065
gender_female	-0.25	0.11	.023
CLASS_comp	0.04	0.03	.193
overall_pre_interest	0.01	0.07	.936
classroom_versus_field_enrichment	0.04	0.07	.512
Random Parts
σ²	0.658
τ_{00, beep_ID_new}	0.061
τ_{00, participant_ID}	0.469
τ_{00, program_ID}	0.049
N_{beep_ID_new}	228
N_{participant_ID}	180
N_{program_ID}	9
ICC_{beep_ID_new}	0.050
ICC_{participant_ID}	0.379
ICC_{program_ID}	0.039
Observations	2483
R² / Ω₀²	.530 / .522

m1i_f <- lmer(challenge ~ 1 +
                  youth_activity_rc +
                  gender_female +
                  CLASS_comp +
                  overall_pre_interest +
                  classroom_versus_field_enrichment +
                  
                  (1|program_ID) + (1 + overall_pre_interest|participant_ID) + (1|beep_ID_new),
              data = df_d)

summary(m1i_f)

## Linear mixed model fit by REML ['lmerMod']
## Formula: challenge ~ 1 + youth_activity_rc + gender_female + CLASS_comp +  
##     overall_pre_interest + classroom_versus_field_enrichment +  
##     (1 | program_ID) + (1 + overall_pre_interest | participant_ID) +  
##     (1 | beep_ID_new)
##    Data: df_d
## 
## REML criterion at convergence: 6593.8
## 
## Scaled residuals: 
##     Min      1Q  Median      3Q     Max 
## -2.8162 -0.6446 -0.0319  0.5593  3.3399 
## 
## Random effects:
##  Groups         Name                 Variance Std.Dev. Corr 
##  beep_ID_new    (Intercept)          0.06129  0.2476        
##  participant_ID (Intercept)          0.87545  0.9357        
##                 overall_pre_interest 0.08377  0.2894   -0.76
##  program_ID     (Intercept)          0.04968  0.2229        
##  Residual                            0.65846  0.8115        
## Number of obs: 2483, groups:  
## beep_ID_new, 228; participant_ID, 180; program_ID, 9
## 
## Fixed effects:
##                                          Estimate Std. Error t value
## (Intercept)                             2.217e+00  2.570e-01   8.628
## youth_activity_rcBasic Skills Activity  5.694e-02  7.286e-02   0.782
## youth_activity_rcCreating Product       2.599e-01  8.057e-02   3.226
## youth_activity_rcField Trip Speaker    -1.171e-01  1.411e-01  -0.830
## youth_activity_rcLab Activity           1.117e-01  1.370e-01   0.816
## youth_activity_rcProgram Staff Led     -1.513e-01  8.228e-02  -1.839
## gender_female                          -2.477e-01  1.105e-01  -2.241
## CLASS_comp                              3.528e-02  2.773e-02   1.272
## overall_pre_interest                    6.355e-05  6.997e-02   0.001
## classroom_versus_field_enrichment       4.396e-02  6.562e-02   0.670
## 
## Correlation of Fixed Effects:
##             (Intr) y__BSA yt__CP y__FTS yt__LA y__PSL gndr_f CLASS_ ovrl__
## yth_ctv_BSA  0.007                                                        
## yth_ctvt_CP  0.100  0.466                                                 
## yth_ctv_FTS -0.041  0.283  0.187                                          
## yth_ctvt_LA  0.032  0.284  0.277  0.170                                   
## yth_ctv_PSL -0.030  0.451  0.364  0.193  0.232                            
## gender_feml -0.254 -0.008  0.004  0.001 -0.007 -0.013                     
## CLASS_comp  -0.315 -0.356 -0.452 -0.136 -0.270 -0.214  0.005              
## ovrll_pr_nt -0.817 -0.005 -0.008 -0.015 -0.006  0.016  0.043 -0.012       
## clssrm_vr__ -0.146  0.090 -0.201  0.255  0.059 -0.006 -0.011 -0.058  0.009

anova(m1i, m1i_f) # the model with the random slope does not seem to fit better

## Data: df_d
## Models:
## m1i: challenge ~ 1 + youth_activity_rc + gender_female + CLASS_comp + 
## m1i:     overall_pre_interest + classroom_versus_field_enrichment + 
## m1i:     (1 | program_ID) + (1 | participant_ID) + (1 | beep_ID_new)
## m1i_f: challenge ~ 1 + youth_activity_rc + gender_female + CLASS_comp + 
## m1i_f:     overall_pre_interest + classroom_versus_field_enrichment + 
## m1i_f:     (1 | program_ID) + (1 + overall_pre_interest | participant_ID) + 
## m1i_f:     (1 | beep_ID_new)
##       Df    AIC    BIC  logLik deviance  Chisq Chi Df Pr(>Chisq)
## m1i   14 6590.6 6672.1 -3281.3   6562.6                         
## m1i_f 16 6592.5 6685.5 -3280.2   6560.5 2.1626      2     0.3392

m1ii <- lmer(relevance ~ 1 + 
                 youth_activity_rc +
                 gender_female + 
                 CLASS_comp +
                 overall_pre_interest +
                 classroom_versus_field_enrichment +
                 
                 (1|program_ID) + (1|participant_ID) + (1|beep_ID_new),
             data = df_d)

sjPlot::sjt.lmer(m1ii, p.kr = F, show.re.var = T, show.ci = F, show.se = T)

	relevance
	B	std. Error	p
Fixed Parts
(Intercept)	2.23	0.23	<.001
youth_activity_rc (Basic Skills Activity)	0.10	0.04	.026
youth_activity_rc (Creating Product)	0.17	0.05	<.001
youth_activity_rc (Field Trip Speaker)	0.21	0.08	.010
youth_activity_rc (Lab Activity)	0.05	0.08	.556
youth_activity_rc (Program Staff Led)	0.11	0.05	.029
gender_female	-0.24	0.11	.029
CLASS_comp	0.03	0.02	.051
overall_pre_interest	0.10	0.06	.111
classroom_versus_field_enrichment	-0.06	0.04	.108
Random Parts
σ²	0.412
τ_{00, beep_ID_new}	0.008
τ_{00, participant_ID}	0.473
τ_{00, program_ID}	0.020
N_{beep_ID_new}	228
N_{participant_ID}	180
N_{program_ID}	9
ICC_{beep_ID_new}	0.008
ICC_{participant_ID}	0.519
ICC_{program_ID}	0.022
Observations	2483
R² / Ω₀²	.596 / .594

m1ii_f <- lmer(relevance ~ 1 + 
                   youth_activity_rc +
                   gender_female + 
                   CLASS_comp +
                   overall_pre_interest +
                   classroom_versus_field_enrichment +
                   
                   (1|program_ID) + (1 + overall_pre_interest|participant_ID) + (1|beep_ID_new),
               data = df_d)

summary(m1ii_f)

## Linear mixed model fit by REML ['lmerMod']
## Formula: relevance ~ 1 + youth_activity_rc + gender_female + CLASS_comp +  
##     overall_pre_interest + classroom_versus_field_enrichment +  
##     (1 | program_ID) + (1 + overall_pre_interest | participant_ID) +  
##     (1 | beep_ID_new)
##    Data: df_d
## 
## REML criterion at convergence: 5403.2
## 
## Scaled residuals: 
##     Min      1Q  Median      3Q     Max 
## -3.9108 -0.5360  0.0289  0.5731  4.0679 
## 
## Random effects:
##  Groups         Name                 Variance Std.Dev. Corr 
##  beep_ID_new    (Intercept)          0.007588 0.08711       
##  participant_ID (Intercept)          0.799636 0.89422       
##                 overall_pre_interest 0.074581 0.27309  -0.72
##  program_ID     (Intercept)          0.016179 0.12720       
##  Residual                            0.411662 0.64161       
## Number of obs: 2483, groups:  
## beep_ID_new, 228; participant_ID, 180; program_ID, 9
## 
## Fixed effects:
##                                        Estimate Std. Error t value
## (Intercept)                             2.28356    0.22280  10.249
## youth_activity_rcBasic Skills Activity  0.09918    0.04447   2.230
## youth_activity_rcCreating Product       0.16685    0.04996   3.340
## youth_activity_rcField Trip Speaker     0.20544    0.08000   2.568
## youth_activity_rcLab Activity           0.04773    0.08039   0.594
## youth_activity_rcProgram Staff Led      0.10890    0.04966   2.193
## gender_female                          -0.23917    0.10782  -2.218
## CLASS_comp                              0.03264    0.01679   1.944
## overall_pre_interest                    0.08493    0.06508   1.305
## classroom_versus_field_enrichment      -0.06284    0.03959  -1.587
## 
## Correlation of Fixed Effects:
##             (Intr) y__BSA yt__CP y__FTS yt__LA y__PSL gndr_f CLASS_ ovrl__
## yth_ctv_BSA  0.005                                                        
## yth_ctvt_CP  0.073  0.459                                                 
## yth_ctv_FTS -0.021  0.301  0.197                                          
## yth_ctvt_LA  0.030  0.296  0.287  0.190                                   
## yth_ctv_PSL -0.026  0.458  0.363  0.205  0.241                            
## gender_feml -0.298 -0.007  0.003  0.001 -0.008 -0.012                     
## CLASS_comp  -0.214 -0.359 -0.449 -0.152 -0.284 -0.216  0.004              
## ovrll_pr_nt -0.879  0.000 -0.008 -0.016 -0.008  0.018  0.060 -0.015       
## clssrm_vr__ -0.097  0.085 -0.207  0.258  0.059 -0.009 -0.012 -0.079  0.013

anova(m1ii, m1ii_f) # the model with the random slope does not seem to fit better

## Data: df_d
## Models:
## m1ii: relevance ~ 1 + youth_activity_rc + gender_female + CLASS_comp + 
## m1ii:     overall_pre_interest + classroom_versus_field_enrichment + 
## m1ii:     (1 | program_ID) + (1 | participant_ID) + (1 | beep_ID_new)
## m1ii_f: relevance ~ 1 + youth_activity_rc + gender_female + CLASS_comp + 
## m1ii_f:     overall_pre_interest + classroom_versus_field_enrichment + 
## m1ii_f:     (1 | program_ID) + (1 + overall_pre_interest | participant_ID) + 
## m1ii_f:     (1 | beep_ID_new)
##        Df    AIC    BIC  logLik deviance  Chisq Chi Df Pr(>Chisq)
## m1ii   14 5391.8 5473.2 -2681.9   5363.8                         
## m1ii_f 16 5393.8 5486.9 -2680.9   5361.8 1.9577      2     0.3757

m1iii <- lmer(learning ~ 1 + 
                  youth_activity_rc +
                  gender_female + 
                  CLASS_comp +
                  overall_pre_interest +
                  classroom_versus_field_enrichment +
                  
                  (1|program_ID) + (1|participant_ID) + (1|beep_ID_new),
              data = df_d)

sjPlot::sjt.lmer(m1iii, p.kr = F, show.re.var = F, show.ci = F, show.se = T)

	learning
	B	std. Error	p
Fixed Parts
(Intercept)	2.28	0.21	<.001
youth_activity_rc (Basic Skills Activity)	0.16	0.06	.005
youth_activity_rc (Creating Product)	0.02	0.06	.747
youth_activity_rc (Field Trip Speaker)	0.04	0.10	.681
youth_activity_rc (Lab Activity)	0.12	0.10	.242
youth_activity_rc (Program Staff Led)	0.03	0.06	.596
gender_female	-0.07	0.10	.510
CLASS_comp	0.06	0.02	.006
overall_pre_interest	0.08	0.06	.164
classroom_versus_field_enrichment	0.01	0.05	.879
Random Parts
N_{beep_ID_new}	228
N_{participant_ID}	180
N_{program_ID}	9
ICC_{beep_ID_new}	0.008
ICC_{participant_ID}	0.354
ICC_{program_ID}	0.004
Observations	2482
R² / Ω₀²	.420 / .414

m1iii_f <- lmer(learning ~ 1 + 
                    youth_activity_rc +
                    gender_female + 
                    CLASS_comp +
                    overall_pre_interest +
                    classroom_versus_field_enrichment +
                    
                    (1|program_ID) + (1 + overall_pre_interest|participant_ID) + (1|beep_ID_new),
                data = df_d)

summary(m1iii_f)

## Linear mixed model fit by REML ['lmerMod']
## Formula: learning ~ 1 + youth_activity_rc + gender_female + CLASS_comp +  
##     overall_pre_interest + classroom_versus_field_enrichment +  
##     (1 | program_ID) + (1 + overall_pre_interest | participant_ID) +  
##     (1 | beep_ID_new)
##    Data: df_d
## 
## REML criterion at convergence: 6623.4
## 
## Scaled residuals: 
##     Min      1Q  Median      3Q     Max 
## -3.1271 -0.5639  0.1119  0.5813  2.8205 
## 
## Random effects:
##  Groups         Name                 Variance Std.Dev. Corr 
##  beep_ID_new    (Intercept)          0.008630 0.09290       
##  participant_ID (Intercept)          0.907311 0.95253       
##                 overall_pre_interest 0.083892 0.28964  -0.81
##  program_ID     (Intercept)          0.002774 0.05266       
##  Residual                            0.710832 0.84311       
## Number of obs: 2482, groups:  
## beep_ID_new, 228; participant_ID, 180; program_ID, 9
## 
## Fixed effects:
##                                         Estimate Std. Error t value
## (Intercept)                             2.339625   0.214340  10.915
## youth_activity_rcBasic Skills Activity  0.157320   0.056209   2.799
## youth_activity_rcCreating Product       0.019408   0.063401   0.306
## youth_activity_rcField Trip Speaker     0.041391   0.099993   0.414
## youth_activity_rcLab Activity           0.118621   0.101310   1.171
## youth_activity_rcProgram Staff Led      0.033598   0.062770   0.535
## gender_female                          -0.062370   0.101470  -0.615
## CLASS_comp                              0.057521   0.021117   2.724
## overall_pre_interest                    0.060915   0.061264   0.994
## classroom_versus_field_enrichment       0.009493   0.049412   0.192
## 
## Correlation of Fixed Effects:
##             (Intr) y__BSA yt__CP y__FTS yt__LA y__PSL gndr_f CLASS_ ovrl__
## yth_ctv_BSA  0.005                                                        
## yth_ctvt_CP  0.095  0.460                                                 
## yth_ctv_FTS -0.020  0.300  0.200                                          
## yth_ctvt_LA  0.045  0.298  0.290  0.194                                   
## yth_ctv_PSL -0.040  0.454  0.365  0.204  0.242                            
## gender_feml -0.291 -0.010  0.005  0.007 -0.016 -0.019                     
## CLASS_comp  -0.273 -0.357 -0.447 -0.157 -0.289 -0.214  0.006              
## ovrll_pr_nt -0.869  0.004 -0.013 -0.031 -0.016  0.033  0.062 -0.030       
## clssrm_vr__ -0.132  0.071 -0.208  0.257  0.059 -0.020 -0.016 -0.073  0.022

anova(m1iii, m1iii_f) # the model with the random slope does not seem to fit better

## Data: df_d
## Models:
## m1iii: learning ~ 1 + youth_activity_rc + gender_female + CLASS_comp + 
## m1iii:     overall_pre_interest + classroom_versus_field_enrichment + 
## m1iii:     (1 | program_ID) + (1 | participant_ID) + (1 | beep_ID_new)
## m1iii_f: learning ~ 1 + youth_activity_rc + gender_female + CLASS_comp + 
## m1iii_f:     overall_pre_interest + classroom_versus_field_enrichment + 
## m1iii_f:     (1 | program_ID) + (1 + overall_pre_interest | participant_ID) + 
## m1iii_f:     (1 | beep_ID_new)
##         Df    AIC    BIC  logLik deviance  Chisq Chi Df Pr(>Chisq)
## m1iii   14 6614.1 6695.6 -3293.1   6586.1                         
## m1iii_f 16 6616.5 6709.5 -3292.2   6584.5 1.6794      2     0.4318

df$positive_affect <- jmRtools::composite_mean_maker(df,
                                                     happy, excited)

# m1iv <- lmer(positive_affect ~ 1 + 
#                  youth_activity_rc +
#                  gender_female + 
#                  CLASS_comp +
#                  (1|program_ID) + (1|participant_ID) + (1|beep_ID_new),
#              na.action = "na.omit",
#              data = df)
# 
# sjPlot::sjt.lmer(m1iv, p.kr = F, show.re.var = F, show.ci = F, show.se = T)

3. QUESTION 3

How do in-the-moment experiences of youth (i.e., challenge, relevance, learning, and affect) cultivate situational interest and engagement in STEM activities?

Broadly, these in-the-moment experiences predict situational interest and engagement, though the subsequent models may be of greater interest to us (and our audience).

3A. Correlations for situational experiences (interest and engagement)

rowname	overall_engagement	interest	challenge	relevance	learning
overall_engagement
interest	.69
challenge	.30	.28
relevance	.65	.61	.39
learning	.68	.56	.30	.65
positive_affect	.65	.56	.27	.52	.48

3B. Models for situational experiences (interest and engagement; with CLASS composite)

m3i0 <- lmer(interest ~ 1 +
                 overall_pre_interest + 
                 (1|program_ID) + (1|participant_ID) + (1|beep_ID_new),
             data = df)

sjPlot::sjt.lmer(m3i0, p.kr = F, show.re.var = F, show.ci = F, show.se = T)

## Computing p-values via Wald-statistics approximation (treating t as Wald z).

	interest
	B	std. Error	p
Fixed Parts
(Intercept)	2.44	0.19	<.001
overall_pre_interest	0.14	0.06	.018
Random Parts
N_{beep_ID_new}	248
N_{participant_ID}	181
N_{program_ID}	9
ICC_{beep_ID_new}	0.040
ICC_{participant_ID}	0.328
ICC_{program_ID}	0.017
Observations	2738
R² / Ω₀²	.448 / .438

m3ii0 <- lmer(overall_engagement ~ 1 + 
                  overall_pre_interest + 
                  (1|program_ID) + (1|participant_ID) + (1|beep_ID_new),
              data = df)

sjPlot::sjt.lmer(m3ii0, p.kr = F, show.re.var = F, show.ci = F, show.se = T)

## Computing p-values via Wald-statistics approximation (treating t as Wald z).

	overall_engagement
	B	std. Error	p
Fixed Parts
(Intercept)	2.50	0.16	<.001
overall_pre_interest	0.11	0.05	.030
Random Parts
N_{beep_ID_new}	248
N_{participant_ID}	181
N_{program_ID}	9
ICC_{beep_ID_new}	0.034
ICC_{participant_ID}	0.429
ICC_{program_ID}	0.000
Observations	2738
R² / Ω₀²	.523 / .518

3C. Models for situational experiences (interest and engagement; with CLASS composite)

m3i <- lmer(interest ~ 1 +
                challenge + relevance + learning +
                gender_female + 
                classroom_versus_field_enrichment +
                CLASS_comp + 
                youth_activity_rc +
                (1|program_ID) + (1|participant_ID) + (1|beep_ID_new),
            data = df)

sjPlot::sjt.lmer(m3i, p.kr = F, show.re.var = F, show.ci = F, show.se = T)

	interest
	B	std. Error	p
Fixed Parts
(Intercept)	0.97	0.11	<.001
challenge	0.03	0.02	.042
relevance	0.44	0.02	<.001
learning	0.24	0.02	<.001
gender_female	0.01	0.05	.822
classroom_versus_field_enrichment	0.04	0.05	.391
CLASS_comp	0.00	0.02	.821
youth_activity_rc (Basic Skills Activity)	-0.13	0.06	.025
youth_activity_rc (Creating Product)	-0.05	0.06	.431
youth_activity_rc (Field Trip Speaker)	0.01	0.11	.939
youth_activity_rc (Lab Activity)	-0.03	0.11	.775
youth_activity_rc (Program Staff Led)	-0.07	0.06	.302
Random Parts
N_{beep_ID_new}	228
N_{participant_ID}	201
N_{program_ID}	9
ICC_{beep_ID_new}	0.045
ICC_{participant_ID}	0.114
ICC_{program_ID}	0.024
Observations	2692
R² / Ω₀²	.564 / .562

m3ii <- lmer(overall_engagement ~ 1 + 
                 challenge + relevance + learning + 
                 gender_female + 
                 classroom_versus_field_enrichment +
                 CLASS_comp + 
                 youth_activity_rc +
                 (1|program_ID) + (1|participant_ID) + (1|beep_ID_new),
             data = df)

sjPlot::sjt.lmer(m3ii, p.kr = F, show.re.var = F, show.ci = F, show.se = T)

	overall_engagement
	B	std. Error	p
Fixed Parts
(Intercept)	0.96	0.08	<.001
challenge	0.04	0.01	.003
relevance	0.33	0.02	<.001
learning	0.32	0.01	<.001
gender_female	0.04	0.05	.361
classroom_versus_field_enrichment	0.10	0.03	.005
CLASS_comp	0.00	0.01	.851
youth_activity_rc (Basic Skills Activity)	0.01	0.04	.830
youth_activity_rc (Creating Product)	-0.01	0.04	.814
youth_activity_rc (Field Trip Speaker)	0.07	0.07	.378
youth_activity_rc (Lab Activity)	0.03	0.07	.672
youth_activity_rc (Program Staff Led)	-0.07	0.04	.130
Random Parts
N_{beep_ID_new}	228
N_{participant_ID}	201
N_{program_ID}	9
ICC_{beep_ID_new}	0.036
ICC_{participant_ID}	0.233
ICC_{program_ID}	0.031
Observations	2692
R² / Ω₀²	.706 / .705

m3iii <- lmer(interest ~ 1 +
                  challenge + relevance + learning + 
                  overall_pre_interest +
                  classroom_versus_field_enrichment +
                  gender_female + 
                  CLASS_comp + 
                  youth_activity_rc +
                  (1|program_ID) + (1|participant_ID) + (1|beep_ID_new),
              data = df)

sjPlot::sjt.lmer(m3iii, p.kr = F, show.re.var = F, show.ci = F, show.se = T)

	interest
	B	std. Error	p
Fixed Parts
(Intercept)	0.78	0.15	<.001
challenge	0.03	0.02	.073
relevance	0.45	0.02	<.001
learning	0.24	0.02	<.001
overall_pre_interest	0.06	0.03	.067
classroom_versus_field_enrichment	0.06	0.05	.215
gender_female	0.02	0.05	.669
CLASS_comp	0.01	0.02	.744
youth_activity_rc (Basic Skills Activity)	-0.15	0.06	.009
youth_activity_rc (Creating Product)	-0.05	0.07	.413
youth_activity_rc (Field Trip Speaker)	0.01	0.11	.938
youth_activity_rc (Lab Activity)	-0.04	0.11	.685
youth_activity_rc (Program Staff Led)	-0.10	0.07	.148
Random Parts
N_{beep_ID_new}	228
N_{participant_ID}	180
N_{program_ID}	9
ICC_{beep_ID_new}	0.047
ICC_{participant_ID}	0.114
ICC_{program_ID}	0.022
Observations	2482
R² / Ω₀²	.561 / .558

m3iv <- lmer(overall_engagement ~ 1 + 
                 challenge + relevance + learning + 
                 overall_pre_interest +
                 classroom_versus_field_enrichment +
                 gender_female + 
                 CLASS_comp + 
                 (1|program_ID) + (1|participant_ID) + (1|beep_ID_new),
             data = df)

sjPlot::sjt.lmer(m3iv, p.kr = F, show.re.var = F, show.ci = F, show.se = T)

	overall_engagement
	B	std. Error	p
Fixed Parts
(Intercept)	0.77	0.12	<.001
challenge	0.04	0.01	.003
relevance	0.33	0.02	<.001
learning	0.31	0.01	<.001
overall_pre_interest	0.06	0.03	.049
classroom_versus_field_enrichment	0.08	0.03	.010
gender_female	0.05	0.05	.284
CLASS_comp	0.00	0.01	.765
Random Parts
N_{beep_ID_new}	231
N_{participant_ID}	180
N_{program_ID}	9
ICC_{beep_ID_new}	0.038
ICC_{participant_ID}	0.239
ICC_{program_ID}	0.026
Observations	2513
R² / Ω₀²	.705 / .704

4. QUESTION 4

Are situational (momentary) interest and engagement in STEM activities across several weeks associated with changes in: (a) individual (sustained) interest in STEM; (c) future goals and aspirations related to STEM?

These models seem to demonstrate that challenge, learning, and relevance (in terms of individual-level) have limited effects, though adding additional covariates, especially the program-level variables for time spent on different activities, could be important.

4A. Descriptives for situational experiences and future goals and plans

participant_df <- df %>% 
    select(participant_ID, challenge, relevance, learning, positive_affect, good_at, post_future_goals_plans) %>% 
    group_by(participant_ID) %>% 
    mutate_at(vars(challenge, relevance, learning, positive_affect, good_at), funs(mean, sd)) %>%
    select(participant_ID, contains("mean"), contains("sd"), post_future_goals_plans) %>% 
    distinct()

df_ss <- left_join(df, participant_df)

df_ss <- select(df_ss, 
                participant_ID, program_ID,
                challenge_mean, relevance_mean, learning_mean, positive_affect_mean, good_at_mean,
                challenge_sd, relevance_sd, learning_sd, positive_affect_sd, good_at_sd,
                overall_post_interest, overall_pre_interest, post_future_goals_plans,
                future_goals)

df_ss <- distinct(df_ss)

df_ss$program_ID <- as.integer(df_ss$program_ID)

df_ss <- left_join(df_ss, pm)

4B. Processing data for predicting individual-level effects

4D. Using individual-level effects to predict post-interest and future goals and aspirations

m4bi <- lmer(overall_post_interest ~ 
                 pred_challenge + pred_learning + pred_relevance +
                 scale(overall_pre_interest, scale = F) + (1|program_ID), 
             data = mod_df)
sjPlot::sjt.lmer(m4bi, p.kr = F, show.re.var = F, show.ci = F, show.se = T)

## Computing p-values via Wald-statistics approximation (treating t as Wald z).

	overall_post_interest
	B	std. Error	p
Fixed Parts
(Intercept)	2.29	0.33	<.001
pred_challenge	-0.17	0.11	.141
pred_learning	0.55	0.23	.016
pred_relevance	-0.14	0.22	.513
scale(overall_pre_interest, scale = F)	0.54	0.08	<.001
Random Parts
N_{program_ID}	9
ICC_{program_ID}	0.064
Observations	142
R² / Ω₀²	.447 / .446

m4bii <- lmer(post_future_goals_plans ~ 
                  pred_challenge + pred_learning + pred_relevance +
                  scale(pre_future_goals_plans, scale = F) + (1|program_ID), 
              data = mod_df)
sjPlot::sjt.lmer(m4bii, p.kr = F, show.re.var = F, show.ci = F, show.se = T)

## Computing p-values via Wald-statistics approximation (treating t as Wald z).

	post_future_goals_plans
	B	std. Error	p
Fixed Parts
(Intercept)	2.14	0.38	<.001
pred_challenge	-0.11	0.13	.421
pred_learning	-0.08	0.27	.774
pred_relevance	0.43	0.26	.092
scale(pre_future_goals_plans, scale = F)	0.41	0.09	<.001
Random Parts
N_{program_ID}	9
ICC_{program_ID}	0.000
Observations	134
R² / Ω₀²	.268 / .268

STEM-IE Models

Joshua Rosenberg

04 October, 2017