Longer-term outcomes question (RQ #4)

RQ: Are challenge, relevance, possibly learning and positive affect during STEM activities across several weeks associated with changes in: (a) individual (sustained) interest in STEM and future goals and aspirations related to STEM? (We have been answering this question.)

RQ: Are situational (momentary) interest and engagement in STEM activities across several weeks associated with changes in: (a) individual (sustained) interest in STEM and future goals and aspirations related to STEM?

Processing data and correlations

A bit more processing.

mod_df <- mod_df %>% 
    left_join(act) %>%
    left_join(loc) %>% 
    rename(lab = `ACT_Lab Activity`,
           create = `ACT_Creating Product`,
           not_focused = `ACT_Not Focused`,
           basic = `ACT_Basic Skills Activity`,
           psl = `ACT_Program Staff Led`,
           fts = `ACT_Field Trip Speaker`,
           class_space = `LOC_Classroom Space`,
           in_comm = `LOC_Outdoors - Community`) %>% 
    mutate(doing = lab + create)

mod_df %>% 
    select(contains("pred"), overall_pre_interest, overall_post_interest, overall_post_interest, pre_future_goals_plans,
           post_future_goals_plans) %>% 
    corrr::correlate() %>% 
    corrr::rearrange() %>% 
    fashion() %>% 
    knitr::kable()

rowname	pred_relevance	pred_interest	pred_engagement	pred_learning	pred_positive_affect	pred_challenge	post_future_goals_plans	pre_future_goals_plans	overall_post_interest	overall_pre_interest
pred_relevance		1.00	.93	.86	.67	.58	.30	.28	.17	.11
pred_interest	1.00		.93	.86	.67	.58	.30	.28	.17	.11
pred_engagement	.93	.93		.90	.76	.52	.30	.27	.27	.14
pred_learning	.86	.86	.90		.66	.46	.28	.23	.21	.10
pred_positive_affect	.67	.67	.76	.66		.42	.25	.25	.24	.17
pred_challenge	.58	.58	.52	.46	.42		.08	.05	-.06	.01
post_future_goals_plans	.30	.30	.30	.28	.25	.08		.47	.47	.20
pre_future_goals_plans	.28	.28	.27	.23	.25	.05	.47		.33	.39
overall_post_interest	.17	.17	.27	.21	.24	-.06	.47	.33		.59
overall_pre_interest	.11	.11	.14	.10	.17	.01	.20	.39	.59

mod_df %>% 
    select(lab, create, not_focused, basic, psl, fts) %>% 
    corrr::correlate() %>% 
    corrr::rearrange() %>% 
    fashion() %>% 
    knitr::kable()

rowname	fts	lab	create	not_focused	basic	psl
fts		-.14	.15	-.16	-.28	-.15
lab	-.14		-.47	.39	-.27	.07
create	.15	-.47		-.06	-.45	-.65
not_focused	-.16	.39	-.06		-.72	-.65
basic	-.28	-.27	-.45	-.72		.79
psl	-.15	.07	-.65	-.65	.79

These correlations suggest that (at the program level):

basic and program staff led are distinct from one another and not correlated with the other variables
not focused and lab are somewhat correlated, but distinct
lab and create are negatively correlated
field trip speaker is not very highly correlated with many other variables

Perhaps we could use:

basic
program staff led
lab or create (combined)?

mod_df <- mutate(mod_df,
                 lab_or_create = lab + create)

Exploring missing data

Specific variables:

mod_df %>% 
    select(participant_ID, program_name, gender, pred_challenge, pred_relevance, overall_pre_interest, overall_post_interest, pre_future_goals_plans, post_future_goals_plans, lab, create, basic) %>% 
    skimr::skim()

## Numeric Variables
## # A tibble: 9 x 13
##                       var    type missing complete     n       mean
##                     <chr>   <chr>   <dbl>    <dbl> <dbl>      <dbl>
## 1                   basic numeric       0      203   203 0.21187398
## 2                  create numeric       0      203   203 0.17663357
## 3                     lab numeric       0      203   203 0.02509145
## 4   overall_post_interest numeric      44      159   203 3.09433962
## 5    overall_pre_interest numeric      22      181   203 3.04419890
## 6 post_future_goals_plans numeric      50      153   203 2.80718954
## 7  pre_future_goals_plans numeric      25      178   203 2.73033708
## 8          pred_challenge numeric       0      203   203 2.27636813
## 9          pred_relevance numeric       0      203   203 2.57620061
## # ... with 7 more variables: sd <dbl>, min <dbl>, `25% quantile` <dbl>,
## #   median <dbl>, `75% quantile` <dbl>, max <dbl>, hist <chr>
## 
## Character Variables
## # A tibble: 3 x 9
##              var      type complete missing empty     n   min   max
## *          <chr>     <chr>    <dbl>   <dbl> <dbl> <dbl> <dbl> <dbl>
## 1         gender character      201       2     0   203     1     1
## 2 participant_ID character      203       0     0   203     4     5
## 3   program_name character      203       0     0   203    18    29
## # ... with 1 more variables: n_unique <dbl>

Overall :

mod_df %>% 
    select(participant_ID, program_name, gender, pred_challenge, pred_relevance, overall_pre_interest, overall_post_interest, pre_future_goals_plans, post_future_goals_plans, lab, create, basic) %>% 
    complete.cases() %>% 
    table() # this suggests that 70 obs are not complete cases

## .
## FALSE  TRUE 
##    70   133

Examining raw data (not run):

mod_df %>% 
    select(participant_ID, program_name, gender, pred_challenge, pred_relevance, overall_pre_interest, overall_post_interest, pre_future_goals_plans, post_future_goals_plans, lab, create, basic) %>% 
    mutate_if(is.numeric, round) %>% 
    knitr::kable()

Models with momentary predictors of interest and engagement (challenge, relevance, possibly learning and positive affect)

m4a <- lmer(overall_post_interest ~ 
                pred_challenge + pred_relevance + 
                gender +
                scale(overall_pre_interest, scale = F) + 
                (1|program_ID), 
            data = mod_df)

sjPlot::sjt.lmer(m4a, p.kr = F, show.re.var = F, show.ci = F, show.se = T)

## Warning in checkMatrixPackageVersion(): Package version inconsistency detected.
## TMB was built with Matrix version 1.2.10
## Current Matrix version is 1.2.11
## Please re-install 'TMB' from source or restore original 'Matrix' package

	overall_post_interest
	B	std. Error	p
Fixed Parts
(Intercept)	2.75	0.27	<.001
pred_challenge	-0.21	0.11	.073
pred_relevance	0.29	0.11	.011
gender (M)	0.15	0.13	.250
scale(overall_pre_interest, scale = F)	0.54	0.08	<.001
Random Parts
N_{program_ID}	9
ICC_{program_ID}	0.044
Observations	141
R² / Ω₀²	.420 / .420

Relevance is a significant positive predictor.

With video codes added in:

m4ai <- lmer(overall_post_interest ~ 
                 pred_challenge + pred_relevance + 
                 gender +
                 scale(lab_or_create) + scale(psl) + scale(basic) + 
                 scale(overall_pre_interest, scale = F) + 
                 (1|program_ID), 
             data = mod_df)

sjPlot::sjt.lmer(m4ai, p.kr = F, show.re.var = F, show.ci = F, show.se = T)

	overall_post_interest
	B	std. Error	p
Fixed Parts
(Intercept)	2.79	0.27	<.001
pred_challenge	-0.22	0.11	.056
pred_relevance	0.29	0.11	.010
gender (M)	0.13	0.13	.293
scale(lab_or_create)	-0.01	0.10	.942
scale(psl)	0.03	0.13	.819
scale(basic)	-0.19	0.12	.108
scale(overall_pre_interest, scale = F)	0.52	0.08	<.001
Random Parts
N_{program_ID}	9
ICC_{program_ID}	0.020
Observations	141
R² / Ω₀²	.425 / .425

Relevance is a significant positive predictor; challenge is (nearly) a negative significant predictor. Basic is approaching a negative, statistically significant relationship.

m4b <- lmer(post_future_goals_plans ~ 
                pred_challenge + pred_relevance + 
                gender +
                scale(pre_future_goals_plans, scale = F) + (1|program_ID), 
            data = mod_df)

sjPlot::sjt.lmer(m4b, p.kr = F, show.re.var = F, show.ci = F, show.se = T)

	post_future_goals_plans
	B	std. Error	p
Fixed Parts
(Intercept)	2.05	0.31	<.001
pred_challenge	-0.13	0.13	.332
pred_relevance	0.33	0.13	.010
gender (M)	0.37	0.14	.011
scale(pre_future_goals_plans, scale = F)	0.36	0.09	<.001
Random Parts
N_{program_ID}	9
ICC_{program_ID}	0.000
Observations	133
R² / Ω₀²	.302 / .302

Relevance is a significant posive predictor of changes in future goals and plans; gender is, as well.

With video codes added in:

m4bi <- lmer(post_future_goals_plans ~ 
                 pred_challenge + pred_relevance + 
                 gender +
                 scale(lab_or_create) + scale(psl) + scale(basic) + 
                 scale(pre_future_goals_plans, scale = F) + (1|program_ID), 
             data = mod_df)

sjPlot::sjt.lmer(m4bi, p.kr = F, show.re.var = F, show.ci = F, show.se = T)

	post_future_goals_plans
	B	std. Error	p
Fixed Parts
(Intercept)	2.03	0.31	<.001
pred_challenge	-0.12	0.13	.357
pred_relevance	0.34	0.13	.011
gender (M)	0.38	0.15	.011
scale(lab_or_create)	0.02	0.10	.855
scale(psl)	0.05	0.13	.686
scale(basic)	0.01	0.12	.961
scale(pre_future_goals_plans, scale = F)	0.36	0.09	<.001
Random Parts
N_{program_ID}	9
ICC_{program_ID}	0.000
Observations	133
R² / Ω₀²	.305 / .305

Relevance is a significant posive predictor of changes in future goals and plans; gender is, as well.

Models with momentary interest and engagement as predictors

Without video codes added in

Just interest:

m4a <- lmer(overall_post_interest ~ 
                pred_interest + 
                gender +
                scale(overall_pre_interest, scale = F) + 
                (1|program_ID), 
            data = mod_df)

sjPlot::sjt.lmer(m4a, p.kr = F, show.re.var = F, show.ci = F, show.se = T)

	overall_post_interest
	B	std. Error	p
Fixed Parts
(Intercept)	2.51	0.28	<.001
pred_interest	0.18	0.10	.063
gender (M)	0.13	0.13	.304
scale(overall_pre_interest, scale = F)	0.56	0.08	<.001
Random Parts
N_{program_ID}	9
ICC_{program_ID}	0.038
Observations	141
R² / Ω₀²	.404 / .403

Interest is approaching statistical signficance as a positive predictor.

Just engagement:

m4b <- lmer(overall_post_interest ~ 
                pred_engagement +
                gender +
                scale(overall_pre_interest, scale = F) + 
                (1|program_ID), 
            data = mod_df)

sjPlot::sjt.lmer(m4b, p.kr = F, show.re.var = F, show.ci = F, show.se = T)

	overall_post_interest
	B	std. Error	p
Fixed Parts
(Intercept)	2.00	0.31	<.001
pred_engagement	0.37	0.11	<.001
gender (M)	0.10	0.12	.406
scale(overall_pre_interest, scale = F)	0.54	0.07	<.001
Random Parts
N_{program_ID}	9
ICC_{program_ID}	0.046
Observations	141
R² / Ω₀²	.441 / .441

Engagement predicts changes in post interest.

Both interest and engagement:

m4c <- lmer(overall_post_interest ~ 
                pred_interest + pred_engagement +
                gender +
                scale(overall_pre_interest, scale = F) + 
                (1|program_ID), 
            data = mod_df)

sjPlot::sjt.lmer(m4c, p.kr = F, show.re.var = F, show.ci = F, show.se = T)

	overall_post_interest
	B	std. Error	p
Fixed Parts
(Intercept)	1.87	0.30	<.001
pred_interest	-0.89	0.24	<.001
pred_engagement	1.33	0.28	<.001
gender (M)	0.17	0.12	.161
scale(overall_pre_interest, scale = F)	0.53	0.07	<.001
Random Parts
N_{program_ID}	9
ICC_{program_ID}	0.046
Observations	141
R² / Ω₀²	.490 / .490

Interest is a negative significant predicotr of changes in interest; engagement is a positive predictor of changes.

With video codes added in

Just interest:

m5a <- lmer(post_future_goals_plans ~
                pred_interest + 
                gender +
                scale(lab_or_create) + scale(psl) + scale(basic) + 
                scale(pre_future_goals_plans, scale = F) + 
                (1|program_ID), 
            data = mod_df)

sjPlot::sjt.lmer(m5a, p.kr = F, show.re.var = F, show.ci = F, show.se = T)

	post_future_goals_plans
	B	std. Error	p
Fixed Parts
(Intercept)	1.86	0.32	<.001
pred_interest	0.27	0.11	.015
gender (M)	0.37	0.15	.013
scale(lab_or_create)	0.03	0.10	.779
scale(psl)	0.05	0.13	.677
scale(basic)	0.01	0.11	.908
scale(pre_future_goals_plans, scale = F)	0.37	0.09	<.001
Random Parts
N_{program_ID}	9
ICC_{program_ID}	0.000
Observations	133
R² / Ω₀²	.300 / .300

Interest is approaching statistical signficance as a positive predictor of future goals and plans. Males are associated with changes in future goals and plans.

Just engagement:

m5b <- lmer(post_future_goals_plans ~
                pred_engagement +
                gender +
                scale(lab_or_create) + scale(psl) + scale(basic) + 
                scale(pre_future_goals_plans, scale = F) + 
                (1|program_ID), 
            data = mod_df)

sjPlot::sjt.lmer(m5b, p.kr = F, show.re.var = F, show.ci = F, show.se = T)

	post_future_goals_plans
	B	std. Error	p
Fixed Parts
(Intercept)	1.75	0.36	<.001
pred_engagement	0.31	0.13	.015
gender (M)	0.38	0.15	.010
scale(lab_or_create)	0.04	0.10	.662
scale(psl)	0.05	0.13	.678
scale(basic)	0.03	0.12	.776
scale(pre_future_goals_plans, scale = F)	0.38	0.09	<.001
Random Parts
N_{program_ID}	9
ICC_{program_ID}	0.000
Observations	133
R² / Ω₀²	.300 / .300

Engagement predicts changes in future goals and plans, as does gender.

Both interest and engagement:

m5c <- lmer(post_future_goals_plans ~
                pred_interest + pred_engagement +
                gender +
                scale(lab_or_create) + scale(psl) + scale(basic) + 
                scale(pre_future_goals_plans, scale = F) + 
                (1|program_ID), 
            data = mod_df)

sjPlot::sjt.lmer(m5c, p.kr = F, show.re.var = F, show.ci = F, show.se = T)

	post_future_goals_plans
	B	std. Error	p
Fixed Parts
(Intercept)	1.78	0.37	<.001
pred_interest	0.14	0.30	.631
pred_engagement	0.16	0.35	.658
gender (M)	0.37	0.15	.013
scale(lab_or_create)	0.04	0.10	.723
scale(psl)	0.05	0.13	.676
scale(basic)	0.02	0.12	.846
scale(pre_future_goals_plans, scale = F)	0.37	0.09	<.001
Random Parts
N_{program_ID}	9
ICC_{program_ID}	0.000
Observations	133
R² / Ω₀²	.301 / .301

Males are associated with changes in future goals and plans.

STEM-IE Longer-term Outcomes Paper

Joshua Rosenberg

10/12/2017