Final_Project
Greg Gallagher
2024-04-22
Preamble
library(tidyverse)## Warning: package 'tidyverse' was built under R version 4.3.2## Warning: package 'ggplot2' was built under R version 4.3.2## Warning: package 'tidyr' was built under R version 4.3.2## Warning: package 'readr' was built under R version 4.3.2## Warning: package 'purrr' was built under R version 4.3.2## Warning: package 'dplyr' was built under R version 4.3.2## Warning: package 'stringr' was built under R version 4.3.2## Warning: package 'forcats' was built under R version 4.3.2## Warning: package 'lubridate' was built under R version 4.3.2## ── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ──
## ✔ dplyr 1.1.4 ✔ readr 2.1.5
## ✔ forcats 1.0.0 ✔ stringr 1.5.1
## ✔ ggplot2 3.4.4 ✔ tibble 3.2.1
## ✔ lubridate 1.9.3 ✔ tidyr 1.3.1
## ✔ purrr 1.0.2
## ── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
## ✖ dplyr::filter() masks stats::filter()
## ✖ dplyr::lag() masks stats::lag()
## ℹ Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errorslibrary(purrr)
library(plotrix)## Warning: package 'plotrix' was built under R version 4.3.2library(readxl)
library(ggthemes)## Warning: package 'ggthemes' was built under R version 4.3.2library(ggrepel)## Warning: package 'ggrepel' was built under R version 4.3.2library(effsize)## Warning: package 'effsize' was built under R version 4.3.3library(pwrss)## Warning: package 'pwrss' was built under R version 4.3.3##
## Attaching package: 'pwrss'
##
## The following object is masked from 'package:stats':
##
## power.t.testDataset
data0 <- read_excel("~/IUPUI/Research/PERC_2024/Datasets/Skill_Growth_F18_F23.xlsx")
view(data0)Abstract
While computational methods of problem-solving have begun to be a focal point of many STEM curricula, little information is currently available to evaluate the efficacy of these initiatives. Using data gathered via undergraduate student surveys, we present our efforts to evaluate the efficacy of a departmental project with the goal of normalizing computational methods of problem-solving across the undergraduate physics curriculum. We examine the effects of students re-evaluating their initial (defined as when they began the program) skills by introducing a survey that asks students to evaluate their initial competency in a variety of computational skills. It is shown that students tend to overestimate their competency in computational skills earlier in the program (i.e., 100-, 200-level students), which supports our assertion that asking students to rate their present and initial skills in a survey is a better indicator of student growth than simply asking students to rate their present skill level alone.
Introduction
Introducing computational methods of problem solving in physics has become a focus for many undergraduate physics programs over the last few decades, perhaps evidenced best by a recent report by the American Association of Physics Teachers (AAPT) suggesting methods of integrating computational methods into undergraduate physics programs.These suggestions were soon reinforced by an APS-AAPT joint task force which, in its final report, emphasized the importance of computational skills for physics majors as they look toward future careers.
While most agree with the importance of physics students gaining computational skills through the undergraduate curriculum, there is little information available for institutions which have introduced computation as an undergraduate-curriculum-wide focus. Some notable exceptions to this would be the physics programs at Oregon State University and Lawrence University, which have both sought to incorporate computation holistically across the undergraduate curriculum.
Further compounding the issue has been the apparent lack of instruments to evaluate the efficacy of these efforts to integrate computational methods into the undergraduate curriculum, highlighted by the notable exclusion of assessment tools from physics education projects such as the PICUP project and PhysPort. This apparent lack of assessment tools is primarily what the survey evaluated in this project seeks to address. The survey can be broken down into parts which seek to accomplish two goals: first is to measure student attitudes toward common computational tools applied to physics, and the second is to measure student growth in specific computational skills as they progress through the undergraduate physics program. It is the second part of the survey - measuring student growth - that this project evaluates.
Context
Survey Instrument
The survey instrument evaluated in this paper was originally developed as a method to evaluate an initiative undertaken in 2016 by the faculty of the physics department at Indiana University Purdue University Indianapolis (IUPUI) requiring that at least 25% of assignments in each undergraduate physics course contain a computational component.
The overarching goal of this initiative has always been to normalize computational methods of problem-solving in physics. It was apparent from the onset that there was a need to evaluate the efficacy of this initiative, ultimately resulting in the development of the survey instrument evaluated in this study. The development, review, and refinement of the present instrument took place in stages. First, initial items were developed by project leaders and further discussed and adjusted by the full group of faculty members in physics. The instrument begins with demographic questions (names, student IDs, physics courses taken that semester). These are followed by items asking students to rate their agreement with statements related to computational physics, e.g., “Using computational methods helps me understand physics topics’’ (five-point Likert scale). Students are then asked to rate both their present and initial abilities on ten computational skills, e.g., numerical integration, on a 1-10 scale (initial is defined as”at the time they began the program’’). During the second stage of development, the instrument was reviewed by an evaluation expert with instrument development experience at IUPUI’s STEM Education and Innovation Research Institute (SEIRI), which resulted in a few items being reworded for better clarity.
This survey was first implemented at the conclusion of the fall semester of 2018 and has been given at the end of each semester since. While the survey did go through some re-tooling at the end of the Spring 2020 semester where four more Likert-scale items were added, the self-efficacy items evaluated in this paper have not changed.
The survey instrument evaluated in this project contains 33 individual survey items, which can be broken up into three distinct parts. The first two items ask students to provide information such as their student ID number (which is later removed as a part of the de-identifying process before the researchers view the data) and course information. The next 9 items ask students “attitudinal” prompts to gauge when students develop “expert-like” attitudes. The latter 22 items include 11 prompts for students to report their current self-perceived competency in a variety of computational skills on a scale of 1-10. The other 11 prompts ask students to then report their initial self-perceived competency for those same 11 skills, again on a scale of 1-10. It is these last 22 prompts that this project evaluates.
For convenience, the skills students are asked to rate their competency in are (in order): Excel(1), LaTeX(2), MATLAB(3), Other Language(4), Dimensionless Analysis(5), Testing Code(6), Creating Visualizations(7), Matrix Operations(8), Differential Equations(9), Numerical Integration/Differentiation(10), and Data Analysis(11).
Methods
Survey Methods and Participants
The survey is given at the end of each Fall and Spring semester to all students enrolled in courses that are a part of the physics undergraduate curriculum. They receive two prompts via email and are asked to complete the survey online. The survey data is then organized and de-identified before being sent to the researchers.
As a part of the process, we require that each survey used in the analysis be complete (i.e., all 22 items must have a response), and that we use only one completed survey per participant. These requirements resulted in dropping 79 surveys due to incomplete surveys and 22 surveys due to repeats. It is important to note that if students completed multiple surveys, we kept the last complete survey for use in the analysis. The total number of participants after dropping surveys is then 516, with N(100)=247, N(200)=210, N(300)=36, and N(400)=23.
Growth Analysis
With the goal of determining how much students gain competency in computational skills, a metric must be defined for measuring the “growth” in skills as they progress through the program. To that end, this study defines “growth” as \(Growth = \bar{P}_{X00} - \bar{I}_{X00}\), where \(\bar{P}_{X00}\) and \(\bar{I}_{X00}\) are the means of the present and initial responses for each academic level (100, 200, 300, 400).
Typically, measuring student gains in competency as they progress through the entire program would result a simple comparison of the mean present competencies for each survey item across academic level. While this method is more efficient and typical, it ignores the possibility of a student re-evaluating the scale, specifically their initial competency, as they gain more knowledge. Justin Kruger and David Dunning, in fact, showed that people tend to overestimate their competency when they are unaware of their own competency level.
We then propose two distinct definitions of student growth for comparison: \(G_1\) and \(G_2\). \(G_1\) is defined simply as: \[ G_1 = \bar{P}_{400} - \bar{P}_{100}, \] where \(\bar{P}_{400}\) is defined as the mean of the present competency of 400-level students, and \(\bar{P}_{100}\) is defined as the mean of the present competency of 100-level students. This is usually the metric surveys use to measure growth, as it relies only on participants reporting their current skill competencies.
The second method, \(G_2\), includes the possibility that students will re-evaluate their own initial starting point (i.e., they revisit their own definition of the scale) as they gain knowledge, and is defined as: \[ G_2 = (\bar{P}_{400} - \bar{I}_{400}) - (\bar{P}_{100} - \bar{I}_{100}), \]
where the inclusion of the initial competency is given by \(\bar{I}_{400}\) and \(\bar{I}_{100}\), which are the means of the initial competencies of the 400-level and 100-level students, respectively. Rearranging this provides a clearer picture of the effect of introducing the initial terms: \[ G_2 = (\bar{P}_{400} - \bar{P}_{100}) - (\bar{I}_{400} - \bar{I}_{100}). \]
Using the definition of \(G_1\), and introducing the term \(I_{eff} = (\bar{I}_{100} - \bar{I}_{400})\) which we define as the effect of the re-evaluation of students’ initial competency, the above equation can be given simply as: \[ G_2 = G_1 + I_{eff} \]
Ideally, if students did not re-evaluate their own skills, \(I_{eff} = 0\), but as we will show later on (see: Results), this assumption is not reflective of reality. It is also important to note that \(I_{eff}\) includes both the 400- and 100-level students in order to maintain a consistent comparison with \(G_1\) in terms of when the students are surveyed. The surveys are given at the end of each semester, so \(G_1\) measures the skill difference between a 400 and 100-level student at the end of the semester, while the prompts for the initial state define initial as when the student first began the undergraduate program. For this reason, \(G_2\) subtracts off the growth made by the students in the first semester so that the comparison is consistent with the timeline of when surveys are given.
Results
Initial Skill Competency
First, let’s plot the mean initial competency of each skill per undergraduate level surveyed (i.e., 100-, 200-. 300-. and 400-level), as well as with the associated standard errors as error bars. For this analysis, the fourth survey item -“Other Language” - was not included in this analysis due to the high variety of other programming languages which leads to little relevant information to be gathered from this analysis.
i_means <-
data0 |>
group_by(Overall_Level)|>
summarise(avg_i1 = mean(Initial1, na.rm = TRUE),
se_i1 = std.error(Initial1, na.rm = TRUE),
avg_i2 = mean(Initial2, na.rm = TRUE),
se_i2 = std.error(Initial2, na.rm = TRUE),
avg_i3 = mean(Initial3, na.rm = TRUE),
se_i3 = std.error(Initial3, na.rm = TRUE),
avg_i5 = mean(Initial5, na.rm = TRUE),
se_i5 = std.error(Initial5, na.rm = TRUE),
avg_i6 = mean(Initial6, na.rm = TRUE),
se_i6 = std.error(Initial6, na.rm = TRUE),
avg_i7 = mean(Initial7, na.rm = TRUE),
se_i7 = std.error(Initial7, na.rm = TRUE),
avg_i8 = mean(Initial8, na.rm = TRUE),
se_i8 = std.error(Initial8, na.rm = TRUE),
avg_i9 = mean(Initial9, na.rm = TRUE),
se_i9 = std.error(Initial9, na.rm = TRUE),
avg_i10 = mean(Initial10, na.rm = TRUE),
se_i10 = std.error(Initial10, na.rm = TRUE),
avg_i11 = mean(Initial11, na.rm = TRUE),
se_i11 = std.error(Initial11, na.rm = TRUE))
view(i_means)i1_plot <-
ggplot(i_means)+
geom_col(mapping = aes(x=as.character(Overall_Level), y=avg_i1))+
geom_errorbar(mapping=aes(x = as.character(Overall_Level), ymin = avg_i1 - se_i1, ymax = avg_i1 + se_i1))
i2_plot <-
ggplot(i_means)+
geom_col(mapping = aes(x=as.character(Overall_Level), y=avg_i2))+
geom_errorbar(mapping=aes(x = as.character(Overall_Level), ymin = avg_i2 - se_i2, ymax = avg_i2 + se_i2))
i3_plot <-
ggplot(i_means)+
geom_col(mapping = aes(x=as.character(Overall_Level), y=avg_i3))+
geom_errorbar(mapping=aes(x = as.character(Overall_Level), ymin = avg_i3 - se_i3, ymax = avg_i3 + se_i3))
i5_plot <-
ggplot(i_means)+
geom_col(mapping = aes(x=as.character(Overall_Level), y=avg_i5))+
geom_errorbar(mapping=aes(x = as.character(Overall_Level), ymin = avg_i5 - se_i5, ymax = avg_i5 + se_i5))
i6_plot <-
ggplot(i_means)+
geom_col(mapping = aes(x=as.character(Overall_Level), y=avg_i6))+
geom_errorbar(mapping=aes(x = as.character(Overall_Level), ymin = avg_i6 - se_i6, ymax = avg_i6 + se_i6))
i7_plot <-
ggplot(i_means)+
geom_col(mapping = aes(x=as.character(Overall_Level), y=avg_i7))+
geom_errorbar(mapping=aes(x = as.character(Overall_Level), ymin = avg_i7 - se_i7, ymax = avg_i7 + se_i7))
i8_plot <-
ggplot(i_means)+
geom_col(mapping = aes(x=as.character(Overall_Level), y=avg_i8))+
geom_errorbar(mapping=aes(x = as.character(Overall_Level), ymin = avg_i8 - se_i8, ymax = avg_i8 + se_i8))
i9_plot <-
ggplot(i_means)+
geom_col(mapping = aes(x=as.character(Overall_Level), y=avg_i9))+
geom_errorbar(mapping=aes(x = as.character(Overall_Level), ymin = avg_i9 - se_i9, ymax = avg_i9 + se_i9))
i10_plot <-
ggplot(i_means)+
geom_col(mapping = aes(x=as.character(Overall_Level), y=avg_i10))+
geom_errorbar(mapping=aes(x = as.character(Overall_Level), ymin = avg_i10 - se_i10, ymax = avg_i10 + se_i10))
i11_plot <-
ggplot(i_means)+
geom_col(mapping = aes(x=as.character(Overall_Level), y=avg_i11))+
geom_errorbar(mapping=aes(x = as.character(Overall_Level), ymin = avg_i11 - se_i11, ymax = avg_i11 + se_i11))
i1_plot+
labs(title="Excel-Initial")+
theme(plot.title = element_text(hjust = 0.5))
i2_plot+
labs(title="LaTeX-Initial")+
theme(plot.title = element_text(hjust = 0.5))
i3_plot+
labs(title="MATLAB-Initial")+
theme(plot.title = element_text(hjust = 0.5))
i5_plot+
labs(title="Dimensionless Analysis-Initial")+
theme(plot.title = element_text(hjust = 0.5))
i6_plot+
labs(title="Testing Code-Initial")+
theme(plot.title = element_text(hjust = 0.5))
i7_plot+
labs(title="Creating Visualizations-Initial")+
theme(plot.title = element_text(hjust = 0.5))
i8_plot+
labs(title="Matrix Operations-Initial")+
theme(plot.title = element_text(hjust = 0.5))
i9_plot+
labs(title="Differential Equations-Initial")+
theme(plot.title = element_text(hjust = 0.5))
i10_plot+
labs(title="Numerical Int/Diff-Initial")+
theme(plot.title = element_text(hjust = 0.5))
i11_plot+
labs(title="Data Analysis-Initial")+
theme(plot.title = element_text(hjust = 0.5))
## Present Skill Competency
Then, we do the same thing, but for the present competency of each item.
p_means <-
data0 |>
group_by(Overall_Level)|>
summarise(avg_p1 = mean(Present1, na.rm = TRUE),
se_p1 = std.error(Present1, na.rm = TRUE),
avg_p2 = mean(Present2, na.rm = TRUE),
se_p2 = std.error(Present2, na.rm = TRUE),
avg_p3 = mean(Present3, na.rm = TRUE),
se_p3 = std.error(Present3, na.rm = TRUE),
avg_p5 = mean(Present5, na.rm = TRUE),
se_p5 = std.error(Present5, na.rm = TRUE),
avg_p6 = mean(Present6, na.rm = TRUE),
se_p6 = std.error(Present6, na.rm = TRUE),
avg_p7 = mean(Present7, na.rm = TRUE),
se_p7 = std.error(Present7, na.rm = TRUE),
avg_p8 = mean(Present8, na.rm = TRUE),
se_p8 = std.error(Present8, na.rm = TRUE),
avg_p9 = mean(Present9, na.rm = TRUE),
se_p9 = std.error(Present9, na.rm = TRUE),
avg_p10 = mean(Present10, na.rm = TRUE),
se_p10 = std.error(Present10, na.rm = TRUE),
avg_p11 = mean(Present11, na.rm = TRUE),
se_p11 = std.error(Present11, na.rm = TRUE))
view(p_means)p1_plot <-
ggplot(p_means)+
geom_col(mapping = aes(x=as.character(Overall_Level), y=avg_p1))+
geom_errorbar(mapping=aes(x = as.character(Overall_Level), ymin = avg_p1 - se_p1, ymax = avg_p1 + se_p1))
p2_plot <-
ggplot(p_means)+
geom_col(mapping = aes(x=as.character(Overall_Level), y=avg_p2))+
geom_errorbar(mapping=aes(x = as.character(Overall_Level), ymin = avg_p2 - se_p2, ymax = avg_p2 + se_p2))
p3_plot <-
ggplot(p_means)+
geom_col(mapping = aes(x=as.character(Overall_Level), y=avg_p3))
geom_errorbar(mapping=aes(x = as.character(Overall_Level), ymin = avg_p3 - se_p3, ymax = avg_p3 + se_p3))## mapping: x = ~as.character(Overall_Level), ymin = ~avg_p3 - se_p3, ymax = ~avg_p3 + se_p3
## geom_errorbar: na.rm = FALSE, orientation = NA
## stat_identity: na.rm = FALSE
## position_identityp5_plot <-
ggplot(p_means)+
geom_col(mapping = aes(x=as.character(Overall_Level), y=avg_p5))+
geom_errorbar(mapping=aes(x = as.character(Overall_Level), ymin = avg_p5 - se_p5, ymax = avg_p5 + se_p5))
p6_plot <-
ggplot(p_means)+
geom_col(mapping = aes(x=as.character(Overall_Level), y=avg_p6))+
geom_errorbar(mapping=aes(x = as.character(Overall_Level), ymin = avg_p6 - se_p6, ymax = avg_p6 + se_p6))
p7_plot <-
ggplot(p_means)+
geom_col(mapping = aes(x=as.character(Overall_Level), y=avg_p7))+
geom_errorbar(mapping=aes(x = as.character(Overall_Level), ymin = avg_p7 - se_p7, ymax = avg_p7 + se_p7))
p8_plot <-
ggplot(p_means)+
geom_col(mapping = aes(x=as.character(Overall_Level), y=avg_p8))+
geom_errorbar(mapping=aes(x = as.character(Overall_Level), ymin = avg_p8 - se_p8, ymax = avg_p8 + se_p8))
p9_plot <-
ggplot(p_means)+
geom_col(mapping = aes(x=as.character(Overall_Level), y=avg_p9))+
geom_errorbar(mapping=aes(x = as.character(Overall_Level), ymin = avg_p9 - se_p9, ymax = avg_p9 + se_p9))
p10_plot <-
ggplot(p_means)+
geom_col(mapping = aes(x=as.character(Overall_Level), y=avg_p10))+
geom_errorbar(mapping=aes(x = as.character(Overall_Level), ymin = avg_p10 - se_p10, ymax = avg_p10 + se_p10))
p11_plot <-
ggplot(p_means)+
geom_col(mapping = aes(x=as.character(Overall_Level), y=avg_p11))+
geom_errorbar(mapping=aes(x = as.character(Overall_Level), ymin = avg_p11 - se_p11, ymax = avg_p11 + se_p11))
p1_plot+
labs(title="Excel-Present")+
theme(plot.title = element_text(hjust = 0.5))
p2_plot+
labs(title="LaTeX-Present")+
theme(plot.title = element_text(hjust = 0.5))
p3_plot+
labs(title="MATLAB-Present")+
theme(plot.title = element_text(hjust = 0.5))
p5_plot+
labs(title="Dimensionless Analysis-Present")+
theme(plot.title = element_text(hjust = 0.5))
p6_plot+
labs(title="Testing Code-Present")+
theme(plot.title = element_text(hjust = 0.5))
p7_plot+
labs(title="Creating Visualizations-Present")+
theme(plot.title = element_text(hjust = 0.5))
p8_plot+
labs(title="Matrix Operations-Present")+
theme(plot.title = element_text(hjust = 0.5))
p9_plot+
labs(title="Differential Equations-Present")+
theme(plot.title = element_text(hjust = 0.5))
p10_plot+
labs(title="Numerical Int/Diff-Present")+
theme(plot.title = element_text(hjust = 0.5))
p11_plot+
labs(title="Data Analysis-Present")+
theme(plot.title = element_text(hjust = 0.5))
It is worth mentioning that as a part of a previous data dive, a one-way omnibus ANOVA was conducted for each group of questions (initial, present) and shows statistically significant (\(p \leq 0.05\)) differences in the initial competency across levels for items Q3 (\(p=0.019\)), Q5 (\(p<0.001\)), Q9 (\(p=0.044\)), Q10 (\(p<0.001\)), and Q11 (\(p<0.001\)). The present competency levels showed statistically significant differences among levels for all but one survey item (Q1). Items Q2, Q3, Q6, Q7, Q8, and Q9 all had significance of \(p<0.001\). Items Q5 (\(p<0.01\)), Q10 (\(p=0.050\)), and Q11 (\(p=0.030)\) showed significant difference among levels as well.
400-Level Student Growth
One of the most important questions to ask about this data is simply: “How much do students feel they have gained in competency for each skill by the time they leave the program?” A good way to answer this question is to simply look at the 400-level student data and measure the growth per each item. First though, we want to ensure we can learn meaningful information from such a plot, so we can conduct a pairwise t-test between the present and initial responses for the 400-level students.
First, filtering the entire dataset for just the 400-level students, then completing a pairwise t-test for each survey item:
data400 <-
data0 |>
filter(Overall_Level == 400)attach(data400)
ttest1 <-
t.test(Initial1, Present1, paired=TRUE)
ttest2 <-
t.test(Initial2, Present2, paired=TRUE)
ttest3 <-
t.test(Initial3, Present3, paired=TRUE)
ttest5 <-
t.test(Initial5, Present5, paired=TRUE)
ttest6 <-
t.test(Initial6, Present6, paired=TRUE)
ttest7 <-
t.test(Initial7, Present7, paired=TRUE)
ttest8 <-
t.test(Initial8, Present8, paired=TRUE)
ttest9 <-
t.test(Initial9, Present9, paired=TRUE)
ttest10 <-
t.test(Initial10, Present10, paired=TRUE)
ttest11 <-
t.test(Initial11, Present11, paired=TRUE)
ttest1##
## Paired t-test
##
## data: Initial1 and Present1
## t = -4.917, df = 22, p-value = 6.443e-05
## alternative hypothesis: true mean difference is not equal to 0
## 95 percent confidence interval:
## -3.770804 -1.533544
## sample estimates:
## mean difference
## -2.652174ttest2##
## Paired t-test
##
## data: Initial2 and Present2
## t = -9.9519, df = 22, p-value = 1.318e-09
## alternative hypothesis: true mean difference is not equal to 0
## 95 percent confidence interval:
## -6.830024 -4.474324
## sample estimates:
## mean difference
## -5.652174ttest3##
## Paired t-test
##
## data: Initial3 and Present3
## t = -13.615, df = 22, p-value = 3.391e-12
## alternative hypothesis: true mean difference is not equal to 0
## 95 percent confidence interval:
## -7.565233 -5.565202
## sample estimates:
## mean difference
## -6.565217ttest5##
## Paired t-test
##
## data: Initial5 and Present5
## t = -8.5994, df = 22, p-value = 1.743e-08
## alternative hypothesis: true mean difference is not equal to 0
## 95 percent confidence interval:
## -5.828073 -3.563232
## sample estimates:
## mean difference
## -4.695652ttest6##
## Paired t-test
##
## data: Initial6 and Present6
## t = -7.5399, df = 22, p-value = 1.556e-07
## alternative hypothesis: true mean difference is not equal to 0
## 95 percent confidence interval:
## -6.153515 -3.498659
## sample estimates:
## mean difference
## -4.826087ttest7##
## Paired t-test
##
## data: Initial7 and Present7
## t = -6.7882, df = 22, p-value = 8.048e-07
## alternative hypothesis: true mean difference is not equal to 0
## 95 percent confidence interval:
## -5.449087 -2.898739
## sample estimates:
## mean difference
## -4.173913ttest8##
## Paired t-test
##
## data: Initial8 and Present8
## t = -8.6098, df = 22, p-value = 1.707e-08
## alternative hypothesis: true mean difference is not equal to 0
## 95 percent confidence interval:
## -6.635975 -4.059677
## sample estimates:
## mean difference
## -5.347826ttest9##
## Paired t-test
##
## data: Initial9 and Present9
## t = -9.206, df = 22, p-value = 5.32e-09
## alternative hypothesis: true mean difference is not equal to 0
## 95 percent confidence interval:
## -5.700189 -3.604159
## sample estimates:
## mean difference
## -4.652174ttest10##
## Paired t-test
##
## data: Initial10 and Present10
## t = -7.8262, df = 22, p-value = 8.484e-08
## alternative hypothesis: true mean difference is not equal to 0
## 95 percent confidence interval:
## -6.269956 -3.643087
## sample estimates:
## mean difference
## -4.956522ttest11##
## Paired t-test
##
## data: Initial11 and Present11
## t = -8.3444, df = 22, p-value = 2.912e-08
## alternative hypothesis: true mean difference is not equal to 0
## 95 percent confidence interval:
## -5.591260 -3.365261
## sample estimates:
## mean difference
## -4.478261In the dataset, growth is already a defined column for each item, so we simply want the mean growth for the 400-level students for each item (as well as the standard error).
Finding the means and standard errors for the growth:
data_g400 <-
data400 |>
summarise(avg_g1 = mean(growth1),
avg_g2 = mean(growth2),
avg_g3 = mean(growth3),
avg_g5 = mean(growth5),
avg_g6 = mean(growth6),
avg_g7 = mean(growth7),
avg_g8 = mean(growth8),
avg_g9 = mean(growth9),
avg_g10 = mean(growth10),
avg_g11 = mean(growth11),
se_g1 = std.error(growth1),
se_g2 = std.error(growth2),
se_g3 = std.error(growth3),
se_g5 = std.error(growth5),
se_g6 = std.error(growth6),
se_g7 = std.error(growth7),
se_g8 = std.error(growth8),
se_g9 = std.error(growth9),
se_g10 = std.error(growth10),
se_g11 = std.error(growth11))
plot_g400_1 <-
ggplot(data_g400)+
geom_col(mapping = aes(x=as.character("Excel"), y=avg_g1))+
geom_errorbar(mapping = aes(x=as.character("Excel"), ymin = avg_g1 - se_g1, ymax = avg_g1 + se_g1))
plot_g400_1
plot_g400_2 <-
ggplot(data_g400)+
geom_col(mapping = aes(x=as.character("LaTeX"), y=avg_g2))+
geom_errorbar(mapping = aes(x=as.character("LaTeX"), ymin = avg_g2 - se_g2, ymax = avg_g2 + se_g2))
plot_g400_2
plot_g400_3 <-
ggplot(data_g400)+
geom_col(mapping = aes(x=as.character("MATLAB"), y=avg_g3))+
geom_errorbar(mapping = aes(x=as.character("MATLAB"), ymin = avg_g3 - se_g3, ymax = avg_g3 + se_g3))
plot_g400_3
plot_g400_5 <-
ggplot(data_g400)+
geom_col(mapping = aes(x=as.character("Dimensionless Analysis"), y=avg_g5))+
geom_errorbar(mapping = aes(x=as.character("Dimensionless Analysis"), ymin = avg_g5 - se_g5, ymax = avg_g5 + se_g5))
plot_g400_5
plot_g400_6 <-
ggplot(data_g400)+
geom_col(mapping = aes(x=as.character("Testing Code"), y=avg_g6))+
geom_errorbar(mapping = aes(x=as.character("Testing Code"), ymin = avg_g6 - se_g6, ymax = avg_g6 + se_g6))
plot_g400_6
plot_g400_7 <-
ggplot(data_g400)+
geom_col(mapping = aes(x=as.character("Visualizations"), y=avg_g7))+
geom_errorbar(mapping = aes(x=as.character("Visualizations"), ymin = avg_g7 - se_g7, ymax = avg_g7 + se_g7))
plot_g400_7
plot_g400_8 <-
ggplot(data_g400)+
geom_col(mapping = aes(x=as.character("Matrix Ops"), y=avg_g8))+
geom_errorbar(mapping = aes(x=as.character("Matrix Ops"), ymin = avg_g8 - se_g8, ymax = avg_g8 + se_g8))
plot_g400_8
plot_g400_9 <-
ggplot(data_g400)+
geom_col(mapping = aes(x=as.character("Diff Eqs"), y=avg_g9))+
geom_errorbar(mapping = aes(x=as.character("Diff Eqs"), ymin = avg_g9 - se_g9, ymax = avg_g9 + se_g9))
plot_g400_9
plot_g400_10 <-
ggplot(data_g400)+
geom_col(mapping = aes(x=as.character("Numerical Int/Diff"), y=avg_g10))+
geom_errorbar(mapping = aes(x=as.character("Numerical Int/Diff"), ymin = avg_g10 - se_g10, ymax = avg_g10 + se_g10))
plot_g400_10
plot_g400_11 <-
ggplot(data_g400)+
geom_col(mapping = aes(x=as.character("Data Analysis"), y=avg_g11))+
geom_errorbar(mapping = aes(x=as.character("Data Analysis"), ymin = avg_g11 - se_g11, ymax = avg_g11 + se_g11))
plot_g400_11
Survey Method Comparison
To begin, let’s run independent t-tests for each survey method, \(G_1\) and \(G_2\), beginning with \(G_1\). It is important to note that as part of a previous data dive, Hedge’s g was calculated for effect size for each method. For \(G_1\), all but three survey items(1, 10, and 11) show \(g>0.5\), which indicates at least a moderate effect. For method \(G_2\), all but the first survey prompt showed \(g >0.8\), indicating a large effect for most survey items.
Running the independent t-test for \(G_2\):
target <- c(100, 400)
data_g <-
data0 |>
filter(Overall_Level %in% target)
g1test1 <- t.test(growth1~Overall_Level, data=data_g)
g1test1##
## Welch Two Sample t-test
##
## data: growth1 by Overall_Level
## t = -2.0377, df = 24.913, p-value = 0.05233
## alternative hypothesis: true difference in means between group 100 and group 400 is not equal to 0
## 95 percent confidence interval:
## -2.28024332 0.01233274
## sample estimates:
## mean in group 100 mean in group 400
## 1.518219 2.652174g1test2 <- t.test(growth2~Overall_Level, data=data_g)
g1test2##
## Welch Two Sample t-test
##
## data: growth2 by Overall_Level
## t = -9.5098, df = 23.007, p-value = 1.951e-09
## alternative hypothesis: true difference in means between group 100 and group 400 is not equal to 0
## 95 percent confidence interval:
## -6.649993 -4.273788
## sample estimates:
## mean in group 100 mean in group 400
## 0.1902834 5.6521739g1test3 <- t.test(growth3~Overall_Level, data=data_g)
g1test3##
## Welch Two Sample t-test
##
## data: growth3 by Overall_Level
## t = -10.419, df = 27.422, p-value = 4.896e-11
## alternative hypothesis: true difference in means between group 100 and group 400 is not equal to 0
## 95 percent confidence interval:
## -6.355134 -4.265179
## sample estimates:
## mean in group 100 mean in group 400
## 1.255061 6.565217g1test5 <- t.test(growth5~Overall_Level, data=data_g)
g1test5##
## Welch Two Sample t-test
##
## data: growth5 by Overall_Level
## t = -6.4813, df = 24.694, p-value = 9.211e-07
## alternative hypothesis: true difference in means between group 100 and group 400 is not equal to 0
## 95 percent confidence interval:
## -4.801386 -2.484655
## sample estimates:
## mean in group 100 mean in group 400
## 1.052632 4.695652g1test6 <- t.test(growth6~Overall_Level, data=data_g)
g1test6##
## Welch Two Sample t-test
##
## data: growth6 by Overall_Level
## t = -4.7467, df = 24.892, p-value = 7.257e-05
## alternative hypothesis: true difference in means between group 100 and group 400 is not equal to 0
## 95 percent confidence interval:
## -4.493795 -1.773764
## sample estimates:
## mean in group 100 mean in group 400
## 1.692308 4.826087g1test7 <- t.test(growth7~Overall_Level, data=data_g)
g1test7##
## Welch Two Sample t-test
##
## data: growth7 by Overall_Level
## t = -4.9343, df = 24.006, p-value = 4.907e-05
## alternative hypothesis: true difference in means between group 100 and group 400 is not equal to 0
## 95 percent confidence interval:
## -4.398110 -1.803967
## sample estimates:
## mean in group 100 mean in group 400
## 1.072874 4.173913g1test8 <- t.test(growth8~Overall_Level, data=data_g)
g1test8##
## Welch Two Sample t-test
##
## data: growth8 by Overall_Level
## t = -5.9404, df = 24.879, p-value = 3.419e-06
## alternative hypothesis: true difference in means between group 100 and group 400 is not equal to 0
## 95 percent confidence interval:
## -5.124949 -2.485682
## sample estimates:
## mean in group 100 mean in group 400
## 1.542510 5.347826g1test9 <- t.test(growth9~Overall_Level, data=data_g)
g1test9##
## Welch Two Sample t-test
##
## data: growth9 by Overall_Level
## t = -6.5097, df = 24.774, p-value = 8.459e-07
## alternative hypothesis: true difference in means between group 100 and group 400 is not equal to 0
## 95 percent confidence interval:
## -4.461732 -2.316300
## sample estimates:
## mean in group 100 mean in group 400
## 1.263158 4.652174g1test10 <- t.test(growth10~Overall_Level, data=data_g)
g1test10##
## Welch Two Sample t-test
##
## data: growth10 by Overall_Level
## t = -5.292, df = 24.361, p-value = 1.898e-05
## alternative hypothesis: true difference in means between group 100 and group 400 is not equal to 0
## 95 percent confidence interval:
## -4.778194 -2.098412
## sample estimates:
## mean in group 100 mean in group 400
## 1.518219 4.956522g1test11 <- t.test(growth11~Overall_Level, data=data_g)
g1test11##
## Welch Two Sample t-test
##
## data: growth11 by Overall_Level
## t = -6.0719, df = 24.292, p-value = 2.715e-06
## alternative hypothesis: true difference in means between group 100 and group 400 is not equal to 0
## 95 percent confidence interval:
## -4.475389 -2.205829
## sample estimates:
## mean in group 100 mean in group 400
## 1.137652 4.478261Running the independent t-test for \(G_1\):
g2test1 <- t.test(Present1~Overall_Level, data=data_g)
g2test1##
## Welch Two Sample t-test
##
## data: Present1 by Overall_Level
## t = -0.99031, df = 27.036, p-value = 0.3308
## alternative hypothesis: true difference in means between group 100 and group 400 is not equal to 0
## 95 percent confidence interval:
## -1.2387696 0.4322214
## sample estimates:
## mean in group 100 mean in group 400
## 7.161943 7.565217g2test2 <- t.test(Present2~Overall_Level, data=data_g)
g2test2##
## Welch Two Sample t-test
##
## data: Present2 by Overall_Level
## t = -9.085, df = 24.539, p-value = 2.535e-09
## alternative hypothesis: true difference in means between group 100 and group 400 is not equal to 0
## 95 percent confidence interval:
## -6.339517 -3.994579
## sample estimates:
## mean in group 100 mean in group 400
## 1.789474 6.956522g2test3 <- t.test(Present3~Overall_Level, data=data_g)
g2test3##
## Welch Two Sample t-test
##
## data: Present3 by Overall_Level
## t = -11.061, df = 36.389, p-value = 3.411e-13
## alternative hypothesis: true difference in means between group 100 and group 400 is not equal to 0
## 95 percent confidence interval:
## -5.444470 -3.757783
## sample estimates:
## mean in group 100 mean in group 400
## 3.485830 8.086957g2test5 <- t.test(Present5~Overall_Level, data=data_g)
g2test5##
## Welch Two Sample t-test
##
## data: Present5 by Overall_Level
## t = -4.1525, df = 30.962, p-value = 0.0002391
## alternative hypothesis: true difference in means between group 100 and group 400 is not equal to 0
## 95 percent confidence interval:
## -2.6823415 -0.9152646
## sample estimates:
## mean in group 100 mean in group 400
## 5.983806 7.782609g2test6 <- t.test(Present6~Overall_Level, data=data_g)
g2test6##
## Welch Two Sample t-test
##
## data: Present6 by Overall_Level
## t = -4.3624, df = 29.615, p-value = 0.0001431
## alternative hypothesis: true difference in means between group 100 and group 400 is not equal to 0
## 95 percent confidence interval:
## -3.156013 -1.142526
## sample estimates:
## mean in group 100 mean in group 400
## 5.372470 7.521739g2test7 <- t.test(Present7~Overall_Level, data=data_g)
g2test7##
## Welch Two Sample t-test
##
## data: Present7 by Overall_Level
## t = -5.4435, df = 30.92, p-value = 6.069e-06
## alternative hypothesis: true difference in means between group 100 and group 400 is not equal to 0
## 95 percent confidence interval:
## -2.898003 -1.318156
## sample estimates:
## mean in group 100 mean in group 400
## 6.109312 8.217391g2test8 <- t.test(Present8~Overall_Level, data=data_g)
g2test8##
## Welch Two Sample t-test
##
## data: Present8 by Overall_Level
## t = -5.9988, df = 34.235, p-value = 8.421e-07
## alternative hypothesis: true difference in means between group 100 and group 400 is not equal to 0
## 95 percent confidence interval:
## -3.141831 -1.552061
## sample estimates:
## mean in group 100 mean in group 400
## 5.870445 8.217391g2test9 <- t.test(Present9~Overall_Level, data=data_g)
g2test9##
## Welch Two Sample t-test
##
## data: Present9 by Overall_Level
## t = -3.9727, df = 31.451, p-value = 0.0003867
## alternative hypothesis: true difference in means between group 100 and group 400 is not equal to 0
## 95 percent confidence interval:
## -2.8024430 -0.9018344
## sample estimates:
## mean in group 100 mean in group 400
## 5.234818 7.086957g2test10 <- t.test(Present10~Overall_Level, data=data_g)
g2test10##
## Welch Two Sample t-test
##
## data: Present10 by Overall_Level
## t = -2.8678, df = 32.491, p-value = 0.007204
## alternative hypothesis: true difference in means between group 100 and group 400 is not equal to 0
## 95 percent confidence interval:
## -1.7113597 -0.2904006
## sample estimates:
## mean in group 100 mean in group 400
## 7.129555 8.130435g2test11 <- t.test(Present11~Overall_Level, data=data_g)
g2test11##
## Welch Two Sample t-test
##
## data: Present11 by Overall_Level
## t = -3.0217, df = 32.478, p-value = 0.004873
## alternative hypothesis: true difference in means between group 100 and group 400 is not equal to 0
## 95 percent confidence interval:
## -1.8098215 -0.3528259
## sample estimates:
## mean in group 100 mean in group 400
## 6.570850 7.652174Discussion
The results presented above provide an avenue to evaluate students’ self-reported competency in computational skills as they progress through the undergraduate program. The figure of the initial means (supported by the previously run ANOVA) shows that students re-evaluate their own initial competencies in skills as they gain knowledge through the program, supporting the assertion that only measuring students’ present self-efficacy loses valuable information. This is further shown by comparing the means of the growth methods, where a clear and consistent difference is shown between growth measures. The results of both independent t-tests show p-values of \(p<0.05\) for most of the survey items (with Excel being the notable exception), indicating significant difference between the variables in the definitions of \(G_1\), and \(G_2\). By incorporating the effect of this re-evaluation (\(I_{eff}\)), \(G_2\) is shown to have a consistent, and in many cases substantial, increase in students’ self-efficacy when compared to the more typically used method, \(G_1\).
It is interesting to note that there seem to be multiple trends observed when comparing the plots of initial and present means. For example, survey items Q5 through Q8 all show significant decreases in initial means while showing significant increases in present means as they go through the program. Whereas survey items Q10 and Q11 show less difference in present means while the initial means change more dramatically than the first group. This could be the result of students’ gaining more understanding of those topics as they progress through the program. For example, lower-level students are less likely to fully understand what it means to use computational methods to perform numerical integration/differentiation, so an interpretation of the prompt could be viewed simply as taking a derivative/integral, which would lead to less gain in present skill but a dramatic change in initial skill as they gain understanding of that survey item.
Conclusions
While there are plenty of avenues left unexplored, we can certainly draw some conclusions based on the results of this analysis. It has been shown that students re-evaluate their initial state with regards to competency in computational skills, which supports the use of an “initial” and “present” state when surveying student self-efficacy. This is further reinforced by the consistent, and in many cases substantial, difference in the growth of student self-efficacy when factoring in the effect of the initial state. The data presented also show clear growth in students’ self-efficacy with computational skills as they progress through the program.
We hope to continue gathering more data to investigate these trends further, with one future goal of investigating where students make the largest gains in self-efficacy. This could lead to answering questions such as what courses are responsible for the largest gains in self-efficacy. Understanding what courses lead to the most significant gains can allow the department to consider applying similar methods to other courses with the intention of further increasing gains in self-efficacy.