library(tidyverse)
## Warning: package 'tidyverse' was built under R version 4.3.2
## Warning: package 'ggplot2' was built under R version 4.3.2
## Warning: package 'tidyr' was built under R version 4.3.2
## Warning: package 'readr' was built under R version 4.3.2
## Warning: package 'purrr' was built under R version 4.3.2
## Warning: package 'dplyr' was built under R version 4.3.2
## Warning: package 'stringr' was built under R version 4.3.2
## Warning: package 'forcats' was built under R version 4.3.2
## Warning: package 'lubridate' was built under R version 4.3.2
## ── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ──
## ✔ dplyr     1.1.4     ✔ readr     2.1.5
## ✔ forcats   1.0.0     ✔ stringr   1.5.1
## ✔ ggplot2   3.4.4     ✔ tibble    3.2.1
## ✔ lubridate 1.9.3     ✔ tidyr     1.3.1
## ✔ purrr     1.0.2     
## ── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
## ✖ dplyr::filter() masks stats::filter()
## ✖ dplyr::lag()    masks stats::lag()
## ℹ Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errors
library(purrr)
library(plotrix)
## Warning: package 'plotrix' was built under R version 4.3.2
library(readxl)
library(ggthemes)
## Warning: package 'ggthemes' was built under R version 4.3.2
library(ggrepel)
## Warning: package 'ggrepel' was built under R version 4.3.2
library(effsize)
## Warning: package 'effsize' was built under R version 4.3.3
library(pwrss)
## Warning: package 'pwrss' was built under R version 4.3.3
## 
## Attaching package: 'pwrss'
## 
## The following object is masked from 'package:stats':
## 
##     power.t.test

Introduction

The purpose of this week’s data dive is to gain experience applying an ANalysis Of VAriance (ANOVA) to our data set. ANOVA tests are integral to a vast array of applied statistical models due to a combination of the conceptual simplicity and wide-ranging applicability across statistical fields.

Specifically, this data dive asks to run an ANOVA on what we deem as the most important response variable against an explanatory we expect to impact the response variable. For my own sanity, I will use dependent/independent variable for response/explanatory variable, respectively, for the duration of this datadive.

As a secondary objective, this data dive asks to run a linear regression for two continuous variables, one that is identified as the dependent variable chosen for the ANOVA test, and another that should have a linear relationship with the dependent variable.

To begin, just as with every other data dive, let’s define the dataset we will use for this analysis:

data0 <- read_excel("~/IUPUI/Research/PERC_2024/Datasets/F18-F23_Survey_Data2_Clean.xlsx")

view(data0)

While it is a relatively straightforward request to identify the “most important” column of the data set, this dataset contains responses to a survey that undergraduate physics majors are prompted to take at the end of every semester in order to gauge student attitudes regarding computational methods in physics and how they would rate their own abilities (on a scale of 1-10) in a variety of specific computational skills, both for when they started the program and where they are now.

All of the above is to say that it is impossible to pick a column that would be identified as the “most important” for this data. In order to compensate for this, I will simply pick a column that I think would be interesting to look at, and could tell us something important about the data. I am interested in how students re-evaluate their initial skill as they progress through the program, as this could provide some valuable insight into a potential Dunning-Krueger (DK) effect being observed; where subjects in the study showed that as they gain information about a topic, their own perceived knowledge about the subject grows sharply before plummeting.

The connection here is to determine whether students begin to re-evaluate their own initial skill as they progress through the program and “gain more knowledge.” So, I will choose the “initial10” column as the dependent variable for the ANOVA test as the prompt asks students to rate their skill for “Numerical integration and differentiation” at the point when they started the program. This choice is due to the possible misunderstanding of how “numerical integration and differentiation” in computational physics is different from simply taking derivatives and integrals of functions. I would expect that 100/200 level students would interpret the question as the latter definition since they essentially “don’t know what they don’t know” and could be more likely to overestimate their own ability in this area. This also leads clearly to the choice of an independent variable, as the most important information would be the student’s overall level, which is defined as the highest course level that the student has taken (i.e., 100, 200, 300, and 400).

Null Hypothesis

The next step is to devise a null hypothesis that can be tested using the dependent variable (“initial10”), and the independent variable (“Overall_Level”). This is somewhat trivial, but it is good practice, so the null hypothesis I would propose would be:

Students do not reevaluate their initial state for performing numerical integration/differentiation in computational physics as they progress through the undergraduate physics program.

ANOVA Test

I10 <- aov(Initial10 ~ Overall_Level, data = data0)

summary(I10)
##                Df Sum Sq Mean Sq F value Pr(>F)    
## Overall_Level   1    184  184.40   23.11  2e-06 ***
## Residuals     528   4214    7.98                   
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

The low p-value is already bad news bears for the null hypothesis, as it indicates that it is exceedingly unlikely that the means are equal. This is, however, good news for looking into any potential DK effects, as it indicates that students seem to re-evaluate their initial skill as they gain more knowledge progressing through the undergraduate program. While this quick test does not show comparisons between levels, it does at least indicate the potential need to do a more detailed analysis to see how students might re-evaluate their own skills.

While not the most statistically-sound method (as a Tukey HSD would provide more detailed information) of looking into the relationship between levels, generating a simple plot of the mean response to “initial10” per level would at least give some clues as to if a DK effect would be worth looking into.

plot_data <-
  data0 |>
  group_by(Overall_Level) |>
  summarise(avg_i10 = mean(Initial10, na.rm = TRUE),
            se_i10 = std.error(Initial10, na.rm = TRUE))
view(plot_data)
plot1 <-
  ggplot(plot_data)+
  geom_col(mapping = aes(x=as.character(Overall_Level), y=avg_i10))+
  geom_errorbar(mapping = aes(x = as.character(Overall_Level), ymin = avg_i10 - se_i10, ymax = avg_i10 + se_i10))

plot1

Again, while not the most statistically-sound method, this plot at does at least indicate that students seem to re-evaluate their initial skill with numerical integration and differentiation as they progress through the program in a direction consistent with what a DK effect would suggest; further reinforcing a need to evaluate this effect using a post-hoc test like a Tukey HSD (though for time, this will be a future plan).

Linear Regression

Maintaining the choice of “Initial10” as the dependent variable, a second continuous column that I would expect to affect “Initial10” would be “Present10”, where students are asked to evaluate their current skill level using numerical integration and differentiation. Conceptually, it is expected that as a student gains more knowledge regarding numerical integration/differentiation, their current skill level will increase while the previous ANOVA indicates that student re-evaluate their initial skill level that lowers as they progress through the program. This would indicate a linear relationship between initial/present skill level that is likely to be negative. Put simply: It would be expected that as a student gains present skill level, they re-evaluate their initial skill level to be lower than originally thought.

While we could go straight into a linear model, I am somewhat curious to just get a quick snapshot to see if this relationship does, in fact, look linear.

plot2_data<-
    data0 |>
  group_by(Overall_Level) |>
  summarise(avg_i10 = mean(Initial10, na.rm = TRUE),
            se_i10 = std.error(Initial10, na.rm = TRUE),
            avg_p10 = mean(Present10, na.rm=TRUE),
            se_p10= std.error(Present10, na.rm=TRUE))
view(plot2_data)
plot2 <-
  ggplot(data = plot2_data)+
  geom_point(mapping= aes(x = avg_p10, y=avg_i10))

plot2

While very crude, especially since I chose to only use the means instead of the complete dataset, this method does indicate a relationship somewhat similar to what I would have expected, which at least provides a little support to running a linear model.

Running the linear model:

m10 <-
  lm(avg_i10 ~ avg_p10, plot2_data)

m10
## 
## Call:
## lm(formula = avg_i10 ~ avg_p10, data = plot2_data)
## 
## Coefficients:
## (Intercept)      avg_p10  
##      20.815       -2.155

The coefficients from this linear model give the y-intercept (20.815), and the slope value (-2.155). Thankfully, this is what we expected and lends more support to a potential DK effect occurring as students gain more knowledge about a topic while progressing through the undergraduate program.