library(tidyverse)
## Warning: package 'tidyverse' was built under R version 4.3.2
## Warning: package 'ggplot2' was built under R version 4.3.2
## Warning: package 'tidyr' was built under R version 4.3.2
## Warning: package 'readr' was built under R version 4.3.2
## Warning: package 'purrr' was built under R version 4.3.2
## Warning: package 'dplyr' was built under R version 4.3.2
## Warning: package 'stringr' was built under R version 4.3.2
## Warning: package 'forcats' was built under R version 4.3.2
## Warning: package 'lubridate' was built under R version 4.3.2
## ── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ──
## ✔ dplyr     1.1.4     ✔ readr     2.1.5
## ✔ forcats   1.0.0     ✔ stringr   1.5.1
## ✔ ggplot2   3.4.4     ✔ tibble    3.2.1
## ✔ lubridate 1.9.3     ✔ tidyr     1.3.1
## ✔ purrr     1.0.2     
## ── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
## ✖ dplyr::filter() masks stats::filter()
## ✖ dplyr::lag()    masks stats::lag()
## ℹ Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errors
library(purrr)
library(plotrix)
## Warning: package 'plotrix' was built under R version 4.3.2
library(readxl)
library(ggthemes)
## Warning: package 'ggthemes' was built under R version 4.3.2
library(ggrepel)
## Warning: package 'ggrepel' was built under R version 4.3.2
library(effsize)
## Warning: package 'effsize' was built under R version 4.3.3
library(pwrss)
## Warning: package 'pwrss' was built under R version 4.3.3
## 
## Attaching package: 'pwrss'
## 
## The following object is masked from 'package:stats':
## 
##     power.t.test

Introduction

This week’s data dive involves testing some hypotheses based on the exploration of the dataset. From the data, we are asked to generate two different null hypotheses and test them using two different methods (Neyman-Pearson and Fischer’s Significance Test, respectively).

First, let’s define clear null hypotheses to be tested, ensuring that they can be disproven.

Null Hypothesis 1

The growth (defined as present level - initial level) in skill 5 does not change as the level of the student (i.e., 100, 200, 300, 400) increases.

Null Hypothesis 2

The growth (as defined above) in skill 6 does not change as the level of student increases.

Testing Hypothesis 1

Let’s begin by defining the dataset to be used:

data0 <- read_excel("~/IUPUI/Research/PERC_2024/Datasets/F18-F23_Survey_Data2_Clean.xlsx")

view(data0)

Once the dataset has been imported, we need to define columns that will allow us to test our hypothesis, so we need to define two new columns that will indicate growth as it has been defined above.

  growth_data <-
    data0 |>
    mutate(data0, growth5 = Present5 - Initial5) |>
    mutate(data0, growth6 = Present6 - Initial6)

view(growth_data)

In order to build a relationship between variables, however, we cannot simply use the growth_data set because there are multiple surveys taken in the same overall_level, semester, and course. So instead, it would be better to simply average the growth data so that for each level and each semester, there is only one entry which allows us to actually build a independent/dependent variable pair.

So, let’s take our original data set and build a new one that will contain only the information we need (i.e., the average growth per level and the standard error in that growth):

avg_growth <-
  growth_data |>
  group_by(Overall_Level) |>
  summarise(avg_growth5 = mean(growth5, na.rm=TRUE),
            se_growth5 = std.error(growth5, na.rm=TRUE),
            avg_growth6 = mean(growth6, na.rm=TRUE),
            se_growth6 = std.error(growth6, na.rm=TRUE))
view(avg_growth)

In order to determine if there is enough data to perform the test, we have to define the different parameters.

A significant difference is assumed to be 0.5 which I’m choosing to be the difference required between levels to be considered significant.Alpha will be chosen to be 0.05 for a variety of reasons, but most notably for consistency. Beta will then be defined as 0.1, based on the idea that I’m willing to risk “missing out” 10% of the time, so that the power (1-beta) is then given as 0.9.

To test for the amount of data required, we can run the following (based on the week 7 notes):

data_test <- pwrss.t.2means(mu1 = 0.5,
                            sd1 = sd(pluck(avg_growth, "avg_growth5")),
                            kappa = 1,
                            power = 0.9,
                            alpha = .05,
                            alternative = "not equal")
##  Difference between Two means 
##  (Independent Samples t Test) 
##  H0: mu1 = mu2 
##  HA: mu1 != mu2 
##  ------------------------------ 
##   Statistical power = 0.9 
##   n1 = 251 
##   n2 = 251 
##  ------------------------------ 
##  Alternative = "not equal" 
##  Degrees of freedom = 500 
##  Non-centrality parameter = 3.254 
##  Type I error rate = 0.05 
##  Type II error rate = 0.1
plot(data_test)
## Warning in qt(1 - prob.extreme, df = df, ncp = ncp, lower.tail = TRUE): full
## precision may not have been achieved in 'pnt{final}'

Based on the test above, I need 251 responses at minimum for the above test. Thankfully, there are well over 250 survey responses so I can conclude from the plot that the null hypothesis is most likely false, and while that is expected, it remains a worthwhile endeavor to make sure.

Testing Hypothesis 2

F_test <- fisher.test(avg_growth$Overall_Level, y=avg_growth$avg_growth6)

F_test
## 
##  Fisher's Exact Test for Count Data
## 
## data:  avg_growth$Overall_Level and avg_growth$avg_growth6
## p-value = 1
## alternative hypothesis: two.sided

Obviously the p-value result of 1 is likely not correct, because a p-value of 1 would indicate that the overall level has no impact on the growth and this is just clearly not what is observable. This is certainly something to consider working through in greater detail moving forward.