Week_3_Data_Dive
Greg Gallagher
2024-02-06
library(purrr)## Warning: package 'purrr' was built under R version 4.3.2library(tidyverse)## Warning: package 'tidyverse' was built under R version 4.3.2## Warning: package 'ggplot2' was built under R version 4.3.2## Warning: package 'tidyr' was built under R version 4.3.2## Warning: package 'readr' was built under R version 4.3.2## Warning: package 'dplyr' was built under R version 4.3.2## Warning: package 'forcats' was built under R version 4.3.2## Warning: package 'lubridate' was built under R version 4.3.2## ── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ──
## ✔ dplyr 1.1.4 ✔ readr 2.1.5
## ✔ forcats 1.0.0 ✔ stringr 1.5.0
## ✔ ggplot2 3.4.4 ✔ tibble 3.2.1
## ✔ lubridate 1.9.3 ✔ tidyr 1.3.0
## ── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
## ✖ dplyr::filter() masks stats::filter()
## ✖ dplyr::lag() masks stats::lag()
## ℹ Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errorslibrary(dplyr)Introduction
Using the same data set as for the first data dive, this file will explore grouping and visualizing some of the data.
First, I have to define the dataframe and ignore any NAs, as an NA indicates that a student did not answer a survey item, not that the value of the response is 0.
First, setting a variable, “data0” in this case, as the Excel spreadsheet (this could also be done if the data was a CSV file).I want to exclude any NA values as those are simply cases where a student did not answer a question. The ‘view’ command is just there for me to make sure that the data has imported correctly.
library(readxl)
data0 <- read_excel("~/IUPUI/Research/Computational_Data/Comp_Data/F18-F23_Survey_Data.xlsx")
na.pass(data0)## # A tibble: 617 × 39
## Semester RID Overall_Level Course Comp1 Comp2 Comp3 Comp4 Comp5 CompD1
## <chr> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
## 1 F18 488171 300 310 5 4 3 3 4 4
## 2 F18 299278 100 152 3 4 4 3 3 3
## 3 F18 238232 100 152 2 2 1 2 4 4
## 4 F18 649079 100 152 4 4 3 2 3 3
## 5 F18 133382 100 152 3 4 3 4 5 4
## 6 F18 620319 100 152 2 1 1 1 4 4
## 7 F18 190508 100 152 4 4 4 4 4 4
## 8 F18 729552 100 152 3 2 2 2 4 4
## 9 F18 935469 200 251 4 4 4 4 4 4
## 10 F18 759379 100 152 4 4 4 4 4 4
## # ℹ 607 more rows
## # ℹ 29 more variables: CompD2 <dbl>, Comp_New1 <dbl>, Comp_New2 <dbl>,
## # Comp_New3 <dbl>, Comp_New4 <dbl>, Initial1 <dbl>, Initial2 <dbl>,
## # Initial3 <dbl>, Initial4 <dbl>, Initial4_TXT <chr>, Initial5 <dbl>,
## # Initial6 <dbl>, Initial7 <dbl>, Initial8 <dbl>, Initial9 <dbl>,
## # Initial10 <dbl>, Initial11 <dbl>, Present1 <dbl>, Present2 <dbl>,
## # Present3 <dbl>, Present4 <dbl>, Present4_TXT <chr>, Present5 <dbl>, …View(data0)For this data dive, we are instructed to investigate 3 columns of grouped data. To make this simple, let’s just find the average response on survey item 1 (i.e., “Comp1”) based on semester, highest course level, and course number. There will be some confusion in course number, as the students are asked to put all courses they are taking from the physics department, so some will have multiple numbers associated with them. To address this issue, I went ahead and chose specific courses to use (which will introduce a fair degree of bias), so in the future it might be helpful to consider other ways to handle this specific issue (especially since it could be interesting to look at the student responses for specific courses).
Data Exploration
Data grouped by Semester
With the data set defined as “data0”, let’s start by finding the mean response to the first survey item, “Comp1” for each semester. To do that, I will group the data by semester and then find the mean and standard deviation (while ignoring missing values). The next objective is to make a bar plot of the Comp1 data grouped by semester.
data0 |>
group_by(Semester) |>
summarise(avg_Comp1_Sem = mean(Comp1, na.rm = TRUE),
sd_Comp1_Sem = sd(Comp1, na.rm = TRUE))## # A tibble: 11 × 3
## Semester avg_Comp1_Sem sd_Comp1_Sem
## <chr> <dbl> <dbl>
## 1 F18 3.88 0.833
## 2 F19 4.20 0.675
## 3 F20 4.37 0.714
## 4 F21 4.45 0.587
## 5 F22 4.67 0.546
## 6 F23 4.67 0.586
## 7 S19 4.22 0.898
## 8 S20 4.02 0.961
## 9 S21 4.27 0.714
## 10 S22 4.47 0.775
## 11 S23 4.31 0.859data0 |>
group_by(Semester) |>
count(Semester)## # A tibble: 11 × 2
## # Groups: Semester [11]
## Semester n
## <chr> <int>
## 1 F18 25
## 2 F19 103
## 3 F20 72
## 4 F21 72
## 5 F22 55
## 6 F23 36
## 7 S19 36
## 8 S20 60
## 9 S21 67
## 10 S22 59
## 11 S23 32plot1 <-
ggplot(data = data0, mapping = aes(x = Semester, y = Comp1, na.rm = TRUE))+
geom_boxplot()
plot1## Warning: Removed 25 rows containing non-finite values (`stat_boxplot()`).
It is fairly obvious from the number of responses, that the Fall of 2018 had the lowest number of total responses, meaning that out of the data set, it is less likely that a student from Fall of 2018 would be randomly selected from this list. To determine that probability, we can simply do the following:
P_F18 <- 25/617
P_F18## [1] 0.04051864It is worth noting that this means there is only ~4% chance that a random selection would be a student from the Fall of 2018 in a random selection. There are few testable hypotheses as to why this is the case, but it is likely the case because this was the first semester the survey was given, so there were likely fewer reminders given to students to take the survey.
From this data, it is clear that there is no distinguishable trend in student attitudes for the first survey item across the duration of the survey.
Data grouped by overall course level
Next, I will group the data by level and then find the mean and standard deviation the same way as before.
data0 |>
group_by(Overall_Level) |>
summarise(avg_Comp1_lvl = mean(Comp1, na.rm = TRUE),
sd_Comp1_lvl = sd(Comp1, na.rm = TRUE))## # A tibble: 4 × 3
## Overall_Level avg_Comp1_lvl sd_Comp1_lvl
## <dbl> <dbl> <dbl>
## 1 100 4.26 0.838
## 2 200 4.32 0.676
## 3 300 4.62 0.540
## 4 400 4.74 0.449data0 |>
group_by(Overall_Level) |>
count(Overall_Level)## # A tibble: 4 × 2
## # Groups: Overall_Level [4]
## Overall_Level n
## <dbl> <int>
## 1 100 307
## 2 200 247
## 3 300 40
## 4 400 23 plot2 <-
ggplot(data = data0, mapping = aes(x = as.character(Overall_Level), y = Comp1, na.rm = TRUE))+
geom_boxplot()
plot2## Warning: Removed 25 rows containing non-finite values (`stat_boxplot()`).
Clearly, as the level of the student increases, the probability that a random survey would be taken by a student at that level also decreases, with an incredibly sharp decrease after the 200 level. My hypothesis as to why this is true would be simply that fewer students continue on in the physics program at IUPUI as some 100 and 200 level classes are taken by additional majors other than just physics students. The probability that a random sample be a 400-level student would be found simply as:
P_400 <- 23/617
P_400## [1] 0.03727715, indicating that the chance a survey was taken by a 400-level student to be ~3.73%. It would be interesting to dive into the registration numbers to see if the response rate is consistent, even with decreased population numbers, but we currently do not have registration numbers unfortunately.
It is interesting to note that there does appear to be an overall positive trend in students gaining an expert-like attitude with regards to the first survey item. This is helpful for determining how successful IUPUI has been in developing students to have more expert-like attitudes as they progress through the physics program at IUPUI.
Data grouped by specific course
Finally, I would like to do the same exploration, but instead group by the specific course.
data0 |>
group_by(Course) |>
summarise(avg_Comp1_crs = mean(Comp1, na.rm = TRUE),
sd_Comp1_crs = sd(Comp1, na.rm = TRUE))## # A tibble: 12 × 3
## Course avg_Comp1_crs sd_Comp1_crs
## <dbl> <dbl> <dbl>
## 1 152 4.26 0.838
## 2 251 4.31 0.679
## 3 299 4.71 0.488
## 4 300 4.83 0.408
## 5 310 4.47 0.624
## 6 330 4.4 0.548
## 7 342 4.82 0.405
## 8 353 5 NA
## 9 400 5 NA
## 10 401 5 NA
## 11 418 4.75 0.5
## 12 442 4.71 0.470data0 |>
group_by(Course) |>
count(Course)## # A tibble: 12 × 2
## # Groups: Course [12]
## Course n
## <dbl> <int>
## 1 152 307
## 2 251 240
## 3 299 7
## 4 300 6
## 5 310 17
## 6 330 5
## 7 342 11
## 8 353 1
## 9 400 1
## 10 401 1
## 11 418 4
## 12 442 17plot3 <-
ggplot(data = data0, mapping = aes(x = as.character(Course), y = Comp1, na.rm = TRUE))+
geom_boxplot()
plot3## Warning: Removed 25 rows containing non-finite values (`stat_boxplot()`).
Again, we can find the probability of the lowest response rate, but in this case, three separate categories all have the same probability, which is defined by
P_353 <- 1/617
P_353## [1] 0.001620746, and indicates that there is just a ~0.2% chance that a student enrolled in PHYS 353, 400, or 401 would be randomly selected. This is interesting, though it is important to note that many students are enrolled in multiple physics classes each semester, so it’s likely that a student in 400 would also be taking another class. It would be beneficial in the future to find a method of accounting for this.
It is unsurprising that these results seem to reinforce what was found when exploring the data by overall level. It is important to note that many of these courses have <10 responses per course, so the results are not nearly as statistically significant as we see for the first two groups.
Data Combination
When looking at the data, it is interesting to see that for the first survey item, Comp1, there are no upper-level (defined by course level = 200, 300, or 400) students that answer below a 2. This seems to suggest that students who are at least two semesters into the program, have some baseline expert-like attitude of a 2/5.
To visualize this data, it would be helpful to make a bar graph of the number of students with each response (1-5), and to be color-coded based on the course level (100, 200, 300, 400). Admittedly, being a little inexperienced with R, it is not immediately obvious how I would make this plot. I have tried a number of different methods, but cannot seem to get it figured out unfortunately. I could not get the code to run, but I have provided the code I tried (not in a Chunk as the markdown file would then not knit properly) below:
p <- data0 |>
ggplot() +
geom_bar(mapping = aes(x = as.character(Comp1), fill = Overall_Level), position="stack") +
theme_minimal() +
scale_fill_brewer(palette = 'Dark2')
p## Warning: The following aesthetics were dropped during statistical transformation: fill
## ℹ This can happen when ggplot fails to infer the correct grouping structure in
## the data.
## ℹ Did you forget to specify a `group` aesthetic or to convert a numerical
## variable into a factor?