The purpose of this week’s data dive is to get experience with sampling subsets of data in order to explore how sampling a population is used when representing the population as a whole. First, beginning with a preamble that will load relevant libraries and define the first data set:
library(tidyverse)## Warning: package 'tidyverse' was built under R version 4.3.2## Warning: package 'ggplot2' was built under R version 4.3.2## Warning: package 'tidyr' was built under R version 4.3.2## Warning: package 'readr' was built under R version 4.3.2## Warning: package 'purrr' was built under R version 4.3.2## Warning: package 'dplyr' was built under R version 4.3.2## Warning: package 'stringr' was built under R version 4.3.2## Warning: package 'forcats' was built under R version 4.3.2## Warning: package 'lubridate' was built under R version 4.3.2## ── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ──
## ✔ dplyr 1.1.4 ✔ readr 2.1.5
## ✔ forcats 1.0.0 ✔ stringr 1.5.1
## ✔ ggplot2 3.4.4 ✔ tibble 3.2.1
## ✔ lubridate 1.9.3 ✔ tidyr 1.3.1
## ✔ purrr 1.0.2
## ── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
## ✖ dplyr::filter() masks stats::filter()
## ✖ dplyr::lag() masks stats::lag()
## ℹ Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errorslibrary(purrr)
library(plotrix)## Warning: package 'plotrix' was built under R version 4.3.2library(readxl)
data0 <- read_excel("~/IUPUI/Research/Computational_Data/Comp_Data/F18-F23_Survey_Data_Clean.xlsx")
view(data0)Once the data frame is identified (in this case as “data0”), it must be randomly sampled (with replacement), and each sample must contain ~50% of the data. The samples will then be stored as individual data frames such that the first sample will be defined as “data1”, the second as “data2”, and so on.
Let’s take the first sample, and noting that the full dataset includes 530 rows, this means that the sample sizes in each case will be 265 rows. In the prompt, we are asked to take a sample with replacement, so for the first sample, we can do the following:
data1 <-
data0[sample(nrow(data0), size = 265, replace = TRUE),]
view(data1)From here, we can compare what the sample gives with what the full data set gives. There are two investigations I’d like to do: first we can compare the probability results from data dive 3 (where I looked at the probability of a random row being attributed to a student’s level), and then second we can compare the average response per item. We could look at the average for each category , but I think it would be easier to simply choose one of the columns and compare the results that way. While not as robust as comparing each column, it gets the job done and saves a little time in the process.
First, from data dive 3, we found the probability of a random sample to be that of a 400-level student and found the probability to be ~0.0405. To find the probability in this data set, we can simply do:
data1 |>
count(Overall_Level)## # A tibble: 4 × 2
## Overall_Level n
## <dbl> <int>
## 1 100 138
## 2 200 99
## 3 300 17
## 4 400 11This shows the number of 400 level students to be 8 out of the 265 students. To find the probability, we can simply do:
p_400_1 <- 8/265
p_400_1## [1] 0.03018868This result shows a slightly lower probability for this sample that a student chosen at random would be at the 400-level. While this result is lower, it is well within the realm of reasonable agreement, supporting the idea that this random sample would be indicative of the entire data set.
Next, we can compare the average response for survey item 1 (column Comp1 in the dataset). To do this, let’s find the average per overall level for responses to comp1.
data1 |>
group_by(Overall_Level) |>
summarise(avg_Comp1_lvl = mean(Comp1, na.rm = TRUE))## # A tibble: 4 × 2
## Overall_Level avg_Comp1_lvl
## <dbl> <dbl>
## 1 100 4.22
## 2 200 4.22
## 3 300 4.71
## 4 400 4.82To compare, lets find the average for the entire data set per overall level.
data0 |>
group_by(Overall_Level) |>
summarise(avg_Comp1_lvl = mean(Comp1, na.rm = TRUE))## # A tibble: 4 × 2
## Overall_Level avg_Comp1_lvl
## <dbl> <dbl>
## 1 100 4.25
## 2 200 4.32
## 3 300 4.65
## 4 400 4.74It is worth noting that each of these results is incredibly close to one another, indicating that the random sample could be a good subset of the total population.
My plan is to simply repeat the above process for the next 4 samples (for a total of 5 samples). Since I have explained the process above, I will simply provide the code and then walk through a comparison at the end of each sample subsection.
data2 <-
data0[sample(nrow(data0), size = 265, replace = TRUE),]
view(data2)data2 |>
count(Overall_Level)## # A tibble: 4 × 2
## Overall_Level n
## <dbl> <int>
## 1 100 130
## 2 200 108
## 3 300 15
## 4 400 12p_400_2 <- 14/265
p_400_2## [1] 0.05283019data2 |>
group_by(Overall_Level) |>
summarise(avg_Comp1_lvl = mean(Comp1, na.rm = TRUE))## # A tibble: 4 × 2
## Overall_Level avg_Comp1_lvl
## <dbl> <dbl>
## 1 100 4.32
## 2 200 4.23
## 3 300 4.4
## 4 400 4.75It is interesting that the probability of a 400 level student being a random selection is slightly higher than the value for the whole population, but again it is well within a reasonable amount. And if you average the two sample probabilities, you get ~0.04, which is the 400-level probability for the entire data set.
The averages remain fairly close together as well. The 200-level difference is slightly larger than the other levels, but it is still well within a reasonable range (~0.08) to be representative of the larger population.
data3 <-
data0[sample(nrow(data0), size = 265, replace = TRUE),]
view(data3)data3 |>
count(Overall_Level)## # A tibble: 4 × 2
## Overall_Level n
## <dbl> <int>
## 1 100 123
## 2 200 109
## 3 300 21
## 4 400 12p_400_3 <- 12/265
p_400_3## [1] 0.04528302data3 |>
group_by(Overall_Level) |>
summarise(avg_Comp1_lvl = mean(Comp1, na.rm = TRUE))## # A tibble: 4 × 2
## Overall_Level avg_Comp1_lvl
## <dbl> <dbl>
## 1 100 4.21
## 2 200 4.35
## 3 300 4.81
## 4 400 4.75The probability is remarkably close to the full population value and the averages remain very close.
data4 <-
data0[sample(nrow(data0), size = 265, replace = TRUE),]
view(data4)data4 |>
count(Overall_Level)## # A tibble: 4 × 2
## Overall_Level n
## <dbl> <int>
## 1 100 121
## 2 200 118
## 3 300 20
## 4 400 6p_400_4 <- 12/265
p_400_4## [1] 0.04528302data4 |>
group_by(Overall_Level) |>
summarise(avg_Comp1_lvl = mean(Comp1, na.rm = TRUE))## # A tibble: 4 × 2
## Overall_Level avg_Comp1_lvl
## <dbl> <dbl>
## 1 100 4.21
## 2 200 4.35
## 3 300 4.65
## 4 400 4.67Again, the results show fair agreement with the full population.
data5 <-
data0[sample(nrow(data0), size = 265, replace = TRUE),]
view(data5)data5 |>
count(Overall_Level)## # A tibble: 4 × 2
## Overall_Level n
## <dbl> <int>
## 1 100 126
## 2 200 100
## 3 300 23
## 4 400 16p_400_5 <- 13/265
p_400_5## [1] 0.0490566data5 |>
group_by(Overall_Level) |>
summarise(avg_Comp1_lvl = mean(Comp1, na.rm = TRUE))## # A tibble: 4 × 2
## Overall_Level avg_Comp1_lvl
## <dbl> <dbl>
## 1 100 4.16
## 2 200 4.27
## 3 300 4.57
## 4 400 4.75It is somewhat interesting in this data set that the 400-level average is ~0.2 less than the average for the full population. Given how close the others have been, it is interesting to note that this sample might be slightly less reliable, but if we were to average all of the 400-level samples(rounded to 3 decimal places):
Sample_Avg <- (4.538+4.750+5.000+4.714+4.750)/5
Sample_Avg## [1] 4.7504The above gives a result of ~4.750, while the total population gives an average of ~4.739. The difference between the two averages is ~0.232%, which indicates the general agreement across the samples with the full set. That said, there are still some discrepancies, as indicated in the discussions.
Sampling data can give us a good sense of the total population with a reasonable agreement as demonstrated above. Considering that not every student will take the survey, the full data set is essentially a sample in and of itself, which is true of survey data in general. Going through this activity reinforces the idea that surveys can be a good representation of the total population, further reinforcing the validity of survey data in general.
It would be interesting to look at this data through the lens of the different response items. I only chose to look at one single column, but it could be interesting to look at the results for other columns as well and see if the same patterns arise, but for the sake of time management, that is a consideration for the future.