1 Objective :

  • Data cleaning, documentation, and reporting of missingness and data loss, producing reproducible summary reports and well-documented, analysis-ready datasets for statistical modeling.

To accomplish this task, we will combine all data into one large data set then investigate, report and reduce the data set.

2 What is my data?

  • This data were from the same professor teaching the course across different years. This data is were not randomly generated, at most representative therefore of the professors student trends.

  • Convenience Sample

The study population consisted of students enrolled in selected offerings of the course taught by the same instructor across three academic quarters. These quarters were selected based on data availability rather than random sampling; consequently, the analytic sample should be interpreted as a convenience sample of these course offerings.

setwd("/Users/isaiahmireles/Desktop/Misconceptions")

# Midterm : 
# F23 
F23_M_VA <- 
  read.csv("F23_Midterm_Version_Set_Scores/Midterm_Version_A_scores.csv")
F23_M_VB <- 
  read.csv("F23_Midterm_Version_Set_Scores/Midterm_Version_B_scores.csv")

# W24 
W24_M_VA <- 
  read.csv("W24_Midterm_Version_Set_Scores/Midterm_Version_A_scores.csv")
W24_M_VB <- 
  read.csv("W24_Midterm_Version_Set_Scores/Midterm_Version_B_scores.csv")

# S24 
S24_M <- 
  read.csv("24S-STATS-10-LEC-4_Midterm/S24_Midterm_student_responses copy.csv")

# ------------------------------------------------------------------------------- #

# Final :
# F23
F23_F_VA <- 
  read.csv("F23_Final_Exam_Version_Set_Scores/Final_Exam_Version_A_scores.csv")
F23_F_VB <- 
  read.csv("F23_Final_Exam_Version_Set_Scores/Final_Exam_Version_B_scores.csv")

# W24 
W24_F_VA <- 
  read.csv("W24_Final_Exam_Version_Set_Scores/Final_Exam_Version_A_scores.csv")
W24_F_VB <- 
  read.csv("W24_Final_Exam_Version_Set_Scores/Final_Exam_Version_B_scores.csv")

# S24 
S24_F_VA <- 
  read.csv("S24_Final_Exam_Version_Set_Scores/Final_Exam_Version_A_scores.csv")
S24_F_VB <- 
  read.csv("S24_Final_Exam_Version_Set_Scores/Final_Exam_Version_B_scores.csv")

Side Notes :

  • Rather than considering only the loss of rows (student observations), we are particularly interested in the loss of individual students, as students may have repeated observations. Therefore, counting distinct students using identifiable features provides a more meaningful measure of data loss and helps characterize which unique students are retained or excluded during the cleaning process.
library(tidyverse)
# Generate large lst
exam_data <- list(
  F23_M_VA = F23_M_VA,
  F23_M_VB = F23_M_VB,
  F23_F_VA = F23_F_VA,
  F23_F_VB = F23_F_VB,
  W24_M_VA = W24_M_VA,
  W24_M_VB = W24_M_VB,
  W24_F_VA = W24_F_VA,
  W24_F_VB = W24_F_VB,
  S24_M = S24_M,
  S24_F_VA = S24_F_VA,
  S24_F_VB = S24_F_VB
)

3 Add Useful Feature-Labels

# Add useful features for each dataset
exam_data <- exam_data |>
  imap(\(data, name) {
    data |>
      mutate(
        term = str_extract(name, "F23|W24|S24"),
        exam = if_else(
          str_detect(name, "_M"),
          "Midterm",
          "Final"
        ),
        version = case_when(
          str_detect(name, "_VA$") ~ "A",
          str_detect(name, "_VB$") ~ "B",
          TRUE ~ "Both"
        )
      )
  })

4 Ct Status (Graded, Missing)

status_summary <- exam_data |>
  map(\(data) {
    data |>
      count(Status) |>
      mutate(prop = n / sum(n))
  }) |>
  list_rbind(names_to = "dataset")

status_summary
status_summary |>
  ggplot(aes(
    x = Status,
    y = prop,
    fill = Status
  )) +
  geom_col() +
  geom_text(
    aes(label = scales::percent(prop, accuracy = 0.1)),
    vjust = -0.3,
    size = 3
  ) +
  facet_wrap(~ dataset) +
  scale_fill_manual(
    values = c(
      "Graded" = "green",
      "Missing" = "red"
    )
  ) +
  scale_y_continuous(
    labels = scales::percent,
    limits = c(0, 1)
  ) +
  labs(
    title = "Grading Status by Dataset",
    x = "Status",
    y = "Proportion",
    fill = "Status"
  )

  • notice that in general, about half the data are Graded except for S24_M which contains version=="Both".

4.1 Filter Status == "Graded"

graded_exam_data <- exam_data |>
  map(\(data) {
    data |>
      filter(Status == "Graded")
  })

4.2 Colnames of each data set :

col_names_wide <- exam_data |>
  map(\(data) {
    tibble(column = names(data))
  }) |>
  list_rbind(names_to = "dataset") |>
  mutate(position = row_number(), .by = dataset) |>
  pivot_wider(
    names_from = dataset,
    values_from = column
  )

col_names_wide

  • there are 2 key issues :

    • Identifiable std. features (Anonymity issues)

    • different colnames per dat

5 How many unique students are within each data?