1 Preface

Purpose :

Clean, Join & Report Missingness
Define : What is my Data
Demonstrate the ability to interpret Data in Real world Context
Demonstrate the ability to infer the scope of generality of data

Deliverable(s) :

Create Docu. for data (data_dictionary.csv,README.md)
Create Clean Dat. (S10_students.csv)
Create Report (S10_Data_Cleaning.Rmd)
Create Codebook (CodeBk.pdf)
Create Loss Diagram (Std_Loss.png)
Explicitly define Sampling Terms :
- Element
- Population
- Sampling Unit
- Frame
- Sample
- Probability Sample
- Sampling Design

Meaning :

data_dictionary.csv : Defines each variable in the final dataset, including its meaning, type, coding, source, and missingness.
README.md : Summarizes the dataset, its provenance, structure, limitations, access restrictions, and how to use the project files.
cohorts.csv : Final cleaned and de-identified analytic dataset used for analysis.
S10_Data_Cleaning.Rmd : Reproducible record of how the raw data were cleaned, linked, anonymized, validated, and converted into cohorts.csv.

Source(s) of Guidance :

Data Privacy Statement :

here we require that the data be anonymized – ie. shared data may not be linked to particular student. Direct & Indirect identifiers must be removed.

1.1 Data Sets

ls() |> as.data.frame()

Quarter : Fall (F), Winter (W), Spring (S)
Exam : Midterm (M), Final (F)
Version (VA, VB)

For example :

“F_F_VA” means Version A (VA) of the Fall (F) Final (F).
“F_M_VA” means Version A (VA) of the Fall (F) Midterm (M).

1.2 Anonymization

F_M_VA[,1:16] |> colnames()

##  [1] "First.Name"               "Last.Name"               
##  [3] "SID"                      "Email"                   
##  [5] "Sections"                 "Total.Score"             
##  [7] "Max.Points"               "Status"                  
##  [9] "Submission.ID"            "Submission.Time"         
## [11] "Lateness..H.M.S."         "View.Count"              
## [13] "Submission.Count"         "X1..Question.1..1.0.pts."
## [15] "X2..Question.2..1.0.pts." "X3..Question.3..1.0.pts."

Data here contains uniquely identifiable information – both directly ( First.Name, Last.Name, Submission.ID) & indirectly (Sections, Email). Therefore we are to strip that information to generate properly anonymized Data.

1.3 Abbreviations

ct : Count
inf. : Information
ID : Identification
rw : Row(s)
obs. : Observation(s)
func. : Function(s)
pct : Percent
num. : Number(s)
mult. : Multiple(s)

library(tidyverse)

2 Data Cleaning

2.1 Ideal Scenarios

n_rw <-
  F_F_VA |> nrow() +
  F_F_VB |> nrow() + 
  W_F_VA |> nrow() + 
  W_F_VB |> nrow() + 
  S_F_VA |> nrow() + 
  S_F_VB |> nrow()  

paste0("Sum of all Rows = ", n_rw, " rows")

## [1] "Sum of all Rows = 1520 rows"

ideally each row is a student but realistically not.
- At most we may have 1520 Students

2.2 Colnames

data_sets <- ls() |>
  keep(~ is.data.frame(get(.x)))

?keep

# check how keep works
x <- 1:10
# purrr::keep()
# x |> keep(\(x){x>=5})
# [1]  5  6  7  8  9 10
# keeps all those w/ cond. 
rm(x)

colnames_all <- set_names(
  map(
    data_sets,
    ~ colnames(get(.x))
  ),
  data_sets
)

unique_col <- 
  colnames_all |> 
  unlist() |> 
  unique()

unique_col |> as.data.frame()

3 Fall Cohort/Class

3.1 Midterm

3.1.1 Direct ID inf.

# Comb. Versions
F_M <-
  bind_rows(
  F_M_VA |> mutate(version="A", recency = 3),
  F_M_VB |> mutate(version="B", recency = 3)
)

paste0(F_M |> nrow(), " rw for Fall Midterm")

## [1] "818 rw for Fall Midterm"

paste0("Therefore ", round(F_M |> nrow()/n_rw, 2), "% (most) obs. are from Fall")

## [1] "Therefore 0.54% (most) obs. are from Fall"

3.1.1.1 `First.Name, Last.Name`

all(F_M$First.Name == toupper(F_M$First.Name))

## [1] FALSE

all(F_M$Last.Name == toupper(F_M$Last.Name))

## [1] FALSE

notice not all are upper case.

# make all upper
F_M <-
  F_M |> mutate(
  First.Name = toupper(First.Name),
  Last.Name = toupper(Last.Name)
  )

# toupper("a")
# rw id
F_M <- F_M |> mutate(rw_idx = row_number())

all(F_M$First.Name == toupper(F_M$First.Name))

## [1] TRUE

all(F_M$Last.Name == toupper(F_M$Last.Name))

## [1] TRUE

3.1.1.1.1 `First.Name, Last.Name` pairs ct

F_M |> distinct(First.Name, Last.Name) |> nrow()

## [1] 301

F_M |> nrow()

## [1] 818

Despite 818 rows, only 301 unique First.Name, Last.Name pairs were observed.
Meaning we have the same person(s) repeated mult. times for the same exam & quarter
To further inspect this, we must anonymize our std. of identifiable features.

3.1.2 `key_tbl_F_M`

# add unique std. id. for each unique FL-Name
key_tbl_F_M <-
  F_M |>
  distinct(First.Name, Last.Name) |>
  mutate(std_id = paste0("student_", row_number()))

# Join std_id
F_M <-
  F_M |>
  left_join(
    key_tbl_F_M,
    by = c("First.Name", "Last.Name")
  )

rep_obs_ct <-
  F_M |> 
  group_by(std_id) |> 
  summarize(ct = n()) |> 
  arrange(desc(ct)) 

rep_obs_ct

Most std. appear twice (2), one appears 4 times (student_110), one 216 times (student_40)

3.1.2.1 `student_40`

F_M |> filter(std_id=="student_40")

F_M |> filter(std_id=="student_40") |> count(Status)

round(F_M |> filter(std_id=="student_40") |> nrow() /nrow(F_M), 2)

## [1] 0.26

Notice some std. are so called UNIDENTIFIED STUDENT (26%)
All students were graded
These students scores cannot be connected across exams
I suspect these are students whom dropped the course

3.1.2.1.1 Does the dist. of `UNIDENTIFIED STUDENT` systematically differ (bias)?

# generate binary label 
F_M <-
  F_M |>
  mutate(
    dropped = factor(
      ifelse(std_id == "student_40", 1, 0),
      levels = c(0, 1),
      labels = c("Kept", "Dropped")
    )
  )
# non-ordinal, categorical
F_M$dropped |> unique()

## [1] Kept    Dropped
## Levels: Kept Dropped

# view dat
F_M |> 
  filter(dropped=="Dropped") |>
  select(dropped, First.Name, Last.Name) |> 
  slice(1)

F_M <- F_M |> mutate(pct=Total.Score/Max.Points)

# graph
library(patchwork)

graph_dat <-
  F_M |>
  filter(Status=="Graded")

# mean, median, mode : based on all dat
mean_pct <- mean(graph_dat$pct)
median_pct <- median(graph_dat$pct)

mode_pct <-
  graph_dat$pct |>
  na.omit() |>
  table() |>
  which.max() |>
  names() |>
  as.numeric()

p1 <-
  graph_dat |> 
  ggplot(aes(
    x = pct,
    y = after_stat(density),
    fill = dropped
  )) +
  geom_histogram(
    position = "identity",
    alpha = 0.4
  ) +
  # mean, median, mode
  geom_vline(xintercept=mean_pct, linetype="dashed") +
  geom_vline(xintercept=median_pct, linetype="dashed") +
  geom_vline(xintercept=mode_pct, linetype="dashed")

p2 <-
  graph_dat |>
  ggplot(aes(
    x = pct,
    color = dropped
  )) +
  geom_density(
    na.rm = TRUE
  ) +
  # mean, median, mode
  geom_vline(xintercept=mean_pct, linetype="dashed") +
  geom_vline(xintercept=median_pct, linetype="dashed") +
  geom_vline(xintercept=mode_pct, linetype="dashed")

p1 / p2

as we can see graphically, our dist appear to come from the same dist. The density of class student_40 and non-student_40 ("Kept", "Dropped") appears to be not systematic.
Looks like both classes ("Kept", "Dropped") maintain the modality (bimodal)
- Dist. graphically appears to be mixed beta dist.

# summary table
tibble(
  Statistic = c("Mean", "Median", "Mode"),
  Value = c(mean_pct, median_pct, mode_pct)
)

3.1.2.2 `student_110`

F_M |> 
  filter(std_id=="student_110") |>
  mutate(First.Name = "Person", Last.Name = "Name") |> 
  select(std_id, First.Name, Last.Name, Status, Total.Score, rw_idx)

here we see that student_110 has 4 repeated obs. – where only one is actually graded. We will discard the ungraded version.

3.1.2.3 Cause :

Cause <- 
  F_M[F_M$std_id=="student_110",] |> 
  distinct(Email, version) 
Cause[,]$Email <- rep(c("Email 1", "Email 2"), 2)
Cause; rm(Cause)

# keep NOT ungraded obs of student_110 
F_M <- F_M[-c(373, 782, 812),]

F_M |> 
  filter(std_id=="student_110") |>
  nrow() # good, 1 obs.

3.1.2.4 `2 obs. student(s)`

# v for vector, vector of ct == 2 std_id's
v <- rep_obs_ct[-(1:2),] |> filter(ct == 2) |> pull(std_id)

# rep_obs_ct |> filter(ct == 2) |> nrow() * 2
F_M |> filter(std_id %in% v) |> count(version)

# 598 == 299*2

here we see of those whom are repeated, half are version A, half version B
- It is likely that we inc. them whether graded or ungraded. Ie. std whom took VA but not VB could be included for both although only graded for 1.

F_M |> count(Status) |> mutate(pct = round(n/sum(n), 2))

Here we see, most std. exams were graded

# ct : (Status, version) pairs
F_M |> 
  filter(std_id %in% v) |> 
  count(Status, version)

# ct : (Status, version) pairs
F_M |> 
  filter(std_id %in% v) |> 
  count(Status, version) |> 
  group_by(Status) |> 
  summarize(ct = sum(n)) |> 
  mutate(pct = round(ct/sum(ct),2))

after grouping by Status, again about half are Graded

F_M |>
  filter(std_id %in% v) |>
  group_by(std_id) |>
  summarise(
    n_graded = sum(Status == "Graded"),
    n_missing = sum(Status == "Missing"),
    n_versions = n_distinct(version),
    .groups = "drop"
  ) |>
  count(n_graded, n_missing, n_versions)

S10 Data Cleaning [Report]

Isaiah C. Mireles

1 Preface

1.1 Data Sets

1.2 Anonymization

1.3 Abbreviations

2 Data Cleaning

2.1 Ideal Scenarios

2.2 Colnames

3 Fall Cohort/Class

3.1 Midterm

3.1.1 Direct ID inf.

3.1.1.1 `First.Name, Last.Name`

3.1.1.1.1 `First.Name, Last.Name` pairs ct

3.1.2 `key_tbl_F_M`

3.1.2.1 `student_40`

3.1.2.1.1 Does the dist. of `UNIDENTIFIED STUDENT` systematically differ (bias)?

3.1.2.2 `student_110`

3.1.2.3 Cause :

3.1.2.4 `2 obs. student(s)`

S10 Data Cleaning [Report]

Isaiah C. Mireles

1 Preface

1.1 Data Sets

1.2 Anonymization

1.3 Abbreviations

2 Data Cleaning

2.1 Ideal Scenarios

2.2 Colnames

3 Fall Cohort/Class

3.1 Midterm

3.1.1 Direct ID inf.

3.1.1.1 First.Name, Last.Name

3.1.1.1.1 First.Name, Last.Name pairs ct

3.1.2 key_tbl_F_M

3.1.2.1 student_40

3.1.2.1.1 Does the dist. of UNIDENTIFIED STUDENT systematically differ (bias)?

3.1.2.2 student_110

3.1.2.3 Cause :

3.1.2.4 2 obs. student(s)

3.1.1.1 `First.Name, Last.Name`

3.1.1.1.1 `First.Name, Last.Name` pairs ct

3.1.2 `key_tbl_F_M`

3.1.2.1 `student_40`

3.1.2.1.1 Does the dist. of `UNIDENTIFIED STUDENT` systematically differ (bias)?

3.1.2.2 `student_110`

3.1.2.4 `2 obs. student(s)`