Comparison of performance between Case and Control Groups - RAP Evaluation

This notebook presents the data and calculations made to evaluate if there is a statistically significant difference between the performance of students that were exposed to the automated feedback of the RAP system and the performance of students in the control group (just practice, but no automated feedback).

Data

The data used in this analysis is an anonymized version of the data collected during the final measurement. It can be downloaded here.

data <- read.csv("final_measurement.csv")
head(data)

The “Score” column represents the final score assigned by the human expert (professor). The “Group” column indicate if that student was in the Control or Case group.

We can also obtain basic statistics about the students in the different groups:

summary(data$Group)

   Case Control 
     85      95 

summary(data[data$Group=="Case",]$Score)

   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
  13.00   16.00   18.00   17.15   18.00   20.00 

sd(data[data$Group=="Case",]$Score)

[1] 1.867682

summary(data[data$Group=="Control",]$Score)

   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
  12.00   15.00   17.00   16.62   18.00   20.00 

sd(data[data$Group=="Control",]$Score)

[1] 2.297913

Visualization

We can show that distribution of scores will be presented through a boxplot separating the Case and Control groups

library(hrbrthemes)
library(viridis)
library(ggplot2)
data %>%
  ggplot( aes(x=Group, y=Score, fill=Group)) +
  geom_boxplot() +
  scale_fill_viridis(discrete = TRUE, alpha=0.6) +
  geom_jitter(color="black", size=0.4, alpha=0.9) +
  theme_ipsum() +
  theme(
    legend.position="none",
    plot.title = element_text(size=11)
  ) +
  ggtitle("") +
  xlab("")

Difference Test

To test the difference between the means, we use a t-test with the alternative hypothesis that the scores in the case group are greater than the scores in the control group.

t.test(Score ~ Group, data = data, alternative = "greater", paired = FALSE)

    Welch Two Sample t-test

data:  Score by Group
t = 1.7111, df = 176.42, p-value = 0.04441
alternative hypothesis: true difference in means is greater than 0
95 percent confidence interval:
 0.01790396        Inf
sample estimates:
   mean in group Case mean in group Control 
             17.15294              16.62105 

While visually there seems to be a considerable overlap, the test shows that there is a small but significant difference between the two distributions.

Effect Size

To estimate the relative strenght of the intervention we calculate the effect size using the Cohen d. The result is 0.25 that confirms that the effect is small.

suppressMessages(library(effsize))
case=data[data$Group=="Case",]$Score
control=data[data$Group=="Control",]$Score
cohen.d(case,control)

Cohen's d

d estimate: 0.252575 (small)
95 percent confidence interval:
      lower       upper 
-0.04322303  0.54837310