1 Purpose of the Analysis

This analysis aims to evaluate the quality of test items in assessing students’ understanding of mathematical concepts using the Rasch Model. This model was chosen because it can assess item fit and estimate students’ abilities objectively along a single ability dimension.

2 Data Description

Data Description:

  • Number of respondents: 500 students
  • Number of test items: 15 multiple-choice items (true/false)
  • Scoring: 1 = correct, 0 = incorrect
  • Model used: Rasch
  • Estimation performed using the mirt package in R
#>  item1  item2  item3  item4  item5  item6  item7  item8  item9 item10 item11 
#>      0      0      0      0      0      0      0      0      0      0      0 
#> item12 item13 item14 item15 
#>      0      0      0      0

From the data above, there are no NA or missing values.

3 Model Rasch

df_rasch <- mirt(data = df, 
               model = 1,
               itemtype = "Rasch", technical = list(NCYCLES = 1000000),
               verbose = FALSE)

# save model
saveRDS(df_rasch, file = "model_rasch.RDS")

4 Parameter Item

### Load Model

model_rasch <- readRDS("model_rasch.RDS")

df_rasch_param <- coef(model_rasch, IRTpars = TRUE, simplify = TRUE)

df_rasch_param <- df_rasch_param %>% 
  as.data.frame() %>% 
  rename(Difficulty_Level = items.b,
         Discrimination = F1) %>% 
  select(Difficulty_Level, Discrimination)

df_rasch_param

Interpretation

  • The easiest item is item4, and
  • The most difficult item is item14.

5 Item Fit

item_fit <- itemfit(model_rasch)

item_fit %>% 
  mutate(information = 
           case_when(
            p.S_X2 < 0.05 ~ "no Fit",
            TRUE ~ "Fit"
           ))

Interpretation

Items 1, 2, 4, 5, 8, 11, 13, 14, and 15 show misfit (p < 0.05), meaning that students responses to these items do not align with the predictions of the Rasch Model.

6 Theta

theta_est <- fscores(model_rasch)
theta_est %>% 
  data.frame() %>% 
  rename(theta = F1) %>% 
  head(15)
df_theta <- cbind(df, theta_est)
df_theta %>% 
  rename(theta = F1) %>% 
  summarise(Min = min(theta),
         Max = max(theta),
         Mean = mean(theta),
         SD = sd(theta))

Interpretation

  • The estimated ability scores (theta) range from -1.325 to +0.9121, with a mean of -0.00000638 (SD = 0.4302).

7 ICC

plot(model_rasch, type = "trace")

This graph shows the probability of answering correctly relative to the ability level (theta). Each curve represents a single item.

8 Conclusion

  • The Rasch Model is not entirely suitable for this data, as most items show misfit.
  • The distribution of item difficulty levels is fairly good for measuring a range of student abilities.
  • Nine items (items 1, 2, 4, 5, 8, 11, 13, 14, and 15) need to be reviewed due to misfit.

9 Recommendation

Replace or revise the nine items (items 1, 2, 4, 5, 8, 11, 13, 14, and 15) because of misfit.