CEP 923 Final Project: Approaches to Dealing with Speededness

Presentation Overview:

• Speeded test basics and issues
• Approaches to speeded tests
• PISA Problem Solving Data Analysis

Whenever tests involve a time limit, the rate at which students work will affect their performance, and this can undermine the validity of the test for measuring the construct of interest.

• Speed vs Power Tests
• Score Approach vs Fairness Approach
• Error Score Partitioning
  • Omitted items
  • Items not reached
• Impact on Validity and Reliability

• Parameter Estimate Bias
• Effect on Factor Structure
• Differential Item Functioning
• Content Validity
• Issues with IRT assumptions
• Approaches to Identifying Speededness
  • Multiple Administration
  • Single Administration

Detecting vs Removing Effects of Speededness

• Incorrect vs Missing

Detecting Speededness effects

• 2 class mixture Rasch Model (Bolt et al 2002)
• Mixture Rasch (Wollack 2003)
• Goegebeur et al. (2008)

Improved Model Estimation

• Maximum Marginal Likelihood Estimation (Glas and Pimentel 2008)
• Performance Decrement Parameters (Jin and Want 2014)
• Bayesian MCMC estimation 2PL model (Chang et al. 2014)

2003 PISA Problems Solving Items: 4 Items 10 quesitons

• Examination of different approaches
• Simulation Study
• Application of Models to PISA Data
  • Speededness Effects
  • Testlet Effects