Simulation

For each construct, we simulated a complete response matrix \(X_{pi}\) in which each student \(p\) responded to each item \(i\). We treated the estimated item parameters from the original calibration (in ConQuest parameterization), and the associated WLE person estimates as the true \(\delta_i\), \(\tau_{ik}\), and \(\theta_p\). With these margins fixed, we used the ConQuest generate command to simulate a random response matrix using the PCM as the data-generating model.

We then fit the PCM to this response matrix, with all the structural parameters (item parameters \(\delta_i\), \(\tau_{ik}\), and person distribution² \(\mu_\theta\) and \(\sigma^{2}_\theta)\)) anchored.

Below, we refer to results from the original calibrations as original, and the results from fitting the PCM to the simulated response matrices as simulated. We also refer to the WLE estimates from the original calibrations as the generating \(\theta\) and those from fitting the PCM to the simulated response matrices as the recovered \(\theta\) . Since the item parameters were anchored, our interest is primarily in the measurement errors (WLE standard errors) and the associated reliability estimates.

Analytic Methods

Additionally, we used an analytic method to estimate measurement error and reliability, based on the additivity of Fisher information across items (assuming local independence).

In the PCM, item information function³ is:

\[ \begin{equation} I_i(\theta) = a_i^2 \left[ \sum_{k=1}^m {k^2 P_{ik}(\theta)} - \left(\sum_{k=1}^m {k P_{ik}(\theta)}\right)^2 \right] \end{equation} \]

where \(a_i = 1\) in Rasch models, and \(P_{ik}(\theta)\) is the usual PCM likelihood⁴:

\[ \begin{equation}\begin{split} P_{ik}(\theta) &= Pr(X_{pi} = k | \theta_p=\theta, \delta_{ik} ) \\ &= \frac { \exp \sum_{k=0}^a ( \theta_p - \delta_{ik}) } { \sum_{h=0}^{A_i} \exp \sum_{k=0}^h ( \theta_p - \delta_{ik}) } \end{split}\end{equation} \] where \(\delta_{ik}\) are the step difficulties (in Masters’ parameterization).

Because we are acting as if all the students responded to all the items, we compute the test information function as the sum of all the item information functions. The Standard Error of Measurement (SEM) function is the inverse square root of the test information function: \[ \begin{align} I(\theta) &= \sum_{i} I_i(\theta) \\ \text{SEM}(\theta) &= \frac{1}{\sqrt{I(\theta)}} \\ \end{align} \] We refer to these as analytic standard errors.

For reliability, we use a formula from Classical Test Theory: \[ \begin{equation} \rho_{xx'} = \frac {\sigma^2_T} {\sigma^2_T + \sigma^2_E} \end{equation} \] For the true score variance \(\sigma^2_T\) we use the population variance from the original calibration. We compute the error variance \(\sigma^2_E\) by margining out the person distribution from analytic measurement error variance function (\(\text{SEM}^2\) or \(1/I(\theta)\)). We do this by taking a weighted average along a gird of \(\theta\) values, with weights from the Normal PDF, having mean and variance from the original calibration. We refer to these as analytic reliability estimates.

We plan to repeat this analysis using the empirical distribution of \(\theta\) (i.e., taking an unweighted average of the of the analytic measurement error variance function at the the WLE estimates from the original calibration).

Short vs. Simulated Test Reliability

In the table and figure below, analytic represents the analytic reliability estimates described above. The original reliability estimates from ConQuest are orig.EAP, orig.MLE, and orig.WLE (based on EAP, MLE, and WLE. respectively). Likewise, sim.EAP, sim.MLE, and sim.WLE are the simulated reliability estimates.

Initial Observations

• The original MLE and WLE reliabilities for MAPm are 0. We do not yet know the cause of these impossibly low estimates.
• The simulated EAP reliabilities for MMRc and IMR are 1. We do not yet know if this is a random artifact of the simulation, or an indicator of a more serious problem.
• In each construct, the analytic reliabilites are generally greater than the original reliabilities, but less than the simulated reliabilities.