Question for Week 14

Use simulation to investigate the recovery of the mean and variance of ability distribution using JML and MML. Vary test length to see if the results change.

To answer this question, the following four simulations were run. Each simulation included,
1. 1000 students
2. Student ability thetas are random normal with M=0, and SD=1
3. Item difficulty deltas from -2 to 2

To test for the effect of the number of items, item numbers ranged from 20 to 80 for increments of 1 (length 61).

1. Effect of Number of Items on Mean Student Ability for MML- and WLE-derived(afterthought) procedures

Results from Figure 1 suggest that the MML-derived mean estimates are, in each case, fixed to 0.00. The means derived as an afterthought tend to vary more as the number of items increases.

Figure 1.

2. Effect of Number of Items on Variance in Student Ability for MML- and WLE-derived(afterthought) procedures

Results from Figure 2 suggest that the variance in ability for the MML- and WLE-derived(afterthought) procedures are quite different. The variance from the MML estimation hovers around 1 in alignment with the generated variance. However, the WLE-derived(afterthought) variances tend to be larger than the generated variances.

Figure 2.

3. Effect of Number of Items on Average Student Ability for JML- and MML-procedures

In Figure 3, the MML-derived ability averages are fixed at zero (as described above). However, the JML average ability estimates are not (with ability estimated jointly from both student and item information)–for this reason, the JML mean ability estimates are different, i.e., lower. It is noted that as the number of items increases, the MML- and JML-derived estimates begin to converge.

Figure 3.

4. Effect of Number of Items on Variance in Student Ability for JML- and MML-procedures

In Figure 4 we note that the JML estimated variance in ability is greater than the MML estimated variance in ability for each item number condition. However, as the number of items increases, the difference in variance between the estimation procedures seems to reduce. This is because an increase in the number of items results in a reduction of measurement error in both estimation procedures.

Figure 4.

Summary

As the length of the test increases, measurement error reduces and the variance in student ability approaches the predetermined level of variance, regardless of the estimation procedure. However, because the MML estimation accounts for the increase in the variance by attenuating the observed variance, MML estimation tends to better replicate the true variance of the test taking group.

END