Simulation of Student Performance on Tests

Dr Robert P. Batzinger and Ms Maleewan Suyalangka
5 May 2016

A word about this project

PYU Logo

This is a joint research project between the Department of Computer Science within the Faculty of Science and the Office of Educational Quality Assurance as part of an ongoing effort to improve the effectiveness and quality of education at Payap University in Chiang Mai, Thailand

Goals of this study

  • To simulate the test taking behavoir of students for the following purposes:

    • To illustrate the effect of class size, student competence and question types on test scores
    • To determine the minimum class size required for proper exam question analysis
    • To deveIop an understanding of the limitations in using exam results as a sole means to identify hard/easy test items questions

  • To gain practical experience using RStudio and RPubs as a statistical research environment

Test model for simulation:

  • This research assumes a general student population with these characteristics

    • Mean: 70
    • Range: 50 - 90
    • Standard deviation: 10

  • Simulated test cases are generated by R:

rnorm(20,70,10)

 [1] 53 53 56 58 63 63 66 68 68 69 70 71 73 75 77 80 82 82 83 86

Common random distributions

  • Uniformed - linear distribution
  • Normal - common distribution in nature
  • Binomial - common distribution in combinoric
  • Poisson - common for intervals between occurrences

plot of chunk unnamed-chunk-3

Distribution of random populations

plot of chunk unnamed-chunk-4

  • Uniform distribution models elements closely linked to linear probabilities
  • Normal distribution models common natural order of things

Individual scores by ability

Normally-distributed, individual students taking an exam of normally distributed difficulty will generate an uniformly distributed range of grades.

Large class (n=10,000) plot of chunk unnamed-chunk-5

Small class (n=3) plot of chunk unnamed-chunk-6

Class averages

Average scores for a test of 50 questions of normal-distributed difficulty by sets of classes with sizes ranging between 1 and 128 of normal-distributed students drawn randomly

plot of chunk unnamed-chunk-7

Two assessments of test difficulty

  • Analysis against a standard based on the full population
  easy = (exam[i] <= qnorm(.2,70,10))
  med =  ((exam[i] > qnorm(.2,70,10))  && 
            (exam[i] < qnorm(.8,70,10)))
  hard = (exam[i] >= qnorm(.8,70,10))
  • Analysis based on student test performance
  pass = (examdifficulty <= studentability)
  fail = (examdifficulty > studentability)
  easy = sum(fail)/j < .20
  hard = sum(pass)/j < .20
  med = 1 - (easy + hard)

Analysis against a population

Normally distributed test items classified by the degree of difficulty within the general population does not change with class size.

plot of chunk unnamed-chunk-10

Analysis by student performace

Difficulty measured by the student test performance varies with the size of the class

plot of chunk unnamed-chunk-11

Analysis against a population

Questions of uniformly distributed difficulty classified by a population of normally distributed ability is independant of class size.

plot of chunk unnamed-chunk-12

Analysis by student performance

Normally distributed students tend to find questions of uniformly distributed difficulty to be both harder and easier than a set of questions that have normally distributed difficulty

plot of chunk unnamed-chunk-13

Analysis by performance of weaker students

If the class average ability is lower (60 instead of 70), there is an asymetric increase in the number of perceived hard questions and a decrease in the number of questions found to be easy.

plot of chunk unnamed-chunk-14

Analysis of top student performance

If the class average ability is raised (80 instead of 70), there is an asymetric increase in the number of perceived easy questions and a decrease in the number of questions found to be hard.

plot of chunk unnamed-chunk-15

Effect of guessing

  • Questions with a limited number of responses encourage guessing
  • Correct guesses are distributed uniformly in a randomized test 
luck = fail * (runif(j,0,1) < 0.5)
fail = fail - luck
pass = pass + luck

Guessing on True/False Questions

  • Raises the number of passes by 25%
  • Greatly reduces the number of hard questions found
  • Increases the number of perceived easy questions.

plot of chunk unnamed-chunk-17

Guessing on multiple choice questions

  • Raises the average score (12.5% for 4 choices, 10% for 5 choices)
  • Reduces the number of perceived hard questions
  • Increases the number of easy questions

plot of chunk unnamed-chunk-18