class: center, middle, inverse, title-slide # Conducting Simulation Studies in Psychometrics ## Educational Measurement: Issues and Practice ### 王慧心 顏立平 ### 2018.6.4 --- #Outline - Introduction - Uses of, and Common Terms in, Simulation - Simulation used in educational and psychological measurement - Principles and recommendations --- #Introduction -Disavetage of real data: - Time consuming - Costly to collect - May be inadequate to support the analyses, inferences, and conclusions - Research questions simply cannot be answered using empirical data --- #Introduction -History - The quality of beer (Student, 1908) - Nuclear warfare (von Neumann & Ulam, 1949) - Evaluating model – data fit (Hambleton, Swaminathan, & Rogers, 1991) - Item response theory (Harwell, Stone, Hsu, & Kirisci, 1996) - 2012 National Council on Measurement in Education (NCME) --- #Uses of, and Common Terms in, Simulation -Simulation methodologies - Monte Carlo (MC) simulation -Resampling methodologies - Bootstrap - Jackknife --- #Uses of, and Common Terms in, Simulation -Term - Factor - Condition - Replication - Bias - Standard error - Mean squared error --- #Simulation used in educational and psychological measurement -Types of Data Simulated - IRT Parameters - Raw(True) Scores - Response Times -Type of Research Utilizing Simulation - IRT Model Fit - Reliability --- #Simulation used in educational and psychological measurement -Number of Replications - 499 or Fewer (33%) - 1,000–4,999 (19%) -Results Present - Bias/RMSE - Type I/Power - ANOVA - Model Fit --- #Principles and recommendations -Steps taken in an IRT simulation study(Harwell et al.,1996) - Specifying a research question - Delineating conditions - Choosing an experimental design - Generating data - Estimating parameters - Replicating the procedure - Analyzing results --- #Principles and recommendations -Information for reader - Type and version of software - Type of hardware - Random number of seeds set - Distribution - Number of replications --- #Principles and recommendations -Analyzing results - Correlation - Bias - Mean absolute difference (MAD) - SE - Mean square error (MSE) - RMSE