Tutorial for Item Analysis (CTT) Shiny App

author: Lok Chau & Yan Liu

May 2, 2018

1 - Introduction

This Shiny App is to provide a user-friendly interface for users to conduct item analysis based on Classical Test Theory (CTT). Item analysis is used for examining responses to individual test items in order to assess the quality of items and of the test as a whole. It is valuable in improving items or eliminating poorly written or ambiguous item. This Shiny App provides several strategies used for item analysis, including reliability estimates, item difficulty, item discrimination (i.e.,item total correlation).

2 - Import File

This Shiny App supports both SPSS and csv data files. Note that for csv data file, the first row must be the headers of the variables (i.e., variable names).

Figure 1. Options screen.

Figure 1. Options screen.

In this user manual, we will use the sample data file CTT.csv. This sample data file consists of 10 multiple choice (MC) items, with raw, unscored resposnes from 100 examinees. These 10 MC items are mathematical items with two possible outcomes (correct/inccorect). To import data, you need to follow the following instruction:

  • Click on the ‘Browse’ button (Figure 1) and read in your own data file
  • Select part of items: Click on the bar under “Select items to be included in the analysis” and select the items for your analysis (Figure 2)
  • Include all variables: click Select all
  • Remove individual variables: click on the variable that you want to remove and then click “Delete” on your own keyboard
  • Click on “Confirm” to display the selected variables
Figure 2. Select items to be analyzed.

Figure 2. Select items to be analyzed.

By default, the first 10 rows of the data file will be displayed on the right (Figure 3). Users can change this default by clicking on “Show 10 entries” and select different number of rows.

Figure 3. Data view of the selected items

Figure 3. Data view of the selected items

3 - Scoring

If the item responses in the data file have been properly scored, you can skip this step and directly proceed to step-4 analysis. The sample data used for this demonstration are not scored with the original responses, A, B, C, or D. We need to score them to either correct (1) or incorrect(0).

This Shiny application provides an easy way to score the item responses (Figure 4).

  • Simply select Dichotomous under Dichotomous/Polytomous scoring:
  • Enter the key for each item.
  • click on Score items

The scored item responses will be displayed on the right. Note. The keys are case-sensitive and need to be in upper case.

For this sample data file, the correct keys to these 10 items are:

  • Item1 : D
  • Item2 : C
  • Item3 : A
  • Item4 : D
  • Item5 : D
  • Item6 : A
  • Item7 : D
  • Item8 : B
  • Item9 : D
  • Item10 : A
Figure 4. Interface for entering item keys.

Figure 4. Interface for entering item keys.

Case of polytomous items

If the items are polytomous and the original responses are recorded, there are two ways to score the items.

Same keys for all items

For psychological tests, you may have the case of the same scoring rule applying to all the items. If so, you can select Same keys for all items. For instance, the items are on Likert scale with three options of ‘Disagree’, ‘Neutral’, and ‘Agree’. You would want to score them as 1, 2, and 3 respectively.

The scoring rule needs to be expressed in the follow ways:

  • Use : to define each pair of original response and the corresponding score. For example, Disagree is scored as 1, so Disagree:1.
  • Use , to separate each pair.

Using the above example, the scoring rule is expressed as Disagree:1,Netural:2,Agree:3. Note that the scoring rule is case sensitive and no spaces are needed. In addition, you assume that no item needs to be reverse coded.

Different keys for each item

If there are muliple scoring rules to the items, you need to select Different keys for each item. For instance, some items have three choices, while othes have five choices, and not all items keyed in the same direction. In this case, you will need to provide the key for each item. The scoring rules are the same as shown in the example in ’Same keys for all items`.

4 - Analysis

Before conducting item analysis, users need to speicify how the missing values are handles and whether the scale of measurement should be treated as continuous or ordinal categorical.

Figure 5. Analysis options.

Figure 5. Analysis options.

Missing values

There are two ways to handle missing values. In Listwise deletion, the entire record is excluded from the analysis if any of the responses to the item is missing. In Treat as incorrect responses, all missing responses are treated as incorrect responses and scored to 0.

Scale of measurement of responses

You have the options to choose different correlation matrix for your item analysis based on the measurement scale of your data.

  • Pearson correlation: Click “Continuous”
  • Polycoric correlation: Click “Ordinal categorical”

5 - Results

The results are shown under the summary and plot tabs on the right.

  • Summary: You can find results of item analysis, including test level statistics, reliabitily if item deleted, item statistics (several types of item-total correlations)
  • Plots: Hitogram of total scores and bar chat for each item