The Application of Differential Normative Criteria to the Gifted Education Screening Phase: Implications for Demographic Representation
Scott Peters & Matthew McBee
4/6/2019
Abstract
Research question: Is using a group-specific norm for the nomination theshold, without altering the confirmatory test requirements, an effective strategy for reducing underrepresentation?
Quick Background
- How to measure underrepresentation
- OCR data
- Two-stage identification psychometrics
- Causes of demographic underpresentation
Representation index (RI)
Compares a group’s representation in gifted programs against their representation in the general population.
\[\text{RI} = \frac{\%_{\text{gifted}}(\text{group})}{\%_{\text{population}}(\text{group})}\]
For example, if a group comprises 20% of the population but only 10% of the gifted students:
\[\text{RI} = \big(\frac{.1}{.2}\big) = 0.5\]
OCR data - African American student representation
Peters, Gentry, Whiting, & McBee (2019) https://osf.io/325m9/
Relative Identitication Rate
The relative identification rate (RIR) is the ratio of two RIs.
\[\text{RIR} = \frac{\text{RI}_1}{\text{RI}_2}\]
Typically this is arranged such that the dominant or typically-represented group’s \(RI\) is in the denoninator.
Two-Stage Identification Process
The typical gifted identification process has two stages:
Nomination (controls who gets assessed)
Confirmatory testing (controls who gets identified)
Sensitivity and the Nomination Cutoff
In this example: test cutoff = 90th percentile, test reliability = 0.95, nomination validity = 0.5
Causes of underrepresentation
Logically underrepresentation must result from some combination of
Assessment bias
Actual differences in mean ability or achievement
Bias can operate at the nomination or confirmatory test phase. McBee (2016*) showed that observed Georgia data can be generated under a model in which there is no test bias. Both pure nomination bias and pure actual differences were plausible causes, or many combinations thereof.
True mean score differences can result from unequal opportunities to learn (OTLs, Peters & Engerrand, 2016).
Method
Our results are based on numerical calculations using the giftedCalcs package (McBee, Peters, & Godkin, in preparation).
To install giftedCalcs, run the following code in the R console.
install.packages("devtools")
devtools::install_github("mcbeem/giftedCalcs")
library(giftedCalcs)
Assumption
For this analysis, we assumed that the cause of underrepresentation is not assessment bias, but completely related to differential opportunities to learn.
Why?
- Worst-case scenario
- Early evidence from universal screening
- Most important: otherwise, the intervention is undefined
The reservoir of potentially qualifying students
How much must the nomination theshold be adjusted?
Baseline: 90th percentile nomination cutoff for the typically-represented group
|
|
|
Population mean
|
Cutoff z
|
Cutoff relative percentile
|
|
|
|
0
|
1.282
|
0.900
|
|
-0.1
|
1.182
|
0.881
|
|
-0.2
|
1.082
|
0.860
|
|
-0.3
|
0.982
|
0.837
|
|
-0.4
|
0.882
|
0.811
|
|
-0.5
|
0.782
|
0.783
|
|
-0.6
|
0.682
|
0.752
|
|
-0.7
|
0.582
|
0.720
|
|
-0.8
|
0.482
|
0.685
|
|
-0.9
|
0.382
|
0.649
|
|
-1
|
0.282
|
0.611
|
|
|
|
|
Relative identification rates in under universal consideration
|
|
|
Population mean
|
Relative identification rate (RIR)
|
|
|
|
0
|
1.000
|
|
-0.1
|
0.836
|
|
-0.2
|
0.692
|
|
-0.3
|
0.569
|
|
-0.4
|
0.463
|
|
-0.5
|
0.374
|
|
-0.6
|
0.299
|
|
-0.7
|
0.238
|
|
-0.8
|
0.187
|
|
-0.9
|
0.146
|
|
-1
|
0.113
|
|
|
|
Note: Assumes an idealized, unbiased single-stage identification process with zero measurement error.
|
Scenario 1: Baseline low-sensitivity identification process
Identification curves
Baseline: Test cutoff = 90th %ile. Nomination validity = 0.5. Test reliability = 0.95.
Panels are nomination cutoffs.
Relative identification rates
Baseline: Test cutoff = 90th %ile. Nomination validity = 0.5. Test reliability = 0.95.
Panels are nomination cutoffs.
Scenario 2: Baseline higher-quality identification process
Relative identification rates
Baseline: Test cutoff = 90th %ile. Nomination validity = 0.9. Test reliability = 0.95.
Panels are nomination cutoffs.
Discussion
Our analysis was premised on the notion that the cause is 100% real differences, 0% nomination bias, and 0% test bias.
In this case, the only way to reduce disproportionality in identification is by the introduction of a countervailing bias into the process favoring students from underrepresented groups.
Whether this is desirable or not is a question of values and politics.
Baseline low sensitivity is a requirement for this intervention to have any effect, and low sensitivity means that many qualifying students will fail to be identified.
Simply improving the identification process for everyone will increase sensitivity for all students.
We cannot generally endorse this policy.
Postscript: Help me improve giftedCalcs
Postscript: Help me improve giftedCalcs
