Identifying gifted English learners: Reforging the leaky pipeline into appropriate programming and services

AERA 2017

Michael S. Matthews, Matthew T. McBee, and Scott J. Peters

April 29, 2017

These slides are available at www.rpubs.com/mmcbee.

Some background context

Underrepresentation in Gifted Education

Representation Index (RI)

The RI compares eaach demographic group’s representation in gifted education with their share of the population.

An RI of 1.0 indicates perfect proportionality.

\[\begin{equation} \text{RI}=\frac{\text{% gifted}}{\text{% total}} \end{equation}\]

RIs from 2013-2014 in the United States

Group RI
Male 0.96
Female 1.04
Latino/a 0.67
Am Indian/Alaskan 0.61
Asian 2.02
Hawiian/Pacific Islander 0.69
Black 0.6
White 1.13
Multiracial 1.05
LEP/ELL 0.27
IDEA 0.25

Unique Features of ELL / LEP Population

Representation Index by State for ELL Students

Data from the Office of Civil Rights

Florida Identification Process

  1. Teacher / Parent Nomination

  2. Screening via group-administered ability test

  3. Confirmatory testing via individually-administered ability test

Two Standards for Identification

Data from Matthews & Kirsch (2011) and other sources

non-ELL ELL
Screening test cutoff 135 120
Test cutoff 130 115
Population size 175,268 24,732
Number nominated 22,368 432
Number identified under Plan A 11,184 92
Number identified under Plan B 265
Total number identified 11,184 357
Nomination rate 0.128 0.017
Identification rate under Plan A 0.064 0.004
Total identification rate 0.064 0.014
Nomination pass rate under Plan A 0.500 0.213
Total nomination pass rate 0.500 0.826

Performance Statistics

Identification rate

\[\begin{equation}\label{id_rate} \begin{split} \text{identification rate} = p(\text{identified}) = \\ \int_{-\infty}^{\infty} \int_{\kappa}^{\infty} \int_{-\infty}^{\infty} \int_{\nu}^{\infty} \int_{-\infty}^{\infty} \int_{\tau}^{\infty} \ N_6(\boldsymbol{\mu}, \boldsymbol{\Sigma}) \ d_{C_o} d_{C_t} d_{S_o} d_{S_t} d_{N_o} d_{N_t} \end{split} \end{equation}\]

Sensitivity

\[\begin{equation}\label{sensitivity} \begin{split} \text{sensitivity} = p(\text{identified }|\text{ gifted})=\\ \frac{p(\text{gifted, identified})}{p(\text{gifted})} = \frac{\int_{-\infty}^{\infty} \int_{\kappa}^{\infty} \int_{-\infty}^{\infty} \int_{\nu}^{\infty} \int_{\tau}^{\infty} \int_{\tau}^{\infty} \ N_6(\boldsymbol{\mu}, \boldsymbol{\Sigma}) \ d_{C_o} d_{C_t} d_{S_o} d_{S_t} d_{N_o} d_{N_t} }{\int_{\tau}^{\infty} \ N(\mu,\sigma) \ d_{T_t}} \end{split} \end{equation}\]

Performance Statistics continued

Specificity

\[\begin{equation}\label{specificity} \begin{split} \text{specificity} = p(\text{not identified }|\text{ not gifted})= \frac{p(\text{not gifted, not identified})}{p(\text{not gifted})} = \\ \frac{\int_{-\infty}^{\infty} \int_{-\infty}^{\kappa} \int_{-\infty}^{\infty} \int_{-\infty}^{\infty} \int_{-\infty}^{\tau} \int_{-\infty}^{\infty} \ N_6(\boldsymbol{\mu}, \boldsymbol{\Sigma}) \ d_{C_o} d_{C_t} d_{S_o} d_{S_t} d_{N_o} d_{N_t}}{\int_{-\infty}^{\tau} \ N(\mu,\sigma) \ d_{T_t}} + \\ \frac{ \int_{-\infty}^{\infty} \int_{\kappa}^{\infty} \int_{-\infty}^{\infty} \int_{-\infty}^{\nu} \int_{-\infty}^{\tau} \int_{-\infty}^{\infty} \ N_6(\boldsymbol{\mu}, \boldsymbol{\Sigma}) \ d_{C_o} d_{C_t} d_{S_o} d_{S_t} d_{N_o} d_{N_t}}{{\int_{-\infty}^{\tau} \ N(\mu,\sigma) \ d_{T_t}}} +\\ \frac{ \int_{-\infty}^{\infty} \int_{\kappa}^{\infty} \int_{-\infty}^{\infty} \int_{\nu}^{\infty} \int_{-\infty}^{\tau} \int_{-\infty}^{\tau} \ N_6(\boldsymbol{\mu}, \boldsymbol{\Sigma}) \ d_{C_o} d_{C_t} d_{S_o} d_{S_t} d_{N_o} d_{N_t}}{\int_{-\infty}^{\tau} \ N(\mu,\sigma) \ d_{T_t}} \end{split} \end{equation}\]

Performance Statistics continued

False negative rate (FNR)

\[\begin{equation}\label{fnr} \begin{split} \text{false negative rate} = p(\text{not identified } | \text{ gifted}) = \frac{p(\text{not identified, gifted})}{p(\text{gifted})} = 1 - \text{sensitivity} \end{split} \end{equation}\]

False positive rate (FPR)

\[\begin{equation}\label{fpr} \begin{split} \text{false positive rate} = p(\text{identified } | \text{ not gifted}) = \frac{p(\text{identified, not gifted})}{p(\text{not gifted})} = 1 - \text{specificity} \end{split} \end{equation}\]

Performance Statistics continued

Incorrect identification rate (1-PPV)

\[\begin{equation}\label{incorrectrate} \begin{split} \text{incorrect identification rate} = p(\text{not gifted } | \text{ identified}) = \\ \frac{p(\text{not gifted, identified})}{p(\text{identified})} = \\ \frac{\int_{-\infty}^{\infty} \int_{\kappa}^{\infty} \int_{-\infty}^{\infty} \int_{\nu}^{\infty} \int_{-\infty}^{\tau} \int_{\tau}^{\infty} \ N_6(\boldsymbol{\mu}, \boldsymbol{\Sigma}) \ d_{C_o} d_{C_t} d_{S_o} d_{S_t} d_{N_o} d_{N_t} }{\int_{-\infty}^{\infty} \int_{\kappa}^{\infty} \int_{-\infty}^{\infty} \int_{\nu}^{\infty} \int_{-\infty}^{\infty} \int_{\tau}^{\infty} \ N_6(\boldsymbol{\mu}, \boldsymbol{\Sigma}) \ d_{C_o} d_{C_t} d_{S_o} d_{S_t} d_{N_o} d_{N_t} } \end{split} \end{equation}\]

Performance Statistics continued

Nomination hit rate

\[\begin{equation}\label{nom_hitrate} \begin{split} \text{nomination hit rate} = p(\text{identified } | \text{ nominated}) = \frac{p(\text{identified, nominated})}{p(\text{nominated})} = \\ \frac{\int_{-\infty}^{\infty} \int_{\kappa}^{\infty} \int_{-\infty}^{\infty} \int_{\nu}^{\infty} \int_{-\infty}^{\infty} \int_{\tau}^{\infty} \ N_6(\boldsymbol{\mu}, \boldsymbol{\Sigma}) \ d_{C_o} d_{C_t} d_{S_o} d_{S_t} d_{N_o} d_{N_t} } {\int_{\kappa}^{\infty} \ N(\mu,\sigma) \ d_{N_o}} \end{split} \end{equation}\]

Screening test hit rate

\[\begin{equation}\label{screener_hitrate} \begin{split} \text{screener hit rate} = p(\text{identified } | \text{ passed screener}) = \frac{p(\text{identified, passed screener})}{p(\text{passed screener})} = \\ \frac{\int_{-\infty}^{\infty} \int_{\kappa}^{\infty} \int_{-\infty}^{\infty} \int_{\nu}^{\infty} \int_{-\infty}^{\infty} \int_{\tau}^{\infty} \ N_6(\boldsymbol{\mu}, \boldsymbol{\Sigma}) \ d_{C_o} d_{C_t} d_{S_o} d_{S_t} d_{N_o} d_{N_t} } {\int_{-\infty}^{\infty} \int_{\kappa}^{\infty} \int_{-\infty}^{\infty} \int_{\nu}^{\infty} \int_{-\infty}^{\infty} \int_{\infty}^{\infty} \ N_6(\boldsymbol{\mu}, \boldsymbol{\Sigma}) \ d_{C_o} d_{C_t} d_{S_o} d_{S_t} d_{N_o} d_{N_t} } \end{split} \end{equation}\]

Performance Statistics continued

Screening test sensitivity

\[\begin{equation}\label{screener_sensitivity} \begin{split} \text{screener sensitivity} = \\ p(\text{nominated, pass screener }|\text{screener true score} > \text{screener cutoff})= \\ \frac{\int_{-\infty}^{\infty} \int_{\kappa}^{\infty} \int_{\nu}^{\infty} \int_{\nu}^{\infty} \int_{-\infty}^{\infty} \int_{\infty}^{\infty} \ N_6(\boldsymbol{\mu}, \boldsymbol{\Sigma}) \ d_{C_o} d_{C_t} d_{S_o} d_{S_t} d_{N_o} d_{N_t} }{\int_{\nu}^{\infty} \ N(\mu,\sigma) \ d_{S_t}} \end{split} \end{equation}\]

Screening test specificity

\[\begin{equation}\label{screener_specificity} \begin{split} \text{screener specificity} = \\ p(\text{not pass screener }|\text{screener true score} < \text{screener cutoff})= \\ \frac{\int_{-\infty}^{\infty} \int_{-\infty}^{\kappa} \int_{-\infty}^{\nu} \int_{-\infty}^{\infty} \int_{-\infty}^{\infty} \int_{-\infty}^{\infty} \ N_6(\boldsymbol{\mu}, \boldsymbol{\Sigma}) \ d_{C_o} d_{C_t} d_{S_o} d_{S_t} d_{N_o} d_{N_t}}{\int_{-\infty}^{\nu} \ N(\mu,\sigma) \ d_{N_t}} + \\ \frac{ \int_{-\infty}^{\infty} \int_{\kappa}^{\infty} \int_{-\infty}^{\nu} \int_{-\infty}^{\nu} \int_{-\infty}^{\infty} \int_{-\infty}^{\infty} \ N_6(\boldsymbol{\mu}, \boldsymbol{\Sigma}) \ d_{C_o} d_{C_t} d_{S_o} d_{S_t} d_{N_o} d_{N_t}}{{\int_{-\infty}^{\nu} \ N(\mu,\sigma) \ d_{N_t}}} \end{split} \end{equation}\]

Performance Statistics continued

Screening test false positive rate

\[\begin{equation}\label{screener_fpr} \begin{split} \text{screener false positive rate} = p(\text{passed screener } | \text{ screener true score} < \text{screener cutoff}) = \\ 1- \text{screener specificity} \end{split} \end{equation}\]

Screening test pass rate

\[\begin{equation}\label{screener_passrate} \begin{split} \text{screener hit rate} = p(\text{ passed screener}) = \\ \int_{-\infty}^{\infty} \int_{\kappa}^{\infty} \int_{-\infty}^{\infty} \int_{\nu}^{\infty} \int_{-\infty}^{\infty} \int_{-\infty}^{\infty} \ N_6(\boldsymbol{\mu}, \boldsymbol{\Sigma}) \ d_{C_o} d_{C_t} d_{S_o} d_{S_t} d_{N_o} d_{N_t} \end{split} \end{equation}\]

Assumptions

  1. The classical test theory model is a reasonably accurate portrayal of measurement errors and resulting diagnostic classification errors.

  2. The informal teacher or parent nomination process can be modeled as an implicit measurement that is compared against some cutoff (see McBee et al, 2016 for discussion).

  3. The nomination, screening, and confirmatory test scores follow a multivariate normal distribution.

  4. The ELL population mean cognitive ability score is 100.

  5. The screening and confirmatory tests do not suffer from systematic bias. This means that their effective cutoffs as specified by the school district (\(\nu=1.33\), \(\tau=1.0\) in the standardized metric).

Structural equation model for generating the Theta matrix

Structural equation model for generating the Theta matrix

\[\begin{equation}\label{FA} \underset{k \times k}{\boldsymbol{\Sigma}} = \underset{k\times p}{\mathbf{\Lambda}} \ \underset{p\times p}{\mathbf{\Phi}} \ \underset{k\times p}{\mathbf{\Lambda}}^{'} + \underset{k\times k}{\mathbf{\Theta}} \end{equation}\] \[\begin{equation} \boldsymbol{\Lambda} = \begin{bmatrix} 1 & 0 & 0\\ \sqrt{\rho_{NN}} & 0 & 0\\ 0 & 1 & 0 \\ 0 & \sqrt{\rho_{SS}} & 0 \\ 0 & 0 & 1 \\ 0 & 0 & \sqrt{\rho_{CC}} \end{bmatrix} \end{equation}\] \[\begin{equation} \boldsymbol{\Phi} = \begin{bmatrix} 1 & r_{SN} & r_{CN}\\ r_{SN} & 1 & r_{CS}\\ r_{CN} & r_{CS} & 1 \end{bmatrix} \end{equation}\] \[\begin{equation}\label{theta} \underset{k\times k}{\mathbf{\Theta}}= \underset{k\times k}{I} - \text{diag}( \underset{k\times p}{\mathbf{\Lambda}} \ \underset{p\times p}{\mathbf{\Phi}} \ \underset{k\times p}{\mathbf{\Lambda}}^{'}) \end{equation}\]

Calculating system performance when there are multiple opportunities for identificiation

Opportunities for Identification by Grade

Grade Opportunities
Kindergarten 1
1st 2
2nd 3
3rd 4
4th 5
5th 6

Calculating Multi-Year Performance Statistics: Memoryless case

The state vector (\(\boldsymbol{S}_i\)) describes the proportion of students in each possible state: true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN)

\(\boldsymbol{S}_0\) is the initial state, before any opportunities for identification. Therefore, all the gifted students begin as false negatives, the non-gifted as true negatives. Using the IQ=115 Plan B cutoff:

\[\begin{equation} \boldsymbol{S}_0 = \begin{bmatrix} 0 & 0.84 & 0 & 0.16 \end{bmatrix} \end{equation}\]

where the order is TP, TN, FP, FN.

Calculating Multi-Year Performance Statistics: Memoryless case continued

The transition matrix (\(\boldsymbol{T}\)) gives the probabilities of moving from one state to another. Rows describe the previous state, columns describe the next state.

\[\begin{equation}\label{transition} \boldsymbol{T} = \begin{bmatrix} 1 & 0 & 0 & 0 \\ 0 & (1-\text{fpr}) & \text{fpr} & 0 \\ 0 & 0 & 1 & 0 \\ \text{sens} & 0 & 0 & (1-\text{sens}) \end{bmatrix} \end{equation}\]

where the order is TP, TN, FP, FN; fpr is the false positive rate and sens the sensitivity.

Note: the values in this matrix assume that gifted identification is permanent once granted.

Calculating Multi-Year Performance Statistics: Memoryless case continued

When there is no system memory, each chance is independent. The proportion of TPs, TNs, FPs, and FNs after \(i\) chances can be calculated using a Markov chain model.

\[\begin{equation} \boldsymbol{S}_i = \boldsymbol{S}_0 \boldsymbol{T}^i \end{equation}\]

Calculating Multi-Year Performance Statistics: general case

Gifted identification probably isn’t memoryless.

A student tested last year is probably less likely to be tested again this year.

This can be modeled by a generalized Markov model, introducing a memory parameter (\(m\)) to the transition matrix equation.

\[\begin{equation}\label{transition_m} \begin{split} \boldsymbol{T}_{i,m} = \begin{bmatrix} 1 & 0 & 0 & 0 \\ 0 & (1-\text{fpr})^{i^m} & \text{fpr}^{i^m} & 0 \\ 0 & 0 & 1 & 0 \\ \text{sens}^{i^m} & 0 & 0 & (1-\text{sens})^{i^m} \end{bmatrix} \end{split} \end{equation}\]

The implication is a different transition matrix for each \(i\) opportunity.

The state vector after k opportunities is:

\[\begin{equation}\label{statevector_m} \boldsymbol{S}_{k,m} = \boldsymbol{S}_0 \prod_{i=1}^k \boldsymbol{T}_{i,m} \end{equation}\]

Grid search (continued)

After eliminating impossible correlations, we calculated the multiyear performance statistics for each of the previous combination of parameters at the following:

memory: 0 to 2 (increment .05)

For each, we calculated marginal perforance statistics by averaging over the statistics after each of six chances. So the marginal is based on \(\frac{1}{6}\) having one chance (Kindergarteners), \(\frac{1}{6}\) having two chances (1st graders), and so on.

Performance statistics were calculated for 41,123 combinations of parameters.

We then selected the subset of these best able to reproduce the ELL data. A total of 2,335 combinations had RMS error values smaller than 0.01.

Results

Plot of values that reproduce the observed ELL data

Density plots of computed sensitivity of ELL identification

Upper panel: nomination cutoff. Middle panel: memory parameter. Bottom panel: nomination validity coefficient

Density plot of plausible values for key parameters

Counterfactual analysis

We considered: what would happen to the performance metrics under less restrictive nomination cutoffs?

We analyzed nomination cutoffs of \(\kappa \in \{100, 105, 110, 115\}\).

We used the plausible values of the nominatation validity and screener validity and updated the calculated system performance statistics.

We first considered single-opportunity performance statistics.

Violin plots of counterfactual performance metrics, single opportunity case.

Upper left panel: sensitivity. Upper right panel: false positive rate. Lower left panel: nomination hit rate. Lower right panel: identification rate."

Then we considered multi-year performance statisics, marginal over six opportunities for identification.

Since the memory parameter \(m\) is unknown, we assumed values of \(m \in \{0, .5, 1, 1.5, 2\}\).

Violin plots for counterfactual sensitivity by memory and nomination cutoff, integrated over six opportunities for identification.

Panels display strength of memory parameter

Violin plots for counterfactual identification rate by memory and nomination cutoff, integrated over six opportunities for identification

Panels display strength of memory parameter. Horizontal reference line indicates actual non-ELL identification rate.

Violin plots for counterfactual false positive rate by memory and nomination cutoff, integrated over six opportunities for identification.

Panels display strength of memory parameter.

Discussion