So far

• The ordinal outcome variable (response) is treated as a continuous variable.
• We compute point estimates of the mean response category for old and new items for each combination of test type/group, and then compute a difference score mean(old) - mean(new).
• This corresponds to a maximum likelihood (point) estimate. Uncertainty regarding this estimate is not quantified.

Musicians vs controls

Absolute pitch, relative pitch, controls

Synaesthetes, absolute pitch, relative pitch, controls

Ordinal data

• Although ordinal data are not metric, they are often analyzed using methods that assume metric responses. This practice may lead to serious errors in inference (Liddell & Kruschke, 2018).
• Ordinal variables: categories have an ordering, but it is unknown
  • what the psychological distance between them is
  • whether distances between categories are the same across participants

Ordinal regression

• Use the framework of signal detection (unequal variance SDT or logistic model with heteroscedastic error)
• Work with raw responses, instead of summarizing data
• Quantify uncertainty at all levels
• Allows multilevel model (shrinkage could be especially important due to low number of subjects)

Unequal Variance (logistic) SDT Model

• Item is either old or new
• Subjects do not provide binary old or new responses, but instead give their responses on a 5-point rating scale
• Subjects rate their confidence in whether the item was old or new (actually, how frequently the item was presented)
• Subjects set a number of criteria for the ratings, such that greater evidence is required for 5-responses, than 4-responses, for example.

\[ P(Y \leq k | X) = F\left( \frac{ c_k - dX} { \sigma_X } \right)\]

for \(k=1\) to \(K-1\), where

• \(K\) is the number of response categories
• \(Y\) is a response rating, taking on the values \(k=1\) to \(K\)
• \(F\) is a cumulative distribution function
• \(c_k\) are response criteria
• \(\sigma_X\) is the standard deviation of the latent distribution

Model specification

We wish to predict subjects’ responses from

• their group membership (synesthesia/absolute pitch/relative pitch/non-musicians)

• the type of test (color only/tone only/both)

• whether the item was previously seen or not (old/new)

• We use the full random effects structure that the experimental design allows:

• Varying coefficients for grouped subjects and varying item intercepts for the linear predictor of cumulative response probability

• Unequal latent variances (discrimination parameter) for old/new items and for each group (the disc parameter will be the exponentiated inverse of the standard deviation \(s\) of \(\tilde{Y}\)).

formula <- bf(response ~ 1 + test * group4 * oldnew + 
                  (1 + test * oldnew | gr(subject, by = group4)) + 
                  (1 + test * oldnew | item)) +
  lf(disc ~ 0 + group4 * oldnew + 
       (0 + oldnew | gr(subject, by = group4)), cmc = FALSE)

priors <- prior(normal(0, 3), class = Intercept) +
  prior(normal(0, 1), class = b) +
  prior(normal(0, 1), class = b, dpar = "disc") +
  prior(student_t(3, 0, 1), class = sd, group = subject) +
  prior(student_t(3, 0, 1), class = sd, group = item) +
  prior(lkj(2), class = cor, group = subject) +
  prior(lkj(2), class = cor, group = item)

fit_1 <- brm(formula = formula,
            family = cumulative(),
            data = data,
            prior = priors,
            init_r = 0.05,
            control = list(adapt_delta = 0.9),
            cores = parallel::detectCores())

Problematic data points

## Pareto k diagnostic values:
##                          Count Pct.    Min. n_eff
## (-Inf, 0.5]   (good)     3162  99.8%   656       
##  (0.5, 0.7]   (ok)          6   0.2%   651       
##    (0.7, 1]   (bad)         0   0.0%   <NA>      
##    (1, Inf)   (very bad)    0   0.0%   <NA>      
## 
## All Pareto k estimates are ok (k < 0.7).

pareto_k_ids(loo_fit_1)

## [1]  337  569 1143 1604 1612 2347

plot(loo_fit_1)

data[pareto_k_ids(loo_fit_1), ] %>% knitr::kable()

Posterior distributions of conditional means

Posterior distributions of conditional means

Posterior distributions of conditional response probabilities

Posterior predictive checks

Score (old - new)

Old / New responses

Todo list

• Hypothesis testing
• Generate predictions for future experiments (e.g. simulate new synaesthetes)
• formalize connection between learning score and SDT model

Questions

• Is there a benefit to using the SDT model?
• Any suggestions?