• Introducing the Problem
• Data and Methods (Aims 1 and 2)
• Clustering Results (Aim 3A)
• Methods and Results of Racial/Ethnic Sensitivity Comparison (Aim 3B)
• Discussion

Slides here: https://rpubs.com/gabrielodom/CTNote_overview_RCMI

• Over 750,000 Americans have died from a drug overdose since 1990
• Since 2018, 2 in 3 drug overdose deaths have been from an opioid (almost 50,000 deaths)
• Nearly 1.3 million Americans are currently using one or more of the following medications to treat opioid use disorder (OUD): methadone, buprenorphine, and naltrexone
• The National Institute on Drug Abuse’s Clinical Trials Network have funded over 100 clinical trials and clinical trial supplemental studies to study the effects of new and existing medications to treat addiction

• Clinical trials assess the effectiveness of medication-based treatments for OUD (MOUD)
• A wide variety of treatment outcomes have been used in clinical trials since 1972 (Table 1 shows outcomes from 41 papers assessing MOUD)
• These endpoints all assess participant success or failure in treatment via the results from scheduled urine drug screenings (UDS) or urine opioid screenings (UOS)
• Accurately measuring the efficacy of MOUD for clinical trial participants is paramount, but no “gold standard” trial outcomes exist
• Moreover, some treatment outcomes may be needlessly sensitive to protected health attributes (such as race, ethnicity, age, and/or sex), which could exacerbate existing health disparities

• Prof. Brandt extracted the MOUD clinical trial outcomes from these 41 papers
• We then:
  • removed outcomes which could not be used to measure subject-specific outcomes (e.g. “proportion of subjects in treatment group with positive UDS in study week 3”)
  • removed outcomes which could not yield a univariate metric of subject success (e.g. “number of positive UDS per three-week window over time”)
  • split papers which assessed multiple outcomes (e.g. complete abstinence between weeks 5-12 AND total number of abstinent weeks)
  • combined papers which used identical outcomes
• In the end, we have 53 outcomes from registered clinical trials to consider, plus a few others of interest to the clinicians.

• Each row is an outcome used in a past clinical trial for MOUD
• As preliminary work, our clinical team (Profs. Brandt and Luo) grouped these by clinical practice into three classes: abstinence, relapse, and substance use reduction
• These classes were further split into subclasses based on their clinical interpretation
• Of note, these metrics also differed in how they considered a scheduled but incomplete UDS (i.e. a “missing” UDS)

• Question 1: When applied to the same clinical trials data, do these outcomes follow the same intuitive clusters (abstinence, relapse, use reduction)?
• Question 2: When applied to the same clinical trials data, and controlling for clinically relevant covariates, are some of these outcomes more sensitive to race/ethnicity than others?

• This is the first large-scale, empirical (data driven) comparison of MOUD clinical trial endpoints
• To date, this is the only standard and code-based library of treatment outcome definitions, which will be useful to all future substance use disorder clinical trials in NIDA’s Clinical Trials Network (CTN) and beyond
• To our knowledge, this is the first racial/ethnic sensitivity analysis of multiple clinical trial outcome definitions

• The CTN-0094 Project research team harmonized data from 3 large-scale MOUD clinical trials:
  • CTN-0027: “Starting reatment with Agonist Replacement Therapies (START)”
  • CTN-0030: “Prescription Opiate Abuse Treatment Study (POATS)”
  • CTN-0051: “Extended-Release Naltrexone vs. Buprenorphine for Opioid Treatment (X:BOT)”
• These three are nationally representative, prospective clinical trials with over 3600 combined participants
• These trials treated participants for 16-24 weeks, and collected rich baseline substance use data, demographics, and medical/psychiatric evaluations

• We have 3560 fully de-identified subjects with demographics and weekly UDS results for the months they were in treatment, 2492 of whom completed randomization to a treatment arm
• We coded race and ethnicity conjointly as “Non-Hispanic White” (\(n_W = 2484\)), “Non-Hispanic Black” (\(n_B = 347\)), “Hispanic” (\(n_H = 507\)), and “Other” (\(n_O = 222\))
• In order to summarize complex substance use patterns over the weeks of the multiple clinical trials:
  • all dates were standardized to the day of trial consent
  • the urinalysis results for each day/week were coded as a “word”
• Unfortunately, this data is currently embargoed by the CTN, but will be released in the R packages ctn0094data and ctn0094DataExtra

• In biology, complex chains of amino acids are often “collapsed” by having one letter represent a single acid (glycine as “G”, leucine as “L”, etc.).
• In medical informatics and patient charting systems, various complex patient states are often represented as a single letter or symbol (❤️ for cardiology, 💊 for pharmacy, 🧪 for labs, etc.).
• We borrowed from these ideas to develop a “word” to represent subject urine screen patterns for a substance (or group of substances) of interest over time
  • +: positive for the substance(s)
  • -: negative for the substance(s)
  • o: subject failed to provide a urine sample
  • *: inconclusive results or mixed results (e.g. subject provided more than one urine sample, and they did not agree)
  • _: no specimens required by the lab/clinic in that interval (weekends, holidays, pre-randomization period, alternating visit days/weeks)

## # A tibble: 3,560 x 5
##    Subject `Start Week` `Randomization Week` `Treatment End` `Treatment UDS`    
##      <int>        <dbl>                <dbl>           <dbl> <chr>              
##  1       1           -4                   NA              15 ooooooooooooooo    
##  2       2           -5                    1              15 *---oo-o-o-o+oo    
##  3       3           -6                    1              23 o-ooo-oooooooooooo~
##  4       4           -4                    1              24 ------------------~
##  5       5           -4                   NA              15 ooooooooooooooo    
##  6       6           -6                    1              15 *oooooooooooooo    
##  7       7           -4                    1              24 ----oooooooooooooo~
##  8       8           -4                   NA              25 oooooooooooooooooo~
##  9       9           -6                    1              22 oooooooooooooooooo~
## 10      10           -6                    1              22 ------+++-++++o+++~
## # ... with 3,550 more rows

• -o---o---o--o+---------- ( 163):
• -++++++++-+++----------- ( 210):
• ----------------------- ( 242):
• -------------------o-o-o ( 4):
• --++*++++++-++++++-+++- ( 17):
• ------------o-oooooooooo ( 13):
• +-+--o----------o-o-oo++o (1103):
• +----o----o---o-o-oo-o-o- ( 33):
• *+++++++++++o++++++++++o ( 233):
• ++++---+--------------o- (2089):

• -o---o---o--o+---------- ( 163): Success
• -++++++++-+++----------- ( 210): Success
• ----------------------- ( 242): Success
• -------------------o-o-o ( 4): Failure
• --++*++++++-++++++-+++- ( 17): Failure
• ------------o-oooooooooo ( 13): Failure
• +-+--o----------o-o-oo++o (1103): Failure
• +----o----o---o-o-oo-o-o- ( 33): Failure
• *+++++++++++o++++++++++o ( 233): Failure
• ++++---+--------------o- (2089): Failure

• -o---o---o--o+---------- ( 163): Success
• -++++++++-+++----------- ( 210): Success
• ----------------------- ( 242): Success
• -------------------o-o-o ( 4): Success
• --++*++++++-++++++-+++- ( 17): Failure
• ------------o-oooooooooo ( 13): Success
• +-+--o----------o-o-oo++o (1103): Success
• +----o----o---o-o-oo-o-o- ( 33): Failure
• *+++++++++++o++++++++++o ( 233): Failure
• ++++---+--------------o- (2089): Success

Once we had the patterns of substance use (opioid use in our case), we wrote a family of algorithms that could be combined to calculate any of the outcome definitions from Table 1 given a use pattern “word”. The CTNote:: package helper functions are:

• Handling missing data: recode_missing_visits() and impute_missing_visits()
• Accounting for the study observation design: collapse_lattice() and view_by_lattice()
• Detecting a use pattern: detect_subpattern() and detect_in_window()
• Measuring lengths of consecutive behavior: measure_retention() and measure_abstinence_period()
• Counting substance use events/proportions: count_matches()

We want computer code that can be read by a non-technical reviewer.

At least 8 weeks of consecutive abstinence

usePatterns_char %>% 
  detect_subpattern(subpattern = "--------")

Number of positive urine screens in the last 4 weeks (NA counts as positive)

usePatterns_char %>% 
  recode_missing_visits(missing_is = "o", missing_becomes = "+") %>% 
  count_matches(match_is = "+", start = -4, end = -1)

Using the CTNote:: package, we wrote code to calculate the treatment outcome for each subject according to each of the outcomes listed in Table 1:

## # A tibble: 2,492 x 67
##      who usePatternUDS cleanLast4Weeks cleanWeeks3to6 comer2006_red dropout_time
##    <dbl> <chr>         <lgl>           <lgl>                  <dbl>        <dbl>
##  1     2 *---oo-o-o-o~ FALSE           FALSE                 0.562        0.385 
##  2     3 o-ooo-oooooo~ FALSE           FALSE                 0.25         0.205 
##  3     4 ------------~ FALSE           TRUE                  1            0.615 
##  4     6 *ooooooooooo~ FALSE           FALSE                 0.0625       0.0769
##  5     7 ----oooooooo~ FALSE           FALSE                 0.5          0.154 
##  6     9 oooooooooooo~ FALSE           FALSE                 0            0.0256
##  7    10 ------+++-++~ FALSE           TRUE                  0.75         0.564 
##  8    11 -+-o+o+--o*-~ FALSE           FALSE                 0.375        0.615 
##  9    12 +-++-*o--o-+~ FALSE           FALSE                 0.438        0.615 
## 10    13 ------------~ FALSE           TRUE                  1            0.410 
## # ... with 2,482 more rows, and 61 more variables: earlyTreatReduction <lgl>,
## #   eissenberg1997_abs <lgl>, fiellin2006_abs <dbl>, fiellin2006_red <dbl>,
## #   fudala2003_red <dbl>, haight2019_red <dbl>, jaffe1972_red <dbl>,
## #   johnson1992_abs <lgl>, johnson1992_red <dbl>, kosten1993_abs <lgl>,
## #   kosten1993B_red <lgl>, krupitsky2004_abs <lgl>, krupitsky2011A_abs <lgl>,
## #   krupitsky2011B_abs <dbl>, lateTreatReduction <lgl>,
## #   lee2016_rel_event <dbl>, lee2016_rel_time <dbl>, ...

• Now that we have all outcomes calculated for all 2492 randomized subjects, we empirically clustered these outcomes to validate (or call in question) the pre-defined clusters of abstinence, relapse, and substance use reduction.
• We performed hierarchical clustering of the outcomes using \(k = 2, 3, \ldots, 15\) clusters, and we noted that using 3 clusters resulted in an “elbow” in the scree plot.
• To remove cluster sensitivity to a single subject, we repeated this clustering on 10,000 random bootstrap samples of the original data, and counted the number of times outcome \(i\) and outcome \(j\) ended up in the same cluster (for \(k = 2, 3, \ldots, 15\))

• We worked with our clinical team to inspect these clusters
• Overall, the clusters are characterized by:
  • Orange cluster (far right): the strictest measures of abstinence, and any missing urine screen is marked positive; these outcome definitions are incredibly sensitive to single values (note the high variability [large number of subclusters])
  • Teal cluster (middle): most other measures of abstinence, relapse, or use reduction, where any missing urine screen is also marked positive; these outcome definitions are not sensitive to single values
  • Pink cluster (far left): these are many different kids of outcomes, but they all share one thing: missing urine screen values are ignored
• In summary, use clinical trial outcomes similar to the ones in the middle cluster.

Consider two statistical models using the same predictors:

\[ \textbf{y} = a_0 + a_1 \textbf{x}_1 + \ldots + a_p \textbf{x}_p + \textbf{e} \qquad (1) \\ \textbf{z} = b_0 + b_1 \textbf{x}_1 + \ldots + b_p \textbf{x}_p + \boldsymbol{\epsilon} \qquad (2) \] Let \(\textbf{x}_1\) measure a composite of race and ethnicity, \(\textbf{y}\) represent outcome 1, and \(\textbf{z}\) represent outcome 2.

• With this setup, we can find a \(p\)-value to measure the statistical significance of race/ethnicity on outcomes 1 and 2 independently.
• However, we can only compare (1) and (2) if the distributions of \(\textbf{e}\) and \(\boldsymbol{\epsilon}\) come from the same family (both Normal, both Binomial, etc.).
• If the two outcomes share the same metric space, we could use this framework to state if outcome 1 is more sensitive to race/ethnicity than outcome 2 (or vice versa).

In order to ensure that the distribution of the model residuals shares the same family (so we can compare \(p\)-values / regression coefficients), we invert the linear models above:

\[ \textbf{x}_1 = a_0 + \textbf{a}_1\textbf{Y} + \ldots + a_p \textbf{x}_p + \textbf{e} \qquad (1^*) \\ \textbf{x}_1 = b_0 + \textbf{b}_1\textbf{Z} + \ldots + b_p \textbf{x}_p + \boldsymbol{\epsilon} \qquad (2^*) \] Notice the differences:

• we are no longer using race/ethnicity to predict the trial outcomes, but we are using the trial outcomes to predict race/ethnicity
• the distributions of \(\textbf{e}\) and \(\boldsymbol{\epsilon}\) are now both Multinomial
• because survival outcomes (2-dimensional) are included, \(\textbf{Y}\) and \(\textbf{Z}\) are now matrices of predictor information and \(\textbf{a}_1\) and \(\textbf{b}_1\) are vectors of regression coefficients

Now that we can build comparable multinomial regression models (with NHW as the reference group), we compare them as follows:

1. Create a “null” model (no outcomes at all); fit \((1^*)\) without \(\textbf{a}_1\textbf{Y}\). This includes covariates for age, sex, treatment medication, diagnoses of mental illness, use of other (non-opioid) illicit substances, clinical trial, and clinical trial site.
2. Compare model \((1^*)\) to the null model using a Likelihood Ratio Test and the Akiake Information Criterion.
3. Compare model \((1^*)\) to \((2^*)\) using the Akiake Information Criterion.

Because we are fitting nearly 60 regression models, we adjust the model \(p\)-values using a false discovery rate (FDR) correction.

• Notice that nearly all the regression coefficients (column “beta”) are negative, meaning that these outcomes are uniformally worse for non-whites
• There are 43 treatment outcomes which do not add significant amounts of predictive information about race/ethnicity to the baseline (null) models (FDR \(\ge\) 0.05)
  • 22 had FDR < 0.05
• There were 36 treatment outcomes in the middle cluster (that we identified had nice statistical properties).
• There are comparatively few outcomes which satisfy both constraints (Table 2)

• We created a novel substance use event coding syntax
• We developed an open-source software package (CTNote) to calculate a wide variety of substance use pattern summary statistics
• We wrote a library of algorithms to calculate published clinical trial treatment outcomes
• We empirically clustered those outcomes using real data, showing that proper outcome metrics:
  • can come from abstinence, relapse, or substance use reduction measures, but
  • cannot ignore missing urine screens or depend too heavily on the urine screen from a single visit
• We compared all outcomes on the basis of sensitivity to participant race/ethnicity and found that some clinical trial outcomes are much too sensitive to patient demographics

• What other important states should be included in the substance use pattern syntax? (Recall that we already include positive [+], negative [-], missing [o], mixed/inconclusive [*], and not assessed [_])
• How can we build on this analysis to construct better clinical trial outcomes for use in future MOUD clinical trials?
• When interpreting racial/ethnic sensitivity in outcomes, does significance mean that the outcome itself is poor, or that the outcome’s structure highlights a clinical trial design feature which itself adds undue trial participation burden to minorities?
  • I.e., is ignoring these “sensitive” trial outcomes doing more harm than good?

• Funding from NIDA UG1DA013035-17 and NIMHD FIU-RCMI Pilot AWD000000009108
• Thank you to Profs. Feaster and Luo and the entire CTN-0094 team
• Thank you to FIU’s RCMI for grantsmanship training