AMR modeling org

• AIM (née Charity Enterpeneurship) had done a mid-depth cost-effectiveness review of a modeling org for AMR they did not ultimately recommend
• One approach they considered was a low-effort, low-expense modeling org that could offer policy guidance
• This was a mid-skill, low-effort attempt to fill in the details by
  • Checking on the available data
  • Investigate candidate methods
• Open Phil is interested in funding this kind of social science research.

Macro Approach

Causal inference

• Allows you to consider the counterfactual by estimating what would have happened to country if they had implemented a different policy or no policy
• Compares the observed outcomes with the potential outcomes that did not actually occur but can be inferred from the data
• Persuasive to policy makers

For example in AMR

• The policy might be veterinarian dispensed antibiotics, the outcome might be antibiotic resistance
• But here the mechanism of the policy might be complex: varies by supply of antibiotics, vet staff, and animals.
  
• And here the data may be partially observed: do AMR infections in animals correspond to those in hospitals?

Better light

• Better to find a lower-bound for feasibility.
• If no sufficient data or method exist to make estimates, the low-effort route is closed.
• Focus on a standard policy (bacterial vaccine introduction)
• With a simple mechanism (infection suppression)
• And a simple outcome (fewer infections)

Bonus: translate this directly to DALYs.

Summary

• Good news! A method that was developed in the last few years is well-adapted to this situation, extracting most of the available detectable signal.
• The limitation is probably rather a matter of readily available
  • administrative data
  • clinical data
• This leaves us with a data scavenger hunt

Intuition: Problem

• From what we know about an outcome today, we want to guess about tomorrow, depending on a choice.
• In particular, we want to know the difference between, say, a binary yes no choice.
  
• But
  • we can’t tell whether the change from today was due to the choice or something else ongoing.
  • we don’t get to live life twice, to see what would have happened otherwise.

Intuition: Solution

• For our choice, suppose we chose yes.
  • call us the “treated unit”
• Today, we look to others with a similar outcome but chose no
  • actually, similar yesterday and before that too.
  • suppose our outcome is about the average of those similar
  • call them “units” in a “candidate donor pool,” and they get equal “weights”
• Tomorrow, we look at the outcomes:
  • take the average again of the donor units choosing “no”
  • call that the “synthetic control”
  • measure the difference between our “yes” and the synthetic control “no”.
    • call that the “ATT” (average treatment of the treated)

Method history

• The idea of matching goes back to Rubins in the 1980s
• Simple version of synthetic control (Abadie and Gardeazabal (2003))
• Enhancements (Abadie, Diamond, and Hainmueller (2010), Doudchenko and Imbens (n.d.))
• Staggered Introduction (Ben-Michael, Feller, and Rothstein (n.d.))
• Best Practices (Abadie (2021))

Intervention Data: WHO vaccine introduction

Outcome Data: ATLAS

• Program Name: Antimicrobial Testing Leadership and Surveillance (ATLAS)
• Initiated by: Pfizer
• Start Year: 2004
• Components: Integrates data from three surveillance programs (TEST, AWARE, INFORM)
• Coverage:
  • Over 760 sites
  • 73 countries
  • 556,000 bacterial isolates
  • 21,000 fungal isolates

ATLAS

Sample Data for method

The software package comes with a dataset that is helpful to benchmark against

Ours: WHO Data

Here we make a choice of when to say the intervention began. The choice is debatable.

Ours: ALTAS

Superimposing the previous choice on the data we have.

Corrections

A documented merge error Catalán et al. (2022), interventions out of the range of outcome data (two observations pre-intervention) and one outlier (Spain).

Fix

Takeaways

• At the start of this project, we thought the effects, if any, would probably be obvious to the eye.
• Effects were not obvious in the sample data, which is somewhat reassuring.
• However, it necessitates other ways of convincing ourselves of a strong effect.

Model

Call:
multisynth(form = n_inf ~ intro, unit = iso_3_code, time = year, 
    data = atlas_analysis)

Average ATT Estimate (Std. Error): -6.802  (14.145)

Global L2 Imbalance: 10.864
Scaled Global L2 Imbalance: 0.464
Percent improvement from uniform global weights: 53.6

Individual L2 Imbalance: 36.810
Scaled Individual L2 Imbalance: 0.752
Percent improvement from uniform individual weights: 24.8   

 Time Since Treatment   Level     Estimate Std.Error lower_bound upper_bound
                    0 Average  -2.92205234  7.995995   -16.68697   14.344963
                    1 Average   2.61805003 12.581574   -22.48560   26.519495
                    2 Average  -0.03792328 13.916355   -28.92870   22.648658
                    3 Average -11.63278694 19.563825   -52.31162   22.070152
                    4 Average  -5.78210725 16.211319   -38.84965   22.410052
                    5 Average  -7.03640675 15.389944   -42.09704   20.128359
                    6 Average -26.90066174 19.664089   -68.24072    9.721004
                    7 Average  -6.05286766 19.267488   -48.74474   26.410116
                    8 Average  -1.34693967 24.204239   -56.69547   36.341045
                    9 Average  -8.92605471 19.831700   -52.79578   23.848750

Synthetic Controls

Here the difference between observed counts and average of “synthetic” countries where the intervention did not occur.

Average Effect

Interpretation

• Direction is the one we want, and increasing over time! Good sanity check.
• In most cases, the only a couple of donors per control, possibly poor fit
  • though sparsity of donors is not uncommon
  • in the famous Basque study introducing the method, only 3.
• Overall effect is moderate (-6.802), but very noisy (14.145)
  • compare our data to the sample data
• We are not using covariates yet that could improve estimates
• We are working with pretty poor outcome data
• Effect varies, and even inverted, in some counties.
• Effect is in terms of infections averted per year per country for the number of patients collected

Connection to DALY

• Effect is in terms of infections averted per year per country for the numbers of patients collected
• We have to relate this to hospital utilization and overall population
• We have good numbers for population per period, and in Europe, decent numbers for utilization
  • Some periods are especially spotty
  • Finding better numbers, and validating them, a future task
• We have a preliminary model of DALYs per infection averted.

Example

Picking here a country where observed counts (with intervention) are lower than average synthetic control (without invention).

Assumptions

Suppose, conservatively

• 10 hospitals sampled (avg. sites/country of ATLAS)
• 161 hospitals (2021 estimate for Hungary)
• About 100 people to 1 hospital discharge (10M pop / ~ 75k patient)
• Hospitals capture a 1/3 of cases
• From ATLAS metadata, most Hungary are infections from hospitals

Results

Takeaways

• There may well be enough signal in existing data (MIC) to estimate the effect of gross interventions “from the outside”
• Modelling finer and more complex interventions remains significantly more challenging
• Noise in the outcome data is the major hurdle, the methods are there

Caveats

So many!

• This is a prototype,
• Mostly a demo of how to wrangle the data,
• The model has not been investigated throughly
• Mostly to see if data is well-suited to it

Future directions

• Pull in data on other MIC data collection efforts
• Add in fixed and time-varying covariates and test effect on estimates
• Deeper investigation of current model
• Calculation of DALYs.

Lessons

• Working with development-stage statistical packages is a real hurdle.
• Data cleaning always takes twice as long as you think, even after you doubled.