1. What is the motivation and methodology of this research?
1.a. Motivation
The motivation here is to show the utility and promise of my R package, epizootic, for building process-explicit models of disease outbreaks in wildlife. I’m using the Mycoplasma outbreak in house finches as a case study to show what we can learn from using epizootic to reconstruct the outbreak and narrow the prior distributions for aspects of the outbreak that are difficult to observe.
1.b. Methods
I simulated the house finch system from 1940 to 2016 (which was as late as my climate and land use data went.) I started in 1940 with the release of house finches in New York because of the scientific consensus that the invasion biology of the house finch is essential to understanding the outbreak. The disease is introduced to the system in Washington D.C. in early 1994, just how it happened in real life.
The simulations include the demography of the house finch, dispersal (both demography and dispersal include density dependence), the epidemiological dynamics of the disease, and seasonality. The prior distributions for these were drawn from the extensive literature on this system. The simulations also include changes in climate and land use, including the house finch’s preferences in climate and land use types, as determined by a species distribution model. The simulations have a daily step but data are only recorded on a seasonal basis.
I ran 10,000 simulations based on stratified sampling of the prior distributions. I collected a variety of data to validate the simulations, all non-overlapping with the data used to parameterize the simulations. In this document, I use these validation metrics to select top-performing simulations and make scientific inferences based on ensembles of top-performing models.
2. How well did the simulations perform?
Our validation metrics are as follows:
Are there finches in Washington DC in 1994? This is crucial because this is the time and place where the Mycoplasma gallisepticum outbreak first occurred. If there are no finches there, there can’t be an outbreak. All the subsequent metrics are conditioned on whether the simulation meets this criterion. All simulations that fail are excluded.
Mycoplasmapresence. There are locations where Project FeederWatch spotted infected finches every year for years on end. This score measures whether the simulations can replicate that persistence. A lower score indicates a better ability to replicate persistence.
Haemorhouspresence/absence. From many years of Breeding Bird Survey and Christmas Bird Count data, we know of places in North America where the house finch is always observed and never observed. A lower score means the simulation can accurately replicate the sites where house finches are always or never present.
Haemorhouspopulation trends. For bird conservation regions throughout North America, Audubon has records of population trends in Haemorhous population size. I used two metrics: population trend since 1970, and population trend since 1993. A lower score means that the population trends in the simulation are more accurate.
Point prevalence. I have data from studies done on Mycoplasma all over North America with prevalence of the disease that they found at a specific point in space and time, with uncertainty around these estimates. I assigned a penalty for the simulated prevalence falling outside this estimated range of prevalence.
First arrival of Mycoplasma. I estimated the first arrival of the Mycoplasma outbreak to different parts of the Northeast based on Project FeederWatch sightings, with uncertainty due to incomplete observation. I assigned a penalty for the simulated date of first arrival falling outside the estimated range.
First arrival of Haemorhous. I estimated the first arrival of the house finch in parts of its invasive range based on sightings from the Breeding Bird Survey, with uncertainty due to incomplete observation. I assigned a penalty for the simulated date of first arrival falling outside the estimated range.
2.a. Presence in Washington, D.C.
First of all, let’s look at the performance on the “presence of finches in DC” metric, since everything else depends on that.
Code
round1_priors |>mutate(dc = round1_metrics$dc) |>select(-mortality_Sa_summer, -mortality_Ra_summer) |>pivot_longer(1:45, names_to ="Prior", values_to ="Value") |>ggplot(aes(x = Value, group = dc, fill = dc)) +geom_density(alpha =0.5) +facet_wrap(~Prior, scales ="free") +scale_fill_paletteer_d("PrettyCols::Light", name ="Birds in DC?")
About half of the simulations (4876) succeeded on this metric. The main determinants of whether the simulations succeeded on this metric had to do with density-dependent controls on dispersal. The simulations with birds in DC maximized the range of population densities at which dispersal would occur.
2.b. Other Validation Metrics
Now, let’s look at the performance on the other seven validation metrics. What I’m looking for here are metrics that are good filters: not all the simulations succeed, but not all of them fail horribly, either. This helps us choose which simulations are the best.
Here I’ve added a blue line representing the top 1% of scores. This is tricky, because some of these metrics are poor filters: almost all simulations do well on them. Though perhaps they look better once extreme outliers are removed:
Code
plotlist2 <-pmap(list(round1_metrics_dc |>filter(if_all(everything(), \(x) x <quantile(x, 0.99, na.rm=T))), quantiles, labels), function(x, y, z) {ggplot(data =NULL, aes(x = x)) +geom_histogram() +geom_vline(xintercept = y, color ="blue") +xlab(z)})plot_grid(plotlist = plotlist2)
Mycoplasma presence remains a poor filter, as does presence and absence of house finches. Another issue is that the performance on finch presence/absence, Mycoplasma arrival, and finch arrival is very poor. I think the best thing I can do at this point is to try ABC on different combinations of validation metrics, choose top models, and check if:
the same top models are chosen by different validation metrics.
whether the ensemble of the top models chosen produces a result that I know to be accurate based on the natural history of the house finch and the conjunctivitis outbreak.
3. Which simulations scored the best on validation metrics?
You can’t reasonably use seven validation metrics at once. Generally, three is considered the ideal number. Choosing a good combination of three validation metrics can be very challenging.
3.a. Correlation
It is important to know which validation metrics are correlated, because you want three that are uncorrelated to one another for maximum effectiveness in choosing models that perform well at multiple functions.
Code
ggpairs(round1_metrics_dc |>filter(if_all(everything(), \(x) x <quantile(x, 0.99, na.rm=T))))
Point prevalence appears to be negatively correlated with Mycoplasma presence, positively correlated with finch presence/absence, positively correlated with population trend from 1993 on, and positively correlated with arrival of Mycoplasma. I will say I would strongly prefer point prevalence over Mycoplasma presence or finch presence/absence because it’s a better filter. Finch presence/absence is also highly correlated with population trend from 1993 on, and population trend from 1993 is a better filter also.
3.b. Point prevalence & finch population size trends
I want to try out the combination of point prevalence and the two population size trend metrics. I’m interested in these because they’re all good filters and performance is pretty good on all three. First, I run Approximate Bayesian Computation:
