The importance of exchangeablity in evaluating treatments
James C Meyer DVM MSc
7/17/2022
Retrospective Studies
In veterinary medicine we frequently are presented with a study that retrieved medical records and followed outcomes in an attempt to evaluate the comparative effectiveness of two or more treatments(interventions). There are many challenges in making the leap from, horses treated with x had a better outcome than horses treated with q , to the cause of improved outcomes was due to treatment x. With retrospective studies missing data is invariably a problem as well as measurement error. Both can lead to significant bias. Additionally immortal time bias will occur if animals in a cohort study cannot experience the outcome during some period of follow-up time. That is their assignment (eg.surgery or no surgery) is based on information acquired after the animal enters the study (time zero). In this brief review the importance of the exchangeability assumption for causal inference in retrospective studies of equine interventions will be explored.
Randomization
The gold standard for determining whether a treatment(intervention) is causally related to an outcome is the randomized controlled trial (RCT):
- Randomly dividing individual animals (or clusters) into treatment and control groups before the action or intervention takes place makes groups of units comparable
- After an action is applied to one group, the other group provides the counterfactual or the best prediction
- In causal terminology they call this exchangeability: the groups are so similar that they could be exchanged; it does not matter which group receives the intervention… the result will be the same
In an observational study the exchangeability assumption is generally violated since the treatment assignment is not random. A representative example study Outcome after medical and surgical intervention in horses with temporohyoid osteoarthropathy, May 2017 Equine Veterinary Journal 49(6) will be used since it provides the study dataset in a Supplement.
Results: A total of 77 horses were identified as having THO during the period 1990-2014. Of these, 25 horses underwent ceratohyoid ostectomy (CHO) and eight underwent partial stylohyoid ostectomy (PSHO). Thirteen of 20, one of 25 and one of eight horses treated by medical therapy, CHO and PSHO, respectively, died or were subjected to euthanasia as a consequence of THO. Compared with CHO, medical therapy was significantly associated with nonsurvival, but there were no significant differences in survival between horses undergoing PSHO and medical therapy.
A summary of collected variables is seen in below.
summary(THO)## treatment Age ...3 Radiographic grade
## Length:54 Min. : 3.00 Length:54 Min. :0.000
## Class :character 1st Qu.: 8.00 Class :character 1st Qu.:1.000
## Mode :character Median :12.00 Mode :character Median :2.000
## Mean :13.38 Mean :1.909
## 3rd Qu.:18.00 3rd Qu.:2.000
## Max. :26.00 Max. :3.000
## NA's :1 NA's :10
## Endoscopic grade Neurologic grade Onset Days
## Min. :1.000 Min. :0.000 Length:54 Min. : 1.00
## 1st Qu.:1.500 1st Qu.:1.000 Class :character 1st Qu.: 3.50
## Median :2.000 Median :2.000 Mode :character Median : 15.50
## Mean :2.047 Mean :1.887 Mean : 73.94
## 3rd Qu.:3.000 3rd Qu.:3.000 3rd Qu.: 59.00
## Max. :3.000 Max. :3.000 Max. :720.00
## NA's :11 NA's :1 NA's :4
## Corneal ulcers Return to work Outcome
## Length:54 Length:54 Length:54
## Class :character Class :character Class :character
## Mode :character Mode :character Mode :character
##
##
##
##
## Follow-up time (months)
## Min. : 0.10
## 1st Qu.: 6.00
## Median : 24.00
## Mean : 43.43
## 3rd Qu.: 72.00
## Max. :144.00
## NA's :1Treatment assignment to surgical or medical treatment was not random. Treatment or confounding by indication is the result. The decision on how to treat a horse is related to veterinary and owner preferences which may be related to the severity of the disease, the age of the animal, the value of the animal, the animal’s use/purpose and many other potential variables known and unknown. Confounding by indication means that the decision on which treatment to institute is influenced by a covariate(s) which also influences the outcome measure. 
A potential confounder by indication in the study of interest is Age. Horse owners may decide how aggressively or whether to treat their horse based upon its age and the outcome of interest survival is also related to age. In addition unobserved confounding variables (U) may be important. 
Using data from the Supplement and generating boxplots of x=treatment, y=Age (Yrs) demonstrates that Age was not exchangeable.
`
boxplot(Age~treatment,data=THO,col="light blue")
The assumption of exchangeability of those treated medically and those treated surgically – or, in general, of those horses receiving different levels of exposure – often gets most of the attention in discussions about causal inference. The preference for randomized experiments over observational studies is because exchangeability is expected by design in the former. In contrast, investigators conducting observational studies need to use their expert knowledge to identify and measure many potential confounders. It is hoped that sufficient data is collected to achieve exchangeability conditional on the measured covariates. However, investigators can never be confident that they have succeeded, even if they have succeeded. Exchangeability cannot be empirically tested in observational studies.
The task of identifying a set of adjustment variables may be daunting. Adjusting for too many variables can introduce bias or increase the variance to unacceptable levels; adjusting for too few variables will leave uncontrolled confounding. Uncontrolled confounding obliterates any hope of discerning a causal association between treatment and outcome. The practice of data driven variable selection prevalent in the veterinary literature is not adequate for the resolution of these issues.