Asymmetric predictability

Overview

• Causal investigations in observational studies pose a great challenge.
• We develop an information geometric causal discovery and inference framework of “predictive asymmetry”.

For \((X, Y)\), predictive asymmetry enables assessment of whether \(X\) is more likely to cause \(Y\) or vice-versa. The asymmetry between cause and effect becomes particularly simple if \(X\) and \(Y\) are deterministically related.

• We propose a new metric called the Directed Mutual Information (\(DMI\)) and establish its key statistical properties.
• \(DMI\) is not only able to detect complex non-linear association patterns, but also is able to detect and infer causal relations.

Motivating problem

Even under simplifying assumptions of no confounding, no feedback loops, and no selection bias, directly assessing bivariate causal relationship is a notoriously hard problem Spirtes and Zhang (2016).

The Directed Mutual Information (DMI)

Mutual information (MI)

• Consider \(U_X = F_X(X)\) and \(U_Y = F_Y(Y)\), where \(F_X\) and \(F_Y\) are the CDF of \(X\) and \(Y\) respectively.

• By Sklar’s theorem (Sklar (1959)), we have \[c_{X Y}\left(u_X, u_Y\right)=\frac{f_{X Y}\left\{F_X^{-1}\left(u_X\right), F_Y^{-1}\left(u_Y\right)\right\}}{f_X\left\{F_X^{-1}\left(u_X\right)\right\} f_Y\left\{F_Y^{-1}\left(u_Y\right)\right\}}, \] so we can re-write \(MI(X, Y) = MI(U_X, U_Y) = E\left\{ \log c(U_X, U_Y) \right\}\), where \(c(U_X, U_Y)\) is the unique copula density function.

• Computational burden of estimation methods based on previous understanding of \(MI\) is greatly reduced (Ma and Sun (2008)).

Purkayastha and Song (2022b) “fastMI: a fast and consistent copula-based estimator of mutual information.”

The Directed Mutual Information (DMI)

Mutual information (MI)

The Directed Mutual Information (DMI)

Entropy and entropy ratio (ER)

• Marginal entropy of \(X\) is defined as \(H(X) = E_X\left\{ - \log f_X(X) \right\}\).
• \(H(X, Y) = E_{XY} \left\{- \log f_{XY}(X, Y) \right\}\) is the joint entropy of \((X, Y)\).
• Conditional entropy of \(X\) conditioned on \(Y\) is given by \(H(X|Y) = H(X, Y) - H(Y)\).

The Directed Mutual Information (DMI)

Entropy ratio

• \(ER(X|Y) = ER(Y|X) = 1/2\) if and only if \(H(X|Y) = H(Y|X)\)
• \(ER(X|Y) \lessgtr 1/2\) implies \(H(X|Y) \lessgtr H(Y|X)\), with \(H(Y|X) < H(X|Y)\) establishing \(X\) as the ‘’dominant predictor variable’’ that exerts more ‘’influence’’ on \(Y\).

\(ER(X|Y) < 1/2\) establishes dominance of \(X\) while \(ER(X|Y) > 1/2\) establishes dominance of \(Y\).

The \(ER\) has a nice transitive property:

• If we assume \(X\) is more predictive than \(Y\), i.e., \(ER(X|Y) > 1/2\) .
• And assume \(Y\) is more predictive than \(Z\), i.e., \(ER(Y|Z) > 1/2\).
• A little algebra yields \(ER(X|Z) > 1/2\).

An ordering of \(ER\) between \((X, Y)\) and \((Y, Z)\) provides insight on asymmetric predictability in a third pair, i.e., \((X, Z)\) within our proposed framework.

The Directed Mutual Information (DMI)

DMI can detect dependence between two random variables and also test for asymmetric predictability between \(X\) and \(Y\). \[\begin{equation} \label{eq:dmi} DMI\left(X\middle| Y\right) = MI(X, Y) \times ER(X|Y). \end{equation}\]

\(DMI(X|Y)\) is a scaled asymmetric function of \(MI\), which is an effective tool to capture symmetric association. The scaling factor \(ER(X|Y)\) aims to capture asymmetric behaviour between \(X\) and \(Y\) by comparing their relative predictive performance when they are associated.

1. \(DMI\left(X\middle| Y\right) \geq 0\) and \(DMI\left(Y\middle| X\right) \geq 0\), with equality if and only if \(X\) and \(Y\) are independent.
2. \(DMI\left(X\middle| Y\right) \lessgtr DMI\left(Y\middle| X\right)\) implies \(ER\left(X\middle| Y\right) \lessgtr ER\left(Y\middle| X\right)\).

Estimating DMI

• Estimating densities:

  • Using advances made by O’Brien et al. (2016), we estimate the underlying joint and marginal densities using Fast-Fourier transformations.
  • This is many magnitudes faster than classical bandwidth-based density estimation, while maintaining comparable error performance, making our method scalable.
  • Estimated densities are used to obtain estimates of \(DMI\) and \(\Delta\).

• Estimating \(DMI\) and \(\Delta\): We use a data-splitting technique for estimation and valid inference, shown by the flowchart below

Theoretical guarantees for valid inference with \(DMI\)

1. Consistency: Assuming the density functions are bounded below and above, with \(\min(n_1, n_2) \rightarrow \infty\), we get consistent estimates of \(DMI(X|Y)\) and \(DMI(Y|X)\).

2. Asymptotic normality: Assuming true \(MI \neq 0\), with \(\min(n_1, n_2) \rightarrow \infty\), we have \[\begin{equation} \sqrt{n_2}(\widehat{DMI}(X|Y) - DMI(X|Y)) \rightarrow N(0, \sigma_1^2)\\ \sqrt{n_2}(\hat{\Delta} - \Delta) \rightarrow N(0, \sigma_2^2) \end{equation}\] where both \(\sigma_1\) and \(\sigma_2\) can be estimated by Monte Carlo tools (Robert (2010)).

Simulation study: testing independence

• In each pattern, the signal parameter \(a\) determines the strength of association between \(X\) and \(Y\) with \(a = 0\) denoting independence.

• Upon increasing \(a\), we increase the signal strength of \(X\) in \(Y\) relative to the independent noise \(\epsilon\), implying a departure from independence (i.e. the null case

Simulation study: testing independence

• In each pattern, the signal parameter \(a\) determines the strength of association between \(X\) and \(Y\) with \(a = 0\) denoting independence.

• Upon increasing \(a\), we increase the signal strength of \(X\) in \(Y\) relative to the independent noise \(\epsilon\), implying a departure from independence (i.e. the null case

Real data application: ELEMENT study.

• \(H_0: DMI = 0\) is test for independence.
  • We perform a permutation-based test of independence for all 21 methylation sites with Systolic and Diastolic BP (SBP/DBP).
  • p-values obtained from each gene are mildly correlated.
  • So we aggregate p-values using the Cauchy combination test Liu and Xie (2019).

Real data application: ELEMENT study.

With most studies on the association between DNAm and blood pressure (Han et al. (2016)) still in their infancy, these findings present evidence that DNAm alteration could be influenced by blood pressure and call for further investigation.

Purkayastha and Song (2022a) “Asymmetric predictability in causal discovery: an information theoretic approach”

In summary…

• Asymmetry is an inherent property of bivariate associations and therefore must not be ignored.

• We present a new causal discovery framework of asymmetric predictability between two random variables \(X\) and \(Y\) using \(DMI\).

• The \(DMI\) is inspired by Shannon’s seminal work on information theory: may simultaneously be used to test for association as well as detect and quantify asymmetry.

• A computationally fast and robust Fourier transformation-based method is used to estimate the \(DMI\) instead of conventional kernel-based methods.

• Moreover, our method consistently performs faster than existing bandwidth-dependent methods - being approximately 4 orders of magnitude faster for bivariate sample sizes of approximately \(10^4\).

• \(DMI\) methodology relies on data-splitting inference, enjoys key large-sample properties necessary for valid inference.

• In absence of knowledge, our framework may serve either as a discovery or confirmatory tool.