Assume we have 3 independent random variables distributed as follows

\[ \begin{equation} \begin{split} X_1 \sim \mathcal{N}(2.5, 1^2); \quad X_2 \sim \mathcal{N}(0, 0.5^2); \quad X_3 \sim \mathcal{N}(-0.5, 0.8^2); \end{split} \end{equation} \]

We are interested in \(\pi_{j,k} = \mathsf{Pr}(X_j > X_k) = \mathsf{Pr}(X_j - X_k > 0)\) for all \(k\). In the present scenario we know what the distributions of the differences between the various pairwise comparisons are. For example, for \(X_1 - X_2\) and \(X_1 - X_3\), we have

\[ \begin{equation} \begin{split} X_1 - X_2 &\sim \mathcal{N}(2.5, 1^2 + 0.5^2) \\ X_1 - X_3 &\sim \mathcal{N}(3.0, 1^2 + 0.8^2) \\ \end{split} \end{equation} \]

Now, say if we had to make a decision on the basis of \(\pi_{j,k} = \mathsf{Pr}(X_j > X_k)\) for all \(k\). That is, we want to pick the best \(X_j\) compared to all the rest. We would be willing to conclude that \(X_j\) is the best if the probabilities of all of the pairwise comparisons were very high. So, let’s say that we define very high as a probability in excess of \(\epsilon = 0.98\). Now, if \(\mathsf{Pr}(X_j > X_k) > \epsilon\) for all \(k \ne j\) then we conclude that \(X_j\) represents a superior strategy. Similarly, we might also be interested in whether the probability that \(X_j\) is the best is very low and in that case we would want to label \(X_j\) as an inferior strategy.

OK, well clearly, in this scenario, we can compute \(\mathsf{Pr}(X_j > X_k)\) directly.

epsilon <- 0.98
pnorm(0, 2.5, sqrt(1^2 + 0.5^2), lower.tail = FALSE)
pnorm(0, 3.0, sqrt(1^2 + 0.8^2), lower.tail = FALSE)
## [1] 0.9873263
## [1] 0.9904252

Both of these are greater than \(\epsilon\) which leads us to conclude that \(X_1\) is superior. That is, it is better than both \(X_2\) and \(X_3\) with very high probability. However, we might wonder whether this implies that \(X_2\) and \(X_3\) are both inferior to \(X_1\). Well, we know that

\[ \begin{equation} \begin{split} X_2 - X_1 &\sim \mathcal{N}(-2.5, 0.5^2 + 1^2) \\ X_3 - X_1 &\sim \mathcal{N}(-3.0, 0.8^2 + 1^2) \end{split} \end{equation} \]

and so

pnorm(0, -2.5, sqrt(0.5^2 + 1^2), lower.tail = FALSE)
pnorm(0, -3.0, sqrt(0.8^2 + 1^2), lower.tail = FALSE)
## [1] 0.01267366
## [1] 0.009574786

which indicates that both have a low probability of the equality \(X_j > X_k\) being true.

It’s common to try and get some internal consistency with regards to the decisions. It happens to be the case that we can when going from inferiority to superiority, but not the other way around.

For example, if we use an inferiority threshold of \(\zeta = (1 - \epsilon) / (|\mathcal{A}| - 1)\) where \(|\mathcal{A}| = 3\) is the number of variables considered then with \(\epsilon = 0.98\) we have an inferiority threshold of

(1 - epsilon) / (3 - 1)
## [1] 0.01

Given this threshold, we find that \(X_3\), but not \(X_2\) are inferior. So, one arm being superior does not imply that the remaining arms are inferior.

However, if we momentarily update the superiority threshold to \(\epsilon = 0.97\) then the inferiority threshold is \(\zeta = 0.015\) and now both \(X_3\), and \(X_2\) are inferior and \(X_1\) is superior. Moreover, this fact generally holds; specifically, if all the variables bar one are inferior, then that remaining arm must be superior.