Confidence intervals are a convenient way to communicate uncertainty in estimates. Paired observations are usually analyzed using the t interval.

About the t distribution.

The t interval technically assumes that the data are iid normal, though it is robust to this assumption. It works well whenever the distribution of the data is roughly symmetric and mound shaped. Paired observations are often analyzed using the t interval by taking differences.

Compare the t and z distributions: t and z distributions

For large degrees of freedom, t quantiles become the same as standard normal quantiles; therefore this interval converges to the same interval as the CLT yielded.

For skewed distributions, the spirit of the t interval assumptions are violated. Also, for skewed distributions, it doesn’t make a lot of sense to center the interval at the mean. In this case, consider taking logs or using a different summary like the median. For highly discrete data, like binary, other intervals are available.

data(sleep)
x1 <- sleep$extra[sleep$group == 1]
x2 <- sleep$extra[sleep$group == 2]
n1 <- length(x1)
n2 <- length(x2)
sp <- sqrt( ((n1 - 1) * sd(x1)^2 + (n2-1) * sd(x2)^2) / (n1 + n2-2))
md <- mean(x1) - mean(x2)
semd <- sp * sqrt(1 / n1 + 1/n2)
md + c(-1, 1) * qt(.975, n1 + n2 - 2) * semd
## [1] -3.363874  0.203874
t.test(x1, x2, paired = FALSE, var.equal = TRUE)$conf
## [1] -3.363874  0.203874
## attr(,"conf.level")
## [1] 0.95
t.test(x1, x2, paired = TRUE)$conf
## [1] -2.4598858 -0.7001142
## attr(,"conf.level")
## [1] 0.95

Acknowledging the pairing explains variation that would otherwise be absorbed into the variation for the group means. As a result, treating the groups as independent results in wider intervals. Even if it didn’t result in a shorter interval, the paired interval would be correct as the groups are not statistically independent!

Paired Groups:

g1 <- sleep$extra[1 : 10]; g2 <- sleep$extra[11 : 20]
difference <- g2 - g1
mn <- mean(difference); s <- sd(difference); n <- 10

mn + c(-1, 1) * qt(.975, n-1) * s / sqrt(n)
## [1] 0.7001142 2.4598858
t.test(difference)
## 
##  One Sample t-test
## 
## data:  difference
## t = 4.0621, df = 9, p-value = 0.002833
## alternative hypothesis: true mean is not equal to 0
## 95 percent confidence interval:
##  0.7001142 2.4598858
## sample estimates:
## mean of x 
##      1.58
t.test(g2, g1, paired = TRUE)
## 
##  Paired t-test
## 
## data:  g2 and g1
## t = 4.0621, df = 9, p-value = 0.002833
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
##  0.7001142 2.4598858
## sample estimates:
## mean of the differences 
##                    1.58
t.test(extra ~ I(relevel(group, 2)), paired = TRUE, data = sleep)
## 
##  Paired t-test
## 
## data:  extra by I(relevel(group, 2))
## t = 4.0621, df = 9, p-value = 0.002833
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
##  0.7001142 2.4598858
## sample estimates:
## mean of the differences 
##                    1.58
t.test(extra ~ I(relevel(group, 2)), paired = FALSE, data = sleep)
## 
##  Welch Two Sample t-test
## 
## data:  extra by I(relevel(group, 2))
## t = 1.8608, df = 17.776, p-value = 0.07939
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
##  -0.2054832  3.3654832
## sample estimates:
## mean in group 2 mean in group 1 
##            2.33            0.75

Independent groups:

A the pooled variance estimate weights the mean variance of each group if you do not have equal numbers of observations in both groups, as long as there is a constant variance across each group.

Based on Rosner, Fundamentals of Biostatistics

sp <- sqrt((7 * 15.34^2 + 20 * 18.23^2) / (8 + 21 - 2))
132.86 - 127.44 + c(-1, 1) * qt(.975, 27) * sp * (1 / 8 + 1 / 21)^.5
## [1] -9.521097 20.361097
## 
## Attaching package: 'dplyr'
## The following objects are masked from 'package:stats':
## 
##     filter, lag
## The following objects are masked from 'package:base':
## 
##     intersect, setdiff, setequal, union
## Warning: package 'plotly' was built under R version 4.0.3
## Loading required package: ggplot2
## 
## Attaching package: 'plotly'
## The following object is masked from 'package:ggplot2':
## 
##     last_plot
## The following object is masked from 'package:stats':
## 
##     filter
## The following object is masked from 'package:graphics':
## 
##     layout
## Warning: `group_by_()` is deprecated as of dplyr 0.7.0.
## Please use `group_by()` instead.
## See vignette('programming') for more help
## This warning is displayed once every 8 hours.
## Call `lifecycle::last_warnings()` to see where this warning was generated.

A plot of the raw data suggest that Diet 1 is less nutritious. Taking the difference in weight as weight gain we see:

# Library
library(ggplot2)

# Most basic violin chart
p <- ggplot(wideCW, aes(x=gain, y=Diet, fill=Diet)) + # fill=name allow to automatically dedicate a color for each group
  geom_violin()

p
## Warning: Removed 5 rows containing non-finite values (stat_ydensity).

It does not seem that there is equal variance between Diet 1 & Diet 4, we can use the t interval to confirm this.

wideCW14 <- subset(wideCW, Diet %in% c(1, 4))
rbind(
t.test(gain ~ Diet, paired = FALSE, var.equal = TRUE, data = wideCW14)$conf,
t.test(gain ~ Diet, paired = FALSE, var.equal = FALSE, data = wideCW14)$conf
)
##           [,1]      [,2]
## [1,] -108.1468 -14.81154
## [2,] -104.6590 -18.29932

The first [1,] case assumes the the intervals are equal the second [2,] that they are not. In both cases the intervals are entirely below zero, suggesting less weight gain on Diet 1 than on Diet 4 (at 95% confidence). The specific interval does change depending on whether or not you have the variances are equal or the variances are not equal, as a rule of thumb when in doubt set var.equal = FALSE in a t test.

For binomial data, there are many ways to compare two groups - Relative risk, risk difference, odds ratio. - Chi-squared tests, normal approximations, exact tests.

For count data, there’s also Chi-squared tests and exact tests.