Simpson’s Paradox

This is an R Quarto document giving an example of Simpson’s Paradox. Looking first at correlation and second at the estimated slopes in regression lines.

The example considers book prices and the number of pages in books. The confounding variable to consider is type of book, either hardcover or paperback.

And some examples of using ggplot2 are included.

Here are some links to get started:

• wikipedia about correlation
• a post on the Analysis with Programming website about the Theory of Linear Least Squares, which gives an introduction to the linear regression and how it is fitted to data.
  
• wikipedia about Simpson’s Paradox

The data.

### synthetic data

# Consider book price (y) by number of pages (x)

z <- c("hardcover","hardcover",
      "hardcover","hardcover",
      "paperback", "paperback","paperback", 
      "paperback")

x1 <- c( 150, 225, 342, 185)
y1 <- c( 27.43, 48.76, 50.25, 32.01 )

x2 <- c( 475, 834, 1020, 790)
y2 <- c( 10.00, 15.73, 20.00, 17.89 )

x <- c(x1, x2)
y <- c(y1, y2)

Plotting a scatterplot.

plot(x,y)

Compute the correlation coefficient \(r\) to see the effect of Simpson’s Paradox. The overall correlation is negative and the correlations, taking into consideration the confounding variable, type of book, the correlations change to positive, for both hardcover and paperback books.

# correlation

cor(y, x)

[1] -0.5949366

cor(y1, x1)

[1] 0.8481439

cor(y2, x2)

[1] 0.9559518

Now compute the slopes in the linear regressions to see the same effect.

Overall, the slope is negative.

# linear regression

lm(y ~ x)

Call:
lm(formula = y ~ x)

Coefficients:
(Intercept)            x  
   41.15238     -0.02665  

For the hardcover books, the slope is positive.

# linear regression

lm(y1 ~ x1)

Call:
lm(formula = y1 ~ x1)

Coefficients:
(Intercept)           x1  
    13.0613       0.1177  

For the paperback books, the slope is positive.

# linear regression

lm(y2 ~ x2)

Call:
lm(formula = y2 ~ x2)

Coefficients:
(Intercept)           x2  
    1.72389      0.01819  

Now let’s visualize Simpon’s Paradox in the scatterplot. First by plotting the names of each point in the scatterplot.

plot(x,y, type="n")
text(x,y,z, cex=0.8)

Now plot the linear regression model ignoring the type of book.

# fitted linear regression

plot(x,y)

model0 <- lm(y ~ x)
abline(model0)

Now plot the linear regression models using the type of book.

# fitted linear regression

plot(x,y)

text(200,40,"hardcover")
model1 <- lm(y1 ~ x1)
abline(model1)

text(600,15,"paperback")
model2 <- lm(y2 ~ x2)
abline(model2)

Now using ggplot2. A very nice tutorial is available from Cookbook R.

library(tidyverse)

── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ──
✔ dplyr     1.2.1     ✔ readr     2.2.0
✔ forcats   1.0.1     ✔ stringr   1.6.0
✔ ggplot2   4.0.3     ✔ tibble    3.3.1
✔ lubridate 1.9.5     ✔ tidyr     1.3.2
✔ purrr     1.2.2     
── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
✖ dplyr::filter() masks stats::filter()
✖ dplyr::lag()    masks stats::lag()
ℹ Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errors

# put the data into a data.frame
#   cond = type of book
#   xvar = x
#   yvar = y

dat <- tibble(cond = z, xvar = x, yvar = y)
dat

Plot the overall data.

dat |> ggplot(aes(x = xvar, y = yvar)) + geom_point(shape = 1)      # Use hollow circles

dat |> ggplot(aes(x = xvar, y = yvar)) +
    geom_point(shape = 1) +    # Use hollow circles
    geom_smooth(method = lm)   # Add linear regression line 

`geom_smooth()` using formula = 'y ~ x'

                               #  (by default includes 95% confidence region)

dat |> ggplot(aes(x = xvar, y = yvar)) +
    geom_point(shap = 1) +    # Use hollow circles
    geom_smooth(method = lm,   # Add linear regression line
                se = FALSE)    # Don't add shaded confidence region

Warning in geom_point(shap = 1): Ignoring unknown parameters: `shap`

`geom_smooth()` using formula = 'y ~ x'

Now plot the regression lines for each type. See Simpson’s Paradox.

dat |> ggplot(aes(x = xvar, y = yvar, color = cond)) + 
    geom_point(shape = 1)

# Same, but with different colors and add regression lines
dat |> ggplot(aes(x = xvar, y = yvar, color = cond)) +
    geom_point(shape = 1) +
    scale_colour_hue(l = 50) + # Use a slightly darker palette than normal
    geom_smooth(method = lm,   # Add linear regression lines
                se = FALSE)    # Don't add shaded confidence region

`geom_smooth()` using formula = 'y ~ x'

Plot using different shapes.

# Set shape by cond
dat |> ggplot(aes(x = xvar, y = yvar, shape = cond)) + geom_point()

# Same, but with different shapes
dat |> ggplot(aes( x= xvar, y = yvar, shape = cond)) + geom_point() +
    scale_shape_manual(values = c(1,2))  # Use a hollow circle and triangle