Simpson’s Paradox
Simpson’s Paradox reveals one of the extreme cases of what confounding variables can cause. In particular, it states that an association between two variables, which was identified for the whole (combined) dataset can be reversed (direction) when considering disaggregated populations (Judea Pearl, 2014).
That sounds pretty strange, right? This example will illustrate how this can happen, and why it is not as counter-intuitive as it seems:
Do you remember Walita? Walita is a guy who is constantly worried about helping his community.
Figure 1 - Walita
In this opportunity, Walita was worried about some students of the Master of Data Science and Innovation (MDSI) at the University of Technology Sydney (UTS), who are taking the Statistical Thinking for Data Science (STDS) subject. These students were feeling discouraged because when comparing to the rest of the class, their performance appeared to be lower.
While looking for strategies to improve, they decided to investigate how increasing their study time would affect their performance. To achieve this, they took a dataset1 containing records of 2019-Autumn class, which included final marks and the students’ average daily time spent studying for STDS.
The following graph shows what they found:
Figure 2 - Initial analysis
As surprised as the STDS’ students were? The relationship between study hours and performance is almost null. To be precise, it looks like there is a weak negative association. After getting these results, the STDS’ students felt even more discouraged. Apparently, even if they wanted to invest more hours studying it would not help them with their performance.
Walita did not like to see them sad, so he decided to investigate more… that conclusion of extra study not paying off seemed counter-intuitive anyway.
Therefore, he started to reflect on it and concluded that there might be an underlying association with a third variable (confounding variable) that they were not taking into account. After revising the available features in the dataset, he found a factor with the students’ background in statistics.
The following graph shows the results Walita got after deciding to include Statistics.Background to differentiate three subpopulations:
Figure 3 - Walita’s analysis
Whooo! That makes more sense. Not only the initial negative association between study time and performance dissapeared, but it showed up being strong and positive: Simpson’s Paradox.
As Walita is very intelligent, he drew an even more important conclusion for his friends: it is misleading to compare one’s performance to the class’ average. Background clearly conditions the relationship between study time and performance, and performance for students with professional studies in statistics are expected to be higher at any daily study time level.
So which was Walita’s advice for his friends taking STDS? Do not compare your performance to others’ as each learning process is unique and valuable! And if for any reason you have to… then at least consider which would be a fairer benchmark (class’ average definitively is not the best option).
Looks like that STDS’ students are feeling so much better now. Moreover, after confirming a positive relationship, they started to felt more encouraged to keep working hard and invest more time studying.
Good work Walita! You are so kind.