POLISCI 3325G - Data Science for Political Science

Randomized Experiments

In this laboratory, we will: (1) create contingency tables, (2) perform an ANOVA test and (3) perform a t-test.

Relevant functions: table(), prop.table(), aov(), TukeyHSD(), t.test().

1. Importing Data

We start by importing a dataset encompassing data from a (fake) survey experiment testing whether exposure to positive or negative vignettes about the state of the Canadian economy impacts satisfaction with the federal government and a feeling thermometer for current the prime minister of the country.

Type of vignette (IV) \(\rightarrow\) Political attitudes (DV)

In order to test this relationship, we randomly assign 1,500 participants in three experimental treatment conditions: “positive” for the group exposed to the positive vignette, “negative” for the group exposed to the negative vignette, and “control” for the group exposed to no vignette at all.

Because the treatment condition is randomly attributed, we assume that the difference we will observe across the scores of the three groups is only explained by the experimental treatment. That’s a property of random assignment: if your sample size is large enough, the disparities on observed and unobserved covariates should be close to zero across all groups. Because of this property, if we observe a statistically significant difference in attitudes for each of the treatment conditions, we can make a causal inference about the effect of these vignettes.

# Set work directory
setwd(dirname(rstudioapi::getSourceEditorContext()$path))

# Loading data
expData <- read.csv("expData.csv")

# Priting out the column names
colnames(expData)

## [1] "id"           "condition"    "satisfaction" "feelTherm"

Here is a description of the different variables within this dataset:

# Priting out the first 10 observations
head(expData,10)

##    id condition satisfaction feelTherm
## 1   1  Negative Dissatisfied  37.62840
## 2   2  Negative      Neutral  29.77305
## 3   3  Negative Dissatisfied  47.29787
## 4   4  Positive    Satisfied  57.72648
## 5   5  Negative Dissatisfied  33.80386
## 6   6  Positive    Satisfied  58.74346
## 7   7   Control Dissatisfied  56.68478
## 8   8   Control    Satisfied  41.79225
## 9   9   Control    Satisfied  63.78172
## 10 10   Control    Satisfied  49.48885

2. Contingency Tables

The first thing we want to do is to display the distribution of our data in a table. Here, we’ll display the relationship between treatment condition and satisfaction with the current government, which are two categorical variables.

2.1 Raw Data

We use the table() function to display the distribution of the raw data. As a convention, we usually display the IV as the rows and the DV as the columns.

# Creating a table 
table(expData$condition,expData$satisfaction)

##           
##            Dissatisfied Neutral Satisfied
##   Control           132     198       170
##   Negative          262     145        93
##   Positive          111     205       184

2.2 Percentaging

We can also display the percentages of observations in each cell, in each row (percentaging across) or in each column (percentaging down) using the prop.table() function .

# Creating a table with overall percentages
prop.table(table(expData$condition,expData$satisfaction))

##           
##            Dissatisfied    Neutral  Satisfied
##   Control    0.08800000 0.13200000 0.11333333
##   Negative   0.17466667 0.09666667 0.06200000
##   Positive   0.07400000 0.13666667 0.12266667

# Creating a table with percentages by rows
prop.table(table(expData$condition,expData$satisfaction),1)

##           
##            Dissatisfied Neutral Satisfied
##   Control         0.264   0.396     0.340
##   Negative        0.524   0.290     0.186
##   Positive        0.222   0.410     0.368

# Creating a table with percentages by columns
prop.table(table(expData$condition,expData$satisfaction),2)

##           
##            Dissatisfied   Neutral Satisfied
##   Control     0.2613861 0.3613139 0.3803132
##   Negative    0.5188119 0.2645985 0.2080537
##   Positive    0.2198020 0.3740876 0.4116331

Just by looking at these tables does it seem like there is a relationship between the treatment condition and satisfaction with the federal government?

3. ANOVA

When working with a categorical explanatory variable (IV) and a continuous dependent variable (DV), you can use an ANOVA (analysis of variance) test. In this case, this test allows us to measure the strength of the relationship between the treatment condition variable (categorical) and the feeling thermometer variable (numerical).

We use the aov() function to peform this test.

# Running an ANOVA test
model <- aov(feelTherm ~ condition, expData) 

# Viewing our results using summary()
summary(model)

##               Df Sum Sq Mean Sq F value Pr(>F)    
## condition      2 104744   52372   495.5 <2e-16 ***
## Residuals   1497 158211     106                   
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Looks like the relationship between the treatment condition and the feeling thermometer towards the PM is statistically significant. But wait: what is the direction of the relationship for each treatment condition?

We cannot know this by looking simply at the ANOVA. Why? Because the ANOVA just tells you if the three groups (or however many treatment groups you have) are significantly different regarding your DV. It might be that the positive treatment had a statistically significant effect on the feeling thermometer, but not the negative one. It might also be that they both had an effect. We just don’t know at this stage.

This is why we use the Tukey’s Honest Significant Difference Test. This allows us to see which pairs differ significantly from each other with regard to the DV.

# Checking whether the difference is sign. across all pairs of IVs
TukeyHSD(model)

##   Tukey multiple comparisons of means
##     95% family-wise confidence level
## 
## Fit: aov(formula = feelTherm ~ condition, data = expData)
## 
## $condition
##                         diff        lwr        upr   p adj
## Negative-Control  -15.967048 -17.492409 -14.441686 0.0e+00
## Positive-Control    3.108001   1.582639   4.633362 5.7e-06
## Positive-Negative  19.075048  17.549687  20.600410 0.0e+00

Here is how to interpret these results:

• the negative group has feeling thermometers that are lower than the control group by -15.967048 on average, and this difference is statistically significant (p \(<\) 0.001).

• the positive group has feeling thermometers that are higher than the control group by 3.108001 on average, and this difference is also statistically significant (p = 0.0000057).

• the positive group has feeling thermometers that are higher than the negative group by 19.075048 on average, and this difference is also statistically significant (p \(<\) 0.001).

4. T-Test

To demonstrate how to perform a t-test, let’s take a short pause from this dataset and imagine a fictive scenario in which two groups of 10 people are either randomly shown an advertisement for the Conservative Party of Canada (treatment), or not (control). Here, let’s say we are interested in the ratings for that party on a scale of 1 to 10 (1 being disliking that party a lot, and 10 being liking that party a lot).

SawAd <- c(6,7,5,8,6,5,4,7,6,6)
DidntSeeAd <- c(3,5,6,4,3,5,6,5,3,5) 

SawAd 

##  [1] 6 7 5 8 6 5 4 7 6 6

DidntSeeAd

##  [1] 3 5 6 4 3 5 6 5 3 5

How do we know if the difference between the results of both groups is statistically significant? We can run a t-test, which attributes a p-value to the difference between the means of both sets of results. That is to say it tells us the probability of observing such a difference (or even more extreme) between the scores of both groups, just by chance alone. At this stage, let’s just say that if the p-value is smaller than 0.05, the difference in means is statistically significant. We’ll come back to this with a class solely on p-values soon.

# Performing a  t-test
t.test(SawAd,DidntSeeAd)

## 
##  Welch Two Sample t-test
## 
## data:  SawAd and DidntSeeAd
## t = 2.8749, df = 17.993, p-value = 0.01008
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
##  0.4038129 2.5961871
## sample estimates:
## mean of x mean of y 
##       6.0       4.5

Voilà! You are now able to analyse experimental results in R.

Exercises