2 CHANGING MINDS ON GAY MARRIAGE In this exercise, we analyze the data from two experiments in which households were canvassed for support on gay marriage.6 Note that the original study was later retracted due to allegations of fabricated data; we will revisit this issue in a follow-up exercise (see section 3.9.1). In this exercise, however, we analyze the original data while ignoring the allegations. Canvassers were given a script leading to conversations that averaged about twenty minutes. A distinctive feature of this study is that gay and straight canvassers were randomly assigned to households, and canvassers revealed whether they were straight or gay in the course of the conversation. The experiment aims to test the “contact hypothesis,” which contends that out-group hostility (towards gay people in this case) diminishes when people from different groups interact with one another. The data file is gay.csv, which is a CSV file. Table 2.7 presents the names and descriptions of the variables in this data set. Each observation of this data set is a respondent giving a response to a four-point survey item on same-sex marriage. There are two different studies in this data set, involving interviews during seven different time periods (i.e., seven waves). In both studies, the first wave consists of the interview before the canvassing treatment occurs.
Import the data (gay.csv)
# Option 1: use read.csv command if you have the data file saved on your computer
# gay <- read.csv(file.choose(), header=TRUE)
# Option 2: use the RCurl package to scrape the data of from the following URL - https://raw.githubusercontent.com/kosukeimai/qss/master/CAUSALITY/gay.csv
library(RCurl)## Warning: package 'RCurl' was built under R version 3.3.3## Loading required package: bitopsx <- getURL("https://raw.githubusercontent.com/kosukeimai/qss/master/CAUSALITY/gay.csv")
gay <- read.csv(text = x)Explore the dataset)
head(gay)## study treatment wave ssm
## 1 1 No Contact 3 5
## 2 1 No Contact 4 5
## 3 1 No Contact 1 5
## 4 1 No Contact 6 5
## 5 1 No Contact 2 5
## 6 1 No Contact 7 51. Using the baseline interview wave before the treatment is administered, examine whether randomization was properly conducted. Base your analysis on the three groups of study 1: “same-sex marriage script by gay canvasser,” “same-sex marriage script by straight canvasser” and “no contact.” Briefly comment on the results.
Using the baseline interview wave, examine whether randomization was conducted properly. Base your analysis on the three groups of Study 1: “Same-sex marriage script by gay canvasser”, “Same-sex marriage script by straight canvasser”, “No contact”.
Subset the data by wave, study and treatment
## Baseline interview wave (1) for first study (1)
nc.w1 <- subset(gay, subset = treatment=="No Contact" & study==1 & wave==1)
ssm.gc.w1 <- subset(gay, subset = treatment=="Same-Sex Marriage Script by Gay Canvasser" & study==1 & wave==1)
ssm.sc.w1 <- subset(gay, subset = treatment=="Same-Sex Marriage Script by Straight Canvasser" & study==1 & wave==1)Briefly comment on the results.
## Outcome is ssm
mean(nc.w1$ssm)## [1] 3.042764mean(ssm.gc.w1$ssm)## [1] 3.025195mean(ssm.sc.w1$ssm)## [1] 3.09971# Those are pretty similar2. The second wave of the survey was implemented two months after canvassing. Using study 1, estimate the average treatment effects of gay and straight canvassers on support for same-sex marriage, separately. Give a brief interpretation of the results.
Second wave (2)
nc.w2 <- subset(gay, subset = treatment=="No Contact" & study==1 & wave==2)
ssm.gc.w2 <- subset(gay, subset = treatment=="Same-Sex Marriage Script by Gay Canvasser" & study==1 & wave==2)
ssm.sc.w2 <- subset(gay, subset = treatment=="Same-Sex Marriage Script by Straight Canvasser" & study==1 & wave==2)# Outcome is ssm
mean(nc.w2$ssm)## [1] 3.039615mean(ssm.gc.w2$ssm)## [1] 3.139489mean(ssm.sc.w2$ssm)## [1] 3.161863mean(ssm.gc.w2$ssm) - mean(nc.w2$ssm)## [1] 0.09987463mean(ssm.sc.w2$ssm) - mean(nc.w2$ssm)## [1] 0.122248Slightly bigger difference when straight canvasser than with gay canvasser - both are positive
Example answer uses SD of wave 1 to look at relative effect size
## sd of baseline support
w1 <- subset(gay, wave == 1)
sd(w1$ssm)## [1] 1.6816153. The study contained another treatment that involves contact, but does not involve using the gay marriage script. Specifically, the authors used a script to encourage people to recycle. What is the purpose of this treatment? Using study 1 and wave 2, compare outcomes from the treatment “same-sex marriage script by gay canvasser” to “recycling script by gay canvasser.” Repeat the same for straight canvassers, comparing the treatment “same-sex marriage script by straight canvasser” to “recycling script by straight canvasser.” What do these comparisons reveal? Give a substantive interpretation of the results.
Placebo
Use Study 1 and wave 2
rec.gc.w2 <- subset(gay, subset = treatment=="Recycling Script by Gay Canvasser" & study==1 & wave==2)
rec.sc.w2 <- subset(gay, subset = treatment=="Recycling Script by Straight Canvasser" & study==1 & wave==2)Average Treatment Effect (ATE) - see page 49
mean(ssm.gc.w2$ssm) - mean(rec.gc.w2$ssm) # Compare recycling script with same sex marriage script (both gay canvasser)## [1] 0.03204239mean(ssm.sc.w2$ssm) - mean(rec.sc.w2$ssm) # Compare recycling script with same sex marriage script (both straight canvasser)## [1] 0.1575845We notice bigger difference in attitude towards same sex marriage for straight canvassers than gay canvassers
4. In study 1, the authors reinterviewed the respondents six different times (in waves 2 to 7) after treatment, at two-month intervals. The last interview, in wave 7, occurs one year after treatment. Do the effects of canvassing last? If so, under what conditions? Answer these questions by separately computing the average effects of straight and gay canvassers with the same-sex marriage script for each of the subsequent waves (relative to the control condition).
Remove the wave specification - all waves
nc <- subset(gay, subset = treatment=="No Contact" & study==1)
ssm.gc <- subset(gay, subset = treatment=="Same-Sex Marriage Script by Gay Canvasser" & study==1)
ssm.sc <- subset(gay, subset = treatment=="Same-Sex Marriage Script by Straight Canvasser" & study==1)Now use tapply to compare waves
tapply(nc$ssm, nc$wave, mean) # No Contact## 1 2 3 4 5 6 7
## 3.042764 3.039615 3.046273 3.035946 3.174272 3.091973 3.313747tapply(ssm.gc$ssm, ssm.gc$wave, mean) # Gay Canvasser## 1 2 3 4 5 6 7
## 3.025195 3.139489 3.127639 3.128571 3.322167 3.178408 3.373116tapply(ssm.sc$ssm, ssm.sc$wave, mean) # Straight Canvasser## 1 2 3 4 5 6 7
## 3.099710 3.161863 3.106681 3.123142 3.272827 3.155488 3.271210Now look at ATE
tapply(ssm.gc$ssm, ssm.gc$wave, mean) - tapply(nc$ssm, nc$wave, mean) # Gay Canvasser vs. No Contact## 1 2 3 4 5 6
## -0.01756893 0.09987463 0.08136612 0.09262577 0.14789543 0.08643548
## 7
## 0.05936835tapply(ssm.sc$ssm, ssm.sc$wave, mean) - tapply(nc$ssm, nc$wave, mean) # Straight Canvasser vs. No Contact## 1 2 3 4 5 6
## 0.05694517 0.12224797 0.06040800 0.08719659 0.09855523 0.06351524
## 7
## -0.04253721We notice a bigger effect of gay canvasser over time
_5. The researchers conducted a second study to replicate the core results of the first study. In this study, same-sex marriage scripts are given only by gay canvassers. For study 2, use the treatments “same-sex marriage script by gay canvasser” and “no contact” to examine whether randomization was appropriately conducted. Use the baseline support from wave 1 for this analysis.__
Study 2 - Just gay canvassers
Baseline interview wave (1) for second study (2)
nc.s2.w1 <- subset(gay, subset = treatment=="No Contact" & study==2 & wave==1)
ssm.gc.s2.w1 <- subset(gay, subset = treatment=="Same-Sex Marriage Script by Gay Canvasser" & study==2 & wave==1)
mean(ssm.gc.s2.w1$ssm)## [1] 2.971729mean(nc.s2.w1$ssm)## [1] 2.970075We see that the means are very similar - suggesting pre-trial randomisation is good
6. For study 2, estimate the treatment effects of gay canvassing using data from wave 2. Are the results consistent with those of study 1?
Second wave (2) for second study (2)
nc.s2.w2 <- subset(gay, subset = treatment=="No Contact" & study==2 & wave==2)
ssm.gc.s2.w2 <- subset(gay, subset = treatment=="Same-Sex Marriage Script by Gay Canvasser" & study==2 & wave==2)
mean(ssm.gc.s2.w2$ssm)## [1] 3.116194mean(nc.s2.w2$ssm)## [1] 2.9923Check with first study - see below ()
mean(ssm.gc.w2$ssm)## [1] 3.139489mean(nc.w2$ssm)## [1] 3.039615Very similar - the gay canvassers ellicit higher responses
7. Using study 2, estimate the average effect of gay canvassing at each subsequent wave and observe how it changes over time. Note that study 2 did not have a fifth or sixth wave, but the seventh wave occurred one year after treatment, as in study 1. Draw an overall conclusion from both study 1 and study 2.
nc.s2 <- subset(gay, subset = treatment=="No Contact" & study==2)
ssm.gc.s2 <- subset(gay, subset = treatment=="Same-Sex Marriage Script by Gay Canvasser" & study==2)Now look at ATE
tapply(ssm.gc.s2$ssm, ssm.gc.s2$wave, mean) - tapply(nc.s2$ssm, nc.s2$wave, mean) # Gay Canvasser vs. No Contact## 1 2 3 4 7
## 0.001653782 0.123893673 0.150420627 0.125173822 0.307062649We do notice really big effects - increasing over time