Github repository Paradigm ##Introduction
This paper by Semrow et al. (2019) studies how membership in different identity categories interact. In this case the researchers find that Asian Americans who are gay are perceived as more American than straight Asian Americans. My primary research is on categories and membership of organizations and individuals with regards to various identity groups and the status attained by membership in different groups. This work interests me inor that it displays how membership in one identity group (LGBTQ) influences membership in another group (American), and is useful to my field in that membership in either group affects individual’s status and labor market outcomes.
I anticipate that my greatest challenge will be offering appropriate characterizations of the individual and of both of the countries, without introducing some confounding aspect of the two countries.
So from the original paper we the estimated effect size between the two groups of interest (varying sexuality of target) was .79, standard deviations were within each group 1.08 (less accepting country) and .45. Utilizing t-tests for equality and \(\alpha=0.05\) at a power level of .8 we would need 19 in each group, at .9 power we would need 25 per group, and at .95 power we would need 30 per group.
However I would reccomend even larger sample sizes since our population being sampled from (M-Turkers) differs from the population selected from in the original study (undergraduate students), since it may be possible that our sample will have greater variance in responses.
This is a 2 by 2 study. A total sample of 50 is the planned size (25 per condition). Each participant will receive two conditions: countries will vary, but sexuality of the person in question will change. The desired sample will be Americans. Although the original study participants were University of Washington students.
The materials as follows will be followed as precisely as possible. Orignal Materials
“Participants [will be] presented with a hypothetical country either named Boden or Thamen. They were told either that gay people are less welcome and accepted in Boden/Thamen than in the United States, or that they are equally welcome and accepted in Boden/Thamen and the United States. Then, they read about a gay man (“X is a gay man”) or straight man (“X is a straight man”). Participants rated the target’s American identity on three questions: “How likely is it that this person is Bodenian/Thamenian or American?,” “How Bodenian/Thamenian or American is this person?,” and “To what extent do you believe this person identifies as Bodenian/Thamenian or American?” on a scale from 1 (very [likely] Bodenian/Thamenian) to 7 (very [likely] American)…[Then] They read about another target with the same sexual orientation (“Y is a gay/straight man”) [but changed country and homophobia condition] and answered the same three questions about his American identity.”
This will be done using a Qualtrics survey with mTurk particpants. Where on one page a country is presented, then on the next page the person is presented along with some questions (then repeated for next country condition).
There seems to be some confusion (from reading the paper) as to whether the person in the survey is described as Asian-American or not. In this study (4), they did not; but in other studies (e.g. study 1) the person in question was described more specifically as “a gay Asian American man.” I have contacted the researchers for further comment.
Additionally results may be confounded by the order of the country conditions. So it will be important to randomize the presentation of the country conditions. Additionally participants may quickly learn about the purpose of the experiment upon being exposed to the second condition. Country names may also confound, so it may be important to randomize association of each country name to homophobia conditions of each country. Further confounds may also be introduced by the name of the target person. It may not be reasonable to keep the same name; so two names (“John” and “Alex”) may be used and counterbalanced too.
The analysis plan is to conduct simple Anova/regression coefficient differences for each of the four conditions and/or t tests of the mean differences per the conditions. It is a simple difference in means. I will exclude participants who failed manipulation checks (either for country homophobia or the person’s sexuality.)
The primary difference will be in the sample. This sample will consist of American mechanical turkers rather than undergraduates at a major American research univeristy. This may make a notable difference as undergraduate students at the University of Washington should be expected to have different views of other countries, ethnicity, and sexuality than the broader American population. If necessary I may later restrict to young Americans.
Sample size, demographics, data exclusions based on rules spelled out in analysis plan
Any differences from what was described as the original plan, or “none”.
For data preparation it is first of all key to exclude particpants who fail the manipulation checks. As respondents at the end of the survey, in this study, will be asked to indicate how much more or less accepting a country was compared to the United States. Those who fail the manipulation checks (e.g those indicating that the homophobic country is more accepting will be excluded, as will those failing to correctly identify the sexuality of the focal person they read about).
####Load Relevant Libraries and Functions
library(tidyverse)## ── Attaching packages ────────────────────────────────── tidyverse 1.2.1 ──## ✔ ggplot2 3.0.0 ✔ purrr 0.3.2
## ✔ tibble 2.1.1 ✔ dplyr 0.8.0.1
## ✔ tidyr 0.8.3 ✔ stringr 1.4.0
## ✔ readr 1.3.1 ✔ forcats 0.4.0## Warning: package 'ggplot2' was built under R version 3.4.4## Warning: package 'tibble' was built under R version 3.4.4## Warning: package 'tidyr' was built under R version 3.4.4## Warning: package 'readr' was built under R version 3.4.4## Warning: package 'purrr' was built under R version 3.4.4## Warning: package 'dplyr' was built under R version 3.4.4## Warning: package 'stringr' was built under R version 3.4.4## Warning: package 'forcats' was built under R version 3.4.4## ── Conflicts ───────────────────────────────────── tidyverse_conflicts() ──
## ✖ dplyr::filter() masks stats::filter()
## ✖ dplyr::lag() masks stats::lag()####Import data
df<-read.csv("Semrow Et Al_October 27, 2019_17.37.csv")
#drop first two rows fo random info #only drop third here because of bad first row of data
df<-df[-c(1,2,3),]
#get mean of dependendent variables from three questions asked for each country
df$Yb <- (as.numeric(df$Identity_B)+as.numeric(df$Like_B)+as.numeric(df$How_B))/3
df$Yt <- (as.numeric(df$Identity_T)+as.numeric(df$Like_T)+as.numeric(df$How_T))/3
#### Data exclusion / filtering
#check manipulations and subset data
DFSexualityManipulatedTrue <- df[df$Countrycondition==df$manipulationcheck1]
#recode as numeric, make a 1 as gay and 2 as straight
df$TargetGay<-abs(as.numeric(df$TargetGay)/2-3)
##check country homophobia
#first figure out which country is homophobic
df$T_Homophobic<-0
df$B_Homophobic<-0
df$B_Homophobic[df$Country1Homophobia=='True' & df$Country1=='Boden']<-1
df$B_Homophobic[df$Country2Homophobia=='True' & df$Country2=='Boden']<-1
df$T_Homophobic[df$Country1Homophobia=='True' & df$Country1=='Thamen']<-1
df$T_Homophobic[df$Country2Homophobia=='True' & df$Country2=='Thamen']<-1
#get homophobic response and non homophobic response
df$Y_Homophobic<-df$T_Homophobic*df$Yt+df$B_Homophobic*df$Yb
df$Y_Equal<-(1-df$T_Homophobic)*df$Yt+(1-df$B_Homophobic)*df$Yb
#get dataset where manipulations pass for the country checks
#check passed if answer is less than 3 for less accepting/homophobic condition, and 4 otherwise
df$PassedCountryCheckT <-1
df$PassedCountryCheckB <-1
#set to fail if you rated homophobic countries as more equally accepting
df$PassedCountryCheckT[as.numeric(df$T_Homophobic)*as.numeric(df$Manip_T_Check)>=4]<-0
df$PassedCountryCheckB[as.numeric(df$B_Homophobic)*as.numeric(df$Manip_B_Check)>=4]<-0
#set to fail if you rated equal countries anything but equal
df$PassedCountryCheckB[(1-as.numeric(df$B_Homophobic))*(as.numeric(df$Manip_B_Check)-4)!=0]<-0
df$PassedCountryCheckT[(1-as.numeric(df$T_Homophobic))*(as.numeric(df$Manip_T_Check)-4)!=0]<-0
#see if you passed the homophobic and accepting Country Checks
df$PassedCountryCheckHomophobic<-df$PassedCountryCheckB*df$B_Homophobic+df$PassedCountryCheckT*df$T_Homophobic
df$PassedCountryCheckEqual<-df$PassedCountryCheckB*(1-df$B_Homophobic)+df$PassedCountryCheckT*(1-df$T_Homophobic)
#get dataset where manipulations pass for the sexuality checks
#recode so that one is gay, 0 is straight
df$PassedSexualityCheckB <-0
df$PassedSexualityCheckB[df$TargetGay==df$Sexuality_Check_B] <-1
df$PassedSexualityCheckT <-0
df$PassedSexualityCheckT[df$TargetGay==df$Sexuality_Check_T] <-1
df$PassedSexCheckHomophobic<-df$PassedSexualityCheckB*df$B_Homophobic+df$PassedSexualityCheckT*df$T_Homophobic
df$PassedSexCheckEqual<-df$PassedSexualityCheckB*(1-df$B_Homophobic)+df$PassedSexualityCheckT*(1-df$T_Homophobic)
#construct variable to notify if either the less accpeting or more accepting condition had a failed check
df$Homophobic_fail<-0
df$Equal_fail<-0
df$Homophobic_fail[df$PassedSexCheckHomophobic+df$PassedCountryCheckHomophobic<2]<-1
df$Equal_fail[df$PassedSexCheckEqual+df$PassedCountryCheckEqual<2]<-1
##keep only the outputs for now, response ID, and gay condition, and passing checks
#Can redo later with those who passed manipulation checks.
#recode as numeric, make a 1 as gay and 2 as straight
df$TargetGay<-abs(as.numeric(df$TargetGay)-2)
df<-(df %>%
select(TargetGay, Y_Homophobic, Y_Equal,Equal_fail,Homophobic_fail) )
#we may construct datasets of each dropping failures for later mean comparisons
dfEqualPasses<-df[df$Equal_fail==0,]
dfHomophobicPasses<-df[df$Homophobic_fail==0,]
#and then specify to if gay or not
dfEqualPassesGay<-dfEqualPasses[dfEqualPasses$TargetGay==1,]
dfHomophobicPassesGay<-dfHomophobicPasses[dfHomophobicPasses$TargetGay==1,]
dfEqualPassesStraight<-dfEqualPasses[dfEqualPasses$TargetGay==0,]
dfHomophobicPassesStraight<-dfHomophobicPasses[dfHomophobicPasses$TargetGay==0,]#now we may get the means
Means <- df %>%
group_by( TargetGay) %>%
summarise(Homophobic = mean(Y_Homophobic),Equal = mean(Y_Equal))
Means## # A tibble: 2 x 3
## TargetGay Homophobic Equal
## <dbl> <dbl> <dbl>
## 1 0 2 3
## 2 1 4.22 2.67##Now conduct a t test of the difference in perceptions of american-ness by group for all data
t.test((df$Y_Homophobic-df$Y_Equal)~df$TargetGay,alternative = "two.sided")##
## Welch Two Sample t-test
##
## data: (df$Y_Homophobic - df$Y_Equal) by df$TargetGay
## t = -3.2857, df = 1.7392, p-value = 0.09792
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
## -6.432038 1.320927
## sample estimates:
## mean in group 0 mean in group 1
## -1.000000 1.555556#t tests for non failures
t.test(dfEqualPassesGay$Y_Equal, dfHomophobicPassesGay$Y_Homophobic, alternative = "two.sided")##
## Welch Two Sample t-test
##
## data: dfEqualPassesGay$Y_Equal and dfHomophobicPassesGay$Y_Homophobic
## t = -4.0249, df = 1.4706, p-value = 0.09171
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
## -3.8061743 0.8061743
## sample estimates:
## mean of x mean of y
## 2.5 4.0Any follow-up analyses desired (not required).
Open the discussion section with a paragraph summarizing the primary result from the confirmatory analysis and the assessment of whether it replicated, partially replicated, or failed to replicate the original result.
Add open-ended commentary (if any) reflecting (a) insights from follow-up exploratory analysis, (b) assessment of the meaning of the replication (or not) - e.g., for a failure to replicate, are the differences between original and present study ones that definitely, plausibly, or are unlikely to have been moderators of the result, and (c) discussion of any objections or challenges raised by the current and original authors about the replication attempt. None of these need to be long.