Replication of Support for Resettling Refugees: The Role of Fixed Versus Growth Mind-Sets by Madan et. al. (2019, Psychological Science)
Ethan Roy ethanroy@stanford.edu
December 11, 2019
Introduction
For this project, I plan to replicate Madan et. al.’s 2019 study, which explores the connection between a fixed versus growth mindset and how those mindsets impact an individual’s openness to supporting refugees in their country. Though my research interests align more closely with the field of educational neuroscience, I am intrigued by the impact a growth mindset can have on student’s learning. By replicating this study, I will be able to reflect more on the impact of mindset (growth or fixed) on an individuals openness to new people, cultures, and ways of thinking.
Study Description
The study being replicated explored the effect of mindset on an individual’s support for refugees in their country. In a series of six experiments, the original authors examined the mindsets (growth or fixed) of individuals in the United States and United Kingdom and their views on if refugees are able to assimilate into a new culture and to what extent are refugees obligated to assimilate to a new society. I plan to replicate the first study, which leverages survey data to ascertain a participant’s support for refugees based on their opinions on the type of person someone is, mindset regarding intelligence (fixed vs growth), and political leanings.
Stimuli and Procedures
Participants will be recruited through Amazon Mechanical Turk. Per the original study, participants will report some demographic information (gender, age, race). Each participant will have to undergo a pre-screening to ensure that they meet two criteria:
- are a United States Citizen
- currently residing in the United States
Once participants have been screened, they will read a brief paragraph outlining the current refugee resettlement policy of the United States announced by the president in September of 2019. They will then answer a four item questionaire to measure their support for resettling refugees and two eight item questionaires measuring their views on the type of person someone is and their mindsets regarding participants. Participants will be randomly assigned so that half of sample responds to the refugee measure first and half of the sample responds to the mindsets measure first. Finally, participants will respond to a three item survey measuring their political orientation.
Potential Challenges
While replicating this study, I foresee the following challenges:
- having a sufficient sample size to generate a sufficient effect size that also meets the bugetary criteria for this replication
- having enough participants after excluding participants who do not meet the inclusion criteria
- subjects not feeling as strongly about the current U.S refugee resettlement policy due to other political happenings moving into the public spotlight.
Methods
Power Analysis
Original effect size, power analysis for samples to achieve 80%, 90%, 95% power to detect that effect size. Considerations of feasibility for selecting planned sample size.
The original effect size was r2 = 0.428. Power analysis using G*Power revealed that the reach certain levels of power the following sample sizes are needed:
- 80% power requires a sample of 22
- 90% power requires a sample of 27
- 95% power requires a sample of 32
Completion of the survey takes about ten minutes. In order to remain consistent with California minimum wage ($7.25/hour), each participant will recieve $1.21 as compensation. To account for the course budget, a minimum sample of 27 will be required to attain 90% power. Though the original paper does not report an explicit exclusion rate, 28 participants will be asked to complete the survey just to account for any participants who do not meet the survey requirements.
Planned Sample
The planned sample for this replication will be 28 participants based on the power analysis conducted above. Participants will be screened to ensure that they are citizens of the United States and currently residing in the United States.
Materials
As quoted from original study:
“We administered four items to measure participants’ support for resettling refugees, each measured on a 7-point scale: (a) “How much do you agree that the US should take in 110,000 refugees in the next 12 months?” (strongly disagree to strongly agree), (b) “How much do you support the policy that the US should take in 110,000 refugees in the next 12 months?” (strongly against to strongly support), (c) “Do you think that the United States is taking in too many refugees?” (reverse-coded; definitely not to definitely yes), and (d) “Do you think that the United States is taking in too few refugees?” (definitely not to definitely yes).
We measured participants’ mind-sets about the kind of person someone is using an established eight-item scale (Levy et al., 1998; sample item: “The kind of person someone is, is something basic about them, and it can’t be changed very much”). We also measured participants’ mind-set about intelligence (fixed vs. growth) using the standard eight-item scale (Dweck, 2000; sample item: “People have a certain amount of intelligence, and they can’t really do much to change it”). For both scales, all items were scored such that higher numbers indicated stronger growth kind-of-person mind-sets or mind-sets about intelligence.
Finally, because the question of accepting refugees is a highly politicized issue, we measured participants’ political orientation using three items, each measured on a 7-point scale ranging from strongly conservative to strongly liberal (Item 1), strongly right to strongly left (Item 2), and strongly Republican to strongly Democrat (Item 3)."
The exact items, measures, and scales were used in this replication.
Procedure
We presented participants with the following paragraph detailing the latest refugee resettlement policy of the United States, which was announced by the U.S. president in September 2019:
The United States recently revised its policy towards refugees. The President recently signed an order that set the cap on the number of refugees admitted to the United States next year at 18,000, the lowest level since the program began four decades ago. This new cap is down from the last years limit of 30,000. This is the third consecutive year the U.S has reduced the limit of refugees admitted to the United States.
Participants were then asked to respond to the three measures described in the Materials. Half of the participants were randomly assigned to respond to the resettling refugees measure first while the other half responded to the mindsets measures first. After responding to both of those measures, participants responded to the political orientation measures.
Analysis Plan
First, participants who are not U.S citizens or are U.S citizens living abroad will be excluded from the final sample.
Next, I will calculate descriptive statistics for each variable (kind of person mindsets, mindsets about intelligence, political orientation, and support for resettling refugees).
Key Analysis of Interest:
I will generate three linear models exploring participants’ support for resettling refugees as the dependent measure and their kind-of-person mind-sets, mind-sets about intelligence, and political orientation as independent variables. The key statistic comes from a t-test on the correlation coefficient after adding the kind-of-person measure to the linear model. The p-value is reported to be 0.0016 in the original paper.
Differences from Original Study
To ensure stable correlations, the original study recruited 400 participants, even though their power analysis indicated they needed 94 participants. The current study only used 28 subjects based on the aforementioned power analysis and bugetary constraints.
Per the suggestion of the original author, participants were asked to read a summary of the most recent refugee policy of the United States, instead of the 2016 policy, the year the original study was conducted. The policy outlined in the original study stated that the U.S would increase the number of refugees it admitted, while the current policy states a reduction in the total number of refugees admitted to the United States.
Methods Addendum (Post Data Collection)
Actual Sample
All 28 participants that responded to the mTurk request were included in the study.
Differences from pre-data collection methods plan
None
Results
Data preparation
Below is the data preparation for the pilot data collected through the direct distribution of my qualtirics survey.
Here we load the knitr library for cleaner table output, the tidyverse library to allow for facilitate working with and exploring the data, the here library to facilitate setting the working directory, and the rgeolocate library to determine the state in which participants completed the survey based on their IP address.
### Pilot Data Preparation
####Load Relevant Libraries and Functions
library(knitr)
library(tidyverse)## ── Attaching packages ──────────────────────────────────────────────────────────────── tidyverse 1.2.1 ──## ✔ ggplot2 3.2.1 ✔ purrr 0.3.3
## ✔ tibble 2.1.3 ✔ dplyr 0.8.3
## ✔ tidyr 1.0.0 ✔ stringr 1.4.0
## ✔ readr 1.3.1 ✔ forcats 0.4.0## ── Conflicts ─────────────────────────────────────────────────────────────────── tidyverse_conflicts() ──
## ✖ dplyr::filter() masks stats::filter()
## ✖ dplyr::lag() masks stats::lag()library(here)## here() starts at /Users/Ethan/Documents/Stanford/Fall 2019/PSYCH 251/mandan2019library(rgeolocate)Next we load the data:
####Import data
#setwd("/Users/Ethan/Documents/Stanford/Fall 2019/Psych 251/mandan2019/Data") #Set working directory
rawData <- read.csv(here("data","finalData.csv"),stringsAsFactors = FALSE)Once the raw data are loaded, we manipulate the data in various ways to allow us to run our statistical tests.
First, we rename the values in the random order column to allow us to more easily determine which measure the particpants viewed first. The value ‘mindsetsFirst’ means that the participant responded to the mindsets measures before the refugee measure and ‘refugeeFirst’ means the participant responded to the refugee measure first.
#### Data exclusion / filtering
#Rename column with randomizer data and rename values in the randomOrder column to reflect whether that participant saw the mindset questions or refugee questions first
rawData<-rawData %>% rename(randomOrder = "FL_4_DO") %>%
mutate(randomOrder=recode(randomOrder,'FL_18|Refugee'='mindsetsFirst','Refugee|FL_18'='refugeeFirst'))Then we filtered out any particpants who did not meet our inclusion criteria.
filteredData <- filter(rawData, rawData$Citizen==1)
#filter out any participants who are not US Citizens per original paperNext, we add a column for participant ID. Since each particicpant only spans one row, we can just use the row number as their participant ID. We also recode some of the numeric values collected by Qualtrics, such as Ethnicity and Gender, so they are human readable.
#### Prepare data for analysis - create columns etc.
filteredData <- filteredData %>% mutate(ID = row_number()) #give each row a participant ID
filteredData <- filteredData %>% mutate(state = (ip_api(filteredData$IPAddress)$country_code)) #get participants state based on IP address
#Could not get qualtrics to export text and numeric data so decoding from numeric to string values for ethinicity field
filteredData$Ethnicity <- factor(filteredData$Ethnicity, levels = c(15,16,17,18,19,20,21), labels = c("European American", "African American", "Asian American", "Latin American", "Native American", "Multiracial", "Other"))
filteredData$Gender <- factor(filteredData$Gender,levels = c(11,12,13), labels = c("Female", "Male", "Other"))We then take the filtered data and make it ‘tidy’ so that each answer recorded by a participant becomes a row. This ‘gather’ generates a column called ‘Measure’, which refers to the type of question (intelligence, kind of person, political, or refugee) a particpant was responding two. The value for this column are then renamed to just include the measure type and not the question number to allow for grouping. We then group based on the participant ID, order in which participants answered each measure, and measure type and find the average. These grouped data are then spread into wide format so each row contains one participant and each column contains the average value of their responses for each survey measure.
tidyData <- filteredData %>%
gather(Measure,Score,contains("Personal"),contains("Intell"),contains("Ref_"),contains("Pol"))
tidyData$Score<-as.numeric(tidyData$Score)
tidyData$Measure[contains("Personal",vars = tidyData$Measure)]<- "personalityMeasure"
tidyData$Measure[contains("Intell",vars = tidyData$Measure)]<- "intelligenceMeasure"
tidyData$Measure[contains("Ref",vars = tidyData$Measure)]<- "refugeeMeasure"
tidyData$Measure[contains("Pol",vars = tidyData$Measure)]<- "politicalMeasure"
#refer all questions within each measure by that measure instead of measure_questionNumber
modelDataTidy <-tidyData %>%
group_by(ID,randomOrder,Measure,state) %>%
summarise(meanScores=mean(Score,na.rm=T))
modelData <- modelDataTidy %>%
spread(Measure,meanScores)
#spread so each participant is in one row and mean scores for each measure are the columns
kable(modelData)| ID | randomOrder | state | intelligenceMeasure | personalityMeasure | politicalMeasure | refugeeMeasure |
|---|---|---|---|---|---|---|
| 1 | mindsetsFirst | CA | 3.500 | 3.500 | 6.000000 | 3.00 |
| 2 | mindsetsFirst | MI | 3.000 | 3.500 | 7.000000 | 2.50 |
| 3 | refugeeFirst | CA | 3.625 | 3.500 | 4.000000 | 4.25 |
| 4 | refugeeFirst | FL | 3.500 | 3.500 | 4.000000 | 4.00 |
| 5 | mindsetsFirst | IA | 4.000 | 3.750 | 4.000000 | 2.75 |
| 6 | refugeeFirst | CA | 3.500 | 3.375 | 6.666667 | 5.50 |
| 7 | refugeeFirst | TN | 3.500 | 3.625 | 4.000000 | 4.00 |
| 8 | refugeeFirst | IL | 3.500 | 3.500 | 7.000000 | 4.00 |
| 9 | refugeeFirst | NC | 3.500 | 3.500 | 7.000000 | 5.50 |
| 10 | refugeeFirst | PA | 3.500 | 3.000 | 3.333333 | 4.75 |
| 11 | mindsetsFirst | IL | 3.125 | 3.250 | 7.000000 | 2.75 |
| 12 | refugeeFirst | MD | 3.000 | 3.500 | 6.000000 | 4.50 |
| 13 | refugeeFirst | NY | 3.125 | 3.750 | 2.666667 | 4.25 |
| 14 | mindsetsFirst | TX | 3.125 | 3.125 | 5.333333 | 3.75 |
| 15 | mindsetsFirst | FL | 3.375 | 3.500 | 6.000000 | 5.00 |
| 16 | mindsetsFirst | NC | 3.125 | 3.250 | 4.000000 | 4.00 |
| 17 | refugeeFirst | OH | 3.375 | 3.250 | 6.000000 | 4.00 |
| 18 | mindsetsFirst | TX | 3.625 | 3.750 | 5.666667 | 3.00 |
| 19 | refugeeFirst | UT | 3.500 | 3.500 | 1.000000 | 5.00 |
| 20 | refugeeFirst | IA | 3.500 | 3.500 | 7.000000 | 4.00 |
| 21 | mindsetsFirst | TX | 3.500 | 3.500 | 2.000000 | 3.00 |
| 22 | refugeeFirst | IL | 3.125 | 3.500 | 7.000000 | 2.50 |
| 23 | mindsetsFirst | MD | 3.500 | 3.500 | 7.000000 | 3.00 |
| 24 | refugeeFirst | MO | 3.500 | 3.500 | 5.333333 | 3.50 |
| 25 | refugeeFirst | FL | 3.500 | 3.500 | 4.000000 | 2.75 |
| 26 | refugeeFirst | MA | 3.125 | 3.625 | 6.666667 | 4.25 |
| 27 | refugeeFirst | MI | 3.500 | 3.500 | 4.333333 | 5.50 |
| 28 | refugeeFirst | MO | 3.500 | 3.500 | 5.000000 | 3.00 |
Participant Demographics
Number of subjects in replication:
length(modelData$ID)## [1] 28Age (in years):
ageBreakdown <- filteredData %>%
summarise(Average=mean(as.numeric(Age)), SD=sd(as.numeric(Age)))
kable(ageBreakdown)| Average | SD |
|---|---|
| 36.03571 | 10.21068 |
Gender:
genderBreakdown <- filteredData %>%
group_by(Gender) %>%
summarise(Count=length(Gender), Percentage = length(Gender)/length(filteredData$Gender)*100)
kable(genderBreakdown)| Gender | Count | Percentage |
|---|---|---|
| Female | 10 | 35.71429 |
| Male | 18 | 64.28571 |
Ethnicity:
ethnicityBreakdown <- filteredData %>%
group_by(Ethnicity) %>%
summarise(Count=length(Ethnicity), Percentage = length(Ethnicity)/length(filteredData$Ethnicity)*100)
kable(ethnicityBreakdown)| Ethnicity | Count | Percentage |
|---|---|---|
| European American | 21 | 75.000000 |
| African American | 1 | 3.571429 |
| Latin American | 3 | 10.714286 |
| Multiracial | 3 | 10.714286 |
Geographic Breakdown
Counts of how many respondants per state and the average political views of those respondants.
geoBreakdown <- modelData %>%
group_by(state) %>%
summarise(Count=length(state), Percentage = (length(state))/length(modelData$state)*100, Average_Political_Score = mean(politicalMeasure))
kable(geoBreakdown[order(-geoBreakdown$Average_Political_Score),])| state | Count | Percentage | Average_Political_Score |
|---|---|---|---|
| IL | 3 | 10.714286 | 7.000000 |
| MA | 1 | 3.571429 | 6.666667 |
| MD | 2 | 7.142857 | 6.500000 |
| OH | 1 | 3.571429 | 6.000000 |
| MI | 2 | 7.142857 | 5.666667 |
| CA | 3 | 10.714286 | 5.555556 |
| IA | 2 | 7.142857 | 5.500000 |
| NC | 2 | 7.142857 | 5.500000 |
| MO | 2 | 7.142857 | 5.166667 |
| FL | 3 | 10.714286 | 4.666667 |
| TX | 3 | 10.714286 | 4.333333 |
| TN | 1 | 3.571429 | 4.000000 |
| PA | 1 | 3.571429 | 3.333333 |
| NY | 1 | 3.571429 | 2.666667 |
| UT | 1 | 3.571429 | 1.000000 |
Political Beliefs
polBreakdownAvg <- paste("Average: ", toString(mean(modelData$politicalMeasure)))
polBreakdownSD <- paste("SD: ", toString(sd(modelData$politicalMeasure)))
polBreakdownAvg## [1] "Average: 5.17857142857143"polBreakdownSD## [1] "SD: 1.68869916784567"Exploratory Visualizations
#exploratory demographic histogram to see age distributions by gender and ethnicity
sortedAges <- sort(filteredData$Age)
ggplot(filteredData, aes(x = Age, group = Ethnicity))+
geom_histogram(stat="count",binwidth = 1) +
theme_bw()+
scale_x_discrete(breaks = c("25","28","32","36","38","41","46","56"))+
#these hardcoded values worked best for the axis to look decent. Otherwise there were big jumps in the tick marks
theme(axis.text.x=element_text(angle=45))+
facet_grid(Gender~Ethnicity)## Warning: Ignoring unknown parameters: binwidth, bins, pad
#exploratory political histogram to see distributions by political views
ggplot(modelData, aes(x = politicalMeasure))+
geom_histogram(stat="count",binwidth = 1) +
theme_bw()+
theme(axis.text.x=element_text(angle=45))
Confirmatory analysis
As done in the original study, I built three linear models. The first one looked at how mindsets about intelligence predict support for refugees. The next model added political orientation to the independent variables and the third model added kind of person mindsets. I also ran another regression examining the effect the order of the survey measures (support for resettling refugees measure first or mindsets measures first) and the interaction between kind of person mindsets and presentation order have on participant’s support for resettling refugees. The key statistic will be a t statistic exploring whether the third model (which includes the kind of person mindset measure) differs significantly from the models that do not include that measure.
Summary of Model 1:
model1 <- lm(refugeeMeasure ~ intelligenceMeasure, d=modelData)
summary(model1)##
## Call:
## lm(formula = refugeeMeasure ~ intelligenceMeasure, data = modelData)
##
## Residuals:
## Min 1Q Median 3Q Max
## -1.3170 -0.8714 0.1286 0.5000 1.6286
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 3.3641 2.7615 1.218 0.234
## intelligenceMeasure 0.1449 0.8101 0.179 0.859
##
## Residual standard error: 0.9537 on 26 degrees of freedom
## Multiple R-squared: 0.00123, Adjusted R-squared: -0.03718
## F-statistic: 0.03201 on 1 and 26 DF, p-value: 0.8594Summary of Model 2:
model2 <- lm(refugeeMeasure ~ intelligenceMeasure+politicalMeasure, d=modelData)
summary(model2)##
## Call:
## lm(formula = refugeeMeasure ~ intelligenceMeasure + politicalMeasure,
## data = modelData)
##
## Residuals:
## Min 1Q Median 3Q Max
## -1.22159 -0.83065 0.05356 0.55354 1.78331
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 4.287057 3.111706 1.378 0.180
## intelligenceMeasure -0.009798 0.850614 -0.012 0.991
## politicalMeasure -0.076582 0.114131 -0.671 0.508
##
## Residual standard error: 0.964 on 25 degrees of freedom
## Multiple R-squared: 0.0189, Adjusted R-squared: -0.05959
## F-statistic: 0.2408 on 2 and 25 DF, p-value: 0.7878Summary of Model 3:
Key Statistical test
model3 <- lm(refugeeMeasure ~ intelligenceMeasure+politicalMeasure+personalityMeasure, d=modelData)
summary(model3)##
## Call:
## lm(formula = refugeeMeasure ~ intelligenceMeasure + politicalMeasure +
## personalityMeasure, data = modelData)
##
## Residuals:
## Min 1Q Median 3Q Max
## -1.1884 -0.8085 0.0184 0.6005 1.7843
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 7.13189 4.37739 1.629 0.116
## intelligenceMeasure 0.22343 0.88938 0.251 0.804
## politicalMeasure -0.07423 0.11448 -0.648 0.523
## personalityMeasure -1.05101 1.13413 -0.927 0.363
##
## Residual standard error: 0.9667 on 24 degrees of freedom
## Multiple R-squared: 0.05279, Adjusted R-squared: -0.06561
## F-statistic: 0.4459 on 3 and 24 DF, p-value: 0.7224Summary of Model 4:
model4 <- lm(refugeeMeasure ~ intelligenceMeasure + politicalMeasure + personalityMeasure + randomOrder + personalityMeasure*randomOrder, d=modelData)
summary(model4)##
## Call:
## lm(formula = refugeeMeasure ~ intelligenceMeasure + politicalMeasure +
## personalityMeasure + randomOrder + personalityMeasure * randomOrder,
## data = modelData)
##
## Residuals:
## Min 1Q Median 3Q Max
## -1.4959 -0.3723 -0.1142 0.3882 1.8182
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 7.84740 5.07937 1.545 0.137
## intelligenceMeasure 0.40413 1.02912 0.393 0.698
## politicalMeasure -0.04391 0.10655 -0.412 0.684
## personalityMeasure -1.64745 1.86416 -0.884 0.386
## randomOrderrefugeeFirst -1.90101 8.75592 -0.217 0.830
## personalityMeasure:randomOrderrefugeeFirst 0.80764 2.51981 0.321 0.752
##
## Residual standard error: 0.8833 on 22 degrees of freedom
## Multiple R-squared: 0.275, Adjusted R-squared: 0.1103
## F-statistic: 1.669 on 5 and 22 DF, p-value: 0.1839Exploratory analyses
Due to the nature of the regressions, the original paper has no figures, however, I have included scatter plots exploring the relationship between the mindsets/political measures and the openness to refugees measure.
Plot of First Linear Model
This plot looks at the relationship between the intelligence measure and the refugee measure.
modelData %>%
ggplot(aes(x = intelligenceMeasure, y = refugeeMeasure)) +
xlim(0,7)+
ylim(0,7)+
geom_jitter(aes(colour = randomOrder), width = 0.2, height = 0.4) +
geom_smooth(method=lm, se=FALSE,fullrange=TRUE)+
theme(panel.background = element_rect(fill=FALSE),
axis.line = element_line(colour='black'))
summary(model1)##
## Call:
## lm(formula = refugeeMeasure ~ intelligenceMeasure, data = modelData)
##
## Residuals:
## Min 1Q Median 3Q Max
## -1.3170 -0.8714 0.1286 0.5000 1.6286
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 3.3641 2.7615 1.218 0.234
## intelligenceMeasure 0.1449 0.8101 0.179 0.859
##
## Residual standard error: 0.9537 on 26 degrees of freedom
## Multiple R-squared: 0.00123, Adjusted R-squared: -0.03718
## F-statistic: 0.03201 on 1 and 26 DF, p-value: 0.8594Plot of Second Linear Model This plot looks at the relationship between the political measure and the refugee measure.
modelData %>%
ggplot(aes(x = politicalMeasure, y = refugeeMeasure)) +
xlim(0,7)+
ylim(0,7)+
geom_jitter(aes(colour = randomOrder), width = 0.2, height = 0.4) +
geom_smooth(method=lm, se=FALSE,fullrange=TRUE)+
theme(panel.background = element_rect(fill=FALSE),
axis.line = element_line(colour='black'))
model = lm(refugeeMeasure~politicalMeasure, d=modelData)
summary(model)##
## Call:
## lm(formula = refugeeMeasure ~ politicalMeasure, data = modelData)
##
## Residuals:
## Min 1Q Median 3Q Max
## -1.21830 -0.83264 0.05302 0.55608 1.78170
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 4.25188 0.58576 7.259 1.04e-07 ***
## politicalMeasure -0.07623 0.10772 -0.708 0.485
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 0.9453 on 26 degrees of freedom
## Multiple R-squared: 0.01889, Adjusted R-squared: -0.01884
## F-statistic: 0.5007 on 1 and 26 DF, p-value: 0.4855Plot of Third Linear Model This plot looks at the relationship between the personality measure and the refugee measure.
modelData %>%
ggplot(aes(x = personalityMeasure, y = refugeeMeasure)) +
xlim(0,7)+
ylim(0,7)+
geom_jitter(aes(colour = randomOrder), width = 0.2, height = 0.4) +
geom_smooth(method=lm, se=FALSE,fullrange=TRUE)+
theme(panel.background = element_rect(fill=FALSE),
axis.line = element_line(colour='black'))
model = lm(refugeeMeasure~personalityMeasure, d=modelData)
summary(model)##
## Call:
## lm(formula = refugeeMeasure ~ personalityMeasure, data = modelData)
##
## Residuals:
## Min 1Q Median 3Q Max
## -1.33239 -0.83239 0.05211 0.56197 1.66761
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 7.0662 3.6742 1.923 0.0655 .
## personalityMeasure -0.9239 1.0566 -0.874 0.3899
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 0.9406 on 26 degrees of freedom
## Multiple R-squared: 0.02857, Adjusted R-squared: -0.008794
## F-statistic: 0.7646 on 1 and 26 DF, p-value: 0.3899Comparison of First Three Models
comp = anova(model1,model2,model3, test="Chisq")
comp## Analysis of Variance Table
##
## Model 1: refugeeMeasure ~ intelligenceMeasure
## Model 2: refugeeMeasure ~ intelligenceMeasure + politicalMeasure
## Model 3: refugeeMeasure ~ intelligenceMeasure + politicalMeasure + personalityMeasure
## Res.Df RSS Df Sum of Sq Pr(>Chi)
## 1 26 23.649
## 2 25 23.231 1 0.41838 0.5034
## 3 24 22.428 1 0.80256 0.3541T-test comparing the random order groups
After graphing the various measures against the refugee measure, I noticed that the random order groups seemed clustered. Here I run a t-test to see if order a participant responded to the survey measures impacted their attititudes towards refugees.
t.test(modelData$refugeeMeasure~modelData$randomOrder)##
## Welch Two Sample t-test
##
## data: modelData$refugeeMeasure by modelData$randomOrder
## t = -2.8524, df = 21.247, p-value = 0.009467
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
## -1.5653156 -0.2457955
## sample estimates:
## mean in group mindsetsFirst mean in group refugeeFirst
## 3.275000 4.180556Discussion
Summary of Replication Attempt
The confirmatory analysis found that an individual’s attitude towards a person being able change (growth mindset) did not significantly predict their attitude towards welcoming refugees into their country (t(24) = -0.927, p = 0.363). This failed to replicate the original result, which found the more a person held a growth mindset, the more welcoming they were to refugees (t(400) = 3.17, p=.0016).
Commentary
Exploratory plotting of the three single variable models reinforced the notion that none of the predictor variables correlated with the criterion variable. However, these plots did highlight a potential difference between the participants who answered the refugee measure first and those who answered the mindsets measures first. After running a two sample t-test, the observed differences based on the graph clusters was found to be significant (p=0.009), indicating that the order the measures were presented impacted how participants responded to the refugee measure. The original authors also conducted an analysis looking at the impact stimulus order had on responses to the refugee measure and did not find a significant result.
Additional exploratory plots revealed that the political views of the current participants skewed heavily to the left (mean=5.178571 sd = 1.688699), which may have had an impact on the present failure to replicate the original study. The original study did not include any these plots but reported an average score on the political measure of 4.38 with a standard deviation of 1.66. This suggests that the original study used either a more moderate or a more politically diverse sample than the present study.
Another possible explaination for the present failure to replicate was proposed by the original author. She suggested that “all the dependent variables across all studies reference to refugee policies that were just announced/top of mind in news media and widely debated”, which may impact the current study. When the original study was conducted, immigration was a salient issue in the U.S media, whereas, at the time of the present study, other issues have moved into the media spotlight. The impact of the news media cycle may also serve to explain the present failure to replicate.
Aside from the concerns about the impact of media coverage, the original authors did not raise any objections or challenges about the present replication attempt.