Replication of Gender stereotypes about intellectual ability emerge early and influence children’s interests by Lin Bian, Sarah-Jane Leslie and Andrei Cimpian (2017, Science)
Lorna Camus
12 December 2017
1 Introduction
Bian, Leslie and Cimpian (2017) investigated whether gender stereotypes regarding brilliance were endorsed by children aged five to seven years old, and whether these influenced children’s interests. Previous studies had looked at the effect of gender stereotypes on performance and interests in adults, along with their role in gender pay gaps. However, the lack of research on the acquisition of these stereotypes led the researchers to investigate this in children. The authors hypothesised that the earlier children acquired gender stereotypes (specifically, the stereotype that men are more intelligent than women), the stronger these would influence children’s interests. This area of research has implications for children’s future academic and career choices, as girls may avoid early subjects and careers typically associated with high-level intellectual ability.
Four studies were conducted to test this hypothesis. The first investigated the acquisition of the stereotype by looking at age differences in stereotype endorsement (and the second replicated it with a larger sample). The third and fourth investigated the stereotype’s influence on children’s interests by looking at whether stereotype endorsement shaped children’s interests in games for children who are ‘really, really smart’ or who try ‘really, really hard’. Study one (and two) found no differences in five year old children’s association of brilliance with their own gender, but found gender differences at ages six and seven, with girls significantly less likely than boys to associate brilliance with their own gender. The authors also tested whether this difference was linked to girls’ perception of their school achievement. They found no significant age difference, suggesting girls’ idea of brilliance is not linked to their perceptions of school performance.
This replication focuses on the second task of the first study, which is explained here. Prior to the tasks of Study 1, the authors presented 12 counterbalanced screener questions to the children to assess their understanding of the key terms ‘smart’ and ‘nice’. The children were asked to decide if a child whose behaviour had been described by the experimenter was ‘smart’ or ‘nice’. If they were wrong they were corrected. The first study had 96 participants of 5, 6 and 7 years old (with 32 children per age group, half boys and half girls). The sample wasn’t ethnically and socioeconomically diverse (mostly middle class children, 75% white), however the authors controlled for these moderators and found they did not significantly moderate the results they were interested in. The study design used both between-subjects factors (gender and age) and within-subjects factors (the three experimental tasks). The second task examined age differences in children’s endorsement of the male=brilliance stereotype by giving children counterbalanced pictures of pairs of same- or different-gender adults. Two were practice trials and involved adults of the same sex as the participant, and four involved a man and a woman. Children were either told one of the adults was ‘really, really smart’ (3 out of 6 trials) or ‘really, really nice’ (other 3 trials), and then had to guess which one matched the description. Both men and women in the pictures were normed for factors that could influence children’s decision, such as professional dress and attractiveness. Children’s responses were scored as a 0 on a trial if they chose the adult of a different gender as their own, and 1 otherwise.
In order to verify this area of research was finding true effects and was not affected by publication bias, we ran a p-Curve analysis. As this research area is new, only three papers were analysed, including the replicated paper (Bian, Leslie & Cimpian, 2017) and another two papers testing the same hypothesis (Cvencek, Meltzoff & Greenwald, 2011; Martinot, Bagès & Désert, 2012). The p-Curve (Figure 1) shows that, for the results included, there is a strong right-skew (towards low p-values), indicating what seems to be true effects (and therefore no publication bias).
Figure 1: p-Curve
2 Methods
2.1 Power Analysis
Included in this section are power analyses for the original effect sizes and for effect sizes 75% as large as the original. I have included both power analyses with and without the inclusion of the interaction between experimental condition and age because the differences are large, and including the interaction considerably slows down running large simulations.
2.1.1 For original effect sizes (without interaction)
Power analysis simulations (see code below) suggest a sample of 40 participants per condition for 80% power for the large effect size found in the original study, as seen in Figure 2. The original sample size reported by Bian, Leslie and Cimpian (2017) was 32 children per age group (96 in total). This replication study will not recruit and report results for 40 participants per condition, rather it will report pilot results.
#downloaded and imported data set for the study being replicated
library(readr)
data <- read_csv("Studies1 and 2 (with demographic information).csv")
#created a data frame with relevant variables (subject ID, age, gender, stereotype scores, and condition (smart or nice))
stereo<-cbind(data$subj, data$age)
stereo<-cbind(stereo, data$gender)
stereo<-cbind(stereo, data$stereo)
stereo<-cbind(stereo, data$trait)
stereo<-as.matrix(stereo)
stereo<-stereo[-(193:480),] #removed data for study 2
stereo1<-data.frame(stereo)
names(stereo1)<-c("subj", "age", "gender", "scores", "trait")n_trials=6 #there were six trials in the original study
n_children=16 #16 children per condition
# we want trait = 1 since we are interested in the scores on the smart condition
means_bian <- ddply(stereo1, .(age, gender, trait), summarize, mean(scores))
sd_bian <- ddply(stereo1, .(age, gender, trait), summarize, sd(scores))
#selected the means and SDs for the right age and gender group and for the right condition
m_5yo_g<-means_bian[2,4]
sd_5yo_g<-sd_bian[2,4]
m_6yo_g<-means_bian[6,4]
sd_6yo_g<-sd_bian[6,4]
m_7yo_g<-means_bian[10,4]
sd_7yo_g<-sd_bian[10,4]
m_5yo_b<-means_bian[4,4]
sd_5yo_b<-sd_bian[4,4]
m_6yo_b<-means_bian[8,4]
sd_6yo_b<-sd_bian[8,4]
m_7yo_b<-means_bian[12,4]
sd_7yo_b<-sd_bian[12,4]
#created a dataframe that crosses age and gender (the variables of interest), and says that there are "N=n_children" participants per combination
bian=expand.grid(
n_trials=n_trials,
N=n_children,
Gender=c("Girl", "Boy"),
Age=c("5", "6", "7"),
Simulations=1
)
#used the mutate function in dplyr to add the mean and SD of the relevant normal distributions to the dataframe. I used ifelse to subset bian into its different conditions (age and gender) and then specify the mean and SD for these
#I have used dplyr:: in some lines below because certain functions (mutate, summarise, ... clash with functions from another package)
bian=
bian %>%
mutate(mean=ifelse(Gender=="Girl"&Age=="5", m_5yo_g,
ifelse(Gender=="Girl"&Age=="6", m_6yo_g,
ifelse(Gender=="Girl"&Age=="7", m_7yo_g,
ifelse(Gender=="Boy"&Age=="5", m_5yo_b,
ifelse(Gender=="Boy"&Age=="6", m_6yo_b, m_7yo_b))))),
sd=ifelse(Gender=="Girl"&Age=="5", sd_5yo_g,
ifelse(Gender=="Girl"&Age=="6", sd_6yo_g,
ifelse(Gender=="Girl"&Age=="7", sd_7yo_g,
ifelse(Gender=="Boy"&Age=="5", sd_5yo_b,
ifelse(Gender=="Boy"&Age=="6", sd_6yo_b,sd_7yo_b)))))) %>%
group_by(Gender, Age, n_trials,N,Simulations) %>% #grouped the variables by gender, age, number of trials, number of children and number of simulations
do(subject_prob = rnorm(.$N, .$mean, .$sd)) %>% #assigned each participant a simulated probability of choosing a male
unnest(subject_prob) %>%
dplyr::mutate(subject_prob=replace(subject_prob, subject_prob<0, 0.01),
subject_prob=replace(subject_prob, subject_prob>1, 0.99)) #since I have generated values from a normal distribution, some values may lie outside of this range - using mutate() and replace() cleans the data by replacing values above 1 with 0.99 and below 0 with 0.01
#using the probability simulated above, we can generate each participant's performance on the 6 test trials. Since the participants need to choose between a man and a woman, we can treat each trial as a binomial event (use rbinom() to do this)
bian=bian %>%
group_by(Gender,Age,n_trials,N,Simulations,subject_prob) %>%
do(data=rbinom(1, .$n_trials, .$subject_prob)/.$n_trials) %>% #data is the generated rbinom() divided by the number of trials to create a proportion
unnest(data)
print(bian)
#generated the mean and SD for each age and gender groups
bian_means = bian %>%
group_by(Gender, Age) %>%
dplyr::summarize(Mean = mean(data), SE = sd(data)/sqrt(n()))
print(bian_means)
#created a graph showing these
ggplot(bian_means, aes(x=Gender, y=Mean, fill=Gender))+
geom_bar(stat="identity")+
geom_linerange(aes(ymin=Mean-SE, ymax=Mean+SE))+
facet_grid(.~Age)
#experimenting with trying to reproduce plot seen in paper
ggplot(bian_means, aes(x=Age, y=Mean, fill=Age, colour=Gender))+
geom_errorbar(aes(ymin=Mean-SE, ymax=Mean+SE))+
geom_line(aes(Gender))n_trials=6 #there were six trials in the original study
# we want trait = 1 since we are interested in the scores on the smart condition
means_bian <- ddply(stereo1, .(age, gender,trait), summarize, mean(scores))
sd_bian <- ddply(stereo1, .(age, gender,trait), summarize, sd(scores))
#selected the means and SDs for the right age and gender group and for the right condition
m_5yo_g<-means_bian[2,4]
sd_5yo_g<-sd_bian[2,4]
m_6yo_g<-means_bian[6,4]
sd_6yo_g<-sd_bian[6,4]
m_7yo_g<-means_bian[10,4]
sd_7yo_g<-sd_bian[10,4]
m_5yo_b<-means_bian[4,4]
sd_5yo_b<-sd_bian[4,4]
m_6yo_b<-means_bian[8,4]
sd_6yo_b<-sd_bian[8,4]
m_7yo_b<-means_bian[12,4]
sd_7yo_b<-sd_bian[12,4]
#created a dataframe that crosses age and gender (the variables of interest), and says that there are N=different numbers of children per condition varying from 10 to 100 (seq()) and that simulates 500 datasets
bian=expand.grid(
n_trials=n_trials,
N=seq(from=10,to=100, by=10),
Gender=c("Girl", "Boy"),
Age=c("5", "6", "7"),
Simulations=1:500
)
#used the mutate function in dplyr to add the mean and SD of the relevant normal distributions to the dataframe. I used ifelse to subset bian into its different conditions (age and gender) and then specify the mean and SD for these
#I have used dplyr:: in some lines below because certain functions (mutate, summarise, ... clash with functions from another package)
bian=
bian %>%
mutate(mean=ifelse(Gender=="Girl"&Age=="5", m_5yo_g,
ifelse(Gender=="Girl"&Age=="6", m_6yo_g,
ifelse(Gender=="Girl"&Age=="7", m_7yo_g,
ifelse(Gender=="Boy"&Age=="5", m_5yo_b,
ifelse(Gender=="Boy"&Age=="6", m_6yo_b,
ifelse(Gender=="Boy"&Age=="7", m_7yo_b,0)))))),
sd=ifelse(Gender=="Girl"&Age=="5", sd_5yo_g,
ifelse(Gender=="Girl"&Age=="6", sd_6yo_g,
ifelse(Gender=="Girl"&Age=="7", sd_7yo_g,
ifelse(Gender=="Boy"&Age=="5", sd_5yo_b,
ifelse(Gender=="Boy"&Age=="6", sd_6yo_b,
ifelse(Gender=="Boy"&Age=="7", sd_7yo_b,0))))))) %>%
group_by(Gender, Age, n_trials,N,Simulations) %>% #grouped the variables by gender, age, number of trials, number of children and number of simulations
do(subject_prob = rnorm(.$N, .$mean, .$sd)) %>% #assigned each participant a simulated probability of choosing a male
unnest(subject_prob) %>%
dplyr::mutate(subject_prob=replace(subject_prob, subject_prob<0, 0.01),
subject_prob=replace(subject_prob, subject_prob>1, 0.99)) #since I have generated values from a normal distribution, some values may lie outside of this range - using mutate() and replace() cleans the data by replacing values above 1 with 0.99 and below 0 with 0.01
#using the probability simulated above, we can generate each participant's performance on the 6 test trials. Since the participants need to choose between a man and a woman, we can treat each trial as a binomial event (use rbinom() to do this)
bian %>%
group_by(Gender, Age,n_trials,N,Simulations,subject_prob) %>%
do(data=rbinom(1, .$n_trials, .$subject_prob)/.$n_trials) %>%
unnest(data)
#ran a regression to test if there is a difference between age and gender groups
summarized_bian = bian %>%
select(n_trials,N,Simulations,subject_prob,Gender,Age) %>%
group_by(Gender,Age,n_trials,N,Simulations,subject_prob) %>%
do(data=rbinom(1, .$n_trials, .$subject_prob)/.$n_trials) %>%
unnest(data) %>%
select(Gender,Age,N,data,Simulations,n_trials,subject_prob) %>%
group_by(N,Simulations) %>%
do(p_value = summary(lm(data~Gender+Age, data=.))$coefficient[2,4]) %>%
unnest(p_value) %>%
mutate(significant=ifelse(p_value<0.05, 1, 0)) %>%
group_by(N) %>%
dplyr::summarise(N_sims=length(p_value), proportion_significant=sum(significant)/N_sims)
summarized_bian#ploted the probability of getting a significant result by the number of children tested.
ggplot(summarized_bian,
aes(x=N,
y=proportion_significant))+
xlab("Number of participants")+
ylab("Proportion of significant results")+
geom_line()
Figure 2: Statistical simulations of sample sizes necessary for 80% power, based on original effect sizes (not including the interaction between age and trial condition).
2.1.2 For smaller effect sizes (without interaction)
Power analysis simulations (see code below) suggest that even a sample of 100 participants per condition cannot detect 20% power for effect sizes 75% as large as those found in the original study, as seen in Figure 3. This suggests that if the original paper overestimated effect sizes, the sample size should be well above 100 to have 80% power.
n_trials=6 #there were six trials in the original study
n_children=16 #16 children per condition
#took the means from the first simulation and made the differences between groups 75% as large as those in the original paper
m_5yo_g<-0.70
sd_5yo_g<-0.22
m_6yo_g<-0.65
sd_6yo_g<-0.23
m_7yo_g<-0.68
sd_7yo_g<-0.26
m_5yo_b<-0.70
sd_5yo_b<-0.21
m_6yo_b<-0.61
sd_6yo_b<-0.24
m_7yo_b<-0.65
sd_7yo_b<-0.25
#created a dataframe that crosses age and gender (the variables of interest), and says that there are "N=n_children" participants per combination
bian=expand.grid(
n_trials=n_trials,
N=n_children,
Gender=c("Girl", "Boy"),
Age=c("5", "6", "7"),
Simulations=1
)
#used the mutate function in dplyr to add the mean and SD of the relevant normal distributions to the dataframe. I used ifelse to subset bian into its different conditions (age and gender) and then specify the mean and SD for these
#I have used dplyr:: in some lines below because certain functions (mutate, summarise, ... clash with functions from another package)
bian=
bian %>%
mutate(mean=ifelse(Gender=="Girl"&Age=="5", m_5yo_g,
ifelse(Gender=="Girl"&Age=="6", m_6yo_g,
ifelse(Gender=="Girl"&Age=="7", m_7yo_g,
ifelse(Gender=="Boy"&Age=="5", m_5yo_b,
ifelse(Gender=="Boy"&Age=="6", m_6yo_b, m_7yo_b))))),
sd=ifelse(Gender=="Girl"&Age=="5", sd_5yo_g,
ifelse(Gender=="Girl"&Age=="6", sd_6yo_g,
ifelse(Gender=="Girl"&Age=="7", sd_7yo_g,
ifelse(Gender=="Boy"&Age=="5", sd_5yo_b,
ifelse(Gender=="Boy"&Age=="6", sd_6yo_b,sd_7yo_b)))))) %>%
group_by(Gender, Age, n_trials,N,Simulations) %>% #grouped the variables by gender, age, number of trials, number of children and number of simulations
do(subject_prob = rnorm(.$N, .$mean, .$sd)) %>% #assigned each participant a simulated probability of choosing a male
unnest(subject_prob) %>%
dplyr::mutate(subject_prob=replace(subject_prob, subject_prob<0, 0.01),
subject_prob=replace(subject_prob, subject_prob>1, 0.99)) #since I have generated values from a normal distribution, some values may lie outside of this range - using mutate() and replace() cleans the data by replacing values above 1 with 0.99 and below 0 with 0.01
#using the probability simulated above, we can generate each participant's performance on the 6 test trials. Since the participants need to choose between a man and a woman, we can treat each trial as a binomial event (use rbinom() to do this)
bian=bian %>%
group_by(Gender,Age,n_trials,N,Simulations,subject_prob) %>%
do(data=rbinom(1, .$n_trials, .$subject_prob)/.$n_trials) %>% #data is the generated rbinom() divided by the number of trials to create a proportion
unnest(data)
print(bian)
#generated the mean and SD for each age and gender groups
bian_means = bian %>%
group_by(Gender, Age) %>%
dplyr::summarize(Mean = mean(data), SE = sd(data)/sqrt(n()))
print(bian_means)
#created a graph showing these
ggplot(bian_means, aes(x=Gender, y=Mean, fill=Gender))+
geom_bar(stat="identity")+
geom_linerange(aes(ymin=Mean-SE, ymax=Mean+SE))+
facet_grid(.~Age)
#experimenting with trying to reproduce plot seen in paper
ggplot(bian_means, aes(x=Age, y=Mean, fill=Age, colour=Gender))+
geom_errorbar(aes(ymin=Mean-SE, ymax=Mean+SE))+
geom_line(aes(Gender))n_trials=6 #there were six trials in the original study
#took the means from the first simulation and made the differences between groups 75% as large as those in the original paper
m_5yo_g<-0.70
sd_5yo_g<-0.22
m_6yo_g<-0.65
sd_6yo_g<-0.23
m_7yo_g<-0.68
sd_7yo_g<-0.26
m_5yo_b<-0.70
sd_5yo_b<-0.21
m_6yo_b<-0.61
sd_6yo_b<-0.24
m_7yo_b<-0.65
sd_7yo_b<-0.25
#created a dataframe that crosses age and gender (the variables of interest), and says that there are N=different numbers of children per condition varying from 10 to 100 (seq()) and that simulates 500 datasets
bian=expand.grid(
n_trials=n_trials,
N=seq(from=10,to=100, by=10),
Gender=c("Girl", "Boy"),
Age=c("5", "6", "7"),
Simulations=1:500
)
#used the mutate function in dplyr to add the mean and SD of the relevant normal distributions to the dataframe. I used ifelse to subset bian into its different conditions (age and gender) and then specify the mean and SD for these
#I have used dplyr:: in some lines below because certain functions (mutate, summarise, ... clash with functions from another package)
bian=
bian %>%
mutate(mean=ifelse(Gender=="Girl"&Age=="5", m_5yo_g,
ifelse(Gender=="Girl"&Age=="6", m_6yo_g,
ifelse(Gender=="Girl"&Age=="7", m_7yo_g,
ifelse(Gender=="Boy"&Age=="5", m_5yo_b,
ifelse(Gender=="Boy"&Age=="6", m_6yo_b,
ifelse(Gender=="Boy"&Age=="7", m_7yo_b,0)))))),
sd=ifelse(Gender=="Girl"&Age=="5", sd_5yo_g,
ifelse(Gender=="Girl"&Age=="6", sd_6yo_g,
ifelse(Gender=="Girl"&Age=="7", sd_7yo_g,
ifelse(Gender=="Boy"&Age=="5", sd_5yo_b,
ifelse(Gender=="Boy"&Age=="6", sd_6yo_b,
ifelse(Gender=="Boy"&Age=="7", sd_7yo_b,0))))))) %>%
group_by(Gender, Age, n_trials,N,Simulations) %>% #grouped the variables by gender, age, number of trials, number of children and number of simulations
do(subject_prob = rnorm(.$N, .$mean, .$sd)) %>% #assigned each participant a simulated probability of choosing a male
unnest(subject_prob) %>%
dplyr::mutate(subject_prob=replace(subject_prob, subject_prob<0, 0.01),
subject_prob=replace(subject_prob, subject_prob>1, 0.99)) #since I have generated values from a normal distribution, some values may lie outside of this range - using mutate() and replace() cleans the data by replacing values above 1 with 0.99 and below 0 with 0.01
#using the probability simulated above, we can generate each participant's performance on the 6 test trials. Since the participants need to choose between a man and a woman, we can treat each trial as a binomial event (use rbinom() to do this)
bian %>%
group_by(Gender, Age,n_trials,N,Simulations,subject_prob) %>%
do(data=rbinom(1, .$n_trials, .$subject_prob)/.$n_trials) %>%
unnest(data)
#ran a regression to test if there is a difference between age and gender groups
summarized_bian = bian %>%
select(n_trials,N,Simulations,subject_prob,Gender,Age) %>%
group_by(Gender,Age,n_trials,N,Simulations,subject_prob) %>%
do(data=rbinom(1, .$n_trials, .$subject_prob)/.$n_trials) %>%
unnest(data) %>%
select(Gender,Age,N,data,Simulations,n_trials,subject_prob) %>%
group_by(N,Simulations) %>%
do(p_value = summary(lm(data~Gender+Age, data=.))$coefficient[2,4]) %>%
unnest(p_value) %>%
mutate(significant=ifelse(p_value<0.05, 1, 0)) %>%
group_by(N) %>%
dplyr::summarise(N_sims=length(p_value), proportion_significant=sum(significant)/N_sims)
summarized_bian#ploted the probability of getting a significant result by the number of children tested.
ggplot(summarized_bian,
aes(x=N,
y=proportion_significant))+
xlab("Number of participants")+
ylab("Proportion of significant results")+
geom_line()
Figure 3: Statistical simulations of sample sizes necessary for 80% power, based on effect sizes 75% as large as those in the original(not including the interaction between age and trial condition)
2.1.3 For original sample sizes (with interaction)
Power analysis simulations including the interaction of trial condition and age (see code below) suggest that even a sample of 100 participants per condition cannot detect 20% power for the original effect sizes, as seen in Figure 4. This suggests that the effect is too small to detect with the sample size reported in the original paper.
#downloaded and imported data set for the study being replicated
library(readr)
data <- read_csv("Studies1 and 2 (with demographic information).csv")
#created a data frame with relevant variables (subject ID, age, gender, stereotype scores, and condition (smart or nice))
stereo<-cbind(data$subj, data$age)
stereo<-cbind(stereo, data$gender)
stereo<-cbind(stereo, data$stereo)
stereo<-cbind(stereo, data$trait)
stereo<-as.matrix(stereo)
stereo<-stereo[-(193:480),] #removed data for study 2
stereo1<-data.frame(stereo)
names(stereo1)<-c("subj", "age", "gender", "scores", "trait")n_trials=6 #there were six trials in the original study
n_children=16 #16 children per condition
# trait = 1 corresponds to the smart condition, trait = 2 to the nice condition
means_bian <- ddply(stereo1, .(age, gender, trait), summarize, mean(scores))
sd_bian <- ddply(stereo1, .(age, gender, trait), summarize, sd(scores))
#selected the means and SDs for the right age and gender group and for the right condition
m_5yo_g_s<-means_bian[2,4]
sd_5yo_g_s<-sd_bian[2,4]
m_6yo_g_s<-means_bian[6,4]
sd_6yo_g_s<-sd_bian[6,4]
m_7yo_g_s<-means_bian[10,4]
sd_7yo_g_s<-sd_bian[10,4]
m_5yo_b_s<-means_bian[4,4]
sd_5yo_b_s<-sd_bian[4,4]
m_6yo_b_s<-means_bian[8,4]
sd_6yo_b_s<-sd_bian[8,4]
m_7yo_b_s<-means_bian[12,4]
sd_7yo_b_s<-sd_bian[12,4]
m_5yo_g_n<-means_bian[1,4]
sd_5yo_g_n<-sd_bian[1,4]
m_6yo_g_n<-means_bian[5,4]
sd_6yo_g_n<-sd_bian[5,4]
m_7yo_g_n<-means_bian[9,4]
sd_7yo_g_n<-sd_bian[9,4]
m_5yo_b_n<-means_bian[3,4]
sd_5yo_b_n<-sd_bian[3,4]
m_6yo_b_n<-means_bian[7,4]
sd_6yo_b_n<-sd_bian[7,4]
m_7yo_b_n<-means_bian[11,4]
sd_7yo_b_n<-sd_bian[11,4]
#created a dataframe that crosses age, gender and trial condition (the variables of interest), and says that there are "N=n_children" participants per combination
bian=expand.grid(
n_trials=n_trials,
N=n_children,
Gender=c("Girl", "Boy"),
Age=c("5", "6", "7"),
Trait=c("Smart", "Nice"),
Simulations=1
)
#used the mutate function in dplyr to add the mean and SD of the relevant normal distributions to the dataframe. I used ifelse to subset bian into its different conditions (age, gender, trait) and then specify the mean and SD for these
#I have used dplyr:: in some lines below because certain functions (mutate, summarise, ... clash with functions from another package)
bian=bian %>%
mutate(mean=ifelse(Gender=="Girl"&Age=="5"&Trait=="Smart", m_5yo_g_s,
ifelse(Gender=="Girl"&Age=="6"&Trait=="Smart", m_6yo_g_s,
ifelse(Gender=="Girl"&Age=="7"&Trait=="Smart", m_7yo_g_s,
ifelse(Gender=="Girl"&Age=="5"&Trait=="Nice", m_5yo_g_n,
ifelse(Gender=="Girl"&Age=="6"&Trait=="Nice", m_6yo_g_n,
ifelse(Gender=="Girl"&Age=="7"&Trait=="Nice", m_7yo_g_n,
ifelse(Gender=="Boy"&Age=="5"&Trait=="Smart", m_5yo_b_s,
ifelse(Gender=="Boy"&Age=="6"&Trait=="Smart", m_6yo_b_s,
ifelse(Gender=="Boy"&Age=="7"&Trait=="Smart", m_7yo_b_s,
ifelse(Gender=="Boy"&Age=="5"&Trait=="Nice", m_5yo_b_n,
ifelse(Gender=="Boy"&Age=="6"&Trait=="Nice", m_6yo_b_n, m_7yo_b_n))))))))))),
sd=ifelse(Gender=="Girl"&Age=="5"&Trait=="Smart", sd_5yo_g_s,
ifelse(Gender=="Girl"&Age=="6"&Trait=="Smart", sd_6yo_g_s,
ifelse(Gender=="Girl"&Age=="7"&Trait=="Smart", sd_7yo_g_s,
ifelse(Gender=="Girl"&Age=="5"&Trait=="Nice", sd_5yo_g_n,
ifelse(Gender=="Girl"&Age=="6"&Trait=="Nice", sd_6yo_g_n,
ifelse(Gender=="Girl"&Age=="7"&Trait=="Nice", sd_7yo_g_n,
ifelse(Gender=="Boy"&Age=="5"&Trait=="Smart", sd_5yo_b_s,
ifelse(Gender=="Boy"&Age=="6"&Trait=="Smart", sd_6yo_b_s,
ifelse(Gender=="Boy"&Age=="7"&Trait=="Smart", sd_7yo_b_s,
ifelse(Gender=="Boy"&Age=="5"&Trait=="Nice", sd_5yo_b_n,
ifelse(Gender=="Boy"&Age=="6"&Trait=="Nice", sd_6yo_b_n, sd_7yo_b_n)))))))))))) %>%
group_by(Gender, Age, Trait, n_trials,N,Simulations) %>% #grouped the variables by gender, age, trait, number of trials, number of children and number of simulations
do(subject_prob = rnorm(.$N, .$mean, .$sd)) %>% #assigned each participant a simulated probability of choosing a male
unnest(subject_prob) %>%
dplyr::mutate(subject_prob=replace(subject_prob, subject_prob<0, 0.01),
subject_prob=replace(subject_prob, subject_prob>1, 0.99)) #since I have generated values from a normal distribution, some values may lie outside of this range - using mutate() and replace() cleans the data by replacing values above 1 with 0.99 and below 0 with 0.01
#using the probability simulated above, we can generate each participant's performance on the 6 test trials. Since the participants need to choose between a man and a woman, we can treat each trial as a binomial event (use rbinom() to do this)
bian=bian %>%
group_by(Gender,Age, Trait, n_trials,N,Simulations,subject_prob) %>%
do(data=rbinom(1, .$n_trials, .$subject_prob)/.$n_trials) %>% #data is the generated rbinom() divided by the number of trials to create a proportion
unnest(data)
print(bian)
#generated the mean and SD for each age, gender and trait groups
bian_means = bian %>%
group_by(Gender, Age, Trait) %>%
dplyr::summarize(Mean = mean(data), SE = sd(data)/sqrt(n()))
print(bian_means)
#created a graph showing these
ggplot(bian_means, aes(x=Gender, y=Mean, fill=Gender))+
geom_bar(stat="identity")+
geom_linerange(aes(ymin=Mean-SE, ymax=Mean+SE))+
facet_grid(Trait~Age)
#experimenting with trying to reproduce plot seen in paper
ggplot(bian_means, aes(x=Age, y=Mean, fill=Age, colour=Gender))+
geom_errorbar(aes(ymin=Mean-SE, ymax=Mean+SE))+
geom_line(aes(Gender))n_trials=6 #there were six trials in the original study
# we want trait = 1 since we are interested in the scores on the smart condition
means_bian <- ddply(stereo1, .(age, gender,trait), summarize, mean(scores))
sd_bian <- ddply(stereo1, .(age, gender,trait), summarize, sd(scores))
#selected the means and SDs for the right age and gender group and for the right condition
m_5yo_g_s<-means_bian[2,4]
sd_5yo_g_s<-sd_bian[2,4]
m_6yo_g_s<-means_bian[6,4]
sd_6yo_g_s<-sd_bian[6,4]
m_7yo_g_s<-means_bian[10,4]
sd_7yo_g_s<-sd_bian[10,4]
m_5yo_b_s<-means_bian[4,4]
sd_5yo_b_s<-sd_bian[4,4]
m_6yo_b_s<-means_bian[8,4]
sd_6yo_b_s<-sd_bian[8,4]
m_7yo_b_s<-means_bian[12,4]
sd_7yo_b_s<-sd_bian[12,4]
m_5yo_g_n<-means_bian[1,4]
sd_5yo_g_n<-sd_bian[1,4]
m_6yo_g_n<-means_bian[5,4]
sd_6yo_g_n<-sd_bian[5,4]
m_7yo_g_n<-means_bian[9,4]
sd_7yo_g_n<-sd_bian[9,4]
m_5yo_b_n<-means_bian[3,4]
sd_5yo_b_n<-sd_bian[3,4]
m_6yo_b_n<-means_bian[7,4]
sd_6yo_b_n<-sd_bian[7,4]
m_7yo_b_n<-means_bian[11,4]
sd_7yo_b_n<-sd_bian[11,4]
#created a dataframe that crosses age, gender and trial condition (the variables of interest), and says that there are N=different numbers of children per condition varying from 10 to 100 (seq()) and that simulates 500 datasets
bian=expand.grid(
n_trials=n_trials,
N=seq(from=10,to=100, by=10),
Gender=c("Girl", "Boy"),
Age=c("5", "6", "7"),
Trait=c("Smart", "Nice"),
Simulations=1:500
)
#used the mutate function in dplyr to add the mean and SD of the relevant normal distributions to the dataframe. I used ifelse to subset bian into its different conditions (age, gender and trait) and then specify the mean and SD for these
#I have used dplyr:: in some lines below because certain functions (mutate, summarise, ... clash with functions from another package)
bian=bian %>%
mutate(mean=ifelse(Gender=="Girl"&Age=="5"&Trait=="Smart", m_5yo_g_s,
ifelse(Gender=="Girl"&Age=="6"&Trait=="Smart", m_6yo_g_s,
ifelse(Gender=="Girl"&Age=="7"&Trait=="Smart", m_7yo_g_s,
ifelse(Gender=="Girl"&Age=="5"&Trait=="Nice", m_5yo_g_n,
ifelse(Gender=="Girl"&Age=="6"&Trait=="Nice", m_6yo_g_n,
ifelse(Gender=="Girl"&Age=="7"&Trait=="Nice", m_7yo_g_n,
ifelse(Gender=="Boy"&Age=="5"&Trait=="Smart", m_5yo_b_s,
ifelse(Gender=="Boy"&Age=="6"&Trait=="Smart", m_6yo_b_s,
ifelse(Gender=="Boy"&Age=="7"&Trait=="Smart", m_7yo_b_s,
ifelse(Gender=="Boy"&Age=="5"&Trait=="Nice", m_5yo_b_n,
ifelse(Gender=="Boy"&Age=="6"&Trait=="Nice", m_6yo_b_n, m_7yo_b_n))))))))))),
sd=ifelse(Gender=="Girl"&Age=="5"&Trait=="Smart", sd_5yo_g_s,
ifelse(Gender=="Girl"&Age=="6"&Trait=="Smart", sd_6yo_g_s,
ifelse(Gender=="Girl"&Age=="7"&Trait=="Smart", sd_7yo_g_s,
ifelse(Gender=="Girl"&Age=="5"&Trait=="Nice", sd_5yo_g_n,
ifelse(Gender=="Girl"&Age=="6"&Trait=="Nice", sd_6yo_g_n,
ifelse(Gender=="Girl"&Age=="7"&Trait=="Nice", sd_7yo_g_n,
ifelse(Gender=="Boy"&Age=="5"&Trait=="Smart", sd_5yo_b_s,
ifelse(Gender=="Boy"&Age=="6"&Trait=="Smart", sd_6yo_b_s,
ifelse(Gender=="Boy"&Age=="7"&Trait=="Smart", sd_7yo_b_s,
ifelse(Gender=="Boy"&Age=="5"&Trait=="Nice", sd_5yo_b_n,
ifelse(Gender=="Boy"&Age=="6"&Trait=="Nice", sd_6yo_b_n, sd_7yo_b_n)))))))))))) %>%
group_by(Gender, Age, Trait, n_trials,N,Simulations) %>% #grouped the variables by gender, age, trait, number of trials, number of children and number of simulations
do(subject_prob = rnorm(.$N, .$mean, .$sd)) %>% #assigned each participant a simulated probability of choosing a male
unnest(subject_prob) %>%
dplyr::mutate(subject_prob=replace(subject_prob, subject_prob<0, 0.01),
subject_prob=replace(subject_prob, subject_prob>1, 0.99)) #since I have generated values from a normal distribution, some values may lie outside of this range - using mutate() and replace() cleans the data by replacing values above 1 with 0.99 and below 0 with 0.01
#using the probability simulated above, we can generate each participant's performance on the 6 test trials. Since the participants need to choose between a man and a woman, we can treat each trial as a binomial event (use rbinom() to do this)
bian %>%
group_by(Gender, Age, Trait, n_trials,N,Simulations,subject_prob) %>%
do(data=rbinom(1, .$n_trials, .$subject_prob)/.$n_trials) %>%
unnest(data)
#ran a regression to test if there is a difference between age and gender groups, and if there is an effect of the interaction between age and condition on each simulation
summarized_bian = bian %>%
select(n_trials,N,Simulations,subject_prob,Gender,Age,Trait) %>%
group_by(Gender,Age,Trait,n_trials,N,Simulations,subject_prob) %>%
do(data=rbinom(1, .$n_trials, .$subject_prob)/.$n_trials) %>%
unnest(data) %>%
select(Gender,Age,Trait,N,data,Simulations,n_trials,subject_prob) %>%
group_by(N,Simulations) %>%
do(p_value = summary(lm(data~Gender+Age+Age:Trait, data=.))$coefficient[2,4]) %>%
unnest(p_value) %>%
mutate(significant=ifelse(p_value<0.05, 1, 0)) %>%
group_by(N) %>%
dplyr::summarise(N_sims=length(p_value), proportion_significant=sum(significant)/N_sims)
summarized_bian#ploted the probability of getting a significant result by the number of children tested.
ggplot(summarized_bian,
aes(x=N,
y=proportion_significant))+
xlab("Number of participants")+
ylab("Proportion of significant results")+
geom_line()
Figure 4: Statistical simulations of sample sizes necessary for 80% power, based on original effect sizes (including the interaction between age and trial condition)
2.1.4 For smaller sample sizes (with interaction)
Power analysis simulations including the interaction of trial condition and age (see code below) suggest that even a sample of 100 participants per condition cannot detect 20% power for effect sizes 75% as large as those found in the original study, as seen in Figure 5. This suggests that if the original paper overestimated effect sizes, the sample size should be well above 100 to have 80% power.
n_trials=6 #there were six trials in the original study
n_children=16 #16 children per condition
#took the means from the first simulation and made the differences between groups 75% as large as those in the original paper
m_5yo_g_s<-0.70
sd_5yo_g_s<-0.22
m_6yo_g_s<-0.65
sd_6yo_g_s<-0.23
m_7yo_g_s<-0.68
sd_7yo_g_s<-0.26
m_5yo_b_s<-0.70
sd_5yo_b_s<-0.21
m_6yo_b_s<-0.61
sd_6yo_b_s<-0.24
m_7yo_b_s<-0.65
sd_7yo_b_s<-0.25
m_5yo_g_n<-0.66
sd_5yo_g_n<-0.29
m_6yo_g_n<-0.60
sd_6yo_g_n<-0.25
m_7yo_g_n<-0.57
sd_7yo_g_n<-0.24
m_5yo_b_n<-0.65
sd_5yo_b_n<-0.31
m_6yo_b_n<-0.66
sd_6yo_b_n<-0.22
m_7yo_b_n<-0.57
sd_7yo_b_n<-0.23
#created a dataframe that crosses age, gender and trial condition (the variables of interest), and says that there are "N=n_children" participants per combination
bian=expand.grid(
n_trials=n_trials,
N=n_children,
Gender=c("Girl", "Boy"),
Age=c("5", "6", "7"),
Trait=c("Smart", "Nice"),
Simulations=1
)
#used the mutate function in dplyr to add the mean and SD of the relevant normal distributions to the dataframe. I used ifelse to subset bian into its different conditions (age, gender and trait) and then specify the mean and SD for these
#I have used dplyr:: in some lines below because certain functions (mutate, summarise, ... clash with functions from another package)
bian=bian %>%
mutate(mean=ifelse(Gender=="Girl"&Age=="5"&Trait=="Smart", m_5yo_g_s,
ifelse(Gender=="Girl"&Age=="6"&Trait=="Smart", m_6yo_g_s,
ifelse(Gender=="Girl"&Age=="7"&Trait=="Smart", m_7yo_g_s,
ifelse(Gender=="Girl"&Age=="5"&Trait=="Nice", m_5yo_g_n,
ifelse(Gender=="Girl"&Age=="6"&Trait=="Nice", m_6yo_g_n,
ifelse(Gender=="Girl"&Age=="7"&Trait=="Nice", m_7yo_g_n,
ifelse(Gender=="Boy"&Age=="5"&Trait=="Smart", m_5yo_b_s,
ifelse(Gender=="Boy"&Age=="6"&Trait=="Smart", m_6yo_b_s,
ifelse(Gender=="Boy"&Age=="7"&Trait=="Smart", m_7yo_b_s,
ifelse(Gender=="Boy"&Age=="5"&Trait=="Nice", m_5yo_b_n,
ifelse(Gender=="Boy"&Age=="6"&Trait=="Nice", m_6yo_b_n, m_7yo_b_n))))))))))),
sd=ifelse(Gender=="Girl"&Age=="5"&Trait=="Smart", sd_5yo_g_s,
ifelse(Gender=="Girl"&Age=="6"&Trait=="Smart", sd_6yo_g_s,
ifelse(Gender=="Girl"&Age=="7"&Trait=="Smart", sd_7yo_g_s,
ifelse(Gender=="Girl"&Age=="5"&Trait=="Nice", sd_5yo_g_n,
ifelse(Gender=="Girl"&Age=="6"&Trait=="Nice", sd_6yo_g_n,
ifelse(Gender=="Girl"&Age=="7"&Trait=="Nice", sd_7yo_g_n,
ifelse(Gender=="Boy"&Age=="5"&Trait=="Smart", sd_5yo_b_s,
ifelse(Gender=="Boy"&Age=="6"&Trait=="Smart", sd_6yo_b_s,
ifelse(Gender=="Boy"&Age=="7"&Trait=="Smart", sd_7yo_b_s,
ifelse(Gender=="Boy"&Age=="5"&Trait=="Nice", sd_5yo_b_n,
ifelse(Gender=="Boy"&Age=="6"&Trait=="Nice", sd_6yo_b_n, sd_7yo_b_n)))))))))))) %>%
group_by(Gender, Age, Trait, n_trials,N,Simulations) %>% #grouped the variables by gender, age, trait, number of trials, number of children and number of simulations
do(subject_prob = rnorm(.$N, .$mean, .$sd)) %>% #assigned each participant a simulated probability of choosing a male
unnest(subject_prob) %>%
dplyr::mutate(subject_prob=replace(subject_prob, subject_prob<0, 0.01),
subject_prob=replace(subject_prob, subject_prob>1, 0.99)) #since I have generated values from a normal distribution, some values may lie outside of this range - using mutate() and replace() cleans the data by replacing values above 1 with 0.99 and below 0 with 0.01
#using the probability simulated above, we can generate each participant's performance on the 6 test trials. Since the participants need to choose between a man and a woman, we can treat each trial as a binomial event (use rbinom() to do this)
bian=bian %>%
group_by(Gender,Age,Trait,n_trials,N,Simulations,subject_prob) %>%
do(data=rbinom(1, .$n_trials, .$subject_prob)/.$n_trials) %>% #data is the generated rbinom() divided by the number of trials to create a proportion
unnest(data)
print(bian)
#generated the mean and SD for each age, gender and trait groups
bian_means = bian %>%
group_by(Gender, Age,Trait) %>%
dplyr::summarize(Mean = mean(data), SE = sd(data)/sqrt(n()))
print(bian_means)
#created a graph showing these
ggplot(bian_means, aes(x=Gender, y=Mean, fill=Gender))+
geom_bar(stat="identity")+
geom_linerange(aes(ymin=Mean-SE, ymax=Mean+SE))+
facet_grid(Trait~Age)
#experimenting with trying to reproduce plot seen in paper
ggplot(bian_means, aes(x=Age, y=Mean, fill=Age, colour=Gender))+
geom_errorbar(aes(ymin=Mean-SE, ymax=Mean+SE))+
geom_line(aes(Gender))n_trials=6 #there were six trials in the original study
#took the means from the first simulation and made the differences between groups 75% as large as those in the original paper
m_5yo_g<-0.70
sd_5yo_g<-0.22
m_6yo_g<-0.65
sd_6yo_g<-0.23
m_7yo_g<-0.68
sd_7yo_g<-0.26
m_5yo_b<-0.70
sd_5yo_b<-0.21
m_6yo_b<-0.61
sd_6yo_b<-0.24
m_7yo_b<-0.65
sd_7yo_b<-0.25
m_5yo_g_n<-0.66
sd_5yo_g_n<-0.29
m_6yo_g_n<-0.60
sd_6yo_g_n<-0.25
m_7yo_g_n<-0.57
sd_7yo_g_n<-0.24
m_5yo_b_n<-0.65
sd_5yo_b_n<-0.31
m_6yo_b_n<-0.66
sd_6yo_b_n<-0.22
m_7yo_b_n<-0.57
sd_7yo_b_n<-0.23
#created a dataframe that crosses age, gender and trial condition (the variables of interest), and says that there are N=different numbers of children per condition varying from 10 to 100 (seq()) and that simulates 500 datasets
bian=expand.grid(
n_trials=n_trials,
N=seq(from=10,to=100, by=10),
Gender=c("Girl", "Boy"),
Age=c("5", "6", "7"),
Trait=c("Smart", "Nice"),
Simulations=1:500
)
#used the mutate function in dplyr to add the mean and SD of the relevant normal distributions to the dataframe. I used ifelse to subset bian into its different conditions (age, gender and trait) and then specify the mean and SD for these
#I have used dplyr:: in some lines below because certain functions (mutate, summarise, ... clash with functions from another package)
bian=bian %>%
mutate(mean=ifelse(Gender=="Girl"&Age=="5"&Trait=="Smart", m_5yo_g_s,
ifelse(Gender=="Girl"&Age=="6"&Trait=="Smart", m_6yo_g_s,
ifelse(Gender=="Girl"&Age=="7"&Trait=="Smart", m_7yo_g_s,
ifelse(Gender=="Girl"&Age=="5"&Trait=="Nice", m_5yo_g_n,
ifelse(Gender=="Girl"&Age=="6"&Trait=="Nice", m_6yo_g_n,
ifelse(Gender=="Girl"&Age=="7"&Trait=="Nice", m_7yo_g_n,
ifelse(Gender=="Boy"&Age=="5"&Trait=="Smart", m_5yo_b_s,
ifelse(Gender=="Boy"&Age=="6"&Trait=="Smart", m_6yo_b_s,
ifelse(Gender=="Boy"&Age=="7"&Trait=="Smart", m_7yo_b_s,
ifelse(Gender=="Boy"&Age=="5"&Trait=="Nice", m_5yo_b_n,
ifelse(Gender=="Boy"&Age=="6"&Trait=="Nice", m_6yo_b_n, m_7yo_b_n))))))))))),
sd=ifelse(Gender=="Girl"&Age=="5"&Trait=="Smart", sd_5yo_g_s,
ifelse(Gender=="Girl"&Age=="6"&Trait=="Smart", sd_6yo_g_s,
ifelse(Gender=="Girl"&Age=="7"&Trait=="Smart", sd_7yo_g_s,
ifelse(Gender=="Girl"&Age=="5"&Trait=="Nice", sd_5yo_g_n,
ifelse(Gender=="Girl"&Age=="6"&Trait=="Nice", sd_6yo_g_n,
ifelse(Gender=="Girl"&Age=="7"&Trait=="Nice", sd_7yo_g_n,
ifelse(Gender=="Boy"&Age=="5"&Trait=="Smart", sd_5yo_b_s,
ifelse(Gender=="Boy"&Age=="6"&Trait=="Smart", sd_6yo_b_s,
ifelse(Gender=="Boy"&Age=="7"&Trait=="Smart", sd_7yo_b_s,
ifelse(Gender=="Boy"&Age=="5"&Trait=="Nice", sd_5yo_b_n,
ifelse(Gender=="Boy"&Age=="6"&Trait=="Nice", sd_6yo_b_n, sd_7yo_b_n)))))))))))) %>%
group_by(Gender, Age, Trait, n_trials,N,Simulations) %>% #grouped the variables by gender, age, trait, number of trials, number of children and number of simulations
do(subject_prob = rnorm(.$N, .$mean, .$sd)) %>% #assigned each participant a simulated probability of choosing a male
unnest(subject_prob) %>%
dplyr::mutate(subject_prob=replace(subject_prob, subject_prob<0, 0.01),
subject_prob=replace(subject_prob, subject_prob>1, 0.99)) #since I have generated values from a normal distribution, some values may lie outside of this range - using mutate() and replace() cleans the data by replacing values above 1 with 0.99 and below 0 with 0.01
#using the probability simulated above, we can generate each participant's performance on the 6 test trials. Since the participants need to choose between a man and a woman, we can treat each trial as a binomial event (use rbinom() to do this)
bian %>%
group_by(Gender, Age,n_trials,N,Simulations,subject_prob) %>%
do(data=rbinom(1, .$n_trials, .$subject_prob)/.$n_trials) %>%
unnest(data)
#ran a regression to test if there is a difference between age and gender groups, and if there is an effect of the interaction between age and condition on each simulation
summarized_bian = bian %>%
select(n_trials,N,Simulations,subject_prob,Gender,Age,Trait) %>%
group_by(Gender,Age,Trait,n_trials,N,Simulations,subject_prob) %>%
do(data=rbinom(1, .$n_trials, .$subject_prob)/.$n_trials) %>%
unnest(data) %>%
select(Gender,Age,Trait,N,data,Simulations,n_trials,subject_prob) %>%
group_by(N,Simulations) %>%
do(p_value = summary(lm(data~Gender+Age+Age:Trait, data=.))$coefficient[2,4]) %>%
unnest(p_value) %>%
mutate(significant=ifelse(p_value<0.05, 1, 0)) %>%
group_by(N) %>%
dplyr::summarise(N_sims=length(p_value), proportion_significant=sum(significant)/N_sims)
summarized_bian#ploted the probability of getting a significant result by the number of children tested.
ggplot(summarized_bian,
aes(x=N,
y=proportion_significant))+
xlab("Number of participants")+
ylab("Proportion of significant results")+
geom_line()
Figure 5: Statistical simulations of sample sizes necessary for 80% power, based on effect sizes 75% as large as those in the original (including the interaction between age and trial condition)
2.2 Participants
This study aims to pilot the replication of Bian, Leslie and Cimpian (2017), and our final sample included five participants, two girls aged 8 and 10 and three adults aged 22 and 23 (two men and one woman). Unfortunately, due to a malfunction of the iPad application for one of the children, followed by a malfunction of the data saving, the data for the two child pilots was lost.
2.3 Materials and procedure
The materials and procedure in the original study were described in the supplementary material as such:
“The study began with a set of 12 screener questions designed to gauge whether children understand the meaning of the key terms”smart" (6 questions) and “nice” (6 questions). The “smart” and “nice” questions were presented to children as separate blocks whose order was counterbalanced. For each of these questions, the experimenter described the behavior of an unfamiliar child (e.g., “This child learns things fast”) and then asked participants whether the relevant trait term could be applied to this child (e.g., “Is this child smart, not smart, or are you not sure?”). In task (ii), children were shown 6 pictures one by one; each picture depicted two individuals. The first 2 trials served as practice trials, and the individuals depicted were all of the same gender as the participant. For the next 4 trials, the pictures consisted of a man and a woman. Children were told that one of the two people was “really, really smart” (on 3 of 6 trials) or “really, really nice” (on the other 3 trials), and they were asked to guess which of the two had the relevant trait. The order of the pictures was counterbalanced." (Bian, Lian & Cimpian, 2017).
We obtained the original stimuli from the authors, but made a few changes to the materials. First, we decided to use a touch screen webpage set up on an iPad as it is more child friendly. Secondly, we used different pictures depicting men and women as they fit the iPad set up better. Thirdly, although we kept the same screener questions, we only used six out of the twelve as the children were also participating in a replication of the third task of the first study, which used the remaining six screener questions. We split the questions evenly to keep descriptions of all the behaviours (nice, smart and neither smart nor nice). Finally, we recorded all the questions in order to reduce potential bias and variation between participants.
The procedure was the same and was coded in the webpage to run without input from the researchers, excluding any help needed by the children to understand the questions. Written informed consent was obtained from the children’s mother prior to the test session and assent from the children before running the iPad task. Children were tested individually in quiet rooms in the lab. The adult pilots were tested individually in their homes and provided consent through the webpage.
The iPad application can be found at the following links (code included below). The second link provides a version of the application where pictures are counterbalanced (so that pictures used for the smart conditions in the first version are used for the nice conditions in the second version, and so that pictures are swapped from the right to the left side and vice versa).
Version 1 Version 2 (counterbalanced)
<html>
<head>
<title>Replication Experiment V1</title>
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.1/jquery.min.js"></script>
<script src="jspsych-5.0.3/jspsych.js"></script>
<script src="jspsych-5.0.3/plugins/jspsych-text.js"></script>
<script src="jspsych-5.0.3/plugins/jspsych-button-response.js"></script>
<script src="jspsych-5.0.3/plugins/jspsych-survey-text.js"></script>
<script src="jspsych-5.0.3/plugins/jspsych-call-function.js"></script>
<link href="jspsych-5.0.3/css/jspsych.css" rel="stylesheet" type="text/css"></link>
</head>
<body>
<script>
function PlaySound () {
document.getElementById(play_stim);
play_stim.play();
};
//the playsound function enables us to use and play sound stimuli in our webpage
function saveData(filename, filedata) {
$.ajax({
type: 'post',
cache: false,
url: 'save_data.php',
data: {filename: filename, filedata: filedata}
})
};
//this uses the php script we created to save data to the group's server (the on finish function at the bottom has been changed to do so)
var id = prompt('Please enter the participant number');
//creates a dialogue box which prompts participant or researcher to put the participant's id in
var consent_block = {
stimulus: "<p>This research is being conducted by the students of PSYL11087 as part"+
" of their MSc degree. You have been asked to participate because you are"+
" a student on a PPLS MSc degree. The Instructor for this class, and supervisor"+
" for this project is Dr Hugh Rabagliati (hugh.rabagliati@ed.ac.uk)."+
" You can contact him by email at any time if you have questions.</p>" +
"<p> In this study, you will be asked to complete one of three tasks:</p>"+
"<ol>"+
"<li> You will listen to a set of spoken sentences embedded in background noise,"+
" and choose which of two pictures the sentence refers to.</li>"+
"<li> You will listen to a set of stories about children, and decide if they"+
" are smart or nice. Then, you will see some pictures of people and decide whether"+
" they look smart or nice.</li>" +
"<li> You will see pictures of people, and match those pictures up to either other pictures"+
" of objects or to specific words.</li></ol></p>"+
"<p></p>"+
"<p></p>"+
"<p>The only information collected will be your responses on this task, and your gender."+
" No identifying infomation about you will be collected. Participating in this study does "+
"not confer either benefits or risks. The study will take less than 10 minutes and "+
" there is no monetary compensation. You may stop participation at any time. This data will"+
" be analyzed and used as a coursework submission for the class PSYL11087.</p>"+
"<p> If you consent to participate, press the Consent button below. If you do not consent"+
" you may end the study or press the decline button.</p>",
is_html: true,
choices:['Consent','Decline'],
type: "button-response",
timing_post_trial: 4000 //this leaves 4000 milliseconds after the button press to end or continue the trial
}; //consent block for adult participants
//creates an empty variable called code which is used below
var code;
//form which allows participants to enter they're age, language and gender.
var survey_trial = {
type: 'survey-text',
preamble: 'Subject number is ' + id,
questions: ["How old is your participant in years?", "What language do they speak at home",
"What gender is your participant?"],
rows: [5,3],
columns: [40,50],
on_finish: function(data){
var responses = JSON.parse(data.responses);
code = responses.Q2; //this saves the responses to the gender question in the variable code, to be used in the conditional function for the same-gender test trials (further down)
}
};
//creates text block which displays instructions for the participant
var instructions_block = {
type: 'button-response',
is_html: true,
choices: ['Start game'], //this displays a button for the participant to press to start
stimulus: "<p> In this game, you will first listen to Alison tell you about children she knows.</p>" +
"<p> You can guess which ones are smart and which ones are nice. </p>" +
"<p> Touch the play button to listen to Alison, then touch the smiley face if you think they are smart or nice, the sad face if you think they aren't, and the confused face if you are not sure. </p>" +
"<p> Next, pictures of people will appear on the screen. </p>" +
"<p> Touch the play button to listen to Alison describing these people. </p>" +
"<p> Alison will tell you you can guess which person looks really really nice or really really smart. </p>" +
"<p> Touch the picture you think Alison is describing. </p>"
};
//this augmented code includes all 6 screening questions and their stimulus, with two pictures side by side and an audio button (made possible by the playSound function)
//the addition of the Key:Value pair with the Key data adds information to the final data file (here on which condition the question belongs to)
var screening_q = [
{
choices: ['Smart', 'Not smart', "I don't know"],
button_html: ['<img src="img/smiley1.png" height="250px" width="250px"/>','<img src="img/smiley2.jpg" height="250px" width="250px"/>','<img src="img/smiley3.jpg" height="250px" width="250px"/>'],
stimulus: '<audio src="audio/q1_s.wav" id="play_stim"></audio>'+'<center><button id="audio_button" onclick="PlaySound()" class="jspsych-btn" style="height:150px;width:150px">Play!</button></center>',
data: {
condition:'smart'
}
},
{
choices: ['Smart', 'Not smart', "I don't know"],
button_html: ['<img src="img/smiley1.png" height="250px" width="250px"/>','<img src="img/smiley2.jpg" height="250px" width="250px"/>','<img src="img/smiley3.jpg" height="250px" width="250px"/>'],
stimulus: '<audio src="audio/q2_s.wav" id="play_stim"></audio>'+'<center><button id="audio_button" onclick="PlaySound()" class="jspsych-btn" style="height:150px;width:150px">Play!</button></center>',
data: {
condition:'smart'
}
},
{
choices: ['Smart', 'Not smart', "I don't know"],
button_html: ['<img src="img/smiley1.png" height="250px" width="250px"/>','<img src="img/smiley2.jpg" height="250px" width="250px"/>','<img src="img/smiley3.jpg" height="250px" width="250px"/>'],
stimulus: '<audio src="audio/q5_s.wav" id="play_stim"></audio>'+'<center><button id="audio_button" onclick="PlaySound()" class="jspsych-btn" style="height:150px;width:150px">Play!</button></center>',
data: {
condition:'not sure'
}
},
{
choices: ['Nice', 'Not nice', "I don't know"],
button_html: ['<img src="img/smiley1.png" height="250px" width="250px"/>','<img src="img/smiley2.jpg" height="250px" width="250px"/>','<img src="img/smiley3.jpg" height="250px" width="250px"/>'],
stimulus: '<audio src="audio/q1_n.wav" id="play_stim"></audio>'+'<center><button id="audio_button" onclick="PlaySound()" class="jspsych-btn" style="height:150px;width:150px">Play!</button></center>',
data: {
condition:'nice'
}
},
{
choices: ['Nice', 'Not nice', "I don't know"],
button_html: ['<img src="img/smiley1.png" height="250px" width="250px"/>','<img src="img/smiley2.jpg" height="250px" width="250px"/>','<img src="img/smiley3.jpg" height="250px" width="250px"/>'],
stimulus: '<audio src="audio/q2_n.wav" id="play_stim"></audio>'+'<center><button id="audio_button" onclick="PlaySound()" class="jspsych-btn" style="height:150px;width:150px">Play!</button></center>',
data: {
condition:'nice'
}
},
{
choices: ['Nice', 'Not nice', "I don't know"],
button_html: ['<img src="img/smiley1.png" height="250px" width="250px"/>','<img src="img/smiley2.jpg" height="250px" width="250px"/>','<img src="img/smiley3.jpg" height="250px" width="250px"/>'],
stimulus: '<audio src="audio/q5_n.wav" id="play_stim"></audio>'+'<center><button id="audio_button" onclick="PlaySound()" class="jspsych-btn" style="height:150px;width:150px">Play!</button></center>',
data: {
condition:' not sure'
}
}
];
//jsPsych.randomization.shuffle produces a random ordering of screening_q (randomly plays nice or smart stimuli) - this is then used below as the timeline for the screening trial.
var screening_q_timeline = jsPsych.randomization.shuffle(screening_q);
//this makes the screening trial into a button response trial
var screening = {
type: 'button-response',
is_html: true,
timeline: screening_q_timeline
};
//this augmented code includes both test trials for male participants and their stimulus, with two pictures side by side and an audio button (made possible by the playSound function)
//the addition of the Key:Value pair with the Key data adds information to the final data file (here on which condition the stimulus belongs to)
//the code that follows allows the test trials to run only if the same gender is entered in the survey
var test_run_m = [
{
choices: ['male','male'],
button_html: ['<img src="img/manA.jpg" height="250px" width="250px"/>','<img src="img/manB.png" height="250px" width="270px"/>'],
stimulus: '<audio src="audio/smart.wav" id="play_stim"></audio>'+'<center><button id="audio_button" onclick="PlaySound()" class="jspsych-btn" style="height:150px;width:150px">Play!</button></center>',
data: {
condition:'practice_smart'
},
},
{
choices: ['male', 'male'],
button_html: ['<img src="img/manC.jpg" height="250px" width="250px"/>', '<img src="img/manD.jpg" height="250px" width="250px"/>'],
stimulus: '<audio src="audio/nice.wav" id="play_stim"></audio>'+'<center><button id="audio_button" onclick="PlaySound()" class="jspsych-btn" style="height:150px;width:150px">Play!</button></center>',
data: {
condition: 'practice_nice'
},
}
];
//jsPsych.randomization.shuffle produces a random ordering of test_run_m (randomly plays nice or smart stimuli) - this is then used below as the timeline for the male test trial
var test_run_m_timeline = jsPsych.randomization.shuffle(test_run_m);
//this makes the male test trial into a button response trial
var test_trial_m = {
type: 'button-response',
is_html: true,
timeline: test_run_m_timeline
};
//this is a conditional function that uses the button response trial above as a timeline, and runs the male test trial only if code=Male, otherwise it does not run
var if_male = {
timeline: [test_trial_m],
conditional_function: function(){
if(code == "Male") {
return true;
} else {
return false;
}
}
};
//this augmented code includes both test trials for female participants and their stimulus, with two pictures side by side and an audio button (made possible by the playSound function)
//the addition of the Key:Value pair with the Key data adds information to the final data file (here on which condition the stimulus belongs to)
//the code that follows allows the test trials to run only if the same gender is entered in the survey
var test_run_f = [
{choices: ['female', 'female'],
button_html: ['<img src="img/womanA.png" height="250px" width="250px"/>', '<img src="img/womanB.png" height="250px" width="250px"/>'],
stimulus: '<audio src="audio/smart.wav" id="play_stim"></audio>'+'<center><button id="audio_button" onclick="PlaySound()" class="jspsych-btn" style="height:150px;width:150px">Play!</button></center>',
data: {
condition: 'practice_smart'
},
},
{
choices: ['female', 'female'],
button_html: ['<img src="img/womanC.jpg" height="250px" width="250px"/>', '<img src="img/womanD.jpg" height="250px" width="250px"/>'],
stimulus: '<audio src="audio/nice.wav" id="play_stim"></audio>'+'<center><button id="audio_button" onclick="PlaySound()" class="jspsych-btn" style="height:150px;width:150px">Play!</button></center>',
data: {
condition: 'practice_nice'
},
}
];
//jsPsych.randomization.shuffle produces a random ordering of test_run_f (randomly plays nice or smart stimuli) - this is then used below as the timeline for the female test trial.
var test_run_f_timeline = jsPsych.randomization.shuffle(test_run_f);
//this makes the female test trial into a button response trial
var test_trial_f = {
type: 'button-response',
is_html: true,
timeline: test_run_f_timeline
};
//this is a conditional function that uses the button response trial above as a timeline, and runs the female test trial only if code=Female, otherwise it does not run
var if_female = {
timeline: [test_trial_f],
conditional_function: function(){
if(code == "Female") {
return true;
} else {
return false;
}
}
};
//this augmented code includes the 4 trials and their stimulus, with two pictures side by side and an audio button (made possible by the playSound function)
//the addition of the Key:Value pair with the Key data adds information to the final data file (here on which condition the stimulus belongs to)
var button_choices = [
{
choices: ['male','female'],
button_html: ['<img src="img/man1.png" height="270px" width="215px"/>','<img src="img/woman1.jpg" height="270px" width="215px"/>'],
stimulus: '<audio src="audio/smart.wav" id="play_stim"></audio>'+'<center><button id="audio_button" onclick="PlaySound()" class="jspsych-btn" style="height:150px;width:150px">Play!</button></center>',
data: {
condition:'smart'
},
},
{
choices: ['male', 'female'],
button_html: ['<img src="img/man2.jpg" height="270px" width="215px"/>', '<img src="img/woman2.jpg" height="270px" width="215px"/>'],
stimulus: '<audio src="audio/nice.wav" id="play_stim"></audio>'+'<center><button id="audio_button" onclick="PlaySound()" class="jspsych-btn" style="height:150px;width:150px">Play!</button></center>',
data: {
condition: 'nice'
},
},
{
choices: ['female', 'male'],
button_html: ['<img src="img/woman3.jpg" height="250px" width="250px"/>', '<img src="img/man3.jpg" height="250px" width="250px"/>'],
stimulus: '<audio src="audio/smart.wav" id="play_stim"></audio>'+'<center><button id="audio_button" onclick="PlaySound()" class="jspsych-btn" style="height:150px;width:150px">Play!</button></center>',
data: {
condition: 'smart'
},
},
{
choices: ['female', 'male'],
button_html: ['<img src="img/woman4.jpg" height="250px" width="250px"/>', '<img src="img/man4.png" height="250px" width="250px"/>'],
stimulus: '<audio src="audio/nice.wav" id="play_stim"></audio>'+'<center><button id="audio_button" onclick="PlaySound()" class="jspsych-btn" style="height:150px;width:150px">Play!</button></center>',
data: {
condition: 'nice'
},
}
];
//jsPsych.randomization.shuffle produces a random ordering of button_choices (randomly shows different pairs of men and women) - this is then used below as the timeline for the real trials
var button_choices_timeline = jsPsych.randomization.shuffle(button_choices);
//this makes the trials into button response trials
var real_trial = {
type: 'button-response',
is_html:true,
timeline: button_choices_timeline
};
//this is a text trial which displays text
var goodbye_trial = {
type: 'text',
text: 'Thanks for your help!'
};
//this makes sure that whenever a trial is saved, the participant ID is saved with it
jsPsych.data.addProperties({subject: id});
//jsPsych.init() tells the website to show the information contained within it to the website users
//the timeline indicates the order in which the plugins will be shown on the webpage
jsPsych.init({
timeline: [consent_block, survey_trial, instructions_block, screening, if_male, if_female, real_trial, goodbye_trial],
on_finish: function() {
saveData(id + ".csv", jsPsych.data.dataAsCSV()); //this function saves a .csv file named after the participant's id
}
})
</script>
</body>
</html>
//this code is identical to the one above, except for the 4 'real' trials, for which I have switched the conditions each picture was on in the first version (if a pair of pictures was in a smart condition in V1, it is now in a nice condition). I also flipped the images (if a picture was on the right in V1 it is now on the left)
<html>
<head>
<title>Replication Experiment V2</title>
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.1/jquery.min.js"></script>
<script src="jspsych-5.0.3/jspsych.js"></script>
<script src="jspsych-5.0.3/plugins/jspsych-text.js"></script>
<script src="jspsych-5.0.3/plugins/jspsych-button-response.js"></script>
<script src="jspsych-5.0.3/plugins/jspsych-survey-text.js"></script>
<script src="jspsych-5.0.3/plugins/jspsych-call-function.js"></script>
<link href="jspsych-5.0.3/css/jspsych.css" rel="stylesheet" type="text/css"></link>
</head>
<body>
<script>
function PlaySound () {
document.getElementById(play_stim);
play_stim.play();
};
function saveData(filename, filedata) {
$.ajax({
type: 'post',
cache: false,
url: 'save_data.php', // this is the path to the PHP script
data: {filename: filename, filedata: filedata}
})
};
var id = prompt('Please enter the participant number');
var consent_block = {
stimulus: "<p>This research is being conducted by the students of PSYL11087 as part"+
" of their MSc degree. You have been asked to participate because you are"+
" a student on a PPLS MSc degree. The Instructor for this class, and supervisor"+
" for this project is Dr Hugh Rabagliati (hugh.rabagliati@ed.ac.uk)."+
" You can contact him by email at any time if you have questions.</p>" +
"<p> In this study, you will be asked to complete one of three tasks:</p>"+
"<ol>"+
"<li> You will listen to a set of spoken sentences embedded in background noise,"+
" and choose which of two pictures the sentence refers to.</li>"+
"<li> You will listen to a set of stories about children, and decide if they"+
" are smart or nice. Then, you will see some pictures of people and decide whether"+
" they look smart or nice.</li>" +
"<li> You will see pictures of people, and match those pictures up to either other pictures"+
" of objects or to specific words.</li></ol></p>"+
"<p></p>"+
"<p></p>"+
"<p>The only information collected will be your responses on this task, and your gender."+
" No identifying infomation about you will be collected. Participating in this study does "+
"not confer either benefits or risks. The study will take less than 10 minutes and "+
" there is no monetary compensation. You may stop participation at any time. This data will"+
" be analyzed and used as a coursework submission for the class PSYL11087.</p>"+
"<p> If you consent to participate, press the Consent button below. If you do not consent"+
" you may end the study or press the decline button.</p>",
is_html: true,
choices:['Consent','Decline'],
type: "button-response",
timing_post_trial: 4000
};
var code;
var survey_trial = {
type: 'survey-text',
preamble: 'Subject number is ' + id,
questions: ["How old is your participant in years?", "What language do they speak at home",
"What gender is your participant?"],
rows: [5,3],
columns: [40,50],
on_finish: function(data){
var responses = JSON.parse(data.responses);
code = responses.Q2;
}
};
var instructions_block = {
type: 'button-response',
is_html: true,
choices: ['Start game'],
stimulus: "<p> In this game, you will first listen to Alison tell you about children she knows.</p>" +
"<p> You can guess which ones are smart and which ones are nice. </p>" +
"<p> Touch the play button to listen to Alison, then touch the smiley face if you think they are smart or nice, the sad face if you think they aren't, and the confused face if you are not sure. </p>" +
"<p> Next, pictures of people will appear on the screen. </p>" +
"<p> Touch the play button to listen to Alison describing these people. </p>" +
"<p> Alison will tell you you can guess which person looks really really nice or really really smart. </p>" +
"<p> Touch the picture you think Alison is describing. </p>"
};
var screening_q = [
{
choices: ['Smart', 'Not smart', "I don't know"],
button_html: ['<img src="img/smiley1.png" height="250px" width="250px"/>','<img src="img/smiley2.jpg" height="250px" width="250px"/>','<img src="img/smiley3.jpg" height="250px" width="250px"/>'],
stimulus: '<audio src="audio/q1_s.wav" id="play_stim"></audio>'+'<center><button id="audio_button" onclick="PlaySound()" class="jspsych-btn" style="height:150px;width:150px">Play!</button></center>',
data: {
condition:'smart'
}
},
{
choices: ['Smart', 'Not smart', "I don't know"],
button_html: ['<img src="img/smiley1.png" height="250px" width="250px"/>','<img src="img/smiley2.jpg" height="250px" width="250px"/>','<img src="img/smiley3.jpg" height="250px" width="250px"/>'],
stimulus: '<audio src="audio/q2_s.wav" id="play_stim"></audio>'+'<center><button id="audio_button" onclick="PlaySound()" class="jspsych-btn" style="height:150px;width:150px">Play!</button></center>',
data: {
condition:'smart'
}
},
{
choices: ['Smart', 'Not smart', "I don't know"],
button_html: ['<img src="img/smiley1.png" height="250px" width="250px"/>','<img src="img/smiley2.jpg" height="250px" width="250px"/>','<img src="img/smiley3.jpg" height="250px" width="250px"/>'],
stimulus: '<audio src="audio/q5_s.wav" id="play_stim"></audio>'+'<center><button id="audio_button" onclick="PlaySound()" class="jspsych-btn" style="height:150px;width:150px">Play!</button></center>',
data: {
condition:'not sure'
}
},
{
choices: ['Nice', 'Not nice', "I don't know"],
button_html: ['<img src="img/smiley1.png" height="250px" width="250px"/>','<img src="img/smiley2.jpg" height="250px" width="250px"/>','<img src="img/smiley3.jpg" height="250px" width="250px"/>'],
stimulus: '<audio src="audio/q1_n.wav" id="play_stim"></audio>'+'<center><button id="audio_button" onclick="PlaySound()" class="jspsych-btn" style="height:150px;width:150px">Play!</button></center>',
data: {
condition:'nice'
}
},
{
choices: ['Nice', 'Not nice', "I don't know"],
button_html: ['<img src="img/smiley1.png" height="250px" width="250px"/>','<img src="img/smiley2.jpg" height="250px" width="250px"/>','<img src="img/smiley3.jpg" height="250px" width="250px"/>'],
stimulus: '<audio src="audio/q2_n.wav" id="play_stim"></audio>'+'<center><button id="audio_button" onclick="PlaySound()" class="jspsych-btn" style="height:150px;width:150px">Play!</button></center>',
data: {
condition:'nice'
}
},
{
choices: ['Nice', 'Not nice', "I don't know"],
button_html: ['<img src="img/smiley1.png" height="250px" width="250px"/>','<img src="img/smiley2.jpg" height="250px" width="250px"/>','<img src="img/smiley3.jpg" height="250px" width="250px"/>'],
stimulus: '<audio src="audio/q5_n.wav" id="play_stim"></audio>'+'<center><button id="audio_button" onclick="PlaySound()" class="jspsych-btn" style="height:150px;width:150px">Play!</button></center>',
data: {
condition:' not sure'
}
}
];
var screening_q_timeline = jsPsych.randomization.shuffle(screening_q);
var screening = {
type: 'button-response',
is_html: true,
timeline: screening_q_timeline
};
var test_run_m = [
{
choices: ['male','male'],
button_html: ['<img src="img/manA.jpg" height="250px" width="250px"/>','<img src="img/manB.png" height="250px" width="250px"/>'],
stimulus: '<audio src="audio/smart.wav" id="play_stim"></audio>'+'<center><button id="audio_button" onclick="PlaySound()" class="jspsych-btn" style="height:150px;width:150px">Play!</button></center>',
data: {
condition:'practice_smart'
},
},
{
choices: ['male', 'male'],
button_html: ['<img src="img/manC.jpg" height="250px" width="250px"/>', '<img src="img/manD.jpg" height="250px" width="250px"/>'],
stimulus: '<audio src="audio/nice.wav" id="play_stim"></audio>'+'<center><button id="audio_button" onclick="PlaySound()" class="jspsych-btn" style="height:150px;width:150px">Play!</button></center>',
data: {
condition: 'practice_nice'
},
}
];
var test_run_m_timeline = jsPsych.randomization.shuffle(test_run_m);
var test_trial_m = {
type: 'button-response',
is_html: true,
timeline: test_run_m_timeline
};
var if_male = {
timeline: [test_trial_m],
conditional_function: function(){
if(code == "Male") {
return true;
} else {
return false;
}
}
};
var test_run_f = [
{choices: ['female', 'female'],
button_html: ['<img src="img/womanA.png" height="250px" width="250px"/>', '<img src="img/womanB.png" height="250px" width="250px"/>'],
stimulus: '<audio src="audio/smart.wav" id="play_stim"></audio>'+'<center><button id="audio_button" onclick="PlaySound()" class="jspsych-btn" style="height:150px;width:150px">Play!</button></center>',
data: {
condition: 'practice_smart'
},
},
{
choices: ['female', 'female'],
button_html: ['<img src="img/womanC.jpg" height="250px" width="250px"/>', '<img src="img/womanD.jpg" height="250px" width="250px"/>'],
stimulus: '<audio src="audio/nice.wav" id="play_stim"></audio>'+'<center><button id="audio_button" onclick="PlaySound()" class="jspsych-btn" style="height:150px;width:150px">Play!</button></center>',
data: {
condition: 'practice_nice'
},
}
];
var test_run_f_timeline = jsPsych.randomization.shuffle(test_run_f);
var test_trial_f = {
type: 'button-response',
is_html: true,
timeline: test_run_f_timeline
};
var if_female = {
timeline: [test_trial_f],
conditional_function: function(){
if(code == "Female") {
return true;
} else {
return false;
}
}
};
var button_choices = [
{
choices: ['female','male'],
button_html: ['<img src="img/woman1.jpg" height="250px" width="250px"/>','<img src="img/man1.png" height="250px" width="250px"/>'],
stimulus: '<audio src="audio/nice.wav" id="play_stim"></audio>'+'<center><button id="audio_button" onclick="PlaySound()" class="jspsych-btn" style="height:150px;width:150px">Play!</button></center>',
data: {
condition:'nice'
},
},
{
choices: ['female', 'male'],
button_html: ['<img src="img/woman2.jpg" height="250px" width="250px"/>', '<img src="img/man2.jpg" height="250px" width="250px"/>'],
stimulus: '<audio src="audio/smart.wav" id="play_stim"></audio>'+'<center><button id="audio_button" onclick="PlaySound()" class="jspsych-btn" style="height:150px;width:150px">Play!</button></center>',
data: {
condition: 'smart'
},
},
{
choices: ['male', 'female'],
button_html: ['<img src="img/man3.jpg" height="250px" width="250px"/>', '<img src="img/woman3.jpg" height="250px" width="250px"/>'],
stimulus: '<audio src="audio/nice.wav" id="play_stim"></audio>'+'<center><button id="audio_button" onclick="PlaySound()" class="jspsych-btn" style="height:150px;width:150px">Play!</button></center>',
data: {
condition: 'nice'
},
},
{
choices: ['male', 'female'],
button_html: ['<img src="img/man4.png" height="250px" width="250px"/>', '<img src="img/woman4.jpg" height="250px" width="250px"/>'],
stimulus: '<audio src="audio/smart.wav" id="play_stim"></audio>'+'<center><button id="audio_button" onclick="PlaySound()" class="jspsych-btn" style="height:150px;width:150px">Play!</button></center>',
data: {
condition: 'smart'
},
}
];
var button_choices_timeline = jsPsych.randomization.shuffle(button_choices);
var real_trial = {
type: 'button-response',
is_html:true,
timeline: button_choices_timeline
};
var goodbye_trial = {
type: 'text',
text: 'Thanks for your help!'
};
jsPsych.data.addProperties({subject: id});
jsPsych.init({
timeline: [consent_block, survey_trial, instructions_block, screening, if_male, if_female, real_trial, goodbye_trial],
on_finish: function() {
saveData(id + ".csv", jsPsych.data.dataAsCSV());
}
})
</script>
</body>
</html>2.4 Analysis plan
The authors used a multilevel mixed-effects linear model to analyse children’s gender stereotype scores (across the three tasks of Study 1), with age (5 vs 6 vs 7 years old), gender (boy vs girl) and trait (‘smart’ vs ‘nice’) as predictors (along with possible interaction terms). A further model treated 6 and 7 years old as a single group, and further tests compared boys’ and girls’ stereotype scores about both traits separately for younger and older children. The same analyses would therefore be run on the data collected in the replication following this pilot study.
3 Results
3.1 Adult pilots
As Figure 3 shows (see code below), we can observe a difference in means between the adult men and woman tested. Whereas the men were equally likely to pick the man or the woman for the smart condition (M=0.50), the woman did not rate the women smart at all.
#importing and binding data sets
library(readr)
library(dplyr)
library(ggplot2)
library(tidyr)
library(readr)
library(wordbankr)
library(dbplyr)
library(lsr)
data1 <- read_csv("C:/Replication_experiment/Lorna/3.csv")
data2 <- read_csv("C:/Replication_experiment/Lorna/4.csv")
data3 <- read_csv("C:/Replication_experiment/Lorna/5.csv")
data1$score <- ifelse(data1$choices=="male", 1, 0)
data2$score <- ifelse(data2$choices=="male", 1, 0)
data3$score <- ifelse(data3$choices=="female", 1, 0)
#created a tibble that combines the first participant's answers to the form and their subject number, then used gsub() to remove the noise characters, then augmented the code to split the responses into different columns
subj_info1=tibble(ptcpt_info = data1$responses[2],
subject = data1$subject[1]) %>%
mutate(ptcpt_info = gsub('[{}"]', "", ptcpt_info)) %>%
mutate(KeyValPairs = strsplit(as.character(ptcpt_info),",")) %>%
unnest(KeyValPairs) %>%
separate(KeyValPairs, into = c("Question", "Answer"), ":") %>%
select (-ptcpt_info) %>%
spread(key = Question, value = Answer)
#used left_join() to join the new tibble with the data set, created a new column with the stereotype scores (where choosing male = 1), then used filter() to keep the conditions of interest (smart and nice), then created another two columns with the mean scores and SEs
data1 =left_join(data1, subj_info1) %>%
mutate(stereo.score = ifelse(grepl('male', choices), 1, 0)) %>%
filter(is.na(condition)==F) %>%
filter(condition != "not sure") %>%
filter(condition != "practice_smart") %>%
filter(condition != "practice_nice") %>%
group_by(subject, condition) %>%
dplyr::summarise(stereo.mean=mean(stereo.score), SE = sd(stereo.score)/sqrt(n()))
#plotted the data using ggplot
ggplot(data = data1, aes(x=condition, y=stereo.mean))+
geom_bar(stat="identity")
#created a tibble that combines the second participant's answers to the form and their subject number, then used gsub() to remove the noise characters, then augmented the code to split the responses into different columns
subj_info2=tibble(ptcpt_info = data2$responses[2],
subject = data2$subject[1]) %>%
mutate(ptcpt_info = gsub('[{}"]', "", ptcpt_info)) %>%
mutate(KeyValPairs = strsplit(as.character(ptcpt_info),",")) %>%
unnest(KeyValPairs) %>%
separate(KeyValPairs, into = c("Question", "Answer"), ":") %>%
select (-ptcpt_info) %>%
spread(key = Question, value = Answer)
#used left_join() to join the new tibble with the data set, created a new column with the stereotype scores (where choosing male = 1), then used filter() to keep the conditions of interest (smart and nice), then created another two columns with the mean scores and SEs
data2 =left_join(data2, subj_info2) %>%
mutate(stereo.score = ifelse(grepl('male', choices), 1, 0)) %>%
filter(is.na(condition)==F) %>%
filter(condition != "not sure") %>%
filter(condition != "practice_smart") %>%
filter(condition != "practice_nice") %>%
group_by(subject, condition) %>%
dplyr::summarise(stereo.mean=mean(stereo.score), SE = sd(stereo.score)/sqrt(n()))
#plotted the data using ggplot
ggplot(data = data2, aes(x=condition, y=stereo.mean))+
geom_bar(stat="identity")
#created a tibble that combines the third participant's answers to the form and their subject number, then used gsub() to remove the noise characters, then augmented the code to split the responses into different columns
subj_info3=tibble(ptcpt_info = data3$responses[2],
subject = data3$subject[1]) %>%
mutate(ptcpt_info = gsub('[{}"]', "", ptcpt_info)) %>%
mutate(KeyValPairs = strsplit(as.character(ptcpt_info),",")) %>%
unnest(KeyValPairs) %>%
separate(KeyValPairs, into = c("Question", "Answer"), ":") %>%
select (-ptcpt_info) %>%
spread(key = Question, value = Answer)
#used left_join() to join the new tibble with the data set, created a new column with the stereotype scores (where choosing male = 1), then used filter() to keep the conditions of interest (smart and nice), then created another two columns with the mean scores and SEs
data3 =left_join(data3, subj_info3) %>%
mutate(stereo.score = ifelse(grepl('female', choices), 1, 0)) %>%
filter(is.na(condition)==F) %>%
filter(condition != "not sure") %>%
filter(condition != "practice_smart") %>%
filter(condition != "practice_nice") %>%
group_by(subject, condition) %>%
dplyr::summarise(stereo.mean=mean(stereo.score), SE = sd(stereo.score)/sqrt(n()))
#plotted the data using ggplot
ggplot(data = data3, aes(x=condition, y=stereo.mean))+
geom_bar(stat="identity")
#used full_join to be able to join all 3 datasets into one
dataset <- full_join(data1, data2)
dataset <- full_join(dataset, data3)
dataset$gender <- "Male" #created a column for the gender
dataset[c(5,6), 5] <- "Female" #renamed the two rows that were for the female participant
dataset$age <- "22" #created a column for age
dataset[c(1,2), 6] <- "23" #renamed the two rows that were for the 23 year old participant
dataset$age <- as.integer(dataset$age) #changed the age variable from character to integer
# plotted the final dataset to show the means and error bars for both ages and both genders - lower bar shows that they say smart less for their own gender
ggplot(dataset, aes(x=gender, y=stereo.mean, fill=gender))+
scale_fill_discrete(name="Gender", labels=c("Female", "Male"))+
xlab("Gender")+
ylab("Mean stereotype score") +
geom_bar(stat="identity")+
geom_linerange(aes(ymin=stereo.mean-SE, ymax=stereo.mean+SE))+
facet_grid(condition~age)
Figure 6: Mean stereotype scores for adult pilots, faceted by age of participants and by trial condition (smart or nice).
3.2 Child pilots
As the data was lost for the two child participants, no results can be reported. However, the child for whom the iPad experiment worked found the study interesting and running the study with children provided some helpful information on our experiment and replication effort.
Firstly, the task of deciding whether a man and a woman was smart or nice seemed confusing for the children (and for the adults too). This is especially problematic for children as they may turn to their parents for help, which could then influence the data. This also suggests the researchers need to help child participants more in order for them to understand and engage with the task. On the other hand, we observed that the children had no problem understanding the procedure for the screener questions, and overall they found the task short and easy.
4 Discussion
As discussed in the Materials and procedure section, we used some materials that were different than the original study. We believe using an iPad application improves the original design by making it more child friendly and more engaging. Moreover, the pictures we used seemed better matched than the ones used in the original paper, although this was not tested. Even so, some adult participants commented on the pictures, which suggests better material that is more thoroughly selected is needed.
The fact that we recorded all the audio stimuli means our design is possibly less likely to be influenced by experimenter effects such as demand characteristics and therefore improved from the original procedure.
Our pilot study is necessarily less sensitive as we had a very small sample size compared to the original study and to the power analysis.
5 References
Bian, L., Leslie, S. J., & Cimpian, A. (2017). Gender stereotypes about intellectual ability emerge early and influence children’s interests. Science, 355(6323), 389-391.
Cvencek, D., Meltzoff, A. N., & Greenwald, A. G. (2011). Math-gender stereotypes in elementary school children. Child development, 82(3), 766-779.
Martinot, D., Bagès, C., & Désert, M. (2012). French children’s awareness of gender stereotypes about mathematics and reading: When girls improve their reputation in math. Sex Roles, 66(3-4), 210-219.