Replication Project - Final Report
Mark Somerville
12 December 2018
1 Introduction
Gendered stereotypes of intellectual ability are very common and are thought to shape the career goals and interests of men and women. A common stereotype assumes that a high level of intellectual ability is more commonly possessed by men rather than women, and this stereotype has often been presented as a partial explanation for the gender disparity in many scientific fields. Negative gendered stereotypes are thought to not only undermine women’s interests in certain fields, but also impair their performance in these domains (Ambady, Shih, Kim, & Pitinsky, 2001; Galdi, Cadinu, Tomasetto, 2014).
Previous research on gender stereotypes have focused predominantly on adult subjects, however cultural stereotypes are likely to be influential throughout development. Research suggests that children develop the ability to label gender groups and recognise their own gender between 18 and 24 months (Martin & Ruble, 2010). Children first begin to acquire and understand basic gender stereotypes by around 30 months, and knowledge of stereotypes increases rapidly between ages of 3 to 5 years, influencing children’s interests and behaviours (Martin, Ruble, & Szkrybalo, 2002; Martin & Ruble, 2010; Miller, Lurye, Zosuls, & Ruble, 2009).
However, little is known about the acquisition of specific gender stereotypes relating to intellectual ability as most research in this area has been conducted on adults. As children are aware of and endorse gendered stereotypes from a young age, and gendered stereotypes specifically related to intellectual ability are common among adults, children may acquire these stereotypes early in development. Throughout schooling, children and young adults make many choices directing their academic interests and future career aspirations. If stereotypes are acquired and endorsed early, they may significantly restrict access and influence girls’ interests in certain fields long before they reach adulthood. As such, it is important to investigate the acquisition and endorsement of these stereotypes in young children
Addressing this gap in the research literature, Bian, Leslia, and Cimpian (2017) report the results of four studies which investigated the acquisition and developmental trajectories of gender stereotypes of cognitive ability and their relation to children’s interests in children aged 5, 6 and 7-years-old
2 Original Study - Bian, Leslia, & Cimpian (2017)
2.1 Studies 1 & 2
Study 1 involved 96 children, aged 5 (mean age = 5.55 years), 6 (mean age = 6.50 years) and 7 (mean age = 7.44 years). There were 48 girls and 48 boys and 32 children in each age group.
In study 1, three stereotype tasks were administered and children were measured on the proportion of trials in which they associated intellectual ability with their own gender.
In task (i) children were told a brief story about a person who was “really, really smart” or “really, really kind”, with the gender of the person left unspecified. The experimenter then laid out 4 photos - 2 adult males, 2 adult females - and asked the child which one of the 4 people might be the character in the story.
In task (ii) children were shown pairs of photos - one adult male, one adult female. They were told that one of the people was “really, really smart” (on half of the trials) or “really, really nice” (on the other half of the trials. The children were then asked to guess which photo the descriptor applied to.
In task (iii) children were asked to complete novel puzzles in which they had to assign 4 puzzle pieces to photos of 4 adults - 2 adult males and 2 adult females. The puzzle pieces consisted of the word “smart”, the word “nice”, a picture of a high-heeled shoe (stereotypically feminine) and a picture of a hammer (stereotypically masculine). The children were given the pieces one by one and asked to match each piece to one of the photos of men and women.
Study 2 involved 144 children aged 5 (mean age = 5.50 years), 6 (mean age = 6.48 years), and 7 (mean age = 7.45 years). There were 72 girls and 72 boys and 48 children in each age group.
Study 2 replicated tasks (i) and (ii) from study 1 with a larger sample, and included both child and adult targets during tasks.
Across the tasks in both studies, two major measurements were recorded. The first was the proportion of trials in which children linked intellectual ability to people of their own gender. These proportions were then averaged to calculate their association of brilliance (intellectual ability) with their own gender. If girls associated brilliance with the male figures in the tasks rather than the females, this was taken as evidence of these children endorsing the stereotype of intellectual ability with males rather than females. Secondly, the number of trials in which children associated either men or women with the descriptor of being “really, really nice” was also recorded.
In both studies there was no statistically significant difference in the association of brilliance with the children’s own gender at age 5, however at ages 6 and 7 girls were significantly less likely than boys to associate their own gender with brilliance. These results suggest that the acquisition of gender stereotypes regarding intellectual ability occurs by age 6. See figure 1.
Figure 1 - Results of Studies 1 (A & B) & 2 (C & D), Boys Scores in Blue, Girls Scores in Red
2.2 Study 3
2.2.1 Methods
Study 3 involved 64 children aged 6 (mean age = 6.52 years) and 7 (mean age = 7.50 years). There were 32 girls and 32 boys, and 32 children in each age group.
To investigate whether gendered beliefs about intelligence relate to children’s interests, the children were introduced to two novel games named “zarky” and “impok”. For each game the experimenter showed a picture and briefly described the game (see figure 2), with one game said to be for children who are “really, really smart” and the other for children who “try really, really hard”. The order was counterbalanced and each game was presented in the “smart” category for half of the participants and the “try hard” category for the other half. After describing the games the experimenter asked the children to recall the relevant information (whether the game was for “smart” children or children who “try hard”) before moving on.
The experimenter proceeded to ask the children four questions to assess to measure their level of interest in the games with the order of the questions being randomized. After answering two questions the children were reminded of the relevant information. Responses to the questions were standardized and then averaged to give a measure of the children’s interest in both types of game.
Children’s gender stereotypes were assessed with task (i) of studies one and two, the gender-neutral story task. Before selecting who the story was about from the selection of photos of men and women, the children were additionally asked to repeat the story while the researcher coded the number of gender pronouns used. The children’s final own-gender-brilliance score was the average of these two scores.
Figure 2 - The Games Used to Assess Children’s Interests in Studies 3 & 4
2.2.2 Results
Girls were found to be less interested than boys in the game for “smart” children (Wald ϲ - 4.02m p = 0.045) but not in the game for children who “try hard” (Wald ϲ = 0.53, p = 0.47). See table 1 and figure 3.
Table 1 - Boys and Girls Interest Scores
| Age | Gender | Smart Game | Try-Hard Game |
|---|---|---|---|
| 6 Years Old | Boys | 0.2 (0.71) | -0.09 (0.81) |
| Girls | -0.17 (0.77) | 0.1 (0.49) | |
| 7 Years Old | Boys | 0.15 (0.69) | -0.03 (0.9) |
| Girls | -0.21 (0.88) | 0.04 (0.58) |
—
Girls’ own-gender brilliance scores (their associations of intelligence with their own gender in the tasks from study 1) were lower than boys’ (t = 2.4, p - 0.02). A product-of-coefficients mediation test was used to investigate the relationship between children’s gender, their stereotype beliefs, and interest in the games. Stereotype beliefs were found to mediate the relationship between gender and interest in the game for “smart” children versus interest in the game for children who “try hard”: indirect effect = -0.11, 95% confidence interval = [-0.33, =0.004]. See figure 4.
Figure 4 - The difference between boys and girls in their interest toward the smart vs. the try-hard game was mediated by their own-gender brilliance scores
2.3 Study 4
Study 4 involved 96 children aged 5 (mean age = 5.4 years) and 6 (mean age = 6.52 years). There were 48 girls and 48 boys, and 48 children in each age group.
The procedure for study 4 was identical to that of study 3, except only the “smart” game was used. The interests of 5 and 6-year-old boys and girls in the game for “smart” children were compared.
No significant gender differences were found in the 5-years-old age group (Wald ϲ = 0.55, p = 0.45). In the 6-years-old age group non-significant trend was found for boys to express more interest in the game for “smart” children (Wald ϲ = 3.66, p = 0.056). Comparing the combined sample of 6 and 7-year-olds from studies 3 and 4, the magnitude of difference in boys’ versus girls’ interest in the game was estimated to be d= 0.51, 95% confidence interval = [0.13, 0.88], p = 0.008.
2.4 Conclusion
The studies suggest that between the age of 5 and 6 years old many children acquire the stereotype that intelligence is a male quality. Specifically, 6-year-old girls are less likley than boys to associate intelligence with their own gender. Additionally, this acquired stereotype begins to influence children’s interests, with young girls less likely to be interested in a game said to be for smart children.
3 P Curve
3.1 Study Selection Rule
Studies included in the p-curve investigated the implicit and explicit endorsement of gender stereotypes about intellectual ability in children aged 5 - 10 years old. As ability in mathematics is stereotyped as requiring ‘brilliance’, mathematics ability is used as a proxy for intellectual ability.
3.2 P Curve Disclosure Table
3.3 P Curve
3.4 P Curve Conclusion
Both the full p-curve and half p-curve analysis indicate the selected studies demonstrate evidential value. The combined test is robust to ambitious p-hacking and so the analysis suggests p-hacking did not occur in the selected studies.
The p-curve additionally suggests that the selected studies do not inidicate evidential value which is inadequate or absent.
The conventional level of significance is p = 0.05. Of the 8 statistical tests included in the analysis, 7 report values of p < 0.02, while one value is around p = 0.04. The p-curve thus indicates the presence of a real effect.
4 Relication Project
4.1 Introduction
This study is a partial replication of experiment 3 in Bian, Leslia, and Cimpian (2017). The study measured the interest of children and adults in two novel games. One game was said to be for “really, really smart” children, and the second was said to be for children who “try really, really hard”.
4.2 Method
4.2.1 Participants
The participants in the pilot study were 2 children, aged 8 and 11 years (both female). A control group 8 adults (5 males, 3 females) were additionally recruited.
The children’s mother provided signed written consent and the children were also asked to assent orally.
4.2.2 Design
Prior to the experiment beginning, participants were assigned to one of 4 conditions. The conditions differed on two points. Firstly, which of the two novel games the participants viewed first. Secondly, which of the two games was presented in the ‘smart’ format, and which was presented in the ‘try hard’ format.
4.2.3 Materials & Stimuli
The experiment was conducted on a touch-screen iPad held between the experiment and participant. All instructions, stimuli, and questions were presented on the iPad.
The pictures and descriptions of the two novel games were the exact same as those used in the original experiment (see figure 2).
The following four questions were used to assess children’s interest in the games. The questions were exactly the same as those used in the original study.
- Imagine I had the Zarky/Impok game right here, in front of you. Would you want to play the Zarky/Impok game, or would you not want to play it?
- Do you like the Zarky/Impok game, or do you not like it?
- Imagine you are playing the Zarky/Impok game. Would playing the Zarky/Impok game make you feel happy or sad?
- If you had the chance to do something tomorrow, would you play the Zarky/Impok game?
Participants answered the questions by responding to a likert scale consisting of 6 faces representing varying degrees of happiness. Two of the questions utilised the full 6-point scale, one question used an abbreviated 4-point scale, and one question used only 2 points. See figure 5.
Figure 5 - Response Scale used to Assess Interest in the Two Novel Games
4.2.4 Procedure
Participants were first asked to enter their gender information and age on the iPad.
The likert scale and its response options were then explained to the participants. Participants answered two simple screener questions to ensure they understood the scale.
Participants were introduced to one of the two novel games ‘Zarky’ or ‘Impok’). The participant was shown the picture of the game and the written explanation of the game and its ruleset was displayed on the iPad. The experimenter read out the full description and ruleset of the game to the child and asked whether they understood how the game worked. The crucial piece of information which differed between the games was whether the game was for children who were ‘really, really smart’ or children who ‘try really, really hard’. The importance of this was emphasised to the participant and participants were asked to remember this.
The experimenter then read out the four questions used to assess the child’s interest in the game. The child responded to the questions by selecting the face which most accurately represented their response to the question. The questions were asked and responded to sequentially, with subsequent questions appearing after the previous question was answered. The order of the questions was randomized.
After the first two questions the experimenter asked the participant to recall the crucial piece of information about the game (whether it was for ‘really smart’ children or children who ‘try really, really hard’). Participants were then reminded of this piece of information if they could not recall.
After answering the four questions, participants were then introduced to the second novel game and the procedure was repeated.
The order of the two games and whether the first game was for ‘really, really smart’ children or children who ‘try really, really hard’ was counterbalanced across participants.
Two interest scores were calculated for each participant for both games by combining response to the four questions. As the four questions used different scales, responses to each question were standardised before they were averaged.
Following completion of the experiment, participants were debriefed and the purpose of the experiment was explained.
4.3 Analysis
4.3.1 Child Pilot Data
For the pilot study, data was collected from 2 children (ages 8.7 years and 11.5 years, both females). Gender was not considered as a variable as only data was only collected from female participants. The overall interest scores in the two novel games was obtained by summing the answers to the individual questions. As the different questions used different scales, standardised scores were then obtained through z-score standardisation. As a sample size of 2 is insufficient for statistical analysis, only descriptive statistics will be reported.
The two children demonstrated different attitudes towards the two games for smart and hardworking children. The older child, aged 11.5 years, showed a preference for the game for Smart children. The younger child, aged 8.7 years, showed a preference for the game for Hardworking children. These results are displayed in Figure 6.
data_child <- rbind(
read.csv("A1_child.csv"),
read.csv("D1_child.csv")
)
data_child = data_child %>%
dplyr::filter(condition != "") %>%
dplyr::select(subject,button_pressed,condition)
data_child$condition <- gsub('[,Q1234]', "", data_child$condition)
gender = data_child %>%
dplyr::filter(condition=="gender information")
gender$button_pressed <- replace(gender$button_pressed, gender$button_pressed==1, "girl")
gender$button_pressed <- replace(gender$button_pressed, gender$button_pressed==0, "boy")
score_smart_a1 = data_child %>%
dplyr::filter(condition=="zarky smart " & subject=="A1_child")
score_hard_a1 = data_child %>%
dplyr::filter(condition=="impok hardworking " & subject=="A1_child")
score_smart_d1 = data_child %>%
dplyr::filter(condition=="zarky smart " & subject=="D1_child")
score_hard_d1 = data_child %>%
dplyr::filter(condition=="impok hardworking " & subject=="D1_child")
game_condition <- data.frame(game_condition = c("smart", "smart", "hard", "hard"))
score <- data.frame(score = c(sum(score_smart_a1$button_pressed)+4,
sum(score_smart_d1$button_pressed)+4,
sum(score_hard_a1$button_pressed)+4,
sum(score_hard_d1$button_pressed)+4))
subject <- data.frame(subject = c("A1_child", "D1_child", "A1_child", "D1_child"))
gender <- data.frame(gender = c(gender$button_pressed, gender$button_pressed))
age <- data.frame(age = c("8.7", "11.5", "8.7", "11.5"))
result_child <- cbind(subject, gender, age, game_condition, score )
result_child = result_child %>%
dplyr::mutate(interest_score = scale(result_child$score))
knitr::kable(result_child)| subject | gender | age | game_condition | score | interest_score |
|---|---|---|---|---|---|
| A1_child | girl | 8.7 | smart | 10 | -1.3 |
| D1_child | girl | 11.5 | smart | 16 | 1.1 |
| A1_child | girl | 8.7 | hard | 14 | 0.3 |
| D1_child | girl | 11.5 | hard | 13 | -0.1 |
levels(result_child$game_condition)[levels(result_child$game_condition)=="hard"] <- "Hardworking"
levels(result_child$game_condition)[levels(result_child$game_condition)=="smart"] <- "Smart"
names(result_child)[names(result_child)=="game_condition"] <- "Game"
levels(result_child$age)[levels(result_child$age)=="8.7"] <- "8.7 Years Old"
levels(result_child$age)[levels(result_child$age)=="11.5"] <- "11.5 Years Old"
ggplot(result_child,
aes(x = Game,
y = interest_score))+
geom_bar(aes(fill = Game), stat = "identity")+
geom_text(aes(label=interest_score))+
geom_hline(aes(yintercept=0),linetype=5)+
labs(title = "Figure. 6: Child Data",
x="Game Type",
y="Interest Score")+
theme(legend.background = element_rect(fill="gray90", size=1, linetype="dotted"))+
facet_grid(.~age)
These results cannot be meaningfully compared to the results of the original experiment due to the small sample size, different age range, and lack of a male sample. However, it is interesting to note that the older girl showed a stronger interest in the game for smart children than the younger girl. As the original experiment demonstrated that gender stereotypes are acquired by age 6 years, this may suggest that the influence of this stereotype subsequently grows weaker as children get older. However, due to the small sample size this is purely speculation.
4.3.2 Adult Pilot Data
Data was additionally collected from 8 adults (5 males, 3 females) with an age range of 22 to 24 years. Interest scores were obtained for adults using the same method as for the child data. As in the child data, there was an insufficient sample size for statistical analysis, and so only descriptive statistics will be reported.
The adult data contradicts the results from the original study. Both men and women showed stronger interest in the game for Smart people, and women showed less interest in the game for Hardworking people than men. These results are displayed in Figure 7.
data_adult <- rbind(
read.csv("A1_adult.csv"),
read.csv("A2_adult.csv"),
read.csv("B1_adult.csv"),
read.csv("B2_adult.csv"),
read.csv("C1_adult.csv"),
read.csv("C2_adult.csv"),
read.csv("D1_adult.csv"),
read.csv("D2_adult.csv")
)
data_adult = data_adult %>%
dplyr::filter(condition != "") %>%
dplyr::select(subject,button_pressed,condition)
data_adult$condition <- gsub('[,Q1234]', "", data_adult$condition)
gender_block = data_adult %>%
dplyr::filter(condition=="gender information") %>%
dplyr::mutate(gender = NULL)
gender_block$gender <- replace(gender_block$gender, gender_block$button_pressed==1, "female")
gender_block$gender <- replace(gender_block$gender, gender_block$button_pressed==0, "male")
gender_block = gender_block %>%
dplyr::select(subject, gender) %>%
dplyr::group_by(subject, gender)
data_adult$condition <- replace(data_adult$condition, data_adult$condition=="zarky smart ", "smart")
data_adult$condition <- replace(data_adult$condition, data_adult$condition=="impok smart ", "smart")
data_adult$condition <- replace(data_adult$condition, data_adult$condition=="zarky hardworking ", "hard")
data_adult$condition <- replace(data_adult$condition, data_adult$condition=="impok hardworking ", "hard")
subject_type <- as.matrix(unique(data_adult$subject))
n=length(subject_type)
score <- function(x){
subject = data_adult %>%
dplyr::filter(subject==subject_type[x])
smart = subject %>%
dplyr::filter(condition=="smart")
score_smart = sum(smart$button_pressed) + 4
hard = subject %>%
dplyr::filter(condition=="hard")
score_hard = sum(hard$button_pressed) + 4
subject_score<- data.frame(subject=c(subject_type[x],subject_type[x]),game_condition=c("smart","hard"), score=c(score_smart,score_hard))
return(subject_score)
}
result<-score(1)
i=2
while(i<=n){
result=rbind(result,score(i))
i<-i+1
}
adult_result = merge(gender_block,result, by = "subject")
adult_result = adult_result %>%
dplyr::mutate(interest_score = scale(adult_result$score))
final_result = adult_result %>%
dplyr::select(gender,game_condition,interest_score) %>%
dplyr::group_by(gender,game_condition) %>%
dplyr::summarise(mean_interest_score = mean(interest_score, na.rm =T))
knitr::kable(final_result)| gender | game_condition | mean_interest_score |
|---|---|---|
| female | hard | -0.7896746 |
| female | smart | 0.4136391 |
| male | hard | -0.1278521 |
| male | smart | 0.3534734 |
levels(final_result$game_condition)[levels(final_result$game_condition)=="hard"] <- "Hardworking"
levels(final_result $game_condition)[levels(final_result$game_condition)=="smart"] <- "Smart"
names(final_result)[names(final_result)=="game_condition"] <- "Game"
ggplot(final_result,
aes(x = Game,
y = mean_interest_score,
fill = Game
)) +
geom_bar(aes(fill = Game), stat="identity")+
geom_text(aes(label=round(mean_interest_score,digits=2)))+
geom_hline(aes(yintercept=0),linetype=5)+
labs(title="Figure 7 - Adult Results", x ="Game Type", y ="Interest Score", fill = "Game Type", label = "mean_interest_score")+
theme(legend.background = element_rect(fill="gray90", size=1, linetype="dotted"))+
facet_grid(~gender)
While results from adult participants cannot be meaningfully compared to results obtained from children in the original experiment, the contradiciton is perhaps noteworthy. In the original study girls showed less interest in the game for smart children than they did for the game for hardworking children. The results found for adults in this replication may be in part due to sampling bias. The participants were primarily postgraduate students from the University of Edinburgh, enrolled in academic fields which value intellectual ability. Accordingly, it may be less likely that the adult women in this study would endorse traditonal gender stereotypes regarding intelligence.
4.4 Java Script Code
The experiment was programmed using Javascript and can be found at the following link:
http://hrexp.ppls.ed.ac.uk/class/replication/Final.html
The complete annotated code for the experiment can be found in the appendix.
5 Discussion
5.1 Differences in the Replication Study
The major difference in the replication of the experiment to the original was in the formatting of the questions assessing participants’ interest in the games and the response scales available.
The original study used a two-stage format for 3 of the 4 questions. Participants were first asked to choose from two options, and then asked to choose from additional options based on their initial choice to derive a more precise answer. For example, participants were asked to respond ‘Yes’ or ‘No’ to the question ‘Do you like the Zarky game or do you not like it?’. Participants were then asked the question a second time and given an additional 3 options to choose from based on their response to the initial question. The response options available to participants were simple worded statements.
The question format and scales used in the original experiment may potentially introduce response bias in a number of ways. First, there is an element of forced choice in the initial stage of the questions as participants are forced to choose imprecisely from an insufficient number of options. This may produce a bias as participants without strong feelings are forced to select an answer which does not accurately reflect their feelings (Choi & Pak, 2005). Secondly, participants are not aware of all possible response options, and so are not able to act on the full information and provide the most accurate of responses. This may again bias participants to select an answer not reflecting their true feelings. Third, repeating questions may bias respondents away from their initial answer by introducing a social desirability bias. Evidence suggests that young children may be particularly vulnerable to a social desirability bias (Crandall & Crandall, 1965), making this a key methodological issue for the original study and replication. Participants who initially responded negatively to the first-stage of a question may subsequently chose the least negative option in the second-stage to avoid seeming harsh/mean. Similarly, participants who initially responded positively may feel pressured to again chose the most positive of options in the second-stage of the question. Research additionally suggests an inconsistent concordance of responses to Yes/No questions to questions using alternative response formats (Mellor & Moore, 2013). By combining multiple response formats within a single question, the original study introduces an additional biasing element which may produce inaccurate responses.
Considering the potential bias induced by the formatting of the original questions, in the replication the two-stage formatting of questions was abandoned. Participants were instead asked each question once, and all possible responses were available to participants. This new formatting reduces potential for bias and results in more accurate responses from participants.
Rather than using worded statements as response options, the replication experiment used a ‘Smiley Face’ Likert scale considered to be more child-friendly. Children have been found to prefer Likert scales to alternative response formats (Laerhoven, Zaag-Loonen, & Derkx, 2004) and the ‘Smiley Face’ Likert scale has been used in a variety of research contexts with children. Research suggests that scales which use emotions from ‘Very Happy’ to ‘Very Unhappy’ induce a social desirability bias, resulting in children only selecting ‘Happy’ responses and not making full use of the scale (Hall, Hume, & Tazzyman, 2016). To achieve full use of the rating scales by children, predominantly happy faces should be used. Accordingly, the study utilised a novel ‘Smiley Face’ Likert scale in which the majority of faces on the scale depicted happiness, and lowest point of the scale depicted a neutral face. The novel scale used in the replication should increase the sensitivity of the rating scale by encouraging full use of all available options, resulting in more accurate responses by children free of the social desirability bias.
5.2 Future Considerations
Based on the pilot replication, a number of issues may be considered when conducting a full replication of the original study.
An issue raised by both of the child participants is that they found it difficult to fully understand the novel games. In particular, the ‘Impok’ game seemed particularly difficult for children to understand. The younger child mentioned thinking she had figured out the ‘Impok’ game after testing had completed, suggesting that she needed additional time to understand the game. This is particularly concerning as the children in the pilot replication were older than in the original study. It may be that the pictures of the games and brief explanations are insufficient for children to grasp how the games work. If this is the case, the children’s responses to the questions assessing their interests may be bias by how well they actually understood the two games. Future replications should consider a more comprehensive explanation of the novel games, or utilising more simple game designs which can be intuitively understood by children. A pilot study should perhaps be run prior to data collection to ensure that the novel games are understandable to the target demographic.
The response scales to the questions assessing interest in the novel games may also need to be revised. The 4 questions utilised 3 different scales (6 points, 4 points, and 2 points) which may have caused unnecessary confusion for participants. The use of different scales also needlessly complicates data analysis. Subsequent studies should use consistent scales, as there appears to be no benefit to The children appeared to demonstrate a preference for neutral responses, but as all the scales were even number an exact neutral response was not permitted. Children appeared to spend most of their time deciding between the two most neutral responses, and so additional options on the scale may permit more accurate reporting of interest. Additionally, the preference for neutral responses perhaps suggests that the novel games did not elicit strong preferences in the children. While this may be due to a simple absence of any effect of the smart versus hardworking style of game, it could also suggest that the experiment was not sufficiently engaging to children. Boredom or fatigue may bias children towards neutral responses. Future pilot studies may thus benefit from consulting with children to ensure that the stimuli and experimental design is more child friendly and fun.
The preference for the game for Smart children exhibited by the older girl in the pilot results also provides direction for future studies. While no conclusions can be drawn from such a limited sample size, the preliminary results suggests that the developmental trajectory of endorsement of gender stereotypes may not be as linear as suggested in the original study. Older children, having had more exposure both to traditional social norms which drive stereotypes as well as alternative messages which contradict prevailing stereotypes may show a reduction in stereotype endorsement compared to younger children. To properly investigate this, future studies should adopt a longitudinal approach wherein gender streotype endorsement and it’s relaitonship with interests is measured at multiple time-points. The developmental trajectory of stereotype endorsement can then be identified using group-based latent trajectory modeling. This approach allows for subgroups characterised by distinct developmental pathways to be identified, and the risk factors for increasing stereotype endorsement could subsequently be identified.
6 References
Ambady, N., Shih, M., Kim, A., & Pittinsky, T. (2001). Stereotype Susceptibility in Children: Effects of Identity Activation on Quantitative Performance. Psychological Science, 12(5), 385-390. https://doi.org/10.1111/1467-9280.00371
Bian, L., Leslie, S. J., & Cimpian, A. (2017). Gender stereotypes about intellectual ability emerge early and influence children’s interests. Science (New York, N.Y.), 355(6323), 389-391. https://doi.org/10.1126/science.aah6524
Choi, B. C. K., & Pak, Anita, W. P. (2005). A Catalog of Biases in Questionaires. Preventing Chronic Disease, 2(1), 1-13. https://doi.org/A13
Crandall, V. C., Crandall, V. J., & Katkovsky, W. (1965). A children’s social desirability questionnaire. Journal of Consulting Psychology, 29(1), 27-36. https://doi.org/10.1037/h0020966
Cvencek, D., Kapur, M., & Meltzoff, A. N. (2015). Math achievement, stereotypes, and math self-concepts among elementary-school students in Singapore. Learning and Instruction, 39, 1-10. https://doi.org/10.1016/j.learninstruc.2015.04.002
Cvencek, D., Meltzoff, A. N., & Greenwald, A. G. (2011). Math-Gender Stereotypes in Elementary School Children. Child Development, 82(3), 766-779. https://doi.org/10.1111/j.1467-8624.2010.01529.x
Cvencek, D., Meltzoff, A. N., & Kapur, M. (2014). Cognitive consistency and math-gender stereotypes in Singaporean children. Journal of Experimental Child Psychology, 117(1), 73-91. https://doi.org/10.1016/j.jecp.2013.07.018
Galdi, S., Cadinu, M., & Tomasetto, C. (2014). The Roots of Stereotype Threat: When Automatic Associations Disrupt Girls’ Math Performance. Child Development, 85(1), 250-263. https://doi.org/10.1111/cdev.12128
Hall, L., Hume, C., & Tazzyman, S. (2016). Five degrees of happiness: effective smiley face Likert Scales for evaluating with children. Proceedings of the The 15th International Conference on Interaction Design and Children - IDC ’16, 311-321. https://doi.org/10.1145/2930674.2930719
Martin, C. L., & Ruble, D. N. (2010). Patterns of Gender Development. Annual Reviews of Psychology, 61, 353-181. https://doi.org/10.1146/annurev.psych.093008.100511
Martin, C. L., Ruble, D. N., & Szkrybalo, J. (2002). Cognitive theories of early gender development. Psychological Bulletin, 128(6), 903-933. https://doi.org/10.1037/0033-2909.128.6.903
Mellor, D., & Moore, K. A. (2014). Mellor, & Moore (2013) The use of Likert scales with children. Journal of Paediatric Psychology, 39(3), 369-379.
Miller, C. F., Lurye, L. E., Zosuls, K. M., & Ruble, D. N. (2009). Accessibility of Gender Stereotype Domains: Developmental and Gender Differences in Children. Sex Roles, 60(11-12), 870-881. https://doi.org/10.1007/s11199-009-9584-x.Accessibility
Passolunghi, M. C., Rueda Ferreira, T. I., & Tomasetto, C. (2014). Math-gender stereotypes and math-related beliefs in childhood and early adolescence. Learning and Individual Differences, 34, 70-76. https://doi.org/10.1016/j.lindif.2014.05.005
Van Laerhoven, H., Van Der Zaag-Loonen, H. J., & Derkx, B. H. F. (2004). A comparison of Likert scale and visual analogue scales as response options in children’s questionnaires. Acta Paediatrica, International Journal of Paediatrics, 93(6), 830-835. https://doi.org/10.1080/08035250410026572
7 Appendix
7.1 Annotated Javascript Code
<html>
<head>
<title>Replication</title>
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.1/jquery.min.js"></script>
<script src="jspsych-5.0.3/jspsych.js"></script>
<link href="jspsych-5.0.3/css/jspsych.css" rel="stylesheet" type="text/css"></link>
<script src="jspsych-5.0.3/plugins/jspsych-text.js"></script>
<script src="jspsych-5.0.3/plugins/jspsych-instructions.js"></script>
<script src="jspsych-5.0.3/plugins/jspsych-survey-likert.js"></script>
<script src="jspsych-5.0.3/plugins/jspsych-survey-text.js"></script>
<script src="jspsych-5.0.3/plugins/jspsych-button-response.js"></script>
<script src="jspsych-5.0.3/plugins/jspsych-call-function.js"></script>
<link href="jspsych-5.0.3/css/jspsych.css" rel="stylesheet" type="text/css"></link>
<style>
body {font-family: "Comic Sans MS", cursive, sans-serif;}
</style>
</head>
<body>
# 4 buttons are visible upon starting the experiment - the experimenter should select one of the buttons to assign the participant to one of 4 conditions (The conditions are whether Zarky or Impok is assigned to the Smart or Hardworking descriptor, and whether participants see the Zarky or Impok game first)
To design 4 buttons to jump to different trails with different conditions.
<button onclick="Zarky_smart_Impok_hard()">trial 1</button>
<button onclick="Zarky_hard_Impok_smart()">trial 2</button>
<button onclick="Impok_smart_Zarky_hard()">trial 3</button>
<button onclick="Impok_hard_Zarky_smart()">trial 4</button>
<script>
# Participant ID and Function for data saving.
var id = prompt('Please enter the participant number');
function saveData(filename, filedata){
$.ajax({
type:'post',
cache: false,
url: 'save_data.php',
data: {filename: filename, filedata: filedata}
});
}
var Introduction = {
type: 'instructions',
pages: [
'Welcome to the experiment! In this experiment we are going to tell you about a new game and ask you some questions. In the questions you will click the face that bests fits your answer for the question. Click next so we can answer some practice questions'
],
show_clickable_nav: true
}
var Finish = {
type: 'instructions',
pages: [
'Thank you for your participation.'
],
show_clickable_nav: true
}
# Descriptions and pictures of the games
var description_zarky_smart = {
type: 'instructions',
pages: [
'<p>I want to tell you about this game that I ask children to play sometimes.</p>',
'<img src="img/Zarky.png"></img>' +
'<p> It is called Zarky, and it is a lot of fun.</p>',
'<img src="img/Zarky.png"></img>' +
'<p> In this game, what you have to do is to bring the red pieces from this side to this side, one piece at a time, without going in a stright line and without getting them stuck in between the blue pieces.</p>',
'<img src="img/Zarky.png"></img>' +
'<p> Oh, and here is something else about the Zarky game, and this is important, so make sure you are paying attention.</p>',
'<img src="img/Zarky.png"></img>' +
'<p> This game is not for everyone. It is only for children who are really, really smart. Only smart children can be good at this game.</p>',
],
show_clickable_nav: true
}
var description_zarky_hard = {
type: 'instructions',
pages: [
'<p>I want to tell you about this game that I ask children to play sometimes.</p>',
'<img src="img/Zarky.png"></img>' +
'<p> It is called Zarky, and it is a lot of fun.</p>',
'<img src="img/Zarky.png"></img>' +
'<p> In this game, what you have to do is to bring the red pieces from this side to this side, one piece at a time, without going in a stright line and without getting them stuck in between the blue pieces.</p>',
'<img src="img/Zarky.png"></img>' +
'<p> Oh, and here is something else about the Zarky game, and this is important, so make sure you are paying attention.</p>',
'<img src="img/Zarky.png"></img>' +
'<p> This game is not for everyone. It is only for children who try really, really hard. Only hardworking children can be good at this game.</p>',
],
show_clickable_nav: true
}
var description_impok_smart = {
type: 'instructions',
pages: [
'<p>I want to tell you about this game that I ask children to play sometimes.</p>',
'<img src="img/Impok.png"></img>' +
'<p> It is called Impok, and it is a lot of fun.</p>',
'<img src="img/Impok.png"></img>' +
'<p> In this game, what you have to do is to figure out how to get the big pyramids next to each other in the black squares and get the small pyramids next to each other in the white squares in only ten moves and without crossing the grey squares.</p>',
'<img src="img/Impok.png"></img>' +
'<p> Oh, and here is something else about the Impok game, and this is important, so make sure you are paying attention.</p>',
'<img src="img/Impok.png"></img>' +
'<p> This game is not for everyone. It is only for children who are really, really smart. Only smart children can be good at this game.</p>',
],
show_clickable_nav: true
}
var description_impok_hard = {
type: 'instructions',
pages: [
'<p>I want to tell you about this game that I ask children to play sometimes.</p>',
'<img src="img/Impok.png"></img>' +
'<p> It is called Impok, and it is a lot of fun.</p>',
'<img src="img/Impok.png"></img>' +
'<p> In this game, what you have to do is to figure out how to get the big pyramids next to each other in the black squares and get the small pyramids next to each other in the white squares in only ten moves and without crossing the grey squares.</p>',
'<img src="img/Impok.png"></img>' +
'<p> Oh, and here is something else about the Impok game, and this is important, so make sure you are paying attention.</p>',
'<img src="img/Impok.png"></img>' +
'<p> This game is not for everyone. It is only for children who try really, really hard. Only hardworking children can be good at this game.</p>',
],
show_clickable_nav: true
};
# The 4 questions used to assess interest in the game - each question uses a different version of the same scale
var Zarky_Question1 = {
type: 'button-response',
prompt: "<p>Imagine I Had the Zarky game right here, in front of you. Would you want to play the Zarky game, or would you not want to play it?</p>",
choices: [
'4', '3', '2', '1'
],
button_html: [
'<img src="figures/EmojiScalePoint1.png" height="100px" width="100px"/>',
'<img src="figures/EmojiScalePoint2.png" height="100px" width="100px"/>',
'<img src="figures/EmojiScalePoint3.png" height="100px" width="100px"/>',
'<img src="figures/EmojiScalePoint4.png" height="100px" width="100px"/>'
],
data: {
condition: 'zarky, smart, Q1'
},
};
var Impok_Question1 = {
type: 'button-response',
prompt: "<p>Imagine I Had the Impok game right here, in front of you. Would you want to play the Impok game, or would you not want to play it?</p>",
choices: [
'4', '3', '2', '1'
],
button_html: [
'<img src="figures/EmojiScalePoint1.png" height="100px" width="100px"/>',
'<img src="figures/EmojiScalePoint2.png" height="100px" width="100px"/>',
'<img src="figures/EmojiScalePoint3.png" height="100px" width="100px"/>',
'<img src="figures/EmojiScalePoint4.png" height="100px" width="100px"/>'
],
data: {
condition: 'impok, hardworking, Q1'
},
};
var Zarky_Question2 = {
type: 'button-response',
prompt: "<p>Do you like the Zarky game, or do you not like it?</p>",
choices: [
'6', '5', '4', '3', '2', '1'
],
button_html: [
'<img src="figures/EmojiScalePoint1.png" height="100px" width="100px"/>',
'<img src="figures/EmojiScalePoint2.png" height="100px" width="100px"/>',
'<img src="figures/EmojiScalePoint3.png" height="100px" width="100px"/>',
'<img src="figures/EmojiScalePoint4.png" height="100px" width="100px"/>',
'<img src="figures/EmojiScalePoint5.png" height="100px" width="100px"/>',
'<img src="figures/EmojiScalePoint6.png" height="100px" width="100px"/>'
],
data: {
condition: 'zarky, smart, Q2'
},
};
var Impok_Question2 = {
type: 'button-response',
prompt: "<p>Do you like the Impok game, or do you not like it?</p>",
choices: [
'6', '5', '4', '3', '2', '1'
],
button_html: [
'<img src="figures/EmojiScalePoint1.png" height="100px" width="100px"/>',
'<img src="figures/EmojiScalePoint2.png" height="100px" width="100px"/>',
'<img src="figures/EmojiScalePoint3.png" height="100px" width="100px"/>',
'<img src="figures/EmojiScalePoint4.png" height="100px" width="100px"/>',
'<img src="figures/EmojiScalePoint5.png" height="100px" width="100px"/>',
'<img src="figures/EmojiScalePoint6.png" height="100px" width="100px"/>'
],
data: {
condition: 'impok, hardworking, Q2'
},
};
var Zarky_Question3 = {
type: 'button-response',
prompt: "<p>Imagine you are playing the Zarky game. Would playing the Zarky game make you feel hapy or sad?</p>",
choices: [
'6', '5', '4', '3', '2', '1'
],
button_html: [
'<img src="figures/EmojiScalePoint1.png" height="100px" width="100px"/>',
'<img src="figures/EmojiScalePoint2.png" height="100px" width="100px"/>',
'<img src="figures/EmojiScalePoint3.png" height="100px" width="100px"/>',
'<img src="figures/EmojiScalePoint4.png" height="100px" width="100px"/>',
'<img src="figures/EmojiScalePoint5.png" height="100px" width="100px"/>',
'<img src="figures/EmojiScalePoint6.png" height="100px" width="100px"/>'
],
data: {
condition: 'zarky, smart, Q3'
},
};
var Impok_Question3 = {
type: 'button-response',
prompt: "<p>Imagine you are playing the Impok game. Would playing the Impok game make you feel hapy or sad?</p>",
choices: [
'6', '5', '4', '3', '2', '1'
],
button_html: [
'<img src="figures/EmojiScalePoint1.png" height="100px" width="100px"/>',
'<img src="figures/EmojiScalePoint2.png" height="100px" width="100px"/>',
'<img src="figures/EmojiScalePoint3.png" height="100px" width="100px"/>',
'<img src="figures/EmojiScalePoint4.png" height="100px" width="100px"/>',
'<img src="figures/EmojiScalePoint5.png" height="100px" width="100px"/>',
'<img src="figures/EmojiScalePoint6.png" height="100px" width="100px"/>'
],
data: {
condition: 'impok, hardworking, Q3'
},
};
var Zarky_Question4 = {
type: 'button-response',
prompt: "<p>If you had the chance to do something tomorrow, would you play the Zarky game?</p>",
choices: [
'1', '0'
],
button_html: [
'<img src="figures/EmojiScalePoint1.png" height="100px" width="100px"/>',
'<img src="figures/EmojiScalePoint6.png" height="100px" width="100px"/>',
],
data: {
condition: 'zarky, smart, Q4'
},
};
var Impok_Question4 = {
type: 'button-response',
prompt: "<p>If you had the chance to do something tomorrow, would you play the Impok game?</p>",
choices: [
'1', '0'
],
button_html: [
'<img src="figures/EmojiScalePoint1.png" height="100px" width="100px"/>',
'<img src="figures/EmojiScalePoint6.png" height="100px" width="100px"/>',
],
data: {
condition: 'impok, hardworking, Q4'
},
};
# Randomizes the questions
var Zarky_questions = [
Zarky_Question1,
Zarky_Question2,
Zarky_Question3,
Zarky_Question4,
]
var Zarky_questions_timeline = jsPsych.randomization.shuffle(Zarky_questions)
var Zarky_outcome = {
timeline: Zarky_questions_timeline,
}
var Impok_questions = [
Impok_Question1,
Impok_Question2,
Impok_Question3,
Impok_Question4,
]
var Impok_questions_timeline = jsPsych.randomization.shuffle(Impok_questions)
var Impok_outcome = {
timeline: Impok_questions_timeline,
}
# Create modules for the two different games assigned to the two different conditions of smart and hardworking: each includes a description of the game and the 4 questions assessing interest in the game.
var Zarky_smart = {
timeline: [
description_zarky_smart,
Zarky_outcome,
]
}
var Impok_hard = {
timeline: [
description_impok_hard,
Impok_outcome,
]
}
var Zarky_hard = {
timeline: [
description_zarky_hard,
Zarky_outcome,
]
}
var Impok_smart = {
timeline: [
description_impok_smart,
Impok_outcome,
]
}
# Age information collection
var survey_trial = {
type: 'survey-text',
preamble: 'Subject number is ' + id,
questions: ["Year of Birth","Month of Birth"],
rows: [5,3],
columns: [40,50]
};
# Gender informationa collection
var gender = {
type: 'button-response',
choices: [ 'male', 'female' ],
data: {
condition: 'gender information'
},
};
# Trials with different conditions. Each trial includes both of the two novel games, with one game assigned to the 'smart' condition, and one game assigned to the 'hardworking' condition
var trial1 = {
timeline: [
Zarky_smart,
Impok_hard,
]
}
var trial2 = {
timeline: [
Zarky_hard,
Impok_smart,
]
}
var trial3 = {
timeline: [
Impok_smart,
Zarky_hard,
]
}
var trial4 = {
timeline: [
Impok_hard,
Zarky_smart,
]
}
# Consent form with two options: consent or decline. The consent block was only used for the adult participants.
var consent_block = {
stimulus: "<p>This research is being conducted by the students of PSYL11087 as part"+
" of their MSc degree. You have been asked to participate because you are"+
" a student on a PPLS MSc degree. The Instructor for this class, and supervisor"+
" for this project is Dr Hugh Rabagliati (hugh.rabagliati@ed.ac.uk)."+
" You can contact him by email at any time if you have questions.</p>" +
"<p> In this study, you will be asked to complete the following task:</p>"+
"<ol>"+
"<li> You will see pictures of games for hardworking and smart kids,"+
" and will be asked to answer questions about the game.</li></ol></p>"+
"<p></p>"+
"<p></p>"+
"<p>The only information collected will be your responses on this task, and your gender."+
" No identifying infomation about you will be collected. Participating in this study does "+
"not confer either benefits or risks. The study will take less than 10 minutes and "+
" there is no monetary compensation. You may stop participation at any time. This data will"+
" be analyzed and used as a coursework submission for the class PSYL11087.</p>"+
"<p> If you consent to participate, press the Consent button below. If you do not consent"+
" you may end the study or press the decline button.</p>",
is_html: true,
choices:['consent','decline'],
type: "button-response",
timing_post_trial: 500
};
# Test questions were used to ensure the participants understood how to use the response scales.
var understanding_question = {
type: 'button-response',
prompt: "<p>Do you like eating worms? </p>",
choices: [
'6', '5', '4', '3', '2', '1'
],
button_html: [
'<img src="figures/EmojiScalePoint1.png" height="100px" width="100px"/>',
'<img src="figures/EmojiScalePoint2.png" height="100px" width="100px"/>',
'<img src="figures/EmojiScalePoint3.png" height="100px" width="100px"/>',
'<img src="figures/EmojiScalePoint4.png" height="100px" width="100px"/>',
'<img src="figures/EmojiScalePoint5.png" height="100px" width="100px"/>',
'<img src="figures/EmojiScalePoint6.png" height="100px" width="100px"/>'
],
data: {
condition: 'understanding'
},
};
var understanding_question2 = {
type: 'button-response',
prompt: "<p>If you had the chance to go to disneyland tomorrow would you want to go?</p>",
choices: [
'1', '0'
],
button_html: [
'<img src="figures/EmojiScalePoint1.png" height="100px" width="100px"/>',
'<img src="figures/EmojiScalePoint6.png" height="100px" width="100px"/>',
],
data: {
condition: 'understanding'
},
};
# Function modules for buttons to jump to each trail.
# Each trail include:
# a consent form,
# a survey trial to collect data of age and gender,
# an introduction to the experiment,
# two understanding questions to make sure participants understand the response scales,
# a trial selected by pressing the button,
# and a thank you block.
function Zarky_smart_Impok_hard(){
jsPsych.data.addProperties({subject: id});
jsPsych.init({
timeline: [consent_block,survey_trial,gender,Introduction, understanding_question, understanding_question2, trial1, Finish],
on_finish: function() {
saveData(id + ".csv", jsPsych.data.dataAsCSV());
}})};
function Zarky_hard_Impok_smart(){
jsPsych.data.addProperties({subject: id});
jsPsych.init({
timeline: [consent_block,survey_trial,gender,Introduction, understanding_question, understanding_question2, trial2, Finish],
on_finish: function() {
saveData(id + ".csv", jsPsych.data.dataAsCSV());
}
})};
function Impok_smart_Zarky_hard(){
jsPsych.data.addProperties({subject: id});
jsPsych.init({
timeline: [consent_block,survey_trial,gender,Introduction, understanding_question, understanding_question2, trial3, Finish],
on_finish: function() {
saveData(id + ".csv", jsPsych.data.dataAsCSV());
}
})};
function Impok_hard_Zarky_smart(){
jsPsych.data.addProperties({subject: id});
jsPsych.init({
timeline: [consent_block,survey_trial, gender,Introduction, understanding_question, understanding_question2, trial4, Finish],
on_finish: function() {
saveData(id + ".csv", jsPsych.data.dataAsCSV());
}
})};
</script>
</body>
</html>