My research interests lie in the realm of Education and the different levers that are involved in the learning sciences. Motivation and growth mindset are very well studied and integral parts of researching how humans learn, how humans view intelligence, and the science behind continuous learning both outside and inside the school environment. For this reason, Haimovitz’s study on children’s implicit theories on intelligence and how they are developed in the home environment is very applicable to my research interests. I believe that understanding how children develop mindsets about their intelligence and how this impacts their motivation to learn.
This study focuses on the distinction between fixed and growth mindset for children in school and the effect that this has on their motivation. The potential that this intelligence mindset has on children is tremendous, and thus it is important to study where and how these mindsets get developed from the home environment — specifically the parents’ own intelligence mindset. There are four different studies that study this link. For the sake of this replication project, I will focus on study 4, which examines “the causal effect of parents’ failure mind-sets on their responses to their children’s hypothetical failure.”
In this study, 132 parents were recruited to take a survey about failure mindsets and tasked to imagine a scenario in which in which their child came home from school with a failing grade on a math quiz. They were then asked what they would do. It was shown that parents that were induced a “failure-is-debilitating mind-set were more likely to react with concerns about their child’s performance and lack of ability.” These results were then used to conclude that parental beliefs about failure are visible and can shape children’s attitudes and beliefs about their own intelligence. This experiment requires a study that is 1) able to induce and differentiate between the two intelligence mindsets for parents, 2) a case where parents can tap into realistic hypothetical failures, and 3) a method to establish causation between parents’ failure mindsets their perspective on their children’s failures. Some challenges include the effectiveness with which we can induce different failure mindsets and accuracy with which the survey questions can accurately portray parental beliefs towards their children.
Link to Repository: git@github.com:psych251/haimovitz_rescue.git
Link to Paper: https://pdfs.semanticscholar.org/7172/b578bfc9a77fbcd34c8e2417b674977f73fc.pdf?te=1&nl=nyt-parenting&emc=edit_ptg_20200108
The prior replication attempt made for the Haimovitz paper from previous years made a great effort towards replicating the effects that were seen on the effect of parents’ failure mindsets on their responses to their children’s hypothetical failures. The replication study followed a similar method route – they both recruited a group of parents from a Amazon Mechnanical Turks, requested survey responses, and used the same coding framework for data analysis. The replication study, used the same crowdsourcing platform and recruited 100 parents (32 less than the original) to achieve 80% power. The sample included the following demographics:
The replication study also added additional funnel debrief questions to collect more demographic information about the subjects’ children and relabeled some survey questions for clearness. The coding scheme remained relatively the same. Lastly, during analysis, two rater coded the responses (one being the replication author, compared to two independent raters in the original study).
The results of the 1st replication study did not replicate the results of the original study. They found that “Parents who were induced to hold a failure-is-debilitating mind-set were less likely to react with concerns about their child’s performance and lack of ability, t(93) = -1.93, p = .056, and more likely to react with support for their child’s learning and mastery, t(94) = 0.21, p = .83, compared with those who were induced to hold a failure-is-enhancing mind-set.”
Original effect size, power analysis for samples to achieve 80%, 90%, 95% power to detect that effect size. Considerations of feasibility for selecting planned sample size. ηp2 = .075 (d = 0.51)
How much power does your planned sample have for original effect? For an attenuated effect that is half the size of the original?
Power analysis to detect this effect size with an alpha of 0.05:
Despite the power analysis, I plan to collect a sample size that is just as large as the original study’s sample size (n=132). This is because my replication study is being jointly with another student in the course – the data collected from this study will be used for both of our analysis. Therefore, we decided to increase the sample size in order to better replicate the results of the original study. To account for anticipated exclusions, we decided to choose a sample size of 140.
All materials - can quote directly from original article - just put the text in quotations and note that this was followed precisely. Or, quote directly and just point out exceptions to what was described in the original article.
From the paper: Two raters, blind to condition, coded the openended responses.
“Participants completed an online survey initially assessing several beliefs, including their perceptions of their child’s competence (assessed with same measure as in Study 1; α = .79). Then we temporarily manipulated failure mind-sets by randomly assigning the parents to complete one of two five-item biased questionnaires, written to foster agreement with either a failure-is-debilitating mind-set (e.g., “Experiencing failure can lead to negative feelings, like shame or sadness, that interfere with learning”) or a failure-isenhancing mind-set (e.g., “Experiencing failure can improve performance in the long run if you learn from it”). All measures used a 6-point rating scale from 1 (strongly disagree) to 6 (strongly agree). One-sample t tests comparing the mean in each priming condition with the scale’s midpoint (3.5) showed that participants’ agreement with the intended mind-set was above the midpoint in both the failure-is-debilitating condition (M = 4.41, SD = 1.07), t(56) = 6.45, p < .001, and the failure-isenhancing condition (M = 5.14, SD = 0.829), t(74) = 17.11, p < .001. We then asked participants to read and vividly imagine a scenario in which their child came home from school with a failing grade on a math quiz, as in Study 2. They then wrote what they would do, think, and feel in response. Finally, participants reported on their failure mind-sets (α = .82), using the same items as in Study 1, as part of a survey that included a few other items.” Can quote directly from original article - just put the text in quotations and note that this was followed precisely. Or, quote directly and just point out exceptions to what was described in the original article.”
The link to the qualtrics survey that I created for this replication study is here: https://stanforduniversity.qualtrics.com/jfe/form/SV_9tVwoAgXkfFqHOK
Quality control measures that I will be adding to the experiment are: changing some of the coding measures to better interpret the open-ended questions, the coders gave a score of 1 each time that a code was present. However, I think that there needs to be more nuance that captures the varying intensity of the different open-ended responses.
To conduct analysis, we first must separate the likert scale questions and the open response questions. The open responses will be analyzed using the coding schema used for both the original and the replication study:
“Two raters, blind to condition, coded the open-ended responses. The first author developed a coding scheme on the basis of an initial reading of the responses and then made clarifying revisions on the basis of feedback from the two raters. The codes were broken down into two main categories of interest: performance-oriented responses and learning-oriented responses. Coders gave a score of 1 each time a code was present.”
The different codes that are separated into two different categories: performance oriented(Ability, Entity Comfort, Contingent Self-Worth, Grades, Social) or learning oriented (Effort, Strategy, Help, Mastery, Interest, Failure_Good). Each open response will be analyzed per these “codes” and each code gets a +1 whenever there is a relevant instantiation in the open responses. Here is an example – If a parent writes this in the open response: “I would tell my son he needs to study harder,” this would receive a +1 in the effort category. Once the response is done being analyzed, we then sum up the number of codes present in the performance category vs. learning category.
To code up the responses, I first plan to run the entire survey and collect all the responses. From there, I will follow the coding scheme as mentioned above. Then, I will compare my analysis to that of my independent rater’s and perform a ICC analysis for reliability purposes. The results will then be used for the confirmatory t-test analysis.
Clarify key analysis of interest here I also attempt to see the link between different demographic factors (i.e. age, race, gender) and its effect on the response towards the child’s failure.
I hope to minimize the differences between my replication study and the previous attempts. First, I hope to utilize the same survey questions and coding schemas as done before. The survey is the same and confirmed with original author Dr. Carol Dweck and Dr. Kyla Haimovitz. In addition, the coding scheme (included in the original_paper folder in the repo) will be followed closely. Lastly, I also have one independent coder to assess my reliability for the coding of the open responses. The only two differences between the the 1st replication study and my study is that: 1)I am using Prolific as my crowdsourcing platform (replication used Amazon Mechanical Turks) and 2) My study will run closer to the original sample size (original n = 132, replication n = 100, current n = 140).
You can comment this section out prior to final report with data collection.
Sample size, demographics, data exclusions based on rules spelled out in analysis plan
Any differences from what was described as the original plan, or “none”.
In this next section, I will filter out the dataframe, identify subjects in the debilitating/enhancing condition and then split up the data into just the Open Responses and just the Survey Responses.
Now, I will load the coded responses (analysis done by blind rater on the open responses)
#load in the coded data your_data <- read.csv("/Users/khaingmon/Desktop/haimovitz_rescue/data/check.csv", fill = TRUE)
coded_data <- read.csv("/Users/khaingmon/Desktop/haimovitz_rescue/data/pilotB_coded_responses.csv", header = TRUE, sep = ",")How did people perform on any quality control checks or positive and negative controls?
After conducting the analysis, the goal was to determine whether or not parents that held a failure-is-debilitating mind-set is more likely to react with concerns about their child’s performance and whether or not this concern was related to their child’s ability. This was done with an unpaired two-tailed t-test. The code for this confirmatory analyses of my pilot study is here:
enhancing_coded <- survey_responses %>%
filter(Condition == "Enhancing")
debilitating_coded <- survey_responses %>%
filter(Condition == "Debilitating")
#separate coded data into FIE/FID conditions
coded_fie <- coded_data %>%
filter(Condition == 'Enhancing')
coded_fid <- coded_data %>%
filter(Condition == 'Debilitating')
#conduct the t-test. For pilot B, there are no results yet because we only have on observation each, but this line of code does conduct t.test correctly.
#t_value <- t.test(coded_fie$sum_performance, coded_fid$sum_performance)
#p_value <- t_value$p.valueThree-panel graph with original, 1st replication, and your replication is ideal here
Any follow-up analyses desired (not required).
Combining across the original paper, 1st replication, and 2nd replication, what is the aggregate effect size?
Open the discussion section with a paragraph summarizing the primary result from the confirmatory analysis and the assessment of whether it replicated, partially replicated, or failed to replicate the original result.
Add open-ended commentary (if any) reflecting (a) insights from follow-up exploratory analysis, (b) assessment of the meaning of the replication (or not) - e.g., for a failure to replicate, are the differences between original and present study ones that definitely, plausibly, or are unlikely to have been moderators of the result, and (c) discussion of any objections or challenges raised by the current and original authors about the replication attempt. None of these need to be long.