The study by Gick and Holyoak (1983) investigated the nature of analogical transfer between semantically distint problems. This paper focused on “how analogies are noticed and then applied to generate solutions to novel problems.” According to Schwartz and Goldstone (2016), Gick and Holyoak’s work demonstrate a way to use analogies by providing students with at least two examples that they use to later identify the scheme. The study participants were more likely to derive the problem scheme as an unintended result of identifying the similarities between two given analogs. The quality of the induced scheme was highly predictive of the subsequent transfer performance. This way of using analogies turned out to be powerful for learning and transfer. For the purpose of this replication project, I will focus on part II of this study, specifically experiment 5.
My research interests lie at the intersection of engineering education and language boundaries within the context of the multilingual science classroom. Specifically, I seek to identify opportunities for children to learn about the ways engineers approach problem solving. There is extensive research on adult engineers approach to problem solving, but very little is known about how to teach engineering thinking to youth. Although engineering has a long history both in higher education and as a profession, the recent K-12th science education standards bring the ideas and practices of engineering to the elementary and secondary levels (NRC, 2012; NGSS, 2013) . Due to the recent inclusion of engineering in the standards, I argue that a successful implementation of the framework demands a better understanding of how we can afford opportunities for learning in K-12th engineering. Contemporary research has yet to define those opportunities. It also needs to address research questions about how analogies are noticed in engineering problems and then applied to generate solutions to novel challenges, similarly to Gick and Holyoak’s work.
This paper followed a between-subjects design as participants were only exposed to a single level of factors in the design. In the case of experiment 5, a total of 143 college students participated in the study. Participants were evenly splited between two conditions, with-principle condition and without-principle condition. There was no control group in this study. The authors used “one pair of dissimilar story analogs: “The General” and “The Fire Chief.” Subjects in the without-principle condition read the two stories just as they appear in Appendix II of the article. Those in the with-principle condition read the identical stories, except that the verbal statement of principle was appended as the final paragraph of each story in the following format: “(The general or the fire chief) attributed his success to an important principle: If you need a large force to accomplish some purpose, but are prevented from applying such a force directly, many smaller forces applied simultaneously from different directions may work just as well”). This statement was designed to focus the subjects’ attention on the critical aspects of the schema implicit in each of the two analogs.
As part of the procedure, subjects were first told to study the two stories carefully for 5 min in preparation for answering questions about them. The stories were then collected, and the remainder of the initial story task was done from memory. Subjects were asked to briefly summarize each story, rate the comprehensibility of each, describe as clearly as possible the ways in which the situations in the two stories seemed similar, and rate their overall similarity. After this initial task was completed, the radiation problem was administered in the usual two-pass manner: first without a hint to use the prior story or statement, and then with such a hint. The challenges of performing this type of experiment are related to sampling and utility. First, I would like to select not a convenience sample, but a sample that it is representative of the study. In an ideal case scenario, I would love to run this study with younger kids. However, due to time constraints and limitations in accessing a sample of bilingual children in K-12th, this may not be possible.
This paper followed a between-subjects design because participants were only exposed to a single level of factors in the design. The total mean of the group with principle condition was 60 while the without prionciple condition was 47. As indicated in the paper, “62% of the subjects in the with-principle condition produced the convergence solution without a hint, as compared to only 40% of those in the without-principle condition, G2( 1) = 6.75, p < .01. The with-principle maintained a slightly smaller advantage in total solution frequency (82% vs 67%), G2(1) = 4.34, p < .05.” For samples to achieve 80%, 90%, 95% power to detect that effect size, I will need an alpha of 0.05 for any of them, Beta of 0.2, 0.1 and 0.05 respectively.
Original effect size, power analysis for samples to achieve 80%, 90%, 95% power to detect that effect size. Considerations of feasibility for selecting planned sample size.
A total of 143 subjects, college students, participated in the original experiment 5. Participants were evenly splited between the two conditions. For the purpose of this replication project, I am considering a sample size of 104 college students. The data will be collected during 2018-2019 Fall quarter until November 15th. To be considered for this study, subjects must be currently enrolled in college and preference will be given to students in their freshman and junior years. As in the original paper, this study does not include a control group.
Planned sample size and/or termination rule, sampling frame, known demographics if any, preselection rules if any.
All materials used in this experiment come from the original article and were followed precisely as indicated by the original authors. The authors used “one pair of dissimilar story analogs: “The General” and “The Fire Chief.” Subjects served in one of two conditions. Those in the without-principle condition read the two stories just as they appear in Appendix II. Those in the with-principle condition read the identical stories, except that the verbal statement of the convergence principle, previously used in Experiment 2, was appended as the final paragraph of each (“…attributed his success to an important principle: If you need a large force to accomplish some purpose, but are prevented from applying such a force directly, many smaller forces applied simultaneously from different directions may work just as well”). The statement was worded exactly as it had been in the earlier experiment (except, of course, that the sentence for “The Fire Chief” began as following: “The fire chief attributed his success. . .“). The statement was thus designed to focus the subjects’ attention on the critical aspects of the schema implicit in each of the two analogs.
All materials - can quote directly from original article - just put the text in quotations and note that this was followed precisely. Or, quote directly and just point out exceptions to what was described in the original article.
Subjects were first told to study the two stories carefully for 5 min in preparation for answering questions about them. The stories were then collected, and the remainder of the initial story task was done from memory. Subjects were asked to briefly summarize each story, rate the comprehensibility of each, describe as clearly as possible the ways in which the situations in the two stories seemed similar, and rate their overall similarity. After this initial task was completed, the radiation problem was adminis- tered in the usual two-pass manner.
Can quote directly from original article - just put the text in quotations and note that this was followed precisely. Or, quote directly and just point out exceptions to what was described in the original article.
Can also quote directly, though it is less often spelled out effectively for an analysis strategy section. The key is to report an analysis strategy that is as close to the original - data cleaning rules, data exclusion rules, covariates, etc. - as possible.
Clarify key analysis of interest here You can also pre-specify additional analyses you plan to do.
Explicitly describe known differences in sample, setting, procedure, and analysis plan from original study. The goal, of course, is to minimize those differences, but differences will inevitably occur. Also, note whether such differences are anticipated to make a difference based on claims in the original article or subsequent published research on the conditions for obtaining the effect.
You can comment this section out prior to final report with data collection.
Sample size, demographics, data exclusions based on rules spelled out in analysis plan
Any differences from what was described as the original plan, or “none”.
Data preparation following the analysis plan.
The analyses as specified in the analysis plan.
Side-by-side graph with original graph is ideal here
Any follow-up analyses desired (not required).
Open the discussion section with a paragraph summarizing the primary result from the confirmatory analysis and the assessment of whether it replicated, partially replicated, or failed to replicate the original result.
Add open-ended commentary (if any) reflecting (a) insights from follow-up exploratory analysis, (b) assessment of the meaning of the replication (or not) - e.g., for a failure to replicate, are the differences between original and present study ones that definitely, plausibly, or are unlikely to have been moderators of the result, and (c) discussion of any objections or challenges raised by the current and original authors about the replication attempt. None of these need to be long.