One of my main research interests is the way in which someone’s language can affect perceptions of them. I am choosing to replicate Study 1 from “Inferring Identity From Language: Linguistic Intergroup Bias Informs Social Categorization” because it falls into my interest area and I might reference their hypothesis in my future work. Additionally, their focus on political party affiliation is relevant for the current political moment. The results of this study suggest favorable linguistic integroup bias (LIB), or “us[ing] abstract language to describe in-group members’ desirable behaviors and concrete language to describe their undesirable behaviors,” increased the likelihood that the participant believed the target shared the same group membership. Most people do not think about whether or not they should describe someone abstractly or concretely when talking about them, but it appears that this distinction has a notable effect. Replicating this finding would provide additional evidence that this phenomenon exists and that differences in language choice can have an effect on perception.
The stimuli in this experiment were passages about a fictional man named Peter that included indications about his political party affiliation as well as his helpful and rude behaviors. This study had four conditions: favorable/unfavorable LIB Democratic, and favorable/unfavorable LIB Republican. In the favorable LIB conditions, helpful behavior was written in abstract language and rude behavior in concrete language. The unfavorable conditions had helpful behavior in concrete language and rude behavior in abstract language. Party affiliation was indicated by past voting behavior. After reading the passages, participants answered questions about the Peter’s likely political group membership, how likely they would be friends with him and information about their own party affiliations and ideology. Participant information was used in the analysis, which was a 2 (LIB condition) x 2 (Peter’s affiliation) x 3 (participant affiliation) ANOVA. Manipulation checks on the effectiveness of the LIB treatment will be conducted as well.
The participants in this replication study will be able to complete this on Amazon Mechanical Turk, which was also the venue for the original study. When conducting this experiment, I will be careful to properly set up the study on both Qualtrics and MTurk so that participants do not run into any issues that inhibit their completion of the study. Other challenges might include cleaning the data from Qualtrics, which has been difficult for me in the past. Hopefully my new knowledge of how to tidy the data will help!
Repository: https://github.com/hnmiecz/replication_paper Original Paper: https://github.com/hnmiecz/replication_paper/blob/master/original_paper/porter2016.pdf
Original effect size, power analysis for samples to achieve 80%, 90%, 95% power to detect that effect size. Considerations of feasibility for selecting planned sample size.
Planned sample size and/or termination rule, sampling frame, known demographics if any, preselection rules if any.
All materials - can quote directly from original article - just put the text in quotations and note that this was followed precisely. Or, quote directly and just point out exceptions to what was described in the original article.
Can quote directly from original article - just put the text in quotations and note that this was followed precisely. Or, quote directly and just point out exceptions to what was described in the original article.
Can also quote directly, though it is less often spelled out effectively for an analysis strategy section. The key is to report an analysis strategy that is as close to the original - data cleaning rules, data exclusion rules, covariates, etc. - as possible.
Clarify key analysis of interest here You can also pre-specify additional analyses you plan to do.
Explicitly describe known differences in sample, setting, procedure, and analysis plan from original study. The goal, of course, is to minimize those differences, but differences will inevitably occur. Also, note whether such differences are anticipated to make a difference based on claims in the original article or subsequent published research on the conditions for obtaining the effect.
You can comment this section out prior to final report with data collection.
Sample size, demographics, data exclusions based on rules spelled out in analysis plan
Any differences from what was described as the original plan, or “none”.
Data preparation following the analysis plan.
The analyses as specified in the analysis plan.
Side-by-side graph with original graph is ideal here
Any follow-up analyses desired (not required).
Open the discussion section with a paragraph summarizing the primary result from the confirmatory analysis and the assessment of whether it replicated, partially replicated, or failed to replicate the original result.
Add open-ended commentary (if any) reflecting (a) insights from follow-up exploratory analysis, (b) assessment of the meaning of the replication (or not) - e.g., for a failure to replicate, are the differences between original and present study ones that definitely, plausibly, or are unlikely to have been moderators of the result, and (c) discussion of any objections or challenges raised by the current and original authors about the replication attempt. None of these need to be long.