Replication of Fluency and the Detection of Misleading Questions: Low Processing Fluency Attenuates The Moses Illusion by Hyunjin Song & Norbert Schwarz (2008, Social Cognition)

Introduction

https://github.com/ashleybellmonteiro/song2008

As a researcher interested in social psychology I found the paper, “Fluency and the Detection of Misleading Questions: Low Processing Fluency Attenuates The Moses Illusion” to be particularly interesting. Although I am primarily an emotion researcher, I enjoy incorporating topics such as user experience, social media, and fluency effects into my work. A big reason I chose this paper was because I don’t have much experience in this exact type of work. Instead of doing studies that I will run all the time, I’ve decided to use this project as an opportunity to try something new. By engaging in this research I hope to not only broaden my understanding of cognitive processes, but also enrich my upcoming research…especially my first year project!

The stimuli and procedures I need for the pretest would all be online, possibly done over Qualtrics. I would first test both the font type and color to see how/if they affect reading ease. This would be done by using five undergraduates per each condition. One sentence will be printed in grey brush script while the other in black Arial. They would rate on a scale of 1-7 the ease in which they could read the text. 1 being very difficult 7 being very easy. When it comes to the main experiment I would recruit 32 more undergrads and randomly assign them to either the easy to read or difficult to read conditions. Then I will have them both read and answer a couple of trivia questions correlating with their given group, easy or difficult to read. The first question would be a control question while the second question would be the distroted question.

A potential challenge of this study may be the trivia questions that were used as stimuli in the previous experiment. My worry is that people may not understand them. Other than that though, I am quite confident this experiment will run smoothly. However, something that may be challenging for me would be R, just because I am so new to coding.

Methods

Main Experiment

Thirty-two prolific participants were randomly assigned to either an easy-to-read or difficult-to-read condition. Instructions (modeled after Erickson & Mattson, 1981) guided participants on how to complete the study by stating, ‘You will read a couple of trivia questions and answer them. You can write the answer in the blank. If you do not know the answer, please write “don’t know.” You may or may not encounter ill-formed questions that do not have correct answers if taken literally. For instance, you might see the question, “Why was President Gerald Ford forced to resign his office?” In fact, Gerald Ford was not forced to resign. Please write “can’t say” for this type of question.’

Depending on the condition to which the participant was assigned, they were presented with either an easy-to-read or hard-to-read font. The control question, which did not have a distortion, was, “Which country is famous for cuckoo clocks, chocolate, banks, and pocket knives?” The answer to this question is Switzerland. The experimental question did contain a distortion and was, “How many animals of each kind did Moses take on the Ark?” This question replaces the correct figure, Noah, with Moses and should be answered by participants with “can’t say.” If participants answered “2,” it indicated the Moses illusion.

Power Analysis

Original effect size, power analysis for samples to achieve 80%, 90%, 95% power to detect that effect size. Considerations of feasibility for selecting planned sample size.

Planned Sample

Planned sample size and/or termination rule, sampling frame, known demographics if any, preselection rules if any.

Materials

All materials - can quote directly from original article - just put the text in quotations and note that this was followed precisely. Or, quote directly and just point out exceptions to what was described in the original article.

Procedure

Can quote directly from original article - just put the text in quotations and note that this was followed precisely. Or, quote directly and just point out exceptions to what was described in the original article.

Analysis Plan

Z-tests

Differences from Original Study

Explicitly describe known differences in sample, setting, procedure, and analysis plan from original study. The goal, of course, is to minimize those differences, but differences will inevitably occur. Also, note whether such differences are anticipated to make a difference based on claims in the original article or subsequent published research on the conditions for obtaining the effect.

Methods Addendum (Post Data Collection)

You can comment this section out prior to final report with data collection.

Actual Sample

Sample size, demographics, data exclusions based on rules spelled out in analysis plan

Differences from pre-data collection methods plan

Any differences from what was described as the original plan, or “none”.

Results

Data preparation

We plan to use Z-tests because the original study used Z tests. T tests are for 30 and under participants while Z tests are better for 30 and over.

The article states:

“To assess the robustness of the results across both experiments, we used Rosen- thal’s (1978) procedures for combining results of independent studies. This analy- sis confirms the overall reliability of the observed patterns. When the question was distorted, participants were less likely to give an erroneous substantive answer, z = 3.26, p < .002, and more likely to recognize the distortion (as indicated by answer- ing ‘can’t say’), z = 2.89, p < .004, when the font was difficult rather than easy to read. When the question was undistorted, participants were less likely to give a cor- rect substantive answer, z = 2.5, p < .02, and more likely to report that they “don’t know, z = 2.45, p < .02, when the font was difficult rather than easy to read.”

Confirmatory analysis

The analyses as specified in the analysis plan.

Side-by-side graph with original graph is ideal here

Exploratory analyses

Any follow-up analyses desired (not required).

Discussion

Summary of Replication Attempt

Open the discussion section with a paragraph summarizing the primary result from the confirmatory analysis and the assessment of whether it replicated, partially replicated, or failed to replicate the original result.

Commentary

Add open-ended commentary (if any) reflecting (a) insights from follow-up exploratory analysis, (b) assessment of the meaning of the replication (or not) - e.g., for a failure to replicate, are the differences between original and present study ones that definitely, plausibly, or are unlikely to have been moderators of the result, and (c) discussion of any objections or challenges raised by the current and original authors about the replication attempt. None of these need to be long.