Replication of Study 3 by Kwan, Dai & Wyer,JR (2017, Journal of Consumer Research)
Introduction
The main objective of this replication project is to thoroughly assess the findings of “Contextual Influences on Message Persuasion: The Effect of Empty Space” by Kwan, Dai, and Wyer from 2017. The original finding was that message recipients infer a weaker opinion when a message is surrounded by empty space, making them less likely to accept its implications. We aim to focus on replicating Study 3 from the original article, where the authors found that people evaluated the statements less favorably and paid less attention to a message when it was surrounded by substantial empty space. By replicating the study’s methods and analyzing the data in a comparable setting, we intend to evaluate the reliability and applicability of the original conclusions. This project will enhance our understanding of the contextual effect and strengthen the credibility of the influence of empty space on persuasion in the field.
Methods
Power Analysis
Original effect size, power analysis for samples to achieve 80%, 90%, 95% power to detect that effect size. Considerations of feasibility for selecting planned sample size.
Planned Sample
Ninety-four US residents were recruited on Mechanical Turk for a monetary incentive. There were no preselection rules to our knowledge. In the paper there were 36 male participants. The amount of the monetary incentive was not disclosed but in an independent study mentioned in the paper the participants were payed $.20.
Materials
We will create ten statements drawn from social media, covering topics such as romance, happiness, and personal values. These statements will vary in length from 5 to 11 words and will be presented uniformly in terms of font type, size, text positioning, line spacing, paragraphing, and background graphics.
Following are the statements that we will use:
Keep calm and carry on. Men never remember but women never forget. The best mirror is an old friend. Happiness shared is happiness doubled. Love is shown more in deeds than in words. Every day is a chance to grow. Take each day one step at a time. A heart in love has no limits. Life is too precious to waste on regrets. Make time for what makes you smile.
In the Limited Space Condition, quotes will be shown in a box sized between 420 × 315 pixels and 660 × 165 pixels, with no extra space around the border. In the Empty Space Condition, quotes will be displayed in a box ranging from 960 × 720 pixels to 960 × 240 pixels, surrounded by significant empty space.
Procedure
Participants first completed a survey called “Quotes-of-the-Year,” during which they evaluated 10 statements sourced from social media platforms like Twitter and Facebook. These statements varied in length from 3 to 11 words and addressed various topics, including romance (e.g., “try to reason about love and you will lose your reason”), happiness (e.g., “life is too short for tears”), and personal values (e.g., “follow your heart”). All statements were pre- sented in the same font type, size, text positioning, line spacing, paragraphing, and background graphics.
In the limited space condition, each quote was displayed in a box measuring between 420 × 315 pixels and 660 × 165 pixels, with no extra space around the border. In contrast, the empty space condition featured a box size ranging from 960 × 720 pixels to 960 × 240 pixels, surrounded by significant empty space.
Participants reviewed all 10 quotes in both conditions. After reading each quote, they indicated how much they liked it and how important they thought it was, using a scale from 1 (not at all) to 7 (very much). Their responses were averaged to create a single measure of message persuasiveness ( = .88). Additionally, the time spent evaluating each quote was recorded, with the total time serving as an indicator of message deliberation. Finally, participants reported their age and gender before being given the first few words of each quote as a cue and asked to recall it as accurately as possible. We will include an attention check at the end of the study post which they are asked to recall the phrases. The attention check is: “The color test you are about to take is very simple, when asked for your favorite color, you must select”Green”. This is an attention check. (The participants will be provided with 5 color options: Red, blue, green, orange and brown).
Design Overview
- One factor was manipulated. The space around the message.
- Three measures: response time, favorability and recall
- Between-subject.
- Yes, each measure was repeated 10 times for the 10 stimuli.
- No, it is not possible to do a within-subject design. Participants’ responses may be influenced by previous conditions, leading to confounding results.
- Yes, we used between-subject design, so participants were unaware of what the manipulation is. Also, we randomized the order of 10 quotes for each participant, so their responses to the first quote is not likely to influence their responses to subsequent questions.
- One potential confound could be the attrition bias, participants may have more difficulty recalling the information in the more space condition, thus they feel frustrated and drop out of the study more. This creates a bias in the remaining data.
Analysis Plan
We plan to run F-tests to compare favorability, response time, and ability to recall between the empty space condition and the limited space condition. Additionally, we plan to build an SPSS model (model 4) to explore whether message recall mediates the effect of empty space on message persuasion, using a bootstrap sample of 5,000.
Differences from Original Study
As the original appendix only provides three pairs of examples from the ten pairs of materials used in the original study, we plan to recreate the other seven pairs of materials using quotes gathered from social media. Based on the findings of the original article, if empty spaces cause people to rate the quotes less favorably and recall them less, the same effect should also exist if the content of the materials changes. Thus, we argue that this difference should not influence our replication of the original findings.
Methods Addendum (Post Data Collection)
You can comment this section out prior to final report with data collection.
Actual Sample
Sample size, demographics, data exclusions based on rules spelled out in analysis plan
Differences from pre-data collection methods plan
Any differences from what was described as the original plan, or “none”.
Results
Data preparation
Data preparation would include: a) Data import and initial inspection my_data <- read.csv(file.choose()) str(my_data) summary(my_data) #inspect head(my_data) #preview
Data Cleaning my_data <- na.omit(my_data) # Removes rows with any NA values # Convert space_condition to factor if not already my_data\(space_condition <- as.factor(my_data\)space_condition) #Checking for duplicates using (dulicated) to ensure no rows are repeated
Defining exclusion criteria #Removing participants with incomplete responses and attention check my_data <- filter(my_data, complete_cases == TRUE)
Final Preparation # Check the structure after data preparation str(my_data) summary(my_data)
Load Relevant Libraries and Functions
library(“dplyr”) library (“lavaan”) #### Import data my_data <- read.csv(file.choose()) #### Data exclusion / filtering sample_n(my_data, 10) favor.ftest <- var.test(empty_favor ~ limited_favor, data = my_data) time.ftest <- var.test(empty_time ~ limited_time, data = my_data) favor.ftest favor.ftest\(estimate favor.ftest\)p.value time.ftest time.ftest\(estimate time.ftest\)p.value #### Prepare data for analysis - create columns etc. # The numbers of correct recall
Specify the quote to match
specific_quote1 <- “Your specific quote here” specific_quote2 <- “Your specific quote here” specific_quote3 <- “Your specific quote here” specific_quote4 <- “Your specific quote here” specific_quote5 <- “Your specific quote here” specific_quote6 <- “Your specific quote here” specific_quote7 <- “Your specific quote here” specific_quote8 <- “Your specific quote here” specific_quote9 <- “Your specific quote here” specific_quote10 <- “Your specific quote here”
Create a new column to indicate if there is an exact match
my_data\(match1 <- ifelse(my_data\)quote1 == specific_quote1, TRUE, FALSE) my_data\(match2 <- ifelse(my_data\)quote2 == specific_quote2, TRUE, FALSE) my_data\(match3 <- ifelse(my_data\)quote3 == specific_quote3, TRUE, FALSE) my_data\(match4 <- ifelse(my_data\)quote4 == specific_quote4, TRUE, FALSE) my_data\(match5 <- ifelse(my_data\)quote5 == specific_quote5, TRUE, FALSE) my_data\(match6 <- ifelse(my_data\)quote6 == specific_quote6, TRUE, FALSE) my_data\(match7 <- ifelse(my_data\)quote7 == specific_quote7, TRUE, FALSE) my_data\(match8 <- ifelse(my_data\)quote8 == specific_quote8, TRUE, FALSE) my_data\(match9 <- ifelse(my_data\)quote9 == specific_quote9, TRUE, FALSE) my_data\(match10 <- ifelse(my_data\)quote10 == specific_quote10, TRUE, FALSE)
Create a new column to indicate the numbers of exact match for a participant
my_data\(recall <- rowSums(my_data\)match1, my_data\(match2, my_data\)match3, my_data\(match4,my_data\)match5,my_data\(match6,my_data\)match7,my_data\(match8,my_data\)match9,my_data$match10)
Compare the numbers of recall for two conditions, and run another F test
Build SPSS Model 4 to run mediation analysis
View the updated data frame
print(data)
#Justification of F test #The main finding from the original study suggests that participants evaluated statements less favorably in the “empty space” condition compared to the “limited space” condition. This indicates that the core hypothesis is centered around a mean difference in favorability scores between these two conditions.
#An ANOVA F-test is designed to compare means between groups, which is precisely what’s needed here. It will provide a single F-statistic and p-value to confirm whether the difference in evaluation scores between the “empty space” and “limited space” conditions is statistically significant.
#The F-test is both powerful and well-suited for comparing two or more groups, especially with continuous data like evaluation scores. It is more straightforward than alternatives (e.g., t-tests) when confirming findings in studies with multiple conditions.
#Further, the original study used an F-test or ANOVA to analyze this comparison, then using the same approach is consistent with replication goals, further justifying its use.
Confirmatory analysis
Define the CFA model
#“favor”, “time”, and “recall” are latent factors with observed variables. # Replace favor_var1
, favor_var2
, etc., with the actual names of observed variables cfa_model <- ’ # Define latent variables favor =~ favor_var1 + favor_var2 + favor_var3 time =~ time_var1 + time_var2 + time_var3 recall =~ recall_var1 + recall_var2 + recall_var3 ’
Fit the model to the data
cfa_fit <- cfa(cfa_model, data = my_data)
Summarize the CFA model results
summary(cfa_fit, fit.measures = TRUE, standardized = TRUE)
View modification indices (to suggest model improvements)
modificationIndices(cfa_fit, sort = TRUE)
Side-by-side graph with original graph is ideal here
Exploratory analyses
Any follow-up analyses desired (not required).
Discussion
Summary of Replication Attempt
Open the discussion section with a paragraph summarizing the primary result from the confirmatory analysis and the assessment of whether it replicated, partially replicated, or failed to replicate the original result.
Commentary
Add open-ended commentary (if any) reflecting (a) insights from follow-up exploratory analysis, (b) assessment of the meaning of the replication (or not) - e.g., for a failure to replicate, are the differences between original and present study ones that definitely, plausibly, or are unlikely to have been moderators of the result, and (c) discussion of any objections or challenges raised by the current and original authors about the replication attempt. None of these need to be long.