Luttrell et al. ( 2019) investigated whether the influence of moral counterattitudinal messages on attitude change depends on the extent that the initial attitudes were moralized. They further explored the mediation effect of the perceptions/thoughts toward the message and the moderation effect of political orientation. To test the hypotheses, they conducted two studies on the attitudes toward recycling.
I will replicate the first study. This study is based on a survey by randomly assigning participants into the moral / practical groups, and testing the differences of their responses toward the moral / practical messages.
The key test is to test the interaction effect between the message type (moral / practical) and moral basis.
I choose this paper to replicate because I have been working on factor analysis and structural equation modeling, thus I am very interested in self-reported data. This paper matches my research experiences and interests as it focused on variables (e.g., attitude) that were measured by scales. Besides, this paper used questionnaires to measure the variables of interest and the reliabilities of the questionnaires were reported in the paper. However, the construct validity was not reported in the paper. In addition to replicating the research, I want to further test the validity of the measurement tools used in this paper.
The potential challenge in this study is the funding to collect sufficient samples. In the original paper, the authors took one dollar to recruit each participant, and they recruited around 200 people.
Original effect size, power analysis for samples to achieve 80%, 90%, 95% power to detect that effect size. Considerations of feasibility for selecting planned sample size.
The authors preregistered a target sample size of 200, which provided 80% power to detect an interaction effect as small as .04. They finally collected 415 samples for study 1. However, to save the money, I plan to collect 50 samples and each participant will get $1 for completing the 5-minute survey.
This experiment will be conducted through qualtrics and participants will be required to fill out a survey (I have designed the survey). The questionnaire on attitude toward recycling, political orientation, perceptions about the message, as well as the stimuli (practical or moral arguments against recycling) used in the replication study will be the same as Luttrell et al. ( 2019) (see the quote below).
Premessage attitudes. Participants rated their attitudes toward recycling using three 9-point semantic-differential scales, ranging from −4 (bad, dislike, and negative) to 4 (good, like, and positive).
Perceived attitude bases. Participants reported the degree to which their attitudes toward recycling were based on their core moral beliefs or on practical concerns using 5-point scales (ranging from not at all to extremely). These items were modeled after items previously used to assess moral conviction (see Skitka, 2010). These items were presented within a small set of randomly ordered questions about people’s bases for their recycling attitudes so participants would not suspect that the study was specifically about the moral basis for those attitudes.
Political orientation. Participants reported their political orientation on two items measuring ideology for social and economic issues. The items were measured on 5-point scales ranging from very liberal to very conservative.
Counterattitudinal message. We created two versions of an antirecycling essay that appealed either to moral or to practical concerns. The moral appeal, entitled “Recycling: Harmful and Immoral,” framed its antirecy- cling position in moral terms (e.g., “Supporting recycling programs would be a grave moral transgression”) and cited particular moral reasons against recycling programs (e.g., “precious pets and animals [are] mercilessly killed by fumes produced in the recycling process”). By contrast, the practical appeal, entitled “Recycling: Costly and Unfeasible,” framed its antirecycling position in prag- matic terms (e.g., recycling is an “inefficient and unfea- sible endeavor for most municipalities to adopt”) and cited particular economic and pragmatic concerns (e.g., “An increase in trucks greatly increases traffic, both on the highways and on city roads”). We chose a practical appeal as the comparison because it reflects a common type of nonmoral persuasive argument (e.g., Mucciaroni, 2011) that could be similarly substantive. Messages were similar in length (426–428 words) and in number of argu- ments, and they were designed to be equally cogent to the sample overall.
Thought listing. Participants next listed the thoughts they had while reading the antirecycling message. Participants were given six thought-listing boxes and told to enter one thought per box; however, they were not required to fill all six. These thoughts were subsequently rated by two independent coders for overall valence (i.e., whether each thought was promessage, antimessage, neutral to the message, or unrelated to the message), following common practice.
Note that since I will code it by myself, there will not be two independent coders.
Postmessage attitudes. After reading the message, participants once again reported their attitudes toward recycling, using the same items used to measure premessage attitudes.
Message ratings. To ensure that participants perceived the messages as intended, we asked how much the mes- sage seemed to make arguments related to moral and practical concerns, each on 7-point scales ranging from not at all to very much.
At the beginning of the survey, participants will be asked to read an introduction to recycling and report their attitude about recycling. Next, participants will be randomly assigned into two groups, and in one group, they will need to read a message arguing against recycling using a moral appeal. In another group, participants will be assigned a message arguing against recycling using a practical appeal. After reading the message, participants in each group need to report their attitudes toward recycling again. Also, they will be asked about their perceptions of the message, political orientation, age, gender, and the primary language they are using.
First, we exclude any cases for which the message would be pro-attitudinal (i.e., attitudes toward recycling below the midpoint.) Next, we excluded any cases that were duplicates of a previous participant’s IP Address.
Calculate Cronbach’s alpha for the measures
Conduct confirmatory factor analysis for the measures
Data were submitted to t-test analyses to determine whether the messages were perceived as intended.
The data were submitted to a multiple regression analysis predicting postmessage attitudes in which pre- message attitudes, moral attitude basis, and message type were entered in the first step of the model and the Moral Attitude Basis × Message Type interaction term was entered in the second step. Message type was effects coded so that −1 corresponded with the moral- argument condition and +1 corresponded with the practical-argument condition. Results for these predic- tors are interpreted in the first steps of the model in which they appear.
We computed the indirect effect and CIs with non- parametric bootstrapping (10,000 iterations) using the mediation package in R (Tingley, Yamamoto, Hirose, Keele, & Imai, 2014). We tested the indirect effect on postmessage attitudes, controlling for premessage atti- tudes, setting the Moral Attitude Basis × Message Type interaction term as the predictor; valenced thoughts as the mediator; premessage attitudes, moral attitude basis, and message type as covariates; and postmessage atti- tudes as the outcome variable.
The fol- lowing analyses tested the interaction of each alternative basis with message type on postmessage attitudes, con- trolling for premessage attitudes.
To assess whether political orientation moderated the Moral Attitude Basis × Message Type interaction, we subjected the data to a hierarchical multiple regression model that tested a three-way interaction on postmessage attitudes.
Sample size will be smaller than the original study.
The thought list will be coded by myself instead of two independent coders in the original study, so I cannot test consistency between coders.
The original paper didn’t conduct confirmatory factor analysis for construct validation, and I will try to conduct in the replication study.
You can comment this section out prior to final report with data collection.
Sample size, demographics, data exclusions based on rules spelled out in analysis plan
Any differences from what was described as the original plan, or “none”.
Currently this part is for coding up the planned analyses . Note that the sample sizes of the pilot study A (N=3, collected from my friends) + B (N=2, collected from prolific) are not enough for the following analysis. I conducted data analysis based on the pilot data and some test responses collected from the teaching assistants and myself (Total sample size = 10) just to test whether the data collection and the code works wel. Responses of TAs and myself will be deleted in the formal analysis.
Data preparation following the analysis plan.
####Import data
pilot_data <- read_csv("../data/pilot_data.csv", show_col_types = FALSE)
nrow(pilot_data)## [1] 11#### Data exclusion / filtering
pilot_data0 <- pilot_data |>
mutate(att1 = (att1_1 + att1_2 + att1_3) / 3) |>
mutate(att2 = (att2_1 + att2_2 + att2_3) / 3) |>
mutate(att.pre = 0 - att1, att.post = 0 - att2) |>
mutate(ideology = (poli_soc + poli_econ) / 2)
table(pilot_data0$cond.F)| Moral Arguement | Practical Arguement |
|---|---|
| 3 | 8 |
The exclusion procedure is the same as the original paper. Because the manipulation is to persuade participants that recycling is bad, and these already had negative attitudes toward recycling.
pilot_data1 <- pilot_data0 |>
filter(att1 >= 0)
#### add columns
pilot_data1$base_mor_low <- scale(pilot_data1$base_mor, scale = F) + sd(pilot_data1$base_mor)
pilot_data1$base_mor_high <- scale(pilot_data1$base_mor, scale = F) - sd(pilot_data1$base_mor)
pilot_data1$base_mor_max <- pilot_data1$base_mor - 5
pilot_data1$cond_mor <- (pilot_data1$cond + 1) / 2
pilot_data1$cond_prac <- (abs(pilot_data1$cond - 1))/2
pilot_data1$ideology_low <- scale(pilot_data1$ideology, scale = F) + sd(pilot_data1$ideology, na.rm = T)
pilot_data1$ideology_high <- scale(pilot_data1$ideology, scale = F) - sd(pilot_data1$ideology, na.rm = T)nrow(pilot_data1) # sample size## [1] 10table(pilot_data1$gender) # gender information| Female | Male | Prefer Not to Say |
|---|---|---|
| 3 | 4 | 2 |
describe(pilot_data1$age) |>
select(mean, sd, median, min, max) |> round(2) # age| mean | sd | median | min | max | |
|---|---|---|---|---|---|
| X1 | 35.33 | 43.69 | 23 | 1 | 150 |
#### reliability
##### pre-message attitude
pilot_data1 |> select(att1_1, att1_2, att1_3) |>
psych::alpha() |> with(total) |> round(2)| raw_alpha | std.alpha | G6(smc) | average_r | S/N | ase | mean | sd | median_r | |
|---|---|---|---|---|---|---|---|---|---|
| 0.9 | 0.93 | 0.91 | 0.81 | 12.4 | 0.05 | 2.27 | 1.56 | 0.83 |
#### post-message attitude
pilot_data1 |> select(att2_1, att2_2, att2_3) |>
psych::alpha() |> with(total) |> round(2)| raw_alpha | std.alpha | G6(smc) | average_r | S/N | ase | mean | sd | median_r | |
|---|---|---|---|---|---|---|---|---|---|
| 0.91 | 0.91 | 0.88 | 0.77 | 9.95 | 0.05 | 1.93 | 1.31 | 0.77 |
#### Political Orientation
pilot_data1 |> select(poli_soc, poli_econ) |>
psych::alpha() |> with(total) |> round(2)| raw_alpha | std.alpha | G6(smc) | average_r | S/N | ase | mean | sd | median_r | |
|---|---|---|---|---|---|---|---|---|---|
| 0.68 | 0.68 | 0.52 | 0.52 | 2.15 | 0.2 | 3.1 | 0.7 | 0.52 |
#### validity
model <- '
att_f1 =~ att1_1 + att1_2 + att1_3
att_f2 =~ att2_1 + att2_2 + att2_3
poli =~ poli_soc + poli_econ
'
var = c("att1_1", "att1_2", "att1_3", "att2_1", "att2_2", "att2_3", "poli_soc", "poli_econ")
cfa_fit <- cfa(model, pilot_data1, var)## Warning in lavaan::lavaan(model = model, data = pilot_data1, ordered = var, : lavaan WARNING:
## the optimizer warns that a solution has NOT been found!# summary(cfa_fit)
# fitMeasures(cfa_fit)
# note that currently the sample size is not enough for the cfa and model didn't convergedisplay.t(y = pilot_data1$essMoral, x = pilot_data1$cond.F) |>
kable(align=rep('c', 8)) |>
kable_styling(full_width = F, position = "center") |>
column_spec(1:8, width = "5em")| Mean1 | SD1 | Mean2 | SD2 | t | df | p | d |
|---|---|---|---|---|---|---|---|
| 4.67 | 2.08 | 3.57 | 1.13 | 1.11 | 8 | 0.3 | 0.77 |
#display.t(y = pilot_data1$essPrac, x = pilot_data1$cond.F) |>
# kable(align=rep('c', 8)) |>
# kable_styling(full_width = F, position = "center") |>
# column_spec(1:8, width = "5em")##### main effect
pers.ME <- lm(att.post ~ att.pre + cond + base_mor, pilot_data1)
summary(pers.ME) ##
## Call:
## lm(formula = att.post ~ att.pre + cond + base_mor, data = pilot_data1)
##
## Residuals:
## Min 1Q Median 3Q Max
## -0.68421 -0.40141 -0.06085 0.15831 1.12112
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) -0.09265 0.67321 -0.138 0.89504
## att.pre 0.68717 0.15392 4.464 0.00426 **
## cond -0.25490 0.26835 -0.950 0.37884
## base_mor -0.05489 0.21888 -0.251 0.81034
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 0.6569 on 6 degrees of freedom
## Multiple R-squared: 0.8331, Adjusted R-squared: 0.7496
## F-statistic: 9.983 on 3 and 6 DF, p-value: 0.009511##### interaction
pers.int <- lm(att.post ~ att.pre + cond * base_mor, pilot_data1)
summary(pers.int) ##
## Call:
## lm(formula = att.post ~ att.pre + cond * base_mor, data = pilot_data1)
##
## Residuals:
## 1 2 3 4 5 6 7 8
## -0.02390 0.15137 -0.55741 -0.04779 0.97239 -0.29419 0.63329 -0.80771
## 9 10
## 0.07169 -0.09773
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) -0.2044 0.7429 -0.275 0.7942
## att.pre 0.6246 0.1983 3.150 0.0254 *
## cond -0.7050 0.8546 -0.825 0.4470
## base_mor -0.0954 0.2437 -0.392 0.7115
## cond:base_mor 0.1492 0.2671 0.559 0.6004
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 0.6981 on 5 degrees of freedom
## Multiple R-squared: 0.8429, Adjusted R-squared: 0.7172
## F-statistic: 6.707 on 4 and 5 DF, p-value: 0.03039##### effect size
R2.ME <- summary(pers.ME) |> with(r.squared)
R2.Full <- summary(pers.int) |> with(r.squared)
f2 <- round((R2.Full - R2.ME)/(1 - R2.Full), 2)
# print(paste("$f^{2}$ = ", f2))cannot conduct this now - requires coding of the thoughts
Following the original paper, I will rate the overall valence (i.e., whether each thought was promessage, antimessage, neutral to the message, or unrelated to the message) of the thoughts that participants list on the free response field, and test the mediation effect of valence thoughts. I cannot test the rater consistency reliability because there is no second rater.
ideo.ME <- lm(att.post ~ att.pre + cond + base_mor + ideology, pilot_data1)
summary(ideo.ME)##
## Call:
## lm(formula = att.post ~ att.pre + cond + base_mor + ideology,
## data = pilot_data1)
##
## Residuals:
## 1 2 3 4 5 6 7 8
## -0.118708 0.004438 -0.325283 0.084927 1.103601 -0.346790 0.564024 -0.823907
## 9 10
## 0.033781 -0.176084
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 0.38806 1.32099 0.294 0.78074
## att.pre 0.70299 0.16946 4.148 0.00892 **
## cond -0.26013 0.28880 -0.901 0.40901
## base_mor -0.04786 0.23591 -0.203 0.84724
## ideology -0.15031 0.34551 -0.435 0.68167
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 0.7063 on 5 degrees of freedom
## Multiple R-squared: 0.8392, Adjusted R-squared: 0.7105
## F-statistic: 6.523 on 4 and 5 DF, p-value: 0.03213ideo.2way <- lm(att.post ~ att.pre + cond * base_mor + cond * ideology + base_mor * ideology, pilot_data1)
summary(ideo.2way)##
## Call:
## lm(formula = att.post ~ att.pre + cond * base_mor + cond * ideology +
## base_mor * ideology, data = pilot_data1)
##
## Residuals:
## 1 2 3 4 5 6 7
## 1.665e-16 7.248e-02 -5.360e-01 -6.539e-16 1.022e+00 6.472e-16 5.108e-01
## 8 9 10
## -7.620e-01 5.708e-16 -3.070e-01
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 6.89043 20.75408 0.332 0.771
## att.pre 0.69932 0.39078 1.790 0.215
## cond -0.70641 11.88297 -0.059 0.958
## base_mor -1.68160 4.74076 -0.355 0.757
## ideology -2.29241 6.91408 -0.332 0.772
## cond:base_mor 0.03179 0.80977 0.039 0.972
## cond:ideology 0.12876 4.15655 0.031 0.978
## base_mor:ideology 0.53772 1.53180 0.351 0.759
##
## Residual standard error: 1.066 on 2 degrees of freedom
## Multiple R-squared: 0.8535, Adjusted R-squared: 0.3408
## F-statistic: 1.665 on 7 and 2 DF, p-value: 0.4256ideo.3way <- lm(att.post ~ att.pre + cond * base_mor * ideology, pilot_data1)
summary(ideo.3way)##
## Call:
## lm(formula = att.post ~ att.pre + cond * base_mor * ideology,
## data = pilot_data1)
##
## Residuals:
## 1 2 3 4 5 6 7
## 1.665e-16 7.248e-02 -5.360e-01 -6.539e-16 1.022e+00 6.472e-16 5.108e-01
## 8 9 10
## -7.620e-01 5.708e-16 -3.070e-01
##
## Coefficients: (1 not defined because of singularities)
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 6.89043 20.75408 0.332 0.771
## att.pre 0.69932 0.39078 1.790 0.215
## cond -0.70641 11.88297 -0.059 0.958
## base_mor -1.68160 4.74076 -0.355 0.757
## ideology -2.29241 6.91408 -0.332 0.772
## cond:base_mor 0.03179 0.80977 0.039 0.972
## cond:ideology 0.12876 4.15655 0.031 0.978
## base_mor:ideology 0.53772 1.53180 0.351 0.759
## cond:base_mor:ideology NA NA NA NA
##
## Residual standard error: 1.066 on 2 degrees of freedom
## Multiple R-squared: 0.8535, Adjusted R-squared: 0.3408
## F-statistic: 1.665 on 7 and 2 DF, p-value: 0.4256# practical bias
lm(att.post ~ att.pre + cond * base_prac, pilot_data1)##
## Call:
## lm(formula = att.post ~ att.pre + cond * base_prac, data = pilot_data1)
##
## Coefficients:
## (Intercept) att.pre cond base_prac cond:base_prac
## 0.74443 0.51616 -0.37376 -0.47205 0.04133# Emotion Basis
pilot_data1$base_emo_low <- scale(pilot_data1$base_emo) + sd(pilot_data1$base_emo)
pilot_data1$base_emo_high <- scale(pilot_data1$base_emo) - sd(pilot_data1$base_emo)
lm(att.post ~ att.pre + cond * base_emo, pilot_data1)##
## Call:
## lm(formula = att.post ~ att.pre + cond * base_emo, data = pilot_data1)
##
## Coefficients:
## (Intercept) att.pre cond base_emo cond:base_emo
## -3.2043 0.7627 0.1120 0.8691 NA# Knowlege Basis
lm(att.post ~ att.pre + cond * base_know + cond * base_mor, pilot_data1)##
## Call:
## lm(formula = att.post ~ att.pre + cond * base_know + cond * base_mor,
## data = pilot_data1)
##
## Coefficients:
## (Intercept) att.pre cond base_know base_mor
## -0.08397 0.63284 -0.88594 -0.05895 -0.03879
## cond:base_know cond:base_mor
## 0.09726 0.08360# Consenus Basis
lm(att.post ~ att.pre + cond * base_cons + cond * base_mor, pilot_data1)##
## Call:
## lm(formula = att.post ~ att.pre + cond * base_cons + cond * base_mor,
## data = pilot_data1)
##
## Coefficients:
## (Intercept) att.pre cond base_cons base_mor
## -0.165733 0.650213 -0.765164 0.004877 -0.098201
## cond:base_cons cond:base_mor
## 0.093576 0.107094The analyses as specified in the analysis plan.
Side-by-side graph with original graph is ideal here
Open the discussion section with a paragraph summarizing the primary result from the confirmatory analysis and the assessment of whether it replicated, partially replicated, or failed to replicate the original result.
Add open-ended commentary (if any) reflecting (a) insights from follow-up exploratory analysis, (b) assessment of the meaning of the replication (or not) - e.g., for a failure to replicate, are the differences between original and present study ones that definitely, plausibly, or are unlikely to have been moderators of the result, and (c) discussion of any objections or challenges raised by the current and original authors about the replication attempt. None of these need to be long.