Replication of Study 1 in Luttrell et al. (2019, Psychological Science)

Introduction

Luttrell et al. ( 2019) investigated the influence of moral message on persuasion. They found the significant interaction effect between the message type (moral / practical) and moral basis (.e., participants with a low moral attitude basis were more persuaded by the practical appeal than the moral appeal, whereas participants with a high moral attitude basis were more persuaded by the moral appeal than the practical appeal). They further explored the mediation effect of the perceptions/thoughts toward the message and the moderation effect of political orientation. To test the hypotheses, they conducted two studies on the attitudes toward recycling.

I will replicate the first study. This study is based on a survey by randomly assigning participants into the moral / practical groups, and testing the differences of their responses toward the moral / practical messages.

Key analysis of Interest

The key test is to test the interaction effect between the message type (moral / practical) and moral basis.

Justification

I choose this paper to replicate because I have been working on factor analysis and structural equation modeling, thus I am very interested in self-reported data. This paper matches my research experiences and interests as it focused on variables (e.g., attitude) that were measured by scales. Besides, this paper used questionnaires to measure the variables of interest and the reliabilities of the questionnaires were reported in the paper. However, the construct validity was not reported in the paper. In addition to replicating the research, I want to further test the validity of the measurement tools used in this paper.

Challenge

The potential challenge in this study is the funding to collect sufficient samples. In the original paper, the authors took one dollar to recruit each participant, and they recruited around 200 people.

Methods

Power Analysis

library(pwr)
pwr.f2.test(u = 1, #df for numerator; conducting the analysis for the single interaction term
            f2 = 0.03, #effect size from the original paper
            sig.level = .05, 
            power = .80)

## 
##      Multiple regression power calculation 
## 
##               u = 1
##               v = 261.5632
##              f2 = 0.03
##       sig.level = 0.05
##           power = 0.8

Planned Sample

The authors preregistered a target sample size of 200, which provided 80% power to detect an interaction effect as small as .04. They finally collected 415 samples for study 1. The power analysis above based on the effect of the original study suggested a target sample size of 262. However, to save the money, I plan to collect 50 samples and each participant will get $1 for completing the 7-minute survey.

Materials

This experiment will be conducted through qualtrics and participants will be required to fill out a survey (see the survey). The questionnaire on attitude toward recycling, political orientation, perceptions about the message, as well as the stimuli (practical or moral arguments against recycling) used in the replication study will be the same as Luttrell et al. ( 2019) (see the quote below).

Premessage attitudes. Participants rated their attitudes toward recycling using three 9-point semantic-differential scales, ranging from −4 (bad, dislike, and negative) to 4 (good, like, and positive).

Perceived attitude bases. Participants reported the degree to which their attitudes toward recycling were based on their core moral beliefs or on practical concerns using 5-point scales (ranging from not at all to extremely). These items were modeled after items previously used to assess moral conviction (see Skitka, 2010). These items were presented within a small set of randomly ordered questions about people’s bases for their recycling attitudes so participants would not suspect that the study was specifically about the moral basis for those attitudes.

Political orientation. Participants reported their political orientation on two items measuring ideology for social and economic issues. The items were measured on 5-point scales ranging from very liberal to very conservative.

Counterattitudinal message. We created two versions of an antirecycling essay that appealed either to moral or to practical concerns. The moral appeal, entitled “Recycling: Harmful and Immoral,” framed its antirecy- cling position in moral terms (e.g., “Supporting recycling programs would be a grave moral transgression”) and cited particular moral reasons against recycling programs (e.g., “precious pets and animals [are] mercilessly killed by fumes produced in the recycling process”). By contrast, the practical appeal, entitled “Recycling: Costly and Unfeasible,” framed its antirecycling position in prag- matic terms (e.g., recycling is an “inefficient and unfea- sible endeavor for most municipalities to adopt”) and cited particular economic and pragmatic concerns (e.g., “An increase in trucks greatly increases traffic, both on the highways and on city roads”). We chose a practical appeal as the comparison because it reflects a common type of nonmoral persuasive argument (e.g., Mucciaroni, 2011) that could be similarly substantive. Messages were similar in length (426–428 words) and in number of argu- ments, and they were designed to be equally cogent to the sample overall.

Thought listing. Participants next listed the thoughts they had while reading the antirecycling message. Participants were given six thought-listing boxes and told to enter one thought per box; however, they were not required to fill all six. These thoughts were subsequently rated by two independent coders for overall valence (i.e., whether each thought was promessage, antimessage, neutral to the message, or unrelated to the message), following common practice.

Note that since I will code it by myself, there will not be two independent coders.

Postmessage attitudes. After reading the message, participants once again reported their attitudes toward recycling, using the same items used to measure premessage attitudes.

Message ratings. To ensure that participants perceived the messages as intended, we asked how much the mes- sage seemed to make arguments related to moral and practical concerns, each on 7-point scales ranging from not at all to very much.

Procedure

At the beginning of the survey, participants will be asked to read an introduction to recycling and report their attitude about recycling. Next, participants will be randomly assigned into two groups, and in one group, they will need to read a message arguing against recycling using a moral appeal. In another group, participants will be assigned a message arguing against recycling using a practical appeal. After reading the message, participants in each group need to report their attitudes toward recycling again. Also, they will be asked about their perceptions of the message, political orientation, age, gender, and the primary language they are using.

Analysis Plan

Exclusion criteria

First, we exclude any cases for which the message would be pro-attitudinal (i.e., attitudes toward recycling below the midpoint.) Next, we excluded any cases that were duplicates of a previous participant’s IP Address.

Reliability of measures

Calculate Cronbach’s alpha for the measures

Validity of measures

Conduct confirmatory factor analysis for the measures

The CFA model didn’t converge given the small sample size.

Manipulation check

Data were submitted to t-test analyses to determine whether the messages were perceived as intended.

Persuasion effects

The data were submitted to a multiple regression analysis predicting postmessage attitudes in which pre- message attitudes, moral attitude basis, and message type were entered in the first step of the model and the Moral Attitude Basis × Message Type interaction term was entered in the second step. Message type was effects coded so that −1 corresponded with the moral- argument condition and +1 corresponded with the practical-argument condition. Results for these predic- tors are interpreted in the first steps of the model in which they appear.

Mediation effect of valenced thoughts

We computed the indirect effect and CIs with non- parametric bootstrapping (10,000 iterations) using the mediation package in R (Tingley, Yamamoto, Hirose, Keele, & Imai, 2014). We tested the indirect effect on postmessage attitudes, controlling for premessage atti- tudes, setting the Moral Attitude Basis × Message Type interaction term as the predictor; valenced thoughts as the mediator; premessage attitudes, moral attitude basis, and message type as covariates; and postmessage atti- tudes as the outcome variable.

Effects of other attitude bases

The fol- lowing analyses tested the interaction of each alternative basis with message type on postmessage attitudes, con- trolling for premessage attitudes.

Moderation effect of political orientation

To assess whether political orientation moderated the Moral Attitude Basis × Message Type interaction, we subjected the data to a hierarchical multiple regression model that tested a three-way interaction on postmessage attitudes.

Differences from Original Study

Sample size will be smaller than the original study.
The thought list will be coded by myself instead of two independent coders in the original study, so I cannot test consistency between coders.
The original paper didn’t conduct confirmatory factor analysis for construct validation, and I will try to conduct in the replication study.

Actual Sample

We recruited 50 participants and adopted the following exclusion criteria: - Replicated IP address - Same responses on all items - Hold a negative attitude basis toward recycling (N = 4)

The final sample size is 4 (17 female, 28 male, and one participant prefer not to say; age: Mean = 28.37, sd = 9.08; Moral Argument Group 16, Practical Argument Group 30).

Results

Data preparation

Data preparation following the analysis plan.

#### Import data
rawData <- read_csv("../data/data.csv", show_col_types = FALSE)
nrow(rawData)

## [1] 50

#### Data exclusion / filtering
data0 <- rawData |> 
  mutate(att1 = (att1_1 + att1_2 + att1_3) / 3) |>
  mutate(att2 = (att2_1 + att2_2 + att2_3) / 3) |>
  mutate(att.pre = 0 - att1, att.post = 0 - att2) |>
  mutate(ideology = (poli_soc + poli_econ) / 2) 
table(data0$cond.F)

Moral Arguement	Practical Arguement
19	31

No replicated IP address and no participants gave the same response on every items. Following the original paper, because the manipulation is to persuade participants that recycling is bad, and these already had negative attitudes toward recycling will be deleted (N = 4).

data1 <- data0 |> 
  filter(att1 >= 0)  
table(data1$cond.F)

Moral Arguement	Practical Arguement
16	30

#### add columns
data1$base_mor_low <- scale(data1$base_mor, scale = F) + sd(data1$base_mor)
data1$base_mor_high <- scale(data1$base_mor, scale = F) - sd(data1$base_mor)
data1$base_mor_max <- data1$base_mor - 5
data1$cond_mor <- (data1$cond + 1) / 2
data1$cond_prac <- (abs(data1$cond - 1))/2
data1$ideology_low <- scale(data1$ideology, scale = F) + sd(data1$ideology, na.rm = T)
data1$ideology_high <- scale(data1$ideology, scale = F) - sd(data1$ideology, na.rm = T)

#### thought listing
data1$pos <- rowSums(select(data1, matches("Th[1-6]VA")) == 1, na.rm = T)
data1$neg <- rowSums(select(data1, matches("Th[1-6]VA")) == 2, na.rm = T)
data1$neu <- rowSums(select(data1, matches("Th[1-6]VA")) == 3, na.rm = T)
data1$irr <- rowSums(select(data1, matches("Th[1-6]VA")) == 4, na.rm = T)
data1 <- mutate(data1, cogres = (pos - neg) / (pos + neg + neu))

#Irrelevant-only responses will have a zero denominator; the intention is that they're zero
data1$cogres[is.nan(data1$cogres)] <- 0


data1$cond.F = factor(data1$cond.F, levels=c("Moral Arguement", "Practical Arguement"))

Demographics

nrow(data1) # sample size

## [1] 46

table(data1$gender) # gender information

Female	Male	Other
17	28	1

summary(data1$age)

Min.	1st Qu.	Median	Mean	3rd Qu.	Max.
20	22	24.5	28.36957	32	56

sd(data1$age)

## [1] 9.083217

Reliability and Validity

#### reliability 
##### pre-message attitude
data1 |> select(att1_1, att1_2, att1_3) |> 
  psych::alpha() |> with(total) |> round(2)

	raw_alpha	std.alpha	G6(smc)	average_r	S/N	ase	mean	sd	median_r
	0.71	0.73	0.65	0.47	2.65	0.07	3.55	0.65	0.44

#### post-message attitude
data1 |> select(att2_1, att2_2, att2_3) |> 
  psych::alpha() |> with(total) |> round(2)

	raw_alpha	std.alpha	G6(smc)	average_r	S/N	ase	mean	sd	median_r
	0.96	0.96	0.95	0.89	25.04	0.01	2.2	1.84	0.91

#### Political Orientation
data1 |> select(poli_soc, poli_econ) |> 
  psych::alpha() |> with(total) |> round(2)

	raw_alpha	std.alpha	G6(smc)	average_r	S/N	ase	mean	sd	median_r
	0.72	0.73	0.58	0.58	2.75	0.08	2.59	0.78	0.58

#### validity
model <- '
att_f1 =~ att1_1 + att1_2 + att1_3
att_f2 =~ att2_1 + att2_2 + att2_3
poli =~ poli_soc + poli_econ
'
var = c("att1_1", "att1_2", "att1_3", "att2_1", "att2_2", "att2_3", "poli_soc", "poli_econ")
#cfa_fit <- bcfa(model, data1, var)
# summary(cfa_fit)
# fitMeasures(cfa_fit)
# the sample size is not enough for the cfa and model didn't converge

the sample size is not enough for the cfa and model didn’t converge

Manipulation Check

Perceived Morality and Practicality Between Message Conditions

Participants perceived the moral message as appealing more to morality (M = 4.94, SD = 1.24) than the practical message (M = 3.83, SD = 1.68) t(44) = 2.31,p = 0.026, d = 0.71.

Participants perceived the practical message as appealing more to practicality (M = 5.53., SD = 1.46) than the moral message (M = 3.62, SD = 1.67) t(44) = -4.03,p <.001, d = -1.25.

### Perceived Morality
display.t(y = data1$essMoral, x = data1$cond.F) |>
  kable(align=rep('c', 8)) |>
  kable_styling(full_width = F, position = "center") |>
  column_spec(1:8, width = "5em")

Mean1	SD1	Mean2	SD2	t	df	p	d
4.94	1.24	3.83	1.68	2.31	44	0.026	0.71

### Perceived Practicality
display.t(y = data1$essPrac, x = data1$cond.F) |>
  kable(align=rep('c', 8)) |>
  kable_styling(full_width = F, position = "center") |>
  column_spec(1:8, width = "5em")

Mean1	SD1	Mean2	SD2	t	df	p	d
3.62	1.67	5.53	1.46	-4.03	44	< .001	-1.25

Persuasion Effects

No overall effect of message type (p = .87), no overall effect of moral attitude basis: b = 0.258, se = .304, t(42) = 0.798, p = 0.429.

No significant interaction between message type and moral attitude basis, b = −0.009, t(41) = −0.029, p =0.977, 𝒇^𝟐 = 0

##### main effect
pers.ME <- lm(att.post ~ att.pre + cond + base_mor, data1)
 summary(pers.ME)

## 
## Call:
## lm(formula = att.post ~ att.pre + cond + base_mor, data = data1)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -1.8187 -1.2074 -0.4848  0.5192  5.4553 
## 
## Coefficients:
##             Estimate Std. Error t value Pr(>|t|)  
## (Intercept)  0.10706    1.59641   0.067   0.9469  
## att.pre      0.91100    0.44324   2.055   0.0461 *
## cond         0.04504    0.28313   0.159   0.8744  
## base_mor     0.25915    0.32000   0.810   0.4226  
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 1.812 on 42 degrees of freedom
## Multiple R-squared:  0.09235,    Adjusted R-squared:  0.02752 
## F-statistic: 1.425 on 3 and 42 DF,  p-value: 0.2491

##### interaction
pers.int <- lm(att.post ~ att.pre + cond * base_mor, data1)
 summary(pers.int)

## 
## Call:
## lm(formula = att.post ~ att.pre + cond * base_mor, data = data1)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -1.8156 -1.2035 -0.4855  0.5140  5.4593 
## 
## Coefficients:
##                Estimate Std. Error t value Pr(>|t|)  
## (Intercept)    0.113811   1.631923   0.070   0.9447  
## att.pre        0.912219   0.450496   2.025   0.0494 *
## cond           0.076581   1.108264   0.069   0.9452  
## base_mor       0.258786   0.324116   0.798   0.4292  
## cond:base_mor -0.008979   0.304785  -0.029   0.9766  
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 1.834 on 41 degrees of freedom
## Multiple R-squared:  0.09237,    Adjusted R-squared:  0.003825 
## F-statistic: 1.043 on 4 and 41 DF,  p-value: 0.3968

##### effect size
R2.ME <- summary(pers.ME) |> with(r.squared)
R2.Full <- summary(pers.int) |> with(r.squared)
f2 <- round((R2.Full - R2.ME)/(1 - R2.Full), 2)
print(paste("f2 = ", f2))

## [1] "f2 =  0"

Visualization

Fig1 <- ggpredict(lm(att.post ~ att.pre + cond.F * base_mor, data1),
          terms = c("base_mor [meansd]", "cond.F")) %>%
  ggplot(aes(x = x, y = predicted, linetype = group)) +
  geom_line(size = 2, color = "black") +
  apatheme +
  ylim(c(-4, 0)) +
  labs(y = "Post-Message Attitudes",
       x = "Moral Attitude Basis",
       linetype = "") +
  theme(legend.position = "top",
        legend.key.width = unit(.5, "inches"),
        text = element_text(size = 16),
        axis.text = element_text(size = 12),
        axis.title.x = element_text(margin = margin(t = 12, b = 5)),
        axis.title.y = element_text(margin = margin(r = 8)),
        legend.text = element_text(size = 14))
ggsave(plot = Fig1, "fig1.png", dpi=300, width = 8, height = 6)
print(Fig1)

Figure 1 from the original paper

Mediation Effect of Valenced Thoughts

Following the original paper, I rated the overall valence (i.e., whether each thought was promessage, antimessage, neutral to the message, or unrelated to the message) of the thoughts that participants list on the free response field, and test the mediation effect of valence thoughts. I cannot test the rater consistency reliability because there is no second rater.

cogres.int <- lm(cogres ~ cond * base_mor, data1)
#summary(cogres.int)

data1$int <- data1$cond * data1$base_mor
model.y <- lm(att.post ~ int + cogres + att.pre + cond + base_mor, data1)
#summary(model.y)
model.m <- lm(cogres ~ int + att.pre + cond + base_mor, data1)
med.model <- mediation::mediate(model.m, model.y, treat = "int", mediator = "cogres", 
                                boot = TRUE, sims = 1000, conf.level = .95)

## Running nonparametric bootstrap

summary(med.model)

## 
## Causal Mediation Analysis 
## 
## Nonparametric Bootstrap Confidence Intervals with the Percentile Method
## 
##                Estimate 95% CI Lower 95% CI Upper p-value
## ACME           -0.08595     -0.30568         0.14    0.38
## ADE             0.07697     -0.37528         0.56    0.81
## Total Effect   -0.00898     -0.47205         0.56    0.92
## Prop. Mediated  9.57274     -5.90286         5.65    0.72
## 
## Sample Size Used: 46 
## 
## 
## Simulations: 1000

Moderation by Political Orientation

ideo.ME <- lm(att.post ~ att.pre + cond + base_mor + ideology, data1)
summary(ideo.ME)

## 
## Call:
## lm(formula = att.post ~ att.pre + cond + base_mor + ideology, 
##     data = data1)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -1.8496 -1.1863 -0.5047  0.4919  5.4162 
## 
## Coefficients:
##             Estimate Std. Error t value Pr(>|t|)  
## (Intercept) -0.15716    2.07044  -0.076   0.9399  
## att.pre      0.89883    0.45234   1.987   0.0536 .
## cond         0.04397    0.28646   0.153   0.8788  
## base_mor     0.26826    0.32678   0.821   0.4164  
## ideology     0.07308    0.35837   0.204   0.8394  
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 1.833 on 41 degrees of freedom
## Multiple R-squared:  0.09327,    Adjusted R-squared:  0.004813 
## F-statistic: 1.054 on 4 and 41 DF,  p-value: 0.3913

ideo.2way <- lm(att.post ~ att.pre + cond * base_mor + cond * ideology + base_mor * ideology, data1)
summary(ideo.2way)

## 
## Call:
## lm(formula = att.post ~ att.pre + cond * base_mor + cond * ideology + 
##     base_mor * ideology, data = data1)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -2.0764 -1.2631 -0.4341  0.6365  5.3698 
## 
## Coefficients:
##                   Estimate Std. Error t value Pr(>|t|)
## (Intercept)        -5.4547     5.9805  -0.912    0.367
## att.pre             0.7394     0.4806   1.538    0.132
## cond               -1.2785     1.7150  -0.745    0.461
## base_mor            1.6286     1.4910   1.092    0.282
## ideology            1.9168     2.0850   0.919    0.364
## cond:base_mor       0.1008     0.3224   0.313    0.756
## cond:ideology       0.3740     0.3965   0.943    0.352
## base_mor:ideology  -0.5391     0.5688  -0.948    0.349
## 
## Residual standard error: 1.865 on 38 degrees of freedom
## Multiple R-squared:  0.1299, Adjusted R-squared:  -0.03036 
## F-statistic: 0.8106 on 7 and 38 DF,  p-value: 0.5841

ideo.3way <- lm(att.post ~ att.pre + cond * base_mor * ideology, data1)
summary(ideo.3way)

## 
## Call:
## lm(formula = att.post ~ att.pre + cond * base_mor * ideology, 
##     data = data1)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -2.2907 -1.2289 -0.3595  0.6860  5.4443 
## 
## Coefficients:
##                        Estimate Std. Error t value Pr(>|t|)
## (Intercept)             -5.8335     6.0885  -0.958    0.344
## att.pre                  0.7327     0.4856   1.509    0.140
## cond                    -3.9692     5.6822  -0.699    0.489
## base_mor                 1.7147     1.5160   1.131    0.265
## ideology                 2.0251     2.1172   0.957    0.345
## cond:base_mor            0.8358     1.5138   0.552    0.584
## cond:ideology            1.3954     2.0931   0.667    0.509
## base_mor:ideology       -0.5646     0.5768  -0.979    0.334
## cond:base_mor:ideology  -0.2833     0.5698  -0.497    0.622
## 
## Residual standard error: 1.884 on 37 degrees of freedom
## Multiple R-squared:  0.1357, Adjusted R-squared:  -0.05119 
## F-statistic: 0.7261 on 8 and 37 DF,  p-value: 0.6676

data1$att.pre.c <- scale(data1$att.pre, scale = F) %>% as.numeric() #Centering the covariate to estimate predicted values at the mean of this variable.

Fig2 <- lm(att.post ~ att.pre.c + cond.F * base_mor * ideology, data1) %>%
  ggpredict(terms = c("base_mor [meansd]", "cond.F", "ideology [1,2,3,4,5]")) %>%
  mutate(ideo.F = factor(facet, labels = c("Very Liberal", "Somewhat Liberal", 
                                           "Neither", "Somewhat Conservative", "Very Conservative"))) %>%
  ggplot(aes(x, predicted, linetype = group)) +
geom_line(size = 2) +
  apatheme +
  ylim(c(-4, 0)) +
  facet_wrap(~ ideo.F, nrow = 1) +
  labs(y = "Post-Message Attitudes",
       x = "Moral Attitude Basis",
       linetype = "",
       title = "A") +
  #guides(fill = F, color = F, 
  #       linetype=guide_legend(keywidth=0.5, default.unit="inch")) +
  theme(legend.position = "top",
        legend.key.width = unit(.5, "inches"),
        text = element_text(family = "Times New Roman", size = 16),
        axis.text = element_text(size = 12),
        axis.title.x = element_text(margin = margin(t = 12, b = 5)),
        axis.title.y = element_text(margin = margin(r = 8)),
        legend.text = element_text(size = 14),
        strip.text = element_text(size = 10),
        plot.title = element_text(size = 16, hjust = -.045))
ggsave(plot = Fig2, "fig2.png", dpi=300, width = 8, height = 6)

## Warning: Removed 1 row(s) containing missing values (geom_path).

print(Fig2)

## Warning: Removed 1 row(s) containing missing values (geom_path).

Fig2B <- ggplot(data1, aes(x = ideology)) + 
  geom_histogram(bins = 5, color = "white", fill = "black") +
  geom_rug(sides = "b", aes(y = 0), position = position_jitter(width = .1), alpha = .2) +
  labs(y = "Frequency", x = "Ideology", title = "B\n") +
  apatheme +
  theme(text = element_text(family = "Times New Roman", size = 12),
        axis.title.x = element_text(size = 16),
        axis.title.y = element_text(size = 16),
        plot.title = element_text(size = 16, hjust = -.05))

ggsave(plot = Fig2B, "fig2B.png", dpi=300, width = 8, height = 6)
print(Fig2B)

Figure 2 from the original paper

Exploring the differences between the replication study and the original study

The influence of age

#####  effect of age
lmage <- lm(att.post ~ att.pre + cond + base_mor + age , data1)
 summary(lmage)

## 
## Call:
## lm(formula = att.post ~ att.pre + cond + base_mor + age, data = data1)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -2.2678 -1.0597 -0.5116  0.7019  4.7167 
## 
## Coefficients:
##             Estimate Std. Error t value Pr(>|t|)  
## (Intercept) -2.34109    1.77028  -1.322   0.1934  
## att.pre      0.97033    0.41643   2.330   0.0248 *
## cond        -0.11117    0.27235  -0.408   0.6853  
## base_mor     0.42008    0.30654   1.370   0.1780  
## age          0.07530    0.02903   2.593   0.0131 *
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 1.7 on 41 degrees of freedom
## Multiple R-squared:  0.2203, Adjusted R-squared:  0.1442 
## F-statistic: 2.895 on 4 and 41 DF,  p-value: 0.03361

summary(data1$att.pre)

Min.	1st Qu.	Median	Mean	3rd Qu.	Max.
-4	-4	-3.666667	-3.550725	-3.333333	-1

Fig1b <- ggpredict(lm(att.post ~ att.pre + cond.F * age, data1),
          terms = c("age [meansd]", "cond.F")) %>%
  ggplot(aes(x = x, y = predicted, linetype = group)) +
  geom_line(size = 2, color = "black") +
  apatheme +
  ylim(c(-4, 0)) +
  labs(y = "Post-Message Attitudes",
       x = "Age",
       linetype = "") +
  #scale_x_continuous(breaks=pretty_breaks(n=5)) +
  theme(legend.position = "top",
        legend.key.width = unit(.5, "inches"),
        text = element_text(size = 16),
        axis.text = element_text(size = 12),
        axis.title.x = element_text(margin = margin(t = 12, b = 5)),
        axis.title.y = element_text(margin = margin(r = 8)),
        legend.text = element_text(size = 14)) 
print(Fig1b)

ggsave(plot = Fig1b, "fig1b_rep.png", dpi=300, width = 6, height = 6)

Data from the original paper

load('../original_paper/original.RData')
lmage_ori <- lm(att.post ~ att.pre + cond + base_mor + age , S1)
 summary(lmage_ori)

## 
## Call:
## lm(formula = att.post ~ att.pre + cond + base_mor + age, data = S1)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -2.6909 -1.1281 -0.5105  0.6671  7.0172 
## 
## Coefficients:
##               Estimate Std. Error t value Pr(>|t|)    
## (Intercept)  0.8764265  0.4409947   1.987   0.0475 *  
## att.pre      0.5287066  0.0927778   5.699 2.31e-08 ***
## cond         0.0722810  0.0833867   0.867   0.3865    
## base_mor    -0.3406607  0.0907586  -3.753   0.0002 ***
## age         -0.0000951  0.0080627  -0.012   0.9906    
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 1.693 on 410 degrees of freedom
## Multiple R-squared:  0.1744, Adjusted R-squared:  0.1663 
## F-statistic: 21.65 on 4 and 410 DF,  p-value: 3.194e-16

S1$cond.F = factor(S1$cond.F, levels=c("Moral Message", "Practical Message"))
Fig1b_ori <- ggpredict(lm(att.post ~ att.pre + cond.F * age, S1),
          terms = c("age [meansd]", "cond.F")) %>%
  ggplot(aes(x = x, y = predicted, linetype = group)) +
  geom_line(size = 2, color = "black") +
  apatheme +
  ylim(c(-4, 0)) +
  labs(y = "Post-Message Attitudes",
       x = "Age",
       linetype = "") +
  #scale_x_continuous(breaks=pretty_breaks(n=5)) +
  theme(legend.position = "top",
        legend.key.width = unit(.5, "inches"),
        text = element_text(size = 16),
        axis.text = element_text(size = 12),
        axis.title.x = element_text(margin = margin(t = 12, b = 5)),
        axis.title.y = element_text(margin = margin(r = 8)),
        legend.text = element_text(size = 14)) 
print(Fig1b_ori)

ggsave(plot = Fig1b_ori, "fig1b_ori.png", dpi=300, width = 6, height = 6)

t.test(data1$age, S1$age)

## 
##  Welch Two Sample t-test
## 
## data:  data1$age and S1$age
## t = -4.3299, df = 59.002, p-value = 5.864e-05
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
##  -9.078028 -3.339468
## sample estimates:
## mean of x mean of y 
##  28.36957  34.57831

Scatter Plot of Pre- and Post-message attitude

ggplot(data = data1) + 
  geom_point(aes(x=att.pre, y=att.post)) + 
  labs(y = "Post-Message Attitudes",
       x = "Pre-Message Attitudes") + 
  apatheme +
  theme(legend.position = "top",
        legend.key.width = unit(.5, "inches"),
        text = element_text(size = 16),
        axis.text = element_text(size = 12),
        axis.title.x = element_text(margin = margin(t = 12, b = 5)),
        axis.title.y = element_text(margin = margin(r = 8)),
        legend.text = element_text(size = 14))

ggsave("point_rep.png", dpi=300, width = 8, height = 6)

ggplot(data = S1) + 
  geom_point(aes(x=att.pre, y=att.post)) + 
  labs(y = "Post-Message Attitudes",
       x = "Pre-Message Attitudes") + 
  apatheme +
  theme(legend.position = "top",
        legend.key.width = unit(.5, "inches"),
        text = element_text(size = 16),
        axis.text = element_text(size = 12),
        axis.title.x = element_text(margin = margin(t = 12, b = 5)),
        axis.title.y = element_text(margin = margin(r = 8)),
        legend.text = element_text(size = 14))

ggsave("point_ori.png", dpi=300, width = 8, height = 6)

Discussion

Summary of Replication Attempt

In the original paper, the result of the key test is: no overall effect of message type (p = .39), but there was an overall effect of moral attitude basis: b = −0.34, 95% CI = [−0.52, −0.16], t(411) = −3.77, p < .001. significant interaction between message type and moral attitude basis, b = −0.27, 95% CI = [−0.42, −0.11], t(410) = −3.34, p < .001, f2 = .03.

The result of this paper is: No overall effect of message type (p = .87), no overall effect of moral attitude basis: b = 0.258, se = .304, t(42) = 0.798, p = 0.429. No significant interaction between message type and moral attitude basis, b = −0.009, t(41) = −0.029, p =0.977, f2 = 0.

So I failed to replicate the study 1 of original paper. Also, figure 1 and figure 2 showed different pattern from the original paper.

Commentary

There are two potential reasons for the different results of the replication study and the original study.

First, the reliability of the pre-message attitude scale is lower (Cronbach’s alpha = 0.71) in the replication study compared to the orginal paper (0.97). The low internal consistency made the measures of pre-message attitude less reliable.
Second, the results based on the exploration of age suggested that the age of the samples in the replication study is much younger than the original study. Also, a trend of the interaction between age and message type was found in the figure. But currently the sample size is not enough to test the influence of age in this study.
Third, the last two figures suggested that most participant in this study hold extreme pro-recycling attitude, which is differnt from the data of the original paper.

Overall, the results of the replication study was very different from the original papaer. The results are not significant, and even the pattern of data was also different. For example, figure 2 on the three way interaction between message type, political orientation and moral attitude basis. A clear pattern of the interaction between political orientation and moral attitude basis was found in this replication study. Future study are sugggested to collect more data to test this effect.

Reference

Luttrell, A., Philipp-Muller, A., & Petty, R. E. (2019). Challenging moral attitudes with moral messages. Psychological Science, 30(8), 1136-1150.