Replication of Study 1 by Hopkins, Weisberg, and Taylor (2016, Cognition)
Author
Howard Chiu (howardchiu@stanford.edu)
Published
March 29, 2024
Introduction
The reductive allure has been most frequently studied in the context of neuroscientific explanations for phenomenon that may not necessarily require them, and has been thought to cause people to perceive these explanations as being higher in quality, even if the neuroscientific jargon offers no additional information.
Beyond evaluating scientific literature and findings, this effect has garnered some attention in scientific communication as its effect is thought to extend even to decisions on medical treatment, juries in legal proceedings, and misappropriated to advance social or political agenda.
This study attempts to examine if the reductive effect generalizes beyond that of neuroscience. Specifically, it investigates whether participants would prefer reductive explanations across 6 scientific domains, and whether this would differ by the science of the phenomenon presented.
Description of stimuli and procedures
Horizontal explanations were defined as referring “only to the science from which the phenomenon itself was drawn (e.g., using biological language to explain a biological phenomenon)”, while reductive explanations were defined as explanations which “included information from the next lower level in the hierarchy (e.g., using chemical language to explain a biological phenomenon)”.
Participants were presented with horizontal and reductive explanations of phenomenon in 6 sciences: physics, chemistry, biology, neuroscience, psychology, and social sciences. The stimuli were carefully constructed so that the reductive information would not provide any explanatory power, allowing the authors to separate out a potential preference for explanations with a reductive form even in the absence of any helpful reductive content.
The original authors observed that there was a main effect of explanation level, where participants rated reductive explanations as of higher quality than horizontal explanations. They also observed that there was a relatively positive view of neuroscience combined with a relatively negative view of psychology, with information in these domains positively and negatively influencing the perceived quality of the reductive explanations respectively.
Challenges
Participants may approach this task, and the explanations provided, with vastly different background knowledge, as was observed in the original study. As the replication sample will be crowd-sourced online via Prolific, participants may be motivated to complete the task as fast as possible rather than attempt to maximise their performance on the task, even with the instructional check at the start of the experiment and the attentional check in the middle of the Rating Explanations task.
Hopkins, E. J., Weisberg, D. S., & Taylor, J. C. V. (2016). The seductive allure is a reductive allure: People prefer scientific explanations that contain logically irrelevant reductive information. Cognition, 155, 67–76. https://doi.org/10.1016/j.cognition.2016.06.011
The primary research questions were “whether participants would prefer reductive explanations and whether this would differ by the science of the phenomenon”.
The measure of interest was the mean rating of quality of reductive and horizontal explanations of various scientific findings provided to participants along a sliding scale ranging from 3 (very good) to -3 (very poor).
The replication utilised the methodology exactly as reported in the original study for the Rating Explanations task, with the exception of using a sample completely recruited from MTurk and no undergraduates. The replication study also omitted tests of scientific literacy, reflective thinking, logical syllogisms, as well as perceptions of science (Sections 2.4.2 to 2.4.5 of the original study) to reduce completion time.
There were a high number of participants who failed the attention checks (70 out of 163 participants) in the replication study.
However, even when the participants who had failed attention checks (either or both) were included in the analysis, the replication study did not manage to replicate the main effect of explanation quality, as well as the significant effect of explanation level.
The repository for the previous replication is hosted here.
Original Findings
The linear mixed-effects regression in the original study revealed a significant main effect of explanation level (as reported in Figure 2 of original paper): reductive explanations (n=131, M=1.26, SD=1.71) were rated signficantly higher on average than horizonal explanations (n=128, M=1.04, SD=1.88). Effect sizes (Cohen’s d for within-subjects tests) were only reported for pairwise differences between adjacent fields in the hierarchy of sciences (Figure 4 of original paper).
Participants (20 MTurk workers and 40 undergraduates out of the complete recruited sample of 167 MTurk workers and 152 undergraduates) who failed attention checks were excluded from the sample (i.e. 60 out of the total recruited sample of 319 participants).
Methods
Power Analysis
The key statistical test for the original study and the subsequent replications is the main fixed effect of condition on rating of explanation quality within the linear mixed-effects model.
No power analysis or sample size justification was provided in the original paper, and the replication sample utilised the same MTurk sample size (n = 167) from the original experiment to preserve power across both experiments.
A power analysis as a t-test of difference between two independent means was conducted as a proxy due to the difficulty of estimating the influence of random effects on power.
As explanation level was a between-subject manipulation, the power calculation is conducted based on the number of items tested rather than the number of participants in the study. In this case, each participant rates 12 explanations, and thus the number of explanations rated in the reductive condition was 876, and the number of explanations rated in the horizontal condition was 888. The main analysis of a two-tailed t-test of two independent means thus gives us a post-hoc power of 72.9%. The post-hoc power analysis was conducted with G*Power 3.1.
For the replication study, the post-hoc power was 36.4% after excluding responses for participants who had failed either attention check. The mean rating of horizontal explanations was 1.61 (SD = 1.50, ntrials = 456) and the mean rating of reductive explanations was 1.42 (SD = 1.47, ntrials = 660).
Planned Sample
Like the original and replication samples, English-speaking adult participants residing in the United States will be recruited online (via Prolific instead of MTurk). Participants who complete the task and pass both attention checks will be included in the final sample for analysis. Details about the attention checks can be found in the Procedure or Control sections below.
Due to the high number of exclusions in in both the original study (60 participants) and the replication study (70 participants), we will be building in a 20% buffer in our planned sample recruitment.
With a planned sample of 200 workers on Prolific, and a 20% exclusion rate for failed attention checks, that would leave us approximately 160 participants. With 80 participants in each condition completing 12 items each, the number of explanations rated in each of the two conditions would be 960. This will give us a planned power of 76.4%.
If all the data collected from the 200 participants can be used in the analysis (yielding 1200 items rated in each condition), this will give us a planned power of 85%.
Materials
The detailed process of constructing the stimuli can be found in the original paper. All materials for the Rating Explanations task can be found in Appendix A of the original paper and will be used without modifications.
Procedure
The following procedure is quoted from the original article:
All participants completed an online survey hosted by Qualtrics. The survey had six components: Rating Explanations, Science Literacy, Reflective Thinking, Logical Syllogisms, Perceptions of Science, and Demographic Information. All participants completed the Rating Explanations component first and the Demographic Information last. The other four components were presented in a random order after the Rating Explanations component. The Rating Explanations task used a 2 (Explanation Level: horizontal, reductive) x 2 (Quality: good, bad) x 6 (Science: physics, biology, chemistry, neuroscience, psychology, social science) design. To be consistent with previous studies of this effect (Weisberg et al., 2008, 2015), explanation level was between-subjects: Participants were randomly assigned to either the horizontal or reductive condition. Quality and science were within-subjects: All participants rated two explanations from each science, one good and one bad.
Unlike the original study, the participants will not be completing the Science Literacy, Reflective Thinking, Logical Syllogisms, and Perceptions of Science sections. They will thus only complete two components - the Rating Explanations, followed by Demographic Information.
Rating Explanations
The following paragraphs are quoted from the original article:
All participants completed the Rating Explanations task first. For this task, participants used a sliding scale ranging from 3 to -3 to indicate their responses. They were first given instructions on how to use the slider; this also served as a check that participants were reading instructions. They were told to use the slider to select 0 on the first page in order to proceed with the survey. If they selected anything other than 0, they were directed to another page asking them again to select 0. Participants who did not select the correct response on this second page were excluded from analyses.
After these general instructions on using the slider, participants were given instructions for the explanations task (modified from Fernandez-Duque et al., 2015):
You will now be presented with descriptions of various scientific findings. All the findings come from solid, replicable research; they are the kind of material you would encounter in a textbook. You will also read an explanation of each finding. Unlike the findings themselves, the explanations of the findings range in quality. Some explanations are better than others: They are more logically sound. Your job is to judge the quality of such explanations, which could range from very poor (-3) to very good (+3).
On each trial, participants were presented with a description of a scientific phenomenon, which was displayed for 10 s before participants could advance to the next screen. On the next screen, an explanation was displayed below the phenomenon, and participants were instructed to rate the quality of the explanation. Participants rated 12 explanations, with an attention check trial administered after the first six (Oppenheimer, Meyvis, & Davidenko, 2009). This trial was similar in format to the others. First, a description of a phenomenon was presented for 10 s. When participants advanced to the next screen, instead of seeing an explanation, they saw text instructing them to select 3 on the scale. Participants who did not select 3 were excluded from analyses.
Demographics
The following procedure is quoted from the original article:
At the end of the survey, participants answered a series of demographic questions, including gender, age, and year in school (for undergraduates) or highest degree completed (for MTurk workers). Participants from both samples were asked to pick the category that most closely matched the field of their highest degree (physical sciences, social sciences, engineering, humanities, health, and business) and to give the exact field. They were also asked whether they had taken any college- or graduate-level courses in anthropology, chemistry, physics, sociology, economics, neuroscience, psychology, political science, biology, or philosophy.
Since undergraduates will not be recruited for this sample, the demographic questions will only include gender, age, highest degree completed, category that closely matched the field of their highest degree (physical sciences, social sciences, engineering, humanities, health, and business), and college- or graduate-level coursework.
In addition, this replication project not be asking about coursework in anthropology and philosophy as these subjects are tangential to the main research questions. Instead, the list of courses will have a combined social sciences item for anthropology/economics/philopsphy so that there will be 6 scientific disciplines in the list which are aligned to the rating explanations. (The original paper also averaged sociology, economics, and political science into a single post-hoc measure when analysing participants’ perceptions of science as they found that the three domains were highly correlated.) This rescue study preserves the coursework question as the original study observed that “the total number of sciences (out of 6) in which a participant had taken courses was a significant predictor of overall difference scores in a linear regression: F (1, 214) = 6.27, p < 0.05, R2 = 0.02.”
Controls
Both attention checks are placed within the Rating Explanations block of the survey.
The first attention check simply requires participants to select 0 on the sliding scale ranging from -3 to 3 after reading instructions on the Rating Explanations task. The second attention check occurs after participants rate 6 out of the 12 explanations. A phenomenon is also presented for 10 seconds, but “when participants advanced to the next screen, instead of seeing an explanation, they saw text instructing them to select 3” on the same sliding scale.
In addition, the description of each scientific phenomena was “displayed for 10 s before participants could advance to the next screen” to ensure that they had sufficient time to read before they could view the explanations provided. This delay should also ensure greater deliberation as participants could not rush through the questions as quickly.
The exact stimuli used for attention checks were not provided in the original paper, so the attention checks in the replication were used. The intermediate attention check is reproduced here for future reference:
People are generally better at reasoning about typical members of a category than unusual members of that category. For example, if a person is told a fact about a type of bird (“All sparrows have livers”) and asked whether that fact is true of all birds, they are more likely to answer correctly if the fact is given about a stereotypical member of the bird category (e.g., sparrows, robins) than if an atypical bird is used (e.g., peacocks, penguins).
Why are people better at reasoning about stereotypical animals?
Research shows that survey participants sometimes fail to pay full attention to the contents of survey items. In order to ensure that tasks are being performed correctly, researchers include questions that require participants to select a particular answer. Instead of rating the quality of this explanation, please select three on the scale below before continuing to the next question.
Analysis Plan
The following procedure for the key analysis of interest is quoted from the original article:
Data from the explanations task were analyzed with a mixed-effects linear regression model (using the lme4 package in R) predicting the rating given on each trial from the sample (MTurk, undergraduates), the quality of the explanation (good, bad), the explanation level (horizontal, reductive), and the science from which the phenomenon was drawn (physics, chemistry, biology, neuroscience, psychology, and social science). Sample and explanation level were between-participants variables and quality and science were within-participants. All possible interactions between variables were tested. […] the best-fitting model included random intercepts by participant and item as well as a random effect of item on the slope for the quality variable.
Similar to the replication but not the original study, the between-participants variable “sample” will not be available for analysis as there will not be an undergraduate population recruited.
Differences from Original Study and 1st replication
Unlike the original study and replication, the sample will be recruited online via Prolific rather than Amazon Mechanical Turk (MTurk). The other differences have been detailed in the sections above (including not having an undergraduate sample, shortened Demographic Information section, and the removal of the Science Literacy, Reflective Thinking, Logical Syllogisms, and Perceptions of Science sections that were deemed tangential to the main research question).
Given that the experimental materials and procedure involved in testing the primary research questions remain identical, and the analysis plan is identical save for one between-participants variable which did not make a difference in the analysis in the original study, these adjustments to the original study are not expected to make a difference based on claims in the original article.
The sample has been increased from the previous replication to increase the power, and the instruction for participants to select a specific value on the slider in the attention check was bolded to increase the likelihood that they would see it and thus comply.
Methods Addendum (Post Data Collection)
Actual Sample
39 participants were excluded for failing both attention checks, within the planned buffer for exclusions. This left a total of 161 participants across both conditions, with 81 participants were randomly assigned to the horizontal condition, and 80 participants were randomly assigned to the reductive condition.
Within this replication sample, there was no significant difference between conditions for the proportion of males (chi-squared test: X-squared = 1.863, df = 2, p = 0.394) and for the mean age (two-tailed t-test: t = -1.234, df = 1858.1, p = 0.218).
Comparing between this replication sample and the original sample, the mean age for this sample was 36.7 as compared to 39.8 in the MTurk sample, 47.2% reported having an associate’s or bachelor’s degree as compared to 46.3% in the MTurk sample, and 14.9% reported having a master’s degree or higher as compared to 15.0% in the MTurk sample.
Thus, demographic variables are unlikely to have affected the subsequent analyses.
Differences from pre-data collection methods plan
None.
Results
Data preparation
Data preparation following the analysis plan. Code has been kindly provided by the author of the replication study, and will mostly be used as is with some edits for debugging and naming conventions.
Load qualtRics, tidyverse, and lme4 libraries.
Read in data from original study, replication study, and current rescue study.
Remove columns unnecessary for primary analysis.
Select participants who passed both attention checks.
Reshape to extract summary statistics for included participants.
Convert ratings into integers and factor variables into categories (good and bad explanations, horizontal and reductive explanations).
Run 2x2x6 linear mixed-effects linear regression predicting explanation rating based on the quality of the explanation (good or bad), the explanation level (horizontal or reductive), and the science from which the phenomenon was drawn (6 sciences).
Visualize plots.
Conduct exploratory analyses if needed.
### Data Preparation#### Load Relevant Libraries and Functionslibrary(qualtRics) #To read data from Qualtricslibrary(tidyverse) #To clean data and plot graphs in ggplot
── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ──
✔ dplyr 1.1.4 ✔ readr 2.1.5
✔ forcats 1.0.0 ✔ stringr 1.5.1
✔ ggplot2 3.5.0 ✔ tibble 3.2.1
✔ lubridate 1.9.3 ✔ tidyr 1.3.1
✔ purrr 1.0.2
── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
✖ dplyr::filter() masks stats::filter()
✖ dplyr::lag() masks stats::lag()
ℹ Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errors
library(lme4) #To run mixed-effects linear regression model in original study
Loading required package: Matrix
Attaching package: 'Matrix'
The following objects are masked from 'package:tidyr':
expand, pack, unpack
library(lmerTest)
Attaching package: 'lmerTest'
The following object is masked from 'package:lme4':
lmer
The following object is masked from 'package:stats':
step
library(brms) #To run Bayesian supplementary analyses
Loading required package: Rcpp
Loading 'brms' package (version 2.21.0). Useful instructions
can be found by typing help('brms'). A more detailed introduction
to the package is available through vignette('brms_overview').
Attaching package: 'brms'
The following object is masked from 'package:lme4':
ngrps
The following object is masked from 'package:stats':
ar
library(broom.mixed)library(kableExtra)
Attaching package: 'kableExtra'
The following object is masked from 'package:dplyr':
group_rows
library(metafor)
Loading required package: metadat
Loading required package: numDeriv
Loading the 'metafor' package (version 4.4-0). For an
introduction to the package please type: help(metafor)
library(sjPlot)
Learn more about sjPlot with 'browseVignettes("sjPlot")'.
── Column specification ────────────────────────────────────────────────────────
cols(
.default = col_double(),
Finished = col_logical(),
Gender = col_character(),
Year = col_character(),
`Course Type` = col_character(),
`Courses Taken` = col_character(),
Condition = col_character(),
Set = col_character()
)
ℹ Use `spec()` for the full column specifications.
data_num <- raw_data |>mutate(ID =1:nrow(raw_data)) #add participant ID columndata_num <- data_num |>select(Att_1, everything()) #move attention check 2 to front of data frame#### Prepare data for analysis - create columns etc.#convert to long formatdata_long <- data_num |>pivot_longer(cols ='P1-HG_1':'S4-RB_1', names_to ="full_item", values_to ="rating", values_drop_na=TRUE)#specify item attributesdata_long <- data_long |>mutate(science =substr(full_item, start=1, stop=1))data_long <- data_long |>mutate(item =substr(full_item, start=1, stop=2))data_long <- data_long |>mutate(quality =substr(full_item, start=5, stop=5))data_long <- data_long |>select(-full_item)data_long <- data_long |>filter(rating!="")#recode item namesdata_long$quality <-ifelse(data_long$quality=="G", "good", "bad")data_long$science <-ifelse(data_long$science=="P", "physics", data_long$science)data_long$science <-ifelse(data_long$science=="C", "chemistry", data_long$science)data_long$science <-ifelse(data_long$science=="B", "biology", data_long$science)data_long$science <-ifelse(data_long$science=="N", "neuroscience", data_long$science)data_long$science <-ifelse(data_long$science=="Y", "psychology", data_long$science)data_long$science <-ifelse(data_long$science=="S", "social", data_long$science)#convert columns to intended data types for analysisdata_long$rating <-as.integer(as.character(data_long$rating))#exclude participants who failed attention check 1data_excluded_both <- data_long |>filter(Slider1_1==0)#exclude participants who failed attention check 2data_excluded_both <- data_excluded_both |>filter(Att_1==3)
Results of control measures
#total number of participantsunfiltered_rows = data_long %>%nrow()unfiltered_total = (unfiltered_rows /12)#total number of participants who passed attention checksfiltered_rows = data_excluded_both %>%nrow()filtered_total = (filtered_rows /12)# number of excluded participants num_excluded = unfiltered_total - filtered_total num_excluded
[1] 39
#final number of included participants filtered_total
# Assuming data_excluded_both is your datasetgroup1_age <-subset(data_excluded_both, Condition =="H")$Agegroup2_age <-subset(data_excluded_both, Condition =="R")$Age# Perform independent t-testt_test_result <-t.test(group1_age, group2_age)t_test_result
Welch Two Sample t-test
data: group1_age and group2_age
t = -1.2337, df = 1858.1, p-value = 0.2175
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
-1.7820674 0.4058016
sample estimates:
mean of x mean of y
36.36250 37.05063
Confirmatory analysis
#Read in and format datadata <- data_excluded_both# data <- data_long# Convert to factordata$science <-as.factor(data$science)data$quality <-as.factor(data$quality)data$condition <-as.factor(data$Condition)#Reorder levels of the factors to facilitate contrast coding in regressions:data$science <-relevel(data$science, ref ="social")data$science <-relevel(data$science, ref ="psychology")data$science <-relevel(data$science, ref ="neuroscience")data$science <-relevel(data$science, ref ="biology")data$science <-relevel(data$science, ref ="chemistry")data$science <-relevel(data$science, ref ="physics")#Backward difference coding; each level is compared to the mean of the one before it contrasts(data$quality) <-matrix(c(-1/2,1/2), nrow=2)contrasts(data$condition) <-matrix(c(-1/2,1/2), nrow=2)#Deviation coding: each level is compared to the grand mean (physics is in the intercept)contrasts(data$science) <-matrix(c(-1,1,0,0,0,0,-1,0,1,0,0,0,-1,0,0,1,0,0,-1,0,0,0,1,0,-1,0,0,0,0,1), nrow=6)
In the original experiment, participant, science and items are random effects, explanation level and quality are fixed effects, and participant rating is the dependent variable.
The original authors also note that “the bestfitting model included random intercepts by participant and item as well as a random effect of item on the slope for the quality variable”.
This replication project does not have different samples of participants. Therefore, to the best of my understanding, the lmer equation is
Correlation matrix not shown by default, as p = 19 > 12.
Use print(x, correlation=TRUE) or
vcov(x) if you need it
# tidy_lmert <- tidy(lmert)# Filter rows where p.value is less than 0.05# filtered_lmert <- tidy_lmert |> # filter(p.value < 0.05)# Display filtered results using kable# kable(filtered_lmert, digits = 3, caption = "Statistically significant terms in lmer at p<0.05")
As mentioned above, the key statistical test for the original study and the subsequent replications is the main fixed effect of condition on rating of explanation quality within the linear mixed-effects model.
This rescue did not find a main effect of explanation level (t = 1.57, p = 0.12), as the original study did (t = 2.23, p < 0.05). The replication also did not find a main effect of explanation level (p = 0.16).
It is worth noting that the model for the original study was not explicitly specified; however, explanation level was not statistically significant as a main effect even when we use the original data to fit either the rescue or the replication models.
#read original dataoriginal_data <-read.csv("../data/original_hopkins2016_data.csv", header =TRUE)#relable science and explanation level values to their full namesoriginal_data$science <-ifelse(original_data$science=="phys", "physics", original_data$science)original_data$science <-ifelse(original_data$science=="chem", "chemistry", original_data$science)original_data$science <-ifelse(original_data$science=="bio", "biology", original_data$science)original_data$science <-ifelse(original_data$science=="neuro", "neuroscience", original_data$science)original_data$science <-ifelse(original_data$science=="psych", "psychology", original_data$science)original_data$science <-ifelse(original_data$science=="soc", "social", original_data$science)original_data$condition <-ifelse(original_data$condition=="H", "Horizontal", original_data$condition)original_data$condition <-ifelse(original_data$condition=="2", "Reductive", original_data$condition)original_data$science <-factor(original_data$science, levels =c("physics", "chemistry", "biology", "neuroscience", "psychology", "social"),labels =c("physics", "chemistry", "biology", "neuroscience", "psychology", "social"))original_data$quality <-factor(original_data$quality, levels =c("good", "bad"),labels =c("good", "bad"))#group data frame by explanation quality, level, and scientific domainoriginal_grouped <- original_data %>%group_by(quality, condition, science) %>%summarise(avg_rating =mean(rating),sem_rating =sd(rating) /sqrt(n()))
`summarise()` has grouped output by 'quality', 'condition'. You can override
using the `.groups` argument.
Correlation matrix not shown by default, as p = 19 > 12.
Use print(x, correlation=TRUE) or
vcov(x) if you need it
#replication chuey_data <-read.csv("../data/hopkins2016_replication_data.csv", header =TRUE)#delete junk rowschuey_data <- chuey_data[-c(1),]chuey_data <- chuey_data[-c(1),]#add participant numberschuey_data_num <- chuey_data %>%mutate(ID =1:nrow(chuey_data))#deletes demographic data that is unnecessary for analysis and moves attention check 2 to the frontchuey_data_num <- chuey_data_num %>%select(-('Year':'Feedback'))chuey_data_num <- chuey_data_num %>%select(Att_1, everything())#convert to long formatchuey_data_long <- chuey_data_num %>%pivot_longer(cols ='P1.HG_1':'S4.RB_1', names_to ="full_item", values_to ="rating", values_drop_na=TRUE)#specify item attributeschuey_data_long <- chuey_data_long %>%mutate(science =substr(full_item, start=1, stop=1))chuey_data_long <- chuey_data_long %>%mutate(item =substr(full_item, start=1, stop=2))chuey_data_long <- chuey_data_long %>%mutate(quality =substr(full_item, start=5, stop=5))chuey_data_long <- chuey_data_long %>%select(-full_item)chuey_data_long <- chuey_data_long %>%filter(rating!="")#recode item nameschuey_data_long$quality <-ifelse(chuey_data_long$quality=="G", "good", "bad")chuey_data_long$science <-ifelse(chuey_data_long$science=="P", "physics", chuey_data_long$science)chuey_data_long$science <-ifelse(chuey_data_long$science=="C", "chemistry", chuey_data_long$science)chuey_data_long$science <-ifelse(chuey_data_long$science=="B", "biology", chuey_data_long$science)chuey_data_long$science <-ifelse(chuey_data_long$science=="N", "neuroscience", chuey_data_long$science)chuey_data_long$science <-ifelse(chuey_data_long$science=="Y", "psychology", chuey_data_long$science)chuey_data_long$science <-ifelse(chuey_data_long$science=="S", "social", chuey_data_long$science)#convert columns to intended data types for analysischuey_data_long$rating <-as.integer(as.character(chuey_data_long$rating))chuey_data_long$science <-factor(chuey_data_long$science, levels =c("physics", "chemistry", "biology", "neuroscience", "psychology", "social"),labels =c("physics", "chemistry", "biology", "neuroscience", "psychology", "social"))chuey_data_long$quality <-factor(chuey_data_long$quality, levels =c("good", "bad"),labels =c("good", "bad"))chuey_data_long$condition <-factor(chuey_data_long$Condition, levels =c("H", "R"),labels =c("Horizontal", "Reductive"))#exclude participants who failed attention check 1chuey_data_excluded_both <- chuey_data_long %>%filter(Slider1_1==0)#exclude participants who failed attention check 2chuey_data_excluded_both <- chuey_data_excluded_both %>%filter(Att_1==3)#Read in and format datachuey_data <- chuey_data_excluded_both#Reorder levels of the factors to facilitate contrast coding in regressions:chuey_data$science <-relevel(chuey_data$science, ref ="social")chuey_data$science <-relevel(chuey_data$science, ref ="psychology")chuey_data$science <-relevel(chuey_data$science, ref ="neuroscience")chuey_data$science <-relevel(chuey_data$science, ref ="biology")chuey_data$science <-relevel(chuey_data$science, ref ="chemistry")chuey_data$science <-relevel(chuey_data$science, ref ="physics")#Backward difference coding; each level is compared to the mean of the one before it contrasts(chuey_data$quality) <-matrix(c(-1/2,1/2), nrow=2)contrasts(chuey_data$condition) <-matrix(c(-1/2,1/2), nrow=2)#Deviation coding: each level is compared to the grand mean (physics is in the intercept)contrasts(chuey_data$science) <-matrix(c(-1,1,0,0,0,0,-1,0,1,0,0,0,-1,0,0,1,0,0,-1,0,0,0,1,0,-1,0,0,0,0,1), nrow=6)#Mixed effects model aggregated across scienceschuey_explanations <-lmer(rating ~ (quality+condition+science)^2+ quality:condition + (1| ID) + (quality | item), data = chuey_data)tab_model(chuey_explanations)
rating
Predictors
Estimates
CI
p
(Intercept)
1.52
1.33 – 1.71
<0.001
quality1
-0.58
-0.80 – -0.36
<0.001
condition1
-0.21
-0.50 – 0.08
0.161
science1
0.14
-0.18 – 0.47
0.377
science2
0.07
-0.25 – 0.40
0.650
science3
0.30
-0.02 – 0.62
0.069
science4
-0.20
-0.52 – 0.12
0.216
science5
-0.51
-0.83 – -0.19
0.002
quality1 × condition1
-0.13
-0.43 – 0.17
0.402
quality1 × science1
0.36
-0.13 – 0.85
0.151
quality1 × science2
-0.12
-0.61 – 0.37
0.638
quality1 × science3
0.13
-0.36 – 0.62
0.615
quality1 × science4
-0.20
-0.69 – 0.29
0.415
quality1 × science5
-0.28
-0.77 – 0.21
0.269
condition1 × science1
0.32
-0.02 – 0.66
0.062
condition1 × science2
-0.03
-0.37 – 0.31
0.859
condition1 × science3
-0.13
-0.46 – 0.21
0.467
condition1 × science4
-0.26
-0.60 – 0.07
0.126
condition1 × science5
0.00
-0.34 – 0.34
0.998
Random Effects
σ2
1.58
τ00ID
0.36
τ00item
0.09
τ11item.quality1
0.16
ρ01item
0.75
ICC
0.24
N ID
93
N item
24
Observations
1116
Marginal R2 / Conditional R2
0.087 / 0.306
Exploratory analyses
The analyses were repeated using Bayesian modelling, and the results obtained were similar to the lmer. Again, the findings that the fixed effect of explanation quality, interaction effect between explanation level and explanation quality, and explanation level and biology are robust in this sample.
Running /Library/Frameworks/R.framework/Resources/bin/R CMD SHLIB foo.c
using C compiler: ‘Apple clang version 15.0.0 (clang-1500.3.9.4)’
using SDK: ‘MacOSX14.4.sdk’
clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I"/Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library/Rcpp/include/" -I"/Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library/RcppEigen/include/" -I"/Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library/RcppEigen/include/unsupported" -I"/Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library/BH/include" -I"/Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library/StanHeaders/include/src/" -I"/Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library/StanHeaders/include/" -I"/Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library/RcppParallel/include/" -I"/Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library/rstan/include" -DEIGEN_NO_DEBUG -DBOOST_DISABLE_ASSERTS -DBOOST_PENDING_INTEGER_LOG2_HPP -DSTAN_THREADS -DUSE_STANC3 -DSTRICT_R_HEADERS -DBOOST_PHOENIX_NO_VARIADIC_EXPRESSION -D_HAS_AUTO_PTR_ETC=0 -include '/Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library/StanHeaders/include/stan/math/prim/fun/Eigen.hpp' -D_REENTRANT -DRCPP_PARALLEL_USE_TBB=1 -I/opt/R/arm64/include -fPIC -falign-functions=64 -Wall -g -O2 -c foo.c -o foo.o
In file included from <built-in>:1:
In file included from /Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library/StanHeaders/include/stan/math/prim/fun/Eigen.hpp:22:
In file included from /Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library/RcppEigen/include/Eigen/Dense:1:
In file included from /Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library/RcppEigen/include/Eigen/Core:19:
/Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library/RcppEigen/include/Eigen/src/Core/util/Macros.h:679:10: fatal error: 'cmath' file not found
#include <cmath>
^~~~~~~
1 error generated.
make: *** [foo.o] Error 1
# tidy_brm <- tidy(my_bayes_explanations)# Filter rows where p.value is less than 0.05# filtered_brm <- tidy_brm |> # filter(conf.low > 0 | conf.high < 0)# Display filtered results using kable# kable(filtered_brm, digits = 3, caption = "Statistically significant terms in brm at p<0.05")
When exploring other instantiations of the lmer, we also tried to test science as a random effect rather than a fixed effect. Again, there was a significant effect of explanation quality, an interaction effect between explanation quality and explanation level, but no significant effect of explanation level alone. The other interaction effect of condition and biology, and the fixed effect of psychology was also no longer statistically significant.
Linear mixed model fit by REML. t-tests use Satterthwaite's method [
lmerModLmerTest]
Formula: rating ~ condition + quality + condition:quality + (1 | science) +
(1 | ID) + (1 + quality | item)
Data: data
REML criterion at convergence: 7151.7
Scaled residuals:
Min 1Q Median 3Q Max
-3.3466 -0.5352 0.2146 0.6849 2.3913
Random effects:
Groups Name Variance Std.Dev. Corr
ID (Intercept) 0.30628 0.5534
item (Intercept) 0.12731 0.3568
quality1 0.27344 0.5229 -0.66
science (Intercept) 0.03275 0.1810
Residual 2.09888 1.4487
Number of obs: 1932, groups: ID, 161; item, 24; science, 6
Fixed effects:
Estimate Std. Error df t value Pr(>|t|)
(Intercept) 1.2703 0.1173 8.9541 10.827 1.92e-06 ***
condition1 0.1777 0.1131 163.8723 1.571 0.11804
quality1 0.8703 0.1256 21.3709 6.928 6.91e-07 ***
condition1:quality1 -0.4380 0.1373 1632.7210 -3.191 0.00145 **
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Correlation of Fixed Effects:
(Intr) cndtn1 qulty1
condition1 -0.001
quality1 -0.348 0.001
cndtn1:qlt1 0.001 -0.022 -0.001
# tidy_lmert2 <- tidy(lmert2)# Filter rows where p.value is less than 0.05# filtered_lmert2 <- tidy_lmert2 |> # filter(p.value < 0.05)# Display filtered results using kable# kable(filtered_lmert2, digits = 3, caption = "Statistically significant terms in lmer at p<0.05 where science is random effect")
Running /Library/Frameworks/R.framework/Resources/bin/R CMD SHLIB foo.c
using C compiler: ‘Apple clang version 15.0.0 (clang-1500.3.9.4)’
using SDK: ‘MacOSX14.4.sdk’
clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I"/Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library/Rcpp/include/" -I"/Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library/RcppEigen/include/" -I"/Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library/RcppEigen/include/unsupported" -I"/Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library/BH/include" -I"/Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library/StanHeaders/include/src/" -I"/Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library/StanHeaders/include/" -I"/Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library/RcppParallel/include/" -I"/Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library/rstan/include" -DEIGEN_NO_DEBUG -DBOOST_DISABLE_ASSERTS -DBOOST_PENDING_INTEGER_LOG2_HPP -DSTAN_THREADS -DUSE_STANC3 -DSTRICT_R_HEADERS -DBOOST_PHOENIX_NO_VARIADIC_EXPRESSION -D_HAS_AUTO_PTR_ETC=0 -include '/Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library/StanHeaders/include/stan/math/prim/fun/Eigen.hpp' -D_REENTRANT -DRCPP_PARALLEL_USE_TBB=1 -I/opt/R/arm64/include -fPIC -falign-functions=64 -Wall -g -O2 -c foo.c -o foo.o
In file included from <built-in>:1:
In file included from /Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library/StanHeaders/include/stan/math/prim/fun/Eigen.hpp:22:
In file included from /Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library/RcppEigen/include/Eigen/Dense:1:
In file included from /Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library/RcppEigen/include/Eigen/Core:19:
/Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library/RcppEigen/include/Eigen/src/Core/util/Macros.h:679:10: fatal error: 'cmath' file not found
#include <cmath>
^~~~~~~
1 error generated.
make: *** [foo.o] Error 1
Warning: There were 17 divergent transitions after warmup. See
https://mc-stan.org/misc/warnings.html#divergent-transitions-after-warmup
to find out why this is a problem and how to eliminate them.
Warning: Examine the pairs() plot to diagnose sampling problems
summary(brm2)
Warning: There were 17 divergent transitions after warmup. Increasing
adapt_delta above 0.8 may help. See
http://mc-stan.org/misc/warnings.html#divergent-transitions-after-warmup
Family: gaussian
Links: mu = identity; sigma = identity
Formula: rating ~ condition + quality + condition:quality + (1 | science) + (1 | ID) + (1 + quality | item)
Data: data (Number of observations: 1932)
Draws: 4 chains, each with iter = 2000; warmup = 1000; thin = 1;
total post-warmup draws = 4000
Multilevel Hyperparameters:
~ID (Number of levels: 161)
Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
sd(Intercept) 0.56 0.05 0.47 0.66 1.00 1595 2515
~item (Number of levels: 24)
Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS
sd(Intercept) 0.39 0.08 0.26 0.56 1.00 1304
sd(quality1) 0.55 0.12 0.35 0.83 1.00 1658
cor(Intercept,quality1) -0.58 0.21 -0.90 -0.13 1.00 1238
Tail_ESS
sd(Intercept) 2027
sd(quality1) 2670
cor(Intercept,quality1) 1562
~science (Number of levels: 6)
Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
sd(Intercept) 0.24 0.17 0.02 0.67 1.01 726 1470
Regression Coefficients:
Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
Intercept 1.28 0.14 1.01 1.60 1.00 1158 1323
condition1 0.18 0.12 -0.05 0.41 1.00 1972 2072
quality1 0.86 0.14 0.59 1.13 1.00 1771 2135
condition1:quality1 -0.44 0.14 -0.71 -0.18 1.00 4893 2176
Further Distributional Parameters:
Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
sigma 1.45 0.02 1.40 1.50 1.00 4500 2719
Draws were sampled using sampling(NUTS). For each parameter, Bulk_ESS
and Tail_ESS are effective sample size measures, and Rhat is the potential
scale reduction factor on split chains (at convergence, Rhat = 1).
# tidy_brm2 <- tidy(brm2)# Filter rows where p.value is less than 0.05# filtered_brm2 <- tidy_brm2 |> # filter(conf.low > 0 | conf.high < 0)# Display filtered results using kable# kable(filtered_brm2, digits = 3, caption = "Statistically significant terms in brm at p<0.05 where science is random effect")
When the same model with science as a random effect was run with Bayesian modelling, the fixed effects of explanation quality, and interaction between explanation level and quality was statistically significant.
In both Bayesian models, regardless of whether science was modelled as a fixed or random effect, there was a significant random effect of participant, item, and effect of item on the slope for the quality variable.
#plot bar graph faceted by scientific domainggplot(original_grouped, aes(x=quality, y=avg_rating, fill=condition)) +geom_bar(position="dodge", stat="identity") +scale_fill_brewer(palette="Set1") +geom_errorbar(aes(ymin = (avg_rating - sem_rating), ymax = (avg_rating + sem_rating)), position ="dodge") +facet_wrap(~science, nrow=1) + ggthemes::theme_few() +scale_y_continuous(name='explanation rating', limits=c(-3, 3), breaks=seq(-3, 3, 1)) +theme(strip.text =element_text(size =8, margin =margin())) +ggtitle("Results of original Hopkins (2016) study")
This meta analysis will focus on the seductive allure of reductive explanations, in which some studies specifically compare psychology and neuroscience explanations, and others (such as Hopkins et al., 2016) compare reductive explanations across multiple domains.
For a more comprehensive treatment of the existing literature, please refer to Bennett and McLaughlin (2023) here. However, this mini meta analysis includes studies that sampled experts, which was an exclusion criteria in Bennet and McLaughlin (e.g. Study 3 in Weisberg et al. (2008) and Hopkins et al. (2019)), and also includes the replication studies from this class (Chuey (2019) and Chiu (2023)).
In both meta analyses,
We found a mild but highly significant effect, with substantial heterogeneity.
It is worth noting that none of the studies in this mini meta analysis had a significant effect size on their own. In addition, Study 2 in Rhodes et al. (2014) used a 100-point scale for participants to rate the quality of the articles, which inflated the standard deviation.
meta <-read_csv("../data/metaanalysis.csv")
Rows: 12 Columns: 7
── Column specification ────────────────────────────────────────────────────────
Delimiter: ","
chr (3): name, target, DOI
dbl (4): pub_date, n_total, d, var_d
ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.
meta <- meta[order(meta$d), ]metamodel <-rma(yi = d, vi = var_d, ni = n_total, data = meta, method ="REML", knha =TRUE)summary(metamodel)
Random-Effects Model (k = 12; tau^2 estimator: REML)
logLik deviance AIC BIC AICc
-13.5021 27.0042 31.0042 31.8000 32.5042
tau^2 (estimated amount of total heterogeneity): 0 (SE = 0.5135)
tau (square root of estimated tau^2 value): 0
I^2 (total heterogeneity / total variability): 0.00%
H^2 (total variability / sampling variability): 1.00
Test for Heterogeneity:
Q(df = 11) = 1.2731, p-val = 0.9998
Model Results:
estimate se tval df pval ci.lb ci.ub
0.4161 0.1125 3.6991 11 0.0035 0.1685 0.6637 **
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
forest(metamodel,ilab = meta$name,cex =0.7)
funnel(metamodel)
Visual inspection of the funnel plot shows that all of the selected studies fall within expected variability in effect size estimates due to chance alone, and the choice of the studies reported is unlikely to be significantly affected by publication bias or study heterogeneity.
Summary of Replication Attempt
Methodological Similarity
According to the scale in Figure 1 of LeBel et al. (2017), this replication is a Very Close Replication of the original study. The following are my ratings of the various design facets of this replication study:
design_facets <-data.frame(Facets =c('Effect, Hypothesis','IV Construct','DV Construct','IV Operationalization','DV Operationalization','Population','IV Stimuli','DV Stimuli','Procedural Details','Physical Setting','Contextual Variables'),Rating =c('Same', 'Same', 'Same', 'Same', 'Same', 'Same', 'Same', 'Same', 'Different', 'Different', 'Different'))# Display the data frame as a table using kablekable(design_facets, caption ="Replication Taxonomy for various Design Facets", align ="c")
Replication Taxonomy for various Design Facets
Facets
Rating
Effect, Hypothesis
Same
IV Construct
Same
DV Construct
Same
IV Operationalization
Same
DV Operationalization
Same
Population
Same
IV Stimuli
Same
DV Stimuli
Same
Procedural Details
Different
Physical Setting
Different
Contextual Variables
Different
Replication of Results
We did not manage to replicate the fixed effect of explanation level in the key test (t = 1.57, p = 0.12). In other words, we did not observe the seductive allure effect. This is consistent with the replication by Chuey (p = 0.062), and in contrast with the original study, which observed a significant main effect of explanation level (t = 2.23, p < .05).
We replicated secondary findings including the fixed effect of explanation quality (0.25), as well as the interaction effect between quality of explanation and explanation level (0.25).
This has been consistent for all of our confirmatory and exploratory analyses. I would rate the replicability of results as partial replication (0.5).
Commentary
With regards to the methodology, although the 7-point Likert scale seems rudimentary, the same authors continued to use this measure for rating explanation quality, even in subsequent papers (Hopkins, Weisberg, & Taylor, 2019, 2023).
There have also been documented effects about using radio buttons versus a slider scale, as well as presenting the options vertically rather than horizontally, but the exact magnitude of difference would require further examination.
Another modification to increase the odds of observing the reductive allure might be to make explanation level a within-subject variable, as suggested in Bennett and McLaughlin (2023).
With regards to the substantive content of the research, Bennett and McLaughlin provide a succinct summary
However, with the exception of domain-specific expertise, no consistent individual difference has yet been found that predicts a predisposition toward the SANE effect. The types of materials and outcomes in each study are much more relevant to the strength of the effect.
The working hypothesis is that deep domain-specific expertise (see Experiment 3 of Hopkins et al. 2016, and Hopkins et al. 2019) may help to reduce the seductive allure such that jargon is only useful if it is relevant and appropriate to the phenomenon being discussed.
Many studies, including the ones included in this meta analysis have established a robust effect of explanation quality (e.g. Weisberg et al., 2008, Hopkins et al., 2016, 2023), and the interaction between explanation quality and explanation level.
Perhaps the seductive allure of reductive explanations is more precisely defined in its interaction with explanation quality, that participants exposed to this effect are still able to tell good from bad explanations, but bad explanations are possibly used as the heuristic in the absence of explanatory information or lack of comprehension, and perceived as less bad or of a higher quality if they contain reductive language.
Investigator Independence
I have no actual, potential or perceived conflict of interest in relation to this replication study. There was no communication with the authors of the original study.
References
Bennett, E. M., & McLaughlin, P. J. (2023). Neuroscience explanations really do satisfy: A systematic review and meta-analysis of the seductive allure of neuroscience. Public Understanding of Science, 09636625231205005. https://doi.org/10.1177/09636625231205005
Hopkins, E. J., Weisberg, D. S., & Taylor, J. C. V. (2016). The seductive allure is a reductive allure: People prefer scientific explanations that contain logically irrelevant reductive information. Cognition, 155, 67–76. https://doi.org/10.1016/j.cognition.2016.06.011
Hopkins, E. J., Weisberg, D. S., & Taylor, J. C. V. (2019). Does expertise moderate the seductive allure of reductive explanations? Acta Psychologica, 198, 102890. https://doi.org/10.1016/j.actpsy.2019.102890
LeBel, E. P., McCarthy, R. J., Earp, B. D., Elson, M., & Vanpaemel, W. (2018). A Unified Framework to Quantify the Credibility of Scientific Findings. Advances in Methods and Practices in Psychological Science, 1(3), 389–402. https://doi.org/10.1177/2515245918787489
Rhodes, R. E., Rodriguez, F., & Shah, P. (2014). Explaining the alluring influence of neuroscience information on scientific reasoning. Journal of Experimental Psychology: Learning, Memory, and Cognition, 40(5), 1432–1440. https://doi.org/10.1037/a0036844
Weisberg, D. S., Keil, F. C., Goodstein, J., Rawson, E., & Gray, J. R. (2008). The Seductive Allure of Neuroscience Explanations. Journal of Cognitive Neuroscience, 20(3), 470–477. https://doi.org/10.1162/jocn.2008.20040
Weisberg, D. S., Taylor, J. C. V., & Hopkins, E. J. (2015). Deconstructing the seductive allure of neuroscience explanations. Judgment and Decision Making, 10(5), 429–441. https://doi.org/10.1017/S193029750000557X
