Replication of Study ‘Effects of donation collection methods on donation amount: Nudging donation for the cause and overhead’ Suk & Mudita (2022, Psychology & Marketing)
Author
Fan Yang (fay006@ucsd.edu)
Published
December 11, 2024
Introduction
The study “Effects of donation collection methods on donation amount: Nudging donation for the cause and overhead” explores how different donation collection methods can nudge individuals to contribute varying amounts to charitable causes and operational overhead. Using a choice architecture approach, the study examines whether presenting overhead expenses first or last can mitigate “overhead aversion”, a common reluctance among donors to fund operational costs, as opposed to direct causes. The authors test three methods across several studies: an allocation method, where donors allocate a set amount between cause and overhead; a cause-first addition, where donors contribute to the cause first; and an overhead-first addition, where donors specify overhead contributions before deciding on cause donations.
In Study 2 of the original paper, Suk and Mudita found that presenting the overhead request before the cause request (overhead-first condition) led to higher total donations compared to the cause-first condition. The goal of this replication project is to test the reliability of this key finding by replicating the original experiment. In this setup, participants will engage in a charitable donation scenario where they allocate a fixed amount of money between a cause and overhead, depending on whether they encounter the cause-first or overhead-first sequence. A key challenge in this study is ensuring that the presentation order truly influences donation behavior across various settings and donor profiles. If confirmed, these findings could help nonprofits design donation requests that encourage higher overall contributions.
To calculate the required sample size for our replication, we used the effect size from the original study (d = 0.41) in our power analysis. A statistical power of 80% and a significance level of α = 0.05 were set to ensure the ability to detect the original effect size. Using G*Power, we estimated that 188 participants would be required to achieve these parameters. A two-tailed t-test for independent means was chosen, and the input parameters included the means and standard deviations of the two groups from the original study.
Planned Sample
Sample Size: The study aims to recruit approximately 188 participants, with about 94 participants per condition (cause-first vs. overhead-first).
Sampling Frame: Participants will be recruited from Prolific, with eligibility limited to adults aged 18 years and older. There will be no other specific controls for demographics like gender and geographic location.
Termination Rule: Data collection will end once a minimum of 188 participants is reached.
Materials
Donation Input Form: Participants are presented with a donation form (Same with the Appendix C of the original paper) where they could enter amounts to allocate between two categories: (1) “For the cause” (the direct support for the charitable mission) and (2) “For covering charitable organization’s operating expenses” (overhead). The order of these categories will depend on the condition to which participants are assigned (cause-first or overhead-first). The total donation amount is automatically computed based on the amounts inputted for each category, ensuring participants do exceed the allowed limit.
Satisfaction Scale: After completing the donation decisions, participants will rate their satisfaction with their donation using a 7-point Likert scale (1 = not satisfied at all, 7 = very satisfied).
Unrelated Survey: We include an unrelated survey (content not specified in the original study) about daily routine and lifestyle before the actual experiment.
Comprehension checks: Participants are presented with two comprehension check questions after the donation context information. The first asks about the probability of receiving an additional $5 for donation, with options of 50%, 20%, or 10%. The second asks how charities use their donations, with the correct answer being “All of the above” (to support a cause and cover necessary expenses).
Procedure
1. Welcome and Consent: Participants will be asked for their consent to participate in the study.
2. Unrelated Survey: To reduce demand characteristics, we included an unrelated survey on daily routines and lifestyle preferences before the donation task. This survey serves to mask the study’s true focus on donation behavior by making the experiment appear as a general lifestyle study. By including questions about sleep, hobbies, and daily activities, we aim to decrease participants’ awareness of the study’s emphasis on charitable donation choices, thereby minimizing potential biases in their responses.
3. Explanation of Prize: Participants are informed about a 20% chance to receive an additional prize of 5 USD, which they can choose to keep or donate.
4. Information on Charitable Donations: Participants will receive information on how charitable organizations collect and allocate donations, explaining that donations often go toward both (1) supporting the cause and (2) covering operational costs. They will also be informed that “some organizations receive donations separately for cause and overhead,” to clarify the purpose of each type of donation. At last, participants will be presented with general information of a donation campaign.
5. Comprehension Checks: Two comprehension check questions will ensure participants understand the setup, including how charities use donations (supporting both causes and operational expenses) and the nature of their compensation (a guaranteed amount with a 20% chance to earn an additional amount to keep or donate).
6. Experimental Conditions: Two conditions are assigned randomly to the participants.
Condition 1 (Cause-First): Participants will first specify how much they wish to donate to the cause, followed by how much to the overhead.
Condition 2 (Overhead-First): Participants will first specify how much they wish to donate to the overhead, followed by how much to the cause.
7. Donation Allocation and Total Calculation: The donation form will automatically calculate and display the total donation amount based on participants’ allocations to the cause and overhead, ensuring they stay within the allowed limit. Participants will be reminded that they may donate any portion of the allowed amount and will keep whatever they choose not to donate.
8. Donation Satisfaction and Experience: After completing the allocation, participants will rate their satisfaction with their donation decision on a 7-point Likert scale ranging from 1 (not satisfied at all) to 7 (very satisfied). They will also be asked about their previous donation experience.
9. Demographic information: Questions about basic demographic information (age, nationality, gender).
7. Closing and Compensation: At the end, participants will be thanked for their participation. All participants will receive a baseline payment for completing the study. For those selected in the “20% winners” category, additional compensation of 5 USD will be sent. They will be provided with a link to the charity and decide if they will donate or not.
Analysis Plan
1. Donation Amount Analysis: The key test of the study evaluates whether the condition (cause-first vs. overhead-first) has a significant effect on total donations. This analysis directly tests the hypothesis that the order in which donation options are presented influences participants’ overall donation behavior.
To assess the impact of donation collection method (cause-first vs. overhead-first) on donation behavior, we will conduct regression analyses with total donation amount as the dependent variable, and donation method along with demographic variables (e.g., age and gender) as independent variables. Separate regressions for cause and overhead donations will further analyze how the collection method influences each allocation independently. Additionally, an independent samples t-test will compare the donation amounts between the two conditions. Successful replication of our findings will be defined as achieving p < 0.05, confirming statistical significance under conventional criteria.
2. Satisfaction Analysis: To analyze satisfaction as exploratory analyses, we will start with a simple regression using satisfaction as the dependent variable and condition (cause-first vs. overhead-first) as the independent variable. We will then expand to a multiple regression model that includes demographic variables (e.g., age, gender) alongside condition to assess if demographics influence satisfaction. Finally, a comprehensive model will include amounts donated to cause and overhead as additional predictors.
Differences from Original Study
Our sample size is determined based on a power analysis using the effect size from the original study resulting in a target sample size of 188, different from 143 in the original Study 2.
The original study recruited participants exclusively from the UK, while our replication collected data online from participants in the US.
In the original study, the bonus amount was £6. In our replication project,20% of participants will receive 5 USD and can choose to donate.
The unrelated survey used in our replication differs from the original study’s survey, as the specific details of the survey were not provided in the original paper.
Methods Addendum (Post Data Collection)
Actual Sample
We initially recruited 188 participants through Prolific for our study. However, after applying the comprehension checks, only 136 participants qualified for the analysis. This final sample was smaller than the 143 participants in the original study and included 49.3% women, with an average age of 37.6 years. Of the 136 participants, 65 were in the overhead-first group, and 71 were in the cause-first group.
Differences from pre-data collection methods plan
Due to time and budget constraints, we analyzed the 136 valid responses we had instead of continuing to collect data to reach the target sample size of 188 participants.
Design Overview
Factors Manipulated
One factor was manipulated: the order of donation requests (cause-first vs. overhead-first).
Measures Taken and Repeated
Four measures were taken: total donation amount, amount donated to cause, amount donated to overhead, and satisfaction with the donation decision. Each measure was taken once.
Study Design
The study used a between-participants design, with each participant assigned to one condition (cause-first or overhead-first).
Consequences of Design Change
If a within-participants design had been used, where participants experienced both conditions, order effects could have biased the results, as participants’ decisions in the second condition might be influenced by their experience in the first. It could also make the study’s purpose more apparent, increasing demand characteristics.
Steps to Reduce Demand Characteristics
An unrelated survey before the experiment masked the study’s purpose, reducing the chance of participants guessing the research’s intent.
Potential confounds
Prior Donation Experience: Participants with more experience in charitable giving may have stronger preferences regarding donation allocations, which could influence their responses independently of the experimental condition.
Order of Cause and Overhead in Instructions: If one of the conditions (cause or overhead) is introduced first in the instructions, participants might interpret it as more important, leading them to allocate more funds to that category. This could introduce unintended bias based on the presentation order.
Limitations to generalizability
The online sample may not represent the broader population, especially those who donate regularly in real-world contexts. This limits the study’s ability to generalize findings, as real-life donors might have different motivations and behaviors.
In real life, people typically do not receive money or incentives when making donations, as they did in this study. This financial incentive might have influenced participants’ donation behaviors, as they were not actually giving from their own funds.
The study set an upper limit on the donation amount (6 pounds), which is unlike real-life situations where donors can choose any amount they wish to give. Participants might subconsciously use the maximum amount as a reference point, which can shape their perception of what an appropriate donation amount is, potentially influencing their decision-making.
Results
Data Preparation
Data is collected through a custom website, using JsPsych packages integrated into our HTML code to guide participants through the experiment. The collected data is automatically saved in .csv format on our OSF database. The process continue until we have 94 valid participants per condition.
Load Relevant Libraries and Functions
Necessary packages (e.g., tidyverse) will be loaded to handle data import, analysis, and visualization. Key packages include those for reading and analyzing .csv data, conducting regression analyses, and performing t-tests.
library(jsonlite)library(tidyverse)
── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ──
✔ dplyr 1.1.4 ✔ readr 2.1.5
✔ forcats 1.0.0 ✔ stringr 1.5.1
✔ ggplot2 3.5.1 ✔ tibble 3.2.1
✔ lubridate 1.9.3 ✔ tidyr 1.3.1
✔ purrr 1.0.2
── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
✖ dplyr::filter() masks stats::filter()
✖ purrr::flatten() masks jsonlite::flatten()
✖ dplyr::lag() masks stats::lag()
ℹ Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errors
library(effsize)
The functions used for data preprocessing:
# Larger function to concatenate individual processed filesprocess_csv_files <-function(file_paths, output_file) {# Nested function to process a single file process_file <-function(file_path) {# Reads file testdata <-read_csv(file_path, show_col_types =FALSE)# Shortens file to specified columns tdcols <-subset(testdata, select =-c(rt, stimulus, trial_type, plugin_version, question_order))# Shortens file to specified rows tdrows <- tdcols[-c(1:13, 16:17), , drop =FALSE]# Rearranges elements of cleaned data into one cohesive row tdarrange <-data.frame(check1 = tdrows$response[1],check2 = tdrows$response[2],cause = tdrows$causeDonation[3],overhead = tdrows$expenseDonation[3],total = tdrows$totalDonation[3],condition = tdrows$questionOrder[3],satis_usual = tdrows$response[4],age = tdrows$age[5],nationality = tdrows$nationality[5],gender = tdrows$response[6],comment = tdrows$response[7] )# Parses the satisfaction responses, unnests JSON formatting tdparsed <- tdarrange %>%mutate(satis_usual =lapply(satis_usual, function(x) {tryCatch(fromJSON(x), error =function(e) NA) }))# Unnests the satisfaction/frequency column into two separate columns tdsep <- tdparsed %>%unnest_wider(col = satis_usual, names_sep ="_")# Renames columns tdclean <- tdsep %>%rename(satisfaction = satis_usual_Q0,frequency = satis_usual_Q1)# Returns cleaned datareturn(tdclean) }# Apply nested function to each file processed_files <-lapply(file_paths, process_file)# Combines files, adds identifying number all_data <-bind_rows(processed_files, .id ="file_id")# Saves the combined datawrite_csv(all_data, output_file)# Returns combined datareturn(all_data)}## Cleans JSON formatting# Function removes all of the special charcters from the dataremove_special_characters <-function(entry) {gsub("[^a-zA-Z0-9\\s]", "", entry)}
Import data
We have established a pipeline from the website to an OSF database using DataPipe, which automatically saves each participant’s responses in .csv format as results come in. We will download the individual .csv files, clean the data, and combine them into a single .csv file.
Codes for importing data:
# Vector of filepaths for data pipe outputs to be strung togetherfile_paths <-list.files(path ="./Sample", full.names =TRUE)# Defines filepath for .csv fileoutput_file <-"./sampledata.csv"
Prepare data for analysis - create columns etc.
We created specific columns for the following variables: - Participant ID; - Condition (cause-first or overhead-first); - Demographic Variables: age, gender, nationality; - Satisfaction: recorded on a Likert scale; - Frequency: previous donation experience; - Amount Donated: Separate columns for donations to overhead and cause; - Total Amount Donated: sum of donations to cause and overhead. - comprehension check answers: for filtering data
# Runs function and produces .csv fileresult <-process_csv_files(file_paths, output_file)head(result, 2)
# A tibble: 2 × 13
file_id check1 check2 cause overhead total condition satisfaction frequency
<chr> <chr> <chr> <dbl> <dbl> <dbl> <chr> <int> <int>
1 1 "{\"chan… "{\"d… 0 0 0 "[\"For … 6 1
2 2 "{\"chan… "{\"d… 2 2 4 "[\"For … 3 3
# ℹ 4 more variables: age <dbl>, nationality <chr>, gender <chr>, comment <chr>
codes for cleaned data and columns:
# Applies function to all relevant columnscleanresult <- result %>%mutate(check1 =sapply(check1, remove_special_characters)) %>%mutate(check2 =sapply(check2, remove_special_characters)) %>%mutate(condition =sapply(condition, remove_special_characters)) %>%mutate(gender =sapply(gender, remove_special_characters)) %>%mutate(comment =sapply(comment, remove_special_characters))# Edits data for legibilitycleanresult <- cleanresult %>%mutate(condition =recode( condition,ForthecauseForcoveringcharitableorganizationsoperatingexpense ='causefirst',ForcoveringcharitableorganizationsoperatingexpenseForthecause ='overfirst' ),check1 =gsub("chanceresponse", "", check1),check2 =gsub("donationuseresponse", "", check2),gender =gsub("gender", "", gender),comment =gsub("Q0", "", comment) )head(cleanresult, 2)
# A tibble: 2 × 13
file_id check1 check2 cause overhead total condition satisfaction frequency
<chr> <chr> <chr> <dbl> <dbl> <dbl> <chr> <int> <int>
1 1 20 Allofthe… 0 0 0 causefir… 6 1
2 2 50 Allofthe… 2 2 4 overfirst 3 3
# ℹ 4 more variables: age <dbl>, nationality <chr>, gender <chr>, comment <chr>
Data exclusion / filtering
Participants who fail both comprehension checks will be excluded from the analysis to ensure data quality and participant understanding.
The confirmatory analysis focuses on testing whether the order of donation requests significantly affect the donation amounts. To analyze the data, we performed several regression analyses. A linear regression model was fit to predict total donations using condition, age, and gender as predictors, followed by separate models for overhead and cause donations. Results will be deemed successful if the p-value is < 0.05, indicating a statistically significant difference between the conditions in their effect on total donations, overhead donations, or cause donations.
The condition (cause-first vs. overhead-first) did not have a statistically significant effect on total donations in this replication, in contrast to the original study, which found a significant effect favoring the overhead-first condition.
Age and gender showed statistically significant effects on donation behavior, with older participants and male participants donating less.
The overall model fit is poor, indicating that the predictors included in the model explain only 6.8% of the variance in total of the variability in total donation amounts.
# Regression analysis on total donation amountstotalreg <-lm('total ~ condition + age + gender', data = filterresult)summary(totalreg)
Call:
lm(formula = "total ~ condition + age + gender", data = filterresult)
Residuals:
Min 1Q Median 3Q Max
-2.4897 -1.3151 -0.5092 0.8497 3.8677
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 2.80183 0.56851 4.928 2.48e-06 ***
conditionoverfirst 0.34397 0.30452 1.130 0.2608
age -0.02734 0.01284 -2.129 0.0351 *
genderMale -0.68530 0.30992 -2.211 0.0288 *
genderNonbinary -0.11264 1.26692 -0.089 0.9293
genderPrefernottosay 0.38164 1.78738 0.214 0.8313
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Residual standard error: 1.759 on 130 degrees of freedom
Multiple R-squared: 0.06833, Adjusted R-squared: 0.03249
F-statistic: 1.907 on 5 and 130 DF, p-value: 0.09747
The mean total donation in the overhead-first condition (M = 1.75, SD = 1.93) is slightly higher than in the cause-first condition (M = 1.47, SD = 1.65), reflecting a similar trend observed in the original study. However, the effect size (d = 0.157) is negligible and falls within a 95% confidence interval of (-0.183) to (0.497), indicating that the difference between conditions is not statistically or meaningful.
# cohen's d for total donation amountsprint(data.frame(Statistic =c("Mean", "Standard Deviation"),Overhead =c(round(mean(splitdata$overfirst$total), 2),round(sd(splitdata$overfirst$total), 2)),Cause =c(round(mean(splitdata$causefirst$total), 2),round(sd(splitdata$causefirst$total), 2)) ))
Statistic Overhead Cause
1 Mean 1.75 1.47
2 Standard Deviation 1.93 1.65
Cohen's d
d estimate: 0.1571732 (negligible)
95 percent confidence interval:
lower upper
-0.1828745 0.4972209
Analyses on Overhead Donations
The replication results for overhead donations did not replicate the significant effect of condition observed in the original study (( = 0.135, p = 0.332 )). However, age (( = -0.015, p = 0.012 )) and gender (male, ( = -0.406, p = 0.005 )) were significant predictors, with older participants and males donating less to overhead. The model’s explanatory power was relatively low. These results suggest that while condition did not significantly influence overhead donations, demographic factors such as age and gender played a more substantial role in the replication.
# Regression on Overhead Donationsoverheadreg <-lm('overhead ~ condition + age + gender', data = filterresult)summary(overheadreg)
Call:
lm(formula = "overhead ~ condition + age + gender", data = filterresult)
Residuals:
Min 1Q Median 3Q Max
-1.0663 -0.5410 -0.2396 0.3893 4.0831
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 1.290311 0.258375 4.994 1.87e-06 ***
conditionoverfirst 0.134722 0.138400 0.973 0.33215
age -0.014946 0.005836 -2.561 0.01157 *
genderMale -0.405731 0.140854 -2.881 0.00465 **
genderNonbinary -0.386863 0.575788 -0.672 0.50285
genderPrefernottosay 0.083347 0.812332 0.103 0.91844
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Residual standard error: 0.7996 on 130 degrees of freedom
Multiple R-squared: 0.09625, Adjusted R-squared: 0.06149
F-statistic: 2.769 on 5 and 130 DF, p-value: 0.02067
The mean overhead donation in the overhead-first condition (M = 0.64, SD = 0.93) was slightly higher than in the cause-first condition (M = 0.54, SD = 0.71). However, the effect size (d = 0.119) was negligible, with a 95% confidence interval ranging from (-0.221) to (0.459), indicating that the difference is not statistically significant.
# cohen's d for overhead donationsprint(data.frame(Statistic =c("Mean", "Standard Deviation"),Overhead =c(round(mean(splitdata$overfirst$overhead), 2),round(sd(splitdata$overfirst$overhead), 2)),Cause =c(round(mean(splitdata$causefirst$overhead), 2),round(sd(splitdata$causefirst$overhead), 2)) ))
Statistic Overhead Cause
1 Mean 0.64 0.54
2 Standard Deviation 0.93 0.71
Cohen's d
d estimate: 0.1192818 (negligible)
95 percent confidence interval:
lower upper
-0.2205443 0.4591080
Analyses on Cause Donations
The effect of condition on cause donations was also not significant (( = 0.209, p = 0.311 )). None of the other predictors, including age and gender, showed significant effects on cause donations. The model’s explanatory power was very low (( R^2 = 0.035 )), indicating limited ability to explain variations in cause donations. The replication’s low model fit suggests additional factors influencing cause donations were not captured.
# Regression on Cause Donationscausereg <-lm('cause ~ condition + age + gender', data = filterresult)summary(causereg)
Call:
lm(formula = "cause ~ condition + age + gender", data = filterresult)
Residuals:
Min 1Q Median 3Q Max
-1.4233 -0.8757 -0.2950 0.4362 3.9802
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 1.511520 0.384364 3.933 0.000136 ***
conditionoverfirst 0.209244 0.205886 1.016 0.311369
age -0.012392 0.008681 -1.427 0.155843
genderMale -0.279574 0.209536 -1.334 0.184455
genderNonbinary 0.274219 0.856553 0.320 0.749374
genderPrefernottosay 0.298290 1.208439 0.247 0.805422
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Residual standard error: 1.19 on 130 degrees of freedom
Multiple R-squared: 0.0349, Adjusted R-squared: -0.00222
F-statistic: 0.9402 on 5 and 130 DF, p-value: 0.4574
The mean cause donation in the overhead-first condition (M = 1.11, SD = 1.33) was slightly higher than in the cause-first condition (M = 0.93, SD = 1.04). However, the effect size (d = 0.154) was negligible, with a 95% confidence interval of (-0.186) to (0.493), suggesting that the difference is neither statistically significant.
# cohen's d for cause donationsprint(data.frame(Statistic =c("Mean", "Standard Deviation"),Overhead =c(round(mean(splitdata$overfirst$cause), 2),round(sd(splitdata$overfirst$cause), 2)),Cause =c(round(mean(splitdata$causefirst$cause), 2),round(sd(splitdata$causefirst$cause), 2)) ))
Statistic Overhead Cause
1 Mean 1.11 0.93
2 Standard Deviation 1.33 1.04
Cohen's d
d estimate: 0.1536157 (negligible)
95 percent confidence interval:
lower upper
-0.1864086 0.4936400
The confirmatory analysis found no significant effect of condition on total, cause, or overhead donations. Demographic factors, such as age and gender, were more predictive of donation behavior than conditions.
Side-by-side graph for donation amounts
# Calculates means and standard errors for donation amounts among both conditionssummary_data <- filterresult %>%group_by(condition) %>%summarise(total_mean =mean(total),total_se =sd(total) /sqrt(n()),cause_mean =mean(cause),cause_se =sd(cause) /sqrt(n()),overhead_mean =mean(overhead),overhead_se =sd(overhead) /sqrt(n()) ) %>%# Shapes data for ease of use in ggplotpivot_longer(cols =-condition,names_to =c("category", ".value"),names_sep ="_" )summary_data$category <-factor(summary_data$category, levels =c("total", "cause", "overhead"))# Creates base for plotggplot(summary_data, aes(x = category, y = mean, fill = condition)) +geom_bar(stat ="identity", position =position_dodge(width =0.4), width =0.4, color ="gray20") +# Adds error barsgeom_errorbar(aes(ymin = mean - se, ymax = mean + se),position =position_dodge(width =0.4), width =0.2, color ="gray20") +# Specifies aesthetics for color and labelscale_fill_manual(values =c("causefirst"="white", "overfirst"="gray"),labels =c("Cause First", "Overhead First")) +labs(title ="Donation amounts by donation collection method",x ="Donation collection method",y ="Amount ($)",fill ="Condition" ) +# Specifies axesscale_y_continuous(expand =c(0, 0),limits =c(0, 2.25),breaks =seq(0, 2.25, .5) ) +# Specifies styletheme_minimal() +theme(legend.position.inside =c(0.9, 0.9), panel.background =element_rect(fill ="white", color ="gray20"), plot.background =element_rect(fill ="white", color =NA) )
Figure from the original study
Exploratory analyses
Satisfaction analyses
We conducted three regression models to explore predictors of satisfaction: (1) a simple model using condition as the predictor, (2) a model using condition, age and gender as covariates, and (3) a full model including condition, age, gender, and the amounts donated to the cause and overhead. These analyses were conducted to examine any potential relationships between satisfaction, donation behaviors, and demographic variables.
# Exploratory analyses based on predicting donor satisfactionsatreg1 <-lm('satisfaction ~ condition', data = filterresult)summary(satreg1)
Call:
lm(formula = "satisfaction ~ condition", data = filterresult)
Residuals:
Min 1Q Median 3Q Max
-4.6197 -1.5231 0.4286 1.3803 1.4769
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 4.61972 0.16561 27.895 <2e-16 ***
conditionoverfirst -0.09664 0.23956 -0.403 0.687
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Residual standard error: 1.395 on 134 degrees of freedom
Multiple R-squared: 0.001213, Adjusted R-squared: -0.006241
F-statistic: 0.1627 on 1 and 134 DF, p-value: 0.6873
satreg2 <-lm('satisfaction ~ condition + age + gender', data = filterresult)summary(satreg2)
Call:
lm(formula = "satisfaction ~ condition + age + gender", data = filterresult)
Residuals:
Min 1Q Median 3Q Max
-4.6192 -1.0820 0.2557 1.0503 1.8313
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 3.975362 0.437604 9.084 1.48e-15 ***
conditionoverfirst -0.192983 0.234404 -0.823 0.4118
age 0.009905 0.009884 1.002 0.3182
genderMale 0.679652 0.238561 2.849 0.0051 **
genderNonbinary 0.309134 0.975199 0.317 0.7518
genderPrefernottosay -2.222978 1.375827 -1.616 0.1086
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Residual standard error: 1.354 on 130 degrees of freedom
Multiple R-squared: 0.08735, Adjusted R-squared: 0.05225
F-statistic: 2.489 on 5 and 130 DF, p-value: 0.03454
satreg3 <-lm('satisfaction ~ condition + age + gender + cause + overhead', data = filterresult)summary(satreg3)
Call:
lm(formula = "satisfaction ~ condition + age + gender + cause + overhead",
data = filterresult)
Residuals:
Min 1Q Median 3Q Max
-4.4931 -1.0278 0.1928 1.1649 2.0190
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 3.77690 0.47713 7.916 1.02e-12 ***
conditionoverfirst -0.22562 0.23371 -0.965 0.33619
age 0.01102 0.01005 1.096 0.27502
genderMale 0.69673 0.24420 2.853 0.00505 **
genderNonbinary 0.20016 0.97175 0.206 0.83713
genderPrefernottosay -2.28227 1.36538 -1.672 0.09706 .
cause 0.23161 0.11838 1.957 0.05258 .
overhead -0.11751 0.17610 -0.667 0.50579
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Residual standard error: 1.344 on 128 degrees of freedom
Multiple R-squared: 0.1154, Adjusted R-squared: 0.06704
F-statistic: 2.386 on 7 and 128 DF, p-value: 0.02512
The exploratory analyses revealed that the condition (cause-first vs. overhead-first) did not significantly predict satisfaction in any of the three regression models, indicating that the order of donation prompts was not a meaningful driver of satisfaction. Across all models, the explained variance was low, with ( R^2 ) values ranging from 0.001 to 0.115, suggesting that other unmeasured factors may better explain variations in satisfaction. Gender emerged as a consistent predictor, with male participants reporting significantly higher satisfaction levels compared to females, while other gender categories showed less consistent trends. The inclusion of age as a predictor did not significantly contribute to the models.
When donation amounts were included in the analysis, cause donations showed a marginally significant positive relationship with satisfaction, suggesting that participants may feel more satisfied when contributing to a cause. However, overhead donations did not show a significant effect on satisfaction. Overall, the findings indicate that demographic factors, particularly gender, and to some extent cause donations, were more influential in shaping satisfaction than the experimental condition or overhead donations. These results emphasize the importance of accounting for individual differences and donation behaviors when analyzing satisfaction outcomes.
Satisfaction vs Donation Amount by Condition
# satisfaction plotggplot(filterresult, aes(x = total, y = satisfaction, color = condition)) +geom_point(alpha =0.5) +geom_smooth(method ="lm", se =TRUE) +labs(title ="Satisfaction vs Donation Amount by Condition",x ="Total Donation", y ="Satisfaction") +theme_minimal()
`geom_smooth()` using formula = 'y ~ x'
The plot shows a weak positive relationship between total donations and satisfaction, with slightly higher donations linked to marginally greater satisfaction. Confidence intervals for the two conditions overlap, indicating no significant difference in the satisfaction-donation relationship between the cause-first and overhead-first conditions. Satisfaction levels remain relatively high and consistent across donation amounts, suggesting other factors likely drive satisfaction.
Distribution of Total donation amounts
ggplot(filterresult, aes(x = total, color = condition, fill = condition)) +geom_density(alpha =0.4) +labs(title ="Distribution of Total Donations by Condition",x ="Total Donation Amount",y ="Density",fill ="Condition",color ="Condition") +theme_minimal()
The density plot shows a similar distribution of total donation amounts for both conditions, with the highest density at lower donation values (close to 0). While the overhead-first condition appears to have a slightly higher density for larger donations, the overall overlap suggests minimal differences between the two conditions.
Plot for mean donation amounts and distribution
ggplot(filterresult, aes(x = condition, y = total, color = condition)) +stat_summary(fun.data ="mean_cl_boot", geom ="pointrange") +geom_jitter(width =0.2, alpha =0.4) +labs(title ="Mean Donation by Condition",x ="Condition", y ="Total Donation Amount") +theme_minimal()
The plot shows the distribution and mean total donation amounts by condition (cause-first vs. overhead-first). While the mean donation amount appears higher in the overhead-first condition, there is substantial overlap in the distributions between the two conditions. This suggests minimal variation in total donations attributable to the condition, supporting the conclusion that the condition did not significantly influence donation behavior. The wide spread and overlapping points further indicate individual differences driving donation patterns rather than the experimental manipulation.
Discussion
Summary of Replication Attempt
Our replication did not replicate the significant findings from the original study. Specifically, the condition (cause-first vs. overhead-first) did not significantly affect total donations, cause donations or overhead donations in our confirmatory analyses. As a result, the replication did not support the original study’s conclusion that the order of donation prompts influences donation behavior. This outcome suggests potential contextual or methodological factors that may have moderated the original results.
Commentary
Insights from Follow-Up Exploratory Analysis
The exploratory analyses highlighted demographic factors, such as gender and age, as more relevant predictors of satisfaction and donation behaviors than the condition itself. Another observation was the high proportion of participants in our replication who chose not to donate, potentially reflecting differing motivations or perceptions of the task compared to the original study’s participants. Additionally, cause donations showed a marginal relationship with satisfaction, while overhead donations did not.
Assessment of the Meaning of the Replication
The differences between our study and the original likely moderated the results. First, our sample was collected from the US, while the original study recruited UK participants. Cultural and socioeconomic differences may have influenced participants’ attitudes toward donations. Second, the bonus prize offered in our replication ($5) was smaller than the original study’s £6 (~$7.60), potentially altering participants’ perceptions of the task. Both factors plausibly contributed to the observed discrepancies.
Objections or Challenges Raised
One challenge in our study was the reduced sample size following comprehension check exclusions, leaving us with 136 participants compared to the original 143. This smaller sample size may have limited the power to detect subtle effects. Additionally, the use of Prolific as a recruitment platform likely influenced participants’ focus on monetary incentives, which may have affected their willingness to donate. These factors complicate direct comparisons between the two studies and highlight areas for improvement in future replication efforts.
Statement of Contributions
Emma Gu: Conceptualization, Methodology, Software, Writing - original draft, and Writing - review & editing.
Fan Yang: Methodology, Software, Writing - original draft, and Writing - review & editing.
Nina Rice: Formal analysis, Software, Validation, Visualization, Writing - original draft, and Writing - review & editing.
William S. de Melo: Data curation, Formal analysis, Investigation, Software, Validation, Writing - original draft, and Writing - review & editing.
Conclusion
While the replication did not fully replicate the original study’s significant findings, it showed similar trends, with slightly higher donations in the overhead-first condition. Future efforts should increase the sample size to account for exclusions, match monetary incentives to the original study, and consider alternatives to crowdworking platforms to reduce financial biases. These adjustments could help clarify whether the original effects are robust across contexts and improve the reliability of replication attempts.
