Replication of Study ‘Effects of donation collection methods on donation amount: Nudging donation for the cause and overhead’ Suk & Mudita (2022, Psychology & Marketing)
Author
Fan Yang (fay006@ucsd.edu)
Published
November 22, 2024
Introduction
The study “Effects of donation collection methods on donation amount: Nudging donation for the cause and overhead” explores how different donation collection methods can nudge individuals to contribute varying amounts to charitable causes and operational overhead. Using a choice architecture approach, the study examines whether presenting overhead expenses first or last can mitigate “overhead aversion”, a common reluctance among donors to fund operational costs, as opposed to direct causes. The authors test three methods across several studies: an allocation method, where donors allocate a set amount between cause and overhead; a cause-first addition, where donors contribute to the cause first; and an overhead-first addition, where donors specify overhead contributions before deciding on cause donations.
In Study 2 of the original paper, Suk and Mudita found that presenting the overhead request before the cause request (overhead-first condition) led to higher total donations compared to the cause-first condition. The goal of this replication project is to test the reliability of this key finding by replicating the original experiment. In this setup, participants will engage in a charitable donation scenario where they allocate a fixed amount of money between a cause and overhead, depending on whether they encounter the cause-first or overhead-first sequence. A key challenge in this study is ensuring that the presentation order truly influences donation behavior across various settings and donor profiles. If confirmed, these findings could help nonprofits design donation requests that encourage higher overall contributions.
In our replication of Study 2, the sample size was initially determined using the effect sizes observed in Study 1 (d = 0.41). To achieve a statistical power of 80%, we estimated that a sample size of 188 participants would be necessary. This calculation was conducted using G*Power, and it ensures that our replication study will have adequate power to detect a similar effect, assuming the effect size is consistent with that observed in the original study.
Planned Sample
Sample Size: The study aims to recruit approximately 188 participants, with about 94 participants per condition (cause-first vs. overhead-first).
Sampling Frame: Participants will be recruited from Prolific, with eligibility limited to adults aged 18 years and older. There will be no other specific controls for demographics like gender and geographic location.
Termination Rule: Data collection will end once a minimum of 188 participants is reached.
Materials
Donation Input Form: Participants will be presented with a donation form (Same with the Appendix C of the original paper) where they could enter amounts to allocate between two categories: (1) “For the cause” (the direct support for the charitable mission) and (2) “For covering charitable organization’s operating expenses” (overhead). The order of these categories will depend on the condition to which participants are assigned (cause-first or overhead-first). The total donation amount was automatically computed based on the amounts inputted for each category, ensuring participants did not exceed the allowed limit.
Satisfaction Scale: After completing the donation decisions, participants will rate their satisfaction with their donation using a 7-point Likert scale (1 = not satisfied at all, 7 = very satisfied).
Unrelated Survey: We will include an unrelated survey (content not specified in the original study) about daily routine and lifestyle at before the actual experiment to reduce demand characteristics.
Procedure
1. Welcome and Consent: Participants will be asked for their consent to participate in the study.
2. Unrelated Survey: To reduce demand characteristics, we included an unrelated survey on daily routines and lifestyle preferences before the donation task. This survey serves to mask the study’s true focus on donation behavior by making the experiment appear as a general lifestyle study. By including questions about sleep, hobbies, and daily activities, we aim to decrease participants’ awareness of the study’s emphasis on charitable donation choices, thereby minimizing potential biases in their responses.
3. Explanation of Prize: Participants are informed about a 20% chance to receive an additional prize of 5 USD, which they can choose to keep or donate.
4. Information on Charitable Donations: Participants will receive information on how charitable organizations collect and allocate donations, explaining that donations often go toward both (1) supporting the cause and (2) covering operational costs. They will also be informed that “some organizations receive donations separately for cause and overhead,” to clarify the purpose of each type of donation. At last, participants will be presented with general information of a donation campaign.
5. Comprehension Checks: Two comprehension check questions will ensure participants understand the setup, including how charities use donations (supporting both causes and operational expenses) and the nature of their compensation (a guaranteed amount with a 20% chance to earn an additional amount to keep or donate).
6.Experimental Conditions: Two conditions are assigned randomly to the participants.
Condition 1 (Cause-First): Participants will first specify how much they wish to donate to the cause, followed by how much to the overhead.
Condition 2 (Overhead-First): Participants will first specify how much they wish to donate to the overhead, followed by how much to the cause.
7.Donation Allocation and Total Calculation: The donation form will automatically calculate and display the total donation amount based on participants’ allocations to the cause and overhead, ensuring they stay within the allowed limit. Participants will be reminded that they may donate any portion of the allowed amount and will keep whatever they choose not to donate.
8. Donation Satisfaction and Experience: After completing the allocation, participants will rate their satisfaction with their donation decision on a 7-point Likert scale ranging from 1 (not satisfied at all) to 7 (very satisfied). They will also be asked about their previous donation experience.
9. Demographic information: Questions about basic demographic information (age, nationality, gender).
7. Closing and Compensation: At the end, participants will be thanked for their participation. All participants will receive a baseline payment for completing the study. For those selected in the “20% winners” category, additional compensation of 5 USD will be sent. They will be provided with a link to the charity and decide if they will donate or not.
Analysis Plan
1. Donation Amount Analysis: To assess the impact of donation collection method (cause-first vs. overhead-first) on donation behavior, we will conduct regression analyses with total donation amount as the dependent variable, and donation method along with demographic variables (e.g., age and gender) as independent variables. Separate regressions for cause and overhead donations will further analyze how the collection method influences each allocation independently. Additionally, an independent samples t-test will compare the donation amounts between the two conditions. Successful replication of our findings will be defined as achieving p < 0.05, confirming statistical significance under conventional criteria.
2. Satisfaction Analysis: To analyze satisfaction, we will start with a simple regression using satisfaction as the dependent variable and condition (cause-first vs. overhead-first) as the independent variable. We will then expand to a multiple regression model that includes demographic variables (e.g., age, gender) alongside condition to assess if demographics influence satisfaction. Finally, a comprehensive model will include amounts donated to cause and overhead as additional predictors.
Differences from Original Study
Our sample size is determined based on a power analysis using the effect size from Study 1, resulting in a target sample size of 188, different from 143 in the original Study 2.
The original study recruited participants exclusively from the UK, while our study collects data online without geographic restrictions.
In the original study, the bonus amount was £6. In our replication project,20% of participants will receive 5 USD and can choose to donate.
Methods Addendum (Post Data Collection)
You can comment this section out prior to final report with data collection.
Actual Sample
Sample size, demographics, data exclusions based on rules spelled out in analysis plan
Differences from pre-data collection methods plan
Any differences from what was described as the original plan, or “none”.
Design Overview
How many factors were manipulated?
One factor was manipulated: the order of donation requests (cause-first vs. overhead-first).
How many measures were taken?
Four measures were taken: total donation amount, amount donated to cause, amount donated to overhead, and satisfaction with the donation decision.
Did it use a within-participants or between-participants design?
The study used a between-participants design, with each participant assigned to one condition (cause-first or overhead-first).
Were measures repeated?
No, measures were not repeated. Each participant made a single donation decision and rated their satisfaction once.
What would have been the consequence of changing one of these decisions? (e.g. moving from within to between participants). Could it have been either way?
If a within-participants design had been used, where participants experienced both conditions, order effects could have biased the results, as participants’ decisions in the second condition might be influenced by their experience in the first. It could also make the study’s purpose more apparent, increasing demand characteristics. Therefore, a between-participants design is more suitable for this experiment.
Were steps taken to reduce demand characteristics?
Yes, an important step taken to reduce demand characteristics was the unrelated survey prior to the actual donation experiment in the original study. This initial survey served to mask the study’s true intent, reducing the chances that participants would guess that the research was examining the effect of donation order on donation amounts.
Can you think of any potential confounds to the study?
Prior Donation Experience: Participants with more experience in charitable giving may have stronger preferences regarding donation allocations, which could influence their responses independently of the experimental condition.
Order of Cause and Overhead in Instructions: If one of the conditions (cause or overhead) is introduced first in the instructions, participants might interpret it as more important, leading them to allocate more funds to that category. This could introduce unintended bias based on the presentation order.
Do you see any limitations to generalizability?
The online sample may not represent the broader population, especially those who donate regularly in real-world contexts. This limits the study’s ability to generalize findings, as real-life donors might have different motivations and behaviors.
In real life, people typically do not receive money or incentives when making donations, as they did in this study. This financial incentive might have influenced participants’ donation behaviors, as they were not actually giving from their own funds.
The study set an upper limit on the donation amount (6 pounds), which is unlike real-life situations where donors can choose any amount they wish to give. Participants might subconsciously use the maximum amount as a reference point, which can shape their perception of what an appropriate donation amount is, potentially influencing their decision-making.
Results
Data Preparation
Data will be collected through a custom website, using JsPsych packages integrated into our HTML code to guide participants through the experiment. The collected data is automatically saved in .csv format on our OSF database. The process will continue until we have 94 valid participants per condition.
Load Relevant Libraries and Functions
Necessary packages (e.g., tidyverse) will be loaded to handle data import, analysis, and visualization. Key packages include those for reading and analyzing .csv data, conducting regression analyses, and performing t-tests.
library(jsonlite)library(tidyverse)
── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ──
✔ dplyr 1.1.4 ✔ readr 2.1.5
✔ forcats 1.0.0 ✔ stringr 1.5.1
✔ ggplot2 3.5.1 ✔ tibble 3.2.1
✔ lubridate 1.9.3 ✔ tidyr 1.3.1
✔ purrr 1.0.2
── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
✖ dplyr::filter() masks stats::filter()
✖ purrr::flatten() masks jsonlite::flatten()
✖ dplyr::lag() masks stats::lag()
ℹ Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errors
The functions used for data preprocessing:
# Larger function to concatenate individual processed filesprocess_csv_files <-function(file_paths, output_file) {# Nested function to process a single file process_file <-function(file_path) {# Reads file testdata <-read_csv(file_path, show_col_types =FALSE)# Shortens file to specified columns tdcols <-subset(testdata, select =-c(rt, stimulus, trial_type, plugin_version, question_order))# Shortens file to specified rows tdrows <- tdcols[-c(1:13, 16:17), , drop =FALSE]# Rearranges elements of cleaned data into one cohesive row tdarrange <-data.frame(check1 = tdrows$response[1],check2 = tdrows$response[2],cause = tdrows$causeDonation[3],overhead = tdrows$expenseDonation[3],total = tdrows$totalDonation[3],condition = tdrows$questionOrder[3],satis_usual = tdrows$response[4],age = tdrows$age[5],nationality = tdrows$nationality[5],gender = tdrows$response[6],comment = tdrows$response[7] )# Parses the satisfaction responses, unnests JSON formatting tdparsed <- tdarrange %>%mutate(satis_usual =lapply(satis_usual, function(x) {tryCatch(fromJSON(x), error =function(e) NA) }))# Unnests the satisfaction/frequency column into two separate columns tdsep <- tdparsed %>%unnest_wider(col = satis_usual, names_sep ="_")# Renames columns tdclean <- tdsep %>%rename(satisfaction = satis_usual_Q0,frequency = satis_usual_Q1)# Returns cleaned datareturn(tdclean) }# Apply nested function to each file processed_files <-lapply(file_paths, process_file)# Combines files, adds identifying number all_data <-bind_rows(processed_files, .id ="file_id")# Saves the combined datawrite_csv(all_data, output_file)# Returns combined datareturn(all_data)}
Import data
We have established a pipeline from the website to an OSF database using DataPipe, which automatically saves each participant’s responses in .csv format as results come in. We will download the individual .csv files, clean the data, and combine them into a single .csv file.
Codes for importing data:
# Vector of filepaths for data pipe outputs to be strung togetherfile_paths <-c("pilotB/wke8v3rwk2.csv","pilotB/vk7098a6qc.csv","pilotB/c5gpk5juov.csv","pilotB/6hf9baq07x.csv","pilotB/33lelrenhh.csv" )# Defines filepath for .csv fileoutput_file <-"testpilotBdata.csv"
Data exclusion / filtering
Participants who fail both comprehension checks will be excluded from the analysis to ensure data quality and participant understanding.
Prepare data for analysis - create columns etc.
We will create specific columns for the following variables: - Participant ID; - Condition (cause-first or overhead-first); - Demographic Variables: age, gender, nationality; - Satisfaction: recorded on a Likert scale; - Frequency: previous donation experience; - Amount Donated: Separate columns for donations to overhead and cause; - Total Amount Donated: sum of donations to cause and overhead.
Codes for cleaned data:
# Runs function and produces .csv fileresult <-process_csv_files(file_paths, output_file)## Cleans JSON formatting# Function removes all of the special charcters from the dataremove_special_characters <-function(entry) {gsub("[^a-zA-Z0-9\\s]", "", entry)}# Applies function to all relevant columnscleanresult <- result %>%mutate(check1 =sapply(check1, remove_special_characters)) %>%mutate(check2 =sapply(check2, remove_special_characters)) %>%mutate(condition =sapply(condition, remove_special_characters)) %>%mutate(gender =sapply(gender, remove_special_characters)) %>%mutate(comment =sapply(comment, remove_special_characters))# Edits data for legibilitycleanresult <- cleanresult %>%mutate(condition =recode( condition,ForthecauseForcoveringcharitableorganizationsoperatingexpense ='causefirst',ForcoveringcharitableorganizationsoperatingexpenseForthecause ='overfirst' ),check1 =gsub("chanceresponse", "", check1),check2 =gsub("donationuseresponse", "", check2),gender =gsub("gender", "", gender),comment =gsub("Q0", "", comment) )# Produces .csv file write.csv(cleanresult,"cleanPilotBdata.csv", row.names =FALSE)
Confirmatory analysis
The confirmatory analysis will focus on testing whether the order of donation requests significantly affects the total amount donated. We will conduct an independent samples t-test comparing the mean total donation amounts between the two conditions: cause-first and overhead-first. Additionally, a series of regression analyses will be conducted to further investigate the effect of donation order while controlling for demographic covariates, such as age and gender. Together, these analyses address the hypothesis that the overhead-first condition will lead to higher total donations.
Pilot A Confirmatory analysis
Using the pilot A data of 16 samples we collected, we performed a multiple regression analysis on the total donation amount to establish the analysis structure:
cleanresultA <-read.csv("pilotdata.csv")# Test regression analysis (total donations ~ other variables)outputA <-lm('total ~ condition + age + gender', data = cleanresultA)summary(outputA)
Call:
lm(formula = "total ~ condition + age + gender", data = cleanresultA)
Residuals:
Min 1Q Median 3Q Max
-4.291 -1.135 0.000 2.192 2.698
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 3.64156 2.04436 1.781 0.109
conditionoverfirst -2.96111 1.63132 -1.815 0.103
age 0.07856 0.05377 1.461 0.178
genderMale 1.87193 2.41019 0.777 0.457
genderNonbinary -5.44834 3.07433 -1.772 0.110
genderOther 5.35566 3.16656 1.691 0.125
genderPrefernottosay -10.49270 6.16016 -1.703 0.123
Residual standard error: 2.845 on 9 degrees of freedom
Multiple R-squared: 0.6202, Adjusted R-squared: 0.367
F-statistic: 2.449 on 6 and 9 DF, p-value: 0.1097
Pilot B Confirmatory Analysis
We conducted a pilot B study with 5 participants recruited through Prolific. To analyze the data, we performed several regression analyses. A linear regression model was fit to predict total donations using condition, age, and gender as predictors, followed by separate models for overhead and cause donations. Despite offering the opportunity to donate, no participants chose to make a donation. The average time that participants took during Pilot B was 2 minutes and 39 seconds.
file_id check1 check2 cause overhead total condition satisfaction
1 1 50 Alloftheabove 0 0 0 overfirst 6
2 2 20 Alloftheabove 0 0 0 causefirst 4
3 3 20 Alloftheabove 0 0 0 causefirst 6
4 4 20 Alloftheabove 0 0 0 overfirst 0
5 5 20 Alloftheabove 0 0 0 overfirst 4
frequency age nationality gender comment
1 3 54 American Female NA
2 2 35 American Female NA
3 1 52 American Female NA
4 0 29 American Male NA
5 2 23 American Female NA
The confirmatory analysis tests the primary hypothesis from the original study regarding the effect of donation collection methods (cause-first vs. overhead-first) on donation behavior. Results will be deemed successful if the p-value is < 0.05, indicating a statistically significant difference between the conditions in their effect on total donations, overhead donations, or cause donations.
Regression on Total Donations (Key Analysis)
totaloutput <-lm('total ~ condition + age + gender', data = cleanresult)summary(totaloutput)
Call:
lm(formula = "total ~ condition + age + gender", data = cleanresult)
Residuals:
1 2 3 4 5
0 0 0 0 0
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 0 0 NaN NaN
conditionoverfirst 0 0 NaN NaN
age 0 0 NaN NaN
genderMale 0 0 NaN NaN
Residual standard error: 0 on 1 degrees of freedom
Multiple R-squared: NaN, Adjusted R-squared: NaN
F-statistic: NaN on 3 and 1 DF, p-value: NA
Regression on Overhead Donations:
overoutput <-lm('overhead ~ condition + age + gender', data = cleanresult)summary(overoutput)
Call:
lm(formula = "overhead ~ condition + age + gender", data = cleanresult)
Residuals:
1 2 3 4 5
0 0 0 0 0
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 0 0 NaN NaN
conditionoverfirst 0 0 NaN NaN
age 0 0 NaN NaN
genderMale 0 0 NaN NaN
Residual standard error: 0 on 1 degrees of freedom
Multiple R-squared: NaN, Adjusted R-squared: NaN
F-statistic: NaN on 3 and 1 DF, p-value: NA
Regression on Cause Donations:
causeoutput <-lm('cause ~ condition + age + gender', data = cleanresult)summary(causeoutput)
Call:
lm(formula = "cause ~ condition + age + gender", data = cleanresult)
Residuals:
1 2 3 4 5
0 0 0 0 0
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 0 0 NaN NaN
conditionoverfirst 0 0 NaN NaN
age 0 0 NaN NaN
genderMale 0 0 NaN NaN
Residual standard error: 0 on 1 degrees of freedom
Multiple R-squared: NaN, Adjusted R-squared: NaN
F-statistic: NaN on 3 and 1 DF, p-value: NA
The confirmatory analyses could not provide meaningful insights, as all donation variables (total, cause, overhead) were 0 for all participants. This outcome is likely due to the smaller sample size and the absence of an incentive structure, which may have limited participant engagement and motivation to donate.
Exploratory analyses
The exploratory analysis focuses on satisfaction, as it is not directly relevant to the main hypotheses regarding donation behavior but provides additional insights into participants’ experiences. In this pilot B analysis, we conducted three regression models to explore predictors of satisfaction: (1) a simple model using condition as the predictor, (2) a model using condition, age and gender as covariates, and (3) a full model including condition, age, gender, and the amounts donated to the cause and overhead. These analyses were conducted to examine any potential relationships between satisfaction, donation behaviors, and demographic variables.
donation1 <-lm('satisfaction ~ condition', data = cleanresult)summary(donation1)
Call:
lm(formula = "satisfaction ~ condition", data = cleanresult)
Residuals:
1 2 3 4 5
2.6667 -1.0000 1.0000 -3.3333 0.6667
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 5.000 1.856 2.694 0.0742 .
conditionoverfirst -1.667 2.396 -0.696 0.5367
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Residual standard error: 2.625 on 3 degrees of freedom
Multiple R-squared: 0.1389, Adjusted R-squared: -0.1481
F-statistic: 0.4839 on 1 and 3 DF, p-value: 0.5367
donation2 <-lm('satisfaction ~ condition + age + gender', data = cleanresult)summary(donation2)
Call:
lm(formula = "satisfaction ~ condition + age + gender", data = cleanresult)
Residuals:
1 2 3 4 5
-1.904e-01 -3.472e-01 3.472e-01 1.839e-17 1.904e-01
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 1.6592 1.0518 1.578 0.360
conditionoverfirst 0.3840 0.5711 0.672 0.623
age 0.0768 0.0224 3.429 0.181
genderMale -4.2704 0.7181 -5.947 0.106
Residual standard error: 0.56 on 1 degrees of freedom
Multiple R-squared: 0.9869, Adjusted R-squared: 0.9477
F-statistic: 25.18 on 3 and 1 DF, p-value: 0.1452
donation3 <-lm('satisfaction ~ condition + age + gender + cause + overhead', data = cleanresult)summary(donation3)
Call:
lm(formula = "satisfaction ~ condition + age + gender + cause + overhead",
data = cleanresult)
Residuals:
1 2 3 4 5
-1.904e-01 -3.472e-01 3.472e-01 1.839e-17 1.904e-01
Coefficients: (2 not defined because of singularities)
Estimate Std. Error t value Pr(>|t|)
(Intercept) 1.6592 1.0518 1.578 0.360
conditionoverfirst 0.3840 0.5711 0.672 0.623
age 0.0768 0.0224 3.429 0.181
genderMale -4.2704 0.7181 -5.947 0.106
cause NA NA NA NA
overhead NA NA NA NA
Residual standard error: 0.56 on 1 degrees of freedom
Multiple R-squared: 0.9869, Adjusted R-squared: 0.9477
F-statistic: 25.18 on 3 and 1 DF, p-value: 0.1452
In the first model, condition alone explained minimal variation in satisfaction (R² = 0.139, p = 0.537). Adding demographic variables (age and gender) improved the model’s fit (R² = 0.987), with age and gender showing stronger but still non-significant effects (p = 0.181 and p = 0.106, respectively). Including donation variables (cause and overhead) in the full model did not contribute to the analysis due to their lack of variability. Overall, these results suggest that neither condition nor the tested covariates had a meaningful impact on satisfaction in this pilot study. We need a larger sample and more data to draw firmer conclusions.
Discussion
Summary of Replication Attempt
Open the discussion section with a paragraph summarizing the primary result from the confirmatory analysis and the assessment of whether it replicated, partially replicated, or failed to replicate the original result.
Commentary
Add open-ended commentary (if any) reflecting (a) insights from follow-up exploratory analysis, (b) assessment of the meaning of the replication (or not) - e.g., for a failure to replicate, are the differences between original and present study ones that definitely, plausibly, or are unlikely to have been moderators of the result, and (c) discussion of any objections or challenges raised by the current and original authors about the replication attempt. None of these need to be long.