Replication of Study 2 in ‘Effects of donation collection methods on donation amount: Nudging donation for the cause and overhead’ by Suk & Mudita (2023, Psychology & Marketing)
Link to the repository for the full project: https://github.com/ucsd-psych201a/suk2023
Introduction
I have chosen to replicate Study 2 in the paper “Effects of donation collection methods on donation amount: Nudging donation for the cause and overhead” because it aligns closely with my research interests in charitable giving and prosocial behavior, particularly in addressing the challenge of overhead aversion. Overhead aversion refers to the tendency of donors to be reluctant to contribute to a charity’s operational costs, often preferring their donations to go directly to the cause. This aversion can negatively impact charities by limiting their ability to cover essential costs such as staffing and infrastructure, which are critical to achieving long-term goals. The paper I’ve chosen examines how different donation collection methods, such as varying the sequence in which donors are asked to contribute to the cause and overhead, can mitigate overhead aversion and increase the total donation amount without reducing donor satisfaction. The paper’s findings are intriguing and I’m curious to see whether the results can be replicated. Its findings have strong implications for helping nonprofit organizations overcome overhead aversion and increase contributions sustainably, which aligns with my research focus on effective charitable giving strategies.
To replicate the experiment, the main stimuli will include a charitable donation scenario, where participants are asked to make donation decisions using different collection methods. Participants will be asked to imagine they have $6 to donate, and then go through different donation processes. The key manipulation involves varying how the donation options are presented: participants will first decide how much to give to the cause, followed by how much to contribute to overhead (cause first), or vice versa (overhead first).
One challenge will be designing the donation collection screens in a way that mimics real-world charity websites, making the stimuli both realistic and easy for participants to understand. Another potential difficulty is ensuring that participants take the scenario seriously, especially in an online setting where hypothetical donations might not fully capture real-world behaviors. Additionally, recruiting a sufficient and diverse sample of participants with varying levels of donation experience could be challenging, as previous donation experience has been shown to influence outcomes.
Methods
Power Analysis
Original effect size, power analysis for samples to achieve 80%, 90%, 95% power to detect that effect size. Considerations of feasibility for selecting planned sample size.
Planned Sample
The target sample size for this study is approximately 140 participants, with around 70 individuals assigned to each condition. Participants have to be at least 18 years old, and there will be no other restrictions or controls placed on age, gender, or other demographic characteristics. Recruitment will continue until the target sample size is reached.
Materials
Donation Input Form: Participants will be asked to indicate how much they would like to donate if they win a prize. Participants will have the option to write down “$0” if they wish to keep all prize money without making a donation.
Donation Satisfaction Scale: After making their donation decision, participants will rate their satisfaction on a Likert scale (1 = not at all, 7 = very much).
Donation Information: All participants will receive an overview of how charitable organizations use donations. They will be informed that donations may be allocated to (1) helping the cause and (2) covering the organization’s operational costs, such as overhead. Additionally, participants will learn that some charities accept separate donations for the cause and the overhead, as in the original article: “This information will be presented in both conditions to ensure that the participants will expect that they could donate separately for the cause and the overhead.
Procedure
Participant Recruitment: Participants will be recruited from the Prolific Academic online panel. They will be informed that 20% of the randomly selected participants will receive an extra $6.
Donation Decision: Participants will be informed that they can choose to donate some or all of the prize money, if won, to charity. The donation will be completely voluntary, and the participants will be asked to write down $0 if they want to keep all the bonus prize money without donation.
Information on Charitable Giving: All participants will receive an overview of how charitable organizations collect and spend donation money. They will be informed that donations can be used for (1) helping the cause and (2) covering the organization’s operations, or overhead costs. Participants will also be told that some organizations accept separate donations for cause and overhead expenses.
Donation Campaign Overview: Participants will be presented with information on a donation campaign to support disabled children born with physical disabilities in an Asian country. The campaign will be described as being run by a trustworthy charity, but no specific charity name will be mentioned. This will be done to avoid potential influence of participants’ attitude toward existing charities.
Condition Assignment: Participants will be randomly assigned to one of two conditions:
Cause-First Condition: Participants will first indicate the amount they want to donate to the cause. On the next page, they will indicate the amount for overhead. “On this page, the amount donated for the cause will be shown in the box, and the participants will be allowed to change the amount if they want to.” The total will then be displayed on the screen.
Overhead-First Condition: In this condition, participants will specify their donation amount for overhead first, followed by the cause. The procedure will be identical to the cause-first condition except for the order.
Total Display and Reminder: The total donation amount will be calculated automatically and displayed on-screen. Participants will be reminded that they can donate any amount up to $6 and that they will keep any remaining funds as prize money.
Prize Allocation and Feedback: After completing the study, “20% of the participants will be randomly selected as prize winners.” The total donation from winners will be donated to a charity supporting children overseas, and each winner will receive their non-donated prize amount. All participants will receive a feedback message explaining whether they won, their prize amount, and how the donations were used.
Analysis Plan
Total Donation Amount Analysis:
A regression analysis will be conducted to examine the effect of donation collection method (cause-first vs. overhead-first) on the total donation amount. The primary hypothesis is that donation amounts will differ based on whether the overhead or the cause donation is collected first. As in the original study, we will expect to find that participants in the overhead-first condition donate more overall than those in the cause-first condition.
Gender and age will be included as covariates in the model to test for their potential influence on donation amounts, with expectations that these demographic variables will not significantly impact the outcome, based on the original results.
Donation for Overhead and Cause Analysis:
Separate regression analyses will be conducted to examine the amounts donated to overhead and to the cause. The main predictor in each model will be the donation collection method (cause-first vs. overhead-first), with the hypothesis that participants in the overhead-first condition will donate more to the overhead than those in the cause-first condition. Likewise, an unexpected but noteworthy finding in the original study was a higher cause donation in the overhead-first condition, which will be tested for replication.
Gender and age will again be included as covariates to control for any potential demographic effects on donation distribution.
Donation Satisfaction Analysis:
Satisfaction with donation decisions will be tested using a series of regression analyses in three steps:
Step 1: A simple regression with donation collection method as the independent variable, to test if satisfaction levels differ between the cause-first and overhead-first conditions.
Step 2: Adding demographic covariates (age and gender) to the model, allowing us to assess any effects of these variables on satisfaction levels, with an expected significant influence of age as observed in the original study.
Step 3: Adding donation amounts for the cause and overhead as covariates to assess whether the donation amounts influence satisfaction levels. Based on the original study, we expect higher donation amounts to be associated with greater satisfaction. At this step, we will also re-evaluate the effect of donation collection method to see if it becomes significant after controlling for donation amounts.
Comparative Analysis (t-tests):
- A t-test will be conducted to compare the mean amounts donated to the cause and to the overhead across both conditions. This analysis will provide insight into any systematic differences in how participants allocate their donations between these two categories.
Differences from Original Study
Sample and Setting:
- The sample will be recruited in the same way as in the original study, via an online panel, but may not include an initial unrelated survey prior to participation in the donation study. This omission is due to practical constraints and aims to streamline recruitment and participation. The sample will not be restricted to UK.
Instructions:
- The original study’s instructions will be used verbatim where possible. Otherwise, a closely aligned set of instructions will be developed that preserves the intent and detail of the original materials.
Methods Addendum (Post Data Collection)
You can comment this section out prior to final report with data collection.
Actual Sample
Sample size, demographics, data exclusions based on rules spelled out in analysis plan
Differences from pre-data collection methods plan
Any differences from what was described as the original plan, or “none”.
Design Overview
The original study utilized a between-participants design, with one manipulated factor and three distinct measures. Measures were not repeated, and each participant was exposed to only one experimental condition. A between-participants design was likely chosen to avoid any potential awkwardness or perceived redundancy that could arise from donating to the same charity multiple times, which might also reveal the experimental purpose to participants. This decision aligns well with the study’s objective, as a within-participants design may have increased the risk of participants discerning the treatment and thus responding differently.
To minimize demand characteristics, an unrelated survey was administered prior to the donation task, which helped to distract participants and potentially reduced any preconceived expectations about the study’s purpose. This approach likely maintained the integrity of responses by focusing participants’ attention elsewhere before the key donation ask.
One critique of the study’s design is that including an unrelated survey could introduce unintended influences on participant behavior, potentially acting as a confound. For instance, the content or tone of the survey might subtly impact their mindset when making donation decisions. Additionally, the order in which each type of donation (i.e., cause vs. overhead costs) was explained could also act as a confounding factor, possibly affecting participants’ understanding or perception of each donation type (e.g., through recency effect).
A few factors limit the generalizability of the study’s findings. Notably, only 20% of participants received actual payment, which may have influenced how participants perceived and engaged with the donation task, knowing they were not guaranteed real funds to donate. Moreover, donations were made using money provided by the experimenters rather than participants’ own money. This setup may not accurately capture genuine donation behavior, as participants may behave differently when donating from personal funds. These factors suggest that the results might be specific to the experimental context and may not fully generalize to real-world donation scenarios.
Results
Data preparation
Data will be collected through jsPsych. Various packages will be needed for reading and analyzing .csv data, as well as conducting regression analyses and t-tests. Based on participants’ comprehension check responses, we will filter results to include only those who passed. The dataset will include the following columns: Participant ID, Condition, Demographics (Age, Gender, Nationality), Amount Donated to Overhead, Amount Donated to Cause, Total Donation Amount, Donation Satisfaction, and Donation Status (indicating if the participant was among the 20% selected to win and/or donate additional compensation).
Confirmatory analysis
Total Donation Amount: A regression analysis will test if the donation collection method (cause-first vs. overhead-first) influences total donation amount, with age and gender as covariates, replicating the original finding that overhead-first leads to higher donations.
Donation Allocation: Separate regression analyses will assess differences in amounts allocated to cause and overhead, confirming if the overhead-first method increases donations to both categories.
Donation Satisfaction:Three regression analyses will test if satisfaction varies by donation order, adding age, gender, and donation amounts in steps to control for their effects.
Comparative Donation Types: A t-test will compare mean donations to cause vs. overhead to confirm any consistent allocation preference across conditions.
Side-by-side graph with original graph is ideal here
Exploratory analyses
Any follow-up analyses desired (not required).
Discussion
Summary of Replication Attempt
Open the discussion section with a paragraph summarizing the primary result from the confirmatory analysis and the assessment of whether it replicated, partially replicated, or failed to replicate the original result.
Commentary
Add open-ended commentary (if any) reflecting (a) insights from follow-up exploratory analysis, (b) assessment of the meaning of the replication (or not) - e.g., for a failure to replicate, are the differences between original and present study ones that definitely, plausibly, or are unlikely to have been moderators of the result, and (c) discussion of any objections or challenges raised by the current and original authors about the replication attempt. None of these need to be long.