Replication of Study 2 from Effects of donation collection methods on donation amount: Nudging donation for the cause and overhead by Suk & Mudita (2022, Psychology & Marketing)

Introduction

Charity organizations rely on donations to both support their advertised cause and maintain their overhead expenses. However, donors support charities less when they think their donations are used more for overheads than for a cause, a heuristic known as overhead aversion. In their paper, Effects of donation collection methods on donation amount: Nudging donation for the cause and overhead, Drs. Kwanho Suk and Triza Mudita contribute to research on reducing/adapting to overhead aversion. They do this by administering surveys through crowdsourcing websites, with which they assess the affects of choice architecture on participants’ willingness to donate to charity.

In the second of their three documented experiments, Suk and Mudita found that participants give more money when asked to donate to a charity’s cause first, before being asked to donate to its overhead expenses, as opposed to overheads first, cause second. Based on this finding, charity organizations could get more money to achieve their goals simply by changing the order in which donations are asked for. We therefore replicate Suk and Mudita’s experiment in this replication to assess the reliability of their data.

Methods

Power Analysis

Planned Sample

We aim to collect data from at least 140 participants through crowdsourcing methods, to assure we have at least 70 participants in each of our two conditions. Our plan is to terminate data collection once this minimum is met. Although the majority of participants from the original study came from the UK, we do not control for nationality, or any other demographic variables.

Materials

In order to conduct this experiment, we will create a custom website using jsPsych to administer surveys like the ones Suk and Mudita created. We will create one for the cause first, overheads second condition, like the one in Appendix C from the original study, and one for the overheads first, cause second condition, which reverses the order of the first two questions.

The first item in the cause first condition will read, “For the cause:” and have a text box where participants can fill in the amount they want to donate to the charity’s cause. The second item will read, “For covering charitable organization’s operating expenses,” and have a fillable text box as well. Below those boxes will be an item reading “Total,” listing the sum total of both of the participants inputs. All values will be indicated in pounds. For the overheads first condition, the order of the first two items is reversed.

After they confirm their total donation, we will also ask participants, “How satisfied are you with your donation?”, prompting them to input a value on a 7-point Likert scale (1 indicating not at all, 7 indicating complete satisfaction). This value will be an additional independent variable in a series of regression analyses conducted after data collection.

Procedure

Our procedure mirrors that of the original study. At the start of our experiment, we ask for participants’ consent. Then, we ask participants to list their age, gender, and nationality. We also state that one fifth of our participants (28 participants minimum) will earn 6 pounds (7.77 USD) upon completion of the survey. To ensure participants understand this, we administer a comprehension check.

We then inform participants of the way charity organizations use donations to both support their cause and to maintain their operation expenses. We will also inform participants that some charities receive these two kinds of donations separately. As the last part of the information portion of our experiment, we present a non-specific donation campaign for children born with an arm or leg disability in an Asian country. This is followed by another comprehension check to assess whether our participants understood what they had been told.

Upon reading through the relevant information, participants will be randomly assigned into either the cause first or overhead first conditions. In each condition, all participants are given the opportunity to donate some, all, or none of the 6 pounds they stand to gain by completing this experiment to the donation campaign. Depending on which condition they are placed in, they will be asked to provide the amount they want to donate to the cause first, before being able to indicate the amount they want to donate to the overheads, or vice-versa.

Regardless of condition, participants will be able to confirm the total amount of money they would be donating to both the cause and the overhead expenses. Participants are reminded that they can donate as much as they want so long as their total does not exceed 6 pounds, and that they can also choose to keep the full six pounds. Lastly, participants are asked to rate their satisfaction with their donation on a Likert scale from 1 to 7.

Upon having collected data from 70 participants in each condition, we will randomly select 20% of our participants to earn six pounds. In accordance with what they said they would donate if they earned the six pounds, we will donate the amount they specified to a real charity organization, World Vision, and send them the remainder.

Analysis Plan

A series of tests will be administered to our data to match those conducted in the original experiment. First, we conduct a regression analysis to determine whether gender, age, and/or condition covary with the total amount donated. Then, we conduct two separate regression analyses to determine the impact of these factors on amounts donated to our charity’s cause and overheads specifically. We also conduct a t-test to assess the difference between the mean amounts donated between both conditions.

We then utilize our participants’ self-reported satisfaction in a second round of tests. We perform a regression analysis using condition as an independent variable and satisfaction as a dependent variable to see if the order in which donations are asked for affects donor sentiment. We then add demographic variables and the amounts donated in each condition for causes and overheads to this regression in order to assess interactions between these variables.

Differences from Original Study

The original experiment was administered as an optional continuation of an unrelated survey. After completing this survey, participants volunteered to take the actual experiment, in return for a 20% chance of earning six pounds. We have chosen not to include this unrelated survey, and instead directly administer the experiment, giving all of our participants this 20% chance.

Design Overview

Analyze the design choices:
How many factors were manipulated?

One - the order in which participants were asked to donate to the cause and overheads of the charity.

How many measures were taken?

Three - the donation to the cause, the donation to the overheads, and participants’ satisfaction with their donation.

Did it use a within-participants or between-participants design?

A between-participants design was used.

Were measures repeated?

No measures were repeated.

What would have been the consequence of changing one of these decisions? (e.g. moving from within to between participants). Could it have been either way?

It couldn’t have been done as a within-participants experiment. If we were to use a within-participants design, we would be asking participants to donate to the same charity twice in 6 pound increments. This raises concerns with demand characteristics, as it makes what we’re testing for more obvious. We could change this by asking participants to donate to two different charities, but that would introduce the potential confound that participants might prefer one charity to the other. Therefore, this experiment must be between-participants.

Were steps taken to reduce demand characteristics?

In the original study, an unrelated survey was administered before the actual experiment, and the actual experiment was treated as an optional add-on. This reduces demand characteristics because it frames the actual experiment as something separate from what the participants think the study is. Because the participants aren’t treating the actual experiment as part of the survey, they aren’t thinking about what the experimenters want to see. Therefore, the unrelated survey functions as a kind of cover story which protects from demand characteristics.

Can you think of any potential confounds to the study?

One potential confound is the order in which participants are informed about the causes and overheads of charities. This experiment is specifically testing the effect of order on donations; it stands to reason that the order in which we inform participants about how charities work may affect our manipulation. For instance, if we describe how charities need donations for their overhead expenses before we describe how charities need donations to support their cause, participants may see donating to the overhead expenses as more important, affecting their responses.

Do you see any limitations to generalizability?

Though this study has interesting findings, I am concerned about its generalization to real-life situations. Would people actually donate more to charities that put their cause first and overheads second when asking for donations? It’s a difficult inference because in real situations, participants do not stand to gain any money like they are with this study. Some people would probably be fine with donating 3 pounds to charity if they were also receiving 3 pounds, but not if they had to donate entirely from their own pockets, receiving no compensation in return.

Additionally, participants cannot donate more than 6 pounds in this study, but in real life, there is no upper limit to how much one could donate. The constraints within the experiment are comfortable for participants to think about because a clear range for which their donation can fall under

Additionally, participants are asked to provide how much they would donate based on the assumption that they would be one of the lucky 20% to earn 6 pounds at the end of the study. Participants may not be taking their answers seriously because they expect to be one of the 80% that earns nothing. At those odds, participants may be more generous than they normally are because they want to prove to themselves that they are generous without being restrained by the possibility of keeping some of the money because they expect not to receive any.

Methods Addendum (Post Data Collection)

Actual Sample

Differences from pre-data collection methods plan

Results

Data Preparation

Using jsPsych, we will establish a pipeline from our custom website to a dataset that populates in .csv format as results come in. We will monitor this dataset continually, filtering out participants that fail the confirmation checks, until we ensure that we have 70 participants in each condition.

Load Relevant Libraries and Functions

We will need to load Tidyverse, which is necessary for reading, cleaning, and analyzing .csv data. We will also need lmtest to perform regression analyses with the lm function. T-tests can be conducted in base R using the t.test function.

Import data

We will import the data using the tidyverse library’s read_csv() function.

Data exclusion / filtering

If participants fail both of the confirmation checks we administer before they list how much money they would donate, their results will be excluded from the study.

Prepare data for analysis - create columns etc.

Each observation will be labeled with a participant ID to eliminate the need for personally identifiable information. We will need columns in our dataset for the experimental condition, age, gender, nationality, satisfaction, amounts donated to the cause and overheads, the total amount donated, and whether that participant was selected to receive 6 pounds or not.

Confirmatory analysis

We will use a t-test to determine the statistical significance of the difference between the mean total amounts donated in the cause first and overheads first conditions. We will also use a regression analysis to see if age and gender covary with the experimental condition and the total amount donated.

Link to repository: https://github.com/ucsd-psych201a/suk2023

Exploratory analyses

Any follow-up analyses desired (not required).

Discussion

Summary of Replication Attempt

Open the discussion section with a paragraph summarizing the primary result from the confirmatory analysis and the assessment of whether it replicated, partially replicated, or failed to replicate the original result.

Commentary

Add open-ended commentary (if any) reflecting (a) insights from follow-up exploratory analysis, (b) assessment of the meaning of the replication (or not) - e.g., for a failure to replicate, are the differences between original and present study ones that definitely, plausibly, or are unlikely to have been moderators of the result, and (c) discussion of any objections or challenges raised by the current and original authors about the replication attempt. None of these need to be long.