Replication of Study 2 from Effects of donation collection methods on donation amount: Nudging donation for the cause and overhead by Suk & Mudita (2022, Psychology & Marketing)
Introduction
Charity organizations rely on donations to both support their advertised cause and maintain their overhead expenses. However, donors support charities less when they think their donations are used more for overheads than for a cause, a heuristic known as overhead aversion. In their paper, Effects of donation collection methods on donation amount: Nudging donation for the cause and overhead, Drs. Kwanho Suk and Triza Mudita contribute to research on reducing/adapting to overhead aversion. They do this by administering a series of experiments with which they assess the affects of choice architecture on participants’ willingness to donate to charity.
In the second of their three documented experiments, Suk and Mudita found that participants give more money when asked to donate to a charity’s overhead expenses first, before being asked to donate to its causes, as opposed to cause first, overheads second. Based on this finding, charity organizations could get more money to achieve their goals simply by changing the order in which donations are asked for. We therefore replicate Suk and Mudita’s experiment in this replication to assess the reliability of their data.
Methods
Power Analysis
Using G*power, we conducted an a priori power analysis of the original finding that the total amount donated by participants was greater for those in the overhead first condition as opposed to the cause first condition. Our power analysis assumed a t-test for two independent means. Using the finding’s effect size (d = .41) yielded an estimate of 188 participants needed to replicate this finding with a statistical power of .8.
Planned Sample
We aim to collect data from at least 188 participants through Prolific, as per our power analysis, to assure we have at least 94 participants in each of our two conditions. Our plan is to terminate data collection once this minimum is met. Although the majority of participants from the original study came from the UK, we do not control for nationality, or any other demographic variables.
Materials
In order to conduct this experiment, we create a custom website using js-Psych to administer a website like the one Suk and Mudita devised. We start with a 10-question unrelated survey that asks participants about various habits, such as their sleep patterns, hobbies, and daily lives. This survey functions as a cover story; once participants complete it, they progress to the real experiment under the premise that they have completed the bulk of the work, making our actual manipulation seem more like an add-on. This will reduce demand characteristics, as our participants will not treat the actual experiment as something the researchers are looking for answers to.
We follow the example set by Appendix C from the original study to create two experimental conditions, cause first and overhead first. The first item in the cause first condition will read, “For the cause:” and have a text box where participants can fill in the amount they want to donate to the charity’s cause. The second item will read, “For covering charitable organization’s operating expenses,” and have a text box as well. Below those boxes will be an item reading “Total,” listing the sum total of both of the participants’ inputs. All values are indicated in US dollars. For the overheads first condition, the order of the first two items is reversed.
After they confirm their total donation, we also ask participants to indicate their agreement with two questions, “I am satisfied with my donation,” and “I usually donate to charity.” These items are answered by inputting values on a 7-point Likert scale (1 indicating “not at all,” 7 indicating “very much”). These values will be independent variables in a series of regression analyses conducted after data collection.
Procedure
Our procedure mirrors that of the original study. At the start of our experiment, we ask for participants’ consent. We then administer our unrelated survey. After completing the survey, participants move on to the experiment, and are informed they have a 20% chance to earn 5 USD upon completion. To ensure participants understand this, we administer a comprehension check here.
We then inform participants of the way charity organizations use donations to both support their cause and to maintain their operation expenses. We also inform participants that some charities receive these two kinds of donations separately. As the last part of the information portion of our experiment, we tell participants they have the opportunity, provided they win the extra 5 dollars, to give to a non-specific donation campaign for children born with disabilities in an Asian country. This is followed by another comprehension check to assess whether our participants understood what they had been told.
Upon reading through the relevant information, participants will be randomly assigned into either the cause first or overhead first condition. In each condition, all participants are given the opportunity to donate some, all, or none of the 5 dollars they stand to gain by completing this experiment to the donation campaign. Depending on which condition they are placed in, they will be asked to provide the amount they want to donate to the cause first, before being able to indicate the amount they want to donate to the overheads, or vice-versa.
Regardless of condition, participants will be able to confirm the total amount of money they would be donating to both the cause and the overhead expenses. Participants are reminded that they can donate as much as they want so long as their total does not exceed 5 USD, and that they can also choose to keep the full amount. Then, participants are asked to rate their satisfaction with their donation, as well as how often they donate to charity, on Likert scales from 1 to 7. At the very end of our survey, we ask participants to indicate their age, gender, and nationality before debriefing.
Upon having collected data from at least 94 participants in each condition, we will randomly select 20% of our participants to earn the full 5 USD (around 38 out of at least 188 participants), regardless of how much they wanted to donate in their responses. We will also send them a link to a real charity in case they would like to follow through with their intent to donate.
Analysis Plan
A series of tests will be administered to our data to match those conducted in the original experiment. First, we conduct a regression analysis to determine whether any of the data we have recorded can explain the variation within the total amount donated by participants. Then, we conduct two separate regression analyses to determine the impact of these factors on amounts donated to our charity’s cause and overheads specifically.
We then utilize our participants’ self-reported satisfaction in a second round of tests. We perform a regression analysis using condition as an independent variable and satisfaction as a dependent variable to see if the order in which donations are asked for affects donor sentiment. We then add demographic variables as independent variables for another regression analysis, and the amounts donated in each condition for causes and overheads in a third in order to assess interactions among all of our variables.
Differences from Original Study
The majority of the original study’s participants were from the UK, so the bonus incentive offered for completing the experiment was 6 pounds. However, as our participant base is American, we must convert this amount from pounds to US dollars. Additionally, due to budgeting concerns, we must also lower this amount to a 5 dollar compensation for 20% of our participants, which converts to 3.99 pounds. In essence, we offer roughly two-thirds of the original compensation to our sample of American participants.
Design Overview
Only one factor was manipulated, that being the order in which participants were asked to donate to the cause and overheads of the charity. Four measures are taken, the donation to the cause, the donation to the overheads, participants’ satisfaction with their donation, and participants’ frequency with which they donate. Additional variables, like age, gender, and nationality, are also recorded.
This study employs a between-participants design; participants may only supply data in one condition of our experiment. No measures are repeated. Like in the original study, an unrelated survey is administered before the actual experiment, and the actual experiment is treated as an add-on. This reduces demand characteristics because it frames the actual experiment as something separate from what the participants think the study is. Because the participants aren’t treating the actual experiment as part of the survey, they aren’t thinking about what the experimenters want to see. Therefore, the unrelated survey functions as a kind of cover story which protects from demand characteristics.
Methods Addendum (Post Data Collection)
Actual Sample
Differences from pre-data collection methods plan
Results
Data Preparation
Using jsPsych, we establish a pipeline from our custom website to an OSF database that populates in .csv format as results come in. As we conduct this experiment online using Prolific, we specify that data collection will end after 188 participants respond to our experiment. We will then analyze our data for participants who fail our comprehension checks and exclude them from the analysis. If necessary, we reopen the study to collect more data to replace the proportion of participants who fail our comprehension checks.
Load Relevant Libraries and Functions
For our data cleaning and analyses, we need tidyverse, which is necessary for reading, cleaning, and analyzing .csv data. Additionally, jsonlite will assist in converting the raw output from the datapipe into a usable format.
library(jsonlite)
library(tidyverse)We utilize the following functions in our analysis:
## For data formatting
read_csv()
subset()
data.frame()
mutate()
lapply()
sapply()
tryCatch()
fromJSON()
unnest_wider()
bind_rows()
write_csv()
## For removing JSON formatting
mutate()
recode()
gsub()
## For analyses
lm()
summary()
group_by()
summarize()Import data
Individual responses to our experiment are added to our OSF repository as individual .csv files. Using OSF’s interface, we can take multiple .csv files and put them into a folder, and then download that folder as .zip file. We import the data in this folder to R by using this function, which takes the folder’s filepath and creates a vector with the filepaths of all the .csv files within.
file_paths <- c(
list.files(path = "...", full.names = TRUE)
)Data exclusion / filtering
If participants fail either of the confirmation checks we administer before they list how much money they would donate, their results will be excluded from the study. We may also exclude results if they enter non-serious answers for demographic variables or other inputs.
Prepare data for analysis - create columns etc.
This function is designed to take the convert our raw data into a useable format. The nested function takes the outputs from each response’s .csv file and turns them into a row with columns for comprehension checks, the amounts donated for which purposes, which condition the participant was exposed to, and other variables. Then, the outer function joins these individual rows to create a single .csv file.
process_csv_files <- function(file_paths, output_file) {
# Nested function to process a single file
process_file <- function(file_path) {
# Reads file
testdata <- read_csv(file_path)
# Shortens file to specified columns
tdcols <- subset(testdata, select = -c(rt, stimulus, trial_type, plugin_version, question_order))
# Shortens file to specified rows
tdrows <- tdcols[-c(1:13, 16:17), , drop = FALSE]
# Rearranges elements of cleaned data into one cohesive row
tdarrange <- data.frame(
check1 = tdrows$response[1],
check2 = tdrows$response[2],
cause = tdrows$causeDonation[3],
overhead = tdrows$expenseDonation[3],
total = tdrows$totalDonation[3],
condition = tdrows$questionOrder[3],
satis_usual = tdrows$response[4],
age = tdrows$age[5],
nationality = tdrows$nationality[5],
gender = tdrows$response[6],
comment = tdrows$response[7]
)
# Parses the satisfaction responses, unnests JSON formatting
tdparsed <- tdarrange %>%
mutate(satis_usual = lapply(satis_usual, function(x) {
tryCatch(fromJSON(x), error = function(e) NA)
}))
# Unnests the satisfaction/frequency column into two separate columns
tdsep <- tdparsed %>%
unnest_wider(col = satis_usual, names_sep = "_")
# Renames columns
tdclean <- tdsep %>%
rename(satisfaction = satis_usual_Q0,
frequency = satis_usual_Q1)
# Returns cleaned data
return(tdclean)
}
# Apply nested function to each file
processed_files <- lapply(file_paths, process_file)
# Combines files, adds identifying number
all_data <- bind_rows(processed_files, .id = "file_id")
# Saves the combined data
write_csv(all_data, output_file)
# Returns combined data
return(all_data)
}This .csv file is further cleaned through a series of steps designed to remove all traces of JSON formatting from the data. Additionally, variables are renamed for ease of comprehension and analysis. The final product becomes the basis for our analyses.
## Cleans JSON formatting
# Function removes all of the special charcters from the data
remove_special_characters <- function(entry) {
gsub("[^a-zA-Z0-9\\s]", "", entry)
}
# Applies function to all relevant columns
cleanresult <- result %>%
mutate(check1 = sapply(check1, remove_special_characters)) %>%
mutate(check2 = sapply(check2, remove_special_characters)) %>%
mutate(condition = sapply(condition, remove_special_characters)) %>%
mutate(gender = sapply(gender, remove_special_characters)) %>%
mutate(comment = sapply(comment, remove_special_characters))
# Edits data for legibility
cleanresult <- cleanresult %>%
mutate(
condition = recode(
condition,
ForthecauseForcoveringcharitableorganizationsoperatingexpense = 'causefirst',
ForcoveringcharitableorganizationsoperatingexpenseForthecause = 'overfirst'
),
check1 = gsub("chanceresponse", "", check1),
check2 = gsub("donationuseresponse", "", check2),
gender = gsub("gender", "", gender),
comment = gsub("Q0", "", comment)
)Confirmatory analyses
Our confirmatory analyses consist of a series of three regression models. These models use the amount donated to the charity’s cause, the amount donated to the charity’s overheads, and the total amount donated as target variables, and the condition, age, and gender of participants as explanatory variables. For our inference criteria, our findings must exhibit a p-value of less than .05 and the correct direction (for instance, that participants donate more in the overhead first condition than in the cause first condition, not the other way around).
Pilot A
This pilot utilizes a sample of personally chosen volunteers who were not randomly selected. To confirm our preparations for analysis work, we conduct a regression analysis to determine the statistical significance of the independent variables we have collected on the total amounts donated. With 16 data points from our pilot study, we produce this output, ensuring our pipelines and formatting work as intended.
> output <- lm('total ~ condition + age + gender', data = cleanresult)
> summary(output)
Call:
lm(formula = "total ~ condition + age + gender", data = cleanresult)
Residuals:
Min 1Q Median 3Q Max
-4.291 -1.135 0.000 2.192 2.698
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 3.64156 2.04436 1.781 0.109
conditionoverfirst -2.96111 1.63132 -1.815 0.103
age 0.07856 0.05377 1.461 0.178
genderMale 1.87193 2.41019 0.777 0.457
genderNonbinary -5.44834 3.07433 -1.772 0.110
genderOther 5.35566 3.16656 1.691 0.125
genderPrefernottosay -10.49270 6.16016 -1.703 0.123
Residual standard error: 2.845 on 9 degrees of freedom
Multiple R-squared: 0.6202, Adjusted R-squared: 0.367
F-statistic: 2.449 on 6 and 9 DF, p-value: 0.1097From this analysis, we can see that none of our variables have a significant relationship with total donation. We then calculate Cohen’s d:
outputmean <- cleanresult %>% group_by(condition) %>% summarize(mean = mean(total))
outputsd <- cleanresult %>% summarize(sd = sd(total))
CohensD <- (outputmean$mean[1] - outputmean$mean[2])/outputsd$sd[1]Our Cohen’s D was .5483, indicating a medium effect size. The means of the total donations among the cause first and overheads first conditions are 5.8889 and 3.9286, respectively. The standard deviation of total donations among both conditions is 3.5754. From our pilot sample, we observe the opposite effect as what the original study found, but this is likely due to our poor sample size. More accurate results will be obtained from our real sample.
Pilot B
This pilot utilizes a random sample of 5 participants found on Prolific. On average, it took participants around 2 minutes and 39 seconds to respond to the entire survey. Although we attempt to conduct our confirmatory analyses, all of the participants responded that they would not donate any money for the charity’s cause or overheads. This means that all observations of the target variables for our confirmatory analyses are 0. Therefore, the outputs of all of our regression analysis follow this example:
> totaloutput <- lm('total ~ condition + age + gender', data = cleanresult)
> summary(totaloutput)
Call:
lm(formula = "total ~ condition + age + gender", data = cleanresult)
Residuals:
1 2 3 4 5
0 0 0 0 0
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 0 0 NaN NaN
conditionoverfirst 0 0 NaN NaN
age 0 0 NaN NaN
genderMale 0 0 NaN NaN
Residual standard error: 0 on 1 degrees of freedom
Multiple R-squared: NaN, Adjusted R-squared: NaN
F-statistic: NaN on 3 and 1 DF, p-value: NAThis makes our confirmatory analyses impossible to conduct, given that there is no variation among our target variables. Similarly, this makes our calculations of effect size invalid. From these results, the pilot data suggests that condition, age and gender do not affect participants’ donations.
Exploratory analyses
In Suk and Mudita’s original study, three exploratory analyses unrelated to the paper’s main hypotheses were conducted. These were regression analyses with donor satisfaction as the target variable that became increasingly more complex as more explanatory variables were added. In the first regression analysis, condition was the only explanatory variable. In the second, age and gender were added for a total of three features, and in the third, amounts donated for the charity’s cause and overheads were added for a total of five features.
Pilot B
Although all participants responded with “0” as the amount they would donate to both the cause and overhead conditions, there was variation in their reported satisfaction, making this series of exploratory analyses possible.
Satisfaction predicted by condition
> donation1 <- lm('satisfaction ~ condition', data = cleanresult)
> summary(donation1)
Call:
lm(formula = "satisfaction ~ condition", data = cleanresult)
Residuals:
1 2 3 4 5
0.6667 -3.3333 1.0000 -1.0000 2.6667
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 5.000 1.856 2.694 0.0742 .
conditionoverfirst -1.667 2.396 -0.696 0.5367
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
Residual standard error: 2.625 on 3 degrees of freedom
Multiple R-squared: 0.1389, Adjusted R-squared: -0.1481
F-statistic: 0.4839 on 1 and 3 DF, p-value: 0.5367Satisfaction predicted by condition, age, and gender
> donation2 <- lm('satisfaction ~ condition + age + gender', data = cleanresult)
> summary(donation2)
Call:
lm(formula = "satisfaction ~ condition + age + gender", data = cleanresult)
Residuals:
1 2 3 4 5
1.904e-01 -6.939e-18 3.472e-01 -3.472e-01 -1.904e-01
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 1.6592 1.0518 1.578 0.360
conditionoverfirst 0.3840 0.5711 0.672 0.623
age 0.0768 0.0224 3.429 0.181
genderMale -4.2704 0.7181 -5.947 0.106
Residual standard error: 0.56 on 1 degrees of freedom
Multiple R-squared: 0.9869, Adjusted R-squared: 0.9477
F-statistic: 25.18 on 3 and 1 DF, p-value: 0.1452Satisfaction predicted by condition, age, gender, and donations
> donation3 <- lm('satisfaction ~ condition + age + gender + cause + overhead', data = cleanresult)
> summary(donation3)
Call:
lm(formula = "satisfaction ~ condition + age + gender + cause + overhead",
data = cleanresult)
Residuals:
1 2 3 4 5
1.904e-01 -6.939e-18 3.472e-01 -3.472e-01 -1.904e-01
Coefficients: (2 not defined because of singularities)
Estimate Std. Error t value Pr(>|t|)
(Intercept) 1.6592 1.0518 1.578 0.360
conditionoverfirst 0.3840 0.5711 0.672 0.623
age 0.0768 0.0224 3.429 0.181
genderMale -4.2704 0.7181 -5.947 0.106
cause NA NA NA NA
overhead NA NA NA NA
Residual standard error: 0.56 on 1 degrees of freedom
Multiple R-squared: 0.9869, Adjusted R-squared: 0.9477
F-statistic: 25.18 on 3 and 1 DF, p-value: 0.1452None of our models are significant, although the p-value somewhat improves between the first and second analyses. Due to the nature of our collected data on cause and overhead donations, the second and third regressions are functionally equivalent. These results suggest the lack of a significant relationship between our participants’ satisfaction in their donations and any of our explanatory variables.
Discussion
Summary of Replication Attempt
Pilot Studies
In both pilot studies, the confirmatory analyses failed to replicate. In the first pilot study, our confirmatory analysis was not significant, and seemed to lean in the opposite direction, where mean total donations were greater in the cause-first condition. In the second pilot study, our confirmatory analyses could not be conducted, due to there being no variation in our target variables.
Commentary
Pilot Studies
Pilot B illuminated that on Prolific, a crowdsourcing platform in which people take surveys for monetary incentives, putting participants in a sitution where they can donate to charity may not be effective. Given that none of the participants were willing to donate, a significant portion of our actual sample may also choose to opt out of donation. Our analysis may be greatly restricted by the abundance of 0 values in our target variables; although it is anticipated that at least a few participants will give non-zero values for their donations, the rest of the participants may weigh down our linear regressions.
This may be because our participants are based in America, and not the UK. This difference in the circumstances surrounding nationality may have played some role in influencing our results. Although, as we only recruited five participants for our pilot study, we should not be overeager to assume that a majority of participants in our actual sample will opt out of donation. This is because small sample sizes tend to have more extreme effects.
Additionally, our study was only active for around five minutes before we had reached 5 participants. The crowdworkers who are consistently monitoring Prolific for new studies to take may be in greater need of money, and would be less likely to donate funds they stand to gain. In this case, when we increase the sample size we specify on Prolific, we may get a more diverse range of results.
Our data on the Open Science Framework is stored here:
Our survey can be taken here:
https://ucsd-psych201a.github.io/suk2023/website/
The HTML code for the survey can be seen here:
https://github.com/ucsd-psych201a/suk2023/blob/main/website/index.html
Pilot data can be found here:
https://github.com/ucsd-psych201a/suk2023/blob/main/data/pilotdata.csv
Link to repository: