Replication of Study 1a by Rubin et al. (2025, Nature Human Behaviour)

Introduction

Justification

Rubin et al. seek to understand whether people perceive empathy differently when it comes from another person compared to an AI system. In Study 1a, Rubin et al. focus specifically on whether the perceived source of empathy affected participants’ perceived sense of empathy. As a part of my PhD research, I have ongoing work focused on the relationship between AI companion usage and user well-being. In prior work, we found that interacting with AI companions on Character.AI have a negative relationship with user well-being. One of the benefits of AI companions that our study participants reported was that these systems could provide emotional support and reduce loneliness; however, a detriment is that using AI companions can displace human-human relationships. Nonetheless, our current work is limited to only survey results and correlational results. Replicating Rubin et al.’s work on perceived empathy depending on human vs. AI sources is a good first step in teasing apart more of the causal mechanisms related to my line of work.

Stimuli and Procedures

For Study 1a, Participants are randomly assigned into one of two conditions: one group is told that they would receive a response generated by an AI and the second group is told that they would receive a response from a human. After, participants are asked to describe a recent emotional experience. After 60 seconds, they are then shown a response to their experience. In both conditions, the response participants are shown is AI-generated. Participants then rate how much empathy they felt the response as well as answering questions that disaggregated dimensions of empathy.

There are two main challenges I anticipate with this study. The first will be developing the experimental code. The authors provide the prompts that they used to generate the AI-responses in the Methods section, but they do not provide the code for the experimental platform, which I will need to develop myself. One modification I will make from the original methods is to use GPT-4.1, which is the latest state-of-the-art non-reasoning model for generating responses, rather than GPT-4, which is no longer offered from OpenAI. The second challenge is whether I will be able to recruit a sufficient number of participants as the original study recruited ~800. While I believe recruiting this number of participants is feasible for an online study, I believe it is out-of-scope and budget for the class. Nonetheless, I ran a power analysis using the effect size (Cohen’s d = 0.34) reported in the paper; to achieve a power of 80% at an alpha level of 0.05, we would need a total sample size of 216 participants (or 108 per condition). Given that the study is on the shorter-side, I believe this is feasible. Nonetheless, it is possible that my study could be underpowered.

Repo Link

Repository: https://github.com/psych251/rubin2025/tree/main

Original Paper: https://github.com/psych251/rubin2025/blob/main/original_paper/s41562-025-02247-w.pdf

Experimental Paradigm: https://stanforduniversity.qualtrics.com/jfe/form/SV_1YAwJsT5JNf428u

Methods

Power Analysis

In the paper, the authors report an effect size of 0.34. I estimate power at 0.80, 0.90, 0.95 for the effect size in the paper as well as a more conservative Cohen’s d of 0.3.

library(pwr)
d_original <- 0.34
d_conservative <- 0.3

## power levels to test
powers <- list(.80, .90, .95)

## power analyses for t-tests with effect size reported in original paper
for (power in powers) {
  print(power)
  print(pwr.t.test(d = d_original, power = power, type = "two.sample"))
}

[1] 0.8

     Two-sample t test power calculation 

              n = 136.7605
              d = 0.34
      sig.level = 0.05
          power = 0.8
    alternative = two.sided

NOTE: n is number in *each* group

[1] 0.9

     Two-sample t test power calculation 

              n = 182.755
              d = 0.34
      sig.level = 0.05
          power = 0.9
    alternative = two.sided

NOTE: n is number in *each* group

[1] 0.95

     Two-sample t test power calculation 

              n = 225.7869
              d = 0.34
      sig.level = 0.05
          power = 0.95
    alternative = two.sided

NOTE: n is number in *each* group

for (power in powers) {
  print(power)
  print(pwr.t.test(d = d_conservative, power = power, type = "two.sample"))
}

[1] 0.8

     Two-sample t test power calculation 

              n = 175.3847
              d = 0.3
      sig.level = 0.05
          power = 0.8
    alternative = two.sided

NOTE: n is number in *each* group

[1] 0.9

     Two-sample t test power calculation 

              n = 234.4627
              d = 0.3
      sig.level = 0.05
          power = 0.9
    alternative = two.sided

NOTE: n is number in *each* group

[1] 0.95

     Two-sample t test power calculation 

              n = 289.7353
              d = 0.3
      sig.level = 0.05
          power = 0.95
    alternative = two.sided

NOTE: n is number in *each* group

Planned Sample

Based on the power analyses for the more conservative effect size (Cohen’s d = .30), I will collect data from 350 participants and data collection will stop once complete data from all 350 participants has been collected from Prolific. The experiment should take about 4 minutes to complete.

Materials

The participants are asked to describe an emotional experience and then showed a response generated using a large language model (LLMs). Given the stochastic nature of LLMs and the different inputs that participants provide, the responses the model provides will vary. Nonetheless, we use the prompt provided by the authors in the SI when generating responses. We deviate slightly from the original work’s material in that we use Gemini-2.5.-Flash to generate responses rather than GPT-4o. These models should be similar in capabilities and are both closed-sourced models.

Procedure

We follow the procedure as described in the original article as follows:

“We told the participants they were paired with either an AI or another participant. The participants shared a recent emotional experience and waited for 60 seconds; they were told either that the AI was generating a response or that the other participant was writing one, depending on the experimental condition. Participants in both conditions were then shown an AI-generated response, with the AI having been prompted to respond to their specific experience and include all three aspects of empathy.”

Analysis Plan

First, we will exclude participants who fail the attention check question in the task and those who do not provide an emotional response, which we will manually validate. Next, following Rubin et al., we removed “any participants that were more than 2.5 standard deviations away from the mean of the dependent variable for that analysis” and ensured that “all independent categorical variables were effect-coded.”

The first analysis of interest is comparing which response participants found more empathic by conducting a Welch’s t-test on the general empathy question between conditions.

The second analysis is comparing which response participants found more postively resonant again by conducting a Welch’s t-test on the positive resonance measure (mean across the three questions) between conditions.

Finally, to see whether condition affected different types of empathy (e.g., cognitive, affective, motivational), we fit a ” a linear mixed-effect model to predict empathy with condition, aspect of empathy (cognitive, affective or motivational) and their interaction.”

Differences from Original Study

There are three main differences. First, we will be using a different LLM (Gemini-2.5-Flash instead of GPT 4-0613) to generate the responses. We select Gemini-2.5-Flash due to practical cost constraints with the replication study, as Gemini offers a more generous rate limit for lower tiered API accounts compared to OpenAI. We select Gemini-2.5-Flash as it is a more cost-efficient model while still having capabilities comparable to GPT-4. This change should not produce material differences given that it is a model with similar capabilities to that used in the original study. Furthermore, in the article, they also performed the same study using an open-source Llama model to find similar results. Second, we are recruiting a much smaller sample size of 350 participants in comparison to the original study, which had 725 participants. Third, we are adding an additional validation to ensure that the participants provided a description of a recent emotional event. Since the LLM will generate a response regardless of whether the participant inputs an emotional response, we require a post-hoc verification that the participants’ input matched our question. Thus, I will manually review the submitted responses and remove those that do not describe a recent event.

Methods Addendum (Post Data Collection)

You can comment this section out prior to final report with data collection.

Actual Sample

Sample size, demographics, data exclusions based on rules spelled out in analysis plan

Differences from pre-data collection methods plan

Any differences from what was described as the original plan, or “none”.

Results

Data preparation

Data preparation following the analysis plan.

Confirmatory analysis

First, we conduct two t-tests comparing perceived empathy and positive resonance between conditions.

empathy_ttest <- t.test(formula = Empathy_1~Condition, data = Q1_filtered)
print(empathy_ttest)

    Welch Two Sample t-test

data:  Empathy_1 by Condition
t = -0.27735, df = 1.7423, p-value = 0.8109
alternative hypothesis: true difference in means between group AI and group Human is not equal to 0
95 percent confidence interval:
 -9.467062  8.467062
sample estimates:
   mean in group AI mean in group Human 
                8.5                 9.0 

Q1PlotViolinE1 <- 
ggplot(data = Q1_filtered, aes(x = Condition, y = general_empathy,
                                                    fill = Condition, color = Condition))+
  ggdist::stat_halfeye( # for distributions
    xmax = 3, # creates distance between them - the max height
    adjust = 1, # adjust bandwidth
    justification = -.3, # less close to boxplot
    alpha = 0.6,
    # remove the slub interval
    .width = 0,
    point_colour = NA) + 
  geom_boxplot(color = "black", 
               width = 0.25,
               # removing outliers
               outlier.color = NA,
               alpha = 0.8)+
  ggdist::stat_dots(aes(fill = Condition), side = "left", 
                    justification = (1.2), 
                    binwidth = 0.15, overflow = "compress")+
  scale_color_manual(values = c("indianred1","darkturquoise"))+
  scale_fill_manual(values = c("indianred1","darkturquoise"))+
  theme(axis.text.x = element_text(face = "bold", size = 13),
        legend.position = "none")+
  #geom_point(aes(fill = Condition),shape=21, position=position_jitterdodge(0.2), inherit.aes = T)+
  stat_summary(geom = "errorbar", fun.data = mean_cl_normal,
               position = position_dodge(.75), linewidth=.3, width =.1, colour="black")+
  stat_summary(fun=mean,  geom="point",
               shape=4, size=2, colour="black",
               position = position_dodge(0.75))+
  xlab("Condition")+
  ylim(c(0,11))+
  ylab("General Empathy")+
  theme_classic() +
  ggtitle("Effect of Condition on Perceived Empathy")

Q1PlotViolinE1

resonance_ttest <- t.test(formula = positive_resonance~Condition, data = Resonance_filtered)
print(resonance_ttest)

    Welch Two Sample t-test

data:  positive_resonance by Condition
t = -2.0004, df = 1.0799, p-value = 0.2805
alternative hypothesis: true difference in means between group AI and group Human is not equal to 0
95 percent confidence interval:
 -107.82202   73.82202
sample estimates:
   mean in group AI mean in group Human 
           51.66667            68.66667 

PosResPlotE1 <- ggplot(data = Resonance_filtered, aes(x = Condition, y = positive_resonance,
                                                    fill = Condition, color = Condition))+
  ggdist::stat_halfeye( # for distributions
    xmax = 3, # creates distance between them - the max height
    adjust = 1, # adjust bandwidth
    justification = -.3, # less close to boxplot
    alpha = 0.6,
    # remove the slub interval
    .width = 0,
    point_colour = NA) + 
  geom_boxplot(color = "black", 
               width = 0.25,
               # removing outliers
               outlier.color = NA,
               alpha = 0.8)+
  ggdist::stat_dots(aes(fill = Condition), side = "left", 
                    justification = (1.2), 
                    binwidth = 0.15, overflow = "compress")+
  scale_color_manual(values = c("indianred1","darkturquoise"))+
  scale_fill_manual(values = c("indianred1","darkturquoise"))+
  theme(axis.text.x = element_text(face = "bold", size = 13),
        legend.position = "none")+
  #geom_point(aes(fill = Condition),shape=21, position=position_jitterdodge(0.2), inherit.aes = T)+
  stat_summary(geom = "errorbar", fun.data = mean_cl_normal,
               position = position_dodge(.75), linewidth=.3, width =.1, colour="black")+
  stat_summary(fun=mean,  geom="point",
               shape=4, size=2, colour="black",
               position = position_dodge(0.75))+
  xlab("Condition")+
  ylab("Positivity Resonance")+
  theme_classic() +
  ggtitle("Effect of Condition on Positivity Resonance")


PosResPlotE1

ggsave("PosResPlotE1.png", PosResPlotE1, width = 5, height = 5)

model <- lmer(empathy.s~Condition*empathy_type+(1|ID),data=filtered_data)
summary(model)

Linear mixed model fit by REML. t-tests use Satterthwaite's method [
lmerModLmerTest]
Formula: empathy.s ~ Condition * empathy_type + (1 | ID)
   Data: filtered_data

REML criterion at convergence: 23.7

Scaled residuals: 
    Min      1Q  Median      3Q     Max 
-1.0331 -0.6351  0.0000  0.6351  1.0331 

Random effects:
 Groups   Name        Variance Std.Dev.
 ID       (Intercept) 0.2346   0.4843  
 Residual             1.3126   1.1457  
Number of obs: 12, groups:  ID, 4

Fixed effects:
                                        Estimate Std. Error      df t value
(Intercept)                              -0.2039     0.8795  5.7363  -0.232
ConditionHuman                            0.2447     1.2439  5.7363   0.197
empathy_typeaffective                    -0.2447     1.1457  4.0000  -0.214
empathy_typemotivational                  0.4895     1.1457  4.0000   0.427
ConditionHuman:empathy_typeaffective     -0.1224     1.6203  4.0000  -0.076
ConditionHuman:empathy_typemotivational   0.1224     1.6203  4.0000   0.076
                                        Pr(>|t|)
(Intercept)                                0.825
ConditionHuman                             0.851
empathy_typeaffective                      0.841
empathy_typemotivational                   0.691
ConditionHuman:empathy_typeaffective       0.943
ConditionHuman:empathy_typemotivational    0.943

Correlation of Fixed Effects:
                    (Intr) CndtnH empthy_typf empthy_typm CndtnHmn:mpthy_typf
ConditinHmn         -0.707                                                   
empthy_typf         -0.651  0.461                                            
empthy_typm         -0.651  0.461  0.500                                     
CndtnHmn:mpthy_typf  0.461 -0.651 -0.707      -0.354                         
CndtnHmn:mpthy_typm  0.461 -0.651 -0.354      -0.707       0.500             

The analyses as specified in the analysis plan.

Exploratory analyses

ggplot(data = Q1_filtered, aes(x=other_source, y=general_empathy, color=Condition)) +
  geom_point() + 
  geom_smooth(method = "lm") +
  ylab("Perceived empathy from response")+
  xlab("Human aided by AI / AI edited by human")+
  theme_classic()

`geom_smooth()` using formula = 'y ~ x'

Warning in qt((1 - level)/2, df): NaNs produced
Warning in qt((1 - level)/2, df): NaNs produced

Warning in max(ids, na.rm = TRUE): no non-missing arguments to max; returning
-Inf
Warning in max(ids, na.rm = TRUE): no non-missing arguments to max; returning
-Inf

Discussion

Summary of Replication Attempt

Open the discussion section with a paragraph summarizing the primary result from the confirmatory analysis and the assessment of whether it replicated, partially replicated, or failed to replicate the original result.

Commentary

Add open-ended commentary (if any) reflecting (a) insights from follow-up exploratory analysis, (b) assessment of the meaning of the replication (or not) - e.g., for a failure to replicate, are the differences between original and present study ones that definitely, plausibly, or are unlikely to have been moderators of the result, and (c) discussion of any objections or challenges raised by the current and original authors about the replication attempt. None of these need to be long.