Introduction

The authors were interested in the age-old question of what makes people who they are, and how people’s beliefs about personal identity are structured. Specifically, they proposed that people’s representations of personal identity incorporate the causal relationships among features of identity. One possibility is that people reason about their self-concept similarly to how they reason about concepts in general, such that more causally central features are more defining of identity.

In Experiment 1, subjects were shown 16 features of personal identity and were asked for each feature to: (1) rate how disruptive it would be to their identity if the feature were to change in a meaningful way, and (2) indicate which other features are caused by that feature and how strong the causal relationship is. Subjects either thought about their own identity (Self condition), a close other’s identity (Close-other condition), or a generic other’s identity (Generic-other condition). Here, we specifically test the the Self condition. The target finding for replication is that features with more causal connections were rated as being more disruptive to identity.

Methods

Link to testable replication paradigm: https://web.stanford.edu/~masaba/254/Chen2016/experiment/chen_exp.html

Power Analysis

Original Values

Original r value = .34

Original sample size (in Self condition): 80 subjects

Original power: .878

Power Analysis

Power analysis for replication given original effect size.

80% power: N=65

90% power: N=86

95% power: N=106

We decided to run the replication with N=65 for 80% power.

Planned Sample

The planned sample size is 65, and participants will be English-speaking adults from the United States. According to the original study, participants were excluded due to a scripting error (5), failing attention checks (4), providing identical answers to every question (2). We will use these same criteria for excluding participants.

Materials

The task was originally presented in an online survey via Amazon’s Mechanical Turk. The specific features that the authors chose for the experiment are: cherished memories of time with parents/family, important childhood memories, memories of important life milestones, level of wholesomeness, level of honesty, level of loyalty, intelligence level, degree of shyness, reliability, goals for personal life, favorite hobbies/activities, aesthetic preferences, knowledge of math, knowledge of music, height, and level of hunger.

Here is the exact wording for the key questions, quoted from Experiment 1 Materials via OSF.

[Identity Task] “…we would first like to understand how a change in each of the features below would change your identity. That is, for each of the features below, imagine that you are completely different on that dimension (e.g., for height, if you are tall, imagine that something changed so that you are now short). Do you think that you would still be the same person you are now, or would you be a different person?

Please indicate your answer with each of the sliders below where 0 means “I would be the exact same person” and 100 means “I would be a completely different person.”"

[Causal Task] “Think about your X.

Which of the other features of your personal identity listed below, if any, are caused by your X? You may select as many or as few features as you see fit. In the below list, please select all the features that you believe are caused by the above feature."

In the paper, the materials and procedure were combined into a single “methods” which I have copied below.

Procedure

Quoted from Experiment 1 Methods on p. 1399-1400:

“The power analysis from a pilot experiment (for details, see Appendix S1 in the Supplemental Material available online) suggested a sample size of 80 per cell. Two hundred fifty Amazon Mechanical Turk respondents from the United States were randomly assigned to one of three conditions (self, close other, or generic other). Five participants were excluded because of a scripting error, 4 because they failed an attention check, and 2 because they gave identical answers to every question, for a total of 11 exclusions. This resulted in a final sample of 239. Results were similar when we included all participants who provided usable data (for details, see Appendix S1).

All participants completed both a causal-relationships task and an identity-disruption survey. To measure centrality, we asked participants in the self condition to report the causal connections among the features of their own identity. To measure perceived disruption to identity, we asked these participants to rate the extent to which a change in each identity feature would disrupt their own identity. Participants in the close-other condition did the same for a nonromantic close other whom they specified. Participants in the generic-other condition completed the tasks for a generic other person. The order of the tasks was counterbalanced across participants within each condition.

In the causal-relationships task, participants reported the causal relationships that they perceived among 16 features of personal identity (see Table 1). Twelve of the 16 features were intended to be of high importance and were chosen from four categories of personal identity that had been identified as important in previous research: autobiographical memories, personality, morality, and preferences and desires (e.g., Strohminger & Nichols, 2014). The remaining 4 were intended to be of low importance. Two were found in previous research to be less important for identity (instances of semantic memories; Strohminger & Nichols, 2014), and 2 (fillers) were found to be unimportant to identity in a pretest.

After practicing the causal-relationships task with an unrelated concept and receiving feedback, participants completed 16 randomized trials. In each trial, a different feature was the target. Participants indicated which of the other 15 features, if any, was caused by the target feature (see Fig. 1). Then, for each feature selected as a direct effect, participants rated the strength of its relationship (1 = weak, 2 = moderate, 3 = strong) with the target feature.

In the identity-disruption survey, participants rated the extent to which change in each feature would disrupt the identity of the person that corresponded to their condition (i.e., self, close other, or generic other). They rated disruption on a scale of 0 to 100; larger numbers indicated greater disruption. (For the wording of the question, see Appendix S1 in the Supplemental Material.)"

We followed this exact procedure with the main exception being that we only ran the Self condition.

Analysis Plan

Quoted from Experiment 1 Results on p. 1400:

“Our analyses used the number of causal connections (i.e., the number of other features to which a target fea- ture was directly linked, either as a cause or as an effect) as the measure of causal centrality. More links indicated greater centrality. Our findings were similar when we used an alternative approach, causal depth (the dependency model; Sloman et al., 1998), as the measure of causal centrality (for details of analysis and results, see Appendix S2 in the Supplemental Material).” […] the average individual-level correlations (within individual participants, across all items) were positive in all conditions (see Table 2). The majority of participants in all conditions rated changes in the fea- tures with more causal connections as being more disruptive (rs was positive for 77%, 84%, and 74% of participants in the self, close-other, and generic-other conditions, respectively)."

Table 2 shows a positive correlation between disruption ratings and causal centrality (rs = .34, 95% CI [.25, .44], t(78) = 7.29, p < .001). The key analysis of interest will be an individual-level Spearman correlation on disruption ratings and number of causal connections for each feature. The authors (1) calculated the spearman correlations of disruption ratings and number of causal connections for each subject, (2) performed a fisher transformation (from rho to z) on each subject’s rho, and (3) running a one-sample t-test on the z-scores to ask whether the average rho was greater than zero.

Differences from Original Study

There are minimal differences between the original study and the replication. In the identity task, it was unclear how participants provide answers (e.g., via a rating scale, or a write-in text-box). Here, we’ve used a sliding scale from 0 to 100.

Methods Addendum (Post Data Collection)

Actual Sample

60 adults (20 female, M_Age(SD) = 38.58(12.33), 20 - 66) were recruited from Amazon’s Mechanical Turk. An additional 10 participants were recruited but excluded due to technical error (4), failing attention checks (4), and providing the same response for all questions in the identity task (1).

Differences from pre-data collection methods plan

None.

Results

Data preparation

Data preparation following the analysis plan.

Load Relevant Libraries and Functions

Import data

Data exclusion / filtering

#remove attention check rows 
cd = cd.raw %>%
  filter(causal_target != "attention_check") %>%
  rename(identity_feature = causal_target) #rename to match feature coulum in id

##Exclusion criteria 1: Failing one or both attention checks 
attn_check = cd.raw %>%
  group_by(workerid) %>%
  filter(causal_target == "attention_check") %>%
  mutate(attn_score = if_else(causal_checked == "pass",1,0)) %>%
  summarise(attn_score = sum(attn_score)) #score of 2 means that participants passed both checks

attn_check_fail = subset(attn_check,attn_check$attn_score < 2) #table of participants who failed attention check

#remove participants who failed attention check

##Exclusion criteria 2: Responding with all same ratings in identity task
ratings_check_fail = id.raw %>%
  group_by(workerid) %>%
  summarise(sd = sd(identity_ratings)) %>%
  filter(sd == 0)

id = id.raw[!(id.raw$workerid %in% ratings_check_fail$workerid),] #remove participants
id = id.raw[!(id.raw$workerid %in% attn_check_fail$workerid),]
cd = cd[!(cd$workerid %in% attn_check_fail$workerid),] 
cd = cd[!(cd$workerid %in% ratings_check_fail$workerid),]

Prepare data for analysis - create columns etc.

#create new column that holds number of features that were checked for each target feature
cd_counts_checked = cd %>%
  group_by(workerid, identity_feature) %>%
  filter(causal_checked != "none") %>%
  mutate(count = n())

#for "none" responses, put "0" in the new column
cd_counts_none = cd %>%
  group_by(workerid, identity_feature) %>%
  filter(causal_checked == "none") %>%
  mutate(count = 0)

#join these two data frames 
cd_counts = full_join(cd_counts_checked,cd_counts_none)
## Joining, by = c("workerid", "identity_feature", "causal_checked", "causal_strengths", "count")
#final data frame 
cd_counts_final = cd_counts %>%
  group_by(workerid, identity_feature) %>%
  distinct(identity_feature, .keep_all = TRUE) %>% #just one row for each feature
  select(workerid, identity_feature,count)

identity_causal = full_join(id,cd_counts_final) #combine identity ratings and causal df
## Joining, by = c("workerid", "identity_feature")
## Warning in full_join_impl(x, y, by$x, by$y, suffix$x, suffix$y): joining
## factors with different levels, coercing to character vector
#fix identity_causal frame
identity_causal$identity_ratings = as.numeric(identity_causal$identity_ratings)
identity_causal = rename(identity_causal, num_connections = count)
identity_causal = na.omit(identity_causal)

#make table displaying mean number of connections and mean disruption rating for each feature
identity_causal_tbl = identity_causal %>%
  group_by(identity_feature) %>%
  summarize(mean_rating = mean(identity_ratings),
            mean_connections = mean(num_connections))

Confirmatory analysis

Individual-level correlations.

alpha=.05
z_critical = qnorm(alpha/2, lower.tail=FALSE)

#take individual correlations, find rho and p, and fisher transform rhos 
indiv_cor = identity_causal %>%
  group_by(workerid) %>%
  mutate(r = cor.test(identity_ratings,num_connections)$estimate,
         p = cor.test(identity_ratings,num_connections)$p.value) %>%
  select(workerid, r, p) %>%
  distinct(workerid, .keep_all = TRUE) %>%
#  ungroup() %>%
  mutate(fisher_z = fisherz(r)) #fisher transform on rhos
## Warning in cor(x, y): the standard deviation is zero

## Warning in cor(x, y): the standard deviation is zero
#excluding participant who gave 1 connection for all features
indiv_cor = na.omit(indiv_cor)

indiv_cor_t = t.test(indiv_cor$fisher_z) #one-sample t-test
indiv_cor_t  
## 
##  One Sample t-test
## 
## data:  indiv_cor$fisher_z
## t = 6.2738, df = 59, p-value = 4.522e-08
## alternative hypothesis: true mean is not equal to 0
## 95 percent confidence interval:
##  0.1965575 0.3806572
## sample estimates:
## mean of x 
## 0.2886074
indiv_cor_table = indiv_cor %>%
  ungroup() %>%
  summarise(num = length(workerid),
            mean_indiv_r = mean(r),
            sem_indiv_r = sd(r)/sqrt(num),
            low_95 = mean_indiv_r - (sem_indiv_r * z_critical),
            upp_95 = mean_indiv_r + (sem_indiv_r * z_critical))

indiv_cor_table
## # A tibble: 1 × 5
##     num mean_indiv_r sem_indiv_r    low_95    upp_95
##   <int>        <dbl>       <dbl>     <dbl>     <dbl>
## 1    60    0.2578597  0.03974556 0.1799598 0.3357596

We test the correlation between disruption ratings and causal centrality and find the following result: rs = 0.26, 95% CI [0.18, 0.34], t(59) = 6.27, p < .001.

Exploratory analyses

Aggregate-level correlation. In addition to the key result, the authors also performed a group-level correlation of disruption ratings and causal centrality as evidence that people reason about the causal structure of personal identity. They collapsed by item across subjects to calculate a mean disruption rating and a mean causal centrality score for each feature. Then, a spearman correlation was run on these aggregate disruption ratings and causal centrality scores.

#aggregate-level spearman correlation
aggregate.test = cor.test(identity_causal_tbl$mean_rating,identity_causal_tbl$mean_connections, method = "spearman")
## Warning in cor.test.default(identity_causal_tbl$mean_rating,
## identity_causal_tbl$mean_connections, : Cannot compute exact p-value with
## ties
aggregate.test
## 
##  Spearman's rank correlation rho
## 
## data:  identity_causal_tbl$mean_rating and identity_causal_tbl$mean_connections
## S = 177.63, p-value = 0.001079
## alternative hypothesis: true rho is not equal to 0
## sample estimates:
##       rho 
## 0.7387787
#scatter plot 
ggplot(identity_causal_tbl,aes(mean_rating,mean_connections)) +
  geom_point() +
  geom_smooth(method="lm") +
  ylab("Number of Causal Connections") +
  xlab("Disruption Ratings (0-100)") +
  ylim(0,5)+
  xlim(0,100)+
  theme_bw()

We test the correlation between mean disruption ratings (averaged across subjects) and mean causal centrality and find the following result: rs = 0.74, p = 0.001.

Linear Mixed-Effects Model. Finally, we ask whether causal centrality predicts identity disruption ratings in a linear mixed-effects model with subject and feature as random effects.

lme.test = lmer(identity_ratings ~ num_connections + (1|workerid) + (1|identity_feature), data=identity_causal)

sum.lme.test = summary(lme.test)
sum.lme.test
## Linear mixed model fit by REML ['lmerMod']
## Formula: 
## identity_ratings ~ num_connections + (1 | workerid) + (1 | identity_feature)
##    Data: identity_causal
## 
## REML criterion at convergence: 9354.6
## 
## Scaled residuals: 
##     Min      1Q  Median      3Q     Max 
## -3.2453 -0.7390  0.0742  0.6893  3.3425 
## 
## Random effects:
##  Groups           Name        Variance Std.Dev.
##  workerid         (Intercept) 254.62   15.957  
##  identity_feature (Intercept)  87.66    9.363  
##  Residual                     737.73   27.161  
## Number of obs: 976, groups:  workerid, 61; identity_feature, 16
## 
## Fixed effects:
##                 Estimate Std. Error t value
## (Intercept)      50.7821     3.4064  14.908
## num_connections   1.8708     0.6038   3.098
## 
## Correlation of Fixed Effects:
##             (Intr)
## num_cnnctns -0.321

In this linear-mixed effects model of identity disruption ratings with causal centrality as a main effect and subject and feature as random effects, we find the following result: causal centrality (est. = 1.87, t = 3.1).

Demographics

dd = dd.raw[(dd.raw$workerid %in% cd$workerid),]
dd = dd.raw[(dd.raw$workerid %in% indiv_cor$workerid),]

dd = dd %>%
  rename(age = age.value,
        lang = age.value.1,
        ethnicity = age.value.2,
        gender = age.value.3) %>%
  select(age, lang, ethnicity, gender) 

dd$age = as.numeric(dd$age)
dd$lang = factor(dd$lang)
dd$ethnicity = factor(dd$ethnicity)
dd$gender = factor(dd$gender)

dd.age = dd %>%
  summarise(mean_age = mean(age),
            sd_age = sd(age),
            min_age = min(age),
            max_age = max(age))
dd.age
##   mean_age  sd_age min_age max_age
## 1 38.58333 12.3306      20      66
table(dd$gender)
## 
##     female       male noresponse      other 
##         21         37          1          1

Discussion

Summary of Replication Attempt

The key analysis of interest was an individual-level Spearman correlation on disruption ratings and causal centrality (number of causal connections) for each feature. The authors found a positive correlation between disruption ratings and causal centrality, and indeed, we also found this result: rs = 0.26, 95% CI [0.18, 0.34], t(59) = 6.27, p < .001. We consider this a successful replication.

Commentary

In addition to the individual-level spearman correlation, we also ran two exploratory analyses. In an aggregate-level correlation, we found a positive correlation between mean disruption ratings and causal centrality (rs = 0.74, p = 0.001). In a linear-mixed effects-model, we found that causal centrality significantly predicts disruption ratings (est. = 1.87, t = 3.1). These exploratory analyses provide additional support for the authors’ claims. All in all, the results from this replication suggest that people do incorporate causal relationships between different features of identity into their self-concept.