Replication of Study 1 – Development of Holistic Episodic Recollection by Ngo et al. (2019, Psychological Science)

Published

December 2, 2023

Introduction

Ngo et al. (2019) sought to determine the extent to which different components of a memory are interconnected and whether cueing one part of a memory aids in the retrieval of another part of the same memory. In my research program, I am interested in understanding how infants learn and recognize statistical regularities in their environment. More specifically, I’m interested in how they leverage these regularities to form novel object representations. The concept of holistic memory retrieval from the study aligns closely with how we organize cognitive representations of the world. Their experimental approach emphasizes the identification of patterns, similarities, and associations, which is foundational to my work. By building on this foundation, I hope to predict and comprehend these elements in infants. My goal is to gain insights into how infants adeptly engage with their environment, especially in terms of how they retrieve memories, as highlighted in the study by Ngo et al. (2019).

I am planning to gather data from at least 31 young adults using Prolific. This is an attempt to not only match but exceed the sample size of the study (N = 31). The experiment will present participants with a variety of stimuli including images of familiar cartoon characters like Cinderella and Moana, cartoon illustrations of everyday objects such as a jump rope or a watch, and cartoon settings that can either be indoors like a bedroom or outdoors like a playground. Using this combination of stimuli, I will design 24 unique events. Each event will include one character, object, and scene, and will be shown to participants for a duration of six seconds - this is the encoding or memory-building phase. Right after this phase, participants will complete a memory task. They will be given a cue from one of the events (either a character, object, or scene) and given four options to choose from. Of these, one will be the correct association from the encoding phase, while the other three will be distractions. 

One of the challenges I foresee is maintaining timing accuracy. The study hinges on the precise timing of stimuli, and there’s the risk of potential lags or inconsistencies. As participants complete the experiment in real-time, it will also be important to have a reliable system in place to capture and record data without any losses. 

Project repository: https://github.com/psych251/ngo19_rescue

Original paper: https://github.com/psych251/ngo19_rescue/blob/main/original_paper/ngo-et-al-2019-development-of-holistic-episodic-recollection.pdf

Experimental Paradigm: http://173.236.204.40/st/ngo19rep/index.html?PROLIFIC_PID=progress3_demo&STUDY_ID=1&SESSION_ID=progress3

Summary of prior replication attempt

In comparing the methodologies between the original study and the 1st replication, I noticed 3 main differences. First, the original study had a larger participant pool of 31 individuals, whereas the replication was conducted with a notably smaller group of just 12 participants. This decision in the replication was influenced by effect size considerations: while the original study reported an effect size of 1.32, the replication aimed for an effect size of 0.8. This reduction in the targeted effect size may have implications for the study’s power and sensitivity. Additionally, the testing environments differed between the original study and the 1st replication. In the original, participants used a 13-inch laptop screen in a controlled setting. In the replication, participants did the experiment remotely on their own laptop or desktop. This means the replication couldn’t control for screen size, resolution, or potential distractions in participants’ environments.

Methods

Power Analysis

Original effect size was d = 1.32. Replication effect size was d = 0.15.

From G*Power Analysis for samples to achieve: 80%: N = 6 90%: N = 7 95% power: N = 8

Considerations of feasibility for selecting planned sample size. - sample size should be bigger than N = 8. Replication had N = 11 participants but only got effect size of d = 0.15. When I put this effect size into G*Power, I need a sample size of N = 277 to get 80% power. However, this is not feasible, budget-wise, so I will be using the Small Telescopes method (Simonsohn, 2014) and doing 2.5 times the original sample size.

Planned Sample

I planned to have a sample size of at least N = 31 but with considerations to the power analysis above and the small telescopes method. I will collect 77 participants, since that is 2.5 times the original sample size. In Prolific, I will set the following parameters: participants must be located in the USA, aged 18-31 years old, have English as their first language, and possess normal or corrected vision, with no issues of color blindness; additionally, I will ensure a balanced sample of males and females.

Materials

I will precisely follow the methods quoted directly from the original study in my experiment.

“We sampled 24 cartoon images of distinct scenes (12 indoor scenes, e.g., an aquarium; 12 outdoor scenes, e.g., a playground), 24 cartoon images of com- mon objects (e.g., a watch), and 24 images of cartoon characters from nonoverlapping movies or books (12 males, e.g., Pinocchio; 12 females, e.g., Alice) from the Google Images search engine. From this pool of selected images, we then constructed 24”events,” each consist- ing of a scene (e.g., an aquarium), a person (e.g., Alice), and an object (e.g., a wallet). The event assignment of the elements was randomized, with the exception that items with preexperimental associations (e.g., books and library) were not assigned to the same event. Every possible cue–test combination of each event was tested, resulting in six test trials per event (1 = cue: scene, test: person; 2 = cue: scene, test: object; 3 = cue: person, test: scene; 4 = cue: person, test: object; 5 = cue: object, test: scene; 6 = cue: object, test: person) and totaling 144 test trials.”

Procedure

In my experiment, I will adhere closely to the methods directly quoted from the original study. While the original study was conducted in-person on a 13-inch laptop screen—likely aiding in attention control and minimizing external distractions—I will ensure participants use only a laptop or desktop. Additionally, the experiment will automatically launch in fullscreen mode and notify me if a participant exits this mode.

“All participants were tested individually. The task procedure administered to children consisted of two encoding-test blocks, which occurred immediately after one another. Each block consisted of 12 encoding and 72 test trials, all presented on a 13-in. laptop screen. Prior to encoding, participants were told that they would see many different stories and that they should pay close attention to all of the different elements, including the scene, person, and object in each story. Then, participants viewed a series of events (12 s each; 0.5 s intertrial interval). A short audio-recorded narrative accompanied each event (e.g.,”Alice went to the aquarium, but she dropped her wallet there; the wallet was lost in the aquarium”; see Fig. 1a). Each narrative consisted of three sentences, with each sentence highlighting one pairwise association within the event. The order of the pairwise associations within each narrative was not fixed or counterbalanced across the events. The narrative was constructed this way to engage children in the task and to increase the likelihood that children would pay attention to all of the elements in an event. Prior to encoding, we provided one example (a playground, Elastigirl, a hat) in order to acquaint the participants with the encoding task.

Immediately after the encoding phase of each block, participants performed a self-paced four-alternative forced-choice task. We tested participants on every possible cue–retrieval combination of each studied event, resulting in 6 test trials per event, which totaled 72 test trials per block. On each trial, a cue and four options were presented simultaneously on the screen (see Fig. 2a). Among four options, one was a target—the correct item because it belonged to the same event as the cue. The three lures were same-category elements from different events. The lures always came from the events that contained same-sex characters, so that participants could not eliminate lures on the basis of general mnemonic heuristics (e.g., remembering that there was a female character who went to the aquarium). Across all 24 events, any two test trials that had overlapping cue items (e.g., AB1 and AC1) or in which tested items (e.g., BA1 and CA1) shared only one foil item (out of three) with respect to their event membership. For example, for the AB test trial of Event 1, the foils included the B elements from Events 2, 3, and 4, whereas for the AC trial of Event 1, the foils included the C elements from Events 3, 5, and 7 (one B and one C foil, both from Event 3). Furthermore, all items served as foils an equal number of times across all 144 test trials. Children were asked to point to one of the four options that belonged to the same story as the cue on the left side of the screen. Positions of the correct answer were counterbalanced across the entire test phase. There were no missing responses, as the response time was unrestricted. The memory task took approximately 40 min.

The adult task procedure was similar to the child task procedure but with a few differences. First, the whole procedure was administered in a single session comprising 24 encoding events and 144 test trials. Second, no narratives were implemented at the encoding phase to avoid potential ceiling performance in young adults. Third, each encoding trial was presented for 6 s (see Fig. 1b).”

Fig. 1.

Procedure of the child (a) and adult (b) multielement-event task. In the child task, participants viewed 24 events presented in two encoding sessions, each consisting of 12 events. Each event lasted 12 s and was accompanied by an audio-recorded narrative. The test phase of each block consisted of 72 test trials. In the adult task procedure, participants studied 24 events (6 s each) together and without the recorded narrative. The test phase consisted of 144 test trials. Note that the characters shown in each event were well-known cartoon characters (e.g., Alice, Pinocchio), which have been replaced in this illustration for copyright concerns.

Fig. 2.

A schematic depiction of the task design and the 2 × 2 contingency table used to estimate retrieval dependency. Examples of six retrieval types per event in the test phase are shown in (a). Each element of a studied event took a turn serving as the cue (item presented on the left side of the screen) and the tested element (one of the four options presented inside the red box). The schematic (b) shows how the proportion of joint retrieval for AB and AC pairs was computed for each participant. The contingency table shows the proportion of events that fell within each of the four categories: Both AB and AC pairs were retrieved correctly, both AB and AC pairs were retrieved incorrectly, AB was retrieved correctly and AC was retrieved incorrectly, and AB was retrieved incorrectly and AC was retrieved correctly. The proportion of events in the blue-outlined boxes (both pairs correct and both pairs incorrect) were added, and the sum was divided by the total number of events. Note that the characters shown in each event were well-known cartoon characters (e.g., Alice, Pinocchio), which have been replaced in this illustration for copyright concerns.

Controls

To confirm that participants are actively engaged and attentive throughout the experiment, I will introduce key press attention checks every 4 trials. For these checks, participants will be prompted with instructions like, “Please press the [chosen key] key to continue.” The chosen keys for these checks will be diverse, including “j”, “k”, “f”, “d”, “s”, and “a”, to prevent any rote memorization or autopilot responses. These attention checks will help filter out participants who might not be genuinely engaged in the task, ensuring that our data reflects genuine cognitive responses.

To further validate the task’s functionality and participants’ comprehension of the instructions, I am considering the incorporation of a positive control. This would consist of a set of trials where, based on the contextual similarity of the stimuli, we anticipate almost all participants to answer correctly. These contextually congruent trials are designed on the hypothesis that when stimuli are more related or familiar in context, participants are more likely to respond accurately. However, this is based on my own speculation, so I’m not sure if I should add this.

Analysis Plan

I will conduct the following analysis precisely as directly quoted below with the young adult group. Although this quote mentions conducting a one-sample t test for 4 to 6 year olds, the original authors did the same analysis for their young adult pool as well.

“The primary questions of this research were whether holistic recollection is evident at the ages of 4 and 6 years, during a crucial developmental window of robust gains in episodic memory, and whether holistic recollection changes with age. To answer the first question, we conducted a one-sample t test to determine whether dependency (data – independent model) exceeded zero for each age group.”

To clarify the key test I will be running in my own words, I will conduct a one-sample t-test to determine if the mean dependency (difference between data and independent model) in my study significantly exceeds zero, which would indicate holistic recollection. I will be testing the ‘Dependency’ variable computed from the proportions of joint retrieval for different item pairs.

I also suspect that individual trials with contextually similar stimuli lead to better performance than trials with non-contextually similar stimuli so I plan to run a follow-up paired-samples t-test. I will label which trials are contextually similar and which are not, compute the performance measure (e.g., accuracy, response time) for each trial type (contextually similar vs. non-contextually similar), and use a paired-samples t-test to compare the mean performance across the two types of trials.

Differences from Original Study and 1st replication

The original study was conducted with 31 participants in a controlled, in-person setting using a 13-inch laptop. In contrast, the first replication used a smaller sample size of 12 participants and was administered remotely via Prolific, allowing participants to use any size of laptop or desktop. Both studies followed the same analysis plan.

For the second replication, I aim to collect data from 77 participants, mirroring the original study’s sample size. This step is to determine if the discrepancies in the first replication arose due to a power issue. If, even with the larger sample, the results still fail to align with the original study, I’ll use a paired samples t-test, which was not used in either the original or first replication. The goal is to probe if the variance emerged from the contextual similarity of the stimuli pairs. If a significant association is identified between contextual stimuli and accuracy, I plan to rerun the study, excluding those particular contextual pairs.

Methods Addendum (Post Data Collection)

You can comment this section out prior to final report with data collection.

Actual Sample

Sample size, demographics, data exclusions based on rules spelled out in analysis plan

Differences from pre-data collection methods plan

Any differences from what was described as the original plan, or “none”.

Results

Data preparation

## Load Relevant Libraries and Functions
library(tidyverse)
── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ──
✔ dplyr     1.1.3     ✔ readr     2.1.4
✔ forcats   1.0.0     ✔ stringr   1.5.0
✔ ggplot2   3.4.3     ✔ tibble    3.2.1
✔ lubridate 1.9.2     ✔ tidyr     1.3.0
✔ purrr     1.0.2     
── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
✖ dplyr::filter() masks stats::filter()
✖ dplyr::lag()    masks stats::lag()
ℹ Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errors
library(BayesFactor)
Loading required package: coda
Loading required package: Matrix

Attaching package: 'Matrix'

The following objects are masked from 'package:tidyr':

    expand, pack, unpack

************
Welcome to BayesFactor 0.9.12-4.5. If you have questions, please contact Richard Morey (richarddmorey@gmail.com).

Type BFManual() to open the manual.
************
## Import data
### get all data files from directory
data_files <- list.files(file.path("~/Desktop/ngo19_rescue/exp/data"), pattern = "*.csv", full.names = TRUE)
data_files
 [1] "/Users/stanfordvpnl/Desktop/ngo19_rescue/exp/data/5ac54531e1546900019c0487_656687ca8ed41959d6031277_65669795d07b24c4dd19d61a_data_1701223108201.csv"
 [2] "/Users/stanfordvpnl/Desktop/ngo19_rescue/exp/data/5b0b89c31e55760001b958e9_656687ca8ed41959d6031277_65668fbc04d8282c116389f7_data_1701220915900.csv"
 [3] "/Users/stanfordvpnl/Desktop/ngo19_rescue/exp/data/5ee00907ed486334f7063e3a_656687ca8ed41959d6031277_656697d357acb6ba1c87d064_data_1701223613874.csv"
 [4] "/Users/stanfordvpnl/Desktop/ngo19_rescue/exp/data/60469cfa17b0fd0dca2c2286_656687ca8ed41959d6031277_65669b821f70541c740d81b6_data_1701224121047.csv"
 [5] "/Users/stanfordvpnl/Desktop/ngo19_rescue/exp/data/605fbc12c61d8d10250f6e51_656687ca8ed41959d6031277_65669b8e7388e6de56a8658c_data_1701223956296.csv"
 [6] "/Users/stanfordvpnl/Desktop/ngo19_rescue/exp/data/60ff21f137a14dc7dce1eeba_656687ca8ed41959d6031277_65668ff23b2b8bb220f8f8db_data_1701220942117.csv"
 [7] "/Users/stanfordvpnl/Desktop/ngo19_rescue/exp/data/60ff83da6b1bee23abb427a4_656687ca8ed41959d6031277_6566959f35fd00183bd58d59_data_1701222925677.csv"
 [8] "/Users/stanfordvpnl/Desktop/ngo19_rescue/exp/data/6147d7528a5ce61af91eedb2_656687ca8ed41959d6031277_6566987bfc6f02e8719da879_data_1701223531354.csv"
 [9] "/Users/stanfordvpnl/Desktop/ngo19_rescue/exp/data/62bf498225d76690dec43d8f_656687ca8ed41959d6031277_65668fc887073b8018931426_data_1701221138545.csv"
[10] "/Users/stanfordvpnl/Desktop/ngo19_rescue/exp/data/63062501b80b139870ea2af4_656687ca8ed41959d6031277_656699f535fd00183bd58e22_data_1701224080480.csv"
### read individual data files to list
all_data <- lapply(data_files, read.csv)

### unpack list as singular tidy data frame
all_data <- all_data |>
  map_df(as_tibble)

if(!221 %in% all_data$trial_index) {
  print("Trial index 221 is missing after import.")
} else {
  print("Trial index 221 is present after import.")
}
[1] "Trial index 221 is present after import."
### unpack participant demographics
demographics_trials <- all_data |>
  subset(trial_type == "survey-html-form")

### string together json strings
json_text <- paste0(demographics_trials$response, collapse = ',')
json_text <- paste0('[', json_text, ']')

### unpack json as data frame
demographics_trials <- jsonlite::fromJSON(json_text) |>
  as_tibble() |>
  mutate(subject_id = rownames(demographics_trials), .before = "Gender")

### get participant demographic stats
#### age
mean_age <- mean(as.numeric(demographics_trials$Age))
sd_age <- sd(as.numeric(demographics_trials$Age))
min_age <- min(as.numeric(demographics_trials$Age))
max_age <- max(as.numeric(demographics_trials$Age))

#### gender
table(demographics_trials$Gender)

Female   Male  Other 
     4      5      1 
## Data exclusion / filtering
### clean data into long format

all_data_cleaned <- all_data |>
  group_by(subject_id) |> 
  #filter(row_number() > 78 & row_number() <= 221) |> 
  #slice(78:221) |>
  filter(trial_index >= 78 & trial_index <= 221) |>
  ungroup() |>
  filter(!is.na(correct))

if(!221 %in% all_data_cleaned$trial_index) {
  print("Trial index 221 is missing")
} else {
  print("Trial index 221 is present")
}
[1] "Trial index 221 is present"
print("Data after filtering:")
[1] "Data after filtering:"
print(head(all_data_cleaned))
# A tibble: 6 × 21
  success timeout failed_images failed_audio failed_video trial_type trial_index
  <chr>   <chr>   <chr>         <chr>        <chr>        <chr>            <int>
1 ""      ""      ""            ""           ""           image-key…          78
2 ""      ""      ""            ""           ""           image-key…          79
3 ""      ""      ""            ""           ""           image-key…          80
4 ""      ""      ""            ""           ""           image-key…          81
5 ""      ""      ""            ""           ""           image-key…          82
6 ""      ""      ""            ""           ""           image-key…          83
# ℹ 14 more variables: time_elapsed <int>, internal_node_id <chr>,
#   subject_id <chr>, study_id <chr>, session_id <chr>, view_history <chr>,
#   rt <chr>, stimulus <chr>, response <chr>, task <chr>,
#   correct_response <chr>, attention_duration <int>, correct <chr>,
#   retrieval_group <chr>
# Group the cleaned data by subject_id, then count the number of rows for each group
participant_row_counts <- all_data_cleaned |>
  group_by(subject_id) |>
  summarise(num_rows = n())


# Print the row counts to see the number of rows for each participant
print(participant_row_counts)
# A tibble: 10 × 2
   subject_id               num_rows
   <chr>                       <int>
 1 5ac54531e1546900019c0487      144
 2 5b0b89c31e55760001b958e9      144
 3 5ee00907ed486334f7063e3a      144
 4 60469cfa17b0fd0dca2c2286      144
 5 605fbc12c61d8d10250f6e51      144
 6 60ff21f137a14dc7dce1eeba      144
 7 60ff83da6b1bee23abb427a4      144
 8 6147d7528a5ce61af91eedb2      144
 9 62bf498225d76690dec43d8f      144
10 63062501b80b139870ea2af4      144
all_data_cleaned <- all_data_cleaned |>
  mutate(accuracy = ifelse(correct == "true", 1, 0)) |>
  select(subject_id, trial_index, time_elapsed, rt, stimulus, task,
         retrieval_group, response, correct_response, correct, accuracy) |>
  separate(stimulus, c(NA, NA, NA, NA, "id"), sep = "_", remove = FALSE, extra = "merge", fill = "right")

if(!221 %in% all_data_cleaned$trial_index) {
  print("Trial index 221 is missing")
} else {
  print("Trial index 221 is present")
}
[1] "Trial index 221 is present"
# Calculate accuracy summaries
accuracy_summary <- all_data_cleaned |>
  filter(task == "response") |> 
  group_by(retrieval_group) |>
  summarise(mean = mean(accuracy), sd = sd(accuracy), n = n(), sem = sd/sqrt(n))

print(accuracy_summary)
# A tibble: 6 × 5
  retrieval_group  mean    sd     n    sem
  <chr>           <dbl> <dbl> <int>  <dbl>
1 Ab              0.775 0.418   240 0.0270
2 Ac              0.671 0.471   240 0.0304
3 Ba              0.779 0.416   240 0.0268
4 Bc              0.717 0.452   240 0.0291
5 Ca              0.671 0.471   240 0.0304
6 Cb              0.721 0.450   240 0.0290
## Prepare data for analysis - create columns 
### compute individual accuracy by grouping
accuracy_summary <- all_data_cleaned |>
  filter(task == "response") |> # only use retrieval data
  group_by(retrieval_group) |>
  summarise(mean = mean(accuracy), sd = sd(accuracy), n = n(), sem = sd/sqrt(n))
### compute accuracy by participant
accuracy_summary_part <- all_data_cleaned |>
  filter(task == "response") |> # only use retrieval data
  group_by(subject_id) |>
  summarise(mean = mean(accuracy), sd = sd(accuracy), n = n(), sem = sd/sqrt(n))

### compute overall accuracy
overall_accuracy_summary <- all_data_cleaned |>
  filter(task == "response") |> # only use retrieval data
  summarise(mean = mean(accuracy), sd = sd(accuracy), n = n(), sem = sd/sqrt(n))


### compute Ab_Ac_Accuracy
Ab_Ac_Accuracy <- all_data_cleaned |>
  filter(task == "response") |> # only use retrieval data
  filter(retrieval_group == "Ab" | retrieval_group == "Ac") |>
  group_by(subject_id) |>
  summarise(mean = mean(accuracy))


### compute Ba_Bc_Accuracy
Ba_Bc_Accuracy <- all_data_cleaned |>
  filter(task == "response") |> # only use retrieval data
  filter(retrieval_group == "Ba" | retrieval_group == "Bc") |>
  group_by(subject_id) |>
  summarise(mean = mean(accuracy))

### compute Ca_Cb_Accuracy
Ca_Cb_Accuracy <- all_data_cleaned |>
  filter(task == "response") |> # only use retrieval data
  filter(retrieval_group == "Ca" | retrieval_group == "Cb") |>
  group_by(subject_id) |>
  summarise(mean = mean(accuracy))

### compute Ba_Ca_Accuracy
Ba_Ca_Accuracy <- all_data_cleaned |>
  filter(task == "response") |> # only use retrieval data
  filter(retrieval_group == "Ba" | retrieval_group == "Ca") |>
  group_by(subject_id) |>
  summarise(mean = mean(accuracy))

### compute Ac_Bc_Accuracy
Ac_Bc_Accuracy <- all_data_cleaned |>
  filter(task == "response") |> # only use retrieval data
  filter(retrieval_group == "Ac" | retrieval_group == "Bc") |>
  group_by(subject_id) |>
  summarise(mean = mean(accuracy))

### compute Ab_Cb_Accuracy
Ab_Cb_Accuracy <- all_data_cleaned |>
  filter(task == "response") |> # only use retrieval data
  filter(retrieval_group == "Ab" | retrieval_group == "Cb") |>
  group_by(subject_id) |>
  summarise(mean = mean(accuracy))

### compute 6 2x2 contingency tables for retrieval dependency key analysis

#### define dependent model function
compute_data_model <- function(group_one, group_two, num_events = 24) {
  test_set <- all_data_cleaned |>
    filter(task == "response") |> # only use retrieval data
    select(subject_id, id, retrieval_group, accuracy) |> # select only the necessary columns
    filter(retrieval_group == group_one | retrieval_group == group_two) |> # filter out relevant groups from function params
    pivot_wider(id_cols = c(subject_id, id), names_from = retrieval_group, values_from = accuracy) |> # make long data frame wide for data dependent calculations
    rowwise() |>
    mutate(sum = sum(eval(parse(text = group_one)) + eval(parse(text = group_two)))) # get sums to compute all_correct and all_incorrect proportion of contingency table
  

  # get unique participants
  ps <- unique(all_data_cleaned$subject_id)
  data_models <- rep(NA, length(ps))
  
  for(ii in 1:length(ps)) {
    # get data for just the one participant in the loop
    data_subset <- test_set |>
      filter(subject_id == ps[ii])
    
    # get proportion of all correct
    prop_all_correct <- sum(data_subset$sum == 2) / num_events
    
    # get proportion of all incorrect
    prop_all_incorrect <- sum(data_subset$sum == 0) / num_events
    
    # compute data model and store per participant
    data_model_calc <- prop_all_correct + prop_all_incorrect
    data_models[ii] <- data_model_calc 
  }
  
  return(data_models)
}

#### 1) compute Data_Ab_Ac
Data_Ab_Ac <- compute_data_model("Ab", "Ac")

#### 2) compute Data_Ba_Bc
Data_Ba_Bc <- compute_data_model("Ba", "Bc")

#### 3) compute Data_Ca_Cb
Data_Ca_Cb <- compute_data_model("Ca", "Cb")

#### 4) compute Data_Ba_Ca
Data_Ba_Ca <- compute_data_model("Ba", "Ca")

#### 5) compute Data_Ac_Bc
Data_Ac_Bc <- compute_data_model("Ac", "Bc")

#### 6) compute Data_Ab_Cb
Data_Ab_Cb <- compute_data_model("Ab", "Cb")

#### define independent model function
compute_ind_model <- function(group_one, group_two) {
  P_AB <- all_data_cleaned |>
    filter(retrieval_group == group_one) |>
    group_by(subject_id) |>
    summarise(mean = mean(accuracy))

  P_AC <- all_data_cleaned |>
    filter(retrieval_group == group_two) |>
    group_by(subject_id) |>
    summarise(mean = mean(accuracy))

  cor_cor <- P_AB$mean * P_AC$mean
  incor_cor <- P_AC$mean * (1 - P_AB$mean)
  cor_incor <- P_AB$mean * (1 - P_AC$mean)
  incor_incor <- (1 - P_AB$mean) * (1 - P_AC$mean)

  return(cor_cor + incor_incor)
}

#### 1) compute Independent_Model_Ab_Ac
Independent_Model_Ab_Ac <- compute_ind_model("Ab", "Ac")

#### 2) compute Independent_Model_Ba_Bc
Independent_Model_Ba_Bc <- compute_ind_model("Ba", "Bc")

#### 3) compute Independent_Model_Ca_Cb
Independent_Model_Ca_Cb <- compute_ind_model("Ca", "Cb")

#### 4) compute Independent_Model_Ba_Ca
Independent_Model_Ba_Ca <- compute_ind_model("Ba", "Ca")

#### 5) compute Independent_Model_Ac_Bc
Independent_Model_Ac_Bc <- compute_ind_model("Ac", "Bc")

#### 6) compute Independent_Model_Ab_Cb
Independent_Model_Ab_Cb <- compute_ind_model("Ab", "Cb")

#### 1) compute Dependency_AbAc
Dependency_AbAc <- Data_Ab_Ac - Independent_Model_Ab_Ac

#### 2) compute Dependency_BaBc
Dependency_BaBc <- Data_Ba_Bc - Independent_Model_Ba_Bc
  
#### 3) compute Dependency_CaCb
Dependency_CaCb <- Data_Ca_Cb - Independent_Model_Ca_Cb
  
#### 4) compute Dependency_BaCa
Dependency_BaCa <- Data_Ba_Ca - Independent_Model_Ba_Ca

#### 5) compute Dependency_AcBc
Dependency_AcBc <- Data_Ac_Bc - Independent_Model_Ac_Bc
  
#### 6) compute Dependency_AbCb
Dependency_AbCb <- Data_Ab_Cb - Independent_Model_Ab_Cb

### compute Collapsed_Data
total_data_model <- (Data_Ab_Ac + Data_Ba_Bc + Data_Ca_Cb + Data_Ba_Ca + Data_Ac_Bc + Data_Ab_Cb) / 6

### compute Collapsed_Ind_Model
total_ind_model <- (Independent_Model_Ab_Ac + Independent_Model_Ba_Bc + Independent_Model_Ca_Cb + Independent_Model_Ba_Ca + Independent_Model_Ac_Bc + Independent_Model_Ab_Cb) / 6

### compute Dependency
Dependency <- total_data_model - total_ind_model
Dependency
 [1]  0.020254630  0.131944444  0.028935185  0.081018519  0.038773148
 [6]  0.055555556 -0.003472222  0.151620370  0.101851852  0.135416667
Dependency_avg <- mean(Dependency)
Dependency_avg
[1] 0.07418981
## construct complete data frame with all relevant stats summarized for each participant
summary_data <- data.frame(subject_id = accuracy_summary_part$subject_id,
                           Accuracy_mean = accuracy_summary_part$mean,
                           Accuracy_sd = accuracy_summary_part$sd,
                           Accuracy_sem = accuracy_summary_part$sem,
                           Accuracy_n = accuracy_summary_part$n,
                           Accuracy_AbAc = Ab_Ac_Accuracy$mean,
                           Accuracy_BaBc = Ba_Bc_Accuracy$mean,
                           Accuracy_CaCb = Ca_Cb_Accuracy$mean,
                           Accuracy_BaCa = Ba_Ca_Accuracy$mean,
                           Accuracy_AcBc = Ac_Bc_Accuracy$mean,
                           Accuracy_AbCb = Ab_Cb_Accuracy$mean,
                           Data_AbAc = Data_Ab_Ac,
                           Data_BaBc = Data_Ba_Bc,
                           Data_CaCb = Data_Ca_Cb,
                           Data_BaCa = Data_Ba_Ca,
                           Data_AcBc = Data_Ac_Bc,
                           Data_AbCb = Data_Ab_Cb,
                           Independent_AbAc = Independent_Model_Ab_Ac,
                           Independent_BaBc = Independent_Model_Ba_Bc,
                           Independent_CaCb = Independent_Model_Ca_Cb,
                           Independent_BaCa = Independent_Model_Ba_Ca,
                           Independent_AcBc = Independent_Model_Ac_Bc,
                           Independent_AbCb = Independent_Model_Ab_Cb,
                           Dependency_AbAc = Dependency_AbAc,
                           Dependency_BaBc = Dependency_BaBc,
                           Dependency_CaCb = Dependency_CaCb,
                           Dependency_BaCa = Dependency_BaCa,
                           Dependency_AcBc = Dependency_AcBc,
                           Dependency_AbCb = Dependency_AbCb,
                           Data_model_overall = total_data_model,
                           Independent_model_overall = total_ind_model,
                           Dependency_overall = Dependency)

DT::datatable(summary_data)
## count number of participants to exclude for having overall task accuracy == 100%
perfect_acc_exclusions_idx <- which(summary_data$Accuracy_mean == 1)
if(length(perfect_acc_exclusions_idx) > 0) {
  summary_data_w_exclusions <- summary_data[perfect_acc_exclusions_idx,]
} else {
  print("No exclusions based on accuracy made!")
}
[1] "No exclusions based on accuracy made!"
num_acc_exclusions <- length(perfect_acc_exclusions_idx)
num_acc_exclusions
[1] 0
print("Final dataset check before exclusions:")
[1] "Final dataset check before exclusions:"
print(head(all_data_cleaned))
# A tibble: 6 × 12
  subject_id trial_index time_elapsed rt    stimulus id    task  retrieval_group
  <chr>            <int>        <int> <chr> <chr>    <chr> <chr> <chr>          
1 5ac54531e…          78       359568 4337  img/tes… 24.j… resp… Ca             
2 5ac54531e…          79       362661 3077  img/tes… 12.j… resp… Ca             
3 5ac54531e…          80       366204 3509  img/tes… 18.j… resp… Ca             
4 5ac54531e…          81       369677 3431  img/tes… 22.j… resp… Ca             
5 5ac54531e…          82       373772 4058  img/tes… 13.j… resp… Ca             
6 5ac54531e…          83       377554 3750  img/tes… 2.jpg resp… Ca             
# ℹ 4 more variables: response <chr>, correct_response <chr>, correct <chr>,
#   accuracy <dbl>

Confirmatory Analysis

if(any(is.na(Dependency)) | any(is.nan(Dependency))) {
  print("Dependency contains NA or NaN values, which are not valid for t-test.")
} else {
### testing key confirmatory analysis
key_test <- t.test(Dependency, alternative = "greater", mu = 0)
print(key_test)

### get bayes factor for key confirmatory analysis t-test
key_test_bf <- ttestBF(Dependency, mu = 0, nullInterval = c(0, Inf))
print(key_test_bf)
print(1/key_test_bf) ## BF in support of the null
}

    One Sample t-test

data:  Dependency
t = 4.3251, df = 9, p-value = 0.0009594
alternative hypothesis: true mean is greater than 0
95 percent confidence interval:
 0.04274566        Inf
sample estimates:
 mean of x 
0.07418981 

Bayes factor analysis
--------------
[1] Alt., r=0.707 0<d<Inf    : 47.48033   ±0%
[2] Alt., r=0.707 !(0<d<Inf) : 0.09924256 ±0%

Against denominator:
  Null, mu = 0 
---
Bayes factor type: BFoneSample, JZS

            denominator
numerator    Alt., r=0.707 0<d<Inf Alt., r=0.707 !(0<d<Inf)
  Null, mu=0            0.02106135                 10.07632

Overall Accuracy

## compute 95% CI of acc mean assuming normal distribution
acc_mean <- mean(summary_data$Accuracy_mean)
acc_sd <- sd(summary_data$Accuracy_mean)
n <- 12
error <- qnorm(0.975)*acc_sd/sqrt(n)
lower_ci <- acc_mean - error
upper_ci <- acc_mean + error

Proportion of Joint Retrieval

Original vs. Replication Attempt

### Overall Accuracy
#### make plot
acc_plot <- ggplot(summary_data, aes(x = "", y = Accuracy_mean)) +
  geom_boxplot(width = 0.15, fill = "#ad93ad", outlier.shape = NA) +
  geom_jitter(width = 0, col = "#5c475b", fill = "#FFFFFF", alpha = .7, shape = 21, size = 1, stroke = 1) +
  xlab(" ") +
  ylab("Overall Accuracy") +
  scale_y_continuous(breaks = seq(0.25, 1.00, 0.25), limits = c(0.25, 1.00)) +
  theme_classic() +
  theme(panel.grid = element_blank(),
        panel.border = element_blank(),
        legend.title = element_blank(),
        legend.text = element_text(color = "grey20", size = 12),
        axis.text.x = element_text(color = "grey20", size = 16),
        axis.text.y = element_text(color = "grey20", size = 16),  
        axis.title.x = element_text(color = "grey20", size = 16, face = "bold"),
        axis.title.y = element_text(color = "grey20", size = 16, vjust = 0.5, face = "bold"))


### Proportion of Joint Retrieval
#### get relevant data in long format
summary_data_joint_ret_long <- pivot_longer(summary_data,
                                            c(Data_model_overall, Independent_model_overall),
                                            names_to = "source",
                                            values_to = "prop_joint_retrieval") |>
  rowwise() |>
  mutate(source_cleaned = ifelse(source == "Data_model_overall", "Data", "Ind. Model")) |>
  select(subject_id, source, source_cleaned, prop_joint_retrieval)

#### make plot
ret_plot <- ggplot(summary_data_joint_ret_long, aes(x = source_cleaned, y = prop_joint_retrieval, fill = source_cleaned)) +
  geom_boxplot(width = 0.15, outlier.shape = NA, position = position_dodge(1)) +
  geom_jitter(aes(col = source_cleaned, x = source_cleaned), position = position_jitter(width = 0), alpha = .7, shape = 21, size = 1, stroke = 1) +
  xlab("Young Adults") +
  ylab("Proportion of\nJoint Retrieval") +
  scale_y_continuous(breaks = seq(0.40, 1.00, 0.1), limits = c(0.40, 1.00)) +
  scale_fill_manual(values = c("#ad93ad", "#dbb5d6")) +
  scale_color_manual(values = c("#5c475b", "#573350")) +
  theme_classic() +
  theme(panel.grid = element_blank(),
        panel.border = element_blank(),
        legend.title = element_blank(),
        legend.text = element_text(color = "grey20", size = 10),
        legend.position = "none",
        axis.text.x = element_text(color = "grey20", size = 13),
        axis.text.y = element_text(color = "grey20", size = 16),  
        axis.title.x = element_text(color = "grey20", size = 22, vjust = 0, face = "bold"),
        axis.title.y = element_text(color = "grey20", size = 16, vjust = 0.5, face = "bold"))


### Dependency
#### make plot
dep_plot <- ggplot(summary_data, aes(x = "", y = Dependency_overall)) +
  geom_boxplot(width = 0.15, fill = "#ad93ad", outlier.shape = NA) +
  geom_jitter(width = 0, col = "#5c475b", fill = "#FFFFFF", alpha = .7, shape = 21, size = 1, stroke = 1) +
  geom_hline(yintercept = 0.0, col = "red", lty = "dashed") +
  xlab(" ") +
  ylab("Dependency") +
  scale_y_continuous(breaks = seq(-0.50, 0.50, 0.1), limits = c(-0.50, 0.50)) +
  theme_classic() +
  theme(panel.grid = element_blank(),
        panel.border = element_blank(),
        legend.title = element_blank(),
        legend.text = element_text(color = "grey20", size = 12),
        axis.text.x = element_text(color = "grey20", size = 16),
        axis.text.y = element_text(color = "grey20", size = 16),  
        axis.title.x = element_text(color = "grey20", size = 16, face = "bold"),
        axis.title.y = element_text(color = "grey20", size = 16, vjust = 0.5, face = "bold"))
### Combine Plots for Rendered Report
ggpubr::ggarrange(acc_plot, ret_plot, dep_plot, ncol = 3, nrow = 1, widths = c(1,2,1),
                  labels = c('(a)', '(b)', '(c)'))

Fig. 6: Attempted replication of key data from the current study. (a) Jittered distribution of overall accuracy on the cued-recognition memory task. (b) Jittered distribution of the proportion of joint retrieval for the actual data and the independent statistical model (Ind. Model). (c) Jittered distribution of retrieval dependency scores. The dashed red line on the figure indicates the test value of 0 used in the t-test (i.e., retrieval dependency scores significantly greater than 0).

Calculate Cohen’s d

round(effectsize::cohens_d(key_test)$Cohens_d, 2)
[1] 1.37

Results of control measures

Three-panel graph with original, 1st replication, and your replication is ideal here

Discussion

Mini meta analysis

Combining across the original paper, 1st replication, and 2nd replication, what is the aggregate effect size?

Summary of Replication Attempt

Open the discussion section with a paragraph summarizing the primary result from the confirmatory analysis and the assessment of whether it replicated, partially replicated, or failed to replicate the original result.

Commentary

Add open-ended commentary (if any) reflecting (a) insights from follow-up exploratory analysis, (b) assessment of the meaning of the replication (or not) - e.g., for a failure to replicate, are the differences between original and present study ones that definitely, plausibly, or are unlikely to have been moderators of the result, and (c) discussion of any objections or challenges raised by the current and original authors about the replication attempt. None of these need to be long.