Introduction

I will replicate Chopra et al., The First Crank of the Cultural Ratchet. This is an important study on how language enables the kind of teaching that leads to cultural learning over generations. The study involves pairs of participants. One participant, the “teacher,” is asked to learn a category and teach it to someone else. The second participant is the “learner” and is asked to learn the category from the teacher’s explanation. Both participants take a test of their understanding and the difference between student and teacher accuracy is analyzed.

Project repository (on Github): https://github.com/psych251/chopra2019

Original paper (as hosted in your repo): https://github.com/psych251/chopra2019/blob/master/original_paper/0060.pdf

Methods

Power Analysis

I will fill this in after we cover power analysis in class

Planned Sample

I plan to run 224 participants as this was the sample size of the original study. (Noah can contribute funding for this because it is part of my research agenda)

Materials

“Participants learned concepts generated by 5 different rules (i.e. logical forms): Single Feature, Conjunction, Disjunction, Conjunctive Disjunction, and Disjunctive Conjunction. Rules were realized in specific concepts by varying Booleanproperties of programatically generated images of creatures, from five different kinds: flowers, bugs, birds, fish, and trees (see Figure 1 for an example). Each kind had 5 to 7 Boolean features that we used to realize our concepts. Each of the 5 rules was realized twice in each creature kind, yielding a total of 50 concepts (listed on the axis of Figure 3). For each concept, we generated 100 specific creatures, split into 50 for training and 50 for testing. We ensured some positive examples of the concept even for very restrictive rules by first randomly selecting 6 positive instances of the concept and then adding 44 items chosen at random from all remaining items (i.e., according to the true concept base rate).”

The materials in the replication will precisely follow these materials.

Procedure

“Every pair of participants was placed in a game, where one was assigned the role of the “teacher” (initial learner) and the other was assigned the role of the “student” (secondary learner). Each game consisted of 5 rounds, each with a new concept from a new rule. Each of a game’s 5 concepts used a different creature kind, and each concept was presented with a different nonce word as the species name. The ordering of concepts was randomized so that there was no standard ordering of rule types across the games. On each round, participants went through three phases: concept learning, concept communicating, and concept testing (Figure 1). During the concept learning phase, the teacher was presented a grid of training creatures and was instructed to click on individual creatures to reveal whether or not they belonged to the species defined by the concept. Once the teacher clicked on every creature in the grid, they were presented a message advising them to review the creatures for as long as they needed. When the teacher ended the concept learning phase, they proceeded to the concept communicating phase, where they entered an online chatroom and were were instructed to teach the concept to the student. Participants were provided no additional instructions for the chatroom, and they were allowed to talk freely. In order to prevent a teacher from rushing through the chatroom without properly communicating with their student, only the student was given the ability proceed to the final concept testing phase. In the final phase both participants were (separately) given the same grid of test creatures and asked to tag the creatures that they believed belonged to the species. Neither participant had access to their chatroom messages during this phase. Once both participants completed concept testing for a concept, they were provided feedback in the form of their own and their partner’s score, computed as: # of hits − # of false alarms. We encouraged them to learn concepts thoroughly and communicate effectively with a monetary bonus equal to the sum of both players’ scores (in cents). Participants were made aware of the task structure and bonus mechanic prior to starting the first round; they had to answer 5 comprehension questions correctly to begin to the game.”

I will precisely follow the same procedure in my replication.

Analysis Plan

In the original paper, “[p]articipants were restricted to those with U.S. IP addresses and at least a 95% work approval rating; in addition, participants who self-reported a native language other than English or failed to partake in the experiment (accepted the hit but then discussed matters entirely unrelated to the experiment) were excluded.”

I will apply the same exclusion criteria to participant data as is described above, though since I am using Prolific instead of MTurk I will not be able to filter based on approval rating.

After applying the exclusion criteria, I will compute three metrics of interest for each trial: the teacher’s accuracy, the student’s accuracy, and the Hamming distance between teacher and student answers. Hamming distance is simply the number of choices that were different between student and teacher.

The key analysis of interest for this project is measuring the Pearson correlation between a teacher’s concept accuracy and the Hamming distance between the teacher and student’s predictions. The results from the original paper found a strong negative correlation, indicating that teachers who understand the concept well are also good at accurately conveying their knowledge of the concept to students.

I also plan on replicating additional analyses from the original paper. I will analyze the Pearson correlation between student and teacher accuracies, both at the individual level and across the 50 concepts.

The original authors also “built a Bayesian mixed-effects model predicting whether or not a participant responded accurately during the concept testing phase as a function of the rule, the participant’s role (teacher vs. student), and their interaction. [They] included random intercepts and effect of rule for participants and random intercepts and effect of role for each of the 50 concepts.”

I will build the same model and report the effect of role, as was done in the original paper.

Differences from Original Study

While the original paper collected data on Amazon Mechanical Turk, I will use Prolific due to recent data quality problems on MTurk and the fact that Amazon is evil. Otherwise, I will follow the same methods for running the experiment and analyzing the data. I do not expect this to make a difference to the findings since the task is fundamentally the same and the original effects reported were very strong.

Methods Addendum (Post Data Collection)

You can comment this section out prior to final report with data collection.

Actual Sample

Sample size, demographics, data exclusions based on rules spelled out in analysis plan

Differences from pre-data collection methods plan

Any differences from what was described as the original plan, or “none”.

Results

Note: I do not have a link to my paradigm here as I have not gotten it to work online yet. I can run my experiment locally and everything works fine, but it has trouble pulling data and pushing it to the database when running online. I am currently debugging it and hope to get it working before my midterm presentation on Wednesday.

Data preparation

My data will be in JSON format as I am saving participant actions to a MongoDB database. I will first write a Python script that converts this data to a CSV file which R can parse more easily.

First, I will exclude data from participants whose I.P. addresses were from outside the US, self-reported a first language other than English, or did not make any statements in the chat that were relevant to the task at hand.

I will then use the data to compute the three dependent variables of interest: student accuracy, teacher accuracy, and student-teacher Hamming distance. Accuracies are defined as the proportion of test cases correctly categorized and student-teacher Hamming distance is defined as the number of examples that the student and teacher categorized differently. These variables will be labelled by concept to enable me to aggregate accuracies for each concept and rule type.

Here is the link to my python script that preprocesses the pilot data and here is the link to the raw data.

###Data Preparation

# I did this in Python because it handles JSON better.

##Load Relevant Libraries and Functions
library(tidyverse)
## -- Attaching packages --------------------------------------- tidyverse 1.3.1 --
## v ggplot2 3.3.5     v purrr   0.3.4
## v tibble  3.1.5     v dplyr   1.0.7
## v tidyr   1.1.4     v stringr 1.4.0
## v readr   2.0.2     v forcats 0.5.1
## -- Conflicts ------------------------------------------ tidyverse_conflicts() --
## x dplyr::filter() masks stats::filter()
## x dplyr::lag()    masks stats::lag()
####Import data
df_scores <- read.csv("../replication_analysis/processed_data/scores.csv")
df_tests <- read.csv("../replication_analysis/processed_data/tests.csv")

#### Data exclusion / filtering

# I have yet to implement this. I will do it before running pilot B.

#### Prepare data for analysis - create columns etc.

# split explorer and student test data
explorer_tests <- df_tests |>
  filter(role == "explorer") |>
  select(game_id, role, round_num, predicted_label, stim_num, true_label)
colnames(explorer_tests)[4] = "explorer_pred_label"

student_tests <- df_tests |>
  filter(role == "student") |>
  select(game_id, role, round_num, predicted_label, stim_num)
colnames(student_tests)[4] = "student_pred_label"

# combine student and teaching test data
combined_tests <- explorer_tests |>
  inner_join(student_tests, by=c("game_id", "round_num", "stim_num")) |>
  rowwise() |>
  mutate(disagree = if(student_pred_label == explorer_pred_label) 0 else 1,
         exp_correct = if(explorer_pred_label == true_label) 1 else 0)
# compute hamming distance
hamming_distances <- combined_tests |>
  group_by(game_id, round_num) |>
  summarize(explorer_student = sum(disagree),
            explorer_accuracy = sum(exp_correct) / n(), .groups="keep")

Confirmatory analysis

The main test I will run is measuring the Pearson correlation between teacher accuracy and student-teacher Hamming distance. The original paper found a significant negative correlation, which indicates that teachers with more accurate understandings of the concept also conveyed their understanding to students with higher fidelity.

Here is the code I used to compute this in Pilot A:

ggplot(hamming_distances, aes(x=explorer_accuracy, y=explorer_student)) +
  geom_point() +
  ylab("explorer-student Hamming distance") +
  xlab("explorer accuracy")

cor.test(hamming_distances$explorer_student, hamming_distances$explorer_accuracy,
         method="pearson")
## 
##  Pearson's product-moment correlation
## 
## data:  hamming_distances$explorer_student and hamming_distances$explorer_accuracy
## t = -8.5107, df = 4, p-value = 0.001046
## alternative hypothesis: true correlation is not equal to 0
## 95 percent confidence interval:
##  -0.9972083 -0.7711926
## sample estimates:
##        cor 
## -0.9734812

I will also conduct two additional analyses: measuring the Pearson correlation between student and teacher accuracies (both at the level of concepts and individual learners) and fitting a Bayesian mixed effects model to predict participants’ accuracy using their role, the rule being learned, and an interaction term between the two.

I have yet to run these on my pilot data.

plot from the original paper

Exploratory analyses

Any follow-up analyses desired (not required).

Discussion

Summary of Replication Attempt

Open the discussion section with a paragraph summarizing the primary result from the confirmatory analysis and the assessment of whether it replicated, partially replicated, or failed to replicate the original result.

Commentary

Add open-ended commentary (if any) reflecting (a) insights from follow-up exploratory analysis, (b) assessment of the meaning of the replication (or not) - e.g., for a failure to replicate, are the differences between original and present study ones that definitely, plausibly, or are unlikely to have been moderators of the result, and (c) discussion of any objections or challenges raised by the current and original authors about the replication attempt. None of these need to be long.