Replication of Study X by Sample & Sample (20xx, Psychological Science)

Introduction

In recent decades, recommender systems have had an increasingly important role in shaping the consumer experience. These systems help users navigate a vast amount of information by filtering and presenting personalized content and product recommendations. As a student in the Computational Social Sciences program, I am particularly interested in exploring how these systems influence decision-making processes, social behaviors, and information exposure. My goal is to explore how understanding consumer psychology in the context of recommender systems can contribute to improving both scientific theory and the design of fairer, more effective recommendation models.

This particular research used both a between-subjects design experiment and a more quantitative analysis to explore the effect of its recommender systems. For the purposes of this class, I will only be replicating the quantitative analysis. To reproduce these findings, I will use the data, which is available in .csv format through the Open Science Framework project, by conducting Exploratory Data Analysis and visualizing the descriptive statistics of the data. I will also replicate several multilevel regression models in Python to study the relationship between the variables.

One potential challenge I might encounter is understanding the data holistically. Because I am only replicating the computational aspect of a study that incorporated both qualitative and quantitative methods, I may lack important context or insights that were derived from the qualitative analysis, which could limit my ability to fully interpret the results and their broader implications. Another challenge lies in the fact that the paper describes the qualitative aspect of the study much more in depth than the quantitative aspect, making it difficult to fully grasp the computational techniques and methods used. This lack of detailed explanation may lead to uncertainty in replicating the quantitative analysis accurately.

Link to repository: https://github.com/80line/nobel2024

Link to paper: https://github.com/80line/nobel2024/blob/main/original_paper/Nobel_2024.pdf

Methods

Power Analysis

Original effect size, power analysis for samples to achieve 80%, 90%, 95% power to detect that effect size. Considerations of feasibility for selecting planned sample size.

Planned Sample

Planned sample size and/or termination rule, sampling frame, known demographics if any, preselection rules if any.

Materials

All materials - can quote directly from original article - just put the text in quotations and note that this was followed precisely. Or, quote directly and just point out exceptions to what was described in the original article.

Procedure

Can quote directly from original article - just put the text in quotations and note that this was followed precisely. Or, quote directly and just point out exceptions to what was described in the original article.

Analysis Plan

Can also quote directly, though it is less often spelled out effectively for an analysis strategy section. The key is to report an analysis strategy that is as close to the original - data cleaning rules, data exclusion rules, covariates, etc. - as possible.

Clarify key analysis of interest here You can also pre-specify additional analyses you plan to do.

Differences from Original Study

Explicitly describe known differences in sample, setting, procedure, and analysis plan from original study. The goal, of course, is to minimize those differences, but differences will inevitably occur. Also, note whether such differences are anticipated to make a difference based on claims in the original article or subsequent published research on the conditions for obtaining the effect.

Methods Addendum (Post Data Collection)

You can comment this section out prior to final report with data collection.

Actual Sample

Sample size, demographics, data exclusions based on rules spelled out in analysis plan

Differences from pre-data collection methods plan

Any differences from what was described as the original plan, or “none”.

Results

Data preparation

Data preparation following the analysis plan.

Confirmatory analysis

The analyses as specified in the analysis plan.

Side-by-side graph with original graph is ideal here

Exploratory analyses

Any follow-up analyses desired (not required).

Discussion

Summary of Replication Attempt

Open the discussion section with a paragraph summarizing the primary result from the confirmatory analysis and the assessment of whether it replicated, partially replicated, or failed to replicate the original result.

Commentary

Add open-ended commentary (if any) reflecting (a) insights from follow-up exploratory analysis, (b) assessment of the meaning of the replication (or not) - e.g., for a failure to replicate, are the differences between original and present study ones that definitely, plausibly, or are unlikely to have been moderators of the result, and (c) discussion of any objections or challenges raised by the current and original authors about the replication attempt. None of these need to be long.