Replication of Recommender Systems Study by Adeline Liem (2024, Psychological Science)

Introduction

In recent decades, recommender systems have had an increasingly important role in shaping the consumer experience. These systems help users navigate a vast amount of information by filtering and presenting personalized content and product recommendations. As a student in the Computational Social Sciences program, I am particularly interested in exploring how these systems influence decision-making processes, social behaviors, and information exposure. My goal is to explore how understanding consumer psychology in the context of recommender systems can contribute to improving both scientific theory and the design of fairer, more effective recommendation models.

This particular research used both a between-subjects design experiment and a more quantitative analysis to explore the effect of its recommender systems. For the purposes of this class, I will only be replicating the quantitative analysis. To reproduce these findings, I will use the data, which is available in .csv format through the Open Science Framework project, by conducting Exploratory Data Analysis and visualizing the descriptive statistics of the data. I will also replicate several multilevel regression models in Python to study the relationship between the variables.

One potential challenge I might encounter is understanding the data holistically. Because I am only replicating the computational aspect of a study that incorporated both qualitative and quantitative methods, I may lack important context or insights that were derived from the qualitative analysis, which could limit my ability to fully interpret the results and their broader implications. Another challenge lies in the fact that the paper describes the qualitative aspect of the study much more in depth than the quantitative aspect, making it difficult to fully grasp the computational techniques and methods used. This lack of detailed explanation may lead to uncertainty in replicating the quantitative analysis accurately.

Link to repository: https://github.com/80line/nobel2024 Link to paper: https://github.com/80line/nobel2024/blob/main/original_paper/Nobel_2024.pdf

Methods

Power Analysis

N/A

Planned Sample

In this experiment, researchers collected data based on real customers from a store specializing in carpets, Retailer A, as well as a store that sold photography equipment, Retailer B. This data is split into two datasets, cyberfoto.csv and kilands.csv. Both datasets contain the same information, such as user_id, number of clicks, search session length, number of products displayed to the user, and number of purchases.

Materials

For the computational analysis, the researchers used R to develop their insights. For their packages, they used tidyverse, tidytext, SnowballC, pwr, gridExtra, powerAnalysis, broom, generics, knitr, forestmangr, arsenal, qwraps2, grid, psych, apaTables, skimr, lme4, emmeans, car, and sjplot.

My project involves reproducing their quantitative analysis in Python. I will likely use packages such as pandas, numpy, scipy.stats, matplotlib, seaborn, statsmodels, and scikit-learn.

Procedure

The following procedure is directly quoted from the original paper and will be followed precisely, with the exception of being done in Python instead of R:

“To model consumer choices based on the data different variations of linear mixed-effects regres- sion models were employed (Gill & Womack, 2013).”

“Four different multilevel regression models were applied as determined by each outcome measure in Outcome Measures section. To illustrate the general model, I therefore focus on the right- hand side containing the predictors, which was common for all models. Particularities stemming from the outcome variable for each model will be outlined in the following Specific Models section. Note that as random assignment was employed, no control variables were added to the models.”

“Model 1 was a multilevel logistic regression model estimated using maximum likelihood (ML) approach where outcome variable ymodel1 was a ij binary variable for the occurrence of purchase. The variable was coded as 1 = occurrence of purchase, 0 = no occurrence of purchase. Model 2 was a multilevel logistic regression model estimated using ML approach where outcome variable ymodel2 was a binary variable for the occurrence ij of add-to-cart. The variable was coded as 1 = occurrence of add-to-cart, 0 = no occurrence of add-to-cart. Model 3 was a multilevel negative binomial regression estimated using ML estimation approach where outcome variable ymodel3 was ij the number of products viewed. Model 4 was a multilevel linear regression estimated using restricted maximum likelihood estimation approach where outcome variable ymodel4 was session time in seconds.”

Analysis Plan

My analysis plan follows the original paper’s approach, using multilevel regression models to evaluate choice overload effects of RS algorithm changes on purchase, add-to-cart actions, session time, and product views. Outlier sessions (2.5 SD above the mean) will be excluded, and a random intercept per consumer will account for individual behavior across sessions. For additional analysis, I will also compute descriptive statistics for the features we are interested in. Also, I will conduct a correlation analysis to explore correlations among engagement metrics, like “number of products displayed,” “add-to-carts,” and “clicks” to see if viewing more items correlates with higher purchase rates.

Differences from Original Study

The original study uses a between-subjects experiment where subjects are asked to interact with recommender systems that are categorized in a high attractiveness or low attractiveness condition. From there, certain engagement metrics are assesed and analyzed in R. My study will not replicate the between-subjects experiment, and instead focus on reproducing the results of the quantitative analysis in Python. I will also do some additional computation, including generating descriptive statistics and a correlation analysis.

Methods Addendum (Post Data Collection)

You can comment this section out prior to final report with data collection.

Actual Sample

Sample size, demographics, data exclusions based on rules spelled out in analysis plan

Differences from pre-data collection methods plan

Any differences from what was described as the original plan, or “none”.

Results

Data preparation

Data preparation following the analysis plan.

Confirmatory analysis

The analyses as specified in the analysis plan.

Side-by-side graph with original graph is ideal here

Exploratory analyses

Any follow-up analyses desired (not required).

Discussion

Summary of Replication Attempt

Open the discussion section with a paragraph summarizing the primary result from the confirmatory analysis and the assessment of whether it replicated, partially replicated, or failed to replicate the original result.

Commentary

Add open-ended commentary (if any) reflecting (a) insights from follow-up exploratory analysis, (b) assessment of the meaning of the replication (or not) - e.g., for a failure to replicate, are the differences between original and present study ones that definitely, plausibly, or are unlikely to have been moderators of the result, and (c) discussion of any objections or challenges raised by the current and original authors about the replication attempt. None of these need to be long.