Data 612 - Final Project Proposal

1 Introduction

The goal for the final project is for you to build out a recommender system using a large dataset (ex: 1M+ ratings or 10k+ users, 10k+ items. There are three deliverables, with separate dates:

Planning Document Find an interesting dataset and describe the system you plan to build out. If you would like to use one of the datasets you have already worked with, you should add a unique element or incorporate additional data. (i.e. explicit features you scrape from another source, like image analysis on movie posters).

The overall goal, however, will be to produce quality recommendations by extracting insights from a large dataset. You may do so using Spark, or another distributed computing method, OR by effectively applying one of the more advanced mathematical techniques we have covered. There is no preference for one over the other, as long as the recommender works!

Implementation. In this final project deliverable, you’ll build out the system that you describe in the planning document.

2 Project Highlights

1. I am plann to use recommenderlab, an R package which provides a convenient framework to evaluate and compare various recommendation algorithms and quickly establish the best suited approach.

recommenderlab accepts 2 types of rating matrix for modelling:

real rating matrix consisting of actual user ratings, which requires normalisation.

binary rating matrix consisting of 0’s and 1’s, where 1’s indicate if the product was purchased. This is the matrix type needed for the analysis and it does not require normalization.

2. I will arrange the purchase history in a rating matrix, with orders in rows and products in columns. This format is often called a user_item matrix because “users” (e.g. customers or orders) tend to be on the rows and “items” (e.g. products) on the columns.

3. The recommenderlab has an ability to estimate multiple algorithms at a time, I will create a list with the algorithms and consider schemes which evaluate on a binary rating matrix.

4. I will compare and evaluate the performance of the algorithms using ROC Curve, Precision/Recall, RMSE, MSE and MAE and use the best performing algorithm to predict top items list for the new customer.

3 About the Data

The data for this project comes from the UCI Machine Learning Repository, an online archive of large datasets which includes a wide variety of data types, analysis tasks, and application areas.

In this project I’ll use the Online Retail dataset donated to UCI in 2015 by the School of Engineering at London South Bank University. This dataset contains transactions occurring between 01/Dec/2010 and 09/Dec/2011 for a UK-based and registered online retail company and contains 541,909 observations with eight variables. The data is too large for my GitHub but can be downloaded from: http://archive.ics.uci.edu/ml/machine-learning-databases/00352/

library(ggplot2)
library(kableExtra)
library(recommenderlab)
library(data.table)           
library(readxl)               
library(tidyverse)
library(lubridate)
library(skimr)                
library(knitr)                
library(treemap)
library(readUCI)

3.1 Loading the Online Retail dataset

retail <- read_excel("Online Retail.xlsx",trim_ws = TRUE)

3.2 Data structure

str(retail)

## tibble [541,909 x 8] (S3: tbl_df/tbl/data.frame)
##  $ InvoiceNo  : chr [1:541909] "536365" "536365" "536365" "536365" ...
##  $ StockCode  : chr [1:541909] "85123A" "71053" "84406B" "84029G" ...
##  $ Description: chr [1:541909] "WHITE HANGING HEART T-LIGHT HOLDER" "WHITE METAL LANTERN" "CREAM CUPID HEARTS COAT HANGER" "KNITTED UNION FLAG HOT WATER BOTTLE" ...
##  $ Quantity   : num [1:541909] 6 6 8 6 6 2 6 6 6 32 ...
##  $ InvoiceDate: POSIXct[1:541909], format: "2010-12-01 08:26:00" "2010-12-01 08:26:00" ...
##  $ UnitPrice  : num [1:541909] 2.55 3.39 2.75 3.39 3.39 7.65 4.25 1.85 1.85 1.69 ...
##  $ CustomerID : num [1:541909] 17850 17850 17850 17850 17850 ...
##  $ Country    : chr [1:541909] "United Kingdom" "United Kingdom" "United Kingdom" "United Kingdom" ...

3.3 Remove the Cancelled Transaction

3.4 Exploring the dataset

3.5 Excluding the missing values

4 Pre-processing

Convert data to numeric

4.1 Create the sparse Matrix

4.2 Modified Distributions

5 Algorithms- Modeling

Set up List of Algorithms

6 Comprison of the Model Accuracy

6.1 Model Performance Metrics - RMSE, MSE and MAE

6.2 ROC curve

6.3 Precision and Recall

7 Training, Testing and Evaluation Data sets

Prepare training dataset

Prepare testing set

Prepare evaluation set

8 Recommender System

8.1 Best Performing Model

8.1.1 Making Predictions with newdata = test

8.1.2 Model Evaluation - User Based

9 Conclusion

10 References