Data 612 - Project 4

1 Introduction

The goal of this assignment is give you practice working with accuracy and other recommender system metrics.

In this assignment you’re asked to do at least one or (if you like) both of the following:

• Work in a small group, and/or • Choose a different dataset to work with from your previous projects.

2 Assignment Highlights

1. As in your previous assignments, compare the accuracy of at least two recommender system algorithms against your offline data.

2. Implement support for at least one business or user experience goal such as increased serendipity, novelty, or diversity.

3. Compare and report on any change in accuracy before and after you’ve made the change in #2.

4. As part of your textual conclusion, discuss one or more additional experiments that could be performed and/or metrics that could be evaluated only if online evaluation was possible. Also, briefly propose how you would design a reasonable online evaluation environment.

3 About the Data

For Project 4, I will build on Project 3 using the same MovieLens dataset, and evaluate the models accuracy using RMSE, MSE and MAE.

The MovieLens Latest Small Datasets contain 100,000 ratings and 3,600 tag applications applied to 9,000 movies by 600 users.

library(dplyr)
library(ggplot2)
library(kableExtra)
library(recommenderlab)

3.1 Loading the MovieLense Data

data (MovieLense)
MovieLense

## 943 x 1664 rating matrix of class 'realRatingMatrix' with 99392 ratings.

summary(rowCounts(MovieLense))

##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
##    19.0    32.0    64.0   105.4   147.5   735.0

slotNames(MovieLense)

## [1] "data"      "normalize"

class(MovieLense)

## [1] "realRatingMatrix"
## attr(,"package")
## [1] "recommenderlab"

head(colnames(MovieLense),10)

##  [1] "Toy Story (1995)"                                    
##  [2] "GoldenEye (1995)"                                    
##  [3] "Four Rooms (1995)"                                   
##  [4] "Get Shorty (1995)"                                   
##  [5] "Copycat (1995)"                                      
##  [6] "Shanghai Triad (Yao a yao yao dao waipo qiao) (1995)"
##  [7] "Twelve Monkeys (1995)"                               
##  [8] "Babe (1995)"                                         
##  [9] "Dead Man Walking (1995)"                             
## [10] "Richard III (1995)"

3.2 Data structure

str(MovieLense)

## Formal class 'realRatingMatrix' [package "recommenderlab"] with 2 slots
##   ..@ data     :Formal class 'dgCMatrix' [package "Matrix"] with 6 slots
##   .. .. ..@ i       : int [1:99392] 0 1 4 5 9 12 14 15 16 17 ...
##   .. .. ..@ p       : int [1:1665] 0 452 583 673 882 968 994 1386 1605 1904 ...
##   .. .. ..@ Dim     : int [1:2] 943 1664
##   .. .. ..@ Dimnames:List of 2
##   .. .. .. ..$ : chr [1:943] "1" "2" "3" "4" ...
##   .. .. .. ..$ : chr [1:1664] "Toy Story (1995)" "GoldenEye (1995)" "Four Rooms (1995)" "Get Shorty (1995)" ...
##   .. .. ..@ x       : num [1:99392] 5 4 4 4 4 3 1 5 4 5 ...
##   .. .. ..@ factors : list()
##   ..@ normalize: NULL

3.3 Distribution of ratings

hist(rowCounts(MovieLense), 
     breaks = 20, 
     main = "Raw Distribution of User's Rating"
     )

3.4 Exploring the ratings values

ratingvalues <- as.vector(MovieLense@data)
unique(ratingvalues) # The rating is numeric with the least value as 0.5 and the highest values as 5.

## [1] 5 4 0 3 1 2

tablerating <- table(ratingvalues)
tablerating

## ratingvalues
##       0       1       2       3       4       5 
## 1469760    6059   11307   27002   33947   21077

3.5 Excluding the missing values

ratingvalues <- ratingvalues[ratingvalues !=0]

3.6 Distribution of the Ratings

hist(ratingvalues, 
     breaks = 5, 
     main="Distribution of Ratings",
     xlab="Ratings",
     col="grey",
     freq=TRUE
     )

4 Pre-processing

Convert data to numeric

movies <- as(MovieLense, 'data.frame')
movies$user <- as.numeric(movies$user)
movies$item <- as.numeric(movies$item)

4.1 Create the sparse Matrix

ratingsMat <- sparseMatrix(i = movies$user, j = movies$item, x = movies$rating, 
                               dims = c(length(unique(movies$user)), length(unique(movies$item))),  
                               dimnames = list(paste("u", 1:length(unique(movies$user)), sep = ""), 
                                               paste("m", 1:length(unique(movies$item)), sep = "")))

ratingsReal <- new("realRatingMatrix", data = ratingsMat)
ratingsReal

## 943 x 1664 rating matrix of class 'realRatingMatrix' with 99392 ratings.

I’m selecting the Users who have rated at least 100 movies and those movies that have been watched at least 150 times

ratings <- MovieLense[rowCounts(MovieLense) > 100, colCounts(MovieLense) > 150]

4.2 Modified Distributions

hist(rowCounts(ratings), 
     breaks = 20, 
     main = "Modified Distribution of User Rating"
     )

hist(colCounts(ratings), 
     breaks = 20, 
     main = "Modified Distribution of Movie Rating"
     )

I’m selecting the Users who have rated at least 100 movies and those movies that have been watched at least 150 times

ratings <- MovieLense[rowCounts(MovieLense) > 100, colCounts(MovieLense) > 150]

5 Spliting the Data Set

Use 90% for training the model

Scheme <- evaluationScheme(ratings, method = "split", k = 1,
                        train = 0.9, given= 10, goodRating=4)

Scheme

## Evaluation scheme with 10 items given
## Method: 'split' with 1 run(s).
## Training set proportion: 0.900
## Good ratings: >=4.000000
## Data set: 358 x 200 rating matrix of class 'realRatingMatrix' with 32713 ratings.

ModelAlgorithms <- list(
  "Random" = list(name="RANDOM", param=list(normalize = "Z-score")),
  "Popular" = list(name="POPULAR", param=list(normalize = "Z-score")),
  "UserBased" = list(name="UBCF", param=list(normalize = "Z-score",
                                                 method="Cosine",
                                                 nn=50, minRating=3)),
  "ItemBased" = list(name="IBCF2", param=list(normalize = "Z-score"
                                                 ))
  
  )

6 Algorithms to Predict Ratings

ModelResults <- evaluate(Scheme, ModelAlgorithms, type = "ratings")

## RANDOM run fold/sample [model time/prediction time]
##   1  [0sec/0.08sec] 
## POPULAR run fold/sample [model time/prediction time]
##   1  [0.06sec/0.02sec] 
## UBCF run fold/sample [model time/prediction time]
##   1

## Warning: Unknown parameters: minRating

## Available parameter (with default values):
## method    =  cosine
## nn    =  25
## sample    =  FALSE
## normalize     =  center
## verbose   =  FALSE
## [0.01sec/0.07sec] 
## IBCF2 run fold/sample [model time/prediction time]
##   1

## Timing stopped at: 0 0 0

## Error in .local(data, ...) : 
##   Recommender method IBCF2 not implemented for data type realRatingMatrix .

## Warning in .local(x, method, ...): 
##   Recommender 'ItemBased' has failed and has been removed from the results!

6.1 Comparing the Recommender Systems

plot(ModelResults)
title(main = "Model Error Comparison")

Popular appears to be the best performing Model with the lowest RMSE, MSE and MAE.

6.2 Let’s predict next movies

ModelResults <- evaluate(Scheme, ModelAlgorithms, n=c(1, 5, 10, 15, 20, 25))

## RANDOM run fold/sample [model time/prediction time]
##   1  [0.02sec/0.01sec] 
## POPULAR run fold/sample [model time/prediction time]
##   1  [0.02sec/0.09sec] 
## UBCF run fold/sample [model time/prediction time]
##   1

## Warning: Unknown parameters: minRating

## Available parameter (with default values):
## method    =  cosine
## nn    =  25
## sample    =  FALSE
## normalize     =  center
## verbose   =  FALSE
## [0.01sec/0.16sec] 
## IBCF2 run fold/sample [model time/prediction time]
##   1

## Timing stopped at: 0 0 0

## Error in .local(data, ...) : 
##   Recommender method IBCF2 not implemented for data type realRatingMatrix .

## Warning in .local(x, method, ...): 
##   Recommender 'ItemBased' has failed and has been removed from the results!

7 Model Metrics/Performance

7.1 ROC curve

plot(ModelResults, annotate = TRUE, legend = "topleft")
title("ROC Curve")

7.2 Precision and Recall

plot(ModelResults, "prec/rec", annotate = TRUE) # precision and recall
title("Precision and Recall")

The Popular Model performs better than Random and the User Based model.

8 Implement Support for Business/User Experiance Goal

The Popular and UserBased models outperforms the Random, I will create a hybrid model consisting of Popular and UserBased models.

It may not always be desirable to recommend products that are likely to be most highly rated by a user. Recommending somewhat unexpected products may improve user experience, expand user preferences, provide additional knowledge about a user.

8.1 Training, Testing and Evaluation Data sets

Prepare training dataset

train <- getData(Scheme, "train")

Prepare testing set

test <- getData(Scheme, "known")

Prepare evaluation set

evaluation <- getData(Scheme, "unknown")

9 Recommender Systems

9.1 The Popular Based Model

Building the User Based Model using 25 nearest neighbor. Building the user collaborative filtering system to recommend movies to users based on how similar they are with other users.

Popularmodel <- Recommender(train, method = "POPULAR")

9.1.1 Making Predictions with newdata = test

Popularpred <- predict(Popularmodel, newdata = test, type = "ratings")

9.1.2 Model Evaluation - Popular Model

Popularaccuracy <- calcPredictionAccuracy(Popularpred, evaluation)
Popularaccuracy

##      RMSE       MSE       MAE 
## 0.9076623 0.8238508 0.7153872

9.2 The User Based Model

Building the User Based Model using 25 nearest neighbor. Building the user collaborative filtering system to recommend movies to users based on how similar they are with other users.

Usermodel <- Recommender(train, method = "UBCF", 
                     param=list(normalize = "center", method="Cosine", nn=25))

9.2.1 Making Predictions with newdata = test

Userpred <- predict(Usermodel, newdata = test, type = "ratings")

9.2.2 Model Evaluation - User Based

Useraccuracy <- calcPredictionAccuracy(Userpred, evaluation)
Useraccuracy

##      RMSE       MSE       MAE 
## 0.9604391 0.9224433 0.7627016

9.3 The Hybrid Model

HybridModel <- HybridRecommender(Popularmodel, Usermodel, weights = c(0.99, 
    0.01))

9.3.1 Making Predictions with newdata = test

predHybrid <- predict(HybridModel, newdata = test, type = "ratings")

9.3.2 Model Evaluation - Hybrid Model

accuracyHybrid <- calcPredictionAccuracy(predHybrid, evaluation)
accuracyHybrid

##      RMSE       MSE       MAE 
## 0.9077973 0.8240959 0.7154864

10 Comprison of the Model Accuracy

ModelError <- rbind(
    Popular = calcPredictionAccuracy(Popularpred, evaluation),
    UserBased = calcPredictionAccuracy(Userpred, evaluation),
    HYBRID = calcPredictionAccuracy(predHybrid, evaluation)
    )
ModelError %>% round(3) %>% 
kable()%>%
kable_styling(bootstrap_options = c("striped", "hover"))

	RMSE	MSE	MAE
Popular	0.908	0.824	0.715
UserBased	0.960	0.922	0.763
HYBRID	0.908	0.824	0.715

The prediction accuracy of the Hybrid Model is similar to the accuracy of the Popular Model. Based on a blend of the two models, the RMSE, MSE and MAE of the Hybrid model decreased slightly in accuracy.

11 Conclusion

The purpose of this exercise was to build a recommender system and to practice working with accuracy and other recommender system metrics.

For the given data set, POPULAR and the Userbased Models within recommenderlab provided the best results. It is worth remembering that the choice of the recommender algorithm depends on the context. A Hybrid of both models provided performance accuarcy similar to the Popular Model.

In addition to the methods provided within the recommenderlab library, there are other supervised and unsupervised machine learning algorithm as well as neutral network-driven deep learning algorithms that could be used to build recommendation systems.

11.0.1 References

Breese JS, Heckerman D, Kadie C (1998). “Empirical Analysis of Predictive Algorithms for Collaborative Filtering.” In Uncertainty in Artificial Intelligence. Proceedings of the Fourteenth Conference, pp. 43-52.

Kohavi, Ron (1995). “A study of cross-validation and bootstrap for accuracy estimation and model selection”. Proceedings of the Fourteenth International Joint Conference on Artificial Intelligence, pp. 1137-1143.

Koren, Y., Bell, R., & Volinsky, C. (2009). Matrix Factorization Techniques for Recommender Systems. Computer, 42(8), 30–37. https://doi.org/10.1109/MC.2009.263

Michael Hahsler (2016). recommenderlab: A Framework for Developing and Testing Recommendation Algorithms, R package. https://CRAN.R-project.org/package=recommenderlab