DATA 612 Project 4

1 Project Scope

The goal of this assignment is give you practice working with accuracy and other recommender system metrics. Choose a different dataset to work with from your previous projects.

Deliverables 1. As in your previous assignments, compare the accuracy of at least two recommender system algorithms against your offline data. 2. Implement support for at least one business or user experience goal such as increased serendipity, novelty, or diversity. 3. Compare and report on any change in accuracy before and after you’ve made the change in #2. 4. As part of your textual conclusion, discuss one or more additional experiments that could be performed and/or metrics that could be evaluated only if online evaluation was possible. Also, briefly propose how you would design a reasonable online evaluation environment.

2 Introduction

For this project, I chose Jester5K dataset. The project will

• Explore the Jester5K dataset
• Compare the UCBF, Popular, Random, and Hybrid recommender
• Discuss a reasonable online evaluation

2.1 Data Exploration - Jester 5K dataset

The Jester 5K dataset is a dataset consisting of 5000 users rating 100 of jokes which means a combination of 500,000 users and ratings.

set.seed(123)
data("Jester5k")
Jester5k

## 5000 x 100 rating matrix of class 'realRatingMatrix' with 362106 ratings.

Jester5K has a slot for the data. Looking at this slot, we see that the data slot of Jester5K is of the class dgcMatrix.

slotNames(Jester5k)

## [1] "data"      "normalize"

class(Jester5k@data)

## [1] "dgCMatrix"
## attr(,"package")
## [1] "Matrix"

2.1.1 The Jokes

Each column in the dataset represents a joke for a total of 100 jokes.

length(colnames(Jester5k@data))

## [1] 100

The columns contain the names of the jokes. Below are the first five column names and below that is the text for each of these jokes.

head(colnames(Jester5k@data),5)

## [1] "j1" "j2" "j3" "j4" "j5"

tail(JesterJokes,6)

##                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          j95 
##                                                                                                                                                                                                                                                                                                                                                  "Just a thought .. Before criticizing someone, walk a mile in their shoes. Then when you do criticize them, you will be a mile away and have their shoes !" 
##                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          j96 
##                                                                                                                                                                "Two attorneys went into a diner and ordered two drinks. Then they produced sandwiches from their briefcases and started to eat. The owner became quite concerned and marched over and told them, \"You can't eat your own sandwiches in here!\" The attorneys looked at each other, shrugged their shoulders and then exchanged sandwiches." 
##                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          j97 
##                                                                     "A teacher is explaining to her class how different languages use negatives differently. She says, \"In all languages, a positive followed by a negative or a negative followed by a positive makes a negative. In some languages, two negatives together make a positive, while in others they make a negative. But in no language do two positives make a negative.\" One of the students puts up his hand and says, \"Yeah, right.\"" 
##                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          j98 
## "Age and Womanhood 1. Between the ages of 13 and 18 ... She is like Africa, virgin and unexplored. 2. Between the ages of 19 and 35 ... She is like Asia, hot and exotic. 3. Between the ages of 36 and 45 ... She is like America, fully explored, breathtakingly beautiful,and free with her resources. 4. Between the ages of 46 and 56 ... She is like Europe, exhausted but still has points of interest. 5. After 56 she is like Australia ... Everybody knows it's down there, but who gives a damn?" 
##                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          j99 
##                                                                                                                                                                                                                                                                                                                                                                                            "A bus station is where a bus stops. A train station is where a train stops. On my desk I have a work station..." 
##                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         j100 
##                                                                                                                                                                                                                                                                                                                                                  "Q: Whats the difference between greeting a Queen and greeting the President of the United States? A: You only have to get on one knee to greet the queen."

2.1.2 The user ratings

The dataset contains a totak of 362,106 ratings which means that the dataset is dense aprrox. 72% full.

nratings(Jester5k)

## [1] 362106

The ratings of the jokes range from a -9.95 to a positive 9.9 with an average rating of 0.8525. This result is after removing entries without a “0” rating.

jokes_vector_ratings <-  as.vector(Jester5k@data)
jokes_vector_ratings <- jokes_vector_ratings[jokes_vector_ratings != 0 ]
range(jokes_vector_ratings)

## [1] -9.95  9.90

summary(jokes_vector_ratings)

##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
## -9.9500 -3.0600  1.4600  0.8525  5.1000  9.9000

hist(jokes_vector_ratings, main="Distribution of ratings", col="yellow")
abline(v=mean(jokes_vector_ratings),col="red")

From this visualization, we see that jokes rated from 3 to 4 on a scale of -10 to 10, were the most commonly rated joke. Overall, more jokes are rated positively than negatively.

2.2 Most rated Jokes

ratings_per_joke <- colCounts(Jester5k)
joke_frequency <- data.frame(jokes = names(ratings_per_joke), told = ratings_per_joke)
joke_frequency <- joke_frequency %>% 
                  arrange(desc(told))

joke_frequency[1:3,]

##   jokes told
## 1    j8 5000
## 2   j13 5000
## 3   j15 5000

JesterJokes["j8"]

##                                                                                  j8 
## "Q. Did you hear about the dyslexic devil worshiper? A. He sold his soul to Santa."

JesterJokes["j13"]

##                                                                                                    j13 
## "They asked the Japanese visitor if they have elections in his country. \"Every Morning\" he answers."

JesterJokes["j15"]

##                                                                                    j15 
## "Q: What did the blind person say when given some matzah? A: Who the hell wrote this?"

2.3 Binarize vs. Normalize

The goal of this project is to compare recommender models accounting for finding novel or serendipitous recommendations. If I binarize the data, then I lose the granualarity of the recommendation. All that I would be able to tell would be that the user rated the joke, not how well or highly rated.

Normalizing the data seems to be the better approach given the wide range of the data.

First, I defined a minimum number of 99 users who have rated at least 25 jokes.

Jester_Jokes <- Jester5k[rowCounts(Jester5k) > 99, colCounts(Jester5k) > 25]
Jester_Jokes 

## 1422 x 100 rating matrix of class 'realRatingMatrix' with 142200 ratings.

average_ratings_per_user <- rowMeans(Jester_Jokes)
qplot(average_ratings_per_user) + stat_bin(binwidth = 0.1) +
ggtitle("Distribution of the average rating per user")

## `stat_bin()` using `bins = 30`. Pick better value with `binwidth`.

Normalizing the Jokes

Jester_Jokes_norm <- normalize(Jester_Jokes)
sum(rowMeans(Jester_Jokes_norm ) > 0.00001)

## [1] 0

jokes_vector_ratings_norm <-  as.vector(Jester_Jokes_norm@data)
jokes_vector_ratings_norm  <- jokes_vector_ratings_norm[jokes_vector_ratings_norm  != 0 ]

hist(jokes_vector_ratings_norm, main="Distribution of ratings", col="yellow")
abline(v=mean(jokes_vector_ratings_norm ),col="red")

2.4 User Similarity

In the blocks below, I used RecommenderLab’s similarity function to find similarity among the top 50 users. The more red blocks indicates similarity between the users. This small sample tells us that there is some similarities between the users but not very strong.

similarity_users <- similarity(Jester_Jokes[1:50, ], method ="cosine", which = "users")

image(as.matrix(similarity_users), main = "User similarity")

2.5 User Similarity

Conversely, we see a stronger similarity between the jokes themselves.

similarity_jokes <- similarity(Jester_Jokes[, 1:50], method ="cosine", which = "items")

image(as.matrix(similarity_jokes), main = "Joke similarity")

3 Comparing Collaboration Filtering Models - UBCF and IBCF

In this section, I applied the UBCF and IBCF algorithms to the Jester dataset. For Serendipity, I then applied the POPULAR algorithm.

3.1 K-Fold Split

n_fold <- 6
items_to_keep <- 10
rating_threshold <- 0

eval_sets  <- evaluationScheme(data = Jester_Jokes, method = "cross-validation", k = n_fold, given = items_to_keep, goodRating = rating_threshold)

size_sets <- sapply(eval_sets@runsTrain, length)
size_sets

## [1] 1185 1185 1185 1185 1185 1185

3.2 Evaluation of models

To evaluate the two models, I created a list of UBCF and SVD models with different parameters (pearson and cosine). This list was fed into the evaluate function and below. The visualizations below both show that the UBCF using the “pearson” method had the larger area under the curve.

In conclusion, this project has shown that if we want our model to be more precise with its recommendations, we would choose the UBCF model and recommend five or less movies. If the number of recommendations is 50 or above, then the UBCF-pearson model does not perform better than the other models.

models_to_evaluate <- list(
UBCF_cos = list(name = "UBCF", param = list(method ="cosine")),
UBCF_cor = list(name = "UBCF", param = list(method ="pearson")),
IBCF_cos = list(name = "IBCF", param = list(method ="cosine")),
IBCF_cor = list(name = "IBCF", param = list(method ="pearson"))
)

n_recommendations <- c(1, 5, seq(10, 100, 10))

compare_results <- evaluate(x = eval_sets, method = models_to_evaluate, n= n_recommendations)

## UBCF run fold/sample [model time/prediction time]
##   1  [0.04sec/0.26sec] 
##   2  [0.01sec/0.38sec] 
##   3  [0.02sec/0.27sec] 
##   4  [0.02sec/0.28sec] 
##   5  [0.01sec/0.27sec] 
##   6  [0.01sec/0.4sec] 
## UBCF run fold/sample [model time/prediction time]
##   1  [0sec/0.26sec] 
##   2  [0.02sec/0.29sec] 
##   3  [0.02sec/0.28sec] 
##   4  [0.02sec/0.27sec] 
##   5  [0sec/0.31sec] 
##   6  [0.01sec/0.28sec] 
## IBCF run fold/sample [model time/prediction time]
##   1  [0.06sec/0.05sec] 
##   2  [0.06sec/0.04sec] 
##   3  [0.05sec/0.03sec] 
##   4  [0.05sec/0.05sec] 
##   5  [0.06sec/0.05sec] 
##   6  [0.04sec/0.04sec] 
## IBCF run fold/sample [model time/prediction time]
##   1  [0.07sec/0.03sec] 
##   2  [0.08sec/0.03sec] 
##   3  [0.08sec/0.03sec] 
##   4  [0.08sec/0.03sec] 
##   5  [0.08sec/0.03sec] 
##   6  [0.07sec/0.04sec]

plot(compare_results, annotate = 4, legend = "topleft") + title("ROC curve")

## integer(0)

plot(compare_results, "prec/rec", annotate = 5, legend = "bottomright")
title("Precision-recall")

3.2.1 Serendipity

To account for serendipity, I added and evaluated the POPULAR model without any parameters. We see that this model performs as well as the UBCF models and outperforms the IBCF model.

recommender_models <- recommenderRegistry$get_entries(dataType ="realRatingMatrix")
recommender_models$POPULAR_realRatingMatrix$parameters

## $normalize
## [1] "center"
## 
## $aggregationRatings
## standardGeneric for "colMeans" defined from package "base"
## 
## function (x, na.rm = FALSE, dims = 1, ...) 
## standardGeneric("colMeans")
## <bytecode: 0x00000000234a3960>
## <environment: 0x000000001b1dd280>
## Methods may be defined for arguments: x
## Use  showMethods("colMeans")  for currently available ones.
## 
## $aggregationPopularity
## standardGeneric for "colSums" defined from package "base"
## 
## function (x, na.rm = FALSE, dims = 1, ...) 
## standardGeneric("colSums")
## <bytecode: 0x000000002348e658>
## <environment: 0x000000001b3354e8>
## Methods may be defined for arguments: x
## Use  showMethods("colSums")  for currently available ones.

models_to_evaluate <- list(
UBCF_cos = list(name = "UBCF", param = list(method ="cosine")),
UBCF_cor = list(name = "UBCF", param = list(method ="pearson")),
IBCF_cos = list(name = "IBCF", param = list(method ="cosine")),
IBCF_cor = list(name = "IBCF", param = list(method ="pearson")),
POPULAR= list(name = "POPULAR", param=NULL)
)

n_recommendations <- c(1, 5, seq(10, 100, 10))

compare_results <- evaluate(x = eval_sets, method = models_to_evaluate, n= n_recommendations)

## UBCF run fold/sample [model time/prediction time]
##   1  [0.02sec/0.26sec] 
##   2  [0sec/0.29sec] 
##   3  [0.01sec/0.28sec] 
##   4  [0.01sec/0.38sec] 
##   5  [0.01sec/0.27sec] 
##   6  [0.02sec/0.23sec] 
## UBCF run fold/sample [model time/prediction time]
##   1  [0sec/0.27sec] 
##   2  [0.02sec/0.28sec] 
##   3  [0.02sec/0.27sec] 
##   4  [0.01sec/0.39sec] 
##   5  [0sec/0.28sec] 
##   6  [0.02sec/0.28sec] 
## IBCF run fold/sample [model time/prediction time]
##   1  [0.18sec/0.03sec] 
##   2  [0.06sec/0.03sec] 
##   3  [0.07sec/0.03sec] 
##   4  [0.06sec/0.02sec] 
##   5  [0.07sec/0.03sec] 
##   6  [0.06sec/0.03sec] 
## IBCF run fold/sample [model time/prediction time]
##   1  [0.08sec/0.03sec] 
##   2  [0.06sec/0.03sec] 
##   3  [0.09sec/0.03sec] 
##   4  [0.09sec/0.03sec] 
##   5  [0.08sec/0.03sec] 
##   6  [0.08sec/0.03sec] 
## POPULAR run fold/sample [model time/prediction time]
##   1  [0sec/0.37sec] 
##   2  [0.02sec/0.26sec] 
##   3  [0sec/0.25sec] 
##   4  [0sec/0.25sec] 
##   5  [0sec/0.27sec] 
##   6  [0.02sec/0.25sec]

plot(compare_results, annotate = 4, legend = "topleft") + title("ROC curve")

## integer(0)

plot(compare_results, "prec/rec", annotate = 5, legend = "bottomright")
title("Precision-recall")

3.3 Building the Final Model - POPULAR vs. RANDOM

In this final section, I compare the POPULAR model to a RANDOM model. The POPULAR model recommends jokes that are the most popular to users who have not rated the jokes. The RANDOM model assigns ratings on a random basis.

Below, I extracted a sample of 1500 users from the Jester5k dataset and extract the rating_jokes_matrix from the object.

3.3.1 Sample the Data

set.seed(123)
jester_data <- sample(Jester5k,1500)
rating_jokes_matrix <- getRatingMatrix(jester_data)
str(rating_jokes_matrix)

## Formal class 'dgCMatrix' [package "Matrix"] with 6 slots
##   ..@ i       : int [1:108049] 0 1 5 6 7 8 9 10 11 12 ...
##   ..@ p       : int [1:101] 0 985 2080 3071 3999 5498 6734 8234 9734 10670 ...
##   ..@ Dim     : int [1:2] 1500 100
##   ..@ Dimnames:List of 2
##   .. ..$ : chr [1:1500] "u23102" "u4970" "u10139" "u19735" ...
##   .. ..$ : chr [1:100] "j1" "j2" "j3" "j4" ...
##   ..@ x       : num [1:108049] 1.21 -1.31 2.86 -2.43 -2.09 -1.7 5.05 3.4 3.45 -1.75 ...
##   ..@ factors : list()

3.3.2 Normalize the Data

jester_data_norm <- normalize(jester_data, method="z-score")
par(mfrow=c(2,1))
plot(density(getRatings(jester_data)), main="Original")
plot(density(getRatings(jester_data_norm)), main="Normalized")

3.3.3 Train, Test, Split

Below using the evaluation scheme method, 80% of the data is allocated to training. The model will take 20 jokes from each user and put them aside. Once the model is built, predictions will be made on these 20 jokes and those predictions will be evaluated against the actual ratings.

plan <- evaluationScheme(jester_data_norm, method="split", train=0.8, given=10, goodRating=2)
plan

## Evaluation scheme with 10 items given
## Method: 'split' with 1 run(s).
## Training set proportion: 0.800
## Good ratings: >=2.000000
## Data set: 1500 x 100 rating matrix of class 'realRatingMatrix' with 108049 ratings.
## Normalized using z-score on rows.

The getData function extracts the train, test, and test_known sets from the plan. The test set or “known” extracts known ratings from the test set.

train <- getData(plan, type="train")
train

## 1200 x 100 rating matrix of class 'realRatingMatrix' with 86349 ratings.
## Normalized using z-score on rows.

test <- getData(plan, type="unknown")
test

## 300 x 100 rating matrix of class 'realRatingMatrix' with 18700 ratings.

test_known <- getData(plan, "known")
test_known

## 300 x 100 rating matrix of class 'realRatingMatrix' with 3000 ratings.

3.3.4 Train and Test the Random Model

In the blocks below, I built Random model which radomly prredicts the ratings. It will be used as reference model for the POPULAR model.

random_model <- Recommender(train, "RANDOM")
random_model

## Recommender of type 'RANDOM' for 'realRatingMatrix' 
## learned using 1200 users.

getModel(random_model)

## $range
## [1] -6.095311  6.647723
## 
## $labels
##   [1] "j1"   "j2"   "j3"   "j4"   "j5"   "j6"   "j7"   "j8"   "j9"   "j10" 
##  [11] "j11"  "j12"  "j13"  "j14"  "j15"  "j16"  "j17"  "j18"  "j19"  "j20" 
##  [21] "j21"  "j22"  "j23"  "j24"  "j25"  "j26"  "j27"  "j28"  "j29"  "j30" 
##  [31] "j31"  "j32"  "j33"  "j34"  "j35"  "j36"  "j37"  "j38"  "j39"  "j40" 
##  [41] "j41"  "j42"  "j43"  "j44"  "j45"  "j46"  "j47"  "j48"  "j49"  "j50" 
##  [51] "j51"  "j52"  "j53"  "j54"  "j55"  "j56"  "j57"  "j58"  "j59"  "j60" 
##  [61] "j61"  "j62"  "j63"  "j64"  "j65"  "j66"  "j67"  "j68"  "j69"  "j70" 
##  [71] "j71"  "j72"  "j73"  "j74"  "j75"  "j76"  "j77"  "j78"  "j79"  "j80" 
##  [81] "j81"  "j82"  "j83"  "j84"  "j85"  "j86"  "j87"  "j88"  "j89"  "j90" 
##  [91] "j91"  "j92"  "j93"  "j94"  "j95"  "j96"  "j97"  "j98"  "j99"  "j100"

This is a random model that produces a random number between -6.095 and 6.647

random_predict <- predict(random_model, test_known, n=5, type="topNList")
random_predict@items[1]

## [[1]]
## [1] 92 97 72 15 29

The predict function predicts the top 5 jokes that the user would have rated. Here, we see that jokes 92, 97, 72, 15, and 29 are recommended for user 1.

3.4 POPULAR model

popular_model <- Recommender(train, "POPULAR")

## Warning in .local(x, ...): x was already normalized by row!

popular_model

## Recommender of type 'POPULAR' for 'realRatingMatrix' 
## learned using 1200 users.

popular_predict <- predict(popular_model, test_known, n = 5, type ="topNList")
popular_predict@items[1]

## $u10139
## [1] 50 32 27 53 49

popular_predict@ratings[1]

## $u10139
## [1] 0.6516351 0.5460754 0.5273327 0.5096653 0.4996111

new_models_to_evaluate <- list(
RANDOM=list(name="RANDOM", param=NULL),
POPULAR= list(name = "POPULAR", param=NULL)
)

n_recommendations <- c(1, 5, seq(10, 100, 10))

plan_2 <- evaluationScheme(jester_data, method="cross", train=0.9, given = 10,goodRating=5)
results <- evaluate(plan_2, method = new_models_to_evaluate, type = "topNList", n =n_recommendations )

## RANDOM run fold/sample [model time/prediction time]
##   1  [0sec/0.03sec] 
##   2  [0sec/0.03sec] 
##   3  [0sec/0.02sec] 
##   4  [0sec/0.02sec] 
##   5  [0sec/0.02sec] 
##   6  [0sec/0.01sec] 
##   7  [0sec/0.01sec] 
##   8  [0sec/0.01sec] 
##   9  [0sec/0.02sec] 
##   10  [0sec/0.02sec] 
## POPULAR run fold/sample [model time/prediction time]
##   1  [0sec/0.17sec] 
##   2  [0.02sec/0.16sec] 
##   3  [0.01sec/0.14sec] 
##   4  [0.01sec/0.15sec] 
##   5  [0.01sec/0.14sec] 
##   6  [0sec/0.14sec] 
##   7  [0.02sec/0.15sec] 
##   8  [0.01sec/0.16sec] 
##   9  [0sec/0.15sec] 
##   10  [0.01sec/0.15sec]

plot(results, annotate = 4, legend = "topleft") + title("ROC curve")

## integer(0)

plot(results, "prec/rec", annotate = 5, legend = "bottomright")
title("Precision-recall")

avg(results)

## $RANDOM
##             TP         FP        FN        TN precision     recall        TPR
## 1    0.1893333  0.8106667 15.210667 73.789333 0.1893333 0.01239150 0.01239150
## 5    0.8966667  4.1033333 14.503333 70.496667 0.1793333 0.05851166 0.05851166
## 10   1.7900000  8.2100000 13.610000 66.390000 0.1790000 0.11950304 0.11950304
## 20   3.4440000 16.5560000 11.956000 58.044000 0.1722000 0.22689122 0.22689122
## 30   5.1586667 24.8413333 10.241333 49.758667 0.1719556 0.33937770 0.33937770
## 40   6.7846667 33.2153333  8.615333 41.384667 0.1696167 0.44410500 0.44410500
## 50   8.4933333 41.5066667  6.906667 33.093333 0.1698667 0.55256760 0.55256760
## 60  10.2326667 49.7673333  5.167333 24.832667 0.1705444 0.66525255 0.66525255
## 70  11.9526667 58.0473333  3.447333 16.552667 0.1707524 0.77525784 0.77525784
## 80  13.6973333 66.3026667  1.702667  8.297333 0.1712167 0.89333499 0.89333499
## 90  15.4000000 74.6000000  0.000000  0.000000 0.1711111 1.00000000 1.00000000
## 100 15.4000000 74.6000000  0.000000  0.000000 0.1711111 1.00000000 1.00000000
##            FPR
## 1   0.01080674
## 5   0.05499423
## 10  0.10988213
## 20  0.22238532
## 30  0.33352416
## 40  0.44579523
## 50  0.55677104
## 60  0.66719542
## 70  0.77828505
## 80  0.88907150
## 90  1.00000000
## 100 1.00000000
## 
## $POPULAR
##             TP         FP         FN        TN precision     recall        TPR
## 1    0.4413333  0.5586667 14.9586667 74.041333 0.4413333 0.03941909 0.03941909
## 5    2.0613333  2.9386667 13.3386667 71.661333 0.4122667 0.17136769 0.17136769
## 10   3.9573333  6.0426667 11.4426667 68.557333 0.3957333 0.31571215 0.31571215
## 20   6.8120000 13.1880000  8.5880000 61.412000 0.3406000 0.50463039 0.50463039
## 30   8.5766667 21.4233333  6.8233333 53.176667 0.2858889 0.60466710 0.60466710
## 40  10.0560000 29.9440000  5.3440000 44.656000 0.2514000 0.68979073 0.68979073
## 50  11.1713333 38.8286667  4.2286667 35.771333 0.2234267 0.74730948 0.74730948
## 60  12.2026667 47.7973333  3.1973333 26.802667 0.2033778 0.80094223 0.80094223
## 70  13.5013333 56.4986667  1.8986667 18.101333 0.1928762 0.87773655 0.87773655
## 80  14.5113333 65.4886667  0.8886667  9.111333 0.1813917 0.93421910 0.93421910
## 90  15.4000000 74.6000000  0.0000000  0.000000 0.1711111 1.00000000 1.00000000
## 100 15.4000000 74.6000000  0.0000000  0.000000 0.1711111 1.00000000 1.00000000
##             FPR
## 1   0.007155769
## 5   0.037969297
## 10  0.077930627
## 20  0.170987941
## 30  0.280201809
## 40  0.394118300
## 50  0.513858226
## 60  0.635011978
## 70  0.752409655
## 80  0.874205939
## 90  1.000000000
## 100 1.000000000

3.4.1 RMSE

RMSE_Random <- evaluate(plan, method="RANDOM", type="ratings")

## RANDOM run fold/sample [model time/prediction time]
##   1  [0sec/0.01sec]

avg(RMSE_Random)

##         RMSE      MSE      MAE
## res 1.451845 2.107853 1.154725

RMSE_POPULAR <- evaluate(plan, method="POPULAR", type="ratings")

## POPULAR run fold/sample [model time/prediction time]
##   1

## Warning in .local(x, ...): x was already normalized by row!

## [0sec/0.02sec]

avg(RMSE_POPULAR)

##          RMSE       MSE       MAE
## res 0.9940796 0.9881943 0.8014356

4 Conclusion

On the basis of AOC, Precision/Recall, and the RMSE, the POPULAR model outperforms the RANDOM model. IOW, populars jokes are rated higher by users than a random selection of jokes. Another experiment that could be run would be A/B testing. For example, lets say that you have an online streaming music service. This service would deploy two recommender models and based on whether the users actually select the songs and rate them would give you the best evaluation.

To conclude, let’s look at a few of the jokes the POPULAR model recommended.

top_jokes <- c(popular_predict@items[[1]])
for (i in 1:length(top_jokes)){
    top_jokes[i] = (paste0("j",top_jokes[i]))

}
JesterJokes[top_jokes]

##                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             j50 
##                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  "A guy goes into confession and says to the priest, \"Father, I'm 80 years old, widower, with 11 grandchildren. Last night I met two beautiful flight attendants. They took me home and I made love to both of them. Twice.\" The priest said: \"Well, my son, when was the last time you were in confession?\" \"Never Father, I'm Jewish.\" \"So then, why are you telling me?\" \"I'm telling everybody.\"" 
##                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             j32 
##                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               "A man arrives at the gates of heaven. St. Peter asks, \"Religion?\" The man says, \"Methodist.\" St. Peter looks down his list, and says, \"Go to room 24, but be very quiet as you pass room 8.\" Another man arrives at the gates of heaven. \"Religion?\" \"Baptist.\" \"Go to room 18, but be very quiet as you pass room 8.\" A third man arrives at the gates. \"Religion?\" \"Jewish.\" \"Go to room 11, but be very quiet as you pass room 8.\" The man says, \"I can understand there being different rooms for different religions, but why must I be quiet when I pass room 8?\" St. Peter tells him, \"Well the Catholics are in room 8, and they think they're the only ones here." 
##                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             j27 
##                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              "Clinton returns from a vacation in Arkansas and walks down the steps of Air Force One with two pigs under his arms. At the bottom of the steps, he says to the honor guardsman, \"These are genuine Arkansas Razor-Back Hogs. I got this one for Chelsea and this one for Hillary.\" The guardsman replies, \"Nice trade, Sir.\"" 
##                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             j53 
## "One Sunday morning William burst into the living room and said, \"Dad! Mom! I have some great news for you! I am getting married to the most beautiful girl in town. She lives a block away and her name is Susan.\" After dinner, William's dad took him aside. \"Son, I have to talk with you. Your mother and I have been married 30 years.. She's a wonderful wife but she has never offered much excitement in the bedroom, so I used to fool around with women a lot. Susan is actually your half-sister, and I'm afraid you can't marry her.\" William was heart-broken. After eight months he eventually started dating girls again. A year later he came home and very proudly announced, \"Dianne said yes! We're getting married in June.\" Again his father insisted on another private conversation and broke the sad news. \"Dianne is your half-sister too, William. I'm awfully sorry about this.\" William was furious! He finally decided to go to his mother with the news. \"Dad has done so much harm.. I guess I'm never going to get married,\" he complained. \"Every time I fall in love, Dad tells me the girl is my half-sister.\" His mother just shook her head. \"Don't pay any attention to what he says, dear. He's not really your father.\"" 
##                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             j49 
##                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                "Three engineering students were gathered together discussing the possible designers of the human body. One said, \"It was a mechanical engineer. Just look at all the joints.\" Another said, \"No, it was an electrical engineer. The nervous systems many thousands of electrical connections.\" The last said, \"Actually it was a civil engineer. Who else would run a toxic waste pipeline through a recreational area?\""