Read Data

data(MovieLense)
y<-as.matrix(MovieLense@data[1:10,1:100])
y  %>% kable(caption = "Showing Part of the Dataset") %>% kable_styling("striped", full_width = TRUE)

Data Exploration

The dataset we will use for this project is MovieLense. This data is about movies. The table contains the ratings that the users give to movies, which was collected through the MovieLens website (movielens.umn.edu) from 943 users on 1664 movies during the seven-month period from September 19th, 1997 through April 22nd, 1998

It is a 943 rows x 1664 columns rating matrix of class ‘realRatingMatrix’ with 99,392 ratings. Each row of MovieLense corresponds to a user, and each column corresponds to a movie. There are more than 943 x 1664 = 1,500,000 combinations between a user and a movie. Therefore, storing the complete matrix would require more than 1,500,000 cells. However, not every user has watched every movie. Therefore, there are fewer then 100,000 ratings, and the matrix is sparse.

It is also included in our textbook, Building a Recommendation System with R, by Suresh K. Gorakala and Michele Usuelli.

As not every user has watched every movie, the dataset is large and sparse, which we may not use the whole dataset to build our models. The histogram below shows the number of ratings per user.

dim(MovieLense)

## [1]  943 1664

table(MovieLense@data %>% as.vector()) %>%
  data.frame() %>%
  mutate(Pcnt = percent(Freq/sum(Freq))) %>%
  rename(Rating = Var1) %>%
  kable() %>%
  kable_styling(bootstrap_options = c('striped', 'bordered'), full_width = FALSE) %>%
  add_header_above(c('Rating Frequency' = 3))

MovieLense %>%
rowCounts() %>%
  data.frame() %>%
  rename(Rating_Per_User = '.') %>%
    ggplot(aes(x=Rating_Per_User)) +
    geom_histogram(color = 'grey', fill = 'deeppink4') +
    ggtitle('Number of Rating Per User')

Sampling

We filter the users who has equal to or more than 100 ratings to be our sample dataset.

sample <- MovieLense[rowCounts(MovieLense)>=100,] 

Train-Test Splitting with Cross Validation

We implement 5-fold cross validation method to split our sample dataset into training dataset and testing dataset.

eval_set <- sample %>%
  evaluationScheme(method='split', train = 0.8, given=25)


data_train <- getData(eval_set, 'train')
data_test_known <- getData(eval_set, 'known')
data_test_unknown <- getData(eval_set, 'unknown')

Train Model

We then train our model with Alternating Least Squares (ALS) method. Before we start, we record the system time (start time).

start_time_reccom <- Sys.time()
rec_als_recom <- Recommender(data = data_train, method = 'ALS')

Make Prediction

And make prediction on the ratings. After the prediction, we record the system time (end time)

predict_recom <- predict(rec_als_recom, data_test_known, type='ratings')
predict_recom@data@x <- pmax(pmin(predict_recom@data@x, 5),0)
end_time_reccom <- Sys.time()

RMSE

We then find out the model accuracy by calculating the RMSE of our model.

eval_metrics <- calcPredictionAccuracy(predict_recom, data_test_unknown) 

rmse_recom <- eval_metrics[1] %>% as.numeric()

print(str_c('RMSE of Model Built in Recommenderlab: ',rmse_recom %>% as.character()))

## [1] "RMSE of Model Built in Recommenderlab: 0.970627716562321"

Compute Run Time

The system run time of the process is calcualted by the start time and end time we recorded above. It is more than 20 seconds.

run_time_reccom <- end_time_reccom - start_time_reccom
end_time_reccom - start_time_reccom

## Time difference of 25.617 secs

Set Spark Configuration

Before building our model, we first set 80% of our system(PC)'s accessible memory to Spark.

conf <- spark_config()
#conf$`sparklyr.shell.driver-memory` <- '5G'
conf$spark.memory.fraction <- 0.8

Create Local Spark Connection

We then create our local Spark connection.

sc <- spark_connect(master = 'local', config = conf)

Copy Data as Spark Data Frame

As MovieLense is in a realRatingMatrix format, we convert it to a dataframe before copying to our local Spark. Because the recommender algorithm in Spark takes numerical values as input, we convert the items (movies) from factors to numerical representations. Note that we named the user column 'user' and the numerical representations of the item factors 'item' because the recommender algorithm takes column vectors named 'user' and 'item' by default and this setting cannot be changed.

movielense <- as(MovieLense[rowCounts(MovieLense)>=100,], "data.frame") %>%
  rename(movie = item) %>%
  mutate(item = factor(movie) %>% as.numeric(),
         user = user %>% as.numeric()) %>%
  select(user, item, movie, rating)

movie_tbl <- sdf_copy_to(sc, movielense, 'MovieLense',overwrite = TRUE)
movie_tbl

Train-Test-Split

We split our dataset into training set (80%) and testing set (20%).

movie_split <- movie_tbl %>%
  sdf_random_split(training = 0.8, testing = 0.2)

movie_train <- movie_split$training
movie_test <- movie_split$testing

Train Model

Train an ALS recommendation model in Spark using function ml_als.

start_time_spark <- Sys.time()
model_formula = rating ~ user + item
rec_als_spark <- ml_als(movie_train, model_formula, max_iter = 5)

Make Prediction

Predict ratings using function ml_predict.

predict_spark <- ml_predict(rec_als_spark, movie_test)
end_time_spark <- Sys.time()

Make Top 5 Recomendation for Each User

Create top 5 item recommendations for all users.

ml_recommend(rec_als_spark, type = 'item', 5) %>% 
  left_join(movie_tbl %>% select(item, movie) %>% distinct(), by = c('item')) %>%
  select(user, movie) %>%
  group_by(user) %>%
  mutate(rank = rank(movie)) %>%
  mutate(rank = paste('Recommendation', rank %>% as.character())) %>%
  collect() %>%
  spread(key = rank, value = movie) %>%
  arrange(user) %>%
  kable() %>%
  kable_styling(bootstrap_options = c('bordered', 'striped'), full_width = TRUE) %>%
  scroll_box(height = '400px')

RMSE

Calculate RMSE of the Spark recommendation model using testing set.

rmse_spark <- predict_spark %>% 
  filter(!isnan(prediction)) %>%
  summarise((rating - prediction)^2 %>% mean() %>% sqrt()) %>%
  collect() %>%
  as.numeric()
print(str_c('RMSE of Model Built in Spark: ',rmse_spark %>% as.character()))

## [1] "RMSE of Model Built in Spark: 0.915883319849977"

Compute Run Time

Calculate the runtime of our Spark recommendation model. It is less than 10 seconds.

run_time_spark <- end_time_spark - start_time_spark
end_time_spark - start_time_spark

## Time difference of 8.30753 secs

Disconnect to Spark

Disconnect our R from Spark.

spark_disconnect(sc)