Introduction

This project will focus on Movies, in particular understanding what drives audience ratings and helping users categorise films into genres based on their features.As the world rapidly evolves into a ever-growing digital world, there is ample opportunity to improve our genre classification and understand what drives bad and good reviews.

Objective

The core objective of this initiative is to leverage Data Science principles to discover the factors influencing movie ratings and accurately categorise films into their respective genres. We aim to achieve this by:

Predicting Average Movie Ratings: Developing a robust regression model to forecast the average rating a movie is likely to receive, providing valuable insights for creators and consumers alike.
Categorising Film Genres: Building an effective classification model to automatically assign genres to films based on their intrinsic features, streamlining content organization and discovery.

Workflow

Library Setup

Setup all the packages that will be used in this project, from data preprocessing until application deployment

knitr::opts_chunk$set(echo = TRUE)
library(dplyr)
library(knitr)
library(kableExtra)
library(stringr)
library(tidyr)
library(readr)
library(ggplot2)
library(wordcloud)
library(tm)
library(tibble)
library(keras)
library(text2vec)
library(Matrix)
library(tibble)
library(tensorflow)
library(reticulate)
library(tidyverse)
library(randomForest)
library(caret)
library(shiny)

# install_miniconda()
use_miniconda("r-reticulate", required = TRUE)
# install_keras()
# install_tensorflow(version = "2.10.0", method = "conda", envname = "r-reticulate")

Data Exploration

There will be 3 datasets used in this project, which are movies, ratings and tags.
Import and explore the data from the files and combine them into a single tabular dataset.

Movies Dataset ( 27278 x 3 )
movieId	title	genres
1	Toy Story (1995)	Adventure\|Animation\|Children\|Comedy\|Fantasy
2	Jumanji (1995)	Adventure\|Children\|Fantasy
3	Grumpier Old Men (1995)	Comedy\|Romance
4	Waiting to Exhale (1995)	Comedy\|Drama\|Romance
5	Father of the Bride Part II (1995)	Comedy

Ratings Dataset ( 20000263 x 4 )
userId	movieId	rating	timestamp
1	2	3.5	1112486027
1	29	3.5	1112484676
1	32	3.5	1112484819
1	47	3.5	1112484727
1	50	3.5	1112484580

Tags Dataset ( 465564 x 4 )
userId	movieId	tag	timestamp
18	4141	Mark Waters	1240597180
65	208	dark hero	1368150078
65	353	dark hero	1368150079
65	521	noir thriller	1368149983
65	592	dark hero	1368150078

Data Preprocessing on Movies Dataset

Check is there any missing value in movies dataset
Separate years from the movies and assign them to a new column as well as convert into numeric
Apply one-hot encode for genres in movie dataset
Rename (no genres listed) to NoGenre

Missing rate of title column in movie dataset: 0%

New Movies Dataset ( 27278 x 23 )
movieId	title	year	Adventure	Animation	Children	Comedy	Fantasy	Romance	Drama
1	Toy Story	1995	1	1	1	1	1	0	0
2	Jumanji	1995	1	0	1	0	1	0	0
3	Grumpier Old Men	1995	0	0	0	1	0	1	0
4	Waiting to Exhale	1995	0	0	0	1	0	1	1
5	Father of the Bride Part II	1995	0	0	0	1	0	0	0

Data Preprocessing on Rating Dataset

Drop the timestamp column (Provide no much information)
Check is there any missing value in ratings dataset
Calculate the average rating and rating count based on each movie
Merge the data into movie dataset

Missing rate of rating column in ratings dataset: 0%

Movies Ratings Dataset ( 27278 x 25 )
movieId	title	year	Adventure	Animation	Children	Comedy	Fantasy	Romance	Drama	average_rating	count_rating
1	Toy Story	1995	1	1	1	1	1	0	0	3.921240	49695
2	Jumanji	1995	1	0	1	0	1	0	0	3.211977	22243
3	Grumpier Old Men	1995	0	0	0	1	0	1	0	3.151040	12735
4	Waiting to Exhale	1995	0	0	0	1	0	1	1	2.861393	2756
5	Father of the Bride Part II	1995	0	0	0	1	0	0	0	3.064592	12161

Data Preprocessing on Tags Dataset

Drop the timestamp column (Provide no much information)
Check is there any missing value in tags dataset
Combine all tags in a single string for respective movies
Merge the data into movie rating dataset

Missing rate of tag column in tags dataset: 16%
Since there are missing tag in the tag column, remove all empty data

Missing rate of tag column in tags dataset after removed: 0%

Movies Ratings Tags Dataset ( 27278 x 26 )
movieId	title	year	Adventure	Animation	Children	Comedy	Fantasy	Romance	Drama	average_rating	count_rating	all_tags
1	Toy Story	1995	1	1	1	1	1	0	0	3.921240	49695	Watched \| computer animation \| Disney animated feature \| Pixar animation \| TÃ©a Leoni does not star in this movie \| Pixar \| animation \| family \| Tom Hanks \| witty \| adventure \| animated \| clever \| comedy \| fantasy \| bright \| DARING RESCUES \| fanciful \| HEROIC MISSION \| humorous \| light \| rousing \| TOYS COME TO LIFE \| UNLIKELY FRIENDSHIPS \| warm \| time travel \| kids movie \| Disney \| Tim Allen \| action figure \| action figures \| Buzz Lightyear \| CG animation \| toy \| toys \| Woody \| villian hurts toys \| pixar \| disney \| children \| é˜®ä¸€é¸£ \| funny \| 3D \| Cartoon \| ya boy \| cgi \| rated-G \| lots of heart \| Animation \| want to see again \| imdb top 250 \| buddy movie \| the boys \| very good \| Best of Rotten Tomatoes: All Time \| John Lasseter \| USA \| classic \| avi \| buy \| fun \| Want \| CGI \| soothing \| almost favorite \| friendship \| family film \| cute \| story \| voice acting \| first cgi film \| Engaging \| 2009 reissue in Stereoscopic 3-D \| 55 movies every kid should see–Entertainment Weekly \| BD-Video \| CLV \| DVD-Video \| kids and family \| erlend’s DVDs \| innovative \| dolls \| National Film Registry \| Tumey’s To See Again \| Tumey’s VHS
2	Jumanji	1995	1	0	1	0	1	0	0	3.211977	22243	time travel \| adapted from:book \| board game \| childhood recaptured \| game \| herds of CGI animals \| scary \| see also:Zathura \| time \| fantasy \| Robin Williams \| Joe Johnston \| kid flick \| jungle \| animals \| Lebbat \| adventure \| Children \| Fantasy \| Dynamic CGI Action \| Kirsten Dunst \| For children \| childish \| children \| Filmed in BC \| bad cgi \| based on a book \| Chris Van Allsburg \| magic board game \| monkey \| Saturn Award (Best Special Effects) \| Saturn Award (Best Supporting Actress) \| CLV \| horrifying (but not Horror genre) \| not for kids
3	Grumpier Old Men	1995	0	0	0	1	0	1	0	3.151040	12735	old people that is actually funny \| sequel fever \| grun running \| moldy \| old \| comedinha de velhinhos engraÃƒÂ§ada \| Funniest Movies \| sequel \| Jack Lemmon \| Walter Matthau \| NO_FA_GANES \| Howard Deutch \| comedinha de velhinhos engraÃ§ada \| CLV \| comedy
4	Waiting to Exhale	1995	0	0	0	1	0	1	1	2.861393	2756	chick flick \| revenge \| characters \| CLV
5	Father of the Bride Part II	1995	0	0	0	1	0	0	0	3.064592	12161	Diane Keaton \| family \| sequel \| Steve Martin \| wedding \| sequel fever \| Fantasy \| childhood classics \| it thought it was funny but it wasn’t \| watched under duress \| worst movies ever \| steve martin \| Comedy \| Touching \| pregnancy \| remake \| CLV

Data PreProcessing on Full Dataset

Check the summary of the dataset
Check the missing values in the dataset and handle missing values if exists
Drop the movieId (Provide no much information)

Summary of full dataset:

##     movieId          title                year        Adventure      
##  Min.   :     1   Length:27278       Min.   :1891   Min.   :0.00000  
##  1st Qu.:  6931   Class :character   1st Qu.:1976   1st Qu.:0.00000  
##  Median : 68068   Mode  :character   Median :1998   Median :0.00000  
##  Mean   : 59855                      Mean   :1989   Mean   :0.08538  
##  3rd Qu.:100293                      3rd Qu.:2008   3rd Qu.:0.00000  
##  Max.   :131262                      Max.   :2015   Max.   :1.00000  
##                                      NA's   :22                      
##    Animation          Children           Comedy         Fantasy       
##  Min.   :0.00000   Min.   :0.00000   Min.   :0.000   Min.   :0.00000  
##  1st Qu.:0.00000   1st Qu.:0.00000   1st Qu.:0.000   1st Qu.:0.00000  
##  Median :0.00000   Median :0.00000   Median :0.000   Median :0.00000  
##  Mean   :0.03765   Mean   :0.04176   Mean   :0.307   Mean   :0.05176  
##  3rd Qu.:0.00000   3rd Qu.:0.00000   3rd Qu.:1.000   3rd Qu.:0.00000  
##  Max.   :1.00000   Max.   :1.00000   Max.   :1.000   Max.   :1.00000  
##                                                                       
##     Romance           Drama            Action          Crime       
##  Min.   :0.0000   Min.   :0.0000   Min.   :0.000   Min.   :0.0000  
##  1st Qu.:0.0000   1st Qu.:0.0000   1st Qu.:0.000   1st Qu.:0.0000  
##  Median :0.0000   Median :0.0000   Median :0.000   Median :0.0000  
##  Mean   :0.1513   Mean   :0.4892   Mean   :0.129   Mean   :0.1077  
##  3rd Qu.:0.0000   3rd Qu.:1.0000   3rd Qu.:0.000   3rd Qu.:0.0000  
##  Max.   :1.0000   Max.   :1.0000   Max.   :1.000   Max.   :1.0000  
##                                                                    
##     Thriller          Horror           Mystery           Sci-Fi      
##  Min.   :0.0000   Min.   :0.00000   Min.   :0.0000   Min.   :0.0000  
##  1st Qu.:0.0000   1st Qu.:0.00000   1st Qu.:0.0000   1st Qu.:0.0000  
##  Median :0.0000   Median :0.00000   Median :0.0000   Median :0.0000  
##  Mean   :0.1532   Mean   :0.09572   Mean   :0.0555   Mean   :0.0639  
##  3rd Qu.:0.0000   3rd Qu.:0.00000   3rd Qu.:0.0000   3rd Qu.:0.0000  
##  Max.   :1.0000   Max.   :1.00000   Max.   :1.0000   Max.   :1.0000  
##                                                                      
##       IMAX           Documentary           War             Musical       
##  Min.   :0.000000   Min.   :0.00000   Min.   :0.00000   Min.   :0.00000  
##  1st Qu.:0.000000   1st Qu.:0.00000   1st Qu.:0.00000   1st Qu.:0.00000  
##  Median :0.000000   Median :0.00000   Median :0.00000   Median :0.00000  
##  Mean   :0.007185   Mean   :0.09059   Mean   :0.04377   Mean   :0.03798  
##  3rd Qu.:0.000000   3rd Qu.:0.00000   3rd Qu.:0.00000   3rd Qu.:0.00000  
##  Max.   :1.000000   Max.   :1.00000   Max.   :1.00000   Max.   :1.00000  
##                                                                          
##     Western          Film-Noir         NoGenre         average_rating 
##  Min.   :0.00000   Min.   :0.0000   Min.   :0.000000   Min.   :0.500  
##  1st Qu.:0.00000   1st Qu.:0.0000   1st Qu.:0.000000   1st Qu.:2.800  
##  Median :0.00000   Median :0.0000   Median :0.000000   Median :3.235  
##  Mean   :0.02478   Mean   :0.0121   Mean   :0.009018   Mean   :3.133  
##  3rd Qu.:0.00000   3rd Qu.:0.0000   3rd Qu.:0.000000   3rd Qu.:3.565  
##  Max.   :1.00000   Max.   :1.0000   Max.   :1.000000   Max.   :5.000  
##                                                        NA's   :534    
##   count_rating       all_tags        
##  Min.   :    1.0   Length:27278      
##  1st Qu.:    3.0   Class :character  
##  Median :   18.0   Mode  :character  
##  Mean   :  747.8                     
##  3rd Qu.:  205.0                     
##  Max.   :67310.0                     
##  NA's   :534

Features that contain with missing values:

For tags missing value, an empty string will be used to replace it.
As for others column missing values, since it is not too much, thus the data will be truncated

==> Columns that contain with missing values after preprocessing: [There is no missing data]

The cleaned dataset will be saved as full_movies.csv.

EDA on Full Dataset

Plot the trend of movie counts by year
Plot the average rating per genre
Plot the genre distribution
Plot the word cloud for tags

Modeling

Classification Model for Genre Prediction

Build a neural network model to train for the multi-label genre classifier.

Result for classifier model

From the plot, the model for training dataset is perform with high accuracy while the testing dataset is not performing well.
This happened might due to overfitting on the trained model.

Predict a movie genre:
Title –> Alien Resurrection
Tags –> space | monster | thriller | future | horror
Year –> 2024
Count Rating –> 7000

## 1/1 - 0s - 219ms/epoch - 219ms/step

Predicted Genres (Prob > 0.5)
genre	probability
Horror	0.969
Sci-Fi	0.815
Thriller	0.724

From the result of the genre prediction, the genre classifier is successfully predict the genre with a given probability

Regression Model for Rating Prediction

Build a random forest regression model to predict on movie rating.

Regression Model Summary:

##                 Length Class      Mode     
## call                6  -none-     call     
## type                1  -none-     character
## predicted       21377  -none-     numeric  
## mse               100  -none-     numeric  
## rsq               100  -none-     numeric  
## oob.times       21377  -none-     numeric  
## importance         44  -none-     numeric  
## importanceSD       22  -none-     numeric  
## localImportance     0  -none-     NULL     
## proximity           0  -none-     NULL     
## ntree               1  -none-     numeric  
## mtry                1  -none-     numeric  
## forest             11  -none-     list     
## coefs               0  -none-     NULL     
## y               21377  -none-     numeric  
## test                0  -none-     NULL     
## inbag               0  -none-     NULL     
## xNames             22  -none-     character
## problemType         1  -none-     character
## tuneValue           1  data.frame list     
## obsLevels           1  -none-     logical  
## param               2  -none-     list

Regression Model Evaluation Metrics
Metric	Value
RMSE	0.6083
MAE	0.4319
R²	0.1607

From the metrics table, the model’s prediction deviate from the actual values by about 0.62 units

Deployment

Build an application for the movie genre and rating prediction

Shiny applications not supported in static R Markdown documents

Conclusion

In this project, three datasets are pre-processed and merge into a single tabular dataset.
This clean dataset will then be evaluated to gain any potential sight in the EDA.
Besides, the model that trained in this project is successful predict on a movie genre and rating.
Finally, an apps is developed by shiny package for the movie genre and rating prediction.

Predictive Model for Movie Genre and Rating

Callum Joseph Smith, Gan Wei Guang, Kirthiga Manimaran, Yang HongBin, Zhang YiNan

2025-06-08