A. What is Exploratory Data Analysis (EDA) ?

Imagine if you’re in the cinema, but you don’t know what movie you’re going to watch. Surely you will experience an inner conflict within yourself, “what movie that is worth to watch?”; and maybe lead to some further questions such: which movie has the good plot, are the casts and crews famous, how the ratings and reviews proceed, etc. Hence, in order to answer those questions, you can explore the information from internet, or you can do a quick survey and ask some of your friends. Whatever investigating process you would take to gain those questions’ answers, data scientists often call it as ‘Exploratory Data Analysis’.

Notes: It is always a good practice to understand the data first and try to gather as many insights from it.

So, without further ado, let’s jump into the analysis! 😉

B. About the Dataset

In this opportunity, we use IMDb User’s Rating Dataset which contain information of 1272 Indonesian movies. The data were gathered using IMDb-Scraper and then were converted and cleaned into a .csv file.

# Input the Data and A Glimpse of Them
movie <- read.csv("data_input/indonesian_movies.csv", header=T, na.strings=c(""))
head(movie,3) %>% 
  kable %>%
  kable_paper() %>%
  row_spec(0, background = "#FF8181") %>% 
  kable_styling(bootstrap_options = c("striped", "hover", "condensed", "responsive", full_width = F, position = "center"))

C. Data Preprocessing

Data preprocessing is an essential step to do before we proceed to the analysis. Real-world raw data usually has inconsistent formatting, human errors, and can also be incomplete. Data preprocessing resolves such issues and makes datasets more complete and efficient to perform data analysis. It’s a crucial process that can affect the success of data mining and machine learning projects.

In this project, we also do data preprocessing as follows..

# Checking the Data Types
str(movie)

## 'data.frame':    1272 obs. of  11 variables:
##  $ title       : chr  "#FriendButMarried 2" "4 Mantan" "Aku Tahu Kapan Kamu Mati" "Anak Garuda" ...
##  $ year        : int  2020 2020 2020 2020 2020 2020 2020 2020 2020 2020 ...
##  $ description : chr  "Ayudia (Mawar De Jongh) is not satisfied enough in enjoying the moment as a wife, but she suddenly becomes preg"| __truncated__ "Sara, Airin, Rachel, and Amara were accidentally brought together after the death of their ex, Alex. All four r"| __truncated__ "After apparent death, Siena is able to see signs that people will die. However, her friends did not believe in "| __truncated__ "Good Morning Indonesia, a school for poor orphans, invites seven children to become a team. But, jealousy and e"| __truncated__ ...
##  $ genre       : chr  "Biography" "Thriller" "Horror" "Adventure" ...
##  $ rating      : chr  "13+" "17+" "13+" "13+" ...
##  $ users_rating: num  6.5 6.4 5.4 9.1 7.6 6.1 6.2 6.1 5.9 8.5 ...
##  $ votes       : chr  "120" "8" "17" "27" ...
##  $ languages   : chr  "Indonesian" "Indonesian" "Indonesian" "Indonesian" ...
##  $ directors   : chr  "Rako Prijanto" "Hanny Saputra" "Hadrah Daeng Ratu" "Faozan Rizal" ...
##  $ actors      : chr  "['Adipati Dolken', 'Mawar Eva de Jongh', 'Vonny Cornellya', 'Sari Nila', 'Ivan Leonardy', 'Clay Gribble', 'Sara"| __truncated__ "['Ranty Maria', 'Jeff Smith', 'Melanie Berentz', 'Denira Wiraguna', 'Melayu Nicole', 'Cinta Brian', 'Delano Dan"| __truncated__ "['Natasha Wilona', 'Ria Ricis', 'Al Ghazali', 'Fitria Rasyidi', 'Ryma Karimah', 'Sonia Alexa', 'Asri Welas', 'C"| __truncated__ "['Tissa Biani Azzahra', 'Violla Georgie', 'Ajil Ditto', 'Clairine Clay', 'Geraldy Krechoff', 'Rania Putri Sari'"| __truncated__ ...
##  $ runtime     : chr  "100 min" "80 min" "92 min" "129 min" ...

There seems to be some data types that don’t match, such as:

• votes and runtime should be numeric
• year, genre, rating, languagesshould be factor

In order for our analysis to run smoothly, we have to change their types. However, there are several things that make it difficult for us to make those changes. Some of these things are: the use of , for the thousands, the use of the word min at runtime, some inefficient (repeated) rating categories, etc. We must fix these first before we proceed into the next step.

# Replacing Characters/Words/Categories that Don't Fit
movie$votes <- str_replace_all(movie$votes,",","")
movie$runtime <- str_replace_all(movie$runtime,"min","")
movie$rating <- str_replace_all(movie$rating,"Not Rated","Unrated")
movie$rating <- str_replace_all(movie$rating,"Not Rated","Unrated")
movie$rating <- movie$rating %>% replace_na('Unrated')
movie$rating <- str_replace_all(movie$rating,"TV-14","13+")
movie$rating <- str_replace_all(movie$rating,"PG-13","13+")
movie$rating <- str_replace_all(movie$rating,"R","13+")
movie$rating <- str_replace_all(movie$rating,"TV-MA","17+")
movie$rating <- str_replace_all(movie$rating,"D","17+")

Besides, there are also several missing values that we should fill in as long as we can.

# Checking the Number of Missing Values for Each Column
colSums(is.na(movie))

##        title         year  description        genre       rating users_rating 
##            0            0          432           36            0            0 
##        votes    languages    directors       actors      runtime 
##            0            0            7            0          403

p.s: To fix this case, I browsed on internet so that we get the information which were missing.

# Filling the Movie Genre
movie[762,"genre"] <- "Comedy"
movie[825,"genre"] <- "Comedy"
movie[827,"genre"] <- "Drama"
movie[857,"genre"] <- "Drama"
movie[900,"genre"] <- "Drama"
movie[939,"genre"] <- "Adventure"
movie[956,"genre"] <- "Drama"
movie[963,"genre"] <- "Romance"
movie[965,"genre"] <- "Drama"
movie[969,"genre"] <- "Romance"
movie[1009,"genre"] <- "Drama"
movie[1015,"genre"] <- "Drama"
movie[1041,"genre"] <- "Comedy"
movie[1048,"genre"] <- "Comedy"
movie[1071,"genre"] <- "Action"
movie[1076,"genre"] <- "Comedy"
movie[1085,"genre"] <- "Action"
movie[1092,"genre"] <- "Romance"
movie[1100,"genre"] <- "Drama"
movie[1122,"genre"] <- "Romance"
movie[1132,"genre"] <- "Drama"
movie[1137,"genre"] <- "Comedy"
movie[1139,"genre"] <- "Drama"
movie[1147,"genre"] <- "Drama"
movie[1153,"genre"] <- "Comedy"
movie[1156,"genre"] <- "Drama"
movie[1159,"genre"] <- "Action"
movie[1178,"genre"] <- "History"
movie[1182,"genre"] <- "Adventure"
movie[1196,"genre"] <- "Action"
movie[1204,"genre"] <- "History"
movie[1211,"genre"] <- "Comedy"
movie[1224,"genre"] <- "Drama"
movie[1231,"genre"] <- "Romance"
movie[1240,"genre"] <- "Action"
movie[1259,"genre"] <- "History"

# Filling the Movie Director
movie[138,"directors"] <- "Amar Mukhi"
movie[223,"directors"] <- "Tema Patrosza"
movie[427,"directors"] <- "Indra Gunawan"
movie[1023,"directors"] <- "Steady Rimba"
movie[1047,"directors"] <- "Steady Rimba"
movie[1256,"directors"] <- "A.N. Alcaff"
movie[1271,"directors"] <- "Nancing movie Corp."

# Filling the Movie Rating
movie[movie$title == "Iseng","rating"] <- "21+"
movie[movie$title == "Wanita Dalam Gairah","rating"] <- "21+"
movie[movie$title == "Bibir Mer","rating"] <- "21+"

But, there is another problem. If we look closely, we could find some movies with the same titles in our dataset.

# Checking the Duplicates
movie[duplicated(movie$title),] %>% 
  kable %>%
  kable_paper() %>%
  row_spec(0, background = "#FF8181") %>% 
  kable_styling(bootstrap_options = c("striped", "hover", "condensed", "responsive", full_width = F, position = "center"))

However, after I double-checked, this case can be ignored as each title does have two movies with different details (it has the older version and the remake version). So, don’t mind it! 😁

Everything is set! Hence, we can change our data types as follows.

# Changing the Data Types
movie$votes <- as.numeric(movie$votes)
movie$runtime <- as.numeric(movie$runtime)
movie[,c("year","genre","rating","languages")] <- lapply(movie[,c("year","genre","rating","languages")],as.factor)

Ups, don’t be too happy yet 😏

It seems that we also need 2 new columns containing:

• id; to differentiate between movies
• users_rating/(votes/sum(votes)); because it doesn’t seem wise to compare the user ratings between movies that get a lot of votes with those that don’t get a lot of votes, as a movie with more votes usually has more variation in the viewer’s rating, hence, in the final, its rating average is smaller (despite it is selling well and becomes the center of attention)

See below for more details!

# Feature Engineering
movie$ratingp <- movie$users_rating * (movie$votes/sum(movie$votes))
movie$id <- 1:nrow(movie)

Don’t forget to drop the unused levels in the dataset.

# Dropping Unused Levels
movie$rating <- droplevels(movie$rating)

Check again for the last time! 😃

# Checking the Data Types (again)
str(movie)

## 'data.frame':    1272 obs. of  13 variables:
##  $ title       : chr  "#FriendButMarried 2" "4 Mantan" "Aku Tahu Kapan Kamu Mati" "Anak Garuda" ...
##  $ year        : Factor w/ 62 levels "1926","1928",..: 62 62 62 62 62 62 62 62 62 62 ...
##  $ description : chr  "Ayudia (Mawar De Jongh) is not satisfied enough in enjoying the moment as a wife, but she suddenly becomes preg"| __truncated__ "Sara, Airin, Rachel, and Amara were accidentally brought together after the death of their ex, Alex. All four r"| __truncated__ "After apparent death, Siena is able to see signs that people will die. However, her friends did not believe in "| __truncated__ "Good Morning Indonesia, a school for poor orphans, invites seven children to become a team. But, jealousy and e"| __truncated__ ...
##  $ genre       : Factor w/ 15 levels "Action","Adventure",..: 4 14 11 2 7 5 4 11 11 7 ...
##  $ rating      : Factor w/ 5 levels "13+","17+","21+",..: 1 2 1 1 2 5 1 2 2 1 ...
##  $ users_rating: num  6.5 6.4 5.4 9.1 7.6 6.1 6.2 6.1 5.9 8.5 ...
##  $ votes       : num  120 8 17 27 33 153 21 7 67 54 ...
##  $ languages   : Factor w/ 8 levels "Dutch","English",..: 4 4 4 4 4 4 2 4 4 4 ...
##  $ directors   : chr  "Rako Prijanto" "Hanny Saputra" "Hadrah Daeng Ratu" "Faozan Rizal" ...
##  $ actors      : chr  "['Adipati Dolken', 'Mawar Eva de Jongh', 'Vonny Cornellya', 'Sari Nila', 'Ivan Leonardy', 'Clay Gribble', 'Sara"| __truncated__ "['Ranty Maria', 'Jeff Smith', 'Melanie Berentz', 'Denira Wiraguna', 'Melayu Nicole', 'Cinta Brian', 'Delano Dan"| __truncated__ "['Natasha Wilona', 'Ria Ricis', 'Al Ghazali', 'Fitria Rasyidi', 'Ryma Karimah', 'Sonia Alexa', 'Asri Welas', 'C"| __truncated__ "['Tissa Biani Azzahra', 'Violla Georgie', 'Ajil Ditto', 'Clairine Clay', 'Geraldy Krechoff', 'Rania Putri Sari'"| __truncated__ ...
##  $ runtime     : num  100 80 92 129 109 101 90 85 109 117 ...
##  $ ratingp     : num  1.33e-03 8.76e-05 1.57e-04 4.20e-04 4.29e-04 ...
##  $ id          : int  1 2 3 4 5 6 7 8 9 10 ...

Finally, it looks like we’re good to go, let’s proceed to EDA 😍

D. Exploratory Data Analysis

First of all, we must clearly know our data size.

# Checking the Dimension
dim(movie)

## [1] 1272   13

It looks like our intuition was right. We have 1272 observations (1272 movies), with 13 variables related to each movie (11 of them are from the original dataset, while the 2 of them are the variables that we were just made).

Next, let’s take a look into the summary of each variable.

summary(movie)

##     title                year     description              genre    
##  Length:1272        2019   :111   Length:1272        Drama    :469  
##  Class :character   2018   : 97   Class :character   Comedy   :295  
##  Mode  :character   2009   : 79   Mode  :character   Horror   :231  
##                     2011   : 78                      Action   :137  
##                     2008   : 77                      Adventure: 51  
##                     2010   : 76                      Biography: 28  
##                     (Other):754                      (Other)  : 61  
##      rating     users_rating       votes                languages   
##  13+    :188   Min.   :1.200   Min.   :     5.0   Indonesian :1241  
##  17+    :119   1st Qu.:5.300   1st Qu.:    12.0   English    :  21  
##  21+    : 11   Median :6.400   Median :    27.0   Japanese   :   3  
##  SU     : 39   Mean   :6.144   Mean   :   459.4   Dutch      :   2  
##  Unrated:915   3rd Qu.:7.100   3rd Qu.:    76.0   Minangkabau:   2  
##                Max.   :9.400   Max.   :187222.0   French     :   1  
##                                                   (Other)    :   2  
##   directors            actors             runtime          ratingp         
##  Length:1272        Length:1272        Min.   : 45.00   Min.   :0.0000205  
##  Class :character   Class :character   1st Qu.: 88.00   1st Qu.:0.0001191  
##  Mode  :character   Mode  :character   Median : 95.00   Median :0.0002699  
##                                        Mean   : 97.07   Mean   :0.0055506  
##                                        3rd Qu.:105.00   3rd Qu.:0.0008588  
##                                        Max.   :262.00   Max.   :2.4348164  
##                                        NA's   :403                         
##        id        
##  Min.   :   1.0  
##  1st Qu.: 318.8  
##  Median : 636.5  
##  Mean   : 636.5  
##  3rd Qu.: 954.2  
##  Max.   :1272.0  
## 

📋 From the summary above, we can take some points, such as:

1. Of all Indonesian movies listed on IMDb, most movies are:
   1. Released in 2019
   2. Having drama genre
   3. Not rated yet
   4. Using Bahasa Indonesia
2. The IMDb user’s rating for Indonesian movies ranges between 1.2 and 9.4, with mean = 6.144 and median = 6.400. It seems that there are no extreme values as the mean and the median are nearly the same.
3. The runtime of Indonesian movies ranges between 45 minutes and 262 minutes, with mean = 97.07 minutes and median = 95 minutes. Although there are some missing values in the runtime data.
4. The number of votes of Indonesian movies in IMDb ranges between 5 and 187222, with mean = 459.4 and median = 27. The large difference between median and mean indicates that only a few movies have a large number of votes, while generally the number of votes is small.

You’re allowed to add other insights, as long as they correspond with the summary above 👌

Now, we are going to dismantle our data so that we get some new-amazing facts. Let’s go! 🔥

Top 3 Genres of Indonesian Movies in IMDb Dataset

# Looking for the Top 3 Genres
genre_most <- as.data.frame(head(sort(table(movie$genre), decreasing = T),3))
colnames(genre_most)[1] <- 'Genre'
genre_most %>% 
  kable(table.attr = "style='width:40%;'") %>%
  kable_paper() %>%
  row_spec(0, background = "#FF8181") %>% 
  kable_styling(bootstrap_options = c("striped", "hover", "condensed", "responsive", full_width = T, position = "center"))

Genre	Freq
Drama	469
Comedy	295
Horror	231

# Showing A Proportion Table
genre_mostprop <- as.data.frame(head(sort(prop.table(table(movie$genre)), decreasing = T),3))
colnames(genre_mostprop)[1] <- 'Genre'
genre_mostprop %>% 
  kable(table.attr = "style='width:40%;'") %>%
  kable_paper() %>%
  row_spec(0, background = "#FF8181") %>% 
  kable_styling(bootstrap_options = c("striped", "hover", "condensed", "responsive", full_width = T, position = "center"))

Genre	Freq
Drama	0.3687107
Comedy	0.2319182
Horror	0.1816038

The top 3 genres in the dataset are Drama, Comedy, and Horror. This shows that mostly Indonesian film creators often produce these 3 genres as they sell really well in the market. The 3 genres have shown consistent growth in production throughout Indonesian filming history. This result also shows some consistency with the data shown by Managing Partner Ideosource Film Fund (IFF) about the 3 most popular Indonesian film genres from 2008-2018.

❗There will be no offer if there is no demand

The fact that most Indonesian TV shows are filled with soap operas and comedy shows (where their ratings are also high), make it clear that there really is a lot of demand in these particular genres. Many moms like to watch soap operas because they can relate to the plot as it describes their daily life; providing emotional release, personal gratification, companionship, and an escape from reality. Meanwhile, in terms of horror movies, indeed Indonesia is famous for its mystique, so it is not very surprising that this genre is famous.

A Mom is Busy Watching A Soap Opera

top2 <- as.data.frame(head(movie[order(movie$ratingp, decreasing = T),'title'],2))
colnames(top2)[1] <- 'Title'
top2 %>% 
  kable(table.attr = "style='width:40%;'") %>%
  kable_paper() %>%
  row_spec(0, background = "#FF8181") %>% 
  kable_styling(bootstrap_options = c("striped", "hover", "condensed", "responsive", full_width = T, position = "center"))

top3 <- as.data.frame(head(movie[order(movie$ratingp, decreasing = T),'title'],3))
colnames(top3)[1] <- 'Title'
top3 %>% 
  kable(table.attr = "style='width:40%;'") %>%
  kable_paper() %>%
  row_spec(0, background = "#FF8181") %>% 
  kable_styling(bootstrap_options = c("striped", "hover", "condensed", "responsive", full_width = T, position = "center"))

top5 <- as.data.frame(head(movie[order(movie$ratingp, decreasing = T),'title'],5))
colnames(top5)[1] <- 'Title'
top5 %>% 
  kable(table.attr = "style='width:40%;'") %>%
  kable_paper() %>%
  row_spec(0, background = "#FF8181") %>% 
  kable_styling(bootstrap_options = c("striped", "hover", "condensed", "responsive", full_width = T, position = "center"))

top10 <- as.data.frame(head(movie[order(movie$ratingp, decreasing = T),'title'],10))
colnames(top10)[1] <- 'Title'
top10 %>% 
  kable(table.attr = "style='width:40%;'") %>%
  kable_paper() %>%
  row_spec(0, background = "#FF8181") %>% 
  kable_styling(bootstrap_options = c("striped", "hover", "condensed", "responsive", full_width = T, position = "center"))

# Removing Some Characters
movie$actors <- str_replace_all(movie$actors,"\\[","")
movie$actors <- str_replace_all(movie$actors,"\\]","")

# Making A Data Frame which Consists of the Actor Name and Acting Frequency
dt <- data.table(movie)
movie_actor <- dt[,strsplit(actors,", ",fixed = T),by = c("id","ratingp")]
freqactor <- (as.data.frame(sort(table(movie_actor$V1), decreasing = T)))[-1,]

# Making A Data Frame which Consists of Senior Actors
topfreqactor <- freqactor[freqactor$Freq >= 20,]
colnames(topfreqactor)[1] <- "actor_name"

# Calculating the Median of User's Rating
actor_rating <- aggregate(ratingp ~ V1, data = movie_actor, FUN = median)
colnames(actor_rating)[1] <- "actor_name"

# Merging the Median Values with the Table of Senior Actors
topratingactor <- merge(x=topfreqactor,y=actor_rating,by="actor_name",all.x=TRUE)

head(topratingactor[order(topratingactor$ratingp, decreasing = T),],3) %>% 
  kable(table.attr = "style='width:80%;'", row.names = FALSE) %>%
  kable_paper() %>%
  row_spec(0, background = "#FF8181") %>% 
  kable_styling(bootstrap_options = c("striped", "hover", "condensed", "responsive", full_width = T, position = "center"))

head(topratingactor[order(topratingactor$ratingp, decreasing = T),],5) %>% 
  kable(table.attr = "style='width:80%;'", row.names = FALSE) %>%
  kable_paper() %>%
  row_spec(0, background = "#FF8181") %>% 
  kable_styling(bootstrap_options = c("striped", "hover", "condensed", "responsive", full_width = T, position = "center"))

# Making A Data Frame which Consists of the Director Name and the Number of Movies that They Produced
freqdirector <- as.data.frame(sort(table(movie$directors), decreasing = T))

# Making A Data Frame which Consists of Senior Directors
topfreqdirector <- freqdirector[freqdirector$Freq >= 5,]
colnames(topfreqdirector)[1] <- "directors"

# Calculating the Median of User's Rating
director_rating <- aggregate(ratingp ~ directors, data = movie, FUN = median)

# Merging the Median Values with the Table of Senior Directors
topratingdirector <- merge(x=topfreqdirector,y=director_rating,by="directors",all.x=TRUE)

head(topratingdirector[order(topratingdirector$ratingp,decreasing = T),],3) %>% 
  kable(table.attr = "style='width:80%;'", row.names = FALSE) %>%
  kable_paper() %>%
  row_spec(0, background = "#FF8181") %>% 
  kable_styling(bootstrap_options = c("striped", "hover", "condensed", "responsive", full_width = T, position = "center"))

head(topratingdirector[order(topratingdirector$ratingp,decreasing = T),],5) %>% 
  kable(table.attr = "style='width:80%;'", row.names = FALSE) %>%
  kable_paper() %>%
  row_spec(0, background = "#FF8181") %>% 
  kable_styling(bootstrap_options = c("striped", "hover", "condensed", "responsive", full_width = T, position = "center"))

E. Download Dataset (Cleaned)