Scraping the Description

#Using CSS selectors to scrape the description section
description_data_html <- html_nodes(webpage,'.ratings-bar+ .text-muted')

#Converting the description data to text
description_data <- html_text(description_data_html)

#Let's have a look at the description data
head(description_data)

## [1] "\n    A secret government agency recruits some of the most dangerous incarcerated super-villains to form a defensive task force. Their first mission: save the world from the apocalypse."        
## [2] "\n    Political involvement in the Avengers' affairs causes a rift between Captain America and Iron Man."                                                                                         
## [3] "\n    A wisecracking mercenary gets experimented on and becomes immortal but ugly, and sets out to track down the man who ruined his looks."                                                      
## [4] "\n    An aspiring model, Jesse, is new to Los Angeles. However, her beauty and youth, which generate intense fascination and jealousy within the fashion industry, may prove themselves sinister."
## [5] "\n    Seven gunmen from a variety of backgrounds are brought together by a vengeful young widow to protect her town from the private army of a destructive industrialist."                        
## [6] "\n    Three girls are kidnapped by a man with a diagnosed 23 distinct personalities. They must try to escape before the apparent emergence of a frightful new 24th."

For the description_data, we will clean it by removing the before each of the strings.

#Data-Preprocessing: removing '\n'
description_data<-gsub("\n","",description_data)

#Let's have another look at the description data 
head(description_data)

## [1] "    A secret government agency recruits some of the most dangerous incarcerated super-villains to form a defensive task force. Their first mission: save the world from the apocalypse."        
## [2] "    Political involvement in the Avengers' affairs causes a rift between Captain America and Iron Man."                                                                                         
## [3] "    A wisecracking mercenary gets experimented on and becomes immortal but ugly, and sets out to track down the man who ruined his looks."                                                      
## [4] "    An aspiring model, Jesse, is new to Los Angeles. However, her beauty and youth, which generate intense fascination and jealousy within the fashion industry, may prove themselves sinister."
## [5] "    Seven gunmen from a variety of backgrounds are brought together by a vengeful young widow to protect her town from the private army of a destructive industrialist."                        
## [6] "    Three girls are kidnapped by a man with a diagnosed 23 distinct personalities. They must try to escape before the apparent emergence of a frightful new 24th."

Scraping the Runtime

#Using CSS selectors to scrape the Movie runtime section
runtime_data_html <- html_nodes(webpage,'.text-muted .runtime')

#Converting the runtime data to text
runtime_data <- html_text(runtime_data_html)

#Let's have a look at the runtime
head(runtime_data)

## [1] "123 min" "147 min" "108 min" "117 min" "132 min" "117 min"

For the runtime_data, we will clean it by removing the mins string after each value, and by connverting it into a numeric data type.

#Data-Preprocessing: removing mins and converting it to numerical
runtime_data<-gsub(" min","",runtime_data)
runtime_data<-as.numeric(runtime_data)

#Let's have another look at the runtime data
head(runtime_data)

## [1] 123 147 108 117 132 117

Scraping the Genre

#Using CSS selectors to scrape the Movie genre section
genre_data_html <- html_nodes(webpage,'.genre')

#Converting the genre data to text
genre_data <- html_text(genre_data_html)

#Let's have a look at the runtime
head(genre_data)

## [1] "\nAction, Adventure, Fantasy            "
## [2] "\nAction, Adventure, Sci-Fi            " 
## [3] "\nAction, Adventure, Comedy            " 
## [4] "\nHorror, Thriller            "          
## [5] "\nAction, Adventure, Western            "
## [6] "\nHorror, Thriller            "

For the genre_data, we will clean it by removing the string, removing excess space, taking only the first genre of each movie before each value, and by converting each genre from text to a factor.

#Data-Preprocessing: removing \n
genre_data<-gsub("\n","",genre_data)

#Data-Preprocessing: removing excess spaces
genre_data<-gsub(" ","",genre_data)

#taking only the first genre of each movie
genre_data<-gsub(",.*","",genre_data)

#Converting each genre from text to factor
genre_data<-as.factor(genre_data)

#Let's have another look at the genre data
head(genre_data)

## [1] Action Action Action Horror Action Horror
## 9 Levels: Action Adventure Animation Biography Comedy Crime ... Thriller

Scraping the IMDB Rating

#Using CSS selectors to scrape the IMDB rating section
rating_data_html <- html_nodes(webpage,'.ratings-imdb-rating strong')

#Converting the ratings data to text
rating_data <- html_text(rating_data_html)

#Let's have a look at the ratings
head(rating_data)

## [1] "6.0" "7.8" "8.0" "6.2" "6.9" "7.3"

For the rating_data, we will clean it by converting the values to numeric data types.

#Data-Preprocessing: converting ratings to numerical
rating_data<-as.numeric(rating_data)

#Let's have another look at the ratings data
head(rating_data)

## [1] 6.0 7.8 8.0 6.2 6.9 7.3

Scraping the Votes Section

#Using CSS selectors to scrape the votes section
votes_data_html <- html_nodes(webpage,'.sort-num_votes-visible span:nth-child(2)')

#Converting the votes data to text
votes_data <- html_text(votes_data_html)

#Let's have a look at the votes data
head(votes_data)

## [1] "536,663" "590,251" "827,314" "73,355"  "169,536" "364,104"

For the votes_data, we will clean it by removing the commas and converting votes to a numerical data type.

#Data-Preprocessing: removing commas
votes_data<-gsub(",","",votes_data)

#Data-Preprocessing: converting votes to numerical
votes_data<-as.numeric(votes_data)

#Let's have another look at the votes data
head(votes_data)

## [1] 536663 590251 827314  73355 169536 364104

Scraping the Directors Section

#Using CSS selectors to scrape the directors section
directors_data_html <- html_nodes(webpage,'.text-muted+ p a:nth-child(1)')

#Converting the directors data to text
directors_data <- html_text(directors_data_html)

#Let's have a look at the directors data
head(directors_data)

## [1] "David Ayer"           "Anthony Russo"        "Tim Miller"          
## [4] "Nicolas Winding Refn" "Antoine Fuqua"        "M. Night Shyamalan"

For the directors_data, we will clean it by converting it to a factor data type.

#Data-Preprocessing: converting directors data into factors
directors_data<-as.factor(directors_data)

Scraping the Actors Section

#Using CSS selectors to scrape the actors section
actors_data_html <- html_nodes(webpage,'.lister-item-content .ghost+ a')

#Converting the gross actors data to text
actors_data <- html_text(actors_data_html)

#Let's have a look at the actors data
head(actors_data)

## [1] "Will Smith"        "Chris Evans"       "Ryan Reynolds"    
## [4] "Elle Fanning"      "Denzel Washington" "James McAvoy"

For the actors_data, we will clean it by converting it to a factor data type.

#Data-Preprocessing: converting actors data into factors
actors_data<-as.factor(actors_data)

Scraping the Metascore Section

#Using CSS selectors to scrape the metascore section
metascore_data_html <- html_nodes(webpage,'.metascore')

#Converting the runtime data to text
metascore_data <- html_text(metascore_data_html)

#Let's have a look at the metascore 
head(metascore_data)

## [1] "40        " "75        " "65        " "51        " "54        "
## [6] "62        "

For the metascore_data, we will clean it by removing the extra space in metascore and by checking the length of the the metascore data.

#Data-Preprocessing: removing extra space in metascore
metascore_data<-gsub(" ","",metascore_data)

#Lets check the length of metascore data
length(metascore_data)

## [1] 97

#Using CSS selectors to scrape the gross revenue section
gross_data_html <- html_nodes(webpage,'.ghost~ .text-muted+ span')

#Converting the gross revenue data to text
gross_data <- html_text(gross_data_html)

#Let's have a look at the votes data
head(gross_data)

## [1] "$325.10M" "$408.08M" "$363.07M" "$1.33M"   "$93.43M"  "$138.29M"

#Data-Preprocessing: removing '$' and 'M' signs
gross_data<-gsub("M","",gross_data)

gross_data<-substring(gross_data,2,6)

#Let's check the length of gross data
length(gross_data)

## [1] 92

Inserting the NAs for missing Gross data

The length of 92 indicates that 8 films are missing gross data. By manually searching the page, we see that The Invisible Ghost (#38), The Autospy of Jane Doe (#47), The Do-Over (#63), Below Her Mouth (#65), Hush (#69), Terrifier (#71), Mean Dreams (#87), and Brimstone (#94) are the titles with missing gross data. We can insert an NA into these rows so that we can still obtain statistic summary data. We will also convert the data into a numeric type so we can observe the summary statistics.

#Filling missing entries with NA
for (i in c(38,47,63,65,69,71,87,94)){

a<-gross_data[1:(i-1)]
b<-gross_data[i:length(gross_data)]
gross_data<-append(a,list("NA"))
gross_data<-append(gross_data,b)

}

#Data-Preprocessing: converting gross to numerical
gross_data<-as.numeric(gross_data)

## Warning: NAs introduced by coercion

## Warning: NAs introduced by coercion

## Warning: NAs introduced by coercion

## Warning: NAs introduced by coercion

## Warning: NAs introduced by coercion

## Warning: NAs introduced by coercion

## Warning: NAs introduced by coercion

## Warning: NAs introduced by coercion

#Let's have another look at the length of gross data
length(gross_data)

## [1] 100

summary(gross_data)

##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max.    NA's 
##    0.02   25.52   55.20   97.14  125.40  532.10       8

Histogram for Runtime of Films by Genre

#Initial Histogram
library('ggplot2')
qplot(data = movies_df,Runtime,fill = Genre,bins = 30)

#Histogram with alternative x-axis intervals and labels
qplot(data = movies_df,Runtime,fill = Genre,bins = 30) +
scale_x_continuous(breaks = seq(0, 200, by = 25)) +
  labs(title = "Number of Films vs Runtime by Genre", 
       x = "Runtime (mins)", y = "Number of Films")

Question 1: Based on the above data, which movie from which Genre had the longest runtime? Answer: The graph shows that the genre with the highest runtime is “Adventure”, but to find out the specific film we need to arrange the data by runtime from longest to shortest.

#arranging the data by runtime
library("dplyr")

## Warning: package 'dplyr' was built under R version 3.6.1

## 
## Attaching package: 'dplyr'

## The following objects are masked from 'package:stats':
## 
##     filter, lag

## The following objects are masked from 'package:base':
## 
##     intersect, setdiff, setequal, union

head(arrange(movies_df, desc(runtime_data)))

##   Rank                              Title
## 1   58                     American Honey
## 2   61                             Dangal
## 3   64                            Silence
## 4   11 Batman v Superman: Dawn of Justice
## 5   94                          Brimstone
## 6    2         Captain America: Civil War
##                                                                                                                                                                                                              Description
## 1     A teenage girl with nothing to lose joins a traveling magazine sales crew, and gets caught up in a whirlwind of hard partying, law bending and young love as she criss-crosses the Midwest with a band of misfits.
## 2                                                               Former wrestler Mahavir Singh Phogat and his two wrestler daughters struggle towards glory at the Commonwealth Games in the face of societal oppression.
## 3                                      In the 17th century, two Portuguese Jesuit priests travel to Japan in an attempt to locate their mentor, who is rumored to have committed apostasy, and to propagate Catholicism.
## 4                                                          Fearing that the actions of Superman are left unchecked, Batman takes on the Man of Steel, while the world wrestles with what kind of a hero it really needs.
## 5                                                                                                                  From the moment the new reverend climbs the pulpit, Liz knows she and her family are in great danger.
## 6                                                                                                                     Political involvement in the Avengers' affairs causes a rift between Captain America and Iron Man.
##   Runtime     Genre Rating Metascore  Votes Gross_Earning_in_Mil
## 1     163 Adventure    7.0        79  32093                 0.66
## 2     161    Action    8.4        NA 125654                12.39
## 3     161 Adventure    7.2        79  85965                 7.08
## 4     151    Action    6.5        44 580049               330.30
## 5     148     Drama    7.1        45  30832                   NA
## 6     147    Action    7.8        75 590251               408.00
##           Director           Actor
## 1    Andrea Arnold      Sasha Lane
## 2    Nitesh Tiwari      Aamir Khan
## 3  Martin Scorsese Andrew Garfield
## 4      Zack Snyder     Ben Affleck
## 5 Martin Koolhoven      Guy Pearce
## 6    Anthony Russo     Chris Evans

This shows that the movie American Honey from the genre “Adventure” has the longest runtime.

Scatterplot for Runtime by Vote Ratings and Genre

##Initial Scatterplot
ggplot(movies_df,aes(x=Runtime,y=Rating))+
geom_point(aes(size=Votes,col=Genre))

#Scatterplot with alternative x-axis intervals and labels
ggplot(movies_df,aes(x=Runtime,y=Rating))+
geom_point(aes(size=Votes,col=Genre))+
scale_x_continuous(breaks = seq(0, 200, by = 25)) +
  labs(title = "Vote Rating vs Runtime by Genre and Number of Votes", 
       x = "Runtime (mins)", y = "Vote Rating")

Question 2: Based on the above data, in the Runtime of 130-160 mins, which genre has the highest votes? Answer: In the runtime of 130-160 mins, the genre “Action” has the highest votes. You can see this by observing that action has the largest circles, which represent number of votes.

Scatterplot for Runtime by Vote Ratings, Genre, and Gross Earnings in Millions

#Initial Scatterplot
ggplot(movies_df,aes(x=Runtime,y=Gross_Earning_in_Mil))+
geom_point(aes(size=Rating,col=Genre))

## Warning: Removed 8 rows containing missing values (geom_point).

#Scatterplot with alternative x-axis intervals and labels
ggplot(movies_df,aes(x=Runtime,y=Gross_Earning_in_Mil))+
geom_point(aes(size=Rating,col=Genre))+
scale_x_continuous(breaks = seq(0, 200, by = 20)) +
  labs(title = "Gross Earnings vs Runtime by Genre and Rating", 
       x = "Runtime (mins)", y = "Gross Earnings (Millions)")

## Warning: Removed 8 rows containing missing values (geom_point).

Question 3: Based on the above data, across all genres which genre has the highest average gross earnings in runtime 100 to 120. Answer: The “Adventure” genre has the highest average gross earnings in runtime 100 to 120.