R Notebook

#Loading the rvest package
library('rvest')

#Specifying the url for desired website to be scraped
url <- 'http://www.imdb.com/search/title?count=100&release_date=2016,2016&title_type=feature'

#Reading the HTML code from the website
webpage <- read_html(url)

.text-primary

#Using CSS selectors to scrape the rankings section
rank_data_html <- html_nodes(webpage,'.text-primary')

#Converting the ranking data to text
rank_data <- html_text(rank_data_html)

#Let's have a look at the rankings
head(rank_data)

[1] "1." "2." "3." "4." "5." "6."

Step 4

#Data-Preprocessing: Converting rankings to numerical
rank_data<-as.numeric(rank_data)

#Let's have another look at the rankings
head(rank_data)

[1] 1 2 3 4 5 6

Step 6

#Using CSS selectors to scrape the title section
title_data_html <- html_nodes(webpage,'.lister-item-header a')

#Converting the title data to text
title_data <- html_text(title_data_html)

#Let's have a look at the title
head(title_data)

[1] "Suicide Squad"                      "Batman v Superman: Dawn of Justice" "Captain America: Civil War"         "Captain Fantastic"                 
[5] "Deadpool"                           "The Accountant"                    

#Using CSS selectors to scrape the description section
description_data_html <- html_nodes(webpage,'.ratings-bar+ .text-muted')

#Converting the description data to text
description_data <- html_text(description_data_html)

#Let's have a look at the description data
head(description_data)

[1] "\n    A secret government agency recruits some of the most dangerous incarcerated super-villains to form a defensive task force. Their first mission: save the world from the apocalypse."                                                             
[2] "\n    Fearing that the actions of Superman are left unchecked, Batman takes on the Man of Steel, while the world wrestles with what kind of a hero it really needs."                                                                                   
[3] "\n    Political involvement in the Avengers' affairs causes a rift between Captain America and Iron Man."                                                                                                                                              
[4] "\n    In the forests of the Pacific Northwest, a father devoted to raising his six kids with a rigorous physical and intellectual education is forced to leave his paradise and enter the world, challenging his idea of what it means to be a parent."
[5] "\n    A wisecracking mercenary gets experimented on and becomes immortal but ugly, and sets out to track down the man who ruined his looks."                                                                                                           
[6] "\n    As a math savant uncooks the books for a new client, the Treasury Department closes in on his activities, and the body count starts to rise."                                                                                                    

Removing “”:

#Data-Preprocessing: removing '\n'
description_data<-gsub("\n","",description_data)

#Let's have another look at the description data 
head(description_data)

[1] "    A secret government agency recruits some of the most dangerous incarcerated super-villains to form a defensive task force. Their first mission: save the world from the apocalypse."                                                             
[2] "    Fearing that the actions of Superman are left unchecked, Batman takes on the Man of Steel, while the world wrestles with what kind of a hero it really needs."                                                                                   
[3] "    Political involvement in the Avengers' affairs causes a rift between Captain America and Iron Man."                                                                                                                                              
[4] "    In the forests of the Pacific Northwest, a father devoted to raising his six kids with a rigorous physical and intellectual education is forced to leave his paradise and enter the world, challenging his idea of what it means to be a parent."
[5] "    A wisecracking mercenary gets experimented on and becomes immortal but ugly, and sets out to track down the man who ruined his looks."                                                                                                           
[6] "    As a math savant uncooks the books for a new client, the Treasury Department closes in on his activities, and the body count starts to rise."                                                                                                    

#Using CSS selectors to scrape the Movie runtime section
runtime_data_html <- html_nodes(webpage,'.text-muted .runtime')

#Converting the runtime data to text
runtime_data <- html_text(runtime_data_html)

#Let's have a look at the runtime
head(runtime_data)

[1] "123 min" "152 min" "147 min" "118 min" "108 min" "128 min"

#Data-Preprocessing: removing mins and converting it to numerical

runtime_data<-gsub(" min","",runtime_data)
runtime_data<-as.numeric(runtime_data)

#Let's have another look at the runtime data
head(runtime_data)

[1] 123 152 147 118 108 128

#Using CSS selectors to scrape the Movie genre section
genre_data_html <- html_nodes(webpage,'.genre')

#Converting the genre data to text
genre_data <- html_text(genre_data_html)

#Let's have a look at the runtime
head(genre_data)

[1] "\nAction, Adventure, Fantasy            " "\nAction, Adventure, Sci-Fi            "  "\nAction, Adventure, Sci-Fi            " 
[4] "\nComedy, Drama            "              "\nAction, Adventure, Comedy            "  "\nAction, Crime, Drama            "      

#Data-Preprocessing: removing \n
genre_data<-gsub("\n","",genre_data)

#Data-Preprocessing: removing excess spaces
genre_data<-gsub(" ","",genre_data)

#taking only the first genre of each movie
genre_data<-gsub(",.*","",genre_data)

#Convering each genre from text to factor
genre_data<-as.factor(genre_data)

#Let's have another look at the genre data
head(genre_data)

[1] Action Action Action Comedy Action Action
Levels: Action Adventure Animation Biography Comedy Crime Drama Horror

#Using CSS selectors to scrape the IMDB rating section
rating_data_html <- html_nodes(webpage,'.ratings-imdb-rating strong')

#Converting the ratings data to text
rating_data <- html_text(rating_data_html)

#Let's have a look at the ratings
head(rating_data)

[1] "6.0" "6.4" "7.8" "7.9" "8.0" "7.3"

#Data-Preprocessing: converting ratings to numerical
rating_data<-as.numeric(rating_data)

#Let's have another look at the ratings data
head(rating_data)

[1] 6.0 6.4 7.8 7.9 8.0 7.3

#Using CSS selectors to scrape the votes section
votes_data_html <- html_nodes(webpage,'.sort-num_votes-visible span:nth-child(2)')

#Converting the votes data to text
votes_data <- html_text(votes_data_html)

#Let's have a look at the votes data
head(votes_data)

[1] "612,311" "643,262" "676,190" "194,560" "913,847" "264,388"

#Data-Preprocessing: removing commas
votes_data<-gsub(",","",votes_data)

#Data-Preprocessing: converting votes to numerical
votes_data<-as.numeric(votes_data)

#Let's have another look at the votes data
head(votes_data)

[1] 612311 643262 676190 194560 913847 264388

#Using CSS selectors to scrape the directors section
directors_data_html <- html_nodes(webpage,'.text-muted+ p a:nth-child(1)')

#Converting the directors data to text
directors_data <- html_text(directors_data_html)

#Let's have a look at the directors data
head(directors_data)

[1] "David Ayer"     "Zack Snyder"    "Anthony Russo"  "Matt Ross"      "Tim Miller"     "Gavin O'Connor"

#Data-Preprocessing: converting directors data into factors
directors_data<-as.factor(directors_data)

#Using CSS selectors to scrape the actors section
actors_data_html <- html_nodes(webpage,'.lister-item-content .ghost+ a')

#Converting the gross actors data to text
actors_data <- html_text(actors_data_html)

#Let's have a look at the actors data
head(actors_data)

[1] "Will Smith"      "Ben Affleck"     "Chris Evans"     "Viggo Mortensen" "Ryan Reynolds"   "Ben Affleck"    

#Data-Preprocessing: converting actors data into factors
actors_data<-as.factor(actors_data)

#Using CSS selectors to scrape the metascore section
metascore_data_html <- html_nodes(webpage,'.metascore')

#Converting the runtime data to text
metascore_data <- html_text(metascore_data_html)

#Let's have a look at the metascore data 
head(metascore_data)

[1] "40        " "44        " "75        " "72        " "65        " "51        "

#Data-Preprocessing: removing extra space in metascore
metascore_data<-gsub(" ","",metascore_data)

#Lets check the length of metascore data
length(metascore_data)

[1] 97

for (i in c(18,57,100)){

metascore_data<-append(x = metascore_data, values = list("NA"), after = i-1)

}

#Data-Preprocessing: converting metascore to numerical
metascore_data<-as.numeric(metascore_data)

NAs introduced by coercionNAs introduced by coercionNAs introduced by coercion

#Let's have another look at length of the metascore data

length(metascore_data)

[1] 100

#Let's look at summary statistics
summary(metascore_data)

   Min. 1st Qu.  Median    Mean 3rd Qu.    Max.    NA's 
  25.00   48.00   62.00   60.44   72.00   99.00       3 

Sttep 10

#Using CSS selectors to scrape the gross revenue section
gross_data_html <- html_nodes(webpage,'.ghost~ .text-muted+ span')

#Converting the gross revenue data to text
gross_data <- html_text(gross_data_html)

#Let's have a look at the votes data
head(gross_data)

[1] "$325.10M" "$330.36M" "$408.08M" "$5.88M"   "$363.07M" "$86.26M" 

#Data-Preprocessing: removing '$' and 'M' signs
gross_data<-gsub("M","",gross_data)

gross_data<-substring(gross_data,2,6)

#Let's check the length of gross data
length(gross_data)

[1] 92

#Filling missing entries with NA
for (i in c(18, 67, 73, 75, 83, 87, 98, 100 )){

gross_data <- append(gross_data, values = list("NA"), after = i-1)


}

#Data-Preprocessing: converting gross to numerical
gross_data<-as.numeric(gross_data)

NAs introduced by coercionNAs introduced by coercionNAs introduced by coercionNAs introduced by coercionNAs introduced by coercionNAs introduced by coercionNAs introduced by coercionNAs introduced by coercion

#Let's have another look at the length of gross data
length(gross_data)

[1] 100

summary(gross_data)

   Min. 1st Qu.  Median    Mean 3rd Qu.    Max.    NA's 
   0.04   14.39   52.30   91.32  116.15  532.10       8 

Step 11

#Combining all the lists to form a data frame
movies_df<-data.frame(Rank = rank_data, Title = title_data,

Description = description_data, Runtime = runtime_data,

Genre = genre_data, Rating = rating_data,

Metascore = metascore_data, Votes = votes_data, 

Gross_Earning_in_Mil = gross_data,

Director = directors_data, Actor = actors_data)

#Structure of the data frame

str(movies_df)

'data.frame':   100 obs. of  11 variables:
 $ Rank                : num  1 2 3 4 5 6 7 8 9 10 ...
 $ Title               : chr  "Suicide Squad" "Batman v Superman: Dawn of Justice" "Captain America: Civil War" "Captain Fantastic" ...
 $ Description         : chr  "    A secret government agency recruits some of the most dangerous incarcerated super-villains to form a defens"| __truncated__ "    Fearing that the actions of Superman are left unchecked, Batman takes on the Man of Steel, while the world "| __truncated__ "    Political involvement in the Avengers' affairs causes a rift between Captain America and Iron Man." "    In the forests of the Pacific Northwest, a father devoted to raising his six kids with a rigorous physical "| __truncated__ ...
 $ Runtime             : num  123 152 147 118 108 128 120 116 107 116 ...
 $ Genre               : Factor w/ 8 levels "Action","Adventure",..: 1 1 1 5 1 1 1 1 3 7 ...
 $ Rating              : num  6 6.4 7.8 7.9 8 7.3 6.8 7.4 7.6 7.9 ...
 $ Metascore           : num  40 44 75 72 65 51 67 70 81 81 ...
 $ Votes               : num  612311 643262 676190 194560 913847 ...
 $ Gross_Earning_in_Mil: num  325.1 330.3 408 5.88 363 ...
 $ Director            : Factor w/ 98 levels "Adam Wingard",..: 23 98 6 61 93 36 40 86 82 27 ...
 $ Actor               : Factor w/ 91 levels "Aamir Khan","Alexander Skarsgård",..: 89 8 19 88 75 8 39 73 7 3 ...

Analyizing scraped data from the web

library('ggplot2')

Want to understand how all the pieces fit together? Read R for Data Science: https://r4ds.had.co.nz/

qplot(data = movies_df,Runtime,fill = Genre,bins = 30)

Question 1: Based on the above data, which movie from which genre had the longest runtime?

From the plot we can see that Drama and Action had the longest runtimes

library(tidyverse)

movies_df %>% arrange(desc(Runtime)) %>% head()

The movies “Silence” and “Dangal” in the drama and action genres had the longest runtimes at 161 minutes each.

ggplot(movies_df,aes(x=Runtime,y=Rating))+
geom_point(aes(size=Votes,col=Genre))

Question 2: Based on the above data, in the Runtime of 130-160 minutes, which genre has the highest votes.

From the plot we can see that biography and drama got the highest votes

ggplot(movies_df,aes(x=Runtime,y=Gross_Earning_in_Mil))+
geom_point(aes(size=Rating,col=Genre))

Question 3: Based on the above data, across all genres which genre has the highest average gross earnings in runtime 100 to 120

Action and Adventure have the highest average gross earnings in the 100-120 minute range for runtime