Web Scraping
Onya Clovis
4/6/2021
pacman::p_load(tidyverse,rvest)install.packages('rvest')## Warning: package 'rvest' is in use and will not be installedScraping the IMDb website using R
scraping the IMDb website for the 100 most popular feature films released in 2016.
#Loading the rvest package
library('rvest')#Specifying the url for desired website to be scraped
url <- 'http://www.imdb.com/search/title?count=100&release_date=2016,2016&title_type=feature'#Reading the HTML code from the website
webpage <- read_html(url)Now, we’ll be scraping the following data from this website.
Rank: The rank of the film from 1 to 100 on the list of 100 most popular feature films released in 2016. Title: The title of the feature film. Description: The description of the feature film. Runtime: The duration of the feature film. Genre: The genre of the feature film, Rating: The IMDb rating of the feature film. Metascore: The metascore on IMDb website for the feature film. Votes: Votes cast in favor of the feature film. Gross_Earning_in_Mil: The gross earnings of the feature film in millions. Director: The main director of the feature film. Note, in case of multiple directors, I’ll take only the first. Actor: The main actor in the feature film. Note, in case of multiple actors, I’ll take only the first.
Use this simple R code to get all the rankings:
Using CSS selectors to scrape the rankings section
rank_data_html <- html_nodes(webpage,'.text-primary')#Converting the ranking data to text
rank_data <- html_text(rank_data_html)#Let’s have a look at the rankings
head(rank_data)## [1] "1." "2." "3." "4." "5." "6."#Data-Preprocessing: Converting rankings to numerical
rank_data<-as.numeric(rank_data)#Let’s have another look at the rankings
head(rank_data)## [1] 1 2 3 4 5 6I will use this selector to scrape all the titles using the following code.
#Using CSS selectors to scrape the title section
title_data_html <- html_nodes(webpage,'.lister-item-header a')#Converting the title data to text
title_data <- html_text(title_data_html)#Let’s have a look at the title
head(title_data)## [1] "Suicide Squad" "Batman v Superman: Dawn of Justice"
## [3] "Captain America: Civil War" "Captain Fantastic"
## [5] "Deadpool" "The Accountant"Now we will be scraping – Description, Runtime, Genre, Rating, Metascore, Votes, Gross_Earning_in_Mil , Director and Actor data.
#Using CSS selectors to scrape the description section
description_data_html <- html_nodes(webpage,'.ratings-bar+ .text-muted')#Converting the description data to text
description_data <- html_text(description_data_html)#Let’s have a look at the description data
head(description_data)## [1] "\n A secret government agency recruits some of the most dangerous incarcerated super-villains to form a defensive task force. Their first mission: save the world from the apocalypse."
## [2] "\n Fearing that the actions of Superman are left unchecked, Batman takes on the Man of Steel, while the world wrestles with what kind of a hero it really needs."
## [3] "\n Political involvement in the Avengers' affairs causes a rift between Captain America and Iron Man."
## [4] "\n In the forests of the Pacific Northwest, a father devoted to raising his six kids with a rigorous physical and intellectual education is forced to leave his paradise and enter the world, challenging his idea of what it means to be a parent."
## [5] "\n A wisecracking mercenary gets experimented on and becomes immortal but ugly, and sets out to track down the man who ruined his looks."
## [6] "\n As a math savant uncooks the books for a new client, the Treasury Department closes in on his activities, and the body count starts to rise."#Data-Preprocessing: removing ‘’
description_data<-gsub("\n","",description_data)#Let’s have another look at the description data
head(description_data)## [1] " A secret government agency recruits some of the most dangerous incarcerated super-villains to form a defensive task force. Their first mission: save the world from the apocalypse."
## [2] " Fearing that the actions of Superman are left unchecked, Batman takes on the Man of Steel, while the world wrestles with what kind of a hero it really needs."
## [3] " Political involvement in the Avengers' affairs causes a rift between Captain America and Iron Man."
## [4] " In the forests of the Pacific Northwest, a father devoted to raising his six kids with a rigorous physical and intellectual education is forced to leave his paradise and enter the world, challenging his idea of what it means to be a parent."
## [5] " A wisecracking mercenary gets experimented on and becomes immortal but ugly, and sets out to track down the man who ruined his looks."
## [6] " As a math savant uncooks the books for a new client, the Treasury Department closes in on his activities, and the body count starts to rise."#Using CSS selectors to scrape the Movie runtime section
runtime_data_html <- html_nodes(webpage,'.text-muted .runtime')#Converting the runtime data to text
runtime_data <- html_text(runtime_data_html)#Let’s have a look at the runtime
head(runtime_data)## [1] "123 min" "152 min" "147 min" "118 min" "108 min" "128 min"#Data-Preprocessing: removing mins and converting it to numerical
runtime_data<-gsub(" min","",runtime_data)
runtime_data<-as.numeric(runtime_data)#Let’s have another look at the runtime data
head(runtime_data)## [1] 123 152 147 118 108 128#Using CSS selectors to scrape the Movie genre section
genre_data_html <- html_nodes(webpage,'.genre')#Converting the genre data to text
genre_data <- html_text(genre_data_html)#Let’s have a look at the runtime
head(genre_data)## [1] "\nAction, Adventure, Fantasy "
## [2] "\nAction, Adventure, Sci-Fi "
## [3] "\nAction, Adventure, Sci-Fi "
## [4] "\nComedy, Drama "
## [5] "\nAction, Adventure, Comedy "
## [6] "\nAction, Crime, Drama "#Data-Preprocessing: removing
genre_data<-gsub("\n","",genre_data)#Data-Preprocessing: removing excess spaces
genre_data<-gsub(" ","",genre_data)#taking only the first genre of each movie
genre_data<-gsub(",.*","",genre_data)#Convering each genre from text to factor
genre_data<-as.factor(genre_data)#Let’s have another look at the genre data
head(genre_data)## [1] Action Action Action Comedy Action Action
## Levels: Action Adventure Animation Biography Comedy Crime Drama Horror#Using CSS selectors to scrape the IMDB rating section
rating_data_html <- html_nodes(webpage,'.ratings-imdb-rating strong')#Converting the ratings data to text
rating_data <- html_text(rating_data_html)#Let’s have a look at the ratings
head(rating_data)## [1] "6.0" "6.4" "7.8" "7.9" "8.0" "7.3"#Data-Preprocessing: converting ratings to numerical
rating_data<-as.numeric(rating_data)#Let’s have another look at the ratings data
head(rating_data)## [1] 6.0 6.4 7.8 7.9 8.0 7.3#Using CSS selectors to scrape the votes section
votes_data_html <- html_nodes(webpage,'.sort-num_votes-visible span:nth-child(2)')#Converting the votes data to text
votes_data <- html_text(votes_data_html)#Let’s have a look at the votes data
head(votes_data)## [1] "612,311" "643,262" "676,190" "194,560" "913,847" "264,388"#Data-Preprocessing: removing commas
votes_data<-gsub(",","",votes_data)#Data-Preprocessing: converting votes to numerical
votes_data<-as.numeric(votes_data)#Let’s have another look at the votes data
head(votes_data)## [1] 612311 643262 676190 194560 913847 264388#Using CSS selectors to scrape the directors section
directors_data_html <- html_nodes(webpage,'.text-muted+ p a:nth-child(1)')#Converting the directors data to text
directors_data <- html_text(directors_data_html)#Let’s have a look at the directors data
head(directors_data)## [1] "David Ayer" "Zack Snyder" "Anthony Russo" "Matt Ross"
## [5] "Tim Miller" "Gavin O'Connor"#Data-Preprocessing: converting directors data into factors
directors_data<-as.factor(directors_data)#Using CSS selectors to scrape the actors section
actors_data_html <- html_nodes(webpage,'.lister-item-content .ghost+ a')#Converting the gross actors data to text
actors_data <- html_text(actors_data_html)#Let’s have a look at the actors data
head(actors_data)## [1] "Will Smith" "Ben Affleck" "Chris Evans" "Viggo Mortensen"
## [5] "Ryan Reynolds" "Ben Affleck"#Data-Preprocessing: converting actors data into factors
actors_data<-as.factor(actors_data)#Using CSS selectors to scrape the metascore section
metascore_data_html <- html_nodes(webpage,'.metascore')#Converting the runtime data to text
metascore_data <- html_text(metascore_data_html)#Let’s have a look at the metascore data
head(metascore_data)## [1] "40 " "44 " "75 " "72 " "65 "
## [6] "51 "#Data-Preprocessing: removing extra space in metascore
metascore_data<-gsub(" ","",metascore_data)#Lets check the length of metascore data
length(metascore_data)## [1] 97The length of the metascore data is 97 while we are scraping the data for 100 movies. The reason this happened is that there are 3 movies that don’t have the corresponding Metascore fields. Unfortunately, if we simply add NA’s to last 3 entries, it will map NA as Metascore for movies 97 to 100 while in reality, the data is missing for some other movies. After a visual inspection, I found that the Metascore is missing for movies 18, 57 and 100. I have written the following function to get around this problem.
for (i in c(18,57,100)){
metascore_data <- append(metascore_data, NA, i-1)
}#Data-Preprocessing: converting metascore to numerical
metascore_data<-as.numeric(metascore_data)#Let’s have another look at length of the metascore data
length(metascore_data)## [1] 100length(metascore_data)
summary(metascore_data)## Min. 1st Qu. Median Mean 3rd Qu. Max. NA's
## 25.00 48.00 62.00 60.44 72.00 99.00 3#Using CSS selectors to scrape the gross revenue section
gross_data_html <- html_nodes(webpage,'.ghost~ .text-muted+ span')#Converting the gross revenue data to text
gross_data <- html_text(gross_data_html)#Let’s have a look at the votes data
head(gross_data)## [1] "$325.10M" "$330.36M" "$408.08M" "$5.88M" "$363.07M" "$86.26M"#Data-Preprocessing: removing ‘$’ and ‘M’ signs
gross_data<-gsub("M","",gross_data)
gross_data<-substring(gross_data,2,6)#Let’s check the length of gross data
length(gross_data)## [1] 92#Filling missing entries with NA
for (i in c(18,67,73,75,83,87,98,100)){
gross_data <- append(gross_data, NA, i-1)
}#Data-Preprocessing: converting gross to numerical
gross_data<-as.numeric(gross_data)#Let’s have another look at the length of gross data
length(gross_data)## [1] 100summary(gross_data)## Min. 1st Qu. Median Mean 3rd Qu. Max. NA's
## 0.04 14.39 52.30 91.32 116.15 532.10 8Now we have successfully scraped all the 11 features for the 100 most popular feature films released in 2016. Let’s combine them to create a dataframe and inspect its structure.
#Combining all the lists to form a data frame
movies_df<-data.frame(Rank = rank_data, Title = title_data,
Description = description_data, Runtime = runtime_data,
Genre = genre_data, Rating = rating_data,
Metascore = metascore_data, Votes = votes_data,
Gross_Earning_in_Mil = gross_data,
Director = directors_data, Actor = actors_data)#Structure of the data frame
str(movies_df)## 'data.frame': 100 obs. of 11 variables:
## $ Rank : num 1 2 3 4 5 6 7 8 9 10 ...
## $ Title : chr "Suicide Squad" "Batman v Superman: Dawn of Justice" "Captain America: Civil War" "Captain Fantastic" ...
## $ Description : chr " A secret government agency recruits some of the most dangerous incarcerated super-villains to form a defens"| __truncated__ " Fearing that the actions of Superman are left unchecked, Batman takes on the Man of Steel, while the world "| __truncated__ " Political involvement in the Avengers' affairs causes a rift between Captain America and Iron Man." " In the forests of the Pacific Northwest, a father devoted to raising his six kids with a rigorous physical "| __truncated__ ...
## $ Runtime : num 123 152 147 118 108 128 120 116 107 116 ...
## $ Genre : Factor w/ 8 levels "Action","Adventure",..: 1 1 1 5 1 1 1 1 3 7 ...
## $ Rating : num 6 6.4 7.8 7.9 8 7.3 6.8 7.4 7.6 7.9 ...
## $ Metascore : num 40 44 75 72 65 51 67 70 81 81 ...
## $ Votes : num 612311 643262 676190 194560 913847 ...
## $ Gross_Earning_in_Mil: num 325.1 330.3 408 5.88 363 ...
## $ Director : Factor w/ 98 levels "Adam Wingard",..: 23 98 6 61 93 36 40 86 82 27 ...
## $ Actor : Factor w/ 91 levels "Aamir Khan","Alexander Skarsgård",..: 89 8 19 88 75 8 39 73 7 3 ...Analyzing scraped data from the web
Here are some interesting visualization out of the data we have just scraped.
library('ggplot2')
qplot(data = movies_df,Runtime,fill = Genre,bins = 30)
Now lets follow the visualizations and answer the questions given below.
Question 1: Based on the above data, which movie from which Genre had the longest runtime?
movies_df %>%
select("Genre", "Runtime","Title") %>%
filter(Runtime >= 160)## Genre Runtime Title
## 1 Drama 161 Silence
## 2 Action 161 DangalThe movies “Silence” and “Dangal” from the Drama and Action genre are the movies with the longest runtime of 161mins.
ggplot(movies_df,aes(x=Runtime,y=Rating))+
geom_point(aes(size=Votes,col=Genre))
Question 2: Based on the above data, in the Runtime of 130-160 mins, which genre has the highest votes?
movies_df %>%
select("Genre", "Runtime","Votes") %>%
filter(Runtime >=130 & Runtime<=160) %>%
arrange(desc(Votes))## Genre Runtime Votes
## 1 Action 147 676190
## 2 Action 152 643262
## 3 Action 133 562876
## 4 Biography 139 444250
## 5 Adventure 132 415318
## 6 Action 144 398149
## 7 Drama 137 251109
## 8 Horror 134 228693
## 9 Action 132 186961
## 10 Biography 134 140061
## 11 Action 144 121667
## 12 Drama 145 117751
## 13 Action 133 93676
## 14 Drama 146 88438
## 15 Biography 141 81739
## 16 Drama 132 65537
## 17 Horror 156 53581
## 18 Animation 130 51657The above code and visualization shows that the “Action” genre has the higest votes with 676168 votes
ggplot(movies_df,aes(x=Runtime,y=Gross_Earning_in_Mil))+
geom_point(aes(size=Rating,col=Genre))## Warning: Removed 8 rows containing missing values (geom_point).
Question 3: Based on the above data, across all genres which genre has the highest average gross earnings in runtime 100 to 120.
Avrg_gross_earnings <- movies_df %>%
select("Genre", "Runtime","Gross_Earning_in_Mil") %>%
filter(Gross_Earning_in_Mil >=100, Gross_Earning_in_Mil<=120) %>%
arrange(desc(Gross_Earning_in_Mil))
Avrg_gross_earnings## Genre Runtime Gross_Earning_in_Mil
## 1 Comedy 100 113.2
## 2 Action 120 103.1
## 3 Horror 134 102.4
## 4 Drama 116 100.5
## 5 Drama 116 100.0The above data shows that the “Comedy” genre has the highest average gross earnings in runtime 100 to 120