WK 8 Web Scraping Tutorial & Homework - IMDB Top 100 Movies 2016
R. LEE
October 27, 2019
Beginner’s Guide on Web Scraping in R (using rvest) with hands-on example
#Loading the rvest package
library('rvest')## Loading required package: xml2#Specifying the url for desired website to be scraped
url <- 'http://www.imdb.com/search/title?count=100&release_date=2016,2016&title_type=feature'
#Reading the HTML code from the website
webpage <- read_html(url)Step 1:
Now, we will start by scraping the Rank field. For that, we’ll use the selector gadget to get the specific CSS selectors that encloses the rankings. You can click on the extension in your browser and select the rankings field with the cursor.
Make sure that all the rankings are selected. You can select some more ranking sections in case you are not able to get all of them and you can also de-select them by clicking on the selected section to make sure that you only have those sections highlighted that you want to scrape for that go.
Step 2:
Once you are sure that you have made the right selections, you need to copy the corresponding CSS selector that you can view in the bottom center.
Step 3:
#Using CSS selectors to scrape the rankings section
rank_data_html <- html_nodes(webpage,'.text-primary')
#Converting the ranking data to text
rank_data <- html_text(rank_data_html)
#Let's have a look at the rankings
head(rank_data)## [1] "1." "2." "3." "4." "5." "6."Step 4:
Once you have the data, make sure that it looks in the desired format. I am preprocessing my data to convert it to numerical format.
#Data-Preprocessing: Converting rankings to numerical
rank_data<-as.numeric(rank_data)
#Let's have another look at the rankings
head(rank_data)## [1] 1 2 3 4 5 6Step 5:
Now you can clear the selector section and select all the titles. You can visually inspect that all the titles are selected. Make any required additions and deletions with the help of your curser. I have done the same here.
Step 6:
Again, I have the corresponding CSS selector for the titles – .lister-item-header a. I will use this selector to scrape all the titles using the following code.
#Using CSS selectors to scrape the title section
title_data_html <- html_nodes(webpage,'.lister-item-header a')
#Converting the title data to text
title_data <- html_text(title_data_html)
#Let's have a look at the title
head(title_data)## [1] "Suicide Squad" "Moonlight" "Rogue One" "The Handmaiden"
## [5] "Split" "La La Land"Step 7:
In the following code, I have done the same thing for scraping – Description, Runtime, Genre, Rating, Metascore, Votes, Gross_Earning_in_Mil , Director and Actor data.
#Using CSS selectors to scrape the description section
description_data_html <- html_nodes(webpage,'.ratings-bar+ .text-muted')
#Converting the description data to text
description_data <- html_text(description_data_html)
#Let's have a look at the description data
head(description_data)## [1] "\n A secret government agency recruits some of the most dangerous incarcerated super-villains to form a defensive task force. Their first mission: save the world from the apocalypse."
## [2] "\n A young African-American man grapples with his identity and sexuality while experiencing the everyday struggles of childhood, adolescence, and burgeoning adulthood."
## [3] "\n The daughter of an Imperial scientist joins the Rebel Alliance in a risky move to steal the Death Star plans."
## [4] "\n A woman is hired as a handmaiden to a Japanese heiress, but secretly she is involved in a plot to defraud her."
## [5] "\n Three girls are kidnapped by a man with a diagnosed 23 distinct personalities. They must try to escape before the apparent emergence of a frightful new 24th."
## [6] "\n While navigating their careers in Los Angeles, a pianist and an actress fall in love while attempting to reconcile their aspirations for the future."#Data-Preprocessing: removing '\n'
description_data<-gsub("\n","",description_data)
#Let's have another look at the description data
head(description_data)## [1] " A secret government agency recruits some of the most dangerous incarcerated super-villains to form a defensive task force. Their first mission: save the world from the apocalypse."
## [2] " A young African-American man grapples with his identity and sexuality while experiencing the everyday struggles of childhood, adolescence, and burgeoning adulthood."
## [3] " The daughter of an Imperial scientist joins the Rebel Alliance in a risky move to steal the Death Star plans."
## [4] " A woman is hired as a handmaiden to a Japanese heiress, but secretly she is involved in a plot to defraud her."
## [5] " Three girls are kidnapped by a man with a diagnosed 23 distinct personalities. They must try to escape before the apparent emergence of a frightful new 24th."
## [6] " While navigating their careers in Los Angeles, a pianist and an actress fall in love while attempting to reconcile their aspirations for the future."#Using CSS selectors to scrape the Movie runtime section
runtime_data_html <- html_nodes(webpage,'.text-muted .runtime')
#Converting the runtime data to text
runtime_data <- html_text(runtime_data_html)
#Let's have a look at the runtime
head(runtime_data)## [1] "123 min" "111 min" "133 min" "145 min" "117 min" "128 min"#Data-Preprocessing: removing mins and converting it to numerical
runtime_data<-gsub(" min","",runtime_data)
runtime_data<-as.numeric(runtime_data)
#Let's have another look at the runtime data
head(runtime_data)## [1] 123 111 133 145 117 128#Using CSS selectors to scrape the Movie genre section
genre_data_html <- html_nodes(webpage,'.genre')
#Converting the genre data to text
genre_data <- html_text(genre_data_html)
#Let's have a look at the runtime
head(genre_data)## [1] "\nAction, Adventure, Fantasy "
## [2] "\nDrama "
## [3] "\nAction, Adventure, Sci-Fi "
## [4] "\nDrama, Romance, Thriller "
## [5] "\nHorror, Thriller "
## [6] "\nComedy, Drama, Music "#Data-Preprocessing: removing \n
genre_data<-gsub("\n","",genre_data)
#Data-Preprocessing: removing excess spaces
genre_data<-gsub(" ","",genre_data)
#taking only the first genre of each movie
genre_data<-gsub(",.*","",genre_data)
#Convering each genre from text to factor
genre_data<-as.factor(genre_data)
#Let's have another look at the genre data
head(genre_data)## [1] Action Drama Action Drama Horror Comedy
## 9 Levels: Action Adventure Animation Biography Comedy Crime ... Mystery#Using CSS selectors to scrape the IMDB rating section
rating_data_html <- html_nodes(webpage,'.ratings-imdb-rating strong')
#Converting the ratings data to text
rating_data <- html_text(rating_data_html)
#Let's have a look at the ratings
head(rating_data)## [1] "6.0" "7.4" "7.8" "8.1" "7.3" "8.0"#Data-Preprocessing: converting ratings to numerical
rating_data<-as.numeric(rating_data)
#Let's have another look at the ratings data
head(rating_data)## [1] 6.0 7.4 7.8 8.1 7.3 8.0#Using CSS selectors to scrape the votes section
votes_data_html <- html_nodes(webpage,'.sort-num_votes-visible span:nth-child(2)')
#Converting the votes data to text
votes_data <- html_text(votes_data_html)
#Let's have a look at the votes data
head(votes_data)## [1] "551,880" "237,227" "492,359" "85,409" "378,279" "451,304"#Data-Preprocessing: removing commas
votes_data<-gsub(",","",votes_data)
#Data-Preprocessing: converting votes to numerical
votes_data<-as.numeric(votes_data)
#Let's have another look at the votes data
head(votes_data)## [1] 551880 237227 492359 85409 378279 451304#Using CSS selectors to scrape the directors section
directors_data_html <- html_nodes(webpage,'.text-muted+ p a:nth-child(1)')
#Converting the directors data to text
directors_data <- html_text(directors_data_html)
#Let's have a look at the directors data
head(directors_data)## [1] "David Ayer" "Barry Jenkins" "Gareth Edwards"
## [4] "Chan-wook Park" "M. Night Shyamalan" "Damien Chazelle"#Data-Preprocessing: converting directors data into factors
directors_data<-as.factor(directors_data)
#Using CSS selectors to scrape the actors section
actors_data_html <- html_nodes(webpage,'.lister-item-content .ghost+ a')
#Converting the gross actors data to text
actors_data <- html_text(actors_data_html)
#Let's have a look at the actors data
head(actors_data)## [1] "Will Smith" "Mahershala Ali" "Felicity Jones" "Min-hee Kim"
## [5] "James McAvoy" "Ryan Gosling"#Data-Preprocessing: converting actors data into factors
actors_data<-as.factor(actors_data)#Using CSS selectors to scrape the metascore section
metascore_data_html <- html_nodes(webpage,'.metascore')
#Converting the runtime data to text
metascore_data <- html_text(metascore_data_html)
#Let's have a look at the metascore
head(metascore_data)## [1] "40 " "99 " "65 " "84 " "62 "
## [6] "93 "#Data-Preprocessing: removing extra space in metascore
metascore_data<-gsub(" ","",metascore_data)
#Lets check the length of metascore data
length(metascore_data)## [1] 94Step 8:
The length of the metascore data is 96 while we are scraping the data for 100 movies. The reason this happened is that there are 4 movies that don’t have the corresponding Metascore fields.
Step 9:
It is a practical situation which can arise while scraping any website. Unfortunately, if we simply add NA’s to last 4 entries, it will map NA as Metascore for movies 96 to 100 while in reality, the data is missing for some other movies. After a visual inspection, I found that the Metascore is missing for movies 39, 73, 80 and 89. I have written the following function to get around this problem.
for (i in c(13,29,40,43,66,75)){
a<-metascore_data[1:(i-1)]
b<-metascore_data[i:length(metascore_data)]
metascore_data<-append(a,list("NA"))
metascore_data<-append(metascore_data,b)
}
#Data-Preprocessing: converting metascore to numerical
metascore_data<-as.numeric(metascore_data)## Warning: NAs introduced by coercion
## Warning: NAs introduced by coercion
## Warning: NAs introduced by coercion
## Warning: NAs introduced by coercion
## Warning: NAs introduced by coercion
## Warning: NAs introduced by coercion#Let's have another look at length of the metascore data
length(metascore_data)## [1] 100#Let's look at summary statistics
summary(metascore_data)## Min. 1st Qu. Median Mean 3rd Qu. Max. NA's
## 21.00 47.00 59.50 58.68 71.75 99.00 6Step 10:
The same thing happens with the Gross variable which represents gross earnings of that movie in millions. I have use the same solution to work my way around:
#Using CSS selectors to scrape the gross revenue section
gross_data_html <- html_nodes(webpage,'.ghost~ .text-muted+ span')
#Converting the gross revenue data to text
gross_data <- html_text(gross_data_html)
#Let's have a look at the votes data
head(gross_data)## [1] "$325.10M" "$27.85M" "$532.18M" "$2.01M" "$138.29M" "$151.10M"#Data-Preprocessing: removing '$' and 'M' signs
# Option 1 - this is the original code from tutorial
gross_data<-gsub("M","",gross_data)
gross_data<-substring(gross_data,2,6)
# Option 2 - this line of code accomplishes Option 1 in one line. Provided by classmate Steve Duky.
# gross_data<-gsub("[^0-9.]*","",gross_data)
#Let's check the length of gross data
length(gross_data)## [1] 90#Filling missing entries with NA
for (i in c(29,33,40,41,43,72,74,75,76,100)){
a<-gross_data[1:(i-1)]
b<-gross_data[i:length(gross_data)]
gross_data<-append(a,list("NA"))
gross_data<-append(gross_data,b)
}# check the length of gross data again
length(gross_data)## [1] 102#Data-Preprocessing: converting gross to numerical
# 2 lines of addt'l code provided by Prof - a workaround for gross_data having 2 extra liines when NA is added to empty cells
unlist(gross_data) ## [1] "325.1" "27.85" "532.1" "2.01" "138.2" "151.1" "67.21" "363.0"
## [9] "330.3" "248.7" "100.5" "234.0" "3.78" "1.33" "86.26" "93.43"
## [17] "56.25" "55.12" "43.02" "155.4" "341.2" "0.78" "100.0" "408.0"
## [25] "5.02" "2.13" "128.3" "10.91" "NA" "232.6" "58.70" "10.64"
## [33] "NA" "7.10" "87.24" "36.26" "89.22" "71.90" "153.7" "NA"
## [41] "NA" "102.4" "NA" "270.4" "66.18" "162.4" "12.63" "51.74"
## [49] "65.08" "113.2" "5.88" "3.02" "0.23" "4.21" "47.37" "158.8"
## [57] "75.40" "126.6" "34.34" "125.0" "47.70" "31.15" "62.68" "127.4"
## [65] "103.1" "12.39" "5.20" "169.6" "8.58" "26.86" "0.18" "NA"
## [73] "97.69" "NA" "NA" "NA" "364.0" "48.39" "52.85" "20.78"
## [81] "45.54" "26.83" "1.72" "368.3" "14.26" "38.56" "61.43" "60.32"
## [89] "30.98" "0.04" "0.66" "35.54" "14.27" "67.27" "46.01" "79.21"
## [97] "54.65" "40.10" "11.10" "NA" "11.10"gross_data <- gross_data[-c(101,102)]
gross_data<-as.numeric(gross_data)## Warning: NAs introduced by coercion
## Warning: NAs introduced by coercion
## Warning: NAs introduced by coercion
## Warning: NAs introduced by coercion
## Warning: NAs introduced by coercion
## Warning: NAs introduced by coercion
## Warning: NAs introduced by coercion
## Warning: NAs introduced by coercion
## Warning: NAs introduced by coercion
## Warning: NAs introduced by coercion#Let's have another look at the length of gross data
length(gross_data)## [1] 100Error with gross_data - cannot be converted to double. See record #101 in data list:
type - NULL value - pairlist of length 0
This could be due to a quirk in the for loop above.
With this error, not possible to create Chart 3 - professor says to skip.
summary(gross_data)## Min. 1st Qu. Median Mean 3rd Qu. Max. NA's
## 0.04 14.26 55.69 92.87 122.05 532.10 10Step 11:
Now we have successfully scraped all the 11 features for the 100 most popular feature films released in 2016. Let’s combine them to create a dataframe and inspect its structure.
#Combining all the lists to form a data frame
movies_df<-data.frame(Rank = rank_data, Title = title_data,
Description = description_data, Runtime = runtime_data,
Genre = genre_data, Rating = rating_data,
Metascore = metascore_data, Votes = votes_data,
Gross_Earning_in_Mil = gross_data,
Director = directors_data, Actor = actors_data)
#Structure of the data frame
str(movies_df)## 'data.frame': 100 obs. of 11 variables:
## $ Rank : num 1 2 3 4 5 6 7 8 9 10 ...
## $ Title : Factor w/ 100 levels "10 Cloverfield Lane",..: 64 51 58 75 62 42 33 24 10 50 ...
## $ Description : Factor w/ 100 levels " 19-year-old Billy Lynn is brought home for a victory tour after a harrowing Iraq battle. Through flashbacks"| __truncated__,..: 20 29 83 28 88 98 100 27 47 63 ...
## $ Runtime : num 123 111 133 145 117 128 139 108 151 107 ...
## $ Genre : Factor w/ 9 levels "Action","Adventure",..: 1 7 1 7 8 5 4 1 1 3 ...
## $ Rating : num 6 7.4 7.8 8.1 7.3 8 8.1 8 6.5 7.6 ...
## $ Metascore : num 40 99 65 84 62 93 71 65 44 81 ...
## $ Votes : num 551880 237227 492359 85409 378279 ...
## $ Gross_Earning_in_Mil: num 325.1 27.85 532.1 2.01 138.2 ...
## $ Director : Factor w/ 99 levels "Adam Wingard",..: 27 14 36 19 59 22 63 94 99 82 ...
## $ Actor : Factor w/ 90 levels "Aamir Khan","Alexander Skarsgård",..: 87 53 30 65 36 70 4 71 9 8 ...6. Analyzing scraped data from the web
Once you have the data, you can perform several tasks like analyzing the data, drawing inferences from it, training machine learning models over this data, etc. I have gone on to create some interesting visualization out of the data we have just scraped. Follow the visualizations and answer the questions given below. Post your answers in the comment section below.
library('ggplot2')
qplot(data = movies_df,Runtime,fill = Genre,bins = 30)
Question 1: Based on the above data, which movie from which Genre had the longest runtime?
American Honey in the Adventure genre had the longest runtime (163 min).
library(tidyverse)## ── Attaching packages ────────────────────────────────────────────── tidyverse 1.2.1 ──## ✔ tibble 2.1.3 ✔ purrr 0.3.2
## ✔ tidyr 1.0.0 ✔ dplyr 0.8.3
## ✔ readr 1.3.1 ✔ stringr 1.4.0
## ✔ tibble 2.1.3 ✔ forcats 0.4.0## ── Conflicts ───────────────────────────────────────────────── tidyverse_conflicts() ──
## ✖ dplyr::filter() masks stats::filter()
## ✖ readr::guess_encoding() masks rvest::guess_encoding()
## ✖ dplyr::lag() masks stats::lag()
## ✖ purrr::pluck() masks rvest::pluck()longest <- movies_df %>%
arrange(desc(Runtime))
head(longest)## Rank Title
## 1 91 American Honey
## 2 34 Silence
## 3 66 Dangal
## 4 100 The Wailing
## 5 9 Batman v Superman: Dawn of Justice
## 6 24 Captain America: Civil War
## Description
## 1 A teenage girl with nothing to lose joins a traveling magazine sales crew, and gets caught up in a whirlwind of hard partying, law bending and young love as she criss-crosses the Midwest with a band of misfits.
## 2 In the 17th century, two Portuguese Jesuit priests travel to Japan in an attempt to locate their mentor, who is rumored to have committed apostasy, and to propagate Catholicism.
## 3 Former wrestler Mahavir Singh Phogat and his two wrestler daughters struggle towards glory at the Commonwealth Games in the face of societal oppression.
## 4 Soon after a stranger arrives in a little village, a mysterious sickness starts spreading. A policeman, drawn into the incident, is forced to solve the mystery in order to save his daughter.
## 5 Fearing that the actions of Superman are left unchecked, Batman takes on the Man of Steel, while the world wrestles with what kind of a hero it really needs.
## 6 Political involvement in the Avengers' affairs causes a rift between Captain America and Iron Man.
## Runtime Genre Rating Metascore Votes Gross_Earning_in_Mil
## 1 163 Adventure 7.0 79 33006 0.66
## 2 161 Drama 7.1 79 88266 7.10
## 3 161 Action 8.4 NA 130710 12.39
## 4 156 Horror 7.4 81 38131 NA
## 5 151 Action 6.5 44 588479 330.30
## 6 147 Action 7.8 75 606345 408.00
## Director Actor
## 1 Andrea Arnold Sasha Lane
## 2 Martin Scorsese Andrew Garfield
## 3 Nitesh Tiwari Aamir Khan
## 4 Hong-jin Na Jun Kunimura
## 5 Zack Snyder Ben Affleck
## 6 Anthony Russo Chris Evansggplot(movies_df,aes(x=Runtime,y=Rating))+
geom_point(aes(size=Votes,col=Genre))
Question 2: Based on the above data, in the Runtime of 130-160 mins, which genre has the highest votes?
Eyeballing the scatterplot above, the action genre (red circles) had the highest votes.
The code below confirms. Action received more than 2.3 million votes.
# List the highest votes per genre
highest_votes <- movies_df %>%
filter (Runtime >= 130 & Runtime <= 160) %>%
group_by(Genre) %>%
summarize(total_votes = sum(Votes)) %>%
arrange(desc(total_votes))
highest_votes ## # A tibble: 5 x 2
## Genre total_votes
## <fct> <dbl>
## 1 Action 2334098
## 2 Biography 450955
## 3 Adventure 376238
## 4 Drama 306467
## 5 Horror 241781ggplot(movies_df,aes(x=Runtime,y=Gross_Earning_in_Mil))+
geom_point(aes(size=Rating,col=Genre))## Warning: Removed 10 rows containing missing values (geom_point).
Question 3: Based on the above data, across all genres which genre has the highest average gross earnings in runtime 100 to 120.
The Animation genre had the highest average gross earnings in runtime 100-120.
The average gross earnings per film was $216 mil.
# List the average gross earnings per genre
highest_avg_gross <- movies_df %>%
filter (Runtime >= 100 & Runtime <= 120) %>%
group_by(Genre) %>%
mutate(num = 1) %>%
summarize(avg_Gross = sum(Gross_Earning_in_Mil)/sum(num)) %>%
arrange(desc(avg_Gross))
highest_avg_gross ## # A tibble: 8 x 2
## Genre avg_Gross
## <fct> <dbl>
## 1 Animation 216.
## 2 Adventure 185.
## 3 Action 80.7
## 4 Drama 52.5
## 5 Biography 47.4
## 6 Horror 46.8
## 7 Comedy 29.1
## 8 Crime NACHECK THE ACCURACY OF ABOVE CODE USING A DIFFERENT METHOD
Dividing tot_Gross by n for each genre yields the same numbers above.
# List the total gross earnings per genre
highest_tot_gross <- movies_df %>%
filter (Runtime >= 100 & Runtime <= 120) %>%
group_by(Genre) %>%
summarize(tot_Gross = sum(Gross_Earning_in_Mil))
highest_tot_gross ## # A tibble: 8 x 2
## Genre tot_Gross
## <fct> <dbl>
## 1 Action 1614.
## 2 Adventure 369.
## 3 Animation 865.
## 4 Biography 94.8
## 5 Comedy 232.
## 6 Crime NA
## 7 Drama 367.
## 8 Horror 140.# List the number of films per genre
genre_count <- movies_df %>%
filter (Runtime >= 100 & Runtime <= 120) %>%
group_by(Genre) %>%
count(Genre)
genre_count## # A tibble: 8 x 2
## # Groups: Genre [8]
## Genre n
## <fct> <int>
## 1 Action 20
## 2 Adventure 2
## 3 Animation 4
## 4 Biography 2
## 5 Comedy 8
## 6 Crime 3
## 7 Drama 7
## 8 Horror 3