Exact copy from the guild

4. Scraping a webpage using R

Now, let’s get started with scraping the IMDb website for the 100 most popular feature films released in 2016. You can access them here.

#Loading the rvest package
library('rvest')
## Warning: package 'rvest' was built under R version 4.1.3
#Specifying the url for desired website to be scraped
url <- 'http://www.imdb.com/search/title?count=100&release_date=2016,2016&title_type=feature'

#Reading the HTML code from the website
webpage <- read_html(url)

##Step 1: Now, we will start by scraping the Rank field. For that, we’ll use the selector gadget to get the specific CSS selectors that encloses the rankings. You can click on the extension in your browser and select the rankings field with the cursor.

##Make sure that all the rankings are selected. You can select some more ranking sections in case you are not able to get all of them and you can also de-select them by clicking on the selected section to make sure that you only have those sections highlighted that you want to scrape for that go.

##Step 2: Once you are sure that you have made the right selections, you need to copy the corresponding CSS selector that you can view in the bottom center.

##Step 3: Once you know the CSS selector that contains the rankings, you can use this simple R code to get all the rankings:

#Using CSS selectors to scrape the rankings section
rank_data_html <- html_nodes(webpage,'.text-primary')

#Converting the ranking data to text
rank_data <- html_text(rank_data_html)

#Let's have a look at the rankings
head(rank_data)
## [1] "1." "2." "3." "4." "5." "6."

##Step 4: Once you have the data, make sure that it looks in the desired format. I am preprocessing my data to convert it to numerical format.

#Data-Preprocessing: Converting rankings to numerical
rank_data<-as.numeric(rank_data)

#Let's have another look at the rankings
head(rank_data)
## [1] 1 2 3 4 5 6

##Step 5: Now you can clear the selector section and select all the titles. You can visually inspect that all the titles are selected. Make any required additions and deletions with the help of your curser. I have done the same here.

##Step 6: Again, I have the corresponding CSS selector for the titles – .lister-item-header a. I will use this selector to scrape all the titles using the following code.

#Using CSS selectors to scrape the title section
title_data_html <- html_nodes(webpage,'.lister-item-header a')

#Converting the title data to text
title_data <- html_text(title_data_html)

#Let's have a look at the title
head(title_data)
## [1] "Batman v Superman: Dawn of Justice Ultimate Edition"
## [2] "Fantastic Beasts and Where to Find Them"            
## [3] "Suicide Squad"                                      
## [4] "Deadpool"                                           
## [5] "Batman v Superman: Dawn of Justice"                 
## [6] "Hacksaw Ridge"

##Step 7: In the following code, I have done the same thing for scraping – Description, Runtime, Genre, Rating, Metascore, Votes, Gross_Earning_in_Mil , Director and Actor data.

#Using CSS selectors to scrape the description section
description_data_html <- html_nodes(webpage,'.ratings-bar+ .text-muted')

#Converting the description data to text
description_data <- html_text(description_data_html)

#Let's have a look at the description data
head(description_data)
## [1] "\nBatman is manipulated by Lex Luthor to fear Superman. Superman´s existence is meanwhile dividing the world and he is framed for murder during an international crisis. The heroes clash and force the neutral Wonder Woman to reemerge."
## [2] "\nThe adventures of writer Newt Scamander in New York's secret community of witches and wizards seventy years before Harry Potter reads his book in school."                                                                              
## [3] "\nA secret government agency recruits some of the most dangerous incarcerated super-villains to form a defensive task force. Their first mission: save the world from the apocalypse."                                                    
## [4] "\nA wisecracking mercenary gets experimented on and becomes immortal but ugly, and sets out to track down the man who ruined his looks."                                                                                                  
## [5] "\nFearing that the actions of Superman are left unchecked, Batman takes on the Man of Steel, while the world wrestles with what kind of a hero it really needs."                                                                          
## [6] "\nWorld War II American Army Medic Desmond T. Doss, who served during the Battle of Okinawa, refuses to kill people and becomes the first man in American history to receive the Medal of Honor without firing a shot."
#Data-Preprocessing: removing '\n'
description_data<-gsub("\n","",description_data)

#Let's have another look at the description data 
head(description_data)
## [1] "Batman is manipulated by Lex Luthor to fear Superman. Superman´s existence is meanwhile dividing the world and he is framed for murder during an international crisis. The heroes clash and force the neutral Wonder Woman to reemerge."
## [2] "The adventures of writer Newt Scamander in New York's secret community of witches and wizards seventy years before Harry Potter reads his book in school."                                                                              
## [3] "A secret government agency recruits some of the most dangerous incarcerated super-villains to form a defensive task force. Their first mission: save the world from the apocalypse."                                                    
## [4] "A wisecracking mercenary gets experimented on and becomes immortal but ugly, and sets out to track down the man who ruined his looks."                                                                                                  
## [5] "Fearing that the actions of Superman are left unchecked, Batman takes on the Man of Steel, while the world wrestles with what kind of a hero it really needs."                                                                          
## [6] "World War II American Army Medic Desmond T. Doss, who served during the Battle of Okinawa, refuses to kill people and becomes the first man in American history to receive the Medal of Honor without firing a shot."
#Using CSS selectors to scrape the Movie runtime section
runtime_data_html <- html_nodes(webpage,'.text-muted .runtime')

#Converting the runtime data to text
runtime_data <- html_text(runtime_data_html)

#Let's have a look at the runtime
head(runtime_data)
## [1] "182 min" "132 min" "123 min" "108 min" "152 min" "139 min"
#Data-Preprocessing: removing mins and converting it to numerical

runtime_data<-gsub(" min","",runtime_data)
runtime_data<-as.numeric(runtime_data)

#Let's have another look at the runtime data
head(runtime_data)
## [1] 182 132 123 108 152 139
#Using CSS selectors to scrape the Movie genre section
genre_data_html <- html_nodes(webpage,'.genre')

#Converting the genre data to text
genre_data <- html_text(genre_data_html)

#Let's have a look at the runtime
head(genre_data)
## [1] "\nAction, Adventure, Sci-Fi            " 
## [2] "\nAdventure, Family, Fantasy            "
## [3] "\nAction, Adventure, Fantasy            "
## [4] "\nAction, Adventure, Comedy            " 
## [5] "\nAction, Adventure, Sci-Fi            " 
## [6] "\nBiography, Drama, History            "
#Data-Preprocessing: removing \n
genre_data<-gsub("\n","",genre_data)

#Data-Preprocessing: removing excess spaces
genre_data<-gsub(" ","",genre_data)

#taking only the first genre of each movie
genre_data<-gsub(",.*","",genre_data)

#Convering each genre from text to factor
genre_data<-as.factor(genre_data)

#Let's have another look at the genre data
head(genre_data)
## [1] Action    Adventure Action    Action    Action    Biography
## Levels: Action Adventure Animation Biography Comedy Crime Drama Horror
#Using CSS selectors to scrape the IMDB rating section
rating_data_html <- html_nodes(webpage,'.ratings-imdb-rating strong')

#Converting the ratings data to text
rating_data <- html_text(rating_data_html)

#Let's have a look at the ratings
head(rating_data)
## [1] "8.4" "7.3" "5.9" "8.0" "6.5" "8.1"
#Data-Preprocessing: converting ratings to numerical
rating_data<-as.numeric(rating_data)

#Let's have another look at the ratings data
head(rating_data)
## [1] 8.4 7.3 5.9 8.0 6.5 8.1
#Using CSS selectors to scrape the votes section
votes_data_html <- html_nodes(webpage,'.sort-num_votes-visible span:nth-child(2)')

#Converting the votes data to text
votes_data <- html_text(votes_data_html)

#Let's have a look at the votes data
head(votes_data)
## [1] "10,799"  "447,862" "669,491" "989,485" "683,431" "497,235"
#Data-Preprocessing: removing commas
votes_data<-gsub(",","",votes_data)

#Data-Preprocessing: converting votes to numerical
votes_data<-as.numeric(votes_data)

#Let's have another look at the votes data
head(votes_data)
## [1]  10799 447862 669491 989485 683431 497235
#Using CSS selectors to scrape the directors section
## directors_data_html <- html_nodes(webpage,'.text-muted+ p a:nth-child(1)')

#Converting the directors data to text
## directors_data <- html_text(directors_data_html)

#Let's have a look at the directors data
## head(directors_data)
#Data-Preprocessing: converting directors data into factors
## directors_data<-as.factor(directors_data)

#Using CSS selectors to scrape the actors section
## actors_data_html <- html_nodes(webpage,'.lister-item-content .ghost+ a')

#Converting the gross actors data to text
## actors_data <- html_text(actors_data_html)

#Let's have a look at the actors data
## head(actors_data)
#Data-Preprocessing: converting actors data into factors
## actors_data<-as.factor(actors_data)
#But, I want you to closely follow what happens when I do the same thing for Metascore data.

#Using CSS selectors to scrape the metascore section
## metascore_data_html <- html_nodes(webpage,'.metascore')

#Converting the runtime data to text
## metascore_data <- html_text(metascore_data_html)

#Let's have a look at the metascore data 
## head(metascore_data)
#Data-Preprocessing: removing extra space in metascore
## metascore_data<-gsub(" ","",metascore_data)

#Lets check the length of metascore data
## length(metascore_data)

##Step 8: The length of the metascore data is 96 while we are scraping the data for 100 movies. The reason this happened is that there are 4 movies that don’t have the corresponding Metascore fields.

##Step 9: It is a practical situation which can arise while scraping any website. Unfortunately, if we simply add NA’s to last 4 entries, it will map NA as Metascore for movies 96 to 100 while in reality, the data is missing for some other movies. After a visual inspection, I found that the Metascore is missing for movies 39, 73, 80 and 89. I have written the following function to get around this problem.

## for (i in c(39,73,80,89)){

## a<-metascore_data[1:(i-1)]

## b<-metascore_data[i:length(metascore_data)]

## metascore_data<-append(a,list("NA"))

## metascore_data<-append(metascore_data,b)

## }

#Data-Preprocessing: converting metascore to numerical
## metascore_data<-as.numeric(metascore_data)

#Let's have another look at length of the metascore data

## length(metascore_data)
#Let's look at summary statistics
## summary(metascore_data)

##Step 10: The same thing happens with the Gross variable which represents gross earnings of that movie in millions. I have use the same solution to work my way around:

#Using CSS selectors to scrape the gross revenue section
gross_data_html <- html_nodes(webpage,'.ghost~ .text-muted+ span')

#Converting the gross revenue data to text
gross_data <- html_text(gross_data_html)

#Let's have a look at the votes data
head(gross_data)
## [1] "$234.04M" "$325.10M" "$363.07M" "$330.36M" "$67.21M"  "$151.10M"
#Data-Preprocessing: removing '$' and 'M' signs
gross_data<-gsub("M","",gross_data)

gross_data<-substring(gross_data,2,6)

#Let's check the length of gross data
length(gross_data)
## [1] 89
#Filling missing entries with NA
for (i in c(17,39,49,52,57,64,66,73,76,77,80,87,88,89)){

a<-gross_data[1:(i-1)]

b<-gross_data[i:length(gross_data)]

gross_data<-append(a,list("NA"))

gross_data<-append(gross_data,b)

}

#Data-Preprocessing: converting gross to numerical
gross_data<-as.numeric(gross_data)
## Warning: NAs introduced by coercion

## Warning: NAs introduced by coercion

## Warning: NAs introduced by coercion

## Warning: NAs introduced by coercion

## Warning: NAs introduced by coercion

## Warning: NAs introduced by coercion

## Warning: NAs introduced by coercion

## Warning: NAs introduced by coercion

## Warning: NAs introduced by coercion

## Warning: NAs introduced by coercion

## Warning: NAs introduced by coercion

## Warning: NAs introduced by coercion

## Warning: NAs introduced by coercion

## Warning: NAs introduced by coercion
#Let's have another look at the length of gross data
length(gross_data)
## [1] 103
summary(gross_data)
##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max.    NA's 
##    0.18   18.71   57.64   99.38  126.60  532.10      14

##Step 11: Now we have successfully scraped all the 11 features for the 100 most popular feature films released in 2016. Let’s combine them to create a dataframe and inspect its structure.

#Combining all the lists to form a data frame
## movies_df<-data.frame(Rank = rank_data, Title = title_data,

## Description = description_data, Runtime = runtime_data,

## Genre = genre_data, Rating = rating_data,
## Genre = genre_data, Rating = rating_data)
## Metascore = metascore_data, 

## Votes = votes_data,  
## Votes = votes_data)

## Gross_Earning_in_Mil = gross_data,
## Gross_Earning_in_Mil = gross_data)

## Director = directors_data, Actor = actors_data)

## movies_df<-data.frame(Rank = rank_data, Title = title_data, Description = description_data, Runtime = runtime_data, Genre = genre_data, Rating = rating_data, Metascore = metascore_data, Votes = votes_data,   Gross_Earning_in_Mil = gross_data)  

## movies_df<-data.frame(Rank = rank_data, Title = title_data, Description = description_data, Runtime = runtime_data, Genre = genre_data, Rating = rating_data, Votes = votes_data, Gross_Earning_in_Mil = gross_data) 

movies_df<-data.frame(Rank = rank_data, Title = title_data, Description = description_data, Runtime = runtime_data, Genre = genre_data, Rating = rating_data, Votes = votes_data) 

#Structure of the data frame

str(movies_df)
## 'data.frame':    100 obs. of  7 variables:
##  $ Rank       : num  1 2 3 4 5 6 7 8 9 10 ...
##  $ Title      : chr  "Batman v Superman: Dawn of Justice Ultimate Edition" "Fantastic Beasts and Where to Find Them" "Suicide Squad" "Deadpool" ...
##  $ Description: chr  "Batman is manipulated by Lex Luthor to fear Superman. Superman´s existence is meanwhile dividing the world and "| __truncated__ "The adventures of writer Newt Scamander in New York's secret community of witches and wizards seventy years bef"| __truncated__ "A secret government agency recruits some of the most dangerous incarcerated super-villains to form a defensive "| __truncated__ "A wisecracking mercenary gets experimented on and becomes immortal but ugly, and sets out to track down the man"| __truncated__ ...
##  $ Runtime    : num  182 132 123 108 152 139 128 144 115 107 ...
##  $ Genre      : Factor w/ 8 levels "Action","Adventure",..: 1 2 1 1 1 4 5 1 1 3 ...
##  $ Rating     : num  8.4 7.3 5.9 8 6.5 8.1 8 6.9 7.5 7.6 ...
##  $ Votes      : num  10799 447862 669491 989485 683431 ...

##6. Analyzing scraped data from the web ## Once you have the data, you can perform several tasks like analyzing the data, drawing inferences from it, training machine learning models over this data, etc. I have gone on to create some interesting visualization out of the data we have just scraped. Follow the visualizations and answer the questions given below. Post your answers in the comment section below.

library('ggplot2')

qplot(data = movies_df,Runtime,fill = Genre,bins = 30)

#Question 1: Based on the above data, which movie from which Genre had the longest runtime?

ggplot(movies_df,aes(x=Runtime,y=Rating))+
geom_point(aes(size=Votes,col=Genre))

## geom_point(aes(size=Rank,col=Genre))

#Question 2: Based on the above data, in the Runtime of 130-160 mins, which genre has the highest votes?

## Gross_Earning_in_Mil == gross_data
## ggplot(movies_df,aes(x=Runtime,y=Gross_Earning_in_Mil))+
## geom_point(aes(size=Rating,col=Genre))
 
#Question 3: Based on the above data, across all genres which genre has the highest average gross earnings in runtime 100 to 120.

##End Notes ##I believe this article would have given you a complete understanding of the web scraping in R. Now, you also have a fair idea of the problems which you might come across and how you can make your way around them. As most of the data on the web is present in an unstructured format, web scraping is a really handy skill for any data scientist.

##Also, you can post the answers to the above three questions in the comment section below. Did you enjoy reading this article? Do share your views with me. If you have any doubts/questionsns feel free to drop them below.