Introduction

Data and information on the web is growing exponentially. All of us today use Google as our first source of knowledge - be it about finding reviews about a place or understanding a new term. There is a lot of information available on the web.

With the amount of data available over the web, it opens new horizons of possibility for a Data Scientist. As such, web scraping is a must have skill for anyone looking to unlock the massive amounts of freely available information on the web. In today’s world, all the data that you need is already available on the internet - the only thing limiting you from using it is the ability to access it. Hopefully this post allows you to overcome the access barrier.

Most of the data available over the web is not readily available in a format suitable for analysis. It is mostly semi-structured in HTML format and is not downloadable. Therefore, it requires knowledge and expertise to access this data and reshape it to eventually build a useful model.

1. What is Web Scraping?

Web scraping is a technique for converting the data present in unstructured format (HTML tags) over the web to the structured format which can easily be accessed and used.

Almost all the main languages provide ways for performing web scraping. For this example, R is used for scraping the data for the most popular feature films of 2019 from the IMDb website.

We’ll get a number of features for each of the 100 popular feature films released in 2019. We’ll look at some of the most common problems that occur while scraping data from the internet.

2. Why do we need Web Scraping?

As mentioned before, the amount of data avaiable on the web couple with the analytical possibilities using this data, the application of web scraping are only limited to your creativity. We are going to scrape data from IMDB, we could use this data to scraping movie rating data to create movie recommendation engines. Some other applications of web scraping might be:

  • Scraping text data from Wikipedia and other sources for making NLP-based systems or training deep learning models for tasks like topic recognition from the given text.
  • Scraping labeled image data from websites like Google, Flickr, etc to train image classification models.
  • Scraping data from social media sites like Facebook and Twitter for performing tasks Sentiment analysis, opinion mining, etc.
  • Scraping user reviews and feedbacks from e-commerce sites like Amazon, Flipkart, etc.

3. Ways to scrape data

There are several ways of scraping data from the web. Some of the popular ways are:

  • Human Copy-Paste: This is a slow and efficient way of scraping data from the web. This involves humans themselves analyzing and copying the data to local storage.
  • Text pattern matching: Another simple yet powerful approach to extract information from the web is by using regular expression matching facilities of programming languages. You can learn more about regular expressions here.
  • API Interface: Many websites like Facebook, Twitter, LinkedIn, etc. provides public and/ or private APIs which can be called using the standard code for retrieving the data in the prescribed format.
  • DOM Parsing: By using web browsers, programs can retrieve the dynamic content generated by client-side scripts. It is also possible to parse web pages into a DOM tree, based on which programs can retrieve parts of these pages.

We’ll use the DOM parsing here and rely on the CSS selectors of the webpage for finding the relevant fields which contain the desired information. Before we begin, there are a few prerequisites that one need in order to proficiently scrape data from any website.

4. Pre-requisites

The prerequisites for performing web scraping in R are divided into two buckets:

  • To get started with web scraping, you must have a working knowledge of R language. If you are just starting or want to brush up the basics, I’ll highly recommend following this learning path in R. During the course of this article, we’ll be using the rvest package in R authored by Hadley Wickham. You can access the documentation for rvest package here. Make sure you have this package installed. If you don’t have this package by now, you can follow the following code to install it (make sure you remove the comment #).
#install.packages('rvest')
  • Adding, knowledge of HTML and CSS will be an added advantage. One of the best sources I could find for learning HTML and CSS is this I have observed that most of the Data Scientists are not very sound with technical knowledge of HTML and CSS. Therefore, we’ll be using an open source software named Selector Gadget which will be more than sufficient for anyone in order to perform Web scraping. You can access and download the Selector Gadget extension here. Make sure that you have this extension installed by following the instructions from the website. I have done the same. I’m using Google chrome and I can access the extension in the extension bar to the top right.

Using this you can select the parts of any website and get the relevant tags to get access to that part by simply clicking on that part of the website. Note that, this is a way around to actually learning HTML & CSS and doing it manually. But to master the art of Web scraping, I’ll highly recommend you to learn HTML & CSS in order to better understand and appreciate what’s happening under the hood.

4. Scraping a webpage using R

Now, let’s get started with scraping the IMDb website for the 100 most popular feature films released in 2019. You can access them here.

#Loading the rvest package
library('rvest')
## Loading required package: xml2
#Specifying the url for desired website to be scraped
url <- 'http://www.imdb.com/search/title?count=100&release_date=2019,2019&title_type=feature'

#Reading the HTML code from the website
webpage <- read_html(url)

Now, we’ll be scraping the following data from this website.

  • Rank: The rank of the film from 1 to 100 on the list of 100 most popular feature films released in 2019.
  • Title: The title of the feature film.
  • Description: The description of the feature film.
  • Runtime: The duration of the feature film.
  • Genre: The genre of the feature film,
  • Rating: The IMDb rating of the feature film.
  • Metascore: The metascore on IMDb website for the feature film.
  • Votes: Votes cast in favor of the feature film.
  • Gross_Earning_in_Mil: The gross earnings of the feature film in millions.
  • Director: The main director of the feature film. Note, in case of multiple directors, I’ll take only the first.
  • Actor: The main actor in the feature film. Note, in case of multiple actors, I’ll take only the first.

Here’s a screenshot that contains how all these fields are arranged.

Step 1: Now, we will start by scraping the Rank field. For that, we’ll use the selector gadget to get the specific CSS selectors that encloses the rankings. You can click on the extension in your browser and select the rankings field with the cursor.

Make sure that all the rankings are selected. You can select some more ranking sections in case you are not able to get all of them and you can also de-select them by clicking on the selected section to make sure that you only have those sections highlighted that you want to scrape for that go.

Step 2: Once you are sure that you have made the right selections, you need to copy the corresponding CSS selector that you can view in the bottom center. In this case it’s .text-primary

Step 3: Once you know the CSS selector that contains the rankings, you can use this simple R code to get all the rankings:

#Using CSS selectors to scrape the rankings section
rank_data_html <- html_nodes(webpage,'.text-primary')

#Converting the ranking data to text
rank_data <- html_text(rank_data_html)

#Let's have a look at the rankings
head(rank_data)
## [1] "1." "2." "3." "4." "5." "6."

Step 4: Once you have the data, we can coerce it to the desired format. I am converting these ranks from character to numerical format.

#Data-Preprocessing: Converting rankings to numerical
rank_data<-as.numeric(rank_data)

#Let's have another look at the rankings
head(rank_data)
## [1] 1 2 3 4 5 6

Step 5: Now you can clear the selector section and select all the titles. You can visually inspect that all the titles are selected. Make any required additions and deletions with the help of your curser. I have done the same here.

Step 6: Again, I have the corresponding CSS selector for the titles - .lister-item-header a. I will use this selector to scrape all the titles using the following code.

#Using CSS selectors to scrape the title section
title_data_html <- html_nodes(webpage,'.lister-item-header a')

#Converting the title data to text
title_data <- html_text(title_data_html)

#Let's have a look at the title
head(title_data)
## [1] "Parasite"                         "Jojo Rabbit"                     
## [3] "1917"                             "Knives Out"                      
## [5] "Uncut Gems"                       "Once Upon a Time... in Hollywood"

Step 7: In the following code, I have done the same thing for scraping Description, Runtime, Genre, Rating, Metascore, Votes, Gross_Earning_in_Mil, Director, and Actor data.

#Using CSS selectors to scrape the description section
description_data_html <- html_nodes(webpage,'.ratings-bar+ .text-muted')

#Converting the description data to text
description_data <- html_text(description_data_html)

#Let's have a look at the description data
head(description_data)
## [1] "\n    A poor family, the Kims, con their way into becoming the servants of a rich family, the Parks. But their easy life gets complicated when their deception is threatened with exposure."                                
## [2] "\n    A young boy in Hitler's army finds out his mother is hiding a Jewish girl in their home."                                                                                                                             
## [3] "\n    April 6th, 1917. As a regiment assembles to wage war deep in enemy territory, two soldiers are assigned to race against time and deliver a message that will stop 1,600 men from walking straight into a deadly trap."
## [4] "\n    A detective investigates the death of a patriarch of an eccentric, combative family."                                                                                                                                 
## [5] "\n    With his debts mounting and angry collectors closing in, a fast-talking New York City jeweler risks everything in hope of staying afloat and alive."                                                                  
## [6] "\n    A faded television actor and his stunt double strive to achieve fame and success in the film industry during the final years of Hollywood's Golden Age in 1969 Los Angeles."
#Data-Preprocessing: removing '\n'
description_data<-gsub("\n","",description_data)

#Let's have another look at the description data 
head(description_data)
## [1] "    A poor family, the Kims, con their way into becoming the servants of a rich family, the Parks. But their easy life gets complicated when their deception is threatened with exposure."                                
## [2] "    A young boy in Hitler's army finds out his mother is hiding a Jewish girl in their home."                                                                                                                             
## [3] "    April 6th, 1917. As a regiment assembles to wage war deep in enemy territory, two soldiers are assigned to race against time and deliver a message that will stop 1,600 men from walking straight into a deadly trap."
## [4] "    A detective investigates the death of a patriarch of an eccentric, combative family."                                                                                                                                 
## [5] "    With his debts mounting and angry collectors closing in, a fast-talking New York City jeweler risks everything in hope of staying afloat and alive."                                                                  
## [6] "    A faded television actor and his stunt double strive to achieve fame and success in the film industry during the final years of Hollywood's Golden Age in 1969 Los Angeles."
#Using CSS selectors to scrape the Movie runtime section
runtime_data_html <- html_nodes(webpage,'.text-muted .runtime')

#Converting the runtime data to text
runtime_data <- html_text(runtime_data_html)

#Let's have a look at the runtime
head(runtime_data)
## [1] "132 min" "108 min" "119 min" "131 min" "135 min" "161 min"
#Data-Preprocessing: removing mins and converting it to numerical
runtime_data<-gsub(" min","",runtime_data)
runtime_data<-as.numeric(runtime_data)

#Let's have another look at the runtime data
head(runtime_data)
## [1] 132 108 119 131 135 161
#Using CSS selectors to scrape the Movie genre section
genre_data_html <- html_nodes(webpage,'.genre')

#Converting the genre data to text
genre_data <- html_text(genre_data_html)

#Let's have a look at the runtime
head(genre_data)
## [1] "\nComedy, Drama, Thriller            "
## [2] "\nComedy, Drama, War            "     
## [3] "\nDrama, War            "             
## [4] "\nComedy, Crime, Drama            "   
## [5] "\nCrime, Drama, Thriller            " 
## [6] "\nComedy, Drama            "
#Data-Preprocessing: removing \n
genre_data<-gsub("\n","",genre_data)

#Data-Preprocessing: removing excess spaces
genre_data<-gsub(" ","",genre_data)

#taking only the first genre of each movie
genre_data<-gsub(",.*","",genre_data)

#Convering each genre from text to factor
genre_data<-as.factor(genre_data)

#Let's have another look at the genre data
head(genre_data)
## [1] Comedy Comedy Drama  Comedy Crime  Comedy
## 9 Levels: Action Adventure Animation Biography Comedy Crime ... Thriller
#Using CSS selectors to scrape the IMDB rating section
rating_data_html <- html_nodes(webpage,'.ratings-imdb-rating strong')

#Converting the ratings data to text
rating_data <- html_text(rating_data_html)

#Let's have a look at the ratings
head(rating_data)
## [1] "8.6" "8.0" "8.5" "8.0" "7.6" "7.7"
#Data-Preprocessing: converting ratings to numerical
rating_data<-as.numeric(rating_data)

#Let's have another look at the ratings data
head(rating_data)
## [1] 8.6 8.0 8.5 8.0 7.6 7.7
#Using CSS selectors to scrape the votes section
votes_data_html <- html_nodes(webpage,'.sort-num_votes-visible span:nth-child(2)')

#Converting the votes data to text
votes_data <- html_text(votes_data_html)

#Let's have a look at the votes data
head(votes_data)
## [1] "277,675" "139,449" "196,858" "191,726" "106,465" "394,494"
#Data-Preprocessing: removing commas
votes_data<-gsub(",","",votes_data)

#Data-Preprocessing: converting votes to numerical
votes_data<-as.numeric(votes_data)

#Let's have another look at the votes data
head(votes_data)
## [1] 277675 139449 196858 191726 106465 394494
#Using CSS selectors to scrape the directors section
directors_data_html <- html_nodes(webpage,'.text-muted+ p a:nth-child(1)')

#Converting the directors data to text
directors_data <- html_text(directors_data_html)

#Let's have a look at the directors data
head(directors_data)
## [1] "Bong Joon Ho"      "Taika Waititi"     "Sam Mendes"       
## [4] "Rian Johnson"      "Benny Safdie"      "Quentin Tarantino"
#Data-Preprocessing: converting directors data into factors
directors_data<-as.factor(directors_data)

#Using CSS selectors to scrape the actors section
actors_data_html <- html_nodes(webpage,'.lister-item-content .ghost+ a')

#Converting the gross actors data to text
actors_data <- html_text(actors_data_html)

#Let's have a look at the actors data
head(actors_data)
## [1] "Kang-ho Song"         "Roman Griffin Davis"  "Dean-Charles Chapman"
## [4] "Daniel Craig"         "Adam Sandler"         "Leonardo DiCaprio"
#Data-Preprocessing: converting actors data into factors
actors_data<-as.factor(actors_data)

Pay attention to what happens when I run this patterned code for Metascore.

#Using CSS selectors to scrape the metascore section
metascore_data_html <- html_nodes(webpage,'.metascore')

#Converting the runtime data to text
metascore_data <- html_text(metascore_data_html)

#Let's have a look at the metascore data 
head(metascore_data)
## [1] "96        " "58        " "78        " "82        " "90        "
## [6] "83        "
#Data-Preprocessing: removing extra space in metascore
metascore_data<-gsub(" ","",metascore_data)

#Lets check the length of metascore data
length(metascore_data)
## [1] 96
len<- length(metascore_data) #this is just for me
mlen<-100-length(metascore_data)#you don't need these

Step 8: The length of the metascore data is 96 while we are scraping the data for 100 movies. The reason this happened is that there are 4 movies that don’t have the corresponding Metascore fields.

Step 9: Because of the formatting of html, or the nature of the fields, this situation can arise while scraping any website. Unfortunately, if we simply add NA’s to last 4 entries, it will map NA as Metascore for movies 96 to 100 while in reality, the data is missing for some other movies. After a visual inspection, I found that the Metascore is missing for movies 32, 49, 81, and 89. The following function can get around this problem.

for (i in c(32,49,81,89)){
  a<-metascore_data[1:(i-1)]
  b<-metascore_data[i:length(metascore_data)]
  metascore_data<-append(a,list("NA"))
  metascore_data<-append(metascore_data,b)
}

#Data-Preprocessing: converting metascore to numerical
metascore_data<-as.numeric(metascore_data)
## Warning: NAs introduced by coercion

## Warning: NAs introduced by coercion

## Warning: NAs introduced by coercion

## Warning: NAs introduced by coercion
#Let's have another look at length of the metascore data
length(metascore_data)
## [1] 100
#Let's look at summary statistics
summary(metascore_data)
##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max.    NA's 
##   24.00   51.75   62.00   62.02   72.25   96.00       4
#Using CSS selectors to scrape the gross revenue section
gross_data_html <- html_nodes(webpage,'.ghost~ .text-muted+ span')

#Converting the gross revenue data to text
gross_data <- html_text(gross_data_html)

#Let's have a look at the votes data
head(gross_data)
## [1] "$0.35M"   "$135.37M" "$192.73M" "$27.33M"  "$0.43M"   "$433.03M"
#Data-Preprocessing: removing '$' and 'M' signs
gross_data<-gsub("M","",gross_data)
gross_data<-substring(gross_data,2,6)

#Let's check the length of gross data
length(gross_data)
## [1] 44

Oops, it happened again. Gross earnings only has values for 44 movies.

#fix first entry and last entry so loop will work
gross_start<-NA #Parasite missing earnings
gross_end<-NA #So is Buffaloed
gross_data<-append(gross_start,gross_data)
gross_data<-append(gross_data,gross_end)
gross_data<-as.numeric(gross_data)

#Filling missing entries with NA
for (i in c(3,4,5,8,9,10,11,12,13,14,16,17,20,21,22,23,24,28,29,30,31,32,33,35,36,37,38,45,46,49,51,52,53,54,60,61,65,66,67,68,70,72,73,76,77,78,80,81,82,89,90,93,95,97)){
  a<-gross_data[1:(i-1)]
  b<-gross_data[i:length(gross_data)]
  gross_data<-append(a,list("NA"))
  gross_data<-append(gross_data,b)
}

#Data-Preprocessing: converting gross to numerical
gross_data<-as.numeric(as.character(gross_data))
## Warning: NAs introduced by coercion
#Let's have another look at the length of gross data
length(gross_data)
## [1] 100
summary(gross_data)
##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max.    NA's 
##    0.03   21.30   41.16  122.37  149.45  858.30      56

Step 11: Now we have successfully scraped all the 11 features for the 100 most popular feature films released in 2019. We can combine them to create a dataframe and inspect its structure.

#Combining all the lists to form a data frame
movies_df<-data.frame(Rank = rank_data, Title = title_data,
Description = description_data, Runtime = runtime_data,
Genre = genre_data, Rating = rating_data,
Metascore = metascore_data, Votes = votes_data,
Gross_Earning_in_Mil = gross_data,
Director = directors_data, Actor = actors_data)

#Structure of the data frame
str(movies_df)
## 'data.frame':    100 obs. of  11 variables:
##  $ Rank                : num  1 2 3 4 5 6 7 8 9 10 ...
##  $ Title               : Factor w/ 100 levels "1917","21 Bridges",..: 62 46 1 51 93 61 47 80 33 53 ...
##  $ Description         : Factor w/ 100 levels "    A 30-something woman navigating through love and heartbreak over the course of one year. During that time, "| __truncated__,..: 23 31 43 9 97 10 59 38 36 66 ...
##  $ Runtime             : num  132 108 119 131 135 161 122 113 152 135 ...
##  $ Genre               : Factor w/ 9 levels "Action","Adventure",..: 5 5 7 5 6 5 6 1 1 7 ...
##  $ Rating              : num  8.6 8 8.5 8 7.6 7.7 8.6 8.1 8.2 8 ...
##  $ Metascore           : num  96 58 78 82 90 83 59 51 81 91 ...
##  $ Votes               : num  277675 139449 196858 191726 106465 ...
##  $ Gross_Earning_in_Mil: num  NA 0.35 NA NA NA ...
##  $ Director            : Factor w/ 99 levels "Adrian Grunberg",..: 13 87 82 73 10 72 93 34 38 33 ...
##  $ Actor               : Factor w/ 93 levels "Aaron Paul","Adam Driver",..: 44 69 19 16 3 51 38 57 56 74 ...

6. Analyzing scraped data from the web

Once you have the data, you can analyze and drawing inferences, train machine learning models over this data, etc. I have gone on to create some interesting visualization out of the data we have just scraped. Follow the visualizations and answer the questions given below.

Question 1: Which Genre had the longest runtimes?

library(ggplot2)
ggplot(movies_df,aes(x=reorder(Genre,Runtime,FUN=median),y=Runtime)) + geom_boxplot(aes(fill = reorder(Genre,Runtime,FUN=median)))+xlab("Genre")+scale_fill_discrete(guide = guide_legend(title = "Genre"))+theme_bw(base_size = 10)

Question 2: Which genre has the highest votes?

ggplot(movies_df,aes(x=reorder(Genre,-Rating,fun=sum),y=Rating,fill=Genre))+geom_bar(stat='identity') + scale_fill_hue(c=40) + theme(legend.position="none") + xlab("Genre")

Question 3: Which genre has the highest average gross earnings in runtime 100 to 120.

ggplot(movies_df,aes(x=Runtime,y=Gross_Earning_in_Mil))+
geom_point(aes(size=Rating,col=Genre))
## Warning: Removed 56 rows containing missing values (geom_point).

End Notes

Now you might understand some of the practical complications of web scraping in R and a good idea of how to work around them. Most of the data on the web is freely accessible, it’s just in an unstructured format. Web scraping is a really handy skill for any data scientist.