Web Scraping

Installing rvest package

#install.packages('rvest')
library('rvest')
library('tidyverse')

## -- Attaching packages --------------------------------------- tidyverse 1.3.1 --

## v ggplot2 3.3.5     v purrr   0.3.4
## v tibble  3.1.4     v dplyr   1.0.7
## v tidyr   1.1.3     v stringr 1.4.0
## v readr   2.0.1     v forcats 0.5.1

## -- Conflicts ------------------------------------------ tidyverse_conflicts() --
## x dplyr::filter()         masks stats::filter()
## x readr::guess_encoding() masks rvest::guess_encoding()
## x dplyr::lag()            masks stats::lag()

Step 1: Now, we will start by scraping the Rank field. For that, we’ll use the selector gadget to get the specific CSS selectors that encloses the rankings. You can click on the extension in your browser and select the rankings field with the cursor.

Step 2: Once you are sure that you have made the right selections, you need to copy the corresponding CSS selector that you can view in the bottom center.

#Specifying the url for desired website to be scraped
url <- 'https://www.imdb.com/search/title/?count=100&release_date=2016,2016&title_type=feature'

#Reading the HTML code from the website
webpage <- read_html(url)

Step 3: Once you know the CSS selector that contains the rankings, you can use this simple R code to get all the rankings:

#Using CSS selectors to scrape the rankings section
rank_data_html <- html_nodes(webpage,'.text-primary')

#Converting the ranking data to text
rank_data <- html_text(rank_data_html)

#Let's have a look at the rankings
head(rank_data)

## [1] "1." "2." "3." "4." "5." "6."

Step 4: Once you have the data, make sure that it looks in the desired format. I am preprocessing my data to convert it to numerical format.

#Data-Preprocessing: Converting rankings to numerical
rank_data<-as.numeric(rank_data)

#Let's have another look at the rankings
head(rank_data)

## [1] 1 2 3 4 5 6

Step 5: Now you can clear the selector section and select all the titles. You can visually inspect that all the titles are selected. Make any required additions and deletions with the help of your curser. I have done the same here.

Step 6: Again, I have the corresponding CSS selector for the titles – .lister-item-header a. I will use this selector to scrape all the titles using the following code.

#Using CSS selectors to scrape the title section
title_data_html <- html_nodes(webpage,'.lister-item-header a')

#Converting the title data to text
title_data <- html_text(title_data_html)

#Let's have a look at the title
head(title_data)

## [1] "Arrival"                       "Ghostbusters: Answer the Call"
## [3] "Train to Busan"                "Suicide Squad"                
## [5] "Hacksaw Ridge"                 "Hush"

Step 7: In the following code, I have done the same thing for scraping – Description, Runtime, Genre, Rating, Metascore, Votes, Gross_Earning_in_Mil , Director and Actor data.

#Using CSS selectors to scrape the description section
description_data_html <- html_nodes(webpage,'.ratings-bar+ .text-muted')

#Converting the description data to text
description_data <- html_text(description_data_html)

#Let's have a look at the description data
head(description_data)

## [1] "\nA linguist works with the military to communicate with alien lifeforms after twelve mysterious spacecraft appear around the world."                                                                                  
## [2] "\nFollowing a ghost invasion of Manhattan, paranormal enthusiasts Erin Gilbert and Abby Yates, nuclear engineer Jillian Holtzmann, and subway worker Patty Tolan band together to stop the otherworldly threat."       
## [3] "\nWhile a zombie virus breaks out in South Korea, passengers struggle to survive on the train from Seoul to Busan."                                                                                                    
## [4] "\nA secret government agency recruits some of the most dangerous incarcerated super-villains to form a defensive task force. Their first mission: save the world from the apocalypse."                                 
## [5] "\nWorld War II American Army Medic Desmond T. Doss, who served during the Battle of Okinawa, refuses to kill people and becomes the first man in American history to receive the Medal of Honor without firing a shot."
## [6] "\nA deaf and mute writer who retreated into the woods to live a solitary life must fight for her life in silence when a masked killer appears at her window."

#Data-Preprocessing: removing '\n'
description_data<-gsub("\n","",description_data)

#Let's have another look at the description data 
head(description_data)

## [1] "A linguist works with the military to communicate with alien lifeforms after twelve mysterious spacecraft appear around the world."                                                                                  
## [2] "Following a ghost invasion of Manhattan, paranormal enthusiasts Erin Gilbert and Abby Yates, nuclear engineer Jillian Holtzmann, and subway worker Patty Tolan band together to stop the otherworldly threat."       
## [3] "While a zombie virus breaks out in South Korea, passengers struggle to survive on the train from Seoul to Busan."                                                                                                    
## [4] "A secret government agency recruits some of the most dangerous incarcerated super-villains to form a defensive task force. Their first mission: save the world from the apocalypse."                                 
## [5] "World War II American Army Medic Desmond T. Doss, who served during the Battle of Okinawa, refuses to kill people and becomes the first man in American history to receive the Medal of Honor without firing a shot."
## [6] "A deaf and mute writer who retreated into the woods to live a solitary life must fight for her life in silence when a masked killer appears at her window."

#Using CSS selectors to scrape the Movie runtime section
runtime_data_html <- html_nodes(webpage,'.text-muted .runtime')

#Converting the runtime data to text
runtime_data <- html_text(runtime_data_html)

#Let's have a look at the runtime
head(runtime_data)

## [1] "116 min" "117 min" "118 min" "123 min" "139 min" "82 min"

#Data-Preprocessing: removing mins and converting it to numerical

runtime_data<-gsub(" min","",runtime_data)
runtime_data<-as.numeric(runtime_data)

#Let's have another look at the runtime data
head(runtime_data)

## [1] 116 117 118 123 139  82

#Using CSS selectors to scrape the Movie genre section
genre_data_html <- html_nodes(webpage,'.genre')

#Converting the genre data to text
genre_data <- html_text(genre_data_html)

#Let's have a look at the runtime
head(genre_data)

## [1] "\nDrama, Sci-Fi            "             
## [2] "\nAction, Comedy, Fantasy            "   
## [3] "\nAction, Horror, Thriller            "  
## [4] "\nAction, Adventure, Fantasy            "
## [5] "\nBiography, Drama, History            " 
## [6] "\nHorror, Thriller            "

#Data-Preprocessing: removing \n
genre_data<-gsub("\n","",genre_data)

#Data-Preprocessing: removing excess spaces
genre_data<-gsub(" ","",genre_data)

#taking only the first genre of each movie
genre_data<-gsub(",.*","",genre_data)

#Convering each genre from text to factor
genre_data<-as.factor(genre_data)

#Let's have another look at the genre data
head(genre_data)

## [1] Drama     Action    Action    Action    Biography Horror   
## 9 Levels: Action Adventure Animation Biography Comedy Crime Drama ... Mystery

#Using CSS selectors to scrape the IMDB rating section
rating_data_html <- html_nodes(webpage,'.ratings-imdb-rating strong')

#Converting the ratings data to text
rating_data <- html_text(rating_data_html)

#Let's have a look at the ratings
head(rating_data)

## [1] "7.9" "6.5" "7.6" "5.9" "8.1" "6.6"

#Data-Preprocessing: converting ratings to numerical
rating_data<-as.numeric(rating_data)

#Let's have another look at the ratings data
head(rating_data)

## [1] 7.9 6.5 7.6 5.9 8.1 6.6

#Using CSS selectors to scrape the votes section
votes_data_html <- html_nodes(webpage,'.sort-num_votes-visible span:nth-child(2)')

#Converting the votes data to text
votes_data <- html_text(votes_data_html)

#Let's have a look at the votes data
head(votes_data)

## [1] "642,307" "214,407" "193,102" "654,870" "472,253" "119,264"

#Data-Preprocessing: removing commas
votes_data<-gsub(",","",votes_data)

#Data-Preprocessing: converting votes to numerical
votes_data<-as.numeric(votes_data)

#Let's have another look at the votes data
head(votes_data)

## [1] 642307 214407 193102 654870 472253 119264

#Using CSS selectors to scrape the directors section
directors_data_html <- html_nodes(webpage,'.text-muted+ p a:nth-child(1)')

#Converting the directors data to text
directors_data <- html_text(directors_data_html)

#Let's have a look at the directors data
head(directors_data)

## [1] "Denis Villeneuve" "Paul Feig"        "Sang-ho Yeon"     "David Ayer"      
## [5] "Mel Gibson"       "Mike Flanagan"

#Data-Preprocessing: converting directors data into factors
directors_data<-as.factor(directors_data)

#Using CSS selectors to scrape the actors section
actors_data_html <- html_nodes(webpage,'.lister-item-content .ghost+ a')

#Converting the gross actors data to text
actors_data <- html_text(actors_data_html)

#Let's have a look at the actors data
head(actors_data)

## [1] "Amy Adams"          "Melissa McCarthy"   "Gong Yoo"          
## [4] "Will Smith"         "Andrew Garfield"    "John Gallagher Jr."

#Data-Preprocessing: converting actors data into factors
actors_data<-as.factor(actors_data)

Metascore data.

#Using CSS selectors to scrape the metascore section
metascore_data_html <- html_nodes(webpage,'.metascore')

#Converting the runtime data to text
metascore_data <- html_text(metascore_data_html)

#Let's have a look at the metascore data 
head(metascore_data)

## [1] "81        " "60        " "72        " "40        " "71        "
## [6] "67        "

#Data-Preprocessing: removing extra space in metascore
metascore_data<-gsub(" ","",metascore_data)

#Lets check the length of metascore data
length(metascore_data)

## [1] 96

Step 8: The length of the metascore data is 96 while we are scraping the data for 100 movies. The reason this happened is that there are 4 movies that don’t have the corresponding Metascore fields.

Step 9: It is a practical situation which can arise while scraping any website. Unfortunately, if we simply add NA’s to last 4 entries, it will map NA as Metascore for movies 96 to 100 while in reality, the data is missing for some other movies. After a visual inspection, I found that the Metascore is missing for movies 44, 45, 46 and 54. I have written the following function to get around this problem.

ratings_bar_data <- html_nodes(webpage,'.ratings-bar') %>% 
# scrape the ratings bar and convert to text
  html_text2()
head(ratings_bar_data) # look at the ratings bar

## [1] "7.9\nRate this\n 1 2 3 4 5 6 7 8 9 10 7.9/10 X \n81 Metascore"
## [2] "6.5\nRate this\n 1 2 3 4 5 6 7 8 9 10 6.5/10 X \n60 Metascore"
## [3] "7.6\nRate this\n 1 2 3 4 5 6 7 8 9 10 7.6/10 X \n72 Metascore"
## [4] "5.9\nRate this\n 1 2 3 4 5 6 7 8 9 10 5.9/10 X \n40 Metascore"
## [5] "8.1\nRate this\n 1 2 3 4 5 6 7 8 9 10 8.1/10 X \n71 Metascore"
## [6] "6.6\nRate this\n 1 2 3 4 5 6 7 8 9 10 6.6/10 X \n67 Metascore"

metascore_data <- str_match(ratings_bar_data, "\\d{2} Metascore") %>%  
# extract Metascore 
  str_match("\\d{2}") %>% 
  as.numeric() # convert to number  
length(metascore_data)

## [1] 100

metascore_data

##   [1] 81 60 72 40 71 67 71 67 62 65 81 65 72 81 57 75 94 48 84 65 52 44 54 51 79
##  [26] 66 70 72 51 81 78 65 82 NA 59 76 41 99 25 74 32 48 96 58 NA 57 51 88 68 68
##  [51] 62 65 49 35 NA 58 45 42 74 35 23 69 44 79 18 42 42 60 78 23 81 59 77 46 68
##  [76] 58 66 66 78 52 42 36 49 45 55 77 62 70 76 28 79 77 55 90 32 NA 72 58 51 47

summary(metascore_data)

##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max.    NA's 
##   18.00   48.75   63.50   61.02   74.25   99.00       4

Step 10: The same thing happens with the Gross variable which represents gross earnings of that movie in millions. I have use the same solution to work my way around:

#Using CSS selectors to scrape the gross revenue section
gross_data_html <- html_nodes(webpage,'.ghost~ .text-muted+ span')

#Converting the gross revenue data to text
gross_data <- html_text(gross_data_html)

#Let's have a look at the votes data
head(gross_data)

## [1] "$100.55M" "$128.34M" "$2.13M"   "$325.10M" "$67.21M"  "$89.22M"

#Data-Preprocessing: removing '$' and 'M' signs
gross_data<-gsub("M","",gross_data)

gross_data<-substring(gross_data,2,6)

#Let's check the length of gross data
length(gross_data)

## [1] 89

#Filling missing entries with NA

# scrape the votess bar and convert to text
votes_bar_data <- html_nodes(webpage,'.sort-num_votes-visible') %>% 
  html_text2()
head(votes_bar_data)  # look at the votes bar data

## [1] "Votes: 642,307 | Gross: $100.55M" "Votes: 214,407 | Gross: $128.34M"
## [3] "Votes: 193,102 | Gross: $2.13M"   "Votes: 654,870 | Gross: $325.10M"
## [5] "Votes: 472,253 | Gross: $67.21M"  "Votes: 119,264"

gross_data <- str_match(votes_bar_data, "\\$.+$") # extract the gross earnings
gross_data <- gsub("M","",gross_data) # clean data: remove 'M' sign 
gross_data <- substring(gross_data,2,6) %>% # clean data: remove '$' sign                    
as.numeric()
length(gross_data)

## [1] 100

summary(gross_data)

##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max.    NA's 
##    0.15   14.43   52.85   87.35  102.40  532.10      11

Step 11: Now we have successfully scraped all the 11 features for the 100 most popular feature films released in 2016. Let’s combine them to create a dataframe and inspect its structure.

#Combining all the lists to form a data frame
movies_df<-data.frame(Rank = rank_data, Title = title_data,

Description = description_data, Runtime = runtime_data,

Genre = genre_data, Rating = rating_data,

Metascore = metascore_data, Votes = votes_data,                                                             Gross_Earning_in_Mil = gross_data,

Director = directors_data, Actor = actors_data)

#Structure of the data frame

str(movies_df)

## 'data.frame':    100 obs. of  11 variables:
##  $ Rank                : num  1 2 3 4 5 6 7 8 9 10 ...
##  $ Title               : chr  "Arrival" "Ghostbusters: Answer the Call" "Train to Busan" "Suicide Squad" ...
##  $ Description         : chr  "A linguist works with the military to communicate with alien lifeforms after twelve mysterious spacecraft appea"| __truncated__ "Following a ghost invasion of Manhattan, paranormal enthusiasts Erin Gilbert and Abby Yates, nuclear engineer J"| __truncated__ "While a zombie virus breaks out in South Korea, passengers struggle to survive on the train from Seoul to Busan." "A secret government agency recruits some of the most dangerous incarcerated super-villains to form a defensive "| __truncated__ ...
##  $ Runtime             : num  116 117 118 123 139 82 88 116 117 134 ...
##  $ Genre               : Factor w/ 9 levels "Action","Adventure",..: 7 1 1 1 4 8 6 7 8 8 ...
##  $ Rating              : num  7.9 6.5 7.6 5.9 8.1 6.6 7.1 7.5 7.3 7.3 ...
##  $ Metascore           : num  81 60 72 40 71 67 71 67 62 65 ...
##  $ Votes               : num  642307 214407 193102 654870 472253 ...
##  $ Gross_Earning_in_Mil: num  100.5 128.3 2.13 325.1 67.21 ...
##  $ Director            : Factor w/ 97 levels "Alex Proyas",..: 26 72 84 21 59 61 31 94 54 41 ...
##  $ Actor               : Factor w/ 91 levels "Aamir Khan","Aaron Poole",..: 5 68 39 91 6 48 82 5 42 89 ...

I have now successfully scraped the IMDb website for the 100 most popular feature films released in 2016.

Analyzing scraped data from the web

Follow the visualizations and answer the questions given below.

library('ggplot2')
library(plotly)

## 
## Attaching package: 'plotly'

## The following object is masked from 'package:ggplot2':
## 
##     last_plot

## The following object is masked from 'package:stats':
## 
##     filter

## The following object is masked from 'package:graphics':
## 
##     layout

library(tidyverse)
qplot(data = movies_df,Runtime,fill = Genre,bins = 30)

Question 1: Based on the above data, which movie from which Genre had the longest runtime?

p1 <- ggplot(movies_df,aes(x=Runtime,y=Rating, text=row.names(Title)))+
geom_point(aes(size=Votes,col=Genre))

ggplotly(p1)

Answer: Based on data above, Action and Drama had the longest runtime at 161 minutes. I used 161 minutes to filter to get the Title of the movies.

movies_df %>% 
  filter(Runtime == "161")

##   Rank   Title
## 1   64 Silence
## 2   96  Dangal
##                                                                                                                                                                         Description
## 1 In the 17th century, two Portuguese Jesuit priests travel to Japan in an attempt to locate their mentor, who is rumored to have committed apostasy, and to propagate Catholicism.
## 2                          Former wrestler Mahavir Singh Phogat and his two wrestler daughters struggle towards glory at the Commonwealth Games in the face of societal oppression.
##   Runtime  Genre Rating Metascore  Votes Gross_Earning_in_Mil        Director
## 1     161  Drama    7.2        79 106526                 7.10 Martin Scorsese
## 2     161 Action    8.4        NA 172807                12.39   Nitesh Tiwari
##             Actor
## 1 Andrew Garfield
## 2      Aamir Khan

Therefore, Silence(Drama) and Dangal(Action) had the longest runtime at 161 minutes.

Question 2: Based on the above data, in the Runtime of 130-160 mins, which genre has the highest votes?

runtime130_160 <- movies_df %>% filter(Runtime %in% (130:160) )
#runtime130_160 

options(scipen=999)
library(ggplot2)
library(viridis)

## Loading required package: viridisLite

library(hrbrthemes)

## NOTE: Either Arial Narrow or Roboto Condensed fonts are required to use these themes.

##       Please use hrbrthemes::import_roboto_condensed() to install Roboto Condensed and

##       if Arial Narrow is not on your system, please see https://bit.ly/arialnarrow

Votes130_160 <- runtime130_160 %>%
group_by(Genre) %>%
 dplyr::summarise(Votes = sum(Votes,na.rm=TRUE))
Votes130_160

## # A tibble: 6 x 2
##   Genre       Votes
##   <fct>       <dbl>
## 1 Action    2707894
## 2 Adventure  430794
## 3 Animation   65120
## 4 Biography  703986
## 5 Drama      535493
## 6 Horror     312085

#qplot(data = runtime130_160,Genre,fill = Genre,bins = 30)
p2 <- ggplot(runtime130_160, aes(fill=Genre,x=Genre, y=Votes)) + 
  geom_bar(stat = "identity")

#ggplotly(p2)
p2 + 
    ggtitle("The highest votes in runtime 130 to 160 by Genre")

Answer: Action has the highest votes.

Question 3: Based on the above data, across all genres which genre has the highest average gross earnings in runtime 100 to 120.

runtime100_120 <- movies_df %>% filter(Runtime %in% (100:120))
#runtime100_120

gross_average <- runtime100_120 %>%
  group_by(Genre) %>%
  dplyr::summarize(Mean = mean(Gross_Earning_in_Mil, na.rm=TRUE))
gross_average

## # A tibble: 9 x 2
##   Genre       Mean
##   <fct>      <dbl>
## 1 Action     82.2 
## 2 Adventure 185.  
## 3 Animation 216.  
## 4 Biography  35.9 
## 5 Comedy     23.4 
## 6 Crime      75.4 
## 7 Drama      59.0 
## 8 Horror     46.8 
## 9 Mystery     1.31

p3 <- ggplot(gross_average, aes(fill=Genre,x=Genre, y=Mean)) + 
  geom_bar(stat = "identity")+ 
    ggtitle("Gross Average Earnings in runtime 100 to 120")+
  theme(axis.text.x = element_text(angle = 90, vjust = 0.5, hjust=1))
ggplotly(p3)   

Answer: Animation has the highest average gross earnings.