library(fansi)
library(ggplot2)
library(tidyverse)

## -- Attaching packages --------------------------------------- tidyverse 1.3.1 --

## v tibble  3.1.2     v dplyr   1.0.6
## v tidyr   1.1.3     v stringr 1.4.0
## v readr   1.4.0     v forcats 0.5.1
## v purrr   0.3.4

## -- Conflicts ------------------------------------------ tidyverse_conflicts() --
## x dplyr::filter() masks stats::filter()
## x dplyr::lag()    masks stats::lag()

Now, let’s get started with scraping the IMDb website for the 100 most popular feature films released in 2016.

#Loading the rvest package
library('rvest')

## 
## Attaching package: 'rvest'

## The following object is masked from 'package:readr':
## 
##     guess_encoding

#Specifying the url for desired website to be scraped
url <- 'http://www.imdb.com/search/title?count=100&release_date=2016,2016&title_type=feature'

#Reading the HTML code from the website
webpage <- read_html(url)

Now, we’ll be scraping the following data from this website.

Rank: The rank of the film from 1 to 100 on the list of 100 most popular feature films released in 2016. Title: The title of the feature film. Description: The description of the feature film. Runtime: The duration of the feature film. Genre: The genre of the feature film, Rating: The IMDb rating of the feature film. Metascore: The metascore on IMDb website for the feature film. Votes: Votes cast in favor of the feature film. Gross_Earning_in_Mil: The gross earnings of the feature film in millions. Director: The main director of the feature film. Note, in case of multiple directors, I’ll take only the first. Actor: The main actor in the feature film. Note, in case of multiple actors, I’ll take only the first.

Make sure that all the rankings are selected. You can select some more ranking sections in case you are not able to get all of them and you can also de-select them by clicking on the selected section to make sure that you only have those sections highlighted that you want to scrape for that go.

Step 2: Once you are sure that you have made the right selections, you need to copy the corresponding CSS selector that you can view in the bottom center.

Step 3: Once you know the CSS selector that contains the rankings, you can use this simple R code to get all the rankings:

#Using CSS selectors to scrape the rankings section
rank_data_html <- html_nodes(webpage,'.text-primary')

#Converting the ranking data to text
rank_data <- html_text(rank_data_html)

#Let's have a look at the rankings
head(rank_data)

## [1] "1." "2." "3." "4." "5." "6."

Step 4: Once you have the data, make sure that it looks in the desired format. I am preprocessing my data to convert it to numerical format.

#Data-Preprocessing: Converting rankings to numerical
rank_data<-as.numeric(rank_data)

#Let's have another look at the rankings
head(rank_data)

## [1] 1 2 3 4 5 6

Step 5: Now you can clear the selector section and select all the titles. You can visually inspect that all the titles are selected. Make any required additions and deletions with the help of your curser. I have done the same here.

Step 6: Again, I have the corresponding CSS selector for the titles – .lister-item-header a. I will use this selector to scrape all the titles using the following code.

#Using CSS selectors to scrape the title section
title_data_html <- html_nodes(webpage,'.lister-item-header a')

#Converting the title data to text
title_data <- html_text(title_data_html)

#Let's have a look at the title
head(title_data)

## [1] "Suicide Squad"     "The Conjuring 2"   "Captain Fantastic"
## [4] "Sing"              "Deadpool"          "Hidden Figures"

Step 7: In the following code, I have done the same thing for scraping – Description, Runtime, Genre, Rating, Metascore, Votes, Gross_Earning_in_Mil , Director and Actor data.

#Using CSS selectors to scrape the description section
description_data_html <- html_nodes(webpage,'.ratings-bar+ .text-muted')

#Converting the description data to text
description_data <- html_text(description_data_html)

#Let's have a look at the description data
head(description_data)

## [1] "\nA secret government agency recruits some of the most dangerous incarcerated super-villains to form a defensive task force. Their first mission: save the world from the apocalypse."                                                             
## [2] "\nEd and Lorraine Warren travel to North London to help a single mother raising four children alone in a house plagued by a supernatural spirit."                                                                                                  
## [3] "\nIn the forests of the Pacific Northwest, a father devoted to raising his six kids with a rigorous physical and intellectual education is forced to leave his paradise and enter the world, challenging his idea of what it means to be a parent."
## [4] "\nIn a city of humanoid animals, a hustling theater impresario's attempt to save his theater with a singing competition becomes grander than he anticipates even as its finalists find that their lives will never be the same."                   
## [5] "\nA wisecracking mercenary gets experimented on and becomes immortal but ugly, and sets out to track down the man who ruined his looks."                                                                                                           
## [6] "\nThe story of a team of female African-American mathematicians who served a vital role in NASA during the early years of the U.S. space program."

Data processing to remove n

#Data-Preprocessing: removing '\n'
description_data<-gsub("\n","",description_data)

#Let's have another look at the description data 
head(description_data)

## [1] "A secret government agency recruits some of the most dangerous incarcerated super-villains to form a defensive task force. Their first mission: save the world from the apocalypse."                                                             
## [2] "Ed and Lorraine Warren travel to North London to help a single mother raising four children alone in a house plagued by a supernatural spirit."                                                                                                  
## [3] "In the forests of the Pacific Northwest, a father devoted to raising his six kids with a rigorous physical and intellectual education is forced to leave his paradise and enter the world, challenging his idea of what it means to be a parent."
## [4] "In a city of humanoid animals, a hustling theater impresario's attempt to save his theater with a singing competition becomes grander than he anticipates even as its finalists find that their lives will never be the same."                   
## [5] "A wisecracking mercenary gets experimented on and becomes immortal but ugly, and sets out to track down the man who ruined his looks."                                                                                                           
## [6] "The story of a team of female African-American mathematicians who served a vital role in NASA during the early years of the U.S. space program."

Select the movies run time

#Using CSS selectors to scrape the Movie runtime section
runtime_data_html <- html_nodes(webpage,'.text-muted .runtime')

#Converting the runtime data to text
runtime_data <- html_text(runtime_data_html)

#Let's have a look at the runtime
head(runtime_data)

## [1] "123 min" "134 min" "118 min" "108 min" "108 min" "127 min"

Data processing converting time to numeric

#Data-Preprocessing: removing mins and converting it to numerical

runtime_data<-gsub(" min","",runtime_data)
runtime_data<-as.numeric(runtime_data)

#Let's have another look at the runtime data
head(runtime_data)

## [1] 123 134 118 108 108 127

Scrapping the genre

#Using CSS selectors to scrape the Movie genre section
genre_data_html <- html_nodes(webpage,'.genre')

#Converting the genre data to text
genre_data <- html_text(genre_data_html)

#Let's have a look at the genre
head(genre_data)

## [1] "\nAction, Adventure, Fantasy            "
## [2] "\nHorror, Mystery, Thriller            " 
## [3] "\nComedy, Drama            "             
## [4] "\nAnimation, Comedy, Family            " 
## [5] "\nAction, Adventure, Comedy            " 
## [6] "\nBiography, Drama, History            "

Data processing, various cleaning functions on genre

#Data-Preprocessing: removing \n
genre_data<-gsub("\n","",genre_data)

#Data-Preprocessing: removing excess spaces
genre_data<-gsub(" ","",genre_data)

#taking only the first genre of each movie
genre_data<-gsub(",.*","",genre_data)

#Convering each genre from text to factor
genre_data<-as.factor(genre_data)

#Let's have another look at the genre data
head(genre_data)

## [1] Action    Horror    Comedy    Animation Action    Biography
## Levels: Action Adventure Animation Biography Comedy Crime Drama Horror

Scrapping the IMDB rating sections

#Using CSS selectors to scrape the IMDB rating section
rating_data_html <- html_nodes(webpage,'.ratings-imdb-rating strong')

#Converting the ratings data to text
rating_data <- html_text(rating_data_html)

#Let's have a look at the ratings
head(rating_data)

## [1] "5.9" "7.3" "7.9" "7.1" "8.0" "7.8"

Data processing

#Data-Preprocessing: converting ratings to numerical
rating_data<-as.numeric(rating_data)

#Let's have another look at the ratings data
head(rating_data)

## [1] 5.9 7.3 7.9 7.1 8.0 7.8

Scrapping the votes

#Using CSS selectors to scrape the votes section
votes_data_html <- html_nodes(webpage,'.sort-num_votes-visible span:nth-child(2)')

#Converting the votes data to text
votes_data <- html_text(votes_data_html)

#Let's have a look at the votes data
head(votes_data)

## [1] "622,774" "239,696" "199,901" "138,650" "928,613" "208,148"

Data processing

#Data-Preprocessing: removing commas
votes_data<-gsub(",","",votes_data)

#Data-Preprocessing: converting votes to numerical
votes_data<-as.numeric(votes_data)

#Let's have another look at the votes data
head(votes_data)

## [1] 622774 239696 199901 138650 928613 208148

Scrapping the Directors’ section

#Using CSS selectors to scrape the directors section
directors_data_html <- html_nodes(webpage,'.text-muted+ p a:nth-child(1)')

#Converting the directors data to text
directors_data <- html_text(directors_data_html)

#Let's have a look at the directors data
head(directors_data)

## [1] "David Ayer"     "James Wan"      "Matt Ross"      "Garth Jennings"
## [5] "Tim Miller"     "Theodore Melfi"

Data processing

#Data-Preprocessing: converting directors data into factors
directors_data<-as.factor(directors_data)
#Let's have a look at the directors data
head(directors_data)

## [1] David Ayer     James Wan      Matt Ross      Garth Jennings Tim Miller    
## [6] Theodore Melfi
## 99 Levels: Alex Proyas Ana Lily Amirpour André Øvredal ... Zack Snyder

Scrapping the actors section

#Using CSS selectors to scrape the actors section
actors_data_html <- html_nodes(webpage,'.lister-item-content .ghost+ a')

#Converting the gross actors data to text
actors_data <- html_text(actors_data_html)

#Let's have a look at the actors data
head(actors_data)

## [1] "Will Smith"          "Vera Farmiga"        "Viggo Mortensen"    
## [4] "Matthew McConaughey" "Ryan Reynolds"       "Taraji P. Henson"

Data processing

#Data-Preprocessing: converting actors data into factors
actors_data<-as.factor(actors_data)
#Let's have a look at the actors data
head(actors_data)

## [1] Will Smith          Vera Farmiga        Viggo Mortensen    
## [4] Matthew McConaughey Ryan Reynolds       Taraji P. Henson   
## 90 Levels: Aamir Khan Alexander Skarsgård Amy Adams ... Zoey Deutch

But, I want you to closely follow what happens when I do the same thing for Metascore data.

#Using CSS selectors to scrape the metascore section
metascore_data_html <- html_nodes(webpage,'.metascore')

#Converting the runtime data to text
metascore_data <- html_text(metascore_data_html)

#Let's have a look at the metascore 
head(metascore_data)

## [1] "40        " "65        " "72        " "59        " "65        "
## [6] "74        "

Data processing

#Data-Preprocessing: removing extra space in metascore
metascore_data<-gsub(" ","",metascore_data)

#Lets check the length of metascore data
length(metascore_data)

## [1] 96

Step 8: The length of the metascore data is 96 while we are scraping the data for 100 movies. The reason this happened is that there are 4 movies that don’t have the corresponding Metascore fields.

The Invisible Guest (34), Dangal (64), Love Machine (82), and Good Kids (88).

# Step 9: It is a practical situation which can arise while scraping any website. Unfortunately, if we simply add NA’s to last 4 entries, it will map NA as Metascore for movies 96 to 100 while in reality, the data is missing for some other movies. After a visual inspection, I found that the Metascore is missing for movies 34, 64, 82 and 88. I have written the following function to get around this problem.

for (i in c(34,64,82,88)){

a<-metascore_data[1:(i-1)]

b<-metascore_data[i:length(metascore_data)]

metascore_data<-append(a,list("NA"))

metascore_data<-append(metascore_data,b)

}

Data processing

#Data-Preprocessing: converting metascore to numerical
metascore_data<-as.numeric(metascore_data)

## Warning: NAs introduced by coercion

## Warning: NAs introduced by coercion

## Warning: NAs introduced by coercion

## Warning: NAs introduced by coercion

#Let's have another look at length of the metascore data

length(metascore_data)

## [1] 100

Summary statistics of metascores

#Let's look at summary statistics
summary(metascore_data)

##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max.    NA's 
##   23.00   46.75   59.50   59.15   72.00   99.00       4

Step 10: The same thing happens with the Gross variable which represents gross earnings of that movie in millions. I have use the same solution to work my way around:

Scrape for Gross

#Using CSS selectors to scrape the gross revenue section
gross_data_html <- html_nodes(webpage,'.ghost~ .text-muted+ span')

#Converting the gross revenue data to text
gross_data <- html_text(gross_data_html)

#Let's have a look at the votes data
head(gross_data)

## [1] "$325.10M" "$102.47M" "$5.88M"   "$270.40M" "$363.07M" "$169.61M"

Data processing

#Data-Preprocessing: removing '$' and 'M' signs
gross_data<-gsub("M","",gross_data)

gross_data<-substring(gross_data,2,6)

#Let's check the length of gross data
length(gross_data)

## [1] 89

Missing entries for Gross: 34,50,55,60,62,63,82,87,88,89,96

#Filling missing entries with NA
for (i in c(34,50,55,60,62,63,82,87,88,89,96)){

a<-gross_data[1:(i-1)]

b<-gross_data[i:length(gross_data)]

gross_data<-append(a,list("NA"))

gross_data<-append(gross_data,b)

}

Data processing

#Data-Preprocessing: converting gross to numerical
gross_data<-as.numeric(gross_data)

## Warning: NAs introduced by coercion

## Warning: NAs introduced by coercion

## Warning: NAs introduced by coercion

## Warning: NAs introduced by coercion

## Warning: NAs introduced by coercion

## Warning: NAs introduced by coercion

## Warning: NAs introduced by coercion

## Warning: NAs introduced by coercion

## Warning: NAs introduced by coercion

## Warning: NAs introduced by coercion

## Warning: NAs introduced by coercion

#Let's have another look at the length of gross data
length(gross_data)

## [1] 100

Summarize Gross

summary(gross_data)

##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max.    NA's 
##    0.18   26.86   58.70   96.47  125.00  532.10      11

Step 11: Now we have successfully scraped all the 11 features for the 100 most popular feature films released in 2016. Let’s combine them to create a dataframe and inspect its structure.

#Combining all the lists to form a data frame
movies_df<-data.frame(Rank = rank_data, Title = title_data,

Description = description_data, Runtime = runtime_data,

Genre = genre_data, Rating = rating_data,

Metascore = metascore_data, Votes = votes_data,                                                             Gross_Earning_in_Mil = gross_data,

Director = directors_data, Actor = actors_data)

#Structure of the data frame

str(movies_df)

## 'data.frame':    100 obs. of  11 variables:
##  $ Rank                : num  1 2 3 4 5 6 7 8 9 10 ...
##  $ Title               : chr  "Suicide Squad" "The Conjuring 2" "Captain Fantastic" "Sing" ...
##  $ Description         : chr  "A secret government agency recruits some of the most dangerous incarcerated super-villains to form a defensive "| __truncated__ "Ed and Lorraine Warren travel to North London to help a single mother raising four children alone in a house pl"| __truncated__ "In the forests of the Pacific Northwest, a father devoted to raising his six kids with a rigorous physical and "| __truncated__ "In a city of humanoid animals, a hustling theater impresario's attempt to save his theater with a singing compe"| __truncated__ ...
##  $ Runtime             : num  123 134 118 108 108 127 107 117 132 115 ...
##  $ Genre               : Factor w/ 8 levels "Action","Adventure",..: 1 8 5 3 1 4 3 8 1 1 ...
##  $ Rating              : num  5.9 7.3 7.9 7.1 8 7.8 7.6 7.3 6.9 7.5 ...
##  $ Metascore           : num  40 65 72 59 65 74 81 62 54 72 ...
##  $ Votes               : num  622774 239696 199901 138650 928613 ...
##  $ Gross_Earning_in_Mil: num  325.1 102.4 5.88 270.4 363 ...
##  $ Director            : Factor w/ 99 levels "Alex Proyas",..: 23 42 59 35 95 93 83 56 8 87 ...
##  $ Actor               : Factor w/ 90 levels "Aamir Khan","Alexander Skarsgård",..: 88 86 87 59 73 81 6 39 22 8 ...

Create some interesting visualization out of the data we have just scraped. Follow the visualizations and answer the questions given below. Post your answers in the comment section below.

Create a histogram plot of Runtime colored by genre

qplot(data = movies_df,Runtime,fill = Genre,bins = 30)

Question 1: Based on the above data, which movie from which Genre had the longest runtime

Answer: The data shows 3 movies with the longest run time of about 162.5 minutes. One of these movies was from the Drama genre (purple), the other was from the Adventure genre (brown) and the other was from the Action genre (red).

Plot Runtime by Rating colored by Genre and spot size by Votes,

ggplot(movies_df,aes(x=Runtime,y=Rating))+
geom_point(aes(size=Votes,col=Genre))

Question 2: Based on the above data, in the Runtime of 130-160 mins, which genre has the highest votes?

Answer: Within this range of Runtime, the dots colored red cover the largest area. Hence, the genre with the largest votes in this time range is Action.

Plot Runtime by Gross earnings using Genre as color and Rating as dot size.

ggplot(movies_df,aes(x=Runtime,y=Gross_Earning_in_Mil))+
geom_point(aes(size=Rating,col=Genre))

## Warning: Removed 11 rows containing missing values (geom_point).

Question 3: Based on the above data, across all genres which genre has the highest average gross earnings in runtime 100 to 120.

Answer: In this time range there are two dots with the highest gross earnigs and their earnings are equal, about $380 million each. One is brown from the Adventure genre and the other is red from the Action genre.

Webscrapping Exercise

Samuel Kalibala

7/1/2021

Now, let’s get started with scraping the IMDb website for the 100 most popular feature films released in 2016.

Now, we’ll be scraping the following data from this website.

Make sure that all the rankings are selected. You can select some more ranking sections in case you are not able to get all of them and you can also de-select them by clicking on the selected section to make sure that you only have those sections highlighted that you want to scrape for that go.

Step 2: Once you are sure that you have made the right selections, you need to copy the corresponding CSS selector that you can view in the bottom center.

Step 3: Once you know the CSS selector that contains the rankings, you can use this simple R code to get all the rankings:

Step 4: Once you have the data, make sure that it looks in the desired format. I am preprocessing my data to convert it to numerical format.

Step 5: Now you can clear the selector section and select all the titles. You can visually inspect that all the titles are selected. Make any required additions and deletions with the help of your curser. I have done the same here.

Step 6: Again, I have the corresponding CSS selector for the titles – .lister-item-header a. I will use this selector to scrape all the titles using the following code.

Step 7: In the following code, I have done the same thing for scraping – Description, Runtime, Genre, Rating, Metascore, Votes, Gross_Earning_in_Mil , Director and Actor data.

Data processing to remove n

Select the movies run time

Data processing converting time to numeric

Scrapping the genre

Data processing, various cleaning functions on genre

Scrapping the IMDB rating sections

Data processing

Scrapping the votes

Data processing

Scrapping the Directors’ section

Data processing

Scrapping the actors section

Data processing

But, I want you to closely follow what happens when I do the same thing for Metascore data.

Data processing

Step 8: The length of the metascore data is 96 while we are scraping the data for 100 movies. The reason this happened is that there are 4 movies that don’t have the corresponding Metascore fields.

Data processing

Summary statistics of metascores

Step 10: The same thing happens with the Gross variable which represents gross earnings of that movie in millions. I have use the same solution to work my way around:

Scrape for Gross

Data processing

Missing entries for Gross: 34,50,55,60,62,63,82,87,88,89,96

Data processing

Summarize Gross

Step 11: Now we have successfully scraped all the 11 features for the 100 most popular feature films released in 2016. Let’s combine them to create a dataframe and inspect its structure.

Create some interesting visualization out of the data we have just scraped. Follow the visualizations and answer the questions given below. Post your answers in the comment section below.

Create a histogram plot of Runtime colored by genre

Question 1: Based on the above data, which movie from which Genre had the longest runtime

Answer: The data shows 3 movies with the longest run time of about 162.5 minutes. One of these movies was from the Drama genre (purple), the other was from the Adventure genre (brown) and the other was from the Action genre (red).

Plot Runtime by Rating colored by Genre and spot size by Votes,

Question 2: Based on the above data, in the Runtime of 130-160 mins, which genre has the highest votes?

Answer: Within this range of Runtime, the dots colored red cover the largest area. Hence, the genre with the largest votes in this time range is Action.

Plot Runtime by Gross earnings using Genre as color and Rating as dot size.

Question 3: Based on the above data, across all genres which genre has the highest average gross earnings in runtime 100 to 120.

Answer: In this time range there are two dots with the highest gross earnigs and their earnings are equal, about $380 million each. One is brown from the Adventure genre and the other is red from the Action genre.