#Web Scraping Assignment

Tutorial from https://www.analyticsvidhya.com/blog/2017/03/beginners-guide-on-web-scraping-in-r-using-rvest-with-hands-on-knowledge/

library(tidyverse)
## ── Attaching packages ───────────────────────────────────────────────────────────────────────────────────────── tidyverse 1.3.0 ──
## ✓ ggplot2 3.3.2     ✓ purrr   0.3.4
## ✓ tibble  3.0.3     ✓ dplyr   1.0.2
## ✓ tidyr   1.1.2     ✓ stringr 1.4.0
## ✓ readr   1.3.1     ✓ forcats 0.5.0
## ── Conflicts ──────────────────────────────────────────────────────────────────────────────────────────── tidyverse_conflicts() ──
## x dplyr::filter() masks stats::filter()
## x dplyr::lag()    masks stats::lag()
#Loading the rvest package
library('rvest')
## Loading required package: xml2
## 
## Attaching package: 'rvest'
## The following object is masked from 'package:purrr':
## 
##     pluck
## The following object is masked from 'package:readr':
## 
##     guess_encoding
#Specifying the url for desired website to be scraped
url <- 'http://www.imdb.com/search/title?count=100&release_date=2016,2016&title_type=feature'

#Reading the HTML code from the website
webpage <- read_html(url)

Now, we’ll be scraping the following data from this website.

Step 1: Now, we will start by scraping the Rank field. For that, we’ll use the selector gadget to get the specific CSS selectors that encloses the rankings. You can click on the extension in your browser and select the rankings field with the cursor.

Step 2: Once you are sure that you have made the right selections, you need to copy the corresponding CSS selector that you can view in the bottom center.

Step 3: Once you know the CSS selector that contains the rankings, you can use this simple R code to get all the rankings:

#Using CSS selectors to scrape the rankings section
rank_data_html <- html_nodes(webpage,'.text-primary')

#Converting the ranking data to text
rank_data <- html_text(rank_data_html)

#Let's have a look at the rankings
head(rank_data)
## [1] "1." "2." "3." "4." "5." "6."

Step 4: Once you have the data, make sure that it looks in the desired format. I am preprocessing my data to convert it to numerical format.

#Data-Preprocessing: Converting rankings to numerical
rank_data<-as.numeric(rank_data)

#Let's have another look at the rankings
head(rank_data)
## [1] 1 2 3 4 5 6

Step 5: Now you can clear the selector section and select all the titles. You can visually inspect that all the titles are selected. Make any required additions and deletions with the help of your cursor. I have done the same here.

Step 6: Again, I have the corresponding CSS selector for the titles – .lister-item-header a. I will use this selector to scrape all the titles using the following code.

#Using CSS selectors to scrape the title section
title_data_html <- html_nodes(webpage,'.lister-item-header a')

#Converting the title data to text
title_data <- html_text(title_data_html)

#Let's have a look at the title
head(title_data)
## [1] "Suicide Squad"           "Deadpool"               
## [3] "In a Valley of Violence" "Brimstone"              
## [5] "Train to Busan"          "Hush"

Step 7: In the following code, I have done the same thing for scraping – Description, Runtime, Genre, Rating, Metascore, Votes, Gross_Earning_in_Mil , Director and Actor data.

#Using CSS selectors to scrape the description section
description_data_html <- html_nodes(webpage,'.ratings-bar+ .text-muted')

#Converting the description data to text
description_data <- html_text(description_data_html)

#Let's have a look at the description data
head(description_data)
## [1] "\n    A secret government agency recruits some of the most dangerous incarcerated super-villains to form a defensive task force. Their first mission: save the world from the apocalypse."
## [2] "\n    A wisecracking mercenary gets experimented on and becomes immortal but ugly, and sets out to track down the man who ruined his looks."                                              
## [3] "\n    A mysterious stranger and a random act of violence drag a town of misfits and nitwits into the bloody crosshairs of revenge."                                                       
## [4] "\n    From the moment the new reverend climbs the pulpit, Liz knows she and her family are in great danger."                                                                              
## [5] "\n    While a zombie virus breaks out in South Korea, passengers struggle to survive on the train from Seoul to Busan."                                                                   
## [6] "\n    A deaf and mute writer who retreated into the woods to live a solitary life must fight for her life in silence when a masked killer appears at her window."
#Data-Preprocessing: removing '\n'
description_data<-gsub("\n","",description_data)

#Let's have another look at the description data 
head(description_data)
## [1] "    A secret government agency recruits some of the most dangerous incarcerated super-villains to form a defensive task force. Their first mission: save the world from the apocalypse."
## [2] "    A wisecracking mercenary gets experimented on and becomes immortal but ugly, and sets out to track down the man who ruined his looks."                                              
## [3] "    A mysterious stranger and a random act of violence drag a town of misfits and nitwits into the bloody crosshairs of revenge."                                                       
## [4] "    From the moment the new reverend climbs the pulpit, Liz knows she and her family are in great danger."                                                                              
## [5] "    While a zombie virus breaks out in South Korea, passengers struggle to survive on the train from Seoul to Busan."                                                                   
## [6] "    A deaf and mute writer who retreated into the woods to live a solitary life must fight for her life in silence when a masked killer appears at her window."
#Using CSS selectors to scrape the Movie runtime section
runtime_data_html <- html_nodes(webpage,'.text-muted .runtime')

#Converting the runtime data to text
runtime_data <- html_text(runtime_data_html)

#Let's have a look at the runtime
head(runtime_data)
## [1] "123 min" "108 min" "104 min" "148 min" "118 min" "82 min"
#Data-Preprocessing: removing mins and converting it to numerical

runtime_data<-gsub(" min","",runtime_data)
runtime_data<-as.numeric(runtime_data)

#Let's have another look at the runtime data
head(runtime_data)
## [1] 123 108 104 148 118  82
#Using CSS selectors to scrape the Movie genre section
genre_data_html <- html_nodes(webpage,'.genre')

#Converting the genre data to text
genre_data <- html_text(genre_data_html)

#Let's have a look at the runtime
head(genre_data)
## [1] "\nAction, Adventure, Fantasy            "
## [2] "\nAction, Adventure, Comedy            " 
## [3] "\nAction, Western            "           
## [4] "\nDrama, Mystery, Thriller            "  
## [5] "\nAction, Horror, Thriller            "  
## [6] "\nHorror, Thriller            "
#Data-Preprocessing: removing \n
genre_data<-gsub("\n","",genre_data)

#Data-Preprocessing: removing excess spaces
genre_data<-gsub(" ","",genre_data)

#taking only the first genre of each movie
genre_data<-gsub(",.*","",genre_data)

#Convering each genre from text to factor
genre_data<-as.factor(genre_data)

#Let's have another look at the genre data
head(genre_data)
## [1] Action Action Action Drama  Action Horror
## Levels: Action Adventure Animation Biography Comedy Crime Drama Horror
#Using CSS selectors to scrape the IMDB rating section
rating_data_html <- html_nodes(webpage,'.ratings-imdb-rating strong')

#Converting the ratings data to text
rating_data <- html_text(rating_data_html)

#Let's have a look at the ratings
head(rating_data)
## [1] "6.0" "8.0" "6.0" "7.1" "7.6" "6.6"
#Data-Preprocessing: converting ratings to numerical
rating_data<-as.numeric(rating_data)

#Let's have another look at the ratings data
head(rating_data)
## [1] 6.0 8.0 6.0 7.1 7.6 6.6
#Using CSS selectors to scrape the votes section
votes_data_html <- html_nodes(webpage,'.sort-num_votes-visible span:nth-child(2)')

#Converting the votes data to text
votes_data <- html_text(votes_data_html)

#Let's have a look at the votes data
head(votes_data)
## [1] "591,159" "888,290" "15,514"  "35,619"  "157,442" "99,724"
#Data-Preprocessing: removing commas
votes_data<-gsub(",","",votes_data)

#Data-Preprocessing: converting votes to numerical
votes_data<-as.numeric(votes_data)

#Let's have another look at the votes data
head(votes_data)
## [1] 591159 888290  15514  35619 157442  99724
#Using CSS selectors to scrape the directors section
directors_data_html <- html_nodes(webpage,'.text-muted+ p a:nth-child(1)')

#Converting the directors data to text
directors_data <- html_text(directors_data_html)

#Let's have a look at the directors data
head(directors_data)
## [1] "David Ayer"       "Tim Miller"       "Ti West"          "Martin Koolhoven"
## [5] "Sang-ho Yeon"     "Mike Flanagan"
#Data-Preprocessing: converting directors data into factors
directors_data<-as.factor(directors_data)

#Using CSS selectors to scrape the actors section
actors_data_html <- html_nodes(webpage,'.lister-item-content .ghost+ a')

#Converting the gross actors data to text
actors_data <- html_text(actors_data_html)

#Let's have a look at the actors data
head(actors_data)
## [1] "Will Smith"         "Ryan Reynolds"      "Ethan Hawke"       
## [4] "Guy Pearce"         "Yoo Gong"           "John Gallagher Jr."
#Data-Preprocessing: converting actors data into factors
actors_data<-as.factor(actors_data)
#Using CSS selectors to scrape the metascore section
metascore_data_html <- html_nodes(webpage,'.metascore')

#Converting the runtime data to text
metascore_data <- html_text(metascore_data_html)

#Let's have a look at the metascore data 
head(metascore_data)
## [1] "40        " "65        " "64        " "45        " "72        "
## [6] "67        "
#Data-Preprocessing: removing extra space in metascore
metascore_data<-gsub(" ","",metascore_data)

#Lets check the length of metascore data
length(metascore_data)
## [1] 96

Step 8: The length of the metascore data is 96 while we are scraping the data for 100 movies. The reason this happened is that there are 4 movies that don’t have the corresponding Metascore fields.

Step 9: It is a practical situation which can arise while scraping any website. Unfortunately, if we simply add NA’s to last 4 entries, it will map NA as Metascore for movies 96 to 100 while in reality, the data is missing for some other movies. After a visual inspection, I found that the Metascore is missing for movies 39, 73, 80 and 89. I have written the following function to get around this problem.

for (i in c(21, 35,81,92)){

a<-metascore_data[1:(i-1)]

b<-metascore_data[i:length(metascore_data)]

metascore_data<-append(a,list("NA"))

metascore_data<-append(metascore_data,b)

}

#Data-Preprocessing: converting metascore to numerical
metascore_data<-as.numeric(metascore_data)

#Let's have another look at length of the metascore data

length(metascore_data)
## [1] 100
#Let's look at summary statistics
summary(metascore_data)
##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max.    NA's 
##   21.00   45.75   59.50   58.83   72.00   99.00       4

Step 10: The same thing happens with the Gross variable which represents gross earnings of that movie in millions. I have use the same solution to work my way around:

#Using CSS selectors to scrape the gross revenue section
gross_data_html <- html_nodes(webpage,'.ghost~ .text-muted+ span')

#Converting the gross revenue data to text
gross_data <- html_text(gross_data_html)

#Let's have a look at the votes data
head(gross_data)
## [1] "$325.10M" "$363.07M" "$0.05M"   "$2.13M"   "$138.29M" "$93.43M"
#Data-Preprocessing: removing '$' and 'M' signs
gross_data<-gsub("M","",gross_data)

gross_data<-substring(gross_data,2,6)

#Let's check the length of gross data
length(gross_data)
## [1] 90
#Filling missing entries with NA
for (i in c(4, 6, 29, 35, 44, 77,80, 85, 90, 92)){

a<-gross_data[1:(i-1)]

b<-gross_data[i:length(gross_data)]

gross_data<-append(a,list("NA"))

gross_data<-append(gross_data,b)

}

#Data-Preprocessing: converting gross to numerical
gross_data<-as.numeric(gross_data)

#Let's have another look at the length of gross data
length(gross_data)
## [1] 100
summary(gross_data)
##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max.    NA's 
##    0.02   12.49   52.30   87.28  101.92  532.10      10

Step 11: Now we have successfully scraped all the 11 features for the 100 most popular feature films released in 2016. Let’s combine them to create a dataframe and inspect its structure.

#Combining all the lists to form a data frame
movies_df<-data.frame(Rank = rank_data, Title = title_data,

Description = description_data, Runtime = runtime_data,

Genre = genre_data, Rating = rating_data,

Metascore = metascore_data, Votes = votes_data,                                     

Gross_Earning_in_Mil = gross_data,

Director = directors_data, Actor = actors_data)

#Structure of the data frame

str(movies_df)
## 'data.frame':    100 obs. of  11 variables:
##  $ Rank                : num  1 2 3 4 5 6 7 8 9 10 ...
##  $ Title               : chr  "Suicide Squad" "Deadpool" "In a Valley of Violence" "Brimstone" ...
##  $ Description         : chr  "    A secret government agency recruits some of the most dangerous incarcerated super-villains to form a defens"| __truncated__ "    A wisecracking mercenary gets experimented on and becomes immortal but ugly, and sets out to track down the"| __truncated__ "    A mysterious stranger and a random act of violence drag a town of misfits and nitwits into the bloody cross"| __truncated__ "    From the moment the new reverend climbs the pulpit, Liz knows she and her family are in great danger." ...
##  $ Runtime             : num  123 108 104 148 118 82 117 132 127 139 ...
##  $ Genre               : Factor w/ 8 levels "Action","Adventure",..: 1 1 1 7 1 8 8 1 1 4 ...
##  $ Rating              : num  6 8 6 7.1 7.6 6.6 7.3 6.9 5.4 8.1 ...
##  $ Metascore           : num  40 65 64 45 72 67 62 54 25 71 ...
##  $ Votes               : num  591159 888290 15514 35619 157442 ...
##  $ Gross_Earning_in_Mil: num  325.1 363 0.05 NA 2.13 ...
##  $ Director            : Factor w/ 96 levels "Alex Proyas",..: 21 91 89 55 79 60 53 8 1 58 ...
##  $ Actor               : Factor w/ 89 levels "Aamir Khan","Adam Sandler",..: 88 71 30 35 89 41 37 20 12 5 ...

6. Analyzing scraped data from the web

Once you have the data, you can perform several tasks like analyzing the data, drawing inferences from it, training machine learning models over this data, etc. I have gone on to create some interesting visualization out of the data we have just scraped. Follow the visualizations and answer the questions given below. Post your answers in the comment section below.

library('ggplot2')
library(plotly)
## 
## Attaching package: 'plotly'
## The following object is masked from 'package:ggplot2':
## 
##     last_plot
## The following object is masked from 'package:stats':
## 
##     filter
## The following object is masked from 'package:graphics':
## 
##     layout
## adding plotly to get the details about what's in each bar

p1 <- qplot(data = movies_df,Runtime,fill = Genre,bins = 30) 

p1 <- movies_df %>%
  ggplot(aes(x=Runtime, fill = Genre)) +
  geom_histogram(position="identity", alpha=0.5, binwidth = 5, color = "white")+
  scale_fill_discrete(name = "Genre") +
  labs(title = "Top 100 Movies of 2016 Runtime by Genre")  
ggplotly(p1)

Question 1: Based on the above data, which movie from which Genre had the longest runtime?

Based purely on the plot, the genre with the longest runtime was drama. To figure out the specific movie, I need to do additional coding:

movies_df %>%
  rownames_to_column(var = "Name") %>% 
  filter(Runtime == max(Runtime))
##   Name Rank          Title
## 1   54   54 American Honey
##                                                                                                                                                                                                              Description
## 1     A teenage girl with nothing to lose joins a traveling magazine sales crew, and gets caught up in a whirlwind of hard partying, law bending and young love as she criss-crosses the Midwest with a band of misfits.
##   Runtime Genre Rating Metascore Votes Gross_Earning_in_Mil      Director
## 1     163 Drama      7        80 37481                 0.66 Andrea Arnold
##        Actor
## 1 Sasha Lane

The film with the longest runtime was “American Honey.”

p2 <- movies_df %>%
  ggplot(aes(x=Runtime,y=Rating))+
  geom_point(aes(size=Votes,col=Genre, text = paste("Movie Title:", title_data)), alpha = 0.7) +
  labs(title = "Top 100 Movies of 2016 Runtime by Ratings")  
## Warning: Ignoring unknown aesthetics: text
ggplotly(p2)

Question 2: Based on the above data, in the Runtime of 130-160 mins, which genre has the highest votes?

The genre with the highest votes is action.

Confirming this with more code:

movies_df %>%
  rownames_to_column(var = "Name") %>% 
  filter(Runtime == c(130,160)) %>%
  filter(Votes == max(Votes))
##   Name Rank                     Title
## 1   85   85 A Silent Voice: The Movie
##                                                                                                                                                          Description
## 1     A young man is ostracized by his classmates after he bullies a deaf girl to the point where she moves away. Years later, he sets off on a path for redemption.
##   Runtime     Genre Rating Metascore Votes Gross_Earning_in_Mil     Director
## 1     130 Animation    8.1        78 43097                   NA Naoko Yamada
##        Actor
## 1 Miyu Irino

This is very strange. The code comes up with a film that has only 43092 votes, but the plot shows that Captain America, with 650,403 votes has the most. I need to play with the code to see what’s wrong.

movies_df %>%
  rownames_to_column(var = "Name") %>% 
  filter(Runtime >= 130 & Runtime <= 160) %>%
  filter(Votes == max(Votes))
##   Name Rank                      Title
## 1   32   32 Captain America: Civil War
##                                                                                              Description
## 1     Political involvement in the Avengers' affairs causes a rift between Captain America and Iron Man.
##   Runtime  Genre Rating Metascore  Votes Gross_Earning_in_Mil      Director
## 1     147 Action    7.8        75 650408                  408 Anthony Russo
##         Actor
## 1 Chris Evans

Aha! It seems the problem was the code was searching just for films with a runtime of 130 minutes or 160 minutes, not within the range of 130 to 160 minutes. Once corrected, it too came up with “Captain America” of the action genre as the highest vote-getter.

p3 <- movies_df %>%
  ggplot(aes(x=Runtime,y=Gross_Earning_in_Mil))+
  geom_point(aes(size = Rating,col = Genre), alpha = 0.5) +
  labs(title = "Top 100 Movies of 2016 Runtime by Gross Earnings in Millions") +
  scale_y_continuous("Gross Earnings in Millions", limits =c(-10, 600))
ggplotly(p3)

Question 3: Based on the above data, across all genres which genre has the highest average gross earnings in runtime 100 to 120.

Based upon the plot, it’s hard to tell which has the highest average gross earnings in runtime 100 to 120. I would guess animation or action, based upon the number of bubbles in that range.

I can confirm this with code (which I will adjust to be between 100 and 120 inclusive:

movies_df %>%
  rownames_to_column(var = "Name") %>% 
  filter(Runtime >= 100 & Runtime <= 120) %>%
  group_by(Genre) %>%
  summarize(averageGross = mean(Gross_Earning_in_Mil)) %>%
  filter(averageGross == max(averageGross))
## `summarise()` ungrouping output (override with `.groups` argument)
## # A tibble: 0 x 2
## # … with 2 variables: Genre <fct>, averageGross <dbl>

For some reason, this doesn’t print anything out. Removing the last filter, I do get a dataframe with the average gross that I can look at, and see that animation has the highest average gross. (Action must have some NA’s interfering with its numbers.)

movies_df %>%
  rownames_to_column(var = "Name") %>% 
  filter(Runtime >= 100 & Runtime <= 120) %>%
  group_by(Genre) %>%
  summarize(averageGross = mean(Gross_Earning_in_Mil)) 
## `summarise()` ungrouping output (override with `.groups` argument)
## # A tibble: 8 x 2
##   Genre     averageGross
##   <fct>            <dbl>
## 1 Action            NA  
## 2 Adventure        149. 
## 3 Animation        216. 
## 4 Biography         35.9
## 5 Comedy            13.9
## 6 Crime             NA  
## 7 Drama             42.7
## 8 Horror            46.8
#remove NA's when taking the mean

movies_df %>%
  rownames_to_column(var = "Name") %>% 
  filter(Runtime >= 100 & Runtime <= 120) %>%
  group_by(Genre) %>%
  summarize(averageGross = mean(Gross_Earning_in_Mil, na.rm = TRUE)) 
## `summarise()` ungrouping output (override with `.groups` argument)
## # A tibble: 8 x 2
##   Genre     averageGross
##   <fct>            <dbl>
## 1 Action            83.4
## 2 Adventure        149. 
## 3 Animation        216. 
## 4 Biography         35.9
## 5 Comedy            13.9
## 6 Crime             41.3
## 7 Drama             42.7
## 8 Horror            46.8

Removing the NA’s adds action back into the categories, and, for some mysterious reason, allows me to use the last filter again!

# Add the last filter which picks out the highest average gross

movies_df %>%
  rownames_to_column(var = "Name") %>% 
  filter(Runtime >= 100 & Runtime <= 120) %>%
  group_by(Genre) %>%
  summarize(averageGross = mean(Gross_Earning_in_Mil, na.rm = TRUE)) %>%
  filter(averageGross == max(averageGross))
## `summarise()` ungrouping output (override with `.groups` argument)
## # A tibble: 1 x 2
##   Genre     averageGross
##   <fct>            <dbl>
## 1 Animation         216.

Animation is the correct answer!