Loading the rvest package

#install.packages('rvest')
library(rvest)

## Warning: package 'rvest' was built under R version 4.1.3

library(stringr)

Specifying the url for desired website to be scraped

url <- 'http://www.imdb.com/search/title?count=100&release_date=2016,2016&title_type=feature'

Reading the HTML code from the website

webpage <- read_html(url)

Using CSS selectors to scrape the rankings section

rank_data_html <- html_nodes(webpage,'.text-primary')

Converting the ranking data to text

rank_data <- html_text(rank_data_html)

Let’s have a look at the rankings

head(rank_data)

## [1] "1." "2." "3." "4." "5." "6."

Data-Preprocessing: Converting rankings to numerical

rank_data<-as.numeric(rank_data)

Let’s have another look at the rankings

head(rank_data)

## [1] 1 2 3 4 5 6

Using CSS selectors to scrape the title section

title_data_html <- html_nodes(webpage,'.lister-item-header a')

Converting the title data to text

title_data <- html_text(title_data_html)

Let’s have a look at the title

head(title_data)

## [1] "Batman v Superman: Dawn of Justice Ultimate Edition"
## [2] "Fantastic Beasts and Where to Find Them"            
## [3] "Suicide Squad"                                      
## [4] "Deadpool"                                           
## [5] "Batman v Superman: Dawn of Justice"                 
## [6] "Hacksaw Ridge"

Using CSS selectors to scrape the description section

description_data_html <- html_nodes(webpage,'.ratings-bar+ .text-muted')

Converting the description data to text

description_data <- html_text(description_data_html)

Let’s have a look at the description data

head(description_data)

## [1] "\nBatman is manipulated by Lex Luthor to fear Superman. Superman´s existence is meanwhile dividing the world and he is framed for murder during an international crisis. The heroes clash and force the neutral Wonder Woman to reemerge."
## [2] "\nThe adventures of writer Newt Scamander in New York's secret community of witches and wizards seventy years before Harry Potter reads his book in school."                                                                              
## [3] "\nA secret government agency recruits some of the most dangerous incarcerated super-villains to form a defensive task force. Their first mission: save the world from the apocalypse."                                                    
## [4] "\nA wisecracking mercenary gets experimented on and becomes immortal but ugly, and sets out to track down the man who ruined his looks."                                                                                                  
## [5] "\nFearing that the actions of Superman are left unchecked, Batman takes on the Man of Steel, while the world wrestles with what kind of a hero it really needs."                                                                          
## [6] "\nWorld War II American Army Medic Desmond T. Doss, who served during the Battle of Okinawa, refuses to kill people and becomes the first man in American history to receive the Medal of Honor without firing a shot."

Data-Preprocessing: removing ‘’

description_data<-gsub("\n","",description_data)

Let’s have another look at the description data

head(description_data)

## [1] "Batman is manipulated by Lex Luthor to fear Superman. Superman´s existence is meanwhile dividing the world and he is framed for murder during an international crisis. The heroes clash and force the neutral Wonder Woman to reemerge."
## [2] "The adventures of writer Newt Scamander in New York's secret community of witches and wizards seventy years before Harry Potter reads his book in school."                                                                              
## [3] "A secret government agency recruits some of the most dangerous incarcerated super-villains to form a defensive task force. Their first mission: save the world from the apocalypse."                                                    
## [4] "A wisecracking mercenary gets experimented on and becomes immortal but ugly, and sets out to track down the man who ruined his looks."                                                                                                  
## [5] "Fearing that the actions of Superman are left unchecked, Batman takes on the Man of Steel, while the world wrestles with what kind of a hero it really needs."                                                                          
## [6] "World War II American Army Medic Desmond T. Doss, who served during the Battle of Okinawa, refuses to kill people and becomes the first man in American history to receive the Medal of Honor without firing a shot."

Using CSS selectors to scrape the Movie runtime section

runtime_data_html <- html_nodes(webpage,'.text-muted .runtime')

Converting the runtime data to text

runtime_data <- html_text(runtime_data_html)

Let’s have a look at the runtime

head(runtime_data)

## [1] "182 min" "132 min" "123 min" "108 min" "152 min" "139 min"

Data-Preprocessing: removing mins and converting it to numerical

runtime_data<-gsub(" min","",runtime_data)
runtime_data<-as.numeric(runtime_data)

Let’s have another look at the runtime data

head(runtime_data)

## [1] 182 132 123 108 152 139

Using CSS selectors to scrape the Movie genre section

genre_data_html <- html_nodes(webpage,'.genre')

Converting the genre data to text

genre_data <- html_text(genre_data_html)

Let’s have a look at the runtime

head(genre_data)

## [1] "\nAction, Adventure, Sci-Fi            " 
## [2] "\nAdventure, Family, Fantasy            "
## [3] "\nAction, Adventure, Fantasy            "
## [4] "\nAction, Adventure, Comedy            " 
## [5] "\nAction, Adventure, Sci-Fi            " 
## [6] "\nBiography, Drama, History            "

Data-Preprocessing: removing

genre_data<-gsub("\n","",genre_data)

Data-Preprocessing: removing excess spaces

genre_data<-gsub(" ","",genre_data)

taking only the first genre of each movie

genre_data<-gsub(",.*","",genre_data)

Convering each genre from text to factor

genre_data<-as.factor(genre_data)

Let’s have another look at the genre data

head(genre_data)

## [1] Action    Adventure Action    Action    Action    Biography
## Levels: Action Adventure Animation Biography Comedy Crime Drama Horror

Using CSS selectors to scrape the IMDB rating section

rating_data_html <- html_nodes(webpage,'.ratings-imdb-rating strong')

Converting the ratings data to text

rating_data <- html_text(rating_data_html)

Let’s have a look at the ratings

head(rating_data)

## [1] "8.4" "7.3" "5.9" "8.0" "6.5" "8.1"

Data-Preprocessing: converting ratings to numerical

rating_data<-as.numeric(rating_data)

Let’s have another look at the ratings data

head(rating_data)

## [1] 8.4 7.3 5.9 8.0 6.5 8.1

Using CSS selectors to scrape the votes section

votes_data_html <- html_nodes(webpage,'.sort-num_votes-visible span:nth-child(2)')

Converting the votes data to text

votes_data <- html_text(votes_data_html)

Let’s have a look at the votes data

head(votes_data)

## [1] "10,375"  "447,783" "669,445" "989,423" "683,393" "497,182"

Data-Preprocessing: removing commas

votes_data<-gsub(",","",votes_data)

Data-Preprocessing: converting votes to numerical

votes_data<-as.numeric(votes_data)

Let’s have another look at the votes data

head(votes_data)

## [1]  10375 447783 669445 989423 683393 497182

Using CSS selectors to scrape the directors section

directors_data_html <- html_nodes(webpage,'.text-muted+ p a:nth-child(1)')

Converting the directors data to text

directors_data <- html_text(directors_data_html)

Let’s have a look at the directors data

head(directors_data)

## [1] "David Yates"     "David Ayer"      "Tim Miller"      "Zack Snyder"    
## [5] "Mel Gibson"      "Damien Chazelle"

Data-Preprocessing: converting directors data into factors

directors_data<-as.factor(directors_data)

Using CSS selectors to scrape the actors section

actors_data_html <- html_nodes(webpage,'.lister-item-content .ghost+ a')

Converting the gross actors data to text

actors_data <- html_text(actors_data_html)

Let’s have a look at the actors data

head(actors_data)

## [1] "Eddie Redmayne"  "Will Smith"      "Ryan Reynolds"   "Ben Affleck"    
## [5] "Andrew Garfield" "Ryan Gosling"

Data-Preprocessing: converting actors data into factors

actors_data<-as.factor(actors_data)

Find metascore data with missing values and replace with NAs (this is an automated method)

ratings_bar_data <- html_nodes(webpage,'.ratings-bar') %>%
  html_text2()

head(ratings_bar_data)

## [1] "8.4\nRate this\n 1 2 3 4 5 6 7 8 9 10 8.4/10 X "              
## [2] "7.3\nRate this\n 1 2 3 4 5 6 7 8 9 10 7.3/10 X \n66 Metascore"
## [3] "5.9\nRate this\n 1 2 3 4 5 6 7 8 9 10 5.9/10 X \n40 Metascore"
## [4] "8.0\nRate this\n 1 2 3 4 5 6 7 8 9 10 8/10 X \n65 Metascore"  
## [5] "6.5\nRate this\n 1 2 3 4 5 6 7 8 9 10 6.5/10 X \n44 Metascore"
## [6] "8.1\nRate this\n 1 2 3 4 5 6 7 8 9 10 8.1/10 X \n71 Metascore"

metascore_data <- str_match(ratings_bar_data, "\\d{2} Metascore") %>%
  str_match("\\d{2}")%>%
  as.numeric()

length(metascore_data)

## [1] 100

metascore_data

##   [1] NA 66 40 65 44 71 94 52 72 81 59 62 78 84 99 81 54 65 64 67 96 48 75 74 88
##  [26] 41 44 78 70 57 79 71 28 25 51 73 72 51 66 81 60 32 66 57 81 47 72 64 68 NA
##  [51] 52 79 77 34 42 51 45 32 69 62 77 58 48 79 47 45 23 76 65 49 NA 74 42 60 76
##  [76] 39 51 36 79 42 18 35 78 67 63 36 77 68 46 55 59 59 NA 60 65 40 61 58 81 26

summary(metascore_data)

##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max.    NA's 
##   18.00   47.00   62.00   60.15   74.00   99.00       4

Find the missing gross earnings (automated) Earnings are part of the votes bar in the html, scrape the votes bar and extract earnings with a regular expression to get the NAs in context.

# scrape the votess bar and convert to text
votes_bar_data <- html_nodes(webpage,'.sort-num_votes-visible') %>% 
 html_text2()
head(votes_bar_data) # look at the votes bar data

## [1] "Votes: 10,375"                    "Votes: 447,783 | Gross: $234.04M"
## [3] "Votes: 669,445 | Gross: $325.10M" "Votes: 989,423 | Gross: $363.07M"
## [5] "Votes: 683,393 | Gross: $330.36M" "Votes: 497,182 | Gross: $67.21M"

gross_data <- str_match(votes_bar_data, "\\$.+$") # extract the gross earnings
gross_data <- gsub("M","",gross_data) # clean data: remove 'M' sign 
gross_data <- substring(gross_data,2,6) %>% # clean data: remove '$' sign 
 as.numeric()
length(gross_data)

## [1] 100

Combining all the lists to form a data frame

movies_df<-data.frame(Rank = rank_data, Title = title_data, Description = description_data, Runtime = runtime_data, Genre = genre_data, Rating = rating_data, Metascore = metascore_data, Votes = votes_data, Gross_Earning_in_Mil = gross_data)

Structure of the data frame

str(movies_df)

## 'data.frame':    100 obs. of  9 variables:
##  $ Rank                : num  1 2 3 4 5 6 7 8 9 10 ...
##  $ Title               : chr  "Batman v Superman: Dawn of Justice Ultimate Edition" "Fantastic Beasts and Where to Find Them" "Suicide Squad" "Deadpool" ...
##  $ Description         : chr  "Batman is manipulated by Lex Luthor to fear Superman. Superman´s existence is meanwhile dividing the world and "| __truncated__ "The adventures of writer Newt Scamander in New York's secret community of witches and wizards seventy years bef"| __truncated__ "A secret government agency recruits some of the most dangerous incarcerated super-villains to form a defensive "| __truncated__ "A wisecracking mercenary gets experimented on and becomes immortal but ugly, and sets out to track down the man"| __truncated__ ...
##  $ Runtime             : num  182 132 123 108 152 139 128 144 115 107 ...
##  $ Genre               : Factor w/ 8 levels "Action","Adventure",..: 1 2 1 1 1 4 5 1 1 3 ...
##  $ Rating              : num  8.4 7.3 5.9 8 6.5 8.1 8 6.9 7.5 7.6 ...
##  $ Metascore           : num  NA 66 40 65 44 71 94 52 72 81 ...
##  $ Votes               : num  10375 447783 669445 989423 683393 ...
##  $ Gross_Earning_in_Mil: num  NA 234 325 363 330 ...

Analyzing scraped data from the web

Tutorial’s visualization for Question 1

library(ggplot2)
qplot(data = movies_df,Runtime,fill = Genre,bins = 30)

Question 1: Based on the above data, which movie from which Genre had the longest runtime?

Based on the tutorial’s visualization the longest runtime were > 150. Using dplyr.

library(dplyr)

## 
## Attaching package: 'dplyr'

## The following objects are masked from 'package:stats':
## 
##     filter, lag

## The following objects are masked from 'package:base':
## 
##     intersect, setdiff, setequal, union

movies_df %>%
  select(Title, Runtime, Genre) %>%
  filter(Runtime > 150)

##                                                 Title Runtime  Genre
## 1 Batman v Superman: Dawn of Justice Ultimate Edition     182 Action
## 2                  Batman v Superman: Dawn of Justice     152 Action
## 3                                             Silence     161  Drama
## 4                                         The Wailing     156  Drama

Batman v Superman: Dawn of Justice Ultimate Edition, an action movie had the longest runtime.

Making a new and better visualization for Question 1

Changing it to more than 120 so we can see more than just 4 movies

q1 <- movies_df %>%
  select(Title, Runtime, Genre) %>%
  filter(Runtime > 120)

qplot(data = q1,Runtime, Title, color = Genre)

Answer 1: Batman v Superman: Dawn of Justice Ultimate Edition, an action movie had the longest runtime.

Question 2: Based on the above data, in the Runtime of 130-160 mins, which genre has the highest votes?

Tutorial’s visualization for Question 2

ggplot(movies_df,aes(x=Runtime,y=Rating))+
geom_point(aes(size=Votes,col=Genre))

Creating a new visualization for Question 2

I noticed the tutorial’s plot uses ratings also not just the votes. I will be using ggplotly.

library(plotly)

## Warning: package 'plotly' was built under R version 4.1.3

## 
## Attaching package: 'plotly'

## The following object is masked from 'package:ggplot2':
## 
##     last_plot

## The following object is masked from 'package:stats':
## 
##     filter

## The following object is masked from 'package:graphics':
## 
##     layout

q2 <- movies_df %>%
   select(Genre, Rating,Votes, Runtime)%>%
   filter(Runtime >= 130 & Runtime <= 160)%>%
  ggplot(aes(Runtime, Rating, color = Genre)) +
  geom_point(aes(size = Votes, col = Genre))+
  labs(x = "Runtime", y = "Rating", size = "Genre sized by Votes")

q2 <- ggplotly(q2)
q2

Answer 2: Animation, Biography, and Drama have the highest ratings, however, Action has the highest votes.

Question 3: Based on the above data, across all genres which genre has the highest average gross earnings in runtime 100 to 120.

Tutorial’s visualization for Question 3

ggplot(movies_df,aes(x=Runtime,y=Gross_Earning_in_Mil))+
geom_point(aes(size=Rating,col=Genre))

## Warning: Removed 11 rows containing missing values (geom_point).

Creating a new visualization for Question 3. Filtering Runtime from 100 to 120. Using ggplotly.

q3 <- movies_df %>%
   select(Genre, Rating,Gross_Earning_in_Mil, Runtime)%>%
   filter(Runtime >= 100 & Runtime <= 120)%>%
  ggplot(aes(Runtime, Gross_Earning_in_Mil, color = Genre)) +
  geom_point(aes(size = Rating, col = Genre))+
  labs(x = "Runtime", y = "Gross Earning in Mil", size = "Genre sized by Rating")

q3 <- ggplotly(q3)
q3

Webscraping Assignment

Ivana Loffa

4/3/2022

Loading the rvest package

Specifying the url for desired website to be scraped

Reading the HTML code from the website

Using CSS selectors to scrape the rankings section

Converting the ranking data to text

Let’s have a look at the rankings

Data-Preprocessing: Converting rankings to numerical

Let’s have another look at the rankings

Using CSS selectors to scrape the title section

Converting the title data to text

Let’s have a look at the title

Using CSS selectors to scrape the description section

Converting the description data to text

Let’s have a look at the description data

Data-Preprocessing: removing ‘’

Let’s have another look at the description data

Using CSS selectors to scrape the Movie runtime section

Converting the runtime data to text

Let’s have a look at the runtime

Data-Preprocessing: removing mins and converting it to numerical

Let’s have another look at the runtime data

Using CSS selectors to scrape the Movie genre section

Converting the genre data to text

Let’s have a look at the runtime

Data-Preprocessing: removing

Data-Preprocessing: removing excess spaces

taking only the first genre of each movie

Convering each genre from text to factor

Let’s have another look at the genre data

Using CSS selectors to scrape the IMDB rating section

Converting the ratings data to text

Let’s have a look at the ratings

Data-Preprocessing: converting ratings to numerical

Let’s have another look at the ratings data

Using CSS selectors to scrape the votes section

Converting the votes data to text

Let’s have a look at the votes data

Data-Preprocessing: removing commas

Data-Preprocessing: converting votes to numerical

Let’s have another look at the votes data

Using CSS selectors to scrape the directors section

Converting the directors data to text

Let’s have a look at the directors data

Data-Preprocessing: converting directors data into factors

Using CSS selectors to scrape the actors section

Converting the gross actors data to text

Let’s have a look at the actors data

Data-Preprocessing: converting actors data into factors

Find metascore data with missing values and replace with NAs (this is an automated method)

Find the missing gross earnings (automated) Earnings are part of the votes bar in the html, scrape the votes bar and extract earnings with a regular expression to get the NAs in context.

Combining all the lists to form a data frame

Structure of the data frame

Analyzing scraped data from the web

Tutorial’s visualization for Question 1

Question 1: Based on the above data, which movie from which Genre had the longest runtime?

Based on the tutorial’s visualization the longest runtime were > 150. Using dplyr.

Batman v Superman: Dawn of Justice Ultimate Edition, an action movie had the longest runtime.

Making a new and better visualization for Question 1

Changing it to more than 120 so we can see more than just 4 movies

Answer 1: Batman v Superman: Dawn of Justice Ultimate Edition, an action movie had the longest runtime.

Question 2: Based on the above data, in the Runtime of 130-160 mins, which genre has the highest votes?

Tutorial’s visualization for Question 2

Creating a new visualization for Question 2

I noticed the tutorial’s plot uses ratings also not just the votes. I will be using ggplotly.

Answer 2: Animation, Biography, and Drama have the highest ratings, however, Action has the highest votes.

Question 3: Based on the above data, across all genres which genre has the highest average gross earnings in runtime 100 to 120.

Tutorial’s visualization for Question 3

Creating a new visualization for Question 3. Filtering Runtime from 100 to 120. Using ggplotly.

Answer 3: Adventure has the highest average gross earnings in runtime 100 to 120.