library('rvest')Web Scraping
#Loading the rvest package
#Specifying the url for desired website to be scraped
url <- 'http://www.imdb.com/search/title?count=100&release_date=2016,2016&title_type=feature'#Reading the HTML code from the website
webpage <- read_html(url)Now, we'll be scraping the following data from this website.
Rank: The rank of the film from 1 to 100 on the list of 100 most popular feature films released in 2016.
Title: The title of the feature film.
Description: The description of the feature film.
Runtime: The duration of the feature film.
Genre: The genre of the feature film,
Rating: The IMDb rating of the feature film.
Metascore: The metascore on IMDb website for the feature film.
Votes: Votes cast in favor of the feature film.
Gross_Earning_in_Mil: The gross earnings of the feature film in millions.
Director: The main director of the feature film. Note, in case of multiple directors, I'll take only the first.
Actor: The main actor in the feature film. Note, in case of multiple actors, I'll take only the first.
Step 1: Now, we will start by scraping the Rank field. For that, we'll use the selector gadget to get the specific CSS selectors that encloses the rankings. You can click on the extension in your browser and select the rankings field with the cursor.
Make sure that all the rankings are selected. You can select some more ranking sections in case you are not able to get all of them and you can also de-select them by clicking on the selected section to make sure that you only have those sections highlighted that you want to scrape for that go.
Step 2: Once you are sure that you have made the right selections, you need to copy the corresponding CSS selector that you can view in the bottom center.
Step 3: Once you know the CSS selector that contains the rankings, you can use this simple R code to get all the rankings:
#Using CSS selectors to scrape the rankings section
rank_data_html <- html_nodes(webpage,'.text-primary')#Converting the ranking data to text
rank_data <- html_text(rank_data_html)#Let’s have a look at the rankings
head(rank_data)[1] "1." "2." "3." "4." "5." "6."Step 4: Once you have the data, make sure that it looks in the desired format. I am preprocessing my data to convert it to numerical format.
#Data-Preprocessing: Converting rankings to numerical
rank_data<-as.numeric(rank_data)#Let’s have another look at the rankings
head(rank_data)[1] 1 2 3 4 5 6Step 5: Now you can clear the selector section and select all the titles. You can visually inspect that all the titles are selected. Make any required additions and deletions with the help of your curser. I have done the same here.
Step 6: Again, I have the corresponding CSS selector for the titles – .lister-item-header a. I will use this selector to scrape all the titles using the following code.
#Using CSS selectors to scrape the title section
title_data_html <- html_nodes(webpage,'.lister-item-header a')#Converting the title data to text
title_data <- html_text(title_data_html)#Let’s have a look at the title
head(title_data)[1] "Terrifier" "Suicide Squad" "Silence" "Hush"
[5] "The Conjuring 2" "Split" Step 7: In the following code, I have done the same thing for scraping – Description, Runtime, Genre, Rating, Metascore, Votes, Gross_Earning_in_Mil , Director and Actor data.
#Using CSS selectors to scrape the description section
description_data_html <- html_nodes(webpage,'.ratings-bar+ .text-muted')#Converting the description data to text
description_data <- html_text(description_data_html)#Let’s have a look at the description data
head(description_data)[1] "\nOn Halloween night, Tara Heyes finds herself as the obsession of a sadistic murderer known as Art the Clown."
[2] "\nA secret government agency recruits some of the most dangerous incarcerated super-villains to form a defensive task force. Their first mission: save the world from the apocalypse."
[3] "\nIn the 17th century, two Portuguese Jesuit priests travel to Japan in an attempt to locate their mentor, who is rumored to have committed apostasy, and to propagate Catholicism."
[4] "\nA deaf and mute writer who retreated into the woods to live a solitary life must fight for her life in silence when a masked killer appears at her window."
[5] "\nEd and Lorraine Warren travel to North London to help a single mother raising four children alone in a house plagued by a supernatural spirit."
[6] "\nThree girls are kidnapped by a man with a diagnosed 23 distinct personalities. They must try to escape before the apparent emergence of a frightful new 24th." #Data-Preprocessing: removing ‘\n’
description_data<-gsub("\n","",description_data)#Let’s have another look at the description data
head(description_data)[1] "On Halloween night, Tara Heyes finds herself as the obsession of a sadistic murderer known as Art the Clown."
[2] "A secret government agency recruits some of the most dangerous incarcerated super-villains to form a defensive task force. Their first mission: save the world from the apocalypse."
[3] "In the 17th century, two Portuguese Jesuit priests travel to Japan in an attempt to locate their mentor, who is rumored to have committed apostasy, and to propagate Catholicism."
[4] "A deaf and mute writer who retreated into the woods to live a solitary life must fight for her life in silence when a masked killer appears at her window."
[5] "Ed and Lorraine Warren travel to North London to help a single mother raising four children alone in a house plagued by a supernatural spirit."
[6] "Three girls are kidnapped by a man with a diagnosed 23 distinct personalities. They must try to escape before the apparent emergence of a frightful new 24th." #Using CSS selectors to scrape the Movie runtime section
runtime_data_html <- html_nodes(webpage,'.text-muted .runtime')#Converting the runtime data to text
runtime_data <- html_text(runtime_data_html)#Let’s have a look at the runtime
head(runtime_data)[1] "85 min" "123 min" "161 min" "82 min" "134 min" "117 min"#Data-Preprocessing: removing mins and converting it to numerical
runtime_data<-gsub(" min","",runtime_data)
runtime_data<-as.numeric(runtime_data)#Let’s have another look at the runtime data
head(runtime_data)[1] 85 123 161 82 134 117#Using CSS selectors to scrape the Movie genre section
genre_data_html <- html_nodes(webpage,'.genre')#Converting the genre data to text
genre_data <- html_text(genre_data_html)#Let’s have a look at the runtime
head(genre_data)[1] "\nHorror, Thriller "
[2] "\nAction, Adventure, Fantasy "
[3] "\nDrama, History "
[4] "\nHorror, Thriller "
[5] "\nHorror, Mystery, Thriller "
[6] "\nHorror, Thriller " #Data-Preprocessing: removing \n
genre_data<-gsub("\n","",genre_data)#Data-Preprocessing: removing excess spaces
genre_data<-gsub(" ","",genre_data)#taking only the first genre of each movie
genre_data<-gsub(",.*","",genre_data)#Convering each genre from text to factor
genre_data<-as.factor(genre_data)#Let’s have another look at the genre data
head(genre_data)[1] Horror Action Drama Horror Horror Horror
9 Levels: Action Adventure Animation Biography Comedy Crime Drama ... Horror#Using CSS selectors to scrape the IMDB rating section
rating_data_html <- html_nodes(webpage,'.ratings-imdb-rating strong')#Converting the ratings data to text
rating_data <- html_text(rating_data_html)#Let’s have a look at the ratings
head(rating_data)[1] "5.6" "5.9" "7.2" "6.6" "7.3" "7.3"#Data-Preprocessing: converting ratings to numerical
rating_data<-as.numeric(rating_data)#Let’s have another look at the ratings data
head(rating_data)[1] 5.6 5.9 7.2 6.6 7.3 7.3#Using CSS selectors to scrape the votes section
votes_data_html <- html_nodes(webpage,'.sort-num_votes-visible span:nth-child(2)')#Converting the votes data to text
votes_data <- html_text(votes_data_html)#Let’s have a look at the votes data
head(votes_data)[1] "47,944" "710,389" "119,555" "149,420" "292,588" "533,236"#Data-Preprocessing: removing commas
votes_data<-gsub(",","",votes_data)#Data-Preprocessing: converting votes to numerical
votes_data<-as.numeric(votes_data)#Using CSS selectors to scrape the directors section
directors_data_html <- html_nodes(webpage,'.text-muted+ p a:nth-child(1)')#Converting the directors data to text
directors_data <- html_text(directors_data_html)#Let’s have a look at the directors data
head(directors_data)[1] "Damien Leone" "David Ayer" "Martin Scorsese"
[4] "Mike Flanagan" "James Wan" "M. Night Shyamalan"#Data-Preprocessing: converting directors data into factors
directors_data<-as.factor(directors_data)#Using CSS selectors to scrape the actors section
actors_data_html <- html_nodes(webpage,'.lister-item-content .ghost+ a')#Converting the gross actors data to text
actors_data <- html_text(actors_data_html)#Let’s have a look at the actors data
head(actors_data)[1] "Jenna Kanell" "Will Smith" "Andrew Garfield"
[4] "John Gallagher Jr." "Vera Farmiga" "James McAvoy" #Data-Preprocessing: converting actors data into factors
actors_data<-as.factor(actors_data)#Using CSS selectors to scrape the metascore section
metascore_data_html <- html_nodes(webpage,'.metascore')#Converting the runtime data to text
metascore_data <- html_text(metascore_data_html)#Let’s have a look at the metascore
head(metascore_data)[1] "40 " "79 " "67 " "65 " "63 "
[6] "71 "#Data-Preprocessing: removing extra space in metascore
metascore_data<-gsub(" ","",metascore_data)#Lets check the length of metascore data
length(metascore_data)[1] 95Step 8: The length of the metascore data is 96 while we are scraping the data for 100 movies. The reason this happened is that there are 4 movies that don't have the corresponding Metascore fields.
Step 9: It is a practical situation which can arise while scraping any website. Unfortunately, if we simply add NA's to last 4 entries, it will map NA as Metascore for movies 96 to 100 while in reality, the data is missing for some other movies. After a visual inspection, I found that the Metascore is missing for movies 39, 73, 80 and 89. I have written the following function to get around this problem.
for (i in c(39,73,80,89)){
a<-metascore_data[1:(i-1)]
b<-metascore_data[i:length(metascore_data)]
metascore_data<-append(a,list("NA"))
metascore_data<-append(metascore_data,b)
}#Data-Preprocessing: converting metascore to numerical
metascore_data[is.na(as.numeric(metascore_data))]Warning: NAs introduced by coercion
Warning: NAs introduced by coercion
Warning: NAs introduced by coercion
Warning: NAs introduced by coercion[[1]]
[1] "NA"
[[2]]
[1] "NA"
[[3]]
[1] "NA"
[[4]]
[1] "NA"#Let’s have another look at length of the metascore data
length(metascore_data)[1] 99#Let’s look at summary statistics
summary(metascore_data) Length Class Mode
[1,] 1 -none- character
[2,] 1 -none- character
[3,] 1 -none- character
[4,] 1 -none- character
[5,] 1 -none- character
[6,] 1 -none- character
[7,] 1 -none- character
[8,] 1 -none- character
[9,] 1 -none- character
[10,] 1 -none- character
[11,] 1 -none- character
[12,] 1 -none- character
[13,] 1 -none- character
[14,] 1 -none- character
[15,] 1 -none- character
[16,] 1 -none- character
[17,] 1 -none- character
[18,] 1 -none- character
[19,] 1 -none- character
[20,] 1 -none- character
[21,] 1 -none- character
[22,] 1 -none- character
[23,] 1 -none- character
[24,] 1 -none- character
[25,] 1 -none- character
[26,] 1 -none- character
[27,] 1 -none- character
[28,] 1 -none- character
[29,] 1 -none- character
[30,] 1 -none- character
[31,] 1 -none- character
[32,] 1 -none- character
[33,] 1 -none- character
[34,] 1 -none- character
[35,] 1 -none- character
[36,] 1 -none- character
[37,] 1 -none- character
[38,] 1 -none- character
[39,] 1 -none- character
[40,] 1 -none- character
[41,] 1 -none- character
[42,] 1 -none- character
[43,] 1 -none- character
[44,] 1 -none- character
[45,] 1 -none- character
[46,] 1 -none- character
[47,] 1 -none- character
[48,] 1 -none- character
[49,] 1 -none- character
[50,] 1 -none- character
[51,] 1 -none- character
[52,] 1 -none- character
[53,] 1 -none- character
[54,] 1 -none- character
[55,] 1 -none- character
[56,] 1 -none- character
[57,] 1 -none- character
[58,] 1 -none- character
[59,] 1 -none- character
[60,] 1 -none- character
[61,] 1 -none- character
[62,] 1 -none- character
[63,] 1 -none- character
[64,] 1 -none- character
[65,] 1 -none- character
[66,] 1 -none- character
[67,] 1 -none- character
[68,] 1 -none- character
[69,] 1 -none- character
[70,] 1 -none- character
[71,] 1 -none- character
[72,] 1 -none- character
[73,] 1 -none- character
[74,] 1 -none- character
[75,] 1 -none- character
[76,] 1 -none- character
[77,] 1 -none- character
[78,] 1 -none- character
[79,] 1 -none- character
[80,] 1 -none- character
[81,] 1 -none- character
[82,] 1 -none- character
[83,] 1 -none- character
[84,] 1 -none- character
[85,] 1 -none- character
[86,] 1 -none- character
[87,] 1 -none- character
[88,] 1 -none- character
[89,] 1 -none- character
[90,] 1 -none- character
[91,] 1 -none- character
[92,] 1 -none- character
[93,] 1 -none- character
[94,] 1 -none- character
[95,] 1 -none- character
[96,] 1 -none- character
[97,] 1 -none- character
[98,] 1 -none- character
[99,] 1 -none- characterStep 10: The same thing happens with the Gross variable which represents gross earnings of that movie in millions. I have use the same solution to work my way around:
#Using CSS selectors to scrape the gross revenue section
gross_data_html <- html_nodes(webpage,'.ghost~ .text-muted+ span')#Converting the gross revenue data to text
gross_data <- html_text(gross_data_html)#Let’s have a look at the votes data
head(gross_data)[1] "$325.10M" "$7.10M" "$102.47M" "$138.29M" "$67.21M" "$2.01M" #Filling missing entries with NA
gross_data<-gsub("M","",gross_data)
gross_data<-substring(gross_data,2,6)length(gross_data)[1] 90#Filling missing entries with NA
for (i in c(17,39,49,52,57,64,66,73,76,77,80,87,88,89)){
a<-gross_data[1:(i-1)]
b<-gross_data[i:length(gross_data)]
gross_data<-append(a,list("NA"))
gross_data<-append(gross_data,b)
}#Data-Preprocessing: converting gross to numerical
gross_data<-as.numeric(gross_data)Warning: NAs introduced by coercion
Warning: NAs introduced by coercion
Warning: NAs introduced by coercion
Warning: NAs introduced by coercion
Warning: NAs introduced by coercion
Warning: NAs introduced by coercion
Warning: NAs introduced by coercion
Warning: NAs introduced by coercion
Warning: NAs introduced by coercion
Warning: NAs introduced by coercion
Warning: NAs introduced by coercion
Warning: NAs introduced by coercion
Warning: NAs introduced by coercion
Warning: NAs introduced by coercion#Let’s have another look at the length of gross data
length(gross_data)[1] 104summary(gross_data) Min. 1st Qu. Median Mean 3rd Qu. Max. NA's
0.01 15.50 55.87 93.09 122.05 532.10 14 lengths <- list(rank_data = length(rank_data),
title_data = length(title_data),
description_data = length(description_data),
runtime_data = length(runtime_data),
genre_data = length(genre_data),
rating_data = length(rating_data),
metascore_data = length(metascore_data),
votes_data = length(votes_data),
gross_data = length(gross_data),
directors_data = length(directors_data),
actors_data = length(actors_data))
print(lengths)$rank_data
[1] 100
$title_data
[1] 100
$description_data
[1] 100
$runtime_data
[1] 100
$genre_data
[1] 100
$rating_data
[1] 100
$metascore_data
[1] 99
$votes_data
[1] 100
$gross_data
[1] 104
$directors_data
[1] 100
$actors_data
[1] 99# Correcting the length of metascore_data
if (length(metascore_data) < 100) {
metascore_data <- c(metascore_data, rep(NA, 100 - length(metascore_data)))
}
# Correcting the length of actors_data
if (length(actors_data) < 100) {
actors_data <- c(actors_data, rep(NA, 100 - length(actors_data)))
}
# Correcting the length of gross_data
# Assuming that the extra 4 entries are at the end and should be removed
if (length(gross_data) > 100) {
gross_data <- head(gross_data, 100)
}#Combining all the lists to form a data frame
movies_df<-data.frame(Rank = rank_data, Title = title_data,
Description = description_data, Runtime = runtime_data,
Genre = genre_data, Rating = rating_data,
Metascore = metascore_data, Votes = votes_data, Gross_Earning_in_Mil = gross_data,
Director = directors_data, Actor = actors_data)#Structure of the data frame
str(movies_df)'data.frame': 100 obs. of 110 variables:
$ Rank : num 1 2 3 4 5 6 7 8 9 10 ...
$ Title : chr "Terrifier" "Suicide Squad" "Silence" "Hush" ...
$ Description : chr "On Halloween night, Tara Heyes finds herself as the obsession of a sadistic murderer known as Art the Clown." "A secret government agency recruits some of the most dangerous incarcerated super-villains to form a defensive "| __truncated__ "In the 17th century, two Portuguese Jesuit priests travel to Japan in an attempt to locate their mentor, who is"| __truncated__ "A deaf and mute writer who retreated into the woods to live a solitary life must fight for her life in silence "| __truncated__ ...
$ Runtime : num 85 123 161 82 134 117 139 145 128 108 ...
$ Genre : Factor w/ 9 levels "Action","Adventure",..: 9 1 7 9 9 9 4 7 5 3 ...
$ Rating : num 5.6 5.9 7.2 6.6 7.3 7.3 8.1 8.1 8 7.1 ...
$ Metascore..40. : chr "40" "40" "40" "40" ...
$ Metascore..79. : chr "79" "79" "79" "79" ...
$ Metascore..67. : chr "67" "67" "67" "67" ...
$ Metascore..65. : chr "65" "65" "65" "65" ...
$ Metascore..63. : chr "63" "63" "63" "63" ...
$ Metascore..71. : chr "71" "71" "71" "71" ...
$ Metascore..85. : chr "85" "85" "85" "85" ...
$ Metascore..94. : chr "94" "94" "94" "94" ...
$ Metascore..59. : chr "59" "59" "59" "59" ...
$ Metascore..81. : chr "81" "81" "81" "81" ...
$ Metascore..81..1 : chr "81" "81" "81" "81" ...
$ Metascore..55. : chr "55" "55" "55" "55" ...
$ Metascore..65..1 : chr "65" "65" "65" "65" ...
$ Metascore..73. : chr "73" "73" "73" "73" ...
$ Metascore..70. : chr "70" "70" "70" "70" ...
$ Metascore..65..2 : chr "65" "65" "65" "65" ...
$ Metascore..88. : chr "88" "88" "88" "88" ...
$ Metascore..57. : chr "57" "57" "57" "57" ...
$ Metascore..65..3 : chr "65" "65" "65" "65" ...
$ Metascore..78. : chr "78" "78" "78" "78" ...
$ Metascore..81..2 : chr "81" "81" "81" "81" ...
$ Metascore..54. : chr "54" "54" "54" "54" ...
$ Metascore..67..1 : chr "67" "67" "67" "67" ...
$ Metascore..60. : chr "60" "60" "60" "60" ...
$ Metascore..44. : chr "44" "44" "44" "44" ...
$ Metascore..51. : chr "51" "51" "51" "51" ...
$ Metascore..41. : chr "41" "41" "41" "41" ...
$ Metascore..65..4 : chr "65" "65" "65" "65" ...
$ Metascore..74. : chr "74" "74" "74" "74" ...
$ Metascore..71..1 : chr "71" "71" "71" "71" ...
$ Metascore..81..3 : chr "81" "81" "81" "81" ...
$ Metascore..66. : chr "66" "66" "66" "66" ...
$ Metascore..96. : chr "96" "96" "96" "96" ...
$ Metascore..68. : chr "68" "68" "68" "68" ...
$ Metascore..58. : chr "58" "58" "58" "58" ...
$ Metascore..76. : chr "76" "76" "76" "76" ...
$ Metascore..47. : chr "47" "47" "47" "47" ...
$ Metascore..66..1 : chr "66" "66" "66" "66" ...
$ Metascore..NA. : chr "NA" "NA" "NA" "NA" ...
$ Metascore..82. : chr "82" "82" "82" "82" ...
$ Metascore..48. : chr "48" "48" "48" "48" ...
$ Metascore..82..1 : chr "82" "82" "82" "82" ...
$ Metascore..44..1 : chr "44" "44" "44" "44" ...
$ Metascore..51..1 : chr "51" "51" "51" "51" ...
$ Metascore..75. : chr "75" "75" "75" "75" ...
$ Metascore..42. : chr "42" "42" "42" "42" ...
$ Metascore..32. : chr "32" "32" "32" "32" ...
$ Metascore..25. : chr "25" "25" "25" "25" ...
$ Metascore..66..2 : chr "66" "66" "66" "66" ...
$ Metascore..52. : chr "52" "52" "52" "52" ...
$ Metascore..51..2 : chr "51" "51" "51" "51" ...
$ Metascore..99. : chr "99" "99" "99" "99" ...
$ Metascore..72. : chr "72" "72" "72" "72" ...
$ Metascore..58..1 : chr "58" "58" "58" "58" ...
$ Metascore..77. : chr "77" "77" "77" "77" ...
$ Metascore..57..1 : chr "57" "57" "57" "57" ...
$ Metascore..81..4 : chr "81" "81" "81" "81" ...
$ Metascore..37. : chr "37" "37" "37" "37" ...
$ Metascore..48..1 : chr "48" "48" "48" "48" ...
$ Metascore..44..2 : chr "44" "44" "44" "44" ...
$ Metascore..72..1 : chr "72" "72" "72" "72" ...
$ Metascore..32..1 : chr "32" "32" "32" "32" ...
$ Metascore..45. : chr "45" "45" "45" "45" ...
$ Metascore..44..3 : chr "44" "44" "44" "44" ...
$ Metascore..47..1 : chr "47" "47" "47" "47" ...
$ Metascore..66..3 : chr "66" "66" "66" "66" ...
$ Metascore..46. : chr "46" "46" "46" "46" ...
$ Metascore..53. : chr "53" "53" "53" "53" ...
$ Metascore..81..5 : chr "81" "81" "81" "81" ...
$ Metascore..69. : chr "69" "69" "69" "69" ...
$ Metascore..49. : chr "49" "49" "49" "49" ...
$ Metascore..77..1 : chr "77" "77" "77" "77" ...
$ Metascore..NA..1 : chr "NA" "NA" "NA" "NA" ...
$ Metascore..62. : chr "62" "62" "62" "62" ...
$ Metascore..58..2 : chr "58" "58" "58" "58" ...
$ Metascore..35. : chr "35" "35" "35" "35" ...
$ Metascore..33. : chr "33" "33" "33" "33" ...
$ Metascore..68..1 : chr "68" "68" "68" "68" ...
$ Metascore..78..1 : chr "78" "78" "78" "78" ...
$ Metascore..NA..2 : chr "NA" "NA" "NA" "NA" ...
$ Metascore..42..1 : chr "42" "42" "42" "42" ...
$ Metascore..42..2 : chr "42" "42" "42" "42" ...
$ Metascore..36. : chr "36" "36" "36" "36" ...
$ Metascore..46..1 : chr "46" "46" "46" "46" ...
$ Metascore..34. : chr "34" "34" "34" "34" ...
$ Metascore..61. : chr "61" "61" "61" "61" ...
$ Metascore..60..1 : chr "60" "60" "60" "60" ...
$ Metascore..60..2 : chr "60" "60" "60" "60" ...
$ Metascore..NA..3 : chr "NA" "NA" "NA" "NA" ...
$ Metascore..21. : chr "21" "21" "21" "21" ...
$ Metascore..38. : chr "38" "38" "38" "38" ...
$ Metascore..66..4 : chr "66" "66" "66" "66" ...
$ Metascore..26. : chr "26" "26" "26" "26" ...
[list output truncated]Analyzing scraped data from the web
library('ggplot2')
qplot(data = movies_df,Runtime,fill = Genre,bins = 30)Warning: `qplot()` was deprecated in ggplot2 3.4.0.
Question 1: Based on the above data, which movie from which Genre had the longest runtime?
Based on the above data, a movie from the “Drama” genre has the longest runtime.
ggplot(movies_df,aes(x=Runtime,y=Rating))+
geom_point(aes(size=Votes,col=Genre))
Question 2: Based on the above data, in the Runtime of 130-160 mins, which genre has the highest votes?
Based on the above data, in the Runtime of 130-160 mins, the genre with the largest dot appears to be “Action”(represented by the red color.)
ggplot(movies_df,aes(x=Runtime,y=Gross_Earning_in_Mil))+
geom_point(aes(size=Rating,col=Genre))Warning: Removed 14 rows containing missing values (`geom_point()`).
Question 3: Based on the above data, across all genres which genre has the highest average gross earnings in runtime 100 to 120.
Based on the above data, it appears that the “Action” genre has the highest average gross earnings in the runtime range of 100-120 minutes, followed closely by the “Adventure” genre.