Web Scraping with rvest - Rakha Hafish Setiawan

Activate libraries

library(rvest)
library(magrittr)
library(dplyr)

Read website URL

Before initiating website scraping, user must first read the desired URL with read_html function from rvest, in this example “Quotes to Scrape” page was used.

TheURL = "https://quotes.toscrape.com/"
TheURL = read_html(TheURL)

Begin Scraping

When scraping, the html nodes must be obtained from the website, this can be done with either manually by using inspect and going through the html structure in order to select the css, or with a Google Chrome Extension SelectorGadget.

In this example, as seen from the image above, SelectorGadget was used to obtain the quotes and the author of each quotes. After done scraping, user must convert each variable into a data frame.

Text = TheURL %>% html_nodes(".text") %>% html_text() %>% data.frame()

Author = TheURL %>% html_nodes(".author") %>% html_text() %>% data.frame()

Data frame binding

After the finishing the web scraping, combine all the obtained data into a single data frame with dplyr. Note that all the columns have to have matching row amount and renaming of the columns is imperative so that it would not create confusion.

Text %>% 
  bind_cols(Author) %>% 
  rename(Quotes = ....1) %>%
  rename(Author = ....2) %>%
  tibble()

# A tibble: 10 × 2
   Quotes                                                                 Author
   <chr>                                                                  <chr> 
 1 “The world as we have created it is a process of our thinking. It can… Alber…
 2 “It is our choices, Harry, that show what we truly are, far more than… J.K. …
 3 “There are only two ways to live your life. One is as though nothing … Alber…
 4 “The person, be it gentleman or lady, who has not pleasure in a good … Jane …
 5 “Imperfection is beauty, madness is genius and it's better to be abso… Maril…
 6 “Try not to become a man of success. Rather become a man of value.”    Alber…
 7 “It is better to be hated for what you are than to be loved for what … André…
 8 “I have not failed. I've just found 10,000 ways that won't work.”      Thoma…
 9 “A woman is like a tea bag; you never know how strong it is until it'… Elean…
10 “A day without sunshine is like, you know, night.”                     Steve…