Web scrapping with R

Web scrapping is a very useful tool for collecting text from news websites or other sites of interests. When collecting from the former, however, we must consider the design of the website. Every news site employs a different site design and this design impacts where the text is located.

This outline reviews how to use the rvest package in R in conjunction with the selectorgadget.

Selectorgadget

The selectorgadget is a Chrome add-on that helps you select the CSS element of interest. Think of it like this, if you only want the headlines of news articles, it will help you select this desired component.

Example

  1. Navigate to the following URL: https://www.cnn.com/2019/10/01/us/jimmy-carter-95-birthday-trnd/index.html

  2. Open the selectorgadget add-on.

  3. Click on the element you want to select (it will turn green). SelectorGadget will then generate a minimal CSS selector for that element, and will highlight (yellow) everything that is matched by the selector.

  1. As you can see, there are thing we do not want (e.g., videos). To remove these elements, click on a highlighted element. The elements highlighted in red are removed.

  1. Repeat until only the elements you want are selected.
## install necessary package 
install.packages("rvest")
## load library
library(rvest)
## Loading required package: xml2
article <- read_html("https://www.cnn.com/2019/10/01/us/jimmy-carter-95-birthday-trnd/index.html")
text <- html_nodes(article, "#body-text .l-container")   ## node from the selectorgadget

html_text(text) #Extracts attributes, text and tag name from html
## [1] " (CNN)Jimmy Carter, the son of a peanut farmer who served as America's 39th president, turns 95 today. He is the oldest living former U.S. president, a title once held by the late George H. W. Bush, who died in late 2018 at 94.People may leave President Carter a birthday wish from anywhere in the world, through the Carter Center website. Thousands of messages have been left so far from people as close as his home state of Georgia and as far as South Korea, Zambia, and Australia. The organization was founded in 1982 by President Carter and his wife and is \"committed to advancing human rights and alleviating unnecessary human suffering,\" according to the Center's website.Former US President Jimmy Carter and First Lady Rosalynn Carter arrive for the Presidential Inauguration of Donald Trump at the US Capitol on January 20, 2017 in Washington, DC.  (Photo by Saul Loeb - Pool/Getty Images){\"@context\": \"http://schema.org\",\"@type\": \"ImageObject\",\"name\": \"Former US President Jimmy Carter and First Lady Rosalynn Carter arrive for the Presidential Inauguration of Donald Trump at the US Capitol on January 20, 2017 in Washington, DC.  (Photo by Saul Loeb - Pool/Getty Images)\",\"description\": \"Former US President Jimmy Carter and First Lady Rosalynn Carter arrive for the Presidential Inauguration of Donald Trump at the US Capitol on January 20, 2017 in Washington, DC.  (Photo by Saul Loeb - Pool/Getty Images)\",\"url\": \"//cdn.cnn.com/cnnnext/dam/assets/190322153536-48-jimmy-carter-unfurled-large-169.jpg\"}President Carter has been a record-breaker since the day he was born. Read MoreBorn in Plains, Georgia, on October 1, 1924, President Carter holds an illustrious legacy as a champion of human rights, earning him a Nobel Peace Prize in 2002 for his dedication to international democracy building, conflict resolution, and development programs. He is also the recipient of three Grammys for Best Spoken Word Album. Not too bad for the son of a peanut farmer. The Jimmy Carter Presidential Library and Museum in Atlanta will be holding a day of festivities to celebrate their namesake's 95th year. They are offering $0.95 museum admission all day, extended hours until 8 pm, and a free screening of the Georgia-made film Smokey and the Bandit.President Carter and his family will be spending a quiet day at home on his birthday, according to a spokesperson."

Now we have the text of interest. As you can see, even with the selectorgadget it is a bit messy. We will discuss text cleaning on a later date.