I'm assuming …

• that you have R installed,
  
• have a basic familiarity with how R works, and
• are familiar with at least one R interface (e.g. RStudio or the R command prompt).

• Introductions

  • Interested/ just getting started
  • Data-proficient
  • Intermediate hackers
  • Experienced R/python coders

• “Something for everyone and everything for no one.”

  • (hopefully not)

• General process (“steps”) for scraping a web page

• Decide when it's appropriate to scrape a web page

  • Tell if a page is scrapable before you start
  • Make a difficulty/benefit estimation of a page's scrapability.

• Know where to look for help

• Base R : uses functions to sort through text
• rvest :relies on pipes and built-in HTML tags
• Google specific searches

• Stack Overflow

• Alternatives

  • Chrome apps
  • other R packages
  • Python packages (beautifulsoup)
  • many more …

Other Presentations on Webscraping in R

• http://cpsievert.github.io/slides/web-scraping/20150612/#8
• https://stat4701.github.io/edav/2015/04/02/rvest_tutorial/

• Why scrape a web page?

  • Necessary concepts for web scraping
  • Most useful functions in rvest.

  WORM HOLE / TIME WARP

• Examples

  • Two examples in rvest
  • Base R example

Probably not the first option, but …

• Sometimes data are not accessible any other way
• You want to update the data in the future
• You need to download a lot of data files

• magrittr
• HTML tags (in theory)
• “Inspect Element”
• sometimes loops

t(cbind(x, x))

  [,1] [,2] [,3] [,4] [,5] [,6]
x    0    4    8   12   16   20
x    0    4    8   12   16   20

x %>% cbind( ., x ) %>% t()

  [,1] [,2] [,3] [,4] [,5] [,6]
.    0    4    8   12   16   20
x    0    4    8   12   16   20

right-click and selecting “view page source”

• <div>
• <a href=...
• <table>

SciStarter website To scrape the projects info from this page, we need to take a look at the HTML tags.

right click -> “Inspect Element” We will make an R data frame of all the projects on this website.

• Open up R or RStudio
• Install rvest -> install.packages('rvest').
• ?html_nodes()
• ?html_table()
• ?html_text()

Go to https://rpubs.com/ryanthomas/YNUS-Webscraping-in-R for the rest of the workshop.