Lecture 3 - Webscraping

We will be looking at different ways to get data off the web. There are 2 ways

• Person who made content made it easy
• Person who made content didn't think about it/people dont want you to take it!

read.csv("example.csv")

     publicid          origintime longitude latitude  depth
1 2014p172810 2014-03-05T22:33:37    175.93   -36.91 368.75
2 2014p172742 2014-03-05T21:57:26    175.89   -40.60  22.46
3 2014p172698 2014-03-05T21:33:31    175.86   -40.63  27.44

We have a variation of paste, but rather than pasting in 1 value we are actually pasting in each of the elements within country.

countries <- c("NZ1", "AUS", "US1")
urls <- sprintf("https://www.stat.auckland.ac.nz/~paul/stats769/2017/AC34/20130901103549-NAV-%s.csv", countries)
urls

[1] "https://www.stat.auckland.ac.nz/~paul/stats769/2017/AC34/20130901103549-NAV-NZ1.csv"
[2] "https://www.stat.auckland.ac.nz/~paul/stats769/2017/AC34/20130901103549-NAV-AUS.csv"
[3] "https://www.stat.auckland.ac.nz/~paul/stats769/2017/AC34/20130901103549-NAV-US1.csv"

races <- lapply(urls, read.csv)

Accessing a resource on the web can be more more difficult.

• We may need to provide a username and password (authentication)
• We may want to send more than just a URL to the web server (e.g., fill in a web form rather than just get a static URL). For example, the statsnz website requires you to point, click, type etc. to get the data you want.
• We may want to use a “protocol” that isn't HTTP.
• The resource may not be a simple flat text file (e.g. JSON, XML, HTML)

We will investigate R packages that do this!

• When you communicate with a webserver you send a request and recieve a response.
• A request generally contains a URL (Unique Resource Location) for the resource we want, but it may also contain additional parameters, such as user/pw.
• The request could also have a body (e.g. to include information for an HTML form)
• The response will have headers and a body!
• The body could by HTML, a csv file etc
• The header contains information on the format of the response.

The httr package allows you to request a URL and supply a username and password.

library(httr)
GET("https://www.stat.auckland.ac.nz/~paul/")

Response [https://www.stat.auckland.ac.nz/~paul/]
  Date: 2017-11-10 09:50
  Status: 200
  Content-Type: text/html; charset=UTF-8
  Size: 27.3 kB
<!doctype html public "-//w3c//dtd html 4.0 transitional//en">
<html>
<head>
   <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
   <title>Paul Murrell's Home Page</title>

<STYLE TYPE="text/css">
  BODY { padding-left: 100; padding-right: 100 }
</STYLE>

...

The content type informs us the format of the response used int this.

response <- GET("http://httpbin.org/basic-auth/user/passwd",
    authenticate("user", "passwd"))

There is another function in the httr package which cleans stuff up for us aswell.

content(response, as = "parsed")

$authenticated
[1] TRUE

$user
[1] "user"

A web page has some elements that you can fill in to get the data you want. How do you get this data? If you need to interact with a GUI, there may be difficulty to download stuff from a GUI.

When we search for STATS 769 on google we get the following URL.

https://www.google.co.nz/search?q=stats+769&oq=stats+769&aqs=chrome..69i57j69i61l3j0l2.1778j0j7&sourceid=chrome&ie=UTF-8

We can see ?, &, +, = making some sort of pattern in this URL.

This indicates, we can search whatever we want using Google if we can generate these URLs. This is how GET requests work.

flights <- GET("http://www.rita.dot.gov/bts/acts/customized/table?adfy=2016&adfm=1&adty=2016&adtm=4&aos=0&artd=1&arti&arts&asts=1&astns&astt&ascc&ascp=1")

flights

Response [https://www.rita.dot.gov/bts/acts/customized/table?adfy=2016&adfm=1&adty=2016&adtm=4&aos=0&artd=1&arti&arts&asts=1&astns&astt&ascc&ascp=1]
  Date: 2017-11-10 09:50
  Status: 200
  Content-Type: text/html; charset=utf-8
  Size: 46.1 kB
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML+RDFa 1.0//EN"
  "http://www.w3.org/MarkUp/DTD/xhtml-rdfa-1.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" version="XHTML+...
  xmlns:content="http://purl.org/rss/1.0/modules/content/"
  xmlns:dc="http://purl.org/dc/terms/"
  xmlns:foaf="http://xmlns.com/foaf/0.1/"
  xmlns:og="http://ogp.me/ns#"
  xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#"
  xmlns:sioc="http://rdfs.org/sioc/ns#"
  xmlns:sioct="http://rdfs.org/sioc/types#"
...

There are another type of forms called POST forms. POST rquests allow us to include additional information in the body of the request.

• POST(url, body = list(…))
• We are required to know the names of the inputs that we need to fill in!

# POST a form
# Demonstrate interactive use of
# http://fred.stlouisfed.org/series/TOTALSA/downloaddata
# (which unfortunately is no longer a POST form :()
# so use "dummy" form instead
url <- "https://www.stat.auckland.ac.nz/~paul/stats769/2017/Form/output.php"
query <- list(count="5")
response <- POST(url, body = query)
content(response, as="text")

[1] "<html>\n<body>\nTAILS<br>TAILS<br>TAILS<br>HEADS<br>TAILS<br></body>\n</html>\n"

If web servers don't like someone making a shit tonne of requests, you may end up being banned from the web server! Or your IP Address maybe banned ;-)

• Sometimes content these days is generated dynamically. This can be quite complicated to scrape, in particular when the content has been generated by some scripting language such as JavaScript

e.g. www.worldrugby.org/results

• If we did a GET from the httr package on this web page we would end up with something else.

• We use the package rdom. We need phantomjs also on the machine. This isn't really in the scope of the course. BUT, remember this!!

• Use HTTP to access a set of (possibly dynamic) resources on a server.
• Not just requesting a static resource; asking the server to perform an action and then return a result.
• In terms of access, a web API is just a predictable pattern of URLs.

• Read API documentation (including Terms of Use!!)
• The docs explicitly define how to generate URL patterns.
• For example let's look at www.geonet.co.nz
• Different formats include XML, JSON, HTML.

Web APIs are designed to share information with a programmer, rather than having someone reverse-engineer an interface.

• How do we interpret different HTML, JSON etc and parse it! ;-) coming up next week.