CUNY Data 607 - Web APIs

Assignment

The New York Times web site provides a rich set of APIs, as described here: https://developer.nytimes.com/apis. Your task is to choose one of the New York Times APIs, construct an interface in R to read in the JSON data, and transform it into an R DataFrame.

Setup

library(tidyverse)
library(rvest)
library(tidyjson)
library(configr)

First we’ll use the configr package to load our secret App Key from a configuration file ‘config.yml’, and constuct a basic API call to the Times Newswire API for the twenty most recent articles:

cfg <- read.config(file = 'config.yml')
api_endpoint <- 'https://api.nytimes.com/svc/news/v3/content/all/all.json'
api_url <- str_c(api_endpoint,'?api-key=',cfg$app_key)

Call API

Next. we’ll call this API endpoint and use the rvest package to parse the text from the HTML body tag in the response:

html <- read_html(api_url)
json <- html %>% html_elements("body") %>% html_text()

Parse JSON

With the tidyjson package, we use the gather_object function to parse the json into a table, and use the json_types function to examine the resulting structure:

json %>% gather_object %>% json_types

# A tbl_json: 4 x 4 tibble with a "JSON" attribute
  ..JSON                  document.id name        type  
  <chr>                         <int> <chr>       <fct> 
1 "\"OK\""                          1 status      string
2 "\"Copyright (c) ..."             1 copyright   string
3 "500"                             1 num_results number
4 "[{\"slug_name\":\"..."           1 results     array 

(Note that the table type is a special tbl_json object which looks like a dataframe or tibble, but always includes an additional attribute with the original json string. We’ll wait until the final step to drop this additional data.)

The results object contains the article information we want, so we select it using enter_object. Since it is an array type, we need to use gather_array (instead of gather_object) to parse the components into a table.

results <- json %>% 
  enter_object('results') %>% 
  gather_array

Each article (each row in our results table) looks something like this:

For each row/article, we want to extract certain fields from the ..JSON field into their own columns.

Here we are using the spread_values function to define new columns individually instead of using spread_all to unpack them all at once, since there is some inconsistency between fields. Some have nulls for list objects (such as des_facet) resulting in errors.

results_dist <- results %>%
  spread_values(
    section = jstring(section),
    title = jstring(title),
    byline = jstring(byline),
    abstract = jstring(abstract),
    created_date = jstring(created_date)
  ) %>%
  select(!c(document.id, array.index))

For convenience. we also de-select some of the index data we no lnoger need, such as document.id and array.index.

Convert to DataFrame

Notice we still have all the original json in the ..JSON column. We’ll drop that by converting our tbl_json object into a normal dataframe:

results_df <- results_dist %>% as_data_frame.tbl_json()

The resulting dataframe is tidy and ready for further handling: