Access with NYT APIs

Problem Statement

The New York Times web site provides a rich set of APIs, as described here: https://developer.nytimes.com/apis
You’ll need to start by signing up for an API key. Your task is to choose one of the New York Times APIs, construct an interface in R to read in the JSON data, and transform it into an R DataFrame.

Git-Hub

https://github.com/ShovanBiswas/DATA607/tree/master/09

LOad Packages

library(jsonlite)
library(tidyverse)
library(knitr)

Approach

I selected two APIs from NYT’s site, namely Books API and Articles Search API. The former is simpler and the latter is relatively complex. After creating the accounts (of each member), I obtained the the key. In the RMD, I stored the key in a variable and hid it with echo=FALSE.

I Books API

The BOOKS API provides information about book reviews and the NYT best seller list.

Extraction of books by Yuval Noah Harari

From Books API’s best seller list, I filtered out author Harari’s books, and recasted into R dataframe.

url <- paste0("https://api.nytimes.com/svc/books/v3/lists/best-sellers/history.json?author=Yuval+Noah+Harari&api-key=",NYTKEY)
harari <- fromJSON(url) %>% 
  as.data.frame() %>%
  select(4, 5, 7, 11)
names(harari) <- str_replace_all(names(harari), "results.", "")
kable(harari)

Extraction of current best sellers

From Books APIs I selected the current best sellers of NYT.

url <- paste0("https://api.nytimes.com/svc/books/v3/lists/current/hardcover-fiction.json?api-key=",NYTKEY)
current <- fromJSON(url,flatten = TRUE)
x <- current$results      # Having scrutinized the content of "fromJSON(url,flatten = TRUE)", I located that a usable data structure was nested inside of current$results.
y <- x$books              # The exact data frame, where it reided was x$books.
kable(y)

df <- y %>%
  select(1, 8,9,11,12,13,18)
kable(df)

II Articles Search API

Article Search API provides faclity to look up articles by keyword, which can be refined by using filters and facets.

Setting up parameters

search_item <- "Iran+Nuclear"
begin_date <- "20191001"
end_date <- "20191015"

Extraction of chosen articles

There were 36 hits, with 10 hits per page. In max_pages, I computed the number of pages to be 3.

article_url <- paste0("http://api.nytimes.com/svc/search/v2/articlesearch.json?q=",search_item,"&begin_date=",begin_date,"&end_date=",end_date,"&facet_filter=true&api-key=", NYTKEY, sep="")
query <- fromJSON(article_url)
max_pages <- round((query$response$meta$hits[1] / 10) - 1)  # Computing the number of pages

Extraction of chosen articles

Now, I’ll create a dictionary, called pages, and store the individual pages in it.

pages <- list()
for(i in 0:max_pages){
  nyt_search <- fromJSON(paste0(article_url, "&page=", i), flatten = TRUE) %>% data.frame() 
  message("Retrieving page ", i + 1)
  pages[[i + 1]] <- nyt_search 
  Sys.sleep(2) 
}

## Retrieving page 1

## Retrieving page 2

## Retrieving page 3

r-binding all pages into one big data frame, whose class is shown below. At this point, I achieved my aim of extracting news into a data frame.

all_pages <- rbind_pages(pages)   # Now, I rbind the individual pages into one big data frame
class(all_pages)                  # This is a data frame

## [1] "data.frame"

Data Visualization

Now, I’ll explore what data visualization I can do.

all_pages %>%
  group_by(response.docs.type_of_material) %>%
  summarize(count = n()) %>%
  mutate(percent = (count / sum(count)) * 100) %>%
  ggplot() +
  geom_bar(aes(y = percent, x = reorder(response.docs.type_of_material, percent), fill = response.docs.type_of_material), stat = "identity") + coord_flip()

## `summarise()` ungrouping output (override with `.groups` argument)

We observe that majority of the articles are in news category, followed by Op-Ed.

In the following I’ll explore a section wise analysis of news about Iran Nuclear deal.

all_pages %>%
  group_by(response.docs.section_name) %>%
  summarize(count = n()) %>%
  mutate(percent = (count / sum(count)) * 100) %>%
  ggplot() +
  geom_bar(aes(y = percent, x = reorder(response.docs.section_name, percent), fill = response.docs.section_name), stat = "identity") + coord_flip()

## `summarise()` ungrouping output (override with `.groups` argument)

We conclude that the World section contains the maximum amount of responses.

Marker: 607-09