Rpubs link: http://rpubs.com/jefflittlejohn/Data_607_Week_9
Github: https://github.com/littlejohnjeff/DATA607_Fall2018/blob/master/Data%20607%20-%20Week%209%20Assignment%20-%20Working%20with%20APIs%20%26%20the%20NYT%20-%20Jeff%20Littlejohn.Rmd

To learn a little more about using Web APIs to grab data, we’re going to explore an API from The New York Times (NYT). Assignment 9 allows us to select one of many APIs that the newspaper makes available using JSON data. Our task is to

Load needed libraries.

library(jsonlite)
library(dplyr)

## 
## Attaching package: 'dplyr'

## The following objects are masked from 'package:stats':
## 
##     filter, lag

## The following objects are masked from 'package:base':
## 
##     intersect, setdiff, setequal, union

library(knitr)

## Warning: package 'knitr' was built under R version 3.3.3

library(kableExtra)
library(ggplot2)
library(forcats)

Save the key as a variable then add to api url.

nyt_key <- "2c7ad3d6e8a34e74bc335f65ca427163"
#choosing most views for last 30 days from all NYT sections
nyt_url <- "http://api.nytimes.com/svc/mostpopular/v2/mostviewed/all-sections/30.json?api-key="
api_url <- paste(nyt_url,nyt_key,sep="")
api_url

## [1] "http://api.nytimes.com/svc/mostpopular/v2/mostviewed/all-sections/30.json?api-key=2c7ad3d6e8a34e74bc335f65ca427163"

Credit to classmates for pointing out this excellent Story Bench article, which walks one through pulling data specifically from the NYT API.

#load to a dataframe as raw
df_most_viewed_all_30_raw <- fromJSON(URLencode(api_url)) %>% data.frame()
kable(df_most_viewed_all_30_raw[1:10,])

Due to settings on the NYT API, our API call only returned the first 20 results - the 20 most viewed articles for the last 30 days. What if we wanted to get all the most viewed articles from that same period?

Using the Story Bench article’s guidance and our knowledge that the NYT API returns 20 results per call, let’s figure out how many calls we would have to make to get all the most viewed articles?

initialQuery <- fromJSON(api_url)
maxPages <- round((initialQuery$num_results/ 20)-1) 
maxPages

## [1] 95

I don’t really care about getting 95 x 20 results but want more than 20. Let’s go for the top 100. So, instead of 95, I’m going to loop through calls for 5 pages (20 results x 5 pages = 100 articles)

pages <- list()
for(i in 0:4){
  nytSearch <- fromJSON(paste0(api_url, "&page=", i), flatten = TRUE) %>% data.frame() 
  message("Retrieving page ", i)
  pages[[i+1]] <- nytSearch 
  Sys.sleep(1) 
}

## Retrieving page 0

## Retrieving page 1

## Retrieving page 2

## Retrieving page 3

## Retrieving page 4

Use rbind_pages to combine a list of data frames into a single data frame, which is particularly useful for JSON API data with limits per request.

#rbind pages: https://www.rdocumentation.org/packages/jsonlite/versions/1.5/topics/rbind_pages
df_most_viewed_all_30_raw_combined <- rbind.pages(pages)
kable(df_most_viewed_all_30_raw_combined[1:10,])

Clean up the data frame a little, removing some unneeded columns and renaming some remaining ones. We are leaving some cool metadata like “PER” (persons) and “ORG” (organizations) that could really be useful in data projects.

df_nyt_most_viewed_30_days <- df_most_viewed_all_30_raw_combined[c(4,5,7,8,9,10,11,12)]
kable(df_nyt_most_viewed_30_days)

Let’s look at a count of the most viewed articles in the last 30 days by section.

#graph
ggplot(df_nyt_most_viewed_30_days,aes(x=results.section)) + geom_bar() + coord_flip()

Articles in the US Section dominate the most viewed articles in the last 30 days - as of the creation of this r code in late October 2018.