The New York Times web site provides a rich set of APIs, as described here: https://developer.nytimes.com/apis. Based on the example provided in the NY Times web site fro the movie reviews API, I have constructed an interface to read in the JSON data, and transform it into an R dataframe.
NOTE: This R markdown file must be “Knit with Parameters” with a valid API key entered for the NY Times APIs to successfully respond. This file utilized the movie reviews API.
library(httr)
library(jsonlite)
library(tidyverse)## ── Attaching packages ────────────────────────────────────────────────────────────────────────────────── tidyverse 1.3.0 ──## ✔ ggplot2 3.2.1 ✔ purrr 0.3.3
## ✔ tibble 2.1.3 ✔ dplyr 0.8.3
## ✔ tidyr 1.0.2 ✔ stringr 1.4.0
## ✔ readr 1.3.1 ✔ forcats 0.4.0## ── Conflicts ───────────────────────────────────────────────────────────────────────────────────── tidyverse_conflicts() ──
## ✖ dplyr::filter() masks stats::filter()
## ✖ purrr::flatten() masks jsonlite::flatten()
## ✖ dplyr::lag() masks stats::lag()With the API Key provided as a parameter, the API request is sent given the URI seen in the code. As the NY Times example showed, the following API request is for all movies reviewed by the NY Times during the decade of the 1980s with ‘big’ in the title.
The response is checked to ensure a successful status was returned, or else the processing will stop. Also, if the response object is not JSON, the processing will stop.
# Url to call for all movies with 'big' in the title for the decade of the 1980s
url_base <- "https://api.nytimes.com/svc/movies/v2/reviews/search.json?query=big&opening-date=1980-01-01;1990-01-01&api-key="
# Get API key from input parameter
api_key <- params$apiKey
# Concatenate the URL base with the api key
url <- paste(url_base, api_key)
# Make the GET request to the API resource
r <- GET(url)
# Check the response status
warn_for_status(r)
stop_for_status(r)
# Ensure response is json, if not, halt processing
if (http_type(r) != "application/json") {
stop("API did not return json", call. = FALSE)
}From the API response object in JSON format, the response is converted to a list. From the overall response list, the results item is flattened to a list itself and then loaded into a dataframe.
# Convert the JSON reponse to a list
resp <- (fromJSON(content(r, "text"), simplifyVector = FALSE))
# Just use the 'results' item from the list as that is where the meaningful data resides
results <- resp$results
# Flatten the list, as the links column is actually a list in a list
results_flat <- do.call(rbind, results)
# Convert the flattened list to a Dataframe
df <- as.data.frame(results_flat)With the flattened results item list now in a dataframe format, the links column is transformed to remove the excess information and retain only the URL of the movie review. Initially, the links column was a list, so the transformation is required to ensure just the URL remains.
# Clean up the link column because it was a list itself, and I only want to maintain the actual URL
df <- df %>% separate(link, into = c("article", "link_url", "link_title"), sep = ",")
# Remove unwanted columns after separating the initial link column
df <- subset(df, select=-c(article,link_title))
# Clean up link_url column
# - Remove the 'url = ' text from each cell
df <- df %>% separate(link_url, into = c("url", "link_url"), sep = " = ")
df <- subset(df, select=-c(url))
# Remove the extra quotes that still remain on the link value
df$link_url <- gsub("\"", "", df$link_url)Display the final dataframe of the movies reviewed by the NY Times in the 1980s with the word ‘big’ in the title.
# Output the final Dataframe
df## display_title mpaa_rating critics_pick
## 1 The Big Blue R 0
## 2 Big Top Pee-Wee PG 0
## 3 Big Business PG 0
## 4 Big PG 0
## 5 Bright Lights, Big City R 0
## 6 Anguish R 0
## 7 Near Dark R 0
## 8 Big Shots PG-13 0
## 9 The Big Town R 0
## 10 The Big Easy R 0
## 11 Nutcracker: The Motion Picture G 0
## 12 Big Trouble in Little China PG-13 0
## 13 Big Trouble R 0
## 14 The Boys Next Door R 1
## 15 Pee-Wee's Big Adventure PG 0
## 16 The Big Chill R 1
## byline headline
## 1 Janet Maslin Big Blue, The (Movie)
## 2 Caryn James BIG TOP PEE-WEE (MOVIE)
## 3 Vincent Canby BIG BUSINESS (MOVIE)
## 4 Janet Maslin BIG (MOVIE)
## 5 Janet Maslin BRIGHT LIGHTS, BIG CITY (MOVIE)
## 6 CARYN JAMES ANGUISH (MOVIE)
## 7 Caryn James NEAR DARK (MOVIE)
## 8 Walter Goodman BIG SHOTS (MOVIE)
## 9 Caryn James Big Town, The (Movie)
## 10 Lawrence Van Gelder BIG EASY, THE (MOVIE)
## 11 Janet Maslin NUTCRACKER: THE MOTION PICTURE (MOVIE)
## 12 Walter Goodman BIG TROUBLE IN LITTLE CHINA (MOVIE)
## 13 Vincent Canby BIG TROUBLE (MOVIE)
## 14 Vincent Canby BOYS NEXT DOOR, THE (MOVIE)
## 15 Vincent Canby PEE WEE'S BIG ADVENTURE (MOVIE)
## 16 VINCENT CANBY BIG CHILL, THE (MOVIE)
## summary_short
## 1 Deep-sea diver. Good wet photography but movie never surfaces.
## 2 Farmer, stranded circus star and talking pig. Kiddie-cute and remarkably unfunny.
## 3 City twins mixed up with country twins. Cheerful and breezy.
## 4 Boy turns into grown-up overnight. Disarming fantasy-comedy.
## 5 McInerney's cocaine-sniffing fact checker. Atmospheric but limited.
## 6 Psycho son of weird mom. Clever idea but on balance, gory mediocrity.
## 7 Oklahoma cowboy falls prey to bloodthirsty vampires.
## 8 Two Chicago youngsters in adventure over stolen watch.
## 9 Gambler in 1950's Chicago. Colorful but predictable.
## 10 Murder-corruption probe in New Orleans. Flavorsome and crisply credible, if curiously blithe.
## 11 Dancing by the Pacific Northwest Ballet; music by Tchaikovsky; original story by E.T.A. Hoffman; sets and costumes by Maurice Sendak; host, Tony Randall. A sugar-plum short on sweetness.
## 12 San Francisco trucker in Chinatown. Fast, colorful, juicy.
## 13 ``Double Indemnity'' spoof, from Cassavetes. It works.
## 14 Teens on L.A. crime spree.
## 15 Herman odyssey, searching for bike. Barren comedy.
## 16 Old college friends reunited. Sharp, melancholy tale and good ensemble work.
## publication_date opening_date date_updated
## 1 1988-08-20 1988-08-19 2017-11-02 04:17:39
## 2 1988-07-22 1988-07-22 2017-11-02 04:17:39
## 3 1988-06-10 1988-06-10 2017-11-02 04:17:39
## 4 1988-06-03 1988-06-03 2017-11-02 04:17:39
## 5 1988-04-01 1988-04-01 2017-11-02 04:17:38
## 6 1988-01-29 1988-01-08 2017-11-02 04:17:38
## 7 1987-10-04 1987-10-02 2017-11-02 04:17:38
## 8 1987-10-02 1987-10-02 2017-11-02 04:17:38
## 9 1987-09-25 1987-09-25 2017-11-02 04:17:38
## 10 1987-08-28 1987-08-28 2017-11-02 04:17:37
## 11 1986-11-26 1986-11-26 2017-11-02 04:17:37
## 12 1986-07-02 1986-07-02 2017-11-02 04:17:36
## 13 1986-05-30 1986-05-30 2017-11-02 04:17:36
## 14 1986-03-14 1985-10-01 2017-11-02 04:17:34
## 15 1985-08-09 1985-08-09 2017-11-02 04:17:34
## 16 1983-09-23 1983-09-30 2017-11-02 04:17:32
## link_url
## 1 http://www.nytimes.com/1988/08/20/movies/review-film-rival-divers-brave-the-depths-of-the-sea.html
## 2 http://www.nytimes.com/1988/07/22/movies/reviews-film-the-world-of-the-circus-according-to-pee-wee.html
## 3 http://www.nytimes.com/1988/06/10/movies/review-film-a-tomlin-midler-comedy-of-errors.html
## 4 http://www.nytimes.com/1988/06/03/movies/review-film-tom-hanks-as-a-13-year-old-in-big.html
## 5 http://www.nytimes.com/1988/04/01/movies/review-film-a-tale-of-the-dark-side-bright-lights-big-city.html
## 6 http://www.nytimes.com/1988/01/29/movies/film-anguish-horror-from-bigas-luna.html
## 7 http://www.nytimes.com/1987/10/04/movies/film-near-dark-a-tale-of-vampires-on-the-road.html
## 8 http://www.nytimes.com/1987/10/02/movies/film-big-shots-a-comedy.html
## 9 http://www.nytimes.com/1987/09/25/movies/matt-dillon-in-the-big-town.html
## 10 http://www.nytimes.com/1987/08/28/movies/at-the-movies.html
## 11 http://www.nytimes.com/1986/11/26/movies/screen-nutcracker.html
## 12 http://www.nytimes.com/1986/07/02/movies/the-screen-big-trouble-wild-stunts.html
## 13 http://www.nytimes.com/1986/05/30/movies/screen-big-trouble-with-arkin-and-falk.html
## 14 http://www.nytimes.com/1986/03/14/movies/screen-spheeris-s-boys-next-door.html
## 15 http://www.nytimes.com/1985/08/09/movies/screen-pee-wee-s-big-adventure-a-comedy.html
## 16 http://www.nytimes.com/1983/09/23/movies/screen-the-big-chill-reunion-of-60-s-activists.html
## multimedia
## 1 NULL
## 2 NULL
## 3 NULL
## 4 NULL
## 5 NULL
## 6 NULL
## 7 NULL
## 8 NULL
## 9 NULL
## 10 NULL
## 11 NULL
## 12 NULL
## 13 NULL
## 14 NULL
## 15 NULL
## 16 NULLIn conclusion, the assignment was straightforward, and the NY Times developer site provided an easy step-by-step guide to creating a developer account in order to receive a valid API key. Using the Knit with Parameters was used in order to prevent the display of my personal API key in a public Github repo. This user-entered API key does make this file not immediately reproducible by another user, but still easy enough for another use to supply their own valid NY Times API key. The transformations of the JSON response to the dataframe were minimal as JSON already has a well-defined structure. Overall, a valuable exercise in using APIs for retrieving data from the RStudio platform.