DATA 607 - Week 9

Introduction

Your task is to choose one of the New York Times APIs, construct an interface in R to read in the JSON data, and transform it to an R dataframe

For this assignment, I chose to work with the NYT Movie Reviews API. The API interface that I constructed is a wrapper function in R, getMovies, which allows the user to specify various search parameters, including:

Search words (string): keywords to search on; matches movie titles and indexed terms in the movie reviews
Critics’ Picks (“Y” or “N”): search only movies selected as a NYT Critics’ Pick (“Y”), or search all movies (“N”)
Opening date (string): search movies that have opening dates in the NY area on or after a single date, or between a date range
Offset (integer): the API by default returns a list of up to 20 movies sorted by publication date of the review; if the search results in more than 20 movies, this input specifies how many results to exclude first before returning up to 20 results.

The wrapper function reads in the JSON data returned by the API query, and transforms this into a dataframe for convenient viewing.

Contruct the wrapper function

To start with preliminaries, we load several libraries needed to construct the function. We will use the httr package to access the API, and jsonlite to parse the JSON data returned.

Also, for communicating with the API, we use the API key from a saved environment variable and set up a user-agent string.

library(httr)
library(jsonlite)
library(stringr)
library(dplyr)
library(knitr)

# get api key from environment variable
key <- Sys.getenv("NYT_api_key")
# set user agent
ua <- user_agent(paste("kecbenson@gmail.com", R.version$platform, R.version$version.string))
ua

## <request>
## Options:
## * useragent: kecbenson@gmail.com x86_64-w64-mingw32 R version 3.5.1 (2018-07-02)

Next, we define getMovies as a function of several inputs. Some brief comments on the code below:

A. First set default values for the inputs as:
- No search keywords: search = NULL
- Include Critics’ Picks as well as non-Critics’ Picks: critics = “N”
- No opening date: open = NULL
- No offset (start with first search result returned): offset = 0
- Don’t use the R viewer: view = “N”.
B. Validate input values and return an error message in case of invalid inputs.
- critics must be “Y” or “N”
- open must be in date format of “yyyy-mm-dd” (single date) or “yyyy-mm-dd;yyyy-mm-dd” format (date range)
- offset must be an integer (actually the API just truncates this input to arrive at an integer, so it can be any numeric value)
- For view we can be more lenient, since this is only used in our R function; this can be any of a set of “Y”/“N” character equivalents.
C. Next define the query list based on the API key and input values, and send the GET request to the API.
D. When the JSON data is returned from the API, use fromJSON to parse the data and return a list object. Note that we set the flatten argument to TRUE, since the dataframe we want has embedded dataframes in two columns.
E. Provide error handling in case the API returns an error.
F. The dataframe we want is the 5th item in the parsed list, so extract it and then select useful columns to display. In this step we rename the columns, and also convert the 0/1 data in the “Critics’ Pick” column to “Y”/“N”.
G. Finally, view (if selected) and return the dataframe.

# A. set default values for inputs
getMovies <- function(search = NULL, critics = "N", open = NULL, offset = 0, view = "N") {
    
    # B. check valid inputs for critics, open, offset, and view
    if ( !(critics %in% c("Y", "N")) ) {
        stop("Wrong NYT Critics' Picks parameter.  Choose either 'Y' or 'N'.")
    }
    if ( !is.null(open) ) {
        if ( !str_detect(open, "^(\\d{4}-\\d{2}-\\d{2}(;\\d{4}-\\d{2}-\\d{2})?)$") ) {
            stop("Wrong opening date parameter.  Choose a single date in 'yyyy-mm-dd' format 
                 or a date range in 'yyyy-mm-dd;yyyy-mm-dd' format.")
        }
    }
    if ( !is.numeric(offset) ) {
        stop("Wrong offset parameter.  Choose an integer value.")
    }
    if ( !(view %in% c("Y", "N", "y", "n", "YES", "NO", "Yes", "No", "yes", "no")) ) {
        stop("Wrong view parameter.  Choose either 'Y' or 'N'.")
    }
    
    
    # C. define url & query list and send GET request 
    url <- "https://api.nytimes.com/svc/movies/v2/reviews/search.json"
    query_list <- list("api-key" = key, 
                       query = search, 
                       "critics-pick" = critics,
                       "opening-date" = open,
                       offset = offset
                       ) 
    resp <- GET(url, query = query_list, ua) 

    # D. parse JSON data into list, flatten nested lists
    parsed <- fromJSON(content(resp, "text"), flatten = TRUE)

    # E. provide API error handling 
    if (http_error(resp)) {
        stop(
            sprintf(
                "Failed request to NYT Movie Reviews API\n[%s] %s",
                status_code(resp),
                parsed$message
            ),
            call. = FALSE
        )
    }

    # F. pass 5th list item to dataframe and select & rename columns   
    df <- parsed[[5]] %>% select(
        "Title" = 1,
        "Rating" = 2,
        "Opening Date" = 8,
        "Critics' Pick" = 3,
        "Reviewer" = 4,
        "Headline" = 5,
        "Review Date" = 7,
        "Review URL" = 11
        )
    # convert 0,1 in critics' pick column to "Y", "N"
    df$"Critics' Pick" <- ifelse(df$"Critics' Pick" == 1, "Y", "N")
    
    # G. view and return dataframe    
    if ( view %in% c("Y", "y", "YES", "Yes", "yes") ) View(df)
    return(df)
}

Test the wrapper function

First let’s test the validation of the function arguments.

# test input validation
getMovies(critics = "yes")

## Error in getMovies(critics = "yes"): Wrong NYT Critics' Picks parameter.  Choose either 'Y' or 'N'.

getMovies(open = "2018-9-15")

## Error in getMovies(open = "2018-9-15"): Wrong opening date parameter.  Choose a single date in 'yyyy-mm-dd' format or a date range in 'yyyy-mm-dd;yyyy-mm-dd' format.

getMovies(open = "2018-09-15;2018-10-1")

## Error in getMovies(open = "2018-09-15;2018-10-1"): Wrong opening date parameter.  Choose a single date in 'yyyy-mm-dd' format or a date range in 'yyyy-mm-dd;yyyy-mm-dd' format.

getMovies(offset = "N")

## Error in getMovies(offset = "N"): Wrong offset parameter.  Choose an integer value.

getMovies(view = 1)

## Error in getMovies(view = 1): Wrong view parameter.  Choose either 'Y' or 'N'.

Next let’s test the handling of API error messages.

# test API error handling (e.g., no API key)
key <- "abc"
getMovies()

## Error: Failed request to NYT Movie Reviews API
## [403] Invalid authentication credentials

Now let’s do a sample search and confirm the result is a valid dataframe.

# re-set API key
key <- Sys.getenv("NYT_api_key")    

# search star trek movies & confirm result is a valid dataframe
df_test <- getMovies("star trek") 
str(df_test)

## 'data.frame':    13 obs. of  8 variables:
##  $ Title        : chr  "Star Trek Beyond" "Star Trek Into Darkness" "Star Trek" "Star Trek: Nemesis" ...
##  $ Rating       : chr  "PG-13" "PG-13" "PG-13" "PG-13" ...
##  $ Opening Date : chr  "2016-07-22" "2013-05-16" "2009-05-08" "2002-12-13" ...
##  $ Critics' Pick: chr  "N" "N" "Y" "N" ...
##  $ Reviewer     : chr  "A. O. SCOTT" "A. O. SCOTT" "MANOHLA DARGIS" "Stephen Holden" ...
##  $ Headline     : chr  "Review: Star Trek Beyond Sticks to Its Brand. Thats Not Necessarily Bad." "Kirk and Spock, in Their Roughhousing Days" "A Franchise Goes Boldly Backward" "Star Trek: Nemesis (Movie)  " ...
##  $ Review Date  : chr  "2016-07-20" "2013-05-15" "2009-05-07" "2002-12-13" ...
##  $ Review URL   : chr  "http://www.nytimes.com/2016/07/22/movies/review-star-trek-beyond.html" "http://www.nytimes.com/2013/05/16/movies/star-trek-into-darkness-directed-by-j-j-abrams.html" "http://www.nytimes.com/2009/05/08/movies/08trek.html" "http://www.nytimes.com/movie/review?res=9C02E1D9133AF930A25751C1A9649C8B63" ...

kable(df_test, caption = "Star Trek Films")

Star Trek Films
Title	Rating	Opening Date	Critics’ Pick	Reviewer	Headline	Review Date	Review URL
Star Trek Beyond	PG-13	2016-07-22	N	A. O. SCOTT	Review: ‘Star Trek Beyond’ Sticks to Its Brand. Thats Not Necessarily Bad.	2016-07-20	http://www.nytimes.com/2016/07/22/movies/review-star-trek-beyond.html
Star Trek Into Darkness	PG-13	2013-05-16	N	A. O. SCOTT	Kirk and Spock, in Their Roughhousing Days	2013-05-15	http://www.nytimes.com/2013/05/16/movies/star-trek-into-darkness-directed-by-j-j-abrams.html
Star Trek	PG-13	2009-05-08	Y	MANOHLA DARGIS	A Franchise Goes Boldly Backward	2009-05-07	http://www.nytimes.com/2009/05/08/movies/08trek.html
Star Trek: Nemesis	PG-13	2002-12-13	N	Stephen Holden	Star Trek: Nemesis (Movie)	2002-12-13	http://www.nytimes.com/movie/review?res=9C02E1D9133AF930A25751C1A9649C8B63
Star Trek: Insurrection	PG	1998-12-11	N	Stephen Holden	Star Trek: Insurrection (Movie)	1998-12-11	http://www.nytimes.com/movie/review?res=9505E2DB103AF932A25751C1A96E958260
Star Trek: First Contact	PG-13	1996-11-22	N	Janet Maslin	Star Trek: First Contact (Movie)	1996-11-22	http://www.nytimes.com/movie/review?res=9A0CE4DA163DF931A15752C1A960958260
Star Trek: Generations	PG	1994-11-18	N	Janet Maslin	STAR TREK GENERATIONS (MOVIE)	1994-11-18	http://www.nytimes.com/movie/review?res=9B05E5DD1031F93BA25752C1A962958260
Star Trek VI: the Undiscovered Country	PG	1991-12-06	N	Janet Maslin	STAR TREK VI: THE UNDISCOVERED COUNTRY (MOVIE)	1991-12-06	http://www.nytimes.com/movie/review?res=9D0CE6DD153EF935A35751C1A967958260
Star Trek V: the Final Frontier	PG	1989-06-09	N	Caryn James	STAR TREK V: THE FINAL FRONTIER (MOVIE)	1989-06-09	http://www.nytimes.com/movie/review?res=950DEEDC173FF93AA35755C0A96F948260
Star Trek IV	PG	1986-11-26	N	Janet Maslin	STAR TREK IV: THE VOYAGE HOME (MOVIE)	1986-11-26	http://www.nytimes.com/movie/review?res=9A0DE1DE153FF935A15752C1A960948260
Star Trek III	PG	1984-06-01	N	Janet Maslin	STAR TREK III: THE SEARCH FOR SPOCK (MOVIE)	1984-06-01	http://www.nytimes.com/movie/review?res=9B05E3DF153BF932A35755C0A962948260
Star Trek II: the Wrath of Khan	PG	1982-06-04	Y	Janet Maslin	STAR TREK II: THE WRATH OF KHAN (MOVIE)	1982-06-04	http://www.nytimes.com/movie/review?res=9C0CEFDB103BF937A35755C0A964948260
Star Trek: the Motion Picture	G	1979-12-07	N	VINCENT CANBY	Star Trek -The Motion Picture (Movie)	1979-12-08	http://www.nytimes.com/movie/review?res=9E02E7DB163BE732A2575BC0A9649D946890D6CF

Show sample output

Finally let’s show some sample output.

# search star wars movies selected as critics' picks
getMovies("star wars", critics = "Y") %>% kable(caption = "Star Wars Films Selected as NYT Critics' Picks")

Star Wars Films Selected as NYT Critics’ Picks
Title	Rating	Opening Date	Critics’ Pick	Reviewer	Headline	Review Date	Review URL
Star Wars: The Last Jedi	PG-13	2017-12-15	Y	MANOHLA DARGIS	Review: ‘Star Wars: The Last Jedi’ Embraces the Magic and Mystery	2017-12-12	http://www.nytimes.com/2017/12/12/movies/star-wars-the-last-jedi-review.html
Star Wars: Episode I - The Phantom Menace	PG	1999-05-19	Y	Janet Maslin	Star Wars: Episode I-the Phantom Menace (Movie)	1999-05-19	http://www.nytimes.com/movie/review?res=9C04E2DE133EF93AA25756C0A96F958260
Star Wars	PG	1977-05-25	Y	VINCENT CANBY	Star Wars (Movie)	1977-05-26	http://www.nytimes.com/movie/review?res=EE05E7DF1738EE70BC4E51DFB366838C669EDE

# search critics' picks in September
getMovies(critics = "Y", open = "2018-09-01;2018-09-30") %>% kable(caption = "NYT Critics' Picks in September 2018")

NYT Critics’ Picks in September 2018
Title	Rating	Opening Date	Critics’ Pick	Reviewer	Headline	Review Date	Review URL
Free Solo	PG-13	2018-09-28	Y	JEANNETTE CATSOULIS	Review: In ‘Free Solo,’ Braving El Capitan With Only Fingers and Toes	2018-09-27	http://www.nytimes.com/2018/09/27/movies/free-solo-review-alex-honnold-el-capitan.html
Bad Reputation	R	2018-09-28	Y	GLENN KENNY	Review: ‘Bad Reputation’ Argues for Joan Jetts Importance	2018-09-27	http://www.nytimes.com/2018/09/27/movies/bad-reputation-review-joan-jett.html
Monsters and Men	R	2018-09-28	Y	JEANNETTE CATSOULIS	Review: In ‘Monsters and Men,’ Racism Disrupts a Neighborhood	2018-09-27	http://www.nytimes.com/2018/09/27/movies/monsters-and-men-review.html
The Sisters Brothers	R	2018-09-21	Y	MANOHLA DARGIS	Review: Blood Is Never Simple in ‘The Sisters Brothers’	2018-09-20	http://www.nytimes.com/2018/09/20/movies/the-sisters-brothers-review-joaquin-phoenix.html
Quincy		2018-09-21	Y	GLENN KENNY	Review: ‘Quincy’ Captures a Lifelong Love Affair With Music	2018-09-20	http://www.nytimes.com/2018/09/20/movies/quincy-review-quincy-jones.html
Tea with the Dames		2018-09-21	Y	GLENN KENNY	Review: In ‘Tea With the Dames,’ Four Legends Dish on Acting and Love	2018-09-20	http://www.nytimes.com/2018/09/20/movies/tea-with-the-dames-review.html
Fahrenheit 11/9	R	2018-09-21	Y	GLENN KENNY	Review: In ‘Fahrenheit 11/9,’ Michael Moore Targets Trump, and Us	2018-09-20	http://www.nytimes.com/2018/09/20/movies/fahrenheit-11-9-review-michael-moore.html
Mandy	Not Rated	2018-09-14	Y	GLENN KENNY	Review: Cults, Demons and Nicolas Cage in ‘Mandy’	2018-09-17	http://www.nytimes.com/2018/09/17/movies/mandy-review-nicolas-cage.html
The Land of Steady Habits		2018-09-14	Y	A.O. SCOTT	Review: In ‘The Land of Steady Habits,’ Suburban Malaise, With a Twist	2018-09-13	http://www.nytimes.com/2018/09/13/movies/the-land-of-steady-habits-review.html
Hale County This Morning, This Evening		2018-09-14	Y	GLENN KENNY	Review: A Multiplicity of Moments in Under 80 Minutes in ‘Hale County’	2018-09-13	http://www.nytimes.com/2018/09/13/movies/hale-county-this-morning-this-evening-review-ramell-ross.html
The Public Image is Rotten		2018-09-14	Y	GLENN KENNY	Review: Johnny Rotten Mellows (a Bit) With Age	2018-09-13	http://www.nytimes.com/2018/09/13/movies/the-public-image-is-rotten-review-sex-pistols.html
Lizzie	R	2018-09-14	Y	JEANNETTE CATSOULIS	Review: In ‘Lizzie,’ an Oppressed Daughter Driven to Murder	2018-09-11	http://www.nytimes.com/2018/09/11/movies/lizzie-review-chloe-sevigny-kristen-stewart-borden.html
Five Fingers for Marseilles		2018-09-07	Y	TEO BUGBEE	Review: ‘Five Fingers for Marseilles’ Takes the Western to South Africa	2018-09-06	http://www.nytimes.com/2018/09/06/movies/five-fingers-for-marseilles-review.html
Bisbee ’17	PG	2018-09-05	Y	A.O. SCOTT	Review: In ‘Bisbee 17,’ Anti-Union Violence Haunts an Arizona Town	2018-09-04	http://www.nytimes.com/2018/09/04/movies/bisbee-17-review-documentary.html

# search movies opening in October, starting after 45 results
getMovies(open = "2018-10-01", offset = 45) %>% kable(caption = "Movies that Opened in October 2018")

Movies that Opened in October 2018
Title	Rating	Opening Date	Critics’ Pick	Reviewer	Headline	Review Date	Review URL
Bayou Caviar		2018-10-05	N	BEN KENIGSBERG	Review: In ‘Bayou Caviar,’ the Main Course Is a Cold-Blooded Noir	2018-10-04	http://www.nytimes.com/2018/10/04/movies/bayou-caviar-review.html
Shine	R	2018-10-05	N	KEN JAWOROWSKI	Review: ‘Shine’ Puts the Focus on Salsa Dancing in Spanish Harlem	2018-10-04	http://www.nytimes.com/2018/10/04/movies/shine-review.html
Viking Destiny	R	2018-10-05	N	JEANNETTE CATSOULIS	Review: In ‘Viking Destiny,’ a Warrior Princess Reclaims Her Throne	2018-10-04	http://www.nytimes.com/2018/10/04/movies/viking-destiny-review.html
A Star Is Born	R	2018-10-05	Y	MANOHLA DARGIS	Review: ‘A Star Is Born’ Brings Gorgeous Heartbreak	2018-10-03	http://www.nytimes.com/2018/10/03/movies/a-star-is-born-review-lady-gaga-bradley-cooper.html
The Hate U Give	PG-13	2018-10-19	N	AISHA HARRIS	Review: In ‘The Hate U Give,’ a Police Shooting Forces a Teen to Find Her Voice	2018-10-03	http://www.nytimes.com/2018/10/03/movies/the-hate-u-give-review-amandla-stenberg.html
The Great Buster		2018-10-05	N	A.O. SCOTT	Review: ‘The Great Buster’ Brings a Deadpan Genius Back to Life	2018-10-03	http://www.nytimes.com/2018/10/03/movies/the-great-buster-a-celebration-review-documentary.html
Moynihan		2018-10-03	N	BEN KENIGSBERG	Review: ‘Moynihan’ Is a Conventional, Entertaining Political Documentary	2018-10-02	http://www.nytimes.com/2018/10/02/movies/moynihan-review-documentary.html

For further development

To enhance the usefulness of the wrapper function, some suggestions for further development include:

Develop the ability to return all search results, instead of limiting the results to 20. For instance, if the NYT API message header includes details on how many results there are in total, then we could determine the number of GET requests needed to retrieve all the search results, and then after getting all the results, we could combine them into a single dataframe.
Include other features of the NYT Movie Reviews API; for instance the API allows searching by reviewer name, or re-ordering the results by title, opening date, or review publication date. Also the API has another search module that includes information on each reviewer.

DATA 607 - Week 9 - Web APIs

Kevin Benson

October 27, 2018

Introduction

Contruct the wrapper function

Test the wrapper function

Show sample output

For further development