Introduction

In this assignment, I demonstrate a pair of functions that, together, return a dataframe containing information from a specified New York Times bestseller list. The functions interact with an NYT API to collect data in JSON format. They return a dataframe containing the contents of the bestseller list.

Setup

The jsonlite package provides functionality for interpreting JSON content downloaded from the API. Tidyverse is here for its usual housekeeping abilities. The api_key and base_url will be used to construct URLs that access the API.

library(jsonlite)
library(tidyverse)
library(knitr)
api_key <- 'dfFgsQLgKyrYxcmXVRqyz4wkVxYjQkao'
base_url <- 'https://api.nytimes.com/svc/books/v3/lists'

Pulling from the API

The NYT publishes bestseller lists for many categories and genres. Some of these lists are actively updated, and some are retired. The function get_list_names() returns a dataframe showing the names and publication dates of NYT bestseller lists. This output serves as a menu of what lists we might request from the API.

get_list_names <- function() {
  a <- str_c(base_url,'/','names','.json?api-key=',api_key) %>%
    fromJSON() %>%
    as.data.frame() %>%
    select(results.list_name, 
           results.oldest_published_date, 
           results.newest_published_date) %>%
    rename('list name' = results.list_name, 
           'first-published' = results.oldest_published_date, 
           'most-recent' = results.newest_published_date)
  return(a)
}

Once we’ve decided on a list name and a date (or “current”), we can query the API with get_list(). This function first reformats the list name for use in the URL. Then it returns a dataframe containing the contents of the specified list.

get_list <- function(date, list_name) {
  list_name <- list_name %>%
    str_to_lower() %>%
    str_replace_all(' ','-')
  
  result_url <- str_c(base_url,'/',date,'/',list_name,'/','.json?api-key=',api_key)
  result_content_as_df <- fromJSON(result_url)$results$books %>%
    select('rank':'contributor')
  
  return(result_content_as_df)
}

Vignette

We’re interested in viewing the contents of a NYT bestseller list in a nonfiction genre from the summer of 2020. Let’s look at the lists and their publication dates to help us choose.

menu <- get_list_names()
kable(menu[1:10,])
list name first-published most-recent
Combined Print and E-Book Fiction 2011-02-13 2021-04-18
Combined Print and E-Book Nonfiction 2011-02-13 2021-04-18
Hardcover Fiction 2008-06-08 2021-04-18
Hardcover Nonfiction 2008-06-08 2021-04-18
Trade Fiction Paperback 2008-06-08 2021-04-18
Mass Market Paperback 2008-06-08 2017-01-29
Paperback Nonfiction 2008-06-08 2021-04-18
E-Book Fiction 2011-02-13 2017-01-29
E-Book Nonfiction 2011-02-13 2017-01-29
Hardcover Advice 2008-06-08 2013-04-21

We see the list “Paperback Nonfiction” has been active since 2008, so we request the 2020-08-01 edition from the API.

pbnf <- get_list('2020-08-01', 'Paperback Nonfiction')
kable(pbnf[,c(1,3,7,8,9,11,12)])
rank weeks_on_list primary_isbn13 publisher description title author
1 98 9780807047415 Beacon Historical and cultural analyses on what causes defensive moves by white people and how this inhibits cross-racial dialogue. WHITE FRAGILITY Robin DiAngelo
2 9 9781580058827 Seal A look at the contemporary racial landscape of the United States. SO YOU WANT TO TALK ABOUT RACE Ijeoma Oluo
3 12 9781631494536 Liveright A case for how the American government abetted racial segregation in metropolitan areas across the country. THE COLOR OF LAW Richard Rothstein
4 111 9780143034759 Penguin A biography of the first Treasury secretary and one of the Founding Fathers of the United States. The basis of the Tony Award-winning Broadway musical. ALEXANDER HAMILTON Ron Chernow
5 214 9780812984965 Spiegel & Grau A civil rights lawyer and MacArthur grant recipient’s memoir of his decades of work to free innocent people condemned to death. JUST MERCY Bryan Stevenson
6 195 9781620971932 New Press A law professor on the “war on drugs” and its role in the disproportionate incarceration of Black men. THE NEW JIM CROW Michelle Alexander
7 9 9781568585987 Bold Type Winner of the 2016 National Book Award for nonfiction. A look at anti-Black racist ideas and their effect on the course of American history. STAMPED FROM THE BEGINNING Ibram X Kendi
8 75 9780399588198 Spiegel & Grau A memoir about growing up biracial in apartheid South Africa by the host of “The Daily Show.” BORN A CRIME Trevor Noah
9 30 9780679763888 Vintage An account of the Great Migration of 1915-70, in which six million African-Americans abandoned the South. THE WARMTH OF OTHER SUNS Isabel Wilkerson
10 7 9780465060689 Basic The president emerita of Spelman College examines whether self-segregation is a problem or a coping strategy. WHY ARE ALL THE BLACK KIDS SITTING TOGETHER IN THE CAFETERIA? Beverly Daniel Tatum
11 91 9780143127741 Penguin How trauma affects the body and mind, and innovative treatments for recovery. THE BODY KEEPS THE SCORE Bessel van der Kolk
12 15 9780062319791 Harper Perennial The late true-crime journalist’s search for “the Golden State Killer.” I’LL BE GONE IN THE DARK Michelle McNamara
13 16 9780143036494 Penguin An overview of the 1918 flu epidemic and cautionary tale for similar kinds of large-scale outbreaks. THE GREAT INFLUENZA John M. Barry
14 17 9781571313560 Milkweed Editions A botanist and member of the Citizen Potawatomi Nation espouses having an understanding and appreciation of plants and animals. BRAIDING SWEETGRASS Robin Wall Kimmerer
15 111 9780062316110 Harper Perennial How Homo sapiens became Earth’s dominant species. SAPIENS Yuval Noah Harari

From this list, we can see that many of the books relate to social justice and race in the United States. There are no titles new to the list for our specified date. Two books have occupied positions on the list for much longer than the others: Just Mercy and The New Jim Crow.

Conclusion

APIs offer a convenient way to interact with the data stored on a website. When they are available, they offer considerable advantages over web-scraping. However, because a publisher can alter the functionality of an API at any time, software that depends on APIs may need to be updated frequently.