The objective of this assignment is to explore data through a public API, such as the New York Times. The general tasks are to load, clean, and analyze the data are similar to previous assignments, this time with the new challenge of using a live source that replicates a real life work environment with automated and integrated applications. With the right permissions, we’re able to access data from another system (NYT) and connect it to ours.
I’ve chosen the books endpoint, which return the current ranked entries for a named NYT Best Sellers list. We are pulling from the lists hardcover-fiction and hardcover-nonfiction to answer: which books on each list have been there the longest, and how do their current ranks compare to where they stood last week?
Step 1: Create an API key and securely save it in an environment variable.
Step 2: Make the requests to an endpoint. For this assignment, I’ll use the Books endpoint. I’ll build the call with httr2 and wrap in tryCatch so we can failures are noted in the cycle.
Step 3: Parse the JSON results. I’ll use resp_body_json() to grab the results array from each response.
Step 4: Clean up the results by trimming whitespace, removing columns we won’t need or stripping any HTML tags.
Step 5: Flatten into a clean data frame.
Step 6: Analyze and build visualizations.
Ensuring the amount and structure of the results is the one we want back. We have limitations of number of requests per minute due to being on a Free tier, and I will need to “sleep” to give time between requests. Some potential cleaning of the data if there’s any empty or null values, that should be turned to NA in loading, if possible.
Here we will setup the environment with the packages we will use for our load, clean and analysis. We’ll also assign my API key to a variable and confirm it’s set within our R environment. API keys should not be made public to avoid giving others access and ensure the appropriate calls are made, so we’ll keep it stored securely.
library(httr2)
library(rvest)
library(stringr)
library(dplyr)
Attaching package: 'dplyr'The following objects are masked from 'package:stats':
filter, lagThe following objects are masked from 'package:base':
intersect, setdiff, setequal, unionlibrary(purrr)
api_key <- Sys.getenv("NYT_API_KEY")
if (!nchar(api_key)) {
stop(
"NYT_API_KEY is not set. Add it to your .Renviron file ",
"(run usethis::edit_r_environ()) and restart R."
)
}Here we define the fetch_bestsellers function that’ll send a request to the Books endpoint with our API Key based on a specific list name (ex. Fiction, Non-fiction, etc.), and return the data if successful or an error if not.
fetch_bestsellers <- function(list_name) {
resp <- tryCatch(
request("https://api.nytimes.com/svc/books/v3/lists/current") |>
req_url_path_append(paste0(list_name, ".json")) |>
req_url_query(`api-key` = api_key) |>
req_perform(),
error = function(e) {
stop("Request failed (list = ", list_name, "): ", conditionMessage(e))
}
)
resp_check_status(resp)
# results is a metadata wrapper — the actual books sit one level deeper
resp_body_json(resp)[["results"]][["books"]]
}We ran into some issues with getting the plain text from the description field. We’re wrapping the response with <p>tags in order for the input to always look like HTML to the read_html()function, and not treat it as a file path.
strip_html <- function(x) {
if (is.na(x) || !nchar(str_trim(x))) return(NA_character_)
paste0("<p>", x, "</p>") |>
read_html() |>
html_text2() |>
str_trim()
}Here we are naming the Book fields we want to use further on for analysis after our API call, filtering out the other fields that are not needed.
parse_book <- function(book, list_name) {
tibble(
list_name = list_name,
rank = book[["rank"]] %||% NA_integer_,
rank_last_week = book[["rank_last_week"]] %||% NA_integer_,
weeks_on_list = book[["weeks_on_list"]] %||% NA_integer_,
title = str_squish(book[["title"]] %||% NA_character_),
author = str_squish(book[["author"]] %||% NA_character_),
publisher = str_trim(book[["publisher"]] %||% NA_character_),
description = strip_html(book[["description"]] %||% NA_character_),
isbn13 = book[["primary_isbn13"]] %||% NA_character_
)
}Here we actually make the API calls, specifically for the Fiction and Nonfiction book lists, then combine the data frames. We include a sleep in between API calls to respect NYT server’s rate limits and avoid appearing as a bot. We were waiting for 1 second, but ran into errors during testing, so it’s currently bumped up to 4 seconds.
books_fiction <- fetch_bestsellers("hardcover-fiction")
Sys.sleep(4)
books_nonfiction <- fetch_bestsellers("hardcover-nonfiction")
df_fiction <- map(books_fiction, parse_book, list_name = "hardcover-fiction") |> bind_rows()
df_nonfiction <- map(books_nonfiction, parse_book, list_name = "hardcover-nonfiction") |> bind_rows()
df_all <- bind_rows(df_fiction, df_nonfiction)
glimpse(df_all)Rows: 30
Columns: 9
$ list_name <chr> "hardcover-fiction", "hardcover-fiction", "hardcover-fi…
$ rank <int> 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 1, 2…
$ rank_last_week <int> 1, 0, 0, 0, 0, 3, 2, 0, 5, 6, 8, 4, 9, 12, 0, 1, 0, 0, …
$ weeks_on_list <int> 4, 1, 1, 1, 1, 4, 3, 1, 25, 3, 6, 2, 32, 21, 1, 2, 1, 1…
$ title <chr> "THRESHING DAY", "MISS WOLCOTT'S GHOST", "A MARRIED LIT…
$ author <chr> "Rebecca Yarros", "Louise Penny", "Abby Jimenez", "Jame…
$ publisher <chr> "Red Tower", "Minotaur", "Forever", "Little, Brown", "B…
$ description <chr> "Thirteen short stories, set in the Empyrean world, in …
$ isbn13 <chr> "9781682818084", "9781250412607", "9781538780046", "978…df_all <- df_all |>
# rank_last_week = 0 means the book is new this week, not ranked 0th
mutate(rank_last_week = na_if(rank_last_week, 0L)) |>
# weeks_on_list = 0 also flags a debut — at minimum it's been on one week
mutate(weeks_on_list = if_else(weeks_on_list == 0L, 1L, weeks_on_list)) |>
# catch any empty strings left after HTML stripping
mutate(description = na_if(str_trim(description), "")) |>
# negative = moved up, positive = moved down, NA = new this week
mutate(
rank_change = case_when(
is.na(rank_last_week) ~ NA_integer_,
TRUE ~ rank - rank_last_week
)
)
# Filter out any books that might appear twice by checking the ISBN number
df_all |>
count(list_name, isbn13) |>
filter(n > 1)# A tibble: 0 × 3
# ℹ 3 variables: list_name <chr>, isbn13 <chr>, n <int>df_all |>
arrange(list_name, desc(weeks_on_list)) |>
group_by(list_name) |>
slice_head(n = 5) |>
select(list_name, rank, title, author, weeks_on_list, rank_change) |>
print(n = 10)# A tibble: 10 × 6
# Groups: list_name [2]
list_name rank title author weeks_on_list rank_change
<chr> <int> <chr> <chr> <int> <int>
1 hardcover-fiction 13 THE DUNGEON ANAR… Matt … 32 4
2 hardcover-fiction 9 THE CALAMITY CLUB Kathr… 25 4
3 hardcover-fiction 14 THE GATE OF THE … Matt … 21 2
4 hardcover-fiction 11 HOLLOW BONES Jodi … 6 3
5 hardcover-fiction 1 THRESHING DAY Rebec… 4 0
6 hardcover-nonfiction 12 SWAN SONG Charl… 5 4
7 hardcover-nonfiction 7 THE STEPS Sylve… 4 0
8 hardcover-nonfiction 11 THE AMERICAN WAY… Malco… 4 2
9 hardcover-nonfiction 10 BLOCKERS Micha… 3 4
10 hardcover-nonfiction 13 ONE HELLUVA RIDE Micha… 3 9df_all |>
group_by(list_name) |>
summarise(
avg_weeks_on_list = round(mean(weeks_on_list, na.rm = TRUE), 1),
median_rank_change = median(rank_change, na.rm = TRUE),
pct_new_this_week = round(mean(is.na(rank_last_week)) * 100, 1)
)# A tibble: 2 × 4
list_name avg_weeks_on_list median_rank_change pct_new_this_week
<chr> <dbl> <dbl> <dbl>
1 hardcover-fiction 7.1 4 40
2 hardcover-nonfiction 2.3 3.5 33.3Throughout the assignment, Claude (Sonnet 4.6) was used to help plan, explain concepts, flag data issues, and debug errors.
Anthropic. (2026). Claude (Sonnet 4.6) [Large language model].