Assignment 9

NY Times API

Approach

I am going to use the “Best Sellers” endpoint of the NYT Books API to get a summary of the highest selling books in all current bestseller lists. I want to find out which publishers are dominating this weeks lists and which books are appearing on multiple lists.

Once I have confirmed that I can use my API key, I will extract the nested JSON data and create a clean table with one row for each book on each list. This will include the list name, rank, number of weeks on the list, book title, author, publisher, ISBN and price.

To clean the data I will replace any empty strings with actual missing values. I will also convert the ranking, weeks on list and price to numerical values. I will also identify any books that are brand new to the list this week. Finally I will remove any nested columns that aren’t required such as the buy link. I will also leave the books that are repeated as they are part of my question.

Finally I will calculate how many times each publisher appears, graph the top 10 and find any books that are appearing on multiple lists.

Code Base

The Best Sellers overview endpoint is at https://api.nytimes.com/svc/books/v3/lists/overview.json. It returns the top books from every current best-seller list, including fiction, nonfiction, and children’s, in a single call. The JSON response is nested, with a list of best-seller lists that each contain a list of books.

The request uses two parameters. The first is api-key, which authenticates the request and is read from the NYT_API_KEY environment variable rather than hard-coded. The second is published_date, which I left out so that the API goes to the most recent overview.

To keep the key secure, I stored it in an environment variable. I opened my .Renviron file, added the line NYT_API_KEY=your_key_here, saved it, and restarted R. The key never appears in this file or in the GitHub repository.

library(tidyverse)
Warning: package 'tidyverse' was built under R version 4.6.1
── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ──
✔ dplyr     1.2.1     ✔ readr     2.2.0
✔ forcats   1.0.1     ✔ stringr   1.6.0
✔ ggplot2   4.0.3     ✔ tibble    3.3.1
✔ lubridate 1.9.5     ✔ tidyr     1.3.2
✔ purrr     1.2.2     
── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
✖ dplyr::filter() masks stats::filter()
✖ dplyr::lag()    masks stats::lag()
ℹ Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errors
library(httr2)
Warning: package 'httr2' was built under R version 4.6.1
api_key <- Sys.getenv("NYT_API_KEY")
resp <- request("https://api.nytimes.com/svc/books/v3/lists/overview.json") |>
  req_url_query(`api-key` = api_key) |>
  req_perform()

resp_status(resp)
[1] 200
body <- resp_body_json(resp, simplifyVector = FALSE)

body$results$lists[[4]]$books[[7]]$title
[1] "THE STEPS"
books_raw <- tibble(lists = body$results$lists) |>
  unnest_wider(lists) |>
  select(list_name, display_name, books) |>
  unnest_longer(books) |>
  unnest_wider(books)

books <- books_raw |>
  transmute(
    list_name,
    rank = as.integer(rank),
    rank_last_week = as.integer(rank_last_week),
    weeks_on_list = as.integer(weeks_on_list),
    title = str_to_title(title),
    author,
    publisher,
    isbn13 = primary_isbn13,
    description,
    age_group,
    price = parse_number(as.character(price))
  ) |>

  mutate(
    across(where(is.character), ~ na_if(str_squish(.x), "")),
    new_this_week = rank_last_week == 0,
    price = na_if(price, 0)
  )

head(books)
# A tibble: 6 × 12
  list_name      rank rank_last_week weeks_on_list title author publisher isbn13
  <chr>         <int>          <int>         <int> <chr> <chr>  <chr>     <chr> 
1 Combined Pri…     1              0             1 Miss… Louis… Minotaur  97812…
2 Combined Pri…     2              0             1 Chai… Lee C… Bantam    97982…
3 Combined Pri…     3              1             4 Thre… Rebec… Red Tower 97816…
4 Combined Pri…     4              0             1 The … James… Little, … 97803…
5 Combined Pri…     5              4            47 Theo… Allen… Atria     97816…
6 Combined Pri…     6              0             1 A Ma… Abby … Forever   97815…
# ℹ 4 more variables: description <chr>, age_group <chr>, price <dbl>,
#   new_this_week <lgl>

Which publishers appear most often?

publisher_counts <- books |>
  filter(!is.na(publisher)) |>
  count(publisher, sort = TRUE)

head(publisher_counts)
# A tibble: 6 × 2
  publisher                  n
  <chr>                  <int>
1 Audible Studios           11
2 Morrow                     9
3 Scholastic                 9
4 Grand Central              8
5 Little, Brown              8
6 Simon & Schuster Audio     8
publisher_counts |>
  slice_head(n = 20) |>
  mutate(publisher = fct_reorder(publisher, n)) |>
  ggplot(aes(n, publisher)) +
  geom_col(fill = "#234312") 

  labs(
    title = "Publishers with the most books on NYT best-seller lists",
    subtitle = "Top books from each list in the weekly overview",
    x = "Number of list appearances",
    y = NULL
  ) +
  theme_minimal()
NULL

Which books climbed the most compared to last week?

books |>
  filter(!new_this_week) |>
  mutate(rank_change = rank_last_week - rank) |>
  arrange(desc(rank_change)) |>
  select(title, author, list_name, rank_change) |>
  slice_head(n = 10)
# A tibble: 10 × 4
   title                                            author list_name rank_change
   <chr>                                            <chr>  <chr>           <int>
 1 Creepy Pair Of Underwear                         Aaron… Children…           4
 2 Wings Of Reverie                                 Anna … Young Ad…           4
 3 Harry Potter And The Sorcerer’s Stone (Full-Cas… J.K. … Audio Ch…           4
 4 Harry Potter And The Chamber Of Secrets (Full-C… J.K. … Audio Ch…           4
 5 Red Rising                                       Pierc… Paperbac…           3
 6 The Divorce                                      Freid… Paperbac…           3
 7 Creepy Carrots!                                  Aaron… Children…           3
 8 The Wild Robot                                   Peter… Children…           3
 9 The Let Them Theory                              Mel R… Advice, …           2
10 How To Catch A Monster                           Adam … Children…           2

How long do best sellers usually last?

books |>
  distinct(isbn13, .keep_all = TRUE) |>
  ggplot(aes(weeks_on_list)) +
  geom_histogram(binwidth = 1, fill = "#785945", boundary = 0) +
  coord_cartesian(xlim = c(0, 50)) +
  labs(
    title = "Weeks on the best-seller list",
    x = "Weeks on list",
    y = "Number of books"
  ) +
  theme_minimal()

Conclusion

In one call of the Books API overview endpoint, we can get an insight into the world of best-sellers. The publishers’ list demonstrates that the top-selling book lists are dominated by just a few big publishers, and the multi-lists table reveals which books appear in several lists at once. Since this is just one snapshot per week, the results will only reflect this specific week; however, with different published_date parameter values, it could be used to compare results over time (one API call per 5 minutes).