Week 9 - NY Times API

Author

Affiliation

Muhammad Ali

CUNY School of Professional Studies

Published

November 2, 2026

Using the New York Times API

Approach

Background & Information

In this assignment, the focus will be to accessing data from the New York Times via an API.

Accessing data with an API Key is the ultimate goal of web scraping, as it is a way to get data in real time from sources such as The New York Times, a census of NYC, sports data such as Formula 1, Premier League, the Super Bowl and so on rather than an old “snapshot” from 2 years ago.

But for this, we will be using the NY Times as mentioned and we will be accessing an endpoint to answer a question about the data that comes to mind.

Terminology

API: Application Programming Interface. Set of rules that allows two softwares to communicate, follow rules, and to share data with each other. In this case, requesting data from a website.
API Key: A string of text that acts as a secure passcode for an application to ask a server for the data it is requesting. This is not to be shared with anyone!
Endpoint: A specific digital location or web address where a software application can access the digital resources it needs. Like a library book located in a certain section and we the user want to access and use that book.
Request: The user asking the server if they can extract information without changing anything on the server end.
Response: An answer in the form of status codes (200, 400, etc.) from the server to the user. For this definition, getting a response code of 200 means we are permitted to send a request to the server.

Data Source

New York Times Developer Network - Where to request for an API Key. First create an account!
New York Times APIs - Available Endpoints
For this, I will be using the Books API endpoint

Question

With the Books API Endpoint, I will be conducting analysis to ask the following questions:

What is the longest that a book has been on the best sellers list?
What genre stays on the list the longest?
Who are the most publishers in this dataset?
Which authors recur the most?

Plan Outline

Create an account on the NY Times and request an API Key on the Books API
Create requests to the server to extract data
Parse from JSON format to use in R tibble
Conduct analysis

Codebase

Install Packages

Show code

# Install packages once if they are missing.
# install.packages(c("httr2", "tidyverse", "usethis"))

# Received an error with my Renviron file for the API key, so this will allow me to use it.
usethis::edit_r_environ()

Load Packages

Show code

library(tidyverse)
library(httr2)
library(usethis)

API Request

Request Parameters

Base URL: https://api.nytimes.com/svc/books/v3
Endpoint: /lists/overview.json
API Key: nyt_api_key Read from using Sys.getenv() Saved using .Renviron file

Show code

api_key <- Sys.getenv("nyt_api_key")
api_url <- "https://api.nytimes.com/svc/books/v3/lists/overview.json"

req <- request(api_url) |>
  req_headers(
    Accept = "application/json",
    `User-Agent` = "DATA-607-week9-homework"
  ) |>
  req_url_query(`api-key` = api_key)

req

<httr2_request>
GET https://api.nytimes.com/svc/books/v3/lists/overview.json?api-key=h95TzFCpRH1nBrNItWunT8wOLZeOv47G1cZvMt5PbC9jUzwK
Headers:
* Accept    : "application/json"
* User-Agent: "DATA-607-week9-homework"
Body: empty

Show code

req_dry_run(req)

GET /svc/books/v3/lists/overview.json HTTP/1.1
accept: application/json
accept-encoding: deflate, gzip
host: api.nytimes.com
user-agent: DATA-607-week9-homework

Performing & Confirming The Request

Show code

resp <- req |>
  req_perform()

# Confirm that the server accepted the request.
resp_status(resp)

[1] 200

Show code

resp_status_desc(resp)

[1] "OK"

Show code

resp_content_type(resp)

[1] "application/json"

Read the JSON body

As we have loaded in the API data, we need to investigate where the information we want is located and we located it in lists and in books.

Show code

books_json <- resp |>
  resp_body_json(simplifyVector = TRUE)

names(books_json)

[1] "status"      "copyright"   "num_results" "results"

Show code

names(books_json$results) # "lists" is most likely where the book information is located.

[1] "previous_published_date"    "published_date"            
[3] "next_published_date"        "published_date_description"
[5] "bestsellers_date"           "lists"                     
[7] "monthly_uri"                "weekly_uri"

Show code

names(books_json$results$lists) # books is where the books we want are stored, and it is in a dataframe

[1] "display_name"        "list_name"           "list_name_encoded"  
[4] "normal_list_ends_at" "updated"             "list_id"            
[7] "uri"                 "books"               "corrections"

Parse JSON to Tibble

We are working with a JSON file, so what we want to do is unnest the json to use in a tabular format adding things from the list dataframe as well as the books dataframe.

Show code

list_column <- books_json$results$lists

# Flatten the list and books
flat_books <- tibble(
  list_name = list_column$list_name,
  books_df = list_column$books
) |>
  unnest(books_df)

head(flat_books)

list_name	amazon_product_url	author	book_image	book_image_height	book_image_width	book_uri	contributor	created_date	description	primary_isbn13	publisher	rank	rank_last_week	title	updated_date	weeks_on_list	isbns	buy_links
Combined Print & E-Book Fiction	https://www.amazon.com/dp/B0GHZJ3BXR?tag=thenewyorktim-20	Louise Penny	https://static01.nyt.com/bestsellers/images/9781250412584.jpg	500	329	nyt://book/18f8e6ad-c7f8-534c-ace5-b701d51837da	by Louise Penny	2026-10-28T18:01:20.498Z	The 21st book in the Chief Inspector Gamache series. Mysteries from the past confound Gamache and his team’s murder investigation.	9781250412584	Minotaur	1	0	MISS WOLCOTT’S GHOST	2026-10-28T18:01:20.498Z	1	, 9781250412584	Amazon , Apple Books , Barnes & Noble , Books-A-Million , Bookshop.org , https://www.amazon.com/dp/B0GHZJ3BXR?tag=thenewyorktim-20 , https://goto.applebooks.apple/9781250412584?at=10lIEQ , https://barnesandnoble.sjv.io/c/197432/1961179/22009?prodsku=9781250412584&u=https%3A%2F%2Fwww.barnesandnoble.com%2Fw%2Fmiss-wolcotts-ghost-louise-penny%2F1149344440%3Fean%3D9781250412584&intsrc=APIG_15890, https://www.anrdoezrs.net/click-7990613-35140?url=https%3A%2F%2Fwww.booksamillion.com%2Fp%2FMiss-Wolcotts-Ghost-Chief-Inspector%2FLouise-Penny%2F9781250412607 , https://bookshop.org/a/3546/9781250412584
Combined Print & E-Book Fiction	https://www.amazon.com/dp/B0GJJJJWPF?tag=thenewyorktim-20	Lee Child and Andrew Child	https://static01.nyt.com/bestsellers/images/9798217375165.jpg	500	329	nyt://book/5f2b4817-27ee-519f-b5ab-c5caf3b7ea80	by Lee Child and Andrew Child	2026-10-28T18:01:20.671Z	The 31st book in the Jack Reacher series. Gen. Susan Turner needs Reacher’s help to solve the murder of a team member on covert duty.	9798217375165	Bantam	2	0	CHAIN REACTION	2026-10-28T18:01:20.671Z	1	, 9798217375165	Amazon , Apple Books , Barnes & Noble , Books-A-Million , Bookshop.org , https://www.amazon.com/dp/B0GJJJJWPF?tag=thenewyorktim-20 , https://goto.applebooks.apple/9798217375165?at=10lIEQ , https://barnesandnoble.sjv.io/c/197432/1961179/22009?prodsku=9798217375165&u=https%3A%2F%2Fwww.barnesandnoble.com%2Fw%2Fchain-reaction-lee-child%2F1150266107%3Fean%3D9798217375165&intsrc=APIG_15890, https://www.anrdoezrs.net/click-7990613-35140?url=https%3A%2F%2Fwww.booksamillion.com%2Fp%2FChain-Reaction-Reacher-Novel%2FLee-Child%2F9798217375158 , https://bookshop.org/a/3546/9798217375165
Combined Print & E-Book Fiction	https://www.amazon.com/dp/168281808X?tag=thenewyorktim-20	Rebecca Yarros	https://static01.nyt.com/bestsellers/images/9781682818084.jpg	400	312	nyt://book/ce1ec3e9-1213-5b7d-922f-c8859b1b06fe	by Rebecca Yarros	2026-10-07T18:21:49.532Z	Thirteen short stories, set in the Empyrean world, in which cadets compete to earn the bond of a dragon.	9781682818084	Red Tower	3	1	THRESHING DAY	2026-10-07T18:21:49.532Z	4	, 9781682818084	Amazon , Apple Books , Barnes and Noble , Books-A-Million , Bookshop.org , https://www.amazon.com/dp/168281808X?tag=thenewyorktim-20 , https://goto.applebooks.apple/9781682818527?at=10lIEQ , https://barnesandnoble.sjv.io/c/197432/1961179/22009?prodsku=9781682818084&u=https%3A%2F%2Fwww.barnesandnoble.com%2Fw%2Fthreshing-day-rebecca-yarros%2F1150799611%3Fean%3D9781682818084&intsrc=APIG_15890, https://www.anrdoezrs.net/click-7990613-35140?url=https%3A%2F%2Fwww.booksamillion.com%2Fp%2FThreshing-Day-Wing-Claw-Collection%2FRebecca-Yarros%2F9781682818084 , https://bookshop.org/a/3546/9781682818084
Combined Print & E-Book Fiction	https://www.amazon.com/dp/0316569747?tag=thenewyorktim-20	James Patterson	https://static01.nyt.com/bestsellers/images/9780316569743.jpg	500	331	nyt://book/32d75190-4c76-509d-a0bf-e01c1dd4e264	by James Patterson	2026-10-28T17:27:56.957Z	The 36th book in the Alex Cross series. A family crisis and the D.C. underworld put the psychologist in a tight spot.	9780316569743	Little, Brown	4	0	THE FAMILY CROSS	2026-10-28T17:27:56.957Z	1	, 9780316569743	Amazon , Apple Books , Barnes & Noble , Books-A-Million , Bookshop.org , https://www.amazon.com/dp/0316569747?tag=thenewyorktim-20 , https://goto.applebooks.apple/9780316569743?at=10lIEQ , https://barnesandnoble.sjv.io/c/197432/1961179/22009?prodsku=9780316569743&u=https%3A%2F%2Fwww.barnesandnoble.com%2Fw%2Fthe-family-cross-james-patterson%2F1149543134%3Fean%3D9780316569743&intsrc=APIG_15890, https://www.anrdoezrs.net/click-7990613-35140?url=https%3A%2F%2Fwww.booksamillion.com%2Fp%2FFamily-Cross%2FJames-Patterson%2F9780316569743 , https://bookshop.org/a/3546/9780316569743
Combined Print & E-Book Fiction	https://www.amazon.com/dp/1668236516?tag=thenewyorktim-20	Allen Levi	https://static01.nyt.com/bestsellers/images/9781668236512.jpg	0	0	nyt://book/f3e1f4c8-34a3-54a3-8305-277666e21b4b	by Allen Levi	2026-03-04T23:40:20.88Z	A man travels to a small Southern town, where he purchases pencil drawings of local residents and exchanges them for stories.	9781668236512	Atria	5	4	THEO OF GOLDEN	2026-03-04T23:40:20.88Z	47	, 9781668236512	Amazon , Apple Books , Barnes & Noble , Books-A-Million , Bookshop.org , https://www.amazon.com/dp/1668236516?tag=thenewyorktim-20 , https://goto.applebooks.apple/9781668236512?at=10lIEQ , https://barnesandnoble.sjv.io/c/197432/1961179/22009?prodsku=9781668236512&u=https%3A%2F%2Fwww.barnesandnoble.com%2Fw%2Ftheo-of-golden-allen-levi%2F1143923011%3Fean%3D9781668236512&intsrc=APIG_15890, https://www.anrdoezrs.net/click-7990613-35140?url=https%3A%2F%2Fwww.booksamillion.com%2Fp%2FTheo-Golden%2FAllen-Levi%2F9781668236512 , https://bookshop.org/a/3546/9781668236512
Combined Print & E-Book Fiction	https://www.amazon.com/dp/1538780046?tag=thenewyorktim-20	Abby Jimenez	https://static01.nyt.com/bestsellers/images/9781538780046.jpg	400	312	nyt://book/4c5deee0-e7f6-54c2-808d-d041c067cb58	by Abby Jimenez	2026-10-28T18:20:48.055Z	A collection of three holiday novellas: “Sweater Weather,” “Stealing Christmas” and “Marry Little Christmas.”	9781538780046	Forever	6	0	A MARRIED LITTLE CHRISTMAS	2026-10-28T18:20:48.055Z	1	, 9781538780046	Amazon , Apple Books , Barnes and Noble , Books-A-Million , Bookshop.org , https://www.amazon.com/dp/1538780046?tag=thenewyorktim-20 , https://goto.applebooks.apple/9781538786208?at=10lIEQ , https://barnesandnoble.sjv.io/c/197432/1961179/22009?prodsku=9781538780046&u=https%3A%2F%2Fwww.barnesandnoble.com%2Fw%2Fa-married-little-christmas-abby-jimenez%2F1149363322%3Fean%3D9781538780046&intsrc=APIG_15890, https://www.anrdoezrs.net/click-7990613-35140?url=https%3A%2F%2Fwww.booksamillion.com%2Fp%2FMarried-Little-Christmas-Deluxe%2FAbby-Jimenez%2F9781538780046 , https://bookshop.org/a/3546/9781538780046

Show code

glimpse(flat_books)

Rows: 240
Columns: 29
$ list_name            <chr> "Combined Print & E-Book Fiction", "Combined Prin…
$ age_group            <chr> "", "", "", "", "", "", "", "", "", "", "", "", "…
$ amazon_product_url   <chr> "https://www.amazon.com/dp/B0GHZJ3BXR?tag=thenewy…
$ article_chapter_link <chr> "", "", "", "", "", "", "", "", "", "", "", "", "…
$ asterisk             <int> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0…
$ author               <chr> "Louise Penny", "Lee Child and Andrew Child", "Re…
$ book_image           <chr> "https://static01.nyt.com/bestsellers/images/9781…
$ book_image_height    <int> 500, 500, 400, 500, 0, 400, 500, 500, 400, 400, 4…
$ book_image_width     <int> 329, 329, 312, 331, 0, 312, 329, 394, 312, 312, 3…
$ book_review_link     <chr> "", "", "", "", "", "", "", "", "", "", "", "", "…
$ book_uri             <chr> "nyt://book/18f8e6ad-c7f8-534c-ace5-b701d51837da"…
$ contributor          <chr> "by Louise Penny", "by Lee Child and Andrew Child…
$ contributor_note     <chr> "", "", "", "", "", "", "", "", "", "", "", "", "…
$ created_date         <chr> "2026-10-28T18:01:20.498Z", "2026-10-28T18:01:20.…
$ dagger               <int> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0…
$ description          <chr> "The 21st book in the Chief Inspector Gamache ser…
$ first_chapter_link   <chr> "", "", "", "", "", "", "", "", "", "", "", "", "…
$ price                <chr> "0.00", "0.00", "0.00", "0.00", "0.00", "0.00", "…
$ primary_isbn10       <chr> "", "", "", "", "", "", "", "", "", "", "", "", "…
$ primary_isbn13       <chr> "9781250412584", "9798217375165", "9781682818084"…
$ publisher            <chr> "Minotaur", "Bantam", "Red Tower", "Little, Brown…
$ rank                 <int> 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15…
$ rank_last_week       <int> 0, 0, 1, 0, 4, 0, 3, 0, 5, 2, 0, 7, 9, 8, 10, 1, …
$ sunday_review_link   <chr> "", "", "", "", "", "", "", "", "", "", "", "", "…
$ title                <chr> "MISS WOLCOTT'S GHOST", "CHAIN REACTION", "THRESH…
$ updated_date         <chr> "2026-10-28T18:01:20.498Z", "2026-10-28T18:01:20.…
$ weeks_on_list        <int> 1, 1, 4, 1, 47, 1, 4, 1, 25, 3, 1, 42, 3, 106, 6,…
$ isbns                <list> [<data.frame[1 x 2]>], [<data.frame[1 x 2]>], [<…
$ buy_links            <list> [<data.frame[5 x 2]>], [<data.frame[5 x 2]>], [<…

Creating Table

We have a large flattened table, but we want only the relevant information regarding books such as the title, author, rank on the list, and such, which are selected below and explained with the provided data dictionary.

Show code

books_table <- flat_books |>
  select(
    book_title = title,
    author,
    rank,
    book_genre = list_name,
    weeks_on_list,
    publisher
    )

head(books_table)

book_title	author	rank	book_genre	weeks_on_list	publisher
MISS WOLCOTT’S GHOST	Louise Penny	1	Combined Print & E-Book Fiction	1	Minotaur
CHAIN REACTION	Lee Child and Andrew Child	2	Combined Print & E-Book Fiction	1	Bantam
THRESHING DAY	Rebecca Yarros	3	Combined Print & E-Book Fiction	4	Red Tower
THE FAMILY CROSS	James Patterson	4	Combined Print & E-Book Fiction	1	Little, Brown
THEO OF GOLDEN	Allen Levi	5	Combined Print & E-Book Fiction	47	Atria
A MARRIED LITTLE CHRISTMAS	Abby Jimenez	6	Combined Print & E-Book Fiction	1	Forever

Data Dictionary
Column Name	Definition
`book_title`	Name of the book
`author`	The author(s) of the book
`rank`	Book’s current position on the list for this week
`book_genre`	The genre(s) that the book is listed as
`weeks_on_list`	How long the book has been on the list
`publisher`	Who published the book

Reproducibility Metadata

This part is crucial as it lists the time of retrieving the source data and url with values.

And to be safe so we don’t abuse the requests on the server, I will save it in a csv file so I can do offline work on this data.

Show code

retrieved_at <- format(Sys.time(), tz = "UTC", usetz = TRUE)

matches_snapshot <- books_table |>
  mutate(
    retrieved_at = retrieved_at,
    request_url = api_url,
    .before = 1
  )

head(matches_snapshot,3)

retrieved_at	request_url	book_title	author	rank	book_genre	weeks_on_list	publisher
2026-11-02 09:58:27 UTC	https://api.nytimes.com/svc/books/v3/lists/overview.json	MISS WOLCOTT’S GHOST	Louise Penny	1	Combined Print & E-Book Fiction	1	Minotaur
2026-11-02 09:58:27 UTC	https://api.nytimes.com/svc/books/v3/lists/overview.json	CHAIN REACTION	Lee Child and Andrew Child	2	Combined Print & E-Book Fiction	1	Bantam
2026-11-02 09:58:27 UTC	https://api.nytimes.com/svc/books/v3/lists/overview.json	THRESHING DAY	Rebecca Yarros	3	Combined Print & E-Book Fiction	4	Red Tower

Show code

# Save to CSV
write.csv(matches_snapshot, "books_table.csv", row.names = FALSE)

Analysis

Load Data

For some reason there is a new column created, so I had to clean that column up and rearrange the columns

Show code

data <- read_csv("https://raw.githubusercontent.com/HaiderrX/CUNY-SPS-MSDS/refs/heads/main/DATA607/Week%209%20-%20Working%20With%20Web%20APIs/books_table.csv")

head(data)

retrieved_at	request_url	book_title	author	rank	book_genre	weeks_on_list	publisher
2026-10-31 10:51:46 UTC	https://api.nytimes.com/svc/books/v3/lists/overview.json	MISS WOLCOTT’S GHOST	Louise Penny	1	Combined Print & E-Book Fiction	1	Minotaur
2026-10-31 10:51:46 UTC	https://api.nytimes.com/svc/books/v3/lists/overview.json	CHAIN REACTION	Lee Child and Andrew Child	2	Combined Print & E-Book Fiction	1	Bantam
2026-10-31 10:51:46 UTC	https://api.nytimes.com/svc/books/v3/lists/overview.json	THRESHING DAY	Rebecca Yarros	3	Combined Print & E-Book Fiction	4	Red Tower
2026-10-31 10:51:46 UTC	https://api.nytimes.com/svc/books/v3/lists/overview.json	THE FAMILY CROSS	James Patterson	4	Combined Print & E-Book Fiction	1	Little, Brown
2026-10-31 10:51:46 UTC	https://api.nytimes.com/svc/books/v3/lists/overview.json	THEO OF GOLDEN	Allen Levi	5	Combined Print & E-Book Fiction	47	Atria
2026-10-31 10:51:46 UTC	https://api.nytimes.com/svc/books/v3/lists/overview.json	A MARRIED LITTLE CHRISTMAS	Abby Jimenez	6	Combined Print & E-Book Fiction	1	Forever

Show code

glimpse(data)

Rows: 240
Columns: 8
$ retrieved_at  <chr> "2026-10-31 10:51:46 UTC", "2026-10-31 10:51:46 UTC", "2…
$ request_url   <chr> "https://api.nytimes.com/svc/books/v3/lists/overview.jso…
$ book_title    <chr> "MISS WOLCOTT'S GHOST", "CHAIN REACTION", "THRESHING DAY…
$ author        <chr> "Louise Penny", "Lee Child and Andrew Child", "Rebecca Y…
$ rank          <dbl> 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 1, 2,…
$ book_genre    <chr> "Combined Print & E-Book Fiction", "Combined Print & E-B…
$ weeks_on_list <dbl> 1, 1, 4, 1, 47, 1, 4, 1, 25, 3, 1, 42, 3, 106, 6, 2, 1, …
$ publisher     <chr> "Minotaur", "Bantam", "Red Tower", "Little, Brown", "Atr…

Show code

clean_data <- data |> select(
  book_title,
  author,
  book_genre,
  rank,
  weeks_on_list,
  publisher,
  retrieved_at,
  request_url
)

Show code

head(clean_data)

book_title	author	book_genre	rank	weeks_on_list	publisher	retrieved_at	request_url
MISS WOLCOTT’S GHOST	Louise Penny	Combined Print & E-Book Fiction	1	1	Minotaur	2026-10-31 10:51:46 UTC	https://api.nytimes.com/svc/books/v3/lists/overview.json
CHAIN REACTION	Lee Child and Andrew Child	Combined Print & E-Book Fiction	2	1	Bantam	2026-10-31 10:51:46 UTC	https://api.nytimes.com/svc/books/v3/lists/overview.json
THRESHING DAY	Rebecca Yarros	Combined Print & E-Book Fiction	3	4	Red Tower	2026-10-31 10:51:46 UTC	https://api.nytimes.com/svc/books/v3/lists/overview.json
THE FAMILY CROSS	James Patterson	Combined Print & E-Book Fiction	4	1	Little, Brown	2026-10-31 10:51:46 UTC	https://api.nytimes.com/svc/books/v3/lists/overview.json
THEO OF GOLDEN	Allen Levi	Combined Print & E-Book Fiction	5	47	Atria	2026-10-31 10:51:46 UTC	https://api.nytimes.com/svc/books/v3/lists/overview.json
A MARRIED LITTLE CHRISTMAS	Abby Jimenez	Combined Print & E-Book Fiction	6	1	Forever	2026-10-31 10:51:46 UTC	https://api.nytimes.com/svc/books/v3/lists/overview.json

Show code

glimpse(clean_data)

Rows: 240
Columns: 8
$ book_title    <chr> "MISS WOLCOTT'S GHOST", "CHAIN REACTION", "THRESHING DAY…
$ author        <chr> "Louise Penny", "Lee Child and Andrew Child", "Rebecca Y…
$ book_genre    <chr> "Combined Print & E-Book Fiction", "Combined Print & E-B…
$ rank          <dbl> 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 1, 2,…
$ weeks_on_list <dbl> 1, 1, 4, 1, 47, 1, 4, 1, 25, 3, 1, 42, 3, 106, 6, 2, 1, …
$ publisher     <chr> "Minotaur", "Bantam", "Red Tower", "Little, Brown", "Atr…
$ retrieved_at  <chr> "2026-10-31 10:51:46 UTC", "2026-10-31 10:51:46 UTC", "2…
$ request_url   <chr> "https://api.nytimes.com/svc/books/v3/lists/overview.jso…

Let us ask a first question:

What is longest on a list in this dataset?