Week 9: NYT Article Search API

Question

How has New York Times coverage of artificial intelligence changed from 2016 to 2026, and which sections cover it most?

In our Project 3 analysis, LLM / Gen AI skills appeared in about 1 in 4 data job postings. This document checks whether the news shows the same rise.

The API

API	Article Search API (NYT Developer Network)
Endpoint	`https://api.nytimes.com/svc/search/v2/articlesearch.json`
What it returns	Article metadata from 1851 to today: headline, abstract, publication date, section, byline, keywords
Results per page	10 articles
Rate limit	5 requests per minute, 500 per day

Show code

library(tidyverse)
library(httr2)

Authentication

The key is stored in my .Renviron file as NYT_API_KEY (set with usethis::edit_r_environ()), so it never appears in this document. The key is sent as a URL parameter, so request objects and URLs are never printed.

Show code

api_key <- Sys.getenv("NYT_API_KEY")
if (api_key == "") stop("NYT_API_KEY is not set. Add it to .Renviron and restart R.")

Request Function

Every request goes through one function. NYT allows about one request every 12 seconds, so the function waits 12 seconds before each call. If NYT still answers “429 Too Many Requests”, it waits 60 seconds and tries again (up to 5 tries).

Show code

nyt_search <- function(...) {
  Sys.sleep(12)                                              # NYT asks for ~12 seconds between calls
  request("https://api.nytimes.com/svc/search/v2/articlesearch.json") |>
    req_url_query(..., `api-key` = api_key) |>
    req_retry(max_tries = 5, backoff = \(attempt) 60) |>     # on a 429, wait 60s and try again
    req_perform() |>
    resp_body_json()
}

Pull 1: AI Articles per Year

Parameter	Value
`q`	`"artificial intelligence"` (exact phrase)
`begin_date` / `end_date`	January 1 to December 31 of each year, 2016 to 2026

Each response includes response$metadata$hits, the total number of matching articles, so only the first page is needed for each year (11 requests).

Show code

count_hits <- function(q = NULL, begin, end) {
  nyt_search(q = q, begin_date = begin, end_date = end)$response$metadata$hits
}

ai_yearly <- tibble(year = 2016:2026) |>
  mutate(ai_articles = map_int(year, \(y) count_hits(q = '"artificial intelligence"',
                                                     begin = paste0(y, "0101"),
                                                     end   = paste0(y, "1231"))))
ai_yearly

year	ai_articles
2016	325
2017	462
2018	639
2019	665
2020	417
2021	417
2022	425
2023	1529
2024	1739
2025	2371
2026	3266

Pull 2: All Articles per Year

To compare years fairly, AI articles are divided by everything the Times published that year. A request with no search term returns every article, but the API caps hits at 10,000, and a full year of the Times is far above that (every year came back as exactly 10,000). One month is under the cap (January 2025: 4,612), so totals are counted month by month and added up.

That is 130 requests, about 30 minutes at NYT’s rate limit, so this code was run once (November 1, 2026) and the result was saved to GitHub. The chunk below shows the code but is not re-run when the document is rendered.

Show code

monthly <- expand_grid(year = 2016:2026, month = 1:12) |>
  mutate(start = make_date(year, month, 1),
         end   = ceiling_date(start, "month") - days(1)) |>
  filter(start <= today()) |>                                # skip future months
  mutate(all_articles = map2_int(
    start, end,
    \(s, e) count_hits(begin = format(s, "%Y%m%d"), end = format(e, "%Y%m%d")),
    .progress = TRUE
  ))

write_csv(monthly, "nyt_monthly_totals.csv")

Show code

monthly <- read_csv(
  "https://raw.githubusercontent.com/dillonleeper/DATA-607/main/assignments/week09/nyt_monthly_totals.csv",
  show_col_types = FALSE
)

all_yearly <- monthly |>
  group_by(year) |>
  summarise(all_articles = sum(all_articles))

AI Coverage Over Time

Show code

yearly <- ai_yearly |>
  left_join(all_yearly, by = "year") |>
  mutate(share_pct = ai_articles / all_articles * 100)

yearly |>
  mutate(share_pct    = round(share_pct, 2),
         across(c(ai_articles, all_articles), \(x) format(x, big.mark = ","))) |>
  rename(Year = year, `AI articles` = ai_articles,
         `All articles` = all_articles, `AI share (%)` = share_pct)

Year	AI articles	All articles	AI share (%)
2016	325	68,897	0.47
2017	462	59,543	0.78
2018	639	57,565	1.11
2019	665	54,212	1.23
2020	417	62,462	0.67
2021	417	64,100	0.65
2022	425	58,683	0.72
2023	1,529	53,686	2.85
2024	1,739	54,552	3.19
2025	2,371	50,417	4.70
2026	3,266	42,999	7.60

Show code

ggplot(yearly, aes(x = year, y = share_pct)) +
  geom_vline(xintercept = 2022.9, linetype = "dashed", colour = "grey60") +
  annotate("text", x = 2022.8, y = max(yearly$share_pct) * 0.9,
           label = "ChatGPT released\n(Nov 2022)", hjust = 1, size = 3.2, colour = "grey40") +
  geom_line(colour = "#2c5f8a", linewidth = 1) +
  geom_point(colour = "#2c5f8a", size = 2.5) +
  geom_label(aes(label = paste0(round(share_pct, 1), "%")), vjust = -0.6, size = 3.2,
             label.size = 0, fill = "white") +
  scale_x_continuous(breaks = 2016:2026, labels = \(x) if_else(x == 2026, "2026*", as.character(x))) +
  scale_y_continuous(labels = \(x) paste0(x, "%"), expand = expansion(mult = c(0.05, 0.15))) +
  labs(x = NULL, y = "Share of all articles",
       title = "NYT articles mentioning \"artificial intelligence\"",
       caption = "*2026 runs through October") +
  theme_minimal(base_size = 12) +
  theme(panel.grid.minor = element_blank())

Figure 1: Share of all New York Times articles that mention “artificial intelligence”, by year

Note: AI coverage grew slowly from 0.5% of articles in 2016 to 1.2% in 2019, then fell back to about 0.7% in 2020 to 2022, possibly because other news (the pandemic, the 2020 election) took up more space. After ChatGPT’s release in November 2022, the share grew 3.9 times in one year, to 2.8% in 2023, and reached 7.6% so far in 2026, about 1 in 13 articles. Meanwhile the Times published fewer articles overall, so raw counts alone would understate how much bigger a share AI has become.

Pull 3: Which Sections Cover AI

Parameter	Value
`q`	`"artificial intelligence"`
`begin_date` / `end_date`	`20260101` to `20261231`
`sort`	`newest`
`page`	`0` to `9` (10 articles per page, 100 articles)

Show code

get_page <- function(page) {
  nyt_search(q = '"artificial intelligence"',
             begin_date = "20260101", end_date = "20261231",
             sort = "newest", page = page)$response$docs
}

docs <- map(0:9, get_page) |> list_flatten()
length(docs)

[1] 100

From Nested JSON to a Tidy Data Frame

Each article is a nested list. headline and byline are lists with several fields, and keywords is a list of lists (one per keyword, each with a name, value, and rank). Each article becomes one row:

Show code

articles <- map(docs, \(d) tibble(
  id       = d$`_id`,
  pub_date = as_date(ymd_hms(d$pub_date)),                      # "2025-10-31T16:45:06Z" -> date
  headline = d$headline$main,                                   # main headline only
  section  = d$section_name,
  byline   = d$byline$original,
  keywords = paste(map_chr(d$keywords, "value"), collapse = ", ") # all keywords in one column
)) |>
  list_rbind() |>
  mutate(section = if_else(is.na(section) | section == "", "Unknown", section)) |>
  distinct(id, .keep_all = TRUE)                                # in case an article appears on two pages

glimpse(articles)

Rows: 100
Columns: 6
$ id       <chr> "nyt://article/0b3054b8-4404-5cad-b6f8-3e99d899e3e2", "nyt://…
$ pub_date <date> 2026-11-01, 2026-11-01, 2026-11-01, 2026-11-01, 2026-11-01, …
$ headline <chr> "Binance Reports $3.64 Billion in Outflows as It Resumes Norm…
$ section  <chr> "Business", "U.S.", "Opinion", "Opinion", "U.S.", "U.S.", "U.…
$ byline   <chr> "By Kate Conger", "", "", "By Toni Sims", "By Matt Zdun and T…
$ keywords <chr> "Virtual Currency, Robberies and Thefts, Cyberattacks and Hac…

Show code

articles |>
  select(pub_date, section, headline) |>
  head(10)

pub_date	section	headline
2026-11-01	Business	Binance Reports $3.64 Billion in Outflows as It Resumes Normal Transactions
2026-11-01	U.S.	Does the Needle use A.I.?
2026-11-01	Opinion	Using Denver as a Model to Help the Homeless
2026-11-01	Opinion	Could A.I. Be Conscious? It’s Definitely Possible.
2026-11-01	U.S.	Meet the Megadonors Pumping $2 Billion Into the 2026 Midterms
2026-11-01	U.S.	Culture War Tensions Roil Beverly Hills School Board Race
2026-11-01	U.S.	Threats, Gaffes and Men’s Posteriors: Trump’s Talk Gets Wilder
2026-11-01	Arts	Museums Turn to A.I. to Create Personalized Tour Experiences
2026-10-31	Technology	How Binance’s $6.3 Billion Hack Went Down
2026-10-31	Business	The New Hot Job in Law Doesn’t Require a J.D.

Show code

section_counts <- articles |> count(section, sort = TRUE)

section_counts |>
  slice_max(n, n = 8, with_ties = FALSE) |>
  ggplot(aes(x = n, y = fct_reorder(section, n))) +
  geom_col(fill = "#2c5f8a") +
  geom_text(aes(label = n), hjust = -0.3, size = 3.2) +
  scale_x_continuous(expand = expansion(mult = c(0, 0.1))) +
  labs(x = "Articles", y = NULL,
       title = "Where AI coverage appears",
       subtitle = paste0(nrow(articles), " most recent 2026 articles, ",
                         format(min(articles$pub_date), "%b %d"), " to ",
                         format(max(articles$pub_date), "%b %d"))) +
  theme_minimal(base_size = 12) +
  theme(panel.grid.major.y = element_blank())

Figure 2: Sections of the 100 most recent 2026 NYT articles that mention “artificial intelligence”

Note: Business has the most AI articles (30 of 100), ahead of Technology (17). AI is covered mostly as a business story, not only as technology news.

Data Cleaning Decisions

The 10,000 hit cap. A search with no keyword reports at most 10,000 hits, so yearly totals were all “10,000”. Totals were counted month by month instead.
Phrase search is a lower bound. The query matches the exact phrase “artificial intelligence”. Articles that only say “A.I.” (the Times’ style in headlines) or only name a product like ChatGPT are not counted. The rule is the same for every year, so the trend is still a fair comparison.
Shares, not raw counts. The Times published fewer articles over time (about 69,000 in 2016 vs. 50,000 in 2025), and 2026 is a partial year, so years are compared by share of all articles.
Nested fields. Only headline$main and byline$original are kept. keywords (a list of lists) is collapsed into one comma-separated column. multimedia and other unused fields are dropped.
Dates. pub_date is text with a time and time zone, converted to a date.
Missing values. Articles with no section are labeled “Unknown” instead of dropped, so section counts add up to the total.
Duplicates. Articles are de-duplicated by id in case one appears on two pages.
Section sample. The section breakdown uses the 100 newest 2026 articles, which covers only the most recent weeks, not the whole year.
Live data. AI counts and articles are pulled when the document is rendered, so the newest numbers can change slightly from render to render. The monthly totals are from November 1, 2026.

AI Disclosure

Claude (Anthropic) helped write and debug the code and organize the text. I ran each request myself, checked the results, and found the 10,000 cap in the output.

Anthropic. (2026). Claude [Large language model]. https://claude.ai/. Accessed November 1, 2026.