Analyzing Flooding Coverage Using the New York Times API

DATA 607 | Data Acquisition and Management

Author

Sarah Abdelrahman

Published

November 1, 2026

1. Research Question and Purpose

Research question: Among the ten most recent articles returned by the New York Times (NYT) Article Search API for the keyword flooding in 2025, which newspaper sections appear most frequently?

Flooding is relevant to geographic information systems (GIS), disaster management, and environmental planning. This assignment demonstrates how to acquire and clean public article metadata using R. It examines news coverage, not the actual frequency, severity, or geographic distribution of floods. To keep the exercise reproducible and focused, it uses the first page of search results (a maximum of ten articles), rather than all articles published during the year.

2. API Selection and Request Parameters

API: New York Times Article Search API, version 2. This endpoint supports article searches and returns JSON records containing headlines, dates, section names, and links.

Endpoint: https://api.nytimes.com/svc/search/v2/articlesearch.json

Parameter Value Reason
q flooding Search term
begin_date 20250101 Start date (January 1, 2025)
end_date 20251231 End date (December 31, 2025)
sort newest Sort most recent first
page 0 First results page, up to ten articles
api-key Read from NYT_API_KEY Authenticate securely; never embed the secret in this file

The API documentation specifies that each page contains at most ten results. The request uses only one page; the findings therefore describe the returned sample and should not be generalized to the entire year’s coverage.

3. Preparation and Secure Authentication

One-time setup in RStudio (do not insert an API key into this .qmd):

  1. Create an application at https://developer.nytimes.com/ and enable Article Search API. If a key has been shared publicly or in a chat, revoke/regenerate it before using it.
  2. In the RStudio Console, install packages if needed: install.packages(c("httr2", "jsonlite", "dplyr", "purrr", "tibble", "knitr"))
  3. In the Console, run file.edit("~/.Renviron") and add the line NYT_API_KEY=YOUR_NEW_PRIVATE_KEY, replacing the placeholder with your actual key. Save the file, restart RStudio, and keep .Renviron private.
  4. Open this .qmd file and select Render. Submit the .qmd document only, as requested.

The code below loads libraries, retrieves the key from the environment, and stops with a clear message if the key is missing. It never prints the key.

library(httr2)
library(jsonlite)
library(dplyr)
library(purrr)
library(tibble)
library(knitr)

api_key <- Sys.getenv("NYT_API_KEY", unset = "")
if (!nzchar(api_key)) {
  stop("NYT_API_KEY is not set. Add it to ~/.Renviron, save, and restart RStudio.")
}

4. Send the Request and Parse JSON

httr2 makes an HTTPS GET request with the parameters above. jsonlite::fromJSON() parses the returned JSON into R objects. Authentication and request errors are reported without displaying the secret key or full authenticated URL.

endpoint <- "https://api.nytimes.com/svc/search/v2/articlesearch.json"

# Construct the query without printing the authenticated URL.
request_obj <- httr2::request(endpoint) |>
  httr2::req_url_query(
    q = "flooding",
    begin_date = "20250101",
    end_date = "20251231",
    sort = "newest",
    page = 0,
    `api-key` = api_key
  )

# Suppress potentially sensitive URLs in any surfaced HTTP errors.
response_obj <- tryCatch(
  httr2::req_perform(request_obj),
  error = function(e) stop(
    "API request failed. Check your internet connection, key, API access, and rate limits.",
    call. = FALSE
  )
)

status_code <- httr2::resp_status(response_obj)
if (status_code != 200L) {
  stop(paste("NYT API HTTP status:", status_code,
             "— verify your key, enabled API, and request quota."), call. = FALSE)
}

json_text <- httr2::resp_body_string(response_obj)
parsed <- tryCatch(
  jsonlite::fromJSON(json_text, simplifyVector = FALSE),
  error = function(e) stop("The server response was not valid JSON.", call. = FALSE)
)

if (!identical(parsed$status, "OK")) {
  stop("The API did not return an OK status.", call. = FALSE)
}

articles <- parsed$response$docs
if (is.null(articles) || length(articles) == 0L) {
  stop("No articles matched these filters. Verify the response or adjust the query.", call. = FALSE)
}

cat("API request successful. Article records returned:", length(articles), "\n")
API request successful. Article records returned: 10

5. Convert Nested Records into a Tidy Data Frame

The API’s JSON contains nested objects, especially headline (whose main field contains the headline). The helper function below safely extracts a single value even when a field is absent, empty, or null. Each article becomes one row; each selected attribute becomes one column. The original missing values remain NA for transparent quality checks.

get_field <- function(x, path) {
  value <- x
  for (part in path) {
    if (is.null(value) || !is.list(value)) return(NA_character_)
    value <- value[[part]]
  }
  if (is.null(value) || length(value) != 1L || is.list(value)) {
    return(NA_character_)
  }
  result <- as.character(value)
  if (is.na(result) || !nzchar(trimws(result))) NA_character_ else result
}

articles_tidy <- purrr::map_dfr(articles, function(article) {
  tibble::tibble(
    article_id = get_field(article, c("_id")),
    headline = get_field(article, c("headline", "main")),
    publication_date = as.Date(substr(get_field(article, c("pub_date")), 1, 10)),
    section = get_field(article, c("section_name")),
    news_desk = get_field(article, c("news_desk")),
    url = get_field(article, c("web_url"))
  )
}) |>
  distinct(article_id, .keep_all = TRUE) |>
  arrange(desc(publication_date))

if (nrow(articles_tidy) == 0L) {
  stop("No usable article rows remain after transformation.", call. = FALSE)
}

knitr::kable(articles_tidy, caption = "Tidy NYT flooding article sample (2025)")
Tidy NYT flooding article sample (2025)
article_id headline publication_date section news_desk url
nyt://article/01c2f94b-11e9-5694-bbab-2bd2ccbe6065 Veto of Water Project Is Trump’s Latest Targeted Hit on Colorado 2025-12-31 U.S. National https://www.nytimes.com/2025/12/31/us/politics/trump-veto-water-project-colorado.html
nyt://article/2811318c-21cf-54df-9537-4a12289c0f6b In Ukraine, a New Arsenal of Killer A.I. Drones Is Being Born 2025-12-31 Magazine Magazine https://www.nytimes.com/2025/12/31/magazine/ukraine-ai-drones-war-russia.html
nyt://article/5afeee42-c5e5-5ad9-8e48-2388f141dfc5 Trump Vetoes 2 Bills, Drawing Accusations of Retaliation 2025-12-31 U.S. Washington https://www.nytimes.com/2025/12/31/us/politics/trump-vetoes-retribution.html
nyt://article/3271e540-d54a-5607-8fa5-bf7d36cac203 The Year in Video 2025-12-31 Briefing NYTNow https://www.nytimes.com/2025/12/31/briefing/the-year-in-video.html
nyt://article/f1019e0b-d4fb-5a8d-a2c7-03d1a559b043 I Counted Trump’s Censorship Attempts. Here’s What I Found. 2025-12-31 Opinion OpEd https://www.nytimes.com/2025/12/31/opinion/trump-first-amendment-dissent.html
nyt://article/58339c43-0fb6-5f53-8935-cd58c56188f6 There Is a Way Out of This Mess 2025-12-31 Opinion OpEd https://www.nytimes.com/2025/12/31/opinion/trump-gilded-age.html
nyt://interactive/131be8fa-25ef-548b-9013-df64def05a2c How Russia’s War Machine Brutalizes and Exploits Its Own Soldiers 2025-12-31 World Foreign https://www.nytimes.com/interactive/2025/12/31/world/europe/russia-military-abuse-soldiers.html
nyt://article/fd577706-3c11-533e-b2be-257a95a9da65 New Rules for the New Year 2025-12-31 Real Estate RealEstate https://www.nytimes.com/2025/12/31/realestate/gardening-rules-new-year-plants.html
nyt://article/370db18d-1866-5776-926d-10fb72c3b020 Brigitte Bardot’s Legacy of Racist Rhetoric 2025-12-31 World Foreign https://www.nytimes.com/2025/12/31/world/europe/brigitte-bardot-racism-far-right.html
nyt://article/66ec65bd-fb10-55c0-b460-ee6afda72da8 Slow Pace of Flood Recovery Stirs Anger in Far-Flung Corner of Indonesia 2025-12-31 World Foreign https://www.nytimes.com/2025/12/31/world/asia/indonesia-flood-recovery-aceh.html

6. Check Empty and Missing Data

Before analysis, the table below counts missing or blank entries in each variable. Dates are considered missing if R could not parse them. This check helps identify incomplete metadata without concealing gaps.

missing_summary <- tibble::tibble(
  variable = names(articles_tidy),
  missing_or_blank = vapply(articles_tidy, function(column) {
    if (is.character(column)) {
      sum(is.na(column) | trimws(column) == "")
    } else {
      sum(is.na(column))
    }
  }, integer(1)),
  total_rows = nrow(articles_tidy)
) |>
  mutate(percent_missing = round(100 * missing_or_blank / total_rows, 1))

knitr::kable(missing_summary,
             caption = "Missing or blank values by column")
Missing or blank values by column
variable missing_or_blank total_rows percent_missing
article_id 0 10 0
headline 0 10 0
publication_date 0 10 0
section 0 10 0
news_desk 0 10 0
url 0 10 0

7. Summarize Articles by Section

For counting purposes only, missing section labels are grouped as "Not specified". This preserves those articles instead of dropping them. The frequency table below answers the research question for the retrieved sample.

section_counts <- articles_tidy |>
  mutate(section = if_else(is.na(section) | trimws(section) == "",
                           "Not specified", section)) |>
  count(section, name = "article_count", sort = TRUE) |>
  mutate(share_percent = round(100 * article_count / sum(article_count), 1))

knitr::kable(section_counts,
             caption = "Section distribution among retrieved flooding articles")
Section distribution among retrieved flooding articles
section article_count share_percent
World 3 30
Opinion 2 20
U.S. 2 20
Briefing 1 10
Magazine 1 10
Real Estate 1 10

Among the 10 unique records retained, the most frequent section is World, with 3 article(s), representing 30% of this sample. These counts represent only the articles returned by this specific API request, not the total number of NYT articles on flooding in 2025.

8. Data-Cleaning Decisions and Limitations

  • Nested JSON: Extracted headline.main from the nested headline object and selected consistently named metadata fields.
  • Missing/empty values: Converted absent, null, or blank scalar fields to NA; counted missing values explicitly before replacing any labels.
  • Dates: Converted the first ten characters of the ISO-style publication timestamp to an R Date; unparseable dates remain NA.
  • Duplicates: Used the API article identifier to remove repeated records, if present.
  • Section names: Assigned "Not specified" only in the aggregated frequency table; the tidy source table preserves missing sections as NA.
  • Scope: An API search can retrieve articles that mention flooding without describing a distinct flood event. The first page is a small, selected sample sorted by recency, and section counts do not measure flood incidence or media coverage overall. Results may change if the API index changes.

9. Conclusion

This exercise demonstrates secure API authentication, an HTTP request, JSON parsing, conversion of nested records into a tidy R data frame, missing-data validation, and a simple summary. The analysis illustrates how reproducible data-acquisition workflows can support research on environmentally relevant topics, including subjects relevant to GIS.

Reference

The New York Times. Article Search API (v2): API specification. https://github.com/nytimes/public_api_specs/blob/master/article_search/article_search_v2.yaml