As data scientists, API’s are very valuable to efficiently share data stored across and accessed through different spaces. They also make it very easy to edit or recieve information on projects hosted elsewhere. For our purposes, the NYT Developers APIs are useful to start exploring how to search for data stored by others on their websites.
To get started we first must load in a select group of packages
library(httr)## Warning: package 'httr' was built under R version 4.4.3library(tidyverse)## ── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ──
## ✔ dplyr 1.1.4 ✔ readr 2.1.5
## ✔ forcats 1.0.0 ✔ stringr 1.5.1
## ✔ ggplot2 3.5.1 ✔ tibble 3.2.1
## ✔ lubridate 1.9.4 ✔ tidyr 1.3.1
## ✔ purrr 1.0.2
## ── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
## ✖ dplyr::filter() masks stats::filter()
## ✖ dplyr::lag() masks stats::lag()
## ℹ Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errorslibrary(jsonlite)## Warning: package 'jsonlite' was built under R version 4.4.3##
## Attaching package: 'jsonlite'
##
## The following object is masked from 'package:purrr':
##
## flattenlibrary(lubridate)
library(stringr)First we must connect to the NYT best sellers API. In order to do this we must use a personalized API key. I have stored my API key in a .csv file saved on github, and can read it in from there.
api_key <- read.csv("https://raw.githubusercontent.com/scrummett/DATA607/refs/heads/main/NYT-API-key.csv")
my_key <- api_key$API.Key[1]From here we can set up the connection with the URL provided by the NYT website.
url <- paste0("https://api.nytimes.com/svc/books/v3/reviews.json?author=Haruki%20Murakami&api-key=", my_key)
nyt_connect <- GET(url)
nyt_connect## Response [https://api.nytimes.com/svc/books/v3/reviews.json?author=Haruki%20Murakami&api-key=hQ9rPjwW70HDichGUxarpZv4yyN1WQSs]
## Date: 2025-03-30 21:43
## Status: 200
## Content-Type: application/json; charset=UTF-8
## Size: 9.04 kBWe can look at “Status” to see “200”, confirming that we are connected to the API. From here we can begin to explore the JSON file.
We have access to the raw data that includes reviews for all of Haruki Murakami’s books from the NYT, however it is not in a very readable form for us yet. We must convert it to a JSON file (content), and then to a R dataframe (fromJSON).
nyt_books <- fromJSON(content(nyt_connect, "text", encoding = "UTF-8"))
str(nyt_books)## List of 4
## $ status : chr "OK"
## $ copyright : chr "Copyright (c) 2025 The New York Times Company. All Rights Reserved."
## $ num_results: int 20
## $ results :'data.frame': 20 obs. of 9 variables:
## ..$ url : chr [1:20] "http://www.nytimes.com/2011/11/10/books/1q84-by-haruki-murakami-review.html" "http://www.nytimes.com/1999/02/17/books/books-of-the-times-an-obsessive-attraction-that-cripples-two-lives.html" "http://www.nytimes.com/1997/11/02/books/east-meets-west.html" "http://www.nytimes.com/1993/03/28/books/as-japanese-as-burt-bacharach.html" ...
## ..$ publication_dt: chr [1:20] "2011-11-10" "1999-02-17" "1997-11-02" "1993-03-28" ...
## ..$ byline : chr [1:20] "JANET MASLIN" "RICHARD BERNSTEIN" "JAMIE JAMES" "DAVID LEAVITT" ...
## ..$ book_title : chr [1:20] "1Q84" "South of the Border, West of the Sun" "Wind-Up Bird Chronicle" "Elephant Vanishes : Stories" ...
## ..$ book_author : chr [1:20] "Haruki Murakami" "Haruki Murakami" "Haruki Murakami" "Haruki Murakami" ...
## ..$ summary : chr [1:20] "In “1Q84,” the Japanese novelist Haruki Murakami writes about characters in a Tokyo with two moons." "" "" "" ...
## ..$ uuid : chr [1:20] "00000000-0000-0000-0000-000000000000" "00000000-0000-0000-0000-000000000000" "00000000-0000-0000-0000-000000000000" "00000000-0000-0000-0000-000000000000" ...
## ..$ uri : chr [1:20] "nyt://book/00000000-0000-0000-0000-000000000000" "nyt://book/00000000-0000-0000-0000-000000000000" "nyt://book/00000000-0000-0000-0000-000000000000" "nyt://book/00000000-0000-0000-0000-000000000000" ...
## ..$ isbn13 :List of 20
## .. ..$ : chr [1:9] "9780307476463" "9780307593313" "9780307957023" "9780345802934" ...
## .. ..$ : chr "9780375402517"
## .. ..$ : chr "9780517414712"
## .. ..$ : chr "9780679750536"
## .. ..$ : chr [1:9] "9780307476463" "9780307593313" "9780307957023" "9780345802934" ...
## .. ..$ : chr [1:4] "9780385352109" "9780385352116" "9780804166737" "9780804170123"
## .. ..$ : chr "9781400044610"
## .. ..$ : chr "9780679420576"
## .. ..$ : chr "9784770016836"
## .. ..$ : chr "9780679446699"
## .. ..$ : chr "9780375704024"
## .. ..$ : chr "9780375411694"
## .. ..$ : chr "9781860469671"
## .. ..$ : chr "9781860469671"
## .. ..$ : chr "9781860469671"
## .. ..$ : chr "9780307265838"
## .. ..$ : chr [1:5] "9780307269195" "9780307269478" "9780307373083" "9780307389831" ...
## .. ..$ : chr "9781400079278"
## .. ..$ : chr "9780451494627"
## .. ..$ : chr "9780525520047"We have successfully converted our data into something more readable, and can see that there are 20 observations in a dataframe under “results”. This is where we will continue our work by creating a dataframe of just these results.
results <- nyt_books$results
results## url
## 1 http://www.nytimes.com/2011/11/10/books/1q84-by-haruki-murakami-review.html
## 2 http://www.nytimes.com/1999/02/17/books/books-of-the-times-an-obsessive-attraction-that-cripples-two-lives.html
## 3 http://www.nytimes.com/1997/11/02/books/east-meets-west.html
## 4 http://www.nytimes.com/1993/03/28/books/as-japanese-as-burt-bacharach.html
## 5 http://www.nytimes.com/2011/11/06/books/review/1q84-by-haruki-murakami-translated-by-jay-rubin-and-philip-gabriel-book-review.html
## 6 http://www.nytimes.com/2014/08/10/books/review/haruki-murakamis-colorless-tsukuru-tazaki-and-his-years-of-pilgrimage.html
## 7 http://www.nytimes.com/2006/09/17/books/review/Rafferty2.t.html
## 8 http://www.nytimes.com/1993/05/12/books/books-of-the-times-from-japan-big-macs-and-marlboros-in-stories.html
## 9 http://www.nytimes.com/1994/01/03/books/books-of-the-times-looking-for-america-or-is-it-japan.html
## 10 http://www.nytimes.com/1997/10/31/books/books-of-the-times-on-a-nightmarish-trek-through-history-s-web.html
## 11 http://www.nytimes.com/2000/09/24/books/rubber-souls.html
## 12 http://www.nytimes.com/2001/06/10/books/lost-in-orbit.html
## 13 http://www.nytimes.com/2002/08/18/books/a-shock-to-the-system.html
## 14 http://www.nytimes.com/2002/08/20/books/books-of-the-times-worlds-where-anything-normal-would-seem-bizarre.html
## 15 http://www.nytimes.com/2002/08/25/books/and-bear-in-mind.html
## 16 http://www.nytimes.com/2007/06/03/books/review/Kirn-t.html
## 17 http://www.nytimes.com/2008/08/10/books/review/Dyer-t.html
## 18 http://www.nytimes.com/2005/02/06/books/review/kafka-on-the-shore-realitys-culdesacs.html
## 19 https://www.nytimes.com/2017/05/09/books/review/men-without-women-haruki-murakami.html
## 20 https://www.nytimes.com/2018/10/16/books/review/haruki-murakami-killing-commendatore.html
## publication_dt byline
## 1 2011-11-10 JANET MASLIN
## 2 1999-02-17 RICHARD BERNSTEIN
## 3 1997-11-02 JAMIE JAMES
## 4 1993-03-28 DAVID LEAVITT
## 5 2011-11-06 KATHRYN SCHULZ
## 6 2014-08-10 PATTI SMITH
## 7 2006-09-17 TERRENCE RAFFERTY
## 8 1993-05-12 HERBERT MITGANG
## 9 1994-01-03 HERBERT MITGANG
## 10 1997-10-31 MICHIKO KAKUTANI
## 11 2000-09-24 JANICE P. NIMURA
## 12 2001-06-10 DANIEL ZALEWSKI
## 13 2002-08-18 JEFF GILES
## 14 2002-08-20 MICHIKO KAKUTANI
## 15 2002-08-25
## 16 2007-06-03 WALTER KIRN
## 17 2008-08-10 GEOFF DYER
## 18 2005-02-06 LAURA MILLER
## 19 2017-05-09 JAY FIELDEN
## 20 2018-10-16 HARI KUNZRU
## book_title book_author
## 1 1Q84 Haruki Murakami
## 2 South of the Border, West of the Sun Haruki Murakami
## 3 Wind-Up Bird Chronicle Haruki Murakami
## 4 Elephant Vanishes : Stories Haruki Murakami
## 5 1Q84 Haruki Murakami
## 6 Colorless Tsukuru Tazaki and His Years of Pilgrimage Haruki Murakami
## 7 Blind Willow, Sleeping Woman: Twenty-Four Stories Haruki Murakami
## 8 The Elephant Vanishes: Stories Haruki Murakami
## 9 Dance Dance Dance: A Novel Haruki Murakami
## 10 The Wind-Up Bird Chronicle Haruki Murakami
## 11 Norwegian Wood Haruki Murakami
## 12 Sputnik Sweetheart Haruki Murakami
## 13 After the Quake Haruki Murakami
## 14 After the Quake Haruki Murakami
## 15 After the Quake Haruki Murakami
## 16 After Dark Haruki Murakami
## 17 What I Talk About When I Talk About Running Haruki Murakami
## 18 Kafka on the Shore Haruki Murakami
## 19 Men Without Women: Stories Haruki Murakami
## 20 Killing Commendatore Haruki Murakami
## summary
## 1 In “1Q84,” the Japanese novelist Haruki Murakami writes about characters in a Tokyo with two moons.
## 2
## 3
## 4
## 5 Haruki Murakami has translated Raymond Chandler into Japanese, and there’s a lot of Marlowe to his madness.
## 6 Patti Smith reviews Haruki Murakami’s dreamlike new novel, in which a man seeks out old friends to understand why he was banished from their circle.
## 7 A career-spanning grab bag of short stories from Haruki Murakami.
## 8
## 9
## 10
## 11
## 12
## 13
## 14
## 15
## 16 Haruki Murakami’s novel features two sisters, one in an enchanted sleep, the other up all night.
## 17 In his latest book -- part training guide, part memoir -- Haruki Murakami connects the disciplines of running and writing.
## 18 <p> KAFKA ON THE SHORE By Haruki Murakami. Translated by Philip Gabriel. 436 pp. Alfred A. Knopf. $25.95.</p>.
## 19 In the seven stories that make up “Men Without Women,” Haruki Murakami strikes a blue note: a rainy Tokyo full of unfaithful women.
## 20 The Japanese novelist’s latest book, “Killing Commendatore,” features a stymied artist, a haunted painting and a host of paranormal mysteries.
## uuid
## 1 00000000-0000-0000-0000-000000000000
## 2 00000000-0000-0000-0000-000000000000
## 3 00000000-0000-0000-0000-000000000000
## 4 00000000-0000-0000-0000-000000000000
## 5 00000000-0000-0000-0000-000000000000
## 6 00000000-0000-0000-0000-000000000000
## 7 00000000-0000-0000-0000-000000000000
## 8 00000000-0000-0000-0000-000000000000
## 9 00000000-0000-0000-0000-000000000000
## 10 00000000-0000-0000-0000-000000000000
## 11 00000000-0000-0000-0000-000000000000
## 12 00000000-0000-0000-0000-000000000000
## 13 00000000-0000-0000-0000-000000000000
## 14 00000000-0000-0000-0000-000000000000
## 15 00000000-0000-0000-0000-000000000000
## 16 00000000-0000-0000-0000-000000000000
## 17 00000000-0000-0000-0000-000000000000
## 18 00000000-0000-0000-0000-000000000000
## 19 00000000-0000-0000-0000-000000000000
## 20 00000000-0000-0000-0000-000000000000
## uri
## 1 nyt://book/00000000-0000-0000-0000-000000000000
## 2 nyt://book/00000000-0000-0000-0000-000000000000
## 3 nyt://book/00000000-0000-0000-0000-000000000000
## 4 nyt://book/00000000-0000-0000-0000-000000000000
## 5 nyt://book/00000000-0000-0000-0000-000000000000
## 6 nyt://book/00000000-0000-0000-0000-000000000000
## 7 nyt://book/00000000-0000-0000-0000-000000000000
## 8 nyt://book/00000000-0000-0000-0000-000000000000
## 9 nyt://book/00000000-0000-0000-0000-000000000000
## 10 nyt://book/00000000-0000-0000-0000-000000000000
## 11 nyt://book/00000000-0000-0000-0000-000000000000
## 12 nyt://book/00000000-0000-0000-0000-000000000000
## 13 nyt://book/00000000-0000-0000-0000-000000000000
## 14 nyt://book/00000000-0000-0000-0000-000000000000
## 15 nyt://book/00000000-0000-0000-0000-000000000000
## 16 nyt://book/00000000-0000-0000-0000-000000000000
## 17 nyt://book/00000000-0000-0000-0000-000000000000
## 18 nyt://book/00000000-0000-0000-0000-000000000000
## 19 nyt://book/00000000-0000-0000-0000-000000000000
## 20 nyt://book/00000000-0000-0000-0000-000000000000
## isbn13
## 1 9780307476463, 9780307593313, 9780307957023, 9780345802934, 9781446484197, 9781446484203, 9781455830497, 9781469258843, 9788483832967
## 2 9780375402517
## 3 9780517414712
## 4 9780679750536
## 5 9780307476463, 9780307593313, 9780307957023, 9780345802934, 9781446484197, 9781446484203, 9781455830497, 9781469258843, 9788483832967
## 6 9780385352109, 9780385352116, 9780804166737, 9780804170123
## 7 9781400044610
## 8 9780679420576
## 9 9784770016836
## 10 9780679446699
## 11 9780375704024
## 12 9780375411694
## 13 9781860469671
## 14 9781860469671
## 15 9781860469671
## 16 9780307265838
## 17 9780307269195, 9780307269478, 9780307373083, 9780307389831, 9781433243912
## 18 9781400079278
## 19 9780451494627
## 20 9780525520047Now we can begin to tidy the data.
First, I want to separate out the dates of the reviews as days, months, and years.
results <- results |>
mutate(
year = year(publication_dt),
month = month(publication_dt),
day = day(publication_dt)) |>
select(-publication_dt)Next we can separate out the ISBN numbers for his books.
results <- results |>
mutate(isbn13 = str_remove_all(results$isbn13, 'c\\(|\\)|"'))## Warning: There was 1 warning in `mutate()`.
## ℹ In argument: `isbn13 = str_remove_all(results$isbn13, "c\\(|\\)|\"")`.
## Caused by warning in `stri_replace_all_regex()`:
## ! argument is not an atomic vector; coercingresults <- results |>
separate_rows(isbn13, sep = ", ")
results## # A tibble: 43 × 11
## url byline book_title book_author summary uuid uri isbn13 year month
## <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <dbl> <dbl>
## 1 http://… JANET… 1Q84 Haruki Mur… "In “1… 0000… nyt:… 97803… 2011 11
## 2 http://… JANET… 1Q84 Haruki Mur… "In “1… 0000… nyt:… 97803… 2011 11
## 3 http://… JANET… 1Q84 Haruki Mur… "In “1… 0000… nyt:… 97803… 2011 11
## 4 http://… JANET… 1Q84 Haruki Mur… "In “1… 0000… nyt:… 97803… 2011 11
## 5 http://… JANET… 1Q84 Haruki Mur… "In “1… 0000… nyt:… 97814… 2011 11
## 6 http://… JANET… 1Q84 Haruki Mur… "In “1… 0000… nyt:… 97814… 2011 11
## 7 http://… JANET… 1Q84 Haruki Mur… "In “1… 0000… nyt:… 97814… 2011 11
## 8 http://… JANET… 1Q84 Haruki Mur… "In “1… 0000… nyt:… 97814… 2011 11
## 9 http://… JANET… 1Q84 Haruki Mur… "In “1… 0000… nyt:… 97884… 2011 11
## 10 http://… RICHA… South of … Haruki Mur… "" 0000… nyt:… 97803… 1999 2
## # ℹ 33 more rows
## # ℹ 1 more variable: day <int>The data is now tidied. For the sake of looking at a nicer set of data, I am going to filter out two columns where every value is the same.
results <- results |>
select(-uuid, -uri)
results## # A tibble: 43 × 9
## url byline book_title book_author summary isbn13 year month day
## <chr> <chr> <chr> <chr> <chr> <chr> <dbl> <dbl> <int>
## 1 http://www.ny… JANET… 1Q84 Haruki Mur… "In “1… 97803… 2011 11 10
## 2 http://www.ny… JANET… 1Q84 Haruki Mur… "In “1… 97803… 2011 11 10
## 3 http://www.ny… JANET… 1Q84 Haruki Mur… "In “1… 97803… 2011 11 10
## 4 http://www.ny… JANET… 1Q84 Haruki Mur… "In “1… 97803… 2011 11 10
## 5 http://www.ny… JANET… 1Q84 Haruki Mur… "In “1… 97814… 2011 11 10
## 6 http://www.ny… JANET… 1Q84 Haruki Mur… "In “1… 97814… 2011 11 10
## 7 http://www.ny… JANET… 1Q84 Haruki Mur… "In “1… 97814… 2011 11 10
## 8 http://www.ny… JANET… 1Q84 Haruki Mur… "In “1… 97814… 2011 11 10
## 9 http://www.ny… JANET… 1Q84 Haruki Mur… "In “1… 97884… 2011 11 10
## 10 http://www.ny… RICHA… South of … Haruki Mur… "" 97803… 1999 2 17
## # ℹ 33 more rowsWhile interfacing through an API was new and sometimes difficult, I found that the amount of R code needed (at least here) was spare. The real challenge for me was understanding exactly what URLs I would need, and how to establish the connection with my key. After that, converting the information into something more readable and then tidying was fairly simple. I appreciate the ease of use that the NYT API provided, as I can imagine how connecting might get more difficult with less clear instructions.