I selected the most popular emailed articles on NYT’s website for the past 30 days. My question is what was the most popular emailed article types on NYT’s website for the past 30 days?
Tackling the problem: First I’ll need to review NYT’s API instructions to use their public APIs. After authenticating the API Key and getting the HTTP request I’ll create an r environment to store the key. Then I’ll use the HTTP request and API key to make the request to the end point. I’ll begin parsing the JSON response using the httr2 package and reference Prof Gomez’s week9 slide deck. Then I’ll bring everything into a dataframe with some cleaned variables.
Anticipated data challenges: I anticipate I will run into the most issues successfully making the API request, and parsing the JSON file.
#Install packages needed pacman::p_load(# Package Install and Management pacman, # package install/load janitor, #clean up data names # Project and File Management readr, # import data httr, #github passkey usethis, #r environment RPostgres, #SQL DBI, #SQL jsonlite, #working with json rvest, #html files rstuioapi, #api knitr, #displaying data in quarto# General Data Management dplyr, # data management tidyr, # data management skimr, # data management lubridate, # work with dates zoo, # work with dates tidyverse, # work with dates scales, # ggplot gmodels, # freq tables SAS patchwork #data visualization, side by side )#API specificlibrary(tibble)library(httr2)library(RSQLite)
Make a request to the endpoint
Code
# The endpoint returns records for most popular NYT articlesapi_url <-paste0("https://api.nytimes.com/svc/mostpopular/v2/emailed/30.json")# request() starts an HTTP request with the API URL.req <-request(api_url) |>req_url_query(`api-key`= nyt_key_pop)# req_headers() adds information the server expects from our client.# req_dry_run() previews the request without sending it.req_dry_run(req)
# req_perform() sends the request and keeps the server response.resp <- req |>req_perform()# Confirm that the server accepted the request.resp_status(resp) #200
[1] 200
Code
resp_status_desc(resp)
[1] "OK"
Code
resp_content_type(resp)
[1] "application/json"
Parse the JSON response
Code
## Read the JSON body # Parse the JSON response into an R list.pop_json <- resp |>resp_body_json(simplifyVector =TRUE)# Inspect the top-level fields before extracting data.names(pop_json)
[1] "status" "copyright" "num_results" "results"
Transform the result into a clean R data frame, dropped nested, and add retrieval time
Only data cleaning ID variables in order to see the whole number (it was in scientific notation). Not cleaning up date variables since saving to SQL file, and as.date format doesn’t work.
Code
## Create a data frame from results (will say data.frame)# Keep the match details df <- pop_json$results%>%mutate(id =as.character(id), asset_id =as.character(asset_id))%>%select(-where(is.list)) # need to drop any other nested columns #sapply(matches_snapshot, class) #names(matches_snapshot)[sapply(matches_snapshot, is.list)]#An API can change between requests. Save the retrieval time and source URL with the values you used.# Use UTC so each timestamp has one unambiguous time zone.retrieved_at <-format(Sys.time(), tz ="UTC", usetz =TRUE)matches_snapshot <- df |>mutate(retrieved_at = retrieved_at,request_url = api_url,.before =1)
Most popular emailed article types on NYT’s website for the past 30 days?
4 (20%) popular opinion pieces made up the most types of popular emailed articles the past month.
Code
#which type of article was popular this last month? counts <- matches_snapshot %>%count(nytdsection, name ="n") %>%mutate(pct_total = n /sum(n)) %>%arrange(desc(n))#graphcounts |>ggplot(aes(x =fct_reorder(nytdsection, n, .desc =TRUE), y = n)) +geom_col(fill ="#1f77b4") +geom_text(aes(label =paste0(comma(n), " (", percent(pct_total, accuracy =0.1), ")")),position =position_stack(vjust =0.8),color ="white",size =2.5,fontface ="bold" ) +labs(title ="Most popular emailed article types on NYT's website for the past 30 days?" ) +scale_y_continuous(labels =label_comma()) +labs(x =NULL, y ="Count")
Store data in .sqlite
Code
# Connect to a local SQLite file in the Week 9 folder.con <-dbConnect(SQLite(), "nyt_df.sqlite", extended_types =TRUE)# Append this retrieval without deleting earlier snapshots.dbWriteTable(con, "df", matches_snapshot, overwrite =TRUE)# Always close the connection when finished.dbDisconnect(con)
Use saved data rather than opening API connection
For reproducible work, save the request URL, retrieval time, and source documentation with every API snapshot.
Code
# Reconnect and read saved rows from the local database.con <-dbConnect(SQLite(), "nyt_df.sqlite", extended_types =TRUE)saved_matches <-dbGetQuery( con,"SELECT * FROM df ORDER BY retrieved_at DESC, published_date DESC")dbDisconnect(con)skim(saved_matches)
Data summary
Name
saved_matches
Number of rows
20
Number of columns
19
_______________________
Column type frequency:
character
17
numeric
2
________________________
Group variables
None
Variable type: character
skim_variable
n_missing
complete_rate
min
max
empty
n_unique
whitespace
retrieved_at
0
1
23
23
0
1
0
request_url
0
1
58
58
0
1
0
uri
0
1
50
54
0
20
0
url
0
1
72
105
0
20
0
id
0
1
15
15
0
20
0
asset_id
0
1
15
15
0
20
0
source
0
1
14
14
0
1
0
published_date
0
1
10
10
0
13
0
updated
0
1
19
19
0
20
0
section
0
1
4
11
0
11
0
subsection
0
1
0
11
13
6
0
nytdsection
0
1
4
11
0
11
0
adx_keywords
0
1
65
284
0
20
0
byline
0
1
13
39
0
20
0
type
0
1
7
11
0
2
0
title
0
1
27
78
0
20
0
abstract
0
1
48
172
0
20
0
Variable type: numeric
skim_variable
n_missing
complete_rate
mean
sd
p0
p25
p50
p75
p100
hist
column
20
0
NaN
NA
NA
NA
NA
NA
NA
eta_id
0
1
0
0
0
0
0
0
0
▁▁▇▁▁
Conclusion
After many trials and errors I successfully accessed the NYT public API’s, authenticated the correct key I needed, added it to my .renviroment script, and made the request to the end point.
I followed the outline of the week 8 and 9 slide deck to parse the JSON script and flatten it to a dataframe in order to answer my original question “what was the most popular emailed article types on NYT’s website for the past 30 days?”
I saved the data frame as a sqlite file and was able to successfully re-connect.
Next, I would want to set up automatic data collection so I could do further data analysis more than 30 days of data.
