Assignment

Your task is to choose one of the New York Times APIs, construct an interface in R to read in the JSON data, and transform it into an R DataFrame.

Setup

In addition to standard tidyverse usage, we’ll leverage the jsonlite for querying the API, lubridate for formatting dates, and kableExtra to display our results.

library(tidyverse)
library(jsonlite)
library(lubridate)
library(kableExtra)

For this assignment, I’ve focused on the New York Time’s Article Search API. The API instructions note that equests will be limited to 10 articles unless query hits are spaced at least 6 seconds apart. For the sake of this exercise, we’ll ignore timing and work within the 10-article limit.

The API requires a key, which users can obtain by creating an account at https://developer.nytimes.com/. I’ll read in my key from a .txt file.

key <- read_lines('data/nyt_api_key.txt')

We’ll begin with a static example. Per the instructions on the NYT API page, we’ll use the base API url and add key terms to filter our search. For now, we’ll just use a simple keyword search (q=) within a specific date range (using &begin_date= and &end_date=).

base_url <- 'https://api.nytimes.com/svc/search/v2/articlesearch.json?'
search_term <- 'gerrymandering'
begin_date <- '20230101'
end_date <- '20230331'

url <- str_c(base_url,
             'q=',search_term,
             '&begin_date=',begin_date,
             '&end_date=',end_date,
             '&api-key=',key)

request <- fromJSON(url)

The JSON we obtain from our request is a large, heavily nested list. After some exploration, it becomes clear that the actual content of the query is in the response$docs$ sublist. I create a variable of that sublist, then parse out key pieces of content from within that sublist into a dataframe. For now, I’ve focused on key article-related details (e.g. date, author, desk) along with whatever actual text content is available (headline, abstract and lead paragraph).

response <- request$response$docs

df <- data.frame(pub_date = as_date(response$pub_date),
                 byline = response$byline$original,
                 news_desk = response$news_desk,
                 document_type = response$document_type,
                 type_of_material = response$type_of_material,
                 word_count = response$word_count,
                 headline = if_else(!is.na(response$headline$print_headline),
                                    response$headline$print_headline,
                                    response$headline$main),
                 abstract = response$abstract,
                 lead_paragraph = response$lead_paragraph
                 )

We can preview the first three rows of the resulting dataframe to get a sense of our results.

df[1:3,] %>% kbl() %>% kable_classic()
pub_date byline news_desk document_type type_of_material word_count headline abstract lead_paragraph
2023-03-21 By Reid J. Epstein Politics article News 1157 In Wisconsin Court Race, A Hostile Cultural Clash In their lone debate, Janet Protasiewicz, a liberal, and Daniel Kelly, a conservative, unleashed searing attacks on each other. At stake is a crucial seat on the battleground state’s Supreme Court. MADISON, Wis. — The dueling contenders in Wisconsin’s consequential and costly Supreme Court race collided on Tuesday in their lone debate, a hostile affair that illustrated their stark disagreements over cultural issues and the role of a justice on the state’s high court.
2023-03-15 By Reid J. Epstein Politics article News 1043 Wisconsin Conservative Court Candidate Faces Formidable TV Ad Gap Daniel Kelly, the conservative candidate for a swing seat on Wisconsin’s Supreme Court, promised that help was on the way. But his campaign has already been outspent on TV by $9.1 million to nothing. As conservatives in Wisconsin seek to maintain control of the State Supreme Court in an all-important election for a crucial swing seat, they would appear to be fighting uphill.
2023-03-10 By Reid J. Epstein Politics article News 1127 A Full-Time Battle Looms Over Congressional Maps What used to be a once-a-decade redistricting fight between political parties is now in perpetual motion, and up to 29 seats in 14 states are already at risk of being redrawn. WASHINGTON — For just about all of the nation’s history, politicians would fight over redistricting for a short period after each once-a-decade census, then forget about congressional maps until the next reapportionment.

Now that we’ve tested our approach, we can create a function to enable more efficient programming of queries. I’ve added some functionality to (i) add additional search parameters, and (ii) allow certain search parameters to be optional.

nyt_query <- function(search_term, begin_date = NULL, 
                      end_date = NULL, news_desk = NULL, 
                      type_of_material = NULL) {
  url <- str_c(base_url,
               'q=',search_term,
               if_else(is.null(begin_date), '',
                       str_c('&begin_date=',begin_date)),
               if_else(is.null(end_date), '',
                       str_c('&end_date=',end_date)),
               if_else(is.null(news_desk), '',
                       str_c('&fq=news_desk:(',news_desk,')')),
               if_else(is.null(type_of_material), '',
                       str_c('&fq=type_of_material:',type_of_material)),
               '&api-key=',key)
  
  request <- fromJSON(url)
  
  response <- request$response$docs
  
  df <- data.frame(pub_date = as_date(response$pub_date),
                   byline = response$byline$original,
                   news_desk = response$news_desk,
                   document_type = response$document_type,
                   type_of_material = response$type_of_material,
                   word_count = response$word_count,
                   headline = if_else(!is.na(response$headline$print_headline),
                                      response$headline$print_headline,
                                      response$headline$main),
                   abstract = response$abstract,
                   lead_paragraph = response$lead_paragraph
  )
  
  return(df)
}

We first test this by re-producing our original query to confirm we obtain comparable results.

nyt_query(search_term = 'gerrymandering', 
          begin_date = '20230101')[1:3,] %>% 
  kbl() %>% kable_classic()
pub_date byline news_desk document_type type_of_material word_count headline abstract lead_paragraph
2023-03-21 By Reid J. Epstein Politics article News 1157 In Wisconsin Court Race, A Hostile Cultural Clash In their lone debate, Janet Protasiewicz, a liberal, and Daniel Kelly, a conservative, unleashed searing attacks on each other. At stake is a crucial seat on the battleground state’s Supreme Court. MADISON, Wis. — The dueling contenders in Wisconsin’s consequential and costly Supreme Court race collided on Tuesday in their lone debate, a hostile affair that illustrated their stark disagreements over cultural issues and the role of a justice on the state’s high court.
2023-03-15 By Reid J. Epstein Politics article News 1043 Wisconsin Conservative Court Candidate Faces Formidable TV Ad Gap Daniel Kelly, the conservative candidate for a swing seat on Wisconsin’s Supreme Court, promised that help was on the way. But his campaign has already been outspent on TV by $9.1 million to nothing. As conservatives in Wisconsin seek to maintain control of the State Supreme Court in an all-important election for a crucial swing seat, they would appear to be fighting uphill.
2023-03-10 By Reid J. Epstein Politics article News 1127 A Full-Time Battle Looms Over Congressional Maps What used to be a once-a-decade redistricting fight between political parties is now in perpetual motion, and up to 29 seats in 14 states are already at risk of being redrawn. WASHINGTON — For just about all of the nation’s history, politicians would fight over redistricting for a short period after each once-a-decade census, then forget about congressional maps until the next reapportionment.

Finally, we can try a few additional queries with different parameters.

nyt_query(search_term = 'svb', 
          begin_date = '20230301',
          end_date = '20230320',
          type_of_material = 'News') %>% 
  kbl() %>% kable_classic()
pub_date byline news_desk document_type type_of_material word_count headline abstract lead_paragraph
2023-03-17 By Lauren Hirsch and Matthew Goldstein Business article News 386 In the Wake Of SVB Failure, Parent Files For Bankruptcy The move would place SVB Financial Group, which owns other businesses aside from the collapsed Silicon Valley Bank, into a court-led process as creditors looked to recover funds. SVB Financial Group, the former parent company of Silicon Valley Bank, the lender that regulators seized last week after a devastating run on deposits, filed for bankruptcy on Friday.
2023-03-14 By Maureen Farrell Business article News 1974 Failed Lender Mixed Hubris And Missteps While its leader extolled innovation and the future of tech, the bank paid less attention to risk management and was caught flat-footed by economic change. Gregory Becker, the chief executive of Silicon Valley Bank, leaned back in his chair at a technology conference last week in San Francisco’s luxurious Palace Hotel, and delivered a bullish message.
2023-03-14 By Alan Rappeport, Lauren Hirsch, Jeanna Smialek and Jim Tankersley Washington article News 1821 Moving Quickly to Avert a Financial Avalanche Officials were initially unsure about the need for the measures they eventually announced to shore up the financial system, but changed their minds quickly. WASHINGTON — On Friday afternoon, the deputy Treasury secretary, Wally Adeyemo, met with Jamie Dimon, the chief executive of JPMorgan Chase & Company, at Mr. Dimon’s office in New York.
2023-03-17 By Erin Griffith, Mike Isaac and Sheera Frenkel Business article News 1577 Cozy Tech Ties Helped a Bank Thrive, and Fall More than many other banks, SVB catered to how risky tech start-ups and their backers do not adhere to normal business practices. When Kleiner Perkins, one of Silicon Valley’s highest-profile venture capital firms, wanted to build a bridge between two of its office buildings around 2005, it decided to take out a loan. It turned to Silicon Valley Bank, just 43 feet away on Sand Hill Road in the heart of the venture industry in Menlo Park, Calif.
2023-03-15 By Rob Copeland and Maureen Farrell Business article News 631 Shards of Failed Bank Are for Sale, but No One Is Buying Yet Big rivals are thus far shying away from scooping up the bank’s assets. Five days after seizing control of Silicon Valley Bank, federal regulators were still in the process of auctioning off its remaining parts, but no single buyer appeared willing to take on everything.
2023-03-18 By Emma Goldberg SundayBusiness article News 2803 A Bank Fails, and the Clock Is Ticking for a Start-Up The collapse of Silicon Valley Bank set major financial disruptions in motion. It also made one founder’s week a scramble. Sara Mauskopf’s day-to-day concerns — her customers, her employees, her own three children — don’t typically involve the Federal Deposit Insurance Corporation. She is the chief executive of Winnie, a start-up that helps families find child care and helps child care providers find customers, serving over 250,000 day cares and millions of families. Ms. Mauskopf, 38, has raised over $15 million for Winnie since its founding in 2016 and employs 29 people. All of her company’s money was in Silicon Valley Bank.
2023-03-13 By Tiffany May Business article News 366 Roku, Roblox and Other Companies Used Silicon Valley Bank The failed bank had said it provided services to nearly half of all venture-backed technology and life-science companies in the United States. Silicon Valley Bank, which described itself as the “financial partner of the innovation economy,” provided services to nearly half of all venture-backed technology and life-science companies in the United States, according to its website.
2023-03-10 By Joe Rennison Business article News 831 Bank Failure Sends a Jolt To Markets The collapse of Silicon Valley Bank added to worries about the economy. The S&P 500 suffered its sharpest weekly decline of the year. The collapse of a small bank in California on Friday set off a wave of concern over the health of the banking sector, amplifying fears for the broader economy and sending global stock markets lower.
2023-03-17 By Lauren Hirsch and Rob Copeland Business article News 886 First Republic Seeks to Sell Shares of Itself to Get Cash Private equity firms and other buyers have been circling failed and struggling banks that are desperately trying to sell off assets or even stakes. Just one day after the biggest U.S. banks gave it a $30 billion infusion, First Republic Bank was in talks to sell a piece of itself to other banks or private equity firms, three people with knowledge of the process said, an indication that the imperiled lender is far from conquering its troubles.
2023-03-10 By Andrew Ross Sorkin, Ravi Mattu, Bernhard Warner, Sarah Kessler, Michael J. de la Merced, Lauren Hirsch and Ephrat Livni Business article News 1696 Troubles at Silicon Valley Bank Rock the Global Market Shares in the bank, a go-to funder of tech start-ups, have fallen more than 60 percent in the past day as investors fear for its finances, and those of similarly sized lenders. Shares in U.S. banks appear poised to slide again on Friday, after the sector suffered a $52 billion sell-off on Thursday, its worst drop in three years. Driving investor concerns is the health of Silicon Valley Bank, one of the biggest lenders to tech start-ups, which is urgently raising cash to shore up its finances. 
nyt_query(search_term = 'svb', 
          begin_date = '20230301',
          end_date = '20230320',
          news_desk = 'Business',
          type_of_material = 'News') %>% 
  kbl() %>% kable_classic()
pub_date byline news_desk document_type type_of_material word_count headline abstract lead_paragraph
2023-03-17 By Lauren Hirsch and Matthew Goldstein Business article News 386 In the Wake Of SVB Failure, Parent Files For Bankruptcy The move would place SVB Financial Group, which owns other businesses aside from the collapsed Silicon Valley Bank, into a court-led process as creditors looked to recover funds. SVB Financial Group, the former parent company of Silicon Valley Bank, the lender that regulators seized last week after a devastating run on deposits, filed for bankruptcy on Friday.
2023-03-14 By Maureen Farrell Business article News 1974 Failed Lender Mixed Hubris And Missteps While its leader extolled innovation and the future of tech, the bank paid less attention to risk management and was caught flat-footed by economic change. Gregory Becker, the chief executive of Silicon Valley Bank, leaned back in his chair at a technology conference last week in San Francisco’s luxurious Palace Hotel, and delivered a bullish message.
2023-03-17 By Erin Griffith, Mike Isaac and Sheera Frenkel Business article News 1577 Cozy Tech Ties Helped a Bank Thrive, and Fall More than many other banks, SVB catered to how risky tech start-ups and their backers do not adhere to normal business practices. When Kleiner Perkins, one of Silicon Valley’s highest-profile venture capital firms, wanted to build a bridge between two of its office buildings around 2005, it decided to take out a loan. It turned to Silicon Valley Bank, just 43 feet away on Sand Hill Road in the heart of the venture industry in Menlo Park, Calif.
2023-03-15 By Rob Copeland and Maureen Farrell Business article News 631 Shards of Failed Bank Are for Sale, but No One Is Buying Yet Big rivals are thus far shying away from scooping up the bank’s assets. Five days after seizing control of Silicon Valley Bank, federal regulators were still in the process of auctioning off its remaining parts, but no single buyer appeared willing to take on everything.
2023-03-13 By Tiffany May Business article News 366 Roku, Roblox and Other Companies Used Silicon Valley Bank The failed bank had said it provided services to nearly half of all venture-backed technology and life-science companies in the United States. Silicon Valley Bank, which described itself as the “financial partner of the innovation economy,” provided services to nearly half of all venture-backed technology and life-science companies in the United States, according to its website.
2023-03-10 By Joe Rennison Business article News 831 Bank Failure Sends a Jolt To Markets The collapse of Silicon Valley Bank added to worries about the economy. The S&P 500 suffered its sharpest weekly decline of the year. The collapse of a small bank in California on Friday set off a wave of concern over the health of the banking sector, amplifying fears for the broader economy and sending global stock markets lower.
2023-03-17 By Lauren Hirsch and Rob Copeland Business article News 886 First Republic Seeks to Sell Shares of Itself to Get Cash Private equity firms and other buyers have been circling failed and struggling banks that are desperately trying to sell off assets or even stakes. Just one day after the biggest U.S. banks gave it a $30 billion infusion, First Republic Bank was in talks to sell a piece of itself to other banks or private equity firms, three people with knowledge of the process said, an indication that the imperiled lender is far from conquering its troubles.
2023-03-10 By Andrew Ross Sorkin, Ravi Mattu, Bernhard Warner, Sarah Kessler, Michael J. de la Merced, Lauren Hirsch and Ephrat Livni Business article News 1696 Troubles at Silicon Valley Bank Rock the Global Market Shares in the bank, a go-to funder of tech start-ups, have fallen more than 60 percent in the past day as investors fear for its finances, and those of similarly sized lenders. Shares in U.S. banks appear poised to slide again on Friday, after the sector suffered a $52 billion sell-off on Thursday, its worst drop in three years. Driving investor concerns is the health of Silicon Valley Bank, one of the biggest lenders to tech start-ups, which is urgently raising cash to shore up its finances.
2023-03-13 By Jin Yu Young Business article News 172 HSBC to Buy Silicon Valley Bank’s British Subsidiary Prime Minister Rishi Sunak said the country’s financial system was in good health and would not suffer from the demise of Silicon Valley Bank. Silicon Valley Bank’s British subsidiary will be acquired by the banking giant HSBC, the Bank of England announced on Monday as it emphasized the stability of the nation’s banking system.
2023-03-15 By Andrew Ross Sorkin, Ravi Mattu, Bernhard Warner, Sarah Kessler, Michael J. de la Merced, Lauren Hirsch and Ephrat Livni Business article News 1865 Goldman Sachs Eyes a Big Payout From Silicon Valley Bank Deal The Wall Street giant is likely to be paid more than $100 million for its role in a bond purchase that ultimately failed to save the California bank from collapse. As an adviser to Silicon Valley Bank, Goldman Sachs last week tried to pull off a last-minute capital raise to save the firm from collapse. But the Wall Street giant also had another role in the bank’s final days, for which it’s expected to collect a massive fee: It bought a cache of the bank’s debt in a deal that ultimately led to concerns about the bank’s viability. 
nyt_query(search_term = 'svb', 
          begin_date = '20230301',
          end_date = '20230320',
          type_of_material = 'Op-Ed') %>% 
  kbl() %>% kable_classic()
pub_date byline news_desk document_type type_of_material word_count headline abstract lead_paragraph
2023-03-16 By Elizabeth Spiers OpEd article Op-Ed 1185 I Was an S.V.B. Client. I Blame the Venture Capitalists. It started with a culture of insular self-congratulation. It ended in an avalanche. In 2016, I started a New York-based creative agency that specialized in branded content. Among creative agencies, the trend at the time was for names that sounded like punk bands, and I unfortunately chose the Insurrection. As of last week, the only thing that aged worse than the name was my choice of bank: Silicon Valley Bank, which has now become the most spectacular example of a bank failure since the 2008 financial crisis. (I briefly lost access to our company’s funds, but I’m fine; my deposits were low enough to be covered by F.D.I.C. guarantees.)
2023-03-14 By Jamelle Bouie OpEd article Op-Ed 1189 The Boys Who Cried ‘Woke!’ There’s a reason Republicans want to distract our attention from what’s really going on. As soon as it was clear that Silicon Valley Bank would not survive the weekend, conservative influencers and Republican politicians had a culprit in sight.
2023-03-18 By Ezra Klein OpEd article Op-Ed 1749 Three Ways the Financial World Has Gone Topsy-Turvy Banking is a critical form of public infrastructure that we pretend is a private act of risk management. Over the past week, an observation by Matt Klein, a financial journalist, has gotten passed around quite a bit. “This was more a case of a ‘bank-run by idiots’ rather than a ‘bank run by idiots,’” he wrote, referring to the collapse of Silicon Valley Bank.