DATA607WK9Assignment
PK O’Flaherty
2022-03-29
President Volodymyr Zelensky of Ukraine, from www.nytimes.com/2022/03/27/world/europe/russia-media-zelensky
Web APIs
Getting Started
Instructions
Choose one of the New York Times APIs, construct an interface in R to read in the JSON data, and transform it into an R DataFrame.
Libraries
We need the rjson package in order to read from JSON to R Programming.
# Load packages --------------------------------------
library(rjson)
library(knitr)Approach
A detailed account of how we approached the task
First we created an account at The New York Times Developer Network to obtain an API key.
We selected the most viewed articles for the last seven days from the ‘Most Popular’ API.
Once we read in the JSON file and saw how it was more branched than tabular and attempts at wrangling it (contenting, parsing, flattening, unlisting) were unsuccessful for us, we switched to extracting specific columns instead of representing the whole JSON file as a data frame.
By experimentation we were able to reference specific content, determine it was a string, and use a for loop to create a vector of just that content for all twenty entries.
We repeated that for four columns to create our dataframe.
Preparing Data
New York Times API
Here we are calling the most viewed articles for the last seven days from the ‘Most Popular’ API, using our NYT Developer API key.
# The API is called and read from JSON into R in one step
r <- fromJSON(file = "https://api.nytimes.com/svc/mostpopular/v2/viewed/1.json?api-key=UNraXJrekYbc5zM3BzNszXT33RTbMm4U")Interface in R
The data is already in R. We can see below what the 20th most viewed article of the last 7 days was and see the structure of the data.
Content in order presented below that could be of interest include:
$url$published_date$adx_keyword$byline(authors)$title$abstract$'media_metadata'(tags related to photos including the picture of Zelensky above)
# View 20th result
r$results[20]## [[1]]
## [[1]]$uri
## [1] "nyt://article/bffbd3bd-620f-54d3-a4e5-3b5e7d4a7257"
##
## [[1]]$url
## [1] "https://www.nytimes.com/2022/03/27/world/europe/russia-media-zelensky.html"
##
## [[1]]$id
## [1] 1e+14
##
## [[1]]$asset_id
## [1] 1e+14
##
## [[1]]$source
## [1] "New York Times"
##
## [[1]]$published_date
## [1] "2022-03-27"
##
## [[1]]$updated
## [1] "2022-03-28 03:25:46"
##
## [[1]]$section
## [1] "World"
##
## [[1]]$subsection
## [1] "Europe"
##
## [[1]]$nytdsection
## [1] "world"
##
## [[1]]$adx_keywords
## [1] "Russian Invasion of Ukraine (2022);Freedom of the Press;Censorship;Putin, Vladimir V;Zelensky, Volodymyr;Novaya Gazeta;Russia;Ukraine"
##
## [[1]]$column
## NULL
##
## [[1]]$byline
## [1] "By Anton Troianovski and Ivan Nechepurenko"
##
## [[1]]$type
## [1] "Article"
##
## [[1]]$title
## [1] "Zelensky Gives Interview to Russian Journalists. Moscow Orders It Quashed."
##
## [[1]]$abstract
## [1] "The remarkable interview was still published by journalists outside of Russia, an episode that laid bare the extraordinary, and partly successful, efforts at censorship by Moscow."
##
## [[1]]$des_facet
## [1] "Russian Invasion of Ukraine (2022)" "Freedom of the Press"
## [3] "Censorship"
##
## [[1]]$org_facet
## [1] "Novaya Gazeta"
##
## [[1]]$per_facet
## [1] "Putin, Vladimir V" "Zelensky, Volodymyr"
##
## [[1]]$geo_facet
## [1] "Russia" "Ukraine"
##
## [[1]]$media
## [[1]]$media[[1]]
## [[1]]$media[[1]]$type
## [1] "image"
##
## [[1]]$media[[1]]$subtype
## [1] "photo"
##
## [[1]]$media[[1]]$caption
## [1] "President Volodymyr Zelensky of Ukraine spoke with some prominent members of the Russian media via video on Sunday."
##
## [[1]]$media[[1]]$copyright
## [1] "Ukrainian Presidential Press Service, via Reuters"
##
## [[1]]$media[[1]]$approved_for_syndication
## [1] 1
##
## [[1]]$media[[1]]$`media-metadata`
## [[1]]$media[[1]]$`media-metadata`[[1]]
## [[1]]$media[[1]]$`media-metadata`[[1]]$url
## [1] "https://static01.nyt.com/images/2022/03/27/world/27russia-media1/27russia-media1-thumbStandard.jpg"
##
## [[1]]$media[[1]]$`media-metadata`[[1]]$format
## [1] "Standard Thumbnail"
##
## [[1]]$media[[1]]$`media-metadata`[[1]]$height
## [1] 75
##
## [[1]]$media[[1]]$`media-metadata`[[1]]$width
## [1] 75
##
##
## [[1]]$media[[1]]$`media-metadata`[[2]]
## [[1]]$media[[1]]$`media-metadata`[[2]]$url
## [1] "https://static01.nyt.com/images/2022/03/27/world/27russia-media1/27russia-media1-mediumThreeByTwo210.jpg"
##
## [[1]]$media[[1]]$`media-metadata`[[2]]$format
## [1] "mediumThreeByTwo210"
##
## [[1]]$media[[1]]$`media-metadata`[[2]]$height
## [1] 140
##
## [[1]]$media[[1]]$`media-metadata`[[2]]$width
## [1] 210
##
##
## [[1]]$media[[1]]$`media-metadata`[[3]]
## [[1]]$media[[1]]$`media-metadata`[[3]]$url
## [1] "https://static01.nyt.com/images/2022/03/27/world/27russia-media1/27russia-media1-mediumThreeByTwo440.jpg"
##
## [[1]]$media[[1]]$`media-metadata`[[3]]$format
## [1] "mediumThreeByTwo440"
##
## [[1]]$media[[1]]$`media-metadata`[[3]]$height
## [1] 293
##
## [[1]]$media[[1]]$`media-metadata`[[3]]$width
## [1] 440
##
##
##
##
##
## [[1]]$eta_id
## [1] 0Transform to an R Dataframe
Here we create four vectors to read into the dataframe.
Code Summary
+ Create the URL vector
+ Create the title vector
+ Create the keywords vector
+ Create the abstract vector
+ Construct and view dataframe
# Create the URL vector
url <- c(1:20)
for (x in c(1:20)){
url[x] <- r$results[x][[1]]$url
}# Create the title vector
title <- c(1:20)
for (x in c(1:20)){
title[x] <- r$results[x][[1]]$title
}# Create the keywords vector
keywords <- c(1:20)
for (x in c(1:20)){
keywords[x] <- r$results[x][[1]]$adx_keywords
}# Create the abstract vector
abstract <- c(1:20)
for (x in c(1:20)){
abstract[x] <- r$results[x][[1]]$abstract
}# Construct and view dataframe
df <- data.frame(url, title, keywords, abstract)
kable(df)| url | title | keywords | abstract |
|---|---|---|---|
| https://www.nytimes.com/2022/03/28/movies/oscars-will-smith-slap-reactions.html | Will Smith Apologizes to Chris Rock After Academy Condemns His Slap | Academy Awards (Oscars);Television;Comedy and Humor;Alopecia Areata;Assaults;Smith, Will;Smith, Jada Pinkett;Rock, Chris | “I was out of line and I was wrong,” said Smith, who hit Rock at the Oscars after the comedian made a joke about his wife. The film organization opened an inquiry into the incident. |
| https://www.nytimes.com/2022/03/27/movies/will-smith-chris-rock-oscars.html | Will Smith hits Chris Rock after joke about his wife, Jada. | Academy Awards (Oscars);Movies;Rock, Chris;Smith, Jada Pinkett;Smith, Will | The apparently unscripted moment stunned viewers and audiences alike. The telecast was muted and paused in the United States. |
| https://www.nytimes.com/2022/03/28/us/politics/trump-election-crimes.html | Federal Judge Finds Trump Most Likely Committed Crimes Over 2020 Election | Project: Democracy;Presidential Election of 2020;Storming of the US Capitol (Jan, 2021);Voter Fraud (Election Fraud);United States Politics and Government;Suits and Litigation (Civil);Decisions and Verdicts;Trump, Donald J;Eastman, John C;Carter, David O | “The illegality of the plan was obvious,” the judge wrote in a civil case. Separately, the Jan. 6 panel voted to recommend contempt of Congress charges for two former Trump aides. |
| https://www.nytimes.com/2022/03/28/movies/the-oscars-2022.html | Onstage Slap Rattles Oscars, Before ‘CODA’ Triumphs | Movies;Academy Awards (Oscars);Deafness;Hispanic-Americans;Video Recordings, Downloads and Streaming;Branagh, Kenneth;Campion, Jane;Chastain, Jessica;Eilish, Billie;Hall, Regina;Heder, Sian (1977- );Knowles, Beyonce;Smith, Will;Zimmer, Hans;Thompson, Ahmir (Questlove);Rock, Chris;Smith, Jada Pinkett;Kotsur, Troy;DeBose, Ariana;Apple TV | On a night when “CODA” became the first film from a streaming service to win best picture, the focus was on Will Smith’s altercation with Chris Rock. |
| https://www.nytimes.com/2022/03/27/movies/oscars-winner-list.html | 2022 Oscar Winners: Complete List | Movies;Academy Awards (Oscars);Chastain, Jessica;Smith, Will;Campion, Jane | The complete list of winners for the 94th Academy Awards. |
| https://www.nytimes.com/2022/03/28/movies/oscars-best-worst.html | The Best and Worst Moments of the 2022 Oscars | Academy Awards (Oscars);Movies;DeBose, Ariana;Hall, Regina;Knowles, Beyonce;Kotsur, Troy;Lady Gaga;Megan Thee Stallion (Rapper);Minnelli, Liza;Williams, Serena;Williams, Venus;Smith, Will;Rock, Chris;Sykes, Wanda;Schumer, Amy | The night made history, from That Slap to milestone wins by Troy Kotsur and Ariana DeBose, and a rousing performance by Beyoncé. |
| https://www.nytimes.com/2022/03/27/business/media/chris-wallace-cnn-fox-news.html | Chris Wallace Says Life at Fox News Became ‘Unsustainable’ | News and News Media;Television;Content Type: Personal Profile;Wallace, Chris (1947- );CNN;Fox News Channel | As he starts a new streaming show at CNN, the longtime TV anchor reflects on his decision to leave Fox News after 18 years. |
| https://www.nytimes.com/2022/03/27/climate/manchin-coal-climate-conflicts.html | How Joe Manchin Aided Coal, and Earned Millions | Conflicts of Interest;Global Warming;Greenhouse Gas Emissions;United States Politics and Government;Politics and Government;Coal;Ethics and Official Misconduct;Mines and Mining;Energy and Power;Factories and Manufacturing;Electric Light and Power;Lobbying and Lobbyists;Manchin, Joe III;Biden, Joseph R Jr;Manchin, Joseph IV;Enersystems Inc;Senate Committee on Energy and Natural Resources;Senate;American Bituminous Power Partners (AmBit);NRG Energy Inc;Edison International;Sumitomo Corporation;West Virginia;Grant Town (W Va) | At every step of his political career, Joe Manchin helped a West Virginia power plant that is the sole customer of his private coal business. Along the way, he blocked ambitious climate action. |
| https://www.nytimes.com/2022/03/25/arts/music/taylor-hawkins-foo-fighters-dead.html | Taylor Hawkins, Foo Fighters’ Drummer, Dies at 50 | Deaths (Obituaries);Pop and Rock Music;Hawkins, Taylor (1972-2022);Foo Fighters | Hard-hitting and charismatic, he was direct about his hopes for the group’s future, even after two decades. “I want to be the biggest band in the world,” he said. |
| https://www.nytimes.com/2022/03/28/crosswords/spelling-bee-forum.html | Spelling Bee Forum | Feeling stuck on today’s puzzle? We can help. | |
| https://www.nytimes.com/2022/03/28/us/crash-pennsylvania-snow.html | Snow Squall Leads to 50-Car Pileup on Pennsylvania Highway | Traffic Accidents and Safety;Snow and Snowstorms;Deaths (Fatalities);Schuylkill County (Pa);Pennsylvania | Several tractor-trailers were involved in the chain-reaction crash on Interstate 81, about 50 miles northeast of Harrisburg, Pa. The wreckage extended for about a mile, the authorities said. |
| https://www.nytimes.com/2022/03/28/opinion/letters/will-smith-chris-rock-oscars.html | No Joke: When Will Smith Slapped Chris Rock at the Oscars | Academy Awards (Oscars);Alopecia Areata;Smith, Will;Smith, Jada Pinkett;Rock, Chris | A sampling of the reaction to the slap that was the talk of the Academy Awards. |
| https://www.nytimes.com/2022/03/28/nyregion/nyc-elevator-outage-20-exchange-place.html | ‘High-Rise Hell’: N.Y.C. Skyscraper’s Elevator Breakdowns Strand Tenants | Elevators and Escalators;Real Estate and Housing (Residential);Buildings (Structures);Consolidated Edison Inc;Financial District (Manhattan, NY) | A luxury residential building in the financial district with more than 750 apartments has been experiencing lengthy elevator outages since the fall. |
| https://www.nytimes.com/2022/03/28/well/live/alopecia-hair-loss.html | What Is Alopecia? | Alopecia Areata;Autoimmune Diseases;Baldness;Drugs (Pharmaceuticals);Smith, Jada Pinkett | Jada Pinkett Smith’s hair loss condition played a major role in an incident at the Oscars. Here’s what we know about it. |
| https://www.nytimes.com/2022/03/28/technology/nokia-russia-surveillance-system-sorm.html | When Nokia Pulled Out of Russia, a Vast Surveillance System Remained | Surveillance of Citizens by Government;Russian Invasion of Ukraine (2022);Politics and Government;Computers and the Internet;Data-Mining and Database Marketing;Cyberwarfare and Defense;Government Contracts and Procurement;Telephones and Telecommunications;Espionage and Intelligence Services;Corporate Social Responsibility;Putin, Vladimir V;Nokia Oyj;Federal Security Service;Russia | The Finnish company played a key role in enabling Russia’s cyberspying, documents show, raising questions of corporate responsibility. |
| https://www.nytimes.com/2022/03/26/world/europe/vladimir-putin-russia.html | The Making of Vladimir Putin | Politics and Government;Russian Invasion of Ukraine (2022);International Relations;United States International Relations;Content Type: Personal Profile;Putin, Vladimir V;Russia;Ukraine;USSR (Former Soviet Union) | Tracing Putin’s 22-year slide from statesman to tyrant. |
| https://www.nytimes.com/2022/03/27/us/politics/biden-ukraine.html | After Biden’s Fiery Speech, Nine Unscripted Words Reverberate | United States International Relations;Russian Invasion of Ukraine (2022);United States Politics and Government;Speeches and Statements;Biden, Joseph R Jr;Putin, Vladimir V;Ukraine;Russia | Administration officials were forced to walk back the ad-lib that Russian President Vladimir V. Putin “cannot remain in power,” which captured the attention of foreign policy experts, lawmakers and allies. |
| https://www.nytimes.com/2022/03/28/arts/television/oscars-review-chris-rock-will-smith.html | An Onstage Slap Gives the Oscars More Drama Than It Bargained For | Academy Awards (Oscars);Movies;Hall, Regina;DeBose, Ariana;Knowles, Beyonce;Rock, Chris;Smith, Will;Schumer, Amy;Sykes, Wanda;Academy Museum of Motion Pictures | The show strained to offer something for everyone. But it could still create shocking moments. |
| https://www.nytimes.com/2022/03/28/science/bobcat-python-eggs.html | Bobcats With a Taste for Python Eggs Might Be the Guardians of Florida’s Swamp | Snakes;Bobcats;Eggs;Research;your-feed-science;your-feed-animals;Ecology and Evolution (Journal);Big Cypress National Preserve;Florida | Cameras captured the wild feline purloining a Burmese python’s eggs, giving hope that the state’s native species are responding to a voracious, invasive predator. |
| https://www.nytimes.com/2022/03/27/world/europe/russia-media-zelensky.html | Zelensky Gives Interview to Russian Journalists. Moscow Orders It Quashed. | Russian Invasion of Ukraine (2022);Freedom of the Press;Censorship;Putin, Vladimir V;Zelensky, Volodymyr;Novaya Gazeta;Russia;Ukraine | The remarkable interview was still published by journalists outside of Russia, an episode that laid bare the extraordinary, and partly successful, efforts at censorship by Moscow. |
Conclusion
I’m more confident I can pull data from API and JSON in the future. Once I’ve read the data into R I can use iterative code chunks to isolate individual fields of interest. For example:
r$results -> r$results[1] ->
r$results[1][[1]] ->
r$results[1][[1]]$url
I would like more experience with JSON and API calls and the functions used to manipulate complicated, branching data structures.
As a flex we could try to make a word cloud out of the keywords from all twenty articles.
Word Cloud
Here we generate a word cloud using a towardsdatascience article linked in the References section below.
#install.packages("wordcloud")
library(tm)
library(wordcloud)
#library(RColorBrewer)
library(magrittr)# Create a corpus
text <- df$keywords
docs <- Corpus(VectorSource(text))# Clean the corpus
docs <- docs %>%
tm_map(removeNumbers) %>%
tm_map(removePunctuation) %>%
tm_map(stripWhitespace)
docs <- tm_map(docs, content_transformer(tolower))
docs <- tm_map(docs, removeWords, stopwords("english"))# Create a Document Term Matrix to assign relevance to words
dtm <- TermDocumentMatrix(docs)
matrix <- as.matrix(dtm)
words <- sort(rowSums(matrix),decreasing=TRUE)
df <- data.frame(word = names(words),freq=words)# Generate word cloud
set.seed(2158)
wordcloud(words = df$word, freq = df$freq,
min.freq = 1, max.words=200,
random.order=FALSE, rot.per=0.35,
colors=brewer.pal(8, "Dark2"))
References
I followed this Medium article almost to the line to generate the word cloud.
Celine Van den Rul (2019) towardsdatascience.com/create-a-word-cloud-with-r