Aquiring Data

Setting up

In this setup, I set a directory location, read in my private access key, and use paste0 inside the GET method to combine the url, my query subject and the key to article searches for the New York Times API.

setwd("/Volumes/Document_Drive/CUNY_MS_ANALYTICS/DATA607/week_9")
source("ny_times_key.R")
subject <- "Russia+Manafort"
url <- "http://api.nytimes.com/svc/search/v2/articlesearch.json?q="
request <- GET(paste0(url, subject, article_key)) #pasting parts and GETTING Data

Convert Data From JSON to Data Frame

Here I simply convert the data to raw character and pull from JSON and save it into a coerced data frame with many nested data frames and lists.

article <- rawToChar(request$content) # converting to character data
article_content <- fromJSON(article, simplifyVector = TRUE) #extracting from JSON
article_data<-data.frame(article_content) #converting to data frame
article_frame<-data.frame(article_data[,c(2:5, 9:23)]) #selecting wanted data

Reformatting Data

Sort, Subset and Select Most Useful Frames

The first thing that I did was apply meaningful and easy names to read. They I pulled the two nested data frames I had decided to keep in the set, byline and reheadline. I created dataframes of each with unique readable names and bound them back to the original article_frame. Then I simply used the named, subset command to reorder and select the final fields for the data frame. 
colnames(article_frame) <- c("copyright" ,"web_url" , "snippet" ,"rprint_page" ,     
                             "rheadline" ,"keywords" ,  "pub_date" ,"document_type" ,  
                             "new_desk" ,"section_name" ,  "byline" ,"type_of_material",
                             "rdocs_id" , "word_count" ,  "score"  ,"docs_uri"  ,           
                             "meta_hits" ,"meta_offset" ,  "meta_time" )
byline<- data.frame(by_line= article_frame$byline$original)
headlines <- data.frame(cbind(article_frame$rheadline$main, article_frame$rheadline$print_headline, article_frame$rheadline$kicker))
colnames(headlines)<- c("main_headline", "print_headline", "kicker" )

final_frame <- cbind(article_frame, headlines, byline)
final_frame <- final_frame[, c("copyright" ,"web_url" , "snippet" ,"rprint_page" ,     
                               "main_headline", "print_headline", "kicker", "pub_date","document_type" ,  
                               "new_desk" ,"section_name" , "by_line" ,"type_of_material",
                               "rdocs_id" , "word_count" , "score"  ,"docs_uri" ,           
                               "meta_hits" ,"meta_offset" , "meta_time" )]

Final Table

Resulting Data

Here is a sortable data table of the resulting 10 articles returned.

knitr::kable(final_frame)
copyright web_url snippet rprint_page main_headline print_headline kicker pub_date document_type new_desk section_name by_line type_of_material rdocs_id word_count score docs_uri meta_hits meta_offset meta_time
Copyright (c) 2017 The New York Times Company. All Rights Reserved. https://www.nytimes.com/2017/10/07/world/europe/ukraine-russia-manafort-corruption.html Russian hacking, secret payments to Paul Manafort, suspicions of missile technology smuggled to North Korea — seemingly all illicit roads lead back to one country. 10 Schooled in Scandal: What Makes Ukraine a Hotbed of Intrigue From Manafort to Russia, Intrigue Finds a Warm Home in Ukraine NA 2017-10-07T12:24:26+0000 article Foreign Europe By ANDREW HIGGINS and ANDREW E. KRAMER News 59d8c78595d0e0246f20e900 1800 3.971438 nyt://article/5d0402e2-ce49-546e-91e9-866e2a7f4e0c 246 0 16
Copyright (c) 2017 The New York Times Company. All Rights Reserved. https://www.nytimes.com/2017/09/21/us/politics/law-firm-faces-questions-for-ukraine-work-with-manafort.html The Justice Department wants information about Skadden’s work on behalf of a Russia-aligned former president of Ukraine. 18 Skadden, Big New York Law Firm, Faces Questions on Work With Manafort New York Law Firm Faces Questions on Work Arranged by Manafort NA 2017-09-21T17:42:35+0000 article Washington Politics By KENNETH P. VOGEL and ANDREW E. KRAMER News 59c3fa1b95d0e0246f20b551 1349 3.360509 nyt://article/9a5343ae-4807-5733-8383-b24a6f8d09d8 246 0 16
Copyright (c) 2017 The New York Times Company. All Rights Reserved. https://www.nytimes.com/2017/09/22/world/europe/morgan-freeman-russia-video.html A Russian news outlet called the actor an “American propaganda loudspeaker” after he appeared in a video accusing Russia of undermining last year’s election. NA Morgan Freeman Angers Russians Over Video About 2016 Election Morgan Freeman Angers Russians Over Video About 2016 Election NA 2017-09-23T00:04:27+0000 article Foreign Europe By CHRISTOPHER MELE News 59c5a51495d0e0246f20bbe7 553 2.882524 nyt://article/f6ada17a-47c5-5e49-b7e4-d8d78c983d6b 246 0 16
Copyright (c) 2017 The New York Times Company. All Rights Reserved. https://www.nytimes.com/2017/09/20/us/politics/manafort-kurdish-referendum.html President Trump’s former campaign chairman has continued soliciting international business even as his past international work is under investigation. 18 Manafort Working on Kurdish Referendum Opposed by U.S. Under Investigation by the F.B.I., and Working Against the United States NA 2017-09-21T00:14:00+0000 article Washington Politics By KENNETH P. VOGEL and JO BECKER News 59c3044e95d0e0246f20acaf 1177 2.432215 nyt://article/2408b784-b6a1-5157-91b5-97ec30995dd4 246 0 16
Copyright (c) 2017 The New York Times Company. All Rights Reserved. https://www.nytimes.com/2017/10/22/world/europe/russia-sergei-magnitsky-william-browder.html William F. Browder campaigned for justice for Sergei Magnitsky, a tax lawyer who died in jail. Now Russian prosecutors are accusing him of murder in the case. 8 Turning Tables in Magnitsky Case, Russia Accuses a Nemesis of Murder Conspiracy Claims Devour Magnitsky Murder Inquiry NA 2017-10-22T21:48:20+0000 article Foreign Europe By ANDREW E. KRAMER News 59ed122d95d0e0246f2117c8 757 2.387515 nyt://article/a6dd42b8-531a-5c81-9635-5aac88ceedd9 246 0 16
Copyright (c) 2017 The New York Times Company. All Rights Reserved. https://www.nytimes.com/2017/10/27/us/politics/trump-tower-veselnitskaya-russia.html The information that a Russian lawyer brought to a June 2016 meeting with Trump campaign aides had been discussed with a top Russian government official. 1 Talking Points Brought to Trump Tower Meeting Were Shared With Kremlin A Kremlin Link To a Memo Taken To Trump Tower NA 2017-10-27T16:57:30+0000 article Washington Politics By SHARON LaFRANIERE and ANDREW E. KRAMER News 59f3658495d0e0246f21291b 1474 2.328831 nyt://article/4316ba72-bcb3-53ca-981e-56d613102a97 246 0 16
Copyright (c) 2017 The New York Times Company. All Rights Reserved. https://www.nytimes.com/2017/09/18/us/politics/mueller-russia-investigation.html Robert S. Mueller III, the special counsel, has aggressively used warrants and subpoenas for the Russia investigation, sending a message in Washington. 1 With a Picked Lock and a Threatened Indictment, Mueller’s Inquiry Sets a Tone Mueller Adopts Aggressive Tone In Russia Inquiry NA 2017-09-18T22:55:24+0000 article Washington Politics By SHARON LaFRANIERE, MATT APUZZO and ADAM GOLDMAN News 59c04eff95d0e0246f2096f8 1644 2.247485 nyt://article/f7cd7922-571f-55b0-8073-4fefbf3943d0 246 0 16
Copyright (c) 2017 The New York Times Company. All Rights Reserved. https://www.nytimes.com/reuters/2017/10/25/us/politics/25reuters-usa-trump-russia-manafort.html The U.S. attorney’s office in Manhattan is investigating Paul Manafort for potential money laundering, according to media reports, adding to other federal and state probes targeting President Donald Trump’s former campaign manager. NA Manhattan U.S. Attorney Adds to Probes of Ex-Trump Aide Manafort: Reports Manhattan U.S. Attorney Adds to Probes of Ex-Trump Aide Manafort: Reports NA 2017-10-25T16:39:28+0000 article None Politics By REUTERS News 59f0be4895d0e0246f21212a 438 2.217107 nyt://article/f1e41ff5-a209-5f7a-80cf-9949bd08c83a 246 0 16
Copyright (c) 2017 The New York Times Company. All Rights Reserved. https://www.nytimes.com/2017/09/27/technology/twitter-russia-election.html Facebook has faced the most scrutiny over social media’s role in the 2016 election. But Twitter may have been used even more extensively by Russians. 1 Twitter, With Accounts Linked to Russia, to Face Congress Over Role in Election Twitter Seen as Key Battlefield In Russian Influence Campaign NA 2017-09-28T00:05:20+0000 article Business NA By DAISUKE WAKABAYASHI and SCOTT SHANE News 59cc3ccb95d0e0246f20cc90 1253 2.113224 nyt://article/e4de539b-04a8-5668-9325-ecaefe1d830c 246 0 16
Copyright (c) 2017 The New York Times Company. All Rights Reserved. https://www.nytimes.com/2017/09/23/opinion/sunday/facebook-zuckerberg-dowd.html Oops! Facebook admits it created a democracy-crushing Frankenstein monster that it can’t control. 9 Will Mark Zuckerberg ‘Like’ This Column? Will Zuck ‘Like’ This Column? Op-Ed Columnist 2017-09-23T18:44:13+0000 article OpEd Sunday Review By MAUREEN DOWD Op-Ed 59c6ab8595d0e0246f20bd35 1025 2.049768 nyt://article/265a70bc-8b52-59e5-9dfc-452838323f73 246 0 16