Accessing NYT API for analysis of the term domestic terrorist
Justin Herman
March 27, 2018
#rm(list=ls())
#install.packages("devtools")
#devtools::install_github("mkearney/nytimes")
library(data.table)
library(plyr)
library(tidyverse)
library(splitstackshape)## Warning: package 'splitstackshape' was built under R version 3.4.4library(magrittr)
library(rlang)
library(gridExtra)
library(knitr)
library(kableExtra)
library(data.table)
library(ggplot2)
library(RCurl)
library(httr)
library(rtimes)## Warning: package 'rtimes' was built under R version 3.4.4library(jsonlite)
library(tidyverse)Intro
- I found a really useful guide to accessing NYT API here http://www.storybench.org/working-with-the-new-york-times-api-in-r/
Set up a times key(doesn’t print)—–
Initial attempt
- This attempt was a failure in that it returned too many results
- I attmepted to use date filter
term <- "domestic+terrorist"
begin_date <- "20000420"
end_date <- "20160428"
baseurl <- paste0("http://api.nytimes.com/svc/search/v2/articlesearch.json?fq=body=",term,
"&begin_date=",begin_date,"&end_date=",end_date,
"&facet_filter=true&api-key=",NYTIMES_KEY, sep="")
initialQuery <- fromJSON(baseurl)
initialQuery[[3]][2]## $meta
## $meta$hits
## [1] 191555
##
## $meta$offset
## [1] 0
##
## $meta$time
## [1] 31Sys.sleep(1)- 417k repsonses, this is not the correct way to search for the term
ATTEMPT 2
Gabrielle helped me to understand how to query for multiple word phrases
Get request gives me an idea of my query limits
### another way
new_search <- "\"domestic terrorist\""
articleSearchURL <- "http://api.nytimes.com/svc/search/v2/articlesearch.json"
APIquery <- list("api-key"=NYTIMES_KEY,'q'=new_search)
rawArticle <- GET(articleSearchURL, query = APIquery)
rawArticle[[3]]## $`content-type`
## [1] "application/json;charset=UTF-8"
##
## $date
## [1] "Fri, 30 Mar 2018 14:13:01 GMT"
##
## $server
## [1] "nginx/1.12.2"
##
## $via
## [1] "kong/0.9.5"
##
## $`x-kong-proxy-latency`
## [1] "46"
##
## $`x-kong-upstream-latency`
## [1] "66"
##
## $`x-ratelimit-limit-day`
## [1] "1000"
##
## $`x-ratelimit-limit-second`
## [1] "1"
##
## $`x-ratelimit-remaining-day`
## [1] "559"
##
## $`x-ratelimit-remaining-second`
## [1] "0"
##
## $`transfer-encoding`
## [1] "chunked"
##
## $connection
## [1] "keep-alive"
##
## attr(,"class")
## [1] "insensitive" "list"Sys.sleep(1)Using my initial api request with correct search terms
- Encode the url becuase the term had a space
orig_url<- "http://api.nytimes.com/svc/search/v2/articlesearch.json?api-key="
term <- "\"domestic terrorist\""
baseurl <- paste0(orig_url,
NYTIMES_KEY,
'&q=',term,
"&facet_filter=true", sep="")
baseurl <- URLencode(baseurl)
initialQuery <- fromJSON(baseurl)
Sys.sleep(1)
initialQuery[[3]][2]## $meta
## $meta$hits
## [1] 247
##
## $meta$offset
## [1] 0
##
## $meta$time
## [1] 55- This worked and my query shows me that there are 247 repsonses
- In the next section I will loop through an api request and get all 247 responses
Create loop to record all metadata
- I took the loop from the walk through I posted earlier
domestic_terroirst <- list()
for(i in 0:24){
nytSearch <- fromJSON(paste0(baseurl, "&page=", i), flatten = TRUE) %>% data.frame()
domestic_terroirst[[i+1]] <- nytSearch
Sys.sleep(1)
}Build df by rbinding the results from the loop
- Display column names
domestic_terroirst_df <- rbind_pages(domestic_terroirst)
head(domestic_terroirst_df)## status
## 1 OK
## 2 OK
## 3 OK
## 4 OK
## 5 OK
## 6 OK
## copyright
## 1 Copyright (c) 2018 The New York Times Company. All Rights Reserved.
## 2 Copyright (c) 2018 The New York Times Company. All Rights Reserved.
## 3 Copyright (c) 2018 The New York Times Company. All Rights Reserved.
## 4 Copyright (c) 2018 The New York Times Company. All Rights Reserved.
## 5 Copyright (c) 2018 The New York Times Company. All Rights Reserved.
## 6 Copyright (c) 2018 The New York Times Company. All Rights Reserved.
## response.docs.web_url
## 1 https://www.nytimes.com/aponline/2018/03/29/us/ap-us-austin-bombings.html
## 2 https://www.nytimes.com/2016/02/28/books/review/a-doubters-almanac-by-ethan-canin.html
## 3 https://www.nytimes.com/2017/10/19/opinion/columnists/fbi-blacks-civil-rights.html
## 4 https://www.nytimes.com/2017/08/13/us/politics/charlottesville-sessions-justice-department.html
## 5 https://www.nytimes.com/2017/08/15/us/politics/right-wing-extremism-charlottesville.html
## 6 https://www.nytimes.com/2017/08/15/opinion/trump-obama-terrorism-charlottesville-isis.html
## response.docs.snippet
## 1 Austin's police chief said Thursday that a "domestic terrorist" set off a series of explosions that killed two people and severely wounded four others in Texas' capital, offering a stronger characterization of the suspected bomber after drawing cr...
## 2 A mathematics genius inflicts emotional and physical pain on his wife and children.
## 3 A chilling report with echoes of the bureau’s persecution of black activists a half century ago talks of the threat from “black identity extremists.”
## 4 The announcement of a civil rights investigation puts the spotlight on what role the Justice Department may play in such cases under Attorney General Jeff Sessions.
## 5 Shortly after taking office, the Trump administration pulled back a $400,000 grant to a Chicago-based group founded to combat right-wing white extremist groups.
## 6 We must call out evil in all its forms by name.
## response.docs.source
## 1 AP
## 2 The New York Times
## 3 The New York Times
## 4 The New York Times
## 5 The New York Times
## 6 The New York Times
## response.docs.multimedia
## 1 NULL
## 2 0, 0, 0, wide, xlarge, thumbnail, NA, NA, NA, NA, NA, NA, image, image, image, images/2016/02/28/books/review/28ULLMAN/0228-BKS-Ullman-thumbWide.jpg, images/2016/02/28/books/review/28ULLMAN/0228-BKS-Ullman-articleLarge.jpg, images/2016/02/28/books/review/28ULLMAN/0228-BKS-Ullman-thumbStandard.jpg, 126, 628, 75, 190, 600, 75, wide, xlarge, thumbnail, NA, NA, NA, images/2016/02/28/books/review/28ULLMAN/0228-BKS-Ullman-thumbWide.jpg, NA, NA, 126, NA, NA, 190, NA, NA, NA, 600, NA, NA, images/2016/02/28/books/review/28ULLMAN/0228-BKS-Ullman-articleLarge.jpg, NA, NA, 628, NA, NA, NA, 75, NA, NA, images/2016/02/28/books/review/28ULLMAN/0228-BKS-Ullman-thumbStandard.jpg, NA, NA, 75
## 3 0, 0, 0, xlarge, wide, thumbnail, NA, NA, NA, NA, NA, NA, image, image, image, images/2017/10/20/opinion/18rosenthalWeb/18rosenthalWeb-articleLarge.jpg, images/2017/10/20/opinion/18rosenthalWeb/18rosenthalWeb-thumbWide.jpg, images/2017/10/20/opinion/18rosenthalWeb/18rosenthalWeb-thumbStandard.jpg, 400, 126, 75, 600, 190, 75, xlarge, wide, thumbnail, NA, NA, NA, 600, NA, NA, images/2017/10/20/opinion/18rosenthalWeb/18rosenthalWeb-articleLarge.jpg, NA, NA, 400, NA, NA, NA, images/2017/10/20/opinion/18rosenthalWeb/18rosenthalWeb-thumbWide.jpg, NA, NA, 190, NA, NA, 126, NA, NA, NA, 75, NA, NA, images/2017/10/20/opinion/18rosenthalWeb/18rosenthalWeb-thumbStandard.jpg, NA, NA, 75
## 4 0, 0, 0, xlarge, wide, thumbnail, NA, NA, NA, NA, NA, NA, image, image, image, images/2017/08/14/us/14dc-hatecrimes/14dc-hatecrimes-articleLarge.jpg, images/2017/08/14/us/14dc-hatecrimes/14dc-hatecrimes-thumbWide.jpg, images/2017/08/14/us/14dc-hatecrimes/14dc-hatecrimes-thumbStandard.jpg, 400, 126, 75, 600, 190, 75, xlarge, wide, thumbnail, NA, NA, NA, 600, NA, NA, images/2017/08/14/us/14dc-hatecrimes/14dc-hatecrimes-articleLarge.jpg, NA, NA, 400, NA, NA, NA, images/2017/08/14/us/14dc-hatecrimes/14dc-hatecrimes-thumbWide.jpg, NA, NA, 190, NA, NA, 126, NA, NA, NA, 75, NA, NA, images/2017/08/14/us/14dc-hatecrimes/14dc-hatecrimes-thumbStandard.jpg, NA, NA, 75
## 5 0, 0, 0, xlarge, wide, thumbnail, NA, NA, NA, NA, NA, NA, image, image, image, images/2017/08/16/us/00DC-Extremism-1/merlin-to-scoop-125979140-689952-articleLarge.jpg, images/2017/08/16/us/00DC-Extremism-1/merlin-to-scoop-125979140-689952-thumbWide.jpg, images/2017/08/16/us/00DC-Extremism-1/00DC-Extremism-1-thumbStandard.jpg, 400, 126, 75, 600, 190, 75, xlarge, wide, thumbnail, NA, NA, NA, 600, NA, NA, images/2017/08/16/us/00DC-Extremism-1/merlin-to-scoop-125979140-689952-articleLarge.jpg, NA, NA, 400, NA, NA, NA, images/2017/08/16/us/00DC-Extremism-1/merlin-to-scoop-125979140-689952-thumbWide.jpg, NA, NA, 190, NA, NA, 126, NA, NA, NA, 75, NA, NA, images/2017/08/16/us/00DC-Extremism-1/00DC-Extremism-1-thumbStandard.jpg, NA, NA, 75
## 6 0, 0, 0, xlarge, wide, thumbnail, NA, NA, NA, NA, NA, NA, image, image, image, images/2017/08/15/opinion/15Stephens-web/15Stephens-web-articleLarge.jpg, images/2017/08/15/opinion/15Stephens-web/15Stephens-web-thumbWide.jpg, images/2017/08/15/opinion/15Stephens-web/15Stephens-web-thumbStandard.jpg, 400, 126, 75, 600, 190, 75, xlarge, wide, thumbnail, NA, NA, NA, 600, NA, NA, images/2017/08/15/opinion/15Stephens-web/15Stephens-web-articleLarge.jpg, NA, NA, 400, NA, NA, NA, images/2017/08/15/opinion/15Stephens-web/15Stephens-web-thumbWide.jpg, NA, NA, 190, NA, NA, 126, NA, NA, NA, 75, NA, NA, images/2017/08/15/opinion/15Stephens-web/15Stephens-web-thumbStandard.jpg, NA, NA, 75
## response.docs.keywords
## 1 NULL
## 2 subject, persons, creative_works, Books and Literature, Canin, Ethan, A Doubter's Almanac (Book), 1, 2, 3, NA, NA, NA
## 3 organizations, subject, subject, subject, organizations, subject, subject, Federal Bureau of Investigation, Blacks, Police Brutality, Misconduct and Shootings, Race and Ethnicity, Black Panther Party, Surveillance of Citizens by Government, Discrimination, 1, 2, 3, 4, 5, 6, 7, N, N, N, N, N, N, N
## 4 subject, subject, glocations, subject, persons, subject, subject, organizations, persons, persons, Charlottesville, Va, Violence (August, 2017), Murders, Attempted Murders and Homicides, Charlottesville (Va), Hate Crimes, Sessions, Jefferson B III, Law and Legislation, Terrorism, Justice Department, Fields, James Alex Jr (1997- ), Heyer, Heather D (1985-2017), 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, N, N, N, N, N, N, N, N, N, N
## 5 subject, subject, subject, organizations, persons, persons, persons, glocations, United States Politics and Government, Fringe Groups and Movements, Neo Nazi Groups, Homeland Security Department, Trump, Donald J, Johnson, Jeh C, Gorka, Sebastian, Chicago (Ill), 1, 2, 3, 4, 5, 6, 7, 8, N, N, N, N, N, N, N, N
## 6 subject, persons, glocations, persons, subject, subject, subject, Charlottesville, Va, Violence (August, 2017), Trump, Donald J, Charlottesville (Va), Fields, James Alex Jr (1997- ), United States Politics and Government, Terrorism, Muslims and Islam, 1, 2, 3, 4, 5, 6, 7, N, N, N, N, N, N, N
## response.docs.pub_date response.docs.document_type
## 1 2018-03-29T15:39:05+0000 article
## 2 2016-02-28T00:00:00Z article
## 3 2017-10-19T08:55:01+0000 article
## 4 2017-08-13T22:34:17+0000 article
## 5 2017-08-15T21:30:26+0000 article
## 6 2017-08-15T07:21:05+0000 article
## response.docs.new_desk response.docs.type_of_material
## 1 None News
## 2 BookReview Review
## 3 OpEd Op-Ed
## 4 National News
## 5 Washington News
## 6 OpEd Op-Ed
## response.docs._id response.docs.word_count response.docs.score
## 1 5abd089c47de81a901219619 339 0.3267278700
## 2 56cb439c38f0d80fb3eb21c2 1263 0.0008585348
## 3 59e8686c95d0e0246f210e5e 753 0.0006722512
## 4 5990d3f795d0e0246f1ff8c3 983 0.0004800893
## 5 599367f795d0e0246f1ffe25 1180 0.0004774901
## 6 5992a0e995d0e0246f1ffc18 820 0.0004243940
## response.docs.uri
## 1 nyt://article/d79b47d5-afac-5363-ad38-294c3549cbea
## 2 <NA>
## 3 nyt://article/70be092d-485b-57f6-8cea-f6315dd572cd
## 4 nyt://article/293a0f00-2efe-56aa-a7b2-20318b0136e7
## 5 nyt://article/82792d2f-482f-5a0c-a8f4-eff7528c44b0
## 6 nyt://article/37575477-795e-5430-9477-e0901fdf5b08
## response.docs.abstract
## 1 <NA>
## 2 Ellen Ullman reviews novel A Doubter's Almanac by Ethan Canin.
## 3 <NA>
## 4 <NA>
## 5 <NA>
## 6 <NA>
## response.docs.print_page response.docs.section_name
## 1 <NA> <NA>
## 2 13 Book Review
## 3 <NA> Columnists
## 4 14 Politics
## 5 15 Politics
## 6 <NA> <NA>
## response.docs.headline.main
## 1 Austin Police Chief Calls Bomber 'Domestic Terrorist'
## 2 ‘A Doubter’s Almanac,’ by Ethan Canin
## 3 The F.B.I.’s Black Phantom Menace
## 4 A Hate Crime? How the Charlottesville Car Attack May Become a Federal Case
## 5 Revocation of Grants to Help Fight Hate Under New Scrutiny After Charlottesville
## 6 Trump, Obama and the Politics of Evasion
## response.docs.headline.kicker response.docs.headline.content_kicker
## 1 <NA> <NA>
## 2 <NA> <NA>
## 3 Op-Ed Columnist Op-Ed Columnist
## 4
## 5 <NA> <NA>
## 6 Op-Ed Columnist Op-Ed Columnist
## response.docs.headline.print_headline
## 1 Austin Police Chief Calls Bomber 'Domestic Terrorist'
## 2 Domestic Terrorist
## 3
## 4 Was the Car Attack a Hate Crime? How the Justice Dept. May View It
## 5 U.S. Rescinded Grant To Combat Extremism Of Right-Wing Groups
## 6 Trump, Obama and the Politics of Evasion
## response.docs.headline.name response.docs.headline.seo
## 1 NA <NA>
## 2 NA <NA>
## 3 NA <NA>
## 4 NA <NA>
## 5 NA <NA>
## 6 NA <NA>
## response.docs.headline.sub response.docs.byline.original
## 1 <NA> By THE ASSOCIATED PRESS
## 2 <NA> By ELLEN ULLMAN
## 3 <NA> By ANDREW ROSENTHAL
## 4 <NA> By CHARLIE SAVAGE
## 5 <NA> By RON NIXON and EILEEN SULLIVAN
## 6 <NA> By BRET STEPHENS
## response.docs.byline.person
## 1 NULL
## 2 Ellen, NA, ULLMAN, NA, NA, reported, , 1
## 3 Andrew, NA, ROSENTHAL, NA, NA, reported, , 1
## 4 Charlie, NA, SAVAGE, NA, NA, reported, , 1
## 5 Ron, Eileen, NA, NA, NIXON, SULLIVAN, NA, NA, NA, NA, reported, reported, , , 1, 2
## 6 Bret, NA, STEPHENS, NA, NA, reported, , 1
## response.docs.byline.organization response.meta.hits
## 1 THE ASSOCIATED PRESS 247
## 2 <NA> 247
## 3 <NA> 247
## 4 <NA> 247
## 5 <NA> 247
## 6 <NA> 247
## response.meta.offset response.meta.time
## 1 0 14
## 2 0 14
## 3 0 14
## 4 0 14
## 5 0 14
## 6 0 14rm(domestic_terroirst)Exploratory analysis
- I love this chooseone function, credit to Amber Thomas, she provided it in a kaggle competition and I used it in project 3
- I lapply it for some summary analysis
chooseOne = function(question){
domestic_terroirst_df %>%
filter(!UQ(sym(question)) == "") %>%
dplyr::group_by_(question) %>%
dplyr::summarise(count = n()) %>%
dplyr::mutate(percent = (count / sum(count)) * 100) %>%
dplyr::arrange(desc(count))
}
my_names <- colnames(domestic_terroirst_df)
the_names <- my_names[(c(2,3,10,11,12,15,19,20,25))]
lapply(the_names,function(x)chooseOne(x))## [[1]]
## # A tibble: 1 x 3
## copyright count perce~
## <fctr> <int> <dbl>
## 1 Copyright (c) 2018 The New York Times Company. All Rights ~ 247 100
##
## [[2]]
## # A tibble: 243 x 3
## response.docs.web_url count perce~
## <chr> <int> <dbl>
## 1 https://artsbeat.blogs.nytimes.com/2010/05/03/the-return-~ 2 0.810
## 2 https://cityroom.blogs.nytimes.com/2011/09/29/a-tribute-t~ 2 0.810
## 3 https://www.nytimes.com/2009/12/20/opinion/20rich.html 2 0.810
## 4 https://www.nytimes.com/2012/05/21/world/europe/italian-s~ 2 0.810
## 5 https://atwar.blogs.nytimes.com/2009/12/09/american-suppo~ 1 0.405
## 6 https://bittman.blogs.nytimes.com/2013/09/10/on-911-my-tw~ 1 0.405
## 7 https://carpetbagger.blogs.nytimes.com/2010/12/01/a-new-c~ 1 0.405
## 8 https://douthat.blogs.nytimes.com/2010/11/30/how-partisan~ 1 0.405
## 9 https://economix.blogs.nytimes.com/2013/08/12/the-fiscal-~ 1 0.405
## 10 https://ethicist.blogs.nytimes.com/2009/11/17/free-speech~ 1 0.405
## # ... with 233 more rows
##
## [[3]]
## # A tibble: 29 x 3
## response.docs.new_desk count percent
## <chr> <int> <dbl>
## 1 National Desk 48 22.7
## 2 National 32 15.2
## 3 Foreign 22 10.4
## 4 OpEd 19 9.00
## 5 Metropolitan Desk 14 6.64
## 6 Foreign Desk 12 5.69
## 7 Editorial Desk 10 4.74
## 8 Metro 7 3.32
## 9 Culture 6 2.84
## 10 Editorial 6 2.84
## # ... with 19 more rows
##
## [[4]]
## # A tibble: 15 x 3
## response.docs.type_of_material count percent
## <chr> <int> <dbl>
## 1 News 148 59.9
## 2 Blog 36 14.6
## 3 Op-Ed 26 10.5
## 4 Review 10 4.05
## 5 Summary 9 3.64
## 6 Editorial 4 1.62
## 7 An Analysis; News Analysis 2 0.810
## 8 Article 2 0.810
## 9 Biography 2 0.810
## 10 Front Page 2 0.810
## 11 Letter 2 0.810
## 12 Brief 1 0.405
## 13 Chronology 1 0.405
## 14 Obituary (Obit) 1 0.405
## 15 Text 1 0.405
##
## [[5]]
## # A tibble: 247 x 3
## response.docs._id count percent
## <chr> <int> <dbl>
## 1 4fc4931945c1498b0da680dc 1 0.405
## 2 4fc49bb945c1498b0da8db1a 1 0.405
## 3 4fc4a2a245c1498b0daac6ec 1 0.405
## 4 4fc4a32445c1498b0daae8d7 1 0.405
## 5 4fd115358eb7c8105d5cb186 1 0.405
## 6 4fd1170a8eb7c8105d5ce428 1 0.405
## 7 4fd136a28eb7c8105d608803 1 0.405
## 8 4fd149fd8eb7c8105d6273c6 1 0.405
## 9 4fd14fd88eb7c8105d630bf1 1 0.405
## 10 4fd1720e8eb7c8105d665789 1 0.405
## # ... with 237 more rows
##
## [[6]]
## # A tibble: 30 x 3
## response.docs.uri count percent
## <chr> <int> <dbl>
## 1 nyt://article/01ed2997-5dfd-5f21-875a-4ae589ca913d 1 3.33
## 2 nyt://article/0854e100-f844-52f7-8fd7-ffd74d0469d2 1 3.33
## 3 nyt://article/0a7a086d-1dcd-5f97-9cc0-58cb30de48b5 1 3.33
## 4 nyt://article/1562239c-63d0-5468-a28a-e2d7e3999445 1 3.33
## 5 nyt://article/1d9a8eb2-68f1-5f91-b2e1-a9856ec732dd 1 3.33
## 6 nyt://article/1dda5a39-9ef4-5e7d-afab-27cb34acb154 1 3.33
## 7 nyt://article/293a0f00-2efe-56aa-a7b2-20318b0136e7 1 3.33
## 8 nyt://article/37341795-8949-521e-8354-ac44fac175f3 1 3.33
## 9 nyt://article/37575477-795e-5430-9477-e0901fdf5b08 1 3.33
## 10 nyt://article/4b78e65a-a864-54b6-9064-4232c679a2ec 1 3.33
## # ... with 20 more rows
##
## [[7]]
## # A tibble: 236 x 3
## response.docs.headline.main count perc~
## <chr> <int> <dbl>
## 1 NEWS SUMMARY 6 2.43
## 2 A Rally Cry Echoes Online 2 0.810
## 3 A Tribute to a Reporter With Legendary Drive and Swagger 2 0.810
## 4 Fatal Bombing at Italian School Is Thought to Be the Work ~ 2 0.810
## 5 Leftward, Ho? 2 0.810
## 6 NEW YORK DIRECTOR OF F.B.I. IS REMOVED ON BREAK-INCHARGES;~ 2 0.810
## 7 Tiger Woods, Person of the Year 2 0.810
## 8 'Big Trouble' 1 0.405
## 9 'This Whole Transition Thing' 1 0.405
## 10 "\u2018A Doubter\u2019s Almanac,\u2019 by Ethan Canin" 1 0.405
## # ... with 226 more rows
##
## [[8]]
## # A tibble: 76 x 3
## response.docs.headline.kicker count percent
## <chr> <int> <dbl>
## 1 The Caucus 9 8.41
## 2 Op-Ed Columnist 6 5.61
## 3 Op-Ed Contributor 4 3.74
## 4 The Lede 4 3.74
## 5 Opinionator 3 2.80
## 6 THREATS AND RESPONSES: DOMESTIC SECURITY 3 2.80
## 7 ArtsBeat 2 1.87
## 8 City Room 2 1.87
## 9 Editorial 2 1.87
## 10 India Ink 2 1.87
## # ... with 66 more rows
##
## [[9]]
## # A tibble: 4 x 3
## response.docs.headline.sub count perce~
## <chr> <int> <dbl>
## 1 6 Arrested in Canada Raid Attended the Same Mosque 1 25.0
## 2 A Plague of Stunt Casting Is Being Spread Over the Airwave~ 1 25.0
## 3 Bank Data Is Sifted by U.S. in Secret to Block Terror 1 25.0
## 4 Lawmaker Will Begin Look Into Homegrown Islamic Terrorism; 1 25.0Create visual displays
domestic_terroirst_df %>%
filter(!UQ(sym('response.docs.new_desk')) == "") %>%
dplyr::group_by(response.docs.new_desk) %>%
dplyr::summarize(count=n()) %>%
ggplot() +
geom_bar(aes(y=count, x=reorder(response.docs.new_desk,count), fill=fct_infreq(response.docs.new_desk)), stat = "identity") +
coord_flip()+
theme(legend.position="none")
domestic_terroirst_df %>%
filter(!UQ(sym('response.docs.section_name')) == "") %>%
dplyr::group_by(response.docs.section_name) %>%
dplyr::summarize(count=n()) %>%
dplyr::mutate(percent = (count / sum(count))*100) %>%
ggplot() +
geom_bar(aes(y=percent, x=reorder(response.docs.section_name,count), fill=response.docs.section_name), stat = "identity") +
coord_flip()+
theme(legend.position="none")
domestic_terroirst_df %>%
dplyr::group_by(response.docs.type_of_material) %>%
dplyr::summarize(count=n()) %>%
dplyr::mutate(percent = (count / sum(count))*100) %>%
ggplot() +
geom_bar(aes(y=percent, x=reorder(response.docs.type_of_material,count), fill=response.docs.type_of_material), stat = "identity") + coord_flip()+ theme(legend.position="none")
Attempt at time series results graph
domestic_terroirst_df$response.docs.pub_date <- as.Date(gsub("T.*", "", domestic_terroirst_df$response.docs.pub_date
))
ggplot(domestic_terroirst_df,aes(x=response.docs.pub_date)) +stat_bin(aes(y = cumsum(..count..)),binwidth = 1)
Now that I have an idea of what columns I want, I will make a request for more specified feilds
- Just a test run
cleaner_domestic_terroirst <- list()
fields <- c("headline", "web_url", "abstract","news_desk","word_count" ,"pub_date")
orig_url<- "http://api.nytimes.com/svc/search/v2/articlesearch.json?api-key="
term <- "\"domestic terrorist\""
baseurl <- paste0(orig_url,NYTIMES_KEY,
'&q=',term,
"&fl=", paste(fields, collapse = ","),
sep="")
## Test Run
baseurl <- URLencode(baseurl)
df_2 <- fromJSON(baseurl, flatten = TRUE)
my_df <- df_2$response$docs
dim(my_df)## [1] 10 13my_df## web_url
## 1 https://www.nytimes.com/aponline/2018/03/29/us/ap-us-austin-bombings.html
## 2 https://www.nytimes.com/2016/02/28/books/review/a-doubters-almanac-by-ethan-canin.html
## 3 https://www.nytimes.com/2017/10/19/opinion/columnists/fbi-blacks-civil-rights.html
## 4 https://www.nytimes.com/2017/08/15/us/politics/right-wing-extremism-charlottesville.html
## 5 https://www.nytimes.com/2017/08/13/us/politics/charlottesville-sessions-justice-department.html
## 6 https://www.nytimes.com/2017/08/15/opinion/trump-obama-terrorism-charlottesville-isis.html
## 7 https://www.nytimes.com/2017/08/13/us/charlottesville-protests-white-nationalists-trump.html
## 8 https://www.nytimes.com/2017/06/15/us/politics/before-the-shooting-assailant-began-day-at-ymca-as-usual.html
## 9 https://www.nytimes.com/2017/08/01/opinion/womens-march-progressives-hate.html
## 10 https://www.nytimes.com/2017/08/13/opinion/university-virginia-uva-protests-charlottesville.html
## pub_date new_desk word_count score
## 1 2018-03-29T15:39:05+0000 None 339 0.3249993300
## 2 2016-02-28T00:00:00Z BookReview 1263 0.0008542584
## 3 2017-10-19T08:55:01+0000 OpEd 753 0.0006835898
## 4 2017-08-15T21:30:26+0000 Washington 1180 0.0004799950
## 5 2017-08-13T22:34:17+0000 National 983 0.0004764059
## 6 2017-08-15T07:21:05+0000 OpEd 820 0.0004303845
## 7 2017-08-13T17:13:45+0000 National 1527 0.0002912214
## 8 2017-06-15T16:18:33+0000 National 846 0.0002860388
## 9 2017-08-01T14:30:56+0000 OpEd 1826 0.0002677684
## 10 2017-08-13T20:21:24+0000 OpEd 2199 0.0002524908
## abstract
## 1 <NA>
## 2 Ellen Ullman reviews novel A Doubter's Almanac by Ethan Canin.
## 3 <NA>
## 4 <NA>
## 5 <NA>
## 6 <NA>
## 7 <NA>
## 8 <NA>
## 9 <NA>
## 10 <NA>
## headline.main
## 1 Austin Police Chief Calls Bomber 'Domestic Terrorist'
## 2 ‘A Doubter’s Almanac,’ by Ethan Canin
## 3 The F.B.I.’s Black Phantom Menace
## 4 Revocation of Grants to Help Fight Hate Under New Scrutiny After Charlottesville
## 5 A Hate Crime? How the Charlottesville Car Attack May Become a Federal Case
## 6 Trump, Obama and the Politics of Evasion
## 7 White House Acts to Stem Fallout From Trump’s First Charlottesville Remarks
## 8 Before Shooting, Suspect Began Day at Y.M.C.A. as Usual
## 9 When Progressives Embrace Hate
## 10 What U.Va. Students Saw in Charlottesville
## headline.kicker headline.content_kicker
## 1 <NA> <NA>
## 2 <NA> <NA>
## 3 Op-Ed Columnist Op-Ed Columnist
## 4 <NA> <NA>
## 5
## 6 Op-Ed Columnist Op-Ed Columnist
## 7 <NA> <NA>
## 8 <NA> <NA>
## 9 Opinion Opinion
## 10 On Campus On Campus
## headline.print_headline
## 1 Austin Police Chief Calls Bomber 'Domestic Terrorist'
## 2 Domestic Terrorist
## 3
## 4 U.S. Rescinded Grant To Combat Extremism Of Right-Wing Groups
## 5 Was the Car Attack a Hate Crime? How the Justice Dept. May View It
## 6 Trump, Obama and the Politics of Evasion
## 7 A White House Statement on Virginia Is Also Found Wanting
## 8 Suspect in Shooting Began Day With Routine Predawn Stop at Y.M.C.A.
## 9 When Progressives Embrace Hate
## 10 What U.Va. Students Saw in Charlottesville
## headline.name headline.seo headline.sub
## 1 NA NA NA
## 2 NA NA NA
## 3 NA NA NA
## 4 NA NA NA
## 5 NA NA NA
## 6 NA NA NA
## 7 NA NA NA
## 8 NA NA NA
## 9 NA NA NA
## 10 NA NA NAConclusion
- I really wanted to go further with this, and may do so during spring break. I will attempt to look into doing some time series analysis on the API results, and compare to other terms.