Assignment Overview

The assignment was to use the NY Times API to read in some data, and then transform that data in an R dataframe, that could be then queried as desired.

The basic functions to access the API and transform to a dataframe are below and are compact and simple. And should hopefully be mostly self explanatory. Comments will be given when we call these functions.

library(RCurl)
## Loading required package: bitops
library(stringr)
library(RJSONIO)
## Warning: package 'RJSONIO' was built under R version 3.3.1
readApiKeyFromFile <-function(fileName)
{
    apiKey <- scan(fileName, what = "character")
}

readTimesAPI <- function(URL, key, q, pageNumber)
{
    params <- str_c(URL, "q=", q, "&page=", pageNumber, "&apiKey=", key)
    return(getURI(params))
}

parseJSON <-function (results)
{
    return(fromJSON(results, simplify = FALSE))
}

#This puts all the fields from a given row into a one row df
pullFields <- function(docRow) 
{           #rbind, gets around dataframe creation errors, even though we aren't joining anyrows together here
    as.data.frame(rbind(docRow[seq_along(1:length(docRow))]), stringsAsFactors = FALSE)
}

#This takes the passed in "document section" from the parsed JSON and creates a dataframe of it 
makeDF <- function(doc)
{
    
    df <- pullFields(doc[[1]])
    for(i in 2:length(doc))
    {
        df <- rbind(df, pullFields(doc[[i]]))
    }
    return(df)
}

First We will populate some variables.

  • NY Times URL
  • Read an my APIKEY from a file (so if rerunning this you will need to poulate a file with your personal API key.
  • The query string we wish to perform a search for.
  • The page number of query results we want to retrieve (the api returns 10 rows per “page”" of data)
timesURL <- "https://api.nytimes.com/svc/search/v2/articlesearch.json?"
apiKey <- readApiKeyFromFile("timesAPIKey.txt")
query <- "brexit"
page <- 2

Next we will call three functions to get a dataframe. The first is to read the times api, using the variables we populated, and we will pass the results right into the parseJSON function which maniuplates the JSON from the API into an easier to work with format (a list). Lastly call a function that (perhaps not efficiently) creates a dataframe by going row by row of the list and concatenating with RBind into a dataframe.

parsed <- parseJSON(readTimesAPI(timesURL, apiKey, query, page))

df <- makeDF(parsed$response$docs)

That is the entire program. Lets run some commands to show that it actually works. The below will show the head of the Dataframe, and then query a couple of the columns to show what was returned.

head(df)
##                                                                                                                  web_url
## 1 http://www.nytimes.com/2016/10/17/world/europe/without-a-constitution-brexit-is-guided-by-a-prerogative-but-whose.html
## 2                   http://www.nytimes.com/2016/10/16/business/mutfund/brexit-blurs-the-european-investment-outlook.html
## 3                            http://www.nytimes.com/2016/09/16/business/international/bank-of-england-brexit-carney.html
## 4                                         http://www.nytimes.com/2016/09/15/world/europe/theresa-may-britain-brexit.html
## 5                                             http://www.nytimes.com/2016/09/26/sports/golf/ryder-cup-brexit-europe.html
## 6         http://www.nytimes.com/2016/11/01/business/dealbook/mark-carney-to-lead-bank-of-england-through-june-2019.html
##                                                                                                                                                                   snippet
## 1         A court is considering whether a set of archaic rules known as the royal prerogative allows British leaders to bypass Parliament in leaving the European Union.
## 2                        Though markets have stabilized since the vote for Britain to leave the European Union, where investors and businesses go from here is uncertain.
## 3                   Mr. Carney, the first foreigner to head Britain<U+0092>s central bank, is a crucial figure in the effort to manage the fallout of the so-called Brexit vote.
## 4                                        David Cameron<U+0092>s successor has said little about how her country will withdraw from the European Union, and tough decisions loom.
## 5 After Britain<U+0092>s vote to leave the European Union, protocol changes and a shift in public perception may be in the offing. But some see it as a non-issue for the event.
## 6                                 Some lawmakers who favored an exit from the European Union had called for Mr. Carney to resign, saying he politicized the central bank.
##                                                                                                                                                            lead_paragraph
## 1         A court is considering whether a set of archaic rules known as the royal prerogative allows British leaders to bypass Parliament in leaving the European Union.
## 2                        Though markets have stabilized since the vote for Britain to leave the European Union, where investors and businesses go from here is uncertain.
## 3                   Mr. Carney, the first foreigner to head Britain<U+0092>s central bank, is a crucial figure in the effort to manage the fallout of the so-called Brexit vote.
## 4                                        David Cameron<U+0092>s successor has said little about how her country will withdraw from the European Union, and tough decisions loom.
## 5 After Britain<U+0092>s vote to leave the European Union, protocol changes and a shift in public perception may be in the offing. But some see it as a non-issue for the event.
## 6                                 Some lawmakers who favored an exit from the European Union had called for Mr. Carney to resign, saying he politicized the central bank.
##                                                                                                                                                                                                                                                     abstract
## 1 Memo From Britain; legal case opens at Royal Courts of Justice that could possibly keep Britain inside 28-nation European Union, at least pending more comprehensive plan for withdrawal; case highlights peculiarity of Britain's unwritten constitution.
## 2                                                                                                                                                                                                                                                       NULL
## 3                                                                                                                                                                                                                                                       NULL
## 4                                                                                                                                                                                                                                                       NULL
## 5                                                                                                                                                                                                                                                       NULL
## 6                                                                                                                                                                                                                                                       NULL
##   print_page blog             source
## 1          7 NULL                   
## 2         18 NULL                   
## 3          1 NULL                   
## 4         11 NULL                   
## 5          4 NULL                   
## 6          6 NULL The New York Times
##                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       multimedia
## 1                   190, images/2016/10/17/world/17ARTICLE50/17ARTICLE50-thumbWide.jpg, 126, wide, images/2016/10/17/world/17ARTICLE50/17ARTICLE50-thumbWide.jpg, 126, 190, image, 600, images/2016/10/17/world/17ARTICLE50/17ARTICLE50-articleLarge.jpg, 409, xlarge, 600, images/2016/10/17/world/17ARTICLE50/17ARTICLE50-articleLarge.jpg, 409, image, 75, images/2016/10/17/world/17ARTICLE50/17ARTICLE50-thumbStandard.jpg, 75, thumbnail, 75, images/2016/10/17/world/17ARTICLE50/17ARTICLE50-thumbStandard.jpg, 75, image
## 2                                     190, images/2016/10/16/business/16BREXIT/16BREXIT-thumbWide.jpg, 126, wide, images/2016/10/16/business/16BREXIT/16BREXIT-thumbWide.jpg, 126, 190, image, 600, images/2016/10/16/business/16BREXIT/16BREXIT-articleLarge.jpg, 365, xlarge, 600, images/2016/10/16/business/16BREXIT/16BREXIT-articleLarge.jpg, 365, image, 75, images/2016/10/16/business/16BREXIT/16BREXIT-thumbStandard.jpg, 75, thumbnail, 75, images/2016/10/16/business/16BREXIT/16BREXIT-thumbStandard.jpg, 75, image
## 3                                     190, images/2016/09/15/business/16CARNEY/16CARNEY-thumbWide.jpg, 126, wide, images/2016/09/15/business/16CARNEY/16CARNEY-thumbWide.jpg, 126, 190, image, 600, images/2016/09/15/business/16CARNEY/16CARNEY-articleLarge.jpg, 400, xlarge, 600, images/2016/09/15/business/16CARNEY/16CARNEY-articleLarge.jpg, 400, image, 75, images/2016/09/15/business/16CARNEY/16CARNEY-thumbStandard.jpg, 75, thumbnail, 75, images/2016/09/15/business/16CARNEY/16CARNEY-thumbStandard.jpg, 75, image
## 4                                           190, images/2016/09/15/world/15May-web/15May-web-thumbWide.jpg, 126, wide, images/2016/09/15/world/15May-web/15May-web-thumbWide.jpg, 126, 190, image, 600, images/2016/09/15/world/15May-web/15May-web-articleLarge.jpg, 401, xlarge, 600, images/2016/09/15/world/15May-web/15May-web-articleLarge.jpg, 401, image, 75, images/2016/09/15/world/15May-web/15May-web-thumbStandard.jpg, 75, thumbnail, 75, images/2016/09/15/world/15May-web/15May-web-thumbStandard.jpg, 75, image
## 5                               190, images/2016/09/26/sports/golf/26ryder/26ryder-thumbWide.jpg, 126, wide, images/2016/09/26/sports/golf/26ryder/26ryder-thumbWide.jpg, 126, 190, image, 600, images/2016/09/26/sports/golf/26ryder/26ryder-articleLarge.jpg, 400, xlarge, 600, images/2016/09/26/sports/golf/26ryder/26ryder-articleLarge.jpg, 400, image, 75, images/2016/09/26/sports/golf/26ryder/26ryder-thumbStandard.jpg, 75, thumbnail, 75, images/2016/09/26/sports/golf/26ryder/26ryder-thumbStandard.jpg, 75, image
## 6 190, images/2016/11/01/business/01DB-CARNEY/01DB-CARNEY-thumbWide.jpg, 126, wide, images/2016/11/01/business/01DB-CARNEY/01DB-CARNEY-thumbWide.jpg, 126, 190, image, 600, images/2016/11/01/business/01DB-CARNEY/01DB-CARNEY-articleLarge.jpg, 862, xlarge, 600, images/2016/11/01/business/01DB-CARNEY/01DB-CARNEY-articleLarge.jpg, 862, image, 75, images/2016/11/01/business/01DB-CARNEY/01DB-CARNEY-thumbStandard.jpg, 75, thumbnail, 75, images/2016/11/01/business/01DB-CARNEY/01DB-CARNEY-thumbStandard.jpg, 75, image
##                                                                                                                                                                                 headline
## 1 Without a Constitution, <U+0091>Brexit<U+0092> Is Guided by a Prerogative. But Whose?, Memo From Britain, Memo From Britain, Without a Constitution, a Prerogative Guides <U+0091>Brexit.<U+0092> But Whose Is It?
## 2                                                                                 <U+0091>Brexit<U+0092> Blurs the European Investment Outlook, <U+0091>Brexit<U+0092>s<U+0092> Potential Fallout Blurs European Investment
## 3                                                        As Britain Confronts <U+0091>Brexit,<U+0092> a Canadian Takes Center Stage: Mark Carney, Polished Outsider Takes Reins Amid Chaos of <U+0091>Brexit<U+0092>
## 4                                 Theresa May, Britain<U+0092>s Prime Minister, Plays the <U+0091>Brexit<U+0092> Waiting Game, Memo From Britain, Memo From Britain, Waiting to Choose Her Moment on <U+0091>Brexit<U+0092>
## 5                                              <U+0091>Brexit<U+0092> Places the Unity of Europe<U+0092>s Ryder Cup Team in Doubt, On Golf, On Golf, <U+0091>Brexit<U+0092> Opens the Door to a Ryder Cup Fissure in Europe
## 6                                                                                     Mark Carney to Lead Bank of England Through June 2019, Bank Governor Says He<U+0092>ll See Brexit Through
##                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    keywords
## 1                                                                                                                                                                                                                                                                                                                   1, N, subject, Great Britain Withdrawal from EU (Brexit), 2, Y, organizations, European Union, 4, N, persons, May, Theresa M, 5, Y, glocations, Great Britain, 6, N, glocations, Europe
## 2                                                                                                                                                                                                                                                                                                                                                      1, N, subject, Great Britain Withdrawal from EU (Brexit), 2, N, glocations, Europe, 3, N, subject, Stocks and Bonds, 4, N, glocations, Great Britain
## 3                                                                                                                                                                                                                                                    1, Y, organizations, Bank of England, 2, Y, persons, Carney, Mark J, 3, N, subject, Banking and Financial Institutions, 4, N, subject, Great Britain Withdrawal from EU (Brexit), 5, Y, glocations, Great Britain, 6, Y, organizations, European Union
## 4                                                                                                                                                                                                                 1, Y, persons, May, Theresa M, 2, N, subject, Great Britain Withdrawal from EU (Brexit), 3, Y, organizations, European Union, 4, N, persons, Johnson, Boris, 5, N, subject, International Relations, 6, N, subject, International Trade and World Market, 7, Y, glocations, Great Britain
## 5 1, N, subject, Golf, 2, N, subject, Ryder Cup (Golf), 3, N, subject, Great Britain Withdrawal from EU (Brexit), 4, N, organizations, European Union, 5, N, organizations, Professional Golfers Assn European Tour, 1, N, subject, Great Britain Withdrawal from EU (Brexit), 2, N, persons, Langer, Bernhard, 3, N, organizations, European Union, 4, N, persons, Farage, Nigel (1964- ), 5, N, subject, Ryder Cup (Golf), 6, N, glocations, Great Britain, 7, N, glocations, Europe, 8, N, subject, Golf
## 6                                                                                                                                                                                                                                                                                                                                                                             1, N, organizations, Bank of England, 2, N, persons, Carney, Mark J, 3, N, subject, Great Britain Withdrawal from EU (Brexit)
##               pub_date document_type      news_desk section_name
## 1 2016-10-16T23:48:23Z       article        Foreign        World
## 2 2016-10-16T04:00:00Z       article SundayBusiness Business Day
## 3 2016-09-16T04:00:00Z       article       Business Business Day
## 4 2016-09-15T04:00:00Z       article        Foreign        World
## 5 2016-09-26T04:00:00Z       article         Sports       Sports
## 6 2016-10-31T20:04:12Z       article       Business Business Day
##          subsection_name
## 1                 Europe
## 2           Mutual Funds
## 3 International Business
## 4                 Europe
## 5                   Golf
## 6               DealBook
##                                                      byline
## 1         , reported, Stephen, 1, CASTLE, By STEPHEN CASTLE
## 2     , reported, Conrad, 1, DE AENLLE, By CONRAD DE AENLLE
## 3    Peter, S., GOODMAN, 1, reported, , By PETER S. GOODMAN
## 4     , , reported, Steven, 1, ERLANGER, By STEVEN ERLANGER
## 5 , reported, Christopher, 1, CLAREY, By CHRISTOPHER CLAREY
## 6                   , reported, Chad, 1, BRAY, By CHAD BRAY
##   type_of_material                      _id word_count slideshow_credits
## 1             News 580411d07c459f606398a087       NULL              NULL
## 2             News 5801385b7c459f6063989cdf       NULL              NULL
## 3             News 57da20657c459f227dbab07e       NULL              NULL
## 4             News 57d9bbff7c459f227dbaaf7d       NULL              NULL
## 5             News 57e823007c459f227dbad552       NULL              NULL
## 6             News 5817a3c17c459f606398cc68        534              NULL
df$web_url[1:5]
## $web_url
## [1] "http://www.nytimes.com/2016/10/17/world/europe/without-a-constitution-brexit-is-guided-by-a-prerogative-but-whose.html"
## 
## $web_url
## [1] "http://www.nytimes.com/2016/10/16/business/mutfund/brexit-blurs-the-european-investment-outlook.html"
## 
## $web_url
## [1] "http://www.nytimes.com/2016/09/16/business/international/bank-of-england-brexit-carney.html"
## 
## $web_url
## [1] "http://www.nytimes.com/2016/09/15/world/europe/theresa-may-britain-brexit.html"
## 
## $web_url
## [1] "http://www.nytimes.com/2016/09/26/sports/golf/ryder-cup-brexit-europe.html"
df$news_desk[1]
## $news_desk
## [1] "Foreign"

Lets finally rerun a different query for completeness, and show the data that is returned from that query.

query <- "comey"
page <- 0
parsed <- parseJSON(readTimesAPI(timesURL, apiKey, query, page))

df <- makeDF(parsed$response$docs)
head(df)
## data frame with 0 columns and 0 rows
df$web_url[1:5]
## NULL