Working with Web APIs
Data acquisition and management
CUNY MSDS DATA 607
Rose Koh
2018/03/27
- The New York Times web site provides a rich set of APIs, as described here:
- http://developer.nytimes.com/docs
- You’ll need to start by signing up for an API key.
- Your task is to choose one of the New York Times APIs, construct an interface in R to read in the JSON data, and transform it to an R dataframe.
Load packages
library(httr)
library(jsonlite)
API info
# https://github.com/rstudio/webinars/blob/master/31-Web-APIs/frozen.R
# Here's your API Key for the Movie Reviews API: dcf704313e574ad1a677254242a549b0
# http://developer.nytimes.com/movie_reviews_v2.json#/README
# http://developer.nytimes.com/movie_reviews_v2.json#/Documentation
# http://developer.nytimes.com/movie_reviews_v2.json#/Console/GET/reviews/%7Bresource-type%7D.json
api.movie.reviews <- "&api-key=dcf704313e574ad1a677254242a549b0"
url <- "https://api.nytimes.com/svc/movies/v2/reviews/picks.json?order=by-opening-date?offset=20"
httr
httr.way <- GET(paste0(url, api.movie.reviews))
httr.way$status_code
## [1] 200
http_status(httr.way)
## $category
## [1] "Success"
##
## $reason
## [1] "OK"
##
## $message
## [1] "Success: (200) OK"
headers(httr.way)
## $`accept-ranges`
## [1] "bytes"
##
## $`access-control-allow-headers`
## [1] "Accept, Content-Type, X-Forwarded-For, X-Prototype-Version, X-Requested-With"
##
## $`access-control-allow-methods`
## [1] "GET, OPTIONS"
##
## $`access-control-allow-origin`
## [1] "*"
##
## $`access-control-expose-headers`
## [1] "Content-Length, X-JSON"
##
## $age
## [1] "0"
##
## $`cache-control`
## [1] "max-age=600"
##
## $`content-type`
## [1] "application/json; charset=UTF-8"
##
## $date
## [1] "Tue, 27 Mar 2018 17:05:16 GMT"
##
## $expires
## [1] "Tue, 27 Mar 2018 17:15:16 GMT"
##
## $server
## [1] "Apache/2.2.15 (CentOS)"
##
## $vary
## [1] "Origin"
##
## $via
## [1] "kong/0.9.5"
##
## $`x-cache`
## [1] "MISS"
##
## $`x-kong-proxy-latency`
## [1] "115"
##
## $`x-kong-upstream-latency`
## [1] "331"
##
## $`x-ratelimit-limit-day`
## [1] "1000"
##
## $`x-ratelimit-limit-second`
## [1] "5"
##
## $`x-ratelimit-remaining-day`
## [1] "974"
##
## $`x-ratelimit-remaining-second`
## [1] "4"
##
## $`x-varnish`
## [1] "13599269"
##
## $`content-length`
## [1] "16346"
##
## $connection
## [1] "keep-alive"
##
## attr(,"class")
## [1] "insensitive" "list"
httr.list <- content(httr.way, "parse")
#httr.list$results
httr.way$headers$`content-type`
## [1] "application/json; charset=UTF-8"
jsonlite
json.way <- fromJSON(paste0(url, api.movie.reviews))
json.df <- json.way$results
Result
# column names
names(json.df)
## [1] "display_title" "mpaa_rating" "critics_pick"
## [4] "byline" "headline" "summary_short"
## [7] "publication_date" "opening_date" "date_updated"
## [10] "link" "multimedia"
# Is it in data frame format?
is.data.frame(json.df)
## [1] TRUE
# head of the data frame
head(json.df)
## display_title mpaa_rating critics_pick byline
## 1 Ismael's Ghosts R 1 GLENN KENNY
## 2 Beauty and the Dogs 1 JEANNETTE CATSOULIS
## 3 The Workshop 1 JENNIFER SZALAI
## 4 I Kill Giants Not Rated 1 JEANNETTE CATSOULIS
## 5 Isle of Dogs PG-13 1 MANOHLA DARGIS
## 6 Pacific Rim Uprising PG-13 1 GLENN KENNY
## headline
## 1 Review: Wife Returns in ‘Ismael’s Ghosts.’ Confusion Ensues.
## 2 Review: Seeking Justice After a Rape in ‘Beauty and the Dogs’
## 3 Review: ‘The Workshop’ Plunges Into a Class Divided
## 4 Review: Slaying Monsters (and Avoiding Grief) in ‘I Kill Giants’
## 5 Review: Wes Anderson’s Bleakly Beautiful ‘Isle of Dogs’
## 6 Review: In ‘Pacific Rim Uprising,’ Scary Monsters and Super Robots
## summary_short
## 1 In Arnaud Desplechin’s remarkable sprawl of a film, a wife comes back after a 20-year disappearance, adding to an already convoluted situation.
## 2 In this film inspired by a true story, the director Kaouther Ben Hania leans hard on official corruption and a bureaucracy dominated by male power.
## 3 This new film from the French director Laurent Cantet follows a diverse group of students enrolled in a summer writing class taught by a famous novelist.
## 4 This magical movie marries adult themes and childlike wonder as a troubled girl embarks on a fantastical quest.
## 5 Set in a fantasy Japan, Mr. Anderson’s new movie tells a heroic story of abandoned dogs, voiced by Bill Murray, Liev Schreiber and others.
## 6 Kaiju come back for world domination in this sci-fi sequel.
## publication_date opening_date date_updated link.type
## 1 2018-03-22 2018-03-23 2018-03-27 16:44:28 article
## 2 2018-03-22 <NA> 2018-03-27 16:44:27 article
## 3 2018-03-22 <NA> 2018-03-27 16:44:27 article
## 4 2018-03-22 2018-03-23 2018-03-27 16:44:26 article
## 5 2018-03-22 2018-03-23 2018-03-27 16:44:23 article
## 6 2018-03-21 2018-03-23 2018-03-27 16:44:23 article
## link.url
## 1 http://www.nytimes.com/2018/03/22/movies/review-ismaels-ghosts-arnaud-desplechin.html
## 2 http://www.nytimes.com/2018/03/22/movies/beauty-and-the-dogs-review.html
## 3 http://www.nytimes.com/2018/03/22/movies/the-workshop-review-laurent-cantet.html
## 4 http://www.nytimes.com/2018/03/22/movies/i-kill-giants-review.html
## 5 http://www.nytimes.com/2018/03/22/movies/isle-of-dogs-review-wes-anderson.html
## 6 http://www.nytimes.com/2018/03/21/movies/pacific-rim-uprising-review-john-boyega.html
## link.suggested_link_text
## 1 Read the New York Times Review of Ismael's Ghosts
## 2 Read the New York Times Review of Beauty and the Dogs
## 3 Read the New York Times Review of The Workshop
## 4 Read the New York Times Review of I Kill Giants
## 5 Read the New York Times Review of Isle of Dogs
## 6 Read the New York Times Review of Pacific Rim Uprising
## multimedia.type
## 1 mediumThreeByTwo210
## 2 mediumThreeByTwo210
## 3 mediumThreeByTwo210
## 4 mediumThreeByTwo210
## 5 mediumThreeByTwo210
## 6 mediumThreeByTwo210
## multimedia.src
## 1 https://static01.nyt.com/images/2018/03/23/arts/23ismaelsghost/merlin_135652056_545215ce-7d65-4267-a0cb-29bddcee101a-mediumThreeByTwo210.jpg
## 2 https://static01.nyt.com/images/2018/03/23/arts/23beautyandthedogspix/23beautyandthedogspix-mediumThreeByTwo210-v2.jpg
## 3 https://static01.nyt.com/images/2018/03/23/arts/23theworkshop/23theworkshop-mediumThreeByTwo210.jpg
## 4 https://static01.nyt.com/images/2018/03/23/arts/23killgiants1/merlin_135651834_da35096f-806d-435e-836e-ce347967699b-mediumThreeByTwo210.jpg
## 5 https://static01.nyt.com/images/2018/03/23/arts/23isleofdogs4/23isleofdogs4-mediumThreeByTwo210.jpg
## 6 https://static01.nyt.com/images/2018/03/23/arts/23pacific-1/23pacific-1-mediumThreeByTwo210.jpg
## multimedia.width multimedia.height
## 1 210 140
## 2 210 140
## 3 210 140
## 4 210 140
## 5 210 140
## 6 210 140