DATA 607 Assignment Six

Overview of approach.

I approached this assignment by going to the NY Times Developer page and went through the process of obtaining an API key. I reviewed the different options to obtain data. I selected the Article Search API to look up articles by keyword. I was interested in viewing historical events. I decided to see what the NY Times covered on December 8, 1941 one day following the attack on Pearl Harbor. To search for these articles, I used the key word “Japan” and isolated articles to 12/08/1941.

Loading of Libraries. The httr package contains tools for Working with URLs and HTTP. It contains the GET command to retrieve whatever is specified by the URL. The jsonlite package allows to work with JSON data in R.

library(httr)
library(jsonlite)
library(tidyverse)

## -- Attaching packages --------------------------------------- tidyverse 1.3.1 --

## v ggplot2 3.3.5     v purrr   0.3.4
## v tibble  3.1.4     v dplyr   1.0.7
## v tidyr   1.1.3     v stringr 1.4.0
## v readr   2.0.1     v forcats 0.5.1

## -- Conflicts ------------------------------------------ tidyverse_conflicts() --
## x dplyr::filter()  masks stats::filter()
## x purrr::flatten() masks jsonlite::flatten()
## x dplyr::lag()     masks stats::lag()

Establish connection to NY Times database with an API key. The API key was acquired by visiting the Developers-Getting Started webpage at https://developer.nytimes.com/get-started. An object defined as URL is created for the path to the NY Times data. The GET command retrieves the json data.

url <- "https://api.nytimes.com/svc/search/v2/articlesearch.json?q=japan&begin_date=19411208&end_date=19411208&api-key=6T9n9dJsBDgrXobS7L4ilAvFz1UvDJSu"
retrieve <- GET(url, accept_json())
retrieve

## Response [https://api.nytimes.com/svc/search/v2/articlesearch.json?q=japan&begin_date=19411208&end_date=19411208&api-key=6T9n9dJsBDgrXobS7L4ilAvFz1UvDJSu]
##   Date: 2021-10-23 01:52
##   Status: 200
##   Content-Type: application/json
##   Size: 10.8 kB

Verify if the retrieval of json data is error free. If FALSE, it is error free.

http_error(retrieve)

## [1] FALSE

The content command extracts content as text from the body. It shows raw data which is not structured and readable. This is data in a text format.

contents <- content(retrieve, as="text", encoding = "UTF-8")

The fromJSON command reads the content in JSON format and converts strings into R objects.

R_objects <- fromJSON(contents)
R_objects

## $status
## [1] "OK"
## 
## $copyright
## [1] "Copyright (c) 2021 The New York Times Company. All Rights Reserved."
## 
## $response
## $response$docs
##                                                                                                                                                                                         abstract
## 1                                                                                                                                                              Closing under police orders illus
## 2                                                                                                                                                                               put on war basis
## 3                                                                                                                                                                                             lr
## 4                                                                                                                                                                                               
## 5  SINGAPORE, Monday, Dec. 8 -- British planes are bombing Japanese ships off the coast of Malaya. Their attack followed a Japanese bombing of Singapore in which there were a few casualties.  
## 6                                                                                                                                                                                               
## 7                                                                                                          Canada declares war; Canadian troops in Hong Kong noted; Vancouver Japanese offer aid
## 8                                                                                                                                                                Australia declares war on Japan
## 9                                                                                                                                                                               Jap natls seized
## 10                                                                                                                                                                          On Japan's war entry
##                                                                                                               web_url
## 1  https://www.nytimes.com/1941/12/08/archives/the-new-war-in-the-pacific-japanese-and-chinese-reactions-burning.html
## 2  https://www.nytimes.com/1941/12/08/archives/entire-city-put-on-war-footing-japanese-rounded-up-by-fbi-sent-to.html
## 3                                                     https://www.nytimes.com/1941/12/08/archives/war-with-japan.html
## 4                                            https://www.nytimes.com/1941/12/08/archives/japanese-premiers-story.html
## 5                                            https://www.nytimes.com/1941/12/08/archives/japanese-ships-attacked.html
## 6                             https://www.nytimes.com/1941/12/08/archives/japanese-invasion-of-thailand-reported.html
## 7                                       https://www.nytimes.com/1941/12/08/archives/vancouver-japanese-offer-aid.html
## 8                             https://www.nytimes.com/1941/12/08/archives/australia-declares-war-on-the-japanese.html
## 9                                          https://www.nytimes.com/1941/12/08/archives/japanese-nationals-seized.html
## 10                                      https://www.nytimes.com/1941/12/08/archives/holds-japan-commits-harakiri.html
##                                                                                  snippet
## 1                                                      Closing under police orders illus
## 2                                                                       put on war basis
## 3                                                                                     lr
## 4                                                                                       
## 5                                                                                       
## 6                                                                                       
## 7  Canada declares war; Canadian troops in Hong Kong noted; Vancouver Japanese offer aid
## 8                                                        Australia declares war on Japan
## 9                                                                       Jap natls seized
## 10                                                                  On Japan's war entry
##    lead_paragraph print_page             source multimedia
## 1                          5 The New York Times       NULL
## 2                          1 The New York Times       NULL
## 3                         22 The New York Times       NULL
## 4                          5 The New York Times       NULL
## 5                          2 The New York Times       NULL
## 6                          2 The New York Times       NULL
## 7                         14 The New York Times       NULL
## 8                          2 The New York Times       NULL
## 9                         14 The New York Times       NULL
## 10                        18 The New York Times       NULL
##                                                                                                     headline.main
## 1                  The New War in the Pacific: Japanese and Chinese Reactions; BURNING OF PAPERS WATCHED BY 1,000
## 2  ENTIRE CITY PUT ON WAR FOOTING; Japanese Rounded Up by FBI, Sent to Ellis Island -- Vital Services Are Guarded
## 3                                                                                                  WAR WITH JAPAN
## 4                                                                                        Japanese Premier's Story
## 5                                                                                         Japanese Ships Attacked
## 6                                                                          Japanese Invasion of Thailand Reported
## 7                                                                                    Vancouver Japanese Offer Aid
## 8                                                                          Australia Declares War on the Japanese
## 9                                                                                       Japanese Nationals Seized
## 10                                                                                  Holds Japan Commits Hara-Kiri
##    headline.kicker headline.content_kicker
## 1             <NA>                      NA
## 2             <NA>                      NA
## 3             <NA>                      NA
## 4                1                      NA
## 5                1                      NA
## 6                1                      NA
## 7             <NA>                      NA
## 8             <NA>                      NA
## 9             <NA>                      NA
## 10            <NA>                      NA
##                                                                                           headline.print_headline
## 1                  The New War in the Pacific: Japanese and Chinese Reactions; BURNING OF PAPERS WATCHED BY 1,000
## 2  ENTIRE CITY PUT ON WAR FOOTING; Japanese Rounded Up by FBI, Sent to Ellis Island -- Vital Services Are Guarded
## 3                                                                                                  WAR WITH JAPAN
## 4                                                                                        Japanese Premier's Story
## 5                                                                                         Japanese Ships Attacked
## 6                                                                          Japanese Invasion of Thailand Reported
## 7                                                                                    Vancouver Japanese Offer Aid
## 8                                                                          Australia Declares War on the Japanese
## 9                                                                                       Japanese Nationals Seized
## 10                                                                                  Holds Japan Commits Hara-Kiri
##    headline.name headline.seo headline.sub
## 1             NA           NA           NA
## 2             NA           NA           NA
## 3             NA           NA           NA
## 4             NA           NA           NA
## 5             NA           NA           NA
## 6             NA           NA           NA
## 7             NA           NA           NA
## 8             NA           NA           NA
## 9             NA           NA           NA
## 10            NA           NA           NA
##                                                                                                                                                                                                                                                                                                                                                                                                keywords
## 1                                                                      glocations, glocations, glocations, glocations, organizations, organizations, organizations, organizations, organizations, persons, Japan, Japan, Japan, Japan, FOREIGN SERVICE, FOREIGN SERVICE, FOREIGN SERVICE, FOREIGN SERVICE, Nippon Club, MONISHINA , MORITO, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, N, N, N, N, N, N, N, N, N, N
## 2  glocations, glocations, glocations, glocations, organizations, subject, subject, subject, subject, subject, Japan, United States, United States, United States, FOREIGN SERVICE, FOREIGN POPULATION, ARMAMENT AND DEFENSE, PERSONNEL, MILITARY, ASSIGNMENTS AND FORCE MOVEMENTS, ARMAMENT AND DEFENSE, STATE, LOCAL AND CIVILIAN ACTION, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, N, N, N, N, N, N, N, N, N, N
## 3                                                                                                                                                                                                                                                              glocations, glocations, subject, subject, FAR EAST AND PACIFIC AREAS, United States, ARMAMENT AND DEFENSE, LABOR, 1, 2, 3, 4, N, N, N, N
## 4                                                                                                                                                                                                                                                                                                                                                                                                  NULL
## 5                                                                                                                                                                                                                                                                                                                                                                                                  NULL
## 6                                                                                                                                                                                                                                                                                                                                                                                                  NULL
## 7                                                                                                                                                                                                                                                                                             glocations, glocations, subject, Canada, FAR EAST AND PACIFIC AREAS, FOREIGN POPULATION, 1, 2, 3, N, N, N
## 8                                                                                                                                                                                                                                                                                                                                                          glocations, FAR EAST AND PACIFIC AREAS, 1, N
## 9                                                                                                                                                                                                                                                                                                                                         glocations, subject, Canada, ARMAMENT AND DEFENSE, 1, 2, N, N
## 10                                                                                                                                                                                                                                                                                                                          glocations, persons, FAR EAST AND PACIFIC AREAS, GIBBS , PHILIP, 1, 2, N, N
##                    pub_date document_type news_desk section_name
## 1  1941-12-08T05:00:00+0000       article      None     Archives
## 2  1941-12-08T05:00:00+0000       article      None     Archives
## 3  1941-12-08T05:00:00+0000       article      None     Archives
## 4  1941-12-08T05:00:00+0000       article      None     Archives
## 5  1941-12-08T05:00:00+0000       article      None     Archives
## 6  1941-12-08T05:00:00+0000       article      None     Archives
## 7  1941-12-08T05:00:00+0000       article      None     Archives
## 8  1941-12-08T05:00:00+0000       article      None     Archives
## 9  1941-12-08T05:00:00+0000       article      None     Archives
## 10 1941-12-08T05:00:00+0000       article      None     Archives
##                                       byline.original
## 1                       Special to THE NEW YORK TIMES
## 2                                                <NA>
## 3                                                <NA>
## 4                                                <NA>
## 5  By F. Tillman Durdinwireless To the New York Times
## 6                                                <NA>
## 7                                                <NA>
## 8                                                <NA>
## 9                                                <NA>
## 10                                               <NA>
##                                         byline.person byline.organization
## 1                   NA, NA, NA, NA, NA, reported, , 1                  NA
## 2                                                NULL                  NA
## 3                                                NULL                  NA
## 4                                                NULL                  NA
## 5  F., Tillman, Durdinwireless, NA, NA, reported, , 1                  NA
## 6                                                NULL                  NA
## 7                                                NULL                  NA
## 8                                                NULL                  NA
## 9                                                NULL                  NA
## 10                                               NULL                  NA
##    type_of_material                                                _id
## 1          Archives nyt://article/35b2bf41-2cd0-563c-a81f-712aac369b19
## 2          Archives nyt://article/6de54920-0773-5b1d-89d6-4d5896726569
## 3          Archives nyt://article/9c5e92d4-99df-5399-947c-459ada05bd37
## 4          Archives nyt://article/241dc503-75e9-5d26-8715-fbd1a2bd5cc7
## 5          Archives nyt://article/ffa7035e-8c24-50a3-ae43-f41e06218e05
## 6          Archives nyt://article/4eebb4c6-1926-5054-ad60-539e56d85dd1
## 7          Archives nyt://article/d6894d92-4f3d-538d-ab90-1d9918b3627f
## 8          Archives nyt://article/b30fc00c-5e61-594c-ae63-4c48fd924ef4
## 9          Archives nyt://article/a3ba68f9-63ef-5664-87da-644cf4237fe3
## 10         Archives nyt://article/02d8b545-f0ba-5389-9f2b-9976f5167bba
##    word_count                                                uri
## 1           0 nyt://article/35b2bf41-2cd0-563c-a81f-712aac369b19
## 2           0 nyt://article/6de54920-0773-5b1d-89d6-4d5896726569
## 3           0 nyt://article/9c5e92d4-99df-5399-947c-459ada05bd37
## 4           0 nyt://article/241dc503-75e9-5d26-8715-fbd1a2bd5cc7
## 5           0 nyt://article/ffa7035e-8c24-50a3-ae43-f41e06218e05
## 6           0 nyt://article/4eebb4c6-1926-5054-ad60-539e56d85dd1
## 7           0 nyt://article/d6894d92-4f3d-538d-ab90-1d9918b3627f
## 8           0 nyt://article/b30fc00c-5e61-594c-ae63-4c48fd924ef4
## 9           0 nyt://article/a3ba68f9-63ef-5664-87da-644cf4237fe3
## 10          0 nyt://article/02d8b545-f0ba-5389-9f2b-9976f5167bba
## 
## $response$meta
## $response$meta$hits
## [1] 90
## 
## $response$meta$offset
## [1] 0
## 
## $response$meta$time
## [1] 15

Convert to a data frame using jsonlite package. Use class function to verify it is a data frame.

df <- as.data.frame(R_objects)
class(df)

## [1] "data.frame"

df

##    status                                                           copyright
## 1      OK Copyright (c) 2021 The New York Times Company. All Rights Reserved.
## 2      OK Copyright (c) 2021 The New York Times Company. All Rights Reserved.
## 3      OK Copyright (c) 2021 The New York Times Company. All Rights Reserved.
## 4      OK Copyright (c) 2021 The New York Times Company. All Rights Reserved.
## 5      OK Copyright (c) 2021 The New York Times Company. All Rights Reserved.
## 6      OK Copyright (c) 2021 The New York Times Company. All Rights Reserved.
## 7      OK Copyright (c) 2021 The New York Times Company. All Rights Reserved.
## 8      OK Copyright (c) 2021 The New York Times Company. All Rights Reserved.
## 9      OK Copyright (c) 2021 The New York Times Company. All Rights Reserved.
## 10     OK Copyright (c) 2021 The New York Times Company. All Rights Reserved.
##                                                                                                                                                                           response.docs.abstract
## 1                                                                                                                                                              Closing under police orders illus
## 2                                                                                                                                                                               put on war basis
## 3                                                                                                                                                                                             lr
## 4                                                                                                                                                                                               
## 5  SINGAPORE, Monday, Dec. 8 -- British planes are bombing Japanese ships off the coast of Malaya. Their attack followed a Japanese bombing of Singapore in which there were a few casualties.  
## 6                                                                                                                                                                                               
## 7                                                                                                          Canada declares war; Canadian troops in Hong Kong noted; Vancouver Japanese offer aid
## 8                                                                                                                                                                Australia declares war on Japan
## 9                                                                                                                                                                               Jap natls seized
## 10                                                                                                                                                                          On Japan's war entry
##                                                                                                 response.docs.web_url
## 1  https://www.nytimes.com/1941/12/08/archives/the-new-war-in-the-pacific-japanese-and-chinese-reactions-burning.html
## 2  https://www.nytimes.com/1941/12/08/archives/entire-city-put-on-war-footing-japanese-rounded-up-by-fbi-sent-to.html
## 3                                                     https://www.nytimes.com/1941/12/08/archives/war-with-japan.html
## 4                                            https://www.nytimes.com/1941/12/08/archives/japanese-premiers-story.html
## 5                                            https://www.nytimes.com/1941/12/08/archives/japanese-ships-attacked.html
## 6                             https://www.nytimes.com/1941/12/08/archives/japanese-invasion-of-thailand-reported.html
## 7                                       https://www.nytimes.com/1941/12/08/archives/vancouver-japanese-offer-aid.html
## 8                             https://www.nytimes.com/1941/12/08/archives/australia-declares-war-on-the-japanese.html
## 9                                          https://www.nytimes.com/1941/12/08/archives/japanese-nationals-seized.html
## 10                                      https://www.nytimes.com/1941/12/08/archives/holds-japan-commits-harakiri.html
##                                                                    response.docs.snippet
## 1                                                      Closing under police orders illus
## 2                                                                       put on war basis
## 3                                                                                     lr
## 4                                                                                       
## 5                                                                                       
## 6                                                                                       
## 7  Canada declares war; Canadian troops in Hong Kong noted; Vancouver Japanese offer aid
## 8                                                        Australia declares war on Japan
## 9                                                                       Jap natls seized
## 10                                                                  On Japan's war entry
##    response.docs.lead_paragraph response.docs.print_page response.docs.source
## 1                                                      5   The New York Times
## 2                                                      1   The New York Times
## 3                                                     22   The New York Times
## 4                                                      5   The New York Times
## 5                                                      2   The New York Times
## 6                                                      2   The New York Times
## 7                                                     14   The New York Times
## 8                                                      2   The New York Times
## 9                                                     14   The New York Times
## 10                                                    18   The New York Times
##    response.docs.multimedia
## 1                      NULL
## 2                      NULL
## 3                      NULL
## 4                      NULL
## 5                      NULL
## 6                      NULL
## 7                      NULL
## 8                      NULL
## 9                      NULL
## 10                     NULL
##                                                                                       response.docs.headline.main
## 1                  The New War in the Pacific: Japanese and Chinese Reactions; BURNING OF PAPERS WATCHED BY 1,000
## 2  ENTIRE CITY PUT ON WAR FOOTING; Japanese Rounded Up by FBI, Sent to Ellis Island -- Vital Services Are Guarded
## 3                                                                                                  WAR WITH JAPAN
## 4                                                                                        Japanese Premier's Story
## 5                                                                                         Japanese Ships Attacked
## 6                                                                          Japanese Invasion of Thailand Reported
## 7                                                                                    Vancouver Japanese Offer Aid
## 8                                                                          Australia Declares War on the Japanese
## 9                                                                                       Japanese Nationals Seized
## 10                                                                                  Holds Japan Commits Hara-Kiri
##    response.docs.headline.kicker response.docs.headline.content_kicker
## 1                           <NA>                                    NA
## 2                           <NA>                                    NA
## 3                           <NA>                                    NA
## 4                              1                                    NA
## 5                              1                                    NA
## 6                              1                                    NA
## 7                           <NA>                                    NA
## 8                           <NA>                                    NA
## 9                           <NA>                                    NA
## 10                          <NA>                                    NA
##                                                                             response.docs.headline.print_headline
## 1                  The New War in the Pacific: Japanese and Chinese Reactions; BURNING OF PAPERS WATCHED BY 1,000
## 2  ENTIRE CITY PUT ON WAR FOOTING; Japanese Rounded Up by FBI, Sent to Ellis Island -- Vital Services Are Guarded
## 3                                                                                                  WAR WITH JAPAN
## 4                                                                                        Japanese Premier's Story
## 5                                                                                         Japanese Ships Attacked
## 6                                                                          Japanese Invasion of Thailand Reported
## 7                                                                                    Vancouver Japanese Offer Aid
## 8                                                                          Australia Declares War on the Japanese
## 9                                                                                       Japanese Nationals Seized
## 10                                                                                  Holds Japan Commits Hara-Kiri
##    response.docs.headline.name response.docs.headline.seo
## 1                           NA                         NA
## 2                           NA                         NA
## 3                           NA                         NA
## 4                           NA                         NA
## 5                           NA                         NA
## 6                           NA                         NA
## 7                           NA                         NA
## 8                           NA                         NA
## 9                           NA                         NA
## 10                          NA                         NA
##    response.docs.headline.sub
## 1                          NA
## 2                          NA
## 3                          NA
## 4                          NA
## 5                          NA
## 6                          NA
## 7                          NA
## 8                          NA
## 9                          NA
## 10                         NA
##                                                                                                                                                                                                                                                                                                                                                                                  response.docs.keywords
## 1                                                                      glocations, glocations, glocations, glocations, organizations, organizations, organizations, organizations, organizations, persons, Japan, Japan, Japan, Japan, FOREIGN SERVICE, FOREIGN SERVICE, FOREIGN SERVICE, FOREIGN SERVICE, Nippon Club, MONISHINA , MORITO, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, N, N, N, N, N, N, N, N, N, N
## 2  glocations, glocations, glocations, glocations, organizations, subject, subject, subject, subject, subject, Japan, United States, United States, United States, FOREIGN SERVICE, FOREIGN POPULATION, ARMAMENT AND DEFENSE, PERSONNEL, MILITARY, ASSIGNMENTS AND FORCE MOVEMENTS, ARMAMENT AND DEFENSE, STATE, LOCAL AND CIVILIAN ACTION, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, N, N, N, N, N, N, N, N, N, N
## 3                                                                                                                                                                                                                                                              glocations, glocations, subject, subject, FAR EAST AND PACIFIC AREAS, United States, ARMAMENT AND DEFENSE, LABOR, 1, 2, 3, 4, N, N, N, N
## 4                                                                                                                                                                                                                                                                                                                                                                                                  NULL
## 5                                                                                                                                                                                                                                                                                                                                                                                                  NULL
## 6                                                                                                                                                                                                                                                                                                                                                                                                  NULL
## 7                                                                                                                                                                                                                                                                                             glocations, glocations, subject, Canada, FAR EAST AND PACIFIC AREAS, FOREIGN POPULATION, 1, 2, 3, N, N, N
## 8                                                                                                                                                                                                                                                                                                                                                          glocations, FAR EAST AND PACIFIC AREAS, 1, N
## 9                                                                                                                                                                                                                                                                                                                                         glocations, subject, Canada, ARMAMENT AND DEFENSE, 1, 2, N, N
## 10                                                                                                                                                                                                                                                                                                                          glocations, persons, FAR EAST AND PACIFIC AREAS, GIBBS , PHILIP, 1, 2, N, N
##      response.docs.pub_date response.docs.document_type response.docs.news_desk
## 1  1941-12-08T05:00:00+0000                     article                    None
## 2  1941-12-08T05:00:00+0000                     article                    None
## 3  1941-12-08T05:00:00+0000                     article                    None
## 4  1941-12-08T05:00:00+0000                     article                    None
## 5  1941-12-08T05:00:00+0000                     article                    None
## 6  1941-12-08T05:00:00+0000                     article                    None
## 7  1941-12-08T05:00:00+0000                     article                    None
## 8  1941-12-08T05:00:00+0000                     article                    None
## 9  1941-12-08T05:00:00+0000                     article                    None
## 10 1941-12-08T05:00:00+0000                     article                    None
##    response.docs.section_name
## 1                    Archives
## 2                    Archives
## 3                    Archives
## 4                    Archives
## 5                    Archives
## 6                    Archives
## 7                    Archives
## 8                    Archives
## 9                    Archives
## 10                   Archives
##                         response.docs.byline.original
## 1                       Special to THE NEW YORK TIMES
## 2                                                <NA>
## 3                                                <NA>
## 4                                                <NA>
## 5  By F. Tillman Durdinwireless To the New York Times
## 6                                                <NA>
## 7                                                <NA>
## 8                                                <NA>
## 9                                                <NA>
## 10                                               <NA>
##                           response.docs.byline.person
## 1                   NA, NA, NA, NA, NA, reported, , 1
## 2                                                NULL
## 3                                                NULL
## 4                                                NULL
## 5  F., Tillman, Durdinwireless, NA, NA, reported, , 1
## 6                                                NULL
## 7                                                NULL
## 8                                                NULL
## 9                                                NULL
## 10                                               NULL
##    response.docs.byline.organization response.docs.type_of_material
## 1                                 NA                       Archives
## 2                                 NA                       Archives
## 3                                 NA                       Archives
## 4                                 NA                       Archives
## 5                                 NA                       Archives
## 6                                 NA                       Archives
## 7                                 NA                       Archives
## 8                                 NA                       Archives
## 9                                 NA                       Archives
## 10                                NA                       Archives
##                                     response.docs._id response.docs.word_count
## 1  nyt://article/35b2bf41-2cd0-563c-a81f-712aac369b19                        0
## 2  nyt://article/6de54920-0773-5b1d-89d6-4d5896726569                        0
## 3  nyt://article/9c5e92d4-99df-5399-947c-459ada05bd37                        0
## 4  nyt://article/241dc503-75e9-5d26-8715-fbd1a2bd5cc7                        0
## 5  nyt://article/ffa7035e-8c24-50a3-ae43-f41e06218e05                        0
## 6  nyt://article/4eebb4c6-1926-5054-ad60-539e56d85dd1                        0
## 7  nyt://article/d6894d92-4f3d-538d-ab90-1d9918b3627f                        0
## 8  nyt://article/b30fc00c-5e61-594c-ae63-4c48fd924ef4                        0
## 9  nyt://article/a3ba68f9-63ef-5664-87da-644cf4237fe3                        0
## 10 nyt://article/02d8b545-f0ba-5389-9f2b-9976f5167bba                        0
##                                     response.docs.uri response.meta.hits
## 1  nyt://article/35b2bf41-2cd0-563c-a81f-712aac369b19                 90
## 2  nyt://article/6de54920-0773-5b1d-89d6-4d5896726569                 90
## 3  nyt://article/9c5e92d4-99df-5399-947c-459ada05bd37                 90
## 4  nyt://article/241dc503-75e9-5d26-8715-fbd1a2bd5cc7                 90
## 5  nyt://article/ffa7035e-8c24-50a3-ae43-f41e06218e05                 90
## 6  nyt://article/4eebb4c6-1926-5054-ad60-539e56d85dd1                 90
## 7  nyt://article/d6894d92-4f3d-538d-ab90-1d9918b3627f                 90
## 8  nyt://article/b30fc00c-5e61-594c-ae63-4c48fd924ef4                 90
## 9  nyt://article/a3ba68f9-63ef-5664-87da-644cf4237fe3                 90
## 10 nyt://article/02d8b545-f0ba-5389-9f2b-9976f5167bba                 90
##    response.meta.offset response.meta.time
## 1                     0                 15
## 2                     0                 15
## 3                     0                 15
## 4                     0                 15
## 5                     0                 15
## 6                     0                 15
## 7                     0                 15
## 8                     0                 15
## 9                     0                 15
## 10                    0                 15

The dataframe contain a lot of extraneous variables. I isolated four variables that would probably be most important to any analyst, that is, the publication date, a snippet of information, the pages and the URL.

df1 <- select(df, response.docs.pub_date, response.docs.snippet, response.docs.print_page, response.docs.web_url)
df1

##      response.docs.pub_date
## 1  1941-12-08T05:00:00+0000
## 2  1941-12-08T05:00:00+0000
## 3  1941-12-08T05:00:00+0000
## 4  1941-12-08T05:00:00+0000
## 5  1941-12-08T05:00:00+0000
## 6  1941-12-08T05:00:00+0000
## 7  1941-12-08T05:00:00+0000
## 8  1941-12-08T05:00:00+0000
## 9  1941-12-08T05:00:00+0000
## 10 1941-12-08T05:00:00+0000
##                                                                    response.docs.snippet
## 1                                                      Closing under police orders illus
## 2                                                                       put on war basis
## 3                                                                                     lr
## 4                                                                                       
## 5                                                                                       
## 6                                                                                       
## 7  Canada declares war; Canadian troops in Hong Kong noted; Vancouver Japanese offer aid
## 8                                                        Australia declares war on Japan
## 9                                                                       Jap natls seized
## 10                                                                  On Japan's war entry
##    response.docs.print_page
## 1                         5
## 2                         1
## 3                        22
## 4                         5
## 5                         2
## 6                         2
## 7                        14
## 8                         2
## 9                        14
## 10                       18
##                                                                                                 response.docs.web_url
## 1  https://www.nytimes.com/1941/12/08/archives/the-new-war-in-the-pacific-japanese-and-chinese-reactions-burning.html
## 2  https://www.nytimes.com/1941/12/08/archives/entire-city-put-on-war-footing-japanese-rounded-up-by-fbi-sent-to.html
## 3                                                     https://www.nytimes.com/1941/12/08/archives/war-with-japan.html
## 4                                            https://www.nytimes.com/1941/12/08/archives/japanese-premiers-story.html
## 5                                            https://www.nytimes.com/1941/12/08/archives/japanese-ships-attacked.html
## 6                             https://www.nytimes.com/1941/12/08/archives/japanese-invasion-of-thailand-reported.html
## 7                                       https://www.nytimes.com/1941/12/08/archives/vancouver-japanese-offer-aid.html
## 8                             https://www.nytimes.com/1941/12/08/archives/australia-declares-war-on-the-japanese.html
## 9                                          https://www.nytimes.com/1941/12/08/archives/japanese-nationals-seized.html
## 10                                      https://www.nytimes.com/1941/12/08/archives/holds-japan-commits-harakiri.html

The variable names were long. Therefore, I changed the names.

War_Articles <- rename(df1, pub_date = response.docs.pub_date, snippet = response.docs.snippet, pages = response.docs.print_page, URL = response.docs.web_url)
War_Articles

##                    pub_date
## 1  1941-12-08T05:00:00+0000
## 2  1941-12-08T05:00:00+0000
## 3  1941-12-08T05:00:00+0000
## 4  1941-12-08T05:00:00+0000
## 5  1941-12-08T05:00:00+0000
## 6  1941-12-08T05:00:00+0000
## 7  1941-12-08T05:00:00+0000
## 8  1941-12-08T05:00:00+0000
## 9  1941-12-08T05:00:00+0000
## 10 1941-12-08T05:00:00+0000
##                                                                                  snippet
## 1                                                      Closing under police orders illus
## 2                                                                       put on war basis
## 3                                                                                     lr
## 4                                                                                       
## 5                                                                                       
## 6                                                                                       
## 7  Canada declares war; Canadian troops in Hong Kong noted; Vancouver Japanese offer aid
## 8                                                        Australia declares war on Japan
## 9                                                                       Jap natls seized
## 10                                                                  On Japan's war entry
##    pages
## 1      5
## 2      1
## 3     22
## 4      5
## 5      2
## 6      2
## 7     14
## 8      2
## 9     14
## 10    18
##                                                                                                                   URL
## 1  https://www.nytimes.com/1941/12/08/archives/the-new-war-in-the-pacific-japanese-and-chinese-reactions-burning.html
## 2  https://www.nytimes.com/1941/12/08/archives/entire-city-put-on-war-footing-japanese-rounded-up-by-fbi-sent-to.html
## 3                                                     https://www.nytimes.com/1941/12/08/archives/war-with-japan.html
## 4                                            https://www.nytimes.com/1941/12/08/archives/japanese-premiers-story.html
## 5                                            https://www.nytimes.com/1941/12/08/archives/japanese-ships-attacked.html
## 6                             https://www.nytimes.com/1941/12/08/archives/japanese-invasion-of-thailand-reported.html
## 7                                       https://www.nytimes.com/1941/12/08/archives/vancouver-japanese-offer-aid.html
## 8                             https://www.nytimes.com/1941/12/08/archives/australia-declares-war-on-the-japanese.html
## 9                                          https://www.nytimes.com/1941/12/08/archives/japanese-nationals-seized.html
## 10                                      https://www.nytimes.com/1941/12/08/archives/holds-japan-commits-harakiri.html

I created a tibble to attempt to make the dataframe look more appealing.

as_tibble(War_Articles)

## # A tibble: 10 x 4
##    pub_date                 snippet                  pages URL                  
##    <chr>                    <chr>                    <chr> <chr>                
##  1 1941-12-08T05:00:00+0000 "Closing under police o~ 5     https://www.nytimes.~
##  2 1941-12-08T05:00:00+0000 "put on war basis"       1     https://www.nytimes.~
##  3 1941-12-08T05:00:00+0000 "lr"                     22    https://www.nytimes.~
##  4 1941-12-08T05:00:00+0000 ""                       5     https://www.nytimes.~
##  5 1941-12-08T05:00:00+0000 ""                       2     https://www.nytimes.~
##  6 1941-12-08T05:00:00+0000 ""                       2     https://www.nytimes.~
##  7 1941-12-08T05:00:00+0000 "Canada declares war; C~ 14    https://www.nytimes.~
##  8 1941-12-08T05:00:00+0000 "Australia declares war~ 2     https://www.nytimes.~
##  9 1941-12-08T05:00:00+0000 "Jap natls seized"       14    https://www.nytimes.~
## 10 1941-12-08T05:00:00+0000 "On Japan's war entry"   18    https://www.nytimes.~

DATA 607 Assignment Six

Joseph Foy

10/24/2021

Assignment Description - Web APIs

Assignment Rubric - (30 points)

Preparing Data (25 points)

Reproducibility (2 points)

Using R Markdown text and headers (2 points)

Workflow (2 points)

Included a brief description of the assigned problem.

Included an overview of your approach. Explained your reasoning.

Provided a conclusion (including any findings and recommendations).

Submission (1 points)

Published to rpubs and provided a link in your assignment submission.

Published to GitHub and provided a link in your assignment submission.

Overview of approach.

Loading of Libraries. The httr package contains tools for Working with URLs and HTTP. It contains the GET command to retrieve whatever is specified by the URL. The jsonlite package allows to work with JSON data in R.

Establish connection to NY Times database with an API key. The API key was acquired by visiting the Developers-Getting Started webpage at https://developer.nytimes.com/get-started. An object defined as URL is created for the path to the NY Times data. The GET command retrieves the json data.

Verify if the retrieval of json data is error free. If FALSE, it is error free.

The content command extracts content as text from the body. It shows raw data which is not structured and readable. This is data in a text format.

The fromJSON command reads the content in JSON format and converts strings into R objects.

Convert to a data frame using jsonlite package. Use class function to verify it is a data frame.

The dataframe contain a lot of extraneous variables. I isolated four variables that would probably be most important to any analyst, that is, the publication date, a snippet of information, the pages and the URL.

The variable names were long. Therefore, I changed the names.

I created a tibble to attempt to make the dataframe look more appealing.

Conclusion

The NY Times data is vast and would take a lot of time to understand how to extract it. The API key access process was rather simple and allows for easy access to information, albeit learning search terms is a complex task.