The objective of this assignment is to use the New Times Search Article web API to manipulate an retrieve json data.
To perform this tas, the jsonlite and Knitr libraries are going to be used for this assignment
library(jsonlite)
library(knitr,quietly = T)First a key is needed to perform the scraping from the New York Time API. The key will be added to html link to retrieve desire data from articles. Finally, the data will be converted into r data frame.
article_key <- "&api-key=b75da00e12d54774a2d362adddcc9bef"
url <- "http://api.nytimes.com/svc/search/v2/articlesearch.json?q=LEBANON"
req <- fromJSON(paste0(url, article_key))
articles <- req$response$docs
colnames(articles)## [1] "web_url" "snippet" "lead_paragraph"
## [4] "abstract" "print_page" "blog"
## [7] "source" "multimedia" "headline"
## [10] "keywords" "pub_date" "document_type"
## [13] "news_desk" "section_name" "subsection_name"
## [16] "byline" "type_of_material" "_id"
## [19] "word_count" "slideshow_credits"The data frame has some vector elements that contain nested lists in addition to some redundant information. Also, some cleaning is formatting is needed to get the data into good shape.
mainheadline<-articles$headline$main
date <-as.Date(articles$pub_date)
pages<-articles$print_page
keyvalue=NULL
for(i in 1:length(articles$keywords))
{
tmp<-articles$keywords[[i]]$value
keyvalue[i]<-gsub(", ",", ",toString(tmp))
}
role=NULL
for(i in 1:length(articles$keywords))
{
tmp<-articles$byline$person[[i]]$role
role[i]<-gsub(", ",", ",toString(tmp))
}
reporter=NULL
for(i in 1:length(articles$keywords))
{
tmp1<-articles$byline$person[[i]]$firstname
tpm2<-articles$byline$person[[i]]$lastname
tmp<-paste(tmp1, tpm2, sep=" ")
reporter[i]<-gsub(", ",", ",toString(tmp))
}Reconstructing the subset cleaned data into data frame.
articlesdfsource<-data.frame(date,pages,role,reporter,articles$news_desk,articles$subsection_name,articles$section_name,articles$source,articles$document_type,articles$type_of_material)
names(articlesdfsource) <- c("date","pages","role","reporter","news_desk",
"subsection_name","section_name","source",
"document_type","type_of_material")
#colnames(articlesdfsource)
kable(articlesdfsource)| date | pages | role | reporter | news_desk | subsection_name | section_name | source | document_type | type_of_material |
|---|---|---|---|---|---|---|---|---|---|
| 2016-10-31 | 4 | reported | Thanassis CAMBANIS | Foreign | Middle East | World | The New York Times | article | News |
| 2006-07-18 | 1 | reported | Hassan | Foreign | Middle East | World | The New York Times | article | News |
| 2016-10-31 | NA | World / Middle East | Middle East | World | Reuters | multimedia | Video | ||
| NA | NA | NA | NA | World | NA | topic | timestopic | ||
| NA | NA | NA | NA | World | NA | topic | timestopic | ||
| NA | NA | NA | NA | NA | NA | topic | traveltopic | ||
| 2016-11-03 | NA | NA | Middle East | World | Reuters | article | News | ||
| 2016-09-13 | NA | reported | Tina ROSENBERG | OpEd | NA | Opinion | article | Op-Ed | |
| 1944-07-16 | BR3 | reported | Edith POPE | NA | NA | NA | The New York Times | article | Review |
| 2006-08-25 | 10 | reported | Hassan | Foreign | Middle East | World | The New York Times | article | News |
articleinfo <- data.frame(mainheadline,keyvalue,articles$snippet,articles$lead_paragraph)
names(articleinfo) <- c("Main_headline","keywords_value","snippet","lead_paragraph")
colnames(articleinfo)## [1] "Main_headline" "keywords_value" "snippet" "lead_paragraph"kable(articleinfo)The Lebanese Republic is named for its stunning snow-covered mountains and renowned for its cedar forests. It was once a part of the Roman Empire, and later became a major center of Christianity and Islam. It is now the most religiously diverse country in the Middle East, boasting significant numbers of Maronite Christians, Shiite and Sunni Muslims, Druze and Greek Orthodox. It borders on Syria, Israel and the Mediterranean Sea.
Before Lebanon’s civil war began in 1975, it boasted significant financial prosperity and was a popular tourist destination. The war exacted an enormous economic and human toll, and recovery has since been hampered by additional conflicts, including a recent influx of Syrian refugees.
Lebanon’s government is a parliamentary democracy established on the principles of confessionalism, a system that distributes power equally among its 18 recognized religious sects. Tensions between the government and Hezbollah have sprouted violence and legislative deadlock. The country has been without a president since Michel Sulayman’s term expired in 2014. Presidential duties are currently being carried out by acting President and Prime Minister Tammam Salam.
Learn more about Lebanon. Scroll below to view our archive of articles and chronology of latest news.
Lebanon The Lebanese Republic is named for its stunning snow-covered mountains and renowned for its cedar forests. It was once a part of the Roman Empire, and later became a major center of Christianity and Islam. It is now the most religiously diverse…The Lebanese Republic is named for its stunning snow-covered mountains and renowned for its cedar forests. It was once a part of the Roman Empire, and later became a major center of Christianity and Islam. It is now the most religiously diverse country in the Middle East, boasting significant numbers of Maronite Christians, Shiite and Sunni Muslims, Druze and Greek Orthodox. It borders on Syria, Israel and the Mediterranean Sea.
Before Lebanon’s civil war began in 1975, it boasted significant financial prosperity and was a popular tourist destination. The war exacted an enormous economic and human toll, and recovery has since been hampered by additional conflicts, including a recent influx of Syrian refugees.
Lebanon’s government is a parliamentary democracy established on the principles of confessionalism, a system that distributes power equally among its 18 recognized religious sects. Tensions between the government and Hezbollah have sprouted violence and legislative deadlock. The country has been without a president since Michel Sulayman’s term expired in 2014. Presidential duties are currently being carried out by acting President and Prime Minister Tammam Salam.
Learn more about Lebanon. Scroll below to view our archive of articles and chronology of latest news.
Lebanon Articles, photos and video about Lebanon from The New York Times, with reader advice on where and when to go. Articles, photos and video about Lebanon from The New York Times, with reader advice on where and when to go.
Lebanon’s Saad Hariri Named Prime Minister, Wins Speaker’s Support Aoun, Michel (1935- ), Lebanon, Beirut (Lebanon) Lebanon’s president asked Sunni Muslim leader Saad al-Hariri to form a new government after he won the support of a majority of lawmakers and the endorsement of the influential Parliament speaker. Lebanon’s president asked Sunni Muslim leader Saad al-Hariri to form a new government after he won the support of a majority of lawmakers and the endorsement of the influential Parliament speaker.
For Refugees in Lebanon, Cash Instead of Camps Lebanon Cash Consortium, World Food Program, United Nations High Commissioner for Refugees, Middle East and Africa Migrant Crisis, Syria, Lebanon, Refugees and Displaced Persons, Banking and Financial Institutions, Humanitarian Aid, Food, International Rescue Committee, United Nations Children’s Fund With debit cards rather than bagged food, refugees buy their produce, live in more dignity, and raise less resentment among their Lebanese neighbors. With debit cards rather than bagged food, refugees buy their produce, live in more dignity, and raise less resentment among their Lebanese neighbors.
Heroine of the Long Ago; LEBANON. By Caroline Miller. 234 pp. New York: Doubleday, Doran & Co. $2.50. A Poetic Heroine LEBANON, MILLER, C, BOOKS AND LITERATURE, REVIEWS GROWING up in Georgia over a hundred years ago, Lebanon Fairgale was so purely a child of nature that her dealings with human beings led to misunderstandings and persecution through no fault of her own: the history of her misfortunes is Caroline… GROWING up in Georgia over a hundred years ago, Lebanon Fairgale was so purely a child of nature that her dealings with human beings led to misunderstandings and persecution through no fault of her own: the history of her misfortunes is Caroline Miller’s first novel since the Pulitzer prize-winning “Lamb in His Bosom.”
At Funeral, a Sunni Village Condemns Hezbollah’s Presence Lebanon, Hezbollah, Israel Criticism of Hezbollah is rare in southern Lebanon, but many residents of Marwaheen blame the group for bringing on an Israeli attack that killed 23 people. Criticism of Hezbollah is rare in southern Lebanon, but many residents of Marwaheen blame the group for bringing on an Israeli attack that killed 23 people.
articlesdf<-data.frame(date,pages,mainheadline,keyvalue,role,reporter,articles$news_desk, articles$subsection_name,articles$section_name,articles$source,articles$snippet,articles$document_type,articles$type_of_material,articles$lead_paragraph)
names(articlesdf) <- c("date","pages","Main_headline","keywords_value","role","reporter","news_desk",
"subsection_name","section_name","source","snippet",
"document_type","type_of_material","lead_paragraph")
colnames(articlesdf)## [1] "date" "pages" "Main_headline"
## [4] "keywords_value" "role" "reporter"
## [7] "news_desk" "subsection_name" "section_name"
## [10] "source" "snippet" "document_type"
## [13] "type_of_material" "lead_paragraph"Even though there are libraries that help facilitate scraping, still some work have to be done such as formatting and cleaning.