Pick three of your favorite books on one of your favorite subjects. At least one of the books should have more than one author. For each book, include the title, authors, and two or three other attributes that you find interesting.
Take the information that you’ve selected about these three books, and separately create three files which store the book’s information in HTML (using an html table), XML, and JSON formats (e.g. “books.html”, “books.xml”, and “books.json”). To help you better understand the different file structures, I’d prefer that you create each of these files “by hand” unless you’re already very comfortable with the file formats.
Write R code, using your packages of choice, to load the information from each of the three sources into separate R data frames. Are the three data frames identical?
Your deliverable is the three source files and the R code. If you can, package your assignment solution up into an .Rmd file and publish to rpubs.com. [This will also require finding a way to make your three text files accessible from the web].
library(jsonlite)
library(XML)
library(dplyr)
library(RCurl)
library(stringr)
library(plyr)
Load using HTML
#Load from HTML
url_html <- "https://raw.githubusercontent.com/monuchacko/cuny_msds/master/data_607/bks1.html"
html_data <- getURL(url_html)
# parse html
html_parse = htmlParse(html_data, asText=TRUE)
text_plain <- xpathSApply(html_parse, "//text()[not(ancestor::script)][not(ancestor::style)][not(ancestor::noscript)][not(ancestor::form)]", xmlValue)
dataCols <- capture.output(cat(paste(text_plain, collapse = " ")))
html_df <- data.frame(dataCols)
html_df
## dataCols
## 1 Top Three Books
## 2
## 3 A Tale of Two Brothers
## 4 Alex Simmons
## 5 Topics
## 6 Broken promise
## 7 Rafiki saved
## 8 A royal crisis
## 9 Audience
## 10 Teachers
## 11 School Kids
## 12 Casual Readers
## 13
## 14 Publish Year
## 15
## 16 1994
## 17 Understanding Computers and Cognition
## 18 Terry Winograd
## 19 Topics
## 20 Towards a new orientation
## 21 Breakdown and the ontology of design
## 22 Computers and representation
## 23 Audience
## 24 IT Professionals
## 25 Executives
## 26 Managers
## 27
## 28 Publish Year
## 29
## 30 1986
## 31 One Universe:: At Home in the Cosmos
## 32 Charles Tsun-Chu Liu
## 33 Topics
## 34 Matter
## 35 Energy
## 36 Frontiers
## 37 Audience
## 38 Scientist
## 39 High School Students
## 40 Teachers
## 41
## 42 Publish Year
## 43
## 44 1998
## 45
#write.csv(html_df, file = "html_df.csv")
Load using XML
#Load from XML
url_xml <- "https://raw.githubusercontent.com/monuchacko/cuny_msds/master/data_607/bks1.xml"
xml_data <- getURL(url_xml)
xml_list <- xmlToList(xml_data)
xml_df <- ldply(xml_list, data.frame)
xml_df
## .id title author
## 1 book A Tale of Two Brothers Alex Simmons
## 2 book Understanding Computers and Cognition Terry Winograd
## 3 book One Universe:: At Home in the Cosmos Charles Tsun-Chu Liu
## topics.topic topics.topic.1
## 1 Broken promise Rafiki saved
## 2 Towards a new orientation Breakdown and the ontology of design
## 3 Matter Energy
## topics.topic.2 audiences.audience audiences.audience.1
## 1 A royal crisis Teachers School Kids
## 2 Computers and representation IT Professionals Executives
## 3 Frontiers Scientist High School Students
## audiences.audience.2 publish_year
## 1 Casual Readers 1994
## 2 Managers 1986
## 3 Teachers 1988
Load using JSON
#Load from JSON
url_json <- "https://raw.githubusercontent.com/monuchacko/cuny_msds/master/data_607/bks1.json"
json_data = jsonlite::fromJSON(url_json)
json_df <- ldply (json_data, data.frame)
json_df
## .id book.title book.authors
## 1 books A Tale of Two Brothers Alex Simmons
## 2 books Understanding Computers and Cognition Terry Winograd
## 3 books One Universe:: At Home in the Cosmos Charles Tsun-Chu Liu
## book.topics
## 1 Broken promise, Rafiki saved, A royal crisis
## 2 Towards a new orientation, Breakdown and the ontology of design, Computers and representation
## 3 Matter, Energy, Frontiers
## book.audiences book.publish_year
## 1 Teachers, School Kids, Casual Readers 1994
## 2 IT Professionals, Executives, Managers 1986
## 3 Scientist, High School Students, Teachers 1988
Conclusion:
The HTML dataset looks different than XML and JSON. This could be because the DOM in HTML is fault tolerance. HTML works even if the data is not well formatted. Extracting data from live websites could be difficult and might require high maintenance. On the other hand XML and JSON for well formatted and does not allow format errors. This type of data is well suited for data extraction.