Week 7 - HTML and JSON

Author

Muhammad Ali

HTML & JSON Scraping

Approach

Background and Information

This assignment focuses on building the foundation for web scraping, working with HTML and JSON. Not only it is to build these foundations but it will create a path for one to understand and work with APIs.

Terminology

  • HTML: A structural code for documents that creates the foundation of websites that we see on the internet.

  • JSON: Short for JavaScript Object Notation, it is used to store hierarchical data (nested information) and communicates between user and website.

  • API: Set of rules that allows two softwares to communicate, follow rules, and to share data with each other. In this case, requesting data from a website.

  • Web Scraping: A process where a computer program automatically extract data from websites and stores it in a dataset.

Goal

The goal for this assignment is to manually create two data files and to store it in:

  • HTML file

  • JSON file

What both of these files will contain are attributes about books such as

  • Title

  • Author(s)

And additional attributes to list a few:

  • Genre

  • Publisher

  • ISBN

  • Rating

The Plan

  1. In order to proceed with this, first we need to find 3 books to do this on, which will be mentioned when starting the codebase portion of this assignment.
  2. After deciding the three books, manually create an .html and .json file for the books.
  3. Load the two files into R using libraries that handle these file types and compare how identical the dataframes are or aren’t.

Codebase

Install Packages

Code 
#install.packages(c("tidyverse", "rvest", "robotstxt", "jsonlite"))

Import Libraries

Code 
library(tidyverse)
library(rvest)
library(robotstxt)
library(jsonlite)

Books Selected:

  • R for Data Science

  • Python Data Science Handbook

  • Hands-On Programming with R

Reading the HTML

Code 
books_html <- read_html("https://raw.githubusercontent.com/HaiderrX/CUNY-SPS-MSDS/refs/heads/main/DATA607/Week%207%20-%20Web%20Technologies/books.html", encoding = "UTF-8") |>
  html_element("table") |>
  html_table()

books_html
# A tibble: 3 × 4
  title                        authors                            year publisher
  <chr>                        <chr>                             <int> <chr>    
1 R for Data Science           Hadley Wickham, Mine Çetinkaya-R…  2023 O'Reilly…
2 Python Data Science Handbook Jake VanderPlas                    2023 O'Reilly…
3 Hands-On Programming with R  Garrett Grolemund                  2014 O'Reilly…

Reading JSON File

Code 
books_json <- fromJSON("https://raw.githubusercontent.com/HaiderrX/CUNY-SPS-MSDS/refs/heads/main/DATA607/Week%207%20-%20Web%20Technologies/books.json") |>
  as_tibble()

books_json
# A tibble: 3 × 4
  title                        authors                            year publisher
  <chr>                        <chr>                             <int> <chr>    
1 R for Data Science           Hadley Wickham, Mine Çetinkaya-R…  2023 O'Reilly…
2 Python Data Science Handbook Jake VanderPlas                    2023 O'Reilly…
3 Hands-On Programming with R  Garrett Grolemund                  2014 O'Reilly…
Code 
str(books_json)
tibble [3 × 4] (S3: tbl_df/tbl/data.frame)
 $ title    : chr [1:3] "R for Data Science" "Python Data Science Handbook" "Hands-On Programming with R"
 $ authors  : chr [1:3] "Hadley Wickham, Mine Çetinkaya-Rundel, Garrett Grolemund" "Jake VanderPlas" "Garrett Grolemund"
 $ year     : int [1:3] 2023 2023 2014
 $ publisher: chr [1:3] "O'Reilly Media" "O'Reilly Media" "O'Reilly Media"

Comparing HTML and JSON files

Code 
identical(as_tibble(books_html), as_tibble(books_json))
[1] TRUE
Code 
all.equal(as_tibble(books_html), as_tibble(books_json))
[1] TRUE

After running identical() and all.equal(), both HTML and JSON tibbles are equivalent to each other. However initially they weren’t as the encoding for the HTML was not set to “UTF-8” which means to allow any digit or character in text which includes the ‘Ç’ that was presented in one of the authors. JSON already handles UTF-8 encoding but for HTML it has to be manually placed into the code or we can place it in as an argument for the function read_html() in R.

Prior to converting them to tibbles, they were both not equivalent as the console was printing out:

identical(books_html, books_json) [1] FALSE 
all.equal(books_html, books_json) [1] "Attributes: < Component “class”: Lengths (3, 1) differ (string compare on first 1) >" [2] "Attributes: < Component “class”: 1 string mismatch >"

This was due to the fact that JSON was uploaded as a dataframe and the HTML was already uploaded as a tibble, which means in order for them to be equivalent with each other JSON must be converted to a tibble.

Conclusion

The road to understanding web scraping first starts with understanding how the data is being presented and stored, it could be in a dataframe, a tibble, or in this case HTML and JSON.

In this assignment creating an .html and .json file highlighted the structure of how things are formatted, nested or in various tables (soon to come). But for this assignment it was relatively easy to follow the pattern to creating.

After loading both data files into R, visually comparing them it would seem that they are exactly the same, however under the hood it is different which showcases that one must be cautious when reading the data to make sure it is formatted properly, in this example making sure that the HTML was UTF-8 encoded and the JSON is converted to a tibble at the end.

But as these are the foundations, there will be more to come as the complexities with web scraping and eventually working with APIs will become a challenge.