Besides the usual and useful tidyverse library, three additional libraries are downloaded. They are: arsenal, jsonlite, and rvest. I have discovered these in the various sources I explored, including the textbook “R for Data Science” textbook.
library(tidyverse)
── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ──
✔ dplyr 1.2.1 ✔ readr 2.2.0
✔ forcats 1.0.1 ✔ stringr 1.6.0
✔ ggplot2 4.0.3 ✔ tibble 3.3.1
✔ lubridate 1.9.5 ✔ tidyr 1.3.2
✔ purrr 1.2.2
── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
✖ dplyr::filter() masks stats::filter()
✖ dplyr::lag() masks stats::lag()
ℹ Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errors
library(rvest)
Attaching package: 'rvest'
The following object is masked from 'package:readr':
guess_encoding
library(jsonlite)
Attaching package: 'jsonlite'
The following object is masked from 'package:purrr':
flatten
library(arsenal)
Attaching package: 'arsenal'
The following object is masked from 'package:lubridate':
is.Date
The basic data
For this exercise I have retrieved information about three American plays from the XXth century which have been made into movies. The information was retrieved from the CUNY library system (https://guides.newman.baruch.cuny.edu/SPS/#).
Information about the author(s), year and city of publication, and the publisher of each book was recorded both in html and JSON format. Both files have been uploaded in GitHub, from where they were retrieved.
{html_document}
<html>
[1] <body>\r\n <section><h2 data-id="1"> The Odd Couple</h2>\r\n <p> ...
[2] <body>\r\n<section><h2 data-id="2"> The Man who came to Dinner </h2>\r\n ...
[3] <body>\r\n<section><h2 data-id="3"> The Front Page </h2>\r\n <p> Au ...
Title Author Year City
1 The Odd Couple Neil Simon 1966 New York
2 The Man who came to Dinner George Kaufman, Moss Hart 1939 New York
3 The Front Page Ben Hecht, Charles MacArthur 1928 New York
Publisher
1 Random House
2 Random House
3 Covici-Friede
If I did not use the raw version from GitHub, I could not read the JSON file in R. Although both the regular and raw versions seemed to work for the hmtl file, I have used the raw version for both files.
From getting the data to getting a data frame
It can be seen above that adding “simplifyVector = TRUE” to the read_json command allows converting the data directly into a data frame when reading the JSON file. However, this is not the case with the html file. At least, I have not seen such a command.
Thus, I have looked for the elements in the html file,
section <- bookshtml |>html_elements("section")
section
{xml_nodeset (3)}
[1] <section><h2 data-id="1"> The Odd Couple</h2>\r\n <p> Author: <span ...
[2] <section><h2 data-id="2"> The Man who came to Dinner </h2>\r\n <p> A ...
[3] <section><h2 data-id="3"> The Front Page </h2>\r\n <p> Author: <spa ...
As this file has three nodes, it seems to have the structure needed to be converted into a data frame. I had to do this step several times, as I was getting strings with 0 nodes. In such cases, I did not manage to create proper data frames. Also, as I proceeded with the comparison with the JSON file (please, see below), I realized the names I was using for the columns did not match the ones in the JSON file. This involved looking at and renaming the various elements (e.g, the author, the year of publication, etc.) several times in the original html file.
Following the example in the textbook, all these elements were explored one by one. In the process I also realized some were not properly read in R from the html file. It was puzzling as I could see all elements when I opened the file in the browser. However, R refused to take them (the issue was corrected when I used straight quotation marks instead of curly ones in the label of the span class). Then, I used them all together in one command to create a tibble. The tibble has been used to generate the data frame, also in the same command.
# A tibble: 3 × 5
Title Author Year City Publisher
<chr> <chr> <chr> <chr> <chr>
1 The Odd Couple Neil Simon 1966 New … Random H…
2 The Man who came to Dinner George Kaufman and Moss Hart 1939 New … Random H…
3 The Front Page Ben Hecht and Charles MacArt… 1928 New … Covici-F…
Compare Object
Function Call:
comparedf(x = datahtml, y = booksJSON)
Shared: 5 non-by variables and 3 observations.
Not shared: 0 variables and 0 observations.
Differences found in 0/4 variables compared.
0 variables compared have non-identical attributes.
The five variables (columns in the data frame) coincide. Also, the values, along each of the three rows, do coincide.
Table: Summary of data.frames
version arg ncol nrow
-------- ---------- ----- -----
x datahtml 5 3
y booksJSON 5 3
Table: Summary of overall comparison
statistic value
------------------------------------------------------------ ------
Number of by-variables 0
Number of non-by variables in common 5
Number of variables compared 4
Number of variables in x but not y 0
Number of variables in y but not x 0
Number of variables compared with some values unequal 0
Number of variables compared with all values equal 4
Number of observations in common 3
Number of observations in x but not y 0
Number of observations in y but not x 0
Number of observations with some compared variables unequal 0
Number of observations with all compared variables equal 3
Number of values unequal 0
Table: Variables not shared
------------------------
No variables not shared
------------------------
Table: Other variables not compared
var.x pos.x class.x var.y pos.y class.y
------- ------ ---------- ------- ------ --------
Author 2 character Author 2 list
Table: Observations not shared
---------------------------
No observations not shared
---------------------------
Table: Differences detected by variable
var.x var.y n NAs
---------- ---------- --- ----
Title Title 0 0
Year Year 0 0
City City 0 0
Publisher Publisher 0 0
Table: Differences detected
------------------------
No differences detected
------------------------
Table: Non-identical attributes
----------------------------
No non-identical attributes
----------------------------
As I have not changed the default setting to compare by any specific variable, there are no “by” variables, they are all “non-by” variables. This was explained better in https://mayoverse.github.io/arsenal/articles/comparedf.html than the original website where I had found the comparedf command.
Two further points
I tried to create two different entries for the authors in the JSON file (following the explanation to use square brackets to include an array instead of a value in the key-value pair I have found in: https://www.freecodecamp.org/news/what-is-json-a-json-file-example/ ). However, it did not work. Also, I could not find a way to do the same thing in html.
While everything seems to match, it is intriguing that only four, not all five, variables were compared. The Author variable was not compared but it is not clear to me why. It could be related to the issue in the previous paragraph. I have tried to explore (not using generative AI) but I could not find an explanation. Nevertheless, I would add, I would not have been able to realize this without the summary command.