library(tidyverse)
library(readxl)
library(tidytext)
library(SnowballC)
library(RTextTools)
library(ranger)
library(rsample)
east_books_alma <- read_excel("Physical titles export EAST 2020-2025 (1).xlsx")
east_books_primo <- read_excel("Export_All_5848032800008261 (2).xlsx")
library(htmltools)
library(xml2)
primo_cleaned <- east_books_primo %>%
mutate(title_clean = vapply(Title, function(x) {
xml_text(read_html(paste0("<p>", x, "</p>")))
}, character(1)))
alma_cleaned <- east_books_alma %>%
mutate(title_clean = gsub(" /.*", "", Title)) %>%
arrange(title_clean)
library(lubridate)
library(stringr)
merged <- merge(alma_cleaned, primo_cleaned, by = "title_clean")
merged <- merged %>%
mutate(publish_year = str_extract(`Creation Date.y`, "\\d{4}")) Statistical Learning Project
Predicting Library of Congress Call Numbers Using Description and Subject Text Data
Acknowledgements
I want to extend a very large thanks to the Interlibrary Loan staff members at Middlebury College, in particular Rachel Manning. This whole project would not have been possible without her helping me to acquire the data I needed.
Introduction
For the past two years, I have worked part-time at the Interlibrary Loan Office at the Middlebury College Library. In that time, I have become familiar with the Library of Congress Classification (LCC) System, which sorts items in library collections via subject, each represented with a letter or collection of letters. The main classes are shown below:
| Class | Subject |
|---|---|
| A | General Works |
| B | Philosophy, Psychology, Religion |
| C | Auxiliary Sciences of History |
| D | World History and History of Europe, Asia, Africa, Australia, New Zealand, etc. |
| E | History of the Americas |
| F | History of the Americans |
| G | Geography, Anthropology, Recreation |
| H | Social Sciences |
| J | Political Science |
| K | Law |
| L | Education |
| M | Music and Books on Music |
| N | Fine Arts |
| P | Language and Literature |
| Q | Science |
| R | Medicine |
| S | Agriculture |
| T | Technology |
| U | Military Science |
| V | Naval Science |
| Z | Bibliography, Library Science, Information Resources |
Within each class, there are also several subclasses, each representing different categories of the parent class. For example, the Q class (science) has subcategories QA for Mathematics, QB for Astronomy, and more. Within each of the subclasses, there are further divisions based on subject, which are represented using numbers.
At Middlebury, we have a very substantial library collection with hundreds of thousands of print items, all organized using the LCC system. We also have additional information about each of the books in the collection available online and searchable using the Primo interface, as well as in a back-end program named Alma that deals with circulation.
For this project, I will attempt to find an algorithm that will use the text data included in the book titles, subject tags, descriptions, and chapter names to classify books into their main classes. The application of this would be for automatically assigning LCC categories to works.
The Data
In order to begin this project, I wanted to find a sample of books that was as representative of Middlebury’s print book collection as possible. Due to limitations with the search capabilities of the library software, it was unfortunately neither possible for me to extract the data for all books in the collection and then take a sample myself nor to “randomly” sample from the collection directly in the software. So, I narrowed my search down to books that have been designated as necessary for the library to keep according to the Eastern Academic Scholars’ Trust (EAST), which aims to maintain and protect certain items in print collections across over 100 different libraries. This is because I feared that any other search term that I would filter by would not be at all representative of the overall collection. This is not to say that the EAST books are fully representative; the list of books to be retained is heavily skewed towards items in the Language and Literature category (P). I will discuss the implications of this as they come up in my analysis.
Due to restrictions with the file exporting capabilities of my boss, who provided the dataset, she restricted the subset of books to categorize to those published between 2005 and 2010. It seems like because of some quirks in the library system’s software, it included a minimal number of books published at times other than these, as can be seen in Figure 1. I assumed these were likely mistakes, and since they represented such a small number in the overall dataset and were simply extra text data the algorithms could use, I left them in. There are obviously reasons why narrowing down publication dates could be disadvantageous. For example, we may be excluding many older books which would include more outdated terms and language which would not be represented in our current data but may be represented in the library collection. If our goal with this algorithm were to actually use it for classification of new books, this shouldn’t matter much (although it would be better to use books published even more recently in that case). As much as I know that this narrowing of publishing dates is an issue, this was the only data I had access to and thus I could not dig into the effect of this in a quantitative manner.
As I alluded to earlier, the data came from two different sources: Alma, which had data that included the call number, and Primo, which included the description and chapter names. So, I had to clean the title column so that the dataframes could merge, as can be seen below as well.
merged %>%
group_by(publish_year) %>%
count() %>%
ggplot() +
geom_col(aes(x = n, y = publish_year)) +
theme_bw() +
labs(title = "Histogram of Publishing Year of Books Included in the Dataset",
x = "Count",
y = "Publishing Year")
Approach 1: A new supervised learning model, Support Vector Machines
When trying to determine what new model would be most appropriate for conducting this analysis, I looked to see if there was any prior research into using machine learning techniques to classify library call numbers. What I found was a paper by Eibe Frank and Gordon W. Paynter titled “Predicting Library of Congress Classifications from Library of Congress Subject Headings”. The authors opted to use support vector machines as their algorithm of choice, which led me to believe that it would be an appropriate choice for this project and that it would work in the context of using text data to classify data points. In this paper, the authors chose to only use the Library of Congress subject headings and to ignore description text data as well as titles.
Here, because I was working with a smaller data set, I decided to include more text data, including the descriptions and contents of books to the analysis. As you will see in the code below, I combined all of the text data available into one large “text” column, which was then turned into a document term matrix using the framework of the RTextTools package. Additionally, I do want to note that not every book entry included a description. This likely leads to lower confidence predictions and more misclassifications. I decided to keep these books in the dataset despite this as they still do contribute some text data and can still be classified despite that classification being more likely to be wrong. If I had more time, I would also look into whether some classes were more affected by the missing descriptions.
merged %>%
group_by(has_description = !is.na(Description)) %>%
count() %>%
ggplot() +
geom_col(aes(x = n, y = has_description)) +
theme_bw() +
labs(title = "Number of books with and without descriptions")
merged %>%
group_by(has_subject = !is.na(Subject.x)) %>%
count() %>%
ggplot() +
geom_col(aes(x = n, y = has_subject)) +
theme_bw() +
labs(title = "Number of books with and without subject")
So, before we begin, how do support vector machines even work?
Suppose we are trying to classify data points with two variables into two separate classes, Class A and Class B. Support vector machines try to draw the “best” line through the data such that on one side of the line, all the points are predicted to be in class A and on the other side of the line all the points are predicted to be in class B. We can now extend this to multiple classes and multiple dimensions, finding the best set of hyperplanes in a multidimensional setting. In the context of LCC, I chose to create a “Document Term Matrix”, where each row is a book and the columns are unique words, with cells containing the frequency of that particular word in that book’s text data. This then allows us to have an n-dimensional dataset, where n is the number of words seen in all of the books’ text data combined, and each book has a count of how many times each word was used in its text data. From there, we can imagine each of these books being a point in this n-dimensional space, and then drawing the “best” set of planes that separates these different classes.
Note 1: What does “best” mean?
Look at the figure below demonstrating the books in class M (music) and class Q (technology) that include the words “symphony” and “algebra”. In the first figure we see that it is quite easy to draw a line that perfectly separates the two groups. However, assume there was a book whose text data included the word symphony three times and the word algebra eight times. We’d have to redraw our line in order to include that single point, and that redrawing would significantly change how our line looks and how it would potentially predict the classification of future points. This means that the variance of this approach would be very high, and in order to have an algorithm that is more robust to certain data changes, we need to potentially allow for some misclassifications. So, SVM accounts for this by introducing a soft margin, which means it allows some points to be misclassified in order to find a decision boundary that not only separates the classes but also leaves as much space as possible between itself and the nearest points from each class — this space is known as the margin, and a wider margin helps the model generalize better to new, unseen data.
merged <- merged %>%
mutate(description_clean = vapply(Description, function(x) {
xml_text(read_html(paste0("<p>", x, "</p>")))
}, character(1))) %>%
mutate(contents_clean = vapply(Contents, function(x) {
xml_text(read_html(paste0("<p>", x, "</p>")))
}, character(1))) %>%
mutate(subject_clean = vapply(Subject.y, function(x) {
xml_text(read_html(paste0("<p>", x, "</p>")))
}, character(1)))
merged_text <- merged %>%
mutate(text = paste(title_clean, description_clean, contents_clean, subject_clean, Genre)) %>%
select(call_num = `LC Call Number`, text) %>%
mutate(main_class = substr(call_num, start = 1, stop = 1)) %>%
filter(!is.na(main_class)) %>%
distinct(text, .keep_all = TRUE)Show the code
#IGNORE THIS CHUNK OF CODE. IT IS NEEDED FOR THE EXPLANATION OF THE SVM TO RUN, BUT WILL BE EXPLAINED LATER IN THE PCA SECTION.
dfm_merged <- merged_text %>%
unnest_tokens(input = "text",
output = "Word") %>%
filter(!(Word %in% stop_words$word)) %>%
mutate(Word = SnowballC::wordStem(Word)) %>%
pivot_wider(id_cols = c("call_num", "main_class"),
names_from = "Word",
values_from = "Word",
values_fn = length,
values_fill = 0)dfm_merged %>%
select(main_class, symphoni, algebra) %>%
filter(main_class %in% c("M", "Q")) %>%
filter(!(symphoni == 0) | !(algebra == 0)) %>%
ggplot() +
geom_jitter(aes(x = symphoni, y = algebra, color = main_class), width = 0.15) +
theme_minimal() +
geom_vline(xintercept = 0.5) +
labs(title = "Potential Classifier that Perfectly Separates Classes")
dfm_merged %>%
select(main_class, symphoni, algebra) %>%
filter(main_class %in% c("M", "Q")) %>%
filter(!(symphoni == 0) | !(algebra == 0)) %>%
ggplot() +
geom_jitter(aes(x = symphoni, y = algebra, color = main_class), width = 0.1) +
theme_minimal() +
geom_abline(intercept = -1, slope = 2.5, color = "red") +
geom_point(data = data.frame(symphoni = 3, algebra = 8), aes(x = symphoni, y = algebra), color = "Blue", size = 3) +
labs(title = "Potential Classifier that Perfectly Separates Classes",
subtitle = "With Hypothetical New Point Added")
SVM explanation
Note 2:
In this analysis, I use a linear classifier (flat hyperplanes) in my SVM algorithm. However, it is possible (and in many other cases preferable) to use other “kernels” for this algorithm, which would be able to better deal with data that has a clear non-linear classification pattern. Nevertheless, since I am using very sparse text data, the linear kernel is preferable because there are a lot of features and changing kernels does not seem to increase performance.
doc_matrix <- create_matrix(merged_text$text, removeStopwords = TRUE, stemWords = TRUE)
container <- create_container(doc_matrix, as.factor(merged_text$main_class), trainSize = 1:2500, testSize = 2501:4950, virgin = FALSE)
svm_model <- train_model(container, "SVM", kernel = "linear")
svm_classify <- classify_model(container, svm_model)
# create_analytics function didn't work; manually compared predications to actual resutls.
# analytics <- create_analytics(container = container, classification_results = cbind(svm_classify), b = 1)
preds_results <- tibble(actual = merged_text[2501:4950, ]$main_class, preds = svm_classify$SVM_LABEL) %>%
mutate(is_correct = (actual == preds))preds_results %>%
group_by(actual) %>%
summarize(class_acc = mean(is_correct),
class_count = n()) %>%
ggplot() +
geom_col(aes(x = class_acc, y = reorder(actual, class_count), fill = class_count)) +
scale_fill_gradientn(colors = c("lightblue", "blue", "darkblue"),
values = scales::rescale(c(0, 100, 500))) +
theme_minimal() +
labs(x = "Class Accuracy", y = "Main Class", title = "Accuracy of SVM by Class, ordered by Class Count", subtitle = "Training using first 2500 rows")
I was told by my boss that the order of the books in the dataset seemed random. However, I wanted to make sure that my results were replicable with a random sample instead of what happened to be the first 2500 rows, since I was worried that there was some underlying ordering to the data that I wasn’t aware of that may have impacted the results. So, the following chunk of code and charts show the results when I shuffle the dataset and repeat the same process.
mt_shuffled <- merged_text %>% sample_frac(1)
doc_matrix <- create_matrix(mt_shuffled$text, removeStopwords = TRUE, stemWords = TRUE)
container <- create_container(doc_matrix, as.factor(mt_shuffled$main_class), trainSize = 1:2500, testSize = 2501:4950, virgin = FALSE)
svm_model <- train_model(container, "SVM", kernel = "linear")
svm_classify <- classify_model(container, svm_model)
# create_analytics function didn't work; manually compared predications to actual resutls.
# analytics <- create_analytics(container = container, classification_results = cbind(svm_classify), b = 1)
preds_results <- tibble(actual = mt_shuffled[2501:4950, ]$main_class, preds = svm_classify$SVM_LABEL) %>%
mutate(is_correct = (actual == preds))preds_results %>%
group_by(actual) %>%
summarize(class_acc = mean(is_correct),
class_count = n()) %>%
ggplot() +
geom_col(aes(x = class_acc, y = reorder(actual, class_count), fill = class_count)) +
scale_fill_gradientn(colors = c("lightblue", "blue", "darkblue"),
values = scales::rescale(c(0, 100, 500))) +
theme_minimal() +
labs(x = "Class Accuracy", y = "Main Class", title = "Accuracy of SVM by Class, ordered by Class Count", subtitle = "Trainig using random 2500 rows")
# Trying to find most common misclassifications
preds_results %>%
group_by(actual) %>%
summarize(class_acc = mean(is_correct),
class_count = n(),
is_correct = is_correct,
preds = preds) %>%
ungroup() %>%
group_by(preds) %>%
summarize(class_count_p = n(),
class_count = n(),
is_correct = is_correct,
preds = preds) %>%
ungroup() %>%
filter(is_correct == FALSE) %>%
group_by(preds) %>%
summarize(count = n(),
class_count = class_count) %>%
distinct() %>%
ggplot() +
geom_col(aes(x = count, y = reorder(preds, class_count), fill = class_count)) +
scale_fill_gradientn(colors = c("lightblue", "blue", "darkblue"),
values = scales::rescale(c(0, 100, 500))) +
theme_minimal() +
labs(title = "Number of Predictions that Incorrectly Predicted this Class")
What do these results show us?
Here, we can see multiple things.
The difference between the effectiveness of SVM on the randomly sampled data vs. simply taking the first half of the rows from the dataset initially given to me seems minimal, indicating that the dataset was ordered pseudo-randomly.
The accuracy is by far the highest for category P, which makes sense since it makes up 2046 out of 4950 rows in our data set. Beyond P having the highest accuracy, there doesn’t seem to be a strong relationship between the number of books in a category present in the dataset (here represented using the fill color) and the accuracy of the predictions on our testing data.
Unsurprisingly, the category that the most books were miscategorized as was P, our most populous category. Based on even just the intuition of how SVMs work, it is clear to see how imbalanced datasets are likely to impact the quality of predictions made, and this is backed up by previous research. As discussed in the paper by Batuwita and Palade just linked, there are techniques available to reduce the bias towards the majority class. If there were more time available for this project, I would explore the effectiveness of resampling the data to have a more even number of books of each class.
Approach 2: Ranger and Principal Component Analysis
The next approach I wanted to explore involved replicating the methods we saw in class using ranger and Principal Component Analysis (PCA).
What does PCA look like in the context of call number classification? Here, we will first replicate the methods we saw in class to make a table that has a row for every book in the dataset, columns for every single word that appears in any of the book entries, and values representing the number of times said word appeared in either the book title, subjects, and description of the book (basically a document term matrix again!). Then, the PCA algorithm is able to compress all those word‐count columns into a handful of “principal components”—new axes that explain most of the differences in how books use words. That is, the algorithm is able to group together words by how “similar” they are in the context of the dataset and make new scales based on those similarities, each being a separate principal component. We can then use the “ranger” function, which builds hundreds of decision trees given the books’ true call numbers and their PCA scores in each component. It can then find the best decision tree by determining which of the decisions made were most common when taking different samplings of the data.
I want to note that at first I tried to perform PCA on the entire dataset and although it worked, it took a very long time to run. So, initially, I instead took a random sample of half of the rows in my merged_text dataframe (we will soon see that this is a flawed approach in this scenario, and will adjust accordingly).
sample_rows <- sample(1:nrow(merged_text),
size = nrow(merged_text)/2)
sample_data <- merged_text[sample_rows, ]
dfm_sample <- sample_data %>%
unnest_tokens(input = "text",
output = "Word") %>%
filter(!(Word %in% stop_words$word)) %>%
mutate(Word = SnowballC::wordStem(Word)) %>%
pivot_wider(id_cols = c("call_num", "main_class"),
names_from = "Word",
values_from = "Word",
values_fn = length,
values_fill = 0)
pca_sample <- prcomp(dfm_sample %>%
select(-call_num, -main_class), scale = TRUE)
rf1 <- ranger(factor(main_class) ~ PC1_score + PC2_score + PC3_score + PC4_score + PC5_score + PC6_score + PC7_score + PC8_score,
data = dfm_sample %>%
mutate(PC1_score = pca_sample$x[, "PC1"],
PC2_score = pca_sample$x[, "PC2"],
PC3_score = pca_sample$x[, "PC3"],
PC4_score = pca_sample$x[, "PC4"],
PC5_score = pca_sample$x[, "PC5"],
PC6_score = pca_sample$x[, "PC6"],
PC7_score = pca_sample$x[, "PC7"],
PC8_score = pca_sample$x[, "PC8"]),
importance = "impurity")
cf <- rf1$confusion.matrix
library(kableExtra)
kable(cf)| A | B | C | D | E | F | G | H | J | K | L | M | N | P | Q | R | S | T | U | V | Z | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| A | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| B | 0 | 7 | 0 | 6 | 1 | 0 | 0 | 15 | 0 | 2 | 1 | 1 | 2 | 112 | 8 | 0 | 0 | 0 | 0 | 0 | 0 |
| C | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 9 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| D | 0 | 8 | 0 | 24 | 0 | 0 | 0 | 8 | 3 | 2 | 0 | 1 | 1 | 154 | 2 | 0 | 0 | 0 | 1 | 0 | 0 |
| E | 0 | 4 | 0 | 6 | 0 | 0 | 0 | 2 | 0 | 1 | 0 | 0 | 2 | 47 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| F | 0 | 3 | 0 | 3 | 1 | 0 | 0 | 2 | 0 | 0 | 0 | 0 | 0 | 83 | 1 | 0 | 0 | 0 | 0 | 0 | 0 |
| G | 0 | 4 | 0 | 2 | 0 | 0 | 0 | 8 | 0 | 0 | 0 | 0 | 0 | 57 | 9 | 0 | 0 | 0 | 0 | 0 | 0 |
| H | 0 | 9 | 0 | 8 | 0 | 0 | 2 | 40 | 0 | 1 | 3 | 0 | 2 | 142 | 20 | 1 | 0 | 1 | 0 | 0 | 0 |
| J | 0 | 3 | 0 | 7 | 0 | 0 | 0 | 9 | 0 | 2 | 2 | 0 | 0 | 33 | 1 | 0 | 0 | 0 | 0 | 0 | 0 |
| K | 0 | 1 | 0 | 5 | 0 | 0 | 0 | 5 | 1 | 3 | 0 | 0 | 0 | 25 | 4 | 0 | 0 | 0 | 0 | 0 | 0 |
| L | 0 | 3 | 0 | 1 | 0 | 0 | 0 | 15 | 1 | 0 | 1 | 0 | 0 | 37 | 3 | 1 | 0 | 0 | 0 | 0 | 0 |
| M | 0 | 2 | 0 | 3 | 0 | 0 | 0 | 5 | 0 | 0 | 0 | 0 | 0 | 54 | 2 | 0 | 0 | 0 | 0 | 0 | 0 |
| N | 0 | 5 | 0 | 3 | 1 | 0 | 0 | 3 | 0 | 1 | 2 | 1 | 2 | 120 | 4 | 0 | 0 | 0 | 0 | 0 | 0 |
| P | 0 | 10 | 0 | 16 | 0 | 2 | 2 | 29 | 0 | 0 | 3 | 1 | 6 | 915 | 20 | 0 | 0 | 0 | 0 | 0 | 0 |
| Q | 0 | 5 | 0 | 0 | 0 | 0 | 0 | 15 | 1 | 1 | 0 | 0 | 1 | 55 | 63 | 1 | 0 | 0 | 0 | 0 | 0 |
| R | 0 | 0 | 0 | 1 | 0 | 0 | 1 | 10 | 0 | 0 | 0 | 0 | 1 | 13 | 9 | 0 | 0 | 0 | 0 | 0 | 0 |
| S | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 7 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| T | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 7 | 0 | 0 | 1 | 0 | 1 | 40 | 5 | 0 | 0 | 0 | 0 | 0 | 0 |
| U | 0 | 0 | 0 | 2 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 8 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| V | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| Z | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 7 | 1 | 0 | 0 | 0 | 0 | 0 | 0 |
This is pretty horrible! This seems to predict that almost every book in our data set will be in the Literature category because the class P is much more strongly represented in our data set than any other class.
So, let us try to use stratified sampling instead to ensure that all classes are (more) fairly represented. This will perhaps help with our problem of over predicting the class P over everything else.
merged_text_sample <- merged_text %>%
group_by(main_class) %>%
mutate(count = n()) %>%
slice(if (first(count) < 50) 1:n() else sample(n(), 50)) %>%
ungroup()
dfm_sample_strat <- merged_text_sample %>%
unnest_tokens(input = "text",
output = "Word") %>%
filter(!(Word %in% stop_words$word)) %>%
mutate(Word = SnowballC::wordStem(Word)) %>%
pivot_wider(id_cols = c("call_num", "main_class"),
names_from = "Word",
values_from = "Word",
values_fn = length,
values_fill = 0)
pca_sample_strat <- prcomp(dfm_sample_strat %>%
select(-call_num, -main_class), scale = TRUE)
n_pcs <- ncol(pca_sample_strat$x)
explained <- cumsum(pca_sample_strat$sdev^2 / sum(pca_sample_strat$sdev^2))
n_pcs <- which(explained >= 0.9)[1]
pc_scores <- as.data.frame(pca_sample_strat$x[, 1:n_pcs])
df_pca <- bind_cols(dfm_sample_strat, pc_scores)
pc_vars <- paste0("PC", 1:n_pcs)
#idea for how to use many PCA scores at once from ChatGPT
formula <- as.formula(paste("factor(main_class) ~", paste(pc_vars, collapse = " + ")))
rf_model <- ranger(formula, data = df_pca, importance = "impurity")
cf2 <- rf_model$confusion.matrix
kable(cf2)| A | B | C | D | E | F | G | H | J | K | L | M | N | P | Q | R | S | T | U | V | Z | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| A | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| B | 0 | 15 | 0 | 4 | 2 | 3 | 1 | 1 | 3 | 1 | 7 | 1 | 3 | 4 | 3 | 1 | 0 | 1 | 0 | 0 | 0 |
| C | 0 | 1 | 0 | 4 | 0 | 2 | 3 | 2 | 2 | 2 | 1 | 0 | 2 | 1 | 0 | 1 | 0 | 2 | 0 | 0 | 0 |
| D | 0 | 1 | 0 | 19 | 3 | 4 | 2 | 1 | 3 | 4 | 0 | 3 | 1 | 4 | 1 | 1 | 0 | 3 | 0 | 0 | 0 |
| E | 0 | 1 | 0 | 4 | 20 | 6 | 3 | 4 | 1 | 1 | 3 | 1 | 0 | 3 | 1 | 2 | 0 | 0 | 0 | 0 | 0 |
| F | 0 | 3 | 0 | 5 | 6 | 24 | 2 | 1 | 3 | 0 | 0 | 2 | 1 | 3 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| G | 0 | 7 | 0 | 4 | 2 | 6 | 10 | 1 | 1 | 1 | 6 | 2 | 2 | 4 | 2 | 1 | 0 | 1 | 0 | 0 | 0 |
| H | 0 | 3 | 0 | 4 | 3 | 3 | 2 | 1 | 4 | 4 | 9 | 4 | 2 | 2 | 4 | 3 | 0 | 2 | 0 | 0 | 0 |
| J | 0 | 2 | 0 | 8 | 3 | 1 | 1 | 7 | 12 | 3 | 7 | 0 | 2 | 2 | 0 | 2 | 0 | 0 | 0 | 0 | 0 |
| K | 0 | 2 | 0 | 4 | 4 | 1 | 2 | 2 | 1 | 21 | 2 | 1 | 1 | 0 | 4 | 4 | 0 | 1 | 0 | 0 | 0 |
| L | 0 | 1 | 0 | 1 | 2 | 1 | 1 | 3 | 2 | 0 | 30 | 2 | 1 | 1 | 3 | 2 | 0 | 0 | 0 | 0 | 0 |
| M | 0 | 1 | 0 | 3 | 1 | 3 | 1 | 1 | 0 | 0 | 0 | 26 | 3 | 8 | 0 | 2 | 0 | 1 | 0 | 0 | 0 |
| N | 0 | 3 | 0 | 4 | 1 | 1 | 1 | 2 | 1 | 2 | 2 | 2 | 16 | 7 | 2 | 1 | 0 | 5 | 0 | 0 | 0 |
| P | 0 | 6 | 0 | 6 | 0 | 1 | 4 | 0 | 0 | 2 | 0 | 7 | 1 | 17 | 1 | 3 | 0 | 2 | 0 | 0 | 0 |
| Q | 0 | 2 | 0 | 0 | 0 | 0 | 1 | 2 | 2 | 0 | 3 | 5 | 1 | 1 | 19 | 11 | 0 | 3 | 0 | 0 | 0 |
| R | 0 | 0 | 0 | 0 | 2 | 1 | 0 | 0 | 1 | 4 | 2 | 2 | 2 | 0 | 8 | 27 | 0 | 1 | 0 | 0 | 0 |
| S | 0 | 0 | 0 | 0 | 1 | 5 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 4 | 0 | 0 | 0 | 0 |
| T | 0 | 2 | 0 | 1 | 1 | 2 | 1 | 2 | 1 | 0 | 4 | 5 | 5 | 9 | 2 | 5 | 0 | 10 | 0 | 0 | 0 |
| U | 0 | 1 | 0 | 0 | 3 | 0 | 0 | 0 | 4 | 1 | 2 | 2 | 1 | 0 | 1 | 1 | 0 | 1 | 0 | 0 | 0 |
| V | 0 | 0 | 0 | 0 | 1 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 1 | 0 | 0 | 0 | 0 | 0 |
| Z | 0 | 2 | 0 | 1 | 1 | 0 | 0 | 0 | 0 | 1 | 3 | 1 | 3 | 0 | 5 | 1 | 0 | 2 | 0 | 0 | 2 |
rowsums1 <- cf %>% rowSums()
correct1 <- NULL
for(row in 1:nrow(cf)) {
correct1[row] <- cf[row, row]
}
cf1_accuracy <- tibble(class = rownames(cf), acc = correct1/rowsums1)
rowsums2 <- cf2 %>% rowSums()
correct2 <- NULL
for(row in 1:nrow(cf2)) {
correct2[row] <- cf2[row, row]
}
cf2_accuracy <- tibble(class = rownames(cf2), acc = correct2/rowsums2)cf1_accuracy %>%
ggplot() +
geom_col(aes(x = acc, y = reorder(class, acc)), fill = "darkblue") +
theme_minimal() +
labs(title = "Accuracy of Random Sampling PCA Ranger",
x = "Accuracy Rate",
y = "Main Class")
cf2_accuracy %>%
ggplot() +
geom_col(aes(x = acc, y = reorder(class, acc)), fill = "darkblue") +
theme_minimal() +
labs(title = "Accuracy of Stratified Sampling PCA Ranger",
x = "Accuracy Rate",
y = "Main Class")
PCA Ranger Accuracy
Here, we see that although our random sample correctly predicted P with very high accuracy, our stratified sample worked better for every other class. It would be interesting to see how much better the algorithm would get if we had a larger sample size for some of these classes, as class A for example only has two entries!
Conclusion
With both approaches, we were able to predict the LCC main class of library books at a much higher rate than random chance (which in this case would be a 1 in 21 chance, or a 4.8% success rate). It seems like with the limited data we had, SVMs did a better job at accurately predicting class (we can see this by simply comparing our accuracy column charts for both methods). However, if we had more data available, or the time to better tweak the ranger algorithm, then the results may change! This is all reason for more future investigation into the topic :)