library(tidytext)
library(janeaustenr)
library(gutenbergr)
library(dplyr)
library(tidyr)
library(stringr)
library(ggplot2)
library(knitr)Sentiment Analysis
Sentiment Analysis
Approach
In this assignment, I plan to reproduce the sentiment analysis example from Chapter 2 of Text Mining with R using Jane Austen’s novels. I will then extend the analysis by examining Mary Shelley’s Frankenstein, which is available through Project Gutenberg. I will use R and the tidytext package to organize the text into individual words and analyze sentiment using the Bing, NRC, and AFINN lexicons. I will also introduce the Loughran lexicon to compare how different sentiment dictionaries classify positive and negative words. I plan to create visualizations showing changes in sentiment throughout the novel and identify the most frequently occurring positive and negative words.
One challenge I anticipate is that sentiment dictionaries may classify the same words differently. Additionally, these methods do not always consider the context in which words are used, which could affect the results. Since the Loughran lexicon was developed for financial text it may not accurately capture sentiment in a literary novel. My goal is to compare the findings with the original Jane Austen example and examine how the choice of text and sentiment lexicon influences the results.
Data Sources: Text Mining with R, Chapter 2: https://www.tidytextmining.com/sentiment Project Gutenberg, Frankenstein
References: Silge, J., & Robinson, D. (2017). Text Mining with R: A Tidy Approach. O’Reilly Media. https://www.tidytextmining.com/ Shelley, M. (1818). Frankenstein; or, The Modern Prometheus. Project Gutenberg.
Codebase
Step 1: Reproducing the Chapter 2 example
tidy_books <- austen_books() %>%
group_by(book) %>%
mutate(
linenumber = row_number(),
chapter = cumsum(str_detect(text, regex("^chapter [\\divxlc]", ignore_case = TRUE)))
) %>%
ungroup() %>%
unnest_tokens(word, text)
kable(head(tidy_books))| book | linenumber | chapter | word |
|---|---|---|---|
| Sense & Sensibility | 1 | 0 | sense |
| Sense & Sensibility | 1 | 0 | and |
| Sense & Sensibility | 1 | 0 | sensibility |
| Sense & Sensibility | 3 | 0 | by |
| Sense & Sensibility | 3 | 0 | jane |
| Sense & Sensibility | 3 | 0 | austen |
nrc_joy <- get_sentiments("nrc") %>% filter(sentiment == "joy")
emma_joy <- tidy_books %>%
filter(book == "Emma") %>%
inner_join(nrc_joy, by = "word") %>%
count(word, sort = TRUE)
kable(head(emma_joy, 10))| word | n |
|---|---|
| good | 359 |
| friend | 166 |
| hope | 143 |
| happy | 125 |
| love | 117 |
| deal | 92 |
| found | 92 |
| present | 89 |
| kind | 82 |
| happiness | 76 |
austen_sentiment <- tidy_books %>%
inner_join(get_sentiments("bing"), by = "word") %>%
count(book, index = linenumber %/% 80, sentiment) %>%
pivot_wider(names_from = sentiment, values_from = n, values_fill = 0) %>%
mutate(sentiment = positive - negative)
ggplot(austen_sentiment, aes(index, sentiment, fill = book)) +
geom_col(show.legend = FALSE) +
facet_wrap(~book, ncol = 2, scales = "free_x") +
labs(title = "Bing sentiment across Jane Austen's novels", x = "80-line segment", y = "Positive minus negative words") +
theme_minimal()pride_prejudice <- tidy_books %>% filter(book == "Pride & Prejudice")
austen_afinn <- pride_prejudice %>%
inner_join(get_sentiments("afinn"), by = "word") %>%
group_by(index = linenumber %/% 80) %>%
summarise(sentiment = sum(value), .groups = "drop") %>%
mutate(method = "AFINN")
austen_bing_nrc <- bind_rows(
pride_prejudice %>% inner_join(get_sentiments("bing"), by = "word") %>% mutate(method = "Bing"),
pride_prejudice %>% inner_join(get_sentiments("nrc"), by = "word") %>%
filter(sentiment %in% c("positive", "negative")) %>% mutate(method = "NRC")
) %>%
count(method, index = linenumber %/% 80, sentiment) %>%
pivot_wider(names_from = sentiment, values_from = n, values_fill = 0) %>%
mutate(sentiment = positive - negative) %>%
select(index, sentiment, method)
bind_rows(austen_afinn, austen_bing_nrc) %>%
ggplot(aes(index, sentiment, fill = method)) +
geom_col(show.legend = FALSE) +
facet_wrap(~method, ncol = 1, scales = "free_y") +
labs(title = "Pride and Prejudice: three sentiment dictionaries", x = "80-line segment", y = "Sentiment score") +
theme_minimal()Step 2: Extension to Frankenstein
frankenstein <- readRDS("frankenstein.rds")
kable(head(frankenstein))| gutenberg_id | title | linenumber | word |
|---|---|---|---|
| 84 | Frankenstein; or, the modern prometheus | 1 | frankenstein |
| 84 | Frankenstein; or, the modern prometheus | 3 | or |
| 84 | Frankenstein; or, the modern prometheus | 3 | the |
| 84 | Frankenstein; or, the modern prometheus | 3 | modern |
| 84 | Frankenstein; or, the modern prometheus | 3 | prometheus |
| 84 | Frankenstein; or, the modern prometheus | 5 | by |
cat("Number of word tokens:", nrow(frankenstein))Number of word tokens: 75266
For the extended analysis, I selected Mary Shelley’s Frankenstein as a different text corpus. The novel was downloaded from Project Gutenberg using the gutenbergr package and converted into a tidy format using unnest_tokens(). Each word was placed in a separate row while retaining its original line number. This preparation allows the novel to be analyzed using sentiment lexicons and compared with the original Jane Austen example.
Loughran is the additional lexicon beyond the book’s Bing, NRC, and AFINN examples. It was designed for financial texts and includes categories such as positive, negative, uncertainty, and litigious. Because it was not developed for novels, it is useful as a comparison but may miss literary language or misclassify its context.
loughran <- get_sentiments("loughran")
kable(loughran %>% count(sentiment, sort = TRUE))| sentiment | n |
|---|---|
| negative | 2355 |
| litigious | 904 |
| positive | 354 |
| uncertainty | 297 |
| constraining | 184 |
| superfluous | 56 |
The Loughran sentiment lexicon was added to extend the original analysis. Unlike Bing, which classifies words as positive or negative, Loughran includes additional categories such as uncertainty, litigious, constraining, and superfluous. The lexicon contains 2,355 negative words and 354 positive words, along with words in the other categories. Because Loughran was developed for financial text, its classifications may not fully capture sentiment in a literary work such as Frankenstein.
To compare the same positive/negative counting method, only positive and negative categories are used. Results are calculated in 80-line blocks.
lexicons <- bind_rows(
get_sentiments("bing") %>% mutate(method = "Bing"),
get_sentiments("nrc") %>% filter(sentiment %in% c("positive", "negative")) %>% mutate(method = "NRC"),
loughran %>% filter(sentiment %in% c("positive", "negative")) %>% mutate(method = "Loughran")
)
frank_scores <- frankenstein %>%
inner_join(lexicons, by = "word", relationship = "many-to-many") %>%
count(method, index = linenumber %/% 80, sentiment) %>%
pivot_wider(names_from = sentiment, values_from = n, values_fill = 0) %>%
mutate(score = positive - negative)
ggplot(frank_scores, aes(index, score, fill = method)) +
geom_col(show.legend = FALSE) +
facet_wrap(~method, ncol = 1, scales = "free_y") +
labs(title = "Frankenstein sentiment by dictionary", x = "80-line segment", y = "Positive minus negative words") +
theme_minimal()The Bing, NRC, and Loughran sentiment lexicons were used to analyze sentiment throughout Mary Shelley’s Frankenstein. The results show that sentiment varies across the novel, with negative scores becoming more common in several later sections. Bing and NRC identify both positive and negative segments, while Loughran produces predominantly negative scores with a smaller numerical range. These differences may be explained by the different vocabularies and classification methods used by each lexicon. Because Loughran was developed for financial text, it may not fully capture the emotional language used in a literary novel. Overall, the findings demonstrate that the choice of sentiment lexicon can influence the interpretation of a text.
frank_summary <- frank_scores %>%
group_by(method) %>%
summarise(positive_matches = sum(positive), negative_matches = sum(negative),
net_score = sum(score), .groups = "drop")
kable(frank_summary)| method | positive_matches | negative_matches | net_score |
|---|---|---|---|
| Bing | 2983 | 3742 | -759 |
| Loughran | 856 | 1400 | -544 |
| NRC | 4447 | 3904 | 543 |
The sentiment analysis of Mary Shelley’s Frankenstein produced different results depending on the lexicon used. Bing identified 2,983 positive and 3,742 negative word occurrences, resulting in a net sentiment score of −759. Similarly, Loughran identified 856 positive and 1,400 negative occurrences, producing a net score of −544. In contrast, NRC identified 4,447 positive and 3,904 negative occurrences, resulting in a positive net score of 543. These findings demonstrate that different sentiment lexicons can produce different interpretations of the same text. While Bing and Loughran suggest that negative language is more prevalent in Frankenstein, NRC identifies slightly more positive than negative language. This highlights the importance of considering the choice of lexicon when interpreting sentiment analysis results.
frankenstein %>%
inner_join(get_sentiments("bing"), by = "word") %>%
count(sentiment, word, sort = TRUE) %>%
group_by(sentiment) %>%
slice_max(n, n = 10, with_ties = FALSE) %>%
ungroup() %>%
ggplot(aes(x = reorder(word, n), y = n, fill = sentiment)) +
geom_col(show.legend = FALSE) +
coord_flip() +
facet_wrap(~sentiment, scales = "free_y") +
labs(title = "Common sentiment words in Frankenstein (Bing)", x = NULL, y = "Occurrences") +
theme_minimal()The Bing sentiment lexicon was used to identify the most frequently occurring positive and negative words in Mary Shelley’s Frankenstein. The most common negative words included “death,” “miserable,” “misery,” and “despair,” reflecting themes of suffering and tragedy in the novel. The most common positive words included “like,” “love,” “great,” and “happiness,” suggesting that the novel also contains language associated with affection and positive emotions. These findings help explain the overall negative sentiment score identified by the Bing lexicon while showing that both positive and negative language appear throughout the novel.
Conclusion
In the original example, Jane Austen’s novels were analyzed using 80-line segments, and the sentiment dictionaries showed how positive and negative word matches changed throughout the stories. In the extension, the same method was applied to Mary Shelley’s Frankenstein and the Loughran lexicon was added. The results showed differences in sentiment depending on the dictionary used. The Bing lexicon identified 2,983 positive and 3,742 negative word occurrences in Frankenstein, resulting in a net sentiment score of -759. Similarly, Loughran produced a negative score of -544. In contrast, NRC produced a positive score of 543. The graphs also showed that sentiment varied throughout the novel, with several negative segments. These findings demonstrate how different sentiment dictionaries can produce different interpretations of the same text.
One limitation of sentiment analysis is that word-level dictionaries do not always capture context, negation, sarcasm, or the identity of the speaker. Additionally, the Loughran lexicon was developed for financial text rather than fiction, which may affect its accuracy when analyzing Frankenstein. Since the novels also differ in length, raw sentiment totals should not be directly compared across texts.