Sentiment Analysis of Notes from Underground

Author

Esra Dogan

Approach

Motivation

In Chapter 2 of Text Mining with R, Silge and Robinson use sentiment analysis to show how the emotional tone changes across Jane Austen’s novels. I will repeat this analysis with Fyodor Dostoevsky’s Notes from Underground, a book I personally find very pessimistic, and see whether the sentiment results confirm this. I will also add a fourth lexicon to see whether it agrees with the three lexicons used in the book.

Plan

  1. Reproduce the primary example from Chapter 2 in a Quarto file, using the Jane Austen novels from the janeaustenr package and the three lexicons (AFINN, Bing and NRC) from tidytext. I will cite the book and the original code from the authors’ GitHub repository.
  2. Download the English translation of Notes from Underground from Project Gutenberg with the gutenbergr package.
  3. Tidy the text the same way as in the book: split it into words with unnest_tokens() and keep track of line numbers and the two parts of the book.
  4. Apply the same steps as the original example: join with the Bing lexicon, count positive and negative words in sections of text, and plot net sentiment across the book.
  5. Add one more lexicon: the syuzhet lexicon from the syuzhet package, which was built specifically for narrative fiction. I will compare it with AFINN, Bing and NRC on Notes from Underground, as the book does with Pride and Prejudice.
  6. Compare the results with the original example: is the overall sentiment lower than in Austen’s novels, and is there a difference between Part I and Part II?

Anticipated Challenges

  • AFINN and NRC are downloaded through the textdata package, which asks for confirmation the first time. This can stop the Quarto file from rendering, so I will download them once beforehand.
  • Project Gutenberg texts include extra text at the beginning and end (titles, license), which I may need to remove.
  • The book is a translation from Russian, so the results reflect the translator’s word choices, not Dostoevsky’s original words.
  • The book is much shorter than Austen’s novels, so I may need smaller sections than 80 lines to see changes across the story.
  • As with “miss” in Austen’s novels, some words may be classified incorrectly, so I will check which words contribute the most to the sentiment scores.

What Is Sentiment Analysis?

Sentiment analysis measures the emotional tone of a text. It uses a lexicon, a list of words labeled as positive or negative (for example, “happy” is positive and “cry” is negative). The text is split into single words, each word is looked up in the lexicon, and the positive and negative words are counted. If a part of the text has more negative words than positive ones, it is treated as negative.

This method is simple, but it has limits: it looks at each word on its own, so it misses context such as “not happy”.

Part 1: Reproducing the Chapter 2 Example

The code in this section comes from Chapter 2 of Text Mining with R (Silge & Robinson, 2017). It uses six Jane Austen novels from the janeaustenr package.

First, the text is split into one word per row. The line number of each word is kept so that we know where in the novel it appears.

library(tidyverse)
── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ──
✔ dplyr     1.2.1     ✔ readr     2.2.0
✔ forcats   1.0.1     ✔ stringr   1.6.0
✔ ggplot2   4.0.3     ✔ tibble    3.3.1
✔ lubridate 1.9.5     ✔ tidyr     1.3.2
✔ purrr     1.2.2     
── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
✖ dplyr::filter() masks stats::filter()
✖ dplyr::lag()    masks stats::lag()
ℹ Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errors
library(tidytext)
library(janeaustenr)

tidy_books <- austen_books() |>
  group_by(book) |>
  mutate(
    linenumber = row_number(),
    chapter = cumsum(str_detect(text, regex("^chapter [\\divxlc]", ignore_case = TRUE)))
  ) |>
  ungroup() |>
  unnest_tokens(word, text)

Sentiment across each novel

Each novel is divided into sections of 80 lines. For each section, the number of negative words is subtracted from the number of positive words (Bing lexicon). Bars above zero are more positive parts of the story; bars below zero are more negative parts.

jane_austen_sentiment <- tidy_books |>
  inner_join(get_sentiments("bing"), by = "word") |>
  count(book, index = linenumber %/% 80, sentiment) |>
  pivot_wider(names_from = sentiment, values_from = n, values_fill = 0) |>
  mutate(sentiment = positive - negative)
Warning in inner_join(tidy_books, get_sentiments("bing"), by = "word"): Detected an unexpected many-to-many relationship between `x` and `y`.
ℹ Row 435434 of `x` matches multiple rows in `y`.
ℹ Row 5051 of `y` matches multiple rows in `x`.
ℹ If a many-to-many relationship is expected, set `relationship =
  "many-to-many"` to silence this warning.
ggplot(jane_austen_sentiment, aes(index, sentiment, fill = book)) +
  geom_col(show.legend = FALSE) +
  facet_wrap(~book, ncol = 2, scales = "free_x")

Most bars in Austen’s novels are above zero, so her novels are mostly positive, with some negative moments along the way.

Comparing three lexicons on Pride and Prejudice

The same novel is measured with three different lexicons: AFINN (scores from -5 to +5), Bing (positive or negative) and NRC (positive or negative, plus emotions).

pride_prejudice <- tidy_books |>
  filter(book == "Pride & Prejudice")

afinn <- pride_prejudice |>
  inner_join(get_sentiments("afinn"), by = "word") |>
  group_by(index = linenumber %/% 80) |>
  summarise(sentiment = sum(value)) |>
  mutate(method = "AFINN")

bing_and_nrc <- bind_rows(
  pride_prejudice |>
    inner_join(get_sentiments("bing"), by = "word") |>
    mutate(method = "Bing et al."),
  pride_prejudice |>
    inner_join(get_sentiments("nrc") |>
                 filter(sentiment %in% c("positive", "negative")),
               by = "word", relationship = "many-to-many") |>
    mutate(method = "NRC")
) |>
  count(method, index = linenumber %/% 80, sentiment) |>
  pivot_wider(names_from = sentiment, values_from = n, values_fill = 0) |>
  mutate(sentiment = positive - negative)

bind_rows(afinn, bing_and_nrc) |>
  ggplot(aes(index, sentiment, fill = method)) +
  geom_col(show.legend = FALSE) +
  facet_wrap(~method, ncol = 1, scales = "free_y")

The three lexicons give different numbers, but their ups and downs happen in roughly the same places. In other words, they agree on the overall shape of the story.

Most common positive and negative words

This chart shows which words contribute the most to the positive and negative scores.

bing_word_counts <- tidy_books |>
  inner_join(get_sentiments("bing"), by = "word") |>
  count(word, sentiment, sort = TRUE)
Warning in inner_join(tidy_books, get_sentiments("bing"), by = "word"): Detected an unexpected many-to-many relationship between `x` and `y`.
ℹ Row 435434 of `x` matches multiple rows in `y`.
ℹ Row 5051 of `y` matches multiple rows in `x`.
ℹ If a many-to-many relationship is expected, set `relationship =
  "many-to-many"` to silence this warning.
bing_word_counts |>
  group_by(sentiment) |>
  slice_max(n, n = 10) |>
  ungroup() |>
  mutate(word = reorder(word, n)) |>
  ggplot(aes(n, word, fill = sentiment)) +
  geom_col(show.legend = FALSE) +
  facet_wrap(~sentiment, scales = "free_y") +
  labs(x = "Contribution to sentiment", y = NULL)

The most common “negative” word is “miss”. In Austen’s novels, however, it is mostly used as a title (“Miss Bennet”), not as a feeling. This shows that a lexicon can misclassify words because it does not understand context.

Part 2: A New Corpus: Notes from Underground

I downloaded the English translation of Fyodor Dostoevsky’s Notes from Underground from Project Gutenberg with the gutenbergr package, and prepared it the same way as the Austen novels.

library(gutenbergr)

underground <- gutenberg_download(600)
Using mirror https://gutenberg.pglaf.org.
tidy_underground <- underground |>
  mutate(linenumber = row_number()) |>
  unnest_tokens(word, text)

Sentiment across the book

As with Austen, the book is divided into sections of 80 lines, and each bar shows positive minus negative words.

underground_sentiment <- tidy_underground |>
  inner_join(get_sentiments("bing"), by = "word") |>
  count(index = linenumber %/% 80, sentiment) |>
  pivot_wider(names_from = sentiment, values_from = n, values_fill = 0) |>
  mutate(sentiment = positive - negative)
Warning in inner_join(tidy_underground, get_sentiments("bing"), by = "word"): Detected an unexpected many-to-many relationship between `x` and `y`.
ℹ Row 41634 of `x` matches multiple rows in `y`.
ℹ Row 5050 of `y` matches multiple rows in `x`.
ℹ If a many-to-many relationship is expected, set `relationship =
  "many-to-many"` to silence this warning.
ggplot(underground_sentiment, aes(index, sentiment)) +
  geom_col(fill = "grey30") +
  labs(title = "Sentiment across Notes from Underground (Bing lexicon)",
       x = "Section of the book (80 lines each)", y = "Net sentiment")

Unlike Austen’s novels, most bars here fall below zero. The negative tone is not limited to a few dark scenes; it continues through most of the book.

Is it more negative than Austen?

To compare books of different lengths fairly, I calculated the percentage of sentiment words in each book that are negative.

bind_rows(
  tidy_books |> mutate(book = as.character(book)),
  tidy_underground |> mutate(book = "Notes from Underground")
) |>
  inner_join(get_sentiments("bing"), by = "word") |>
  count(book, sentiment) |>
  pivot_wider(names_from = sentiment, values_from = n) |>
  mutate(percent_negative = round(100 * negative / (negative + positive), 1)) |>
  arrange(desc(percent_negative))
Warning in inner_join(bind_rows(mutate(tidy_books, book = as.character(book)), : Detected an unexpected many-to-many relationship between `x` and `y`.
ℹ Row 435434 of `x` matches multiple rows in `y`.
ℹ Row 5051 of `y` matches multiple rows in `x`.
ℹ If a many-to-many relationship is expected, set `relationship =
  "many-to-many"` to silence this warning.
# A tibble: 7 × 4
  book                   negative positive percent_negative
  <chr>                     <int>    <int>            <dbl>
1 Notes from Underground     2065     1490             58.1
2 Northanger Abbey           2518     3244             43.7
3 Sense & Sensibility        3671     4933             42.7
4 Pride & Prejudice          3652     5052             42  
5 Mansfield Park             4828     6749             41.7
6 Emma                       4809     7157             40.2
7 Persuasion                 2201     3473             38.8

In Notes from Underground, 58.1% of the sentiment words are negative. In Austen’s novels, this number is between 38.8% (Persuasion) and 43.7% (Northanger Abbey). Notes from Underground is the only book where negative words outnumber positive ones, so the data supports my impression that it is a very pessimistic book.

Most common positive and negative words

tidy_underground |>
  inner_join(get_sentiments("bing"), by = "word") |>
  count(word, sentiment, sort = TRUE) |>
  group_by(sentiment) |>
  slice_max(n, n = 10, with_ties = FALSE) |>
  ungroup() |>
  mutate(word = reorder(word, n)) |>
  ggplot(aes(n, word, fill = sentiment)) +
  geom_col(show.legend = FALSE) +
  facet_wrap(~sentiment, scales = "free_y") +
  labs(x = "Contribution to sentiment", y = NULL)
Warning in inner_join(tidy_underground, get_sentiments("bing"), by = "word"): Detected an unexpected many-to-many relationship between `x` and `y`.
ℹ Row 41634 of `x` matches multiple rows in `y`.
ℹ Row 5050 of `y` matches multiple rows in `x`.
ℹ If a many-to-many relationship is expected, set `relationship =
  "many-to-many"` to silence this warning.

This chart is a useful check, just like the “miss” example in Austen. Some “positive” words, such as “well” and “like”, are often not positive in this book: the narrator frequently says “well” as a filler word, and “like” is often used for comparison (“like a mouse”). This means the book may be even more negative than the numbers show.

Part 3: Adding a Fourth Lexicon: syuzhet

The syuzhet lexicon comes from the syuzhet package. It was built for narrative fiction and gives each word a score between -1 and 1. Below, I compare it with the three lexicons from the book on Notes from Underground.

library(syuzhet)

syuzhet_lexicon <- get_sentiment_dictionary("syuzhet")

four_numeric <- bind_rows(
  tidy_underground |>
    inner_join(get_sentiments("afinn"), by = "word") |>
    group_by(index = linenumber %/% 80) |>
    summarise(sentiment = sum(value)) |>
    mutate(method = "AFINN"),
  tidy_underground |>
    inner_join(syuzhet_lexicon, by = "word") |>
    group_by(index = linenumber %/% 80) |>
    summarise(sentiment = sum(value)) |>
    mutate(method = "syuzhet")
)
Warning in inner_join(tidy_underground, syuzhet_lexicon, by = "word"): Detected an unexpected many-to-many relationship between `x` and `y`.
ℹ Row 35358 of `x` matches multiple rows in `y`.
ℹ Row 1368 of `y` matches multiple rows in `x`.
ℹ If a many-to-many relationship is expected, set `relationship =
  "many-to-many"` to silence this warning.
four_binary <- bind_rows(
  tidy_underground |>
    inner_join(get_sentiments("bing"), by = "word") |>
    mutate(method = "Bing et al."),
  tidy_underground |>
    inner_join(get_sentiments("nrc") |>
                 filter(sentiment %in% c("positive", "negative")),
               by = "word", relationship = "many-to-many") |>
    mutate(method = "NRC")
) |>
  count(method, index = linenumber %/% 80, sentiment) |>
  pivot_wider(names_from = sentiment, values_from = n, values_fill = 0) |>
  mutate(sentiment = positive - negative)
Warning in inner_join(tidy_underground, get_sentiments("bing"), by = "word"): Detected an unexpected many-to-many relationship between `x` and `y`.
ℹ Row 41634 of `x` matches multiple rows in `y`.
ℹ Row 5050 of `y` matches multiple rows in `x`.
ℹ If a many-to-many relationship is expected, set `relationship =
  "many-to-many"` to silence this warning.
bind_rows(four_numeric, four_binary) |>
  ggplot(aes(index, sentiment, fill = method)) +
  geom_col(show.legend = FALSE) +
  facet_wrap(~method, ncol = 1, scales = "free_y") +
  labs(title = "Four lexicons on Notes from Underground",
       x = "Section of the book (80 lines each)", y = "Net sentiment")

Each lexicon uses a different scale, so the heights of the bars cannot be compared directly. What matters is the shape: whether the lexicons find positive and negative parts in the same places.

How the Results Differ from the Original Example

  • Overall tone: Austen’s novels are mostly positive, while Notes from Underground is mostly negative (58.1% negative words compared with about 39–44% in Austen).
  • Shape of the story: Austen’s novels move up and down around a positive level, which fits stories that end happily. In Notes from Underground, the negative tone continues through most of the book.
  • Misclassified words: In Austen, “miss” was wrongly counted as negative. In Dostoevsky, words like “well” and “like” may be wrongly counted as positive.
  • Lexicons: As in the book’s example, the lexicons give different values, so it is better to compare their overall patterns than their exact numbers.
  • Part I and Part II: I planned to compare the two parts of the book, but I kept the analysis simpler and looked at the book as a whole.

Conclusion

Sentiment analysis supports my impression of Notes from Underground: it is much more negative than any of Jane Austen’s novels. At the same time, the analysis has clear limits. It looks at single words without context, the lexicons were built mostly from modern texts, and the book is a translation, so the results reflect the translator’s word choices. Still, even this simple method clearly shows the difference in tone between the two authors.

Use of AI

I used Claude (Anthropic) as a tutor for this assignment. It helped me choose the extra lexicon, adapt the code from the book to a new text, fix errors, and write simple explanations of the results.

References