Introduction

In Text Mining with R, Chapter 2 looks at Sentiment Analysis. In this assignment, you should start by getting the primary example code from chapter 2 working in an R Markdown document. You should provide a citation to this base code. You’re then asked to extend the code in two ways:

Work with a different corpus of your choosing, and
Incorporate at least one additional sentiment lexicon (possibly from another R package that you’ve found through research).

As usual, please submit links to both an .Rmd file posted in your GitHub repository and to your code on rpubs.com. You make work on a small team on this assignment.

Example Code from Chapter 2

We being by loading our libraries, as well as our lexicons. They are the general lexicons avaialble in the tidytext user.

library(tidytext)
library(janeaustenr)
library(dplyr)
## 
## Attaching package: 'dplyr'
## The following objects are masked from 'package:stats':
## 
##     filter, lag
## The following objects are masked from 'package:base':
## 
##     intersect, setdiff, setequal, union
library(stringr)

get_sentiments(lexicon = c("bing", "afinn", "loughran", "nrc"))
## # A tibble: 6,786 × 2
##    word        sentiment
##    <chr>       <chr>    
##  1 2-faces     negative 
##  2 abnormal    negative 
##  3 abolish     negative 
##  4 abominable  negative 
##  5 abominably  negative 
##  6 abominate   negative 
##  7 abomination negative 
##  8 abort       negative 
##  9 aborted     negative 
## 10 aborts      negative 
## # … with 6,776 more rows
get_sentiments("afinn")
## # A tibble: 2,477 × 2
##    word       value
##    <chr>      <dbl>
##  1 abandon       -2
##  2 abandoned     -2
##  3 abandons      -2
##  4 abducted      -2
##  5 abduction     -2
##  6 abductions    -2
##  7 abhor         -3
##  8 abhorred      -3
##  9 abhorrent     -3
## 10 abhors        -3
## # … with 2,467 more rows
get_sentiments("bing")
## # A tibble: 6,786 × 2
##    word        sentiment
##    <chr>       <chr>    
##  1 2-faces     negative 
##  2 abnormal    negative 
##  3 abolish     negative 
##  4 abominable  negative 
##  5 abominably  negative 
##  6 abominate   negative 
##  7 abomination negative 
##  8 abort       negative 
##  9 aborted     negative 
## 10 aborts      negative 
## # … with 6,776 more rows
get_sentiments("nrc")
## # A tibble: 13,872 × 2
##    word        sentiment
##    <chr>       <chr>    
##  1 abacus      trust    
##  2 abandon     fear     
##  3 abandon     negative 
##  4 abandon     sadness  
##  5 abandoned   anger    
##  6 abandoned   fear     
##  7 abandoned   negative 
##  8 abandoned   sadness  
##  9 abandonment anger    
## 10 abandonment fear     
## # … with 13,862 more rows

Loading Austen Books

tidy_books <- austen_books() %>%
  group_by(book) %>%
  mutate(
    linenumber = row_number(),
    chapter = cumsum(str_detect(text, 
                                regex("^chapter [\\divxlc]", 
                                      ignore_case = TRUE)))) %>%
  ungroup() %>%
  unnest_tokens(word, text)

nrc_joy <- get_sentiments("nrc") %>% 
  filter(sentiment == "joy")

tidy_books %>%
  filter(book == "Emma") %>%
  inner_join(nrc_joy) %>%
  count(word, sort = TRUE)
## Joining with `by = join_by(word)`
## # A tibble: 301 × 2
##    word          n
##    <chr>     <int>
##  1 good        359
##  2 friend      166
##  3 hope        143
##  4 happy       125
##  5 love        117
##  6 deal         92
##  7 found        92
##  8 present      89
##  9 kind         82
## 10 happiness    76
## # … with 291 more rows
library(tidyr)

jane_austen_sentiment <- tidy_books %>%
  inner_join(get_sentiments("bing")) %>%
  count(book, index = linenumber %/% 80, sentiment) %>%
  pivot_wider(names_from = sentiment, values_from = n, values_fill = 0) %>% 
  mutate(sentiment = positive - negative)
## Joining with `by = join_by(word)`
## Warning in inner_join(., get_sentiments("bing")): Each row in `x` is expected to match at most 1 row in `y`.
## ℹ Row 435434 of `x` matches multiple rows.
## ℹ If multiple matches are expected, set `multiple = "all"` to silence this
##   warning.

Plotting the Data

library(ggplot2)

ggplot(jane_austen_sentiment, aes(index, sentiment, fill = book)) +
  geom_col(show.legend = FALSE) +
  facet_wrap(~book, ncol = 2, scales = "free_x")

pride_prejudice <- tidy_books %>% 
  filter(book == "Pride & Prejudice")

afinn <- pride_prejudice %>% 
  inner_join(get_sentiments("afinn")) %>% 
  group_by(index = linenumber %/% 80) %>% 
  summarise(sentiment = sum(value)) %>% 
  mutate(method = "AFINN")
## Joining with `by = join_by(word)`
bing_and_nrc <- bind_rows(
  pride_prejudice %>% 
    inner_join(get_sentiments("bing")) %>%
    mutate(method = "Bing et al."),
  pride_prejudice %>% 
    inner_join(get_sentiments("nrc") %>% 
                 filter(sentiment %in% c("positive", 
                                         "negative"))
    ) %>%
    mutate(method = "NRC")) %>%
  count(method, index = linenumber %/% 80, sentiment) %>%
  pivot_wider(names_from = sentiment,
              values_from = n,
              values_fill = 0) %>% 
  mutate(sentiment = positive - negative)
## Joining with `by = join_by(word)`
## Joining with `by = join_by(word)`
## Warning in inner_join(., get_sentiments("nrc") %>% filter(sentiment %in% : Each row in `x` is expected to match at most 1 row in `y`.
## ℹ Row 215 of `x` matches multiple rows.
## ℹ If multiple matches are expected, set `multiple = "all"` to silence this
##   warning.

Sentiment Lexicon Graphs

bind_rows(afinn, 
          bing_and_nrc) %>%
  ggplot(aes(index, sentiment, fill = method)) +
  geom_col(show.legend = FALSE) +
  facet_wrap(~method, ncol = 1, scales = "free_y")

get_sentiments("nrc") %>% 
  filter(sentiment %in% c("positive", "negative")) %>% 
  count(sentiment)
## # A tibble: 2 × 2
##   sentiment     n
##   <chr>     <int>
## 1 negative   3316
## 2 positive   2308
get_sentiments("bing") %>% 
  count(sentiment)
## # A tibble: 2 × 2
##   sentiment     n
##   <chr>     <int>
## 1 negative   4781
## 2 positive   2005

Analyzing the words

bing_word_counts <- tidy_books %>%
  inner_join(get_sentiments("bing")) %>%
  count(word, sentiment, sort = TRUE) %>%
  ungroup()
## Joining with `by = join_by(word)`
## Warning in inner_join(., get_sentiments("bing")): Each row in `x` is expected to match at most 1 row in `y`.
## ℹ Row 435434 of `x` matches multiple rows.
## ℹ If multiple matches are expected, set `multiple = "all"` to silence this
##   warning.
bing_word_counts
## # A tibble: 2,585 × 3
##    word     sentiment     n
##    <chr>    <chr>     <int>
##  1 miss     negative   1855
##  2 well     positive   1523
##  3 good     positive   1380
##  4 great    positive    981
##  5 like     positive    725
##  6 better   positive    639
##  7 enough   positive    613
##  8 happy    positive    534
##  9 love     positive    495
## 10 pleasure positive    462
## # … with 2,575 more rows

Graphing the Contribution to Sentiment

bing_word_counts %>%
  group_by(sentiment) %>%
  slice_max(n, n = 10) %>% 
  ungroup() %>%
  mutate(word = reorder(word, n)) %>%
  ggplot(aes(n, word, fill = sentiment)) +
  geom_col(show.legend = FALSE) +
  facet_wrap(~sentiment, scales = "free_y") +
  labs(x = "Contribution to sentiment",
       y = NULL)

custom_stop_words <- bind_rows(tibble(word = c("miss"),  
                                      lexicon = c("custom")), 
                               stop_words)

custom_stop_words
## # A tibble: 1,150 × 2
##    word        lexicon
##    <chr>       <chr>  
##  1 miss        custom 
##  2 a           SMART  
##  3 a's         SMART  
##  4 able        SMART  
##  5 about       SMART  
##  6 above       SMART  
##  7 according   SMART  
##  8 accordingly SMART  
##  9 across      SMART  
## 10 actually    SMART  
## # … with 1,140 more rows
austen_chapters <- austen_books() %>%
  group_by(book) %>%
  unnest_tokens(chapter, text, token = "regex", 
                pattern = "Chapter|CHAPTER [\\dIVXLC]") %>%
  ungroup()

austen_chapters %>% 
  group_by(book) %>% 
  summarise(chapters = n())
## # A tibble: 6 × 2
##   book                chapters
##   <fct>                  <int>
## 1 Sense & Sensibility       51
## 2 Pride & Prejudice         62
## 3 Mansfield Park            49
## 4 Emma                      56
## 5 Northanger Abbey          32
## 6 Persuasion                25
bingnegative <- get_sentiments("bing") %>% 
  filter(sentiment == "negative")

wordcounts <- tidy_books %>%
  group_by(book, chapter) %>%
  summarize(words = n())
## `summarise()` has grouped output by 'book'. You can override using the
## `.groups` argument.
tidy_books %>%
  semi_join(bingnegative) %>%
  group_by(book, chapter) %>%
  summarize(negativewords = n()) %>%
  left_join(wordcounts, by = c("book", "chapter")) %>%
  mutate(ratio = negativewords/words) %>%
  filter(chapter != 0) %>%
  slice_max(ratio, n = 1) %>% 
  ungroup()
## Joining with `by = join_by(word)`
## `summarise()` has grouped output by 'book'. You can override using the
## `.groups` argument.
## # A tibble: 6 × 5
##   book                chapter negativewords words  ratio
##   <fct>                 <int>         <int> <int>  <dbl>
## 1 Sense & Sensibility      43           161  3405 0.0473
## 2 Pride & Prejudice        34           111  2104 0.0528
## 3 Mansfield Park           46           173  3685 0.0469
## 4 Emma                     15           151  3340 0.0452
## 5 Northanger Abbey         21           149  2982 0.0500
## 6 Persuasion                4            62  1807 0.0343

Extending the Code

I have chosen to do Charales Dickens and gather 3 books from his large library.

Loading Charles Dickens

We are going to load a collection of Charales Dickens books. First, we load it into a tidy format. We’ve managed to obtain the book library from the gutenbergr library.

library(gutenbergr)
gutenberg_metadata
## # A tibble: 69,199 × 8
##    gutenberg_id title              author guten…¹ langu…² guten…³ rights has_t…⁴
##           <int> <chr>              <chr>    <int> <chr>   <chr>   <chr>  <lgl>  
##  1            1 "The Declaration … Jeffe…    1638 en      Politi… Publi… TRUE   
##  2            2 "The United State… Unite…       1 en      Politi… Publi… TRUE   
##  3            3 "John F. Kennedy'… Kenne…    1666 en      <NA>    Publi… TRUE   
##  4            4 "Lincoln's Gettys… Linco…       3 en      US Civ… Publi… TRUE   
##  5            5 "The United State… Unite…       1 en      United… Publi… TRUE   
##  6            6 "Give Me Liberty … Henry…       4 en      Americ… Publi… TRUE   
##  7            7 "The Mayflower Co… <NA>        NA en      <NA>    Publi… TRUE   
##  8            8 "Abraham Lincoln'… Linco…       3 en      US Civ… Publi… TRUE   
##  9            9 "Abraham Lincoln'… Linco…       3 en      US Civ… Publi… TRUE   
## 10           10 "The King James V… <NA>        NA en      Banned… Publi… TRUE   
## # … with 69,189 more rows, and abbreviated variable names ¹​gutenberg_author_id,
## #   ²​language, ³​gutenberg_bookshelf, ⁴​has_text
dickens <- gutenberg_works(author == "Dickens, Charles")

tale_of_two_cities <- dickens[grep("Tale of Two Cities", dickens$title), ]
christmas_carol <- dickens[grep("Christmas Carol", dickens$title), ]
oliver_twist <- dickens[grep("Oliver Twist", dickens$title), ]

# christmas_carol_book <- gutenberg_download(19337)
# oliver_twist_book <- gutenberg_download(730) 
# tale_cities <- gutenberg_download(98) 

dickens_books <- gutenberg_download(c(98, 46, 730))
## Determining mirror for Project Gutenberg from https://www.gutenberg.org/robot/harvest
## Using mirror http://aleph.gutenberg.org

We are still running the same Jane Austen code as before, just slightly altered to fit the gutenbergr library.

dickens_books <- dickens_books %>%
  mutate(book = case_when(
    gutenberg_id == 98 ~ "A Tale of Two Cities",
    gutenberg_id == 46 ~ "Christmas Carol",
    gutenberg_id == 730 ~ "Oliver Twist"
  )) %>%
  select(book, text)

original_books <- dickens_books %>%
  group_by(book) %>%
  mutate(linenumber = row_number(),
         chapter = cumsum(str_detect(text, 
                                     regex("^chapter [\\divxlc]",
                                           ignore_case = TRUE)))) %>%
  ungroup()

Transforming Charles Dickens

We then tokenize our tidy dataframe.

tidy_books <- original_books %>%
  unnest_tokens(word, text)

tidy_books <- tidy_books %>%
  anti_join(stop_words, by = join_by(word))

Analyzing Charles Dickens

We begin to analyze our initial example using the syuzhet library.

library(syuzhet)

sentiment_scores <- tidy_books %>%
  group_by(book) %>%
  mutate(sentiment_score = get_sentiment(word)) %>%
  ungroup()



mean_sentiment <- sentiment_scores %>%
  group_by(book) %>%
  summarize(mean_sentiment_score = mean(sentiment_score))

How the books Scored

Here we find that the sadest of the trio is “A Tale of Two Cities” with a score of -0.01188674, followed by “Oliver Twist” with a score of -0.00769418. Of the three” “Christmas Carol” scored the highest with a score of 0.00123055, meaning that it is “happier” than the other two books.

library(ggplot2)

mean_sentiment
## # A tibble: 3 × 2
##   book                 mean_sentiment_score
##   <chr>                               <dbl>
## 1 A Tale of Two Cities             -0.0119 
## 2 Christmas Carol                   0.00123
## 3 Oliver Twist                     -0.00769
ggplot(mean_sentiment, aes(x = book, y = mean_sentiment_score)) +
  geom_bar(stat = "identity", fill = "blue") +
  ggtitle("Mean Sentiment")

We also look at the positive word count per book and find that “Oliver Twist”, while scoring the second worst of all, came in with nearly three times as many positive wordst than a “Christmas Carol”.

positive_words <- sentiment_scores %>%
  filter(sentiment_score == 1) %>%
  group_by(book) %>%
  count()

positive_words
## # A tibble: 3 × 2
## # Groups:   book [3]
##   book                     n
##   <chr>                <int>
## 1 A Tale of Two Cities   540
## 2 Christmas Carol        170
## 3 Oliver Twist           977
ggplot(positive_words, aes(x = book, y = n)) +
  geom_bar(stat = "identity", fill = "blue") +
  ggtitle("Most positive words")

Conclusion

Analyzing a data set for specific sentiments is particularly beneficial. In this instance, we have analyzed a relatively simple selection of three books; however, it is possible to delve much further into the analysis of literary works using sentiment analysis techniques.