In Text Mining with R, Chapter 2 looks at Sentiment Analysis. In this assignment, you should start by getting the primary example code from chapter 2 working in an R Markdown document. You should provide a citation to this base code. You’re then asked to extend the code in two ways:
Work with a different corpus of your choosing, and
Incorporate at least one additional sentiment lexicon (possibly from another R package that you’ve found through research).
As usual, please submit links to both an .Rmd file posted in your GitHub repository and to your code on rpubs.com. You make work on a small team on this assignment.
We being by loading our libraries, as well as our lexicons. They are the general lexicons avaialble in the tidytext user.
library(tidytext)
library(janeaustenr)
library(dplyr)
##
## Attaching package: 'dplyr'
## The following objects are masked from 'package:stats':
##
## filter, lag
## The following objects are masked from 'package:base':
##
## intersect, setdiff, setequal, union
library(stringr)
get_sentiments(lexicon = c("bing", "afinn", "loughran", "nrc"))
## # A tibble: 6,786 × 2
## word sentiment
## <chr> <chr>
## 1 2-faces negative
## 2 abnormal negative
## 3 abolish negative
## 4 abominable negative
## 5 abominably negative
## 6 abominate negative
## 7 abomination negative
## 8 abort negative
## 9 aborted negative
## 10 aborts negative
## # … with 6,776 more rows
get_sentiments("afinn")
## # A tibble: 2,477 × 2
## word value
## <chr> <dbl>
## 1 abandon -2
## 2 abandoned -2
## 3 abandons -2
## 4 abducted -2
## 5 abduction -2
## 6 abductions -2
## 7 abhor -3
## 8 abhorred -3
## 9 abhorrent -3
## 10 abhors -3
## # … with 2,467 more rows
get_sentiments("bing")
## # A tibble: 6,786 × 2
## word sentiment
## <chr> <chr>
## 1 2-faces negative
## 2 abnormal negative
## 3 abolish negative
## 4 abominable negative
## 5 abominably negative
## 6 abominate negative
## 7 abomination negative
## 8 abort negative
## 9 aborted negative
## 10 aborts negative
## # … with 6,776 more rows
get_sentiments("nrc")
## # A tibble: 13,872 × 2
## word sentiment
## <chr> <chr>
## 1 abacus trust
## 2 abandon fear
## 3 abandon negative
## 4 abandon sadness
## 5 abandoned anger
## 6 abandoned fear
## 7 abandoned negative
## 8 abandoned sadness
## 9 abandonment anger
## 10 abandonment fear
## # … with 13,862 more rows
tidy_books <- austen_books() %>%
group_by(book) %>%
mutate(
linenumber = row_number(),
chapter = cumsum(str_detect(text,
regex("^chapter [\\divxlc]",
ignore_case = TRUE)))) %>%
ungroup() %>%
unnest_tokens(word, text)
nrc_joy <- get_sentiments("nrc") %>%
filter(sentiment == "joy")
tidy_books %>%
filter(book == "Emma") %>%
inner_join(nrc_joy) %>%
count(word, sort = TRUE)
## Joining with `by = join_by(word)`
## # A tibble: 301 × 2
## word n
## <chr> <int>
## 1 good 359
## 2 friend 166
## 3 hope 143
## 4 happy 125
## 5 love 117
## 6 deal 92
## 7 found 92
## 8 present 89
## 9 kind 82
## 10 happiness 76
## # … with 291 more rows
library(tidyr)
jane_austen_sentiment <- tidy_books %>%
inner_join(get_sentiments("bing")) %>%
count(book, index = linenumber %/% 80, sentiment) %>%
pivot_wider(names_from = sentiment, values_from = n, values_fill = 0) %>%
mutate(sentiment = positive - negative)
## Joining with `by = join_by(word)`
## Warning in inner_join(., get_sentiments("bing")): Each row in `x` is expected to match at most 1 row in `y`.
## ℹ Row 435434 of `x` matches multiple rows.
## ℹ If multiple matches are expected, set `multiple = "all"` to silence this
## warning.
library(ggplot2)
ggplot(jane_austen_sentiment, aes(index, sentiment, fill = book)) +
geom_col(show.legend = FALSE) +
facet_wrap(~book, ncol = 2, scales = "free_x")
pride_prejudice <- tidy_books %>%
filter(book == "Pride & Prejudice")
afinn <- pride_prejudice %>%
inner_join(get_sentiments("afinn")) %>%
group_by(index = linenumber %/% 80) %>%
summarise(sentiment = sum(value)) %>%
mutate(method = "AFINN")
## Joining with `by = join_by(word)`
bing_and_nrc <- bind_rows(
pride_prejudice %>%
inner_join(get_sentiments("bing")) %>%
mutate(method = "Bing et al."),
pride_prejudice %>%
inner_join(get_sentiments("nrc") %>%
filter(sentiment %in% c("positive",
"negative"))
) %>%
mutate(method = "NRC")) %>%
count(method, index = linenumber %/% 80, sentiment) %>%
pivot_wider(names_from = sentiment,
values_from = n,
values_fill = 0) %>%
mutate(sentiment = positive - negative)
## Joining with `by = join_by(word)`
## Joining with `by = join_by(word)`
## Warning in inner_join(., get_sentiments("nrc") %>% filter(sentiment %in% : Each row in `x` is expected to match at most 1 row in `y`.
## ℹ Row 215 of `x` matches multiple rows.
## ℹ If multiple matches are expected, set `multiple = "all"` to silence this
## warning.
bind_rows(afinn,
bing_and_nrc) %>%
ggplot(aes(index, sentiment, fill = method)) +
geom_col(show.legend = FALSE) +
facet_wrap(~method, ncol = 1, scales = "free_y")
get_sentiments("nrc") %>%
filter(sentiment %in% c("positive", "negative")) %>%
count(sentiment)
## # A tibble: 2 × 2
## sentiment n
## <chr> <int>
## 1 negative 3316
## 2 positive 2308
get_sentiments("bing") %>%
count(sentiment)
## # A tibble: 2 × 2
## sentiment n
## <chr> <int>
## 1 negative 4781
## 2 positive 2005
bing_word_counts <- tidy_books %>%
inner_join(get_sentiments("bing")) %>%
count(word, sentiment, sort = TRUE) %>%
ungroup()
## Joining with `by = join_by(word)`
## Warning in inner_join(., get_sentiments("bing")): Each row in `x` is expected to match at most 1 row in `y`.
## ℹ Row 435434 of `x` matches multiple rows.
## ℹ If multiple matches are expected, set `multiple = "all"` to silence this
## warning.
bing_word_counts
## # A tibble: 2,585 × 3
## word sentiment n
## <chr> <chr> <int>
## 1 miss negative 1855
## 2 well positive 1523
## 3 good positive 1380
## 4 great positive 981
## 5 like positive 725
## 6 better positive 639
## 7 enough positive 613
## 8 happy positive 534
## 9 love positive 495
## 10 pleasure positive 462
## # … with 2,575 more rows
bing_word_counts %>%
group_by(sentiment) %>%
slice_max(n, n = 10) %>%
ungroup() %>%
mutate(word = reorder(word, n)) %>%
ggplot(aes(n, word, fill = sentiment)) +
geom_col(show.legend = FALSE) +
facet_wrap(~sentiment, scales = "free_y") +
labs(x = "Contribution to sentiment",
y = NULL)
custom_stop_words <- bind_rows(tibble(word = c("miss"),
lexicon = c("custom")),
stop_words)
custom_stop_words
## # A tibble: 1,150 × 2
## word lexicon
## <chr> <chr>
## 1 miss custom
## 2 a SMART
## 3 a's SMART
## 4 able SMART
## 5 about SMART
## 6 above SMART
## 7 according SMART
## 8 accordingly SMART
## 9 across SMART
## 10 actually SMART
## # … with 1,140 more rows
austen_chapters <- austen_books() %>%
group_by(book) %>%
unnest_tokens(chapter, text, token = "regex",
pattern = "Chapter|CHAPTER [\\dIVXLC]") %>%
ungroup()
austen_chapters %>%
group_by(book) %>%
summarise(chapters = n())
## # A tibble: 6 × 2
## book chapters
## <fct> <int>
## 1 Sense & Sensibility 51
## 2 Pride & Prejudice 62
## 3 Mansfield Park 49
## 4 Emma 56
## 5 Northanger Abbey 32
## 6 Persuasion 25
bingnegative <- get_sentiments("bing") %>%
filter(sentiment == "negative")
wordcounts <- tidy_books %>%
group_by(book, chapter) %>%
summarize(words = n())
## `summarise()` has grouped output by 'book'. You can override using the
## `.groups` argument.
tidy_books %>%
semi_join(bingnegative) %>%
group_by(book, chapter) %>%
summarize(negativewords = n()) %>%
left_join(wordcounts, by = c("book", "chapter")) %>%
mutate(ratio = negativewords/words) %>%
filter(chapter != 0) %>%
slice_max(ratio, n = 1) %>%
ungroup()
## Joining with `by = join_by(word)`
## `summarise()` has grouped output by 'book'. You can override using the
## `.groups` argument.
## # A tibble: 6 × 5
## book chapter negativewords words ratio
## <fct> <int> <int> <int> <dbl>
## 1 Sense & Sensibility 43 161 3405 0.0473
## 2 Pride & Prejudice 34 111 2104 0.0528
## 3 Mansfield Park 46 173 3685 0.0469
## 4 Emma 15 151 3340 0.0452
## 5 Northanger Abbey 21 149 2982 0.0500
## 6 Persuasion 4 62 1807 0.0343
I have chosen to do Charales Dickens and gather 3 books from his large library.
We are going to load a collection of Charales Dickens books. First, we load it into a tidy format. We’ve managed to obtain the book library from the gutenbergr library.
library(gutenbergr)
gutenberg_metadata
## # A tibble: 69,199 × 8
## gutenberg_id title author guten…¹ langu…² guten…³ rights has_t…⁴
## <int> <chr> <chr> <int> <chr> <chr> <chr> <lgl>
## 1 1 "The Declaration … Jeffe… 1638 en Politi… Publi… TRUE
## 2 2 "The United State… Unite… 1 en Politi… Publi… TRUE
## 3 3 "John F. Kennedy'… Kenne… 1666 en <NA> Publi… TRUE
## 4 4 "Lincoln's Gettys… Linco… 3 en US Civ… Publi… TRUE
## 5 5 "The United State… Unite… 1 en United… Publi… TRUE
## 6 6 "Give Me Liberty … Henry… 4 en Americ… Publi… TRUE
## 7 7 "The Mayflower Co… <NA> NA en <NA> Publi… TRUE
## 8 8 "Abraham Lincoln'… Linco… 3 en US Civ… Publi… TRUE
## 9 9 "Abraham Lincoln'… Linco… 3 en US Civ… Publi… TRUE
## 10 10 "The King James V… <NA> NA en Banned… Publi… TRUE
## # … with 69,189 more rows, and abbreviated variable names ¹gutenberg_author_id,
## # ²language, ³gutenberg_bookshelf, ⁴has_text
dickens <- gutenberg_works(author == "Dickens, Charles")
tale_of_two_cities <- dickens[grep("Tale of Two Cities", dickens$title), ]
christmas_carol <- dickens[grep("Christmas Carol", dickens$title), ]
oliver_twist <- dickens[grep("Oliver Twist", dickens$title), ]
# christmas_carol_book <- gutenberg_download(19337)
# oliver_twist_book <- gutenberg_download(730)
# tale_cities <- gutenberg_download(98)
dickens_books <- gutenberg_download(c(98, 46, 730))
## Determining mirror for Project Gutenberg from https://www.gutenberg.org/robot/harvest
## Using mirror http://aleph.gutenberg.org
We are still running the same Jane Austen code as before, just slightly altered to fit the gutenbergr library.
dickens_books <- dickens_books %>%
mutate(book = case_when(
gutenberg_id == 98 ~ "A Tale of Two Cities",
gutenberg_id == 46 ~ "Christmas Carol",
gutenberg_id == 730 ~ "Oliver Twist"
)) %>%
select(book, text)
original_books <- dickens_books %>%
group_by(book) %>%
mutate(linenumber = row_number(),
chapter = cumsum(str_detect(text,
regex("^chapter [\\divxlc]",
ignore_case = TRUE)))) %>%
ungroup()
We then tokenize our tidy dataframe.
tidy_books <- original_books %>%
unnest_tokens(word, text)
tidy_books <- tidy_books %>%
anti_join(stop_words, by = join_by(word))
We begin to analyze our initial example using the syuzhet library.
library(syuzhet)
sentiment_scores <- tidy_books %>%
group_by(book) %>%
mutate(sentiment_score = get_sentiment(word)) %>%
ungroup()
mean_sentiment <- sentiment_scores %>%
group_by(book) %>%
summarize(mean_sentiment_score = mean(sentiment_score))
Here we find that the sadest of the trio is “A Tale of Two Cities” with a score of -0.01188674, followed by “Oliver Twist” with a score of -0.00769418. Of the three” “Christmas Carol” scored the highest with a score of 0.00123055, meaning that it is “happier” than the other two books.
library(ggplot2)
mean_sentiment
## # A tibble: 3 × 2
## book mean_sentiment_score
## <chr> <dbl>
## 1 A Tale of Two Cities -0.0119
## 2 Christmas Carol 0.00123
## 3 Oliver Twist -0.00769
ggplot(mean_sentiment, aes(x = book, y = mean_sentiment_score)) +
geom_bar(stat = "identity", fill = "blue") +
ggtitle("Mean Sentiment")
We also look at the positive word count per book and find that “Oliver Twist”, while scoring the second worst of all, came in with nearly three times as many positive wordst than a “Christmas Carol”.
positive_words <- sentiment_scores %>%
filter(sentiment_score == 1) %>%
group_by(book) %>%
count()
positive_words
## # A tibble: 3 × 2
## # Groups: book [3]
## book n
## <chr> <int>
## 1 A Tale of Two Cities 540
## 2 Christmas Carol 170
## 3 Oliver Twist 977
ggplot(positive_words, aes(x = book, y = n)) +
geom_bar(stat = "identity", fill = "blue") +
ggtitle("Most positive words")
Analyzing a data set for specific sentiments is particularly beneficial. In this instance, we have analyzed a relatively simple selection of three books; however, it is possible to delve much further into the analysis of literary works using sentiment analysis techniques.