This exercise consist of three parts. First, the results in Chapter 2 of Text Mining with R by J. Silge and D. Robinson (https://www.tidytextmining.com/) are reproduced.Then there are two extensions, one with a different set of books and another one with a different sentiment lexicon applied to Jane Austen’s novels.
These are some of the libraries and packages that may be needed to reproduce the results in Chapter 2.
library(dplyr)
Attaching package: 'dplyr'
The following objects are masked from 'package:stats':
filter, lag
The following objects are masked from 'package:base':
intersect, setdiff, setequal, union
# A tibble: 301 × 2
word n
<chr> <int>
1 good 359
2 friend 166
3 hope 143
4 happy 125
5 love 117
6 deal 92
7 found 92
8 present 89
9 kind 82
10 happiness 76
# ℹ 291 more rows
It seems to work, However, “young” was not picked up here (it is in the textbook example). All the other ones match, both in terms of rank and the word count. The only exception is “happiness” that is not shown in the book. It seems to have moved up the list as “young” has not appeared here.
Something similar is done across all six novels. However, in these graphs, the prevalence of positive or negative sentiments is plotted along the narrative of each novel. This is done by estimating the prevalence of these sentiments in sections of 80 lines.
Warning in inner_join(tidy_books, get_sentiments("bing")): Detected an unexpected many-to-many relationship between `x` and `y`.
ℹ Row 435434 of `x` matches multiple rows in `y`.
ℹ Row 5051 of `y` matches multiple rows in `x`.
ℹ If a many-to-many relationship is expected, set `relationship =
"many-to-many"` to silence this warning.
There is an error message. However, it does not seem to affect the construction of the following graphs.
The graphs seem to correspond to the ones in the textbook. However they do not show up in the Plots tab in the R pane.
Small detour to explore the lexicons
In the textbook, the book Pride and Prejudice is selected to explore the three sentiment lexicons. Thus, first, Pride and prejudice is set up as a tidy data frame.
# A tibble: 122,204 × 4
book linenumber chapter word
<fct> <int> <int> <chr>
1 Pride & Prejudice 1 0 pride
2 Pride & Prejudice 1 0 and
3 Pride & Prejudice 1 0 prejudice
4 Pride & Prejudice 3 0 by
5 Pride & Prejudice 3 0 jane
6 Pride & Prejudice 3 0 austen
7 Pride & Prejudice 7 1 chapter
8 Pride & Prejudice 7 1 1
9 Pride & Prejudice 10 1 it
10 Pride & Prejudice 10 1 is
# ℹ 122,194 more rows
In the next step, what was done above for all six novels using the Bing et al sentiment lexicon is done with the other two lexicons only for Pride and Prejudice. Then, all three abalyses are shown together.
Joining with `by = join_by(word)`
Joining with `by = join_by(word)`
Warning in inner_join(pride_prejudice, filter(get_sentiments("nrc"), sentiment %in% : Detected an unexpected many-to-many relationship between `x` and `y`.
ℹ Row 215 of `x` matches multiple rows in `y`.
ℹ Row 5178 of `y` matches multiple rows in `x`.
ℹ If a many-to-many relationship is expected, set `relationship =
"many-to-many"` to silence this warning.
Same error message as above. However, the graphs do work out below.
Clearly, each sentiment lexicon provides different results. Part of the reason is due to different counts of positive and negative words in each lexicon, These can be counted and compared (in a bar chart, below).
Warning in inner_join(tidy_books, get_sentiments("bing")): Detected an unexpected many-to-many relationship between `x` and `y`.
ℹ Row 435434 of `x` matches multiple rows in `y`.
ℹ Row 5051 of `y` matches multiple rows in `x`.
ℹ If a many-to-many relationship is expected, set `relationship =
"many-to-many"` to silence this warning.
Similar error message as above
bing_word_counts
# A tibble: 2,585 × 3
word sentiment n
<chr> <chr> <int>
1 miss negative 1855
2 well positive 1523
3 good positive 1380
4 great positive 981
5 like positive 725
6 better positive 639
7 enough positive 613
8 happy positive 534
9 love positive 495
10 pleasure positive 462
# ℹ 2,575 more rows
Same results as in the text book
bing_word_counts |>group_by(sentiment) |>slice_max(n, n =10) |>ungroup() |>mutate(word =reorder(word, n)) |>ggplot(aes(n, word, fill = sentiment)) +geom_col(show.legend =FALSE) +facet_wrap(~sentiment, scales ="free_y") +labs(x ="Contribution to sentiment",y =NULL)
In the textbook, it is mentioned that the word “miss” is counted as a negative word but, in these books, “miss” is an unmarried woman. It is also mentioned that the word “miss” can be added to the list of “stop words” (which in chapter 1 of the textbook are described as: “words that are not useful for an analysis, typically extremely common words such as “the”, “of”, “to”, and so forth in English”).
# A tibble: 1,150 × 2
word lexicon
<chr> <chr>
1 miss custom
2 a SMART
3 a's SMART
4 able SMART
5 about SMART
6 above SMART
7 according SMART
8 accordingly SMART
9 across SMART
10 actually SMART
# ℹ 1,140 more rows
Continuing the analysis of Austen’s novels
A wordcloud can be constructed. This is done for all six novels.
tidy_books |>anti_join(stop_words) |>count(word) |>with(wordcloud(word, n, max.words =100))
Joining with `by = join_by(word)`
In a wordcloud, it is also possible to separate positive and negative words.
Warning in inner_join(tidy_books, get_sentiments("bing")): Detected an unexpected many-to-many relationship between `x` and `y`.
ℹ Row 435434 of `x` matches multiple rows in `y`.
ℹ Row 5051 of `y` matches multiple rows in `x`.
ℹ If a many-to-many relationship is expected, set `relationship =
"many-to-many"` to silence this warning.
Interpreting this wordcloud is a bit complicated as the size of the words show their relative importance within each group (i.e. among the positive and the negative sentiments). Thus, as it says in the textbook: “the sizes of the words are not comparable across sentiments”
Also, the words in this graph are different from the ones in the textbook. And the wordcloud does not appear in the Plots tab in the R panel.
Another type of analysis involves exploring sentences instead of individual words.
# A tibble: 6 × 2
book chapters
<fct> <int>
1 Sense & Sensibility 51
2 Pride & Prejudice 62
3 Mansfield Park 49
4 Emma 56
5 Northanger Abbey 32
6 Persuasion 25
Then, the negative sentiments are counted in each chapter. As different chapters have varying amounts of words, the ratio of negative words to total words in the chapter is calculated, This is used to find the chapter with the highest ratio of negative words.
`summarise()` has regrouped the output.
ℹ Summaries were computed grouped by book and chapter.
ℹ Output is grouped by book.
ℹ Use `summarise(.groups = "drop_last")` to silence this message.
ℹ Use `summarise(.by = c(book, chapter))` for per-operation grouping
(`?dplyr::dplyr_by`) instead.
tidy_books |>semi_join(bingnegative) |>group_by(book, chapter) |>summarize(negativewords =n()) |>left_join(wordcounts, by =c("book", "chapter")) |>mutate(ratio = negativewords/words) |>filter(chapter !=0) |>slice_max(ratio, n =1) |>ungroup()
Joining with `by = join_by(word)`
`summarise()` has regrouped the output.
# A tibble: 6 × 5
book chapter negativewords words ratio
<fct> <int> <int> <int> <dbl>
1 Sense & Sensibility 43 161 3405 0.0473
2 Pride & Prejudice 34 111 2104 0.0528
3 Mansfield Park 46 173 3685 0.0469
4 Emma 15 151 3340 0.0452
5 Northanger Abbey 21 149 2982 0.0500
6 Persuasion 4 62 1807 0.0343
The results are the same as in the textbook
Exploring different books
In order to explore different books, I use the gutenbergr package. It had been retrieved above and I can explore what it contains and how to find information about it with
?gutenbergr
starting httpd help server ... done
??gutenbergr
View(gutenberg_metadata)
The data frame gutenberg_metadata has 8 elements from each book such as author, the language, and the subject matter or category. The latter is called gutenberg_bookshelf. It can be arranged alphabetically. This allowed me to find books on economics. There are about 45 titles. From these ones I have selected four important classic book from the history of economic thought: A. Smith’s The Wealth of Nations (gutenbergr ID: 38194), D. Ricardo’s Principles of Political Economy and Taxation (33310), W. Jevons’ Political Economy (33219), and J. M. Keynes’ General Theory (65278).
As it is explained below, I was not expecting the results I found. As this pertains to the book by Jevons, I explored its content a bit further. Although I did this afterwards, it seems logical to mention it here.
# A tibble: 44,810 × 4
gutenberg_id linenumber chapter word
<int> <int> <int> <chr>
1 33219 1 0 transcribers
2 33219 1 0 note
3 33219 3 0 in
4 33219 3 0 this
5 33219 3 0 book
6 33219 3 0 all
7 33219 3 0 words
8 33219 3 0 and
9 33219 3 0 phrases
10 33219 3 0 surrounded
# ℹ 44,800 more rows
The list of chapters does not match the version of Jevon’s classic book “The Theory of Political Economy”. Also, the title in gutenbergr is only “Political Economy”, not the full title and it is an edition from 1880, which is not the first one (from 1871), nor one of the editions listed in https://www.hetwebsite.net/het/profiles/jevons.htm, which is one of the best websites for history of economic thought. Besides biographies of major economists throughout history it has links to PDFs or online versions of the original books and articles. For instance, in this case, it has a link to https://books.google.com.ar/books?id=Sw8ZAAAAYAAJ&pg=PR3&redir_esc=y#v=onepage&q&f=false (a PDF of a photocopy of the book at the Littauer Library of Harvard University)
In spite of these challenges, I have used the book in the exercise that follows. Some of the conclusions below would have to be taken with caution and require further exploration. Nevertheless, I was able to carry out the exercise.
I looked for other lexicons online, using “tidytext sentiment lexicon” in Google.
I have found a reference to chapter 5.3.1 in “Text Mining with R: A Tidy Approach”!
There I found the Loughran and McDonald dictionary of financial sentiment terms
From my work on macroeconomics years ago, I have always been interested in uncertainty. So I chose uncertainty over the other words they use for sentiment categories (which are: negative, positive, litigious, strong modal, weak modal, and constraining)
# A tibble: 104 × 2
word n
<chr> <int>
1 may 143
2 could 51
3 might 47
4 risk 43
5 fluctuations 41
6 possible 33
7 depends 25
8 perhaps 23
9 fluctuation 22
10 nearly 21
# ℹ 94 more rows
As it seemed to work, I have reproduced the work done for Jane Austen above, to the four classic economics books (trying to analyze the presence of sentiments about uncertainty).
Warning in inner_join(tidy_books_g, get_sentiments("loughran")): Detected an unexpected many-to-many relationship between `x` and `y`.
ℹ Row 3116 of `x` matches multiple rows in `y`.
ℹ Row 2914 of `y` matches multiple rows in `x`.
ℹ If a many-to-many relationship is expected, set `relationship =
"many-to-many"` to silence this warning.
As expected, it does seem there are more occurrences of words related to uncertainty on Keynes’ General Theory than the other three books. Also, they tend to be more negative.
Also, just in case, I also explored combining less lines than in the novels, under the assumption that in novels more lines may be need to capture sentiments than in theoretical books about econonmics. I have used 40 instead of 80 lines.
Warning in inner_join(tidy_books_g, get_sentiments("loughran")): Detected an unexpected many-to-many relationship between `x` and `y`.
ℹ Row 3116 of `x` matches multiple rows in `y`.
ℹ Row 2914 of `y` matches multiple rows in `x`.
ℹ If a many-to-many relationship is expected, set `relationship =
"many-to-many"` to silence this warning.
`summarise()` has regrouped the output.
ℹ Summaries were computed grouped by gutenberg_id and chapter.
ℹ Output is grouped by gutenberg_id.
ℹ Use `summarise(.groups = "drop_last")` to silence this message.
ℹ Use `summarise(.by = c(gutenberg_id, chapter))` for per-operation grouping
(`?dplyr::dplyr_by`) instead.
tidy_books_g |>semi_join(uncertainty_g) |>group_by(gutenberg_id, chapter) |>summarize(negativewords =n()) |>left_join(wordcounts, by =c("gutenberg_id", "chapter")) |>mutate(ratio = negativewords/words) |>filter(chapter !=0) |>slice_max(ratio, n =1) |>ungroup()
Joining with `by = join_by(word)`
`summarise()` has regrouped the output.
Here I was surprised by two elements. First, A. Smith’s book does not appear. It is strange because it does show in the graph.
Secondly, as a proportion of the words in the chapter with more occurrences, the book by Keynes comes in last. I was intrigued by Jevons’ book having more occurrences of words associated to sentiments of uncertainty than Keynes’. At this point I explored in depth the book by Jevons as mentioned above. The results probably need to be taken with a grain of salt.
Exploring uncertainty in Jane Austen’s novels
As I have worked with the Loughran and McDonald lexicon to explore uncertainty, I have used it to analyze Jane Austen’s novels too. I thought it could be interesting to check if Jane Austen used words related to uncertainty (given she deals with family and romantic relationships).
I have used the uncertainty lexicon to explore Jane Austen’s Emma. This is a first step to check if the analysis can work.
# A tibble: 91 × 2
word n
<chr> <int>
1 could 837
2 might 326
3 may 221
4 believe 102
5 perhaps 99
6 doubt 98
7 almost 88
8 possible 84
9 appeared 49
10 believed 46
# ℹ 81 more rows
It seems to work. It is interesting to note that seven of these words are verbs. Among the other words there is one substantive, one adverb, and one adjective. However, the substantive (“doubt”) could also be a verb.
As it does seem to work, I look at all the novels at the same time.
Warning in inner_join(tidy_books, get_sentiments("loughran")): Detected an unexpected many-to-many relationship between `x` and `y`.
ℹ Row 1252 of `x` matches multiple rows in `y`.
ℹ Row 2772 of `y` matches multiple rows in `x`.
ℹ If a many-to-many relationship is expected, set `relationship =
"many-to-many"` to silence this warning.
There is an error message (as it was the case when trying to reproduce the results from chapter 2. However, it does not seem to affect the constructions of the following graphs.
There are many instances of words related to uncertainty on the negative side in all six novels. In particular, “Negative uncertainty” appears towards, but just before, the end. This is particularly the case in Mansfield Park.
Moreover, in all then there seem to be “early crises” with a lot of negative uncertainty related words: Emma (around 50), Mansfield Park, (post 50 and post 100), Pride and Prejudice (=~ 80 and 120).
Finally, in Sense and Sensibility there are longer stretches of “Negative uncertainty” than in the other novels.
It would be interesting to explore if this is any way related to the particular times in which they were written.
