The dataset used in this assignment contains the manifestos of six different serial killers. The following table contains the number of words in each manifesto, the number of unique words, and the density of unique words in each manifesto.
Breivik’s document seems to have the most words, as well as the most lexical diversity. Harper Mercer, however, has the highest lexical density. These statistics are better portrayed in the following graph:
mani_words |>mutate(word_length =nchar(word)) |>ggplot(aes(word_length)) +facet_wrap(vars(author), scales ="free_y") +geom_histogram(binwidth =1) +labs(title ="Word Length Distribution for Manifestos of Mass Murderers") +theme_minimal()
Most Commonly Used Words
The following graph shows a list of the most commonly used words in each author’s manifestos:
mani_words |>group_by(author) |>count(word, sort = T) |>top_n(5) |>ungroup() |>mutate(word =reorder(word, n)) |>ggplot(aes(word, n, fill = author)) +geom_col(show.legend =FALSE) +coord_flip() +facet_wrap(~author, scales ="free") +# creates separate graphs for each authorscale_fill_viridis_d() +# uses a nicer color schemetheme_minimal() +# removes the gray backgroundlabs(x =NULL, y ="Most Common Words In Each Author's Manifesto")
Selecting by n
I don’t think it’s very surprising that words such as “the”, “a”, or “you”, so let’s remove those stopwords from the list.
Instead of including each author’s top 5 most used words, I chose to do 10 to see more of what their document was about.
mani_words |>anti_join(snowball) |>group_by(author) |>count(word, sort = T) |>top_n(10) |>ungroup() |>mutate(word =reorder(word, n)) |>ggplot(aes(word, n, fill = author)) +geom_col(show.legend =FALSE) +labs(x =NULL, y ="Most Common Words In Each Author's Manifesto") +facet_wrap(vars(author), scales ="free") +scale_fill_viridis_d() +theme_minimal() +coord_flip()
Joining with `by = join_by(word)`
Selecting by n
There are several interesting themes emerging here. Harper Mercer’s document seem to center around race as an issue, while Breivik and the Unabomber seem to discuss the issues of our modern society and government.
TF-IDF
A measure of TF-IDF shows the words that are more frequently used in one document compared to every other documents. The following table and graph shows the highest TF-IDF values of each author’s manifestos:
mani_word_counts <- manifestos |># This counts each word per authorunnest_tokens(word, text) |>count(author, word, sort =TRUE) mani_total_words <- mani_word_counts |># This counts total words per authorgroup_by(author) |>summarize(total =sum(n))mani_word_counts <-left_join(mani_word_counts, mani_total_words) # Joins the two
# A tibble: 5,879 × 7
author word n total tf idf tf_idf
<chr> <chr> <int> <int> <dbl> <dbl> <dbl>
1 Cho you 130 1971 0.0660 0.405 0.0267
2 Cho your 34 1971 0.0173 1.10 0.0190
3 Cho fuck 19 1971 0.00964 1.79 0.0173
4 Breivik cultural 42 5186 0.00810 1.79 0.0145
5 Breivik marxism 37 5186 0.00713 1.79 0.0128
6 Auvinen intelligent 10 1750 0.00571 1.79 0.0102
7 Auvinen minded 10 1750 0.00571 1.79 0.0102
8 Cho jesus 18 1971 0.00913 1.10 0.0100
9 Unabomber leftist 27 5100 0.00529 1.79 0.00949
10 Breivik frankfurt 27 5186 0.00521 1.79 0.00933
# ℹ 5,869 more rows
mani_tf_idf |>arrange(-tf_idf) |>mutate(word =factor(word, levels =rev(unique(word)))) |>group_by(author) |>top_n(10) |>ggplot(aes(word, tf_idf, fill = author)) +geom_col(show.legend =FALSE) +labs(x =NULL, y ="TF-IDF") +facet_wrap(~author, scales ="free") +coord_flip() +theme_minimal() +scale_fill_viridis_d() +labs(title ="Most Distinctive Words in Each Manifesto")
Selecting by tf_idf
There are many interesting patterns here. Auvinen, for example, had many instances of mentioning intelligence and retardation. The unabomber had strong opinions on leftist ideaology. Cho has many instances of strong and aggressive language.
Sentiment Analysis
The following words best reflect the sentiment in all the authors’ manifestos, both the most common positively and negatively associated words:
bing <-get_sentiments("bing")mani_words |>inner_join(bing) |>count(word, sentiment, sort =TRUE) |>group_by(sentiment) |>top_n(10) |>ungroup() |>mutate(word =reorder(word, n)) |>ggplot(aes(word, n, fill = sentiment)) +geom_col(show.legend =FALSE) +facet_wrap(vars(sentiment), scales ="free") +labs(y ="Words that Contribute The Most to Each Manifesto's Sentiment",x =NULL) +scale_fill_viridis_d() +coord_flip() +theme_minimal()
Joining with `by = join_by(word)`
Selecting by n
The negative words are the most interesting ones here. I’m surprised that terrorism was mentioned as often as it was, while inferiority also seems to be a surprising recurring theme.
Bigrams
The following table shows the most common bigrams in the manifesto documents:
manifestos |>unnest_tokens(bigram, text, token ="ngrams", n =2) |>select(bigram) -> mani_bigrams
mani_bigrams |>count(bigram, sort = T)
# A tibble: 14,954 × 2
bigram n
<chr> <int>
1 of the 159
2 in the 81
3 to the 69
4 i was 60
5 and the 56
6 to be 43
7 political correctness 37
8 i had 34
9 it is 34
10 it was 30
# ℹ 14,944 more rows
Most of the bigrams in the top 10 aren’t very interesting, except for “political correctness”. This could be that many of the serial killers were politically motivated to perform their crimes. The next table ignores instances of stopwords in bigrams:
# A tibble: 4,115 × 2
bigram n
<chr> <int>
1 political correctness 37
2 frankfurt school 22
3 jesus christ 17
4 cultural marxism 16
5 black man 12
6 human race 12
7 modern leftism 9
8 power process 9
9 surrogate activities 9
10 will never 9
# ℹ 4,105 more rows
Many more interesting words appeared on the top of this list. We have the theme of race again, probably from Harper Mercer, and “modern leftism” likely from the unabomber.
First Words
This final graph shows the most frequent words in all manifestos that came directly after three words, “I”, “you”, and “the”.
first_word <-c("i", "you", "the") # these need to be lowercasemani_bigrams |>count(bigram, sort = T) |>separate(bigram, c("word1", "word2"), sep =" ") |># separate the two wordsfilter(word1 %in% first_word) |># find first words from our listcount(word1, word2, wt = n, sort =TRUE) |>mutate(word2 =factor(word2, levels =rev(unique(word2)))) |># put the words in ordergroup_by(word1) |>top_n(7) |>ggplot(aes(word2, n, fill = word1)) +scale_fill_viridis_d() +# set the color palettegeom_col(show.legend =FALSE) +labs(x =NULL, y =NULL, title ="Word following 'I', 'You', and 'The' in Serial Killers' Manifestos") +facet_wrap(~word1, scales ="free") +coord_flip() +theme_minimal()
Selecting by n
What I found most interesting about this data is that the authors often referred to themselves in the past tense as if they had lost something, and often referred to their audience in the present or future tense. There were also a few interesting words that followed “the”, namely “world” and “system”.
