Intro to Text Analysis

library(tidyverse)
library(DT)
library(tidytext)        # package for text analysis
library(readxl)          # reads excel files, the format I used for the data

1. The following will read in the text and unnest the words for inaugural speeches.

in_notes <- read_excel("inaug_speeches.xlsx")

in_notes

in_words <- in_notes %>%
  unnest_tokens(word, text)

in_words

NA

The previous graphs show that we are looking at inaugural speeches from Washington, Jefferson, Lincoln, FDR, Kennedy, Reagan, and Obama. The second graph shows every word used in each of their speeches.

2. The following shows lexical diversity, lexical density, and the total number of words for each inaugural speech.

in_words %>% 
  group_by(author) %>% 
  summarise(num_words = n(),
            lex_diversity = n_distinct(word), 
            lex_density = n_distinct(word)/n())

Lexical diversity refers to the number of distinct words that the inaugural speech contained. This is one measure of an individual’s vocabulary. However, longer speeches have more distinct words. Because of that, lexical density is measured. Lexical density refers to the number of distinct words divided by the total number of words. The higher the lexical density is, the less repeat words are being used. In the graph above, Washington has the highest lexical density and Lincoln has the lowest.

3. The following shows the mean word length of each inaugural speech.

in_words %>%
  group_by(author) %>% 
  mutate(word_length = nchar(word)) %>% 
  summarize(mean_word_length = mean(word_length)) %>% 
  arrange(-mean_word_length)

The table above shows the mean word length for each inaugural speech. The inaugural speeches were very similar in mean word length, with Washington having the biggest mean word length of 4.9 and Kennedy having the smallest mean word length with 4.4.

4. The following graphs show the mean word length for each inaugural speech.

in_words %>%
  mutate(word_length = nchar(word)) %>% 
  ggplot(aes(word_length)) +
  geom_histogram(binwidth = 1) +
  facet_wrap(vars(author), scales = "free_y") +
  labs(title = "Average Word Length of Inaugural Speeches")

According to the histograms, the presidents all had similar mean word lengths in their inaugural speeches. It is important to note that the y-axis is different for each histogram.

5. The following shows the most common words used in each inaugural speech with stop words removed.

in_words %>%
  anti_join(stop_words) %>% 
  group_by(author) %>% 
  count(word, sort = T) %>%
  top_n(5) %>% 
  ungroup() %>% 
  mutate(word = reorder(word, n)) %>%
  ggplot(aes(word, n, fill = author)) +
  geom_col(show.legend = FALSE) +
  labs(x = NULL, y = "Most Common Words") +
  facet_wrap(vars(author), scales = "free") +
  scale_fill_viridis_d() +
  theme_minimal() +
  coord_flip()

Joining, by = "word"
Selecting by n

The above graphs show the most common words used according to each president’s speech after stop words have been removed. There are some similarities and differences as one can see from looking at them. For example, Washington, Reagan, and Jefferson all had the word “government” as the most used word. FDR and Obama had a similar word they both used that was the most common, “national” and “nation”. From looking at the histograms, it makes sense that these were the most common words that were used in inaugural speeches.

6. The following graphs show words with the highest tf-idfs in each inaugural speech.

in_word_counts <- in_notes %>%             # This counts each word per author
  unnest_tokens(word, text) %>%
  count(author, word, sort = TRUE) 

total_words <- in_word_counts %>%               # This counts total words per author
  group_by(author) %>% 
  summarize(total = sum(n))

in_word_counts <- left_join(in_word_counts, total_words)    # Joins the two

Joining, by = "author"

in_tf_idf <- in_word_counts %>%             # Calculates tf-idf
  bind_tf_idf(word, author, n)

in_tf_idf %>%                                   # Displays it
  arrange(-tf_idf)                          

NA

in_tf_idf %>%
  arrange(-tf_idf) %>%
  mutate(word = factor(word, levels = rev(unique(word)))) %>% 
  group_by(author) %>% 
  top_n(5) %>% 
  ggplot(aes(word, tf_idf, fill = author)) +
  geom_col(show.legend = FALSE) +
  labs(x = NULL, y = "tf-idf") +
  facet_wrap(~author, scales = "free") +
  coord_flip() +
  theme_minimal() +
  scale_fill_viridis_d() +
  labs(title= "Most Distinctive Words in Each Inaugural Speech")

Selecting by tf_idf

The above graphs show what words are unique to each inaugural speech when all speeches are taken into account. In other words, it’s a measure of how often a word appears in one document, divided by how often it appears in other documents. Furthermore, tf-idf automatically excludes most stop words because they will appear in all of the notes. With all of that being said, some inaugural speeches have more unique words than others. For example, Washington’s speech has more than Obama’s. Perhaps, the words in the graphs are a good indicator of what was happening during the time that president gave their inaugural speech.