R Notebook

Kaitlin Kavlie PSYC-541 Lab #8: Text Analysis of Manifestos

1. Using the first code below I read the Manifestos texts into R and created a data set called ‘manifestos’. Then with the second code I unnest the words for each of the manifestos.

manifestos <- read_excel('manifestos.xlsx')

manifestos

manifestos_words <- manifestos %>%
  unnest_tokens(word, text)

manifestos_words

NA

2. By using the code shown below, I created a table that includes both lexical diversity and density, as well as the total number of words for each manifesto.

manifestos_words %>% 
  group_by(author) %>% 
  summarise(num_words = n(),
            lex_diversity = n_distinct(word), 
            lex_density = n_distinct(word)/n())

3. I produced a table with the mean word length of each of the manifestos by using the code shown below.

manifestos_words %>%
  group_by(author) %>% 
  mutate(word_length = nchar(word)) %>% 
  summarize(mean_word_length = mean(word_length)) %>% 
  arrange(-mean_word_length)

4. I generated a graph using the code chunk below, which includes mini histograms of each manifesto’s word lengths.

manifestos_words %>%
  mutate(word_length = nchar(word)) %>% 
  ggplot(aes(word_length)) +
  geom_histogram(binwidth = 1) +
  facet_wrap(vars(author), scales = "free_y") +
  labs(title = "Word lengths by manifesto author")

5. I removed stop words and then created a graph with the most common words in each document.

I used this first code to load the list of stop words.

stop_words

Then I used the code directly below to remove the stop words from the manifestos_words data set.

manifestos_words %>%
  anti_join(stop_words)

Joining, by = "word"

The code chunk below was used to create a graph of the most used words in each of the manifestos.

manifestos_words %>%
  anti_join(stop_words) %>% 
  group_by(author) %>% 
  count(word, sort = T) %>%
  top_n(5) %>% 
  ungroup() %>% 
  mutate(word = reorder(word, n)) %>%
  ggplot(aes(word, n, fill = author)) +
  geom_col(show.legend = FALSE) +
  labs(x = NULL, y = "Most common words in each manifesto") +
  facet_wrap(vars(author), scales = "free") +
  scale_fill_viridis_d() +
  theme_minimal() +
  coord_flip()

Joining, by = "word"
Selecting by n

6. Using the first code shown below, I calculated tf-idfs for each manifesto and a table displaying the results. Then with the last code below I created a graph of the words with the highest tf-idfs in each manifesto

manifestos_word_counts <- manifestos %>%             
  unnest_tokens(word, text) %>%
  count(author, word, sort = TRUE) 

total_words <- manifestos_word_counts %>%               
  group_by(author) %>% 
  summarize(total = sum(n))

manifestos_word_counts <- left_join(manifestos_word_counts, total_words)    

Joining, by = "author"

manifestos_tf_idf <- manifestos_word_counts %>%             
  bind_tf_idf(word, author, n)

manifestos_tf_idf %>%                                   
  arrange(-tf_idf)                          

NA

With this final code below I created a graph of the words with the highest tf-idfs in each manifesto.

manifestos_tf_idf %>%
  arrange(-tf_idf) %>%
  mutate(word = factor(word, levels = rev(unique(word)))) %>% 
  group_by(author) %>% 
  top_n(5) %>% 
  ggplot(aes(word, tf_idf, fill = author)) +
  geom_col(show.legend = FALSE) +
  labs(x = NULL, y = "tf-idf") +
  facet_wrap(~author, scales = "free") +
  coord_flip() +
  theme_minimal() + 
  scale_fill_viridis_d() +
  labs(title = "Most distinctive words in each manifesto")

Selecting by tf_idf