Intro to Text Analysis

library(tidyverse)
library(DT)
library(tidytext)        # package for text analysis
library(readxl)          # reads excel files, the format I used for the data

Assignment

1. Read in the text and unnest the words.

manifestos <- read_excel('manifestos.xlsx')

manifestos

NA

manifestos_words <- manifestos %>%
  unnest_tokens(word, text)

manifestos_words

NA
NA

I used the codes above to first read in the Manifestos document to R, and to “unnest” the words within the spreadsheet.

2. Generate a table that includes both lexical diversity and density, and the total number of words, of each document.

manifestos_words %>% 
  group_by(author) %>% 
  summarise(num_words = n(),
            lex_diversity = n_distinct(word), 
            lex_density = n_distinct(word)/n())

NA

I used the code above to create a table that includes the lexical diversity and density, as well as showing a total word count for each document.

3. Generate a table with the mean word length of each document.

manifestos_words %>%
  group_by(author) %>% 
  mutate(word_length = nchar(word)) %>% 
  summarize(mean_word_length = mean(word_length)) %>% 
  arrange(-mean_word_length)

NA
NA

I used the code above to generate a table to show the mean word length for each document.

4. Generate a graph with mini histograms of each document’s word lengths.

manifestos_words %>%
  mutate(word_length = nchar(word)) %>% 
  ggplot(aes(word_length)) +
  geom_histogram(binwidth = 1) +
  facet_wrap(vars(author), scales = "free_y") +
  labs(title = "Word Lengths per Manifesto")

The code above was used to create mini histograms to compare average word length for the different manifesto documents.

5. Remove stop words and then create a graph with the most common words in each document.

manifestos_words %>%
  anti_join(stop_words) %>% 
  group_by(author) %>% 
  count(word, sort = T) %>%
  top_n(5) %>% 
  ungroup() %>% 
  mutate(word = reorder(word, n)) %>%
  ggplot(aes(word, n, fill = author)) +
  geom_col(show.legend = FALSE) +
  labs(x = NULL, y = "Most Common Words in Each Document") +
  facet_wrap(vars(author), scales = "free") +
  scale_fill_viridis_d() +
  theme_minimal() +
  coord_flip()

Joining, by = "word"Selecting by n

The code above was used to remove the stop words and then graph the most common words in each document. Cho used the most vulgar language of the group.

6. Calculate tf-idfs and create a graph of the words with the highest tf-idfs in each document.

manifestos_word_counts <- manifestos %>%             
  unnest_tokens(word, text) %>%
  count(author, word, sort = TRUE) 

total_words <- manifestos_word_counts %>%               
  group_by(author) %>% 
  summarize(total = sum(n))

manifestos_word_counts <- left_join(manifestos_word_counts, total_words)    

Joining, by = "author"

manifestos_tf_idf <- manifestos_word_counts %>%             
  bind_tf_idf(word, author, n)

manifestos_tf_idf %>%                                   
  arrange(-tf_idf)                          

NA

manifestos_tf_idf %>%
  arrange(-tf_idf) %>%
  mutate(word = factor(word, levels = rev(unique(word)))) %>% 
  group_by(author) %>% 
  top_n(5) %>% 
  ggplot(aes(word, tf_idf, fill = author)) +
  geom_col(show.legend = FALSE) +
  labs(x = NULL, y = "tf-idf") +
  facet_wrap(~author, scales = "free") +
  coord_flip() +
  theme_minimal() + 
  scale_fill_viridis_d() +
  labs(title = "Most Unique Words in Each Manifesto")

Selecting by tf_idf

With the codes above, I first calculated the tf-idfs and created a table to display the results. I then used the data to create a graph of the words with the highest tf-idfs in each manifesto.