Random Seeds

As part of LDA, we know that we find the probability that each word \(i\) in a document \(d\) comes from a certain topic. For instance, we could have:

\[\gamma_1 = \left( .2, .1, .7, .1 \right)\]

This means that there is a 20% probability that a word \(i\) in document 1 comes from Topic 1, a 10% probability it comes from Topic 2, a 70% probability it comes from Topic 3, and a 10% probability it comes from Topic 4.

Once we find these probabilities, part of the model involves sampling a topic \(t_{i1}\) for each word \(i\) in document 1 using these probabilities. We can do that using this code:

sample( 1:4, 1, prob = c(.2,.1,.7,1))

This function has three inputs:

• 1:4: this tells the computer what our options are for sampling. In this case, it is the topics 1-4.
• 1: this tells the computer how many topics we want to sample. Each word \(i\) can have only one topic, so we choose 1.
• prob =: this is a vector that tells the computer the probability of sampling each of the topics.

When you run the sample code above, you get a number 1, 2, 3, or 4. Great…however, suppose you close your Markdown file and come back to it later. We want the computer to choose the SAME topic when you run your code again. Otherwise, you would get completely different results each time you drew a random sample! The job of theset.seed() function is to do just that. It ensures that each time you run your chunk, you will get the same random sample. Let’s try that.

You should notice that every time you hit play on this chunk, the result can change. This is what would happen if you closed your Markdown and re-opened it, or if you gave your code to someone else to run. This is not something we want to have happen, so we set a random seed to fix this problem.

Now we notice that no matter how often we run the chunk, we get the same values. Yes!! This means that we can close our R and come back to it later, and our results will not change. This also means that we can send our code to someone else, and they will get the same random sample that we did. This means that setting a seed can help make your code reproducible.

Setting a random seed (which is what we will call using the set.seed() command) will prove very useful for any kind of random sampling we do in this course.

Back to the model: Word Probabilities

Recall that we were talking about the one line of code we need to run LDA before we got onto the tangent about random seeds:

chapters_lda <- LDA(chapters_dtm, k = 4, control = list(seed = 1234))

There are two things we want to retrieve from this output: our estimates of \(\gamma\) and our estimates of \(\beta\). We will start with \(\beta\).

In order to retrieve the estimates of \(\beta\) from LDA, we can use the following:

beta_estimates <- tidy(chapters_lda, matrix = "beta")

The \(\beta_t\) terms show us the probability of getting each of \(W\) words in documents from topic \(t\). This means that one way to determine the words that make up topic \(t\) is to look at the words that have the highest probability. In other words, we look at the words in \(\beta_t\) that have the highest probability for each topic \(t\).

To do this, we can use code that should be very familiar to us to by now.

top_terms <- beta_estimates |>
  group_by(topic) |>
  slice_max(beta, n = 10) |>
  ungroup() |>
  # Order the data in decreasing beta
  # order within each topic 
  arrange(topic, -beta)

Another way we can visualize the terms defining each topic is by using a graph. Again, this code should be familiar.

# Arrange the data so facet wrap keeps them in order
top_terms_plot <- top_terms |>
  mutate(term = reorder_within(term, beta, topic))

# Create the plot 
ggplot(top_terms_plot,aes(beta, term, fill = factor(topic))) +
  geom_col(show.legend = FALSE) +
  facet_wrap(~ topic, scales = "free") +
  scale_y_reordered() 

Topic Probability Estimates

At this point, we can see the words that seem to define each topic. We hope each of these topics relates to one of the 4 books in our data set. The next step is to see if the model was able to accurately separate the chapters into books. In other words, did the model actually create topics for each book, and did it do so correctly?

To determine all of this, we need to switch gears from estimating \(\beta\) to estimating \(\gamma\).

In order to retrieve the estimates of \(\gamma\), we can use the following:

gamma_estimates <- tidy(chapters_lda, matrix = "gamma")

Let’s look at the first document, which is the first chapter in the book The War of the Worlds.

subset(gamma_estimates, document == "The War of the Worlds_1")

Okay, great!! This means that for each document, we can take a look and see if the highest probability in \(\gamma_d\) is the correct book!

There is only one small problem right now, and it has to do with how one of the columns is named. Right now, you’ll notice the first document is named The War of the Worlds_1. This is because this document is the 1st chapter in the book The War of the Worlds.

Our task is going to be to see if the model correctly identified the books the chapter came from. This means it would be helpful to have just the book title. To clean up the document names, we can use the following:

 gamma_estimates <- gamma_estimates |>
  separate(document, c("title", "chapter"), sep = "_", convert = TRUE)

This changes the title column to just indicate which book a document came from.

There are a few ways we can go about seeing how well our model was able to separate documents into books. One option is to look at all the documents (chapters) in each book and find the \(\gamma\) vector for each one. We can then take the average of all these vectors to determine which topics the documents tended to be placed in.

 gamma_estimates|>
  group_by(title,topic)|>
  summarise(mean(gamma))

Again, recall that the hope is that the \(\gamma_d\) vector will place most of the probability on the topic that is actually associated with the correct book.

We can also visualize the information in a graph.

gamma_estimates |>
  mutate(title = reorder(title, gamma * topic)) |>
  ggplot(aes(factor(topic), gamma)) +
  geom_boxplot() +
  facet_wrap(~ title) +
  labs(x = "topic", y = expression(gamma))

What we can see from the graph and the table output is that for each book, the highest probability is associated with a different topic. This means that the model was able to correctly separate into the 4 different books!

Choosing T

Now, all of this relied on us knowing that there were 4 topics (4 books), so we could correctly choose \(T=4\). In practice, there are a few different tools that we can use to choose \(T\). One possibility is to choose a large \(T\), like \(T\) = 10. Once we see the words defining the topics, we may be able to see that some topics can be combined, which means we can train the model again using a lower choice of \(T\). There are also versions of LDA that add an extra layer to the model that estimates \(T\). There are a lot of choices depending on our modeling goals!

References

Data

Johnston M, Robinson D (2023). gutenbergr: Download and Process Public Domain Works from Project Gutenberg. R package version 0.2.4, https://CRAN.R-project.org/package=gutenbergr.

Code

This analysis and code was adapted from Chapter 6 of “Text Mining with R: A Tidy Approach”, written by Julia Silge and David Robinson (https://www.tidytextmining.com/topicmodeling). The book was last built on 2024-06-20.