Word Co-occurrence Analysis

This analysis uses the 300 YouTube comments collected in my previous exercise. The comments came from three videos related to Palestine and were previously saved in youtube_comments_multi_video.csv.

Step 1: Load Packages and Data

library(tidyverse)
library(tidytext)
library(widyr)
library(igraph)
library(ggraph)

comments <- read_csv("youtube_comments_multi_video.csv")

comments <- comments %>%
  select(video_label, comment_number, comment_text) %>%
  filter(!is.na(comment_text), comment_text != "")

Step 2: Tokenization and Processing

comment_words <- comments %>%
  unnest_tokens(word, comment_text)

comment_words_clean <- comment_words %>%
  anti_join(stop_words, by = "word") %>%
  filter(str_detect(word, "[a-z]")) %>%
  filter(!str_detect(word, "^[0-9]+$"))

Step 3: Review the Most Frequent Words

comment_words_clean %>%
  count(word, sort = TRUE) %>%
  slice_head(n = 25)
## # A tibble: 25 × 2
##    word             n
##    <chr>        <int>
##  1 israel          49
##  2 people          34
##  3 palestine       30
##  4 genocide        27
##  5 israeli         23
##  6 it’s            23
##  7 free            22
##  8 gaza            22
##  9 hamas           21
## 10 palestinians    20
## # ℹ 15 more rows

Step 4: Add Custom Stopwords

After reviewing the word-frequency table, I added people and it’s because they appeared frequently but were too general to explain the main topics in the comments.

custom_stopwords <- tibble(
  word = c("people", "it’s")
)

comment_words_clean <- comment_words_clean %>%
  anti_join(custom_stopwords, by = "word")

comment_words_clean %>%
  count(word, sort = TRUE) %>%
  slice_head(n = 25)
## # A tibble: 25 × 2
##    word             n
##    <chr>        <int>
##  1 israel          49
##  2 palestine       30
##  3 genocide        27
##  4 israeli         23
##  5 free            22
##  6 gaza            22
##  7 hamas           21
##  8 palestinians    20
##  9 october         19
## 10 war             19
## # ℹ 15 more rows

Step 5: Create Word Co-occurrence Pairs

Words are treated as co-occurring when they appear in the same comment. Repeated uses of a word within one comment are counted only once for that comment.

word_pairs <- comment_words_clean %>%
  distinct(comment_number, word) %>%
  pairwise_count(
    item = word,
    feature = comment_number,
    sort = TRUE,
    upper = FALSE
  )

word_pairs %>%
  slice_head(n = 25)
## # A tibble: 25 × 3
##    item1     item2        n
##    <chr>     <chr>    <dbl>
##  1 palestine free        13
##  2 october   7th         11
##  3 palestine israel       8
##  4 israel    genocide     7
##  5 gaza      genocide     6
##  6 israel    israeli      6
##  7 israel    free         6
##  8 israel    october      5
##  9 gaza      october      5
## 10 israel    idf          4
## # ℹ 15 more rows

Step 6: Filter the Word Pairs

The tutorial originally used filter(n > 1). I changed the value to n > 3, so a word pair must appear together in at least four separate comments to remain in the final network.

word_pairs_filtered <- word_pairs %>%
  filter(n > 3)

word_pairs_filtered
## # A tibble: 21 × 3
##    item1     item2        n
##    <chr>     <chr>    <dbl>
##  1 palestine free        13
##  2 october   7th         11
##  3 palestine israel       8
##  4 israel    genocide     7
##  5 gaza      genocide     6
##  6 israel    israeli      6
##  7 israel    free         6
##  8 israel    october      5
##  9 gaza      october      5
## 10 israel    idf          4
## # ℹ 11 more rows

Step 7: Create the Co-occurrence Network

word_graph <- word_pairs_filtered %>%
  graph_from_data_frame(directed = FALSE)

set.seed(123)

ggraph(word_graph, layout = "fr") +
  geom_edge_link(
    aes(width = n),
    alpha = 0.4,
    show.legend = FALSE
  ) +
  geom_node_point(size = 4) +
  geom_node_text(
    aes(label = name),
    repel = TRUE,
    size = 3.5
  ) +
  theme_void() +
  labs(
    title = "Word Co-occurrence Network of YouTube Comments",
    subtitle = "Word pairs appearing together in at least four comments"
  )

Reflection Questions

Question 1

Which additional words did you add to the stopword list manually? Why does it make sense to remove those words as stopwords? Also, does the final network graph change after removing these additional stopwords? If so, briefly explain how and why.

I manually added people and it’s to the stopword list. Both appeared frequently in the word table, but they were too general to identify a specific topic or opinion in the comments. The word people can refer to many different groups without showing which group or issue is being discussed, while it’s is a common contraction that does not carry useful subject information. The final network changed after removing these terms because their nodes and all of their connections disappeared. This reduced generic connections and made the relationships among topic-specific words easier to see.

Question 2

Change n in filter(n > 1) to a different value. What is your final value of n? Will it give you a better result?

I changed the function to filter(n > 3), so my final value of n is 3. A pair must therefore appear together in at least four comments before it is included in the graph. This gives me a better result because the analysis contains 300 comments, and the original threshold retained too many weak connections. The higher threshold produces a less cluttered network while preserving the word pairs that appeared repeatedly across the comments.

Question 3

On the instructor’s BlueSky post, there were two co-occurrence network graphs. Which one do you like better? Why?

I prefer the second graph because it is easier to read and the strongest word relationships are more visible. The first graph contains more connections, but the number of overlapping edges makes it difficult to identify the main clusters. The second graph is more effective for communicating the results because it removes weaker connections and directs attention toward the most repeated relationships.

Question 4

Which word pairs surprised you? Did “buy” cluster more with optimism words such as “dip,” or caution words such as “wait” and “valuation”?

The relationship between buy and dip was the most interesting because it suggests that some commenters interpreted a lower price as a buying opportunity. In the network, buy appeared to cluster more with the optimistic word dip than with the caution words wait and valuation. However, the presence of the caution terms shows that not all commenters agreed that buying immediately would be a good decision. The network therefore shows both optimism about the investment and concern about its price and timing.

Question 5

Should the stock ticker $SPCX and SpaceX be treated as the same token instead of separate ones?

Yes, I would generally treat $SPCX and SpaceX as the same token because both refer to the same company in this discussion. Keeping them separate divides the company’s frequency and connections between two nodes and can make the company appear less central than it actually is. Standardizing both forms as spacex would create a clearer network. I would still preserve the original comment text in the raw data in case the use of ticker language becomes important in a later analysis.

Question 6

What is the risk of drawing conclusions from a co-occurrence network built on only a small sample of comments? What is a good sample size? How would you communicate that limitation in a business report?

A network based on a small sample can make a few repeated phrases or unusually active commenters appear more important than they are in the larger audience. The graph may also change substantially when only a small number of comments are added or removed. There is no single correct sample size for every text analysis, but several hundred comments are more useful for exploratory analysis than only a few dozen, while larger and more representative samples are preferable for business decisions. My dataset contains 300 comments, which is useful for exploring patterns but is not enough to represent every viewer of the three videos. In a business report, I would clearly state that the network describes the collected sample and should not be treated as proof of the opinions of the entire audience.

Question 7

Is there a way to analyze comments or reviews on topics that interest you in relation to a significant event? If so, how would you identify the event? Why is that event significant, and how might it influence the comments or reviews you analyze?

I could extend this project by comparing comments before and after a significant event related to the Israel-Palestine conflict, such as a ceasefire announcement, a major military escalation, or an important government decision. I would identify the event using its confirmed date and divide the comments into a period before the event and a period after it. The event would be significant because it would introduce new information and could change the subjects, emotions, and arguments appearing in the discussion. Comments before the event might focus on predictions or demands, while later comments might focus on reactions, responsibility, consequences, or whether the event changed the situation. Separate co-occurrence networks would make it possible to compare how the discussion changed over time.