library(tidyverse)
library(tidytext)
library(widyr)
library(igraph)
library(ggraph)
library(Matrix)
library(checkdown)
library(readr)

Tasks for you:

You will perform a word co-occurrence analysis using the reviews or comments you scraped in a previous exercise. We will discuss the key steps in class and brainstorm the most efficient approach to completing the analysis.

Step 1: Load the Raw Text

etf_comments <- read_csv("comments1.csv")

etf_text <- etf_comments %>%
  select(comment_id, text) %>%
  rename(id = comment_id)

etf_text
## # A tibble: 1,577 × 2
##       id text                                                                   
##    <dbl> <chr>                                                                  
##  1     1 "A monster of an ETF guide! Let me know if there are other videos you'…
##  2     2 "Which CDs \U0001f4bf would you recommend with the highest dividends a…
##  3     3 "Proper portfolio allocation"                                          
##  4     4 "@Dee-rc2lt this!!!!"                                                  
##  5     5 "Brother , I am proud of you. I am a new financial advisor in this fie…
##  6     6 "Hey.... You can get connected to Mrs Anna with this number here \U000…
##  7     7 "I’m saving and investing around 70 percent of my income in the financ…
##  8     8 "So what's the most effective strategy during this period of volatilit…
##  9     9 "Amazing video, A friend of mine referred me to a financial adviser so…
## 10    10 "Buy a good cross section of an economy, Build a diverse portfolio tha…
## # ℹ 1,567 more rows

Step 2: Tokenize and Clean

custom_stop_words <- bind_rows(
  stop_words,
  tibble(
    word = c(
      "im","ive","dont","didnt","isnt","lol",
      "video","videos","thanks","thank",
      "youtube","watch","watched","channel",
      "guys","guy","people","really","good",
      "great","love","like","comment","comments",
      "etf","etfs"
    ),
    lexicon = "custom"
  )
)

etf_words <- etf_text %>%
  mutate(text = str_replace_all(text, "[^[:alnum:][:space:]$]", " ")) %>%
  unnest_tokens(word, text, token = "words") %>%
  filter(!str_detect(word, "^[0-9]+$")) %>%   # remove numbers
  anti_join(custom_stop_words, by = "word")

etf_words %>%
  count(word, sort = TRUE) %>%
  head(15)
## # A tibble: 15 × 2
##    word           n
##    <chr>      <int>
##  1 market       186
##  2 investing    178
##  3 ve           152
##  4 portfolio    142
##  5 financial    123
##  6 money        123
##  7 stocks       123
##  8 investment   117
##  9 advisor      100
## 10 time          96
## 11 invest        93
## 12 stock         75
## 13 lot           68
## 14 don           67
## 15 buy           65

Step 3: Count Word Pairs

etf_pairs <- etf_words %>%
  pairwise_count(word, id, sort = TRUE, upper = FALSE)

Step 4: Build the Co-occurrence Matrix

top_words <- etf_words %>%
  count(word, sort = TRUE) %>%
  slice_max(n, n = 15) %>%
  pull(word)

etf_matrix <- etf_pairs %>%
  filter(item1 %in% top_words,
         item2 %in% top_words) %>%
  bind_rows(
    etf_pairs %>%
      rename(item1 = item2, item2 = item1) %>%
      filter(item1 %in% top_words,
             item2 %in% top_words)
  ) %>%
  cast_sparse(item1, item2, n) %>%
  as.matrix()

etf_matrix
##            advisor market stock stocks ve financial investment buy time
## financial       47     39     6     14 16         0         20   6    9
## portfolio       27     39    18     26 27        26         25   8   13
## market          28      0    35     33 29        39         20  16   25
## advisor          0     28     6     12 21        47         18   2    5
## investing       12     19    16     27 25        19         19   9   18
## stocks          12     33    12      0 22        14          9  25   18
## ve              21     29     9     22  0        16         18  11   19
## money           10     22    14     13 12        15         17  15   11
## investment      18     20     7      9 18        20          0   5   11
## buy              2     16     8     25 11         6          5   0   16
## stock            6     35     0     12  9         6          7   8    5
## don              9     11     9     17  9         7          6  13    6
## time             5     25     5     18 19         9         11  16    0
## invest           3     11     7     12 10         4         14   7    9
## lot              2      9     4      7 10         4          7   3    6
##            investing money don invest lot portfolio
## financial         19    15   7      4   4        26
## portfolio         24    16  12     10   8         0
## market            19    22  11     11   9        39
## advisor           12    10   9      3   2        27
## investing          0    21  13      9  12        24
## stocks            27    13  17     12   7        26
## ve                25    12   9     10  10        27
## money             21     0  18     13   9        16
## investment        19    17   6     14   7        25
## buy                9    15  13      7   3         8
## stock             16    14   9      7   4        18
## don               13    18   0      9   2        12
## time              18    11   6      9   6        13
## invest             9    13   9      0   6        10
## lot               12     9   2      6   0         8

Step 5: Visualize the Strongest Co-occurrences

To keep the network readable, we’ll only keep pairs that occurred together more than once.

set.seed(580)

etf_pairs %>%
  filter(n > 8) %>%
  graph_from_data_frame() %>%
  ggraph(layout = "fr") +
  geom_edge_link(aes(edge_alpha = n,
                     edge_width = n),
                 color = "firebrick") +
  geom_node_point(size = 4,
                  color = "steelblue") +
  geom_node_text(aes(label = name),
                 repel = TRUE,
                 size = 3.5) +
  theme_void() +
  labs(
    title = "Word Co-occurrence Network: ETF YouTube Comments",
    subtitle = "Edge thickness = number of comments containing both words"
  )

Reflection questions for you

Question 1

For this exercise, I removed some common filler words such as “ETF”, “ETF’s”, “love”, “like”, “video”, and more related words similar to these. Removing these words made sense to me because most of the comments in the video expressed general comments using these words which created a lot of noise in my network graph. By filtering these stopwords from appearing, I was able to get a more concentrated list of words that related to the contents of the video.

Question 2

I started with N>1 and receieved mainly names of people. I’m assuming these were names that were present in the comments which doesn’t add much value towards visualizing the network graph. After some testing, I went with N>8 which really helped giving me a variety of video-related words and showed which word links were the strongest.

Additional discussion Questions (try to use external sources to support your responses)

Question 3

Out of the two options, I prefer the first graph due to the rich quantity of words. With the second graph, I felt it was too simple and lacked variety and only showed few co-occurences. With the first graph, it showed various levels at which words we’re more strongly associated with each other.

Question 4

All the words seem to make sense as pairings but the only one that stood out to me was “details” and “appointment”. They don’t associate as much as the other words towards the video but I can see the case if a commentator wanted more details on an ETF or make an appointment with an adviser.

Question 5

I believe it should be. For my case, ETF and ETF’s both mean the same thing but just formatted differently. By creating them as separate tokes, we’re just adding unnecessary steps since these stopwords both have identical meanings.

Question 6

I feel that it might be interpreted as a generalization since these word pairings are based off one video. It might be a tedious task but I think running this test on other related videos can be a benefit to see if there is a correlation between word pairings to see how common they are. Even with comments, there is always a chance of high comment counts to have bot messages so it would be great to have a sample size of roughly 500 comments to avoid the amount of extra noise coming from the comment data.

Question 7

Yes. To name one, COVID-19 is an interesting event that negatively affected ETF’s. This event triggered a market crash so analyzing the text from videos before the event and afterwards could create interest in how sentiment has shifted around it. With comparing the before and after hrough this same exercise, it would be interesting to compare which words became a central theme and how overall sentiment shifted for the commentators.

Wrap-Up: From Toy Example to Business Insight

The fruit example exists purely to build intuition — once students can predict the network shape by eye, they’re ready to trust the same code on real text, where the patterns aren’t obvious in advance. The SpaceX example mirrors what a sentiment or brand-perception analysis would look like in practice: same five steps (tokenize, clean, pair, matrix, visualize), just messier inputs.

References

Boost Brand Credibility with Co Occurrence SEO. https://www.linkedin.com/posts/umer-abid-78045131a_co-occurrence-seo-is-a- powerful-concept-that-activity-7421759779892809728-MGh_/ https://seopressor.com/blog/why-co-citation-and-co-occurrence-are-such- big-deal/ Kong, J., Scott, A., & Goerg, G. M. (2016). Improving topic clustering on search queries with word co-occurrence and bipartite graph coclustering. https://research.google/pubs/improving-topic-clustering-on-search-queries- with-word-co-occurrence-and-bipartite-graph-co-clustering/ Colladon, A. F. (2018). The semantic brand score. Journal of Business Research, 88, 150-160. Silge, J., & Robinson, D. (2017). Text Mining with R: A Tidy Approach. O’Reilly Media. https://www.tidytextmining.com/ Robinson, D. (2021). widyr: Widen, process, then re-tidy data [R package documentation]. https://CRAN.R-project.org/package=widyr Csardi, G., & Nepusz, T. (2006). The igraph software package for complex network research. InterJournal, Complex Systems, 1695. https://igraph.org Pedersen, T. L. (2024). ggraph: An implementation of grammar of graphics for graphs and networks [R package documentation]. https://CRAN.R- project.org/package=ggraph