Lab 5 - Text Analysis & NLP - Word Co-occurrence Analysis
Keith Concepcion
2026-07-05
library(tidyverse)## ── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ──
## ✔ dplyr 1.2.1 ✔ readr 2.2.0
## ✔ forcats 1.0.1 ✔ stringr 1.6.0
## ✔ ggplot2 4.0.3 ✔ tibble 3.3.1
## ✔ lubridate 1.9.5 ✔ tidyr 1.3.2
## ✔ purrr 1.2.2
## ── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
## ✖ dplyr::filter() masks stats::filter()
## ✖ dplyr::lag() masks stats::lag()
## ℹ Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errorslibrary(tidytext)
library(widyr)
library(igraph)##
## Attaching package: 'igraph'
##
## The following objects are masked from 'package:lubridate':
##
## %--%, union
##
## The following objects are masked from 'package:dplyr':
##
## as_data_frame, groups, union
##
## The following objects are masked from 'package:purrr':
##
## compose, simplify
##
## The following object is masked from 'package:tidyr':
##
## crossing
##
## The following object is masked from 'package:tibble':
##
## as_data_frame
##
## The following objects are masked from 'package:stats':
##
## decompose, spectrum
##
## The following object is masked from 'package:base':
##
## unionlibrary(ggraph)Introduction
For this assignment, I was tasked with performing a word co-occurrence analysis on Youtube Comments related to CoreWeave, the AI cloud infrastructure company that is continuing from my last lab of analyzing public discussion around CoreWeave.
comments <- read.csv("coreweave_comments_raw.csv", stringsAsFactors = FALSE)
nrow(comments)## [1] 119Word Co-occurrence Analysis
Step 1: Load and Tokenize the Comments
coreweave_text <- comments %>%
select(id, textOriginal) %>%
mutate(comment_id = row_number())
coreweave_words <- coreweave_text %>%
unnest_tokens(word, textOriginal) %>%
anti_join(stop_words, by = "word")
head(coreweave_words, 10)## id comment_id word
## 1 Ugz67PNhHDTXH82twz14AaABAg 1 catch
## 2 Ugz67PNhHDTXH82twz14AaABAg 1 episode
## 3 Ugz67PNhHDTXH82twz14AaABAg 1 technology
## 4 Ugz67PNhHDTXH82twz14AaABAg 1 podcast
## 5 Ugz67PNhHDTXH82twz14AaABAg 1 spotify
## 6 Ugz67PNhHDTXH82twz14AaABAg 1 https
## 7 Ugz67PNhHDTXH82twz14AaABAg 1 spoti
## 8 Ugz67PNhHDTXH82twz14AaABAg 1 fi
## 9 Ugz67PNhHDTXH82twz14AaABAg 1 32azgzx
## 10 Ugz67PNhHDTXH82twz14AaABAg 1 appleStep 2: Count Word Pairs Within Each Comment
coreweave_pairs <- coreweave_words %>%
pairwise_count(word, comment_id, sort = TRUE, upper = FALSE)
head(coreweave_pairs, 15)## # A tibble: 15 × 3
## item1 item2 n
## <chr> <chr> <dbl>
## 1 interview company 6
## 2 coreweave ai 6
## 3 interview alex 6
## 4 ai cloud 6
## 5 coreweave company 5
## 6 michael coreweave 4
## 7 interview coreweave 4
## 8 interview questions 4
## 9 company ai 4
## 10 ai data 4
## 11 data center 4
## 12 interview excellent 4
## 13 secret sauce 4
## 14 michael intrator 4
## 15 hedge fund 4Step 3: Identify Top Words and Build the Co-occurrence Matrix
top_words <- coreweave_words %>%
count(word, sort = TRUE) %>%
slice_max(n, n = 15) %>%
pull(word)
top_words## [1] "interview" "ai" "coreweave" "providers" "company" "money"
## [7] "alex" "guys" "cloud" "data" "questions" "they’re"
## [13] "1" "gpus" "tier"coreweave_matrix <- coreweave_pairs %>%
filter(item1 %in% top_words, item2 %in% top_words) %>%
bind_rows(coreweave_pairs %>% rename(item1 = item2, item2 = item1) %>%
filter(item1 %in% top_words, item2 %in% top_words)) %>%
cast_sparse(item1, item2, n) %>%
as.matrix()
coreweave_matrix## company ai alex cloud coreweave questions data 1 guys money gpus tier
## interview 6 2 6 0 4 4 1 2 2 0 3 0
## coreweave 5 6 0 3 0 2 3 4 2 3 2 2
## ai 4 0 2 6 6 1 4 3 1 2 1 2
## company 0 4 1 2 5 2 1 0 0 2 1 0
## questions 2 1 3 0 2 0 0 1 3 0 0 0
## tier 0 2 0 2 2 0 0 0 0 0 0 0
## providers 0 2 0 2 2 0 0 0 0 0 0 2
## guys 0 1 1 0 2 3 0 2 0 1 0 0
## cloud 2 6 0 0 3 0 1 2 0 1 1 2
## data 1 4 1 1 3 0 0 1 0 0 1 0
## alex 1 2 0 0 0 3 1 0 1 0 1 0
## money 2 2 0 1 3 0 0 1 1 0 1 0
## they’re 0 0 1 0 1 0 1 0 0 0 1 0
## 1 0 3 0 2 4 1 1 0 2 1 0 0
## gpus 1 1 1 1 2 0 1 0 0 1 0 0
## providers they’re interview
## interview 0 1 0
## coreweave 2 1 4
## ai 2 0 2
## company 0 0 6
## questions 0 0 4
## tier 2 0 0
## providers 0 0 0
## guys 0 0 2
## cloud 2 0 0
## data 0 1 1
## alex 0 1 6
## money 0 0 0
## they’re 0 0 1
## 1 0 0 2
## gpus 0 1 3Step 4: Visualize the Co-occurrence Network
set.seed(580)
coreweave_pairs %>%
filter(item1 %in% top_words, item2 %in% top_words, n >= 2) %>%
graph_from_data_frame() %>%
ggraph(layout = "fr") +
geom_edge_link(aes(edge_alpha = n, edge_width = n), color = "steelblue") +
geom_node_point(size = 5, color = "darkorange") +
geom_node_text(aes(label = name), repel = TRUE, size = 4) +
theme_void() +
labs(title = "Word Co-occurrence Network: CoreWeave YouTube Comments")## Warning: The `trans` argument of `continuous_scale()` is deprecated as of ggplot2 3.5.0.
## ℹ Please use the `transform` argument instead.
## This warning is displayed once per session.
## Call `lifecycle::last_lifecycle_warnings()` to see where this warning was
## generated.
### Custom Stopwords Task
coreweave_words %>%
count(word, sort = TRUE) %>%
head(20)## word n
## 1 interview 32
## 2 ai 20
## 3 coreweave 18
## 4 providers 14
## 5 company 12
## 6 money 11
## 7 alex 10
## 8 guys 10
## 9 cloud 9
## 10 data 9
## 11 questions 9
## 12 they’re 9
## 13 1 8
## 14 gpus 8
## 15 tier 8
## 16 bubble 7
## 17 build 7
## 18 chips 7
## 19 gpu 7
## 20 michael 7custom_stop_words <- bind_rows(
stop_words,
tibble(word = c("they’re", "1"), lexicon = "custom")
)coreweave_words_clean <- coreweave_text %>%
unnest_tokens(word, textOriginal) %>%
anti_join(custom_stop_words, by = "word")
coreweave_pairs_clean <- coreweave_words_clean %>%
pairwise_count(word, comment_id, sort = TRUE, upper = FALSE)
top_words_clean <- coreweave_words_clean %>%
count(word, sort = TRUE) %>%
slice_max(n, n = 15) %>%
pull(word)
top_words_clean## [1] "interview" "ai" "coreweave" "providers" "company" "money"
## [7] "alex" "guys" "cloud" "data" "questions" "gpus"
## [13] "tier" "bubble" "build" "chips" "gpu" "michael"coreweave_matrix_clean <- coreweave_pairs_clean %>%
filter(item1 %in% top_words_clean, item2 %in% top_words_clean) %>%
bind_rows(coreweave_pairs_clean %>% rename(item1 = item2, item2 = item1) %>%
filter(item1 %in% top_words_clean, item2 %in% top_words_clean)) %>%
cast_sparse(item1, item2, n) %>%
as.matrix()
set.seed(580)
coreweave_pairs_clean %>%
filter(item1 %in% top_words_clean, item2 %in% top_words_clean, n >= 2) %>%
graph_from_data_frame() %>%
ggraph(layout = "fr") +
geom_edge_link(aes(edge_alpha = n, edge_width = n), color = "firebrick") +
geom_node_point(size = 5, color = "steelblue") +
geom_node_text(aes(label = name), repel = TRUE, size = 4) +
theme_void() +
labs(title = "Word Co-occurrence Network: CoreWeave Comments (Custom Stopwords Removed)")
Question 1: Which additional words did you add to the stopword list manually? Why does it make sense to remove those words as stopwords? Also, does the final network graph change after removing these additional stopwords? If so, please briefly explain how and why.
I added “they’re” and “1” as custom stopwords. “They’re” is a generic contraction that carries no meaning, and “1” is a stray numeric fragment, not a real word. Removing them did change the graph and once it was gone, “michael” surfaced in the top 15 words for the first time, appearing and also connecting to “ai,” “coreweave,” and “company.”
Reflection Question 2: Adjusting the Co-occurrence Threshold
set.seed(580)
coreweave_pairs_clean %>%
filter(item1 %in% top_words_clean, item2 %in% top_words_clean, n > 2) %>%
graph_from_data_frame() %>%
ggraph(layout = "fr") +
geom_edge_link(aes(edge_alpha = n, edge_width = n), color = "darkgreen") +
geom_node_point(size = 5, color = "orange") +
geom_node_text(aes(label = name), repel = TRUE, size = 4) +
theme_void() +
labs(title = "Word Co-occurrence Network: n > 2 Threshold")
Question 2: Change “n” in the function, “filter(n > 1)”, in step 5 to a different value. What is your final value of n? Will it give you a better result?
I changed the threshold from n > 1 to n > 2. This produced a better and cleaner, more focused graph by taking out weaker connections like “providers” and “tier,” which only connected with other words twice.I think n > 2 gives a better result for readability and confidence in the connections shown. i think with a larger dataset, this threshold could be raised furth.
Question 3: On my BlueSky post, I had two co-occurrence network graphs. Which one do you like better? Why?
I prefer the first, denser graph. Even though it does like a lot more and harder to read at a glance, it gives those extra clusters (like “specialty,” “roast,” “craft,” “quality”) that show how many distinct ways people talk about coffee. The second graph, takes the network down to just six words, which is easy to read but loses almost all of the the details. This rteminds me of what what I found in my own analysis: raising the co-occurrence threshold makes a graph cleaner, but at the cost of real information.
Question 4: Which word pairs surprised you?
One pair that surprised me was “bubble” appearing at all in my top 15 word list, connected to “cloud” and “ai.” Given that most of the CoreWeave comments I analyzed centered on the interview format and CoreWeave’s business fundamentals (words like “money,” “data,” “providers,” “gpus”), This shows that there’s an undercurrent of broader AI industry skepticism mixed with the company specific discussion.
Question 5: Should we treat stock tickers, $SPCX and SpaceX, as the same token instead of separate ones?
My dataset didn’t include stock tickers, but I ran into a similar tokenization issue with “gpu” and “gpus,” which both appeared separately in my top 15 word list I think the right answer depends on the analysis goal. If we are just looking at to measure raw frequency like “$SPCX” and “SpaceX” (or “gpu” and “gpus”) makes sense. However, if the distinction itself is meaningful, keeping them separate preserves that.. For me, I would lean toward merging singulars like “gpu”/“gpus” using stemming.
Question 6: What’s the risk of drawing conclusions from a co-occurrence network built on only a small sample of comments? What is a good sample size? How would you communicate that limitation in a business report?
My analysis was based on only 119 YouTube comments, which is a small sample by most standards. The main issue is that a handful of vocal or repetitive commenters can dominate the word pair counts. A small sample is also more sensitive to noise. In a business report, I would communicate this limitation directly by stating the exact sample size, noting that the results represent a snapshot rather than a definitive pattern, and recommending that any decisions based on the findings be validated against a larger sample before being treated as conclusive.
Question 7: Is there a way to analyze comments or reviews on topics that interest you in relation to a significant event? If so, how would you identify the event? Why is that event significant, and how might it influence the comments or reviews you analyze?
Yes, I could apply this same word co occurrence approach to analyze comments or reviews around a thins liek an earnings announcements, product launch, or a major industry event but I would look for a clear, dateable trigger that caused a spike in public discussion volume or sentiment shift. That event is significant because it creates a natural before and after comparison point. For example, applying this to CoreWeave specifically, I could compare word networks from comments posted before and after a major funding announcement or a public statement from leadership.
References
Bose, R., Dey, R. K., Roy, S., & Sarddar, D. (2019). Sentiment analysis on online product reviews. In Information and Communication Technology for Sustainable Development: Proceedings of ICT4SD 2018 (pp. 559–569). Springer.
Displayr. (2024, November). Learn text analytics in R: A step-by-step guide. https://www.displayr.com/text-analytics-in-r/
Silge, J., & Robinson, D. (2017). Text mining with R: A tidy approach. O’Reilly Media. https://www.tidytextmining.com/
Xu, J. Z. (2026). Text analysis & NLP - Word co-occurrence analysis demo 1 [Course tutorial]. RPubs. https://rpubs.com/JimmyXu/1445260
Xu, Z. (2020). Understanding changes in a brand’s core positioning and customer engagement: A sentiment analysis of a brand-owned Facebook site. [Citation details as provided in course syllabus