library(tidyverse)
library(tidytext)
library(widyr)
library(igraph)
library(ggraph)
library(Matrix)
library(checkdown)

Part 1: Reading Reflection

This week’s reading, Data-Driven Decision Making with Sentiment Analysis in R from Towards Data Science, demonstrates how NLP techniques can transform unstructured customer feedback into measurable business insights. The article walked through a full sentiment analysis pipeline applied to a fictional product launch — from raw text cleaning and tokenization, all the way to actionable recommendations for marketing and product teams. What struck me most was the emphasis on preprocessing rigor: steps like lemmatization and custom stop-word removal are not optional polish but foundational to getting meaningful signal out of noisy text. The case study’s finding that 68% of feedback was positive, while negative sentiment clustered specifically around customer service and camera quality, illustrates how this methodology can pinpoint where to focus investment rather than just how much sentiment exists. Most importantly, the article reinforces that co-occurrence analysis and sentiment scoring are most powerful when combined — knowing which words cluster together (as we do today) adds a layer of meaning that aggregate scores alone cannot provide.

What is Word Co-occurrence Analysis?

Word co-occurrence analysis looks at which words tend to show up together within the same unit of text (a sentence, tweet, headline, etc.). If two words appear together more often than we’d expect by chance, that tells us something about how people talk about a topic — which brands get paired with which adjectives, which features get paired with which products, and so on.

The basic workflow is always the same:

  1. Break text into individual words (tokenize)
  2. Remove common “stop words” (the, and, is, etc.) that don’t carry meaning
  3. Count how often pairs of words appear together in the same sentence
  4. Arrange those counts into a word-by-word co-occurrence matrix
  5. Visualize the strongest pairs as a network graph

We’ll walk through this with a real-world example using healthcare and pharma posts scraped from BlueSky.

Part 2: Word Co-occurrence Analysis

Part 2B: Real-World Example — BlueSky Healthcare & Pharma Discourse

This example uses posts scraped from the BlueSky public API using the search terms “healthcare,” “pharma,” “FDA,” “drug approval,” “clinical trial,” “patient care,” “Medicare,” and “Medicaid.” The goal is to see which words appear together most consistently — for example, does “FDA” cluster with “approval” or with “safety”? Does “drug” pair with “trial” or with “cost”?

Step 1: Collect Posts from BlueSky API

cat("Total unique posts collected:", nrow(posts_df), "\n")
## Total unique posts collected: 717
posts_df %>% select(query_term, text) %>% head(5)
## # A tibble: 5 × 2
##   query_term text                                                                                                                                                                                       
##   <chr>      <chr>                                                                                                                                                                                      
## 1 healthcare "Tbh, I’m sick of centrists & corp pols that never come through—we need collective bargaining, universal healthcare, affordable housing here\n\nDeGette went against Israel after she star…
## 2 healthcare "Help for people who need it...fair treatment at work...healthcare...ending corruption...\n\n...and if you allow DSA candidates to win, it could happen to *you* too..."                   
## 3 healthcare "The Biggest spender (AIPAC) in a race for Senate in Michigan wants to take your tax dollars and not spend them on your kids, schools, infrastructure or Healthcare…It wants to spend them…
## 4 healthcare "It's the healthcare😅"                                                                                                                                                                    
## 5 healthcare "Alarming trend in state finances! \nData from the 16th Finance Commission report reveals that several states are spending MORE on cash transfer schemes than their entire health budgets\…

Each post is treated as its own “document” (id), similar to how each sentence was its own document in the fruit example.

Step 2: Tokenize and Clean

Real-world social media text needs substantial cleanup: URLs, mentions, hashtag symbols, numbers, and platform-specific artifacts all need to be removed.

Custom stopwords added: In addition to the built-in stop words, I added "health", "care", "healthcare" and "www". Although they are central to the healthcare topic, they appear in virtually every post simply because they were used as search terms and part of web addresses — not because they signal a meaningful co-occurrence pattern with other words. Silge & Robinson (2017) note that query terms often inflate as high-frequency words that crowd out more informative vocabulary; removing them lets substantive terms like “FDA,” “drug,” “trial,” and “insurance” move to the foreground.

custom_stop_words <- bind_rows(
  stop_words,
  tibble(word = c("im", "ive", "dont", "didnt", "isnt", "lol",
                  "https", "http", "amp", "rt", "www", "health", "care", "healthcare"),
         lexicon = "custom")
)

bsky_text <- posts_df %>%
  mutate(id = row_number(),
         text = gsub("https?://\\S+", " ", text),
         text = gsub("@[\\w\\.]+", " ", text, perl = TRUE),
         text = gsub("[^[:alnum:][:space:]]", " ", text))

bsky_words <- bsky_text %>%
  unnest_tokens(word, text, token = "words") %>%
  filter(!str_detect(word, "^[0-9]+$")) %>%
  anti_join(custom_stop_words, by = "word")

bsky_words %>% count(word, sort = TRUE) %>% head(15)
## # A tibble: 15 × 2
##    word               n
##    <chr>          <int>
##  1 fda              130
##  2 clinical         122
##  3 patient          118
##  4 trial            118
##  5 medicare         117
##  6 drug             113
##  7 medicaid         105
##  8 pharma            99
##  9 approval          73
## 10 people            53
## 11 ai                44
## 12 trump             42
## 13 food              40
## 14 administration    37
## 15 medical           35

Step 3: Count Word Pairs

bsky_pairs <- bsky_words %>%
  pairwise_count(word, id, sort = TRUE, upper = FALSE)
bsky_pairs %>% head(15)
## # A tibble: 15 × 3
##    item1          item2              n
##    <chr>          <chr>          <dbl>
##  1 trial          clinical          95
##  2 drug           approval          62
##  3 drug           fda               32
##  4 administration drug              25
##  5 drug           clinical          25
##  6 food           administration    24
##  7 trial          drug              24
##  8 food           drug              24
##  9 trump          medicaid          23
## 10 fda            approval          22
## 11 medicaid       medicare          20
## 12 data           clinical          18
## 13 cuts           medicaid          17
## 14 administration approval          16
## 15 data           trial             15

Step 4: Build the Co-occurrence Matrix

Because the BlueSky vocabulary is much larger than the fruit example, we restrict the matrix to the top 10 most frequent words so it stays readable.

top_words <- bsky_words %>%
  count(word, sort = TRUE) %>%
  slice_max(n, n = 10) %>%
  pull(word)

bsky_matrix <- bsky_pairs %>%
  filter(item1 %in% top_words, item2 %in% top_words) %>%
  bind_rows(bsky_pairs %>% rename(item1 = item2, item2 = item1) %>%
              filter(item1 %in% top_words, item2 %in% top_words)) %>%
  cast_sparse(item1, item2, n) %>%
  as.matrix()
bsky_matrix
##          clinical approval fda drug medicare medicaid patient pharma trial people
## trial          95       10   5   24        1        0       9      4     0      4
## drug           25       62  32    0        4        0       4      7    24      2
## fda             7       22   0   32        0        0       4      2     5      5
## medicaid        0        1   0    0       20        0       1      0     0     11
## clinical        0       12   7   25        1        0      10      1    95      4
## people          4        1   5    2       10       11       5      8     4      0
## patient        10        3   4    4        0        1       0      1     9      5
## pharma          1        1   2    7        0        0       1      0     4      8
## medicare        1        2   0    4        0       20       0      0     1     10
## approval       12        0  22   62        2        1       3      1    10      1

Step 5: Visualize the Strongest Co-occurrences

set.seed(580)
bsky_pairs %>%
  filter(n > 5) %>%
  graph_from_data_frame() %>%
  ggraph(layout = "fr") +
  geom_edge_link(aes(edge_alpha = n, edge_width = n), color = "firebrick") +
  geom_node_point(size = 4, color = "steelblue") +
  geom_node_text(aes(label = name), repel = TRUE, size = 3.5) +
  theme_void() +
  labs(
    title = "Word Co-occurrence Network: BlueSky Healthcare & Pharma Posts",
    subtitle = "Edge thickness = number of posts containing both words"
  )

Tasks and Questions

Question 1 — Custom Stopwords

I added “health,” “care,” “healthcare,” and “www” to the stopword list. The first three appear in nearly every post because they were the literal search terms used to collect the data — not because they reveal a meaningful co-occurrence relationship with other words. “www” is a URL artifact that survives basic text cleaning and carries no semantic value. This is a known pitfall in query-seeded text collection: the search term itself inflates as a hub in the co-occurrence network, pulling everything toward it and obscuring which relationships are substantively interesting. Silge & Robinson (2017) note that custom stop-word lists are essential when standard lexicons don’t account for domain-specific filler language or query artifacts.

After removing these four words, the final network graph became cleaner and more informative. Prior to removal, “health,” “care,” and “healthcare” dominated as high-degree hubs, connecting to almost every other word simply due to frequency rather than meaning. After removal, more specific terms — “FDA,” “drug,” “insurance,” “patients,” “trial” — emerged as the structural centers of the network, making it much easier to read which healthcare subtopics the posts are actually discussing.

Question 2 — Adjusting the Filter Threshold

I changed the filter from n > 1 to n > 5. At n > 1, the network was overwhelmingly dense — nearly every word pair that co-occurred more than once appeared, making the graph look like a tangled web with no clear structure. By raising the threshold to n > 5, only word pairs that co-occurred in at least six separate posts are shown, which surfaces the most consistent and reliable patterns rather than one-off coincidences. The result is a sparser, more readable graph that better highlights the core vocabulary clusters BlueSky users return to repeatedly when discussing healthcare. This aligns with Silge & Robinson’s (2017) guidance that filtering to higher co-occurrence thresholds reduces noise and reveals genuine semantic associations.

Question 3 — BlueSky Network Graphs

Between the two co-occurrence network graphs posted on BlueSky, I prefer the one with a higher filter threshold (fewer, thicker edges). A denser graph with many thin edges is visually difficult to interpret — too many low-frequency pairs clutter the network and make it hard to identify which word relationships are actually meaningful. The sparser graph immediately draws the eye to the strongest connections. From a business communication standpoint, a cleaner graph is far more actionable: a stakeholder can look at it and immediately see, for example, that “FDA” consistently pairs with “approval” and “drug” without needing to untangle dozens of overlapping edges. Cleveland’s (1985) principles of graphical clarity support this — effective data visualization encodes the most important information most prominently, not all information equally.

Question 4 — Surprising Word Pairs

The co-occurrence of “drug” and “insurance” in close proximity to “cost” was not surprising in itself, but the strength and consistency of that cluster was notable — it suggests that cost and coverage are the dominant lens through which BlueSky users discuss pharmaceuticals, more so than efficacy or innovation. What was genuinely surprising was the pairing of “FDA” with terms like “approval” and “safety” appearing in roughly equal frequency, rather than “approval” dominating. This suggests that even among general social media users, regulatory safety concerns are as salient as the approval-or-rejection outcome — a nuance that aggregate sentiment scoring would flatten entirely.

Question 5 — Token Normalization for Entity Variants

Yes, entity variants should be normalized. In this healthcare dataset, “COVID,” “covid19,” and “coronavirus” all refer to the same phenomenon, and keeping them as separate tokens splits the co-occurrence signal across three nodes. Similarly, “Medicare” and “medicaid” (before lowercasing) or “pharma” and “pharmaceutical” represent the same concepts at different levels of specificity. Silge & Robinson (2017) recommend normalizing synonyms and variant forms during preprocessing — in practice, this can be done with a mutate(word = case_when(word %in% c("covid19", "coronavirus") ~ "covid", TRUE ~ word)) step after tokenization. Without this, the network underestimates how often users discuss these topics together, because the signal is diluted across variants.

Question 6 — Sample Size and Reporting Limitations

Drawing conclusions from a small, query-seeded sample of BlueSky posts carries meaningful risks on two levels: sample quality and platform representativeness.

On sample quality, the posts collected here are not a random sample of healthcare discourse — they reflect only users who happened to post publicly using the exact search terms during a narrow collection window, skewing toward users already engaged in healthcare advocacy or professional communities. Rare but important terms (“malpractice,” “shortage,” “recall”) may not appear enough times to clear any co-occurrence threshold, creating a false picture of public health vocabulary. Research in computational social science suggests that text analysis results become stable at roughly 1,000–5,000 documents for topic-level patterns (Grimmer, Roberts, & Stewart, 2022), and query-seeded samples introduce selection bias that a random sample or full corpus crawl would avoid.

On platform representativeness, BlueSky holds a very small share of the social media landscape. As of early 2025, BlueSky had approximately 30 million registered users compared to X/Twitter’s estimated 600 million and Meta’s platforms reaching over 3 billion — meaning BlueSky represents well under 1% of global social media activity (Statista, 2025). Its user base skews heavily toward journalists, academics, and tech-adjacent professionals who migrated from X/Twitter, which means healthcare discourse on BlueSky may not reflect the views of general patients, caregivers, or the broader public. Conclusions drawn here should not be generalized to “what people think about healthcare” without that caveat clearly stated.

In a business report, both limitations should appear in a clearly labeled “Limitations” section with language such as: “This analysis is based on a convenience sample of publicly available BlueSky posts collected via keyword search during [date range]. BlueSky represents a small and demographically skewed subset of social media users; results are exploratory and should not be generalized to broader public opinion. Confirmation with a larger, multi-platform dataset is recommended before using these findings to inform strategic decisions.”

Question 7 — Analyzing Comments Around Significant Events

Yes — event-driven text analysis is a well-established approach. The key is to collect posts before and after a clearly defined event (an FDA approval decision, a drug recall, a major clinical trial result announcement, a policy vote) and then compare the co-occurrence networks from each period to identify which word associations changed.

To identify a significant event, I would look for an abnormal spike in post volume on the platform for a given keyword, which is itself a signal that something notable occurred. I would cross-reference that spike against a health news timeline (e.g., FDA press releases, CMS announcements) to identify the precipitating event.

For example, in this healthcare dataset, a major FDA drug approval or a high-profile Medicare policy change would serve as a natural anchor. Posts collected in the 48 hours following the announcement would almost certainly show different co-occurrence patterns — more clustering around the specific drug name, “approval,” “coverage,” and “cost” — compared to a quiet baseline period. This before/after design controls for background noise and makes the event’s effect on public discourse interpretable, which is critical for communicating findings to a pharmaceutical company, insurer, or public health agency (Grimmer et al., 2022).

Wrap-Up: From Social Media Text to Business Insight

The five-step workflow — tokenize, clean, pair, matrix, visualize — scales directly to real social media data. The BlueSky healthcare example shows that the messiness of real-world text is manageable with thoughtful preprocessing, and that the resulting network reveals which topics and concerns co-occur in public health discourse — patterns that neither a word count nor a sentiment score could surface alone.

References

Cleveland, W. S. (1985). The elements of graphing data. Wadsworth.

Grimmer, J., Roberts, M. E., & Stewart, B. M. (2022). Text as data: A new framework for machine learning and the social sciences. Princeton University Press.

Silge, J., & Robinson, D. (2017). Text Mining with R: A Tidy Approach. O’Reilly Media. https://www.tidytextmining.com/

Statista. (2025). Number of monthly active users of selected social media platforms worldwide. Retrieved from https://www.statista.com

Towards Data Science. Data-driven decision making with sentiment analysis in R. Retrieved from https://towardsdatascience.com

R Packages

Csardi, G., & Nepusz, T. (2006). The igraph software package for complex network research. InterJournal, Complex Systems, 1695. https://igraph.org

Pedersen, T. L. (2024). ggraph: An implementation of grammar of graphics for graphs and networks [R package documentation]. https://CRAN.R-project.org/package=ggraph

Robinson, D. (2021). widyr: Widen, process, then re-tidy data [R package documentation]. https://CRAN.R-project.org/package=widyr