Text Data
Text can also be visualized, but unlike numeric variables, text usually needs some extra preparation before plotting. Words appear in different forms, some words are very common but not very informative, and some meaningful ideas are expressed as phrases rather than as single words.
In this final session, you will learn a simple workflow for visualizing text data. We will begin with word frequencies, and then compare different graphical options:
- word clouds
- ranked bar charts
- lollipop charts
- treemaps
- topic models
- co-occurrence networks
The goal is not only to produce attractive plots, but also to understand what each plot is good for.
1. Get the text data
Let us begin with a data frame containing some old tweets from D.Trump:
rm(list = ls())
link1 = "https://github.com/MagallanesAtAlacip/datafiles/"
link2 = "raw/main/text/trumps.csv"
trumpLink = paste0(link1, link2)
allTweets = read.csv(trumpLink, stringsAsFactors = FALSE)The data has these columns:
names(allTweets)## [1] "created_at" "text" "is_retweet" "favorite_count"
## [5] "retweet_count" "Hour" "Day" "Date"The column with the tweet text contains entries like these:
head(allTweets$text, 2)## [1] ".@DonYoungAK really produces for Alaska. He is an incredible Congressman who loves his State and works tirelessly for it. Strong on Public Lands, Energy, and our Second Amendment. He will never let you down, and has my Complete and Total Endorsement! https://t.co/cUR3zuUfs6"
## [2] "...She will be a great Senator, and has my Complete and Total Endorsement! https://t.co/7yHPhxqQVS"2. Select the observations of interest
This data frame has several columns that allow subsetting. In this example, I will keep only tweets that are not retweets:
SomeTrumpTweets = allTweets[allTweets$is_retweet == FALSE, ]
SomeTrumpTweets = SomeTrumpTweets[, c("created_at", "text")]
head(SomeTrumpTweets, 3)## created_at
## 1 2020-08-13 23:26:50
## 2 2020-08-13 23:23:26
## 3 2020-08-13 23:23:25
## text
## 1 .@DonYoungAK really produces for Alaska. He is an incredible Congressman who loves his State and works tirelessly for it. Strong on Public Lands, Energy, and our Second Amendment. He will never let you down, and has my Complete and Total Endorsement! https://t.co/cUR3zuUfs6
## 2 ...She will be a great Senator, and has my Complete and Total Endorsement! https://t.co/7yHPhxqQVS
## 3 .@CynthiaMLummis is a friend of mine and a great woman. She is running for Senate in the very Special State of Wyoming. Cynthia is Strongly for our Military, our Vets, and protection of the Second Amendment...At this point, we still have complete text messages. To visualize text, we first need to break each message into smaller units.
3. Turn text into plottable words
This process starts by what is called tokenization: splitting phrases into individual words.
library(dplyr)
library(tidytext)
SomeTrumpTweets |>
unnest_tokens(
output = EachWord,
input = text,
token = "words"
) |>head()## created_at EachWord
## 1 2020-08-13 23:26:50 donyoungak
## 2 2020-08-13 23:26:50 really
## 3 2020-08-13 23:26:50 produces
## 4 2020-08-13 23:26:50 for
## 5 2020-08-13 23:26:50 alaska
## 6 2020-08-13 23:26:50 heNotice that we named a new column EachWord, and in fact
it has just one word from the tweet. The tweet message has been splitted
or expanded into multiple rows, one row per word. It is useful to keep
track of which tweet each word came from, like a long format ID.
Currently, the created_at might work, but a simpler
solution is to create a sequential ID:
SomeTrumpTweets|>
mutate(tweet_id = row_number())|>
unnest_tokens(
output = EachWord,
input = text,
token = "words"
) |>head()## created_at tweet_id EachWord
## 1 2020-08-13 23:26:50 1 donyoungak
## 2 2020-08-13 23:26:50 1 really
## 3 2020-08-13 23:26:50 1 produces
## 4 2020-08-13 23:26:50 1 for
## 5 2020-08-13 23:26:50 1 alaska
## 6 2020-08-13 23:26:50 1 heLet’s save that object:
WordsIn_SomeTrumpTweets = SomeTrumpTweets|>
mutate(tweet_id = row_number())|>
unnest_tokens(
output = EachWord,
input = text,
token = "words"
)Notice that the date is repeated for every word coming from the same tweet.
How many word-tokens do we have now?
nrow(WordsIn_SomeTrumpTweets)## [1] 3028This number includes repeated words, which is useful because frequency is often our first summary of text. We still have some pending pre processing.
3.1. Remove common words: stopwords
Many very frequent words carry little substantive meaning. Words such as the, and, of, or to are useful for grammar, but not always useful for analysis. These are called stopwords.
data(stop_words)
head(stop_words)## # A tibble: 6 × 2
## word lexicon
## <chr> <chr>
## 1 a SMART
## 2 a's SMART
## 3 able SMART
## 4 about SMART
## 5 above SMART
## 6 according SMARTNow let us remove them:
WordsIn_SomeTrumpTweets = WordsIn_SomeTrumpTweets |>
anti_join(stop_words, by = c("EachWord" = "word"))
nrow(WordsIn_SomeTrumpTweets)## [1] 1496After removing stopwords, the number of rows decreases because many very common words are filtered out.
This does not mean the remaining words are automatically important. It only means we are removing words that are usually too general to help interpretation.
3.2. Getting rid of other unneeded words
Take a look at the most repeated words (tokens):
sort(table(WordsIn_SomeTrumpTweets$EachWord))|>tail(10)##
## complete news support infrastructure service
## 12 12 13 14 14
## usdot people bus https t.co
## 17 18 20 54 54Then we can safely drop some of those:
badWords = c("https", "t.co")
WordsIn_SomeTrumpTweets_clean = WordsIn_SomeTrumpTweets |>
filter(!EachWord %in% badWords) Now, let’s keep the ones that appeared more than 4 times:
moreThan_4 = WordsIn_SomeTrumpTweets_clean|>
count(EachWord, name = "Counts", sort = TRUE) |>
filter(Counts > 4)
head(moreThan_4, 10)## EachWord Counts
## 1 bus 20
## 2 people 18
## 3 usdot 17
## 4 infrastructure 14
## 5 service 14
## 6 support 13
## 7 complete 12
## 8 news 12
## 9 endorsement 11
## 10 federal 11These are the words that appear most often after basic cleaning.
Before moving to more decorative plots, it is useful to begin with the clearest graph possible.
4. Options for plots
4.1. Ranked bar chart
A ranked bar chart is usually the best option when the goal is comparison.
library(ggplot2)
top_words = moreThan_4 |>
slice_head(n = 15)
ggplot(top_words,
aes(x = reorder(EachWord, Counts), y = Counts)) +
geom_col() +
coord_flip() +
labs(
x = "",
y = "Count",
title = "Most frequent words"
) +
theme_minimal(base_size = 14)
This plot is very useful because it allows precise comparison. You can clearly see:
- which words are most frequent
- their ranking
- the size of the differences between them
For comparing frequencies, bar charts are usually better than word clouds.
4.2. Lollipop chart
A lollipop chart shows the same information as a bar chart, but with a lighter appearance.
ggplot(top_words,
aes(x = reorder(EachWord, Counts), y = Counts)) +
geom_segment(aes(xend = EachWord, y = 0, yend = Counts)) +
geom_point(size = 3) +
coord_flip() +
labs(
x = "",
y = "Count",
title = "Most frequent words: lollipop chart"
) +
theme_minimal(base_size = 14)
This chart still prioritizes ranking and comparison, but it uses less visual weight than bars.
4.1. Word cloud
Now let us build the word cloud.
library(ggwordcloud)
ggplot(moreThan_4,
aes(label = EachWord,
size = Counts,
color = Counts)) +
geom_text_wordcloud_area(eccentricity = 0.7,
rm_outside = TRUE) +
scale_size_area(max_size = 18) +
scale_color_gradient(low = "red", high = "darkred") +
theme_minimal(base_size = 16) +
labs(title = "Word cloud of frequent terms")
Word clouds are often visually attractive, but they are not ideal for accurate comparison. They are better for quick engagement and general impression than for careful measurement.
4.2. Treemap
A treemap is another way to represent frequencies, this time using area of rectangles.
Install “treemapify” using this code:
# install.packages("treemapify", type = "binary")library(treemapify)
ggplot(moreThan_4 |>
slice_head(n = 20),
aes(area = Counts, fill = Counts, label = EachWord)) +
geom_treemap() +
geom_treemap_text(place = "centre", colour = "white", reflow = TRUE) +
labs(title = "Word frequencies as a treemap") +
theme_minimal(base_size = 14)
Treemaps are compact and visually appealing, but they are not as precise as bars for comparing values.
5. A second example: reading from a TXT file
Not all text data comes from tweets. Sometimes you simply have a text file.
Suppose we have this file:
We can read it like this:
LinkText = "https://github.com/MagallanesAtAlacip/datafiles/raw/main/text/sometext.txt"
otherText = read.delim(LinkText, header = FALSE)
head(otherText)## V1
## 1 Seattle is under siege. Over the past five years, the Emerald City has seen an explosion of homelessness, crime, and addiction. In its 2017 point-in-time count of the homeless, King County social-services agency All Home found 11,643 people sleeping in tents, cars, and emergency shelters. Property crime has risen to a rate two and a half times higher than Los Angeles’s and four times higher than New York City’s. Cleanup crews pick up tens of thousands of dirty needles from city streets and parks every year.
## 2 At the same time, according to the Puget Sound Business Journal, the Seattle metro area spends more than $1 billion fighting homelessness every year. That’s nearly $100,000 for every homeless man, woman, and child in King County, yet the crisis seems only to have deepened, with more addiction, more crime, and more tent encampments in residential neighborhoods. By any measure, the city’s efforts are not working.
## 3 Over the past year, I’ve spent time at city council meetings, political rallies, homeless encampments, and rehabilitation facilities, trying to understand how the government can spend so much money with so little effect. While most of the debate has focused on tactical policy questions (Build more shelters? Open supervised injection sites?), the real battle isn’t being waged in the tents, under the bridges, or in the corridors of City Hall but in the realm of ideas, where, for now, four ideological power centers frame Seattle’s homelessness debate. I’ll identify them as the socialists, the compassion brigades, the homeless-industrial complex, and the addiction evangelists. Together, they have dominated the local policy discussion, diverted hundreds of millions of dollars toward favored projects, and converted many well-intentioned voters to the politics of unlimited compassion. If we want to break through the failed status quo on homelessness in places like Seattle—and in Portland, San Francisco, and Los Angeles, too—we must first map the ideological battlefield, identify the flaws in our current policies, and rethink our assumptions.
## 4 Seattle has long been known as one of America’s most liberal locales, but in recent years, the city has marched even further left as socialists, once relegated to the margins, have declared war on the Democratic establishment. Socialist Alternative city councilwoman Kshama Sawant claims that the city’s homelessness crisis is the inevitable result of the Amazon boom, greedy landlords, and rapidly increasing rents. As she told Street Roots News: “The explosion of the homelessness crisis is a symptom of how deeply dysfunctional capitalism is and also how much worse living standards have gotten with the last several decades.” The capitalists of Amazon, Starbucks, Microsoft, and Boeing, in her Marxian optic, generate enormous wealth for themselves, drive up housing prices, and push the working class toward poverty and despair—and, too often, onto the streets.
## 5 On the surface, this argument has its own internal logic. Advocates point to Zillow and McKinsey studies that show a high correlation between rent hikes and homelessness in Seattle, for example. But correlation is not causation, and the survey data paint a remarkably different picture. According to King County’s point-in-time study, only 6 percent of homeless people surveyed cited “could not afford rent increase” as the precipitating cause of their situation, pointing instead to a wide range of other problems—domestic violence, incarceration, mental illness, family conflict, medical conditions, breakups, eviction, addiction, and job loss—as bigger factors. Further, while the Zillow study did find correlation between rising rents and homelessness in four major markets—Seattle, Los Angeles, New York, and Washington, D.C.—it also found that homelessness decreased despite rising rents in Houston, Tampa, Chicago, Phoenix, St. Louis, San Diego, Portland, Detroit, Baltimore, Atlanta, Charlotte, and Riverside. Rent increases are a real burden for the working poor, but the evidence suggests that higher rents alone don’t push people onto the streets.
## 6 Even in a pricey city like Seattle, most working- and middle-class residents respond to economic incentives in logical ways: relocating to less expensive neighborhoods, downsizing to smaller apartments, taking in roommates, moving in with family, or leaving the city altogether. King County is home to more than 1 million residents earning below the median income, and 99 percent of them manage to find a place to live and pay the rent on time. The aggregate-level analyses from Zillow and McKinsey don’t take into account the vast number of options available even to the poorest families; for the socialists, “the rent is too damn high” explains everything.Notice that a new row is created whenever a new paragraph or line break is found.
Now let us tokenize the text and remove stopwords and the above discovered badWords:
otherText_words = otherText |>
unnest_tokens(output = EachWord,
input = V1,
token = "words") |>
anti_join(stop_words, by = c("EachWord" = "word")) |>
filter(!EachWord %in% badWords)
head(otherText_words, 20)## EachWord
## 1 seattle
## 2 siege
## 3 past
## 4 emerald
## 5 city
## 6 explosion
## 7 homelessness
## 8 crime
## 9 addiction
## 10 2017
## 11 time
## 12 count
## 13 homeless
## 14 king
## 15 county
## 16 social
## 17 services
## 18 agency
## 19 home
## 20 foundNow compute frequencies:
txt_counts = otherText_words |>
count(EachWord, name = "Counts", sort = TRUE) |>
filter(Counts > 4) # filtered by count
head(txt_counts, 10)## EachWord Counts
## 1 homeless 47
## 2 homelessness 46
## 3 city 40
## 4 seattle 37
## 5 addiction 21
## 6 percent 20
## 7 people 18
## 8 public 18
## 9 services 18
## 10 housing 166. Other Plots beyond raw counts
Once you have word counts or frequencies, you can prepare your word cloud:
ggplot(txt_counts,
aes(label = EachWord,
size = Counts,
color = Counts)) +
geom_text_wordcloud_area(rm_outside = TRUE) +
scale_size_area(max_size = 18) +
scale_color_gradient(low = "red", high = "darkred") +
theme_minimal(base_size = 16) +
labs(title = "Word cloud from the TXT file")
And you ranked bars:
txt_top15 = txt_counts |>
slice_head(n = 15)
ggplot(txt_top15,
aes(x = reorder(EachWord, Counts), y = Counts)) +
geom_col() +
coord_flip() +
labs(
x = "",
y = "Count",
title = "Most frequent words in the text"
) +
theme_minimal(base_size = 14)
You may also use proportions instead of raw values as labels (counts still are represented on Y axis):
ggplot(txt_top15,
aes(x = reorder(EachWord, Counts), y = Counts)) +
geom_col() +
coord_flip() +
geom_text(
aes(label = paste0(round(100 * Counts / sum(Counts), 1), "%")),
nudge_y = 2
) +
labs(
x = "",
y = "Count",
title = "Most frequent words in the text"
) +
theme_minimal(base_size = 14)
However, other strategies beyond plotting frequencies are possible.
6.1 Co-occurrence networks
So far, we have treated words as isolated units. But language also has structure: some words tend to appear together.
A simple way to explore this is through bigrams,
that is, pairs of words that appear next to each other. The
unnest_tokens() is used again, but
requesting consecutive pairs:
bigrams = SomeTrumpTweets |>
unnest_tokens(bigram, text, token = "ngrams", n = 2)
## see some
bigrams |>head()## created_at bigram
## 1 2020-08-13 23:26:50 donyoungak really
## 2 2020-08-13 23:26:50 really produces
## 3 2020-08-13 23:26:50 produces for
## 4 2020-08-13 23:26:50 for alaska
## 5 2020-08-13 23:26:50 alaska he
## 6 2020-08-13 23:26:50 he isThe output above is not ready yet, we need to split the pairs:
library(tidyr)
bigrams_inColumns = bigrams |>
separate_wider_delim(bigram,
names = c("word1", "word2"),
delim = " ")
bigrams_inColumns## # A tibble: 2,907 × 3
## created_at word1 word2
## <chr> <chr> <chr>
## 1 2020-08-13 23:26:50 donyoungak really
## 2 2020-08-13 23:26:50 really produces
## 3 2020-08-13 23:26:50 produces for
## 4 2020-08-13 23:26:50 for alaska
## 5 2020-08-13 23:26:50 alaska he
## 6 2020-08-13 23:26:50 he is
## 7 2020-08-13 23:26:50 is an
## 8 2020-08-13 23:26:50 an incredible
## 9 2020-08-13 23:26:50 incredible congressman
## 10 2020-08-13 23:26:50 congressman who
## # ℹ 2,897 more rowsSome cleaning and sorted counts:
bigram_counts_clean = bigrams_inColumns|>
filter(!word1 %in% stop_words$word,
!word2 %in% stop_words$word) |>
filter(!word1 %in% badWords,
!word2 %in% badWords) |>
count(word1, word2, sort = TRUE) |>
filter(n >= 5)
# see top
bigram_counts_clean |> head(10)## # A tibble: 7 × 3
## word1 word2 n
## <chr> <chr> <int>
## 1 bus service 12
## 2 total endorsement 10
## 3 federal infrastructure 8
## 4 infrastructure funds 8
## 5 support bus 8
## 6 joe biden 6
## 7 radical left 56.1.1 The creation of networks
As we have pairs, we can think of each word as a network node. If word are nodes, a link between them means they co occur. Notice this network model allows that a word may be connected to more than one other word.
We need igraph here
library(igraph)
bigram_graph = graph_from_data_frame(bigram_counts_clean,directed = F)
bigram_graph## IGRAPH 9e2dd9b UN-- 12 7 --
## + attr: name (v/c), n (e/n)
## + edges from 9e2dd9b (vertex names):
## [1] bus --service total --endorsement
## [3] federal --infrastructure infrastructure--funds
## [5] bus --support joe --biden
## [7] radical --leftAbove you see new info. It is telling you the object
bigram_graph is an IGRAPH object; it is an
Undirected network whose nodes or vertices have
Names (UN); its has 12 nodes and 7 vertices.
The fact tat it is Undirected means we do not care
which word is before the other (that will be a directed network), we
only care about co-occurrence.
The bigram_graph also has some attributes. It has
name and n. The former is an
vertex (or node) attribute, and it is a
character, take a look:
V(bigram_graph)$name## [1] "bus" "total" "federal" "infrastructure"
## [5] "support" "joe" "radical" "service"
## [9] "endorsement" "funds" "biden" "left"The latter (n) is an edge attribute
whose type is numeric:
E(bigram_graph)$n## [1] 12 10 8 8 8 6 5This came from our counts column in
bigram_counts_clean.
You can easily produce a bigram plot like this:
library(ggraph)
ggraph::ggraph(bigram_graph ,layout="fr") +
geom_edge_link() +
geom_node_point() 
The hardest part of drawing networks is deciding where to place nodes. The fr layout helps by simulating attraction between connected nodes and repulsion among all nodes. This usually spreads the graph clearly while bringing related nodes together, making clusters and central nodes easier to see.
We can improve this by adding names to the nodes:
ggraph::ggraph(bigram_graph,layout="fr") +
geom_edge_link() +
geom_node_point() +
geom_node_text(aes(label = name))
We know that those edges have the counts attribute
(n). We can use the line width channel to
present the weight:
ggraph::ggraph(bigram_graph, layout = "fr") +
geom_edge_link(aes(width = n)) +
geom_node_point() +
geom_node_text(aes(label = name))
Let’s add some details for improvement:
ggraph::ggraph(bigram_graph, layout = "fr") + theme_void() +
geom_edge_link(aes(width = n), alpha = 0.4, color='blue') +
geom_node_point(size = 4) +
geom_node_text(aes(label = name), repel = TRUE, size = 5) 
This plot does not focus only on word frequency. Instead, it reveals relationships between words. That is often more meaningful than treating words separately.
This is where we end. I hope you liked the journey!