Mining for Tweets Tutorial
Rachel Saidi
October 19, 2020
STEP 1: Load the rtweet and other needed R packages.
Note you are introducing 2 new packages lower in this lesson: igraph and ggraph.
# load twitter library - the rtweet library is recommended now over twitteR
library(rtweet)
# plotting and pipes - tidyverse!
library(ggplot2)
library(dplyr)##
## Attaching package: 'dplyr'## The following objects are masked from 'package:stats':
##
## filter, lag## The following objects are masked from 'package:base':
##
## intersect, setdiff, setequal, union# text mining library
library(tidytext)
# plotting packages
library(igraph)##
## Attaching package: 'igraph'## The following objects are masked from 'package:dplyr':
##
## as_data_frame, groups, union## The following objects are masked from 'package:stats':
##
## decompose, spectrum## The following object is masked from 'package:base':
##
## unionlibrary(ggraph)Create a twitter developer account
In order to download tweets, you will need a twitter account.
Then you will need a developer account. Follow this CRAN tutorial in order to create a developer account: https://cran.r-project.org/web/packages/rtweet/vignettes/auth.html
As an example, my developer account is: MC data mining. You basically need to provide twitter with a reason that you would like this account, and obviously, it is to learn to mine tweets for your data science class. It is typically a fast approval process.
Once you have created a developer app, you are ready to start.
Access Token/Secret Method
Copy and paste the four keys (along with the name of your app) into an R script file and pass them along to create_token().
store api keys (these are fake example values; replace with your own keys)
api_key <- “…” api_secret_key <- “…” access_token <- “…” access_token_secret <- “…”
#api_key <- "..."
#api_secret_key <- "..."
#access_token <- "..."
#access_token_secret <- "..."
api_key <- "onsqqdn6FkeFG2jeFLeODn6UE"
api_secret_key <- "50AZmbOZizq29Hiiy5a3kheDl93Z3ym5lWTxZLywLLGuswvX3b"
access_token <- "432673917-rOQ8W1Wgi1qAH1NeS3BbNByogRlrhq34kzqWgTrd"
access_token_secret <- "OgGb43o5Rwv5nGQvuocbHmkLVH7imPQsIUfzBjxQpxVoK"
## authenticate via web browser
token <- create_token(
app = "MC data minig",
consumer_key = api_key,
consumer_secret = api_secret_key,
access_token = access_token,
access_secret = access_token_secret)Search for Tweets Related to Climate
Use the tutorial on the following website:
In this example, let’s find tweets that are using the words “climate change” in them.
The first time you run the next line of code, it will take you to your twitter page and you have to agree that you are allowing this app to work with R. Be warned that you can only run this one time per chunk of time (if you try to run without waiting long enough, the query will fail)
climate_tweets <- search_tweets(q = "climate change", n = 10000,
lang = "en",
include_rts = FALSE)
# check data to see if there are emojis
head(climate_tweets$text)## [1] "@T_S_P_O_O_K_Y @JoeBiden As an Environmental Prof., I can say climate change is real and has been happening since the dawn of time. The debate is how a man plays a role in the effects and how man impact change. That said, we can spend huge sums of money which would result in zero impact on the climate"
## [2] "BBC News - Why the US election could decide battle against climate change\nhttps://t.co/pgDZp3puq8"
## [3] "@Kashmiriyoungun @DrMMirajkar1 @TwitterIndia Courage is not a climate, to change by place. \nIf you are so much confident about your courage, you should go to other places and challenge. \nYou can sit in your room and can say that you can rule this world. But you can't even rule your neighbors, is reality. <U+0001F602>"
## [4] "Republicans will believe in Bigfoot before climate change"
## [5] "A 5-point plan to addressing climate change and protecting the planet https://t.co/JA3IxxOf4Q"
## [6] "The thing about Kasich that everyone is missing is that we are on the verge of runaway climate change and we need a complete re-think of the global economy to ensure a just society & avoid planetary catastrophe."Look at the results.
Change the code below to do a new query: If you set your query to q=“climate+change” then the API fill find tweets that use the words together in a string rather than across the entire string. Let’s try it.
# Find tweet using climate+change together in them
climate_tweets <- search_tweets(q = "climate+change", n = 10000, lang = "en",
include_rts = FALSE)## Warning: Rate limit exceeded - 88## Warning: Rate limit exceeded# check data to see if there are emojis
head(climate_tweets$text)## [1] "@greentechmedia Thompson will ge hiring nationwide with good paying jobs. Starting pay will be $25 an hour. America do you want a good paying JOB and to help fight climate change? Write my name on the ballot. Morris Kent Thompson for President American People’s Action Party https://t.co/LcjcdxGBb9"
## [2] "@USATODAY Thompson's tax policy will see you keeping your money. Thompson will be hiring nationwide with good paying jobs. Starting wage will be $25 an hour. America, do you want a good job and to help fight climate change? write my name on the ballot.\nMorris Kent Thompson APAP https://t.co/xy6P1jgbdx"
## [3] "@MSNBC @MaddowBlog Thompson will ge hiring nationwide with good paying jobs. Starting pay will be $25 an hour. America do you want a good paying JOB and to help fight climate change? Write my name on the ballot. Morris Kent Thompson for President American People’s Action Party https://t.co/j3gvjusyQY"
## [4] "@MSNBC Thompson will ge hiring nationwide with good paying jobs. Starting pay will be $25 an hour. America do you want a good paying JOB and to help fight climate change? Write my name on the ballot. Morris Kent Thompson for President American People’s Action Party https://t.co/R8byg8P5ij"
## [5] "@CNNPolitics Thompson will ge hiring nationwide with good paying jobs. Starting pay will be $25 an hour. America do you want a good paying JOB and to help fight climate change? Write my name on the ballot. Morris Kent Thompson for President American People’s Action Party https://t.co/bepDC13qrz"
## [6] "@CNN Thompson will ge hiring nationwide with good paying jobs. Starting pay will be $25 an hour. America do you want a good paying JOB and to help fight climate change? Write my name on the ballot. Morris Kent Thompson for President American People’s Action Party https://t.co/OYNA4vj4D4"Data Clean-Up
Looking at the data above, it becomes clear that there is a lot of clean-up associated with social media data.
First, there are url’s in your tweets. If you want to do a text analysis to figure out what words are most common in your tweets, the URL’s won’t be helpful. Let’s remove those.
# remove urls tidyverse is failing here for some reason
# climate_tweets %>%
# mutate_at(c("stripped_text"), gsub("http.*","",.))
# remove http elements manually
climate_tweets$stripped_text <- gsub("http.*","", climate_tweets$text)
climate_tweets$stripped_text <- gsub("https.*","", climate_tweets$stripped_text)Finally, you can clean up your text. If you are trying to create a list of unique words in your tweets, words with capitalization will be different from words that are all lowercase. Also you don’t need punctuation to be returned as a unique word.
# note the words that are recognized as unique by R
a_list_of_words <- c("Dog", "dog", "dog", "cat", "cat", ",")
unique(a_list_of_words)## [1] "Dog" "dog" "cat" ","## [1] "Dog" "dog" "cat" ","You can use the tidytext::unnest_tokens() function in the tidytext package to magically clean up your text! When you use this function the following things will be cleaned up in the text:
Convert text to lowercase: each word found in the text will be converted to lowercase, so ensure that you don’t get duplicate words due to variation in capitalization.
Punctuation is removed: all instances of periods, commas etc will be removed from your list of words , and Unique id associated with the tweet: will be added for each occurrence of the word
The unnest_tokens() function takes two arguments:
- The name of the column where the unique word will be stored and
- The column name from the data.frame that you are using that you want to pull unique words from.
In your case, you want to use the stripped_text column which is where you have your cleaned up tweet text stored.
# remove punctuation, convert to lowercase, add id for each tweet!
climate_tweets_clean <- climate_tweets %>%
dplyr::select(stripped_text) %>%
unnest_tokens(word, stripped_text)Now you can plot your data. What do you notice?
# plot the top 15 words -- notice any issues?
climate_tweets_clean %>%
count(word, sort = TRUE) %>%
top_n(15) %>%
mutate(word = reorder(word, n)) %>%
ggplot(aes(x = word, y = n)) +
geom_col() +
xlab(NULL) +
coord_flip() +
labs(x = "Count",
y = "Unique words",
title = "Count of unique words found in tweets")## Selecting by n
Your plot of unique words contains some words that may not be useful to use, called “stop_words”. For instance “a” and “to”. You want to remove these words from your analysis as they are fillers used to compose a sentence.
Lucky for us, the tidytext package has a function that will help us clean up stop words! To use this you:
Load the stop_words data included with tidytext. This data is simply a list of words that you may want to remove in a natural language analysis. Then you use anti_join to remove all stop words from your analysis. Let’s give this a try next!
# load list of stop words - from the tidytext package
data("stop_words")
# view first 6 words
head(stop_words)## Warning: `...` is not empty.
##
## We detected these problematic arguments:
## * `needs_dots`
##
## These dots only exist to allow future extensions and should be empty.
## Did you misspecify an argument?## # A tibble: 6 x 2
## word lexicon
## <chr> <chr>
## 1 a SMART
## 2 a's SMART
## 3 able SMART
## 4 about SMART
## 5 above SMART
## 6 according SMART## # A tibble: 6 x 2
## word lexicon
## <chr> <chr>
## 1 a SMART
## 2 a's SMART
## 3 able SMART
## 4 about SMART
## 5 above SMART
## 6 according SMART
nrow(climate_tweets_clean)## [1] 237945## [1] 230984 or something similar
# remove stop words from your list of words
cleaned_tweet_words <- climate_tweets_clean %>%
anti_join(stop_words)## Joining, by = "word"# there should be fewer words now
nrow(cleaned_tweet_words)## [1] 122857## [1] 118701 or something similar##Explore Networks of Words
You might also want to explore words that occur together in tweets. Let’s do that next.
ngrams specifies pairs and 2 is the number of words together
# library(devtools)
# install_github("dgrtwo/widyr")
library(widyr)
# remove punctuation, convert to lowercase, add id for each tweet!
climate_tweets_paired_words <- climate_tweets %>%
dplyr::select(stripped_text) %>%
unnest_tokens(paired_words, stripped_text, token = "ngrams", n = 2)
climate_tweets_paired_words %>%
count(paired_words, sort = TRUE)## Warning: `...` is not empty.
##
## We detected these problematic arguments:
## * `needs_dots`
##
## These dots only exist to allow future extensions and should be empty.
## Did you misspecify an argument?## # A tibble: 120,241 x 2
## paired_words n
## <chr> <int>
## 1 climate change 7181
## 2 change is 827
## 3 of the 757
## 4 in the 724
## 5 change and 644
## 6 of climate 555
## 7 on climate 513
## 8 is a 501
## 9 on the 407
## 10 about climate 389
## # ... with 120,231 more rows## # A tibble: 134,656 x 2
## paired_words n
## <chr> <int>
## 1 climate change 1021
## 2 of the 804
## 3 in the 798
## 4 climatechange is 570
## 5 is a 442
## 6 of climatechange 437
## 7 on the 383
## 8 on climatechange 364
## 9 to the 354
## 10 this is 331
## # . with 134,646 more rowsAgain, eliminate the stop words from the paired words
library(tidyr)##
## Attaching package: 'tidyr'## The following object is masked from 'package:igraph':
##
## crossingclimate_tweets_separated_words <- climate_tweets_paired_words %>%
separate(paired_words, c("word1", "word2"), sep = " ")
climate_tweets_filtered <- climate_tweets_separated_words %>%
filter(!word1 %in% stop_words$word) %>%
filter(!word2 %in% stop_words$word)
# new bigram counts:
climate_words_counts <- climate_tweets_filtered %>%
count(word1, word2, sort = TRUE)
head(climate_words_counts)## Warning: `...` is not empty.
##
## We detected these problematic arguments:
## * `needs_dots`
##
## These dots only exist to allow future extensions and should be empty.
## Did you misspecify an argument?## # A tibble: 6 x 3
## word1 word2 n
## <chr> <chr> <int>
## 1 climate change 7181
## 2 fight climate 185
## 3 covid 19 164
## 4 frack frack 114
## 5 global warming 113
## 6 foreign policy 98## # A tibble: 6 x 3
## word1 word2 n
## <chr> <chr> <int>
## 1 climate change 1021
## 2 climatechange climatecrisis 232
## 3 climatechange globalwarming 147
## 4 global warming 141
## 5 hurricane dorian 113
## 6 climate crisis 100Finally, plot the data
library(igraph)
library(ggraph)
# plot climate change word network
# (plotting graph edges is currently broken)
climate_words_counts %>%
filter(n >= 24) %>%
graph_from_data_frame() %>%
ggraph(layout = "fr") +
geom_edge_link(aes(edge_alpha = n, edge_width = n)) +
geom_node_point(color = "darkslategray4", size = 3) +
geom_node_text(aes(label = name), vjust = 1.8, size = 3) +
labs(title = "Word Network: Tweets using the hashtag - Climate Change",
subtitle = "Text mining twitter data ",
x = "", y = "")