Regression

How do skill and openness to cross-cutting exposure predict trust in social media source, after controling for sex, race and ideologies?

Answering the question above requires using more than one independent variable and some of the independent variables are categorical and others are continuous.This is where regression comes into play. You use regression when your dependent variable is continuous and your independent variables are continuous or when your independent variables are mixed (i.e., some are continuous and others are categorical).

In this example, we use the same dataset, but will create two new continuous variables from the raw data. The two new variables are skill and openness to cross-cutting exposure.

#create variables

#remove missing values and inapplicable cases
clean_data <- clean_data[!is.na(clean_data$q5)&
                        !is.na(clean_data$q3a) &
                        !is.na(clean_data$q3b) &
                        !is.na(clean_data$q3c) &
                        !is.na(clean_data$q3d) &
                        !is.na(clean_data$q3e) &
                        !is.na(clean_data$q3f),]

clean_data<- clean_data[clean_data$q5<=3 &
                                clean_data$q3a <= 2 &
                                clean_data$q3b <= 2 &
                                clean_data$q3c <= 2 &
                                clean_data$q3d <= 2 &
                                clean_data$q3e <= 2 &
                                clean_data$q3f <= 2,]

library(car)
clean_data$q3b_reversed <- recode(clean_data$q3b, "1=2; 2=1")
clean_data$q3c_reversed <- recode(clean_data$q3c, "1=2; 2=1")
clean_data$q3d_reversed <- recode(clean_data$q3d, "1=2; 2=1")
clean_data$q3f_reversed <- recode(clean_data$q3f, "1=2; 2=1")

clean_data$skill <- clean_data$q5
clean_data$skill <- recode(clean_data$skill, "1=3; 2=2;3=1")

clean_data$openness <- clean_data$q3a + clean_data$q3b_reversed + clean_data$q3c_reversed + clean_data$q3d_reversed + clean_data$q3e + clean_data$q3f_reversed

Plot the key variables.

hist(clean_data$skill)

hist(clean_data$openness)

Perform regression

lm.1 <- lm(skill~ openness+ideo+sex+race, data = clean_data)
summary(lm.1)

## 
## Call:
## lm(formula = skill ~ openness + ideo + sex + race, data = clean_data)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -1.8043 -0.5374  0.2979  0.4082  0.9228 
## 
## Coefficients:
##              Estimate Std. Error t value Pr(>|t|)    
## (Intercept)  1.862520   0.162010  11.496  < 2e-16 ***
## openness     0.077866   0.014905   5.224 1.95e-07 ***
## ideo2       -0.060737   0.053647  -1.132 0.257719    
## ideo3        0.007417   0.052175   0.142 0.886966    
## ideo4       -0.016933   0.055969  -0.303 0.762271    
## ideo5        0.050796   0.062358   0.815 0.415412    
## sex2        -0.066499   0.027570  -2.412 0.015965 *  
## race2       -0.165911   0.045488  -3.647 0.000272 ***
## race3       -0.063471   0.069635  -0.911 0.362169    
## race5        0.035560   0.121053   0.294 0.768976    
## race6        0.396826   0.339322   1.169 0.242369    
## race7       -0.358886   0.057703  -6.220 6.17e-10 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 0.5865 on 1821 degrees of freedom
## Multiple R-squared:   0.05,  Adjusted R-squared:  0.04426 
## F-statistic: 8.713 on 11 and 1821 DF,  p-value: 3.219e-15

Take a closer look at the regression model

library(lm.beta)
lm.beta(lm.1)

## 
## Call:
## lm(formula = skill ~ openness + ideo + sex + race, data = clean_data)
## 
## Standardized Coefficients::
##  (Intercept)     openness        ideo2        ideo3        ideo4 
##  0.000000000  0.120411725 -0.044373592  0.005867208 -0.011298187 
##        ideo5         sex2        race2        race3        race5 
##  0.026579526 -0.055425981 -0.084126065 -0.021097938  0.006740099 
##        race6        race7 
##  0.026746125 -0.143328105

#Text cleaning

Why text cleaning?

Textual data are always messy. The data may contain words that, if taken out of context, would be meaningless. You may also encounter a group of different words which convey the same meaning. Or you might have to convert slangs and acronyms into standard English, or emojis into something computer can recognize. Only by cleaning the mess and the noise in the text will you be able to discern useful patterns and signals.

From corpus to DFM There is a lot of interest in quantifying and visualizing textual data. Texts reveal our thoughts, our personality, and the pulse of a society. We broadly refer to the quantification of text as text mining. Thanks to the developments in Natural Language Processing and Information retrieval, we now have a wide selection of easy-to-use R libraries for cleaning, transforming, quantifying, and visualizing text.

Which R library?

Throughout the tutorials on text mining, we will use the library quantaeda, which stands for Quantitative Analysis of Textual Data. You can visit the library website to view many examples. There is another decent text mining library called tidytext. Although we will not use it since there is a considerable overlap in functionality between the two libraries, I still highly recommend checking out tidytext and a free e-book written by the library author.

What data? In this tutorial, we will experiment with a dataset containing 4,995 tweets containing the #metoo hashtag. You can download the data here. Or you can run the code below to download the data into your RStudio. For faster loading, we ask R to show the first 50 tweets and four selected columns. Text analytics typically begins with text files (e.g., a data frame or a CSV file) containing character strings (the actual text) and the metadata of the text. In the example below, we are interested in analyzing the text content in the text column.

library(readr)
library(DT)

metoo_tweets <- read.csv("https://curiositybits.cc/files/metoo.csv")

#show the first 500 arrests
datatable(metoo_tweets[1:50,c("status_id","screen_name","text","retweet_count")], options = list(pageLength = 5)) 

Workflow

A typical workflow in text analytics is:

1. create a corpus from text files;
2. tokenization;
3. create a DFM (document-feature matrix);
4. conduct a variety of text analyses.

Corpus

Think of a corpus as a container of different documents. A corpus is the complete collection of text you want to analyze, Documents are individual units in that collection. For example, if you want to analyze someone’s tweets (the corpus), each unique tweet can be treated as a document. Another example is: if you want to compare tweets sent by GOP and the Democratic Party, you can treat all tweets sent by one party as a single document. This treatment will lead you to have two documents in the corpus. Below we will create a corpus of 4,995 documents (i.e., tweets).

library(quanteda)

metoo_tweets$text <- as.character(metoo_tweets$text) #optional step: convert the text column to characters

tweet_corpus <- corpus(metoo_tweets,docid_field = "status_id",text_field = "text")

tweet_corpus

## Corpus consisting of 4,995 documents and 86 docvars.

Alternatively, you can combine tweets sent by the same users and create a corpus in which a document is an user’s tweets.You will see that the corpus created from the code below contains 4,090 documents (i.e., unique users). In order to create the corpus, we first create a new data frame (called tweets_byusers) by using aggregate() to merge tweets from the same user. Then, in corpus(), we specify docid_field = “screen_name”. This tells R to recognize the city column as the document IDs.

library(dplyr)
tweets_byusers <- aggregate(text~screen_name, data = metoo_tweets, paste0, collapse=". ")

#create a corpus based on the new data frame
user_corpus <- corpus(tweets_byusers,docid_field = "screen_name",text_field = "text")

user_corpus

## Corpus consisting of 4,090 documents and 0 docvars.

Tokenization

Run the code below and see how text is tokenized.

This tokenize the second tweet based on words.

tokens(tweet_corpus[2])

## tokens from 1 document.
## x1103090889126264833 :
##  [1] "Jessi"      "Hempel's"   "The"        "Problem"    "with"      
##  [6] "#METOO"     "and"        "Viral"      "Outrage"    "represent" 
## [11] "the"        "key"        "elements"   "of"         "a"         
## [16] "major"      "issue"      "in"         "society"    ";"         
## [21] "sexual"     "harassment" "and"        "women"      "are"       
## [26] "finally"    "speaking"   "out"        "about"      "their"     
## [31] "past"       "tragedies"  "."          "#COM416"    "https"     
## [36] ":"          "/"          "/"          "t.co"       "/"         
## [41] "15Uv2nGGGo"

tokens(), by default, segments a corpus into tokens where each token represents a word. The code above tokenizes the second document (i.e., the 2nd tweet) in the corpus.

We commonly tokenize corpus based on words. The following code tokenizes the corpus based on sentences so that each unique sentence is treated as a token. We do that by adding what = "sentence to the tokens() function.

tokens(tweet_corpus[2], what = "sentence")

## tokens from 1 document.
## x1103090889126264833 :
## [1] "Jessi Hempel's The Problem with #METOO and Viral Outrage represent the key elements of a major issue in society; sexual harassment and women are finally speaking out about their past tragedies."
## [2] "#COM416 https://t.co/15Uv2nGGGo"

DFM

With a corpus, we can go ahead and create a DFM (document feature matrix). What is a DFM? The best way to understand it is perhaps by playing around one. Run the code below. It will construct a DFM based on the corpus of 4,995 documents.

DFM is a matrix. Numbers in DFM represent how many times a token (commonly referring to a word or a sentence) appears in a given document. In the output above. The word confrontation appears once in document x1103092683717906432 (this refers to the status_id of a tweet).

Please note that the dfm() function performs tokenization under the hood. In other words, by default, dfm() tokenizes your text first before constructing a DFM object.

tweets_dfm <- dfm(tweet_corpus)

tweets_dfm[1:5,1:5] #show the first 5 words in the first 5 documents

## Document-feature matrix of: 5 documents, 5 features (64.0% sparse).
## 5 x 5 sparse Matrix of class "dfm"
##                       features
## docs                   #metoo confrontation can be alarming
##   x1103092683717906432      1             1   1  1        1
##   x1103090889126264833      1             0   0  0        0
##   x1103092651140808705      1             0   0  0        0
##   x1103092268490280961      1             0   0  0        0
##   x1103092648888410112      1             0   0  0        0

tweets_dfm[1:5,1:5] gives you the frequency distribution of the first 5 words across the first 5 documents. The complete matrix (the object named reviews_dfm) is 25,974 by 4,995. It means that this dfm object has 25,974 features (i.e., words) across 4,995 documents (tweets).

After you have a DFM from your text data, you can do all sorts of things. For instance, run the code below will show you the most frequent terms in your corpus.

library(ggplot2)

tweets_dfm %>% 
  textstat_frequency(n = 25) %>% 
  ggplot(aes(x = reorder(feature, frequency), y = frequency)) +
  geom_point() +
  coord_flip() +
  labs(x = NULL, y = "Frequency") +
  theme_minimal()

Filter unwanted words

But, wait a minute! The most frequent terms are: /, ., :? Shouldn’t we take those meaningless and noisy words and punctuation marks out of the analysis? You are right! What I have demonstrated above is a very messy and rudimentary step in text mining. Next we will cover a set of standard text cleaning and transformation procedures.

In the code below, a number of cleaning criteria are added. Observe changes in the output.

tokens(tweet_corpus[2],remove_punct = TRUE,remove_numbers = TRUE, remove_symbols = TRUE, remove_twitter=TRUE, remove_url=TRUE)

## tokens from 1 document.
## x1103090889126264833 :
##  [1] "Jessi"      "Hempel's"   "The"        "Problem"    "with"      
##  [6] "METOO"      "and"        "Viral"      "Outrage"    "represent" 
## [11] "the"        "key"        "elements"   "of"         "a"         
## [16] "major"      "issue"      "in"         "society"    "sexual"    
## [21] "harassment" "and"        "women"      "are"        "finally"   
## [26] "speaking"   "out"        "about"      "their"      "past"      
## [31] "tragedies"

This time, tokens() segments our corpus into word tokens, but in the process, remove punctuation marks, numbers, symbols, Twitter handles and hashtags, and URLs. As discussed early, choosing what cleaning criteria to apply should be decided on a case by case basis. For some tasks, you may want to preserve Twitter handles and hashtags because they hold the key to understanding your data. You may also want to preserve punctuations if you are interested in how punctuations convey emotions.

Now, let’s apply tokens() to tokenize the whole corpus and identify the most frequent terms. The tokenization produces an object call tok. We can create a DFM from review_tok.

tok <- tokens(tweet_corpus,remove_punct = TRUE,remove_numbers = TRUE, remove_symbols = TRUE, remove_twitter=TRUE, remove_url=TRUE)

tweets_dfm <- dfm(tok)

#find 20 most frequent terms
topfeatures(tweets_dfm, 20)

##    metoo      the       to        a      and       of       is       in 
##     4970     3337     2117     1772     1649     1627     1250     1177 
##        i      for      you     that       on     this       it movement 
##     1071      948      875      701      698      680      617      589 
##      are     with    women      amp 
##      557      519      486      447

Hold on, the most frequent terms are: the, to, a, and, of, is? This tells us hardly anything interesting about the data. Well, this is because there are common stop words in our data. Stop words are commonly used function words that we ask algorithms to ignore in processing text data. In many text analytics tasks, we reckon such stop words as meaningless noises. But, there is a school of thoughts that argue how people use function words (e.g., the, is, at, which, and on) reveals important insights about social identities, power dynamics in a communication context, and personality. Read more here.

To remove stop words from tok (the tokenized corpus created from the previous step), use the code below.

tok <- tokens_select(tok, pattern = stopwords('en'), selection = 'remove')

Alternatively, you can directly use remove = stopwords(“english”) inside the dfm() function. It will do the same trick.

tweets_dfm <- dfm(tweet_corpus, remove = stopwords("english"), stem = FALSE, remove_punct = TRUE)

Run the below code blocks. Each will test a new set of cleaning criteria.

tok <- tokens_select(tok, min_nchar=3, selection = 'keep') #remove any word shorter than3 characters.

tok <- tokens_select(tok, pattern = c("metoo","twitter","however"), selection = 'remove') #remove additional filter words.

The code above can be handy when we want to create our own list of filter words. Note that the tokenization process will automatically convert anything into lower cases, so instead of adding “However” as a filter word, we added “however.”

Ever wonder what stem = FALSE means in the code above? Or do you want to try other cleaning criteria? Stemming refers to a cleaning process that chops off the ends of words. Why do we do that? This is because different words (e.g., working, works, and worked) may share a same stem/root word and thus convey similar meaning. In text mining, we might want to treat such words as identical.

With the cleaned DFM, you can visualize 25 most frequent terms.

tweet_corpus <- corpus(metoo_tweets,docid_field = "status_id",text_field = "text")

tok <- tokens(tweet_corpus,remove_punct = TRUE,remove_numbers = TRUE, remove_symbols = TRUE, remove_twitter=TRUE, remove_url=TRUE)

tweets_dfm <- dfm(tok,remove = stopwords("english"))

tweets_dfm %>% 
  textstat_frequency(n = 25) %>% 
  ggplot(aes(x = reorder(feature, frequency), y = frequency)) +
  geom_point() +
  coord_flip() +
  labs(x = NULL, y = "Frequency") +
  theme_minimal()

#Keywords

Now you are on course to try basic text mining techniques to extract insights from textual data. In this tutorial, we will try four techniques: word cloud, n-grams, and keyness.

Word cloud

We can create a word cloud using the most frequent terms in the tweets.

textplot_wordcloud(tweets_dfm, max_words = 100)

Keyness

Just because a term is frequent, does not mean it is a “keyword.” In text mining, keywords are measured by “keyness”. It shows how distinctive a word is for a group of texts when compared to the texts in other groups. For instance, we suspect that verified Twitter users and unverified users tweet about #metoo differently. We can use the textstat_keyness function to find terms that distinguish verified users’ tweets from the tweets from unverified users.

tweets_dfm <- dfm(tok, groups="verified") 

result_keyness <- textstat_keyness(tweets_dfm, target = "TRUE") 

# Plot estimated word keyness
textplot_keyness(result_keyness) 

N-gram

Text mining is a process of data reduction. It simplifies a complex text into tokens. Along the way, lots of information is lost, especially, when you take single words out of context. Next, we will try N-grams. N-grams can preserve multi-word phrases and expressions.

Below we construct bigrams and compare frequent bigrams between two groups of tweets: those sent by verified users and the ones sent by unverified users.

tweet_corpus <- corpus(metoo_tweets,docid_field = "status_id",text_field = "text")

tok <- tokens(tweet_corpus,remove_punct = TRUE,remove_numbers = TRUE, remove_symbols = TRUE, remove_twitter=TRUE, remove_url=TRUE)

tok <- tokens_ngrams(tok, n = 2)
tweets_dfm <- dfm(tok, groups="verified") 

set.seed(132)
textplot_wordcloud(tweets_dfm, comparison = TRUE, max_words = 100)

#Dictionary-based text analysis: sentiment detection

Researchers commonly use pre-defined and pre-tested dictionaries to identify themes and features in texts. The dictionary-based approach has been widely used in sentiment detection.During the 2012 US presidential election, Twitter, in partnership with several polling agencies, launched something called Twitter Political Index. The idea was to track candidates’ popularity among voters based on sentiment expressed in tweets. Back then, such idea was a novelty. Nowadays, sentiment analysis of social media text has been widely applied in marketing/PR, electoral forecasting, and sports analytics. The NPR show Planet Money even built a Twitter bot to automatically trade stocks based on sentiments in Trump’s tweets.

Below we use a library called syuzhet to identify sentiments expressed in tweets. The library uses the NRC Emotion Lexicon which rates words by eight dimensions: joy, anger, anticipation, disgust, fear, sadness, surprise and trust.

library(syuzhet) #this is the library for sentiment detection

#extract sentiments
Sentiment <- get_nrc_sentiment(metoo_tweets$text)

alltweets_senti <- cbind(metoo_tweets, Sentiment)

Next, we sum up across sentiment types and plot the data.

library(ggplot2)
library(ggthemes)
library(reshape2)
library(dplyr)

senti_aggregated <- Sentiment %>% 
  summarise(anger = sum(anger), 
            anticipation = sum(anticipation), 
            disgust = sum(disgust), 
            fear = sum(fear), 
            joy = sum(joy), 
            sadness = sum(sadness), 
            surprise = sum(surprise), 
            trust = sum(trust)) %>% melt

ggplot(data = senti_aggregated, aes(x = variable, y = value)) +
  geom_bar(aes(fill = value), stat = "identity") + geom_smooth(method = "lm", se = FALSE) + scale_color_fivethirtyeight("cyl") +
  theme_fivethirtyeight()

#A network approach towards text: semantic networks

To understand what a semantic network looks like, go ahead and run the code below.

library(quanteda)
library(ggplot2)

tweet_corpus <- corpus(metoo_tweets,docid_field = "status_id",text_field = "text")

tok <- tokens(tweet_corpus,remove_punct = TRUE,remove_numbers = TRUE, remove_symbols = TRUE, remove_twitter=TRUE, remove_url=TRUE)

tweets_dfm <- dfm(tok, remove = stopwords("english")) 

#create a feature co-occurrence matrix (FCM)
tweets_fcm <- fcm(tweets_dfm) 

#extract the top 50 frequent terms from the FCM object
feat <- names(topfeatures(tweets_fcm, 100)) 

#trim the old FCM object into a one that contains only the 50 frequent terms 
fcm_select <- fcm_select(tweets_fcm, pattern = feat)

set.seed(144)
textplot_network(fcm_select, min_freq = 0.8)

What you see in the output are a bunch of words interconnected to each other. A semantic network can tell us the most central concept/ideas in your corpus. We can use mathematical functions to quantify each position in a semantic network. For example, centrally located words tend to have higher betweenness centrality. Even without any mathematic calculation, we can easily spot the most central words based on the visualization. The concept of semantic networks is developed from the graph theory and social network analysis. We will have a separate set of tutorials on online social networks.

How are words connected to one and another? You may wonder about that. In quantaeda, a semantic network is referred to as feature co-occurrence matrix (FCM). It is a type of network based on co-occurrence: more specifically, two words are linked to each other if they appear in the same document. In our case, two words are connected to each other if they occur in the same review.

The semantic network approach can be useful in mapping out central ideas and see how different ideas are connected and clustered. A shortcoming here, though, is that there are too many vague words in the corpus. A refined approach is tagging each word token as noun, adjective, or verb (a process called Part-of-speech tagging) and then create a semantic network based only on one type of words, say, a semantic network entirely based on nouns or adjectives. You may also consider this approach in analyzing Twitter hashtags. You will learn alot by looking at how different hashtags are concentrated and clustered based on their co-occurrence in the same tweet. See the code below.

tweet_corpus <- corpus(metoo_tweets,docid_field = "status_id",text_field = "text")

tweets_dfm <- dfm(tweet_corpus) 

tweets_dfm <- dfm_select(tweets_dfm, pattern = ("#*"))

tweets_fcm <- fcm(tweets_dfm) 

#extract the top 50 frequent terms from the FCM object
feat <- names(topfeatures(tweets_fcm, 100)) 

#trim the old FCM object into a one that contains only the 50 frequent terms 
fcm_select <- fcm_select(tweets_fcm, pattern = feat)

set.seed(144)
textplot_network(fcm_select, min_freq = 0.8)

#Topic models

What is a topic model?

Have you dreamed of a day when algorithms can quickly scan through your textbooks and give you a bullet point summary? How convenient! No more tedious reading! Actually, there are algorithms out there that do automatic summarization of large-scale corpus. They are called topic models. In building topic models, we basically ask computers to discover some abstract topics from the text. The internal logic is this: words about the same topic tend to be used together in the same or adjacent semantic space.

New library?

The topic model feature is built in the quantaeda library. But you may need to install_topicmodels_.

The workflow

Run the code below and wait untill you get an output.

As you can tell, producing topic models takes time–it is computationally intensive if you hope to build a topic model based on a large-scale corpus.

In quanteda, we convert a regular DFM object into a format ready for building topic models (see the convert() function in the code above). We then apply the LDA() function to build a topic mode. LDA stands for Latent Dirichlet allocation. It is one of the most commonly used topic modeling algorithms. terms(lda, 10) gives 10 frequent terms from each topic.

But, what does k=10 mean?

By setting k=10, we ask the LDA algorithm to identify ten topics in the corpus. This is where things get extremely complicated and confusing. The algorithm is agnostic about how many topics are there in the corpus. So you will start with a k value and check if the keywords returned shows any clear distinction of topics. If you struggle to summarize the text based on the keywords returned, chances are it is not a good model. Then you will go back and adjust the k value until you find one that gives you somewhat sensible output.

Tuning up a topic model is an art. It is like tuning up a telescope. So based on the corpus we have, let’s try different k values. This time, run the code in your own laptop.

If you want to plot key words from each topic model, try the following code in the library tidytext.

library(tidytext)
library(topicmodels)

dfm_fortm <- convert(tweets_dfm, to = "topicmodels")
lda <- LDA(dfm_fortm, k = 10)

tuber_topics <- tidy(lda, matrix = "beta")

top_terms <- tuber_topics %>%
  group_by(topic) %>%
  top_n(10, beta) %>%
  ungroup() %>%
  arrange(topic, -beta)

top_terms %>%
  mutate(term = reorder(term, beta)) %>%
  ggplot(aes(term, beta, fill = factor(topic))) +
  geom_col(show.legend = FALSE) +
  facet_wrap(~ topic, scales = "free") +
  coord_flip()

Extracting meaning?

Based on your experience with topic models so far, do you think computers can extract true meanings from textual data?

#What can a network tell us?

Have you wondered how information spreads on Twitter, how Instagram influencers are identified, and how different actors in an online community collaborate or confront one and another? There are the sorts of questions that can be best answered using network analysis and network visualization. In network analysis of internet communities, we visualize and quantify the structure of social relationships and information flows. See a real-world application my team has built to track the upcoming Philippine General Election.

Here, you can see a retweet network based on 9,999 #BreakUpBigTech tweets. In this network, a pair of users represents a retweeting relationship. That is, two users are connected to one and another if one retweets or is retweeted by the other. For simplicity, the graph below only shows users who at least twice retweeted or were retweeted by others.

Guess how the size and color of a node is determined.

#Edges

Where do we begin to visualize a network? It all starts with nodes and edges. The table below shows 20 tweets.

An edgelist shows all edges in a network along with attributes of the edges. An edge is a pair of relationship between two nodes (in this case, users). An edge can be directed: for example, A retweets B will be expressed as User A → User B, whereas B retweets A is expressed as User B → User A. But, in some cases, an edge is undirected. Think about your Facebook relationships. If user A is a friend of user B. By default, user B is also connected to user A.

An edgelist based on the 20 tweets looks like this. The column source lists the Twitter users who retweeted. The column target shows those users who were on the receiving end of the retweets (i.e., users who were retweeted by others). The size column is edge weight, referring to the number of retweeting that occurred between the same pair of users.

#Nodes

In our example, a node is a Twitter user. Below is a list of nodes, with their id, labels, and attributes (e.g., size). Wonder how size is determined? We will cover this in the later part of the tutorial.

#How to turn tweets into network?

Here comes the real deal: how to turn collected tweets into a network. Previously, the process involves several steps of text cleaning to extract relevant @screennames. A new library called graphtweets[http://graphtweets.john-coene.com/index.html] makes the task much easier. Graphtweets works seamlessly on Twitter data collected through rtweet[https://rtweet.info/]. You can easily tweak the code in graphtweets to make it work for data that come from different shapes and sizes. Below, I will use the data frame tweets as a demo. tweets contains 9,999 tweets that use #BreakUpBigTech.

Make sure graphtweets is installed and loaded. We begin by defining a R function. A function is a set of codes organized together to perform a specific task. R has a large number of built-in functions. We can also create our own functions. A self-defined function will save a lot of repetitive work. See how I define a function called extractrt below.

library(graphTweets)

extractrt <- function(df){
  rt <- df %>% 
    gt_edges(screen_name, retweet_screen_name) %>% # get edges
    gt_nodes() %>% # get nodes
    gt_collect() # collect
  
  return(rt)
}

The self-defined function extractrt takes in df (it will have to be a data frame from rtweet in order for the function to work). The function then uses three in-built functions from graphtweets to extract nodes and edges from tweets in df. In a standard data frame returned from rtweet, the sender of a retweet (the user who retweets other) is in the screen_name column, and the retweeted users are in the retweet_screen_name column. See how this particular process is spelled out below.

 gt_edges(screen_name, retweet_screen_name) %>% # get edges

The function extractrt creates and returns an object called rt. rt can be easily converted to an igraph object. igraph is one of the most common network analysis libraries in R. We will deal with igraph later.

After a function is defined in R, we can apply it to a data frame of tweets. Below, we apply extractrt to tweets and create rtnet.

rtnet <- extractrt(tweets)

The reason we call this function extractrt is that it only extracts retweeting relationships. How about Twitter mentions/replies. A Twitter mention/reply signifies a more engaged mode of interaction. A retweet is mostly a passive information relay, but a Twitter mention/reply is an active outreach. In the code below, I define extractmt as a function for extracting edges and nodes in Twitter mentions/replies. This function scans tweets and extracts @screennames in the screen_name and mentions_screen_name column. I then apply extractmt to tweets and create mtnet.

extractmt <- function(df){
  
  mt <- df %>% 
    gt_edges(screen_name, mentions_screen_name) %>% # get edges
    gt_nodes() %>% # get nodes
    gt_collect() # collect
  
  return(mt)
}

mtnet <- extractmt(tweets)

Now that we have two network objects: rtnet and mtnet. We want to take a look at them. But unlike data frames, you cannot just click to view a network object. You can use the two following self-defined functions to get node lists and edgelists from the two network objects.

#define a function called nodes to extract node information from a network object

nodes <- function(net){
  
  c(edges, nodes) %<-% net
  nodes$id <- as.factor(nodes$nodes) 
  nodes$size <- nodes$n 
  nodes <- nodes2sg(nodes)
  nodes <- nodes[,2:5]
  
  return(nodes)
}

#define a function called edges to extract edge information from a network object

edges <- function(net){
  
  c(edges, nodes) %<-% net
  edges$id <- seq(1, nrow(edges))
  edges <- edges2sg(edges)
  
  return(edges)
}

#apply the two self-defined functions
rtnet_nodes <- nodes(rtnet)
rtnet_edges <- edges(rtnet)

mtnet_nodes <- nodes(mtnet)
mtnet_edges <- edges(mtnet)

From the above step, we create four objects: rtnet_nodes, rtnet_edges, mtnet_nodes, mtnet_edges. The four objects are all data frames. Let’s take a look at rtnet_edges. It is an edgelist.

library(DT)
datatable(rtnet_edges, options = list(pageLength = 5)) 

#Convert to igraph

Network analysis is essentially a mathematical process. Any user and any network can be scored based on some attributes. To do this, we will convert our network objects into igraph objects. For example, for the retweet network, we can create an igraph object based on rtnet_edges and rtnet_nodes. See the code and comments below.

# use rtnet_edges as the edgelist and rtnet_nodes as the node list. Set the network type as directed

rt <- graph_from_data_frame(d=rtnet_edges, vertices=rtnet_nodes, directed=T) 

# see edge weight by copying the values from the size column in rtnet_edges

rt <- set_edge_attr(rt, "weight", value= rtnet_edges$size)

# we do the same for the mention network

mt <- graph_from_data_frame(d=mtnet_edges, vertices=mtnet_nodes, directed=T) 
mt <- set_edge_attr(mt, "weight", value= mtnet_edges$size)

But first, let’s just take a look at some network-level indicators.

#Size matters! A quick way to compare different networks (e.g., the retweet network vs. mention network) is looking at its size. Run the code below to get a count of edges and nodes in rtnet and mtnet.

Which network has more users in it? And which network has more connections?

vcount(rt) #this shows the number of nodes/vertices in rt 

## [1] 9092

vcount(mt) #this shows the number of nodes/vertices in mt

## [1] 9322

ecount(rt) #this shows the number of edges in rt 

## [1] 9137

ecount(mt) #this shows the number of edges in mt

## [1] 10399

#Dense or sparse?

A densely connected network (high density score) is a type of network in which many users are interconnected, whereas a sparse network (low density) is a network in which only a few are interconnected. Two contrasting examples of dense and sparse networks are a network of people in a family gathering in which almost everyone knows everyone else, and a network of people sitting on a public bus.

Which network is more interconnected?

edge_density(rt, loops = FALSE) #the density of rt

## [1] 0.0001105433

edge_density(mt, loops = FALSE) #the density of mt

## [1] 0.0001196796

#Centralized or decentralized?

Think of centralization as a question of inequality and who is in control. In a centralized network, a small number of nodes (users) control the information flow. In a retweet network specifically, it means that only a handful of users retweet or are retweeted by others. Centralized and decentralized networks have different ramification for the diffusion of ideas, norms, and effective mobilization.

by setting mode = c(“in”), we calculate the centralization score based on the extent to which users are retweeted by others (as opposed to retweet others).

So, which network is more centralized?

#Calculate centralization
centr_degree(rt, mode = c("in"), loops = TRUE,normalized = TRUE)$centralization

## [1] 0.95006

centr_degree(mt, mode = c("in"), loops = TRUE,normalized = TRUE)$centralization

## [1] 0.936797

#Birds of a feathre flock together?

Have you heard of the saying birds of a feather flock together? In a network, nodes tend to cluster together based on some shared attributes. For instance, Twitter users may retweet mostly content they agree with. Hence, this tendency will result in a cluster of nodes based on similar mindsets or opinions. To what extent is a network reflecting this pattern of clustering can be quantified by using clustering coefficient.

transitivity(rt)

## [1] 1.929652e-06

transitivity(mt)

## [1] 0.0001250248

#Is it reciprocal?

Reciprocity is calculated as the proportion of reciprocated ties. In the retweet network, for example, reciprocity shows the extent to which a pair of users have mutually retweeted one and another.

Which form of Twitter interactions (retweet vs. mention) is more reciprocal?

reciprocity(rt)

## [1] 0.000219082

reciprocity(mt)

## [1] 0.001733436

#Look for influencers

I have introduced previously a range of indicators to quantify a network. Such indicators are only useful when it involves a comparison of different networks. When analyzing one single network, we are more interested in node-level indicators.

A common task in network analysis is identifying influencers? An influencer could mean different things to different people. Here we try a couple of dfferent metrics.

indegree centrality measures the number of incoming connections a user has received. A high indegree in the retweet network means that the user is frequently retweeted by others. Do you agree that the most retweeted users are influencers? And why?

indegree_rt <- sort(degree(rt,mode = "in"),decreasing = TRUE)
indegree_rt[1:10] #show the top 10 users ranked by in-degree

##         ewarren     anandwrites      omanreagan   stclairashley 
##            8638             107              40              40 
##        guardian      chadfelixg          a35362         soyrosa 
##              39              20              19              16 
##         jc_cali myth_capitalism 
##              15              14

outdegree centrality measures the number of outgoing connections a user has. A high outdegree in the retweet network means that the user frequently retweets other users. What would you call such users, mobilizers?

outdegree_rt <- sort(degree(rt,mode = "out"),decreasing = TRUE)
outdegree_rt[1:10] #show the top 10 users ranked by out-degree

## edwood05572006         raqb16   damonbethea1    fuelgrannie   gavin_bonnar 
##             13              6              5              5              4 
##   atheist_cvnt     natemezmer  philippejouan  sharonresists   tedgrunewald 
##              3              3              3              3              3

Betweenness centrality measures the number of times a node lies on the shortest path between other nodes. We use this metric to find users who act as ‘bridges’ between nodes in a network and who influence the information flow around a network.

bt <- sort(betweenness(rt, directed=T, weights=NA), decreasing = TRUE)
bt[1:10] #show the top 10 nodes by betweenness centrality 

##      omanreagan      chadfelixg         soyrosa    damonbethea1 
##              32              18              16              13 
##         jc_cali    commondreams  resistasista76          yitzee 
##               8               4               3               3 
##      tuxcedocat vivek_gkrishnan 
##               2               2

Ever wonder how Google ranks search results? It uses the PageRank algorithm developed by Google’s founders Sergey Brin and Larry Page. We can use PageRank to locate influencers as well.

pr <- page_rank(rt, algo = c("prpack"))
pr <- sort(pr$vector,decreasing = TRUE)
pr[1:10] #show the top 10 users ranked by PageRank

##         ewarren     anandwrites myth_capitalism      omanreagan 
##    0.4338045988    0.0047293401    0.0042530946    0.0042105884 
##   scottrickhoff   stclairashley        guardian      chadfelixg 
##    0.0020597744    0.0017596358    0.0016080549    0.0010092895 
##          a35362         soyrosa 
##    0.0009209153    0.0008592202

#Look for clusters/cliques

We use community detection algorithm to cluster users into different groups (we call such groups clusters or cliques). Users in the same cluster are more connected with one and another than with users outside of the cluster. By using the community detection method, we can reveal important divisions and fragmentation that exist due to different opinions, values, and user characteristics.

Some community detection algorithms require intensive computating. It may take a long time to produce an output.

k-core

Creating k-core is fast and easy. We can use k-core to identify a small subset of users who are the most interconnected. In a k-core, each node has at least k connections with everyone else. Below we extract a 2-core (named twocore) in which each user has at least 2 edges with any other users in the core.

kcore <- coreness(rt, mode="all") 
twocore <- induced_subgraph(rt, kcore>=2)

edge betweenness (Newman-Girvan)

This is one of the community detection algorithm that is computationally intensive. Be patient when it is crunching numbers for you.

The code above creates an object call ceb. It contains the information about which cluster each node belongs to. We can run the code below to see the cluster ID of the first 10 nodes.

ceb <- cluster_edge_betweenness(rt) 

print("there are",length(ceb),"clusters based on this community detection algorithm")

membership(ceb)[1:10] #list only 10 nodes.

#Visualize a network and make it pretty!

There are many ways to visualize a network. You can make the visualization static or interactive as this example shows. You can even create a dynamic one showing the evolution of a network over time an example.

Below, we will try some of the basics using two libraries igraph and VisNetwork. igraph comes with some in-built functions for visualization. VisNetwork takes a step further by making it prettier and interactive.

Before you visualize a network, here are the decisions you need to make:

• do you want to assign colors to nodes based on some node attributes?
• do you want to set the node size based on some attributes?
• do you want to show all nodes?

In our example below, we color nodes based on the clusters they belong to. We set the node size based on PageRank score (the famous scoring technique used by Google), with central nodes represented by bigger nodes. And we don’t want to show all nodes as that will create a messy network; Instead, we would show only the most interconnected subset (using k-core).

In the previous steps, we know the codes for calculating node and network-level metrics (e.g., centrality). Here, we will pass the metrics to nodes and store them as node attributes. This will allows the visualization code to pick up the attributes and use them for sizing and coloring.

In the code below, we add PageRank score (used for node size) and the cluster id (used for assigning color). We use V(rt) to access node attributes and E(rt) to access edge attributes.

library(igraph)
library(visNetwork)
library(scales)

pr <-page_rank(rt, algo = c("prpack"))
V(rt)$size <- pr$vector*100  #set node size by PageRank scores.

wc <- cluster_walktrap(rt)

V(rt)$color <- membership(wc) # set color by subgroup id

Since we visualize only the 2-core. We create a subset of the network.

kcore <- coreness(rt, mode="all") 
twocore <- induced_subgraph(rt, kcore>=2)

Find a visualization algorithm that fits

And we visualize it. Notice that we set layout = “layout_nicely”? This is how we specify which visualization algorithm to use. There is a whole bunch of them: see the listing. If you are curious about visual effects from different algorithms, try layout =“layout_in_circle” or layout =“layout_with_kk” or layout =“layout_with_sugiyama”

visIgraph(twocore,idToLabel = TRUE,layout = "layout_nicely") %>%
  visOptions(highlightNearest = TRUE, nodesIdSelection = TRUE)