The Latest Trump's Tweets

Susan Li

April 3, 2017

Just finished a few hours text mining lesson, can't wait to put my new skill into practice, starting from Trump's tweets.

First, apply API keys from twitter.

## [1] "Using browser based authentication"

The maximize request is 3200 tweets, I got 815, which is not bad.

## [1] 815

Have a look the first three, then convert all these 815 tweets to a data frame.

## [[1]]
## [1] "realDonaldTrump: Looking forward to hosting our heroes from the Wounded Warrior Project (@WWP) Soldier Ride to the @WhiteHouse on Th… https://t.co/QLC0qFD94x"
## 
## [[2]]
## [1] "realDonaldTrump: Getting ready to meet President al-Sisi of Egypt. On behalf of the United States, I look forward to a long and wonderful relationship."
## 
## [[3]]
## [1] "realDonaldTrump: .@FoxNews from multiple sources: \"There was electronic surveillance of Trump, and people close to Trump. This is unprecedented.\" @FBI"

Text cleaning process, which includes convert all letters to lower case, remove URL, remove anything other than English letter and space, remove stopwords, and extra white space.

Look at these three tweets again.

## <<SimpleCorpus>>
## Metadata:  corpus specific: 1, document level (indexed): 0
## Content:  documents: 3
## 
## [1] look forward host hero wound warrior project wwp soldier ride whitehous th                
## [2] get readi meet presid alsisi egypt behalf unit state look forward long wonder relationship
## [3] foxnew multipl sourc electron surveil peopl close unpreced fbi

Need to replace a few words, such as 'peopl' to 'people', 'whitehous' to 'whitehouse', 'countri' to 'country'.

Building term document matrix

This is a matrix of numbers (0 and 1) that keeps track of which documents in a corpus use which terms.

## <<TermDocumentMatrix (terms: 2243, documents: 815)>>
## Non-/sparse entries: 8419/1819626
## Sparsity           : 100%
## Maximal term length: 29
## Weighting          : term frequency (tf)

As you can see, the term-document matrix is composed of 2243 terms and 815 documents(tweets). It is very sparse, with 100% of the entries being zero. Let's have a look at the terms of 'clinton', 'bad' and 'great', and tweets numbered 21 to 30.

##          Docs
## Terms     21 22 23 24 25 26 27 28 29 30
##   clinton  0  0  0  0  0  0  0  0  0  0
##   great    0  1  0  0  0  0  0  0  1  1
##   bad      0  0  0  0  0  0  0  0  0  0

What are the top frequent terms?

##  [1] "get"     "presid"  "state"   "people"  "clinton" "hillari" "just"   
##  [8] "big"     "us"      "join"    "go"      "time"    "will"    "news"   
## [15] "thank"   "now"     "country" "great"   "today"   "elect"   "job"    
## [22] "watch"   "make"    "america" "pm"      "vote"

A picture worth a thousand words.

Word Cloud

Which word/words are associated with 'will'?

## $will
##  bring  dream wealth   back togeth    god    pig 
##   0.28   0.27   0.27   0.22   0.22   0.20   0.20

Which word/words are associated with 'great'? - This is obvious.

## $great
## america    make  togeth 
##    0.35    0.32    0.25

Which word/ words are associated with 'bad'?

## $bad
##         dude       stupid         judg relationship         fool 
##         0.34         0.34         0.25         0.24         0.24 
##          ban 
##         0.23

Clustering Words

##     get  presid   state  people clinton hillari    just      us    join 
##       1       1       1       2       3       3       4       5       6 
##      go    time    will    news   thank     now country   great   today 
##       1       1       7       4       8       1       2       9      10 
##   elect   watch    make america    vote 
##       1       1       1       1       1

We can see the words in the tweets, words "will, great, thank, us, join, today" are not clustered into any group, "hillari, clinton" are clustered into one group, "now, president, elect, time, go, make, america, state, watch, get, vote" are clustered in one group, "people, country" are clustered into one group, and "just, nows" are clustered into one group.

Clustering Tweets with the k-means Algorithm

##     get presid state people clinton hillari  just    us  join    go  time
## 1 0.000  0.042 0.021  0.729   0.000   0.021 0.167 0.021 0.000 0.083 0.042
## 2 0.027  0.068 0.000  0.000   0.057   0.000 0.070 0.000 0.000 0.043 0.062
## 3 0.100  0.036 0.027  0.091   0.009   0.009 0.045 0.000 0.009 0.073 0.082
## 4 0.047  0.093 0.023  0.070   0.395   1.047 0.093 0.000 0.000 0.070 0.023
## 5 0.146  0.042 0.115  0.125   0.010   0.000 0.052 0.021 0.000 0.115 0.073
## 6 0.130  0.037 0.000  0.000   0.000   0.037 0.056 1.093 0.019 0.056 0.037
## 7 0.023  0.000 0.000  0.000   0.000   0.000 0.047 0.163 1.000 0.000 0.047
## 8 0.212  0.038 0.654  0.058   0.077   0.019 0.077 0.019 0.000 0.058 0.096
##    will  news thank   now country great today elect watch  make america
## 1 0.021 0.042 0.021 0.021   0.542 0.021 0.042 0.083 0.062 0.042   0.000
## 2 0.000 0.073 0.119 0.068   0.000 0.000 0.060 0.068 0.054 0.027   0.019
## 3 1.209 0.027 0.118 0.027   0.055 0.136 0.055 0.027 0.055 0.073   0.027
## 4 0.116 0.000 0.000 0.047   0.000 0.000 0.023 0.047 0.000 0.000   0.000
## 5 0.146 0.104 0.240 0.073   0.042 1.125 0.135 0.031 0.083 0.323   0.281
## 6 0.185 0.019 0.185 0.074   0.185 0.074 0.019 0.000 0.000 0.074   0.037
## 7 0.070 0.000 0.023 0.047   0.000 0.000 0.023 0.000 0.070 0.000   0.047
## 8 0.038 0.019 0.135 0.038   0.038 0.000 0.115 0.115 0.058 0.019   0.019
##    vote
## 1 0.062
## 2 0.000
## 3 0.027
## 4 0.023
## 5 0.052
## 6 0.019
## 7 0.023
## 8 0.635

Check the top three words in every cluster

## cluster 1: people country just 
## cluster 2: thank news just 
## cluster 3: will great thank 
## cluster 4: hillari clinton will 
## cluster 5: great make america 
## cluster 6: us will thank 
## cluster 7: join us will 
## cluster 8: state vote get

I have admit that I can't easily distinguish the clusters of Trump's tweets are of different topics.

Sentiment Analysis

The sentiment analysis algorithm used here is based on NRC Word Emotion Association Lexion, available from the tidytext package which developed by Julia Silge and David Robinson. The algorithm associates with eight basic emotions (anger, fear, anticipation, trust, surprise, sadness, joy, and disgust) and two sentiments (negative and positive).

Sometimes there are tweaks I need to do to get rid of the problem characters.

Have a look the head of Trump's tweets sentiment scores:

##   anger anticipation disgust fear joy sadness surprise trust negative
## 1     2            0       0    1   0       1        0     0        0
## 2     0            2       0    0   1       0        1     3        0
## 3     0            0       0    1   0       0        2     0        0
## 4     0            1       0    0   1       1        1     1        0
## 5     1            1       1    0   1       0        1     2        1
## 6     0            0       0    0   0       0        0     1        1
##   positive
## 1        3
## 2        4
## 3        0
## 4        3
## 5        2
## 6        2

Then combine Trump tweets dataframe and Trump sentiment dataframe together.

Let's visualize it!

Trump's tweets appear more positive than negative, more trust than anger. Has Trump's tweets always been positive or only after he won the election?

The positive sentiment scores are always higher than the negative sentiment scores. And both show decrease over time, especially negaive sentiment experienced a significant drop recently. However, the simple text mining process conducted in this post does not make this conclusion. A more sophisticated text analysis of Trump's tweets by David Robinson found that Trump writes only the angrier half from Android, and another postive half from his staff using iPhone.

The end

I really enjoyed working on Trump Tweets analysis. Learning about text mining and social network analysis is very rewarding. Thanks to Julia Silge and Yanchang Zhao's tutorials to make it possible.