March Madness Analysis

CME Group Foundation Business Analytics Lab

---

About this Notebook

---

• On this notebook we are going to analysis tweets from march madness 2018

• Use regular expression to clean the tweets text

• Familiarize with some natural language processing tools

---

Analytics Toolkit: Require Packages

---

• tidyverse: https://www.tidyverse.org/
• syuzhet: https://github.com/mjockers/syuzhet
• cleanNLP: https://github.com/statsmaths/cleanNLP
• wordcloud: https://cran.r-project.org/web/packages/wordcloud/wordcloud.pdf

# Here we are checking if the package is installed
if(!require("tidyverse")){
  install.packages("tidyverse", dependencies = TRUE)
  library("tidyverse")
}
if(!require("syuzhet")){
  install.packages("syuzhet", dependencies = TRUE)
  library("syuzhet")
}

package <U+393C><U+3E31>syuzhet<U+393C><U+3E32> was built under R version 3.4.4

if(!require("cleanNLP")){
  install.packages("cleanNLP", dependencies = TRUE)
  library("cleanNLP")
}

package <U+393C><U+3E31>cleanNLP<U+393C><U+3E32> was built under R version 3.4.4

if(!require("magrittr")){
  install.packages("magrittr", dependencies = TRUE)
  library("magrittr")
}
if(!require("wordcloud")){
  install.packages("wordcloud", dependencies = TRUE)
  library("wordcloud")
}

---

Data Preparation: Cleaning tweets using regular expressions

---

Reading and inspecting the dataset

tweets <- read_csv("data/sentiment_march_madness.csv")
# Change the tweets IDs from longe integer to characters
tweets$tweet_id <- as.character(tweets$tweet_id)
# Extract and delete the links variable to add it at the end
links <- tweets$links
tweets$links <- NULL
# Inspects the first 10 rows
head(tweets)

Now first we need to extract the text from the raw tweets and clean it using regular expressions

replace_reg <- 'https://t.co/[A-Za-z\\d]+|http://[A-Za-z\\d]+|(pic.twitter.com/[A-Za-z\\d]+)|&amp;|&lt;|&gt;|RT|(.*.)\\.com(.*.)\\S+\\s|[^[:alnum:]]|(http|https)\\S+\\s*|(#|@)\\S+\\s*|\\n|\\"'
tweets <- tweets %>% 
  mutate(text = str_replace_all(text, replace_reg, " ")) %>% 
  mutate(text = iconv(text, from = "ASCII", to = "UTF-8", sub = " "))

Error in mutate_impl(.data, dots) : 
  Evaluation error: embedded nul in string: 'How to draw  kawaii  step by step leassons on Google Play   animejapan  KAWAIIcollection  FinalFour  SisterJean  ORLvUTA  c\003\tc\003)c\0024c\0033c\003\034c\003<c\003+h6\005  nitiasa  precure  d;.i\035"c\003)c\002$c\003\0c\003<c\003\023c\003+c\003\t  c\0025c\0033c\003\ac\003<c\003"c\003<c\003\vc\0033c\0020  CBX SomeoneLikeYou  l\v m\031\024l\025< l\n$k,4l\0024l\035\004 l6\025m\025\030m\0254 l5\034j0\025l0=k/< '.

Here we are going to use the Saif Mohammad’s NRC Emotion lexicon toextract the sentiment of the tweet. The NRC emotion lexicon is a list of words and their associations with eight emotions (anger, fear, anticipation, trust, surprise, sadness, joy, and disgust) and two sentiments (negative and positive).

• Source: http://saifmohammad.com/WebPages/NRC-Emotion-Lexicon.htm

nrc_data <- get_nrc_sentiment(tweets$text)
nrc_data <- as_tibble(nrc_data)

head(nrc_data)

Here we are going to merge the NRC Emotion results with the original data to create a complete dataset.

tweets <- bind_cols(tweets, nrc_data)

To have another metrics for semtiment we are going to use another lexicon developed by Professor Minqing Hu and Professor Bing Liu, from University of Illinois at Chicago.

• Source: http://www.cs.uic.edu/~liub/FBS/sentiment-analysis.html

tweets$sentiment_bing <- get_sentiment(char_v = tweets$text, method="bing", language = "english")

Finally add the links variable and save the new complete dataset

tweets$links <- links

write_csv(tweets, "data/march_madness_sent.csv")

head(tweets[3:10])

---

Sentiment Analysis: Natural Language Processing

---

First lets read the dataset and inspect the first 10 rows

tweets <- read_csv("data/sentiment_march_madness.csv")
tweets$tweet_id <- as.character(tweets$tweet_id)
head(tweets[12:21])

For sentiment analysis we are going to use the cleanNLP package that uses Stanford CoreNLP – Natural language software int he backend. First we need to initialize the CoreNLP engine and create an annotation object using the text column, tweet_id and the other columns are given as metadata

cnlp_init_udpipe()

doc <- cnlp_annotate(input = tweets$text, as_strings = TRUE, doc_ids = tweets$tweet_id, meta = tweets[-c(1,2)])

Distribution of tweet/sentence length, max number of works in a tweet 280

tokens <- cnlp_get_token(doc) %>%
  group_by(id, sid) %>%
  summarize(sent_len = n())

quantile(tokens$sent_len, seq(0,1,0.1))

Here we can see the change of sentiment in the tweets

qplot(x = 1:length(tweets$sentiment_bing), 
      y = tweets$sentiment_bing, 
      geom = "line", 
      xlab = "Narrative Time", 
      ylab = "Emotional Valence", 
      main = "Tweets Sentiment Trajectory")

Here we can find the most used entities from the tweets entity table. The document corpus yields an alternative way to see the underlying topics.

tweets_entities <- cnlp_get_token(doc) %>%
  filter(upos == "NOUN") %>%
  group_by(lemma) %>%
  summarize(count = n()) %>%
  top_n(n = 80, count) %>%
  arrange(desc(count)) %>%
  use_series(lemma)

data_frame(tweets_entities)

Here we creating a high-level summary of the tweets text by extracting all direct object object-dependencies.

tweets_summary <- cnlp_get_dependency(doc, get_token = TRUE) %>%
  left_join(cnlp_get_document(doc)) %>%
  select(id = id, start = word, word = lemma_target) %>%
  left_join(word_frequency) %>%
  filter(frequency < 0.0001) %>%
  select(id, start, word) %$%
  sprintf("%s => %s", start, word)

data_frame(tweets_summary)

Look at the tweets with negative sentiment

angry_tweets <- which(tweets$anger > 0)
data_frame(tweet = tweets$text[angry_tweets][1:2])

Look a tweets with positive sentiment

joy_tweets <- which(tweets$joy > 0)
data_frame(tweet = tweets$text[joy_tweets][5:7])

Lets explore the emotions in the tweets more in-depth. Here we are going to extract the variables regarding emotions and create a subset.

value <- as.double(colSums(prop.table(tweets[, 11:18])))
emotion <- names(tweets)[11:18]
emotion <- factor(emotion, levels = names(tweets)[11:18][order(value, decreasing = FALSE)])
emotions <- data_frame(emotion, percent = value * 100)

head(emotions)

Now we can create a plot of the emotions in the march madness tweets

ggplot(data = emotions, aes(x = emotion, y = percent)) + 
  geom_bar(stat = "identity", aes(fill = emotion)) + 
  scale_fill_brewer(palette="RdYlGn") + 
  coord_flip() +
  xlab("Emotion") +
  ylab("Percentage")

---

Reporting: A Wordcloud from March Madness Tweets

---

Wordclouds are always a fun and engaging way to display data. Here we are going to set some stop words that we dont want in the plot.

remove_words <- c( "twitter", "chicago", "loyola", "ramblers", "loyolaramblers","school", "gonna",
                   "university", "luc", "loyolachicago" , "ramblersmbb", "ncaa","ve","basketball" ,
                   "umichbball", "marchmadness2018", "marchmadness", "final", "marchmaddness",
                   "goblue", "finalfour", "sisterjean", "ncaatournament", "ncaatournament2018",
                   "didn","city", "hey", "day", "college", "games", "tourney", "march", "game")

my_stop_words <- bind_rows(data_frame(word = remove_words, lexicon = c("SMART")), stop_words)

Now lets create a dataframe of words and filter using predefined stop words

twt_text <- tibble(text = tweets$text) %>% 
  unnest_tokens(word, text) %>%
  filter(!word %in% my_stop_words$word, str_detect(word, "[a-z]"))

Set a threshold for the min/max frequency of words and scale of the wordcloud

min_freq = 80
max_words = 100
fig_scale = c(3 , 0.5)

The last step is to create the wordcloud by counting the frequency of the words

twt_text %>%
  anti_join(my_stop_words) %>%
  count(word) %>%
  with(wordcloud(word, n, 
                 scale = fig_scale,
                 min.freq = min_freq,
                 max.words = max_words))

---

Task 1: Data Exploration - Tableau

---

1A) Generally describe the data (summary)

summary(tweets)

   tweet_id             text             username           fullname        
 Length:20187       Length:20187       Length:20187       Length:20187      
 Class :character   Class :character   Class :character   Class :character  
 Mode  :character   Mode  :character   Mode  :character   Mode  :character  
                                                                            
                                                                            
                                                                            
      date               datetime                      verified      
 Min.   :2011-02-09   Min.   :2011-02-09 19:42:51   Min.   :0.00000  
 1st Qu.:2018-03-24   1st Qu.:2018-03-24 11:03:02   1st Qu.:0.00000  
 Median :2018-03-25   Median :2018-03-25 00:22:38   Median :0.00000  
 Mean   :2018-03-18   Mean   :2018-03-18 10:40:33   Mean   :0.06192  
 3rd Qu.:2018-03-25   3rd Qu.:2018-03-25 03:08:07   3rd Qu.:0.00000  
 Max.   :2018-04-06   Max.   :2018-04-06 21:24:17   Max.   :1.00000  
     reply             retweets           favorite           anger       
 Min.   :  0.0000   Min.   :   0.000   Min.   :    0.0   Min.   :0.0000  
 1st Qu.:  0.0000   1st Qu.:   0.000   1st Qu.:    0.0   1st Qu.:0.0000  
 Median :  0.0000   Median :   0.000   Median :    1.0   Median :0.0000  
 Mean   :  0.3467   Mean   :   3.146   Mean   :   15.8   Mean   :0.1342  
 3rd Qu.:  0.0000   3rd Qu.:   0.000   3rd Qu.:    3.0   3rd Qu.:0.0000  
 Max.   :591.0000   Max.   :5143.000   Max.   :32180.0   Max.   :4.0000  
  anticipation       disgust             fear             joy       
 Min.   :0.0000   Min.   :0.00000   Min.   :0.0000   Min.   :0.000  
 1st Qu.:0.0000   1st Qu.:0.00000   1st Qu.:0.0000   1st Qu.:0.000  
 Median :0.0000   Median :0.00000   Median :0.0000   Median :0.000  
 Mean   :0.4359   Mean   :0.07143   Mean   :0.1612   Mean   :0.421  
 3rd Qu.:1.0000   3rd Qu.:0.00000   3rd Qu.:0.0000   3rd Qu.:1.000  
 Max.   :7.0000   Max.   :3.00000   Max.   :6.0000   Max.   :8.000  
    sadness          surprise          trust           negative     
 Min.   :0.0000   Min.   :0.0000   Min.   :0.0000   Min.   :0.0000  
 1st Qu.:0.0000   1st Qu.:0.0000   1st Qu.:0.0000   1st Qu.:0.0000  
 Median :0.0000   Median :0.0000   Median :0.0000   Median :0.0000  
 Mean   :0.1122   Mean   :0.1798   Mean   :0.4806   Mean   :0.2395  
 3rd Qu.:0.0000   3rd Qu.:0.0000   3rd Qu.:1.0000   3rd Qu.:0.0000  
 Max.   :5.0000   Max.   :4.0000   Max.   :7.0000   Max.   :6.0000  
    positive      sentiment_bing   
 Min.   :0.0000   Min.   :-5.0000  
 1st Qu.:0.0000   1st Qu.: 0.0000  
 Median :0.0000   Median : 0.0000  
 Mean   :0.6676   Mean   : 0.5141  
 3rd Qu.:1.0000   3rd Qu.: 1.0000  
 Max.   :9.0000   Max.   :11.0000  

This summary shows the different types of sentiments in the tweets such as anger, anticipation, disgust, fear, joy, sadness, surprise, and trust. It shows how many verified people tweeted about Loyola - which was only one. It shows how many retweets, favorites, and replies there were for the tweets. It also shows how many could seem positive or negative.

1B) Use tableau to create at least 5 plots

knitr::include_graphics("imgs/tweets1.png")

This graph shows which tweets were retweeted, favorited, or replied on.

knitr::include_graphics("imgs/tweets2.png")

This plot shows which accounts had the most retweets. @RamblersMBB had the most followed by @marchmadness. The rest of the accounts had way less retweets than these two accounts.

knitr::include_graphics("imgs/tweets3.png")

This shows how many tweets there were for each measure. There were more favorites than were retweets. More tweets were aksi more positive than negative.

knitr::include_graphics("imgs/tweets4.png")

This graph shows that most of the tweets were in 2018 but some date back to 2011.

knitr::include_graphics("imgs/tweets5.png")

This shows that more tweets were positive rather than negative.

1C) Explain each plot make a relation to date of the tweets/time

## done under each graph.

Task 3: Data Analysis

---

2A)Based on your plots and data description make give a general narrative for the image of loyola in twitter

Loyola received more of a positive image on twitter than a negative one. We can see that the account for the Men’s basketball team had the biggest impact on twitter users as it had the most retweets out of any account looked at. People were happy that we were winning and were anxious for each game. This all points toward a positive image for Loyola.

2B) Use descriptive statistics to backup your arguments

summary(tweets)

   tweet_id             text             username           fullname        
 Length:20187       Length:20187       Length:20187       Length:20187      
 Class :character   Class :character   Class :character   Class :character  
 Mode  :character   Mode  :character   Mode  :character   Mode  :character  
                                                                            
                                                                            
                                                                            
      date               datetime                      verified      
 Min.   :2011-02-09   Min.   :2011-02-09 19:42:51   Min.   :0.00000  
 1st Qu.:2018-03-24   1st Qu.:2018-03-24 11:03:02   1st Qu.:0.00000  
 Median :2018-03-25   Median :2018-03-25 00:22:38   Median :0.00000  
 Mean   :2018-03-18   Mean   :2018-03-18 10:40:33   Mean   :0.06192  
 3rd Qu.:2018-03-25   3rd Qu.:2018-03-25 03:08:07   3rd Qu.:0.00000  
 Max.   :2018-04-06   Max.   :2018-04-06 21:24:17   Max.   :1.00000  
     reply             retweets           favorite           anger       
 Min.   :  0.0000   Min.   :   0.000   Min.   :    0.0   Min.   :0.0000  
 1st Qu.:  0.0000   1st Qu.:   0.000   1st Qu.:    0.0   1st Qu.:0.0000  
 Median :  0.0000   Median :   0.000   Median :    1.0   Median :0.0000  
 Mean   :  0.3467   Mean   :   3.146   Mean   :   15.8   Mean   :0.1342  
 3rd Qu.:  0.0000   3rd Qu.:   0.000   3rd Qu.:    3.0   3rd Qu.:0.0000  
 Max.   :591.0000   Max.   :5143.000   Max.   :32180.0   Max.   :4.0000  
  anticipation       disgust             fear             joy       
 Min.   :0.0000   Min.   :0.00000   Min.   :0.0000   Min.   :0.000  
 1st Qu.:0.0000   1st Qu.:0.00000   1st Qu.:0.0000   1st Qu.:0.000  
 Median :0.0000   Median :0.00000   Median :0.0000   Median :0.000  
 Mean   :0.4359   Mean   :0.07143   Mean   :0.1612   Mean   :0.421  
 3rd Qu.:1.0000   3rd Qu.:0.00000   3rd Qu.:0.0000   3rd Qu.:1.000  
 Max.   :7.0000   Max.   :3.00000   Max.   :6.0000   Max.   :8.000  
    sadness          surprise          trust           negative     
 Min.   :0.0000   Min.   :0.0000   Min.   :0.0000   Min.   :0.0000  
 1st Qu.:0.0000   1st Qu.:0.0000   1st Qu.:0.0000   1st Qu.:0.0000  
 Median :0.0000   Median :0.0000   Median :0.0000   Median :0.0000  
 Mean   :0.1122   Mean   :0.1798   Mean   :0.4806   Mean   :0.2395  
 3rd Qu.:0.0000   3rd Qu.:0.0000   3rd Qu.:1.0000   3rd Qu.:0.0000  
 Max.   :5.0000   Max.   :4.0000   Max.   :7.0000   Max.   :6.0000  
    positive      sentiment_bing   
 Min.   :0.0000   Min.   :-5.0000  
 1st Qu.:0.0000   1st Qu.: 0.0000  
 Median :0.0000   Median : 0.0000  
 Mean   :0.6676   Mean   : 0.5141  
 3rd Qu.:1.0000   3rd Qu.: 1.0000  
 Max.   :9.0000   Max.   :11.0000  

This summary shows that the max was 9 for positive which was higher than negative which was 6. It also showed there was more for joy than anger. This shows that the school had more of a positive image than a negative one at this time.

2C)Any recommendations to Loyola’s marketing team

## Loyola’s marketing team can use their Cinderella Story to paint Loyola in a positive image. There was constant talk about this school doing everything right. They can use that narrative to bring in more students to make this school prosperous.

Task 3: Watson Analysis

---

3A)Use watson analytics to explore the data

3B)Give at least 3 plots or discoveries using watson. Explain your findings.

knitr::include_graphics("imgs/tweets6.png")

This shows what drove trust for the team. Positive, joy, and anticipation fostered trust in the team. This means they were trusting the team to win which gave a positive response of happiness.

knitr::include_graphics("imgs/tweets7.png")

Most of the tweets about joy were towards the end of the month around the time of the elite 8 which lead to the final four. This shows the positive image loyola had at the time.

knitr::include_graphics("imgs/tweets8.png")

This shows there were tweets about trust for years, but most were about the team in 2018. It seems like most were before the tournament started probably talking about what may happen.