#Conversational AI: An Analysis of Public Twitter Sentiment Surrounding ChatGPT
Jessica Bethea
2023-02-01
1. PREPARE
1a. Background
ChatGPT is an artificially intelligent chatbot created by OpenAI Labs. As a language model, its purpose is to generate natural language text that is similar to human-written text. It can be fine-tuned for a wide variety of natural language processing tasks, such as language translation, text summarization, solving math problems, creating/debugging/explaining code, and conversation generation. Chat GPT is designed to be used in conversational interfaces, where it can generate human-like responses based on user input.
While it has many capabilities, since its launch in November 2022, conversations surrounding the implications that its use may have been innumerable. AI has been used in recent by individuals and organizations alike to simplify and automate tasks that would otherwise be tedious. As time goes on, AI technology has become more widely accessible. However, while AI is geared toward convenience, there are concerns that come with it such as privacy, transparency in use, bias and much more.
For ChatGPT in particular, one of the main concerns have been among educational organizations and institutions has been students having a new way to engage in academic dishonesty by having ChatGPT create solutions to their work for them. Additionally, the idea of widespread ChatGPT use has seen conversations regarding the replacement of humans in certain job roles. On the flip side, however, there has been conversation that ChatGPT’s capabilities could potentially be leveraged in a positive way.
The overarching goal of this study is to explore these conversations further to get an idea of how people feel about the inception of ChatGPT.
1b. Research Questions
This analysis will address three questions:
- What is the overall public sentiment expressed toward ChatGPT?
- How do these sentiments vary between the AFINN, NRC, BING and Loughran lexicons?
- What words are the most frequently used in the conversations surrounding ChatGPT?
1c. Data Source and Analysis
Data for this study will be collected and wrangled using the Twitter API.
1d. Set up
Loading Packages
#install.packages("dplyr")
#install.packages("readr")
#install.packages("tidyr")
#install.packages("rtweet")
#install.packages("writexl")
#install.packages("readxl")
#install.packages("tidytext")
#install.packages("textdata")
#install.packages("ggplot2")
#install.packages("textdata")
#install.packages("scales")library(dplyr)##
## Attaching package: 'dplyr'## The following objects are masked from 'package:stats':
##
## filter, lag## The following objects are masked from 'package:base':
##
## intersect, setdiff, setequal, unionlibrary(readr)
library(tidyr)
library(rtweet)
library(writexl)
library(readxl)
library(tidytext)
library(textdata)
library(ggplot2)
library(textdata)
library(scales)##
## Attaching package: 'scales'## The following object is masked from 'package:readr':
##
## col_factorLoading the Twitter API
#app_name <- "eci588research"
#api_key <- "mzNQRmiYPV8x8ZwobqvO3oRuN"
#api_secret_key <- "oVZHB7J40GXhnGcLgDSCkiTP0j7fn9boOfVpeKmZzWvET2dKJR"
#access_token <- "1562797323440103428-zXikORK587QKMWuJQJ49zICSOg4h3C"
#access_token_secret <- "ONUgitOJ1eZgsf1Y8WcgZu0aLB8NGSiRwxSSNXlOACqZS"
#token <- create_token(
#app = app_name,
#consumer_key = api_key,
#consumer_secret = api_secret_key,
#access_token = access_token,
#access_secret = access_token_secret)2. WRANGLE
I will first begin by pulling the data from Twitter to see what we get. I anticipate there will be a lot of tweets to sort through regarding ChatGPT, so I will start with pulling around 3,000 just to get a peak at what is there.
#chatgpt_tweets <- search_tweets(q = "#ChatGPT", n=3000)Based on the return, ChatGPT seems like a pretty hot topic already, with 2,969 tweets being returned out of the 3,000 that were asked for. Now, let’s wrangle a bit further.
I noticed that the screen names were hidden in the original search data frame that was created. After researching, this is due to how the newer versions of rtweet function. I used the users_data function to gather information about the users from the chatgpt_all_tweets data that was created and then added the screen name column using the cbind function.
#chatgpt_screen <- users_data(chatgpt_tweets) %>%
#select(screen_name) #chatgpt_tweets <- cbind(chatgpt_screen, chatgpt_tweets)
#chatgpt_tweetsNow, I will save the data frame in my project folder by writing an xlsx.
#write_xlsx(chatgpt_tweets, "chatgpt_tweets.xlsx")2a. Tidy Text
Now that I have my tweets, I will go ahead and tidy the text to see what information can be gathered. First, I read the data frame I saved to get another look.
chatgpt_tweets <- read_xlsx("/Users/shiyan/Desktop/Jessica/chatgpt_tweets.xlsx")
chatgpt_tweets## # A tibble: 2,970 × 44
## screen_name created_at id id_str text full_text truncated
## <chr> <dttm> <dbl> <chr> <chr> <chr> <lgl>
## 1 justinsuntron 2023-02-04 10:04:37 1.62e18 162181… "As … "As an i… FALSE
## 2 EricTopol 2023-02-03 16:51:47 1.62e18 162155… "Ins… "Insight… FALSE
## 3 doctorow 2023-02-03 14:02:49 1.62e18 162150… "#Ch… "#ChatGP… FALSE
## 4 JohnNew94455527 2023-02-04 16:29:39 1.62e18 162190… "RT … "RT @Tec… FALSE
## 5 CTL_UT 2023-02-04 16:29:31 1.62e18 162190… "RT … "RT @abm… FALSE
## 6 JustHayden3 2023-02-04 16:29:13 1.62e18 162190… "Aft… "After m… FALSE
## 7 ftmlifer 2023-02-04 16:29:02 1.62e18 162190… "Any… "Anyone … FALSE
## 8 Phillip95539827 2023-02-04 16:29:01 1.62e18 162190… "RT … "RT @Tec… FALSE
## 9 UniMatrixZ0 2023-02-04 16:29:00 1.62e18 162190… "\"#… "\"#Goog… FALSE
## 10 marcosx86 2023-02-04 16:28:29 1.62e18 162190… "Ouv… "Ouvindo… FALSE
## # … with 2,960 more rows, and 37 more variables: entities <lgl>, source <chr>,
## # in_reply_to_status_id <dbl>, in_reply_to_status_id_str <chr>,
## # in_reply_to_user_id <dbl>, in_reply_to_user_id_str <chr>,
## # in_reply_to_screen_name <chr>, geo <lgl>, coordinates <lgl>, place <lgl>,
## # contributors <lgl>, is_quote_status <lgl>, retweet_count <dbl>,
## # favorite_count <dbl>, favorited <lgl>, favorited_by <lgl>, retweeted <lgl>,
## # scopes <lgl>, lang <chr>, possibly_sensitive <lgl>, …Next, I will filter the language to English to return tweets that only contain English. Additionally, I am only interested in the screen_name, created_at and the text columns for the purpose of this study. I will use these columns to create the chatgpt_text data frame.
chatgpt_tweets## # A tibble: 2,970 × 44
## screen_name created_at id id_str text full_text truncated
## <chr> <dttm> <dbl> <chr> <chr> <chr> <lgl>
## 1 justinsuntron 2023-02-04 10:04:37 1.62e18 162181… "As … "As an i… FALSE
## 2 EricTopol 2023-02-03 16:51:47 1.62e18 162155… "Ins… "Insight… FALSE
## 3 doctorow 2023-02-03 14:02:49 1.62e18 162150… "#Ch… "#ChatGP… FALSE
## 4 JohnNew94455527 2023-02-04 16:29:39 1.62e18 162190… "RT … "RT @Tec… FALSE
## 5 CTL_UT 2023-02-04 16:29:31 1.62e18 162190… "RT … "RT @abm… FALSE
## 6 JustHayden3 2023-02-04 16:29:13 1.62e18 162190… "Aft… "After m… FALSE
## 7 ftmlifer 2023-02-04 16:29:02 1.62e18 162190… "Any… "Anyone … FALSE
## 8 Phillip95539827 2023-02-04 16:29:01 1.62e18 162190… "RT … "RT @Tec… FALSE
## 9 UniMatrixZ0 2023-02-04 16:29:00 1.62e18 162190… "\"#… "\"#Goog… FALSE
## 10 marcosx86 2023-02-04 16:28:29 1.62e18 162190… "Ouv… "Ouvindo… FALSE
## # … with 2,960 more rows, and 37 more variables: entities <lgl>, source <chr>,
## # in_reply_to_status_id <dbl>, in_reply_to_status_id_str <chr>,
## # in_reply_to_user_id <dbl>, in_reply_to_user_id_str <chr>,
## # in_reply_to_screen_name <chr>, geo <lgl>, coordinates <lgl>, place <lgl>,
## # contributors <lgl>, is_quote_status <lgl>, retweet_count <dbl>,
## # favorite_count <dbl>, favorited <lgl>, favorited_by <lgl>, retweeted <lgl>,
## # scopes <lgl>, lang <chr>, possibly_sensitive <lgl>, …chatgpt_text <-
chatgpt_tweets %>%
filter(lang == "en") %>%
select(screen_name, created_at, text)
chatgpt_text## # A tibble: 2,051 × 3
## screen_name created_at text
## <chr> <dttm> <chr>
## 1 justinsuntron 2023-02-04 10:04:37 "As an industry-leading decentralized st…
## 2 EricTopol 2023-02-03 16:51:47 "Insightful on #ChatGPT, LLMs for scienc…
## 3 doctorow 2023-02-03 14:02:49 "#ChatGPT as #ConMan\n\n* give people wh…
## 4 JohnNew94455527 2023-02-04 16:29:39 "RT @TechCompanyNews: New #Startup https…
## 5 CTL_UT 2023-02-04 16:29:31 "RT @abmarkman: @toddkashdan @2GoYH @reb…
## 6 JustHayden3 2023-02-04 16:29:13 "After months of meditating on where hum…
## 7 ftmlifer 2023-02-04 16:29:02 "Anyone dare to ask #chatgpt what the be…
## 8 Phillip95539827 2023-02-04 16:29:01 "RT @TechCompanyNews: New #Startup https…
## 9 UniMatrixZ0 2023-02-04 16:29:00 "\"#Google invests almost $400 million i…
## 10 JenningThommas 2023-02-04 16:28:24 "RT @TechCompanyNews: New #Startup https…
## # … with 2,041 more rowswrite_xlsx(chatgpt_text, "chatgpt_text.xlsx")2b. Tokenize Text
tweet_tokens <-
chatgpt_text %>%
unnest_tokens(output = word,
input = text)
tweet_tokens## # A tibble: 47,433 × 3
## screen_name created_at word
## <chr> <dttm> <chr>
## 1 justinsuntron 2023-02-04 10:04:37 as
## 2 justinsuntron 2023-02-04 10:04:37 an
## 3 justinsuntron 2023-02-04 10:04:37 industry
## 4 justinsuntron 2023-02-04 10:04:37 leading
## 5 justinsuntron 2023-02-04 10:04:37 decentralized
## 6 justinsuntron 2023-02-04 10:04:37 stablecoin
## 7 justinsuntron 2023-02-04 10:04:37 financial
## 8 justinsuntron 2023-02-04 10:04:37 infrastructure
## 9 justinsuntron 2023-02-04 10:04:37 tron
## 10 justinsuntron 2023-02-04 10:04:37 will
## # … with 47,423 more rowstidy_chatgpt <-
tweet_tokens %>%
anti_join(stop_words, by = "word")
tidy_chatgpt## # A tibble: 27,470 × 3
## screen_name created_at word
## <chr> <dttm> <chr>
## 1 justinsuntron 2023-02-04 10:04:37 industry
## 2 justinsuntron 2023-02-04 10:04:37 leading
## 3 justinsuntron 2023-02-04 10:04:37 decentralized
## 4 justinsuntron 2023-02-04 10:04:37 stablecoin
## 5 justinsuntron 2023-02-04 10:04:37 financial
## 6 justinsuntron 2023-02-04 10:04:37 infrastructure
## 7 justinsuntron 2023-02-04 10:04:37 tron
## 8 justinsuntron 2023-02-04 10:04:37 provide
## 9 justinsuntron 2023-02-04 10:04:37 ai
## 10 justinsuntron 2023-02-04 10:04:37 oriented
## # … with 27,460 more rowscount(tidy_chatgpt, word, sort = T)## # A tibble: 5,824 × 2
## word n
## <chr> <int>
## 1 chatgpt 1633
## 2 https 1444
## 3 rt 1306
## 4 ai 1263
## 5 t.co 936
## 6 bepaitoken 515
## 7 power 505
## 8 unlock 493
## 9 revolutionising 492
## 10 visit 492
## # … with 5,814 more rowshttps, rt and t.co appear in the most common words list, but are part of web addresses, so I will remove them.
tidy_chatgpt <-
tweet_tokens %>%
anti_join(stop_words, by = "word") %>%
filter(!word == "https") %>%
filter(!word == "t.co") %>%
filter(!word == "rt")
tidy_chatgpt## # A tibble: 23,784 × 3
## screen_name created_at word
## <chr> <dttm> <chr>
## 1 justinsuntron 2023-02-04 10:04:37 industry
## 2 justinsuntron 2023-02-04 10:04:37 leading
## 3 justinsuntron 2023-02-04 10:04:37 decentralized
## 4 justinsuntron 2023-02-04 10:04:37 stablecoin
## 5 justinsuntron 2023-02-04 10:04:37 financial
## 6 justinsuntron 2023-02-04 10:04:37 infrastructure
## 7 justinsuntron 2023-02-04 10:04:37 tron
## 8 justinsuntron 2023-02-04 10:04:37 provide
## 9 justinsuntron 2023-02-04 10:04:37 ai
## 10 justinsuntron 2023-02-04 10:04:37 oriented
## # … with 23,774 more rows3. EXPLORE
In my exploration, I want to visualize the most common words from the tidy_chatgpt data frame. I will do this using a word cloud. First, I need to install the wordcloud2 package for the wordcloud, and the RColorBrewer package for colors.
#install.packages("wordcloud2")
library(wordcloud2)
#install.packages("RColorBrewer")
library(RColorBrewer)3a. Visualizing Top Tokens
top_tokens <- tidy_chatgpt %>%
ungroup ()%>%
count(word, sort = TRUE) %>%
top_n(50)## Selecting by nwordcloud2(top_tokens)Based on the word cloud, some of the most popular words are:
power
ai
artificial intelligence
unleash
solving
unlock
technology
driven
All of these words are pretty decent representations of what ChatGPT is/does, but I am interested in seeing the sentiments surrounding ChatGPT using these words and others that were identified within the data frames.
Additionally, since ChatGPT has seemingly become popular over the last few months, I’m also interested in seeing how the frequency of tweets has changed over time.
ts_plot(tidy_chatgpt, by = "days", color = "pink")
4. MODEL
4a. Sentiment Analysis
I will be comparing sentiments using the NRC, AFINN, Loughran and BING lexicons. To start, they will be loaded.
afinn <- get_sentiments("afinn")
nrc <- get_sentiments("nrc")
bing <- get_sentiments("bing")
loughran <- get_sentiments("loughran")sentiment_afinn <- inner_join(tweet_tokens, afinn, by = "word")
sentiment_bing <- inner_join(tweet_tokens, bing, by = "word")
sentiment_nrc <- inner_join(tweet_tokens, nrc, by = "word")
sentiment_loughran <- inner_join(tweet_tokens, loughran, by = "word")sentiment_afinn <- select(sentiment_afinn, "word", "value")
sentiment_afinn## # A tibble: 2,855 × 2
## word value
## <chr> <dbl>
## 1 support 2
## 2 accept 1
## 3 reject -1
## 4 worry -3
## 5 true 2
## 6 ethical 2
## 7 confidence 2
## 8 good 3
## 9 luck 3
## 10 nice 3
## # … with 2,845 more rowssentiment_bing <- select(sentiment_bing, "word", "sentiment")
sentiment_bing## # A tibble: 2,859 × 2
## word sentiment
## <chr> <chr>
## 1 leading positive
## 2 insightful positive
## 3 support positive
## 4 reject negative
## 5 worry negative
## 6 ethical positive
## 7 lie negative
## 8 lie negative
## 9 confidence positive
## 10 good positive
## # … with 2,849 more rowssentiment_loughran <- select(sentiment_loughran, "word", "sentiment")
sentiment_loughran## # A tibble: 2,162 × 2
## word sentiment
## <chr> <chr>
## 1 leading positive
## 2 insightful positive
## 3 reject negative
## 4 worry negative
## 5 lie negative
## 6 lie negative
## 7 good positive
## 8 might uncertainty
## 9 best positive
## 10 almost uncertainty
## # … with 2,152 more rowssentiment_nrc <- select(sentiment_nrc, "word", "sentiment")
sentiment_nrc## # A tibble: 7,488 × 2
## word sentiment
## <chr> <chr>
## 1 leading trust
## 2 provide positive
## 3 provide trust
## 4 payment negative
## 5 framework trust
## 6 technology positive
## 7 reject anger
## 8 reject fear
## 9 reject negative
## 10 reject sadness
## # … with 7,478 more rowsNow that we have selected the word and sentiment for each column for easier reading, we’ll calculate sentiment for the BING lexicon.
summary_bing <- sentiment_bing %>%
count(sentiment, sort = TRUE) %>%
spread(sentiment, n) %>%
mutate(sentiment = positive - negative) %>%
mutate(lexicon = "bing") %>%
relocate(lexicon)
summary_bing## # A tibble: 1 × 4
## lexicon negative positive sentiment
## <chr> <int> <int> <int>
## 1 bing 1501 1358 -143Based on the results from the BING lexicon, sentiments toward ChatGPT are overall negative. Next is the AFINN lexicon.
summary_afinn <- sentiment_afinn %>%
summarise(sentiment = sum(value)) %>%
mutate(lexicon = "AFINN") %>%
relocate(lexicon)
summary_afinn## # A tibble: 1 × 2
## lexicon sentiment
## <chr> <dbl>
## 1 AFINN 1592For the AFINN sentiment, the overall sentiment is positive.
summary_nrc <- sentiment_nrc %>%
count(sentiment, sort = TRUE) %>%
filter(sentiment == "positive" | sentiment == "negative") %>%
spread(sentiment, n) %>%
mutate(sentiment = positive - negative) %>%
mutate(lexicon = "nrc") %>%
relocate(lexicon)
summary_nrc## # A tibble: 1 × 4
## lexicon negative positive sentiment
## <chr> <int> <int> <int>
## 1 nrc 922 2421 1499Again, the NRC sentiment is overwhelmingly positive at 1,499.
Lastly, I’ll calculate the Loughran sentiment.
summary_loughran <- sentiment_loughran %>%
count(sentiment, sort = TRUE) %>%
filter(sentiment == "positive" | sentiment == "negative") %>%
spread(sentiment, n) %>%
mutate(sentiment = positive - negative) %>%
mutate(lexicon = "loughran") %>%
relocate(lexicon)
summary_loughran## # A tibble: 1 × 4
## lexicon negative positive sentiment
## <chr> <int> <int> <int>
## 1 loughran 867 1085 218Based on the calculations, the loughran lexicon also returns a positive sentiment of 218.
5. COMMUNICATE
5a. Select
Because I want to ensure that I have really captured the overall twitter sentiment of the ChatGPT language processing model, I will use this section to solidify the following questions:
- What is the overall public Twitter sentiment expressed toward ChatGPT?
- How do these sentiments vary between the AFINN, NRC, BING and Loughran lexicons?
5b. Polish
To polish, I will select the id_str and text columns from the original chatgpt_tweets data frame in order to prepare to conduct a new analysis to calculate a sentiment score for each whole tweet, rather than individual words. This is because the sentiment of an individual word may not fully convey the meaning or intent when used within a phrase in conjunction with other words.
chatgpt_text2 <-
chatgpt_tweets %>%
filter(lang == "en") %>%
select(id_str, text)
chatgpt_text2## # A tibble: 2,051 × 2
## id_str text
## <chr> <chr>
## 1 1621811959883776003 "As an industry-leading decentralized stablecoin financi…
## 2 1621552039607287809 "Insightful on #ChatGPT, LLMs for science\n\"Soon this t…
## 3 1621509516230803458 "#ChatGPT as #ConMan\n\n* give people what they ask for\…
## 4 1621908857755766790 "RT @TechCompanyNews: New #Startup https://t.co/hBh2qfdJ…
## 5 1621908820325801989 "RT @abmarkman: @toddkashdan @2GoYH @rebeccamcinroy @KUT…
## 6 1621908747168763905 "After months of meditating on where humanity might find…
## 7 1621908700322488320 "Anyone dare to ask #chatgpt what the best L1 will be in…
## 8 1621908694446268421 "RT @TechCompanyNews: New #Startup https://t.co/hBh2qfdJ…
## 9 1621908690600075269 "\"#Google invests almost $400 million in #ChatGPT rival…
## 10 1621908540750192642 "RT @TechCompanyNews: New #Startup https://t.co/hBh2qfdJ…
## # … with 2,041 more rowsNow, I’ll attach the sentiment scores from AFINN and unnest tokens, and clean up any words that don’t add to meaning. AFINN uses a -5 to 5 point scale to assign sentiment and is the preferred lexicon for analyzing tweets, compared to the other three lexicons. As we have learned, analyzing sentiment is most effective when utilizing lexicon dictionaries that they were adapted for, in order to get the best possible result.
sentiment_afinn2 <- chatgpt_text2 %>%
unnest_tokens(output = word,
input = text) %>%
anti_join(stop_words, by = "word") %>%
filter(!word == "https") %>%
filter(!word == "t.co") %>%
filter(!word == "rt") %>%
inner_join(afinn, by = "word")
sentiment_afinn2## # A tibble: 1,801 × 3
## id_str word value
## <chr> <chr> <dbl>
## 1 1621552039607287809 support 2
## 2 1621552039607287809 accept 1
## 3 1621552039607287809 reject -1
## 4 1621509516230803458 worry -3
## 5 1621509516230803458 true 2
## 6 1621509516230803458 ethical 2
## 7 1621509516230803458 confidence 2
## 8 1621908820325801989 luck 3
## 9 1621908820325801989 nice 3
## 10 1621908747168763905 god 1
## # … with 1,791 more rowsCalculating scores for each tweet.
afinn_score <- sentiment_afinn2 %>%
group_by(id_str) %>%
summarise(value = sum(value))
afinn_score## # A tibble: 1,333 × 2
## id_str value
## <chr> <dbl>
## 1 1621470934451326977 -1
## 2 1621470953908879361 7
## 3 1621470995499581441 1
## 4 1621470998620155904 2
## 5 1621471006505455617 1
## 6 1621471008212553728 1
## 7 1621471079104675840 1
## 8 1621471091888889856 1
## 9 1621471113523109888 1
## 10 1621471169357438976 4
## # … with 1,323 more rowsNow calculating sentiment and filtering out any sentiment scores that are equal to zero.
afinn_sentiment <- afinn_score %>%
filter(value != 0) %>%
mutate(sentiment = if_else(value < 0, "negative", "positive"))
afinn_sentiment## # A tibble: 1,304 × 3
## id_str value sentiment
## <chr> <dbl> <chr>
## 1 1621470934451326977 -1 negative
## 2 1621470953908879361 7 positive
## 3 1621470995499581441 1 positive
## 4 1621470998620155904 2 positive
## 5 1621471006505455617 1 positive
## 6 1621471008212553728 1 positive
## 7 1621471079104675840 1 positive
## 8 1621471091888889856 1 positive
## 9 1621471113523109888 1 positive
## 10 1621471169357438976 4 positive
## # … with 1,294 more rowsFinally calculating the sentiment ratio.
afinn_ratio <- afinn_sentiment %>%
count(sentiment) %>%
spread(sentiment, n) %>%
mutate(ratio = negative/positive)
afinn_ratio## # A tibble: 1 × 3
## negative positive ratio
## <int> <int> <dbl>
## 1 188 1116 0.168Based on the calculation, positive sentiment leads with 1116, with negative at 188, leading to a ratio of .1684.
Now, I will visualize these results using a pie-chart from the ggplot package.
afinn_counts <- afinn_sentiment %>%
count(sentiment)
afinn_counts %>%
ggplot(aes(x="", y=n, fill=sentiment)) +
geom_bar(width = .6, stat = "identity") +
labs(title = "ChatGPT",
subtitle = "Proportion of Positive & Negative Tweets") +
coord_polar(theta = "y") +
theme_void() +
scale_fill_manual(values=c("orange", 'lightblue'))
Based on the pie chart and the above data, the tweets surrounding ChatGPT are mostly positive, using the AFINN lexicon.
To address the second research question, I will compare the sentiment counts for each of the lexicons in addition to the AFINN that was calculated above.
summary_afinn2 <- sentiment_afinn2 %>%
filter(value != 0) %>%
mutate(sentiment = if_else(value < 0, "negative", "positive")) %>%
count(sentiment, sort = TRUE) %>%
mutate(method = "AFINN")
summary_bing2 <- sentiment_bing %>%
count(sentiment, sort = TRUE) %>%
mutate(method = "bing")
summary_nrc2 <- sentiment_nrc %>%
filter(sentiment %in% c("positive", "negative")) %>%
count(sentiment, sort = TRUE) %>%
mutate(method = "nrc")
summary_loughran2 <- sentiment_loughran %>%
filter(sentiment %in% c("positive", "negative")) %>%
count(sentiment, sort = TRUE) %>%
mutate(method = "loughran") summary_sentiment <- bind_rows(summary_afinn2,
summary_bing2,
summary_nrc2,
summary_loughran2) %>%
arrange(method) %>%
relocate(method)
summary_sentiment## # A tibble: 8 × 3
## method sentiment n
## <chr> <chr> <int>
## 1 AFINN positive 1459
## 2 AFINN negative 342
## 3 bing negative 1501
## 4 bing positive 1358
## 5 loughran positive 1085
## 6 loughran negative 867
## 7 nrc positive 2421
## 8 nrc negative 922total_counts <- summary_sentiment %>%
group_by(method) %>%
summarise(total = sum(n))
total_counts## # A tibble: 4 × 2
## method total
## <chr> <int>
## 1 AFINN 1801
## 2 bing 2859
## 3 loughran 1952
## 4 nrc 3343Out of the 4
sentiment_counts <- left_join(summary_sentiment, total_counts)## Joining, by = "method"sentiment_counts## # A tibble: 8 × 4
## method sentiment n total
## <chr> <chr> <int> <int>
## 1 AFINN positive 1459 1801
## 2 AFINN negative 342 1801
## 3 bing negative 1501 2859
## 4 bing positive 1358 2859
## 5 loughran positive 1085 1952
## 6 loughran negative 867 1952
## 7 nrc positive 2421 3343
## 8 nrc negative 922 3343sentiment_percents <- sentiment_counts %>%
mutate(percent = n/total * 100)
sentiment_percents## # A tibble: 8 × 5
## method sentiment n total percent
## <chr> <chr> <int> <int> <dbl>
## 1 AFINN positive 1459 1801 81.0
## 2 AFINN negative 342 1801 19.0
## 3 bing negative 1501 2859 52.5
## 4 bing positive 1358 2859 47.5
## 5 loughran positive 1085 1952 55.6
## 6 loughran negative 867 1952 44.4
## 7 nrc positive 2421 3343 72.4
## 8 nrc negative 922 3343 27.6sentiment_percents %>%
ggplot(aes(x= percent, y = sentiment, fill=sentiment)) +
geom_bar(width = .8, stat = "identity") +
facet_wrap(~method, ncol = 1) +
coord_flip() +
labs(title = "Twitter Public Sentiment of ChatGPT",
x = "Percentage of Words", y = "") +
scale_fill_manual(values=c('orange', 'lightblue'))
Though the AFINN is best used for Twitter text analysis, it is clear that 3 out of the 4 models returned a positive sentiment value for tweets regarding ChatGPT, with Bing returning a negative sentiment of 52%, which is still close to being halfway positive.
5c. Narrate
Purpose
The purpose of this study was to gauge public sentiment on the ChatGPT AI tool that has recently been released. Due to its capabilities such as language translation, text summarization, solving math problems, creating/debugging/explaining code, and conversation generation, which are currently unriveled by previous AI language processing models, conversation surrounding the tool has been immense. I wanted to get a feel for:
- What the public sentiment is surrounding the ChatGPT tool?
- What words within text were the most frequently used that could help assign sentiment?
- How sentiment varied, if at all, across the lexicon dictionaries (AFINN, BING, Loughran, or NRC) used for text analysis?
With ChatGPT having such important capabilities, in its short amount of time in existence has resulted in feelings ranging from excitement at the innovation, to concern of how it might be used or misused, particularly in academic settings or among individuals within organizations fearful of “human replacement”.
Methods
The data used for this analysis was gathered directly from the Twitter API in real time. The data return provided a wide variety of information in addition to text such as retweets, screen names, quotes, metadata and much more.
Due to the sheer amount of posts available, for the purpose of this study, 3,000 tweets were pulled, with the API returning 2,989 posts across 44 variables based on the search parameters.
All 44 of these variables were not necessary to analyze sentiment, and as such, the data was cleaned and wrangled to focus only on posts that were in the English language and limited to the screenname, text and post time features.
From there, the text was tokenized to get a feel for which words/tokens stood out the most within the posts and was visualized using a word cloud.
Finally, sentiments were calculated using the AFINN, Bing, NRC and Loughran lexicons.
Findings
Ultimately, the analysis showed that as of now, the sentiment toward the ChatGPT tool is overwhelmingly positive across all tested lexicons.
Discussion
Because the technology is so new, and regularly goes through periods where it is difficult to access due to server overloads from interested parties, the data pulled from this study so far might not be representative of true feelings toward the technology. It seems as though many of the tweets are from those that are familiar with or work regularly with AI technology and as such, tend to be eager toward advancements within the field.
In my own experience, as someone who is a Higher Ed instructional designer, these conversations are also in fact happening outside of Twitter, many of which with strong concern about the negative impacts along with excitement at its potential use. Some universities have already begun to implement clauses in their academic integrity policies regard the use of the ChatGPT tool. In my opinion, I think as it is implemented and used on wider scale across the board, the possibility of a change or variation in the sentiment we’ve seen in this study is strong.
In future studies, I would be really interested in perhaps comparing the sentiment between now and some time futher in the future. Additionally, I am interested in comparing sentiments among certain industries, maybe education vs. tech or research to see if sentiments changed based on use cases across different organizations. I would also love to see how individual users, such as those outside of organizations and research groups feel about the technology. It would be really interesting to see how the general public interacts with the tool in daily life.
Lastly, I am highly interested in doing a comparative sentiment against another natural language processing tool that could possibly be an alternative to the ChatGPT tool. As it stands right now, there are hardly any competitors with the sheer capabilities or amount of attention that ChatGPT is garnering at the moment (though there are talks of a possible Google competitor coming soon), which is the reason for the standalone sentiment analysis in this study, but I think it could definitely make a good future study.