Project Proposal

Climate affects nearly every aspect of our lives, from our food sources to our transport infrastructure, from what clothes we wear, to where we go on holiday. It has a huge effect on our livelihoods, our health, and our future. In this project, we attempt to understand the popular sentiment among public based on their overall opinion. We would also explore and identify the frequently used terms related to global warming, for example “climate change”, “environment”, “temperature” etc. In summary, we would extract tweets using Twitter API, classify opinions into categories, create function to calculate score and visualize data to organize results to express views or emotions.

Analytics Plan

Gather Data (Twitter API)
Transform Data (Dplyr, Tidyverse)
Visual Analysis (Ggplot2, Wordcloud)
Machine Learning (H20 or Keras)

Evaluation Plan

Text Mining and NLP Methods (Sentiment Score/Occurence)
Qualitative and Quantitative Measures (Word Clouds/Frequency Plots)
Sophisticated Deep Learning Techniques (H20 or Keras)

Peer Reviewed Comments

Can Tweets be used for Forecasting/Predicting Future?
What Geography/Data are you taking into account?
What Related Words will you use to identify Global Warming?
Which Libraries Used for Sentiment Analysis?
Which ML approaches are you considering for analysis?
Will you use Multi-Lingual, Multipolar data?
Will you employe traditional ML approaches?
Will you consider working on Retweets as well?
How many Data Points are you considering for analysis?
How will you Categorize Comments for Tweets?
How will you differentiate between Fake and Real Accounts?

Project Description

Sentiment analysis gives us insights into the things that automate mining of attitudes, opinions, views and emotions from text, speech, tweets and database sources. In this project, we use tweets extracted using Twitter API, store tweets as text data, classify opinions in text into categories like positive, or negative or neutral, create a function to calculate the score of each type of opinion in the text and try to explore and visualize as much as we can, using R libraries. Tweets can be imported into R using Twitter API, then the text data has to be cleaned before analysis, for example removing emoticons, removing URLs, etc. Along the way, we use sophisticated AI-related techniques (deep learning) like H20 or Keras.

Pulling Tweets into R (Linking Twitter’s API):

Apply for a Twitter Developer Account
Create a Twitter API App
Installing/Loading twitteR package
Linking API into R Studio
Setting Up Authentication
Pulling Related Tweets

Data Preprocessing

Text cleaning is one of the text mining processing to clean the words or other component that is hard to analyze or figure the meaning of the text. Text data or sentence data contains white spaces, punctuations, stop words etc. These characters do not inform much information and are hard to process for the sentiment analysis. For example, English stop words like “the”, “is” etc. do not tell you much information about the sentiment of the text, entities mentioned in the text, or relationships between those entities

Cleaning Date/Time
Checking Duplicate Data
Dealing with Missing Data
Dealing with Categorical data
Removing Whitespaces and Newlines
Removing Special Characters
Scaling the Features
Segmenting Data
Standardizing Casing

Exploratory Data Analysis

Most Liked Tweets

Most Retweeted Tweets

Most Active Users

Data Modelling and Evaluation

Simple RNN

LSTM

Bi-Directional LSTM

GRU

Bi-Directional GRU

1D-Convnet

Results and Key-Takeaways

Twitter data is a powerful source of information on a wide list of topics. More than half of the tweets examined support for a negative sentiment associated with Global Warming. Futhermore, climate change is now one of the two most important issues in politics. In this project, We used this data to find trends related to specific topics, measure popular sentiment, obtain feedback on past decisions and also help make future decisions. We further trained and evaluated our model in Keras with different settings and evaluation measures.

Challenges:

Identifying Subjective Parts
Domain Dependence
Sarcasm Detection
Thwarted Expressions
Order Depedence
Entity Recognition
Handling Comparisons

Next Steps and Future Work

Sentiment Analysis has many different applications in several fields:

Applications in ECommerce: Internet has a large collection of reviews and feedbacks on almost everything. This includes product reviews, feedbacks on political issues, comments about services, etc
Applications in Smart Homes: Based on the current sentiment or emotion of the user, the home could alter its ambiance to create a soothing and peaceful environment
Applications in Marketing: Businesses wish to extract sentiment from the online reviews in order to improve their products and in turn their reputation and help in customer satisfaction.The online opinion decides the success or failure of their product.
Applications in Entertainment: A sentiment predictor system can be helpful in recommender systems as well. The recommender system will not recommend items that receive a lot of negative feedback or fewer ratings. In online communication, we come across abusive language and other negative elements. These can be detected simply by identifying a highly negative sentiment and correspondingly taking action against it.

References

Keras 2.7.0 documentation. [online] Available at: https://www.rdocumentation.org/packages/keras/versions/2.7.0 [Accessed 09 Nov. 2021].
Tidyverse 1.3.1 documentation. [online] Available at: https://www.tidyverse.org/packages/ [Accessed 05 Nov. 2021].
Ggplot2 3.3.5 documentaion. [online] Available at: https://ggplot2.tidyverse.org/ [Accessed 07 Nov. 2021].
RMarkdown 2.1.1 documentaion. [online] Available at: https://rmarkdown.rstudio.com/index.html [Accessed 12 Dec. 2021].
KableExtra 1.3.4 documentaion. [online] Available at: https://cran.r-project.org/web/packages/kableExtra/vignettes/awesome_table_in_html.html [Accessed 12 Dec. 2021].
twitterR 1.1.9 documentaion. [online] Available at: https://www.rdocumentation.org/packages/twitteR/versions/1.1.9 [Accessed 12 Dec. 2021].
Getting Started with Twitter API [online] Available at: https://developer.twitter.com/en/docs/tutorials/getting-started-with-r-and-v2-of-the-twitter-api [Accessed 12 Dec. 2021].
Pulling Tweets into R [online] Available at: https://towardsdatascience.com/pulling-tweets-into-r-e17d4981cfe2 [Accessed 10 Dec. 2021].

Sentiment Analysis using Keras

Team Members