10A Sentiment Analysis

Task

Assignment: Sentiment Analysis with Text Mining with R In Chapter 2 of Text Mining with R, the authors introduce sentiment analysis. Your task is to reproduce and extend that example.

Step 1 — Reproduce the Base Example Obtain the primary example code from Chapter 2.

Create a Quarto (.qmd) file that successfully runs this code.

Include a proper citation to the book and the original example code.

Step 2 — Extend the Analysis Extend the original example in two ways:

Use a different text corpus of your choosing (e.g., news articles, book reviews, speeches, social media, etc.).

Add at least one additional sentiment lexicon, which may come from:

Another R package, or

A lexicon you locate through external research.

You should clearly explain how the results differ from the original example.

Codebase

The code in Step 1 is from Silge and Robinson (2017), Chapter 2, ‘Sentiment analysis with tidy data’, https://www.tidytextmining.com/sentiment.html

Step 1: Reproduce the base example

The sentiments datasets

library(tidytext)
library(janeaustenr)
library(dplyr)

Attaching package: 'dplyr'

The following objects are masked from 'package:stats':

    filter, lag

The following objects are masked from 'package:base':

    intersect, setdiff, setequal, union

library(stringr)
library(ggplot2)
library(syuzhet)

get_sentiments("nrc")

# A tibble: 13,872 × 2
   word        sentiment
   <chr>       <chr>    
 1 abacus      trust    
 2 abandon     fear     
 3 abandon     negative 
 4 abandon     sadness  
 5 abandoned   anger    
 6 abandoned   fear     
 7 abandoned   negative 
 8 abandoned   sadness  
 9 abandonment anger    
10 abandonment fear     
# ℹ 13,862 more rows

get_sentiments("afinn")

# A tibble: 2,477 × 2
   word       value
   <chr>      <dbl>
 1 abandon       -2
 2 abandoned     -2
 3 abandons      -2
 4 abducted      -2
 5 abduction     -2
 6 abductions    -2
 7 abhor         -3
 8 abhorred      -3
 9 abhorrent     -3
10 abhors        -3
# ℹ 2,467 more rows

get_sentiments("bing")

# A tibble: 6,786 × 2
   word        sentiment
   <chr>       <chr>    
 1 2-faces     negative 
 2 abnormal    negative 
 3 abolish     negative 
 4 abominable  negative 
 5 abominably  negative 
 6 abominate   negative 
 7 abomination negative 
 8 abort       negative 
 9 aborted     negative 
10 aborts      negative 
# ℹ 6,776 more rows

Sentiment analysis with inner join

tidy_books <- austen_books() %>%
  group_by(book) %>%
  mutate(
    linenumber = row_number(),
    chapter = cumsum(str_detect(text, 
                                regex("^chapter [\\divxlc]", 
                                      ignore_case = TRUE)))) %>%
  ungroup() %>%
  unnest_tokens(word, text)

nrc_joy <- get_sentiments("nrc") %>% 
  filter(sentiment == "joy")

tidy_books %>%
  filter(book == "Emma") %>%
  inner_join(nrc_joy) %>%
  count(word, sort = TRUE)

Joining with `by = join_by(word)`

# A tibble: 301 × 2
   word          n
   <chr>     <int>
 1 good        359
 2 friend      166
 3 hope        143
 4 happy       125
 5 love        117
 6 deal         92
 7 found        92
 8 present      89
 9 kind         82
10 happiness    76
# ℹ 291 more rows

library(tidyr)

jane_austen_sentiment <- tidy_books %>%
  inner_join(get_sentiments("bing")) %>%
  count(book, index = linenumber %/% 80, sentiment) %>%
  pivot_wider(names_from = sentiment, values_from = n, values_fill = 0) %>% 
  mutate(sentiment = positive - negative)

Joining with `by = join_by(word)`

Warning in inner_join(., get_sentiments("bing")): Detected an unexpected many-to-many relationship between `x` and `y`.
ℹ Row 435434 of `x` matches multiple rows in `y`.
ℹ Row 5051 of `y` matches multiple rows in `x`.
ℹ If a many-to-many relationship is expected, set `relationship =
  "many-to-many"` to silence this warning.

library(ggplot2)
ggplot(jane_austen_sentiment, aes(index, sentiment, fill = book)) +
  geom_col(show.legend = FALSE) +
  facet_wrap(~book, ncol = 2, scales = "free_x")

Comparing the three sentiment dictionaries

pride_prejudice <- tidy_books %>% 
  filter(book == "Pride & Prejudice")

pride_prejudice

# A tibble: 122,204 × 4
   book              linenumber chapter word     
   <fct>                  <int>   <int> <chr>    
 1 Pride & Prejudice          1       0 pride    
 2 Pride & Prejudice          1       0 and      
 3 Pride & Prejudice          1       0 prejudice
 4 Pride & Prejudice          3       0 by       
 5 Pride & Prejudice          3       0 jane     
 6 Pride & Prejudice          3       0 austen   
 7 Pride & Prejudice          7       1 chapter  
 8 Pride & Prejudice          7       1 1        
 9 Pride & Prejudice         10       1 it       
10 Pride & Prejudice         10       1 is       
# ℹ 122,194 more rows

afinn <- pride_prejudice %>% 
  inner_join(get_sentiments("afinn")) %>% 
  group_by(index = linenumber %/% 80) %>% 
  summarise(sentiment = sum(value)) %>% 
  mutate(method = "AFINN")

Joining with `by = join_by(word)`

bing_and_nrc <- bind_rows(
  pride_prejudice %>% 
    inner_join(get_sentiments("bing")) %>%
    mutate(method = "Bing et al."),
  pride_prejudice %>% 
    inner_join(get_sentiments("nrc") %>% 
                 filter(sentiment %in% c("positive", 
                                         "negative"))
    ) %>%
    mutate(method = "NRC")) %>%
  count(method, index = linenumber %/% 80, sentiment) %>%
  pivot_wider(names_from = sentiment,
              values_from = n,
              values_fill = 0) %>% 
  mutate(sentiment = positive - negative)

Joining with `by = join_by(word)`
Joining with `by = join_by(word)`

Warning in inner_join(., get_sentiments("nrc") %>% filter(sentiment %in% : Detected an unexpected many-to-many relationship between `x` and `y`.
ℹ Row 215 of `x` matches multiple rows in `y`.
ℹ Row 5178 of `y` matches multiple rows in `x`.
ℹ If a many-to-many relationship is expected, set `relationship =
  "many-to-many"` to silence this warning.

bind_rows(afinn, 
          bing_and_nrc) %>%
  ggplot(aes(index, sentiment, fill = method)) +
  geom_col(show.legend = FALSE) +
  facet_wrap(~method, ncol = 1, scales = "free_y")

get_sentiments("nrc") %>% 
  filter(sentiment %in% c("positive", "negative")) %>% 
  count(sentiment)

# A tibble: 2 × 2
  sentiment     n
  <chr>     <int>
1 negative   3316
2 positive   2308

get_sentiments("bing") %>% 
  count(sentiment)

# A tibble: 2 × 2
  sentiment     n
  <chr>     <int>
1 negative   4781
2 positive   2005

Most common positive and negative words

bing_word_counts <- tidy_books %>%
  inner_join(get_sentiments("bing")) %>%
  count(word, sentiment, sort = TRUE) %>%
  ungroup()

Joining with `by = join_by(word)`

Warning in inner_join(., get_sentiments("bing")): Detected an unexpected many-to-many relationship between `x` and `y`.
ℹ Row 435434 of `x` matches multiple rows in `y`.
ℹ Row 5051 of `y` matches multiple rows in `x`.
ℹ If a many-to-many relationship is expected, set `relationship =
  "many-to-many"` to silence this warning.

bing_word_counts

# A tibble: 2,585 × 3
   word     sentiment     n
   <chr>    <chr>     <int>
 1 miss     negative   1855
 2 well     positive   1523
 3 good     positive   1380
 4 great    positive    981
 5 like     positive    725
 6 better   positive    639
 7 enough   positive    613
 8 happy    positive    534
 9 love     positive    495
10 pleasure positive    462
# ℹ 2,575 more rows

bing_word_counts %>%
  group_by(sentiment) %>%
  slice_max(n, n = 10) %>% 
  ungroup() %>%
  mutate(word = reorder(word, n)) %>%
  ggplot(aes(n, word, fill = sentiment)) +
  geom_col(show.legend = FALSE) +
  facet_wrap(~sentiment, scales = "free_y") +
  labs(x = "Contribution to sentiment",
       y = NULL)

custom_stop_words <- bind_rows(tibble(word = c("miss"),  
                                      lexicon = c("custom")), 
                               stop_words)

custom_stop_words

# A tibble: 1,150 × 2
   word        lexicon
   <chr>       <chr>  
 1 miss        custom 
 2 a           SMART  
 3 a's         SMART  
 4 able        SMART  
 5 about       SMART  
 6 above       SMART  
 7 according   SMART  
 8 accordingly SMART  
 9 across      SMART  
10 actually    SMART  
# ℹ 1,140 more rows

library(wordcloud)

Loading required package: RColorBrewer

tidy_books %>%
  anti_join(stop_words) %>%
  count(word) %>%
  with(wordcloud(word, n, max.words = 100))

Joining with `by = join_by(word)`

library(reshape2)

Attaching package: 'reshape2'

The following object is masked from 'package:tidyr':

    smiths

tidy_books %>%
  inner_join(get_sentiments("bing")) %>%
  count(word, sentiment, sort = TRUE) %>%
  acast(word ~ sentiment, value.var = "n", fill = 0) %>%
  comparison.cloud(colors = c("gray20", "gray80"),
                   max.words = 100)

Joining with `by = join_by(word)`

Warning in inner_join(., get_sentiments("bing")): Detected an unexpected many-to-many relationship between `x` and `y`.
ℹ Row 435434 of `x` matches multiple rows in `y`.
ℹ Row 5051 of `y` matches multiple rows in `x`.
ℹ If a many-to-many relationship is expected, set `relationship =
  "many-to-many"` to silence this warning.

Step 2: Extend the Analysis

2.1 Social Media Dataset

For the extended analysis, I will use the Social Media Sentiments Analysis Dataset from Kaggle,https://www.kaggle.com/datasets/kashishparmar02/social-media-sentiments-analysis-dataset. Unlike the original example, which analyzes Jane Austen’s novels, this dataset contains user-generated social media posts from different platforms, including Instagram, Facebook, and Twitter.

posts<-read.csv("https://raw.githubusercontent.com/lhamo07/Data-607-Assignment/refs/heads/main/sentimentdataset.csv")%>%
  rename_with(tolower) %>%
    mutate(post_id = row_number())
tidy_words<-posts%>%
  # there is white space after each platform name so trim them before applying filter
      mutate(platform = str_trim(platform))%>%
  select(post_id,platform,text)%>%
  # format sentence into one word per row.
    unnest_tokens(word, text) 

tidy_words %>%
  filter(platform == "Instagram") %>%
  inner_join(nrc_joy, by = "word") %>%
  count(word, sort = TRUE)

           word  n
1           joy 11
2       feeling 10
3      laughter  9
4           art  8
5        beauty  8
6      surprise  8
7        garden  6
8    excitement  5
9        friend  5
10         hope  5
11      journey  5
12     optimism  5
13     serenity  5
14  celebration  4
15   confidence  4
16        dance  4
17   enthusiasm  4
18         glow  4
19         good  4
20    happiness  4
21      hopeful  4
22         love  4
23      victory  4
24     birthday  3
25  celebrating  3
26         dawn  3
27      delight  3
28      embrace  3
29     favorite  3
30        found  3
31      harmony  3
32  masterpiece  3
33        music  3
34        peace  3
35        pride  3
36     symphony  3
37     tranquil  3
38    whimsical  3
39   admiration  2
40 appreciation  2
41    beautiful  2
42      charity  2
43       create  2
44      endless  2
45     festival  2
46   friendship  2
47    gratitude  2
48     inspired  2
49       joyful  2
50     peaceful  2
51    reverence  2
52       thrill  2
53     treasure  2
54      triumph  2
55       weight  2
56     yearning  2
57 accomplished  1
58  achievement  1
59       amused  1
60    amusement  1
61      amusing  1
62         baby  1
63      blessed  1
64    blessings  1
65        bloom  1
66    childhood  1
67        choir  1
68    companion  1
69   completing  1
70  connoisseur  1
71    delicious  1
72   enthusiast  1
73      excited  1
74         food  1
75     football  1
76       freely  1
77          gem  1
78         gift  1
79      glimmer  1
80         grow  1
81         heal  1
82        hobby  1
83      immerse  1
84 independence  1
85  inspiration  1
86         luck  1
87      magical  1
88    orchestra  1
89       quaint  1
90      romance  1
91        share  1
92      special  1
93      success  1
94          sun  1
95     sunshine  1
96    thrilling  1
97  tranquility  1
98   unexpected  1
99     vacation  1

Using nrc_joy to count the number of joy word available in the instagram post. joy word appeared to be 11, feeling(10), laughter (9) and so on.

social_media_sentiment <- tidy_words %>%
  inner_join(get_sentiments("bing")) %>%
  count(platform,post_id, sentiment) %>%
  pivot_wider(names_from = sentiment, values_from = n, values_fill = 0) %>% 
  mutate(sentiment = positive - negative)

Joining with `by = join_by(word)`

Warning in inner_join(., get_sentiments("bing")): Detected an unexpected many-to-many relationship between `x` and `y`.
ℹ Row 3314 of `x` matches multiple rows in `y`.
ℹ Row 2018 of `y` matches multiple rows in `x`.
ℹ If a many-to-many relationship is expected, set `relationship =
  "many-to-many"` to silence this warning.

inner_join() keeps only words that exist in the Bing lexicon. A post with no Bing-listed words has no rows after the join, so it never reaches count() and never appears in the result.

ggplot(social_media_sentiment, aes(post_id, sentiment, fill = platform)) +
  geom_col(show.legend = FALSE) +
  facet_wrap(~platform, ncol = 2, scales = "free_x")

Step 2: Analyze your social media dataset

2.2 Additional Sentiment Method: syuzhet

In addition to the three original sentiment lexicons, I used syuzhet lexicon, it gives sentiment to sentence rather than word.

get_sentiment_dictionary("syuzhet") %>%
  head(10)

          word value
1      abandon -0.75
2    abandoned -0.50
3    abandoner -0.25
4  abandonment -0.25
5     abandons -1.00
6     abducted -1.00
7    abduction -0.50
8   abductions -1.00
9     aberrant -0.60
10  aberration -0.80

syuzhet_sentiment <- posts %>%
  mutate(
    platform = str_trim(platform),
    sentiment = get_sentiment(text, method = "syuzhet")
  ) %>%
  select(post_id, platform, text, sentiment)

ggplot(syuzhet_sentiment, aes(post_id, sentiment, fill = platform)) +
  geom_col(show.legend = FALSE) +
  facet_wrap(~platform, ncol = 2, scales = "free_x")

Comparing the four sentiment dictionaries

instagram_data <- tidy_words %>% 
  filter(platform == "Instagram")

instagram_posts <- syuzhet_sentiment %>%
  filter(platform == "Instagram")

afinn <- instagram_data %>%
  inner_join(get_sentiments("afinn"), by = "word") %>%
  group_by(post_id) %>%
  summarise(sentiment = sum(value), .groups = "drop") %>%
  mutate(method = "AFINN")

bing <- instagram_data %>%
  inner_join(get_sentiments("bing"), by = "word") %>%
  count(post_id, sentiment) %>%
  pivot_wider(
    names_from = sentiment,
    values_from = n,
    values_fill = 0
  ) %>%
  mutate(sentiment = positive - negative) %>%
  select(post_id, sentiment) %>%
  mutate(method = "Bing")

nrc <- instagram_data %>%
  inner_join(
    get_sentiments("nrc") %>%
      filter(sentiment %in% c("positive", "negative")),
    by = "word"
  ) %>%
  count(post_id, sentiment) %>%
  pivot_wider(
    names_from = sentiment,
    values_from = n,
    values_fill = 0
  ) %>%
  mutate(sentiment = positive - negative) %>%
  select(post_id, sentiment) %>%
  mutate(method = "NRC")

Warning in inner_join(., get_sentiments("nrc") %>% filter(sentiment %in% : Detected an unexpected many-to-many relationship between `x` and `y`.
ℹ Row 46 of `x` matches multiple rows in `y`.
ℹ Row 1998 of `y` matches multiple rows in `x`.
ℹ If a many-to-many relationship is expected, set `relationship =
  "many-to-many"` to silence this warning.

syuzhet <- instagram_posts %>%
  mutate(
    sentiment = get_sentiment(text, method = "syuzhet"),
    method = "Syuzhet"
  ) %>%
  select(post_id, sentiment, method)

all_sentiments <- bind_rows(
  afinn,
  bing,
  nrc,
  syuzhet
)

ggplot(all_sentiments,
       aes(x = post_id, y = sentiment, color = method)) +
  geom_col(show.legend = FALSE) +
  facet_wrap(~method, ncol = 2, scales = "free_y") +
  labs(
    title = "Comparison of Sentiment Lexicons for Instagram Posts",
    x = "Post ID",
    y = "Sentiment Score"
  ) +
  theme_minimal()

The four sentiment methods generate very different absolute scores, but they follow almost the same relative pattern across the Instagram timeline, similar to what was observed in Pride and Prejudice novel. However, the Instagram sentiment chart shows much greater fluctuations, with frequent changes in sentiment compared to the more gradual peaks and dips in the novel.

In Pride and Prejudice, AFINN scores peaked as high as +80. In contrast, on Instagram, AFINN’s highest peak barely exceeds +10, while Bing and NRC peak around +6.

In the previous chart, the NRC lexicon remained almost entirely above the zero line, indicating a strong positive baseline shift. However, in the Instagram dataset, NRC scores drop significantly below zero, following the negative dips observed in the other lexicons, particularly around Post 300.

The Syuzhet lexicon uses a much narrower scoring range, approximately -2.5 to +2.5, because it assigns decimal weights to words rather than whole-number scores. Despite this smaller scale, it closely follows the same overall patterns and movements as AFINN and Bing.

bing_word_counts_social_media <- tidy_words %>%
  inner_join(get_sentiments("bing")) %>%
  count(word, sentiment, sort = TRUE) %>%
  ungroup()

Joining with `by = join_by(word)`

Warning in inner_join(., get_sentiments("bing")): Detected an unexpected many-to-many relationship between `x` and `y`.
ℹ Row 3314 of `x` matches multiple rows in `y`.
ℹ Row 2018 of `y` matches multiple rows in `x`.
ℹ If a many-to-many relationship is expected, set `relationship =
  "many-to-many"` to silence this warning.

bing_word_counts

# A tibble: 2,585 × 3
   word     sentiment     n
   <chr>    <chr>     <int>
 1 miss     negative   1855
 2 well     positive   1523
 3 good     positive   1380
 4 great    positive    981
 5 like     positive    725
 6 better   positive    639
 7 enough   positive    613
 8 happy    positive    534
 9 love     positive    495
10 pleasure positive    462
# ℹ 2,575 more rows

bing_word_counts_social_media %>%
  group_by(sentiment) %>%
  slice_max(n, n = 10) %>% 
  ungroup() %>%
  mutate(word = reorder(word, n)) %>%
  ggplot(aes(n, word, fill = sentiment)) +
  geom_col(show.legend = FALSE) +
  facet_wrap(~sentiment, scales = "free_y") +
  labs(x = "Contribution to sentiment",
       y = NULL)

In the austen_books the most common positive word is well and negative is miss whereas in social media dataset the most common positive word is like and negative is lost.

Conclusion

In conclusion, all four lexicons produce relatively similar sentiment patterns, despite differences in their scoring scales. In Pride and Prejudice, sentiment scores are mostly positive, which may reflect the nature of the novel’s storyline. In contrast, social media posts express a wider range of emotions, including both positive and negative sentiments. As a result, Instagram sentiment scores fluctuate more frequently, reflecting the varied emotions expressed in social media content.

Citation and Reference

• Silge, J., & Robinson, D. (2017). Text mining with R: A tidy approach. O’Reilly Media. https://www.tidytextmining.com/

• Anthropic. (2026). Claude (Claude Sonnet 5.5) [Large language model]. https://claude.ai/chat/911d9fd6-ccd1-4343-b8e7-dad5c7a96854