Project 4

Approach: Classifying Emotional Tone in Relationship Communication

Data Collection

For this project, I plan to construct a dataset using text data from publicly available online forums such as Reddit, particularly communities focused on relationships and advice. Comments and posts will serve as the unit of analysis.

I will manually label a subset of the data into the three predefined categories (supportive, neutral, dismissive). This approach will allow me to ensure consistency in how emotional tone is defined and applied. I expect the dataset to include approximately 200–500 labeled observations, which should be sufficient for exploratory modeling and evaluation.

Data Preparation

Before modeling, I will preprocess the text data to make it suitable for analysis. This will include converting all text to lowercase, removing punctuation and special characters, and eliminating common stopwords such as “the” and “and.” I will also tokenize the text into individual words.

After cleaning, I will transform the text into a numerical format using a Term Frequency–Inverse Document Frequency (TF-IDF) representation. This will allow the models to interpret the importance of different words across messages.

Modeling Approach

I plan to use this standard classification models for this task:

• Naive Bayes, which is commonly used for text classification due to its simplicity and effectiveness.

Evaluation Strategy

To evaluate model performance, I will split the dataset into training and testing sets. The training set will be used to build the models, while the testing set will be used to assess how well the models generalize to new data.

I will evaluate performance using accuracy as a general measure, along with precision and recall to better understand how well the model identifies supportive and dismissive messages. I will also use a confusion matrix to analyze where the model makes mistakes, such as confusing neutral messages with dismissive ones.

Expected Challenges

There are several challenges I anticipate in this project. First, emotional tone is inherently subjective, so labeling may vary depending on interpretation. Second, there may be class imbalance if certain types of messages appear more frequently than others. Third, individual messages may lack sufficient context, making classification more difficult. Finally, subtle language cues such as sarcasm or indirect phring may be challenging for the models to detect.

To address these challenges, I will define clear labeling guidelines and interpret my results with these limitations in mind.

Expected Outcomes

I expect this project to show that machine learning models can identify patterns in language associated with different emotional tones. While I do not expect perfect accuracy due to the subjective nature of communication, I anticipate that the models will still provide meaningful insights into how supportive and dismissive language can be distinguished.

Real-World Applications

This type of classification has several potential real-world applications. It could be used to enhance moderation systems in online communities, support mental health and well-being tools, and improve communication analysis in areas such as customer service or relationship platforms.

Codebase

# Load libraries
library(tidyverse)

── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ──
✔ dplyr     1.2.1     ✔ readr     2.1.6
✔ forcats   1.0.1     ✔ stringr   1.6.0
✔ ggplot2   4.0.3     ✔ tibble    3.3.1
✔ lubridate 1.9.4     ✔ tidyr     1.3.2
✔ purrr     1.2.2     
── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
✖ dplyr::filter() masks stats::filter()
✖ dplyr::lag()    masks stats::lag()
ℹ Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errors

library(tidytext)
library(tidymodels)

── Attaching packages ────────────────────────────────────── tidymodels 1.5.0 ──
✔ broom        1.0.12     ✔ rsample      1.3.2 
✔ dials        1.4.3      ✔ tailor       0.1.0 
✔ infer        1.1.0      ✔ tune         2.1.0 
✔ modeldata    1.5.1      ✔ workflows    1.3.0 
✔ parsnip      1.5.0      ✔ workflowsets 1.1.1 
✔ recipes      1.3.2      ✔ yardstick    1.4.0 
── Conflicts ───────────────────────────────────────── tidymodels_conflicts() ──
✖ scales::discard() masks purrr::discard()
✖ dplyr::filter()   masks stats::filter()
✖ recipes::fixed()  masks stringr::fixed()
✖ dplyr::lag()      masks stats::lag()
✖ yardstick::spec() masks readr::spec()
✖ recipes::step()   masks stats::step()

relationship_texts <- tibble(
  text = c(
# supportive
    "I understand why you feel hurt and your feelings are valid.",
    "You deserve someone who listens to you and respects your emotions.",
    "That sounds really difficult and I am sorry you are going through it.",
    "It makes sense that you would feel confused after that conversation.",
    "You handled this with a lot of patience and care.",
    "I think you should talk to them honestly because your feelings matter.",
    "It is okay to need reassurance in a relationship.",
    "You are not wrong for wanting kindness and consistency.",
    "I can see why this situation would make you anxious.",
    "You deserve to be treated with respect.",
    "A healthy relationship should make both people feel heard.",
    "Your reaction makes sense given what happened.",
    "I hope you are able to get the support you need.",
    "It sounds like you tried your best to communicate clearly.",
    "You should not have to beg someone to care about your feelings.",
    "I would feel hurt too if I were in that situation.",
    "It is reasonable to ask for clarity and honesty.",
    "You are allowed to set boundaries.",
    "This sounds painful and I hope things get better for you.",
    "You deserve compassion, not judgment.",
    # neutral
    "It depends on what happened before this conversation.",
    "There may be more context needed before judging the situation.",
    "It is hard to know without hearing both sides.",
    "This could be a misunderstanding between both people.",
    "The best next step may be to have a direct conversation.",
    "More information would help explain their behavior.",
    "This situation could go either way depending on their intentions.",
    "It might be useful to wait and see if the pattern continues.",
    "There are several possible explanations for what happened.",
    "Both people may need time to think about the issue.",
    "It is unclear whether this was intentional or accidental.",
    "A calm conversation could help clarify the problem.",
    "The situation sounds complicated and may need more context.",
    "It depends on whether this has happened before.",
    "There is not enough information to say for sure.",
    "They may have meant something different than how it sounded.",
    "It could be helpful to ask them what they meant.",
    "This might be a communication issue rather than a relationship issue.",
    "It is possible that both people are interpreting things differently.",
    "The outcome depends on how they respond when you bring it up.",
     # dismissive
    "You are overreacting and making this a bigger deal than it is.",
    "That sounds like your problem, not theirs.",
    "I do not think your feelings matter that much here.",
    "You are being too sensitive about something small.",
    "Just get over it and move on.",
    "This is not worth being upset about.",
    "You are making drama out of nothing.",
    "They probably did nothing wrong and you are reading into it.",
    "You need to stop taking everything so personally.",
    "Honestly, this sounds like an immature reaction.",
    "You should not expect people to care about every little feeling.",
    "This is a you problem, not a relationship problem.",
    "You are acting like this is worse than it really is.",
    "I think you are being dramatic.",
    "There is no reason to be hurt by that.",
    "You are creating problems where there are none.",
    "It is not that serious.",
    "You need thicker skin.",
    "This seems like attention seeking.",
    "You are blowing this way out of proportion."
  ),
  tone = c(
    rep("supportive", 20),
    rep("neutral", 20),
    rep("dismissive", 20)
  )
)

relationship_texts <- relationship_texts %>%
  mutate(tone = as.factor(tone))

tidy_text <- relationship_texts %>%
  unnest_tokens(word, text)
tidy_text <- tidy_text %>%
  filter(!word %in% c("you", "am", "im", "ive", "dont"))

Let’s see which words are most important for each tone category.

tf_idf_words <- tidy_text %>%
  count(tone, word, sort = TRUE) %>%
  bind_tf_idf(word, tone, n)

head(tf_idf_words)

# A tibble: 6 × 6
  tone       word      n     tf   idf tf_idf
  <fct>      <chr> <int>  <dbl> <dbl>  <dbl>
1 supportive to       10 0.0568 0     0     
2 dismissive are       9 0.0608 0     0     
3 neutral    it        9 0.0479 0     0     
4 supportive i         8 0.0455 0.405 0.0184
5 dismissive is        7 0.0473 0     0     
6 dismissive this      7 0.0473 0     0     

Let’s visualize this.

tf_idf_words %>%
  group_by(tone) %>%
  slice_max(tf_idf, n = 5) %>%
  ungroup() %>%
  ggplot(aes(x = reorder(word, tf_idf), y = tf_idf, fill = tone)) +
  geom_col(show.legend = FALSE) +
  facet_wrap(~ tone, scales = "free") +
  coord_flip() +
  labs(
    title = "Top TF-IDF Words by Emotional Tone",
    x = "Word",
    y = "TF-IDF"
  )

library(tidyverse)
library(tidytext)
library(tidymodels)

set.seed(123)

data_split <- initial_split(relationship_texts, prop = 0.8, strata = tone)

train_data <- training(data_split)
test_data <- testing(data_split)

library(textrecipes)

text_recipe <- recipe(tone ~ text, data = train_data) %>%
  step_tokenize(text) %>%
  step_stopwords(text) %>%
  step_tfidf(text)

Let’s connect the workflow

library(discrim)

Attaching package: 'discrim'

The following object is masked from 'package:dials':

    smoothness

nb_model <- naive_Bayes() %>%
  set_engine("klaR")

workflow_nb <- workflow() %>%
  add_recipe(text_recipe) %>%
  add_model(nb_model)
nb_fit <- fit(workflow_nb, data = train_data)
predictions <- predict(nb_fit, test_data) %>%
  bind_cols(test_data)

predictions

# A tibble: 12 × 3
   .pred_class text                                                        tone 
   <fct>       <chr>                                                       <fct>
 1 neutral     I understand why you feel hurt and your feelings are valid. supp…
 2 neutral     That sounds really difficult and I am sorry you are going … supp…
 3 neutral     You are not wrong for wanting kindness and consistency.     supp…
 4 neutral     It is reasonable to ask for clarity and honesty.            supp…
 5 supportive  More information would help explain their behavior.         neut…
 6 neutral     It is unclear whether this was intentional or accidental.   neut…
 7 neutral     The situation sounds complicated and may need more context. neut…
 8 supportive  This might be a communication issue rather than a relation… neut…
 9 neutral     You are overreacting and making this a bigger deal than it… dism…
10 neutral     They probably did nothing wrong and you are reading into i… dism…
11 neutral     You are acting like this is worse than it really is.        dism…
12 neutral     You are creating problems where there are none.             dism…

conf_mat(predictions, truth = tone, estimate = .pred_class)

            Truth
Prediction   dismissive neutral supportive
  dismissive          0       0          0
  neutral             4       2          4
  supportive          0       2          0

accuracy(predictions, truth = tone, estimate = .pred_class)

# A tibble: 1 × 3
  .metric  .estimator .estimate
  <chr>    <chr>          <dbl>
1 accuracy multiclass     0.167

The first model achieved an accuracy of 33%, correctly classifying 1 out of 3 test observations. The model tended to predict the “supportive” class for all observations, indicating a bias likely caused by the small dataset size and limited variation in training examples.

In order to train the model better, I added more data. After expanding the dataset to include a balanced set of 60 labeled observations, the model’s accuracy decreased compared to the initial smaller dataset. This is expected, as the larger dataset provides a more realistic and challenging classification task. With more diverse examples, the model is less likely to rely on simple patterns and must better distinguish between subtle differences in emotional tones.

The results highlight that classifying emotional tone is inherently complex, as many words and phrases appear across multiple categories. This leads to overlap between classes, particularly between neutral and supportive messages.