Introduction

Online platforms have become central arenas for the negotiation of gender, identity, and power. Among these platforms, Reddit hosts a wide range of communities dedicated to discussions of men’s issues, feminism, and contemporary gender politics. One such community is r/MensRights, a subreddit created in 2008 that defines itself as an advocacy space “for the social and legal equality of men and boys” and for discussion of “any issue related to the relationship of men to society.” While the stated focus of the subreddit emphasizes legal and social equality, previous research suggests that men’s rights communities are often embedded within the broader manosphere, a loose network of online groups that articulate shared narratives about masculinity, gender relations, and perceived discrimination against men (Chang, 2022).

The emergence of the manosphere is commonly understood as a response to the rise of feminism in the late twentieth century. Early men’s rights movements were frequently characterized by explicitly anti feminist rhetoric, framing feminism as a force that undermines men’s social status, legal rights, and cultural legitimacy (Messner, 2016). According to Messner (2016), such orthodox anti-feminist movements are less likely to be socially acceptable in the post feminist era, where overt misogyny is more readily sanctioned. As a consequence, a more moderate and neo liberal form of men’s rights discourse has gained prominence. This discourse does not always articulate explicit hostility toward feminism, but it implicitly assumes a feminist dominated social order and frames men as an overlooked or disadvantaged group within social and legal institutions.

Ging (2019) further distinguishes between these strands of men’s rights discourse. Orthodox anti feminist men’s rights groups tend to focus on issues such as domestic violence against men, child custody, divorce, and the feminisation of education, often articulated through grievance based and adversarial narratives. In contrast, neo liberal men’s rights discourse emphasizes men’s wellbeing and individual self improvement using a depoliticized language of equality, while still implicitly positioning feminism as the dominant ideological force (Ging, 2019; Messner, 2016). Although this distinction is well established theoretically, it remains unclear how these discourses are reflected in large scale online communities such as r/MensRights.

Hate speech plays a particularly important role in manosphere related spaces. Beyond isolated instances of offensive language, hateful expressions function as discursive tools that reinforce in group identity, legitimize hostility toward women and feminist movements, and normalize antagonistic gender ideologies (Ging, 2019). While existing scholarship has examined the prevalence of toxicity and misogyny in the manosphere, relatively little attention has been paid to the thematic content of hate speech itself, namely what users are actually discussing when they engage in hateful expression.

This study addresses these gaps by combining supervised machine learning and topic modeling to analyze discourse in r/MensRights. Hate speech is first identified using interpretable classification models trained on manually annotated data. Rather than treating hate speech solely as a classification outcome, the study conceptualizes it as a discursive phenomenon whose themes and narratives can reveal deeper ideological structures. Topic modeling is then applied to posts classified as hate speech in order to uncover the dominant themes articulated within this subset of content.

The study is guided by the following research questions:

· RQ1: What is the prevalence and distribution of hate speech within the r/MensRights subreddit?

· RQ2: What thematic patterns characterize hate speech within r/MensRights, and how do these themes relate to established manosphere narratives (Chang, 2022)?

· RQ3: To what extent does the discourse in r/MensRights aligns with an orthodox anti feminist men’s rights movement as opposed to a more moderate neo liberal men’s rights discourse, as described by Messner (2016) and Ging (2019)?

Based on the existing literature, the following hypotheses are proposed:

· H1: Hate speech constitutes a non marginal and systematically observable component of discourse within the r/MensRights subreddit.

· H2: The thematic structure of hate speech in r/MensRights is dominated by topics commonly associated with orthodox anti feminist men’s rights discourse, including domestic violence against men, child custody and divorce, and the feminisation of education (Ging, 2019).

· H3: Hate speech in r/MensRights is primarily organized around grievance based narratives, including claims of misandry, perceived female or feminist dominance, and the delegitimization of feminist movements, reflecting core manosphere frames (Chang, 2022; Ging, 2019).

To answer these questions, the study analyzes a large corpus of Reddit posts collected from r/MensRights over an extended period. By integrating supervised classification and unsupervised topic modeling, the analysis provides a systematic examination of how hate speech and men’s rights discourse are articulated in a prominent online community, thereby contributing to a deeper understanding of the manosphere and gender politics in digital spaces.

Methodology

This project answer the above research questions by combining supervised machine learning and topic modeling to analyze hate speech in r/MensRights. Rather than treating hate speech solely as a classification outcome, the study approaches it as a discursive phenomenon whose themes, narratives, and argumentative patterns can reveal deeper ideological structures.

The dataset consists of 46,365 Reddit posts collected via the Pushshift API, covering the period from June 2009 to November 2022. Prior to any modeling, the full corpus underwent systematic text preprocessing, including lowercasing, removal of punctuation, URLs, HTML artefacts, tokenization, stop-word removal, and lemmatization. We created multiple cleaned versions of the same posts to test the performance of different feature representations (unigram vs. unigram+bigram, lemmatized vs. non-lemmatized, etc.) under the same annotation scheme. This preprocessing pipeline ensured consistency across both the supervised and unsupervised stages of the analysis. To enable supervised classification, a multi-stage annotation and validation strategy was implemented. First, a subset of 300 posts was independently coded by four annotators to assess intercoder reliability. All coders were trained using a shared theoretical framework and discussed relevant categories prior to annotation, after which agreement was evaluated using pairwise Cohen’s kappa coefficients.

Following this calibration phase, a larger sample of 1,200 posts was manually annotated to serve as the labeled dataset for model training and testing. This dataset exhibited moderate class imbalance, with 717 posts labeled as non–hate speech and 483 labeled as hate speech. These annotations were used to train and evaluate multiple interpretable classifiers, including logistic regression, naïve Bayes, and random forest models. Model performance was assessed using precision, recall, F1-score, balanced accuracy, and agreement statistics, with particular emphasis placed on recall for the hate-speech class to maximize the inclusion of relevant content for downstream analysis.

To further assess the reliability of automated classification, an additional validation sample of 300 posts was drawn from the automatically labeled corpus and manually reviewed. This step provided an external quality check on model predictions, allowing for an estimation of false positives and false negatives before proceeding to large-scale thematic analysis.

Posts predicted as hate speech were then subjected to Latent Dirichlet Allocation (LDA) topic modeling.

Ultimately, this project seeks to contribute to a more nuanced understanding of how hate speech functions within online gender-politics communities. By mapping the thematic landscape of hateful discourse in r/MensRights, it aims to clarify the extent to which this subreddit reflects broader manosphere narratives such as misandry claims, perceived female dominance, and anti-feminist ideology, and to demonstrate the value of mixed-method approaches for studying complex social phenomena in digital spaces.

EDA and Preprocessing

Our data was imported using the reddit API in python.

The dataset contains six columns: author, date, title, selftext, reddit_label, and year.

author: the username of the author of the text in the subreddit.
date: the date of publication of the post.
selftext: the original text of the post.
reddit_label: the label automatically assigned to the post by reddit. This feature was introduced around 2014. Thus, posts published before 2014 do not have a label. We could consider the label to be the topics. We assume that the labels automatically assigned by Reddit is not depth enough and does not allow us to capture the full meaning of the post.
year: year the post was published

Cleaning

In order to use the data, we first need to clean our text, as raw Reddit posts contain a lot of noise — including URLs, HTML artifacts, formatting symbols, and punctuation — which can negatively affect the analysis. We therefore apply a conservative cleaning procedure that only removes formatting and boilerplate elements using regular expressions. Specifically, we:

Convert markdown links of the form text to just text
Remove all URLs
Strip HTML tags and escape sequences
Remove backticks and other formatting symbols
Remove punctuation, special characters, and emojis while keeping letters, numbers, spaces, apostrophes, underscores, and hyphens
Collapse multiple spaces and convert all text to lowercase

This procedure preserves the core textual content while removing elements that could introduce noise into the topic modeling.

# loading the dataset
cleanmenright <- read_csv('/Users/PaulMUTAMBA/Documents/Old Pc/HSE Social Info/Third Year/Text Analysis/cleanmenright.csv')
raw_dataset <- cleanmenright %>% as_tibble() 

# making a single text_raw field that contains both title and selftext
posts <- raw_dataset %>% 
  dplyr::select(any_of(c("author","date","title","selftext","reddit_label","year","wspaceselftext"))) %>%
  mutate(id = row_number(), date_parsed = suppressWarnings(mdy(date)), year = if_else(!is.na(year), as.integer(year), as.integer(year(date_parsed))), title = coalesce(as.character(title), ""), selftext = coalesce(as.character(selftext), ""), reddit_label = na_if(as.character(reddit_label), ""), text_raw = str_squish(str_c(title, selftext, sep = " "))) %>%
  filter(text_raw != "")

# building the cleaning function
clean_regex <- function(x) {x %>%
    str_replace_all("\\[([^\\]]+)\\]\\((https?://[^\\)]+)\\)", "\\1") %>% # [txt](url) -> txt
    str_replace_all("https?://\\S+|www\\.\\S+", " ") %>% # turning URLs -> ""
    str_replace_all("</?\\w[^>]*>", " ") %>% # HTML tags -> ""
    str_replace_all("&[^;]+;", " ") %>% # HTML escapes -> ""
    str_replace_all("`{1,3}", " ") %>% # backticks -> ""
    str_replace_all("[^\\p{L}\\p{N}\\s'_-]+", " ") %>% # symbols/punctuation/emoji -> ""
    str_squish() %>% # removing extra whitespaces
    str_to_lower()} # lowercase everything

posts <- posts %>%
  mutate(text_clean = clean_regex(text_raw)) %>%
  filter(text_clean != "")

Next we inspect our cleaned posts to validate and assess the scale and variability of the dataset.

posts <- posts %>% mutate(n_chars = str_length(text_clean), n_words = str_count(text_clean, boundary("word")))

diag_tbl <- tibble(metric = c("Observations:", "Mean characters (clean):", "Mean words per text (clean):", "Median words (clean):", "90% of posts are at or below (clean): ", "Date range (parsed):"),
  value = c(nrow(posts), round(mean(posts$n_chars), 2), round(mean(posts$n_words), 2), round(median(posts$n_words), 2), round(quantile(posts$n_words, 0.90), 2), paste0(min(posts$date_parsed, na.rm=TRUE), " to ", max(posts$date_parsed, na.rm=TRUE))))

print(diag_tbl)

## # A tibble: 6 × 2
##   metric                                   value                   
##   <chr>                                    <chr>                   
## 1 "Observations:"                          44414                   
## 2 "Mean characters (clean):"               1236.91                 
## 3 "Mean words per text (clean):"           228.09                  
## 4 "Median words (clean):"                  131                     
## 5 "90% of posts are at or below (clean): " 505                     
## 6 "Date range (parsed):"                   2009-06-19 to 2022-11-19

Interpretation and implications for further steps:

There are 44414 observations in our dataset and it spans from 19/06/2009 to 19/11/2022.

90% of posts are at or below 505 characters, while median is at only 131 characters, which practically implies that the length distribution is highly right-skewed. That is to say, many of the posts are quite short but there still is a substantial minority of much longer posts containing more tokens that can potentially dominate and bias the model. It is therefore rational to employ TF–IDF + trimming to counter this effect.

Distribution of posts over years:

posts %>% filter(!is.na(year)) %>% count(year) %>%
  ggplot(aes(x = year, y = n)) +
  geom_area() +
  labs(title = "Distribution of posts over years",
       x = "Year", y = "Number of posts") +
  theme_minimal(base_size = 12)

The dataset contains posts from 2009 to 2022. In 2013, the sub-credit was closed due to controversial cases, which explains the sharp decrease in messages that year.

Tokenization, stopwords

Next, we are removing standard stopwords (except negations) and performing tokenization. We are deliberatly keeping negation words (not/no/nor), because within the scope of our research question, they are important contextual cues (in cases of denial, negated accusations, taking positions, etc.)

sw <- stopwords("en")
sw <- setdiff(sw, c("no", "nor", "not"))
corp <- corpus(posts, text_field = "text_clean")
docvars(corp, "year") <- posts$year
toks <- quanteda::tokens(corp, remove_punct = TRUE, remove_numbers = TRUE, remove_symbols = TRUE, remove_separators = TRUE) %>%
  tokens_tolower() %>%
  tokens_remove(sw) %>%
  tokens_select(pattern = "^[a-z][a-z']{1,}$", valuetype = "regex", selection = "keep")
dfm_uni <- dfm(toks)

Word frequencies & wordcloud

top_tbl <- tibble(term = names(topfeatures(dfm_uni, 30)), freq = as.integer(topfeatures(dfm_uni, 30)))

top_tbl

## # A tibble: 30 × 2
##    term    freq
##    <chr>  <int>
##  1 men    76443
##  2 women  65991
##  3 not    64858
##  4 just   35012
##  5 like   31390
##  6 can    29927
##  7 one    24886
##  8 get    24462
##  9 people 24351
## 10 no     24140
## # ℹ 20 more rows

textplot_wordcloud(dfm_uni, max_words = 300)

As we can see, both “men” and “women” are in top terms, as well as “rights”, “feminism/feminist(s)”, “rape”, and also “not” (we intentionally kept negations). This is consistent with our expectations: a big share of posts revolves around gender categories, ideology, and conflict domains (like sexual violence discourse).

Collocations

Next we look at which words pairs occur together more often than expected given their marginal frequencies:

coll <- textstat_collocations(toks, size = 2, min_count = 25) %>% arrange(desc(lambda))

print(head(coll, 15))

##               collocation count count_nested length   lambda        z
## 7135      referrals usccr    39            0      2 17.04292 11.78931
## 4425          los angeles    86            0      2 16.94325 19.45795
## 2698         saudi arabia   111            0      2 15.95779 27.02238
## 2575           boko haram    75            0      2 15.88298 27.61922
## 7450      daniel levinson    56            0      2 15.81755 11.09916
## 7553          murphey doj    32            0      2 15.57231 10.87939
## 7564        debra murphey    32            0      2 15.53691 10.85632
## 7598          cidhoea oas    26            0      2 15.44302 10.76710
## 7599            spcim oas    26            0      2 15.44302 10.76710
## 4001          sonja starr    25            0      2 15.21097 21.10975
## 7828         askdoj usdoj    53            0      2 14.62538 10.28291
## 5006        san francisco    61            0      2 14.60066 17.52971
## 2206       camille paglia    34            0      2 14.37286 29.73834
## 7939 racketeer influenced    29            0      2 14.37084 10.06192
## 5132      scheme artifice    45            0      2 14.33230 17.13261

coll %>% slice_head(n = 25) %>% mutate(collocation = fct_reorder(collocation, lambda)) %>%
  ggplot(aes(x = lambda, y = collocation)) +
  geom_col() +
  labs(title = "Top collocations by lambda", x = "lambda", y = NULL) +
  theme_minimal(base_size = 12)

The top collocations are things like “los angeles”, “saudi arabia”, “boko haram”, as well as name-like pairs (“daniel levinson”, “camille paglia”, etc.) It implies that a part of r/MensRights posts comments on specific cases, certain locations, and public figures.

Supervised ML

To investigate the prevalence and patterns of hate speech in r/MensRights, and therefore answer the RQ1, we decided to use a supervised machine learning approach.

Code Dataset

labelled <- readxl::read_excel("/Users/PaulMUTAMBA/Downloads/Labelled_Posts.xlsx")
cleaned <- read_csv('/Users/PaulMUTAMBA/Downloads/mensrights_text_variants_for_ml.csv')

To predict hate speech within our corpus, we utilize machine learning models trained on a custom labeled dataset, as existing resources predominantly focus on explicit misogyny—characterized by open attacks, insults, or threats. The necessity for a more nuanced dataset is illustrated by the evolution of the “Men’sRights” subreddit; following a shadow ban in 2013 due to controversial content, moderators deleted posts containing explicit hate speech to comply with Reddit policies. However, this suppression did not eliminate the hostility but rather caused it to evolve into implicit misogyny, where members used linguistic creativity to bypass detection. To capture this shift, we sampled and coded 1,200 posts based on the classification scheme by Strathern and Pfeffer (2022), which identifies eleven distinct layers of online misogyny. This scheme defines explicit misogyny through four categories: call for action/violence (inciting physical punishment or institutionalization), personal insult (using slurs and defamation), gendered personal attacks (leveraging stereotypes or sexual motivation), and intellectual inferiority (judging women moral or intellectual worth). To address the “hidden” hate found in groups like the Men’sRights subreddit, the scheme also incorporates seven layers of implicit misogyny: cynicism/sarcasm (derogatory attitudes requiring context), skeptical attitude/distrust (undermining accounts), imputation (flawed motivations like being “attention-seeking”), allegation (refuting accounts with “facts”), speculation/denying credibility (doubting the severity of experiences), demonstration of power (supporting the male counterpart), and taking a position (redefining the perpetrator and victim). To these, we added the category of men’s grievance, which we adapted from the definition of “white grievance” provided by ElSherief et al. (2021) . Just as white grievance involves frustration over a minority group’s perceived privilege and the casting of majority groups as the “real victims” of social change, men’s grievance frames men as the primary victims of gender equality or “misandry”.

Therefore for our study, hate speech is defined as a gender-based antagonistic discourse that includes both explicit and implicit misogyny including male grievance narratives.

Inter-coder Reliability

All four coders first read Strathern and Pfeffer (2022) and ElSherief et al. (2021), and discussed the coding categories to establish a shared understanding of the annotation framework. They then coded 300 posts —15% of all the posts to code— independently. Intercoder reliability was assessed using Pairwise Cohen’s kappa to evaluate agreement beyond chance across all coder pairs. Pairwise Cohen’s kappa quantifies the extent to which two annotators assign the same labels to a set of items beyond what would be expected by random chance. Because kappa is defined for two raters at a time, it is calculated separately for every pair of coders in multi-rater studies, providing a comprehensive picture of agreement across the annotation team.

paul <- readxl::read_excel('/Users/PaulMUTAMBA/Documents/Old Pc/KUL QASS/Collecting and Analysing for SDS/com_sample_posts(AutoRecovered).xlsx')
jane <- read_csv("/Users/PaulMUTAMBA/Documents/Old Pc/KUL QASS/Collecting and Analysing for SDS/Labelled posts (Jane) 2.csv")
moustapha <- read_csv('/Users/PaulMUTAMBA/Downloads/Moustafa - com_sample_posts.csv (2).csv')

tolu <- readxl::read_excel('/Users/PaulMUTAMBA/Documents/Old Pc/KUL QASS/Collecting and Analysing for SDS/Final project/Tolu_label.xlsx', skip = 1)

cs <- data.frame(Paul = paul$Hate_speech, Jane = jane$label, moustafa = moustapha$`Label 0-1`, Tolu = tolu$speech_label)

psych::cohen.kappa(cs)

## 
## Cohen Kappa (below the diagonal) and Weighted Kappa (above the diagonal) 
## For confidence intervals and detail print with all=TRUE
##          Paul Jane moustafa Tolu
## Paul     1.00 0.64     0.61 0.61
## Jane     0.64 1.00     0.68 0.70
## moustafa 0.61 0.68     1.00 0.78
## Tolu     0.61 0.70     0.78 1.00
## 
## Average Cohen kappa for all raters  0.67
## Average weighted kappa for all raters  0.67

Cohen’s kappa values ranged from 0.61 to 0.70, indicating substantial agreement between coders. Across all coder pairs, the average Cohen’s kappa was 0.67, which falls within the range typically interpreted as substantial reliability (Landis & Koch, 1977). Weighted kappa coefficients showed a similar pattern, with an average weighted kappa of 0.67, indicating that the level of agreement remained stable even when partial agreement was taken into account.

These results suggest that the annotation guidelines were applied consistently and that coders shared a common understanding of the hate-speech criteria. The substantial level of inter-coder agreement allowed us to go ahead and label the 1200 posts sampled.

Labelled Dataset

labelled <- readxl::read_excel("/Users/PaulMUTAMBA/Downloads/Labelled_Posts.xlsx")

df_counts <- labelled %>%
  mutate(hate_speech = ifelse(hate_speech == "1", "Hate Speech", "Safe")) %>%
  count(hate_speech)

ggplot(df_counts, aes(x = hate_speech, y = n, fill = hate_speech)) +
  geom_col(width = 0.7) +
  geom_text(
    aes(y = n / 2, label = n),
    color = "white",
    size = 5
  ) +
  scale_fill_brewer(palette = "Set1") +
  labs(
    x = NULL,
    y = "Count",
    fill = "Code",
    title = "Hate Speech Distribution on the Labelled Dataset"
  ) +
  theme_minimal(base_size = 13)

The Labelled Dataset can be found in Appendix B — Labelled Dataset. The manually coded dataset exhibited a moderate class imbalance. Of the 1,200 annotated posts, 717 were labeled as non–hate speech (class 0) and 483 as hate speech (class 1). Although the distribution was not perfectly balanced, both categories were sufficiently represented to support a reliable estimation of intercoder agreement and to enable effective training and evaluation of the supervised classification models.

To ensure that this imbalance did not bias model performance, evaluation metrics that are robust to uneven class distributions, such as precision, recall, and F1-score, were prioritized over accuracy alone in subsequent analyses.

Training Models

Now that we have a labelled cleaned dataset, we can train machine learning models onto it to predicts whether the remaining 43135 posts contains hate speech or not. For this study we used three machine learning models:

Naive Bayses
Logisitc Regression
Random Forest

We ran these models using different Preprocessing and feature representation.

Preprocessing and Feature representation for ML (4 options)

As there is no universal preprocessing pipeline, we intend to test a small set of slightly different options and later choose the optimal one through validation:

First option: unigrams. The most straightforward and basic option.
Second option: unigrams + bigrams. Additionally captures short phrases like “not N”, fixed expressions, certain slurs.
Third option: Lemmatized unigrams. Reduces each words to its roots.
Fourth option: lemmatized unigrams + bigrams. Captures short phrases and reduces sparsity by reducing each word to its root.

In this section, we focus our analysis on the preprocessing pipeline approach that achieved the best performance: unigrams with lemmatization. Appendix A presents the results for all preprocessing variants.

Naive Bayes

Data Set UP: To prepare the data, we transform the cleaned text field into a tokenized corpus. We then use the createDataPartition function to split this corpus into an 80% training set and a 20% testing set. Because createDataPartition specifically generates row indices, we can easily map the original labels to their corresponding documents in the corpus.
Cross Validation: Following the initial data split, we implement a 5-fold cross-validation process using the createFolds function to ensure the model generalizes well to unseen data. Within a loop that iterates through each fold, we partition the training tokens into a sub-training set and a validation set. We then generate a Document-Feature Matrix (DFM) for the training subset, by trimming rare terms to remove noise and prevent the model from learning from infrequent words.
Data Leakage: To maintain consistency and prevent data leakage, we use the dfm_match function to ensure the validation DFM utilizes the exact same features defined during the training phase.
Model Training: After preparing the matrices, we train a Naive Bayes model and generate predictions, carefully forcing the results into a factor format with consistent levels to ensure accurate evaluation. To train our model we use textmodel_nb from quanteda.textmodels which is more suitable for corpus. It does not requires to transform our DFM into a matrix.
Metrics: For each iteration, we calculate the accuracy, precision, recall, and F1-score. Finally, we average these metrics across all five folds to provide a robust and stable assessment of the model’s overall performance.

cleaned <- read_csv('/Users/PaulMUTAMBA/Downloads/mensrights_text_variants_for_ml.csv')
data <- cleaned[labelled$X, , drop = FALSE]

data$label <- labelled$hate_speech

set.seed(1991)
split <- caret::createDataPartition(y= data$label, p = 0.8, list = FALSE) #split partition

fullcorpus <- quanteda::corpus(data, docid_field = "id",
                           text_field = "text_toks_lem")
full_tokens <- quanteda::tokens(fullcorpus, remove_punct = TRUE) 
train_tokens <- full_tokens[split]       
trainY       <- as.factor(data$label)[split]

set.seed(1991)

# Create Folds for cross validation
k <- 5
folds <- createFolds(trainY, k = k, list = TRUE)

# Initialize results storage
cv_results <- data.frame(
  Accuracy = numeric(k),
  Precision = numeric(k),
  Recall = numeric(k),
  F1 = numeric(k)
)

# 2. Cross-Validation Loop
for(i in 1:k) {
  test_idx <- folds[[i]]
  
  # A. Split tokens based on fold indices
  tk_train <- train_tokens[-test_idx]
  tk_test  <- train_tokens[test_idx]
  y_train  <- trainY[-test_idx]
  y_test   <- trainY[test_idx]
  
  # B. Process DFM (Train)
  # Trimming happens here to prevent data leakage from test set
  dfm_train <- dfm(tk_train) %>% 
    dfm_trim(min_docfreq = 5, min_termfreq = 3, docfreq_type = "count") %>%
    dfm_weight(scheme = "boolean")
  
  # C. Process DFM (Test)
  # CRITICAL: Match features to training set (handles unseen/missing words)
  dfm_test <- dfm(tk_test) %>% 
    dfm_match(features = featnames(dfm_train)) %>% # Trim to reduce matrix size
    dfm_weight(scheme = "boolean")
  
  # D. Train Bernoulli Naive Bayes
  nb_model <- textmodel_nb(dfm_train, y_train,distribution = "Bernoulli")
  
  # E. Predict
  raw_preds <- predict(nb_model, newdata = dfm_test)
  
  # FIX: Force predictions to be a factor with the exact same levels as training data.
  # This prevents confusionMatrix errors if a specific class is never predicted in a fold.
  preds <- factor(raw_preds, levels = levels(y_train))
  
  # F. Evaluate
  # (Removed the duplicate confusionMatrix call found in original code)
  cm <- confusionMatrix(preds, y_test, mode = "prec_recall", positive = "1")
  
  cv_results$Accuracy[i]  <- cm$overall["Accuracy"]
  cv_results$Precision[i] <- cm$byClass["Precision"]
  cv_results$Recall[i]    <- cm$byClass["Recall"]
  cv_results$F1[i]        <- cm$byClass["F1"]
  
}

# Output final mean metrics
print(colMeans(cv_results, na.rm = TRUE))

##  Accuracy Precision    Recall        F1 
## 0.6458333 0.5739062 0.4389610 0.4964135

After several attempts, Bernoulli Naive Bayes achieved higher accuracy than Multinomial Naive Bayes. Bernoulli Naive Bayes considers only the presence or absence of a token in a document, whereas Multinomial Naive Bayes accounts for token frequencies, which in this case added noise rather than improving predictions.

The Bernoulli model achieved an overall mean Accuracy of 64.6% during cross-validation. In terms of identifying the positive class (Hate Speech), it reached a mean Precision of 57.4% and a mean Recall of 43.9%, resulting in an F1-score of 0.496. These metrics indicate that while the model is moderately effective at making correct predictions, it currently struggles to capture more than half of the actual hate speech instances within the dataset.

Logistic Regression

For logistic regression, the train function from the caret package can directly handle text represented as document feature matrices, so we adopted this approach. First, the corpus was split into training and test sets, and class labels were converted to valid factor levels. Tokens were transformed into a document feature matrix, trimmed to remove rare terms, and weighted using TF IDF. The test matrix was aligned to the training features to ensure comparability. The resulting matrices were then used to train a regularized logistic regression model (glmnet) with five fold cross validation, automatic down sampling to address class imbalance, and optimization based on the F1 score. Without optimizing for class imbalance the model tended to only predict 1, since only predicting one offered a precision of 60% and a recall of 100%.

train_tokens <- full_tokens[split]
test_tokens <- full_tokens[-split]

y_train <- as.factor(data$label)[split]
y_test  <- as.factor(data$label)[-split]

levels(y_train) <- make.names(levels(y_train))
levels(y_test)  <- make.names(levels(y_test))

set.seed(1991)

dfm_train <- dfm(train_tokens) %>%
  dfm_trim(min_docfreq = 5, min_termfreq = 3) %>% # Trim to reduce matrix size
  dfm_tfidf() #use tfidf

dfm_test <- dfm(test_tokens) %>%
  dfm_match(features = featnames(dfm_train)) %>%
  dfm_tfidf()

x_train_mat <- as.matrix(dfm_train) #Caret works with matrices of dfs, so we have transfom the dfms into matrices
x_test_mat  <- as.matrix(dfm_test)

ctrl <- trainControl(
  method = "cv",                # 5-Fold Cross Validation
  number = 5,
  sampling = "down",            # AUTOMATICALLY fixes class imbalance
  classProbs = TRUE,            # Needed for ROC/AUC 
  summaryFunction = prSummary  # Optimizes for Precision/Recall/F1
)

lr_caret <- train(
  x = x_train_mat,
  y = y_train,
  method = "glmnet",
  metric = "F",                 # Optimize for F1 Score (requires prSummary)
  trControl = ctrl,
  tuneLength = 5
)

preds <- predict(lr_caret, newdata = x_test_mat)
confusionMatrix(preds, y_test, mode = "prec_recall", positive = "X1")

## Confusion Matrix and Statistics
## 
##           Reference
## Prediction  X0  X1
##         X0 119  58
##         X1  23  40
##                                           
##                Accuracy : 0.6625          
##                  95% CI : (0.5989, 0.7221)
##     No Information Rate : 0.5917          
##     P-Value [Acc > NIR] : 0.0144426       
##                                           
##                   Kappa : 0.2606          
##                                           
##  Mcnemar's Test P-Value : 0.0001582       
##                                           
##               Precision : 0.6349          
##                  Recall : 0.4082          
##                      F1 : 0.4969          
##              Prevalence : 0.4083          
##          Detection Rate : 0.1667          
##    Detection Prevalence : 0.2625          
##       Balanced Accuracy : 0.6231          
##                                           
##        'Positive' Class : X1              
##

The Logistic Regression model produced an overall Accuracy of 66.3%. For the positive class (Hate Speech), the model demonstrated a Precision of 63.5% and a Recall of 40.8%, yielding an F1-score of 0.497. While the precision shows a slight improvement over the Naive Bayes approach, the low recall suggests the model still misses a substantial portion of hate speech content, and the Kappa of 0.261 indicates only fair agreement beyond what would be expected by chance.

Random Forest

We did not use the random forest implementation from the caret library because it does not allow direct control over the number of trees, and with our sparse, high dimensional dataset the training time became prohibitive. Instead, we relied on the randomForest package. This implementation does not perform explicit cross validation; rather, it uses the out of bag (OOB) error for model evaluation. OOB error acts as a proxy for cross validation because each tree is trained on a bootstrap sample, leaving out roughly one third of the observations, which are then used to evaluate the model. This repeated internal validation provides an unbiased estimate of generalization performance.

The document feature matrices were converted into data frames, the class label was appended to the training data, and all feature names were standardized to valid R identifiers. A random forest classifier with 70 trees was then trained. Feature name consistency was enforced for the test set.

set.seed(1991)
split <- caret::createDataPartition(y= data$label, p = 0.8, list = FALSE)
corpus <- quanteda::corpus(data, docid_field = "id",
       text_field = "text_toks_lem") %>%  quanteda::tokens(remove_punct=T) %>% dfm() %>% dfm_trim(min_docfreq = 5, min_termfreq = 3, docfreq_type = "count") %>% dfm_tfidf()

train.data <- corpus %>% dfm_subset(rownames(corpus) %in% rownames(corpus)[split])
test.data <- corpus %>% dfm_subset(!rownames(corpus) %in% rownames(corpus)[split])

response <- factor(data$label, levels = c("1", "0"))
trainY <- response[split]
testY <- response[-split]
trainY <- as.factor(data$label)[split]

set.seed(1991)
levels(trainY) <- make.names(levels(trainY))
levels(testY)  <- make.names(levels(testY))

Xtrain.df <- as.data.frame(as.matrix(train.data))
Xtest.df <- as.data.frame(as.matrix(test.data))

Xtrain.df$label <- trainY
colnames(Xtrain.df) <- make.names(colnames(Xtrain.df))
RFmodel <- randomForest::randomForest(label ~ ., data=Xtrain.df, ntree = 70)
colnames(Xtest.df) <- make.names(colnames(Xtest.df))
predRF <- predict(RFmodel, Xtest.df)
print(RFmodel)

## 
## Call:
##  randomForest(formula = label ~ ., data = Xtrain.df, ntree = 70) 
##                Type of random forest: classification
##                      Number of trees: 70
## No. of variables tried at each split: 52
## 
##         OOB estimate of  error rate: 36.15%
## Confusion matrix:
##     X0  X1 class.error
## X0 438 137   0.2382609
## X1 210 175   0.5454545

cm.po3 <- confusionMatrix(data = predRF, reference = testY, positive= "X1", mode = "prec_recall")
cm.po3

## Confusion Matrix and Statistics
## 
##           Reference
## Prediction  X1  X0
##         X1  53  23
##         X0  45 119
##                                           
##                Accuracy : 0.7167          
##                  95% CI : (0.6551, 0.7728)
##     No Information Rate : 0.5917          
##     P-Value [Acc > NIR] : 3.841e-05       
##                                           
##                   Kappa : 0.3925          
##                                           
##  Mcnemar's Test P-Value : 0.01088         
##                                           
##               Precision : 0.6974          
##                  Recall : 0.5408          
##                      F1 : 0.6092          
##              Prevalence : 0.4083          
##          Detection Rate : 0.2208          
##    Detection Prevalence : 0.3167          
##       Balanced Accuracy : 0.6894          
##                                           
##        'Positive' Class : X1              
##

The Random Forest model significantly improved performance, achieving an Accuracy of 71.7%, which is well above the No Information Rate (\(p < 0.001\)). Regarding the identification of hate speech, the model reached a Precision of 69.7% and a Recall of 54.1%, resulting in a much stronger F1-score of 0.609. The Out-of-Bag (OOB) error rate of 36.15% indicates moderate generalization performance, close to the Kappa of 0.393 suggests a moderate level of agreement, indicating that this ensemble approach is more effective at balancing detection and correctness than the previous models.

The table below summarizes the performance metrics for the Bernoulli Naive Bayes, Logistic Regression, and Random Forest models based on the cross-validation and test results.

Model	Accuracy	Precision	Recall	F1-Score
Bernoulli Naive Bayes	64.6%	57.4%	43.9%	0.496
Logistic Regression	66.3%	63.5%	40.8%	0.497
Random Forest	71.7%	69.7%	54.1%	0.609

Comparing the three approaches, the Random Forest model is clearly the top performer across every metric. It provides the highest Accuracy and F1-Score, showing a particularly significant jump in Recall compared to the other two. While the Logistic Regression model offered a slight improvement in Precision over Naive Bayes, it struggled the most with Recall, missing nearly 60% of hate speech instances.

Analyse of predictors

To understand which terms most strongly influenced the detection of hate speech toward women, we examined variable importance using the Mean Decrease in Gini index on our best model — Random Forest. This metric indicates how much each predictor contributes to reducing classification uncertainty across the forest.

randomForest::importance(RFmodel)%>% data.frame() %>% top_n(20, MeanDecreaseGini) %>% arrange(desc(MeanDecreaseGini))

##           MeanDecreaseGini
## woman            10.940085
## man               6.715495
## feminist          6.079088
## rapist            2.487397
## gender            2.083590
## get               1.939315
## reverse           1.904620
## male              1.892107
## fuck              1.820379
## feminism          1.796024
## privilege         1.661072
## time              1.585199
## not               1.572908
## can               1.511135
## make              1.500788
## say               1.430640
## right             1.405415
## think             1.395905
## believe           1.338461
## female            1.334713

randomForest::varImpPlot(RFmodel, n.var = 20)

The most important predictors are explicitly gendered terms, with woman and man showing the highest importance. This suggests that direct references to women and gender comparisons are central cues for identifying misogynistic content. Ideologically loaded terms such as feminist, feminism, and gender also rank highly, reflecting that hate speech toward women often emerges within debates or attacks related to feminism and gender politics. The presence of highly offensive terms like rapist and fuck further indicates that explicit and aggressive language is a strong signal of hateful or abusive expressions targeting women. Additional terms such as male, reverse, and get contribute more moderately, likely capturing contextual patterns in narratives that frame women in opposition to men or invoke claims of “reverse” discrimination or what we coded as Men grievance.

Overall, the importance analysis shows that the model relies primarily on gender specific and derogatory vocabulary, which aligns well with the task of predicting hate speech directed at women rather than general negativity or neutral discourse.

False Positive Posts

Next, let’s analyse the wrongly predicted tweets and learn from their characteristics. The False Positive Posts are those one in the test data that our model predicts as hate speech while we coded them as being safe.

false_positive_indices <- which(predRF == "X1" & testY != "X1")
false_positive_indices <- as.numeric(false_positive_indices)
false_positive_posts <- labelled %>%  filter(X %in% as.numeric(rownames(Xtest.df[false_positive_indices, , drop = FALSE]))) %>% select(title, selftext, hate_speech, X)

head(false_positive_posts$selftext, 5)

## [1] "I'd appreciate you reading through this, and not just downvoting. I am a feminist, and a man, dating and living with a feminist who is not a man. I care about equality. But I have seen some of you getting worked up about feminists acting like all men are rapists. I've decided to try an explain why it seems that way. (There are, obviously, some feminists who do believe that. But most do not.)\r\n \r\n Women on the street face harassment all the time. Everything from a simple \"what are you reading?\" whilst on public transport, to wolf whistles and cat calls multiple times a day, to being groped as they walk down the street can be harassment. (You may wonder why the first is harassment, but imagine if you were asked it by women every day for a year. And not attractive women either. Sometimes women who you actually find physically repulsive. Every day. And they won't stop talking to you without you giving them your name and number. And they threaten you if you won't talk. And they are bigger and stronger than you. That is the experience of some women.)\r\n \r\n Women have no way of telling whether a man is okay or not in advance. Some are great guys. Some are rapists. Some are rapists who think they are great guys. (1 in 12 of college aged men in America will admit to sexual assault/rape if the word \"rape is not used in the question.) So it's easier, and safer for them to not trust any guys in public. It's easier to read a book, listen to music, and try and shut any guy down as fast as possible. Because you have no idea if this guy is going to yell at you. Call you horrible names. Scare you until you think he might well kill or rape you.\r\n \r\n As to why a lot of feminists consider most rape prevention tips to have major issues. There's two sides to that. One side is because it presumes that rape is a force of nature. That all guys are rapists, and that they have no self control. And you people, the Men's Rights Activists, should be as upset by that as anyone! How dare they presume the default state of a man is a rapist. But I haven't seen that from you. \r\n \r\n I've seen whining about how feminists complain about \"victim blaming\", which is the other side. And I'm going to try and explain why we say that, and why it's a problem, using the standard example of theft. So say a friend of yours has their car broken into. You could respond in two ways. Consoling them, saying how shitty the person was for doing that. Even if they parked in a bad area, the person who did it is still shitty, and you comfort them. Or you can jump right in and blame them for parking there, for not taking their valuables with them, for not parking in a garage in a safe area. But if you do that, you're being a shitty friend, and a shitty person. They don't need to be reminded of what they could have done to prevent it. It's too late now. They need to be reminded the crime is not their fault. And that's why we call it \"victim blaming\". Because it blames the victim, not the criminal, for the crime. It acts like criminals are a force of nature, not just ordinary people like me or you. And that is not right.\r\n \r\n I'll happily answer any questions you have, though I can't promise to be fast."
## [2] "So here's what happened, when I was choosing my classes a few months ago they didn't have space in the one I wanted, so instead I chose this one called \"Gender, society and rights\" because I thought it sounded cool (should've thought about it more), first day of class I'm met with a teacher that is nice but is very biased towards women. The whole semester has just been talks about how women have it worse than men and it's all thanks to the patriarchy. \r\n \r\n I think I should make it clear that my teacher isn't a radical feminist and she doesn't hate men. Can't say the same about a certain classmate who believes in \"guilty until proven innocent\".\r\n \r\n Anyways, we're supposed to make a presentation on a social problem about gender so I thought this would be a great opportunity to talk about men's rights and their problems.\r\n \r\n I'm in a team with 3 other guys, 2 of them are cool and I think they didn't want to enter this class either, not sure about the last one.\r\n \r\n What would be some good subjects to talk about? Link me some sources too if you can.\r\n \r\n I haven't discussed this with the other guys yet, I'll tell them during our next meeting, I think they're gonna be against it because no one wants to be \"rebellius\" and frankly I'm scared that I'll just be shamed and called a misogynist by everyone else, but that's another problem. \r\n \r\n Share some cool ideas!, it's almost 3am so I'll read everything in the morning"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               
## [3] "I've heard some people call AMC's Mad men feminist propaganda, but I've found that it often portrays male and female characters and their relationships honestly enough.\r\n \r\n Don Draper has his flaws but is a good man. His wife Betty is a fickle, petulant child.\r\n \r\n Still, it may be guilty of painting men and women in certain ways that don't quite mesh with reality, but I can't put my finger on anything specific.\r\n \r\n Your thoughts?"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          
## [4] "I'm a fervent supporter of abortion. If you think it's morally wrong I can accept that, but I can't accept you imposing your morality on another.\r\n \r\n That said it strikes me as a gross injustice that a man has no rights whatsoever regarding his right to choose to be a parent.\r\n \r\n Earlier I was downvoted on /r/ask for posting what I thought a plausible scenario where the man is upfront about not wanting kids, but trusts that his partner has been honest about taking her birth control. Then it turns out she hadn't been taking it regularly/at all and wants to keep the kid. \r\n \r\n That man is then trapped with 18 years of child support or fathering a child he is not emotionally/financially stable enough to raise and the only response I got was basically \"wrap it up, you can't trust women on that\" which I find pretty sexist. \r\n \r\n On the reverse of that, where a man wants the child but the woman doesn't, that the right to choose must go to the mother, since it is her body that has to bear the child. But seeing as the decision to terminate rests solely with the woman, shouldn't a man have at least a say in whether or not he wants to participate?"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   
## [5] "When I found this subreddit I was very happy to know that it existed as I have long been an advocate for male advocacy. After reading through it for the past few weeks I've become increasingly concerned that this movement may be heading in the wrong direction. I think that this community is making the same mistake that the feminists have made: there is a loud, radical element that is guiding the conversation toward the absurd. We need to learn from that and focus ourselves on the big issues that face men, not devolving into revisionist history or insane theories on gender relations. When we do that we are no better than those who vilify men.\r\n \r\n The point of any rights group is to ensure that equality is protected. Equality. The problem with feminism is that it has whipped itself into such a froth that many of its adherents have lost sight of that. I don't want us to do that. If anything we should be working *with* feminists not against them; hashing out a new social contract that allows for an increased level of egalitarianism. That being said I feel that this movement needs to have second element to it: it needs to be about improving ourselves. Many of the things that the MRM is against are intra-male e.g. circumcision, the disposablility of male life, workplace deaths, etc. We need to ignite a worldwide conversation on what it means to be a man in the modern world. We need to figure out how we can strive for a more equal society and still maintain our dignity as men. So, I implore you brothers to take a step back and take the objective view. A measured, rational approach will help give us credence and silence the nay sayers."

The five false positives share a common feature: they contain contentious gender-related discourse and critical language about feminism or gender politics, but do not meet the threshold for hate speech. Patterns:

1. Critical but non-hostile discourse

Several posts (e.g., FP1, FP4, FP5) adopt a critical stance toward feminism, gender norms, or reproductive rights, yet they do so in a reflective, argumentative, or conciliatory tone rather than a hateful one.

The model appears to have over-relied on ideological keywords (e.g., feminists, patriarchy, sexism, rights, men) as signals of hate speech, even when these terms were used in non-hostile or conciliatory contexts.

2. Presence of Gender Stereotypes Without Explicit Abuse

FP3 contains negative character judgments (e.g., describing a female character as “a fickle, petulant child”), which may resemble misogynistic language but is framed as media criticism, not as an attack on women as a group.

3. Sensitive Topics Triggering Over-Detection

FP4 discusses abortion and reproductive rights, which are topics often associated with polarized discourse. Although the post explicitly condemns sexist language, the surrounding semantic field likely increased the model’s tendency to predict hate speech.

These errors suggest that the classifier tends to fail on the side of over-inclusion, flagging posts that discuss gender conflict even when they aim for dialogue, critique, or moderation. This pattern is consistent with the study’s deliberate prioritization of high recall, but it also highlights the difficulty of distinguishing ideological disagreement from hateful expression in politically charged contexts. For example, in FP#1, we can clearly detect that these patterns emerge and the systematic error takes place.

False Negative Posts

Next, we analyse the False Negative Posts. Those ones are predicted as Safe by the model while we coded them as being Hate speech.

false_negative_indices <- which(predRF == "X0" & testY != "X0")
false_negative_posts <- labelled %>%  filter(X %in% as.numeric(rownames(Xtest.df[false_negative_indices, , drop = FALSE]))) %>% select(title, selftext, hate_speech, X)

head(false_negative_posts$selftext, 5)

## [1] "All that really says is that women like spending other people's money.\r\n \r\n whether it be her husbands money or money taken from men by gun point by the government. I heard somewhere that women receive more money than they pay in taxes."                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     
## [2] "I am active in the menslib community in the SF bay area. MKP, and a few other groups are important passions of mine. Menslib doesn‚Äôt want to hear from men who disagree with their dogma. I pointed out the wage gap study of nurses. Male nurses make several percent more on average. They do it by specializing, working overtime, accepting undesirable shifts, and taking positions with additional responsibility. It is not oppression, unfairness, patriarchy, and sexism. They don‚Äôt want to hear it."                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   
## [3] "So, I decided to come to MRA for help. My good friend, who is a wonderful man and awesome father to his son, is in the middle of an ugly divorce. His eventual ex is doing everything in her power to try and convince him to come back to her. She abused him, emotionally and physically, threatens to take the kid, etc. He long ago moved out, but she shows up at his house under the guise of wanting to see her kid, even though it's really all about continuing her controlling abuse and getting him \"back.\" He tried to file for an injunction against her but basically was laughed out of the court at the notion that he was the battered party, despite her mental illnesses that predispose her toward violence. All the pamphlets (this is FL by the way) are geared toward abused women, nothing about men. She's burned through all his cash drawing out the divorce, and now his best shot at keeping her from stalking him has been burned. \r\n \r\n Is there much he can really do? Any ideas would be welcomed - I don't want to see him lose any custody of his beautiful boy, nor do I want to see him continue to live in fear of her next crazy attack."
## [4] "I know that something like 95% of workplace accidental deaths are men. If we remove those numbers, and the amount of men (and fewer women) that die young in war, is there a major effect on life expectancy, or is it negligible?\r\n \r\n What I'm saying is, do women live longer purely because they are less likely to be killed?"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               
## [5] "So is anyone going to say anything about how all men around their 18th birthday have to register in case the draft needs to be returned. Women don't sign up for this thing. Is there a reasoning that they haven't changed this?\r\n \r\n \r\n This could go both as discrimination against men or women I am aware but still why are only men affected by this in 2020?"

The false negatives, by contrast, reveal how the model sometimes fails to detect hate speech when it is expressed in subtle, indirect, or normalized forms. A qualitative review of the five false negative cases reveals that not all errors can be attributed solely to model failure. Instead, these examples highlight the inherent ambiguity of hate-speech annotation, particularly when misogyny is expressed in indirect, coded, or normalized ways.

Several of the false negatives rely on implicit stereotypes and grievance-based narratives rather than overt insults or threats. For instance, statements portraying women as financially exploitative (FN1) or framing feminist spaces as ideologically hostile to men (FN2), communicate antagonism through implication rather than explicit abuse.

However, posts emphasizing structural disadvantages faced by men, such as workplace deaths or military conscription (FN3, FN4, FN5), were marked as non-hate speech by the model and hate speech by the coders, which points out a big limitation of this type of research. For borderline cases, the model’s conservative classification may be considered as defensible as the human labels. Disagreements among coders during the annotation process suggest that the boundary between legitimate grievance, ideological critique, and hate speech is often difficult to draw. In this sense, some false negatives reflect not only algorithmic limitations but also the interpretive challenges faced by human annotators. The model’s behavior in these cases mirrors the broader uncertainty inherent in defining and operationalizing hate speech in politically charged gender discourse.

This finding points out a key methodological insight: errors in automated hate-speech detection do not always indicate clear misclassification, but may instead reveal the contested and context-dependent nature of the category itself. Recognizing this ambiguity strengthens the interpretation of model performance by situating it within the broader epistemic limits of both computational and human judgment.

Model Tuning

Based on the results above, the next step is to improve the predictive power of our model by increasing recall. As mentioned earlier, our priority is to minimize the number of Hate Speech posts that are not flagged by the model.

p_yes <- predict(RFmodel, newdata = Xtest.df, type = "prob")[, "X1"]

my_roc_curve <- roc(response = testY, predictor = p_yes, levels = c("X1","X0"))
my_roc_curve

## 
## Call:
## roc.default(response = testY, predictor = p_yes, levels = c("X1",     "X0"))
## 
## Data: p_yes in 98 controls (testY X1) > 142 cases (testY X0).
## Area under the curve: 0.7552

#plot(my_roc_curve)

optimal_th = my_roc_curve$threshold[which.max(my_roc_curve$sensitivities + my_roc_curve$specificities)] 
optimal_th

## [1] 0.3928571

roc_2 = roc(testY ~ p_yes, plot=T, print.auc=T,
            print.thres="best", print.thres.patterns="Best threshold: %1.2f")

To further refine the model’s performance, we analyzed the Receiver Operating Characteristic (ROC) curve, which yielded an AUC of 0.76. This indicates a strong discriminatory capability, performing significantly better than a random classifier. By evaluating the curve, we identified an optimal classification threshold of 0.393. This specific cutoff was chosen because it maximizes the balance between Sensitivity and Specificity, effectively prioritizing the detection of hate speech.

pred_optimal = as.factor(ifelse(p_yes>optimal_th , "X1", "X0"))

confusionMatrix(data = pred_optimal, reference = testY, positive= "X1", mode = "prec_recall")

## Confusion Matrix and Statistics
## 
##           Reference
## Prediction X1 X0
##         X1 72 43
##         X0 26 99
##                                           
##                Accuracy : 0.7125          
##                  95% CI : (0.6508, 0.7689)
##     No Information Rate : 0.5917          
##     P-Value [Acc > NIR] : 6.775e-05       
##                                           
##                   Kappa : 0.4206          
##                                           
##  Mcnemar's Test P-Value : 0.05408         
##                                           
##               Precision : 0.6261          
##                  Recall : 0.7347          
##                      F1 : 0.6761          
##              Prevalence : 0.4083          
##          Detection Rate : 0.3000          
##    Detection Prevalence : 0.4792          
##       Balanced Accuracy : 0.7159          
##                                           
##        'Positive' Class : X1              
##

confmat_opt = table(testY, pred_optimal)

Applying this optimized threshold resulted in a more balanced performance profile. While the overall Accuracy remained stable at 71.3%, we observed a substantial improvement in the model’s ability to identify hate speech, with Recall rising to 73.5%. This shift led to a significantly higher F1-score of 0.676 and a Kappa of 0.421, indicating moderate-to-substantial agreement. Although Precision dipped slightly to 62.6% as a trade-off for capturing more positive cases, the optimized model is now far more effective at catching instances of hate speech that the default settings previously missed.

All in all, The Random Forest model emerged as the superior solution, demonstrating a naturally higher capacity to handle the complex, non-linear relationships found in text data. A notable limitation of in our model training is the inability to perform an exhaustive hyperparameter grid search (tuning parameters such as mtry, nodesize, or the number of trees). Due to significant computational power limits, the optimization was primarily focused on threshold tuning rather than a full search across the model’s configuration space. While the threshold adjustment provided substantial gains in recall, further fine-tuning of the Random Forest’s internal structure could potentially reduce the current Out-of-Bag (OOB) error rate and further sharpen precision.

Hate Speech Detection

Given that the tuning model offers good predictive power, we propose implementing it to detect hate speech on the rest of the subreddit’s posts, i.e. 43,135 posts.

full <- cleaned[-labelled$X, , drop = FALSE]

fullcorpus <- quanteda::corpus(full, docid_field = "id",
                           text_field = "text_toks_lem") %>%  quanteda::tokens(remove_punct=T) %>% dfm() %>% dfm_trim(min_docfreq = 5, min_termfreq = 3, docfreq_type = "count") %>% dfm_tfidf()
fulltfidf.df <- as.data.frame(as.matrix(fullcorpus))
colnames(fulltfidf.df) <- make.names(colnames(fulltfidf.df))

prediction_full <- predict(RFmodel, newdata = fulltfidf.df, type = "prob")[, "X1"]

pred_fac <- factor(ifelse(prediction_full > optimal_th, "X1","X0"), levels=c("X1","X0"))

full$prediction <- pred_fac

df_counts <- full %>%
  mutate(prediction = ifelse(prediction == "X1", "Hate Speech", "Safe")) %>%
  count(prediction)

ggplot(df_counts, aes(x = prediction, y = n, fill = prediction)) +
  geom_col(width = 0.7) +
  geom_text(
    aes(y = n / 2, label = n),
    color = "white",
    size = 5
  ) +
  scale_fill_brewer(palette = "Set1") +
  labs(
    x = NULL,
    y = "Count",
    fill = "Predictions",
    title = "Prediction Distribution in the Full Corpus"
  ) +
  theme_minimal(base_size = 13)

On the 43135 posts, 22994 (53%) are predicted as containing hate speech targeting women, and 20141 (47%) are predicted as being safe.

Next we analyse how hate speech varied trhoughout the years in the subreddit.

yearly_dist <- full %>%
  mutate(
    hate = ifelse(prediction == "X1", 1, 0)
  ) %>%
  group_by(year) %>%
  summarise(
    n = n(),
    hate_count = sum(hate),
    prop_hate = hate_count / n
  )


#install.packages("binom")
library(binom)
ci <- binom.confint(
  x = yearly_dist$hate_count,
  n = yearly_dist$n,
  methods = "wilson"
)

yearly_dist <- yearly_dist %>%
  mutate(
    lower = ci$lower,
    upper = ci$upper
  )


ggplot(yearly_dist, aes(x = year, y = prop_hate)) +
  geom_point(size = 3, color = "#D55E00") +
  geom_line(color = "#D55E00", linewidth = 1) +
  geom_errorbar(
    aes(ymin = lower, ymax = upper),
    width = 0.2,
    color = "#D55E00"
  ) +
  scale_y_continuous(
    labels = scales::percent_format(accuracy = 1),
    limits = c(0, NA)
  ) +
  labs(
    x = "Year",
    y = "Proportion of Hate Speech",
    title = "Distribution of Hate Speech Over Time",
    subtitle = "Wilson 95% confidence intervals"
  ) +
  theme_minimal(base_size = 13) +
  theme(
    plot.title = element_text(face = "bold"),
    panel.grid.minor = element_blank()
  )

The figure above presents the yearly proportion of hate speech in the corpus, together with 95% Wilson confidence intervals. Overall, the prevalence of hate speech remains relatively stable over time, fluctuating around 50–55% in most years.

A notable deviation occurs in 2013, where the estimated proportion of hate speech drops sharply and is accompanied by a substantially wider confidence interval. This pattern reflects the closure of the subreddit during that year, which resulted in a markedly reduced number of observations. Consequently, the estimate for 2013 is subject to considerable statistical uncertainty and should not be interpreted as a substantive change in hate speech prevalence.

Excluding this structurally constrained year, the temporal trend suggests moderate year-to-year variation but no evidence of a sustained increase or decrease in hate speech over the observed period.

Predictions Validation

To validate the predictions, we randomly select 300 posts and code them manually to see whether the model perform accurately on unseen data.

sample_validation <- read_csv('/Users/PaulMUTAMBA/Downloads/Validation_set - Sheet1.csv')


sample_validation$prediction <-  ifelse(sample_validation$prediction == "X1", "Hate Speech", "Safe")

sample_validation$prediction <- tolower(sample_validation$prediction)
sample_validation$analysis_result <- tolower(sample_validation$analysis_result)
#table(sample_validation$prediction)
#table(sample_validation$analysis_result)
sample_validation$prediction <- as.factor(sample_validation$prediction)
sample_validation$analysis_result <- as.factor(sample_validation$analysis_result)
cm.vald <- confusionMatrix(data = sample_validation$analysis_result , reference = sample_validation$prediction, positive= "hate speech", mode = "prec_recall")

cm.vald

## Confusion Matrix and Statistics
## 
##              Reference
## Prediction    hate speech safe
##   hate speech         103   24
##   safe                 65  108
##                                           
##                Accuracy : 0.7033          
##                  95% CI : (0.6481, 0.7545)
##     No Information Rate : 0.56            
##     P-Value [Acc > NIR] : 2.387e-07       
##                                           
##                   Kappa : 0.4174          
##                                           
##  Mcnemar's Test P-Value : 2.235e-05       
##                                           
##               Precision : 0.8110          
##                  Recall : 0.6131          
##                      F1 : 0.6983          
##              Prevalence : 0.5600          
##          Detection Rate : 0.3433          
##    Detection Prevalence : 0.4233          
##       Balanced Accuracy : 0.7156          
##                                           
##        'Positive' Class : hate speech     
##

The validation dataset can be found in Appendix C— Validation Dataset. The model achieved an overall accuracy of 70.3% (95% CI: 0.65–0.75), which is significantly higher than the no-information rate of 56% (p < .001). This indicates that the classifier performs substantially better than a baseline model that always predicts the majority class. In terms of class-specific performance, the model achieved a precision of 0.81 and a recall of 0.61 for the hate-speech category, resulting in an F1-score of 0.70. This pattern suggests that while the classifier is highly accurate when it predicts a post as hate speech, it is more conservative in identifying all hate-speech instances, missing some true cases.

Recall was prioritized in model evaluation because the central aim of the study was to analyze the thematic composition of hate-speech discourse rather than to optimize classification accuracy per se. Missing hate-speech posts at the detection stage would have permanently excluded relevant material from the topic-modeling analysis, risking an incomplete or biased representation of dominant narratives. In contrast, allowing some non–hate-speech posts into the hate-speech subset was considered less problematic, as topic modeling is relatively robust to limited noise. Emphasizing recall, therefore, ensured broad coverage of hate-speech content while safeguarding the validity of the subsequent thematic analysis

Topic Modelling

To explore the thematic structure of hate speech in r/MensRights, we applied topic modeling to identify dominant patterns of discussion. Guided by the hypothesis that these patterns are shaped by orthodox men’s rights narratives (Ging, 2019), we aimed to uncover topics related to domestic violence against men, child custody and divorce, and the feminisation of education. This approach allows us to systematically analyze the content of the subreddit and link observed themes to established discourses in the manosphere.

Coherence and Exclusivity

To examine the thematic structure of hate speech in r/MensRights, we employ Latent Dirichlet Allocation (LDA), a widely used topic modeling technique that identifies clusters of words representing distinct themes. The LDA as K—the number of topics as hyperparameter. To determine the most suitable number of topics for our corpus, we evaluate models across a range of K values using topic coherence, which measures the semantic similarity of words within a topic, and exclusivity, which assesses how distinct each topic is from the others. This approach allows us to select a model that captures meaningful and well-separated themes.

hate_df <- full %>%
  filter(prediction == "X1") %>%        
  select(text_toks_lem)  

corp_hate <- corpus(hate_df, text_field = "text_toks_lem") 

dfm_hate <- corp_hate %>%
  quanteda::tokens(remove_punct = TRUE) %>%
  dfm() %>%
  dfm_remove(stopwords("english")) %>%
  dfm_trim(
    min_docfreq = 0.02,
    max_docfreq = 0.7,
    docfreq_type = "prop"   
  )

# ------- Keeping top 600 features to further reduce sparsity ------
top_features <- names(topfeatures(dfm_hate, n = 600))
dfm_hate <- dfm_hate[, top_features]

# cat("Documents:", nrow(dfm_hate), "\n")     
# cat("Features:", nfeat(dfm_hate), "\n")     
# cat("Sparsity:", sparsity(dfm_hate), "\n")   

set.seed(123)
sample_docs <- sample(docnames(dfm_hate), 5000)
dfm_sample <- dfm_hate[sample_docs, ]

dtm_sample <- convert(dfm_sample, to = "topicmodels")

calculate_perplexity <- function(k, dtm) {
  lda_model <- LDA(dtm, k = k, control = list(seed = 123))
  perplexity(lda_model)
}

k_vals <- seq(4, 10, by = 1)
perplexities <- sapply(k_vals, calculate_perplexity, dtm = dtm_sample)

perplexity_df <- data.frame(
  k = k_vals,
  perplexity = perplexities
)


# ggplot(perplexity_df, aes(x = k, y = perplexity)) +
#   geom_line() +
#   geom_point() +
#   labs(
#     title = "Perplexity vs Number of Topics",
#     x = "Number of Topics",
#     y = "Perplexity"
#   ) +
#   theme_classic()

multi_lda <- function(k, dtm){
  LDA(dtm, k = k, control = list(seed = 123))
}

lda_models <- lapply(k_vals, multi_lda, dtm_sample)
names(lda_models) <- k_vals

diag_list <- lapply(lda_models, topic_diagnostics, dfm_sample, 10)
diag_df <- bind_rows(diag_list, .id = "K") %>% mutate(K = as.numeric(K))

diag_summary <- diag_df %>%
  group_by(K) %>%
  summarise(
    coherence = mean(topic_coherence),
    exclusivity = mean(topic_exclusivity)
  )

#diag_summary

## Plot Coherence–Exclusivity Trade-off
ggplot(diag_summary, aes(coherence, exclusivity)) +
  geom_text(aes(label = K, color = factor(K)), size = 5, fontface = "bold") +
  scale_color_brewer(palette = "Dark2") + 
  labs(
    x = "Coherence",
    y = "Exclusivity",
    title = "Coherence–Exclusivity Trade-off"
  ) +
  theme_minimal() +
  theme(
    plot.title = element_text(face = "bold", hjust = 0.5),
    legend.position = "none"
  )

A Model with eight topics offers the most balanced compromise between semantic coherence and topic exclusivity, avoiding the loss in interpretability observed in highly exclusive but less coherent solutions, while still improving distinctiveness compared to the most coherent models.

Therefore we ran a LDA model with K = 8.

LDA

Since we have an extremely large corpus, we decided to use the MalletLDA function for topic modeling since it is highly efficient and can handle large text corpora faster. The MalletLDA runs a LDA model by using JAVA in the background — a programming language that is optmized to work on large matrices unlike R.

posthatespeech <- full %>% select("id","text_raw", "text_tok", "year", "prediction") %>% filter(prediction == "X1")

mallet.instances <- mallet.import(id.array=as.character(posthatespeech$id),
                                  text.array=posthatespeech$text_tok)
topic.model <- MalletLDA(num.topics=8) # number of topics

# fixing seed for reproducibility
topic.model$setRandomSeed(396L)
topic.model$loadDocuments(mallet.instances) 
topic.model$setAlphaOptimization(30, 70) #optimization for MALLET (computational ressources used by the function to run faster the LDA)
vocabulary <- topic.model$getVocabulary() # corpus dictionary
word.freqs <- mallet.word.freqs(topic.model) # frequency table
## top frequent words (by doc frequency)
#word.freqs %>% arrange(desc(doc.freq)) %>% head(10)
topic.model$train(20)
topic.model$maximize(5)

doc.topics <- mallet.doc.topics(topic.model, smoothed=TRUE, normalized=TRUE)

topic.words <- mallet.topic.words(topic.model, smoothed=TRUE, normalized=TRUE)

topic.labels <- mallet.topic.labels(topic.model, topic.words, 10)
topic.labels

## [1] "women men male female less likely males work years many"             
## [2] "child s children support not family pay father law money"            
## [3] "men women not rape man woman sexual sex can people"                  
## [4] "violence gender boys social victims male s domestic girls abuse"     
## [5] "not just like get one no time even know said"                        
## [6] "men women not s feminism feminist feminists rights people can"       
## [7] "man woman young said boy sex time one now body"                      
## [8] "new show media public video including university act justice article"

Above are the topic labels for the eight themes of discussion in Mens’reddit posts flagged as hate speech.

Topic Interpretation

Based solely on the topic labels listed above — the words that characterize each theme — we can’t provide a meaningful interpretation for each of the eight discussion topics. Therefore, we had to reviewed the top posts associated with each topic and, using both the labels and the content of these posts, to derive an interpretation for each theme.

Top post for topic 1

top.docs <- function(doc.topics, topic, docs, top.n=10) {
    head(docs[order(-doc.topics[,topic])], top.n)
}
top.docs(doc.topics, 1, posthatespeech$text_raw, 3)

## [1] "In many countries around the world, the state retirement age is higher for men; as a result, men often enjoy only half the years in retirement women do. One of the major male discriminations around the world is the issue of higher pension ages- in many countries, the state pension age where a person can retire and collect a government pension is [higher for men.](https://en.wikipedia.org/wiki/Retirement_age#Retirement_age_by_country) This is of course despite the fact that men have shorter life expectancies in nearly every country in the world. As a result, men often have only *half* the years in retirement to enjoy. I've selected a few countries to highlight this difference, bit of course there are many more. I can imagine that if the genders were reversed, it would be considered a gender inequality and social issue, but since men are the ones adversely affected, it's ignored. --------------- **AUSTRIA** Male life expectancy: 79 Male pension age: 65 Years in retirement: 14 _ Female life expectancy: 83.9 Female pension age: 60 Years in retirement: 23.9 _ Difference in retirement years: 9.9 So in Austria, women enjoy nearly 10 years longer in retirement. Once again, a country considered very gender-equal and pro-feminist is an example of a country with active male discrimination. ---------------- **Poland** Male life expectancy: 73.6 Male pension age: 65.58 Years in retirement: 8.02 _ Female life expectancy: 81.3 Female pension age: 60.58 Years in retirement: 20.72 _ Difference in retirement years: 12.7 In Poland, women have more than double the years in retirement, more than 250% longer. -------------------- **Croatia** Male life expectancy: 74.7 Male pension age: 65 Years in retirement: 9.7 _ Female life expectancy: 81.2 Female pension age: 61.75 Years in retirement: 19.45 _ Difference in retirement years: 9.75 An almost 10 year gap in Croatia, where women enjoy a little more than double the years in retirement, enjoying just under 20 to men's just under 10. ------------- **CUBA** Male life expectancy: 76.9 Male pension age: 65 Years in retirement: 11.9 _ Female life expectancy: 81.4 Female pension age: 60 Years in retirement: 21.4 _ Difference in retirement years: 9.5 An almost 10 year difference, women enjoy nearly double the time in retirement in Cuba. ------------------ **Vietnam** Male life expectancy: 71.21 Male pension age: 60 Years in retirement: 11.21 _ Female life expectancy: 80.58 Female pension age: 60 Years in retirement: 20.58 _ Difference in retirement years: 9.37 Another example of an almost 10 year difference, where men enjoy only about half of the time in retirement. ---------------- **Venezuela** Male life expectancy: 70.37 Male pension age: 60 Years in retirement: 10.37 -- Female life expectancy: 78.61 Female pension age: 55 Years in retirement: 23.61 -- Difference in retirement years: 23.24 More than double the years in retirement. 23 years and 10 years are two completely different retirements. -------- **Chile** Male life expectancy: 76.71 Male pension age: 65 Years in retirement: 11.71 -- Female life expectancy: 81.74 Female pension age: 60 Years in retirement: 21.74 -- Difference in retirement years: 10.03 Another example of an entire decade difference. ---------------- In addition to age qualifications, many countries have a \"years worked\" rule for eligibility, where a person must work for a certain number of years to qualify. This difference often leads to older men being homeless and on the streets. In the UK for example, men must work for 44 years, compared to 39 years for women. [Here's the government website](https://www.gov.uk/state-pension/eligibility). Note the chart on \"qualifying years\", I made a [screenshot here](https://imgur.com/a/aEEgEsR). So, imagine a man turns 65, but faced a few years of unemployment. Maybe he was injured or disabled, and is now too old to work, but doesn't have the 44 years...say he only has 42. That 42 would be enough if he was a woman, but as a man, he wouldn't qualify. This situation has lead to numerous elderly men on the streets of the UK. It's a blatant legal double standard and discrimination. It's interesting to note that feminists in the UK [protested against](https://www.reuters.com/article/us-britain-women-pensions/british-women-protest-as-their-pension-age-rises-to-meet-mens-idUSKCN1NB2AM) equal pension ages, claiming that it would be in inequality. The list goes on and on in every continent, as you can see from the retirement age map, many countries have higher pension ages for men. This trend is something that goes pretty much entirely ignored. I can't help but think that if the genders were reversed, it would be considered a massive social issue, and cited as an example of male privilege. To me, this is a pretty major disparity and a pretty major difference in one's life. Consider that on top of higher pension ages, many of these countries also practice male only conscription, where men must join the military or perform civil service for less than minimum wage for a set period, usually 1-3 years."                                                                                                                   
## [2] "In many countries around the world, the state retirement age is higher for men; as a result, men often only live to see half the years of retirement women do. One of the major male discriminations around the world is the issue of higher pension ages- in many countries, the state pension age where a person can retire and collect a government pension is [higher for men.](https://en.wikipedia.org/wiki/Retirement_age#Retirement_age_by_country) This is of course despite the fact that men have shorter life expectancies in nearly every country in the world. As a result, men often have only *half* the years in retirement to enjoy. I've selected a few countries to highlight this difference, bit of course there are many more. I can imagine that if the genders were reversed, it would be considered a gender inequality and social issue, but since men are the ones adversely affected, it's ignored. --------------- **AUSTRIA** Male life expectancy: 79 Male pension age: 65 Years in retirement: 14 _ Female life expectancy: 83.9 Female pension age: 60 Years in retirement: 23.9 _ Difference in retirement years: 9.9 So in Austria, women enjoy nearly 10 years longer in retirement. Once again, a country considered very gender-equal and pro-feminist is an example of a country with active male discrimination. ---------------- **Poland** Male life expectancy: 73.6 Male pension age: 65.58 Years in retirement: 8.02 _ Female life expectancy: 81.3 Female pension age: 60.58 Years in retirement: 20.72 _ Difference in retirement years: 12.7 In Poland, women have more than double the years in retirement, more than 250% longer. -------------------- **Croatia** Male life expectancy: 74.7 Male pension age: 65 Years in retirement: 9.7 _ Female life expectancy: 81.2 Female pension age: 61.75 Years in retirement: 19.45 _ Difference in retirement years: 9.75 An almost 10 year gap in Crotia, where women enjoy a little more than double the years in retirement, enjoying just under 20 to men's just under 10. ------------- **CUBA** Male life expectancy: 76.9 Male pension age: 65 Years in retirement: 11.9 _ Female life expectancy: 81.4 Female pension age: 60 Years in retirement: 21.4 _ Difference in retirement years: 9.5 An almost 10 year difference, women enjoy nearly double the time in retirement in Cuba. ------------------ **Vietnam** Male life expectancy: 71.21 Male pension age: 60 Years in retirement: 11.21 _ Female life expectancy: 80.58 Female pension age: 60 Years in retirement: 20.58 _ Difference in retirement years: 9.37 Another example of an almost 10 year difference, where men enjoy only about half of the time in retirement. ---------------- **Venezuela** Male life expectancy: 70.37 Male pension age: 60 Years in retirement: 10.37 -- Female life expectancy: 78.61 Female pension age: 55 Years in retirement: 23.61 -- Difference in retirement years: 23.24 More than double the years in retirement. 23 years and 10 years are two completely different retirements. -------- **Chile** Male life expectancy: 76.71 Male pension age: 65 Years in retirement: 11.71 -- Female life expectancy: 81.74 Female pension age: 60 Years in retirement: 21.74 -- Difference in retirement years: 10.03 Another example of an entire decade difference. ---------------- In addition to age qualifications, many countries have a \"years worked\" rule for eligibility, where a person must work for a certain number of years to qualify. This difference often leads to older men being homeless and on the streets. In the UK for example, men must work for 44 years, compared to 39 years for women. [Here's the government website](https://www.gov.uk/state-pension/eligibility). Note the chart on \"qualifying years\", I made a [screenshot here](https://imgur.com/a/aEEgEsR). So, imagine a man turns 65, but faced a few years of unemployment. Maybe he was injured or disabled, and is now too old to work, but doesn't have the 44 years...say he only has 42. That 42 would be enough if he was a woman, but as a man, he wouldn't qualify. This situation has lead to numerous elderly men on the streets of the UK. It's a blatant legal double standard and discrimination. It's interesting to note that feminists in the UK [protested against](https://www.reuters.com/article/us-britain-women-pensions/british-women-protest-as-their-pension-age-rises-to-meet-mens-idUSKCN1NB2AM) equal pension ages, claiming that it would be in inequality. The list goes on and on in every continent, as you can see from the retirement age map, many countries have higher pension ages for men. This trend is something that goes pretty much entirely ignored. I can't help but think that if the genders were reversed, it would be considered a massive social issue, and cited as an example of male privilege. To me, this is a pretty major disparity and a pretty major difference in one's life. Consider that on top of higher pension ages, many of these countries also practice male only conscription, where men must join the military or perform civil service for less than minimum wage for a set period, usually 1-3 years."                                                                                                              
## [3] "In many countries around the world, the state retirement age is higher for men; as a result, men often enjoy only half the years in retirement women do. Btw this is an article from [menarehuman](https://menarehuman.com/retirement-age/), a website a few of us run here. One of the major male discriminations around the world is the issue of higher pension ages- in many countries, the state pension age where a person can retire and collect a government pension is [higher for men.](https://en.wikipedia.org/wiki/Retirement_age#Retirement_age_by_country) This is of course despite the fact that men have shorter life expectancies in nearly every country in the world. As a result, men often have only *half* the years in retirement to enjoy. I've selected a few countries to highlight this difference, bit of course there are many more. I can imagine that if the genders were reversed, it would be considered a gender inequality and social issue, but since men are the ones adversely affected, it's ignored. --------------- **AUSTRIA** Male life expectancy: 79 Male pension age: 65 Years in retirement: 14 _ Female life expectancy: 83.9 Female pension age: 60 Years in retirement: 23.9 _ Difference in retirement years: 9.9 So in Austria, women enjoy nearly 10 years longer in retirement. Once again, a country considered very gender-equal and pro-feminist is an example of a country with active male discrimination. ---------------- **Poland** Male life expectancy: 73.6 Male pension age: 65.58 Years in retirement: 8.02 _ Female life expectancy: 81.3 Female pension age: 60.58 Years in retirement: 20.72 _ Difference in retirement years: 12.7 In Poland, women have more than double the years in retirement, more than 250% longer. -------------------- **Croatia** Male life expectancy: 74.7 Male pension age: 65 Years in retirement: 9.7 _ Female life expectancy: 81.2 Female pension age: 61.75 Years in retirement: 19.45 _ Difference in retirement years: 9.75 An almost 10 year gap in Crotia, where women enjoy a little more than double the years in retirement, enjoying just under 20 to men's just under 10. ------------- **CUBA** Male life expectancy: 76.9 Male pension age: 65 Years in retirement: 11.9 _ Female life expectancy: 81.4 Female pension age: 60 Years in retirement: 21.4 _ Difference in retirement years: 9.5 An almost 10 year difference, women enjoy nearly double the time in retirement in Cuba. ------------------ **Vietnam** Male life expectancy: 71.21 Male pension age: 60 Years in retirement: 11.21 _ Female life expectancy: 80.58 Female pension age: 60 Years in retirement: 20.58 _ Difference in retirement years: 9.37 Another example of an almost 10 year difference, where men enjoy only about half of the time in retirement. ---------------- **Venezuela** Male life expectancy: 70.37 Male pension age: 60 Years in retirement: 10.37 -- Female life expectancy: 78.61 Female pension age: 55 Years in retirement: 23.61 -- Difference in retirement years: 23.24 More than double the years in retirement. 23 years and 10 years are two completely different retirements. -------- **Chile** Male life expectancy: 76.71 Male pension age: 65 Years in retirement: 11.71 -- Female life expectancy: 81.74 Female pension age: 60 Years in retirement: 21.74 -- Difference in retirement years: 10.03 Another example of an entire decade difference. ---------------- In addition to age qualifications, many countries have a \"years worked\" rule for eligibility, where a person must work for a certain number of years to qualify. This difference often leads to older men being homeless and on the streets. In the UK for example, men must work for 44 years, compared to 39 years for women. [Here's the government website](https://www.gov.uk/state-pension/eligibility). Note the chart on \"qualifying years\", I made a [screenshot here](https://imgur.com/a/aEEgEsR). So, imagine a man turns 65, but faced a few years of unemployment. Maybe he was injured or disabled, and is now too old to work, but doesn't have the 44 years...say he only has 42. That 42 would be enough if he was a woman, but as a man, he wouldn't qualify. This situation has lead to numerous elderly men on the streets of the UK. It's a blatant legal double standard and discrimination. It's interesting to note that feminists in the UK [protested against](https://www.reuters.com/article/us-britain-women-pensions/british-women-protest-as-their-pension-age-rises-to-meet-mens-idUSKCN1NB2AM) equal pension ages, claiming that it would be in inequality. The list goes on and on in every continent, as you can see from the retirement age map, many countries have higher pension ages for men. This trend is something that goes pretty much entirely ignored. I can't help but think that if the genders were reversed, it would be considered a massive social issue, and cited as an example of male privilege. To me, this is a pretty major disparity and a pretty major difference in one's life. Consider that on top of higher pension ages, many of these countries also practice male only conscription, where men must join the military or perform civil service for less than minimum wage for a set period, usually 1-3 years."

Top posts for topic 2

top.docs(doc.topics, 2, posthatespeech$text_raw, 2)

## [1] "Legal Paternal Surrender This is going to be a long post, but I think I've developed a good explanation for why legal parental surrender should be allowed: All reasonable arguments against legal parental surrender rest on the idea that it would violate either a woman's right to bodily autonomy, or the child's (assuming the child isn't aborted) right to a good life (ie. financial support from the father). Legal Parental Surrender: Within the same time frame as an abortion is permitted (minus a week or so to inform the woman of the decision), the man may pay half of the woman's abortion or delivery costs (some people like to quibble over how much the man should have to pay, but it's not really relevant to the underlying idea) and not be held as the child's father. **Edit:** Woman can also utilize legal parental surrender to declare they intend to have the child, but then give it up to the father (with no child support from her), or for adoption if the father also exercises legal parental surrender. This has the added benefit of maybe allowing some men to have children that would otherwise be aborted because the mother doesn't want the child and doesn't want to pay child support. Assume for the moment that legal parental surrender is allowed. A man and a woman have consensual sex and a pregnancy occurs. Here are the possible outcomes: 1. Both want child: everything is good. 2. The man wants the child, the woman does not: the fetus is aborted because the woman has the right to bodily integrity. 3. The woman wants the child, the man does not: the man surrenders legal paternity. The woman has the right to bodily integrity and chooses to get an abortion or not. 4. Neither party wants the child: the fetus is aborted because the woman has the right to bodily integrity. Under this model the responsibility for the child is placed on the party, or parties who want the child, without taking away the woman's right to bodily autonomy. Therefore legal parental surrender does not violate a woman's right to bodily autonomy. The other concern is that legal parental surrender would violate the child's right to a good life. Here is why that is not the case: The only time this concern is raised is in case 3 (where legal paternal surrender is relevant). In this case, the woman has the choice to abort, or have the child without the support of the father (minus his half of the delivery costs). If the woman chooses to abort, no child is born and no rights are violated. If the woman chooses to continue with the pregnancy, one of three outcomes can occur: 1. The woman gives the child up for adoption. This is the same outcome as our current system and does not violate the child's right to a good life. 2. The woman keeps the child and has the ability to care for the child. This is the same outcome as our current system (artificial insemination from a sperm bank) and does not violate the child's right to a good life. 3. The woman keeps the child but does not have the ability to care for the child, and the child is taken by social services (the woman should have given the child up for adoption as in outcome 1). This is the same outcome as our current system (when a child lives with an unfit parent or parents) and does not violate the child's right to a good life. It could be argued that legal parental surrender would increase the number of pregnancies which result in one of these three outcomes (outcome 3 being the socially objectionable one), but it does not create the outcome. Legal parental surrender does not violate the child's right to a good life, only the woman responsible for the child's existence can by not giving the child up for adoption when she cannot support the child. In summary, all legal parental surrender does is place the responsibility for a child only the woman wants solely on the woman who chooses if that child is born. It does not violate anyone's rights. The only reasons to oppose it would be based on one of these four premises: 1. The responsibility for a child that only the woman wants should be placed on both the woman who chooses if the child is born, and on the man who does not. (this is pretty obviously wrong) 2. The social ramifications (the increased incidence of outcome 3) would be so severe that legal paternal surrender must not be allowed, as it would destroy society. (here they have accepted that legal paternal surrender should be allowed, they're just quibbling technicalities) 3. Women have the right to raise a child with the support of a man who does not want a child. (this is pretty obviously wrong) 4. One of the three outcomes (adoption, well-supported single motherhood, or removal of the child by social services) of legal parental surrender actually violates the child's right to a good life. (now they aren't against legal paternal surrender, they are against some aspect of the legal system that society accepts) What do you think Men's Right's? The reason FOR legal parental surrender is pretty self-evident (men having child support forcibly extracted for a child they don't want is wrong), but I hope this clear breakdown of why you can't be AGAINST legal paternal surrender might gain us some ground with the equality feminists and general populace (for the gender feminists no amount of clear explanation will help). **TL;DR Legal paternal surrender doesn't violate anyone's rights.**"
## [2] "Legal Abortion : Her Body, Her Choice, Her Responsibility I believe the topic of legal abortion for men is more than just about men's rights. I think it goes deeper. Fundamentally, you cannot hold a person responsible for something that they have no right to. A man has no right to a woman's body. During the first 6 months of pregnancy, the fetus is not a person. It has no human rights. It can legally be destroyed. A man has no veto power of this decision, nor should he. The non-human entity is inside the woman's body, and it is therefore her right to do with it as she wishes. But with this right, comes responsibility. If a man has no right to her body, he therefore cannot be held responsible. It is therefore the woman's responsibility not to fall pregnant. And during the time that the fetus is not a person, the man should have the right, as does the woman, not to become a parent. He may lose all custody rights of the child if the woman carries it to birth, but he should not be held financially responsible for it."

Top posts for topic 3

top.docs(doc.topics, 3, posthatespeech$text_raw, 2)

## [1] "Women and consent This is just gonna be a short rant I find it annoying how a lot of the time women just put themselves onto men and start touching them and trying to have sex, assuming the man likes it and often times the man does. I hate how they don’t have to ask for consent and they get away with it and are never held accountable. Even if the man likes it, she still didn’t ask for consent and doesn’t respect men’s consent and she could easily try doing that to a man who doesn’t want it. It teaches other women they can get away with it (and they do) and other men feel obligated to just go along with it even if they don’t want to. I think women should still be held accountable and ask for consent before doing these things. They think men should always like sex, always take advantage of sexual situations and always enjoy it and that it’s ok for women to just force themselves onto men. Men need more boundaries surrounding sex so that women don’t get away with this and women need to show they care about a mans consent."
## [2] "Will women recognize their privilege and power when it comes to dating. Women have all the privilege and power when it comes to dating, sex, relationships, reproduction. Men don’t have any power when it comes to those things. Getting relationships and sex is 100x easier for women then men and I hope they realize how unfair it is for men. I’m trying not to complain or be a victim but when you talk to women and they have no idea how the male experience is complete different from women is what annoys me. Why can’t they just understand this and do some introspective thinking. A women told me that because she gets lots of sex and relationships that 90% of men are doing it too. Absolutely delusional. I can see why men are giving up its like trying to talk to a brick wall."

We applied this process to the eight discussion themes in our dataset. Below are the reports for each theme, based on their keywords and the posts which better represent each them.

Topic 1: Gendered inequalities in retirement age and pension outcomes — Pension Disparity

The first topic is primarily concerned with perceived structural discrimination against men in pension and retirement systems, framed through cross national comparisons between men and women. The most salient words women men male female less likely work years many indicate a strong gender comparative logic, with emphasis on differences in life expectancy, statutory retirement age, and years spent in retirement.

The exemplar texts show that discussion within this topic centres on the claim that men are systematically disadvantaged by higher statutory retirement ages despite having shorter life expectancies. Contributors frequently argue that this results in men enjoying significantly fewer years of retirement than women, often quantified through detailed country level statistics. These numerical comparisons are used rhetorically to frame the issue as a form of institutional or state sanctioned gender discrimination against men, particularly in countries that are otherwise portrayed as gender equal or feminist.

More broadly, this topic reflects a recurring men’s rights narrative in which gender equality policies are interpreted as privileging women while neglecting or obscuring disadvantages faced by men. The tone of the discussion positions male disadvantage as socially ignored or politically illegitimate, especially when contrasted with how similar inequalities would allegedly be treated if they affected women. Overall, Topic 1 captures debates around gender, work, longevity, and welfare policy, framed through claims of unequal treatment and injustice toward men in retirement and pension systems.

Topic 2: Child support, paternal responsibility, and legal parental surrender — Parental Rights & Support

Topic 2 centres on debates about fatherhood, child support obligations, and reproductive responsibility, with a strong focus on the legal and financial dimensions of parenting. The most salient terms child children support not family pay father law money indicate that this topic is structured around financial responsibility for children, particularly the obligation of fathers to provide child support within existing legal frameworks.

The representative texts reveal that discussions in this topic are dominated by arguments in favour of “legal parental surrender” or “financial abortion” for men. Contributors contend that because men lack bodily autonomy over pregnancy decisions, they should similarly have the right to opt out of legal and financial responsibility for a child within the same temporal window in which abortion is legally permitted. Child support is framed not primarily as a child’s right, but as an unfair legal burden imposed on men, especially when the decision to continue a pregnancy is presented as solely the woman’s choice.

In general, this topic reflects a recurring men’s rights discourse on reproductive asymmetry, where abortion law and child support legislation are interpreted as structurally privileging women while constraining men. The discussion often reframes fatherhood as a consensual contractual responsibility, arguing that legal and moral obligations should only arise when both parents explicitly choose parenthood. Overall, Topic 2 captures narratives that challenge existing family law by contesting norms of paternal responsibility, emphasizing financial autonomy, and redefining parental obligations in terms of individual choice rather than the welfare of the child.

Topic 3: Sexual consent, rape, and perceived gendered power asymmetries — Consent & Power Dynamics

Topic 3 focuses on discussions of sexual interactions, consent, and rape, framed through claims of gendered double standards between men and women. The most salient terms men women rape sexual sex man woman people indicate that this topic is centrally concerned with who is recognized as a legitimate victim or agent in sexual contexts, and how norms of consent are unevenly applied.

The example texts show that contributors frequently argue that men’s experiences of sexual coercion or unwanted sexual contact are minimized or dismissed, particularly when women are the initiators. Posts emphasize that women are allegedly not held accountable for violating men’s consent and that cultural expectations portray men as always desiring sex, thereby obscuring male vulnerability. This framing challenges dominant public narratives around sexual violence by asserting that men’s consent is routinely ignored and that female sexual misconduct is socially tolerated.

In parallel, the topic also incorporates broader claims about female privilege in dating and sexual markets. Discussions portray women as holding disproportionate power over sex, relationships, and reproduction, while men are depicted as structurally disadvantaged and lacking agency. These arguments often blend grievances about consent and rape with frustrations about dating inequality, reinforcing a narrative in which sexual norms and social attitudes systematically favour women.

Overall, Topic 3 captures a cluster of discourse that reframes sexual consent and rape through a men’s rights lens, emphasizing perceived double standards, underrecognition of male victimhood, and gendered power imbalances in sexual and romantic relationships.

Topic 4: Gendered violence and male victimization — Male Victimization

Topic 4 captures a men’s rights discourse that seeks to reframe violence as a gender inclusive issue, emphasizing underrecognition of abuse against men and boys and advocating for equal visibility, resources, and policy responses.

Topic 5: Personal relationship conflict, emotional distress, and accusations — Relational Conflict

This topic centres on personal experiences of romantic rejection, emotional vulnerability, and interpersonal conflict, often involving women. The key terms are generic and narrative driven, reflecting storytelling rather than policy or ideological debate.

The example texts show men recounting confusing or painful relationship dynamics, including unreciprocated affection, emotional manipulation, breakups, and rejection. A second prominent strand involves fear of false accusations or reputational harm, particularly accusations of sexual misconduct, and the perceived lack of due process or voice for men in these situations.

Overall, Topic 5 captures a discourse of male emotional suffering and insecurity, combining intimate relationship struggles with broader anxieties about trust, credibility, and vulnerability in interactions with women and social institutions.

Topic 6: Definitions of equality, feminist critique, and men’s rights advocacy — Feminism Hate

This topic centres on the ideological friction between men’s rights activists and feminist movements, focusing on how labels like “feminist,” “misogynist,” and “egalitarian” are defined and contested. Unlike purely narrative accounts of personal conflict, this discourse is analytical and argumentative, seeking to establish the boundaries of legitimate advocacy.

The example texts reveal a deep-seated skepticism toward modern feminist waves, with contributors often distinguishing between the abstract goal of gender equality and the specific political actions of feminist groups. A recurring theme is the rejection of the “feminist” label as a prerequisite for believing in equality. Some participants express a sense of alienation, arguing that modern social terminology is used to silence or mischaracterize men who do not actively participate in women’s rights advocacy.

In brief, Topic 6 captures a search for an egalitarian identity that exists outside of, or in opposition to, contemporary feminism.

Topic 7: Double standards in social conduct, sexual boundaries, and male victimization — Social Double Standards

Topic 7 captures a discourse of men navigating social double standards where their own boundaries and safety are frequently overlooked. It combines a critique of traditional “gentlemanly” burdens with a serious concern for how male victims of harassment and abuse are perceived by society and the law.

Topic 8: Institutional corruption, public advocacy, and veteran affairs — Institutional Advocacy

Topic 8 captures a brand of male advocacy that is focused on public policy, institutional ethics, and the role of media and government in addressing grievances. It portrays a community engaged in systematic “calls for justice,” moving away from ideological debate toward the practical monitoring of government agencies and the protection of male-heavy populations like veterans.

Frequency of Topic Models in Documents

doc_topics_tidy <- melt(doc.topics)

colnames(doc_topics_tidy) <- c("Document_ID", "Topic_ID", "Frequency")

doc_topics_tidy$Topic_Label <- case_when(
  doc_topics_tidy$Topic_ID == 1 ~ "Pension Disparity",
  doc_topics_tidy$Topic_ID == 2 ~ "Parental Rights & Support",
  doc_topics_tidy$Topic_ID == 3 ~ "Consent & Power Dynamics",
  doc_topics_tidy$Topic_ID == 4 ~ "Male Victimization",
  doc_topics_tidy$Topic_ID == 5 ~ "Relational Conflict",
  doc_topics_tidy$Topic_ID == 6 ~ "Feminism Hate", 
  doc_topics_tidy$Topic_ID == 7 ~ "Social Double Standards",
  TRUE ~ "Institutional Advocacy" 
)

doc_topics_tidy$Topic_Label <- ordered(doc_topics_tidy$Topic_Label, levels = c(
  "Pension Disparity", 
  "Parental Rights & Support", 
  "Consent & Power Dynamics", 
  "Male Victimization", 
  "Relational Conflict", 
  "Feminism Hate", 
  "Social Double Standards", 
  "Institutional Advocacy"
))

ggplot(data = doc_topics_tidy, aes(x = Document_ID, y = Frequency)) + 
  geom_col(width = 1) + 
  facet_wrap(~Topic_Label) + 
  ylab("Topic Probability") + 
  xlab("Documents") + 
  ggtitle("Frequency of Topics in Documents") + 
  theme_hc() + 
  theme(
    plot.title = element_text(hjust = 0.5), 
    axis.text.x = element_blank(), 
    axis.ticks.x = element_blank()
  )

Relational Conflict and Feminism Hate emerge as the most dominant and pervasive themes, maintaining consistently higher probabilities across a larger volume of documents than any other category. These topics represent the core of the discourse, characterized by dense and frequent activity throughout the dataset.

In contrast, Male Victimization and Institutional Advocacy show significantly lower overall frequency, appearing as specialized or niche subjects that only occasionally reach high probability in specific texts. Consent & Power Dynamics also demonstrates high-intensity spikes, suggesting that while it may not be as constant as the primary ideological themes, it remains a frequent and significant point of focus.

Frequency of Topic Over Time

doc_topics_tidy$year <- posthatespeech$year[doc_topics_tidy$Document_ID]

data1 <- doc_topics_tidy %>%
  
  filter(year != 2013) %>%
  
  group_by(year, Topic_Label) %>%
  summarise(val_per_year = sum(Frequency), .groups = "drop") %>%
  
  group_by(year) %>%
  mutate(tot_year = sum(val_per_year)) %>%
  ungroup() %>%
  
  mutate(norma_val = val_per_year / tot_year) %>%
  mutate(norma_val = replace_na(norma_val, 0))

ggplot(data1, aes(x = year, y = norma_val)) +
  geom_area(fill = "grey50", alpha = 0.5) + 
  geom_line(color = "black") + 
  facet_wrap(~Topic_Label) + 
  ylab("Normalized Frequency") + 
  xlab("Year") + 
  ggtitle("Frequency of Topics over Time") + 
  theme_hc() + 
  theme(plot.title = element_text(hjust = 0.5))

The longitudinal data shows that while most topics have remained relatively stable, there are notable shifts in specific areas of concern. Consent & Power Dynamics has seen a steady increase in normalized frequency since 2009, indicating a growing focus on sexual boundaries and agency. Conversely, Feminism Hate, while still a dominant topic, has experienced a gradual decline from its peak in the early 2010s, though it saw a minor resurgence around 2020.

Most other topics, including Relational Conflict, Male Victimization, and Pension Disparity, have maintained a very consistent presence over the twelve-year period, reflecting the enduring nature of these core grievances. Institutional Advocacy saw a slight increase in attention leading up to 2016 before trending downward, suggesting a period of heightened organized activism followed by a return to baseline levels.

Conclusion

Our analysis shows that across 44,414 r/MensRights posts we collected, hate speech surfaces fairly often, and our model detects it with decent reliability. Human annotation reached substantial agreement (with average Cohen’s κ reaching around 0,67), and among the tested classifiers the tuned Random Forest had the best performance. Threshold was tuned at the cutoff of around 0,393 to improve the recall, and on the 300 posts validation sample the model reached 70,3% accuracy with high precision (~0,81) and moderate recall (~0,61). Ont he remaining 43,135 posts, the model flagged 22,994 posts (53%) as the ones containing hate speech.

Topic modeling of the hate-speech subset (8 topics) shows that the flagged posts’ discourse is primarily structured by grievance domains. That being the case, feminism-focused/ideological conflict and relational conflict narratives were the most persistent and dominant themes. Besides these two, there are also posts (hate speech subset) that address issues like child support and “financial abortion”, double standards in consent/rape and dating, domestic violence framed around male victims, institutions treating men unfairly (pensions/retirement rules). Over time, the proportion of posts about consent/dating power increases from the early 2010s onward, whereas the explicitly anti-feminist narratives are at their strongest in the early 2010s, and they slowly decline over time (with a small spike at around 2020).

Addressing research questions and hypotheses:

RQ1 & H1: supported, hate-speech-labeled content is consistently present and commonenough to be considered a structural part of the subreddit’s discourse.
RQ2& H2-H3: for the most part, the results are consistent with what past research describes: many hate-speech posts follow common manosphere frames (misandry, feminism having too much power, distrust in feminist narratives) and focus on classic men’s rights issues like custody/child support and debates around education, violence, and sex. At the same time, some posts are written more like policy or activism reports, which may appear less openly hostile when you read them on their own.
RQ3: hate-speech subset includes both explicitly anti-feminist & grievance-heavy posts andless hostile posts that communicate in terms of equality/rights. In practice, though, they seem to often cover the same issues: what changes is the tone and wording.

As for the main limitation of our study, we have aknowledge that the boundary between grievance, critique, and hate is genuinely hard to operationalize, and the classifier ultimatelyrelies on gender/ideology keywords, which can lead to flagging of non-hateful but “charged” debate as hate speech.

Overall, the approach that we used allowed us to detect posts containing hate speech at a large scale, and then see what people talk about inside that subset. The main themes are conflict around feminism, relationship problems, reproduction and child support, consent/sexual politics, and complaints about institutions treating men unfairly, and the balance between these themes changes over time (consent/dating talk grows anti-feminist hostility decreases a little). If this study were extended, the most useful next steps would thenbe to clearly separate explicit vs implicit misogyny, sample more of the borderline cases for manual checking, and compare r/MensRights to similar subreddits to see what could bespecific to this community.

Gen AI Use

Generative AI (ChatGPT and Gemini) was used in this project in accordance with KU Leuven’s guidelines on the responsible use of AI in education and research. The tool supported the research process by assisting with the clarification of coding guidelines and by helping to improve the clarity, structure, and academic style of the written report. Particularly the GenAI helped for regular expressions for the preprocess and the creation of a cross validation function for Naive Bayse. All conceptual decisions, including the research design, development of the coding scheme, interpretation of results, and formulation of conclusions, were made exclusively by the authors. All AI-assisted outputs were critically reviewed, edited, and validated by the research team to ensure accuracy, methodological soundness, and full academic responsibility.

References

Chang, W. (2022). The monstrous-feminine in the incel imagination: Investigating the representation of women as “femoids” on /r/Braincels. Feminist Media Studies, 22(2), 254–270. https://doi.org/10.1080/14680777.2020.1804976
ElSherief, M., Ziems, C., Muchlinski, D., Anupindi, V., Seybolt, J., De Choudhury, M., & Yang, D. (2021). Latent hatred: A benchmark for understanding implicit hate speech. arXiv preprint arXiv:2109.05322.
Ging, D. (2019). Alphas, Betas, and Incels: Theorizing the Masculinities of the Manosphere. Men and Masculinities, 22(4), 638–657. https://doi.org/10.1177/1097184X17706401
Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977 Mar;33(1):159-74. PMID: 843571.
Messner, M. A. (2016). Forks in the Road of Men’s Gender Politics: Men’s Rights vs Feminist Allies. International Journal for Crime, Justice and Social Democracy, 5(2), Article 2. https://doi.org/10.5204/ijcjsd.v5i2.301
Strathern, Wienke & Pfeffer, Juergen. (2023). Identifying Different Layers of Online Misogyny. 10.36190/2023.54.

Appendices

Appendix A : Model trainings for each Variants of Preprocessing

https://drive.google.com/file/d/1Tn1KoLDmVc-cpas7spg0R8EBFLis8PNJ/view?usp=sharing

Appendix B : Labelled Dataset

https://docs.google.com/spreadsheets/d/10cqaOGRM3aZ7M5HLa48X988VKxuYMpbP/edit?usp=sharing&ouid=108179007286731714531&rtpof=true&sd=true

Appendix C: Validation Dataset

https://drive.google.com/file/d/19SGJqQvFKZ27ajDVZdqVzLGmDUOwlgm6/view?usp=sharing

Final Project

Student 1: Paul Marie Mweni

Student 2: Jane Shadrina

Student 3: Moustafa Basiony

Student 4: Tolulope Soetan

2026-01-11