The datasets each from true.CSV and fake.CSV comprises metadata, allowing us to explore both surface-level and deeper linguistic features.

Our study is guided by two main objectives:

  1. To identify common linguistic, stylistic or structural patterns in differentiating real and fake news articles. We analyze word usage, bigram patterns, and readability scores to uncover how fake news differs from real news in language style, structure, and complexity. For instance, fake news may use more media-related terms or web-like formatting, while real news emphasizes verified sources and geographic references.

  2. To classify whether a news article is fake or real based on its content and title. Using machine learning models (GLMNet and XGBoost), we train classifiers on textual features from the content and titles. The goal is to develop a predictive model capable of accurately labeling unseen articles as real or fake, based on learned patterns in the dataset.

# data loading and preprocessing here
fake <- read.csv("Fake.csv", stringsAsFactors = FALSE)
real <- read.csv("True.csv", stringsAsFactors = FALSE)

news_df <- bind_rows(
  fake %>% mutate(label = "Fake"),
  real %>% mutate(label = "Real")
) %>%
  mutate(
    text = paste(title, text, sep = " "),
    cleaned = tolower(text) %>%
      str_replace_all("[^a-z\\s]", " ") %>%
      str_squish()
  ) %>%
  select(label, cleaned) %>%
  mutate(label = as.factor(label))

TF-IDF Keyword Analysis

The chart below presents TF-IDF keyword scores, where the x-axis represents the TF-IDF values and the y-axis lists the keywords. Red bars indicate keywords from fake news, while blue bars represent real news.

Fake news keywords (e.g., https, quot, var, filessupport, flicker, youtu) often consist of HTML fragments or media link remnants. These likely result from insufficient text cleaning during data preprocessing. Other terms like henningsen and somodevill are names or media-related references. Such patterns suggest that fake news may originate from unstructured web content or copied material, reflecting lower content quality.

In contrast, real news keywords predominantly include political figures and geographic names. These terms are strongly tied to real-world events, indicating that real news content is more standardized, timely, and thematically focused. This achieves the liguistic pattern analysis of our objective.

data("stop_words")

unigram_df <- news_df %>%
  unnest_tokens(word, cleaned) %>%
  anti_join(stop_words, by = "word") %>%
  count(label, word, sort = TRUE) %>%
  bind_tf_idf(term = word, document = label, n = n)

top_tf_idf <- unigram_df %>%
  group_by(label) %>%
  slice_max(tf_idf, n = 10)

ggplot(top_tf_idf, aes(fct_reorder(word, tf_idf), tf_idf, fill = label)) +
  geom_col(show.legend = FALSE) +
  coord_flip() +
  facet_wrap(~label, scales = "free") +
  labs(title = "Top TF-IDF Words per Class", x = NULL, y = "TF-IDF") +
  theme_minimal()

TF-IDF Bigram Analysis

This chart displays the top TF-IDF bigrams for fake (red) and real (blue) news articles.

Fake news bigrams like “featured image”, “ twitter.com”, and “white house” often relate to visual content or social media, suggesting an attempt to boost engagement or credibility through external media.

Real news bigrams frequently include “reuters”, such as “washington reuters” or “united states”, indicating trusted sources and geographic context—hallmarks of formal journalism.

The x-axis shows TF-IDF scores, highlighting how distinct each phrase is for its news category. In short, fake news leans on media cues, while real news emphasizes credible sources and locations. This explains the stylistic pattern in our objective.

# Load stopwords
data("stop_words")

# Tokenize into bigrams (includes stopwords)
bigrams <- news_df %>%
  unnest_tokens(bigram, cleaned, token = "ngrams", n = 2)

# Remove bigrams where BOTH words are stopwords
bigrams_filtered <- bigrams %>%
  separate(bigram, into = c("word1", "word2"), sep = " ") %>%
  filter(!(word1 %in% stop_words$word & word2 %in% stop_words$word)) %>%
  unite(bigram, word1, word2, sep = " ")

# Count and get top 10 bigrams per label
top_bigrams <- bigrams_filtered %>%
  count(label, bigram, sort = TRUE) %>%
  group_by(label) %>%
  slice_max(n, n = 10)

# Plot the result
ggplot(top_bigrams, aes(fct_reorder(bigram, n), n, fill = label)) +
  geom_col(show.legend = FALSE) +
  coord_flip() +
  facet_wrap(~label, scales = "free") +
  labs(title = "Top Bigrams by Label (Filtered Stopword-only Bigrams)", x = NULL, y = "Frequency") +
  theme_minimal()

### Readability Analysis

The Flesch-Kincaid score below describe how difficult a text is to read. It is based on sentence length and word complexity. Lower scores indicate more complex text, while higher scores suggest easy readability.

In the chart below, we compare the readability of fake and real news articles. In our project this helps identify whether one category tends to use simpler or more complex language. It meets our current objective in identifying the structural patterns.

qcorpus <- corpus(news_df$cleaned)
readability <- textstat_readability(qcorpus, measure = "Flesch.Kincaid")
news_df$readability <- readability$Flesch.Kincaid

ggplot(news_df, aes(label, readability, fill = label)) +
  geom_boxplot() +
  labs(title = "Readability (Flesch-Kincaid) by Label", 
       x = "Label", y = "Readability Score") +
  theme_minimal()

Modelling Introduction

In our project, we apply machine learning pipeline to classify news articles as Fake or Real. We use two classification models:

The dataset consists of two CSV files (Fake.csv and True.csv). We combine them, preprocess the text, build Document-Term Matrices (DTMs), and use them to train models using different text components: title, content, and combined text.

1. Load Libraries

library(dplyr)
library(readr)
## Warning: package 'readr' was built under R version 4.3.3
library(tidyr)
library(tidytext)
library(stringr)
library(tm)
## Warning: package 'tm' was built under R version 4.3.3
## Loading required package: NLP
## Warning: package 'NLP' was built under R version 4.3.3
## 
## Attaching package: 'NLP'
## The following objects are masked from 'package:quanteda':
## 
##     meta, meta<-
## The following object is masked from 'package:ggplot2':
## 
##     annotate
## 
## Attaching package: 'tm'
## The following object is masked from 'package:stopwords':
## 
##     stopwords
## The following object is masked from 'package:quanteda':
## 
##     stopwords
library(caret)
library(Matrix)
library(text2vec)
library(xgboost)
library(parallel)
library(ggplot2)
library(DiagrammeR)
## Warning: package 'DiagrammeR' was built under R version 4.3.3
library(grid)
library(glmnet)

2. Data Preparation

2.1 Load and Label Data

fake <- read.csv("Fake.csv", stringsAsFactors = FALSE) %>% mutate(label = 1)
real <- read.csv("True.csv", stringsAsFactors = FALSE) %>% mutate(label = 0)
all_data <- bind_rows(fake, real) %>% mutate(id = row_number())

We label Fake news with 1 and Real news with 0, then combine both datasets into one.

2.2 Tokenization

prep_fun <- tolower
tok_fun <- word_tokenizer

it_combined <- itoken(paste(all_data$title, all_data$text), preprocessor = prep_fun, tokenizer = tok_fun, ids = all_data$id)
it_title    <- itoken(all_data$title, preprocessor = prep_fun, tokenizer = tok_fun, ids = all_data$id)
it_content  <- itoken(all_data$text,  preprocessor = prep_fun, tokenizer = tok_fun, ids = all_data$id)

We tokenize the text using text2vec, preparing it for vectorization.

2.3 Vectorization

vocab <- create_vocabulary(it_combined, stopwords = stop_words$word)
vectorizer <- vocab_vectorizer(vocab)
dtm_combined <- create_dtm(it_combined, vectorizer)
dtm_title    <- create_dtm(it_title, vectorizer)
dtm_content  <- create_dtm(it_content, vectorizer)

We convert the tokenized text into numerical matrices (DTM) that can be used by machine learning models.

2.4 Train/Test Split

set.seed(7004)
split_idx <- createDataPartition(all_data$label, p = 0.8, list = FALSE)
y_train <- all_data$label[split_idx]
y_test  <- all_data$label[-split_idx]

We split the data into 80% training and 20% testing.

2.5 Visualize Split

df_counts <- data.frame(set = c(rep("Train", length(y_train)), rep("Test", length(y_test))),
                        label = c(ifelse(y_train == 1, "Fake", "Real"), ifelse(y_test == 1, "Fake", "Real")))

df_summary <- df_counts %>% count(set, label)

ggplot(df_summary, aes(x = set, y = n, fill = label)) +
  geom_bar(stat = "identity", position = position_dodge()) +
  labs(title = "Real vs. Fake News in Train/Test", x = "Set", y = "Count") +
  theme_minimal()

This plot helps confirm balanced class distributions across training and testing sets.

3. XGBoost Modeling

3.1 Prepare Datasets

dtm_comb_tr  <- dtm_combined[split_idx, ]
dtm_comb_te  <- dtm_combined[-split_idx, ]
dtm_title_tr <- dtm_title[split_idx, ]
dtm_title_te <- dtm_title[-split_idx, ]
dtm_cont_tr  <- dtm_content[split_idx, ]
dtm_cont_te  <- dtm_content[-split_idx, ]

We subset the DTM based on training/testing split.

3.2 Train and Evaluate Function

train_and_evaluate <- function(dtm_train, y_train, dtm_test, y_test) {
  dtrain<- xgb.DMatrix(data = dtm_train, label = y_train)
  params<- list(objective = "binary:logistic", eval_metric = "logloss")
  model <- xgb.train(params = params, data = dtrain, nrounds = 100)

  probs <- predict(model, dtm_test)
  pred  <- factor(ifelse(probs > 0.5, 1, 0), levels = c(0, 1))
  obs   <- factor(y_test, levels = c(0, 1))
  cm    <- confusionMatrix(pred, obs, positive = "1")

  metrics<- data.frame(
    Accuracy  = cm$overall["Accuracy"],
    Precision = cm$byClass["Pos Pred Value"],
    Recall    = cm$byClass["Sensitivity"],
    F1        = 2 * (cm$byClass["Pos Pred Value"] * cm$byClass["Sensitivity"]) /
                (cm$byClass["Pos Pred Value"] + cm$byClass["Sensitivity"])
  )
  list(
    model   = model,
    metrics = metrics
  )
}

This helper function trains the model and calculates performance metrics and model. - We use binary:logistic to tell the model our targeted output was binary class. - The scoring system that penalize the incorrect prediction is using logloss. - The boosting round is set to be 100. - The threshold is set to be 0.5.

3.3 Model Training

res_combined <- train_and_evaluate(dtm_comb_tr, y_train, dtm_comb_te, y_test)
res_title    <- train_and_evaluate(dtm_title_tr, y_train, dtm_title_te, y_test)
res_content  <- train_and_evaluate(dtm_cont_tr, y_train, dtm_cont_te, y_test)

We train XGBoost models using different inputs: combined, title only, and content only.

combined_model <- res_combined$model
title_model    <- res_title$model
content_model  <- res_content$model

combined_metrics <- res_combined$metrics
title_metrics    <- res_title$metrics
content_metrics  <- res_content$metrics

Extract Results.

3.4 Results

combined_metrics <- res_combined$metrics
title_metrics    <- res_title$metrics
content_metrics  <- res_content$metrics

xgb_results <- bind_rows(Combined = combined_metrics, Title = title_metrics, Content = content_metrics, .id = "Input")
print(xgb_results)
##                 Input  Accuracy Precision    Recall        F1
## Accuracy...1 Combined 0.9974385 0.9983112 0.9968381 0.9975741
## Accuracy...2    Title 0.9036641 0.9629983 0.8503373 0.9031680
## Accuracy...3  Content 0.9978840 0.9989440 0.9970489 0.9979956

Result Explanation: - In fake news detection, we prioritize Recall metric as it measures how well the model identifies actual fake news. - The Content-only XGBoost model achieved the highest scores across all evaluation metrics. - The Title-only model performed the worst, indicating that titles alone are insufficient to differentiate between fake and real news. - The Combined model produced decent results but had the longest run time. - Therefore, we recommend using the Content-only XGBoost model for optimal performance.

4. GLM Modeling

4.1 Train Models

glm_combined <- cv.glmnet(x = dtm_comb_tr, y = y_train, family = "binomial", alpha = 0.5, nfolds = 3, parallel = TRUE)
## Warning: from glmnet C++ code (error code -81); Convergence for 81th lambda
## value not reached after maxit=100000 iterations; solutions for larger lambdas
## returned
## Warning: executing %dopar% sequentially: no parallel backend registered
## Warning: from glmnet C++ code (error code -81); Convergence for 81th lambda
## value not reached after maxit=100000 iterations; solutions for larger lambdas
## returned

## Warning: from glmnet C++ code (error code -81); Convergence for 81th lambda
## value not reached after maxit=100000 iterations; solutions for larger lambdas
## returned
## Warning: from glmnet C++ code (error code -80); Convergence for 80th lambda
## value not reached after maxit=100000 iterations; solutions for larger lambdas
## returned
glm_title    <- cv.glmnet(x = dtm_title_tr, y = y_train, family = "binomial", alpha = 0.5, nfolds = 3, parallel = TRUE)
glm_content  <- cv.glmnet(x = dtm_cont_tr, y = y_train, family = "binomial", alpha = 0.5, nfolds = 3, parallel = TRUE)
## Warning: from glmnet C++ code (error code -85); Convergence for 85th lambda
## value not reached after maxit=100000 iterations; solutions for larger lambdas
## returned
## Warning: from glmnet C++ code (error code -81); Convergence for 81th lambda
## value not reached after maxit=100000 iterations; solutions for larger lambdas
## returned

## Warning: from glmnet C++ code (error code -81); Convergence for 81th lambda
## value not reached after maxit=100000 iterations; solutions for larger lambdas
## returned

## Warning: from glmnet C++ code (error code -81); Convergence for 81th lambda
## value not reached after maxit=100000 iterations; solutions for larger lambdas
## returned
  • Family: Binomial is logistic regression.
  • CV = 3 to perform 3 cross-validation by finding the best lambda.
  • Parallel enables parallel processing to speed up the model run time.

4.2 Evaluation Function

eval_glm <- function(glm_obj, dtm_te, y_true) {
  probs <- predict(glm_obj, dtm_te, s = "lambda.min", type = "response")
  pred  <- factor(ifelse(probs > 0.5, 1, 0), levels = c(0,1))
  obs   <- factor(y_true, levels = c(0,1))
  cm <- confusionMatrix(pred, obs, positive = "1")

  data.frame(
    Accuracy  = cm$overall["Accuracy"],
    Precision = cm$byClass["Pos Pred Value"],
    Recall    = cm$byClass["Sensitivity"],
    F1        = 2 * (cm$byClass["Pos Pred Value"] * cm$byClass["Sensitivity"]) /
                (cm$byClass["Pos Pred Value"] + cm$byClass["Sensitivity"])
  )
}

Same logic as XGBoost evaluation but applied to GLM predictions.

4.3 Results

res_glm_combined <- eval_glm(glm_combined, dtm_comb_te, y_test)
res_glm_title    <- eval_glm(glm_title,    dtm_title_te, y_test)
res_glm_content  <- eval_glm(glm_content,  dtm_cont_te,  y_test)

glm_results <- bind_rows(Combined = res_glm_combined, Title = res_glm_title, Content = res_glm_content, .id = "Input")
print(glm_results)
##                 Input  Accuracy Precision    Recall        F1
## Accuracy...1 Combined 0.9938746 0.9968214 0.9915683 0.9941879
## Accuracy...2    Title 0.9468760 0.9727454 0.9253794 0.9484714
## Accuracy...3  Content 0.9939860 0.9976655 0.9909359 0.9942893

Result Explanation: - The GLM results indicate that the Content-only model performs the best, achieving the highest precision (99.77%) and F1 score (99.43%) while matching the Combined model in accuracy. - Although the Combined model has a slightly higher recall (99.15%) compared to the Content model (99.09%), its performance in other metrics is slightly lower. -The Title-only model lags behind significantly across all metrics, reinforcing that titles alone are insufficient for reliable classification. - Therefore, the Content-based GLM model is recommended for effective and efficient fake news detection.

5. Model Interpretation

5.1 XGBoost Tree Model

5.1.1 XGBoost Combined Interpretation

grid.newpage()
grid.text("XGBoost: Tree #0 Structure", 
          y = unit(0.98, "npc"), 
          gp = gpar(fontsize = 16, fontface = "bold"))
xgb.plot.tree(
  model        = combined_model,
  trees        = 0,
  show_node_id = TRUE
)

5.1.2 XGBoost Title Interpretation

grid.newpage()
grid.text("XGBoost: Tree #0 Structure", 
          y = unit(0.98, "npc"), 
          gp = gpar(fontsize = 16, fontface = "bold"))
xgb.plot.tree(
  model        = title_model,
  trees        = 0,
  show_node_id = TRUE
)

5.1.3 XGBoost Content Interpretation

grid.newpage()
grid.text("XGBoost: Tree #0 Structure", 
          y = unit(0.98, "npc"), 
          gp = gpar(fontsize = 16, fontface = "bold"))
xgb.plot.tree(
  model        = content_model,
  trees        = 0,
  show_node_id = TRUE
)

5.2 Extract Coefficients

get_top_coefs <- function(glm_obj, n = 20) {
  coefs <- as.matrix(coef(glm_obj, s = "lambda.min"))
  df <- data.frame(term = rownames(coefs), coefficient = coefs[,1], stringsAsFactors = FALSE)
  df <- df[df$term != "(Intercept)", ]
  df <- df[order(-abs(df$coefficient)), ]
  head(df, n)
}

coefs_comb <- get_top_coefs(glm_combined)
coefs_title <- get_top_coefs(glm_title)
coefs_cont <- get_top_coefs(glm_content)

The top coefficients show which words most influence fake/real prediction.

5.3 Visualize Top Coefficients

plot_coefs <- function(df, title) {
  ggplot(df, aes(x = reorder(term, coefficient), y = coefficient, fill = coefficient > 0)) +
    geom_bar(stat = "identity", show.legend = FALSE) +
    coord_flip() +
    labs(title = title, y = "Coefficient") +
    theme_minimal() +
    scale_fill_manual(values = c("TRUE" = "steelblue", "FALSE" = "salmon"))
}

plot_coefs(coefs_comb,  "GLM Coefs: Combined Text")

plot_coefs(coefs_title, "GLM Coefs: Title Only")

plot_coefs(coefs_cont,  "GLM Coefs: Content Only")

These bar charts visualize the most influential words driving model predictions.

combined_results <- bind_rows(Content_XGBoost = content_metrics, Content_GLM = res_glm_content, .id = "Input")
print(combined_results)
##                        Input Accuracy Precision    Recall        F1
## Accuracy...1 Content_XGBoost 0.997884 0.9989440 0.9970489 0.9979956
## Accuracy...2     Content_GLM 0.993986 0.9976655 0.9909359 0.9942893

  • Both models perform exceptionally well when using content-only input, but the XGBoost model slightly outperforms the GLM model across all key metrics.

  • XGBoost achieves higher accuracy (99.79% vs. 99.39%), precision (99.89% vs. 99.77%), recall (99.70% vs. 99.09%), and F1 score (99.80% vs. 99.43%).

  • This indicates that XGBoost is not only more accurate but also slightly better at identifying true fake news while minimizing false positives.

  • Hence, for maximum performance in fake news detection, the XGBoost content-only model is the optimal choice.

  • XGBoost outperforms GLM in this project as it excellent at capturing non-linear relationships and complex patterns in high-dimensional text data.

  • GLM (which assumes a linear relationship between features and output), XGBoost can handle noisy, sparse data — such as bag-of-words or DTM — more effectively by learning interactions between words.

Conclusion

End of Report.