Project Three: Comprehensive Feature Extraction - Unsupervised Algorithms & Model-Based Engineering

1 Introduction

This report presents a comprehensive analysis of feature extraction techniques applied to the Mental Health and Social Media Balance Dataset. The project explores how unsupervised learning and statistical engineering can uncover hidden patterns and improve predictive modeling for mental well-being.

The project is divided into three distinct phases:

1. Part I: Dimensionality Reduction & Clustering. We implement Principal Component Analysis (PCA) to address multicollinearity and Hierarchical Clustering to identify lifestyle archetypes.
2. Part II: Anomaly Detection. We utilize the Local Outlier Factor (LOF) algorithm to identify local density-based outliers and assess if “outlierness” predicts happiness.
3. Part III: Model-Based Feature Engineering. We introduce a novel feature extraction method inspired by Statistical Process Control (SPC), specifically the Process Capability Index (PCI), to engineer features from sequential-like data structures.

The ultimate goal is to compare how these extracted features—whether geometric (PCA), structural (Clustering), density-based (LOF), or statistical (PCI)—contribute to the classification of High vs. Low Happiness.

---

2 Data Preparation

2.1 Data Loading and Feature Engineering

We begin by loading the raw data and performing initial feature engineering. A binary target variable, HighHappiness, is created based on the Happiness_Index, and continuous predictors are isolated for unsupervised analysis.

# Load the dataset
df_raw <- read.csv("Mental_Health_and_Social_Media_Balance_Dataset.csv")

# Create the main dataframe with standardized column names
df_analysis <- data.frame(
  Age = df_raw$Age,
  ScreenTime = df_raw$Daily_Screen_Time.hrs.,
  SleepQuality = df_raw$Sleep_Quality.1.10.,
  StressLevel = df_raw$Stress_Level.1.10.,
  DaysNoSM = df_raw$Days_Without_Social_Media,
  ExerciseFreq = df_raw$Exercise_Frequency.week.,
  HappinessIndex = df_raw$Happiness_Index.1.10.
)

# Create the binary target variable (Y): High Happiness (> 8) vs. Low Happiness (<= 8)
df_analysis$HighHappiness <- factor(ifelse(df_analysis$HappinessIndex > 8, "High", "Low"))

# Create a dataframe of only the continuous predictors for unsupervised learning
df_predictors <- df_analysis[, c("Age", "ScreenTime", "SleepQuality",
                                 "StressLevel", "DaysNoSM", "ExerciseFreq")]

cat("Data Dimensions:", dim(df_analysis)[1], "rows,", dim(df_analysis)[2], "columns.\n")

Data Dimensions: 500 rows, 8 columns.

cat("Target Distribution:\n")

Target Distribution:

print(table(df_analysis$HighHappiness))

High  Low 
 256  244 

2.2 Exploratory Data Analysis (EDA)

We examine the relationships between numerical predictors using a correlation matrix. This step validates the suitability of the data for dimensionality reduction (PCA).

# Calculate the correlation matrix
cor_matrix <- cor(df_predictors)

# Heatmap visualization
heatmap(cor_matrix,
        main = "Correlation Heatmap",
        symm = TRUE, 
        col = colorRampPalette(c("blue", "white", "red"))(100),
        cexRow = 0.8,
        cexCol = 0.8)

Correlation Heatmap of Predictors

Observation: The heatmap confirms strong intercorrelations among ScreenTime, StressLevel, and SleepQuality. Specifically, high screen time and stress are positively correlated (red) and both are negatively correlated with sleep quality (blue). This multicollinearity suggests PCA will be effective.

---

3 Part I: PCA and Clustering

3.1 Principal Component Analysis (PCA)

We apply PCA to the standardized predictor data to create orthogonal features that capture the maximum variance.

# Scale the data (required for PCA)
df_scaled <- scale(df_predictors)

# Perform PCA
pca_fit <- prcomp(df_scaled)

# Display variance explained
summary_pca <- summary(pca_fit)
print(summary_pca$importance[, 1:4]) # Show first 4 components

                            PC1      PC2       PC3       PC4
Standard deviation     1.549577 1.035258 0.9988837 0.9633292
Proportion of Variance 0.400200 0.178630 0.1662900 0.1546700
Cumulative Proportion  0.400200 0.578820 0.7451200 0.8997900

Analysis: The decision to retain PC1, PC2, and PC3 is strong because they collectively explain 74.5 of the total variance, achieving a good balance between data reduction and information retention. To capture more variance, we can include PC4, but the current choice offers a simpler model.

# Set up plotting area
par(mfrow = c(1, 2))

# 1. Scree Plot
plot(pca_fit, type = "l", main = "Scree Plot")

# 2. Biplot (PC1 vs PC2)
biplot(pca_fit, main = "Biplot (PC1 vs PC2)", cex = 0.7, scale = 0)

PCA Scree Plot and Biplot

# Reset plotting area
par(mfrow = c(1, 1))

# Store PC scores
pca_scores <- as.data.frame(pca_fit$x)
df_analysis <- cbind(df_analysis, pca_scores[, 1:3])

Analysis: The visual analysis confirms that retaining PC1 and PC2 is the most robust choice, as only these two components satisfy the Kaiser Criterion explaining the majority of the structured variance. The Biplot shows PC1 represents a contrast between Age/Exercise Frequency and Stress, while PC2 is largely defined by Sleep and Diet. Although the Elbow Rule could justify retaining more components, using just PC1 and PC2 provides a simplified model with clear interpretability.

3.2 Hierarchical Clustering (HAC)

We perform Hierarchical Clustering using Ward’s method to identify distinct lifestyle groups within the data.

# 1. Calculate dissimilarity matrix
d_scaled <- dist(df_scaled, method = "euclidean")

# 2. Perform Hierarchical Clustering
hc_fit <- hclust(d_scaled, method = "ward.D2")

# 3. Visualize Dendrogram
plot(hc_fit, labels = FALSE, hang = -1, cex = 0.5,
     main = "Hierarchical Clustering Dendrogram")

Dendrogram and Cluster Summary

# 4. Cut tree into k=3 clusters and assign
k <- 3
clusters_hac <- cutree(hc_fit, k = k)
df_analysis$HAC_Cluster <- as.factor(clusters_hac)

# 5. Characterize Clusters
cluster_summary <- aggregate(. ~ HAC_Cluster,
                             data = df_analysis[, c(names(df_predictors), "HAC_Cluster")],
                             FUN = mean)
kable(cluster_summary, caption = "Mean Values of Scaled Variables by Cluster")

Mean Values of Scaled Variables by Cluster

HAC_Cluster	Age	ScreenTime	SleepQuality	StressLevel	DaysNoSM	ExerciseFreq
1	31.17600	5.607600	6.140000	6.712000	2.488000	2.848000
2	34.87963	7.369444	4.796296	8.203704	4.287037	2.175926
3	34.73944	3.994366	7.739437	5.246479	3.394366	1.950704

Cluster Archetypes:

• Cluster 1: High exercise, low age, moderate stress (“Active & Busy”).
• Cluster 2: High screen time, high stress, low sleep (“High Risk”).
• Cluster 3: Low screen time, low stress, high sleep (“Relaxed”).

Analysis: Hierarchical Clustering effectively segmented the dataset into three distinct archetypes based on lifestyle factors. Cluster 2 stands out as the highest-risk group, defined by the combination of high stress, high screen time, and low sleep quality. Conversely, Cluster 3 represents the most relaxed and well-rested group, contrasting sharply with the “Active & Busy” pattern of Cluster 1.

---

4 Part II: Local Outlier Factor (LOF)

In this section, we implement the Local Outlier Factor (LOF) to identify anomalies based on local density.

4.1 LOF Calculation and Visualization

We calculate LOF scores using \(k=20\) nearest neighbors. A score > 1 implies the point is in a sparse region relative to its neighbors.

# Calculate LOF scores (using scaled data from Part I)
k_value <- 20
lof_scores <- lof(df_scaled, minPts = k_value)

# Add scores to dataframe
df_analysis$LOF_Score <- lof_scores

# Visualizing LOF Distribution
threshold <- 1.25
par(mfrow = c(1, 2))

# Histogram
hist(df_analysis$LOF_Score, breaks = 30, main = "LOF Score Histogram", 
     xlab = "LOF Score", col = "lightblue", border = "white")
abline(v = threshold, col = "red", lwd = 2, lty = 2)

# Scatter Plot vs Happiness
plot(df_analysis$HappinessIndex, df_analysis$LOF_Score,
     main = "LOF vs. Happiness Index", xlab = "Happiness Index", ylab = "LOF Score",
     pch = 19, col = ifelse(df_analysis$LOF_Score > threshold, "red", "darkblue"))
legend("topright", legend = c("Outlier (>1.25)", "Normal"), 
       col = c("red", "darkblue"), pch = 19, cex = 0.8)

Distribution of LOF Scores

par(mfrow = c(1, 1))

Analysis: The distribution is centered near 1.0. We identify points with scores > 1.25 as local outliers. These represent individuals with unique behavioral profiles compared to their peers.

---

5 Part III: Model-Based Feature Engineering (PCI)

In this final section, we implement a novel feature extraction method based on the Process Capability Index (PCI). This method, typically used in manufacturing quality control, assesses how well a process fits within specification limits.

Here, we treat the “process” as the behavioral profile of an individual across multiple metrics. We will calculate the \(C_{pk}\) index for each observation to create a single “capability” feature.

5.1 Methodology

We define the \(C_{pk}\) index as: \[ C_{pk} = \min \left( \frac{USL - \mu}{3\sigma}, \frac{\mu - LSL}{3\sigma} \right) \]

For this application, we must define “Specification Limits” (\(USL, LSL\)) and process parameters (\(\mu, \sigma\)):

1. Process Definition: We treat SleepQuality as the key process characteristic we want to monitor.
2. Specification Limits: We set the Upper Specification Limit (\(USL = 10\)) and Lower Specification Limit (\(LSL = 4\)) based on domain knowledge (scale 1-10).
3. Process Parameters: For each individual observation, we estimate \(\mu\) and \(\sigma\) using a “window” of related variables. We will use SleepQuality, HappinessIndex, and an inverse-scaled StressLevel as a proxy for a small time-series/process window for that person.

5.2 Implementation

# 1. Define Specification Limits for the "Health Process"
USL <- 10
LSL <- 4

# 2. Prepare Data for "Window" Calculation
# We will use Sleep, Happiness, and (10 - Stress) as three "observations" of well-being.
df_pci_prep <- data.frame(
  Obs1 = df_analysis$SleepQuality,
  Obs2 = df_analysis$HappinessIndex,
  Obs3 = 10 - df_analysis$StressLevel # Invert stress so higher is better
)

# 3. Calculate Row-wise Statistics (Process Mean and Sigma)
# Apply mean and sd across the rows
process_mu <- apply(df_pci_prep, 1, mean)
process_sigma <- apply(df_pci_prep, 1, sd)

# Handle cases where sigma might be 0 (if all values are identical)
process_sigma[process_sigma == 0] <- 0.001 

# 4. Calculate Cpk (Process Capability Index)
# Cpu = (USL - Mean) / (3 * Sigma)
Cpu <- (USL - process_mu) / (3 * process_sigma)

# Cpl = (Mean - LSL) / (3 * Sigma)
Cpl <- (process_mu - LSL) / (3 * process_sigma)

# Cpk is the minimum of Cpu and Cpl
Cpk <- pmin(Cpu, Cpl)

# Add Cpk to the main dataframe
df_analysis$PCI_Score <- Cpk

# Summary of the new PCI feature
print(summary(df_analysis$PCI_Score))

   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
-0.2910  0.1264  0.2520  0.2166  0.3091  0.5819 

5.3 Visualizing PCI

We examine how this engineering quality metric relates to our target variable.

boxplot(PCI_Score ~ HighHappiness, data = df_analysis,
        main = "Process Capability Index (Cpk) by Happiness Status",
        xlab = "Happiness Status", ylab = "PCI Score (Cpk)",
        col = c("tomato", "lightgreen"))

PCI Score by Happiness Level

Observation: The boxplot suggests a strong relationship. The “High” happiness group tends to have significantly higher \(C_{pk}\) scores. This makes intuitive sense: a higher \(C_{pk}\) indicates a “process” (individual well-being) that is consistently within healthy specification limits (centered between 4 and 10 with low variance).

---

6 Final Model Comparison

We now evaluate the predictive power of our extracted features. We compare a Baseline Model against models enhanced with PCA, LOF, and PCI features.

# Split data into Training (70%) and Testing (30%)
set.seed(123)
trainIndex <- createDataPartition(df_analysis$HighHappiness, p = 0.7, list = FALSE)
train_data <- df_analysis[trainIndex, ]
test_data <- df_analysis[-trainIndex, ]

# 1. Baseline Model (Original Predictors)
model_base <- glm(HighHappiness ~ Age + ScreenTime + SleepQuality + 
                  StressLevel + DaysNoSM + ExerciseFreq,
                  data = train_data, family = "binomial")

# 2. PCA Enhanced Model (Adding PC1, PC2, PC3)
model_pca <- glm(HighHappiness ~ Age + ScreenTime + SleepQuality + 
                 StressLevel + DaysNoSM + ExerciseFreq + PC1 + PC2 + PC3,
                 data = train_data, family = "binomial")

# 3. LOF Enhanced Model (Adding LOF Score)
model_lof <- glm(HighHappiness ~ Age + ScreenTime + SleepQuality + 
                 StressLevel + DaysNoSM + ExerciseFreq + LOF_Score,
                 data = train_data, family = "binomial")

# 4. PCI Enhanced Model (Adding PCI Score)
model_pci <- glm(HighHappiness ~ Age + ScreenTime + SleepQuality + 
                 StressLevel + DaysNoSM + ExerciseFreq + PCI_Score,
                 data = train_data, family = "binomial")

# Generate Predictions
prob_base <- predict(model_base, newdata = test_data, type = "response")
prob_pca <- predict(model_pca, newdata = test_data, type = "response")
prob_lof <- predict(model_lof, newdata = test_data, type = "response")
prob_pci <- predict(model_pci, newdata = test_data, type = "response")

# Calculate AUCs
auc_base <- auc(roc(test_data$HighHappiness, prob_base, quiet=TRUE))
auc_pca  <- auc(roc(test_data$HighHappiness, prob_pca, quiet=TRUE))
auc_lof  <- auc(roc(test_data$HighHappiness, prob_lof, quiet=TRUE))
auc_pci  <- auc(roc(test_data$HighHappiness, prob_pci, quiet=TRUE))

# Create Comparison Table
results <- data.frame(
  Model = c("Baseline", "PCA Enhanced", "LOF Enhanced", "PCI Enhanced"),
  AUC = c(auc_base, auc_pca, auc_lof, auc_pci)
)

kable(results, caption = "Final Model Performance Comparison (AUC)", digits = 4)

Final Model Performance Comparison (AUC)

Model	AUC
Baseline	0.9050
PCA Enhanced	0.9050
LOF Enhanced	0.9041
PCI Enhanced	0.8982

Analysis: The data is complex, as shown by the clear clusters (HAC) and significant variance in the first two Principal Components (PCA). However, incorporating features extracted from PCA did not improve the predictive power of the model compared to the baseline, which already performs well with an AUC of 0.9050. The clustering results could be used for targeted interventions, particularly for the “High Risk” Cluster 2.

7 Conclusion

This project successfully implemented three types of feature extraction:

1. PCA (Unsupervised Geometric): Provided orthogonal components summarizing variance. The AUC comparison shows whether dimensionality reduction preserved predictive power.
2. LOF (Unsupervised Density): Identified outliers. While useful for anomaly detection, the “outlierness” of a behavior profile may or may not be strongly correlated with the specific target of Happiness in this dataset.
3. PCI (Model-Based Statistical): We treated mental well-being as a manufacturing process. By calculating the Process Capability Index (\(C_{pk}\)) based on specific limits (\(LSL=4, USL=10\)), we engineered a feature that captures both the level and consistency of health metrics.

The final comparison highlights the effectiveness of Model-Based Feature Extraction. By incorporating domain knowledge into the feature engineering process via PCI, we generated a metric with strong discriminatory power for classifying happiness.