Assignment 8: Implement EDA/Clustering Analysis and Interpret (Signature Assignment)
Mina Fard
2024-03-03
Install required packages
install.packages(“caret”) install.packages(“corrplot”) install.packages(“factoextra”)
#Load required libraries
library(caret)
library(ggplot2)
library(corrplot)
library(factoextra)
#Load data set
file_path <- "C:\\Users\\fardm\\OneDrive\\Desktop\\PhD\\american_bankruptcy.csv"
bank_data <- read.csv("C:\\Users\\fardm\\OneDrive\\Desktop\\PhD\\american_bankruptcy.csv")Data Preprocessing:
• Identify variable types and redefine/recategorize as necessary.
Two variables are character type, one is integer and rest are of numeric type
#changing type of the character variables
bank_data$company_name <- as.factor(bank_data$company_name)
bank_data$status_label <- as.factor(bank_data$status_label)• Analyze and report the structure of the data.
#Structure of the data
str(bank_data)## 'data.frame': 78682 obs. of 21 variables:
## $ company_name: Factor w/ 8971 levels "C_1","C_10","C_100",..: 1 1 1 1 1 1 1 1 1 1 ...
## $ status_label: Factor w/ 2 levels "alive","failed": 1 1 1 1 1 1 1 1 1 1 ...
## $ year : int 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 ...
## $ X1 : num 511 486 437 396 432 ...
## $ X2 : num 833 714 526 497 523 ...
## $ X3 : num 18.4 18.6 22.5 27.2 26.7 ...
## $ X4 : num 89 64.4 27.2 30.7 47.5 ...
## $ X5 : num 336 321 287 260 247 ...
## $ X6 : num 35.2 18.5 -58.9 -12.4 3.5 ...
## $ X7 : num 128.3 115.2 77.5 66.3 104.7 ...
## $ X8 : num 373 377 365 143 309 ...
## $ X9 : num 1024 874 639 606 652 ...
## $ X10 : num 741 702 710 687 709 ...
## $ X11 : num 180 180 218 165 249 ...
## $ X12 : num 70.66 45.79 4.71 3.57 20.81 ...
## $ X13 : num 191 160 112 110 129 ...
## $ X14 : num 164 125 150 204 131 ...
## $ X15 : num 201 204 140 124 132 ...
## $ X16 : num 1024 874 639 606 652 ...
## $ X17 : num 401 362 400 392 408 ...
## $ X18 : num 935 810 612 576 604 ...There are 21 variables in the data set and 78682 observations
• Evaluate for missing data, outliers, and anomalies and handle them.
#checking for missing values
apply(is.na(bank_data),2,sum)## company_name status_label year X1 X2 X3
## 0 0 0 0 0 0
## X4 X5 X6 X7 X8 X9
## 0 0 0 0 0 0
## X10 X11 X12 X13 X14 X15
## 0 0 0 0 0 0
## X16 X17 X18
## 0 0 0#Checking for Outliers
boxplot(bank_data$X17)
#Removing outliers
outliers <- boxplot(bank_data$X1, plot = FALSE)$out
bank_data <- bank_data[-which(bank_data$X1 %in% outliers),] No missing value in the data set
• Conduct univariate/multivariate graphical analysis.
Univariate graphical analysis
#status_label variable
ggplot(data = bank_data,aes(status_label)) +
geom_bar(fill= "blue",alpha=0.5)+
theme(legend.position = "None") + theme_minimal()
# Combine all numeric variables into a long format
long_data <- reshape2::melt(bank_data[4:21])
# Plot using ggplot2
ggplot(long_data, aes(x = value)) +
geom_histogram(fill= "blue",alpha=0.5) +
xlim(-10,200) +
facet_wrap(~variable, scales = "free") +
theme_minimal()
#
ggplot(data = bank_data,aes(X1, fill=status_label)) +
geom_density(alpha=0.4)+ xlim(-2,5)+
theme(legend.position = "None") + theme_minimal()
#correlatio plot
cor_mat <- cor(bank_data[4:21])
corrplot(cor_mat)
Numeric variables are not uniformly distributed.
• Evaluate and report on univariate/multivariate statistics.
#outcome variable
table(bank_data$status_label)##
## alive failed
## 62917 4791#mean values of the numeric variables
mean_var <- sapply(bank_data[4:17],mean)
#max values of the numeric variables
max_var <- sapply(bank_data[4:17],max)
#min values of the numeric variables
min_var <- sapply(bank_data[4:17],min)
stat <- data.frame(cbind(mean_var,max_var,min_var))
stat## mean_var max_var min_var
## X1 168.867816 1050.431 -7.7600
## X2 300.304403 15232.449 -366.6450
## X3 26.332285 3010.068 0.0000
## X4 57.376636 3131.000 -4512.1350
## X5 38.478249 903.294 0.0000
## X6 3.740006 24728.000 -7751.2480
## X7 55.685755 940.216 -0.0060
## X8 632.407727 253635.500 0.0001
## X9 446.377034 16199.300 -1964.9990
## X10 544.480343 29253.741 0.0010
## X11 159.463576 19973.000 -0.0230
## X12 31.044197 2491.000 -5083.0000
## X13 146.072754 4476.700 -4367.5710
## X14 98.564795 7423.688 0.0010Data Splitting and Transformation:
• Conduct transformation/scaling.
#scaling numeric variables
bank_data[4:21] <- scale(bank_data[4:21])
bank_data$company_name <- as.numeric(bank_data$company_name)• Split your data into training/test sets without leakage.
# Set random seed
set.seed(123)
# Creating index for splitting the data
index <- createDataPartition(bank_data$status_label, p = 0.8, list = FALSE)
# Split the data into training (80%) and test sets (20%)
trainset <- bank_data[index, ]
testset <- bank_data[-index, ]Clustering and Model Building:
• Build a 2-means cluster model (k=2) and interpret the results.
# Build a 2-means cluster model
kmeans_model <- kmeans(trainset[-2], centers = 2)
#center of each cluster for each numeric variable
kmeans_model$centers## company_name year X1 X2 X3 X4
## 1 2072.394 2006.741 0.05599586 0.08291335 0.02540138 0.05774558
## 2 6434.380 2007.593 -0.04829211 -0.07217393 -0.02081999 -0.05103554
## X5 X6 X7 X8 X9 X10
## 1 0.1002371 0.03467935 0.06246193 0.009633919 0.08130507 0.04383865
## 2 -0.0896018 -0.03289809 -0.05600331 -0.010462544 -0.07054073 -0.04040342
## X11 X12 X13 X14 X15 X16
## 1 0.01900338 0.05897118 0.05398887 0.04787952 0.08266085 0.08130507
## 2 -0.01472996 -0.05307925 -0.04630921 -0.04446276 -0.08277815 -0.07054073
## X17 X18
## 1 0.03891287 0.07929746
## 2 -0.03495843 -0.06858631Positive values indicate that, on average, cluster 1 has higher values for the variables, while negative values indicate that cluster 2 has higher values.
#cluster plot
fviz_cluster(kmeans_model, data = trainset[-2]) +
theme_minimal()
Model Evaluation:
• Discuss how well your clustering performed. • Calculate precision (positive predictive value) on your test set and interpret. • Calculate recall (sensitivity) on your test set and interpret. • Calculate accuracy on your test set and interpret. • Calculate the F1-scores on your test set and interpr
kmeans_test <- kmeans(testset[-2], centers = 2)
# Assign cluster labels based on majority class
cluster_labels <- ifelse(kmeans_test$cluster == 1, "alive", "failed")
# Confusion matrix
conf_matrix <- table(cluster_labels, testset$status_label)
# Precision
precision <- conf_matrix["alive", "failed"] / sum(conf_matrix["alive", ])
precision## [1] 0.07458308#precision is approximately 0.075, suggesting that out of the instances predicted as "alive," only about 7.5% are correctly predicted, while the majority are false positives.
# Recall
recall <- conf_matrix["alive", "failed"] / sum(conf_matrix[, "alive"])
recall## [1] 0.03838512# recall value is 0.038, indicating that the model captures only about 3.8% of the actual "alive" instances.
# Accuracy
accuracy <- sum(diag(conf_matrix)) / sum(conf_matrix)
accuracy## [1] 0.4776604# accuracy is approximately 0.478, suggesting that the model is correct in its predictions for about 47.8% of the instances. However, accuracy can be misleading in imbalanced datasets
# F1-Score
f1_score <- 2 * (precision * recall) / (precision + recall)
f1_score## [1] 0.05068472#F1-score is approximately 0.051, indicating a low overall performance, likely due to the low precision and recall values.Discussion and Future Work
• Discuss limitations of your analysis.
- If the data used for clustering contains noise, outliers, or missing values, it can significantly impact the quality of the clusters. Pre-processing steps such as data cleaning and normalization may be required. -K-means presupposes that clusters are spherical, equally large, and have comparable variances. If the data does not meet these criteria, K-means performance may be unsatisfactory.
• Discuss logical future work.
- We can Explore additional feature engineering techniques to create more informative variables or combine existing ones. This can potentially improve the performance of the clustering algorithm
- Use ensemble clustering methods, which integrate the results of various clustering algorithms, to improve robustness and stability.