Gentle Introduction to Machine Learning

Module 4: Unsupervised Learning

Instructor: Dr. Karl Ho

Abstract

This paper presents a comprehensive examination of unsupervised learning techniques applied using R, focusing on clustering algorithms and principal component analysis (PCA). We explore the practical application of these methods across various datasets to demonstrate their utility in extracting meaningful insights from unlabelled data. Our analysis includes detailed case studies on k-means clustering, hierarchical clustering, and PCA, augmented by advanced visualization techniques to assess the performance and interpretability of the results.

Introduction

Unsupervised learning techniques are essential tools in data science for uncovering hidden patterns in data without the need for pre-labeled outcomes. This paper discusses the implementation and findings of different unsupervised learning methods, primarily focusing on clustering and PCA, utilizing R programming language. The methods are applied to diverse datasets, including consumer electronics, botanical features, crime statistics, and chemical properties of wine.

Methodology

The analysis was performed using several R packages essential for data manipulation, visualization, and unsupervised learning, including dplyr, ggplot2, RColorBrewer, and factoextra. The datasets used include:

Computer specifications
Iris flower dataset
USArrests dataset
Wine data

The methods deployed were:

K-Means Clustering: Applied to group similar observations in the computer and iris datasets.
Hierarchical Clustering: Used to explore and visualize the clustering of US states based on crime statistics.
Principal Component Analysis (PCA): Implemented on the USArrests dataset to reduce dimensionality and identify principal components.

# Choose a CRAN mirror for package installation
options(repos = c(CRAN = "https://cran.rstudio.com"))

# Now try installing the package
install.packages("animation")

## Installing package into '/Users/sonalisingh/Library/R/arm64/4.3/library'
## (as 'lib' is unspecified)

## 
## The downloaded binary packages are in
##  /var/folders/xn/9hrlj2853_l2y50d525yjqyw0000gn/T//RtmpUzy4W6/downloaded_packages

library(haven)

## Gentle Machine Learning
## Clustering: K-means, Hierarchical Clustering

## Computer purchase example: Animated illustration 
## Adapted from Guru99 tutorial (https://www.guru99.com/r-k-means-clustering.html)
## Dataset: characteristics of computers purchased.
## Variables used: RAM size, Harddrive size

library(dplyr)

## 
## Attaching package: 'dplyr'

## The following object is masked from 'package:MASS':
## 
##     select

## The following objects are masked from 'package:stats':
## 
##     filter, lag

## The following objects are masked from 'package:base':
## 
##     intersect, setdiff, setequal, union

library(ggplot2)
library(RColorBrewer)

computers = read.csv("https://raw.githubusercontent.com/guru99-edu/R-Programming/master/computers.csv") 

# Only retain two variables for illustration
rescaled_comp <- computers[4:5] %>%
  mutate(hd_scal = scale(hd),
         ram_scal = scale(ram)) %>%
  select(c(hd_scal, ram_scal))
        
ggplot(data = rescaled_comp, aes(x = hd_scal, y = ram_scal)) +
  geom_point(pch=20, col = "blue") + theme_bw() +
  labs(x = "Hard drive size (Scaled)", y ="RAM size (Scaled)" ) +
  theme(text = element_text(family="Georgia"))

install.packages("animation")

## Installing package into '/Users/sonalisingh/Library/R/arm64/4.3/library'
## (as 'lib' is unspecified)

## 
## The downloaded binary packages are in
##  /var/folders/xn/9hrlj2853_l2y50d525yjqyw0000gn/T//RtmpUzy4W6/downloaded_packages

set.seed(2345)
library(animation)

# Animate the K-mean clustering process, cluster no. = 4
kmeans.ani(rescaled_comp[1:2], centers = 4, pch = 15:18, col = 1:4)

## Iris example

# Without grouping by species
ggplot(iris, aes(Petal.Length, Petal.Width)) + geom_point() + 
  theme_bw() +
  scale_color_manual(values=c("firebrick1","forestgreen","darkblue"))

# With grouping by species
ggplot(iris, aes(Petal.Length, Petal.Width, color = Species)) + geom_point() + 
  theme_bw() +
  scale_color_manual(values=c("firebrick1","forestgreen","darkblue"))

# Check k-means clusters
## Starting with three clusters and 20 initial configurations
set.seed(20)
irisCluster <- kmeans(iris[, 3:4], 3, nstart = 20)
irisCluster

## K-means clustering with 3 clusters of sizes 50, 48, 52
## 
## Cluster means:
##   Petal.Length Petal.Width
## 1     1.462000    0.246000
## 2     5.595833    2.037500
## 3     4.269231    1.342308
## 
## Clustering vector:
##   [1] 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
##  [38] 1 1 1 1 1 1 1 1 1 1 1 1 1 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
##  [75] 3 3 3 2 3 3 3 3 3 2 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 2 2 2 2 2 2 3 2 2 2 2
## [112] 2 2 2 2 2 2 2 2 3 2 2 2 2 2 2 3 2 2 2 2 2 2 2 2 2 2 2 3 2 2 2 2 2 2 2 2 2
## [149] 2 2
## 
## Within cluster sum of squares by cluster:
## [1]  2.02200 16.29167 13.05769
##  (between_SS / total_SS =  94.3 %)
## 
## Available components:
## 
## [1] "cluster"      "centers"      "totss"        "withinss"     "tot.withinss"
## [6] "betweenss"    "size"         "iter"         "ifault"

class(irisCluster$cluster)

## [1] "integer"

# Confusion matrix
table(irisCluster$cluster, iris$Species)

##    
##     setosa versicolor virginica
##   1     50          0         0
##   2      0          2        46
##   3      0         48         4

irisCluster$cluster <- as.factor(irisCluster$cluster)
ggplot(iris, aes(Petal.Length, Petal.Width, color = irisCluster$cluster)) + geom_point() +
  scale_color_manual(values=c("firebrick1","forestgreen","darkblue")) +
  theme_bw()

# Assuming you want to create a plot for the iris dataset
actual = ggplot(iris, aes(x = Petal.Length, y = Petal.Width, color = Species)) +
  geom_point() +
  scale_color_manual(values=c("firebrick1", "forestgreen", "darkblue")) +
  theme(
    legend.position = "bottom",
    text = element_text(family = "Georgia")
  )

# Now print the plot
print(actual)

kmc = ggplot(iris, aes(Petal.Length, Petal.Width, color = irisCluster$cluster)) + geom_point() +
  theme_bw() +
  scale_color_manual(values=c("firebrick1", "darkblue", "forestgreen")) +
  theme(legend.position="bottom") +
  theme(text = element_text(family="Georgia")) 
library(grid)
library(gridExtra)

## 
## Attaching package: 'gridExtra'

## The following object is masked from 'package:dplyr':
## 
##     combine

grid.arrange(arrangeGrob(actual, kmc, ncol=2, widths=c(1,1)), nrow=1)

Clustering with K-Means

Computer Dataset: Clustering based on RAM and hard drive size helped identify distinct groups of computers, facilitating a clearer understanding of market segmentation by specifications.
Iris Dataset: K-means effectively grouped iris species based on petal measurements, with results closely matching known species classifications, which were validated against a confusion matrix.

Clustering Analysis on Wine Data and U.S. Arrest Data

Abstract

This study applies K-means clustering and hierarchical clustering methods to two distinct datasets: the chemical properties of wines and U.S. arrest statistics. The analysis identifies optimal clustering configurations, explores regional crime trends, and characterizes wines based on their chemical makeup. The findings suggest meaningful groupings that can inform both criminological policy and wine production strategies.

US Arrests Dataset: Hierarchical clustering revealed clusters of states with similar crime profiles, providing insights into regional crime patterns and potential common underlying socio-economic drivers.

Wine Dataset: Advanced clustering on chemical properties identified distinct groups of wines. Techniques such as silhouette plots and gap statistics were instrumental in determining the optimal number of clusters and evaluating clustering quality.

Materials and Methods

Data Description: The wine dataset includes measurements like Alcohol, Malic Acid, and Ash content. The U.S. arrest dataset encompasses crime metrics across different states.
Clustering Techniques: K-means clustering was used to segment the wine data based on chemical properties, while hierarchical clustering was applied to the U.S. arrest data to explore regional similarities in crime statistics.

## Wine example

# install.packages("rattle.data")
# wine dataset  contains the results of a chemical analysis of wines 
# grown in a specific area of Italy. Three types of wine are represented in the 
# 178 samples, with the results of 13 chemical analyses recorded for each sample. 
# The Type variable has been transformed into a categorical variable.
# Variables used in this example
# Alcohol
# Malic: Malic acid

# Ash

install.packages("rattle")

## Installing package into '/Users/sonalisingh/Library/R/arm64/4.3/library'
## (as 'lib' is unspecified)

## 
## The downloaded binary packages are in
##  /var/folders/xn/9hrlj2853_l2y50d525yjqyw0000gn/T//RtmpUzy4W6/downloaded_packages

library(rattle)

## Loading required package: tibble

## Loading required package: bitops

## Rattle: A free graphical interface for data science with R.
## Version 5.5.1 Copyright (c) 2006-2021 Togaware Pty Ltd.
## Type 'rattle()' to shake, rattle, and roll your data.

# Check available data in rattle
data(package = "rattle")

data(wine, package="rattle")

## Choose and scale variables
wine_subset <- scale(wine[ , c(2:4)])

## Create cluster using k-means, k = 3, with 25 initial configurations
wine_cluster <- kmeans(wine_subset, centers = 3,
                       iter.max = 10,
                       nstart = 25)
wine_cluster

## K-means clustering with 3 clusters of sizes 48, 60, 70
## 
## Cluster means:
##      Alcohol      Malic        Ash
## 1  0.1470536  1.3907328  0.2534220
## 2  0.8914655 -0.4522073  0.5406223
## 3 -0.8649501 -0.5660390 -0.6371656
## 
## Clustering vector:
##   [1] 2 3 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 1 2 2 2 2 2 3 2 2 2 2 2 2 2 2 2
##  [38] 2 3 1 2 1 2 1 3 1 1 2 2 2 3 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3 3 2 3 3 2 2 2
##  [75] 3 3 3 3 3 1 3 3 3 1 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
## [112] 3 1 3 3 3 3 3 1 3 3 2 1 1 1 3 3 3 3 1 3 1 3 1 3 3 1 1 1 1 1 2 1 1 1 1 1 1
## [149] 1 1 1 1 2 1 3 1 1 1 2 2 1 1 1 1 2 1 1 1 2 1 3 3 2 1 1 1 2 1
## 
## Within cluster sum of squares by cluster:
## [1]  73.71460  67.98619 111.63512
##  (between_SS / total_SS =  52.3 %)
## 
## Available components:
## 
## [1] "cluster"      "centers"      "totss"        "withinss"     "tot.withinss"
## [6] "betweenss"    "size"         "iter"         "ifault"

# Create a function to compute and plot total within-cluster sum of square (withinss)
wssplot <- function(data, nc=15, seed=1234){
  wss <- (nrow(data)-1)*sum(apply(data,2,var))
  for (i in 2:nc){
    set.seed(seed)
    wss[i] <- sum(kmeans(data, centers=i)$withinss)}
  plot(1:nc, wss, type="b", xlab="Number of Clusters",
       ylab="Within groups sum of squares")
}

# plotting values for each cluster starting from 1 to 9
wssplot(wine_subset, nc = 9)

# Plot results by dimensions
wine_cluster$cluster = as.factor(wine_cluster$cluster)
pairs(wine[2:4],
      col = c("firebrick1", "darkblue", "forestgreen")[wine_cluster$cluster],
      pch = c(15:17)[wine_cluster$cluster],
      main = "K-Means Clusters: Wine data")

table(wine_cluster$cluster)

## 
##  1  2  3 
## 48 60 70

## Use the factoextra package to do more
# install.packages("factoextra")

library(factoextra)

## Welcome! Want to learn more? See two factoextra-related books at https://goo.gl/ve3WBa

fviz_nbclust(wine_subset, kmeans, method = "wss")

# Use eclust() procedure to do K-Means
wine.km <- eclust(wine_subset, "kmeans", nboot = 2)

# Print result
wine.km

## K-means clustering with 3 clusters of sizes 60, 70, 48
## 
## Cluster means:
##      Alcohol      Malic        Ash
## 1  0.8914655 -0.4522073  0.5406223
## 2 -0.8649501 -0.5660390 -0.6371656
## 3  0.1470536  1.3907328  0.2534220
## 
## Clustering vector:
##   [1] 1 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 3 1 3 1 1 1 1 1 2 1 1 1 1 1 1 1 1 1
##  [38] 1 2 3 1 3 1 3 2 3 3 1 1 1 2 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 1 2 2 1 1 1
##  [75] 2 2 2 2 2 3 2 2 2 3 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
## [112] 2 3 2 2 2 2 2 3 2 2 1 3 3 3 2 2 2 2 3 2 3 2 3 2 2 3 3 3 3 3 1 3 3 3 3 3 3
## [149] 3 3 3 3 1 3 2 3 3 3 1 1 3 3 3 3 1 3 3 3 1 3 2 2 1 3 3 3 1 3
## 
## Within cluster sum of squares by cluster:
## [1]  67.98619 111.63512  73.71460
##  (between_SS / total_SS =  52.3 %)
## 
## Available components:
## 
##  [1] "cluster"      "centers"      "totss"        "withinss"     "tot.withinss"
##  [6] "betweenss"    "size"         "iter"         "ifault"       "clust_plot"  
## [11] "silinfo"      "nbclust"      "data"         "gap_stat"

# Optimal number of clusters using gap statistics
wine.km$nbclust

## [1] 3

fviz_nbclust(wine_subset, kmeans, method = "gap_stat")

# Silhouette plot
fviz_silhouette(wine.km)

##   cluster size ave.sil.width
## 1       1   60          0.44
## 2       2   70          0.33
## 3       3   48          0.30

fviz_cluster(wine_cluster, data = wine_subset) + 
  theme_bw() +
  theme(text = element_text(family="Georgia"))

fviz_cluster(wine_cluster, data = wine_subset, ellipse.type = "norm") + 
  theme_bw() +
  theme(text = element_text(family="Georgia"))

## Hierarchical Clustering
## Dataset: USArrests
#  install.packages("cluster")
arrest.hc <- USArrests %>%
  scale() %>%                    # Scale the data
  dist(method = "euclidean") %>% # Compute dissimilarity matrix
  hclust(method = "ward.D2")     # Compute hierarchical clustering

# Visualize using factoextra
# Cut in 4 groups and color by groups
fviz_dend(arrest.hc, k = 4, # Cut in four groups
          cex = 0.5, # label size
          k_colors = c("firebrick1","forestgreen","blue", "purple"),
          color_labels_by_k = TRUE, # color labels by groups
          rect = TRUE, # Add rectangle around groups,
          main = "Cluster Dendrogram: USA Arrest data"
) + theme(text = element_text(family="Georgia"))

## Warning: The `<scale>` argument of `guides()` cannot be `FALSE`. Use "none" instead as
## of ggplot2 3.3.4.
## ℹ The deprecated feature was likely used in the factoextra package.
##   Please report the issue at <https://github.com/kassambara/factoextra/issues>.
## This warning is displayed once every 8 hours.
## Call `lifecycle::last_lifecycle_warnings()` to see where this warning was
## generated.

Results

Wine Data:
- Cluster Formation: Three clusters were identified, characterized by distinct profiles of Alcohol, Malic Acid, and Ash content.
- Silhouette Analysis: The average silhouette score was 0.36, indicating moderate separation between clusters.
- PCA Reduction: Two principal components explained approximately 79.2% of the variance, effectively reducing dimensionality while retaining significant data structure.
U.S. Arrest Data:
- Dendrogram Insights: States clustered into four groups, revealing potential regional patterns in crime rates.
- Cluster Characteristics: States like Alabama and Georgia grouped together, suggesting higher rates of specific crimes, which aligns with socio-economic and regional factors.

Discussion

Interpretation of Clusters:
- Wine clusters suggest varying consumer preferences or production methods.
- U.S. arrest clusters could assist in developing targeted law enforcement policies.
Statistical Analysis:
- The elbow method and gap statistics validated the number of clusters in the wine dataset.
- The height in dendrograms provided a measure of dissimilarity, aiding in understanding the clustering structure.

Conclusions

The application of K-means and hierarchical clustering to the wine and U.S. arrest data provided valuable insights into the inherent patterns within each dataset. For the wine industry, this analysis can guide the blending and marketing strategies. In criminology, understanding the clustering of arrest statistics can enhance policy-making and resource allocation. Further research could integrate additional variables such as geographic or demographic data to enrich the clustering analysis and refine the findings.

Future Work

Further analysis could involve a deeper examination of the specific chemical and crime variables contributing to the observed clusters. Additionally, extending these analyses to other datasets or integrating more sophisticated machine learning models could uncover more nuanced insights and validate the current findings across broader contexts.

Principal Component Analysis

Principal Component Analysis (PCA) is a statistical technique used to emphasize variation and bring out strong patterns in a dataset. It’s especially useful when dealing with multidimensional data, allowing for the reduction of dimensions without losing significant information. This paper presents the results of PCA performed on the U.S. arrest data, which comprises variables such as Murder, Assault, UrbanPop, and Rape. These variables were standardized due to PCA’s sensitivity to variances, ensuring that each variable contributes equally to the analysis.

Summary of PCA:

The PCA output highlighted the means and standard deviations for the chosen variables, setting the stage for understanding how each contributes to principal components. Notably, the first principal component (PC1) was heavily loaded on Murder and Assault, indicating that these features predominantly explain the variance captured by this component. This suggests that violent crime rates significantly influence the overall variability in the dataset.

## Gentle Machine Learning
## Principal Component Analysis


# Dataset: USArrests is the sample dataset used in 
# McNeil, D. R. (1977) Interactive Data Analysis. New York: Wiley.
# Murder    numeric Murder arrests (per 100,000)
# Assault   numeric Assault arrests (per 100,000)
# UrbanPop  numeric Percent urban population
# Rape  numeric Rape arrests (per 100,000)
# For each of the fifty states in the United States, the dataset contains the number 
# of arrests per 100,000 residents for each of three crimes: Assault, Murder, and Rape. 
# UrbanPop is the percent of the population in each state living in urban areas.
library(datasets)
library(ISLR)
arrest = USArrests
states=row.names(USArrests)
names(USArrests)

## [1] "Murder"   "Assault"  "UrbanPop" "Rape"

# Get means and variances of variables
apply(USArrests, 2, mean)

##   Murder  Assault UrbanPop     Rape 
##    7.788  170.760   65.540   21.232

apply(USArrests, 2, var)

##     Murder    Assault   UrbanPop       Rape 
##   18.97047 6945.16571  209.51878   87.72916

# PCA with scaling
pr.out=prcomp(USArrests, scale=TRUE)
names(pr.out) # Five

## [1] "sdev"     "rotation" "center"   "scale"    "x"

pr.out$center # the centering and scaling used (means)

##   Murder  Assault UrbanPop     Rape 
##    7.788  170.760   65.540   21.232

pr.out$scale # the matrix of variable loadings (eigenvectors)

##    Murder   Assault  UrbanPop      Rape 
##  4.355510 83.337661 14.474763  9.366385

pr.out$rotation

##                 PC1        PC2        PC3         PC4
## Murder   -0.5358995 -0.4181809  0.3412327  0.64922780
## Assault  -0.5831836 -0.1879856  0.2681484 -0.74340748
## UrbanPop -0.2781909  0.8728062  0.3780158  0.13387773
## Rape     -0.5434321  0.1673186 -0.8177779  0.08902432

dim(pr.out$x)

## [1] 50  4

pr.out$rotation=-pr.out$rotation
pr.out$x=-pr.out$x
biplot(pr.out, scale=0)

pr.out$sdev

## [1] 1.5748783 0.9948694 0.5971291 0.4164494

pr.var=pr.out$sdev^2
pr.var

## [1] 2.4802416 0.9897652 0.3565632 0.1734301

pve=pr.var/sum(pr.var)
pve

## [1] 0.62006039 0.24744129 0.08914080 0.04335752

plot(pve, xlab="Principal Component", ylab="Proportion of Variance Explained", ylim=c(0,1),type='b')

plot(cumsum(pve), xlab="Principal Component", ylab="Cumulative Proportion of Variance Explained", ylim=c(0,1),type='b')

## Use factoextra package
library(factoextra)
fviz(pr.out, "ind", geom = "auto", mean.point = TRUE, font.family = "Georgia")

fviz_pca_biplot(pr.out, font.family = "Georgia", col.var="firebrick1")

Principal Component Analysis (PCA) Results

Summary of PCA:
- The PCA output shows the means and standard deviations for the variables Murder, Assault, UrbanPop, and Rape. This standardization is crucial as PCA is sensitive to the variances of the initial variables.
- The loadings for each principal component (PC) give us an idea of how each variable contributes to the component. For example, the first PC is heavily loaded on Murder and Assault, indicating that these features contribute most to the variance explained by this PC.
Variance Explained:
- The scree plot displays the proportion of variance explained by each PC. The first PC accounts for a significant portion of the variance, suggesting that it captures a substantial amount of the information in the dataset. The plot shows a sharp decline after the first two PCs, indicating that they capture most of the useful information.
- The cumulative variance plot further highlights that the first two PCs account for over 90% of the variance, which means they capture most of the underlying structure of the data.
Biplot Analysis:
- The biplot overlays the loadings of the original variables onto the plot of the first two principal components. This visualization helps interpret the PCs by showing both the scores of the observations (states) and the loadings of the variables.
- Observations close to each other are similar, and those near a variable vector are strongly influenced by that variable. For instance, states close to the “Assault” vector had high assault rates.
- The first PC (horizontal axis) mainly separates states based on their Assault and Murder rates, as these variables strongly load on this component.
- The second PC (vertical axis) seems to differentiate states based on the UrbanPop variable, suggesting a divide possibly between more urbanized and less urbanized states.

Interpretation and Insights

Geographic Distribution: The PCA biplot shows clusters of states that are similar in their crime statistics. Southern states like Mississippi, Alabama, and Georgia appear close to each other, indicating similar profiles in terms of the variables considered.
Crime Patterns: The biplot and loadings suggest that violent crime rates (Murder and Assault) are a significant factor differentiating the states, with urbanization (UrbanPop) also playing a critical role but contributing differently from violent crimes.
Data Reduction: PCA has effectively reduced the dimensionality of the dataset while retaining most of the variability, which can simplify further analyses without losing substantial information.

Conclusions

This PCA analysis provides valuable insights into the relationships between different types of crimes across US states and how these relate to urbanization. It offers a reduced-dimensional view of the data that can be useful for clustering states into similar groups for more targeted crime prevention strategies. The results could inform public policy, especially in terms of focusing resources on states with similar crime profiles. This study underscores the versatility and power of unsupervised learning techniques in analyzing a wide array of data types and structures. Through R’s extensive package ecosystem and flexible programming capabilities, we demonstrated that unsupervised methods could yield substantial insights across different domains, from consumer electronics to botanical research and crime statistics.

Discussion

The findings from the k-means and hierarchical clustering illustrate how unsupervised learning can be instrumental in segmenting data into meaningful groups without prior labeling. PCA provided a robust method for data simplification and feature extraction, proving essential in highlighting the most influential variables.

Re-run on TEDS_2016

library(haven)
TEDS_2016 <- read_stata("https://github.com/datageneration/home/blob/master/DataProgramming/data/TEDS_2016.dta?raw=true")

library(haven)
library(dplyr)
library(ggplot2)
library(RColorBrewer)

# Load the TEDS_2016 dataset
TEDS_2016 <- read_stata("https://github.com/datageneration/home/blob/master/DataProgramming/data/TEDS_2016.dta?raw=true")

# Only retain relevant variables and rescale for clustering
# Assuming 'edu' and 'income' are present and are relevant for your analysis
TEDS_2016_scaled <- TEDS_2016 %>%
  mutate(
    edu_scaled = scale(edu, center = TRUE, scale = TRUE),
    income_scaled = scale(income, center = TRUE, scale = TRUE)
  ) %>%
  select(edu_scaled, income_scaled)

# Plot the scaled data
ggplot(data = TEDS_2016_scaled, aes(x = edu_scaled, y = income_scaled)) +
  geom_point(pch = 20, col = "blue") +
  theme_bw() +
  labs(x = "Education Level (Scaled)", y = "Income Level (Scaled)") +
  theme(text = element_text(family = "Georgia"))

## Warning: Removed 10 rows containing missing values or values outside the scale range
## (`geom_point()`).

# Removing rows with any NA, NaN, or Inf values across specific columns
TEDS_2016_scaled <- TEDS_2016_scaled %>%
  filter(complete.cases(edu_scaled, income_scaled))

# Alternatively, if you want to check and handle NaNs and Infs explicitly:
# Replace NaNs and Infs with NA for uniform handling
TEDS_2016_scaled$edu_scaled[sapply(TEDS_2016_scaled$edu_scaled, is.nan)] <- NA
TEDS_2016_scaled$income_scaled[sapply(TEDS_2016_scaled$income_scaled, is.nan)] <- NA
TEDS_2016_scaled$edu_scaled[sapply(TEDS_2016_scaled$edu_scaled, is.infinite)] <- NA
TEDS_2016_scaled$income_scaled[sapply(TEDS_2016_scaled$income_scaled, is.infinite)] <- NA

# Now, impute NAs if you prefer not to drop them:
TEDS_2016_scaled$edu_scaled[is.na(TEDS_2016_scaled$edu_scaled)] <- mean(TEDS_2016_scaled$edu_scaled, na.rm = TRUE)
TEDS_2016_scaled$income_scaled[is.na(TEDS_2016_scaled$income_scaled)] <- mean(TEDS_2016_scaled$income_scaled, na.rm = TRUE)


# Removing rows with any NA, NaN, or Inf values
TEDS_2016_scaled_clean <- TEDS_2016_scaled %>%
  filter(complete.cases(.))  # This keeps only complete cases

# Alternatively, you can impute missing values, for example, using the mean or median
# This is a basic example for numeric data using mean for imputation
TEDS_2016_scaled$edu_scaled[is.na(TEDS_2016_scaled$edu_scaled)] <- mean(TEDS_2016_scaled$edu_scaled, na.rm = TRUE)
TEDS_2016_scaled$income_scaled[is.na(TEDS_2016_scaled$income_scaled)] <- mean(TEDS_2016_scaled$income_scaled, na.rm = TRUE)

# Now retry k-means clustering
set.seed(2345)  # For reproducibility
k_means_result <- kmeans(TEDS_2016_scaled_clean, centers = 4, nstart = 25)


# Manual animation setup for k-means clustering
library(ggplot2)
set.seed(2345)  # For reproducibility

# Perform k-means clustering
k_means_result <- kmeans(TEDS_2016_scaled, centers = 4, nstart = 25)

# Add cluster assignments to the data
TEDS_2016_scaled$cluster <- factor(k_means_result$cluster)

# Create a plot for each iteration and save as an image
for(i in 1:nrow(k_means_result$centers)) {
  # Create a temporary data frame for centers up to the current iteration
  centers_df <- data.frame(
    edu_scaled = k_means_result$centers[1:i, 1],  # Assuming 1st column is edu_scaled
    income_scaled = k_means_result$centers[1:i, 2]  # Assuming 2nd column is income_scaled
  )
  
  # Plotting the clusters and their centers
  p <- ggplot(TEDS_2016_scaled, aes(x = edu_scaled, y = income_scaled, color = cluster)) +
      geom_point(alpha = 0.5) +
      geom_point(data = centers_df, aes(x = edu_scaled, y = income_scaled), color = "red", size = 5) +
      labs(title = paste("K-Means Clustering - Step", i)) +
      theme_minimal()
  print(p)
  ggsave(paste0("kmeans_step_", i, ".png"))
}

## Saving 7 x 5 in image

## Saving 7 x 5 in image

## Saving 7 x 5 in image

## Saving 7 x 5 in image

# Standard non-animated k-means as a fallback
k_means_result <- kmeans(TEDS_2016_scaled, centers = 4, nstart = 25)
print(k_means_result)

## K-means clustering with 4 clusters of sizes 357, 398, 538, 387
## 
## Cluster means:
##   edu_scaled income_scaled cluster
## 1 -0.8198427    -1.3575805       1
## 2  0.8467121     1.2440108       3
## 3 -0.7814344     0.2080506       4
## 4  0.9718454    -0.3103688       2
## 
## Clustering vector:
##    [1] 2 2 2 3 3 3 1 2 1 3 1 3 3 2 4 3 3 2 4 2 2 3 3 2 1 1 1 3 4 4 3 2 1 3 3 3 3
##   [38] 1 2 4 1 3 3 2 1 3 3 3 1 2 4 3 1 2 1 1 1 3 3 3 1 4 2 1 3 3 4 4 1 3 3 3 4 1
##   [75] 4 1 1 3 3 1 4 3 3 4 4 2 3 4 3 4 1 1 1 1 3 4 4 1 4 2 3 1 2 3 1 3 1 3 3 3 1
##  [112] 2 4 4 3 3 1 1 4 4 1 3 1 1 1 3 2 3 1 3 1 4 3 3 3 4 1 1 1 1 1 1 1 2 1 1 3 1
##  [149] 1 1 3 1 3 4 1 1 1 3 3 4 4 2 1 4 1 3 1 2 2 3 2 4 3 2 3 2 4 3 3 2 1 3 2 2 2
##  [186] 3 4 2 4 1 4 1 3 2 2 2 2 3 3 2 4 1 2 1 1 4 2 1 1 4 2 4 2 2 3 1 1 3 1 4 3 1
##  [223] 4 1 4 3 4 2 3 1 1 4 1 3 4 1 1 1 1 3 3 3 4 4 1 3 3 3 1 2 4 4 3 2 2 1 3 4 3
##  [260] 3 4 1 3 3 2 1 3 4 3 3 3 3 2 3 4 4 4 3 1 2 3 1 4 4 2 2 2 3 2 1 2 3 1 2 3 3
##  [297] 2 4 4 4 2 2 3 4 2 4 2 2 4 4 2 2 4 2 2 1 2 2 3 1 1 4 3 3 2 1 1 3 3 3 3 3 3
##  [334] 3 2 3 3 2 1 3 2 3 1 4 2 2 4 2 3 2 4 2 2 4 4 3 1 3 2 1 4 1 3 2 3 2 2 3 4 4
##  [371] 4 2 3 3 2 2 2 3 4 3 3 1 2 3 1 3 2 1 4 3 2 1 1 1 4 1 1 2 3 4 4 2 1 4 4 1 1
##  [408] 1 3 1 2 3 1 2 3 3 3 2 4 4 3 1 3 3 4 2 3 3 2 4 3 1 4 3 3 3 4 1 1 1 3 3 4 1
##  [445] 3 2 3 3 3 3 4 2 3 3 4 1 4 2 3 1 4 2 1 2 1 2 1 3 2 3 4 4 3 3 4 2 4 4 2 2 3
##  [482] 1 4 1 1 2 2 4 3 3 1 4 3 3 2 3 2 3 3 4 3 4 3 4 3 4 2 4 4 4 3 2 3 4 4 1 4 3
##  [519] 4 3 3 3 1 3 3 1 3 4 1 3 1 3 3 3 1 2 1 4 4 2 2 1 2 2 2 3 1 3 3 3 2 2 3 4 3
##  [556] 3 3 3 1 3 3 3 1 3 3 4 4 1 1 4 2 3 3 4 4 2 3 3 1 3 3 1 3 4 1 4 3 1 3 3 1 3
##  [593] 4 1 4 2 4 3 3 3 1 1 1 1 1 1 4 3 3 3 3 1 1 1 1 3 1 1 1 4 1 1 4 3 1 1 1 1 4
##  [630] 1 4 2 1 3 1 3 2 4 2 4 4 4 4 2 4 2 2 4 1 3 4 2 2 2 2 4 4 3 4 2 4 1 1 4 2 3
##  [667] 2 3 3 2 2 1 1 3 2 2 4 2 2 2 4 3 2 3 4 3 4 2 1 1 1 3 3 1 2 4 1 4 1 4 3 4 3
##  [704] 3 2 2 3 2 3 4 3 3 3 3 1 4 4 4 3 2 1 3 4 2 3 3 2 3 2 4 3 2 1 3 1 3 4 4 4 3
##  [741] 2 3 2 2 3 4 2 2 4 3 1 4 3 3 3 2 2 3 3 1 4 4 2 2 4 3 4 3 2 2 2 3 2 2 4 4 3
##  [778] 3 3 2 3 3 3 1 3 4 2 1 2 3 2 1 3 3 2 2 4 2 2 3 1 4 4 3 4 2 2 3 4 4 1 3 4 2
##  [815] 3 4 4 4 4 3 2 2 2 4 4 2 1 4 4 3 3 2 4 3 3 3 4 3 2 1 2 3 4 2 2 1 3 4 4 4 4
##  [852] 3 1 3 4 4 2 3 1 4 1 3 1 2 4 2 4 2 1 1 1 3 1 3 4 4 3 3 1 3 1 4 2 3 3 1 3 3
##  [889] 1 4 2 1 3 3 1 1 3 1 2 3 2 4 3 2 1 4 4 4 3 4 4 3 3 3 2 3 3 4 2 4 4 3 4 4 2
##  [926] 3 2 2 3 3 1 2 4 1 4 2 3 1 3 3 3 4 3 3 4 3 2 1 4 3 2 3 1 1 4 3 2 3 1 3 1 2
##  [963] 3 3 2 2 1 1 2 2 3 4 2 4 2 1 3 4 2 3 3 2 4 3 1 1 1 2 3 1 2 3 4 2 3 4 1 1 3
## [1000] 3 4 3 3 3 3 3 2 3 3 3 2 3 1 4 4 1 3 3 1 1 3 1 3 1 1 3 4 3 1 1 1 1 2 4 1 4
## [1037] 4 4 4 4 4 4 2 3 1 1 1 4 4 2 1 4 2 4 2 4 2 1 2 3 3 3 2 3 2 4 1 2 2 2 2 3 2
## [1074] 4 1 3 1 4 4 2 2 4 3 3 3 4 2 1 4 4 2 4 2 4 1 3 2 4 2 4 4 3 2 2 2 4 3 1 4 3
## [1111] 3 3 2 1 1 3 1 3 4 2 3 3 1 4 4 3 1 3 3 3 4 3 4 3 3 3 2 1 2 1 2 3 1 1 3 3 3
## [1148] 2 4 3 2 1 4 2 2 3 4 3 4 2 3 3 1 2 2 2 2 4 3 3 4 4 3 2 2 3 3 3 2 2 4 3 4 3
## [1185] 2 3 1 2 2 2 2 3 3 2 4 4 4 4 4 2 4 2 3 3 3 1 2 1 1 1 3 2 3 3 3 2 3 1 1 4 2
## [1222] 2 3 2 1 4 1 3 2 3 4 1 3 1 3 2 4 4 3 1 3 3 3 1 4 4 1 3 3 4 1 3 3 2 3 2 3 4
## [1259] 4 3 4 3 1 3 1 3 2 3 4 3 2 3 3 3 3 3 1 3 4 1 3 1 3 1 1 4 4 4 2 4 2 3 2 4 2
## [1296] 2 3 3 1 3 4 1 1 3 1 1 3 1 4 3 2 3 1 4 2 3 3 3 4 2 2 3 2 2 1 2 3 1 3 4 1 4
## [1333] 3 4 1 3 4 4 4 4 4 3 4 3 3 4 3 2 3 2 4 2 2 2 3 4 4 1 2 2 2 3 4 3 1 3 3 2 4
## [1370] 1 1 4 4 3 3 4 1 3 4 1 1 4 4 4 2 3 4 3 2 1 1 1 3 2 3 4 2 3 2 4 1 3 4 3 3 3
## [1407] 1 2 2 1 3 3 2 4 3 3 4 4 4 3 2 4 2 4 4 3 4 4 2 2 2 2 4 3 3 2 1 1 2 3 2 2 2
## [1444] 2 3 1 2 2 3 4 3 1 2 4 4 3 4 4 1 1 3 2 3 1 3 4 2 4 1 1 1 3 2 1 3 3 3 3 3 4
## [1481] 1 2 2 2 2 2 4 4 3 1 4 4 4 3 4 3 2 2 4 2 4 2 2 4 1 1 3 2 1 4 4 1 2 2 1 1 1
## [1518] 2 1 3 3 2 3 1 2 4 2 1 3 4 2 2 4 2 2 3 3 3 4 3 3 4 1 4 3 2 3 4 1 1 4 4 3 3
## [1555] 1 2 4 4 3 4 4 2 2 3 3 4 1 3 2 4 3 2 3 4 2 4 2 2 3 4 2 3 3 1 1 2 2 2 2 2 3
## [1592] 3 1 3 1 2 2 2 2 4 2 3 2 1 2 4 3 2 2 4 4 4 2 2 3 2 2 4 4 2 3 3 1 4 2 1 3 3
## [1629] 1 3 2 3 3 1 2 2 2 1 1 4 4 2 3 3 3 1 4 2 3 4 2 4 3 4 2 2 1 2 2 2 1 2 1 3 1
## [1666] 2 2 2 1 2 2 4 4 1 1 1 3 1 1 1
## 
## Within cluster sum of squares by cluster:
## [1] 177.6538 154.9004 334.0604 143.5970
##  (between_SS / total_SS =  85.3 %)
## 
## Available components:
## 
## [1] "cluster"      "centers"      "totss"        "withinss"     "tot.withinss"
## [6] "betweenss"    "size"         "iter"         "ifault"

# Adding cluster assignment back to the data
TEDS_2016_scaled$cluster <- as.factor(k_means_result$cluster)

# Plotting results with cluster identification
ggplot(TEDS_2016_scaled, aes(x = edu_scaled, y = income_scaled, color = cluster)) +
  geom_point(alpha = 0.6) +
  theme_bw() +
  scale_color_brewer(palette = "Set1") +
  labs(title = "Clustered Data: Education vs. Income")

Based on the results and plots you provided from the k-means clustering analysis on the TEDS_2016 dataset, let’s analyze and interpret the key findings:

Clustering Analysis Overview

The k-means clustering algorithm was applied to the TEDS_2016 dataset, specifically focusing on two scaled variables: education level and income. The goal appears to be identifying distinct groups within the dataset based on these socioeconomic factors.

Interpretation of Clustering Results

Cluster Formation and Distribution:
- Initial Steps (1-4): The series of images from Step 1 to Step 4 show the progression of the k-means algorithm. Initially, the cluster centers (red dots) are randomly placed, and in subsequent steps, they move towards the dense regions of data points. By Step 4, the centers seem to stabilize, indicating convergence of the algorithm.
- Final Cluster Plot: This plot shows the data points colored according to their cluster assignments. The clear separation along both the education and income scales suggests that the k-means algorithm has effectively partitioned the data into meaningful groups.
Cluster Characteristics:
- Clusters might represent groups differentiated by varying levels of education and income. For instance, one cluster may contain individuals with high education and high income, while another might represent low education and low income, and so forth.
- The distinct clusters could be indicative of socioeconomic stratification within the dataset, which could be valuable for targeted interventions or further socioeconomic research.
Statistical Summary:
- Within Cluster Sum of Squares (WCSS): The results indicate a WCSS for each cluster, with values such as 177.6538, 154.9004, 334.0604, and 143.5970. These values represent the variability within each cluster; lower values would indicate tighter clusters.
- Between_SS / Total_SS = 85.3%: This high percentage suggests that a substantial amount of the total variation in the dataset is explained by the cluster assignments. In other words, the clustering structure is strong, and the clusters are well-separated.

Conclusions

The k-means clustering analysis on the TEDS_2016 data with scaled education and income variables has successfully identified distinct groups that might represent different socioeconomic statuses. The clear separation and significant variance explained by the clusters suggest that the model has done well in capturing the underlying patterns in the data.

Further analysis could involve a deeper demographic study of the members within each cluster to validate assumptions about socioeconomic factors or to develop targeted programs based on the clustering insights. Also, examining cluster stability over different runs (changing seeds and the number of clusters) could validate the robustness of the clustering solution.

References

R Core Team (2021). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria.
James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An Introduction to Statistical Learning. Springer.
Kassambara, A. (2017). Practical Guide to Cluster Analysis in R. STHDA.

Assignment 4

Sonali Singh

Gentle Introduction to Machine Learning

Module 4: Unsupervised Learning

Abstract

Introduction

Methodology

Clustering with K-Means

Clustering Analysis on Wine Data and U.S. Arrest Data

Abstract

Materials and Methods

Results

Discussion

Conclusions

Future Work

Principal Component Analysis

Summary of PCA:

Principal Component Analysis (PCA) Results

Interpretation and Insights

Conclusions

Discussion

Re-run on TEDS_2016

Clustering Analysis Overview

Interpretation of Clustering Results

Conclusions

References

Assignment 4

Sonali Singh

Gentle Introduction to Machine Learning

Module 4: Unsupervised Learning

Abstract

Introduction

Methodology

Clustering with K-Means

Clustering Analysis on Wine Data and U.S. Arrest Data

Abstract

Materials and Methods

Results

Discussion

Conclusions

Future Work

** Principal Component Analysis**

Summary of PCA:

Principal Component Analysis (PCA) Results

Interpretation and Insights

Conclusions

Discussion

Re-run on TEDS_2016

Clustering Analysis Overview

Interpretation of Clustering Results

Conclusions

References

Principal Component Analysis