Image Clustering
In this article I will show how to use unsupervised learning methods for image clustering. The idea is to cluster an image and find its dominant colour. I will compare two Edvard Munch’s masterpieces - “Scream” and “Anxiety”, as the painter is known from diverse colour usage.
Data preprocessing
First step is to open a given image. To open .jpg file, JPEG library from R Documentation is needed.
library(jpeg)
image1 <- readJPEG("Scream.jpg")
image2 <- readJPEG("Anxiety.jpg")In order to process an image, we need to represent it in a matrix of numbers. The most convenient format is RGB. RGB denotes to amount of red, green and blue in a given pixel of an image. It can be easily processed, as it represents the image in a 3 column matrix, where the first column relates to the amount of red colour, the second relates to the amount of green colour and the last one relates to the amount of blue colour in range of 0-255.
First, let’s check the dimension.
dm1 <- dim(image1);dm1[1:2]## [1] 900 725dm2 <- dim(image2);dm2[1:2]## [1] 1119 880Size of the “Scream” image is equal to 900x725 and the size of the “Anxiety” image is equal to 1119x880.
Now, we can change the format of the images from jpg to rgb and display them on the plot.
rgbImage1 <- data.frame(
x=rep(1:dm1[2], each=dm1[1]),
y=rep(dm1[1]:1, dm1[2]),
r.value=as.vector(image1[,,1]),
g.value=as.vector(image1[,,2]),
b.value=as.vector(image1[,,3]))
rgbImage2 <- data.frame(
x=rep(1:dm2[2], each=dm2[1]),
y=rep(dm2[1]:1, dm2[2]),
r.value=as.vector(image2[,,1]),
g.value=as.vector(image2[,,2]),
b.value=as.vector(image2[,,3]))plot(y ~ x, data=rgbImage1, main="Edvard Munch - Scream",
col = rgb(rgbImage1[c("r.value", "g.value", "b.value")]),
asp = 1, pch = ".")
plot(y ~ x, data=rgbImage2, main="Edvard Munch - Anxiety",
col = rgb(rgbImage2[c("r.value", "g.value", "b.value")]),
asp = 1, pch = ".")
Optimal number of k-clusters
I will use Clara algorithm for image clustering. It is based on k-medoids PAM algorithm and it is optimal for large data sets. The size of the data sets of the images are 900 x 725 x 3 and 1119 x 880 x 3, which is 1957500 and 2954160, so they could be classified as large data sets.
First step is to find the optimal number of k - clusters for each image by comparing average silhouette width for every k. Silhouette ranges from -1 to 1 and it describes clustering consistency. A positive value means that the elements in cluster are correctly matched - objects in a given clusters are similar to each other and dissimilar to the objects of the surrounding clusters. The higher value, the better clustering.
I will use “cluster” library to run clara algorithm for 10 consecutive numbers to analyze the average silhouette width.
library(cluster)
n1 <- c()
for (i in 1:10) {
cl <- clara(rgbImage1[, c("r.value", "g.value", "b.value")], i)
n1[i] <- cl$silinfo$avg.width
}
plot(n1, type = 'l',
main = "Optimal number of clusters for Scream",
xlab = "Number of clusters",
ylab = "Average silhouette",
col = "blue")
The results show that 3 clusters are optimal for the “Scream” image.
Let’s repeat the process the “Anxiety” image.
n2 <- c()
for (i in 1:10) {
cl <- clara(rgbImage2[, c("r.value", "g.value", "b.value")], i)
n2[i] <- cl$silinfo$avg.width
}
plot(n2, type = 'l',
main = "Optimal number of clusters for Anxiety",
xlab = "Number of cluster",
ylab = "Average silhouette",
col = "blue")
Running Clara algorithm
Now, let’s run clara with the given number of clusters.
“Scream” image:
scream = rgbImage1[, c("r.value", "g.value", "b.value")]
clara <- clara(scream, 3)
plot(silhouette(clara))
Let’s see the clustered image on the plot with the usage of rgb() function, which creates colours from rgb values.
colours <- rgb(clara$medoids[clara$clustering, ])
plot(y ~ x, data=rgbImage1, main="Edvard Munch - Scream",
col = colours,
asp = 1, pch = ".")
“Anxiety” image:
anxiety = rgbImage2[, c("r.value", "g.value", "b.value")]
clara2 <- clara(anxiety, 3)
plot(silhouette(clara2))
colours2 <- rgb(clara2$medoids[clara2$clustering, ])
plot(y ~ x, data=rgbImage2, main="Edvard Munch - Anxiety",
col = colours2,
asp = 1, pch = ".")
Finding dominant colour
We can find a dominant colour of the image by counting clusters distribution. Let’s use the output of rgb() function, which is given in a hexadecimal format and and represents colours.
Let’s count the percentage colour distribution. Value of the colour frequency column is simply a size of the clusters.
dominantColours <- as.data.frame(table(colours))
max_col <- max(dominantColours$Freq)/sum(dominantColours$Freq)
min_col <- min(dominantColours$Freq)/sum(dominantColours$Freq)
medium_col <- 1-max_col - min_col
dominantColours$distribution <- round((c(min_col, medium_col, max_col) * 100), 2)
dominantColours## colours Freq distribution
## 1 #281818 151321 23.19
## 2 #915E3F 246428 37.77
## 3 #DCA35C 254751 39.04Then, plot it.
dominantColours$colours <- as.character(dominantColours$colours)
pie(dominantColours$Freq, labels = dominantColours$distribution,
col = dominantColours$colours,
xlab = "Colours",
ylab = "Frequency")
Golden is the dominant colour on the clustered “Scream” painting. It takes around 39,04% of the image. Dark brown takes around 23,19% and brown around 37,77% of the image.
Let’s repeat the process for the “Anxiety” image.
dominantColours2 <- as.data.frame(table(colours2))
max_col2 <- max(dominantColours2$Freq)/sum(dominantColours2$Freq)
min_col2 <- min(dominantColours2$Freq)/sum(dominantColours2$Freq)
medium_col2 <- 1-max_col2 - min_col2
dominantColours2$distribution <- round((c(max_col2, medium_col2, min_col2) * 100), 2)
dominantColours2## colours2 Freq distribution
## 1 #1C2D37 584844 59.39
## 2 #B14336 228918 23.25
## 3 #C4B07B 170958 17.36dominantColours2$colours2 <- as.character(dominantColours2$colours2)
pie(dominantColours2$Freq, labels = dominantColours2$distribution,
col = dominantColours2$colours2,
xlab = "Colours",
ylab = "Frequency")
Dark blue is the dominant colour of the clustered “Anxiety” image. It takes around 59,39% of the place. Tomato-red takes around 23,25% and pale-golden around 17,36% of the image.