Introduction

In this paper, I will be performing image clustering using unsupervised learning tools. As an example, I will compare the differences in color between spring and autumn, using Park Konstytucji 3 Maja in Suwałki, Poland, as a case study. The aim of the study is to examine which colors dominate during a given season and whether the shades between spring and autumn are relatively similar or rather distant. For this purpose, I will use the clara clustering method because it is the most suitable for large datasets, such as images. The method is based on the k-medoids PAM algorithm. The images were found using the Google search engine.

Import necessary packages

library(png)
library(ggplot2)
library(gridExtra)
library(cluster)
## Warning: pakiet 'cluster' został zbudowany w wersji R 4.3.2

Data Preparation

Now, let’s examine pictures of the park during both spring and autumn without any transformations.

spring <- readPNG("suwałkiwiosna.png")
autumn <- readPNG("suwałkijesien.png")

data <- data.frame(x = 1:10, y = rnorm(10))

add_background_image <- function(plot, image, title) {
  plot + 
    annotation_raster(image, xmin = -Inf, xmax = Inf, ymin = -Inf, ymax = Inf) +
    labs(title = title) +
    theme(axis.title = element_blank(),
          axis.text = element_blank(),
          axis.ticks = element_blank())
}

im_A <- ggplot(data, aes(x, y)) + geom_point() +
  theme(plot.margin = margin(t=1, l=1, r=1, b=1, unit = "cm"))
im_A <- add_background_image(im_A, spring, "Spring")

im_B <- ggplot(data, aes(x, y)) + geom_point() +
  theme(plot.margin = margin(t=1, l=1, r=1, b=1, unit = "cm"))
im_B <- add_background_image(im_B, autumn, "Autumn")

grid.arrange(im_A, im_B, ncol = 2)

Most of the surface in the photos is occupied by trees. With the naked eye, we can observe that green dominates in spring and orange in autumn. However, the photographs also include buildings, roads, cars, and fields, which may influence the perception of color during the clustering process.

dmspring <- dim(spring)
dmautumn <- dim(autumn)

Dimensions of the park spring image: 1535 2048
Dimensions of the park autumn image: 1279 1920

Converting images to RGB colors:

rgbspring<-data.frame(x=rep(1:dmspring[2], 
                            each=dmspring[1]),  
                      y=rep(dmspring[1]:1, dmspring[2]), 
                      r.value=as.vector(spring[,,1]),  
                      g.value=as.vector(spring[,,2]), 
                      b.value=as.vector(spring[,,3]))

rgbautumn<-data.frame(x=rep(1:dmautumn[2], 
                            each=dmautumn[1]),  
                      y=rep(dmautumn[1]:1, dmautumn[2]), 
                      r.value=as.vector(autumn[,,1]),  
                      g.value=as.vector(autumn[,,2]), 
                      b.value=as.vector(autumn[,,3]))
plot(y~x, data=rgbspring, main="Spring", 
     col=rgb(rgbspring[c("r.value", "g.value", "b.value")]), asp=1, pch=".")

plot(y~x, data=rgbautumn, main="Autumn", 
     col=rgb(rgbautumn[c("r.value", "g.value", "b.value")]), asp=1, pch=".")

Optimal number of clusters

To perform the clustering method, the clara method will be used. First, it is necessary to determine the appropriate number of clusters (representing colors). In this case, the Silhouette Index will be employed. The Silhouette Index takes values in the range from -1 to 1. A higher value indicates that a particular object fits well with its assigned cluster and poorly with neighboring clusters. However, besides calculating the Silhouette values, careful consideration should be given to the expected number of clusters.

The dominant objects in the image are as follows:


The ideal situation would be if the Silhouette Index indicated 3 clusters, as it would align with expectations.

nspring <- c()
for (i in 1:10) {
  clspring <- clara(rgbspring[, c("r.value", "g.value", "b.value")], i)
  nspring[i] <- clspring$silinfo$avg.width
}
nautumn <- c()
for (i in 1:10) {
  clautumn <- clara(rgbautumn[, c("r.value", "g.value", "b.value")], i)
  nautumn[i] <- clautumn$silinfo$avg.width
}
plot(nspring, type = 'l',
     main = "Optimal number of clusters for spring",
     xlab = "Number of clusters",
     ylab = "Average silhouette",
     col = "lightgreen")

plot(nautumn, type = 'l',
     main = "Optimal number of clusters for autumn",
     xlab = "Number of clusters",
     ylab = "Average silhouette",
     col = "orange")

As it can be observed, the optimal number of clusters for both spring and autumn is two. However, setting the number of clusters to two for these images may be insufficient, as it would combine various types of objects such as trees, buildings, or shadows. Therefore, the decision has been made to set the number of clusters to 3 to distinguish the dominant color within each category of objects in the images. The average silhouette width is still high enough.

springRGB = rgbspring[, c("r.value", "g.value", "b.value")]
autumnRGB = rgbautumn[, c("r.value", "g.value", "b.value")]
claraspring3 <- clara(springRGB, 3)
claraautumn3 <- clara(autumnRGB, 3)
plot(silhouette(claraspring3))

plot(silhouette(claraautumn3))

Now, let’s take a look at the algorithm results on the images

coloursspring3<-rgb(claraspring3$medoids[claraspring3$clustering, ])
coloursautumn3<-rgb(claraautumn3$medoids[claraautumn3$clustering, ])
plot(rgbspring$y~rgbspring$x, col=coloursspring3, pch=".", cex=2, asp=1, main="Spring - 3 colours")

plot(rgbautumn$y~rgbautumn$x, col=coloursautumn3, pch=".", cex=2, asp=1, main="Autumn - 3 colours")

#SPRING
dominantColoursspring <- as.data.frame(table(coloursspring3))

max_colspring  <- max(dominantColoursspring$Freq)/sum(dominantColoursspring$Freq)
min_colspring  <- min(dominantColoursspring$Freq)/sum(dominantColoursspring$Freq)
medium_colspring <- 1-max_colspring - min_colspring
dominantColoursspring$colours <- as.character(dominantColoursspring$colours)
dominantColoursspring$distribution <- round((c(max_colspring, medium_colspring, min_colspring) * 100), 2)

colors_spring <- c("#342E18", "#62643F", "#7E7B76")

spring_data <- data.frame(
  colors = factor(colors_spring, levels = colors_spring),
  distribution = round(c(max_colspring, medium_colspring, min_colspring) * 100, 2)
)

plot_spring <- ggplot(spring_data, aes(x = "", y = distribution, fill = colors)) +
  geom_bar(stat = "identity", width = 1) +
  geom_text(aes(label = paste0(distribution, "%")), position = position_stack(vjust = 0.5),
            color = "white") +
  coord_polar("y") +
  theme_void() +
  scale_fill_manual(values = colors_spring) +
  ggtitle("Percentage of main colors - spring")

#AUTUMN
dominantColoursautumn <- as.data.frame(table(coloursautumn3))

max_colautumn  <- max(dominantColoursautumn$Freq)/sum(dominantColoursautumn$Freq)
min_colautumn  <- min(dominantColoursautumn$Freq)/sum(dominantColoursautumn$Freq)
medium_colautumn <- 1-max_colautumn - min_colautumn
dominantColoursautumn$colours <- as.character(dominantColoursautumn$colours)
dominantColoursautumn$distribution <- round((c(max_colautumn, medium_colautumn, min_colautumn) * 100), 2)

colors_autumn <- c("#1F282F", "#634930", "#9F9383")

autumn_data <- data.frame(
  colors = factor(colors_autumn, levels = colors_autumn),
  distribution = round(c(max_colautumn, medium_colautumn, min_colautumn) * 100, 2)
)

plot_autumn <- ggplot(autumn_data, aes(x = "", y = distribution, fill = colors)) +
  geom_bar(stat = "identity", width = 1) +
  geom_text(aes(label = paste0(distribution, "%")), position = position_stack(vjust = 0.5),
            color = "white") +
  coord_polar("y") +
  theme_void() +
  scale_fill_manual(values = colors_autumn) +
  ggtitle("Percentage of main colors - autumn")

grid.arrange(plot_spring, plot_autumn, ncol = 2)

Conclusion

As evident, the color palette of the surveyed area significantly varies depending on the season. In both seasons, the predominant color is responsible for shadows and roofs (dark green in spring and navy blue in winter). Colors primarily associated with trees are green and brown, respectively. It is worth emphasizing that these colors are not vibrant, contrary to what the initial images suggested. The colors of roads, on the other hand, are gray and beige. The percentage shares of respective colors differ, but they can be considered relatively close. Differences may arise due to factors such as: