Spotify Tracks Clustering Using K-Means Algorithm
Rifky Novrian K
2021-05-21
1 Overview
Spotify is a digital streaming service that gives us access to millions of music, podcasts and videos from artists around the world. With Spotify, we can listen to various choices of content at any time legally and easily. Spotify also has a complex algorithm for recommending music based on our listening history.
In this article, I would like to demonstrate an unsupervised learning analysis using the spotify dataset from Kaggle. The analysis includes clustering using the K-Means algorithm, reading the profile of each cluster and seeing what music genre dominates each cluster (looking at the average genre characteristics through the cluster profile). Through clustering, we can recommend similar songs to listeners of a song
2 Library Setup
library(tidyverse)
library(FactoMineR)
library(factoextra)3 Data Preparation
3.1 Import Data
First, we need to load the spotify dataset into R
spotify <- read.csv("SpotifyFeatures.csv")
glimpse(spotify)## Rows: 232,725
## Columns: 18
## $ genre <chr> "Movie", "Movie", "Movie", "Movie", "Movie", "Movie",~
## $ artist_name <chr> "Henri Salvador", "Martin & les fées", "Joseph Willi~
## $ track_name <chr> "C'est beau de faire un Show", "Perdu d'avance (par G~
## $ track_id <chr> "0BRjO6ga9RKCKjfDqeFgWV", "0BjC1NfoEOOusryehmNudP", "~
## $ popularity <int> 0, 1, 3, 0, 4, 0, 2, 15, 0, 10, 0, 2, 4, 3, 0, 0, 0, ~
## $ acousticness <dbl> 0.61100, 0.24600, 0.95200, 0.70300, 0.95000, 0.74900,~
## $ danceability <dbl> 0.389, 0.590, 0.663, 0.240, 0.331, 0.578, 0.703, 0.41~
## $ duration_ms <int> 99373, 137373, 170267, 152427, 82625, 160627, 212293,~
## $ energy <dbl> 0.9100, 0.7370, 0.1310, 0.3260, 0.2250, 0.0948, 0.270~
## $ instrumentalness <dbl> 0.00e+00, 0.00e+00, 0.00e+00, 0.00e+00, 1.23e-01, 0.0~
## $ key <chr> "C#", "F#", "C", "C#", "F", "C#", "C#", "F#", "C", "G~
## $ liveness <dbl> 0.3460, 0.1510, 0.1030, 0.0985, 0.2020, 0.1070, 0.105~
## $ loudness <dbl> -1.828, -5.559, -13.879, -12.178, -21.150, -14.970, -~
## $ mode <chr> "Major", "Minor", "Minor", "Major", "Major", "Major",~
## $ speechiness <dbl> 0.0525, 0.0868, 0.0362, 0.0395, 0.0456, 0.1430, 0.953~
## $ tempo <dbl> 166.969, 174.003, 99.488, 171.758, 140.576, 87.479, 8~
## $ time_signature <chr> "4/4", "4/4", "5/4", "4/4", "4/4", "4/4", "4/4", "4/4~
## $ valence <dbl> 0.8140, 0.8160, 0.3680, 0.2270, 0.3900, 0.3580, 0.533~3.2 Variables Description
Here are some information about the data :
genre: Track genre
artist_name: Artist name
track_name: Title of track
track_id: The Spotify ID for the track.
popularity: Popularity rate (1-100)
acousticness: A confidence measure from 0.0 to 1.0 of whether the track is acoustic. 1.0 represents high confidence the track is acoustic.
danceability: Danceability describes how suitable a track is for dancing based on a combination of musical elements including tempo, rhythm stability, beat strength, and overall regularity. A value of 0.0 is least danceable and 1.0.
duration_ms: The duration of the track in milliseconds.
energy: Energy is a measure from 0.0 to 1.0 and represents a perceptual measure of intensity and activity. Typically, energetic tracks feel fast, loud, and noisy. For example, death metal has high energy, while a Bach prelude scores low on the scale.
instrumentalness: Predicts whether a track contains no vocals. “Ooh” and “aah” sounds are treated as instrumental in this context. Rap or spoken word tracks are clearly “vocal”. The closer the instrumentalness value is to 1.0, the greater likelihood the track contains no vocal content. Values above 0.5 are intended to represent instrumental tracks, but confidence is higher as the value approaches 1.0.
key: The estimated overall key of the track. Integers map to pitches using standard Pitch Class notation . E.g. 0 = C, 1 = C♯/D♭, 2 = D, and so on. If no key was detected, the value is -1.
liveness: Detects the presence of an audience in the recording. Higher liveness values represent an increased probability that the track was performed live. A value above 0.8 provides strong likelihood that the track is live.
loudness: The overall loudness of a track in decibels (dB). Loudness values are averaged across the entire track and are useful for comparing relative loudness of tracks.
mode: Mode indicates the modality (major or minor) of a track, the type of scale from which its melodic content is derived. Major is represented by 1 and minor is 0.
speechiness: Speechiness detects the presence of spoken words in a track. The more exclusively speech-like the recording (e.g. talk show, audio book, poetry), the closer to 1.0 the attribute value. Values above 0.66 describe tracks that are probably made entirely of spoken words. Values between 0.33 and 0.66 describe tracks that may contain both music and speech, either in sections or layered, including such cases as rap music. Values below 0.33 most likely represent music and other non-speech-like tracks.
tempo: The overall estimated tempo of a track in beats per minute (BPM). In musical terminology, tempo is the speed or pace of a given piece and derives directly from the average beat duration.
time_signature: An estimated overall time signature of a track. The time signature (meter) is a notational convention to specify how many beats are in each bar (or measure).
valence: A measure from 0.0 to 1.0 describing the musical positiveness conveyed by a track. Tracks with high valence sound more positive (e.g. happy, cheerful, euphoric), while tracks with low valence sound more negative (e.g. sad, depressed, angry).
4 Data Wrangling
As we can see, some of variables have inappropriate data types so we need to change it and drop variables that we think didn’t related with our case
spotify <- spotify %>%
mutate(genre = as.factor(genre),
key = as.factor(key),
mode = as.factor(mode)) If we look carefully, there are track names data that appears repeatedly, because one track can be sung by several artists with different genres.
check <- spotify %>%
filter(track_name == "Don't Let Me Be Lonely Tonight")
rmarkdown::paged_table(check)For this time, we will try to focus on just one genre only. We choose the jazz genre
spotify <- spotify %>%
filter(genre == "Jazz")We would check NA or Empty value of each variable, We didnt find any NA inside data
colSums(is.na(spotify))## genre artist_name track_name track_id
## 0 0 0 0
## popularity acousticness danceability duration_ms
## 0 0 0 0
## energy instrumentalness key liveness
## 0 0 0 0
## loudness mode speechiness tempo
## 0 0 0 0
## time_signature valence
## 0 0spotify_new <- spotify %>%
select(-c(genre, key, mode, valence, time_signature))
str(spotify_new)## 'data.frame': 9441 obs. of 13 variables:
## $ artist_name : chr "Kelsea Ballerini" "Earth, Wind & Fire" "Leslie Odom Jr." "Etta James" ...
## $ track_name : chr "Miss Me More" "September" "Alexander Hamilton" "At Last" ...
## $ track_id : chr "5NfJGBAL9mgFPRQxKJmiX2" "5nNmj1cLH3r4aA4XDJ2bgY" "4TTV7EcfroSLWzXRY6gLv6" "4Hhv2vrOTy89HFRcjU3QOx" ...
## $ popularity : int 74 79 72 74 69 67 66 68 67 68 ...
## $ acousticness : num 0.014 0.114 0.524 0.707 0.125 0.846 0.729 0.936 0.906 0.81 ...
## $ danceability : num 0.643 0.697 0.609 0.171 0.561 0.569 0.271 0.464 0.601 0.421 ...
## $ duration_ms : int 192840 214827 236738 182400 193750 282320 139227 255227 184004 337733 ...
## $ energy : num 0.72 0.809 0.435 0.33 0.474 0.117 0.165 0.305 0.222 0.0161 ...
## $ instrumentalness: num 0.00 5.21e-04 0.00 3.81e-03 5.07e-06 8.52e-01 1.60e-06 8.46e-02 4.11e-05 2.10e-03 ...
## $ liveness : num 0.0834 0.183 0.118 0.302 0.0922 0.0978 0.118 0.208 0.0723 0.0978 ...
## $ loudness : num -7.15 -8.2 -7.86 -9.7 -9.64 ...
## $ speechiness : num 0.0527 0.0302 0.284 0.0329 0.155 0.0446 0.0351 0.0316 0.0303 0.0374 ...
## $ tempo : num 96 125.9 132 174.4 86.9 ...We excluded variables factor data type for further data scaling pre-processing.
spotify_new <- spotify_new %>%
select(-c(artist_name, track_name, track_id))
str(spotify_new)## 'data.frame': 9441 obs. of 10 variables:
## $ popularity : int 74 79 72 74 69 67 66 68 67 68 ...
## $ acousticness : num 0.014 0.114 0.524 0.707 0.125 0.846 0.729 0.936 0.906 0.81 ...
## $ danceability : num 0.643 0.697 0.609 0.171 0.561 0.569 0.271 0.464 0.601 0.421 ...
## $ duration_ms : int 192840 214827 236738 182400 193750 282320 139227 255227 184004 337733 ...
## $ energy : num 0.72 0.809 0.435 0.33 0.474 0.117 0.165 0.305 0.222 0.0161 ...
## $ instrumentalness: num 0.00 5.21e-04 0.00 3.81e-03 5.07e-06 8.52e-01 1.60e-06 8.46e-02 4.11e-05 2.10e-03 ...
## $ liveness : num 0.0834 0.183 0.118 0.302 0.0922 0.0978 0.118 0.208 0.0723 0.0978 ...
## $ loudness : num -7.15 -8.2 -7.86 -9.7 -9.64 ...
## $ speechiness : num 0.0527 0.0302 0.284 0.0329 0.155 0.0446 0.0351 0.0316 0.0303 0.0374 ...
## $ tempo : num 96 125.9 132 174.4 86.9 ...5 Principal Component Analysis
PCA is very useful to retain information while reducing the dimension of the data. However, we need to make sure that our data is properly scaled in order to get a useful PCA.
spotify_scale <- scale(spotify_new)We have prepared the scaled data to be used for PCA. Next, we will try to generate the principal component from the spotify_scale.
pca_spotify <- PCA(spotify_scale,
scale.unit = FALSE,
graph = F,
ncp = 10) #default: 5)
pca_spotify$eig## eigenvalue percentage of variance cumulative percentage of variance
## comp 1 2.6334620 26.337410 26.33741
## comp 2 1.3340133 13.341546 39.67896
## comp 3 1.1932441 11.933705 51.61266
## comp 4 1.0948894 10.950053 62.56272
## comp 5 0.9802949 9.803987 72.36670
## comp 6 0.8352482 8.353366 80.72007
## comp 7 0.7479393 7.480185 88.20025
## comp 8 0.6098589 6.099235 94.29949
## comp 9 0.3964786 3.965206 98.26469
## comp 10 0.1735121 1.735305 100.00000We have to use 7 Principal Components (PCs) if we only tolerate no more than 15% of information loss.
PCA can be combined with clustering to obtain better visualization of our clustering result, or simply to understand the pattern in our dataset. We will visualize high dimensional data into 2 dimensional plot for various purposes, such as cluster analysis or detecting any outliers.
plot.PCA(pca_spotify,
choix = c("ind"),
habillage = 1,
select = "contrib5",
invisible = "quali")
We found 5 song id considered as outliers : 3311,6516,4800,5089,9222
plot.PCA(pca_spotify, choix = c("var"))
An alternative way to extract the loading information is by using the dimdesc() function to the pca_spotify. We will inspect the loading information from the first dimension/PC by calling pca_dimdesc$Dim.1 since the first dimension is the one that hold the most information.
pca_dimdesc <- dimdesc(pca_spotify)
pca_dimdesc$Dim.1## $quanti
## correlation p.value
## energy 0.90291391 0.000000e+00
## loudness 0.86631176 0.000000e+00
## danceability 0.50185684 0.000000e+00
## speechiness 0.31125248 3.680572e-211
## tempo 0.13410724 3.859506e-39
## liveness 0.11868880 5.728887e-31
## popularity -0.07156306 3.368689e-12
## instrumentalness -0.09928999 4.039131e-22
## acousticness -0.81985575 0.000000e+00
##
## attr(,"class")
## [1] "condes" "list"6 K-Means Clustering
Data clustering is a common data mining technique to create clusters of data that can be identified as “data with the same characteristics”. Before performing data clustering, you will need to remove the identified outlier based the previous individual PCA plot. So, we need to remove them from our initial dataset and once again scale the data.
spotify_no_out <- spotify_new[-c(3311,6516,4800,5089,9222),]
spotify_scale_no_out <- scale(spotify_no_out)
str(spotify_scale_no_out)## num [1:9436, 1:10] 3.46 3.98 3.25 3.46 2.94 ...
## - attr(*, "dimnames")=List of 2
## ..$ : chr [1:9436] "1" "2" "3" "4" ...
## ..$ : chr [1:10] "popularity" "acousticness" "danceability" "duration_ms" ...
## - attr(*, "scaled:center")= Named num [1:10] 4.08e+01 5.00e-01 5.86e-01 2.65e+05 4.73e-01 ...
## ..- attr(*, "names")= chr [1:10] "popularity" "acousticness" "danceability" "duration_ms" ...
## - attr(*, "scaled:scale")= Named num [1:10] 9.59 3.38e-01 1.59e-01 1.13e+05 2.38e-01 ...
## ..- attr(*, "names")= chr [1:10] "popularity" "acousticness" "danceability" "duration_ms" ...6.1 Choosing Optimum K
The next step in building a K-means clustering is to find the optimum cluster number to model our data. We can use the kmeansTunning() function below to find the optimum K using Elbow method
RNGkind(sample.kind = "Rounding")
kmeansTunning <- function(data, maxK) {
withinall <- NULL
total_k <- NULL
for (i in 2:maxK) {
set.seed(1)
temp <- kmeans(data,i)$tot.withinss
withinall <- append(withinall, temp)
total_k <- append(total_k,i)
}
plot(x = total_k, y = withinall, type = "o", xlab = "Number of Cluster", ylab = "Total within")
}
kmeansTunning(spotify_scale_no_out, maxK = 8)
We found 6 cluster is good enough since there is’nt significant decline in total within-cluster sum of squares on higher number of clusters.
6.2 Building Clusters
Once we find the optimum K from the previous section, we will try to do K-means clustering from our data and store it as spot_km.
# k-means with 6 clusters
RNGkind(sample.kind = "Rounding")
set.seed(100)
spot_km <- kmeans(x = spotify_scale_no_out,centers = 6)We can also visualize our data cluster with fviz_cluster function
fviz_cluster(spot_km, data = spotify_scale_no_out)
Next, extract the cluster information from the resulting K-means object using spot_km$cluster and add them as a new column named cluster to the spotify_no_out dataset.
spotify_no_out$cluster <- spot_km$cluster
rmarkdown::paged_table(spotify_no_out)7 Evaluation
7.1 Goodness of Fit
To check whether the cluster we created is good enough, we can see the value in the cluster. The goodness of the clustering results can be seen from 3 values:
- Within Sum of Squares (
$withinss): sum of the distance squared from each observation to the centroid of each cluster.
- Between Sum of Squares (
$betweenss): the sum of the weighted square distances of each centroid to the global average. Weighted based on the number of observations in the cluster.
- Total Sum of Squares (
$totss): the sum of the distances squared from each observation to the global average.
- Total within sum of square: the number of withinss for each cluster
spot_km$betweenss## [1] 37809.03spot_km$withinss## [1] 14562.002 9415.837 10453.138 6340.152 10910.587 4859.256spot_km$totss## [1] 94350spot_km$tot.withinss## [1] 56540.97spot_km$betweenss/spot_km$tot.withinss## [1] 0.6687014From the evaluation of the model, the betweenss / tot.withinss value is 0.6687014 or close to 1 so it can be said that the model is quite good.
8 Cluster Profiling
We will create a Jazz cluster profiling by using a combination of group_by() and summarise_all(), grouped by the previously created cluster column.
cluster_prof <- spotify_no_out %>%
group_by(cluster) %>%
summarise_all("mean")
rmarkdown::paged_table(cluster_prof)From the figures above, the characteristic summary of Jazz Spotify songs in the same cluster are as follows:
- Cluster 1: Lowest acousticness and tempo, highest energy
- Cluster 2: Highest acousticness and instrumentalness, lowest energy, liveness, and loudness
- Cluster 3: Highest popularity, lowest instrumentalness
- Cluster 4: Highest danceability and speechiness, lowest duration
- Cluster 5: Highest duration and tempo
- Cluster 6: Highest liveness