Spotify is a Swedish-based audio streaming and media services provider. Spotify was launched in October 2008 and it is one of the biggest streaming device in the world.
The goal is to examine how well the tracks in the Spotify Dataset can be grouped into different clusters using KMeans Clustering Algorithm.
The following packages are used in the analysis:
| Package Name | Purpose |
|---|---|
| library(kableExtra) | kableExtra |
| library(spotifyr) | spotifyr |
| library(knitr) | knitr |
| library(tidyverse) | tidyverse |
| library(plotly) | plotly |
| library(imager) | imager |
| library(readr) | readr |
| library(ggcorrplot) | ggcorrplot |
Let’s install these packages.
#install.packages("kableExtra",repos = "http://cran.us.r-project.org")
#install.packages("spotifyr",repos = "http://cran.us.r-project.org")
#install.packages("tidyverse",repos = "http://cran.us.r-project.org")
#install.packages("knitr",repos = "http://cran.us.r-project.org")
#install.packages("plotly",repos = "http://cran.us.r-project.org")
#install.packages("imager",repos = "http://cran.us.r-project.org")
#install.packages("readr",repos = "http://cran.us.r-project.org")
#install.packages("ggcorrplot",repos = "http://cran.us.r-project.org")
#install.packages("corpus",repos = "http://cran.us.r-project.org")
#install.packages("wordcloud",repos = "http://cran.us.r-project.org")
#install.packages("tm",repos = "http://cran.us.r-project.org")
#install.packages("randomForest",repos = "http://cran.us.r-project.org")
#install.packages("Metrics",repos = "http://cran.us.r-project.org")
#install.packages("cluster",repos = "http://cran.us.r-project.org")
#install.packages("factoextra",repos = "http://cran.us.r-project.org")
#install.packages("gridExtra",repos = "http://cran.us.r-project.org")
The data this week comes from Spotify via the spotifyr package. Charlie Thompson, Josiah Parry, Donal Phipps, and Tom Wolff authored this package to make it easier to get either your own data or general metadata arounds songs from Spotify’s API. Make sure to check out the spotifyr package website to see how you can collect your own data!
Kaylin Pavlik had a recent blogpost using the audio features to explore and classify songs. She used the spotifyr package to collect about 5000 songs from 6 main categories (EDM, Latin, Pop, R&B, Rap, & Rock).
library(readr)
spotify_songs_data <- readr::read_csv('https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2020/2020-01-21/spotify_songs.csv')| Variable | Class | Description |
|---|---|---|
| track_id | character | Song unique ID |
| track_name | character | Song Name |
| track_artist | character | Song Artist |
| track_popularity | double | Song Popularity (0-100) where higher is better |
| track_album_id | character | Album unique ID |
| track_album_name | character | Song album name |
| track_album_release_date | character | Date when album released |
| playlist_name | character | Name of playlist |
| playlist_id | character | Playlist ID |
| playlist_genre | character | Playlist genre |
| danceability | double | Danceability describes how suitable a track is for dancing based on a combination of musical elements including tempo, rhythm stability, beat strength, and overall regularity. A value of 0.0 is least danceable and 1.0 is most danceable. |
| energy | double | Energy is a measure from 0.0 to 1.0 and represents a perceptual measure of intensity and activity. Typically, energetic tracks feel fast, loud, and noisy. For example, death metal has high energy, while a Bach prelude scores low on the scale. Perceptual features contributing to this attribute include dynamic range, perceived loudness, timbre, onset rate, and general entropy. |
| key | double | The estimated overall key of the track. Integers map to pitches using standard Pitch Class notation . E.g. 0 = C, 1 = C♯/D♭, 2 = D, and so on. If no key was detected, the value is -1. |
| loudness | double | The overall loudness of a track in decibels (dB). Loudness values are averaged across the entire track and are useful for comparing relative loudness of tracks. Loudness is the quality of a sound that is the primary psychological correlate of physical strength (amplitude). Values typical range between -60 and 0 db. |
| mode | double | Mode indicates the modality (major or minor) of a track, the type of scale from which its melodic content is derived. Major is represented by 1 and minor is 0. |
| speechiness | double | Speechiness detects the presence of spoken words in a track. The more exclusively speech-like the recording (e.g. talk show, audio book, poetry), the closer to 1.0 the attribute value. Values above 0.66 describe tracks that are probably made entirely of spoken words. Values between 0.33 and 0.66 describe tracks that may contain both music and speech, either in sections or layered, including such cases as rap music. Values below 0.33 most likely represent music and other non-speech-like tracks. |
| acousticness | double | A confidence measure from 0.0 to 1.0 of whether the track is acoustic. 1.0 represents high confidence the track is acoustic. |
| instrumentalness | double | Predicts whether a track contains no vocals. “Ooh” and “aah” sounds are treated as instrumental in this context. Rap or spoken word tracks are clearly “vocal”. The closer the instrumentalness value is to 1.0, the greater likelihood the track contains no vocal content. Values above 0.5 are intended to represent instrumental tracks, but confidence is higher as the value approaches 1.0. |
| liveness | double | Detects the presence of an audience in the recording. Higher liveness values represent an increased probability that the track was performed live. A value above 0.8 provides strong likelihood that the track is live. |
| valence | double | A measure from 0.0 to 1.0 describing the musical positiveness conveyed by a track. Tracks with high valence sound more positive (e.g. happy, cheerful, euphoric), while tracks with low valence sound more negative (e.g. sad, depressed, angry). |
| tempo | double | The overall estimated tempo of a track in beats per minute (BPM). In musical terminology, tempo is the speed or pace of a given piece and derives directly from the average beat duration. |
| duration_ms | double | Duration of song in milliseconds. |
Let’s look at the dimesions of the data set.
# Dimension of dataset
dim(spotify_songs_data)## [1] 32833 23
The data set has 3 variables and 32833 observations.
Now let’s analyse the structure of the data set.
# Structure of Data
str(spotify_songs_data)## spec_tbl_df [32,833 x 23] (S3: spec_tbl_df/tbl_df/tbl/data.frame)
## $ track_id : chr [1:32833] "6f807x0ima9a1j3VPbc7VN" "0r7CVbZTWZgbTCYdfa2P31" "1z1Hg7Vb0AhHDiEmnDE79l" "75FpbthrwQmzHlBJLuGdC7" ...
## $ track_name : chr [1:32833] "I Don't Care (with Justin Bieber) - Loud Luxury Remix" "Memories - Dillon Francis Remix" "All the Time - Don Diablo Remix" "Call You Mine - Keanu Silva Remix" ...
## $ track_artist : chr [1:32833] "Ed Sheeran" "Maroon 5" "Zara Larsson" "The Chainsmokers" ...
## $ track_popularity : num [1:32833] 66 67 70 60 69 67 62 69 68 67 ...
## $ track_album_id : chr [1:32833] "2oCs0DGTsRO98Gh5ZSl2Cx" "63rPSO264uRjW1X5E6cWv6" "1HoSmj2eLcsrR0vE9gThr4" "1nqYsOef1yKKuGOVchbsk6" ...
## $ track_album_name : chr [1:32833] "I Don't Care (with Justin Bieber) [Loud Luxury Remix]" "Memories (Dillon Francis Remix)" "All the Time (Don Diablo Remix)" "Call You Mine - The Remixes" ...
## $ track_album_release_date: chr [1:32833] "2019-06-14" "2019-12-13" "2019-07-05" "2019-07-19" ...
## $ playlist_name : chr [1:32833] "Pop Remix" "Pop Remix" "Pop Remix" "Pop Remix" ...
## $ playlist_id : chr [1:32833] "37i9dQZF1DXcZDD7cfEKhW" "37i9dQZF1DXcZDD7cfEKhW" "37i9dQZF1DXcZDD7cfEKhW" "37i9dQZF1DXcZDD7cfEKhW" ...
## $ playlist_genre : chr [1:32833] "pop" "pop" "pop" "pop" ...
## $ playlist_subgenre : chr [1:32833] "dance pop" "dance pop" "dance pop" "dance pop" ...
## $ danceability : num [1:32833] 0.748 0.726 0.675 0.718 0.65 0.675 0.449 0.542 0.594 0.642 ...
## $ energy : num [1:32833] 0.916 0.815 0.931 0.93 0.833 0.919 0.856 0.903 0.935 0.818 ...
## $ key : num [1:32833] 6 11 1 7 1 8 5 4 8 2 ...
## $ loudness : num [1:32833] -2.63 -4.97 -3.43 -3.78 -4.67 ...
## $ mode : num [1:32833] 1 1 0 1 1 1 0 0 1 1 ...
## $ speechiness : num [1:32833] 0.0583 0.0373 0.0742 0.102 0.0359 0.127 0.0623 0.0434 0.0565 0.032 ...
## $ acousticness : num [1:32833] 0.102 0.0724 0.0794 0.0287 0.0803 0.0799 0.187 0.0335 0.0249 0.0567 ...
## $ instrumentalness : num [1:32833] 0.00 4.21e-03 2.33e-05 9.43e-06 0.00 0.00 0.00 4.83e-06 3.97e-06 0.00 ...
## $ liveness : num [1:32833] 0.0653 0.357 0.11 0.204 0.0833 0.143 0.176 0.111 0.637 0.0919 ...
## $ valence : num [1:32833] 0.518 0.693 0.613 0.277 0.725 0.585 0.152 0.367 0.366 0.59 ...
## $ tempo : num [1:32833] 122 100 124 122 124 ...
## $ duration_ms : num [1:32833] 194754 162600 176616 169093 189052 ...
## - attr(*, "spec")=
## .. cols(
## .. track_id = col_character(),
## .. track_name = col_character(),
## .. track_artist = col_character(),
## .. track_popularity = col_double(),
## .. track_album_id = col_character(),
## .. track_album_name = col_character(),
## .. track_album_release_date = col_character(),
## .. playlist_name = col_character(),
## .. playlist_id = col_character(),
## .. playlist_genre = col_character(),
## .. playlist_subgenre = col_character(),
## .. danceability = col_double(),
## .. energy = col_double(),
## .. key = col_double(),
## .. loudness = col_double(),
## .. mode = col_double(),
## .. speechiness = col_double(),
## .. acousticness = col_double(),
## .. instrumentalness = col_double(),
## .. liveness = col_double(),
## .. valence = col_double(),
## .. tempo = col_double(),
## .. duration_ms = col_double()
## .. )
## - attr(*, "problems")=<externalptr>
Let’s summarize the data set to observe the min, median and maximum values of numerical variables and length of character variables.
# Summary
summary(spotify_songs_data)## track_id track_name track_artist track_popularity
## Length:32833 Length:32833 Length:32833 Min. : 0.00
## Class :character Class :character Class :character 1st Qu.: 24.00
## Mode :character Mode :character Mode :character Median : 45.00
## Mean : 42.48
## 3rd Qu.: 62.00
## Max. :100.00
## track_album_id track_album_name track_album_release_date
## Length:32833 Length:32833 Length:32833
## Class :character Class :character Class :character
## Mode :character Mode :character Mode :character
##
##
##
## playlist_name playlist_id playlist_genre playlist_subgenre
## Length:32833 Length:32833 Length:32833 Length:32833
## Class :character Class :character Class :character Class :character
## Mode :character Mode :character Mode :character Mode :character
##
##
##
## danceability energy key loudness
## Min. :0.0000 Min. :0.000175 Min. : 0.000 Min. :-46.448
## 1st Qu.:0.5630 1st Qu.:0.581000 1st Qu.: 2.000 1st Qu.: -8.171
## Median :0.6720 Median :0.721000 Median : 6.000 Median : -6.166
## Mean :0.6548 Mean :0.698619 Mean : 5.374 Mean : -6.720
## 3rd Qu.:0.7610 3rd Qu.:0.840000 3rd Qu.: 9.000 3rd Qu.: -4.645
## Max. :0.9830 Max. :1.000000 Max. :11.000 Max. : 1.275
## mode speechiness acousticness instrumentalness
## Min. :0.0000 Min. :0.0000 Min. :0.0000 Min. :0.0000000
## 1st Qu.:0.0000 1st Qu.:0.0410 1st Qu.:0.0151 1st Qu.:0.0000000
## Median :1.0000 Median :0.0625 Median :0.0804 Median :0.0000161
## Mean :0.5657 Mean :0.1071 Mean :0.1753 Mean :0.0847472
## 3rd Qu.:1.0000 3rd Qu.:0.1320 3rd Qu.:0.2550 3rd Qu.:0.0048300
## Max. :1.0000 Max. :0.9180 Max. :0.9940 Max. :0.9940000
## liveness valence tempo duration_ms
## Min. :0.0000 Min. :0.0000 Min. : 0.00 Min. : 4000
## 1st Qu.:0.0927 1st Qu.:0.3310 1st Qu.: 99.96 1st Qu.:187819
## Median :0.1270 Median :0.5120 Median :121.98 Median :216000
## Mean :0.1902 Mean :0.5106 Mean :120.88 Mean :225800
## 3rd Qu.:0.2480 3rd Qu.:0.6930 3rd Qu.:133.92 3rd Qu.:253585
## Max. :0.9960 Max. :0.9910 Max. :239.44 Max. :517810
Checking for missing or null values in the data set. Null values affect the analysis to a great extent if present in large percent.
# Null values
colSums(is.na(spotify_songs_data))## track_id track_name track_artist
## 0 5 5
## track_popularity track_album_id track_album_name
## 0 0 5
## track_album_release_date playlist_name playlist_id
## 0 0 0
## playlist_genre playlist_subgenre danceability
## 0 0 0
## energy key loudness
## 0 0 0
## mode speechiness acousticness
## 0 0 0
## instrumentalness liveness valence
## 0 0 0
## tempo duration_ms
## 0 0
Track_name, track_album_name and track_artist variables contain 5 missing values out of 32833 rows. It is safe to remove these 5 rows without any significant impact on our data
spotify_songs_data <- na.omit(spotify_songs_data)
colSums(is.na(spotify_songs_data))## track_id track_name track_artist
## 0 0 0
## track_popularity track_album_id track_album_name
## 0 0 0
## track_album_release_date playlist_name playlist_id
## 0 0 0
## playlist_genre playlist_subgenre danceability
## 0 0 0
## energy key loudness
## 0 0 0
## mode speechiness acousticness
## 0 0 0
## instrumentalness liveness valence
## 0 0 0
## tempo duration_ms
## 0 0
Now let’s identify if the dataset has any duplicate records.
dim(spotify_songs_data)## [1] 32828 23
length(unique(spotify_songs_data$track_id))## [1] 28352
Number of unique track_id are 28352 however the total row count is 32828. We will remove these duplicate records.
spotify_songs_data <- spotify_songs_data[!duplicated(spotify_songs_data$track_id),]
dim(spotify_songs_data)## [1] 28352 23
As I am analyzing track popularity depending on the track features of the songs, Columns such as track_id, track_name, track_artist, track_album_id, track_album_name, playlist_id, playlist_name, track_album_release_date, playlist_genre, playlist_subgenre, key and mode are not required for clustering.
library(dplyr)
spotify_songs_data <- spotify_songs_data %>%
select(-c(track_id, track_name, track_artist, track_album_id, track_album_name, playlist_id, playlist_name, track_album_release_date, playlist_genre, playlist_subgenre, key, mode))
head(spotify_songs_data,2)## # A tibble: 2 x 11
## track_popularity danceability energy loudness speechiness acousticness
## <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
## 1 66 0.748 0.916 -2.63 0.0583 0.102
## 2 67 0.726 0.815 -4.97 0.0373 0.0724
## # ... with 5 more variables: instrumentalness <dbl>, liveness <dbl>,
## # valence <dbl>, tempo <dbl>, duration_ms <dbl>
Now that the data set is clean let’s have a look at it.
library(knitr)
output_data <- head(spotify_songs_data, n = 2)
kable(output_data, filter = 'top', options = list(pageLength = 25))| track_popularity | danceability | energy | loudness | speechiness | acousticness | instrumentalness | liveness | valence | tempo | duration_ms |
|---|---|---|---|---|---|---|---|---|---|---|
| 66 | 0.748 | 0.916 | -2.634 | 0.0583 | 0.1020 | 0.00000 | 0.0653 | 0.518 | 122.036 | 194754 |
| 67 | 0.726 | 0.815 | -4.969 | 0.0373 | 0.0724 | 0.00421 | 0.3570 | 0.693 | 99.972 | 162600 |
Now I am plotting the correlation matrix to see how various track features are related to each other.
library(ggcorrplot)
songs_corr <- spotify_songs_data %>%
select(track_popularity,danceability,energy,loudness,speechiness,acousticness,instrumentalness, liveness, valence, tempo)
corr <- round(cor(songs_corr), 1)
head(corr[, 1:6])## track_popularity danceability energy loudness speechiness
## track_popularity 1.0 0.0 -0.1 0.0 0.0
## danceability 0.0 1.0 -0.1 0.0 0.2
## energy -0.1 -0.1 1.0 0.7 0.0
## loudness 0.0 0.0 0.7 1.0 0.0
## speechiness 0.0 0.2 0.0 0.0 1.0
## acousticness 0.1 0.0 -0.5 -0.4 0.0
## acousticness
## track_popularity 0.1
## danceability 0.0
## energy -0.5
## loudness -0.4
## speechiness 0.0
## acousticness 1.0
ggcorrplot(corr, hc.order = TRUE, type = "lower",lab = TRUE)Below are the results from the correlation matrix.
Principle component analysis is used in exploratory data analysis and for making predictive models. It is a dimensionality reduction method that is used to reduce the dimension of large datasets into smaller ones which can be easily analyzed and visualized. Although the dimension is reduced, most of the information is still retained in the dataset.
spotify_PCA <- spotify_songs_data %>%
select(danceability, energy, speechiness, acousticness, instrumentalness, liveness, valence, tempo)
spotify_PCA_scaled <- scale(spotify_PCA)
spotify_PCA_results <- prcomp(spotify_PCA_scaled, scale = TRUE)
spotify_PCA_variance <- spotify_PCA_results$sdev^2 / sum(spotify_PCA_results$sdev^2)
plot(cumsum(spotify_PCA_variance), xlab = "Principal Component",
ylab = "Cumulative Proportion of Variance Explained",
type = "b")
K-means clustering is an iterative algorithm that tries to partition the dataset into K pre-defined distinct non-overlapping subgroups (clusters) where each data point belongs to only one group. It assigns data points to a cluster such that the sum of the squared distance between the data points and the cluster’s centroid is at the minimum. The less variation we have within clusters, the more homogeneous (similar) the data points are within the same cluster.
cluster_spotify_data <- spotify_songs_data[, c('instrumentalness', 'danceability', 'valence', 'energy', 'liveness', 'tempo', 'speechiness', 'acousticness')]
cluster_spotify_data_v1 <- scale(cluster_spotify_data[, c('instrumentalness', 'danceability', 'valence', 'energy', 'liveness', 'tempo', 'speechiness', 'acousticness')])k_mean_2 <- kmeans(cluster_spotify_data_v1, centers = 2, nstart = 25)
k_mean_3 <- kmeans(cluster_spotify_data_v1, centers = 3, nstart = 25)
k_mean_4 <- kmeans(cluster_spotify_data_v1, centers = 4, nstart = 25)
k_mean_5 <- kmeans(cluster_spotify_data_v1, centers = 5, nstart = 25)
k_mean_6 <- kmeans(cluster_spotify_data_v1, centers = 6, nstart = 25)Elbow method is one of the several method in determine the optimum value of k. It uses the sum of squared distance (SSE) to choose an ideal value of k clusters based on the distance between the data points and their assigned clusters.
set.seed(100)
library("factoextra")
fviz_nbclust(cluster_spotify_data[1:1000,], kmeans, method = "wss")From the above graph I can conclude that the optimum value of k is 3 clusters.
I created the final model using the optimum number of clusters i.e., 3 clusters.
plot_k_mean <- fviz_cluster(k_mean_3, geom = "point", data = cluster_spotify_data_v1) + ggtitle("k = 3")
plot_k_mean
To check whether the clusters I created are good enough, I can see the value in the cluster. The goodness of the clustering results can be seen from the following 4 values:
k_mean_3$withinss## [1] 69808.41 67015.79 34787.90
k_mean_3$betweenss## [1] 55195.89
k_mean_3$totss## [1] 226808
k_mean_3$tot.withinss## [1] 171612.1
From the evaluation of the model, the betweenss / tot.withinss value is 0.32 which is somewhat close to 1 so it can be said that the model is decent.
k_mean_3$betweenss/k_mean_3$tot.withinss## [1] 0.3216317
Now I am extracting the clustering information from the resulting Kmeans $cluster and add them to as a new column named cluster to the spotify_songs_data dataset.
spotify_songs_data$cluster <- k_mean_3$cluster
head(spotify_songs_data)## # A tibble: 6 x 12
## track_popularity danceability energy loudness speechiness acousticness
## <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
## 1 66 0.748 0.916 -2.63 0.0583 0.102
## 2 67 0.726 0.815 -4.97 0.0373 0.0724
## 3 70 0.675 0.931 -3.43 0.0742 0.0794
## 4 60 0.718 0.93 -3.78 0.102 0.0287
## 5 69 0.65 0.833 -4.67 0.0359 0.0803
## 6 67 0.675 0.919 -5.38 0.127 0.0799
## # ... with 6 more variables: instrumentalness <dbl>, liveness <dbl>,
## # valence <dbl>, tempo <dbl>, duration_ms <dbl>, cluster <int>
To profile the clusters, I am grouping the spotify_songs_data dataset based on the 3 clusters and calculating the mean value of each track feature.
cluster_prof <- spotify_songs_data %>%
group_by(cluster) %>%
summarise_all("mean")
rmarkdown::paged_table(cluster_prof)