Introduction

Background

Spotify is a Swedish-based audio streaming and media services provider. Spotify was launched in October 2008 and it is one of the biggest streaming device in the world.



Objective

Idea

There are various factors responsible for making songs popular. Every person has their own liking towards the music. Music companies that can identify the person’s liking and suggest the best recommendations for their long drives, parties, ‘me time’ depending on the situation and mood wins the market. This is where our prediction model comes into the picture. Our goal is to perform an in-depth analysis of the Spotify dataset and build the best recommendation model.



Goal

Dataset has different variables such as loudness, energy, speechiness, liveness, etc. Our goal is to identify how these factors contribute to the popularity of a song by-

  • Identifying correlation between popularity and other variables.
  • Identifying features of each genre and how to segregate them for recommending songs.
  • Understanding and Analyzing features that contribute to the popularity of the song.
  • Predicting track popularity based on various features.



Proposed Analytical Methodology

  • We plan to perform a linear and multi-linear regression analysis to understand the relationship between track popularity and other variables.
  • In addition to that we would model the dataset using various statistical models such as random forest, fitted etc. and choose the one giving the highest value of R2 and least error.
  • We also plan to perform multi-collinearity testing to understand the interaction between various variables other than track_popularity. This needs to be analyzed because if other variables have interaction among themselves, it will affect standard error and p-value.



Outcomes

  • It will help to identify trends of popular songs and recommend those songs.
  • This analysis will help in recommending good songs for each user as per their historical behaviors on the music application. E.g. if a person is playing a song that has a certain set of features, the app can recommend another song having similar features (e.g. acousticness, liveness, etc.). Depending on the customer’s behavior recommendation system can be improved continuously.
  • It will help musicians to understand if certain features of the songs make them popular.



Packages Used

Required Packages

The following packages are used in the analysis:

Package Name Purpose
library(kableExtra) kableExtra
library(spotifyr) spotifyr
library(knitr) knitr
library(tidyverse) tidyverse
library(plotly) plotly
library(imager) imager
library(readr) readr
library(ggcorrplot) ggcorrplot



Installation

Let’s install these packages.

my_path <- getwd()
.libPaths(my_path)

#install.packages("kableExtra",repos = "http://cran.us.r-project.org")
#install.packages("spotifyr",repos = "http://cran.us.r-project.org")
#install.packages("tidyverse",repos = "http://cran.us.r-project.org")
#install.packages("knitr",repos = "http://cran.us.r-project.org")
#install.packages("plotly",repos = "http://cran.us.r-project.org")
#install.packages("imager",repos = "http://cran.us.r-project.org")
#install.packages("readr",repos = "http://cran.us.r-project.org")
#install.packages("ggcorrplot",repos = "http://cran.us.r-project.org")
#install.packages("corpus",repos = "http://cran.us.r-project.org")
#install.packages("wordcloud",repos = "http://cran.us.r-project.org")
#install.packages("tm",repos = "http://cran.us.r-project.org")
#install.packages("randomForest",repos = "http://cran.us.r-project.org")
#install.packages("Metrics",repos = "http://cran.us.r-project.org")
#install.packages("cluster",repos = "http://cran.us.r-project.org")
#install.packages("factoextra",repos = "http://cran.us.r-project.org")
#install.packages("gridExtra",repos = "http://cran.us.r-project.org")



Data Preparation

Data Source

The data this week comes from Spotify via the spotifyr package. Charlie Thompson, Josiah Parry, Donal Phipps, and Tom Wolff authored this package to make it easier to get either your own data or general metadata arounds songs from Spotify’s API. Make sure to check out the spotifyr package website to see how you can collect your own data!

Kaylin Pavlik had a recent blogpost using the audio features to explore and classify songs. She used the spotifyr package to collect about 5000 songs from 6 main categories (EDM, Latin, Pop, R&B, Rap, & Rock).

#library(readr)
spotify_songs_data <- read.csv('https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2020/2020-01-21/spotify_songs.csv')


Data Dictionary

Variable Class Description
track_id character Song unique ID
track_name character Song Name
track_artist character Song Artist
track_popularity double Song Popularity (0-100) where higher is better
track_album_id character Album unique ID
track_album_name character Song album name
track_album_release_date character Date when album released
playlist_name character Name of playlist
playlist_id character Playlist ID
playlist_genre character Playlist genre
playlist_name character Name of playlist
playlist_name character Name of playlist
danceability double Danceability describes how suitable a track is for dancing based on a combination of musical elements including tempo, rhythm stability, beat strength, and overall regularity. A value of 0.0 is least danceable and 1.0 is most danceable.
energy double Energy is a measure from 0.0 to 1.0 and represents a perceptual measure of intensity and activity. Typically, energetic tracks feel fast, loud, and noisy. For example, death metal has high energy, while a Bach prelude scores low on the scale. Perceptual features contributing to this attribute include dynamic range, perceived loudness, timbre, onset rate, and general entropy.
key double The estimated overall key of the track. Integers map to pitches using standard Pitch Class notation . E.g. 0 = C, 1 = C♯/D♭, 2 = D, and so on. If no key was detected, the value is -1.
loudness double The overall loudness of a track in decibels (dB). Loudness values are averaged across the entire track and are useful for comparing relative loudness of tracks. Loudness is the quality of a sound that is the primary psychological correlate of physical strength (amplitude). Values typical range between -60 and 0 db.
mode double Mode indicates the modality (major or minor) of a track, the type of scale from which its melodic content is derived. Major is represented by 1 and minor is 0.
speechiness double Speechiness detects the presence of spoken words in a track. The more exclusively speech-like the recording (e.g. talk show, audio book, poetry), the closer to 1.0 the attribute value. Values above 0.66 describe tracks that are probably made entirely of spoken words. Values between 0.33 and 0.66 describe tracks that may contain both music and speech, either in sections or layered, including such cases as rap music. Values below 0.33 most likely represent music and other non-speech-like tracks.
acousticness double A confidence measure from 0.0 to 1.0 of whether the track is acoustic. 1.0 represents high confidence the track is acoustic.
instrumentalness double Predicts whether a track contains no vocals. “Ooh” and “aah” sounds are treated as instrumental in this context. Rap or spoken word tracks are clearly “vocal”. The closer the instrumentalness value is to 1.0, the greater likelihood the track contains no vocal content. Values above 0.5 are intended to represent instrumental tracks, but confidence is higher as the value approaches 1.0.
liveness double Detects the presence of an audience in the recording. Higher liveness values represent an increased probability that the track was performed live. A value above 0.8 provides strong likelihood that the track is live.
valence double A measure from 0.0 to 1.0 describing the musical positiveness conveyed by a track. Tracks with high valence sound more positive (e.g. happy, cheerful, euphoric), while tracks with low valence sound more negative (e.g. sad, depressed, angry).
tempo double The overall estimated tempo of a track in beats per minute (BPM). In musical terminology, tempo is the speed or pace of a given piece and derives directly from the average beat duration.
duration_ms double Duration of song in milliseconds.


Data Cleaning

Original Dataset

Let’s look at the dimesions of the data set.

# Dimension of dataset
dim(spotify_songs_data)
## [1] 32833    23

The data set has 3 variables and 32833 observations.


Now let’s analyse the structure of the data set.

# Structure of Data
str(spotify_songs_data)
## 'data.frame':    32833 obs. of  23 variables:
##  $ track_id                : chr  "6f807x0ima9a1j3VPbc7VN" "0r7CVbZTWZgbTCYdfa2P31" "1z1Hg7Vb0AhHDiEmnDE79l" "75FpbthrwQmzHlBJLuGdC7" ...
##  $ track_name              : chr  "I Don't Care (with Justin Bieber) - Loud Luxury Remix" "Memories - Dillon Francis Remix" "All the Time - Don Diablo Remix" "Call You Mine - Keanu Silva Remix" ...
##  $ track_artist            : chr  "Ed Sheeran" "Maroon 5" "Zara Larsson" "The Chainsmokers" ...
##  $ track_popularity        : int  66 67 70 60 69 67 62 69 68 67 ...
##  $ track_album_id          : chr  "2oCs0DGTsRO98Gh5ZSl2Cx" "63rPSO264uRjW1X5E6cWv6" "1HoSmj2eLcsrR0vE9gThr4" "1nqYsOef1yKKuGOVchbsk6" ...
##  $ track_album_name        : chr  "I Don't Care (with Justin Bieber) [Loud Luxury Remix]" "Memories (Dillon Francis Remix)" "All the Time (Don Diablo Remix)" "Call You Mine - The Remixes" ...
##  $ track_album_release_date: chr  "2019-06-14" "2019-12-13" "2019-07-05" "2019-07-19" ...
##  $ playlist_name           : chr  "Pop Remix" "Pop Remix" "Pop Remix" "Pop Remix" ...
##  $ playlist_id             : chr  "37i9dQZF1DXcZDD7cfEKhW" "37i9dQZF1DXcZDD7cfEKhW" "37i9dQZF1DXcZDD7cfEKhW" "37i9dQZF1DXcZDD7cfEKhW" ...
##  $ playlist_genre          : chr  "pop" "pop" "pop" "pop" ...
##  $ playlist_subgenre       : chr  "dance pop" "dance pop" "dance pop" "dance pop" ...
##  $ danceability            : num  0.748 0.726 0.675 0.718 0.65 0.675 0.449 0.542 0.594 0.642 ...
##  $ energy                  : num  0.916 0.815 0.931 0.93 0.833 0.919 0.856 0.903 0.935 0.818 ...
##  $ key                     : int  6 11 1 7 1 8 5 4 8 2 ...
##  $ loudness                : num  -2.63 -4.97 -3.43 -3.78 -4.67 ...
##  $ mode                    : int  1 1 0 1 1 1 0 0 1 1 ...
##  $ speechiness             : num  0.0583 0.0373 0.0742 0.102 0.0359 0.127 0.0623 0.0434 0.0565 0.032 ...
##  $ acousticness            : num  0.102 0.0724 0.0794 0.0287 0.0803 0.0799 0.187 0.0335 0.0249 0.0567 ...
##  $ instrumentalness        : num  0.00 4.21e-03 2.33e-05 9.43e-06 0.00 0.00 0.00 4.83e-06 3.97e-06 0.00 ...
##  $ liveness                : num  0.0653 0.357 0.11 0.204 0.0833 0.143 0.176 0.111 0.637 0.0919 ...
##  $ valence                 : num  0.518 0.693 0.613 0.277 0.725 0.585 0.152 0.367 0.366 0.59 ...
##  $ tempo                   : num  122 100 124 122 124 ...
##  $ duration_ms             : int  194754 162600 176616 169093 189052 163049 187675 207619 193187 253040 ...


Let’s summarize the data set to observe the min, median and maximum values of numerical variables and length of character variables.

# Summary
summary(spotify_songs_data)
##    track_id          track_name        track_artist       track_popularity
##  Length:32833       Length:32833       Length:32833       Min.   :  0.00  
##  Class :character   Class :character   Class :character   1st Qu.: 24.00  
##  Mode  :character   Mode  :character   Mode  :character   Median : 45.00  
##                                                           Mean   : 42.48  
##                                                           3rd Qu.: 62.00  
##                                                           Max.   :100.00  
##  track_album_id     track_album_name   track_album_release_date
##  Length:32833       Length:32833       Length:32833            
##  Class :character   Class :character   Class :character        
##  Mode  :character   Mode  :character   Mode  :character        
##                                                                
##                                                                
##                                                                
##  playlist_name      playlist_id        playlist_genre     playlist_subgenre 
##  Length:32833       Length:32833       Length:32833       Length:32833      
##  Class :character   Class :character   Class :character   Class :character  
##  Mode  :character   Mode  :character   Mode  :character   Mode  :character  
##                                                                             
##                                                                             
##                                                                             
##   danceability        energy              key            loudness      
##  Min.   :0.0000   Min.   :0.000175   Min.   : 0.000   Min.   :-46.448  
##  1st Qu.:0.5630   1st Qu.:0.581000   1st Qu.: 2.000   1st Qu.: -8.171  
##  Median :0.6720   Median :0.721000   Median : 6.000   Median : -6.166  
##  Mean   :0.6548   Mean   :0.698619   Mean   : 5.374   Mean   : -6.720  
##  3rd Qu.:0.7610   3rd Qu.:0.840000   3rd Qu.: 9.000   3rd Qu.: -4.645  
##  Max.   :0.9830   Max.   :1.000000   Max.   :11.000   Max.   :  1.275  
##       mode         speechiness      acousticness    instrumentalness   
##  Min.   :0.0000   Min.   :0.0000   Min.   :0.0000   Min.   :0.0000000  
##  1st Qu.:0.0000   1st Qu.:0.0410   1st Qu.:0.0151   1st Qu.:0.0000000  
##  Median :1.0000   Median :0.0625   Median :0.0804   Median :0.0000161  
##  Mean   :0.5657   Mean   :0.1071   Mean   :0.1753   Mean   :0.0847472  
##  3rd Qu.:1.0000   3rd Qu.:0.1320   3rd Qu.:0.2550   3rd Qu.:0.0048300  
##  Max.   :1.0000   Max.   :0.9180   Max.   :0.9940   Max.   :0.9940000  
##     liveness         valence           tempo         duration_ms    
##  Min.   :0.0000   Min.   :0.0000   Min.   :  0.00   Min.   :  4000  
##  1st Qu.:0.0927   1st Qu.:0.3310   1st Qu.: 99.96   1st Qu.:187819  
##  Median :0.1270   Median :0.5120   Median :121.98   Median :216000  
##  Mean   :0.1902   Mean   :0.5106   Mean   :120.88   Mean   :225800  
##  3rd Qu.:0.2480   3rd Qu.:0.6930   3rd Qu.:133.92   3rd Qu.:253585  
##  Max.   :0.9960   Max.   :0.9910   Max.   :239.44   Max.   :517810


Checking for missing or null values in the data set. Null values affect the analysis to a great extent if present in large percent.

# Null values
colSums(is.na(spotify_songs_data))
##                 track_id               track_name             track_artist 
##                        0                        5                        5 
##         track_popularity           track_album_id         track_album_name 
##                        0                        0                        5 
## track_album_release_date            playlist_name              playlist_id 
##                        0                        0                        0 
##           playlist_genre        playlist_subgenre             danceability 
##                        0                        0                        0 
##                   energy                      key                 loudness 
##                        0                        0                        0 
##                     mode              speechiness             acousticness 
##                        0                        0                        0 
##         instrumentalness                 liveness                  valence 
##                        0                        0                        0 
##                    tempo              duration_ms 
##                        0                        0



Cleaning the dataset

Track_name, track_album_name and track_artist variables contain 5 missing values out of 32833 rows. It is safe to remove these 5 rows without any significant impact on our data

spotify_songs_data <- na.omit(spotify_songs_data)
colSums(is.na(spotify_songs_data))
##                 track_id               track_name             track_artist 
##                        0                        0                        0 
##         track_popularity           track_album_id         track_album_name 
##                        0                        0                        0 
## track_album_release_date            playlist_name              playlist_id 
##                        0                        0                        0 
##           playlist_genre        playlist_subgenre             danceability 
##                        0                        0                        0 
##                   energy                      key                 loudness 
##                        0                        0                        0 
##                     mode              speechiness             acousticness 
##                        0                        0                        0 
##         instrumentalness                 liveness                  valence 
##                        0                        0                        0 
##                    tempo              duration_ms 
##                        0                        0


Playlist_genre, playlist_subgenre, mode and key are categorical columns hence they should be converted to factor from string for further data analysis.

library(dplyr)
spotify_songs_data <-spotify_songs_data %>%
  mutate(playlist_genre=as.factor(spotify_songs_data$playlist_genre),
         playlist_subgenre=as.factor(spotify_songs_data$playlist_subgenre),
         mode=as.factor(mode),
         key=as.factor(key))


Now we can see that categorical variables have factor as data type.

str(spotify_songs_data)
## 'data.frame':    32828 obs. of  23 variables:
##  $ track_id                : chr  "6f807x0ima9a1j3VPbc7VN" "0r7CVbZTWZgbTCYdfa2P31" "1z1Hg7Vb0AhHDiEmnDE79l" "75FpbthrwQmzHlBJLuGdC7" ...
##  $ track_name              : chr  "I Don't Care (with Justin Bieber) - Loud Luxury Remix" "Memories - Dillon Francis Remix" "All the Time - Don Diablo Remix" "Call You Mine - Keanu Silva Remix" ...
##  $ track_artist            : chr  "Ed Sheeran" "Maroon 5" "Zara Larsson" "The Chainsmokers" ...
##  $ track_popularity        : int  66 67 70 60 69 67 62 69 68 67 ...
##  $ track_album_id          : chr  "2oCs0DGTsRO98Gh5ZSl2Cx" "63rPSO264uRjW1X5E6cWv6" "1HoSmj2eLcsrR0vE9gThr4" "1nqYsOef1yKKuGOVchbsk6" ...
##  $ track_album_name        : chr  "I Don't Care (with Justin Bieber) [Loud Luxury Remix]" "Memories (Dillon Francis Remix)" "All the Time (Don Diablo Remix)" "Call You Mine - The Remixes" ...
##  $ track_album_release_date: chr  "2019-06-14" "2019-12-13" "2019-07-05" "2019-07-19" ...
##  $ playlist_name           : chr  "Pop Remix" "Pop Remix" "Pop Remix" "Pop Remix" ...
##  $ playlist_id             : chr  "37i9dQZF1DXcZDD7cfEKhW" "37i9dQZF1DXcZDD7cfEKhW" "37i9dQZF1DXcZDD7cfEKhW" "37i9dQZF1DXcZDD7cfEKhW" ...
##  $ playlist_genre          : Factor w/ 6 levels "edm","latin",..: 3 3 3 3 3 3 3 3 3 3 ...
##  $ playlist_subgenre       : Factor w/ 24 levels "album rock","big room",..: 4 4 4 4 4 4 4 4 4 4 ...
##  $ danceability            : num  0.748 0.726 0.675 0.718 0.65 0.675 0.449 0.542 0.594 0.642 ...
##  $ energy                  : num  0.916 0.815 0.931 0.93 0.833 0.919 0.856 0.903 0.935 0.818 ...
##  $ key                     : Factor w/ 12 levels "0","1","2","3",..: 7 12 2 8 2 9 6 5 9 3 ...
##  $ loudness                : num  -2.63 -4.97 -3.43 -3.78 -4.67 ...
##  $ mode                    : Factor w/ 2 levels "0","1": 2 2 1 2 2 2 1 1 2 2 ...
##  $ speechiness             : num  0.0583 0.0373 0.0742 0.102 0.0359 0.127 0.0623 0.0434 0.0565 0.032 ...
##  $ acousticness            : num  0.102 0.0724 0.0794 0.0287 0.0803 0.0799 0.187 0.0335 0.0249 0.0567 ...
##  $ instrumentalness        : num  0.00 4.21e-03 2.33e-05 9.43e-06 0.00 0.00 0.00 4.83e-06 3.97e-06 0.00 ...
##  $ liveness                : num  0.0653 0.357 0.11 0.204 0.0833 0.143 0.176 0.111 0.637 0.0919 ...
##  $ valence                 : num  0.518 0.693 0.613 0.277 0.725 0.585 0.152 0.367 0.366 0.59 ...
##  $ tempo                   : num  122 100 124 122 124 ...
##  $ duration_ms             : int  194754 162600 176616 169093 189052 163049 187675 207619 193187 253040 ...
##  - attr(*, "na.action")= 'omit' Named int [1:5] 8152 9283 9284 19569 19812
##   ..- attr(*, "names")= chr [1:5] "8152" "9283" "9284" "19569" ...


Now let’s identify if the dat aset has any duplicate records.

dim(spotify_songs_data)
## [1] 32828    23
length(unique(spotify_songs_data$track_id))
## [1] 28352


Number of unique track_id are 28352 however the total row count is 32828. We will remove these duplicate records.

spotify_songs_data <- spotify_songs_data[!duplicated(spotify_songs_data$track_id),]
dim(spotify_songs_data)
## [1] 28352    23


As we are analyzing track popularity depending on the features of the songs. Columns such as track_id, track_name,track_album_id, track_album_name, playlist_id, playlist_name are redundant columns.

spotify_songs_data <- spotify_songs_data %>% select(-c(track_id, track_name,track_album_id,track_album_name,playlist_id,playlist_name))
head(spotify_songs_data,2)
##   track_artist track_popularity track_album_release_date playlist_genre
## 1   Ed Sheeran               66               2019-06-14            pop
## 2     Maroon 5               67               2019-12-13            pop
##   playlist_subgenre danceability energy key loudness mode speechiness
## 1         dance pop        0.748  0.916   6   -2.634    1      0.0583
## 2         dance pop        0.726  0.815  11   -4.969    1      0.0373
##   acousticness instrumentalness liveness valence   tempo duration_ms
## 1       0.1020          0.00000   0.0653   0.518 122.036      194754
## 2       0.0724          0.00421   0.3570   0.693  99.972      162600


To analyze which type of songs are released in which year, we are manipulating track_album_release_date column. Separating month and year columns will help us to analyse data set in an efficient way.

spotify_songs_data$year <- substr(spotify_songs_data$track_album_release_date,1,4)
spotify_songs_data$month <- substr(spotify_songs_data$track_album_release_date,6,7)


Changing data type of month and year column from character to numeric.

spotify_songs_data$year <- as.numeric(spotify_songs_data$year)
spotify_songs_data$month <- as.numeric(spotify_songs_data$month)



Cleaned Dataset

Now that the data set is clean let’s have a look at it.

library(knitr)
output_data <- head(spotify_songs_data, n = 2)
kable(output_data, filter = 'top', options = list(pageLength = 25))
track_artist track_popularity track_album_release_date playlist_genre playlist_subgenre danceability energy key loudness mode speechiness acousticness instrumentalness liveness valence tempo duration_ms year month
Ed Sheeran 66 2019-06-14 pop dance pop 0.748 0.916 6 -2.634 1 0.0583 0.1020 0.00000 0.0653 0.518 122.036 194754 2019 6
Maroon 5 67 2019-12-13 pop dance pop 0.726 0.815 11 -4.969 1 0.0373 0.0724 0.00421 0.3570 0.693 99.972 162600 2019 12



Exploratory Data Analysis

Definition

In statistics, exploratory data analysis is an approach of analyzing data sets to summarize their main characteristics, often using statistical graphics and other data visualization methods.

As of now we have the below columns in our dataset

str(spotify_songs_data)
## 'data.frame':    28352 obs. of  19 variables:
##  $ track_artist            : chr  "Ed Sheeran" "Maroon 5" "Zara Larsson" "The Chainsmokers" ...
##  $ track_popularity        : int  66 67 70 60 69 67 62 69 68 67 ...
##  $ track_album_release_date: chr  "2019-06-14" "2019-12-13" "2019-07-05" "2019-07-19" ...
##  $ playlist_genre          : Factor w/ 6 levels "edm","latin",..: 3 3 3 3 3 3 3 3 3 3 ...
##  $ playlist_subgenre       : Factor w/ 24 levels "album rock","big room",..: 4 4 4 4 4 4 4 4 4 4 ...
##  $ danceability            : num  0.748 0.726 0.675 0.718 0.65 0.675 0.449 0.542 0.594 0.642 ...
##  $ energy                  : num  0.916 0.815 0.931 0.93 0.833 0.919 0.856 0.903 0.935 0.818 ...
##  $ key                     : Factor w/ 12 levels "0","1","2","3",..: 7 12 2 8 2 9 6 5 9 3 ...
##  $ loudness                : num  -2.63 -4.97 -3.43 -3.78 -4.67 ...
##  $ mode                    : Factor w/ 2 levels "0","1": 2 2 1 2 2 2 1 1 2 2 ...
##  $ speechiness             : num  0.0583 0.0373 0.0742 0.102 0.0359 0.127 0.0623 0.0434 0.0565 0.032 ...
##  $ acousticness            : num  0.102 0.0724 0.0794 0.0287 0.0803 0.0799 0.187 0.0335 0.0249 0.0567 ...
##  $ instrumentalness        : num  0.00 4.21e-03 2.33e-05 9.43e-06 0.00 0.00 0.00 4.83e-06 3.97e-06 0.00 ...
##  $ liveness                : num  0.0653 0.357 0.11 0.204 0.0833 0.143 0.176 0.111 0.637 0.0919 ...
##  $ valence                 : num  0.518 0.693 0.613 0.277 0.725 0.585 0.152 0.367 0.366 0.59 ...
##  $ tempo                   : num  122 100 124 122 124 ...
##  $ duration_ms             : int  194754 162600 176616 169093 189052 163049 187675 207619 193187 253040 ...
##  $ year                    : num  2019 2019 2019 2019 2019 ...
##  $ month                   : num  6 12 7 7 3 7 7 8 6 6 ...
##  - attr(*, "na.action")= 'omit' Named int [1:5] 8152 9283 9284 19569 19812
##   ..- attr(*, "names")= chr [1:5] "8152" "9283" "9284" "19569" ...


Let’s analyze the number of unique artist in our data set.

length(unique(spotify_songs_data$track_artist))
## [1] 10692

We have 10692 unique artists in our data set.


Checking artist that has released most number of songs.

library(plotly)
highest_tracks <- spotify_songs_data %>% group_by(Artis_Name = track_artist) %>%
                                        summarise(No_of_tracks = n()) %>%
                                        arrange(desc(No_of_tracks)) %>%
                                        top_n(15, wt = No_of_tracks) %>% 
                                        ggplot(aes(x = Artis_Name, y = No_of_tracks)) + geom_bar(stat = "identity") + coord_flip() + labs(title = "Top Artists having the most track releases", x = "Artist Name", y = "Number of Tracks")
ggplotly(highest_tracks)


Now we will summaries songs per Genre.

Genre_spotify_songs_summary <- spotify_songs_data %>% group_by(Genre_of_song = playlist_genre) %>%
  summarise(No_of_tracks = n()) %>% 
  arrange(desc(No_of_tracks))


#library(utils)
print(Genre_spotify_songs_summary)
## # A tibble: 6 x 2
##   Genre_of_song No_of_tracks
##   <fct>                <int>
## 1 rap                   5398
## 2 pop                   5132
## 3 edm                   4877
## 4 r&b                   4504
## 5 rock                  4305
## 6 latin                 4136


Now, lets analyze the average popularity of songs released every year.

spotify_yearly_popularity <-spotify_songs_data%>%
                            group_by(year)%>%
                            mutate(avg_popularity_year=mean(track_popularity))%>%
                            select(year,avg_popularity_year)%>%
                            unique()

                    
ggplot(spotify_yearly_popularity,aes(x=year,y=avg_popularity_year))+geom_line()+ geom_smooth(method = "lm")+scale_x_continuous(breaks=seq(1950, 2020,5))

We can see that the average popularity year wise has shown decreasing trend till 2016.


Now plotting the correlation matrix to see how variables are related to each other.

library(ggcorrplot)
songs_corr <- spotify_songs_data %>%
  select(track_popularity,danceability,energy,loudness,speechiness,acousticness,instrumentalness, liveness, valence, tempo)

corr <- round(cor(songs_corr), 1)
head(corr[, 1:6])
##                  track_popularity danceability energy loudness speechiness
## track_popularity              1.0          0.0   -0.1      0.0         0.0
## danceability                  0.0          1.0   -0.1      0.0         0.2
## energy                       -0.1         -0.1    1.0      0.7         0.0
## loudness                      0.0          0.0    0.7      1.0         0.0
## speechiness                   0.0          0.2    0.0      0.0         1.0
## acousticness                  0.1          0.0   -0.5     -0.4         0.0
##                  acousticness
## track_popularity          0.1
## danceability              0.0
## energy                   -0.5
## loudness                 -0.4
## speechiness               0.0
## acousticness              1.0
ggcorrplot(corr, hc.order = TRUE, type = "lower",lab = TRUE)


Conclusion

We can summarize below conclusions from the correlation matrix.

  1. Energy and loudness have a strong positive correlation among themselves. This makes sense as usually energetic songs are loud.
  2. Energy and loud are negatively correlated with acoustiness.
  3. If we look at the target variable - track popularity, we observe that liveness, energy and instrumentalness has negative correlation with track popularity. However, acoustiness has a positive correlation with the track popularity.



Classification

Now we will classify the data into three parts.

  1. Low popularity
  2. Medium popularity
  3. High popularity


For this first we need to divide the data set into three parts depending on the track_popularity range. First let’s have a look at the number of songs corresponding to each track_popularity value.

popularity_spotify_songs_summary <- spotify_songs_data %>% group_by(popularity_of_song = track_popularity) %>%
summarise(No_songs = n()) %>% 
arrange(desc(popularity_of_song))
print(popularity_spotify_songs_summary)
## # A tibble: 101 x 2
##    popularity_of_song No_songs
##                 <int>    <int>
##  1                100        1
##  2                 99        1
##  3                 98        5
##  4                 97        3
##  5                 96        1
##  6                 95        2
##  7                 94        5
##  8                 93        7
##  9                 92        5
## 10                 91       10
## # ... with 91 more rows


There is a very small percentage of songs that have song popularity >90. However, large number of songs have 0 popularity. Let’s plot a box plot to understand where the majority of songs belongs to.

boxplot(spotify_songs_data$track_popularity, xlab="Box plot of Track popularity")

Looking at the box plot and the summary, we can roughly make the following assumptions:

  1. Low Popularity : track_popularity < 30
  2. Medium Popularity: track_popularity between 31-60
  3. High popularity: track_popularity > 61
partition <-c(-1,30,60,100)
classifier_tags<-c('Low','Medium','High')
spotify_songs_data$popularity_class<-cut(spotify_songs_data$track_popularity, breaks=partition, labels=classifier_tags)
str(spotify_songs_data)
## 'data.frame':    28352 obs. of  20 variables:
##  $ track_artist            : chr  "Ed Sheeran" "Maroon 5" "Zara Larsson" "The Chainsmokers" ...
##  $ track_popularity        : int  66 67 70 60 69 67 62 69 68 67 ...
##  $ track_album_release_date: chr  "2019-06-14" "2019-12-13" "2019-07-05" "2019-07-19" ...
##  $ playlist_genre          : Factor w/ 6 levels "edm","latin",..: 3 3 3 3 3 3 3 3 3 3 ...
##  $ playlist_subgenre       : Factor w/ 24 levels "album rock","big room",..: 4 4 4 4 4 4 4 4 4 4 ...
##  $ danceability            : num  0.748 0.726 0.675 0.718 0.65 0.675 0.449 0.542 0.594 0.642 ...
##  $ energy                  : num  0.916 0.815 0.931 0.93 0.833 0.919 0.856 0.903 0.935 0.818 ...
##  $ key                     : Factor w/ 12 levels "0","1","2","3",..: 7 12 2 8 2 9 6 5 9 3 ...
##  $ loudness                : num  -2.63 -4.97 -3.43 -3.78 -4.67 ...
##  $ mode                    : Factor w/ 2 levels "0","1": 2 2 1 2 2 2 1 1 2 2 ...
##  $ speechiness             : num  0.0583 0.0373 0.0742 0.102 0.0359 0.127 0.0623 0.0434 0.0565 0.032 ...
##  $ acousticness            : num  0.102 0.0724 0.0794 0.0287 0.0803 0.0799 0.187 0.0335 0.0249 0.0567 ...
##  $ instrumentalness        : num  0.00 4.21e-03 2.33e-05 9.43e-06 0.00 0.00 0.00 4.83e-06 3.97e-06 0.00 ...
##  $ liveness                : num  0.0653 0.357 0.11 0.204 0.0833 0.143 0.176 0.111 0.637 0.0919 ...
##  $ valence                 : num  0.518 0.693 0.613 0.277 0.725 0.585 0.152 0.367 0.366 0.59 ...
##  $ tempo                   : num  122 100 124 122 124 ...
##  $ duration_ms             : int  194754 162600 176616 169093 189052 163049 187675 207619 193187 253040 ...
##  $ year                    : num  2019 2019 2019 2019 2019 ...
##  $ month                   : num  6 12 7 7 3 7 7 8 6 6 ...
##  $ popularity_class        : Factor w/ 3 levels "Low","Medium",..: 3 3 3 2 3 3 3 3 3 3 ...
##  - attr(*, "na.action")= 'omit' Named int [1:5] 8152 9283 9284 19569 19812
##   ..- attr(*, "names")= chr [1:5] "8152" "9283" "9284" "19569" ...


Now that we have classified the data set into three different parts depending on the popularity let’s plot a bar chart to have an understanding of songs across various classes.

ggplot(data=spotify_songs_data, aes(x=popularity_class, color = popularity_class)) +
  geom_bar(stat="count", fill="white")




Modeling

Multiple regression analysis technique is used when we have to analyze the relationship between a single target variable and serveral predictor variable.

Our goal is to predict track popularity based on various features. This analysis can be used by singers and music companies to make modifications in the song to ensure song gets popular. In order to achieve this goal we will divide our data set into two parts train dataset and test dataset.

n_train <- round(0.8 * nrow(spotify_songs_data))

train_indices <- sample(1:nrow(spotify_songs_data), n_train)

spotify_train_data <- spotify_songs_data[train_indices, ]

spotify_test_data <- spotify_songs_data[-train_indices, ]

Now let’s build our first model for multi linear regression.

Spotify_Recommendation_Model_v1 <- lm(track_popularity ~ danceability + speechiness + acousticness + instrumentalness+loudness+energy
                                   + liveness+ tempo , data = spotify_train_data)

summary(Spotify_Recommendation_Model_v1)
## 
## Call:
## lm(formula = track_popularity ~ danceability + speechiness + 
##     acousticness + instrumentalness + loudness + energy + liveness + 
##     tempo, data = spotify_train_data)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -55.612 -17.509   3.163  18.397  65.593 
## 
## Coefficients:
##                    Estimate Std. Error t value Pr(>|t|)    
## (Intercept)       56.915179   1.781999  31.939  < 2e-16 ***
## danceability       6.393076   1.114280   5.737 9.74e-09 ***
## speechiness       -5.858648   1.543022  -3.797 0.000147 ***
## acousticness       6.176595   0.832394   7.420 1.21e-13 ***
## instrumentalness  -9.878267   0.685509 -14.410  < 2e-16 ***
## loudness           1.240849   0.071552  17.342  < 2e-16 ***
## energy           -22.693754   1.321738 -17.170  < 2e-16 ***
## liveness          -4.154319   1.014844  -4.094 4.26e-05 ***
## tempo              0.031066   0.005897   5.268 1.39e-07 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 23.18 on 22673 degrees of freedom
## Multiple R-squared:  0.04419,    Adjusted R-squared:  0.04385 
## F-statistic:   131 on 8 and 22673 DF,  p-value: < 2.2e-16

Now let’s predict the track popularity of test data using this model and find out residuals.

r_sq_v1 <- summary(Spotify_Recommendation_Model_v1)$r.squared

prediction_Spotify_Recommendation_Model_v1 <- predict(Spotify_Recommendation_Model_v1, newdata = spotify_test_data)

residuals_v1 <- spotify_test_data$track_popularity - prediction_Spotify_Recommendation_Model_v1

rmse_v1 <- sqrt(mean(residuals_v1^2, na.rm=TRUE))

The first step in interpreting this spotify model for regression is to analyse p-value. In our case, p-value is < 2.2e-16 which is highly significant. This tell us that there is atleast one variable which is highly related to the target variable.

We can examine the coefficient table to understand which variable is most significant in this analysis.

summary(Spotify_Recommendation_Model_v1)$coefficient
##                      Estimate  Std. Error    t value      Pr(>|t|)
## (Intercept)       56.91517857 1.781999201  31.938947 5.421502e-219
## danceability       6.39307558 1.114280382   5.737403  9.736349e-09
## speechiness       -5.85864751 1.543022086  -3.796866  1.469189e-04
## acousticness       6.17659475 0.832393868   7.420279  1.209857e-13
## instrumentalness  -9.87826653 0.685508589 -14.410128  7.201107e-47
## loudness           1.24084938 0.071552068  17.341908  6.142486e-67
## energy           -22.69375445 1.321738396 -17.169626  1.166679e-65
## liveness          -4.15431913 1.014843885  -4.093555  4.262805e-05
## tempo              0.03106607 0.005897201   5.267934  1.392270e-07

From the summary we can observe that the coefficient of tempo is not very significant whereas energy has very strong impact on the track popularity. Same goes for loudness(less significant). We will remove these two predictors from our version 2 model and check if it improves the accuracy.

Now let’s build our Second model for multi linear regression.

Spotify_Recommendation_Model_v2 <- lm(track_popularity ~ danceability + speechiness + acousticness + instrumentalness +energy
                                   + liveness , data = spotify_train_data)

summary(Spotify_Recommendation_Model_v2)
## 
## Call:
## lm(formula = track_popularity ~ danceability + speechiness + 
##     acousticness + instrumentalness + energy + liveness, data = spotify_train_data)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -50.597 -17.856   3.151  18.613  60.327 
## 
## Coefficients:
##                  Estimate Std. Error t value Pr(>|t|)    
## (Intercept)       41.6585     1.1461  36.346  < 2e-16 ***
## danceability       6.9557     1.0969   6.341 2.32e-10 ***
## speechiness       -5.0677     1.5496  -3.270  0.00108 ** 
## acousticness       6.1389     0.8372   7.333 2.33e-13 ***
## instrumentalness -12.6407     0.6697 -18.876  < 2e-16 ***
## energy            -7.6860     1.0271  -7.483 7.51e-14 ***
## liveness          -4.9040     1.0211  -4.803 1.58e-06 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 23.35 on 22675 degrees of freedom
## Multiple R-squared:  0.03028,    Adjusted R-squared:  0.03002 
## F-statistic:   118 on 6 and 22675 DF,  p-value: < 2.2e-16

Let’s predict the track popularity of test data using this model 2 and find out residuals.

r_sq_v2 <- summary(Spotify_Recommendation_Model_v2)$r.squared

prediction_Spotify_Recommendation_Model_v2 <- predict(Spotify_Recommendation_Model_v2, newdata = spotify_test_data)

residuals_v2 <- spotify_test_data$track_popularity - prediction_Spotify_Recommendation_Model_v2

rmse_v2 <- sqrt(mean(residuals_v2^2, na.rm=TRUE))

Summarizing the second model

summary(Spotify_Recommendation_Model_v2)$coefficient
##                    Estimate Std. Error    t value      Pr(>|t|)
## (Intercept)       41.658478  1.1461490  36.346477 3.412259e-281
## danceability       6.955656  1.0968560   6.341448  2.319080e-10
## speechiness       -5.067726  1.5496091  -3.270326  1.075840e-03
## acousticness       6.138932  0.8371643   7.333008  2.325954e-13
## instrumentalness -12.640746  0.6696709 -18.876055  7.228455e-79
## energy            -7.685996  1.0270713  -7.483411  7.505344e-14
## liveness          -4.903993  1.0210851  -4.802727  1.575178e-06

Now that we have built these two models, lets compare these to see which one if performing better.

R-square tells us how good data fits for linear regression models. This statistic indicates the percentage of the variance in the dependent variable that the independent variables explain collectively.

Usually the larger the R-square is better the regression model fits our observations.

Root Mean Squared Error (RMSE) is a metrics used to evaluate a Regression Model. These metrics tell us how accurate our predictions are and, what is the amount of deviation from the actual values.

print(paste0("R-squared for first model: ", round(r_sq_v1, 4)," ", "R-squared for Second model: ", round(r_sq_v2, 4)))
## [1] "R-squared for first model: 0.0442 R-squared for Second model: 0.0303"
print(paste0("RMSE for first model: ", round(rmse_v1, 2)," ","RMSE for Second model: " ,  round(rmse_v2, 2)))
## [1] "RMSE for first model: 23.07 RMSE for Second model: 23.29"

As per the definitions of R-square and RMSE we select first model to be more accurate than the second.

Model Evaluation/Validation

Prediction

spotify_test_data$prediction <- predict(Spotify_Recommendation_Model_v1, newdata = spotify_test_data)
ggplot(spotify_test_data, aes(x = prediction, y = track_popularity)) + 
  geom_point(color = "blue", alpha = 0.7) + 
  geom_abline(color = "red") +
  ggtitle("Prediction vs. Real values")

The more linear this graph is better should be the accuracy of the model.

Residuals are the measure of how far from the regression line the Data points are. Fitted values are models prediction of mean response value when you input the values of the predictors, factor levels or components into the model.

We have plotted the following graphs: 1. Residuals Vs Fitted values graph 2. Histogram of residuals

spotify_test_data$prediction <- predict(Spotify_Recommendation_Model_v1, newdata = spotify_test_data)

spotify_test_data$residuals <- spotify_test_data$track_popularity - spotify_test_data$prediction
ggplot(data = spotify_test_data, aes(x = prediction, y = residuals)) +
  geom_pointrange(aes(ymin = 0, ymax = residuals), color = "purple", alpha = 0.7) + geom_hline(yintercept = 0, linetype = 4, color = "red") +
  ggtitle("Residuals vs. Linear model prediction")

ggplot(spotify_test_data, aes(x = residuals)) + 
  geom_histogram(bins = 15, fill = "light blue") +
  ggtitle("Histogram of residuals")

From these two plots we can see that the residuals are not normally distributed and residual vs prediction graph has shown come concentrated and linear model pattern. This shows that we are missing on some coefficient or model is not very accurate.

We will try to fit data using Random forest Model.

library(randomForest)
Spotify_Recommendation_Model_v3 <- randomForest(track_popularity~ danceability + speechiness + acousticness + instrumentalness +energy
                                   + liveness , data = spotify_test_data,mtry=,  ntree=100 )
Spotify_Recommendation_Model_v3
## 
## Call:
##  randomForest(formula = track_popularity ~ danceability + speechiness +      acousticness + instrumentalness + energy + liveness, data = spotify_test_data,      mtry = , ntree = 100) 
##                Type of random forest: regression
##                      Number of trees: 100
## No. of variables tried at each split: 2
## 
##           Mean of squared residuals: 557.9803
##                     % Var explained: 0.45

Here is the summary of Random Forest Model.

summary(Spotify_Recommendation_Model_v3)
##                 Length Class  Mode     
## call               5   -none- call     
## type               1   -none- character
## predicted       5670   -none- numeric  
## mse              100   -none- numeric  
## rsq              100   -none- numeric  
## oob.times       5670   -none- numeric  
## importance         6   -none- numeric  
## importanceSD       0   -none- NULL     
## localImportance    0   -none- NULL     
## proximity          0   -none- NULL     
## ntree              1   -none- numeric  
## mtry               1   -none- numeric  
## forest            11   -none- list     
## coefs              0   -none- NULL     
## y               5670   -none- numeric  
## test               0   -none- NULL     
## inbag              0   -none- NULL     
## terms              3   terms  call

Now we will predict the values using random forest model and find out rmse value.

randomForest_prediction_spotify = predict(Spotify_Recommendation_Model_v3,spotify_test_data )

library(Metrics)
rmse_v3<- rmse(randomForest_prediction_spotify, spotify_test_data$track_popularity)/mean(spotify_test_data$track_popularity)
rmse_v3
## [1] 0.2685903

The model has been improved to a great extent and we got very less rmse value which represents better accuracy. Below is the graph of predicted vs actual data points.

spotify_test_data$prediction_random_forest <- predict(Spotify_Recommendation_Model_v3, newdata = spotify_test_data)
ggplot(spotify_test_data, aes(x = prediction_random_forest, y = track_popularity)) + 
  geom_point(color = "blue", alpha = 0.7) +
  ggtitle("Prediction vs. Real values")

K-Means Clustering

K-means clustering is an algorithm that helps to group the data. K represents the number of groups.

cluster_spotify_data <- spotify_songs_data[, c('instrumentalness', 'danceability' ,'loudness','valence','energy','liveness','tempo', 'speechiness', 'acousticness')]


cluster_spotify_data_v1 <- scale(cluster_spotify_data[, c('instrumentalness', 'danceability' ,'loudness','valence','energy','liveness','tempo', 'speechiness', 'acousticness')])
k_mean_2 <- kmeans(cluster_spotify_data_v1, centers = 2, nstart = 25)
k_mean_3 <- kmeans(cluster_spotify_data_v1, centers = 3, nstart = 25)
k_mean_4 <- kmeans(cluster_spotify_data_v1, centers = 4, nstart = 25)
k_mean_5 <- kmeans(cluster_spotify_data_v1, centers = 5, nstart = 25)
k_mean_6 <- kmeans(cluster_spotify_data_v1, centers = 6, nstart = 25)

Now we need to find optimum value of k. Elbow method is one of the several method in determine the optimum value of k. It relies on the principle that select the value of k from where if we add one more cluster it does not improve the total sum square much.

set.seed(100)
library("factoextra")
fviz_nbclust(cluster_spotify_data[1:1000,], kmeans, method = "wss")

From the above graph we can say that the optimum value of k is 3. Now lets plot this cluster.

plot_k_mean <- fviz_cluster(k_mean_3, geom = "point",  data = cluster_spotify_data_v1) + ggtitle("k = 3")
plot_k_mean

Evaluation of clusters

To check whether the cluster we created is good enough, we can see the value in the cluster. The goodness of the clustering results can be seen from 3 values:

Within Sum of Squares (\(withinss): sum of the distance squared from each observation to the centroid of each cluster. Between Sum of Squares (\)betweenss): the sum of the weighted square distances of each centroid to the global average. Weighted based on the number of observations in the cluster. Total Sum of Squares ($totss): the sum of the distances squared from each observation to the global average. Total within sum of square: the number of withinss for each cluster

  1. Betweenss
k_mean_3$betweenss
## [1] 62979.39
  1. Withinss
k_mean_3$withinss
## [1] 48571.09 68635.25 74973.27
  1. totss
k_mean_3$totss
## [1] 255159
  1. tot.withinss
k_mean_3$tot.withinss
## [1] 192179.6
  1. betweenss/k_mean_3$tot.withinss
k_mean_3$betweenss/k_mean_3$tot.withinss
## [1] 0.3277111