AS3-1 Popularity of music records

rm(list=ls(all=T))
options(digits=4, scipen=12)
library(dplyr); library(ggplot2)

---

Introduction

議題：使用歌曲的屬性，預測它會不會進入流行歌曲排行榜的前10名

學習重點：

• 依時間分割資料
• model formula 的寫法
• 高相關(共線性)自變數之間的選擇
• accuracy, sensitivity, specificity的實際意義
• 如何調整臨界機率來權衡：TFR/sensitivity vs. FPR/specificity

---

1 基本的資料處理 Understanding the Data

【1.1】How many observations (songs) are from the year 2010?

setwd("D:/Data")

The working directory was changed to D:/Data inside a notebook chunk. The working directory will be reset when the chunk is finished running. Use the knitr root.dir option in the setup chunk to change the working directory for notebook chunks.

songs<-read.csv("Unit3/songs.csv",header = TRUE,sep = ",",stringsAsFactors = F)
sum(songs$year=="2010")

[1] 373

【1.2】How many songs does the dataset include for which the artist name is “Michael Jackson”?

sum(songs$artistname=="Michael Jackson")

[1] 18

【1.3】Which of these songs by Michael Jackson made it to the Top 10? Select all that apply.

subset(songs,artistname=="Michael Jackson" & Top10==1)

#You Rock My World ;You Are Not Alone ;Black or White;Remember the Time;In The Closet

【1.4】(a) What are the values of timesignature that occur in our dataset? (b) Which timesignature value is the most frequent among songs in our dataset?

table(songs$timesignature)

   0    1    3    4    5    7 
  10  143  503 6787  112   19 

#013457
#4

【1.5】 Which of the following songs has the highest tempo?

max(songs$tempo)

[1] 244.307

subset(songs,tempo==244.307)

---

2 建立模型 Creating Our Prediction Model

【2.1 依時間分割資料】How many observations (songs) are in the training set?

SongsTrain=subset(songs,year<=2009)
SongsTest=subset(songs,year>2009)
nrow(SongsTrain)

[1] 7201

【2.2 建立模型、模型摘要】What is the value of the Akaike Information Criterion (AIC)?

In strsplit(code, "\n", fixed = TRUE) :
  input string 1 is invalid in this locale

SongsTrain = SongsTrain[ , !(names(SongsTrain) %in% nonvars) ]
SongsTest = SongsTest[ , !(names(SongsTest) %in% nonvars) ]
SongsLog1 = glm(Top10 ~ ., data=SongsTrain, family=binomial)
summary(SongsLog1)

Call:
glm(formula = Top10 ~ ., family = binomial, data = SongsTrain)

Deviance Residuals: 
    Min       1Q   Median       3Q      Max  
-1.9220  -0.5399  -0.3459  -0.1845   3.0770  

Coefficients:
                           Estimate Std. Error z value Pr(>|z|)    
(Intercept)               1.470e+01  1.806e+00   8.138 4.03e-16 ***
timesignature             1.264e-01  8.674e-02   1.457 0.145050    
timesignature_confidence  7.450e-01  1.953e-01   3.815 0.000136 ***
loudness                  2.999e-01  2.917e-02  10.282  < 2e-16 ***
tempo                     3.634e-04  1.691e-03   0.215 0.829889    
tempo_confidence          4.732e-01  1.422e-01   3.329 0.000873 ***
key                       1.588e-02  1.039e-02   1.529 0.126349    
key_confidence            3.087e-01  1.412e-01   2.187 0.028760 *  
energy                   -1.502e+00  3.099e-01  -4.847 1.25e-06 ***
pitch                    -4.491e+01  6.835e+00  -6.570 5.02e-11 ***
timbre_0_min              2.316e-02  4.256e-03   5.441 5.29e-08 ***
timbre_0_max             -3.310e-01  2.569e-02 -12.882  < 2e-16 ***
timbre_1_min              5.881e-03  7.798e-04   7.542 4.64e-14 ***
timbre_1_max             -2.449e-04  7.152e-04  -0.342 0.732087    
timbre_2_min             -2.127e-03  1.126e-03  -1.889 0.058843 .  
timbre_2_max              6.586e-04  9.066e-04   0.726 0.467571    
timbre_3_min              6.920e-04  5.985e-04   1.156 0.247583    
timbre_3_max             -2.967e-03  5.815e-04  -5.103 3.34e-07 ***
timbre_4_min              1.040e-02  1.985e-03   5.237 1.63e-07 ***
timbre_4_max              6.110e-03  1.550e-03   3.942 8.10e-05 ***
timbre_5_min             -5.598e-03  1.277e-03  -4.385 1.16e-05 ***
timbre_5_max              7.736e-05  7.935e-04   0.097 0.922337    
timbre_6_min             -1.686e-02  2.264e-03  -7.445 9.66e-14 ***
timbre_6_max              3.668e-03  2.190e-03   1.675 0.093875 .  
timbre_7_min             -4.549e-03  1.781e-03  -2.554 0.010661 *  
timbre_7_max             -3.774e-03  1.832e-03  -2.060 0.039408 *  
timbre_8_min              3.911e-03  2.851e-03   1.372 0.170123    
timbre_8_max              4.011e-03  3.003e-03   1.336 0.181620    
timbre_9_min              1.367e-03  2.998e-03   0.456 0.648356    
timbre_9_max              1.603e-03  2.434e-03   0.659 0.510188    
timbre_10_min             4.126e-03  1.839e-03   2.244 0.024852 *  
timbre_10_max             5.825e-03  1.769e-03   3.292 0.000995 ***
timbre_11_min            -2.625e-02  3.693e-03  -7.108 1.18e-12 ***
timbre_11_max             1.967e-02  3.385e-03   5.811 6.21e-09 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

(Dispersion parameter for binomial family taken to be 1)

    Null deviance: 6017.5  on 7200  degrees of freedom
Residual deviance: 4759.2  on 7167  degrees of freedom
AIC: 4827.2

Number of Fisher Scoring iterations: 6

#4827.2

【2.3 模型係數判讀】The LOWER or HIGHER our confidence about time signature, key and tempo, the more likely the song is to be in the Top 10

#The higher our confidence about time signature, key and tempo, the more likely the song is to be in the Top 10

【2.4 進行推論】What does Model 1 suggest in terms of complexity?

#Mainstream listeners tend to prefer less complex songs

【2.5 檢查異常係數】 (a) By inspecting the coefficient of the variable “loudness”, what does Model 1 suggest? (b) By inspecting the coefficient of the variable “energy”, do we draw the same conclusions as above?

#Mainstream listeners prefer songs with heavy instrumentation 
#no

---

3 處理共線性 Beware of Multicollinearity Issues!

【3.1 檢查相關係數】What is the correlation between loudness and energy in the training set?

cor(SongTrain)

Error in is.data.frame(x) : object 'SongTrain' not found

【3.2 重新建立模型、檢查係數】Look at the summary of SongsLog2, and inspect the coefficient of the variable “energy”. What do you observe?

SongsLog2 = glm(Top10 ~ . - loudness, data=SongsTrain, family=binomial)
summary(SongsLog2)

Call:
glm(formula = Top10 ~ . - loudness, family = binomial, data = SongsTrain)

Deviance Residuals: 
    Min       1Q   Median       3Q      Max  
-2.0983  -0.5607  -0.3602  -0.1902   3.3107  

Coefficients:
                           Estimate Std. Error z value Pr(>|z|)    
(Intercept)              -2.241e+00  7.465e-01  -3.002 0.002686 ** 
timesignature             1.625e-01  8.734e-02   1.860 0.062873 .  
timesignature_confidence  6.885e-01  1.924e-01   3.578 0.000346 ***
tempo                     5.521e-04  1.665e-03   0.332 0.740226    
tempo_confidence          5.497e-01  1.407e-01   3.906 9.40e-05 ***
key                       1.740e-02  1.026e-02   1.697 0.089740 .  
key_confidence            2.954e-01  1.394e-01   2.118 0.034163 *  
energy                    1.813e-01  2.608e-01   0.695 0.486991    
pitch                    -5.150e+01  6.857e+00  -7.511 5.87e-14 ***
timbre_0_min              2.479e-02  4.240e-03   5.847 5.01e-09 ***
timbre_0_max             -1.007e-01  1.178e-02  -8.551  < 2e-16 ***
timbre_1_min              7.143e-03  7.710e-04   9.265  < 2e-16 ***
timbre_1_max             -7.830e-04  7.064e-04  -1.108 0.267650    
timbre_2_min             -1.579e-03  1.109e-03  -1.424 0.154531    
timbre_2_max              3.889e-04  8.964e-04   0.434 0.664427    
timbre_3_min              6.500e-04  5.949e-04   1.093 0.274524    
timbre_3_max             -2.462e-03  5.674e-04  -4.339 1.43e-05 ***
timbre_4_min              9.115e-03  1.952e-03   4.670 3.02e-06 ***
timbre_4_max              6.306e-03  1.532e-03   4.115 3.87e-05 ***
timbre_5_min             -5.641e-03  1.255e-03  -4.495 6.95e-06 ***
timbre_5_max              6.937e-04  7.807e-04   0.889 0.374256    
timbre_6_min             -1.612e-02  2.235e-03  -7.214 5.45e-13 ***
timbre_6_max              3.814e-03  2.157e-03   1.768 0.076982 .  
timbre_7_min             -5.102e-03  1.755e-03  -2.907 0.003644 ** 
timbre_7_max             -3.158e-03  1.811e-03  -1.744 0.081090 .  
timbre_8_min              4.488e-03  2.810e-03   1.597 0.110254    
timbre_8_max              6.423e-03  2.950e-03   2.177 0.029497 *  
timbre_9_min             -4.282e-04  2.955e-03  -0.145 0.884792    
timbre_9_max              3.525e-03  2.377e-03   1.483 0.138017    
timbre_10_min             2.993e-03  1.804e-03   1.660 0.097004 .  
timbre_10_max             7.367e-03  1.731e-03   4.255 2.09e-05 ***
timbre_11_min            -2.837e-02  3.630e-03  -7.815 5.48e-15 ***
timbre_11_max             1.829e-02  3.341e-03   5.476 4.34e-08 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

(Dispersion parameter for binomial family taken to be 1)

    Null deviance: 6017.5  on 7200  degrees of freedom
Residual deviance: 4871.8  on 7168  degrees of freedom
AIC: 4937.8

Number of Fisher Scoring iterations: 6

#Model 2 suggests that songs with high energy levels tend to be more popular. This contradicts our observation in Model 1

【3.3 選擇模型】 do we make the same observation about the popularity of heavy instrumentation as we did with Model 2?

SongsLog3 = glm(Top10 ~ . - energy, data=SongsTrain, family=binomial)
summary(SongsLog3)

#yes

---

4 驗證模型 Validating Our Model

【4.1 正確性】What is the accuracy of Model 3 on the test set, using a threshold of 0.45?

pre  = predict(SongsLog3,newdata=SongsTest,type="response")
table(SongsTest$Top10,pre>=0.45)

   
    FALSE TRUE
  0   309    5
  1    40   19

(309+19)/(309+40+5+19)

[1] 0.8793566

【4.2 底線正確率】What would the accuracy of the baseline model be on the test set? ?

table(SongsTest$Top10)

  0   1 
314  59 

314/(314+59)

[1] 0.8418231

【4.3 正確性 vs. 辨識率】How many songs does Model 3 correctly predict as Top 10 hits in 2010? How many non-hit songs does Model 3 predict will be Top 10 hits?

table(SongsTest$Top10, pre >= 0.45)

   
    FALSE TRUE
  0   309    5
  1    40   19

#19
#5

【Q】不能大幅度增加正確性的模型也會有用嗎？為甚麼？

以不同投資觀點來看，製作公司會願意投資有可能進入Top10的歌曲，然而公司的目標則是盡可能減少因投資失敗而面臨的財務風險，因此增加正確性的模型對於兩方來說是有用處的

【4.4 敏感性 & 明確性】What is the sensitivity and specificity of Model 3 on the test set, using a threshold of 0.45?

19/(19+40)

[1] 0.3220339

309/(309+5)

[1] 0.9840764

【4.5 結論】What conclusions can you make about our model?

#Model 3 favors specificity over sensitivity. 
#Model 3 provides conservative predictions, and predicts that a song will make it to the Top 10 very rarely. So while it detects less than half of the Top 10 songs, we can be very confident in the songs that it does predict to be Top 10 hits.

【Q】從這個結論我們學到什麼？

一開始建模的時候必須要刪掉一些不必要(不會對模型有影響)的變數，像是年份、歌曲名稱。建好模型之後就可以依照係數推論出要如果歌曲要進到Top10的話，自變數對於歌曲的影響。

再來model1發現出loudness和energy的共線性問題，因此為了排除這兩個變數，又額外建立兩個model，解決共線性問題。

最後驗證我們的模型後，在拿Baseline model做比較，發現有確實改善預測正確性的問題。並且檢測模型的sensitivity和specificity之後，specificity的值較高，代表他們幾乎都是預估歌曲不會進Top10，如果真的進的話代表模型真的是有用的

---