AS6-1 Group4 Daily Kos文章分群

主要議題：依字頻表對文章分群

學習重點：

• 依字頻表對文章分群
• 層級式集群分析：Hierarchical Cluster Analysis
• 依據樹狀圖決定要分多少群
• 依據應用決定要分多少群
• K-Means集群分析：K-Means Cluster Analysis
• 從常見字辭推論文集的主題

rm(list=ls(all=T))
Sys.setlocale("LC_ALL","C")

[1] "C"

options(digits=4, scipen=12)
library(dplyr)

---

1. Hierarchical Clustering

1.1 字頻表、距離矩陣、階層式集群分析

Let’s start by building a hierarchical clustering model. First, read the data set into R. Then, compute the distances (using method=“euclidean”), and use hclust to build the model (using method=“ward.D”). You should cluster on all of the variables.

dailykos = read.csv("data/dailykos.csv")
distance = dist(dailykos,method="euclidean")
clusterdailykos = hclust(distance,method="ward.D")

Running the dist function will probably take you a while. Why? Select all that apply.

• We have a lot of observations, so it takes a long time to compute the distance between each pair of observations.
• We have a lot of variables, so the distance computation is long.

Plot the dendrogram of your hierarchical clustering model.

plot(clusterdailykos)
rect.hclust(clusterdailykos, k=7, border="red")

1.2 從樹狀圖判斷群數

Just looking at the dendrogram,

which of the following seem like good choices for the number of clusters? Select all that apply.

• 2
• 3
• 找垂直距離較大的做切割較合適。

1.3 從應用決定群數

In this problem, we are trying to cluster news articles or blog posts into groups. This can be used to show readers categories to choose from when trying to decide what to read. Just thinking about this application,

what are good choices for the number of clusters? Select all that apply.

• 7
• 8

1.4 依群組分割資料

Let’s pick 7 clusters. This number is reasonable according to the dendrogram, and also seems reasonable for the application. Use the cutree function to split your data into 7 clusters.

hierGroups = cutree(clusterdailykos, k = 7)
table(hierGroups) %>% sort(T)

hierGroups
   1    6    5    3    2    7    4 
1266  714  407  374  321  209  139 

Now, we don’t really want to run tapply on every single variable when we have over 1,000 different variables. Let’s instead use the subset function to subset our data by cluster. Create 7 new datasets, each containing the observations from one of the clusters.

How many observations are in cluster 3?

table(hierGroups)

hierGroups
   1    2    3    4    5    6    7 
1266  321  374  139  407  714  209 

• 374

Which cluster has the most observations?

• 1

Which cluster has the fewest observations?

• 4

1.5 找出第一族群中最常見的字辭

Instead of looking at the average value in each variable individually, we’ll just look at the top 6 words in each cluster. To do this for cluster 1, type the following in your R console (where “HierCluster1” should be replaced with the name of your first cluster subset):

tail(sort(colMeans(HierCluster1)))

This computes the mean frequency values of each of the words in cluster 1, and then outputs the 6 words that occur the most frequently. The colMeans function computes the column (word) means, the sort function orders the words in increasing order of the mean values, and the tail function outputs the last 6 words listed, which are the ones with the largest column means.

What is the most frequent word in this cluster, in terms of average value? Enter the word exactly how you see it in the output:

HierCluster1 = subset(dailykos, hierGroups == 1)
HierCluster2 = subset(dailykos, hierGroups == 2)
HierCluster3 = subset(dailykos, hierGroups == 3)
HierCluster4 = subset(dailykos, hierGroups == 4)
HierCluster5 = subset(dailykos, hierGroups == 5)
HierCluster6 = subset(dailykos, hierGroups == 6)
HierCluster7 = subset(dailykos, hierGroups == 7)
tail(sort(colMeans(HierCluster1),F))

     state republican       poll   democrat      kerry       bush 
    0.7575     0.7591     0.9036     0.9194     1.0624     1.7054 

• bush

1.6 找出各族群中最常見的字辭

Now repeat the command given in the previous problem for each of the other clusters, and answer the following questions.

tail(sort(colMeans(HierCluster2),F))

     bush  democrat challenge      vote      poll  november 
    2.847     2.850     4.097     4.399     4.847    10.340 

tapply(dailykos$iraq , hierGroups, mean) %>% sort(T)

     5      3      4      1      2      7      6 
2.4275 0.8182 0.6835 0.5166 0.4237 0.1388 0.1303 

Which words best describe cluster 2?

• november, poll, vote ,challenge

Which cluster could best be described as the cluster related to the Iraq war?

• 5

In 2004, one of the candidates for the Democratic nomination for the President of the United States was Howard Dean, John Kerry was the candidate who won the democratic nomination, and John Edwards with the running mate of John Kerry (the Vice President nominee). Given this information,

tail(sort(colMeans(HierCluster2)))

     bush  democrat challenge      vote      poll  november 
    2.847     2.850     4.097     4.399     4.847    10.340 

tail(sort(colMeans(HierCluster3)))

     elect    parties      state republican   democrat       bush 
     1.647      1.666      2.321      2.524      3.824      4.406 

tail(sort(colMeans(HierCluster4)))

campaign    voter presided     poll     bush    kerry 
   1.432    1.540    1.626    3.590    7.835    8.439 

tail(sort(colMeans(HierCluster5)))

      american       presided administration            war           iraq           bush 
         1.091          1.120          1.231          1.776          2.428          3.941 

tail(sort(colMeans(HierCluster6)))

    race     bush    kerry    elect democrat     poll 
  0.4580   0.4888   0.5168   0.5350   0.5644   0.5812 

tail(sort(colMeans(HierCluster7)))

democrat    clark   edward     poll    kerry     dean 
   2.148    2.498    2.608    2.766    3.952    5.804 

which cluster best corresponds to the democratic party?

• 7

---

2 K-Means Clustering

2.1 K-Means集群分析

Now, run k-means clustering, setting the seed to 1000 right before you run the kmeans function. Again, pick the number of clusters equal to 7. You don’t need to add the iters.max argument.

set.seed(1000)
KMC = kmeans(dailykos, centers = 7)
table(KMC$cluster) %>% sort(T)

   4    6    7    3    5    1    2 
2063  329  308  277  163  146  144 

Subset your data into the 7 clusters (7 new datasets) by using the “cluster” variable of your kmeans output.

How many observations are in Cluster 3?

• 277

Which cluster has the most observations?

• 4

Which cluster has the fewest number of observations?

• 2

2.2 找出各族群中最常見的字辭

Now, output the six most frequent words in each cluster, like we did in the previous problem, for each of the k-means clusters.

dailykos$cluster = KMC$cluster
dailykos = as.data.frame(dailykos)
sapply(split(dailykos[,1:1545], dailykos$cluster),function(x) names(head(sort(colMeans(x),T)))
  )

     1                2           3                4            5            6           7         
[1,] "bush"           "dean"      "iraq"           "bush"       "democrat"   "november"  "kerry"   
[2,] "presided"       "kerry"     "war"            "democrat"   "republican" "poll"      "bush"    
[3,] "administration" "clark"     "bush"           "poll"       "parties"    "vote"      "poll"    
[4,] "kerry"          "edward"    "american"       "kerry"      "state"      "challenge" "campaign"
[5,] "iraq"           "democrat"  "iraqi"          "republican" "senate"     "bush"      "voter"   
[6,] "state"          "primaries" "administration" "elect"      "race"       "democrat"  "presided"

Which k-means cluster best corresponds to the Iraq War?

• 3

Which k-means cluster best corresponds to the democratic party? (Remember that we are looking for the names of the key democratic party leaders.)

• 2

2.3 ~ 2.6 兩種分群結果之間的對應關係

For the rest of this problem, we’ll ask you to compare how observations were assigned to clusters in the two different methods. Use the table function to compare the cluster assignment of hierarchical clustering to the cluster assignment of k-means clustering.

KmeansCluster1 = subset(dailykos, KMC$cluster == 1)
KmeansCluster2 = subset(dailykos, KMC$cluster == 2)
KmeansCluster3 = subset(dailykos, KMC$cluster == 3)
KmeansCluster4 = subset(dailykos, KMC$cluster == 4)
KmeansCluster5 = subset(dailykos, KMC$cluster == 5)
KmeansCluster6 = subset(dailykos, KMC$cluster == 6)
KmeansCluster7 = subset(dailykos, KMC$cluster == 7)
tail(sort(colMeans(KmeansCluster1)))

         state           iraq          kerry administration       presided           bush 
         1.610          1.616          1.637          2.664          2.767         11.432 

tail(sort(colMeans(KmeansCluster2)))

primaries  democrat    edward     clark     kerry      dean 
    2.319     2.694     2.799     3.090     4.979     8.278 

tail(sort(colMeans(KmeansCluster3)))

   iraqi american     bush  cluster      war     iraq 
   1.610    1.686    2.610    3.000    3.025    4.094 

tail(sort(colMeans(KmeansCluster4)))

republican      kerry       poll   democrat       bush    cluster 
    0.6175     0.6495     0.7475     0.7891     1.1474     4.0000 

tail(sort(colMeans(KmeansCluster5)))

    senate      state    parties republican    cluster   democrat 
     2.650      3.521      3.620      4.638      5.000      6.994 

tail(sort(colMeans(KmeansCluster6)))

     bush challenge      vote      poll   cluster  november 
    2.960     4.122     4.447     4.872     6.000    10.371 

tail(sort(colMeans(KmeansCluster7)))

   voter campaign     poll     bush    kerry  cluster 
   1.334    1.383    2.789    5.971    6.481    7.000 

table(hier = hierGroups,KMC = KMC$cluster)

    KMC
hier    1    2    3    4    5    6    7
   1    3   11   64 1045   32    0  111
   2    0    0    0    0    0  320    1
   3   85   10   42   79  126    8   24
   4   10    5    0    0    1    0  123
   5   48    0  171  145    3    1   39
   6    0    2    0  712    0    0    0
   7    0  116    0   82    1    0   10

Which Hierarchical Cluster best corresponds to K-Means Cluster 2?

• 7

Which Hierarchical Cluster best corresponds to K-Means Cluster 3?

• 5

Which Hierarchical Cluster best corresponds to K-Means Cluster 7?

• No Hierarchical Cluster contains at least half of the points in K-Means Cluster 7.

Which Hierarchical Cluster best corresponds to K-Means Cluster 6?

• 2

【討論問題】

字頻表是什麼？它的資料格式？

• 字頻表是對一些特定的單字計算在某些文章或是其他出處出現的次數。
• 在row是資料的出處，可能是某篇文章，或是某天的報章雜誌；在colum是特定的文字，依據想了解的文字進行分析，可多可少。

使用字頻表作集群分析時，區隔變數是什麼？

• 文字的不同(種類)，字的數量是區隔變數的值。
• 去除stop words後，出現頻率高的文字欄位當作選取的文字。

從樹狀圖判斷群數和從應用需求決定群數有什麼差別？

• 樹狀圖會依「群內距離小，群間距離大」的準則切割，因此會從樹狀圖的垂直距離大的地方切割，會依數據的不同可能會有不同的切割數。
• 樹狀圖的最適當切割數會過少，但實際應用切割過少可能導致資料分群不夠精細，因此技術上或應用上會造成不同的取捨。

---