class: middle background-image: url(data:image/png;base64,#LTU_logo.jpg) background-position: top left background-size: 30% # STM1001 Lecture # Introduction to Cluster Analysis ## Science/Health and Data Science streams ### La Trobe University --- # Welcome! ### In this lecture we will cover an Introduction to Cluster Analysis, focusing on how to conduct cluster analyses in R/jamovi. -- * By the end of this lecture you will: -- * understand what cluster analysis is and why we use it -- * have a foundational understanding of `\(k\)`-means clustering -- We will practice cluster analyses in Computer Lab 7B, and the computer lab has some additional extension material if you would like to extend your knowledge. -- *It is recommended that you cover this lecture material before you start Computer Lab 7B.* --- # Overview Over the following slides, we will cover: -- * Motivation for using Clustering -- * What is Clustering? -- * Types of Clustering Methods -- * `\(k\)`-means Clustering -- * Extensions -- Throughout this lecture, we will see how clustering works via an example. --- # 1. Motivation for Clustering We have looked at a range of data sets so far in STM1001. These have covered a variety of variables, from penguins to GDP per capita to onion yield. -- For all of these data sets, for each individual (penguin, country, onion) we have observations which, broadly speaking, fall into one of two groups: -- * A set of **feature** variables (e.g. flipper length, happiness score, onion density etc) -- * A .teal_style[group label] variable (e.g. species, continent, planting location etc) --- # 1. Motivation for Clustering For our analyses of these data, we used the .teal_style[group label] to group together similar **feature variable** observations, e.g.: * Assess **flipper lengths** across different penguin .teal_style[Species] -- * Assess **yield of onions** grown across different .teal_style[Localities] -- * Assess **Extraversion** across different US .teal_style[Regions] -- What happens though if we have data with feature variables, but no group label? -- * How can we determine the group in which the observations should be placed? -- * What if we don't even know what groups exist, or how many there are? -- We will need a new statistical technique to help us with our analyses - clustering! --- # 1. Motivation for Clustering * For example, let us again consider the Big Five data set (Hartnett, 2020) * Recall that the Big Five personality traits are: * **E**xtraversion * **A**greeableness * **C**onscientiousness * **N**euroticism * **O**penness --- # 1. Motivation for Clustering * Suppose now that we do not have access to any grouping variables - all we have access to are the Big Five measurements: <div class="datatables html-widget html-fill-item" id="htmlwidget-d77368a75529d764b02d" style="width:100%;height:auto;"></div> <script type="application/json" data-for="htmlwidget-d77368a75529d764b02d">{"x":{"filter":"none","vertical":false,"data":[["1","2","3","4","5","6","7","8","9","10","11","12","13","14","15","16","17","18","19","20","21","22","23","24","25","26","27","28","29","30","31","32","33","34","35","36","37","38","39","40","41","42","43","44","45","46","47","48","49"],[55.5,50.6,49.9,51.4,45.3,57.6,47,64.8,60.9,63.2,40.7,62.5,48.9,62.8,45.5,53.4,52.2,44.2,35.2,44.4,55.2,52.9,56.8,62.9,33.1,60,46.4,40.2,59.9,32.4,47,51,52.4,54.6,39.7,30.9,54.6,43.3,60,58.7,51.3,55.2,55.8,26.5,48.5,30.6,38.5,69.8,46],[52.7,46.6,52.7,49,47.5,38.6,38.8,21.4,50.7,60,52.9,48.3,50.2,56.6,48.9,48.1,49.7,32.8,37.3,40.7,54.7,61.6,63.3,59.3,52.3,62.9,31.8,53.5,44.6,45.4,29.8,63.6,52.4,45.9,54.3,59.1,42.8,35.2,55.4,56.7,66.59999999999999,52.3,69.40000000000001,60,47.4,55.8,51.8,57.8,40.7],[55.5,58.4,41,43.2,58.8,34.2,36.5,44.1,62.7,68.8,44.5,50.9,56.2,52.2,50.8,51.3,45,24,37.5,32.2,53,52.5,59.7,60.8,56.1,64.3,55.8,38,40.8,58.5,37.7,68.40000000000001,51.4,46.5,54.7,45.8,52.4,48.5,69.59999999999999,55.8,62,54.4,54.5,38.2,45.3,45,42.6,47.6,42.4],[48.7,38.1,56.2,39.1,34.3,53.4,62.4,41.6,40.8,38,44.2,51.2,59.3,49.1,49,62.5,60.4,71,49.4,63.8,48.6,43.4,52,48.3,43,41.6,44,61.8,56.4,51.6,62.7,44.8,49.6,58.5,52.1,39.5,61.4,61.9,45.7,36.1,49,44.3,30.4,55.8,44.7,36.9,79.2,48.6,46.1],[42.7,54.7,40.3,65,57.9,53.9,42.7,77.5,61,56.9,44.7,55.2,44.9,33.7,40.1,43,53.7,50.8,56.6,59.6,43.4,38.5,46.2,45.7,55,34.3,61.3,48.7,57.6,62,64.5,49.6,21.8,46,42.2,58.8,49.6,59.4,55.3,41.9,50.1,52.7,47.7,54.2,57.1,56.6,36.8,35.7,42.4]],"container":"<table class=\"display\">\n <thead>\n <tr>\n <th> <\/th>\n <th>E<\/th>\n <th>A<\/th>\n <th>C<\/th>\n <th>N<\/th>\n <th>O<\/th>\n <\/tr>\n <\/thead>\n<\/table>","options":{"pageLength":5,"columnDefs":[{"className":"dt-right","targets":[1,2,3,4,5]},{"orderable":false,"targets":0},{"name":" ","targets":0},{"name":"E","targets":1},{"name":"A","targets":2},{"name":"C","targets":3},{"name":"N","targets":4},{"name":"O","targets":5}],"order":[],"autoWidth":false,"orderClasses":false,"lengthMenu":[5,10,25,50,100]}},"evals":[],"jsHooks":[]}</script> --- # 1. Motivation for Clustering * Can we use just this information to help us identify distinct groups, or "clusters"? -- * We will see how Clustering analysis can help us do just that --- # 2. What is Clustering? Clusters are reasonably intuitive, and are easy to understand visually. -- * As an example, here we have used feature variables from our `Big Five` data set to cluster the data into two clusters: <img src="data:image/png;base64,#STM1001_Data_Science_Clustering_Lecture_files/figure-html/unnamed-chunk-2-1.svg" width="55%" style="display: block; margin: auto;" /> --- # Purposes of Clustering Clustering is often used as part of an exploratory data analysis procedure, to uncover hidden patterns in a new data set. -- Typically, the main purposes of clustering are to: -- * Analyse the data structure -- * Relate the different elements of the data to each other -- and most importantly -- * Aid in classifying the data into certain groups or classes * Note that in general, we may not be aware of what these classes are, prior to the clustering analysis. -- **Note that clustering is usually not the final part of our analysis.** --- # How to define a Cluster? We have seen that clusters can be relatively easy to identify visually. -- Clusters can however be difficult to define mathematically. We will avoid overly technical definitions, and use the following intuitive notion of clustering: -- * A cluster is a set of entities which are alike, and entities from different clusters are not alike (Everitt, 1974). -- Phrased slightly differently: -- * A cluster is a subset of a data set that consists of observations which, using a chosen metric, are similar to each other, and also dissimilar to other observations (Mirkin, 1996). --- # Types of Clustering Methods There are numerous clustering algorithms (methods), but no single clustering method is universally preferred over all others. -- What this means is that the choice of 'best' clustering method is context specific - different clustering methods can be better for different problems. -- It is important to be able to use different types of clustering methods. In STM1001, we will look at: -- * .teal_style[k-means clustering] -- * .teal_style[Hierarchical clustering] (extension) -- * .teal_style[PAM clustering] and .teal_style[Fuzzy clustering] (extension for Data Science students only) -- **Don’t worry**, we won’t go into all the complicated mathematics involved in these techniques - our focus is on learning how to conduct and interpret cluster analyses in R and jamovi. --- # Conducting Cluster Analyses in jamovi and RStudio In Computer Lab 7B, you will learn how to conduct your own cluster analyses. -- * We will need certain modules and packages for cluster analysis: -- * In **jamovi** (The jamovi project 2022), we can use the `snowCluster` module (Seol 2022; Kassambara and Mundt 2020). -- * In **RStudio** (Posit team 2023), we can use the `cluster` package (R Core Team 2021), along with the `factoextra` (Kassambara and Mundt 2020) and `ggfortify` (Horikoshi et al. 2021) packages. --- # Data Preparations Regardless of software, we must properly prepare our data, before we begin our cluster analyses. -- **Important: Clustering algorithms only work on continuous feature variables.** -- Let's take a look at the Big Five data as an example: <div class="datatables html-widget html-fill-item" id="htmlwidget-f815ed988e5f8f608be0" style="width:100%;height:auto;"></div> <script type="application/json" data-for="htmlwidget-f815ed988e5f8f608be0">{"x":{"filter":"none","vertical":false,"data":[["1","2","3","4","5","6","7","8","9","10","11","12","13","14","15","16","17","18","19","20","21","22","23","24","25","26","27","28","29","30","31","32","33","34","35","36","37","38","39","40","41","42","43","44","45","46","47","48","49"],["Alabama","Arizona","Arkansas","California","Colorado","Connecticut","Delaware","District of Columbia","Florida","Georgia","Idaho","Illinois","Indiana","Iowa","Kansas","Kentucky","Louisiana","Maine","Maryland","Massachusetts","Michigan","Minnesota","Mississippi","Missouri","Montana","Nebraska","Nevada","New Hampshire","New Jersey","New Mexico","New York","North Carolina","North Dakota","Ohio","Oklahoma","Oregon","Pennsylvania","Rhode Island","South Carolina","South Dakota","Tennessee","Texas","Utah","Vermont","Virginia","Washington","West Virginia","Wisconsin","Wyoming"],[55.5,50.6,49.9,51.4,45.3,57.6,47,64.8,60.9,63.2,40.7,62.5,48.9,62.8,45.5,53.4,52.2,44.2,35.2,44.4,55.2,52.9,56.8,62.9,33.1,60,46.4,40.2,59.9,32.4,47,51,52.4,54.6,39.7,30.9,54.6,43.3,60,58.7,51.3,55.2,55.8,26.5,48.5,30.6,38.5,69.8,46],[52.7,46.6,52.7,49,47.5,38.6,38.8,21.4,50.7,60,52.9,48.3,50.2,56.6,48.9,48.1,49.7,32.8,37.3,40.7,54.7,61.6,63.3,59.3,52.3,62.9,31.8,53.5,44.6,45.4,29.8,63.6,52.4,45.9,54.3,59.1,42.8,35.2,55.4,56.7,66.59999999999999,52.3,69.40000000000001,60,47.4,55.8,51.8,57.8,40.7],[55.5,58.4,41,43.2,58.8,34.2,36.5,44.1,62.7,68.8,44.5,50.9,56.2,52.2,50.8,51.3,45,24,37.5,32.2,53,52.5,59.7,60.8,56.1,64.3,55.8,38,40.8,58.5,37.7,68.40000000000001,51.4,46.5,54.7,45.8,52.4,48.5,69.59999999999999,55.8,62,54.4,54.5,38.2,45.3,45,42.6,47.6,42.4],[48.7,38.1,56.2,39.1,34.3,53.4,62.4,41.6,40.8,38,44.2,51.2,59.3,49.1,49,62.5,60.4,71,49.4,63.8,48.6,43.4,52,48.3,43,41.6,44,61.8,56.4,51.6,62.7,44.8,49.6,58.5,52.1,39.5,61.4,61.9,45.7,36.1,49,44.3,30.4,55.8,44.7,36.9,79.2,48.6,46.1],[42.7,54.7,40.3,65,57.9,53.9,42.7,77.5,61,56.9,44.7,55.2,44.9,33.7,40.1,43,53.7,50.8,56.6,59.6,43.4,38.5,46.2,45.7,55,34.3,61.3,48.7,57.6,62,64.5,49.6,21.8,46,42.2,58.8,49.6,59.4,55.3,41.9,50.1,52.7,47.7,54.2,57.1,56.6,36.8,35.7,42.4],["S","W","S","W","W","NE","S","S","S","S","W","MW","MW","MW","MW","S","S","NE","S","NE","MW","MW","S","MW","W","MW","W","NE","NE","W","NE","S","MW","MW","W","W","NE","NE","S","MW","S","S","W","NE","S","W","S","MW","W"]],"container":"<table class=\"display\">\n <thead>\n <tr>\n <th> <\/th>\n <th>State<\/th>\n <th>E<\/th>\n <th>A<\/th>\n <th>C<\/th>\n <th>N<\/th>\n <th>O<\/th>\n <th>Region<\/th>\n <\/tr>\n <\/thead>\n<\/table>","options":{"pageLength":4,"columnDefs":[{"className":"dt-right","targets":[2,3,4,5,6]},{"orderable":false,"targets":0},{"name":" ","targets":0},{"name":"State","targets":1},{"name":"E","targets":2},{"name":"A","targets":3},{"name":"C","targets":4},{"name":"N","targets":5},{"name":"O","targets":6},{"name":"Region","targets":7}],"order":[],"autoWidth":false,"orderClasses":false,"lengthMenu":[4,10,25,50,100]}},"evals":[],"jsHooks":[]}</script> --- # Data Preparations * Here, only `E`, `A`, `C`, `N` and `O` are continuous numeric variables. -- * We won't be able to use the other variables as feature variables (unless we conduct some transformations, which are beyond the scope of this topic). -- * So the data we will use to input into the clustering algorithm will be limited to the numeric variables: <div class="datatables html-widget html-fill-item" id="htmlwidget-8efcff402af4297aa856" style="width:100%;height:auto;"></div> <script type="application/json" data-for="htmlwidget-8efcff402af4297aa856">{"x":{"filter":"none","vertical":false,"data":[["1","2","3","4","5","6","7","8","9","10","11","12","13","14","15","16","17","18","19","20","21","22","23","24","25","26","27","28","29","30","31","32","33","34","35","36","37","38","39","40","41","42","43","44","45","46","47","48","49"],[55.5,50.6,49.9,51.4,45.3,57.6,47,64.8,60.9,63.2,40.7,62.5,48.9,62.8,45.5,53.4,52.2,44.2,35.2,44.4,55.2,52.9,56.8,62.9,33.1,60,46.4,40.2,59.9,32.4,47,51,52.4,54.6,39.7,30.9,54.6,43.3,60,58.7,51.3,55.2,55.8,26.5,48.5,30.6,38.5,69.8,46],[52.7,46.6,52.7,49,47.5,38.6,38.8,21.4,50.7,60,52.9,48.3,50.2,56.6,48.9,48.1,49.7,32.8,37.3,40.7,54.7,61.6,63.3,59.3,52.3,62.9,31.8,53.5,44.6,45.4,29.8,63.6,52.4,45.9,54.3,59.1,42.8,35.2,55.4,56.7,66.59999999999999,52.3,69.40000000000001,60,47.4,55.8,51.8,57.8,40.7],[55.5,58.4,41,43.2,58.8,34.2,36.5,44.1,62.7,68.8,44.5,50.9,56.2,52.2,50.8,51.3,45,24,37.5,32.2,53,52.5,59.7,60.8,56.1,64.3,55.8,38,40.8,58.5,37.7,68.40000000000001,51.4,46.5,54.7,45.8,52.4,48.5,69.59999999999999,55.8,62,54.4,54.5,38.2,45.3,45,42.6,47.6,42.4],[48.7,38.1,56.2,39.1,34.3,53.4,62.4,41.6,40.8,38,44.2,51.2,59.3,49.1,49,62.5,60.4,71,49.4,63.8,48.6,43.4,52,48.3,43,41.6,44,61.8,56.4,51.6,62.7,44.8,49.6,58.5,52.1,39.5,61.4,61.9,45.7,36.1,49,44.3,30.4,55.8,44.7,36.9,79.2,48.6,46.1],[42.7,54.7,40.3,65,57.9,53.9,42.7,77.5,61,56.9,44.7,55.2,44.9,33.7,40.1,43,53.7,50.8,56.6,59.6,43.4,38.5,46.2,45.7,55,34.3,61.3,48.7,57.6,62,64.5,49.6,21.8,46,42.2,58.8,49.6,59.4,55.3,41.9,50.1,52.7,47.7,54.2,57.1,56.6,36.8,35.7,42.4]],"container":"<table class=\"display\">\n <thead>\n <tr>\n <th> <\/th>\n <th>E<\/th>\n <th>A<\/th>\n <th>C<\/th>\n <th>N<\/th>\n <th>O<\/th>\n <\/tr>\n <\/thead>\n<\/table>","options":{"pageLength":4,"columnDefs":[{"className":"dt-right","targets":[1,2,3,4,5]},{"orderable":false,"targets":0},{"name":" ","targets":0},{"name":"E","targets":1},{"name":"A","targets":2},{"name":"C","targets":3},{"name":"N","targets":4},{"name":"O","targets":5}],"order":[],"autoWidth":false,"orderClasses":false,"lengthMenu":[4,10,25,50,100]}},"evals":[],"jsHooks":[]}</script> --- # Data Preparations Before conducting a cluster analysis, it is also generally a good idea to .teal_style[normalise] our data. This is like calculating `\(z\)`-scores for each number in a data set ([recall the standardisation process covered in Topic 3](https://bookdown.org/a_shaker/STM1001_Topic_3/4.2-standardisation.html)). -- * Normalising our data helps remove any impact from different variables being measured on different scales (e.g. flipper length in mm and body mass in grams). * This can prevent any single variable unintentionally overpowering others when being assessed by the cluster algorithm. -- * Normalising the data consists of two steps: -- 1. We compute the sample mean and sample standard deviation for each variable. -- 1. We subtract the relevant sample mean from each observation, and then divide by the relevant sample standard deviation. (Recall the `\(z\)`-score formula: `\(z = \frac{x - \mu}{\sigma}\)`. If we use the sample values instead of the population parameters `\(\mu\)` and `\(\sigma\)`, we have `\(\frac{x - \overline{x}}{s}\)`.) --- # Data Preparations - Big Five Example Note that we don't have to do this normalisation by hand - we can use a simple line of code (R) or click a button (jamovi) to conduct the process. -- Let's take a look at the Big Five data after normalization: -- <div class="datatables html-widget html-fill-item" id="htmlwidget-7e40a9c813873c4504c8" style="width:100%;height:auto;"></div> <script type="application/json" data-for="htmlwidget-7e40a9c813873c4504c8">{"x":{"filter":"none","vertical":false,"data":[[0.5498499999999999,0.05962,-0.01041,0.13966,-0.47063,0.7599399999999999,-0.30055,1.48028,1.0901,1.3202,-0.93084,1.25017,-0.11046,1.28018,-0.45062,0.33975,0.21969,-0.58068,-1.48109,-0.56067,0.51983,0.28973,0.67991,1.29019,-1.69119,1.00005,-0.36057,-0.98086,0.99005,-1.76122,-0.30055,0.09964000000000001,0.2397,0.4598,-1.03089,-1.91129,0.4598,-0.67072,1.00005,0.86999,0.12965,0.51983,0.57986,-2.3515,-0.15048,-1.94131,-1.15094,1.98051,-0.40059],[0.27003,-0.33958,0.27003,-0.09973,-0.24964,-1.13906,-1.11907,-2.85795,0.07016,0.99956,0.29002,-0.16969,0.02019,0.65978,-0.10973,-0.18967,-0.02978,-1.71869,-1.26898,-0.9292,0.4699,1.15946,1.32935,0.9296,0.23006,1.28937,-1.81862,0.34998,-0.53945,-0.4595,-2.01849,1.35933,0.24005,-0.40953,0.42993,0.90962,-0.71933,-1.47884,0.53986,0.66977,1.65913,0.23006,1.93895,0.99956,-0.25963,0.57983,0.18009,0.7796999999999999,-0.9292],[0.54993,0.84,-0.90042,-0.68037,0.88001,-1.58058,-1.35053,-0.59034,1.2701,1.88024,-0.5503400000000001,0.08982,0.61994,0.21985,0.07981000000000001,0.12983,-0.50032,-2.60082,-1.2505,-1.78063,0.29987,0.24986,0.9700299999999999,1.08005,0.60994,1.43014,0.5799299999999999,-1.20049,-0.92042,0.85,-1.2305,1.84023,0.13983,-0.35029,0.46991,-0.4203,0.23985,-0.15024,1.96026,0.5799299999999999,1.20008,0.4399,0.4499,-1.18049,-0.47032,-0.50032,-0.74038,-0.24026,-0.76039],[-0.131,-1.19085,0.6188900000000001,-1.09087,-1.5708,0.33893,1.23881,-0.8409,-0.92089,-1.20085,-0.58094,0.11896,0.92885,-0.09100999999999999,-0.10101,1.2488,1.03883,2.09869,-0.06101,1.37879,-0.141,-0.66093,0.19895,-0.171,-0.70092,-0.8409,-0.60094,1.17881,0.63889,0.15896,1.2688,-0.52095,-0.04101,0.8488599999999999,0.20895,-1.05087,1.13882,1.18881,-0.43096,-1.39083,-0.10101,-0.57094,-1.96075,0.5789,-0.53095,-1.31084,2.91857,-0.141,-0.39097],[-0.72937,0.46959,-0.96916,1.4987,0.78931,0.38966,-0.72937,2.74761,1.09905,0.6894,-0.52954,0.51955,-0.50956,-1.62859,-0.98914,-0.69939,0.36968,0.07993,0.65943,0.95917,-0.65943,-1.149,-0.37967,-0.42963,0.49957,-1.56864,1.12902,-0.12989,0.75934,1.19896,1.44874,-0.03997,-2.81755,-0.39965,-0.77932,0.87924,-0.03997,0.93918,0.52954,-0.8093,0.009990000000000001,0.26977,-0.2298,0.41964,0.70938,0.65943,-1.31885,-1.42876,-0.75934]],"container":"<table class=\"display\">\n <thead>\n <tr>\n <th>E<\/th>\n <th>A<\/th>\n <th>C<\/th>\n <th>N<\/th>\n <th>O<\/th>\n <\/tr>\n <\/thead>\n<\/table>","options":{"pageLength":5,"columnDefs":[{"className":"dt-right","targets":[0,1,2,3,4]},{"name":"E","targets":0},{"name":"A","targets":1},{"name":"C","targets":2},{"name":"N","targets":3},{"name":"O","targets":4}],"order":[],"autoWidth":false,"orderClasses":false,"lengthMenu":[5,10,25,50,100]}},"evals":[],"jsHooks":[]}</script> --- class: middle Once our data is prepared, we are ready to conduct our cluster analysis. --- # 3. k-means Clustering The main type of clustering we will focus on is called `\(k\)`-means clustering. -- `\(k\)`-means clustering is a top-down or *divisive* form of clustering, whereby we take the entire data sample, and divide the observations into similar groups. -- We say that `\(k\)`-means clustering is a *centroid*-based type of clustering, since cluster membership is determined by closeness to cluster means (aka *centroids*). --- # k-means Clustering Steps A simplified description of `\(k\)`-means clustering process is as follows: -- 1. We begin by arbitrarily choosing a number of clusters `\(k\)` to use. -- 1. `\(k\)` initial ***centroids*** (centre points) are initially chosen (normally randomly) -- 1. Each data point from our data set is assigned to a cluster, depending on which ***centroid*** it is closest to, and an iterative process begins: -- * The ***centroid***, i.e. the mean of each cluster, is recalculated -- * Each data point is again assigned the cluster with the closest ***centroid*** -- * This process continues, until the new centroids do not change and the points therefore remain stable within their clusters. At this point the clustering stops. -- * This process can be seen visually at [this link](https://www.naftaliharris.com/blog/visualizing-k-means-clustering/) (Harris, 2014), in a 2-dimensional setting (i.e. where clusters are formed based on two feature variables). Example settings to use: `\(k = 3\)`, Initial centroids: "`I'll choose`", using "`Gaussian Mixture`" data --- # How to choose k? In general, we do not know how many clusters are in our data set. -- * Recall for example our Big Five data set, assuming we only have access to the Big Five measurements and no other variables -- * We cannot easily plot this data and it is not obvious how many clusters there should be -- Like clustering algorithms, there is no universally dominant or correct method for selecting the number of clusters. We will consider two options: -- * .teal_style[The silhouette method] -- * .teal_style[The gap statistic method] Note that only the gap statistic method is available in jamovi --- # How to choose k - The silhouette method The **silhouette** assessment method (Rousseeuw, 1987) measures the similarity of a point to other points in its cluster. -- The silhouette value for each observation can be between 0 and 1. -- * Values close to 0 suggest clusters are very similar * Values close to 1 indicate clusters are very different -- Thus, the higher the average silhouette width value (across all points in a cluster), the better. --- # How to choose k - The silhouette method To interpret a silhouette plot, we simply pick the number of clusters which results in the highest average silhouette width value - which is 3 in the example below: -- <img src="data:image/png;base64,#STM1001_Data_Science_Clustering_Lecture_files/figure-html/unnamed-chunk-6-1.svg" width="55%" style="display: block; margin: auto;" /> --- # How to choose k - the gap statistic method The other clustering assessment method we will consider is the **gap statistic** assessment method (Tibshirani, Walther & Hastie, 2001). -- We won’t go into the mathematical details of this statistic; suffice to say that .bold_style[higher values are considered preferable], like the silhouette method. * However, .bold_style[if the best gap statistic values are similar between two potential cluster numbers, we may select the smaller number of clusters]. --- # How to choose k - the gap statistic method We can interpret the gap statistic plots in a similar way to the silhouette plots. -- Just look for the dashed line highlighting the number of clusters determined to be optimal - in the example below it is 1. -- <img src="data:image/png;base64,#STM1001_Data_Science_Clustering_Lecture_files/figure-html/unnamed-chunk-7-1.svg" width="50%" style="display: block; margin: auto;" /> --- # Determining the optimal number for k * As we have seen, different diagnostic methods may propose different optimal numbers for `\(k\)`, for a given data set. -- * In practice, we can carry out the cluster analysis for various values of `\(k\)` and make a decision after seeing the results. -- * Sometimes, there is no "right" choice for `\(k\)`, so whatever value is chosen, it is important to be able to justify why you have chosen a specific value for `\(k\)`. -- * Returning to the Big Five example, let's choose a value of `\(k = 3\)` and see the results --- # Big Five example Choosing a value of `\(k = 3\)` results in the following cluster plot: .pull-left[ <img src="data:image/png;base64,#STM1001_Data_Science_Clustering_Lecture_files/figure-html/unnamed-chunk-8-1.svg" width="120%" style="display: block; margin: auto;" /> ] -- .pull-right[ * As we can see, three distinct clusters have been identified {{content}} ] -- * Cluster 1 overlaps a little bit with both Cluster 2 and Cluster 3 (compare this to the result with `\(k = 2\)` which we saw earlier) {{content}} -- * However, a reasonable amount of separation has been achieved Note that since there is some randomness involved in the clustering algorithm, the results can vary slightly each time the algorithm is run --- # Big Five example: Some diagnostics As well as the cluster plot, there are a number of other diagnostics we can look at to understand the results. For example, consider the following "Means across clusters" plot: -- <img src="data:image/png;base64,#Means_across_clusters.png" width="400px" style="display: block; margin: auto;" /> -- Taking Cluster 3 as an example, the cluster contains states with above average levels of Extraversion, Agreeableness and Conscientiousness, and below average levels of Neuroticism and Openness -- See if you can interpret the other two clusters. --- # Big Five example: Some diagnostics * We carried out our cluster analysis assuming we did not have any information about the `Region` variable. -- * But, an interesting question to ask is: how closely aligned are the identified clusters to the Regions that exist in the data? --- # Big Five example: Some diagnostics * Because we actually do have the Regions variable, we can compare the Region categories with the identified clusters: <img src="data:image/png;base64,#Regions_clusters.png" width="400px" style="display: block; margin: auto;" /> -- * All Midwest (MW) states except one have been assigned to Cluster 3; -- all Northeast (NE) states except one have been assigned to Cluster 1; -- most West (W) states have been assigned to Cluster 2; -- South (S) states have been assigned to a mix of all clusters. -- * Overall, generally speaking, the clusters do seem to have separated the regions into different groups, although there is some overlap * We will consider some other diagnostic measures in Computer Lab 7B --- # Cluster Analysis Extensions There are many other clustering analysis options out there to discover. In Computer Lab 7B, as extensions, we can look at: -- * .teal_style[Hierarchical clustering] -- * .teal_style[PAM clustering] and .teal_style[Fuzzy clustering] (**Data Science** students only) --- # Extensions: Hierarchical Clustering The hierarchical clustering process is very different to `\(k\)`-means clustering, with clusters visualised via a .teal_style[dendrogram], as shown below: -- <img src="data:image/png;base64,#STM1001_dendrogram.png" width="600px" style="display: block; margin: auto;" /> *Note the y-axis values denote distance between clusters.* --- # Extensions: Combining Cluster Analyses While beyond the scope of this subject, it is worthwhile to note that we can even combine information from multiple cluster analyses - see for example this .teal_style[heatmap] below, produced by dendrogram bi-clustering. <img src="data:image/png;base64,#heatmap.jpg" width="750px" style="display: block; margin: auto;" /> --- name: menti class: middle background-image: url(data:image/png;base64,#menti.jpg) background-size: 115% # Kahoot ## Go to [www.kahoot.it](https://www.kahoot.it) and use ## the code provided --- # End That concludes our Introduction to Cluster Analysis lecture. -- What to do next: * Before Computer Lab 7B, make sure you are up-to-date with the current assessments, and remember that you can ask your computer lab demonstrator during the lab if you have questions about the assessments * If you have any cluster analysis questions, we can resolve them in the computer labs. --- background-image: url(data:image/png;base64,#computerlab.jpg) background-position: bottom background-size: 75% class: center # See you in the computer labs! --- # References Harris, N. (2014). _Visualizing K-Means Clustering_. URL: [https://www.naftaliharris.com/blog/visualizing-k-means-clustering/](https://www.naftaliharris.com/blog/visualizing-k-means-clustering/). Hartnett, J. (2020). _Not awful and boring ideas for teaching statistics_. URL: [https://notawfulandboring.blogspot.com/2020/04/online-day-6-one-way-anova-example.html](https://notawfulandboring.blogspot.com/2020/04/online-day-6-one-way-anova-example.html). Everitt, B. (1974). *Cluster Analysis*. New York: Wiley. Gan, G., C. Ma, & J. Wu. 2007. *Data Clustering: Theory, Algorithms, and Applications*. ASA-SIAM Series on Statistics and Applied Probability ; 20. Philadelphia, Pa.: Society for Industrial; Applied Mathematics (SIAM, 3600 Market Street, Floor 6, Philadelphia, PA 19104). [https://doi.org/10.1137/1.9780898718348](https://doi.org/10.1137/1.9780898718348). Horikoshi, M., Y. Tang, A. Dickey, M. Grenie, R. Thompson, L. Selzer, D. Strbenac, K. Voronin, & D. Pulatov. 2021. *ggfortify: Data Visualization Tools for Statistical Analysis Results*. [https://github.com/sinhrks/ggfortify](https://github.com/sinhrks/ggfortify). The jamovi project. 2022. *Jamovi [Computer Software]*.[https://www.jamovi.org](https://www.jamovi.org). --- # References Kassambara, A., and F. Mundt. 2020. *factoextra: Extract and Visualize the Results of Multivariate Data Analyses.* [http://www.sthda.com/english/rpkgs/factoextra](http://www.sthda.com/english/rpkgs/factoextra). Mirkin, B. 1996. Mathematical Classification and Clustering. 1st ed. 1996.. Nonconvex Optimization and Its Applications, 11. Posit team (2023). RStudio: Integrated Development Environment for R. Posit Software, PBC, Boston, MA. URL [http://www.posit.co/](http://www.posit.co/). R Core Team. 2021. *R: A Language and Environment for Statistical Computing*. Vienna, Austria: R Foundation for Statistical Computing. [https://www.R-project.org/](https://www.R-project.org/) Rousseeuw, P. J. 1987. Silhouette: A Graphical Aid to the Interpretation and Validation of Cluster Analysis. *Computational and Applied Mathematics*. 20: 53–65. Seol, H. 2022. *snowCluster: Multivariate Analysis*. [https://github.com/hyunsooseol/snowCluster](https://github.com/hyunsooseol/snowCluster) --- # References Tibshirani, R., Walther, G., & Hastie, T. 2001. Estimating the number of clusters in a data set via the gap statistic. *Journal of the Royal Statistical Society, Series B*. 63 (2): 411–423. [https://doi.org/10.1111/1467-9868.00293](https://doi.org/10.1111/1467-9868.00293) --- class: middle <font color = "grey"> These notes have been prepared by Amanda Shaker and Rupert Kuveke. The copyright for the material in these notes resides with the authors named above, with the Department of Mathematical and Physical Sciences and with La Trobe University. Copyright in this work is vested in La Trobe University including all La Trobe University branding and naming. Unless otherwise stated, material within this work is licensed under a Creative Commons Attribution-Non Commercial-Non Derivatives License <a href = "https://creativecommons.org/licenses/by-nc-nd/4.0/" target="_blank"> BY-NC-ND. </a> </font>