Machine learning is a branch of Artificial Intelligence that studies programs/algorithms that improve, or “learn” through experience.
In machine learning, programs/algorithms typically learn to perform tasks such as prediction without being explicitly programmed to do so.
Machine learning is generally used for prediction, which uses sample data to learn about outcomes for new data points.
Machine learning implementations are classified into two major categories, depending on the nature of learning.
Supervised Learning - using training data with known outcomes/labels.
Unsupervised Learning - using training data with unknown outcomes/labels.
Supervised learning as the name itself suggests that under the presence of supervision. In short in supervised learning we try to teach the machine with the data using labels and which already have the correct answer in it.
After this, the machine will create an example set of data so that the supervised algorithm analyses the training data and produce the correct output of the labeled data.
For example, if we create a set of data of fruits then we will be labeling as the fruit having a round shape with a dip upside and red in color then it is termed as an apple.
Now when we ask the machine to identify the apple from the basket of fruits then it will use the previous labeling and identify an apple.
Supervised Learning is classified into two categories as below:
Classification: a classification problem is when the output variable is a category, such as “Red” or “Orange” or “countable” or “not countable”.
Regression: a regression is used when the output variable is real value, such as “rupees” or “height”.
Unsupervised learning is the training of machines using information that is not labeled and it works without any guidance.
Here the main task of the machine is to separate the data using the similarities, differences, and patterns without any prior supervision. Hence, the machine is restricted to find the hidden structure in unlabeled data by own-self.
For example, if we provide a group of cats and dogs which are never seen before. Then the machine will differentiate the group of cats and dogs according to their behavior and nature.
Now when we provide the pictures of dogs and cats according to the classification made by the machine it will provide the result.
Clustering: a clustering problem is where the machine identify the inherent groupings in the data, such as grouping customers according to visits in the shop.
Association: an association problem is where we can find the relation between two events or items, such as people buying item A also tends to buy B.
Regression: the regression technique helps the machine learning approach to predict continuous values. For example, the price of a house.
Classification: the input is divided into one or more classes or categories for the learner to produce a model to assign unseen modules. For example, in the case of email fraud, we can divide the emails into two classes i.e “spam” and “not spam”.
Clustering: this technique follows the summarization, finding a group of similar entities. For example, we can gather and take readings of the patients in the hospital.
Association: this technique finds co-occurring events or items. For example, market-basket.
R language is perfect for data visualization. R language provides the best prototype to work with machine learning models.
R language has the best tools and library packages to work with machine learning projects. Developers can use these packages to create the best pre-model, model, and post-model of the machine learning projects.
Social Media Service: Help people to connect all over the world and also show the recommendations of the people we may know.
Online Customer Support: Provide high convenience of customer and efficiency of support agent.
Product Recommendation: A software tool used to recommend the product that you might like to purchase or engage with.
Healthcare: Machine Learning can manage a large amount of data beyond the imagination of normal human being and help to identify the illness of the patient according to symptoms.
Real world example: When you search for some kind of cooking recipe on youTube, you will see the recommendations below with the title “You May Also Like This”. This is a common use of Machine Learning.
K-means clustering (MacQueen 1967) is one of the most commonly used unsupervised machine learning algorithm for partitioning a given data set into a set of k groups (i.e. k clusters), where k represents the number of groups pre-specified by the analyst.
It classifies objects in multiple groups (i.e., clusters), such that objects within the same cluster are as similar as possible (i.e., high intra-class similarity), whereas objects from different clusters are as dissimilar as possible (i.e., low inter-class similarity).
In k-means clustering, each cluster is represented by its center (i.e, centroid) which corresponds to the mean of points assigned to the cluster.
Here, we will learn about:
Specify the number of clusters (K) to be created (by the analyst)
Select randomly k objects from the dataset as the initial cluster centers or means
Assigns each observation to their closest centroid, based on the Euclidean distance between the object and the centroid
For each of the k clusters update the cluster centroid by calculating the new mean values of all the data points in the cluster. The centoid of a Kth cluster is a vector of length p containing the means of all variables for the observations in the kth cluster; p is the number of variables.
Iteratively minimize the total within sum of square. That is, iterate steps 3 and 4 until the cluster assignments stop changing or the maximum number of iterations is reached. By default, the R software uses 10 as the default value for the maximum number of iterations.
The data must contain only continuous variables, as the k-means algorithm uses variable means.
As we don’t want the k-means algorithm to depend to an arbitrary variable unit, we start by scaling the data using the R function scale() as follow:
## Murder Assault UrbanPop Rape
## Alabama 1.24256408 0.7828393 -0.5209066 -0.003416473
## Alaska 0.50786248 1.1068225 -1.2117642 2.484202941
## Arizona 0.07163341 1.4788032 0.9989801 1.042878388
## Arkansas 0.23234938 0.2308680 -1.0735927 -0.184916602
## California 0.27826823 1.2628144 1.7589234 2.067820292
## Colorado 0.02571456 0.3988593 0.8608085 1.864967207
kmeans(x, centers, iter.max = 10, nstart = 1)
x: numeric matrix, numeric data frame or a numeric vector
centers: Possible values are the number of clusters (k).
iter.max: The maximum number of iterations allowed. Default value is 10.
nstart: The number of random starting partitions when centers is a number. Trying nstart > 1 is often recommended.
To create a beautiful graph of the clusters generated with the kmeans() function, will use the factoextra package.
# install.packages("factoextra")
# library(factoextra)
# Compute k-means with k = 4
set.seed(123)
km.res <- kmeans(df, 4, nstart = 25)As the final result of k-means clustering result is sensitive to the random starting assignments, we specify nstart = 25. This means that R will try 25 different random starting assignments and then select the best results corresponding to the one with the lowest within cluster variation.
The default value of nstart in R is one. But, it’s strongly recommended to compute k-means clustering with a large value of nstart such as 25 or 50, in order to have a more stable result.
To create a beautiful graph of the clusters generated with the kmeans() function, will use the factoextra package.
# install.packages("factoextra")
# library(factoextra)
# Compute k-means with k = 4
set.seed(123)
km.res <- kmeans(df, 4, nstart = 25)
# Print the results
print(km.res)## K-means clustering with 4 clusters of sizes 8, 13, 16, 13
##
## Cluster means:
## Murder Assault UrbanPop Rape
## 1 1.4118898 0.8743346 -0.8145211 0.01927104
## 2 -0.9615407 -1.1066010 -0.9301069 -0.96676331
## 3 -0.4894375 -0.3826001 0.5758298 -0.26165379
## 4 0.6950701 1.0394414 0.7226370 1.27693964
##
## Clustering vector:
## Alabama Alaska Arizona Arkansas California
## 1 4 4 1 4
## Colorado Connecticut Delaware Florida Georgia
## 4 3 3 4 1
## Hawaii Idaho Illinois Indiana Iowa
## 3 2 4 3 2
## Kansas Kentucky Louisiana Maine Maryland
## 3 2 1 2 4
## Massachusetts Michigan Minnesota Mississippi Missouri
## 3 4 2 1 4
## Montana Nebraska Nevada New Hampshire New Jersey
## 2 2 4 2 3
## New Mexico New York North Carolina North Dakota Ohio
## 4 4 1 2 3
## Oklahoma Oregon Pennsylvania Rhode Island South Carolina
## 3 3 3 3 1
## South Dakota Tennessee Texas Utah Vermont
## 2 1 4 3 2
## Virginia Washington West Virginia Wisconsin Wyoming
## 3 3 2 2 3
##
## Within cluster sum of squares by cluster:
## [1] 8.316061 11.952463 16.212213 19.922437
## (between_SS / total_SS = 71.2 %)
##
## Available components:
##
## [1] "cluster" "centers" "totss" "withinss" "tot.withinss"
## [6] "betweenss" "size" "iter" "ifault"
## cluster Murder Assault UrbanPop Rape
## 1 1 13.93750 243.62500 53.75000 21.41250
## 2 2 3.60000 78.53846 52.07692 12.17692
## 3 3 5.65625 138.87500 73.87500 18.78125
## 4 4 10.81538 257.38462 76.00000 33.19231
## Murder Assault UrbanPop Rape cluster
## Alabama 13.2 236 58 21.2 1
## Alaska 10.0 263 48 44.5 4
## Arizona 8.1 294 80 31.0 4
## Arkansas 8.8 190 50 19.5 1
## California 9.0 276 91 40.6 4
## Colorado 7.9 204 78 38.7 4
K-means clustering is very simple and fast algorithm. It can efficiently deal with very large data sets. However there are some weaknesses, including:
It assumes prior knowledge of the data and requires the analyst to choose the appropriate number of cluster (k) in advance.
The final results obtained is sensitive to the initial random selection of cluster centers. Why is it a problem? Because, for every different run of the algorithm on the same dataset, you may choose different set of initial centers. This may lead to different clustering results on different runs of the algorithm.
It’s sensitive to outliers.
If you rearrange your data, it’s very possible that you’ll get a different solution every time you change the ordering of your data.
Possible solutions to these weaknesses, include:
Solution to issue 1: Compute k-means for a range of k values, for example by varying k between 2 and 10. Then, choose the best k by comparing the clustering results obtained for the different k values.
Solution to issue 2: Compute K-means algorithm several times with different initial cluster centers. The run with the lowest total within-cluster sum of square is selected as the final clustering solution.
To avoid distortions caused by excessive outliers, it’s possible to use PAM algorithm, which is less sensitive to outliers.
Main Goal: Practice using R to implement a statistical method to analyze multivariate data. This slides shows you how to use the R statistical software to carry out some simple multivariate analyses, with a focus on principal components analysis (PCA). PCA is used in exploratory data analysis and for making decisions in predictive models.
Multi -> Multiple
Variate -> Variables
Multivariate -> Multiple Variables
# library(car)
# wine <- read.table("http://archive.ics.uci.edu/ml/machine-learning-databases/wine/wine # .data", sep=",")
head(wine) ## V1 V2 V3 V4 V5 V6 V7 V8 V9 V10 V11 V12 V13 V14
## 1 1 14.23 1.71 2.43 15.6 127 2.80 3.06 0.28 2.29 5.64 1.04 3.92 1065
## 2 1 13.20 1.78 2.14 11.2 100 2.65 2.76 0.26 1.28 4.38 1.05 3.40 1050
## 3 1 13.16 2.36 2.67 18.6 101 2.80 3.24 0.30 2.81 5.68 1.03 3.17 1185
## 4 1 14.37 1.95 2.50 16.8 113 3.85 3.49 0.24 2.18 7.80 0.86 3.45 1480
## 5 1 13.24 2.59 2.87 21.0 118 2.80 2.69 0.39 1.82 4.32 1.04 2.93 735
## 6 1 14.20 1.76 2.45 15.2 112 3.27 3.39 0.34 1.97 6.75 1.05 2.85 1450
##
## 1 2 3
## 59 71 48
## V2 V3 V4 V5 V6
## 1 14.23 1.71 2.43 15.6 127
## 2 13.20 1.78 2.14 11.2 100
## 3 13.16 2.36 2.67 18.6 101
## 4 14.37 1.95 2.50 16.8 113
## 5 13.24 2.59 2.87 21.0 118
## 6 14.20 1.76 2.45 15.2 112
The purpose of principal component analysis is to find the best low-dimensional representation of the variation in a multivariate data set.
For example, in the case of the wine data set, we have 13 chemical concentrations describing wine samples from three different cultivars.
We can carry out a principal component analysis to investigate whether we can capture most of the variation between samples using a smaller number of new variables (principal components), where each of these new variables is a linear combination of all or some of the 13 chemical concentrations.
To carry out a principal component analysis (PCA) on a multivariate data set, the first step is often to standardise the variables under study using the “scale()” function.
This is necessary if the input variables have very different variances, which is true in this case as the concentrations of the 13 chemicals have very different variances.
Once you have standardised your variables, you can carry out a principal component analysis using the “prcomp()” function in R.
## V2 V3 V4 V5
## Min. :11.03 Min. :0.740 Min. :1.360 Min. :10.60
## 1st Qu.:12.36 1st Qu.:1.603 1st Qu.:2.210 1st Qu.:17.20
## Median :13.05 Median :1.865 Median :2.360 Median :19.50
## Mean :13.00 Mean :2.336 Mean :2.367 Mean :19.49
## 3rd Qu.:13.68 3rd Qu.:3.083 3rd Qu.:2.558 3rd Qu.:21.50
## Max. :14.83 Max. :5.800 Max. :3.230 Max. :30.00
## V6
## Min. : 70.00
## 1st Qu.: 88.00
## Median : 98.00
## Mean : 99.74
## 3rd Qu.:107.00
## Max. :162.00
## standardise the variables
standardisedconcentrations <- as.data.frame(scale(wine[2:14]))
head(standardisedconcentrations)## V2 V3 V4 V5 V6 V7 V8
## 1 1.5143408 -0.56066822 0.2313998 -1.1663032 1.90852151 0.8067217 1.0319081
## 2 0.2455968 -0.49800856 -0.8256672 -2.4838405 0.01809398 0.5670481 0.7315653
## 3 0.1963252 0.02117152 1.1062139 -0.2679823 0.08810981 0.8067217 1.2121137
## 4 1.6867914 -0.34583508 0.4865539 -0.8069748 0.92829983 2.4844372 1.4623994
## 5 0.2948684 0.22705328 1.8352256 0.4506745 1.27837900 0.8067217 0.6614853
## 6 1.4773871 -0.51591132 0.3043010 -1.2860793 0.85828399 1.5576991 1.3622851
## V9 V10 V11 V12 V13 V14
## 1 -0.6577078 1.2214385 0.2510088 0.3611585 1.8427215 1.01015939
## 2 -0.8184106 -0.5431887 -0.2924962 0.4049085 1.1103172 0.96252635
## 3 -0.4970050 2.1299594 0.2682629 0.3174085 0.7863692 1.39122370
## 4 -0.9791134 1.0292513 1.1827317 -0.4263410 1.1807407 2.32800680
## 5 0.2261576 0.4002753 -0.3183774 0.3611585 0.4483365 -0.03776747
## 6 -0.1755994 0.6623487 0.7298108 0.4049085 0.3356589 2.23274072
## [1] 2.1692972 1.5801816 1.2025273 0.9586313 0.9237035 0.8010350 0.7423128
## [8] 0.5903367 0.5374755 0.5009017 0.4751722 0.4108165 0.3215244
## [1] 13
In this case, we see that the total variance is 13, which is equal to the number of standardized variables (13 variables).
This is because for standardized data, the variance of each standardized variable is 1.
The total variance is equal to the sum of the variances of the individual variables, and since the variance of each standardized variable is 1, the total variance should be equal to the number of variables (13 here).
The most obvious change in slope in the scree plot occurs at component 4, which is the “elbow” of the scree plot.
Therefore, it cound be argued based on the basis of the scree plot that the first three components should be retained.
## [1] 4.7058503 2.4969737 1.4460720 0.9189739 0.8532282 0.6416570 0.5510283
## [8] 0.3484974 0.2888799 0.2509025 0.2257886 0.1687702 0.1033779
# Compute k-means with k = 3
set.seed(123)
km.res_1 <- kmeans(standardisedconcentrations, 3, nstart = 25)
# Print the results
print(km.res_1)## K-means clustering with 3 clusters of sizes 51, 62, 65
##
## Cluster means:
## V2 V3 V4 V5 V6 V7
## 1 0.1644436 0.8690954 0.1863726 0.5228924 -0.07526047 -0.97657548
## 2 0.8328826 -0.3029551 0.3636801 -0.6084749 0.57596208 0.88274724
## 3 -0.9234669 -0.3929331 -0.4931257 0.1701220 -0.49032869 -0.07576891
## V8 V9 V10 V11 V12 V13
## 1 -1.21182921 0.72402116 -0.77751312 0.9388902 -1.1615122 -1.2887761
## 2 0.97506900 -0.56050853 0.57865427 0.1705823 0.4726504 0.7770551
## 3 0.02075402 -0.03343924 0.05810161 -0.8993770 0.4605046 0.2700025
## V14
## 1 -0.4059428
## 2 1.1220202
## 3 -0.7517257
##
## Clustering vector:
## [1] 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
## [38] 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 3 3 1 3 3 3 3 3 3 3 3 3 3 3 2
## [75] 3 3 3 3 3 3 3 3 3 1 3 3 3 3 3 3 3 3 3 3 3 2 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
## [112] 3 3 3 3 3 3 3 1 3 3 2 3 3 3 3 3 3 3 3 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
## [149] 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
##
## Within cluster sum of squares by cluster:
## [1] 326.3537 385.6983 558.6971
## (between_SS / total_SS = 44.8 %)
##
## Available components:
##
## [1] "cluster" "centers" "totss" "withinss" "tot.withinss"
## [6] "betweenss" "size" "iter" "ifault"
If you recall from the post about k-means clustering, it requires us to specify the number of clusters, and finding the optimal number of clusters can often be hard.
Hierarchical clustering is an alternative approach which builds a hierarchy from the bottom-up, and doesn’t require us to specify the number of clusters beforehand.
The algorithm works as follows:
Put each data point in its own cluster.
Identify the closest two clusters and combine them into one cluster.
Repeat the above step till all the data points are in a single cluster.
Once this is done, it is usually represented by a dendrogram like structure. A dendrogram is a branching diagram that represents the relationships of similarity among a group of entities.
In the iris dataset, we have three different species of flowers.
## Sepal.Length Sepal.Width Petal.Length Petal.Width
## Min. :4.300 Min. :2.000 Min. :1.000 Min. :0.100
## 1st Qu.:5.100 1st Qu.:2.800 1st Qu.:1.600 1st Qu.:0.300
## Median :5.800 Median :3.000 Median :4.350 Median :1.300
## Mean :5.843 Mean :3.057 Mean :3.758 Mean :1.199
## 3rd Qu.:6.400 3rd Qu.:3.300 3rd Qu.:5.100 3rd Qu.:1.800
## Max. :7.900 Max. :4.400 Max. :6.900 Max. :2.500
## Species
## setosa :50
## versicolor:50
## virginica :50
##
##
##
## [1] setosa setosa setosa setosa setosa setosa
## [7] setosa setosa setosa setosa setosa setosa
## [13] setosa setosa setosa setosa setosa setosa
## [19] setosa setosa setosa setosa setosa setosa
## [25] setosa setosa setosa setosa setosa setosa
## [31] setosa setosa setosa setosa setosa setosa
## [37] setosa setosa setosa setosa setosa setosa
## [43] setosa setosa setosa setosa setosa setosa
## [49] setosa setosa versicolor versicolor versicolor versicolor
## [55] versicolor versicolor versicolor versicolor versicolor versicolor
## [61] versicolor versicolor versicolor versicolor versicolor versicolor
## [67] versicolor versicolor versicolor versicolor versicolor versicolor
## [73] versicolor versicolor versicolor versicolor versicolor versicolor
## [79] versicolor versicolor versicolor versicolor versicolor versicolor
## [85] versicolor versicolor versicolor versicolor versicolor versicolor
## [91] versicolor versicolor versicolor versicolor versicolor versicolor
## [97] versicolor versicolor versicolor versicolor virginica virginica
## [103] virginica virginica virginica virginica virginica virginica
## [109] virginica virginica virginica virginica virginica virginica
## [115] virginica virginica virginica virginica virginica virginica
## [121] virginica virginica virginica virginica virginica virginica
## [127] virginica virginica virginica virginica virginica virginica
## [133] virginica virginica virginica virginica virginica virginica
## [139] virginica virginica virginica virginica virginica virginica
## [145] virginica virginica virginica virginica virginica virginica
## Levels: setosa versicolor virginica
##
## clusterCut setosa versicolor virginica
## 1 50 0 0
## 2 0 21 50
## 3 0 29 0
It looks like the algorithm successfully classified all the flowers of species setosa into cluster 1, and virginica into cluster 2, but had trouble with versicolor.
If you look at the original plot showing the different species (on the next slide), you can understand why.