Introduction to Machine Learning

Introduction to Machine Learning in R

Classification Of Machine Learning

Machine learning implementations are classified into two major categories, depending on the nature of learning.

  1. Supervised Learning - using training data with known outcomes/labels.

  2. Unsupervised Learning - using training data with unknown outcomes/labels.

Supervised Learning

Supervised Learning Continuation

Supervised Learning is classified into two categories as below:

  1. Classification: a classification problem is when the output variable is a category, such as “Red” or “Orange” or “countable” or “not countable”.

  2. Regression: a regression is used when the output variable is real value, such as “rupees” or “height”.

Unsupervised Learning

Unsupervised Learning Continuation

  1. Clustering: a clustering problem is where the machine identify the inherent groupings in the data, such as grouping customers according to visits in the shop.

  2. Association: an association problem is where we can find the relation between two events or items, such as people buying item A also tends to buy B.

Types of Machine Learning Problems

  1. Regression: the regression technique helps the machine learning approach to predict continuous values. For example, the price of a house.

  2. Classification: the input is divided into one or more classes or categories for the learner to produce a model to assign unseen modules. For example, in the case of email fraud, we can divide the emails into two classes i.e “spam” and “not spam”.

  3. Clustering: this technique follows the summarization, finding a group of similar entities. For example, we can gather and take readings of the patients in the hospital.

  4. Association: this technique finds co-occurring events or items. For example, market-basket.

Software

Advantages to Implement Machine Learning Using R Language

  1. R language is perfect for data visualization. R language provides the best prototype to work with machine learning models.

  2. R language has the best tools and library packages to work with machine learning projects. Developers can use these packages to create the best pre-model, model, and post-model of the machine learning projects.

Some Examples of Machine Learning Problems

  1. Social Media Service: Help people to connect all over the world and also show the recommendations of the people we may know.

  2. Online Customer Support: Provide high convenience of customer and efficiency of support agent.

  3. Product Recommendation: A software tool used to recommend the product that you might like to purchase or engage with.

  4. Healthcare: Machine Learning can manage a large amount of data beyond the imagination of normal human being and help to identify the illness of the patient according to symptoms.

  5. Real world example: When you search for some kind of cooking recipe on youTube, you will see the recommendations below with the title “You May Also Like This”. This is a common use of Machine Learning.

K-Means Clustering in R: Algorithm and Practical Examples

K-Means Clustering

Here, we will learn about:

  1. The basic steps of k-means algorithm
  2. How to compute k-means in R software using practical examples
  3. Advantages and disadvantages of k-means clustering

K-Means Clustering

  1. Specify the number of clusters (K) to be created (by the analyst)

  2. Select randomly k objects from the dataset as the initial cluster centers or means

  3. Assigns each observation to their closest centroid, based on the Euclidean distance between the object and the centroid

  4. For each of the k clusters update the cluster centroid by calculating the new mean values of all the data points in the cluster. The centoid of a Kth cluster is a vector of length p containing the means of all variables for the observations in the kth cluster; p is the number of variables.

  5. Iteratively minimize the total within sum of square. That is, iterate steps 3 and 4 until the cluster assignments stop changing or the maximum number of iterations is reached. By default, the R software uses 10 as the default value for the maximum number of iterations.

Example: Data

data("USArrests")      # Loading the data set
df <- scale(USArrests) # Scaling the data

Example: Data

data("USArrests")      # Loading the data set
df <- scale(USArrests) # Scaling the data

head(df) 
##                Murder   Assault   UrbanPop         Rape
## Alabama    1.24256408 0.7828393 -0.5209066 -0.003416473
## Alaska     0.50786248 1.1068225 -1.2117642  2.484202941
## Arizona    0.07163341 1.4788032  0.9989801  1.042878388
## Arkansas   0.23234938 0.2308680 -1.0735927 -0.184916602
## California 0.27826823 1.2628144  1.7589234  2.067820292
## Colorado   0.02571456 0.3988593  0.8608085  1.864967207

Required R Packages and Functions for K-Means Clustering

kmeans(x, centers, iter.max = 10, nstart = 1)

Required R Packages and Functions for K-Means Clustering

To create a beautiful graph of the clusters generated with the kmeans() function, will use the factoextra package.

# install.packages("factoextra")

# library(factoextra)

# Compute k-means with k = 4
set.seed(123)
km.res <- kmeans(df, 4, nstart = 25)

Required R Packages and Functions for K-Means Clustering

To create a beautiful graph of the clusters generated with the kmeans() function, will use the factoextra package.

# install.packages("factoextra")

# library(factoextra)

# Compute k-means with k = 4
set.seed(123)
km.res <- kmeans(df, 4, nstart = 25)

# Print the results
print(km.res)
## K-means clustering with 4 clusters of sizes 8, 13, 16, 13
## 
## Cluster means:
##       Murder    Assault   UrbanPop        Rape
## 1  1.4118898  0.8743346 -0.8145211  0.01927104
## 2 -0.9615407 -1.1066010 -0.9301069 -0.96676331
## 3 -0.4894375 -0.3826001  0.5758298 -0.26165379
## 4  0.6950701  1.0394414  0.7226370  1.27693964
## 
## Clustering vector:
##        Alabama         Alaska        Arizona       Arkansas     California 
##              1              4              4              1              4 
##       Colorado    Connecticut       Delaware        Florida        Georgia 
##              4              3              3              4              1 
##         Hawaii          Idaho       Illinois        Indiana           Iowa 
##              3              2              4              3              2 
##         Kansas       Kentucky      Louisiana          Maine       Maryland 
##              3              2              1              2              4 
##  Massachusetts       Michigan      Minnesota    Mississippi       Missouri 
##              3              4              2              1              4 
##        Montana       Nebraska         Nevada  New Hampshire     New Jersey 
##              2              2              4              2              3 
##     New Mexico       New York North Carolina   North Dakota           Ohio 
##              4              4              1              2              3 
##       Oklahoma         Oregon   Pennsylvania   Rhode Island South Carolina 
##              3              3              3              3              1 
##   South Dakota      Tennessee          Texas           Utah        Vermont 
##              2              1              4              3              2 
##       Virginia     Washington  West Virginia      Wisconsin        Wyoming 
##              3              3              2              2              3 
## 
## Within cluster sum of squares by cluster:
## [1]  8.316061 11.952463 16.212213 19.922437
##  (between_SS / total_SS =  71.2 %)
## 
## Available components:
## 
## [1] "cluster"      "centers"      "totss"        "withinss"     "tot.withinss"
## [6] "betweenss"    "size"         "iter"         "ifault"

K-Means Clustering

aggregate(USArrests, by=list(cluster=km.res$cluster), mean)
##   cluster   Murder   Assault UrbanPop     Rape
## 1       1 13.93750 243.62500 53.75000 21.41250
## 2       2  3.60000  78.53846 52.07692 12.17692
## 3       3  5.65625 138.87500 73.87500 18.78125
## 4       4 10.81538 257.38462 76.00000 33.19231

K-Means Clustering

dd <- cbind(USArrests, cluster = km.res$cluster)
head(dd)
##            Murder Assault UrbanPop Rape cluster
## Alabama      13.2     236       58 21.2       1
## Alaska       10.0     263       48 44.5       4
## Arizona       8.1     294       80 31.0       4
## Arkansas      8.8     190       50 19.5       1
## California    9.0     276       91 40.6       4
## Colorado      7.9     204       78 38.7       4

K-means clustering Advantages and Disadvantages

K-means clustering is very simple and fast algorithm. It can efficiently deal with very large data sets. However there are some weaknesses, including:

  1. It assumes prior knowledge of the data and requires the analyst to choose the appropriate number of cluster (k) in advance.

  2. The final results obtained is sensitive to the initial random selection of cluster centers. Why is it a problem? Because, for every different run of the algorithm on the same dataset, you may choose different set of initial centers. This may lead to different clustering results on different runs of the algorithm.

  3. It’s sensitive to outliers.

  4. If you rearrange your data, it’s very possible that you’ll get a different solution every time you change the ordering of your data.

Remedies

Possible solutions to these weaknesses, include:

  1. Solution to issue 1: Compute k-means for a range of k values, for example by varying k between 2 and 10. Then, choose the best k by comparing the clustering results obtained for the different k values.

  2. Solution to issue 2: Compute K-means algorithm several times with different initial cluster centers. The run with the lowest total within-cluster sum of square is selected as the final clustering solution.

  3. To avoid distortions caused by excessive outliers, it’s possible to use PAM algorithm, which is less sensitive to outliers.

Multivariate Data

Main Goal: Practice using R to implement a statistical method to analyze multivariate data. This slides shows you how to use the R statistical software to carry out some simple multivariate analyses, with a focus on principal components analysis (PCA). PCA is used in exploratory data analysis and for making decisions in predictive models.

Part 1: Multivariate Data

Example 1: Multivariate Data

# library(car)
# wine <- read.table("http://archive.ics.uci.edu/ml/machine-learning-databases/wine/wine #  .data", sep=",")

head(wine) 
##   V1    V2   V3   V4   V5  V6   V7   V8   V9  V10  V11  V12  V13  V14
## 1  1 14.23 1.71 2.43 15.6 127 2.80 3.06 0.28 2.29 5.64 1.04 3.92 1065
## 2  1 13.20 1.78 2.14 11.2 100 2.65 2.76 0.26 1.28 4.38 1.05 3.40 1050
## 3  1 13.16 2.36 2.67 18.6 101 2.80 3.24 0.30 2.81 5.68 1.03 3.17 1185
## 4  1 14.37 1.95 2.50 16.8 113 3.85 3.49 0.24 2.18 7.80 0.86 3.45 1480
## 5  1 13.24 2.59 2.87 21.0 118 2.80 2.69 0.39 1.82 4.32 1.04 2.93  735
## 6  1 14.20 1.76 2.45 15.2 112 3.27 3.39 0.34 1.97 6.75 1.05 2.85 1450
table(wine$V1)
## 
##  1  2  3 
## 59 71 48

Example 1: Multivariate Data

head(wine[2:6])
##      V2   V3   V4   V5  V6
## 1 14.23 1.71 2.43 15.6 127
## 2 13.20 1.78 2.14 11.2 100
## 3 13.16 2.36 2.67 18.6 101
## 4 14.37 1.95 2.50 16.8 113
## 5 13.24 2.59 2.87 21.0 118
## 6 14.20 1.76 2.45 15.2 112

Principal Components Analysis (PCA)

Principal Components Analysis (PCA)

Principal Components Analysis (PCA)

summary(wine[2:6])
##        V2              V3              V4              V5       
##  Min.   :11.03   Min.   :0.740   Min.   :1.360   Min.   :10.60  
##  1st Qu.:12.36   1st Qu.:1.603   1st Qu.:2.210   1st Qu.:17.20  
##  Median :13.05   Median :1.865   Median :2.360   Median :19.50  
##  Mean   :13.00   Mean   :2.336   Mean   :2.367   Mean   :19.49  
##  3rd Qu.:13.68   3rd Qu.:3.083   3rd Qu.:2.558   3rd Qu.:21.50  
##  Max.   :14.83   Max.   :5.800   Max.   :3.230   Max.   :30.00  
##        V6        
##  Min.   : 70.00  
##  1st Qu.: 88.00  
##  Median : 98.00  
##  Mean   : 99.74  
##  3rd Qu.:107.00  
##  Max.   :162.00

Principal Components Analysis (PCA)

## standardise the variables
standardisedconcentrations <- as.data.frame(scale(wine[2:14]))
head(standardisedconcentrations)
##          V2          V3         V4         V5         V6        V7        V8
## 1 1.5143408 -0.56066822  0.2313998 -1.1663032 1.90852151 0.8067217 1.0319081
## 2 0.2455968 -0.49800856 -0.8256672 -2.4838405 0.01809398 0.5670481 0.7315653
## 3 0.1963252  0.02117152  1.1062139 -0.2679823 0.08810981 0.8067217 1.2121137
## 4 1.6867914 -0.34583508  0.4865539 -0.8069748 0.92829983 2.4844372 1.4623994
## 5 0.2948684  0.22705328  1.8352256  0.4506745 1.27837900 0.8067217 0.6614853
## 6 1.4773871 -0.51591132  0.3043010 -1.2860793 0.85828399 1.5576991 1.3622851
##           V9        V10        V11        V12       V13         V14
## 1 -0.6577078  1.2214385  0.2510088  0.3611585 1.8427215  1.01015939
## 2 -0.8184106 -0.5431887 -0.2924962  0.4049085 1.1103172  0.96252635
## 3 -0.4970050  2.1299594  0.2682629  0.3174085 0.7863692  1.39122370
## 4 -0.9791134  1.0292513  1.1827317 -0.4263410 1.1807407  2.32800680
## 5  0.2261576  0.4002753 -0.3183774  0.3611585 0.4483365 -0.03776747
## 6 -0.1755994  0.6623487  0.7298108  0.4049085 0.3356589  2.23274072

Principal Components Analysis (PCA)

wine.pca <- prcomp(standardisedconcentrations)

wine.pca$sdev
##  [1] 2.1692972 1.5801816 1.2025273 0.9586313 0.9237035 0.8010350 0.7423128
##  [8] 0.5903367 0.5374755 0.5009017 0.4751722 0.4108165 0.3215244
sum((wine.pca$sdev)^2)
## [1] 13

Deciding How Many Principal Components to Retain

screeplot(wine.pca, type="lines")

Deciding How Many Principal Components to Retain (Another Approach)

(wine.pca$sdev)^2
##  [1] 4.7058503 2.4969737 1.4460720 0.9189739 0.8532282 0.6416570 0.5510283
##  [8] 0.3484974 0.2888799 0.2509025 0.2257886 0.1687702 0.1033779

K-Means Clustering

# Compute k-means with k = 3
set.seed(123)
km.res_1 <- kmeans(standardisedconcentrations, 3, nstart = 25)

# Print the results
print(km.res_1)
## K-means clustering with 3 clusters of sizes 51, 62, 65
## 
## Cluster means:
##           V2         V3         V4         V5          V6          V7
## 1  0.1644436  0.8690954  0.1863726  0.5228924 -0.07526047 -0.97657548
## 2  0.8328826 -0.3029551  0.3636801 -0.6084749  0.57596208  0.88274724
## 3 -0.9234669 -0.3929331 -0.4931257  0.1701220 -0.49032869 -0.07576891
##            V8          V9         V10        V11        V12        V13
## 1 -1.21182921  0.72402116 -0.77751312  0.9388902 -1.1615122 -1.2887761
## 2  0.97506900 -0.56050853  0.57865427  0.1705823  0.4726504  0.7770551
## 3  0.02075402 -0.03343924  0.05810161 -0.8993770  0.4605046  0.2700025
##          V14
## 1 -0.4059428
## 2  1.1220202
## 3 -0.7517257
## 
## Clustering vector:
##   [1] 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
##  [38] 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 3 3 1 3 3 3 3 3 3 3 3 3 3 3 2
##  [75] 3 3 3 3 3 3 3 3 3 1 3 3 3 3 3 3 3 3 3 3 3 2 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
## [112] 3 3 3 3 3 3 3 1 3 3 2 3 3 3 3 3 3 3 3 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
## [149] 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
## 
## Within cluster sum of squares by cluster:
## [1] 326.3537 385.6983 558.6971
##  (between_SS / total_SS =  44.8 %)
## 
## Available components:
## 
## [1] "cluster"      "centers"      "totss"        "withinss"     "tot.withinss"
## [6] "betweenss"    "size"         "iter"         "ifault"

What is hierarchical clustering?

Hierarchical Clustering

The algorithm works as follows:

Once this is done, it is usually represented by a dendrogram like structure. A dendrogram is a branching diagram that represents the relationships of similarity among a group of entities.

Iris Dataset

In the iris dataset, we have three different species of flowers.

# library(datasets)

data(iris)
summary(iris)
##   Sepal.Length    Sepal.Width     Petal.Length    Petal.Width   
##  Min.   :4.300   Min.   :2.000   Min.   :1.000   Min.   :0.100  
##  1st Qu.:5.100   1st Qu.:2.800   1st Qu.:1.600   1st Qu.:0.300  
##  Median :5.800   Median :3.000   Median :4.350   Median :1.300  
##  Mean   :5.843   Mean   :3.057   Mean   :3.758   Mean   :1.199  
##  3rd Qu.:6.400   3rd Qu.:3.300   3rd Qu.:5.100   3rd Qu.:1.800  
##  Max.   :7.900   Max.   :4.400   Max.   :6.900   Max.   :2.500  
##        Species  
##  setosa    :50  
##  versicolor:50  
##  virginica :50  
##                 
##                 
## 
iris$Species
##   [1] setosa     setosa     setosa     setosa     setosa     setosa    
##   [7] setosa     setosa     setosa     setosa     setosa     setosa    
##  [13] setosa     setosa     setosa     setosa     setosa     setosa    
##  [19] setosa     setosa     setosa     setosa     setosa     setosa    
##  [25] setosa     setosa     setosa     setosa     setosa     setosa    
##  [31] setosa     setosa     setosa     setosa     setosa     setosa    
##  [37] setosa     setosa     setosa     setosa     setosa     setosa    
##  [43] setosa     setosa     setosa     setosa     setosa     setosa    
##  [49] setosa     setosa     versicolor versicolor versicolor versicolor
##  [55] versicolor versicolor versicolor versicolor versicolor versicolor
##  [61] versicolor versicolor versicolor versicolor versicolor versicolor
##  [67] versicolor versicolor versicolor versicolor versicolor versicolor
##  [73] versicolor versicolor versicolor versicolor versicolor versicolor
##  [79] versicolor versicolor versicolor versicolor versicolor versicolor
##  [85] versicolor versicolor versicolor versicolor versicolor versicolor
##  [91] versicolor versicolor versicolor versicolor versicolor versicolor
##  [97] versicolor versicolor versicolor versicolor virginica  virginica 
## [103] virginica  virginica  virginica  virginica  virginica  virginica 
## [109] virginica  virginica  virginica  virginica  virginica  virginica 
## [115] virginica  virginica  virginica  virginica  virginica  virginica 
## [121] virginica  virginica  virginica  virginica  virginica  virginica 
## [127] virginica  virginica  virginica  virginica  virginica  virginica 
## [133] virginica  virginica  virginica  virginica  virginica  virginica 
## [139] virginica  virginica  virginica  virginica  virginica  virginica 
## [145] virginica  virginica  virginica  virginica  virginica  virginica 
## Levels: setosa versicolor virginica

Hierarchical Clustering

clusters <- hclust(dist(iris[, 3:4]))
plot(clusters)

Hierarchical Clustering

clusterCut <- cutree(clusters, 3)
table(clusterCut, iris$Species)
##           
## clusterCut setosa versicolor virginica
##          1     50          0         0
##          2      0         21        50
##          3      0         29         0

Hierarchical Clustering

ggplot(data = iris,
       aes(x = Petal.Length, y = Petal.Width, color = Species)) +
  geom_point()