Principal Component Analysis (PCA)

• Dimensionality Reduction Technique (Variable Reduction Strategy)
• A tool for data visualization and preprocessing
• Transforms a larger number of correlated variables into a much smaller set of uncorrelated variables called principal components
• Captures as much information in the original variables as possible
• Less Noisy Results in the end
• Essence: When presented with a large set of correlated variables, principal components allow us to summarize the dataset into a smaller number of representitive variables that collectively explain most of the variablility in the original set

• Consider a case of visualizing n observations with p features
• Scatterplots of data in 2D is useful in capturing correlation
• However, as p tends to be large, we need alternatives
• Specifically, we want to find a low-dimensional representation of the data that captures as much information as possibe
• PCA seeks a subset of original dimensions that are as interesting as possible, where the concept of interesting is measured by the amount that the observations vary across the dimension
• Each of the dimensions found by PCA is a linear combination of the p features, so this is technically not a form of feature selection.

• In general, we would like to use the smallest number of principal components required to get a good understanding of the data
• However, there is no single threshold that answers this question
• Arguably, the best way to do this is to visualize the data in a scree plot
• It is simply a plot of the cumulative PVE
• We examine the elbow effect: see the scree plot when the percent variation drops off, such that addition prinicipal components don’t really add a significant amount of variance

#clearing the workspace prior to start
rm(list = ls())

#loading up libraries
library(stats)
library(graphics)

#reading the dataset
data <- read.csv("~/Desktop/data.csv")

#checking the dimension of data
dim(data)

[1] 768  10

• Firstly start by looking at the axis, PC1 on the x and PC2 on the y. The arrows show how they are moving across the two dimensions
• We see that two of the variables namely OverallHeight & RoofArea are moving along the first principle component, where as rest of the variables are all moving along both the principal components
• The rown umbers colored in black show how each observation varies across the PC directions. For example, 48th observation points to large roof area where as 768th observation points to large surface area, large wall area, large glazing area and high ga distribution

• The $sdev attribute outputs the standard deviation of each component; the variance explained by each component can computed by squaring these:

pr.var <- pr.out$sdev^2
pr.var

 [1] 5.222885e+00 1.533386e+00 1.218894e+00 1.000177e+00 8.047700e-01
 [6] 1.630918e-01 3.308280e-02 1.935870e-02 4.353904e-03 4.582035e-30

• We can see that maximum variability in data is captured by the very first principal component
• We see a decresing trend in the proportion of variance explained by the rest of the principal components
• Principal components from 7-10 explain least variance in the data

library(FactoMineR)
PCA(data)

**Results for the Principal Component Analysis (PCA)**
The analysis was performed on 768 individuals, described by 10 variables
*The results are available in the following objects:

   name               description                          
1  "$eig"             "eigenvalues"                        
2  "$var"             "results for the variables"          
3  "$var$coord"       "coord. for the variables"           
4  "$var$cor"         "correlations variables - dimensions"
5  "$var$cos2"        "cos2 for the variables"             
6  "$var$contrib"     "contributions of the variables"     
7  "$ind"             "results for the individuals"        
8  "$ind$coord"       "coord. for the individuals"         
9  "$ind$cos2"        "cos2 for the individuals"           
10 "$ind$contrib"     "contributions of the individuals"   
11 "$call"            "summary statistics"                 
12 "$call$centre"     "mean of the variables"              
13 "$call$ecart.type" "standard error of the variables"    
14 "$call$row.w"      "weights for the individuals"        
15 "$call$col.w"      "weights for the variables"