Principal component analysis (PCA), like MDS, is a dimension reduction technique.
It is a way of representing high-dimensional data in lower dimensions, in order to:
Visualize
Identify characteristics of similar observations
Understand underlying structure
While classic metric multidimensional scaling \(\equiv\) PCA, in general the motivation is a bit different.
MDS: find a low-dimensional representation that preserves distance
PCA: find a low-dimensional representation that preserves explained variability
PCA is a linear combination of X’s
Starting point: \(n\times p\) data matrix \(X\), want to represent in \(k\leq p\) dimensions
Let \(X_{ij}\) represent the \(j^{th}\) column for the \(i^{th}\) observation
\(j=1,2,...,p\)
\(i = 1,2,...,n\)
For the \(i^{th}\) data observation, the \(k^{th}\)principal component, (also known as score), \(PC_{ik}\), is a linear combination (recall: weighted sum) of the \(X\)’s:
The vector of weights \((v_{k1},v_{k2},...,v_{kp})\) are called the loadings of the \(k^{th}\) principal component, where \(\sum_{j=1}^pv_{kj}^2 = 1\) (i.e., each vector of loadings has \(||v_k||_2 = 1\).)
\(PC_{1:n,i}\) and \(PC_{1:n, j}\) are uncorrelated
\(\color{blue}{PC_{1:n,1}}=X\color{blue}{v_{1,1:p}}\), \(\color{orange}{PC_{1:n,2}}=X\color{orange}{v_{2,1:p}}\), etc.
\(\color{blue}{\sum_{j=1}^pv_{1j}^2} =\color{orange}{\sum_{j=1}^pv_{2j}^2}= 1\), etc.
\(\color{blue}{PC_{1:n,1}}\) and \(\color{orange}{PC_{1:n,2}}\) are uncorrelated with each other and all other PCs
Goal of PCA
Find loadings to produce PCs that:
Maximize explained variation in reduced \(k\)-dimensional space (what does this mean?!) ;
Are uncorrelated with each other, i.e. each score tells us something entirely unique about the \(k\)-dimensional representation.
Put together, this means:
\(\color{blue}{PC_{1:n,1}}\) explains as much variability in \(X\) as possible
\(\color{orange}{PC_{1:n,2}}\) explains as much remaining variability in \(X\) as possible that hasn’t already been explained by \(\color{blue}{PC_{1:n,1}}\)
\(\color{black}{PC_{1:n,3}}\) explains as much remaining variability in \(X\) as possible that hasn’t already been explained by \((\color{blue}{PC_{1:n,1}}, \color{orange}{PC_{1:n,2}})\)
etc.
Starting small
Let’s gain some intuition behind what it means to preserve “variability” by taking a look at the cars data set.
Here, of course, we have \(p=2\) dimensional data, so “dimension reduction” means going down to \(k=1\) dimension:
Total variability in 2 dimensions: 27.959+664.061 = 1603.9
So, which of the 1D components explains the greatest proportion of this variability?
var(XV1)
XV
XV 27.95918
var(XV2)
XV
XV 664.0608
var(XV3)
XV
XV 455.9569
…so \(V_2\) is the best loading matrix, and PC2 is the best score…right??
But it’s not fair!
Distance is just measured on a larger scale than speed!
It is driving the “total variability,” so any linear combination that weights distance more than speed will explain a larger proportion of variability.
If I measure speed in “miles per minute”, now giving all the weight to speed explains more total variance:
It turns out \(V_3\) is best of these 3, for our scaled data example.
Is \(V_3\) the best there is, or is there one better?
How do we find the optimal loadings, in general?
SVD/eigen analysis of \(R\)!
Given mean-centere and scaled data matrix \(X_S\), the optimal loadings for a reduced \(k\)-dimensional representation are the first \(k\) vectors from \(V\), from singular value decomposition:
\[X_S = UDV^T\]
Finding the optimal loadings for a \(2\rightarrow 1\) transformation:
V <-svd(cars_scaled)$vV[,1]
[1] 0.7071068 0.7071068
Have we seen these before?
c(sqrt(0.5), sqrt(0.5))
[1] 0.7071068 0.7071068
These are also the first \(k\) eigenvectors of the correlation matrix!
eigen(cor(cars))$vector[,1]
[1] 0.7071068 0.7071068
Finding one score
The \(k^{th}\)score or principal component, \(S_k\), can also be found with the SVD ingredients:
If \(k=p\) (i.e., number of PCs = dimension of data matrix) then:
Scores are a rotation of the scaled data, such that \(Cor(PC1, PC2) = 0\).
Maximum variability along first PC; second most along second PC; etc.
3D example: Prehistoric goblets
Suppose you have been asked to analyze data on 25 prehistoric goblets from Thailand (Professor C.F.W. Higham, University of Otago, as taken from Manly, B.F.J. 1986. Multivariate Statistical Methods: A Primer. Chapman and Hall, London, 159pp.)
PC1 alone explains 74.27% of the total variability (amount of information) in these 3 variables
PC2 explains an additional 22.40%
PC1 and PC2 together explain (74.27+22.40)=96.67% (!!) of the variability \(\implies\) we lose very little information representing these 3 variables in 2 dimensions
PC3 adds virtually no information, explaining the remaining 3.33%.
Variance of PCs = eigenvalues of \(R\)!
We’ve already seen how the loadings are the eigenvectors of \(R\), the correlation matrix (aka covariance of scaled data)
The eigenvalues of \(R\) are exactly equal to amount of variability explained by each PC:
Appear to give relatively equal weight to MouthWidth, TotalWidth, and TotalHeight
Large PC1 = wider, taller goblets
Small PC1 = narrower, shorter goblets
Loadings for PC2 (0.82, -0.12, -0.56):
Appears to contrast MouthWidth with TotalHeight
Large PC2 = large MouthWidth but small TotalHeight: short, stout goblets
PC1 vs PC2
Let’s see if we’re right by plotting just the first 2 PCs, using size and color to represent MouthWidth and TotalHeight (scatterplot created with ggplot, ellipses/arrows/annotations added with PPT):
PCA should always be done on scaled \(X\). Assume \(X\) scaled for rest of summary.
Each PC/score is a linear combination of the \(X\) columns.
All PCs are uncorrelated.
PC1 explains as much total variability in \(X\) as possible, PC2 as much remaining that has not been explained by PC1, etc.
There are up to \(p\) possible PCs. If all \(p\) are considered, the matrix of PCs is a rotation of \(X\) to be an “uncorrelated \(p\)-dimensional (American) football.”
Each PC is a linear combination of \(p\) data columns. The linear weights are called loadings.
Summary (continued)
Loadings are:
\(V\) from SVD of \(X\)
Eigenvectors of \(R\)
First \(k\) PCs:
First \(k\) columns of \(UD\) from SVD of \(X\)
Fist \(k\) columns of \(XV\)
Variance of \(PC_k\) = eigenvalue \(k\) of \(R\)
Concept question 1
Consider the below plots of four mean-centered and scaled data sets:
The first PC for each data set is found and the variance of each PC1 shown below:
var(firstpc_1)
[1] 1.750179
var(firstpc_2)
[1] 1.942512
var(firstpc_3)
[1] 1.116269
var(firstpc_4)
[1] 1.377787
Match the PC1 to the data set from which it comes.
Concept question 2
Consider the below plots of four mean-centered and scaled data sets:
The second PC for each data set is found and the variance of each PC2 shown below:
var(secondpc_1)
[1] 0.2498212
var(secondpc_2)
[1] 0.05748802
var(secondpc_3)
[1] 0.8837308
var(secondpc_4)
[1] 0.6222128
Match the PC2 to the data set from which it comes.
Concept question 3
Consider the plot below:
4 possible sets of loadings for the first PC are:
A. (0.43, 0.90)
B. (0.90, -0.43)
C. (0.71, 0.71)
D. (-0.43, 0.90)
Which of these are the loadings for the first PC from this data set?
