2024-03-18

Background

Dimensionality Reduction is the transformation of data from a high-dimensional space into a low-dimensional space so that the low-dimensional representation retains meaningful properties of the original data.

Working in high-dimensional spaces can be undesirable for many reasons; raw data are often sparse as a consequence of the curse of dimensionality, and analyzing the data is usually computationally intractable (hard to control or deal with). Dimensionality reduction is common in fields that deal with large numbers of observations and/or large numbers of variables, such as signal processing, speech recognition, neuroinformatics, and bioinformatics.

Techniques

  • Linear Discriminant Analysis
  • Principal Component Analysis

Example of Linear Discriminant Analysis

What is Linear Discriminant Analysis?

Linear discriminant analysis (LDA) is a generalization of Fisher’s linear discriminant, a method used in statistics, pattern recognition and machine learning to find a linear combination of features that characterizes or separates two or more classes of objects or events.

LDA is closely related to analysis of variance (ANOVA) and regression analysis, which also attempt to express one dependent variable as a linear combination of other features or measurements.

How does LDA work?

Consider a set of observations, \({\vec{x}}\), for each sample with known class \({y}\). The classification problem is then to find a good predictor for the class \({y}\) of any sample of the same distribution (not necessarily from the training set) given only an observation \({\vec{x}}\).

LDA approaches the problem by assuming that the conditional probability density functions \({p({\vec {x}}|y=0)}\) and \({p({\vec {x}}|y=1)}\) are both the normal distribution with mean and covariance parameters \({\left({\vec {\mu }}_{0},\Sigma _{0}\right)}\) and \({\left({\vec {\mu }}_{1},\Sigma _{1}\right)}\), respectively.

How does LDA work? (cont.)

Under this assumption, the Bayes-optimal solution is to predict points as being from the second class if the log of the likelihood ratios is bigger than some threshold T, such that: \({{\frac {1}{2}}({\vec {x}}-{\vec {\mu }}_{0})^{\mathrm {T} }\Sigma _{0}^{-1}({\vec {x}}-{\vec {\mu }}_{0})+{\frac {1}{2}}\ln |\Sigma _{0}|\\-{\frac {1}{2}}({\vec {x}}-{\vec {\mu }}_{1})^{\mathrm {T} }\Sigma _{1}^{-1}({\vec {x}}-{\vec {\mu }}_{1})-{\frac {1}{2}}\ln |\Sigma _{1}|\ >\ T}\)

Then, LDA makes the further simplifying assumption that the class covariances are identical and are full rank. This causes

\({\displaystyle {\vec {x}}^{\mathrm {T} }\Sigma _{0}^{-1}{\vec {x}}={\vec {x}}^{\mathrm {T} }\Sigma _{1}^{-1}{\vec {x}}}\)

and

\({\displaystyle {\vec {x}}^{\mathrm {T} }{\Sigma _{i}}^{-1}{\vec {\mu }}_{i}={{\vec {\mu }}_{i}}^{\mathrm {T} }{\Sigma _{i}}^{-1}{\vec {x}}}\),

simplifying the above equation.

Example of Principal Component Analysis

Example of 3d PCA - Code (pt. 1)

data("iris")

X <- subset(iris, select = -c(Species))

prin_comp <- prcomp(X, rank = 3)

components <- prin_comp[["x"]]
components <- data.frame(components)
components$PC2 <- -components$PC2
components$PC3 <- -components$PC3
components = cbind(components, iris$Species)

Example of 3d PCA - Code (pt. 2)

tot_explained_variance_ratio <- 
  summary(prin_comp)[["importance"]]['Proportion of Variance',]
tot_explained_variance_ratio <- 
  100 * sum(tot_explained_variance_ratio[1:3])

fig <- plot_ly(components, x = ~PC1, 
               y = ~PC2, 
               z = ~PC3, 
               color = ~iris$Species, 
               colors = c('#636EFA','#EF553B','#00CC96') ) %>%
  add_markers(size = 12)

fig <- fig %>%
  layout(
    title = paste('Total Explained Variance = ',
                  tot_explained_variance_ratio),
    scene = list(bgcolor = "#e5ecf6")
)

Example of 3d PCA - Plot

What is PCA?

The goal of PCA is to explain most of the variability in the data with a smaller number of variables than the original data set.

PCA provides a tool to do just this. It finds a low-dimensional representation of a data set that contains as much of the variation as possible. The idea is that each of the n observations lives in p-dimensional space, but not all of these dimensions are equally interesting.

How does PCA Work?

The first principal component of a data set \(X_1, X_2, ..., X_p\) is the linear combination of the features

\(Z_{1} = \phi_{11}X_{1} + \phi_{21}X_{2} + ... + \phi_{p1}X_{p}\)

that has the largest variance and where \(\phi_1\) is the first principal component loading vector, with elements \(\phi_{12}, \phi_{22}, ... , \phi{p2}\). Each vector is normalized, such that the squared sum of each vector’s elements is equal to 1. The loadings of each vector, \(\phi\) that maximizes the variance is calculated by using linear algebra. It turns out that the eigenvector corresponding to the largest eigenvalue of the covariance matrix is the set of loadings that explains the greatest proportion of the variability.

How does PCA Work? (cont.)

After the first principal component, \(Z_{1}\) is determined, we calculate the second principal component, \(Z_{2}\). The second principal component is the linear combination of \(X_1, X_2, ..., X_p\), that has the maximal variance out of all linear combinations that are uncorrelated with \(Z_{1}\). The second principal component takes the form:

\(Z_{2} = \phi_{12}X_{1} + \phi_{22}X_{2} + ... + \phi_{p2}X_{p}\)

From here, the calculations continue, but we are usually only interested in the first two Principal Components.

Works Cited: