Applying Linear PCA vs. Kernel PCA (with Gaussian Kernel) for dimensionality reduction on a few datasets in R

In this article, both the linear PCA and the kernel PCA will be applied on a few shape datasets, to show whether the structuter of the data (in terms of different clusters) in higher dimensions are presevered in the lower dimension or not for both the methods.

1. For the linear PCA, as usual, the dataset is first z-score normalized and then the eigen-analysis of the covariance matrix is done. Then to reduce the dimension, the dataset is projected onto the first few principal components (dominant eigenvectors of the covariance matrix).

2. For the kernel PCA, Gaussian Kernel is used to compute the distances between the datapoints and the Kernel matrix is computed, then normalized. Next the eigen-analysis of the Kernel matrix is done. Then to reduce the dimension, the first few dominant eigenvectors of the kernel matrix are chosen, which implicitly represent the data already projected on the principal components of the infinite dimensional space. The next figure shows the algorithms for the two methods.

3. The following figures / animations show the results obtained with both the dimensionality reduction methods (and for Gaussian kernel PCA with different bandwidth sigma) on a few 2D and 3D datasets (many of them taken from https://cs.joensuu.fi/sipu/datasets/*). As we can see, in many of the cases, the Kernel PCA can retain the non-linear structure* of the data (in terms of well-separated clusters), where the linear PCA can’t, although Kernel PCA is much slower than PCA when we have large number of datapoints.