The following face modeling problem was discussed in one of the lectures in the edX course: ColumbiaX: CSMM.102x Machine Learning.
For this experiment the CBCL Face Database from the MIT Center For Biological and Computation Learning is used. This dataset contains 6977 face images to be trained and 24045 images to be tested with svm (each image is of size 21x21 pixels), the images in these two files are combined and then the first 10000 images are used for modeling the faces with NMF, Kmeans/VQ and SVD, after that the models are compared.

The following figure shows the first 625 images from the dataset.

k-means (Modeling the images with VQ)

The following figures show the averages of full faces learnt by K-means with different numbers (k=25,64,100) of clusters respectively. As can be seen some of the cluster centroids represent good quality average images, where for some others the average image is very noisy and can not be identified as face at all.

k=25

k=25

k=64

k=64

k=100

k=100

SVD (Modeling images with first k orthonormal vectors V)

SVD Finds the singular value decomposition of the image matrix. As can be seen from the below figure with k=100 images represeting the top k singular vectors (from right to left columnwise), the results not interpretable because of negative values and orthogonality constraint.

k=100

k=100

Using NMF with Squared error objective

The following figures show the iterative coordinate-ascent algorithm that is going to be used.

NMF NMF

NMF learns a “parts-based” representation. Each column captures something interpretable. This is a result of the nonnegativity constraint. The follwoing figures show the W matrix learnt by squared-error objective NMF, for different values of k=25, 64, 100 and the k columns of W are shown as images.

k=25

k=25

k=64

k=64

k=100

k=100