Machine Learning (CS 5350/CS 6350) 27 Feb 2007
Low-dimensional Representations
Often we want to find a representation of data from RDin some lower-dimenaional space, RF, for FD.
For F∈ {2,3}, this is useful for visualization. For other F, it’s useful if we believe that the data is noisy, or
not ideal for our learning algorithm (eg., kNN).
There are two varieties of dimensionality reduction techniques: linear and non-linear. We will talk about
one example of each.
Linear Dimensionality Reduction: PCA
Have data matrix in X∈RN×D. Want to linearly project Xinto some Y∈RN×F. We don’t want to lose
much “information.”
Two ways of deriving PCA:
1. Project Xonto basis vectors of highest variance
2. Imagine data was generated by an F-dimensional Gaussian and then noisified into Ddimensions
(1) is standard, hence the name “principle component analysis” (a component is a basis vector).
First, center Xso it has mean 0.
Now, what do we want? We want Ysuch that Y Y >is diagonal (so the basis is orthogonal) and so that
Y=ZX . From this, we get:
Y Y >= (ZX )(ZX )>
=ZX X>Z>
=Z(XX >)Z>
Now, linear algebra (i.e., magic) tells us that if Ais a symmetric matrix, then A=EDE>, where Eis a
matrix of eigenvectors of Aand Dis a diagonal matrix of eigenvalues.
Letting A=XX >, we select Zto be the matrix of eigenvectors of XX >and Dbe the eigenvalues, so:
Z(XX >)Z>=Z(Z>DZ)Z>
= (ZZ >)D(ZZ >)
= (ZZ −1)D(ZZ −1)
=D
using the fact that the inverse of an orthogonal matrix is its transpose.
All you really need to know is that you first center your data, find the eigenvectors corresponding to the top
Feigenvalues, and then use these as the new basis for the data.
You can additionally show that PCA minimizes the reconstruction mean-squared error, within the constraint
of being an orthogonal linear projection.
1
