Machine Learning (CS 5350/CS 6350) 25 Mar 2008
Linear Dimensionality Reduction
Often we want to find a representation of data from RDin some lower-dimenaional space, RK, for KD.
For K∈ {2,3}, this is useful for visualization. For other K, it’s useful if we believe that the data is noisy,
or not ideal for our learning algorithm (eg., kNN).
Have data matrix in X∈RN×D. Want to linearly project Xinto some Z∈RN×K. We don’t want to lose
much “information.”
Two ways of deriving PCA:
•Project Xonto basis vectors of highest variance
•Project Xin such a way that the reconstruction error is minimized
•Imagine data was generated by an K-dimensional Gaussian and then noisified into Ddimensions (wait
until probabilistic models time)
(1) is standard, hence the name “principle component analysis” (a component is a basis vector).
First, center Xso it has mean 0.
Now, supposed we want to find a single dimension to project down on to. Represent this dimension by
a vector u, so that Z=Xu. We want to find uso that V[Z] = V[Xu] is maximized. We have to be
careful to ensure that uis a unit vector; otherwise we get trivial uninteresting solutions. This turns into an
optimization problem:
max
u:||u||=1
V[Xu] = max
u:||u||=1
1
NX
nu>xn2
= max
u:||u||=1
1
N||Xu||2
To solve this, we construct the Lagrangian by adding a multiplier for the constraint ||u||2= 1 (changing to
squared norm doesn’t change the problem); we also throw out the 1
Nbecause it is a constant. Then we take
the gradient and set it equal to zero.
L(u, λ) = ||Xu||2−λ||u||2−1
∇uL= 2(X>X)u−2λu= 0
⇐⇒(X>X)u=λu
So now all we need to do is solve the above for u. Note that this has the form Au=λuwhere Ais defined as
X>X. The solution to this problem is a vector uthat is an eigenvector of Aand can be computed “easily”
(use eig or eigs in Matlab/Octave).
When you get eigenvectors/eigenvalues of a matrix Aof size D×D, you will get D-many pairs (ui,λi).
The λs are measures of the variance along the corresponding u, so we want to select the uiwith largest
corresponding λi.
In general, to project onto K > 1 dimensions, instead of choosing the single eigenvector with largest eigen-
value, we take the top K(they are guaranteed to be orthogonal).
1
