Study with the several resources on Docsity
Earn points by helping other students or get them with a premium plan
Prepare for your exams
Study with the several resources on Docsity
Earn points to download
Earn points by helping other students or get them with a premium plan
Material Type: Notes; Professor: Jacobs; Subject: Computer Science; University: University of Maryland; Term: Unknown 1989;
Typology: Study notes
1 / 7
This page cannot be seen from the preview
Don't miss anything!
that best approximates them.
PCA derivation
(this is all just taken from Duda, Hart and Stork)
Suppose we have a series of vectors, x1…xn, and we want to approximate them with a low-dimensional subspace. What is the best way to do this? If we want to approximate them with a 0 dimensional subspace, we can do this most accurately by approximating them by their mean, m. This is probably intuitive, but if not, Duda, Hart and Stork have a very nice proof (Eq. 80, p. 115).
Next we’ll consider find the best 1-dimensional subspace, written as: x_i is approximated by m+a_ie, where e is a unit vector indicating the direction of the space. Then our goal is to choose a_i and e to minimize:
J(a1, …, an, e) = sum ||(m+ake)-xk}}^
= sum ||ake – (xk – m)||^ = sum ak^2||e||^2 – 2 sum ak e(xk – m) + sum ||xk – m||^ ||e|| = 1. Taking the derivatives w.r.t. ak and setting them to 0 we get: 2ak – 2 e(xk-m) = 0,
ak = e(xk-m).
We can skip this derivation, and just say that of course we get the best choice of ak by projecting xk-m onto e.
Now, if we set ak = e(xk-m), we get J as a function of e J(e) = sum ak^2 – 2 sum ak^2 + sum ||xk-m||^ = - sum [e(xk-m)]^2 + sum ||xk-m||^ = - sum e(xk-m)(xk-m)e + sum ||xk-m||^ So we need to maximize eSe subject to ||e|| = 1. We do this with Lagrange multipliers. We set: U = eSe – lambda (ee – 1), differentiate w.r.t. e and set this to 0. We get: partial u/ partial e = 2Se – 2 lambda e, so Se = lambda e. So e is an eigenvector of S, and we can see that eSe is maximized when e is the eigenvector associated with the largest eigenvalue.
different objects, throw away variations that don’t.
1 2
1 2
1 2
2 , 1 2 , 2 2 , 3
1 , 1 1 , 2 1 , 3
2 2 2 , 3
2 2 , 2
2 2 , 1
2 2 1 , 3
2 1 , 2
2 1 , 1
1 1 2 , 3
1 2 , 2
1 2 , 1
1 1 1 , 3
1 1 , 2
1 1 , 1
1 2
1 2
2 2 2
2 1
2 2 2
2 1
1 1 2
1 1
1 1 2
1 1
n
n
n
m y
m m m
m x
m m m
y
x
y
x
m n
m m
m n
m m
n
n
n
n
z z z
y y y
x x x
s s s t
s s s t
s s s t
s s s t
s s s t
s s s t
v v v
u u u
v v v
u u u
v v v
u u u
Immediately apparent that u and v coordinates lie in a 4D linear subspace
