CSE 494 CSE/CBS 598 (Fall 2007): Numerical Linear Algebra for Data
Exploration— Tensor Decomposition
Instructor: Jieping Ye
1 Introduction
•Principal component analysis (PCA) reduces the dimensionality of a data set by finding a
new set of variables, smaller than the original set of variables. It retains most of the sample’s
variance.
•It is useful for the compression and classification of data.
•The new variables, called principal components (PCs), are uncorrelated, and are ordered by
the fraction of the total information each retains.
2 Computation of PCs
•Let Abe a n×mdata matrix in which each row is a data vector, each column represents a
variable. Ais centered: the estimated mean is subtracted from each row, so Ais of zero row
mean.
•Let wbe the column vector in IRmof projection weights that result in the largest variance
when the data Ais projected along w. We require wTw= 1.
•Projection of a vector uonto wis wTu=Pm
j=1 ujwj. Thus, pro jection of data along wis
Aw.
•The variance of the data after the projection by wis
(Aw)T(Aw) = wTATAw =wTC w
where C=ATAis the covariance matrix of the data (note that Ais centered).
•Compute the optimal projection w, which maximizes the variance subject to constraint
wTw= 1.
•Define the following Lagrange function:
f=wTCw −λ(wTw−1)
where λis the Lagrange multiplier.
•Take the directive of fin terms of wand set it to zero, we get
∂f
∂w = 2C w −2λw = 0.
This leads to the eigenvalue problem: Cw =λw, where C=ATA.
•Show that the solution is given by the singular values and singular vectors of A.
