Dimension Reduction: Principal Component Analysis and Fisher Linear Discriminant - Prof. S, Study notes of Statistics

Dimension reduction techniques, focusing on principal component analysis (pca) and fisher linear discriminant. Pca is a widely used method for reducing the dimensionality of data, while fisher linear discriminant aims to find the best linear separator between classes. An overview of these techniques, their derivation, and their applications.

Typology: Study notes

Pre 2010

Uploaded on 08/27/2009

koofers-user-174
koofers-user-174 🇺🇸

10 documents

1 / 11

Toggle sidebar

This page cannot be seen from the preview

Don't miss anything!

bg1
1
Lecture 4-6: Dimension Reduction
One common method is to assume smooth density functions in empty spaces, e.g.
Lecture notes Stat 231-CS276A S.C. Zhu
Dimension reduction techniques
The other method is to reduce the dimension of the feature space, for example by projecting
a feature vector to a lower dimensional space.
Common techniques for dimension reduction:
1. Principle component analysis (PCA)
2. Fisher linear discriminant analysis
3. Independent component analysis (ICA)
4 Multi
dimensional scaling (MDS)
Lecture notes Stat 231-CS276A, S.C. Zhu
4
.
Multi
-
dimensional scaling (MDS)
5. Over-complete bases coding
6. Transformed component analysis (TCA)
pf3
pf4
pf5
pf8
pf9
pfa

Partial preview of the text

Download Dimension Reduction: Principal Component Analysis and Fisher Linear Discriminant - Prof. S and more Study notes Statistics in PDF only on Docsity!

Lecture 4-6: Dimension Reduction

One common method is to assume smooth density functions in empty spaces, e.g.

Lecture notes Stat 231-CS276A S.C. Zhu

Dimension reduction techniques

The other method is to reduce the dimension of the feature space, for example by projecting

a feature vector to a lower dimensional space.

Common techniques for dimension reduction:

1. Principle component analysis (PCA)

2. Fisher linear discriminant analysis

3. Independent component analysis (ICA)

4 Multi dimensional scaling (MDS)

4. Multi-dimensional scaling (MDS)

5. Over-complete bases coding

6. Transformed component analysis (TCA)

PCA

The principal component analysis (PCA), also called Karhunen-Loeve transform in functional space,

is widely used for dimension reduction. In vision, it becomes popular by the eigen-face example.

There are many ways to derive PCA, here we study it from the perspective of dimension reduction.y y , y p p

Given: a number of n samples { x 1 ,x 2 , …, x n } in d-space.

Objective: project it in a d’< d space, that is, approximate each vector x k by

Criterion: minimize the sum of squared error.

k

d

i

m + ∑ aki ei → x

Lecture notes Stat 231-CS276A, S.C. Zhu

n

k

k

d

i

J d mae m akiei x

What is the intrinsic dimension of the dataset?

For a human face image of 128 x 128 pixels, what is the dimension of all images of a same person

under varying illumination? It must be quite small.

Lecture notes Stat 231-CS276A S.C. Zhu

A cosmology picture

The real dataset may be a mixture of many subspaces of different dimensions.

A clustering technique is to separate subspaces.

Lecture notes Stat 231-CS276A S.C. Zhu

PCA

The result of minimizing the error is:

m is the sample mean,

e 1

e

p ,

ei is the i-th largest eigen-vector of the co-variance matrix

aki is the projection of x k to ei

The book derives this in three separate steps. As this is so well-known, we

don’t unfold the details.

e 2

Example on face representation

Lecture notes Stat 231-CS276A, S.C. Zhu

400 images each labeled with 122 points.

Eigen-vectors for Geometry and Photometry

Fisher linear discriminant

Lecture notes Stat 231-CS276A, S.C. Zhu

Fisher linear discriminant

Lecture notes Stat 231-CS276A Fall, 2005, S.C. Zhu

These are 1D variables

Fisher linear discriminant

This is a typical criterion used in almost all discriminative methods

Lecture notes Stat 231-CS276A, S.C. Zhu

This is a typical criterion used in almost all discriminative methods.

Fisher linear discriminant

Multiple discriminant analysis

Lecture notes Stat 231-CS276A, S.C. Zhu

Multiple discriminant analysis

Examples in applications

Lecture notes Stat 231-CS276A, S.C. Zhu

Example

Task: glass vs no-glass in a face image

Compare the principal conponent analysis and Fisher discriminant analysis

a face image a Fisher-face image

Ref. P.N. Belhumeur et al, “Eigenfaces vs FisherFaces…”, IEEE Trans. PAMI Vol19, no7, 1997.