Classification and Regression - Introduction to Pattern Recognition - Lecture Slides, Slides of Design and Analysis of Algorithms

Download Classification and Regression - Introduction to Pattern Recognition - Lecture Slides and more Slides Design and Analysis of Algorithms in PDF only on Docsity!

Recap

We have been considering learning linear models forclassification and regression.

PR NPTEL course – p.1/

Recap

We have been considering learning linear models forclassification and regression.

We have discussed linear least squares method.

PR NPTEL course – p.2/

Recap

We have been considering learning linear models forclassification and regression.

We have discussed linear least squares method.

We have also looked at some generalizations such aslogistic regression.

In this class we discuss some more issues connectedwith least squares method.

PR NPTEL course – p.4/

Linear Regression

PR NPTEL course – p.5/

Linear Regression

The training data: {

X

i

, y

i

, i

, n

,

X

i

d

, y

i

i

.

The objective is to learn a linear model:

y

X

f

X

W

T

X

w

0

PR NPTEL course – p.7/

Linear Regression

The training data: {

X

i

, y

i

, i

, n

,

X

i

d

, y

i

i

.

The objective is to learn a linear model:

y

X

f

X

W

T

X

w

0

(or,

f

X

W

T

X

, when using augumented vectors).

PR NPTEL course – p.8/

We saw that this framework is also useful for learninglinear classifiers. For example, we can take y

i

and use sign of

f

X

to make the

classification decision.

PR NPTEL course – p.10/

We saw that this framework is also useful for learninglinear classifiers. For example, we can take y

i

and use sign of

f

X

to make the

classification decision.

The criterion is to minimize mean square error:

J

W

n

i

=

W

T

X

i

y

i

2 PR NPTEL course – p.11/

The minimizer of

J

is given by

W

∗

A

T

A

−

1

A

T

Y

where

A

is a

n

×

d

matrix whose rows are the

X

i

and

Y

is

n

×

vector whose components are

y

i

.

PR NPTEL course – p.13/

The minimizer of

J

is given by

W

∗

A

T

A

−

1

A

T

Y

where

A

is a

n

×

d

matrix whose rows are the

X

i

and

Y

is

n

×

vector whose components are

y

i

.

In the general case of fixed basis functions, the

i

th

row

of matrix

A

would be

[

φ

0

X

i

φ

d

′

X

i

)]

. PR NPTEL course – p.14/

In most applications, our observations or data wouldbe noisy.

We can take the

X

i

to be fixed and the observed

y

i

to

be random.

PR NPTEL course – p.16/

In most applications, our observations or data wouldbe noisy.

We can take the

X

i

to be fixed and the observed

y

i

to

be random.

Often, we get data by measuring

y

i

for specific value

of

X

i

. Hence this is a useful scenario.

PR NPTEL course – p.17/

In most applications, our observations or data wouldbe noisy.

We can take the

X

i

to be fixed and the observed

y

i

to

be random.

Often, we get data by measuring

y

i

for specific value

of

X

i

. Hence this is a useful scenario. -

Now the

W

∗

obtained through linear least squares

regression would also be random.

Hence we would like to know its variance.

PR NPTEL course – p.19/

We assume that noise corrupting different

y

i

are iid

and zero-mean.

PR NPTEL course – p.20/