Optimizing y Prediction with Kernel Functions & ǫ-Insensitive Loss in SVR, Slides of Engineering Dynamics

Download Optimizing y Prediction with Kernel Functions & ǫ-Insensitive Loss in SVR and more Slides Engineering Dynamics in PDF only on Docsity!

• We have been discussing SVM method for learningclassifiers. PR NPTEL course – p.1/

• We have been discussing SVM method for learningclassifiers. - The basic idea is to transform the feature space andlearn a linear classifier in the new space. PR NPTEL course – p.2/

• We have been discussing SVM method for learningclassifiers. - The basic idea is to transform the feature space andlearn a linear classifier in the new space. - Using Kernel functions we can do this mappingimplicitly. - Thus Kernels give us an elegant method to learnnonlinear classifiers. PR NPTEL course – p.4/

• We have been discussing SVM method for learningclassifiers. - The basic idea is to transform the feature space andlearn a linear classifier in the new space. - Using Kernel functions we can do this mappingimplicitly. - Thus Kernels give us an elegant method to learnnonlinear classifiers. - We can use the same idea in regression problemsalso. PR NPTEL course – p.5/

Kernel Trick

• We use φ

m

→ H

to map pattern vectors into appropriate high dimensional space.

• Kernel fn allows us to compute innerproducts in

H

implicitly without using (or even knowing) φ

. PR NPTEL course – p.7/

Kernel Trick

• We use φ

m

→ H

to map pattern vectors into appropriate high dimensional space.

• Kernel fn allows us to compute innerproducts in

H

implicitly without using (or even knowing) φ .

• Through kernel functions, many algorithms that useonly innerproducts can be implicitly executed in a highdimensional

H

. ( e.g., Fisher discriminant, regression etc). PR NPTEL course – p.8/

Support Vector Regression

• Now we consider the regression problem. - Given training data

X

1 , y 1

X

n , y n

X

i

m , y i

, want to find ‘best’ function to predict y given

X

. PR NPTEL course – p.10/

Support Vector Regression

• Now we consider the regression problem. - Given training data

X

1 , y 1

X

n , y n

X

i

m , y i

, want to find ‘best’ function to predict y given

X

.

• We search in a parameterized class of functions g

X, W

w 1 φ 1

X

w m ′^ φ m ′^

X

b

W

T

X

b , where φ i

m

are some chosen functions. PR NPTEL course – p.11/

• If we choose, φ i

X

x i (and hence, m

m ′ ) then it is a linear model.

• Denoting

Z

X

m ′ , we are essentially learning a linear model in a transformed space. PR NPTEL course – p.13/

• If we choose, φ i

X

x i (and hence, m

m ′ ) then it is a linear model.

• Denoting

Z

X

m ′ , we are essentially learning a linear model in a transformed space.

• This is in accordance with the basic idea of SVMmethod. PR NPTEL course – p.14/

• If we choose, φ i

X

x i (and hence, m

m ′ ) then it is a linear model.

• Denoting

Z

X

m ′ , we are essentially learning a linear model in a transformed space.

• This is in accordance with the basic idea of SVMmethod. - We want to formulate the problem so that we can usethe Kernel idea. - Then, by using a kernel function, we never need tocompute or even precisely specify the mapping

. PR NPTEL course – p.16/

Loss function

• As in a general regression problem, we need to find W to minimize

i

L

y i , g

X

i

, W

where

L

is a loss function. PR NPTEL course – p.17/

Loss function

• As in a general regression problem, we need to find W to minimize

i

L

y i , g

X

i

, W

where

L

is a loss function.

• This is the general strategy of empirical riskminimization. - We consider a special loss function that allows us touse the kernel trick. PR NPTEL course – p.19/

-insensitive loss

• We employ ǫ -insensitive loss function: L ǫ

y i , g

X

i

, W

If

y i

g

X

i

, W

< ǫ

y i

g

X

i

, W

ǫ otherwise Here, ǫ is a parameter of the loss function. PR NPTEL course – p.20/