Resulting Optimization Problem - Introduction to Pattern Recognition - Lecture Slides, Slides of Advanced Algorithms

Download Resulting Optimization Problem - Introduction to Pattern Recognition - Lecture Slides and more Slides Advanced Algorithms in PDF only on Docsity!

Recap^ •^ We have been discussing the SVM method.

PR NPTEL course – p.1/

Recap^ •^ We have been discussing the SVM method.^ •^ We have looked at the formulation of optimalseparating hyperplane.

PR NPTEL course – p.2/

Recap^ •^ We have been discussing the SVM method.^ •^ We have looked at the formulation of optimalseparating hyperplane.^ •^ The resulting optimization problem is sloved moreeasily in the dual form.^ •^ By using a kernel function we can implicitly map thefeature vectors to a high dimensional space and findan optimal hyperplane there.

PR NPTEL course – p.4/

The optimization problem for SVM^ •^ The optimal hyperplane is a solution of the followingconstrained optimization problem:

PR NPTEL course – p.5/

The optimization problem for SVM^ •^ The optimal hyperplane is a solution of the followingconstrained optimization problem:Find^ W^ ∈ ℜ

m,^ b^ ∈ ℜ^ to^1 T^ minimize W^ W^2 T^ subject to y(W^ X+^ ii^

b)^ ≥^1 ,^ i^ = 1,... , n

-^ Quadratic cost function and linear (inequality)constraints.

PR NPTEL course – p.7/

The optimization problem for SVM^ •^ The optimal hyperplane is a solution of the followingconstrained optimization problem:Find^ W^ ∈ ℜ

m,^ b^ ∈ ℜ^ to^1 T^ minimize W^ W^2 T^ subject to y(W^ X+^ ii^

b)^ ≥^1 ,^ i^ = 1,... , n

-^ Quadratic cost function and linear (inequality)constraints. •^ Kuhn-Tucker conditions are necessary and sufficient.Every local minimum is global minimum.

PR NPTEL course – p.8/

-^ The dual of this problem is:^ maxn^ μ∈ℜ

n∑q(μ) =^ μ−i^ i= n∑ 1 μμyyXij^ ij^2 i,j= TXji^

subject to^ μ

≥^0 ,^ i^ = 1,... , n,i

n∑^ yμ= 0ii^ i=

-^ Then the final solution is:∑^ ∗^ ∗^ W^ =^ μ^ i^

∗^ yX, b=^ y−Xiij^ T∗W^ , j^ such thatj^

μ>^0 j^ PR NPTEL course – p.10/

-^ This problem has no solution if training data are notlinearly separable.

PR NPTEL course – p.11/

-^ This problem has no solution if training data are notlinearly separable. •^ Hence, in general, we use slack variables. •^ The optimization problem now is^ min^ W,b,ξ

1 T^ W^ W^ +^ C 2

n∑^ ξi i= subject to^ y

T^ (W X+^ b)^ ≥ii^ 1 −^ ξ,^ i^ = 1,... , ni ξ≥^0 ,^ i^ = 1,... , ni^

PR NPTEL course – p.13/

-^ The dual problem now is:^ max^ μ^

n∑q(μ) =^ μ−i^ i= n∑ 1 μμyyXij^ ij^2 i,j= TXji^

subject to^0

≤^ μ≤^ C,^ i^ = 1i^

n∑,... , n,^ yi i= μ= 0i^ PR NPTEL course – p.14/

-^ The dual problem now is:^ max^ μ^

n∑q(μ) =^ μ−i^ i= n∑ 1 μμyyXij^ ij^2 i,j= TXji^

subject to^0

≤^ μ≤^ C,^ i^ = 1i^

n∑,... , n,^ yi i= μ= 0i^

-^ The only difference – upper bound on

μ.i

-^ We solve the dual and the final optimal hyperplane is∑^ ∗^ ∗^ W^ =^ μy^ i^

X,ii ∗ T∗ b= y− XW^ , j^ such thatj j

0 < μ< C.j^ PR NPTEL course – p.16/

Non-linear discriminant functions^ •^ The idea is that we map the feature vectors into a highdimensional space and find a linear classifier there.

PR NPTEL course – p.17/

Non-linear discriminant functions^ •^ The idea is that we map the feature vectors into a highdimensional space and find a linear classifier there.^ •^ In general, we can use a mapping,

′m m φ : ℜ→ ℜ^. ′m • In ℜ^ , the training set is^ {(Z, yi ), i^ = 1,... , n}i

,^ Z=^ φ(X)i^ i^ PR NPTEL course – p.19/

Non-linear discriminant functions^ •^ The idea is that we map the feature vectors into a highdimensional space and find a linear classifier there.^ •^ In general, we can use a mapping,

′m m φ : ℜ→ ℜ^. ′m • In ℜ^ , the training set is^ {(Z, yi ), i^ = 1,... , n}i

,^ Z=^ φ(X)i^ i

-^ We can find optimal hyperplane by solving the dual.

PR NPTEL course – p.20/