Understanding SVM Classification and Regression with Reproducing Kernels - Prof. Sridhar M, Study notes of Computer Science

Download Understanding SVM Classification and Regression with Reproducing Kernels - Prof. Sridhar M and more Study notes Computer Science in PDF only on Docsity!

SVM Classification and Regression; Reproducing

Kernels Sridhar Mahadevan [email protected] University of Massachusetts

^ Sridhar Mahadevan: CMPSCI 689 – p.1/

Topics

•^ SVM review: linearly separable case •^ Soft-margin classifiers: dealing with overfitting •^ SVM regression •^ Examples of kernels •^ Mercer’s theorem

^ Sridhar Mahadevan: CMPSCI 689 – p.2/

Separating Hyperplane between Convex

Hulls

**+

• −+ − − − +**^ ^ Sridhar Mahadevan: CMPSCI 689 – p.4/

Optimal Margin Classification

•^ Consider the problem of finding a set of weights

w^ that produces a hyperplane with

the maximum geometric margin.

maxγ^ such that γ,w,b^ y(< w, x>^ +b)^ ≥^ γ, i^ = 1ii^

,... , m ‖w‖ = 1

•^ We rescale the weights by

‖w‖^ to eliminate the non-convex constraint

‖w‖^ = 1,

and instead look to minimize

< w, w >^ while keeping the functional margin

ˆγ^ = 1.

12 min‖w‖such that w 2 y(< w, x>^ +b)^ ≥^1 , i^ = 1ii

,... , m^ ^ Sridhar Mahadevan: CMPSCI 689 – p.5/

Weak Duality Theorem

•^ Suppose^ w^ is a feasible solution to the primal problem, and that

α^ and^ β^ constitute

a solution to the dual problem.^ L(α, βD^

)^ =^ minL(u, α, β^ u^

) ≤ L(w, α, β) XX= f (w) + αg(w) +^ βh(iiii i^ i

w)^ ≤^ f^ (w)

•^ Since^ w^ is a feasible solution to the primal, the last inequality holds true since^ g(w)^ ≤^0 and^ hi

(w) = 0^ (since^ w^ i

is a feasible solution), and since

(α, β)^ is a

feasible solution to the primal,

α≥^0 .i^

•^ This implies the following condition:^ max^ α,β

L(α, β)^ ≤^ minD^ :α≥ 0

{f^ (w) :^ g(w)^ ≤i w

0 , h(w) = 0}i^ ^ Sridhar Mahadevan: CMPSCI 689 – p.7/

Sparsity of Parameters

∗ • Corollary: Let w

be a weight vector that satisfies the primal constraints and

be the Lagrangian variables that satisfies both the dual constraints, that is∗^ f^ (w) =^ LD

∗∗(α, β)^ where^ α

∗∗≥^0 and^ g(w)i i^

∗ ≤ 0 , h(w) = 0i

∗∗ • Then, αg(w) = 0i i^

for^ i^ = 1,... , k.

•^ The proof follows easily by noting that the inequality

X∗∗ f (w) + αg(i^ i^ i

X∗∗∗w) + βh(wi^ i^ i

∗) ≤ f (w)

becomes an equality only when

∗∗ αg(w) = 0^ for^ i i^

i^ = 1,... , k.

•^ This is a^ key^ result: in the SVM context, it implies that the representation will be^ sparse^ (since we only need to maintain

αfor the^ support vectors i^

, that is the

instances that are closest to the separating hyperplane).

^ Sridhar Mahadevan: CMPSCI 689 – p.8/

Duality Gap and Saddle Points

•^ If the primal and dual solutions are not equal, this duality gap can be detected inthe course of solving the dual, and used as a convergence metric. •^ Define a^ saddle point

∗ as the triple w, α

∗∗∗^ , β, where^ w∈^

∗^ Ω, α≥^0 , and

∗ L(w, α, β)^ ≤^ L(w

∗∗∗∗, α, β)^ ≤^ L(w

•^ Theorem:^ The triple

∗∗∗^ w, α, βis a saddle point if and only if

∗^ wis a solution to

the primal problem, and

∗∗^ α, βis a solution to the dual problem, and there is no

duality gap, so^ f^ (w

∗∗∗) =^ L(α, β).D^

•^ Strong Duality Theorem:

If^ f^ (w)^ is convex, and

w^ ∈^ Ω, where^ Ω^ is a convex set,

and^ g, hare affine functions, the duality gap isii^

0.^ ^ Sridhar Mahadevan: CMPSCI 689 – p.10/

Karush Kuhn Tucker Conditions

•^ Assume that the function

f^ (w)^ and the constraints

g(w)^ are convex, andi

h(w)^ isi

an affine set (meaning

h(w) =< a, w >ii

+b).i

•^ Let there be at least one

w^ such that^ g(w)i

<^0 for all^ i. Then, the KKT conditions

assure us the duality gap is

•^ The minimizing values

∗∗∗^ α, β, walso satisfy the following conditions: ∂∗∗∗ L(w, α, β) = 0 ∂wi

,^ i^ = 1,... , n

(1)

∂∗∗∗^ L(w, α, β) = 0 ∂βi

,^ i^ = 1,... , l

(2)

∗∗ αg(w) = 0,^ i^ = 1i i^

,... , k

(3)

∗ g(w)^ ≤^0 , i^ = 1,^ i

... , k

(4)

∗ α≥^0 ,^ i^ = 1,^... , k i^

(5)

^ Sridhar Mahadevan: CMPSCI 689 – p.11/

Dual Form of Optimal Margin Classifier • We can write the Lagrangian for our optimal margin classifier as^1 L(w, b, α) = ‖w^2

X 2 ‖− α(y(< w, xi^ i^ i

>^ +b)^ −^ 1)i

•^ To solve the dual form, we first minimize with respect to

w^ and^ b, and then

maximize w.r.t.^ α^ ∇L(w, b, aw^

mX) = w −^ αyxii i=

mX= 0 ⇒ w =i i=

αyxiii

mX ∇L(w, b, a) =b i=

αy= 0ii^

•^ Mechanical interpretation:

Think of each instance

xas generating a forcei^

αyonii^

the hyperplane. The forces sum to

^ Sridhar Mahadevan: CMPSCI 689 – p.13/

Support Vectors for Optimal Margin

Classifiers

•^ Using these results, we can simplify the Lagrangian into the following form:^0 mX@^ max^ αi^ α^ i

mX 1 − yyααij^ ij 2 i,j=

1 A^ < x, x>s.t.^ ij

Xα≥ 0 and αi i

y= 0ii^

•^ Given the maximizing

α, we use the equationi

Pm∗ ∗ w= αyxii=1^ i^

to find thei

∗maximizing w. A new instance

x^ is classified using a weighted sum of inner

products (over only support vectors!)∗^ < w, x >^

mX+b =^ αy< xii^ i=

X, x > +b = αi i∈SV

y< x, x >^ +bii^ i

•^ The intercept term

∗^ bcan be found from the primal constraints∗max(< w, xy=−^1 ∗ i^ b= −

>) + min(y=−^1 i i^

∗< w, x>)i^2 Sridhar Mahadevan: CMPSCI 689 – p.14/

Dealing with Nonseparable Data

**high bias, low variance − − − − + +− −+ ++− − − non−separable data

• +− + ++ non−separable data+**^ −+ − − −

+− ++ non−separable data high variance, low bias

^ Sridhar Mahadevan: CMPSCI 689 – p.16/

Soft Margin Classifiers

•^ Let us reformulate the concept of margin to allow errors, so positive (or negative)instances can lie on the wrong side of the margin. •^ The^ slack variable

ξrepresents the extent to which a margin constraint is violatedi^ y(< w, x>^ +b)^ ≥^1 −iii^

ξwhere^ ξ≥^0 ,^ i^ i^

i^ = 1,... , l

•^ Similar to ridge regresson, define a variable

λ^ which represents the extent to which

we want to tolerate errors. • A soft-margin classifier solves the following constrained optimization problemMinimize

< w, w >^ +^

lX^2 λ^ ξ^ i i=

subject to^ y(< wii

, x>^ +^ b)^ ≥^1 i^

−^ ξ,^ i^ = 1,... li

where^ ξ≥^0 ,i^

i^ = 1,... l^ ^ Sridhar Mahadevan: CMPSCI 689 – p.17/

 -insensitive loss L

y − <w, x> − b^ 2ε^ L^ 2ε y − <w, x> − b

^ Sridhar Mahadevan: CMPSCI 689 – p.19/

SVM Regression

•^ We introduce^ two

slack variables^ ξandi^

ˆ ξwhich represent the penalty fori^

exceeding or being below the target value by more than

•^ The primal problem can be formulated as

2 Minimize ‖w‖+

lX^22 ˆ λ^ (ξ+ ξ)^ i^ i^ i=

subject to^ (< w, xi

>^ +^ b)^ −^ yi i^

≤^ ^ +^ ξ,^ i^ = 1,... li

and^ y−^ (< w, xi^ i

>^ +^ b)^ ≤^ ^ + i

ˆξ,^ i^ = 1,... li

ˆ where ξ, ξ≥^ ii^

0 ,^ i^ = 1,... l^ ^ Sridhar Mahadevan: CMPSCI 689 – p.20/