Machine Learning (CS 5350/CS 6350) 09 Oct 2008
Maximum margin classifiers
Large margin principle: want w, b that separates the classes maximally. We assume that w, b separates
the data and achieves a functional margin of at least 1. That is, w>x+b≥1 for all positive xand
w>x+b≤ −1 for all negative x. We defined the geometric margin based on the normalized weight vector:
γ= minnyn(u>xn+b), where u=w/||w||. Compute margin as a function of normalized weight vector u,
for positive point x+and negative point x−:
γ≤1
2u>x++b−u>x−−b
γ=1
2w
||w||
>x+−w
||w||
>x−
=1
2||w|| w>x+−w>x−
=1
||w||
This shows that the margin is inversely proportional to the norm of the weight vector, and independent of
the bias. Moreover, having a large margin is equivalent to having a small weight vector norm (why does this
make intuitive sense?).
Now, we write the learning problem as an optimization problem:
minimizew
1
2||w||2
subject to ynw>xn+b−1≥0 (∀n)
Now we need to figure out how to solve this. Enter convex optimization and Lagrange theory. . .
Introduce Lagrange-multipliers α1:N, one for each constraint. Leads to Lagrangian:
L(w,α) = 1
2||w||2−X
n
αnynw>xn+b−1
Now, we want to minimize L(w,α) with respect to both wand α. Take derivatives with respect to w:
∇wL=w−X
n
αnynxn= 0
=⇒w=X
n
αnynxn
∇bL=X
n
ynαn= 0
So, given α,wis deterministic. . . plug back in to L:
L(α) = 1
2X
n
αnynxn
2
−X
n
αn(yn"(X
m
αmymxm)>xn+b#−1)
=1
2X
mX
n
αmαnymynxm
>xn−X
mX
n
αmαnymynxm
>xn−bX
n
αnyn+X
n
αn
=−1
2X
mX
n
αmαnymynxm
>xn+X
n
αn
1
