Download SUPPORT VECTOR MACHINE and more Summaries Computer Science in PDF only on Docsity!
Introduction to Support
Vector Machines
Note to other teachers and users of these slides. Andrew would be delighted if you found this source material useful in giving your own lectures. Feel free to use these slides verbatim, or to modify them to fit your own needs. PowerPoint originals are available. If you make use of a significant portion of these slides in your own lecture, please include this message, or the following link to the source repository of Andrewās tutorials: http://www.cs.cmu.edu/~awm/tutori als
. Comments and corrections gratefully received. Thanks: Andrew Moore CMU And Martin Law Michigan State University
History of SVM
ļ®
SVM is related to statistical learning theory [3]
ļ®
SVM was first introduced in 1992 [1]
ļ®
SVM becomes popular because of its success in
handwritten digit recognition
ļ®
1.1% test error rate for SVM. This is the same as the
error rates of a carefully constructed neural network,
LeNet 4.
ļ® See Section 5.11 in [2] or the discussion in [3] for details ļ®
SVM is now regarded as an important example of
ākernel methodsā, one of the key area in machine
learning
ļ®
Note: the meaning of ākernelā is different from the
ākernelā function for Parzen windows
[1] B.E. Boser et al. A Training Algorithm for Optimal Margin Classifiers. Proceedings of the Fifth Annual Workshop on Computational Learning Theory 5 144-152, Pittsburgh, 1992. [2] L. Bottou et al. Comparison of classifier methods: a case study in handwritten digit recognition. Proceedings of the 12th IAPR International Conference on Pattern Recognition, vol. 2, pp. 77-82. [3] V. Vapnik. The Nature of Statistical Learning Theory. 2nd^ edition, Springer, 1999.
Linear Classifiers
f
x
y
est denotes + denotes - f ( x , w ,b) = sign( w. x - b) How would you classify this data?
Linear Classifiers
f
x
y
est denotes + denotes - f ( x , w ,b) = sign( w. x - b) How would you classify this data?
Linear Classifiers
f
x
y
est denotes + denotes - f ( x , w ,b) = sign( w. x - b) Any of these would be fine.. ..but which is best?
Classifier Margin
f
x
y
est denotes + denotes - f ( x , w ,b) = sign( w. x - b)
Define the
margin of a
linear classifier
as the width
that the
boundary could
be increased by
before hitting a
datapoint.
Maximum Margin
f
x
y
est denotes + denotes - f ( x , w ,b) = sign( w. x + b)
The maximum
margin linear
classifier is the
linear classifier
with the, um,
maximum
margin.
This is the
simplest kind of
SVM (Called an
LSVM)
Support Vectors are those datapoints that the margin pushes up against Linear SVM
Why Maximum Margin?
denotes + denotes - f ( x , w ,b) = sign( w. x - b)
The maximum
margin linear
classifier is the
linear classifier
with the, um,
maximum
margin.
This is the
simplest kind of
SVM (Called an
LSVM)
Support Vectors are those datapoints that the margin pushes up against
Estimate the Margin
ļ®
What is the distance expression for a point x to a
line wx +b= 0?
denotes + denotes - x wx +b = 0 (^2 ) 2 1
d i^ i
b b
d
w
ļ½
x w x w
x
w
X ā Vector W ā Normal Vector b ā Scale Value
W
Large-margin Decision Boundary
ļ®
The decision boundary should be as far away
from the data of both classes as possible
ļ®
We should maximize the margin, m
ļ®
Distance between the origin and the line w t x =-b is
b/|| w ||
Class 1 Class 2
m
Next stepā¦ Optional
ļ®
Converting SVM to a form we can solve
ļ®
Dual form
ļ®
Allowing a few errors
ļ®
Soft margin
ļ®
Allowing nonlinear boundary
ļ®
Kernel functions
The Dual Problem (we ignore the derivation) ļ®
The new objective function is in terms of ļ”i only
ļ®
It is known as the dual problem: if we know w , we
know all ļ”i; if we know all ļ”i, we know w
ļ®
The original problem is known as the primal
problem
ļ®
The objective function of the dual problem needs
to be maximized!
ļ®
The dual problem is therefore:
Properties of ļ”i when we introduce the Lagrange multipliers The result when we differentiate the original Lagrangian w.r.t. b
Characteristics of the Solution
ļ® Many of the ļ”
i are zero (see next page for example)
ļ® (^) w is a linear combination of a small number of data points ļ® (^) This āsparseā representation can be viewed as data compression as in the construction of knn classifier
ļ® x
i with non-zero^ ļ”i are called support vectors (SV)
ļ® The decision boundary is determined only by the SV ļ® (^) Let t j ( j =1, ...,^ s ) be the indices of the^ s^ support vectors. We can write ļ®
For testing with a new data z
ļ® Compute and classify z as class 1 if the sum is positive, and class 2 otherwise ļ® (^) Note: w need not be formed explicitly
6
A Geometrical Interpretation
Class 1 Class 2
1
2
3
4
5
7
8
9
10
