Download Machine Learning: Understanding Kernels and Their Applications in CS 5350/CS 6350 - Prof. and more Study notes Computer Science in PDF only on Docsity!
Machine Learning (CS 5350/CS 6350) 07 Feb 2008
Kernels
Today, we talk about kernels. These are magical beings that allow us to turn (nearly) any linear model into a non-linear model! Consider 1D, non-linearly separable data: “+ + + − − − + + +”. Now, map each location x to a 2D point (x, x^2 ). Then, in 2D, the data is linearly separable. We can generalize: for x ∈ RD^ , map Φ : x 7 → (x^21 , x^22 ,... , x^2 D , x 1 x 2 , x 1 x 3 ,... , x 1 xD ,...... , xD− 1 xD ). Or maybe we want cubic function! This is computationally a nightmare! Recall SVM (dual) optimization:
minimizeα^ ∑ n
αn − 12 ∑ n,m
ynymαnαmxn>xm
subject to
n
ynαn = 0 αn ≥ 0 , (∀n)
And SVM prediction:
f (x′) = sign
[ ∑
n∈SV
αnynxn>x′
]
In both cases, inputs only ever appear in dot products! So, we just replace all xs with Φ(x)s:
minimizeα^ ∑ n
αn − 12 ∑ n,m
ynymαnαmΦ(xn)>Φ(xm)
subject to
n
ynαn = 0 αn ≥ 0 , (∀n) prediction : sign
[ ∑
n∈SV
αnynΦ(xn)>Φ(x′)
]
Insight: for many Φ, we can compute dot products without actually computing the resulting vectors! Consider the quadratic case in two dimensions (with an addition of some constant terms). The dot product can be computed as:
Φ(x)>Φ(z) = (x^21 , x^22 , √ 2 x 1 x 2 )>(z 12 , z 22 , √ 2 z 1 z 2 ) = x^21 z 12 + x^22 z 22 + 2x 1 x 2 z 1 z 2 = (x 1 z 1 + x 2 z 2 )^2 = (x>z)^2
1
Machine Learning (CS 5350/CS 6350) 2
So the cost of computing the dot product in the high-dimensional space is the same as the cost in the low-dimensional space. Can additionally include a linear term: Φ(x) = (x 1 , x 2 , x^21 , x^22 , √ 2 x 1 x 2 ) by using (1 + x>z)^2. Such constructions are called kernel products: KΦ(x, z) = Φ(x)>Φ(z) (often subscript of Φ is dropped from K when it is clear from context).
A bit of theory... We write the original space (the “input space”) as X and the output of Φ (the “feature space”) as F.
- Φ : X → F
- K : X × X → R, K(x, z) = Φ(x)>Φ(z)
- The previous means that F must be a vector space with a dot product: these are called Hilbert spaces.
Question: can I used any function K as a kernel? Answer: No! There must exist some Hilbert space F for which K is a dot product. Question: Ack! How can I know that? Answer: A sufficient condition is that K be positive definite:
∫ dx
dz f (x)K(x, z)f (z) > 0 , (∀f ∈ L 2 )
This is equivalent to saying that the resulting matrix for any data set be positive definite; define Kij = K(xi, xj ) and require that K be positive definite. (Mercer’s condition) Why does this matter? It helps us build kernels...
Building kernels... If K 1 , K 2 are kernels and α is a positive real number, then the following are also all kernels:
- K(x, z) = K 1 (x, z) + K 2 (x, z) – direct sum
- K(x, z) = αK 1 (x, z) – scalar product
- K(x, z) = K 1 (x, z)K 2 (x, z) – direct product
Combining these, any (positive) linear combination of kernels is a kernel. Okay, so where do they come from in the first place? We’ve already seen two:
- Linear: K(x, z) = x>z
- Quadratic: K(x, z) = (1 + x>z)^2
