Download Gaussian Mixture Models for Clustering: Probabilistic Approach to k-Means - Prof. Harold D and more Study notes Computer Science in PDF only on Docsity!
Machine Learning (CS 5350/CS 6350) 15 Feb 2007
Clustering (cont’d)
We can think of k-means in a probabilistic framework. Suppose that we have a Gaussian centered at each of the means, then we can get the probability of the data set as:
p(x1:N | c1:N ) =
n
Nor(xn | μcn , σ^2 I)
Here, μk is the the mean of cluster k. (For now, we assume common variance.)
We can think of the clustering problem as trying to find good μks.
Change of notation: Instead of cn being the cluster for data point n, let zn ∈ { 0 , 1 }K^ be an indicator vector for data point n. I.e., zn,k = 1 if xn is in k and = 0, otherwise.
Model: Generate each data point by first choosing among one of k clusters, each with probability πk. Then generate the data point by a Gaussian centered at μk. In equations:
p(x1:N , z1:N,1:K | μ1:K , σ^2 , π) =
n
k
πkNor(xn | μk, σ^2 I)
}zn,k
n
k
πk(2πσ^2 )−d/^2 exp
[
2 σ^2
||xn − μk||^2
]}zn,k
From this, we get the likelihood of the data by summing over the unknown zs:
p(x | μ, σ^2 , π) =
z1:N,1:K
n
k
πk(2πσ^2 )−d/^2 exp
[
2 σ^2
||xn − μk||^2
]}zn,k
n
zn,1:K
k
πk(2πσ^2 )−d/^2 exp
[
2 σ^2 ||xn − μk||^2
]}zn,k
So now we follow our standard recipe of taking logs and derivatives...
log p(x | μ, σ^2 , π) =
n
log
zn
k
πk(2πσ^2 )−d/^2 exp
[
2 σ^2
||xn − μk||^2
]}zn,k
But at this point we get stuck!
If we knew z, we could do this easily:
log p(x, z | μ, σ^2 , π) =
n
k
zn,k
log πk + log Nor(xn | μk, σ^2 I)
We call the value with z the “complete log likelihood” and the value without z the “incomplete log likelihood.”
The idea for clustering with GMMs is the same as for k-means. We will make an initial guess at z, and then try to iteratively refine it. This turns out to be a special case of the “expectation maximization” algorithm, which we will discuss shortly in more generality.
In k-means, we made “hard” guesses at the clusters: the z vector we considered had a one in a single location and zeros everywhere else. In Gaussian mixture models, we make “soft” guesses. The z vector will satisfy zn,k ≥ 0 for all n, k and
k zn,k^ = 1 for all^ n. Thus, it’s a probabilistic guess at the clustering.
Given some setting of μ and σ^2 , we can make guesses at z by just looking at their expectations:
Ep(z | x,μ,σ (^2) ,π)zn,k = 1 × p(zn,k = 1 | xn, μ, σ^2 , π) + 0 × p(zn,k = 0 | xn, μ, σ^2 , π) = p(zn,k = 1 | xn, μ, σ^2 , π)
= p(xn | zn,k = 1, μ, σ^2 )p(zn,k = 1 | π) ∑ k′^ p(xn^ |^ zn,k′^ = 1, μ, σ (^2) )p(zn,k′ = 1 | π)
Nor(xn | μk, σ^2 )πk ∑ k′^ Nor(xn^ |^ μk′^ , σ (^2) )πk′
These expectations give us a soft clustering for each data point into each of the k clusters.
Now, using these “guesses”, we want to maximize the complete data log likelihood with respect to μ and σ^2. To do this, we take the gradient of the complete likelihood with respect to π, μ and σ^2.
We do π first. For this, we actually need to introduce a Lagrange multiplier to ensure that the constraints on π are satisfied (πk ≥ 0 and
k πk^ = 1). This gives us an augmented likelihood function of:
n
k
zn,k log πk − λ
k
πk − 1
We differentiate this with respect to πk to get:
n
zn,k πk
− λ =
n
zn,k − λπk = 0
Summing over all K, we get that λ =
n
k zn,k^ =^ N^ , so:
πk =
N
n
zn,k
This makes intuitive sense!
Next, we’ll take care of μk. These are somewhat easier since we don’t need to worry about constraints, so there are no Lagrange multipliers.
Putting this all together, we obtain the following algorithm:
- Initialize cluster centers μ1:K , π and σ^2
- Iterate T times...
(a) Compute expectation of z variables by:
Ep(z | x,μ,σ (^2) ,π)zn,k =
Nor(xn | μk, σ^2 )πk ∑ k′^ Nor(xn^ |^ μk′^ , σ (^2) )πk′
(b) Compute new values of π, μ, σ^2 by:
πk =
N
n
zn,k
μk =
n
zn,k ∑ n′^ zn′,k
xn
σ^2 =
dN
n
k
zn,k ||xn − μk||^2
One can obtain a more general solution, where we use full covariance matrices and/or cluster-specific co- variance matrices. For the assignment, you will have to do both. The D’Souza writeup includes the deriva- tions/updates for the more general case.
