Download Bayes' Theorem and Applications: K-Means Clustering and Expectation-Maximization and more Study notes Computer Science in PDF only on Docsity!
Bayes Theorem again•^
Bayes's Theorem begins with a statement of knowledgeprior to getting the data (called the
prior)
-^
The prior is in the form of a probability density. It can bebased on physics, on the results of other experiments, onexpert opinion, or any other source of relevantinformation.
-^
To improve this state of knowledge (hypothesis or priordistribution), a measurement is taken (data)
-^
Bayes's Theorem is the mechanism used to update thestate of knowledge to provide a
posterior
distribution.
-^
Both the prior and the measurement have a jointdistribution
-^
Let the measurement be A and the prior be B.
-^
Since both have occurred, the event AB has taken place.
-^
The probability of both A and B happening together isP(AB).
-^
This probability can be found as the product of theconditional probability of one, given the other, times theprobability of the other.
-^
P(A|B) ´ P(B) =
P(AB)
= P(B|A) ´ P(A)
(assuming both P(A) and P(B) are non zero)
Bayesian inference•^
Bayes’ rule:
-^
What makes a good scientific argument?
-^
P
( H
| D
) is high if:
P
( H
) is high
- Hypothesis strongly predicts the observed data:
P
( D
| H
) is high
P
( D
) is low
) (
) | ( ) ( ) |
(^
D P
H D P H P D H P
=
K-means clustering•^
Brute force difficult because many spheres, many pixels.
-^
Assume all spheres same radius; just need sphere centers.
-^
Iterative method.– If we knew centers, it would be easy to assign pixels to clusters.– If we knew which pixels in each cluster, it would be easy to find
centers.
- So guess centers, assign pixels to clusters, pick centers for
clusters, assign pixels to clusters, ….
simple example.Suppose we want to cluster black and whiteintensities, and we have the intensities: 1 3 8 11.start with centers c1 = 7 and c2=10.assign 1, 3, 8 to c1, 11 to c2.update c1 = (1+3+8)/3 = 4, c2 = 11.assign 1,3 to c1 and 8 and 11 to c2.update c1 = 2, c2 = 9 ½.converged.No assignments change, so the centers don’tchange.
K-means Properties•^
Think of this as trying to find the optimal solution to:–
Given points p1… pn, find centers c1…ck
-^
and find mapping f:{p1…pn}->{c1…ck} that minimizes
C = (p1-f(p1))^2 + …+ (pn-f(pn))^2.
-^
Every step reduces C.– The mean is the pt that minimizes sum of squared distance to a
set of points. So changing the center to be the mean reduces thisdistance.
- When we reassign a point to a closer center, we reduce its
distance to its cluster center.
-^
Convergence: since there are only a finite set of possibleassignments.
E-M (Expectation maximization)•^
Like K-means but withsoft assignment.– Assign point partly to all
clusters based onprobability it belongs toeach.
averages (c
) and variancej
(σ).
− −
− −
j
c c p p
i
j^
j j i i
e
e
p
f
2 (^22) 2
σ σ
Cluster centers are c
.j
Example•^
Matlab: tutorial
-^
Fuzzy assignment allows cluster to creep towards nearbypoints and capture them.
E-M•^
Reading:– Forsyth & Ponce 16.1, 16.2– Forsyth & Ponce 16.3– Yair Weiss:
Motion Segmentation using EM – a short tutorial.
(^1
st^
2 pages.)
Parametric Methods•^
We discussed Ransac, Hough Transform.
-^
The have some limitations– Object must have few parameters.– Finds an answer, but is it the best answer?– Hard to say because problem definition a bit vague.
E-M Definitions•^
Models have parameters: u–
Examples: line has slope/intercept; Gaussian has mean and variance.
-^
Data is what we know from image: y–
Examples: Points we want to fit a line to.
-^
Assignment of data to models: z–
Eg., which points came from line 1.– z(i,j) = 1 means data i came from model j.
-^
Data and assignments (y & z): x.
E-M Definitions•^
Missing Data: We know y. Missing values are u and z.
-^
Mixture model: The data is a mixture of more than onemodel.
Example: Histograms•^
Histogram gives 1D clustering problem.
-^
Constant regions + noise = Gaussians.
-^
Guess mean and variance of pixel intensities.
-^
Compute membership for each pixel.
-^
Compute means as weighted average.
-^
Compute variance as weighted sample variance.
-^
Details:
whiteboard;
Also, Matlab and Weiss.
More subtle points•^
Guess must be reasonable, or we won’t converge toanything reasonable.– Seems good to start with high variance.
-^
How do we stop.– When things don’t change much.– Could look at parameters (u).– Or likelihood of data.
