Clustering Algorithms Overview, Lecture Notes - Computer Science, Study notes of Network Theory

Download Clustering Algorithms Overview, Lecture Notes - Computer Science and more Study notes Network Theory in PDF only on Docsity!

CS181 Lecture 9: Clustering

(and intro to Prob methods)

David C. Parkes

Outline

• Clustering
  • K -means
  • Hierarchical Agglomerative Clustering (HAC)
• Introduction to Probabilistic Methods
  • classification
  • clustering
  • maximum likelihood

2

Today: Clustering

• Clustering : discover groups of similar examples - for dimensionality reduction : project the data down to two or three dimensions - to understand the data - as a way to preprocess a lot unlabeled data, to use for supervised learning

4

Clustering

5

How do we Evaluate Clustering

Algorithms?

• Eyeball the clusters to see if they make intuitive sense?
• Problem: subjective
• Identifying a useful objective is the key challenge in clustering! 10

Possible Objective Criterion

• Intuition: a cluster is a group of examples with small inter-example distances compared to distances to other examples

11

K-means Algorithm

• Choose initial values for ¹ 1 , …, ¹K
• [E] Minimize Err with respect to rik, keeping prototypes ¹ fixed Set rik = 1 for k that minimizes ||xi - ¹k||^2
• [M] Minimize Err with respect to {¹ 1 , …, ¹K},

keeping assignments rik fixed Set  Err /  ¹k = 0 ¹k = i rik x i / i rik

• Repeat until convergence (^14)

K-means Algorithm

• Choose initial values for ¹ 1 , …, ¹K
• [E] Minimize Err with respect to rik, keeping prototypes ¹ fixed Set rik = 1 for k that minimizes ||xi - ¹k||^2
• [M] Minimize Err with respect to {¹ 1 , …, ¹K},

keeping assignments rik fixed

¹k = i rik x i / i rik

• Repeat until convergence (^15)

K-means Algorithm

• Choose initial values for ¹ 1 , …, ¹K
• [E] Minimize Err with respect to rik, keeping prototypes ¹ fixed Set rik = 1 for k that minimizes ||xi - ¹k||^2
• [M] Minimize Err with respect to {¹ 1 , …, ¹K},

keeping assignments rik fixed Set  Err /  ¹k = 0 ¹k = i rik x i / i rik [just centoid!]

• Repeat until convergence [+ restart!] (^17)

• Yellowstone National Park, Wyoming
• 272 data points
  • duration of eruption (y-axis)
  • duration to next eruption (x)
  • standardized

(Wikipedia)^19

Example

21

Sum squared error in K-means; E (blue), M (red)

(Bishop)

22

(Bishop; K-means image segmentation)