Evaluating Cluster Validity in Data Mining: Techniques and Measures - Prof. Jennifer L. Ne, Study notes of Data Analysis & Statistical Methods

Download Evaluating Cluster Validity in Data Mining: Techniques and Measures - Prof. Jennifer L. Ne and more Study notes Data Analysis & Statistical Methods in PDF only on Docsity!

Data Mining

CS57300 / STAT 59800-

Purdue University

April 7, 2009

Descriptive modeling: evaluation

Cluster validity

• For prediction tasks there are a variety of external evaluation metrics

• Accuracy, squared loss, area under ROC, etc.

• For cluster analysis the external evaluation should evaluate the “goodness” of

the resulting clusters

• Why do we want external validation?

• To avoid finding patterns in noise

• To compare clustering algorithms

• To compare two sets of clusters

Random data

October 30, 2007 Data Mining: Concepts and Techniques 4

Clusters found in Random Data

(^00 0). 2 0. 4 0. 6 0. 8 1

1. 1
2. 2
3. 3
4. 4
5. 5
6. 6
7. 7
8. 8
9. 9 1

x

y

Random

Points

x

y

K-means

x

y

DBSCAN

x

y

Complete

Link

Unsupervised

Correlation

• Compute the correlation between the initial similarity matrix and an “ideal”

cluster matrix

• Entry i,j is 1 if i and j are in the same cluster, 0 otherwise

• High correlation indicates that points in the same cluster are close to each

other

• Assumes that the proximity values are [0,1]

• When is this a good measure?

Example

October 30, 2007 Data Mining: Concepts and Techniques 8

Measuring Cluster Validity Via Correlation

! Correlation of ideal similarity and proximity

matrices for the K-means clusterings of the

following two data sets.

x

y

x

y

Corr = - 0. 9235 Corr = - 0. 5810

Visual inspection

• Order the proximity matrix with respect to cluster labels

• Inspect visually

• Good clusterings exhibit clear block pattern

Cohesion and separation

October 30, 2007 Data Mining: Concepts and Techniques 22

Internal Measures: Cohesion and Separation

! Prototype-Based View of Cohesion and Separation.

x " Ci

proximity ( x , c i )^1 "^2 "%^3 $'&#+!'^5 )!^6 - )^4 )^7 %&^8 '*'$^9 +:'^5 ):^6 -

A<)B")="$) 2 %& 8 '*'$ 9 )C")$.")/D> 3 %"()E>:='(" 3 #)('/$ 3 #:")$."#)!&."/'&#)'/)$.")!=>/$"%) 11 E@

Cohesion

• Measures how closely related the objects are within each cluster

• Within cluster sum of squared errors (SSE)

• For each point, the error is the distance to the centroid

• Within cluster pairwise weighting

• Sum distance between all pairs of points in same cluster

Separation

• Measures how distinct a cluster is from the other clusters

• Between cluster SSE (for cluster C)

• For each cluster C’, the error is the distance from the centroid c to the other

centroid c’

• The error is multiplied by the cluster size |C’|

• Between cluster pairwise weighting

• Sum distance between all pairs of points in different clusters

Cohesion and separation

• The sum of the between cluster SSE and within cluster SSE is equal to the

total sum of squared error (distance of each point to overall mean)

• Thus minimizing cohesion is equivalent to maximizing separation

Example

October 30, 2007 Data Mining: Concepts and Techniques 15

Evaluation of Hierarchical Clustering

! Example : Below table shows the cophenetic distance matrix for

the single link clustering of the 6 two-dimensional points

.>"!? 5 $ 5 %# 7 .;# 9 %<$ 75 .=<%:#@.A":. 9 #$ 345. 4 #$

B:# 3 #$< 4 .;# 99 #=# 4 <:#%C.=<%:#@.DEF 874 #; 5 <$.;# 9 %<$ 7

5 G Similarity matrix^ Cophenetic matrix

Cophenetic correlation

• Measure correlation between the original disimilarity matrix and the

cophenetic matrix

CP CC =

Cov(D, CD )

[V ar(D) · V ar(CD )]^1 /^2

Supervised

Class-label evaluation

• If you have class labels why cluster?

• Usually small hand-labeled dataset for evaluation

• But large dataset to cluster automatically

• May want to assess how close clusterings correspond to classes but still

allow for more variation in the clusters

Classification-oriented

• Precision

• The fraction of a cluster that consists of objects of a specified class

• Recall

• The extent to which a cluster contains all objects of a specified class

• Accuracy

• Why is it hard to measure the accuracy of a clustering if you know class

labels?

Similarity-oriented

• Based on premise that any pair of objects in the same cluster should have the

same class and vice versa

• Compare the “ideal” cluster similarity matrix to the “ideal” class similarity

matrix

Approaches

• Correlation between two ideal matrices

• Measures of binary similarity between two ideal matrices

• f 00 = # pairs of objects having diff class and diff cluster

• f 01 = # pairs of objects having diff class and same cluster

• f 10 = # pairs of objects having same class and diff cluster

• f 1 = # pairs of objects having same class and same cluster

Rand =

f 00 + f 11

f 00 + f 01 + f 10 + f 11

Jaccard =

f 11

f 01 + f 10 + f 11

Determining k

• Approach: evaluate over a range of k, look for peak, dip, or knee in evaluation

measure

October 30, 2007 Data Mining: Concepts and Techniques 37

Determining the Correct Number of Clusters

! Various unsupervised cluster evaluation measures can be

used to approximately determine the correct number

of clusters.

! Example : The data set has 10 natural clusters.

5#:;4#<=:=)#>?@24=0A

B.45=C4#/D9;614::4#86477D8D40:#.)

(@E=D5/

SSE Silhouette

Bayesian networks

• Goodness of fit

• AIC, BIC, MDL, BD

• How to assess quality of structure learning?

• Use synthetic data generated from specified Bayes net, try to recover

structure, compare to “gold standard”

Next class

• Reading: Chapter 13 PDM

• Topic: Pattern mining: representation