Download Clustering - Data Mining - Lecture Slides | PSTAT 131 and more Study notes Statistics in PDF only on Docsity!
11
CLUSTERINGCLUSTERING
2
2
Clustering
Clustering
- – ClusteringClustering refers to grouping records, observations, or tasksrefers to grouping records, observations, or tasks
into classes of similar objects
into classes of similar objects
- – A Cluster is a collection of records that are similar to oneA Cluster is a collection of records that are similar to one
anotheranother
- – Records in one cluster are dissimilar to records in otherRecords in one cluster are dissimilar to records in other
clusters
clusters
33
ClusteringClustering
Clustering Tasks in Business and Research
Clustering Tasks in Business and Research
1. Target marketing for niche product, without large marketing
1. Target marketing for niche product, without large marketing
budget. budget.
2. Segment financial behavior into benign and suspicious
2. Segment financial behavior into benign and suspicious
categories.
categories.
3. Gene expression clustering, where genes exhibit similar3. Gene expression clustering, where genes exhibit similar
characteristics.
characteristics.
4
4
Clustering
Clustering
Applying cluster analysis to enormous databases helpfulApplying cluster analysis to enormous databases helpful
Reduces search space for downstream algorithms
Reduces search space for downstream algorithms
77
ClusteringClustering
Between-cluster variation:
Within-cluster variation:
8
8
Hierarchical Clustering Methods
Hierarchical Clustering Methods
- • HierarchicalHierarchical
- – Treelike cluster structure (dendogram) created through recursiveTreelike cluster structure (dendogram) created through recursive partitioningpartitioning
(Divisive Methods) or combining (Agglomerative Methods) existing(Divisive Methods) or combining (Agglomerative Methods) existing clustersclusters
99
Hierarchical Clustering MethodsHierarchical Clustering Methods
- – Agglomerative MethodsAgglomerative Methods
- – Each observation initialized to become own clusterEach observation initialized to become own cluster
- – At each iteration two closest clusters aggregated togetherAt each iteration two closest clusters aggregated together
Number of clusters reduced by one, each step
Number of clusters reduced by one, each step
- – Eventually, all records combined into single clusterEventually, all records combined into single cluster
10
10
Hierarchical Clustering Methods
Hierarchical Clustering Methods
- • Distance Between ClustersDistance Between Clusters
Single Linkage
Single Linkage
- – Known as NearestKnown as Nearest--Neighbor ApproachNeighbor Approach
- – Minimum distance between any record in cluster A, and any recordMinimum distance between any record in cluster A, and any record inin
cluster Bcluster B
1313
SingleSingle--Linkage ClusteringLinkage Clustering
2, 5
15, 16, 18
15, 16 33, 33
2 5 9 15 16 18 25 33 33 45
14
14
Single
Single
Linkage Clustering
Linkage Clustering
2, 5
2, 5, 9
2, 5, 9, 15, 16, 18, 25, 33, 33
2, 5, 9, 15, 16, 18
2, 5, 9, 15, 16, 18, 25
15, 16, 18
15, 16 33, 33
2, 5, 9, 15, 16, 18, 25, 33, 33, 45
2 5 9 15 16 18 25 33 33 45
1515
CompleteComplete--Linkage ClusteringLinkage Clustering
Complete
Complete
linkage explored using sample data
linkage explored using sample data
We want the distance among records in two clusters farthest fromWe want the distance among records in two clusters farthest from each othereach other
minimizedminimized
16
16
Complete
Complete
Linkage Clustering
Linkage Clustering
2, 5
2, 5, 9
2, 5, 9, 15, 16, 18
15, 16, 18
15, 16 33, 33
2, 5, 9, 15, 16, 18, 25, 33, 33, 45
25, 33, 33
25, 33, 33, 45
2 5 9 15 16 18 25 33 33 45
1919
kk - -Means ClusteringMeans Clustering
- • Determining Cluster CentroidDetermining Cluster Centroid
AssumeAssume nn data points (adata points (a 11 , b, b 11 , c, c 11 ), (a), (a 22 , b, b 22 , c, c 22 ), ..., (a), ..., (a nn , b, b nn , c, c nn ))
Centroid of points is center of gravity of pointsCentroid of points is center of gravity of points
Located at point (
Located at point ( Σ
a
a i
i /
n
n ,
b
b i
i /
n
n ,
c
c i
i /
n
n )
For example, points (1, 1, 1), (1, 2, 1), (1, 3, 1), and (2, 1,For example, points (1, 1, 1), (1, 2, 1), (1, 3, 1), and (2, 1, 1) have centroid1) have centroid
20
20
k
k
Means Clustering
Means Clustering
- – kk - -Means algorithm terminates when centroids no longer changeMeans algorithm terminates when centroids no longer change
- – ForFor kk clusters, Cclusters, C 11 , C, C 22 , ...., C, ...., C kk , all records, all records ““ownedowned”” by cluster remain in clusterby cluster remain in cluster
- – Convergence criterion may also cause terminationConvergence criterion may also cause termination
- – For example, no significant reduction in SSE. (WeFor example, no significant reduction in SSE. (We’’ll work with this)ll work with this)
2121
Example ofExample of kk - -MeansMeans
Assume
Assume k
k = 2 to cluster following data points
= 2 to cluster following data points
Step 1:Step 1: kk = 2 specifies number of clusters to partition= 2 specifies number of clusters to partition
Step 2:Step 2: Randomly assignRandomly assign kk = 2 cluster centers= 2 cluster centers
For example, mFor example, m 11 = (1, 1) and m= (1, 1) and m 22 = (2, 1)= (2, 1)
First Iteration
Step 3:Step 3: For each record, find nearest cluster centerFor each record, find nearest cluster center
Euclidean distance from points to mEuclidean distance from points to m 11 and mand m 22 shownshown
(1, 3) (3, 3) (4, 3) (5, 3) (1, 2) (4, 2) (1, 1) (1, 2)
a b c d e f g h
C 2
C 1
C 2
C 1
C 2
C 2
C 2
C 1
Cluster Membership
Distance from m 2.24 2.24 2.83 3.61 1.41 2.24 1.00 0. 2
Distance from m 2.00 2.83 3.61 4.47 1.00 3.16 0.00 1. 1
Point a b c d e f g h
22
22
Example of
Example of
k
k
Means Clustering
Means Clustering
Cluster mCluster m 11 contains {a, e, g} and mcontains {a, e, g} and m 22 has {b, c, d, f, h}has {b, c, d, f, h}
Cluster membership assigned, now SSE calculatedCluster membership assigned, now SSE calculated
Recall clusters constructed whereRecall clusters constructed where betweenbetween--cluster variationcluster variation (BCV) large, as(BCV) large, as
compared tocompared to withinwithin--cluster variationcluster variation (WCV)(WCV)
Ratio BCV/WCV expected to increase for successive iterationsRatio BCV/WCV expected to increase for successive iterations
2 2 2 2 2 2 2 2
1
2
∑ ∑
= ∈
k
i i
pC
i
SSE d pm
SSE surrogate for WCV
( , ) surrogateforBCV
- 0278 ,where
36
1
SSE
( , )
WCV
BCV
1 2
1 2
=
=
= = =
dm m
dmm
2525
Example ofExample of kk - -Means ClusteringMeans Clustering
Cluster centroids updated to mCluster centroids updated to m 11 = (1.25, 1.75) or m= (1.25, 1.75) or m 22 = (4, 2.75)= (4, 2.75)
After Second Iteration, cluster centroids shown to move slightlyAfter Second Iteration, cluster centroids shown to move slightly
0 1 2 3 4 5 6
0
1
2
5
4
3
26
26
Example of
Example of
k
k
Means Clustering
Means Clustering
- • Third (Final) IterationThird (Final) Iteration
Repeat procedure for Steps 3Repeat procedure for Steps 3 – – 44
Now, for each record find nearest cluster center mNow, for each record find nearest cluster center m 11 = (1.25, 1.75) or m= (1.25, 1.75) or m 22 = (4, 2.75)= (4, 2.75)
SSE = 6.23, and BCV/WCV = 0.4703SSE = 6.23, and BCV/WCV = 0.
Again, BCV/WCV has increased compared to previous = 0.3346Again, BCV/WCV has increased compared to previous = 0.
This time, no records shift cluster membershipThis time, no records shift cluster membership
Centroids remain unchanged, therefore algorithm terminatesCentroids remain unchanged, therefore algorithm terminates
2727
Example ofExample of kk - -Means ClusteringMeans Clustering
kk - -Means not guaranteed to find to find global minimum SSEMeans not guaranteed to find to find global minimum SSE
Invoking algorithm using variety of initial cluster centers imprInvoking algorithm using variety of initial cluster centers improvesoves
probability of achieving global minimum
probability of achieving global minimum
One approach places first cluster at random point, with remaininOne approach places first cluster at random point, with remaining clustersg clusters
placed far from previous centersplaced far from previous centers (Moore)(Moore)
What is appropriate value forWhat is appropriate value for kk ??
Potential problem for applyingPotential problem for applying kk - -MeansMeans
Analyst may have
Analyst may have
a priori
a priori
knowledge of
knowledge of
k
k
28
28
SAS K SAS K--Means Cluster AnalysisMeans Cluster Analysis
3131
32
32
3333
34
34
3737
38
38
3939
40
40
