Lecture 21: LWR, Unsuperwised learning
TTIC 31020: Introduction to Machine Learning
Instructor: Greg Shakhnarovich
TTI–Chicago
November 12, 2010
Lecture 21: LWR, Unsuperwised learning TTIC 31020
Prepare for your exams
Study with the several resources on Docsity
Earn points to download
Earn points by helping other students or get them with a premium plan
Guidelines and tips
Prepare for your exams
Study with the several resources on Docsity
Earn points to download
Earn points by helping other students or get them with a premium plan
LWR-Introduction to Machine Learning-Lecture 21-Computer Science, Lecture notes of Introduction to Machine Learning
LWR, Unsuperwised Learning, Kernel Density Estimate, Nearest Neighbor, Kernel Regression, Parametric, Locally Weighted Regression, Linear LWR, Complexity, Nearest Neighbor Search, Locality Sensitive Hashing, LSH, Hash Functions, Unsupervised Learning, Clustering, Gaussian Mixture, K-Means Clustering, K-Means, Optimization, Vector Quantization, Greg Shakhnarovich, Lecture Slides, Introduction to Machine Learning, Computer Science, Toyota Technological Institute at Chicago, United States of Americ
Typology: Lecture notes
2011/2012
1 / 56
This page cannot be seen from the preview
Don't miss anything!
Related documents
Partial preview of the text
Download LWR-Introduction to Machine Learning-Lecture 21-Computer Science and more Lecture notes Introduction to Machine Learning in PDF only on Docsity!
Lecture 21: LWR, Unsuperwised learning
TTIC 31020: Introduction to Machine Learning
Instructor: Greg Shakhnarovich
TTI–Chicago
November 12, 2010
Review
Kernel density estimate:
−6^0 −5 −4 −3 −2 −1 0 1 2 3
Nearest neighbor
Kernel regression
Parametric locally weighted regression
Idea 2: bring back the parameters.
Fit a (simple) parametric model to the neighbors of x 0.
Parametric locally weighted regression
Idea 2: bring back the parameters.
Fit a (simple) parametric model to the neighbors of x 0.
Parametric locally weighted regression
Idea 2: bring back the parameters.
Fit a (simple) parametric model to the neighbors of x 0.
Implicit assumption: the target function is reasonably smooth.
Example: linear LWR
from Atkeson et al.
Spring stiffness ⇔ K(x 0 , xi)
What kind of functions can we estimate with this model?
Complexity of nearest neighbor search
How similar is similar?
Complexity of nearest neighbor search
How similar is similar?
k nearest neighbors
Complexity of nearest neighbor search
How similar is similar?
k nearest neighbors
r-neighbors:
within radius r from x 0
(, r)-neighbors:
within radius (1 + )r
Locality Sensitive Hashing
LSH [Indyk&Motwani]: very fast algorithm for finding a
(, r)-neighbor of x 0 :
- Query time O
dn^1 /(1+)
- (^) Storage O
dn + n1+1/(1+)
Locality Sensitive Hashing
LSH [Indyk&Motwani]: very fast algorithm for finding a
(, r)-neighbor of x 0 :
- Query time O
dn^1 /(1+)
- (^) Storage O
dn + n1+1/(1+)
Practical meaning, with 10^6 examples ×10000 features, for
= 1:
- (^) Assume each feature is a float (4 × 1010 data bytes).
Locality Sensitive Hashing
LSH [Indyk&Motwani]: very fast algorithm for finding a
(, r)-neighbor of x 0 :
- Query time O
dn^1 /(1+)
- (^) Storage O
dn + n1+1/(1+)
Practical meaning, with 10^6 examples ×10000 features, for
= 1:
- (^) Assume each feature is a float (4 × 1010 data bytes).
- The algorithm requires 4. 1 × 1010 bytes storage,
- (^) Query requires about O(10^7 ) byte operations,
- Compared to O(10^10 ) for exhaustive search.
LSH: intuition
Preprocessing: index the data by l hash tables
Hash functions are unlikely to separate close points
LSH: intuition
Preprocessing: index the data by l hash tables
Hash functions are unlikely to separate close points