CS181 Lecture 21: Instance-Based Methods and Histograms in Machine Learning, Study notes of Artificial Intelligence

Download CS181 Lecture 21: Instance-Based Methods and Histograms in Machine Learning and more Study notes Artificial Intelligence in PDF only on Docsity!

CS181 Lecture 21:

Instance-Based Methods

David C. Parkes

Histograms

• Consider M=2 dimensional box

Histograms

• Count proportion of training instances in each bin
• Estimate P ( x ) = N k

/ N £ V

k where Nx = number of instances in k and N = total number of instances and point x in bin k and V k is volume bin N k

/ N = 3/

x V k

P ( x ) = 2

Histograms: Classification

• Choose most common class in bin

x

k - Nearest Neighbors “Density Estimator”

• Given an instance x
• Find k points with smallest L 1 distance (max distance in any one direction).
• Set cube side length h = 2d max (d max is furthest distance of k points) x k = 3 h = 2/ N = 6 V = 4/ P ( x ) = k /( NV h

Effect of k

30 points

Alternative View

• Consider a hypercube with length h=1/ x

N

B

h = 1/ V h

N = 6

P ( x ) = N B

/ (N V

h

Alternative View

• Consider a hypercube with length h=1/ x

N

B

h = 1/ V h

N = 6

P ( x ) = N B

/ (N V

h

Alternative View

• Consider a hypercube with length h=1/ x

N

B

h = 1/ V h

N = 6

P ( x ) = N B

/ (N V

h

Gaussian Kernel Example

x

K

1

K

2 Estimate P ( x ) = (1/2) * ( K 1

+ K

2

Alternate View

x Estimate P ( x ) = (1/2) * ( K 1

+ K

2

K

2

Gaussian Kernel:

Bimodal Density Function

Don’t get confused – a Gaussian kernel density estimator does not assume the data is Gaussian! x