CS 4803B/8803B: Pattern Recognition
Problem Set 4
Date: Mar 27, 2001 Due: April 5, 2001
For the first two problems of this problem set you will compare two different non-parametric
classification techniques in the same data set. The training and testing data can be obtained from
the class Web page (files PS4 1train.mat and PS4 1test.mat) These are two classes with equal prior
probabilities and scalar-valued features. Each classification technique has a free parameter which you
will estimate via the technique of leave-one-out validation (to be explained).
1. Use the Parzen window technique in DHS 4.3 to estimate the density of each class. Use the
Gaussian window (equation (27) in DHS). Unfortunately, the width parameter
h
is still to be
determined for each class.
To estimate
h
, you will use the amazing technique of leave-one-out validation. The idea is to
compute an evidence curve
v
(
h
)
which approximately represents the likelihood of
h
given the
data. Then you can choose the
h
which maximizes
v
(
h
)
. Each class has its own density so this
needs to be done separately for each class. Here's what you do to compute
v
(
h
)
for a particular
value of
h
:
(a) Break the training data into two parts
A
and
B
,where
B
contains only a single sample.
This is what makes the method “leave-one-out” validation.
(b) Compute the Parzen window density estimate
p
n
(
x
)
from
A
only.
(c) Compute
log(
p
n
(
B
))
, the log-probability of the sample you held out.
(d) Iterate back to part (a) and break the training data up in a different way. Do this for every
possible choice of
B
. The average of all of the log-probabilities you get in part (c) is
defined to be
v
(
h
)
.
For each class, plot the evidence curve. Pick three values of
h
from different parts of the curve
and plot the estimated density using each. What would your intuition say is the best
h
?How
well does the evidence curve match your intuition?
Now design a minimum error rate classifier using the
h
that maximizes
v
(
h
)
for each class.
(Remember, minimum error rate would mean posterior is greatest. With equal priors, simply
choose class with greater likelihood - right?) What is its performance on the test data?
2. This time use the
k
-nearest neighbor method in DHS 4.5.4. Here we do not know the right value
of
k
and will estimate it using an evidence curve
v
(
k
)
. Here' s what you do for a particular
k
:
(a) Break the training data into two parts
A
and
B
,where
B
contains only a single sample.
(b) Determine if
B
is classified correctly by its
k
-nearest neighbors from
A
.
1
