Docsity
Docsity
Prepare for your exams
Prepare for your exams

Study with the several resources on Docsity

Earn points to download
Earn points to download

Earn points by helping other students or get them with a premium plan

Guidelines and tips
Guidelines and tips
Sell on Docsity
Sell on Docsity
Docsity AI
Docsity AI
Log inSign up

Prepare for your exams

Study with the several resources on Docsity

Find documents

Prepare for your exams with the study notes shared by other students like you on Docsity

Search for your university

Find the specific documents for your university's exams

Docsity AINEW

Summarize your documents, ask them questions, convert them into quizzes and concept maps

Explore questions

Clear up your doubts by reading the answers to questions asked by your fellow students

Earn points to download

Earn points by helping other students or get them with a premium plan

Share documents
download point icon20 Points

For each uploaded document

Answer questions
download point icon5 Points

For each given answer (max 1 per day)

All the ways to get free points
Get points immediately

Choose a premium plan with all the points you need

Study Opportunities

Choose your next study program

Get in touch with the best universities in the world. Search through thousands of universities and official partners

Community

Ask the community

Ask the community for help and clear up your study doubts

Free resources

Our save-the-student-ebooks!

Download our free guides on studying techniques, anxiety management strategies, and thesis advice from Docsity tutors

CS446 Problem Set 6: Pattern Recognition and Machine Learning - Prof. Dan Roth, Assignments of Computer Science

University of Illinois - Urbana-ChampaignComputer Science
Prof. Dan Roth

A problem set for cs446: pattern recognition and machine learning course, which was handed out in fall 2008. It includes two main problems: the first one is about tree dependent distributions and showing that the choice of a root node for directing the tree does not matter. The second problem is about deriving an expectation-maximization (em) algorithm to estimate unknown parameters in a given distribution. The problem set is due on december 5, 2008.

Typology: Assignments

Pre 2010

Uploaded on 03/10/2009

koofers-user-aks
koofers-user-aks 🇺🇸

10 documents

1 / 2

Toggle sidebar

This page cannot be seen from the preview

Don't miss anything!

bg1
CS446: Pattern Recognition and Machine Learning Fall 2008
Problem Set 6
Handed Out: November 20, 2008 Due: December 5, 2008
•Feel free to talk to other members of the class in doing the homework. I am more concerned that
you learn how to solve the problem than that you demonstrate that you solved it entirely on your
own. You should, however, write down your solution yourself. Please try to keep the solution brief
and clear.
•Feel free to send me email or come to ask questions regarding this handout or conceptual issues.
•Please, no hard copy solutions.
•The homework is due at 4:00 pm on the due date. Email your write-up to the TA. Please put
“<userid>CS446 hw6 submission” as the subject line of the email when you submit your homework
to [email protected].
1. [Tree Dependent Distributions - 25 points]
Assume an undirected tree Tobtained by the algorithm described in class for learning
tree dependent distributions. We would like to show that the step of directing the tree
by choosing an arbitrary node is okay.
(a) State exactly what is meant by the statement “the two directed trees obtained
from Tare the same”.
(b) Show that no matter which node in Tis chosen as a root for the “direction” stage,
the resulting directed trees are all the same (based on your definition above).
2. [Deriving an Expectation-Maximization Algorithm - 75 points]
Assume we have a set of data points x∈ {0,1}n. Denote the ith bit of the jth example
as x(j)
i. Thus, the index iranges from 1 . . . n, and the index jranges from 1 . . . m.
Assume these data points were generated according to the following distribution:
Postulate a hidden random variable Ywith values y= 1,2, where the probability of
y= 1 is pand the probability of y= 2 is 1 −p.
For a specific example x(j), a random value of Yis chosen, but its true value yis
hidden. Note that each example x(j)has a fixed underlying y. If y= 1, x(j)
iis set to
1 with probability αi. If y= 2, x(j)
iis set to 1 with probability βi. Thus, there are
2n+ 1 unknown parameters. You will use EM to develop an algorithm to estimate
these unknown parameters.
1
pf2

Partial preview of the text

Download CS446 Problem Set 6: Pattern Recognition and Machine Learning - Prof. Dan Roth and more Assignments Computer Science in PDF only on Docsity!

CS446: Pattern Recognition and Machine Learning Fall 2008

Problem Set 6

Handed Out: November 20, 2008 Due: December 5, 2008

  • Feel free to talk to other members of the class in doing the homework. I am more concerned that you learn how to solve the problem than that you demonstrate that you solved it entirely on your own. You should, however, write down your solution yourself. Please try to keep the solution brief and clear.
  • Feel free to send me email or come to ask questions regarding this handout or conceptual issues.
  • Please, no hard copy solutions.
  • The homework is due at 4:00 pm on the due date. Email your write-up to the TA. Please put “ CS446 hw6 submission” as the subject line of the email when you submit your homework to [email protected].
  1. [Tree Dependent Distributions - 25 points] Assume an undirected tree T obtained by the algorithm described in class for learning tree dependent distributions. We would like to show that the step of directing the tree by choosing an arbitrary node is okay.

(a) State exactly what is meant by the statement “the two directed trees obtained from T are the same”. (b) Show that no matter which node in T is chosen as a root for the “direction” stage, the resulting directed trees are all the same (based on your definition above).

  1. [Deriving an Expectation-Maximization Algorithm - 75 points] Assume we have a set of data points x ∈ { 0 , 1 }n. Denote the ith bit of the jth example as x( i j). Thus, the index i ranges from 1... n, and the index j ranges from 1... m. Assume these data points were generated according to the following distribution: Postulate a hidden random variable Y with values y = 1, 2, where the probability of y = 1 is p and the probability of y = 2 is 1 − p. For a specific example x(j), a random value of Y is chosen, but its true value y is hidden. Note that each example x(j)^ has a fixed underlying y. If y = 1, x( i j)is set to 1 with probability αi. If y = 2, x( i j)is set to 1 with probability βi. Thus, there are 2 n + 1 unknown parameters. You will use EM to develop an algorithm to estimate these unknown parameters.

(a) Express P (x(j)) first in terms of conditional probabilities and then in terms of the unknown parameters p, αi, βi.

(b) Let q( yj )= P (Y = y|x(j)), i.e., the probability that the data point x(j)^ has y as

the value of its hidden variable Y. Express q( 1 j )and q( 2 j )in terms of the unknown parameters. (c) Derive an expression for the expected log likelihood (LL) of the entire data set x(1), x(2),... , x(m)^ and its associated y settings given new parameter estimates p,˜ α˜i, β˜i.

(d) Maximize the LL and determine the update rules for the parameters according to the EM algorithm. (e) Examine the update rules and try to explain them in English. Describe in English how you would run the algorithm: initialization, iteration, termination. What equations would you use at which steps in the algorithm? (f) Assume that your task is to predict the value of Y given an assignment to all n variables and that you have the parameters of the model. Show how to use these parameters to predict Y.

(g) Show that the decision surface for this prediction is a linear function of the xi’s.