Mixture Densities - Pattern Recognition - Lecture Slides, Slides of Mechanical Engineering

Download Mixture Densities - Pattern Recognition - Lecture Slides and more Slides Mechanical Engineering in PDF only on Docsity!

Recap

• In the last class we considered ML estimation ofmixture densities. PR NPTEL course – p.1/

Recap

• In the last class we considered ML estimation ofmixture densities. - We considered the example of learning a mixture ofGaussian densities. PR NPTEL course – p.2/

Recap

• In the last class we considered ML estimation ofmixture densities. - We considered the example of learning a mixture ofGaussian densities. - We obtained an iterative algorithm to maximize thelog likelihood. - We introduced the EM algorithm and derived theiterative scheme for a Gaussian mixture. PR NPTEL course – p.4/

The EM Algorithm

• We think of the given data x i , i

, n , as incomplete data. PR NPTEL course – p.5/

The EM Algorithm

• We think of the given data x i , i

, n , as incomplete data.

• We formulate the missing data

Z

i , i

, n so that given complete data,

x i

, Z

i

, i

, n , the estimation is easy.

• Then EM algorithm is a general iterative scheme toestimate the parameters. PR NPTEL course – p.7/

The EM Algorithm

• We think of the given data x i , i

, n , as incomplete data.

• We formulate the missing data

Z

i , i

, n so that given complete data,

x i

, Z

i

, i

, n , the estimation is easy.

• Then EM algorithm is a general iterative scheme toestimate the parameters. - It consists of two steps: Expectation andMaximization. PR NPTEL course – p.8/

• The two steps of EM algorithm are as follows: E-step : Compute

Q

θ, θ ( k )

which is expectation of the complete data loglikelihood w.r.t. the conditionaldistribution of hidden variables conditioned onincomplete data and current value of θ as θ ( k ) . PR NPTEL course – p.10/

• The two steps of EM algorithm are as follows: E-step : Compute

Q

θ, θ ( k )

which is expectation of the complete data loglikelihood w.r.t. the conditionaldistribution of hidden variables conditioned onincomplete data and current value of θ as θ ( k ) . Q

θ, θ ( k )

E

Z |x ,θ ( k ) ln( f

x

Z

θ

PR NPTEL course – p.11/

• The two steps of EM algorithm are as follows: E-step : Compute

Q

θ, θ ( k )

which is expectation of the complete data loglikelihood w.r.t. the conditionaldistribution of hidden variables conditioned onincomplete data and current value of θ as θ ( k ) . Q

θ, θ ( k )

E

Z |x ,θ ( k ) ln( f

x

Z

θ

M-step : Compute next value of θ as θ ( k +1) by maximizing Q

θ, θ ( k )

over θ . θ ( k +1) = arg max θ

Q

θ, θ ( k ) )^ PR NPTEL course – p.13/

Recall Example

• We have considered estimation of two componentGaussian mixture f

x

θ

λ 1 φ

x

θ 1

λ 2 φ

x

θ 2

PR NPTEL course – p.14/

• The complete data density is f

x i

, Z

i

θ

2 ∏^ j =

λ j φ

x i

θ j

Z ij PR NPTEL course – p.16/

• The complete data density is f

x i

, Z

i

θ

2 ∏^ j =

λ j φ

x i

θ j

Z ij

• The complete data log likelihood is ln( f

x

Z

θ

n ∑ i =

[

2 ∑^ j =

Z

ij ln( λ j φ

x i

θ j

]

PR NPTEL course – p.17/

Recall Example: the E-step

• Thus, under the E-step, we get Q

θ, θ ( k )

n ∑ i =

[

2 ∑^ j =

E

[

Z

ij

x , θ ( k ) ] ln( λ j φ

x i

θ j

]

PR NPTEL course – p.19/

Recall Example: the E-step

• Thus, under the E-step, we get Q

θ, θ ( k )

n ∑ i =

[

2 ∑^ j =

E

[

Z

ij

x , θ ( k ) ] ln( λ j φ

x i

θ j

]

n ∑ i =

[

2 ∑^ j = γ ij

θ ( k ) ) ln( λ j φ

x i

θ j

]

PR NPTEL course – p.20/