Maximum Likelihood
Chris Piech
CS109
Handout #35
May 13th, 2016
Consider IID random samples X1,X2,. . . Xnwhere Xiis a sample from the density function f(Xi|θ). We are
going to introduce a new way of choosing parameters called Maximum Likelihood Estimation (MLE). We
want to select that parameters (θ) that make the observed data the most likely. Note that we are now using
notation that shows that the density of X depends on its parameters, θ.
First we define the likelihood of our data give parameters θ:
L(θ) =
n
∏
i=1
f(Xi|θ)
This is the probability of all of our data. It evaluates to a product because all Xiare independent. Now we
chose the value of θthat maximizes the likelihood function. Formally ˆ
θ=argmax
θ
L(θ).
A cool property of argmax is that since log is a monotone function, the argmax of a function is the same
as the argmax of the log of the function! That’s nice because logs make the math simpler. Instead of using
likelihood, you should instead use log likelihood: LL(θ).
LL(θ) = log
n
∏
i=1
f(Xi|θ) =
n
∑
i=1
log f(Xi|θ)
To use a maximum likelihood estimator, first write the log likelihood of the data given your parameters. Then
chose the value of parameters that maximize the log likelihood function. Argmax can be computed in many
ways. Most require computing the first derivative of the function.
Bernoulli MLE Estimation
Consider IID random variables X1,X2,. . . Xnwhere Xi∼Ber(p). First we are going to write the PMF of a
Bernoulli in a crazy way: The probability mass function f(Xi|p) = pXi(1−p)1−Xi. Wow! Whats up with
that? First convince yourself that when Xi=0 and Xi=1 this returns the right probabilities. We write the
PMF this way because its derivable.
Now let’s do some MLE estimation:
L(θ) =
n
∏
i=1
pXi(1−p)1−Xi
LL(θ) =
n
∑
i=1
log pXi(1−p)1−Xi
=
n
∑
i=1
Xi(log p)+(1−Xi)log(1−p)
=Ylog p+ (n−Y)log(1−p)where Y=
n
∑
i=1
Xi
Great Scott! Now we simply need to chose the value of pthat maximizes our log-likelihood. One way to do
that is to find the first derivative and set it equal to 0.
δLL(p)
δp=Y1
p+ (n−Y)−1
1−p=0
ˆp=Y
n=∑n
i=1Xi
n
All that work and we get the same thing as method of moments and sample mean...
