EECS 501 ESTIMATION: MLE, MAP, LS Fall 2001
Model: Aknown model of system or process with unknown parameter a.
Data: An observation Rof a random variable rwhose pdf depends on a.
Model→fr|a(R|A): If knew a=A, would know pdf of observation r.
Goal: Estimate afrom Rand conditional pdf fr|a(R|A): Compute ˆa(R).
Example: Flip coin 10 times. Data: #heads in 10 independent flips.
Model: Binomial pmf for r.Unknown parameter: a=Pr[heads].
1. Non-Bayesian: ais an unknown constant (do not know fa(A)).
Given: fr|a(R|A) from model; observation (data) Rof rv r; nothing more.
Advantage: Need very little; no (possibly wrong) prior information.
Soln: Maximum Likelihood Estimator: max likelihood of what happened:r=R.
MLE: ˆaM LE (R) = argmax
A[fr|a(R|A)]. Compute: ∂
∂A [log fr|a(R|A)] = 0.
BLUE: Best (minimum variance) Linear Unbiased Estimator of constant x
from y=Hx +v, E[v] = 0 is ˆx(Y) = (H0H)−1H0Y.Proof: p. 290.
2. Bayesian: ais itself random with known a priori pdf fa(A).
Given: fr|a(R|A) from model; fa(A)=a priori info; observation Rof r.
Advantage: Incorporate a priori in estimate, but this better be right!
Soln: min E[c(e)] where e=a−ˆa(r)=error and c(·)=cost=MEP or LSE:
2a. MEP: Min Error Prob: c(e) = ½0 if |e|< ²;
1 if |e|> ².
“close only counts in horseshoes”
“a miss is as good as a mile”
E[c(e)] = 1 −R∞
−∞ dR Rˆa(R)+²
ˆa(R)−²dA fr,a(R, A) = 1 −2²R∞
−∞ fr,a(R, ˆa(R))dR.
This is minimized when fr,a(R, ˆa(R)) maximized for each R.
MAP: Max A Posteriori: ˆaM AP (R) = ar gmax
A[fr|a(R|A)fa(A)] (compare MLE).
Compute: ∂
∂A [log fr|a(R|A) + log fa(A)] = 0. MEP criterion→MAP solution.
2b. LSE: Least Squares Estimation criterion: c(e) = e2. Penalize big errors.
LSE: ˆaLS (R) = E[a|r=R] = RAfr|a(R|A)fa(A)dA
Rfr|a(R|A0)fa(A0)dA0
Denominator just fr(R) :
no effect on argmax of A
Proof: Page 298. Moment of inertia minimized around center of mass.
Bias: Let abe an unknown constant Aact so that fa(A) = δ(A−Aact).
DEF: ˆa(R) is unbiased if E[ˆa(r)] = Aact ↔E[e] = 0. How to compute:
E[ˆa(r)] = R R ˆa(R)fr|a(R|A)δ(A−Aact )dR dA =Rˆa(R)fr|a(R|Aact)dR.
MSE: ˆa(R) unbiased→E[(ˆa(r)−Aact)2] = σ2
ˆa(r)→MSE =variance of ˆa(R).
