EECS 501 ESTIMATOR PROPERTIES Fall 2001
Problem: Let {x1. . . xN}be iidrv with xi∼N(m, σ2) and m, σ2unknown.
Want: To compute ˆmMLE and ˆσ2
MLE based on observations {X1. . . XN}.
Solution: fx1...xN(X1. . . XN) = QN
i=1 1
√2πσ2e−1
2(Xi−m)2/σ2since xiindpt rvs.
Set: 0 = ∂
∂m log fx1...xN=∂
∂m [−N
2log(2π)−N
2log σ2−1
2PN
i=1(Xi−m)2/σ2]
=1
σ2PN
i=1(Xi−m) = 0 →ˆmM LE =1
NPN
i=1 Xi=sample mean.
Set: 0 = ∂
∂σ2log fx1...xN=∂
∂σ2[−N
2log(2π)−N
2log σ2−1
2PN
i=1(Xi−m)2/σ2]
=−N
2
1
σ2+1
2PN
i=1(Xi−m)2/(σ2)2= 0 →ˆσ2
MLE =1
NPN
i=1(Xi−m)2.
Replace min ˆσ2
MLE with ˆmMLE →ˆσ2
MLE =sample variance.
Note: ˆσM LE =pˆσ2
M LE : MLE commutes with nonlinear functions g(a).
Why? argmax
Afr|a(R|A) = argmax
g(A)fr|g(a)(R|g(A)). No Jacobian for a→g(a).
Q: What are some desirable properties for estimators to have?
DEF: Unbiased estimator has E[ˆa(x1. . . xN)] = A(xinow treated as rvs).
DEF: Asymptotically unbiased estimator has lim
N→∞E[ˆa(x1. . . xN)] = A.
•Sample mean is unbiased: E[ ˆm] = E[1
NPN
i=1 xi] = 1
NPN
i=1 E[xi] = m.
•Sample variance is biased:E[ˆσ2] = E[1
NPN
i=1(Xi−ˆm)2] = N−1
Nσ2.
•Sample variance is asymp. unbiased: lim
N→∞E[ˆσ2] = lim
N→∞
N−1
Nσ2=σ2.
Note: ˆσ2=1
N−1PN
i=1(Xi−1
NPN
j=1 Xj)2is an unbiased estimator of σ2.
Note: If we know mean m, then the sample variance is unbiased estimator.
Note: Algebra for sample variance biased and consistent is on pp. 274-5.
DEF: Seq. of rvs {a1, a2. . .} → a in probability if lim
N→∞P r[|aN−a|> ²] = 0.
DEF: Consistent estimator has lim
N→∞ˆa(x1. . . xN) = a in probability .
Means: More data helps: The distribution of ˆabecomes tighter around a.
•Sample mean is consistent: Use the Chebyschev inequality:
P r[|ˆm−m|> ²] = P r[|ˆm−E[ ˆm]|> ²]≤σ2
ˆm
²2=σ2
N²2→0 as N→ ∞.
Using: ˆmunbiased and σ2
ˆm=1
N2PN
i=1 σ2=σ2
N. We have just proved the:
Thm: Weak Law of Large Numbers: Let {xi}be iidrv with E[xi], σ2
i<∞.
→The sample mean is a consistent estimator of the expectation E[xi].
Means: Mean ˆmof data approaches mean m=E[xi] of random variables.