Lecture Slides on Mathematical Statistics | STAT 710, Study notes of Mathematical Statistics

Download Lecture Slides on Mathematical Statistics | STAT 710 and more Study notes Mathematical Statistics in PDF only on Docsity!

logo

Stat 710: Mathematical Statistics

Lecture 14

Jun Shao

Department of Statistics University of Wisconsin Madison, WI 53706, USA

logo

Lecture 14: Asymptotically efficient estimation

Scoring and RLE

The method of estimating θ by solving sn ( γ) = 0 over γ ∈ Θ is called scoring and the function sn ( γ) is called the score function. RLE’s are not necessarily MLE’s. We may use the techniques discussed in §4.4 to check whether an RLE is an MLE. However, according to Theorem 4.17, when a sequence of RLE’s is consistent, then it is asymptotically efficient. We may not need to search for MLE’s, if asymptotic efficiency is the only criterion to select estimators. Typically a sequence of MLE’s is consistenct (and asymptotically efficient), although there are examples in which an RLE sequence is consistent but an MLE sequence is not.

logo

Example 4.39 (continued)

If θ̂ n = n −^1 ∑ ni = 1 T ( Xi ) ∈ Θ, the range of θ = g ( η) = ∂ ζ ( η)/ ∂ η, then θ̂ n is a unique RLE of θ , which is also a unique MLE of θ since ∂ 2 ζ (η)/∂ η∂ ητ^ = Var( T ( Xi )) is positive definite. Also, η = g −^1 ( θ ) exists and a unique RLE (MLE) of η is η̂ n = g −^1 ( θ̂ n ). However, θ̂ n may not be in Θ and the previous argument fails (e.g., Example 4.29). What Theorem 4.17 tells us in this case is that as n → ∞, P ( θ̂ n ∈ Θ) → 1 and, therefore, θ̂ n (or η̂ n ) is the unique asymptotically efficient RLE (MLE) of θ (or η) in the limiting sense. In an example like this we may directly show that P ( θ̂ n ∈ Θ) → 1, using the fact that θ̂ n → a. s. E [ T ( X 1 )] = g ( η) (the SLLN).

The next theorem provides a similar result for the MLE or RLE in the GLM (§4.4.2). Its proof is similar to the proof of Theorem 4.17.

logo

Example 4.39 (continued)

If θ̂ n = n −^1 ∑ ni = 1 T ( Xi ) ∈ Θ, the range of θ = g ( η) = ∂ ζ ( η)/ ∂ η, then θ̂ n is a unique RLE of θ , which is also a unique MLE of θ since ∂ 2 ζ (η)/∂ η∂ ητ^ = Var( T ( Xi )) is positive definite. Also, η = g −^1 ( θ ) exists and a unique RLE (MLE) of η is η̂ n = g −^1 ( θ̂ n ). However, θ̂ n may not be in Θ and the previous argument fails (e.g., Example 4.29). What Theorem 4.17 tells us in this case is that as n → ∞, P ( θ̂ n ∈ Θ) → 1 and, therefore, θ̂ n (or η̂ n ) is the unique asymptotically efficient RLE (MLE) of θ (or η) in the limiting sense. In an example like this we may directly show that P ( θ̂ n ∈ Θ) → 1, using the fact that θ̂ n → a. s. E [ T ( X 1 )] = g ( η) (the SLLN).

The next theorem provides a similar result for the MLE or RLE in the GLM (§4.4.2). Its proof is similar to the proof of Theorem 4.17.

logo

One-Step MLE

Assume the conditions in Theorem 4.16. Let sn (γ)^ be the score function.

Let θ̂ (^) n (^0 )be an estimator of θ that may not be asymptotically efficient. The estimator ̂ θ (^) n (^1 )= θ̂ (^) n (^0 )− [∇ sn ( θ̂ (^) n (^0 ))]−^1 sn ( θ̂ (^) n (^0 ))

is the first iteration in computing an MLE (or RLE) using the

Newton-Raphson iteration method with θ̂ (^) n (^0 )as the initial value and, therefore, is called the one-step MLE.

Without any further iteration, θ̂ (^) n (^1 )is asymptotically efficient under some conditions.

Theorem 4.

Assume that the conditions in Theorem 4.16 hold and that√ θ̂ (^) n (^0 )is n -consistent for θ (Definition 2.10).

(i) The one-step MLE θ̂ (^) n (^1 )is asymptotically efficient. (ii) The one-step MLE obtained by replacing ∇ sn ( γ) with its expected value, − In (γ) (the Fisher-scoring method), is asymptotically efficient.

logo

One-Step MLE

Assume the conditions in Theorem 4.16. Let sn (γ)^ be the score function.

Let θ̂ (^) n (^0 )be an estimator of θ that may not be asymptotically efficient. The estimator ̂ θ (^) n (^1 )= θ̂ (^) n (^0 )− [∇ sn ( θ̂ (^) n (^0 ))]−^1 sn ( θ̂ (^) n (^0 ))

is the first iteration in computing an MLE (or RLE) using the

Newton-Raphson iteration method with θ̂ (^) n (^0 )as the initial value and, therefore, is called the one-step MLE.

Without any further iteration, θ̂ (^) n (^1 )is asymptotically efficient under some conditions.

Theorem 4.

Assume that the conditions in Theorem 4.16 hold and that√ θ̂ (^) n (^0 )is n -consistent for θ (Definition 2.10).

(i) The one-step MLE θ̂ (^) n (^1 )is asymptotically efficient. (ii) The one-step MLE obtained by replacing ∇ sn ( γ) with its expected value, − In (γ) (the Fisher-scoring method), is asymptotically efficient.

logo

Proof (continued)

Using an argument similar to those in the proof of Theorem 4.17, we can show that ‖ Gn ( θ̂ (^) n (^0 )) − Ik ‖ → p 0.

These results and the fact that

n ( θ̂ (^) n (^0 )− θ ) = Op ( 1 ) imply √ n ( θ̂ (^) n (^1 )− θ ) =

n [ In ( θ )]−^1 sn ( θ ) + op ( 1 ).

This proves (i). The proof for (ii) is similar.

Example 4.

Let X 1 , ..., Xn be i.i.d. from the Weibull distribution W ( θ , 1 ), where θ > 0 is unknown. Note that

sn ( θ ) =

n θ

n

i = 1

log Xi −

n

i = 1

X i^ θ log Xi

logo

Proof (continued)

Using an argument similar to those in the proof of Theorem 4.17, we can show that ‖ Gn ( θ̂ (^) n (^0 )) − Ik ‖ → p 0.

These results and the fact that

n ( θ̂ (^) n (^0 )− θ ) = Op ( 1 ) imply √ n ( θ̂ (^) n (^1 )− θ ) =

n [ In ( θ )]−^1 sn ( θ ) + op ( 1 ).

This proves (i). The proof for (ii) is similar.

Example 4.

Let X 1 , ..., Xn be i.i.d. from the Weibull distribution W ( θ , 1 ), where θ > 0 is unknown. Note that

sn ( θ ) =

n θ

n

i = 1

log Xi −

n

i = 1

X i^ θ log Xi

logo

Example 40 (continued)

Then

∇ sn ( θ ) = −

n θ 2

n

i = 1

X i^ θ (log Xi )^2.

Hence, the one-step MLE of θ is

θ̂ (^) n (^1 )= θ̂ (^) n (^0 )

 1 + n^ + θ̂ (^) n (^0 )(∑ ni = 1 log Xi − (^) ∑ ni = 1 X^ θ̂^ n (^0 ) i log^ Xi^ ) n + ( θ̂ (^) n (^0 ))^2 ∑ ni = 1 X θ̂ (^) n (^0 ) i (log^ Xi^ )

2

Usually one can use a moment estimator (§3.5.2) as the initial

estimator θ̂ (^) n (^0 ). In this example, a moment estimator of θ is the solution of X^ ¯ = Γ(θ −^1 + 1 ).

Results similar to that in Theorem 4.19 can be obtained in the GLM.

logo

Bayes estimators

Bayes estimators are often asymptotically efficient It can be checked if explicit forms of Bayes estimators are available. The following is a general result.

Theorem 4.

Assume the conditions of Theorem 4.16. Let π(γ) be a prior p.d.f. (which may be improper) w.r.t. the Lebesgue measure on Θ and pn ( γ) be the posterior p.d.f., given X 1 , ..., Xn , n = 1 , 2 , .... Assume that there exists an n 0 such that pn 0 ( γ) is continuous and positive for all γ ∈ Θ,

∫ pn 0 ( γ) d γ = 1 and

∫ ‖ γ‖ pn 0 ( γ) d γ < ∞. Suppose further that, for any ε > 0, there exists a δ > 0 such that

n lim→∞ P

sup ‖ γ− θ ‖≥ ε

log ℓ( γ) − log ℓ( θ ) n

− δ

and

logo

Theorem 4.20 (continued)

n^ lim→∞ P

sup ‖ γ− θ ‖≤ δ

‖∇ sn ( γ) − ∇ sn ( θ )‖ n

≥ ε

where ℓ(γ) is the likelihood function and sn ( γ) is the score function. (i) Let p ∗ n (γ) be the posterior p.d.f. of

n ( γ − Tn ), where Tn = θ + [ In ( θ )]−^1 sn ( θ ) and θ is the true parameter value, and let ψ( γ) be the p.d.f. of Nk ( 0 , [ I 1 ( θ )]−^1 ). Then (^) ∫

( 1 + ‖ γ‖)

p ∗ n ( γ) − ψ( γ)

d γ → p 0.

(ii) The Bayes estimator of θ under the squared error loss is asymptotically efficient.

Proof: omitted

logo

Conclusions from Theorem 4.

Result in (i) shows that the posterior p.d.f. is approximately normal with mean θ + [ In ( θ )]−^1 sn ( θ ) and covariance matrix [ In ( θ )]−^1. This result is useful in Bayesian computation; see Berger (1985, §4.9.3). Result (i) shows that the posterior distribution and its first-order moments converge to the degenerate distribution at θ and its first-order moments, which implies the consistency and asymptotic unbiasedness of Bayes estimators such as the posterior means. The Bayes estimator under the squared error loss is asymptotically efficient, which provides an additional support for the early suggestion that the Bayesian approach is a useful method for generating estimators. The results hold regardless of the prior being used, indicating that the effect of the prior declines as n increases.