Download Solving Likelihood Equations using Newton's Method in Statistical Inference and more Assignments Statistics in PDF only on Docsity!
Stat 411 – Homework 03 Due: Friday 02/
- Problem 6.1.3 on pages 317–318.
- Problem 6.1.6 on page 318. [Hint: Refer back to Problem 6.1.3(c).]
- Problem 6.1.9 on page 318. [Hint: Refer back to Problem 6.1.3(a).]
- Problem 6.1.11 on page 319.
- Let θ be a scalar (one-dimensional) parameter and let (θ) denote the log-likelihood function. Then the MLE θˆ is a solution of the likelihood equation′(θ) = 0. It may happen that there is a unique solution to the likelihood equation, but there’s no formula for it (e.g., in a gamma shape parameter problem). In such cases, the solution must be found numerically. A powerful method for such problems is Newton’s method, covered in early calculus courses. Describe how Newton’s method can be used to solve the likelihood equation. You may assume that `(θ) is at least twice differentiable.
- (Graduate Only) Consider independent samples from two normal populations:
X 11 ,... , X 1 n 1 iid ∼ N(θ 1 , 1) and X 21 ,... , X 2 n 2 iid ∼ N(θ 2 , 1),
independent throughout. Suppose that θ 1 ≤ θ 2. Find the MLE of (θ 1 , θ 2 ). You may find it helpful to sketch the parameter space Θ = {(θ 1 , θ 2 ) : θ 1 ≤ θ 2 }.
Stat 411 – Homework 03 Solutions
- Problem 6.1.3.
(a) Let X 1 ,... , Xn be iid with PMF fθ(x) = e−θθx/x!, x = 0, 1 , 2 ,... (a Poisson distribution). Then the likelihood function is
Lx(θ) =
∏^ n
i=
e−θ^
θxi xi!
= const · e−nθθx^1 +···+xn^.
Taking log and then derivative gives
0 set =
∂θ
log Lx(θ) =
∂θ
[
const − nθ + (x 1 + · · · + xn) log θ
]
= −n +
nx θ
From here it’s clear that the MLE is θˆ = x. (b) Let X 1 ,... , Xn be iid with PDF fθ(x) = θxθ−^1 , x ∈ (0, 1) (a beta distribution). The likelihood function is
Lx(θ) =
∏^ n
i=
θxθ i −^1 = θn
( (^) ∏n
i=
xi
)θ− 1 .
Taking log and then derivative gives
set
∂θ
log Lx(θ) =
∂θ
[
n log θ + (θ − 1)
∑^ n
i=
log xi
]
n θ
∑^ n
i=
log xi.
Therefore, the MLE is θˆ = −n(
∑n i=1 log^ xi)
(c) Let X 1 ,... , Xn be iid with PDF fθ(x) = (1/θ)e−x/θ, x > 0 (an exponential distribution with mean θ). The likelihood equation is
Lx(θ) =
∏^ n
i=
θ
e−xi/θ^ =
θn^
e−(x^1 +···+xn)/θ.
Taking log and then derivative gives
0 set =
∂θ
log Lx(θ) =
∂θ
[
−n log θ − nx/θ
]
n θ
nx θ^2
Then the MLE is clearly θˆ = x. (d) Let X 1 ,... , Xn be iid with PDF fθ(x) = e−(x−θ)I[θ,∞)(x) (a shifted exponential distribution). Here, like in the uniform problem from class, we use the indicator function because the support set depends on θ. The likelihood function is
Lx(θ) =
∏^ n
i=
e−(xi−θ)I[θ,∞)(xi) ∝ enθI[θ,∞)(x(1)).
For the right-hand side, the logic is as follows. First, the proportionality constant is e−nx, which can be ignored because it has nothing to do with θ. Second, the product of indicators is 1 if and only if all the indicators are 1; all the indicators are 1 if and only if all the Xi’s are ≥ θ; all the data are ≥ θ if and only if X(1), the sample minimum, is ≥ θ. But enθ^ is strictly increasing and, since θ ≤ x(1), it must be that θˆ = x(1).
- (Graduate only) Let L 12 (θ 1 , θ 2 ) be the full likelihood function, depending on both sample 1 and sample 2. Moreover, let L 1 (θ 1 ) and L 2 (θ 2 ) be the sample-specific likelihood functions. Because the samples are assumed to be independent, the following factorization holds:
L 12 (θ 1 , θ 2 ) = L 1 (θ 1 ) · L 2 (θ 2 ).
Therefore, the maximizer of L 12 (θ 1 , θ 2 ) is just the pair of individual maximiz- ers. Since the two sample-specific likelihoods are normal, the calculation in Prob- lem 6.1.11 above shows that (θˆ 1 , θˆ 2 ) = (x 1 , x 2 ), the pair of sample means. But it is known that θ 2 ≥ θ 1 , i.e., Θ = {(θ 1 , θ 2 ) : θ 2 ≥ θ 1 }. If (x 1 , x 2 ) is inside Θ (i.e., if x 2 ≥ x 1 ), then the MLE is just (x 1 , x 2 ). But it is possible that x 1 > x 2 and, in such a case, an adjustment must be made. The adjustment, as usual, is to push (x 1 , x 2 ) to the closest point in Θ. There’s a number of ways to find this closest point, and here I list two. (Drawing a picture would help.)
- A calculus-based solution. This closest point will obviously be on the boundary of Θ, of the form (y, y). So define a function h(y) = (y − x 1 )^2 + (y − x 2 )^2 that describes the distance between (x 1 , x 2 ) and points (y, y) on the boundary of Θ. Now minimize this function with respect to y by differentiating, etc. One easily finds that the minimum is attained at y = (x 1 + x 2 )/2.
- A geometry-based solution. This closest point is the orthogonal projection of (x 1 , x 2 ) onto Θ. So you need to find a line perpendicular to the diagonal y = x that passes through the point (x 1 , x 2 ). The equation of this perpendicular line is y = −x + (x 1 + x 2 ). Summing these two equations gives 2y = x 1 + x 2 or, equivalently, y = (x 1 + x 2 )/2.
So the MLE in this constrained parameter problem is
ˆθ =
(x 1 , x 2 ) if x 2 ≥ x 1 (x^1 + 2 x^2 , x^1 + 2 x^2 ) otherwise.
