Search in the document preview
Taylor Approximation and the Delta Method
April 28, 2009
1 Taylor Approximation
1.1 Motivating Example: Estimating the odds
Suppose we observe X1, . . . , Xn independent Bernoulli(p) random variables. Typically, we are interested in p but there is also interest in the parameter p1−p , which is known as the odds. For example, if the outcomes of a medical treatment occur with p = 2/3, then the odds of getting better is 2 : 1. Furthermore, if there is another treatment with success probability r, we might also be interested in the odds ratio p1−p/
r 1−r , which gives the relative odds of one treatment over another.
If we wished to estimate p, we would typically estimate this quantity with the observed success probability p̂ =
∑ iXi/n. To estimate the odds, it then seems perfectly natural to use
p̂ 1−p̂ as an
estimate for p1−p . But whereas we know the variance of our estimator p̂ is p(1 − p) (check this be computing var(p̂)), what is the variance of p̂1−p̂? Or, how can we approximate its sampling distribution?
The Delta Method gives a technique for doing this and is based on using a Taylor series approxi- mation.
1.2 The Taylor Series
Definition: If a function g(x) has derivatives of order r, that is g(r)(x) = d r
dxr g(x) exists, then for any constant a, the Taylor polynomial of order r about a is
Tr(x) = r∑
While the Taylor polynomial was introduced as far back as beginning calculus, the major theorem from Taylor is that the remainder from the approximation, namely g(x)− Tr(x), tends to 0 faster than the highest-order term in Tr(x).
Theorem: If g(r)(a) = d r
dxr g(x)|x=a exists, then
g(x)− Tr(x) (x− a)r
∗The material here is almost word for word from pp. 240-245 of Statistical Inference by George Casella and Roger L. Berger and credit is really to them.
For the purposes of the Delta Method, we will only be considering r = 1. Furthermore, we will not be concerned with the remainder term since, (1), we are interested in approximations, and (2), we will have a nice convergence result that says from a probabilistic point of view, the remainder will vanish.
1.3 Applying the Taylor Theorem
Let’s now put the first-order Taylor polynomial to use from a statistical point of view: Let T1, . . . , Tk be random variables with means θ1, . . . , θk, and define T = (T1, . . . , Tk) and θ = (θ1, . . . , θk). Suppose there is a differentiable function g(T ) (say, an estimator of some parameter. In our motivating example, T = p and g(p) = p1−p) for which we want an estimate of variance. Define the partial derivatives as
g′i(θ) = ∂
∂ti g(t)|t1=θ1,...,tk=θk ,
where t = (t1, . . . , tk) is just any k-dimensional coordinates. The first-order Taylor series expansion (this is actually coming from the multivariate version of the Taylor series which shall be addressed later) of g about θ is
g(t) = g(θ) + k∑ i=1
g′i(θ)(ti − θi) + Remainder.
So far, we have done nothing special. Now, let’s turn this into a statistical approximation by bringing in T and dropping the remainder. This gives
g(T ) ≈ g(θ) + k∑ i=1
g′i(θ)(Ti − θi). (1)
Continuing, let’s take expectations on both sides (noticing that everything but the Ti terms on the right-hand side are non-random) to get
Eg(T ) ≈ g(θ) + k∑ i=1
g′i(θ)E(Ti − θi)
= g(θ). (2)
We can also approximate the variance of g(T ) by
Varg(T ) ≈ E[(g(T )− g(θ))2] From Eq. (2).
≈ E ((∑k
i=1g ′ i(θ)(Ti − θi)
)2) From Eq. (1).
= k∑ i=1
g′i(θ) 2VarTi + 2
g′i(θ)g ′ j(θ)Cov(Ti, Tj), (3)
where the last equality comes from expanding the square and using the definitions of variance and covariance. Notice that we have approximated the variance to our estimator g(T ) using only the variances and covariances of the Ti, which if the problem is set up well, are not terribly difficult to compute or estimate. Let’s now put this to work.
1.4 Continuation: Estimating the Odds
Recall that we wanted to gather some properties about p̂1−p̂ as an estimate of p
1−p , where p is a binomial success probability. Using the notation described in the previous section, we take g(p) = p1−p so that g
′(p) = 1 (1−p)2 (this is a univariate this case, so k = 1 and thus there is only one
) ≈ g′(p)2Var(p̂)
1 (1− p)2
)2 p(1− p) n
n(1− p)3 ,
giving us an approximation for the variance of our estimator. ‖
1.5 Example: Approximate Mean and Variance
Suppose X is a random variable with EX = µ 6= 0. If we want to estimate a function g(µ), a first-order approximation like before would give us
g(X) = g(µ) + g′(µ)(X − µ).
Thus, if we use g(X) as an estimator of g(µ), we can say that approximately
Eg(X) ≈ g(µ), Varg(X) ≈ g′(θ)2VarX.
What is the purpose of this? Well, suppose we take g(µ) = 1/µ with µ unknown. If we estimate 1/µ with 1/X, then we can say
) ≈ 1 µ ,
) ≈ (
As we have seen, we can use these Taylor series approximations to estimate the mean and variance estimators. As mentioned earlier, we can generalize this into a convergence result akin to the Central Limit Theorem. This result is known as the Delta Method.
2 The Delta Method
2.1 Slutsky’s Theorem
Before we address the main result, we first state a useful result, named after Eugene Slutsky.
Theorem: (Slutsky’s Theorem) If Wn →W in distribution and Zn → c in probability, where c is a non-random constant, then
WnZn → cW in distribution. Wn + Zn →W + c in distribution.
The proof is omitted.
2.2 Delta Method: A Generalized CLT
Theorem: Let Yn be a sequence of random variables that satisfies √ n(Yn − θ) → N (0, σ2) in
distribution. For a given function and a specific value of θ, suppose that g′(θ) exists and is not 0. Then, √
n(g(Yn)− g(θ))→ N (0, σ2g′(θ)2) in distribution. Proof: The Taylor expansion of g(Yn) around Yn = θ is
g(Yn) = g(θ) + g′(θ)(Yn − θ) + Remainder,
where the remainder → 0 as Yn → θ. From the assumption that Yn satisfies the standard CLT, we have Yn → θ in probability, so it follows that the remainder→ 0 in probability as well. Rearranging terms, we have √
n(g(Yn)− g(θ)) = g′(θ) √ n(Yn − θ) + Remainder.
Applying Slutsky’s Theorem with Wn = g′(θ) √ n(Yn − θ) and Zn as the remainder, we have the
right-hand side converging to N (0, σ2g′(θ)2), and thus the desired result follows.
2.3 Continuation: Approximate Mean and Variance
Before, we considered the case of just estimating g(µ) with g(X). Suppose now we have took an i.i.d. random sample of a population to get X1, . . . , Xn to get a sample mean X̂n1 For µ 6= 0, from the Delta Method we have
X̂ − 1 µ
( 0, (
) in distribution.
This is pretty good! But what if we don’t know the variance of X1? Furthermore, we’re trying to estimate 1/µ and the variance on the right-hand side requires knowledge of µ. This actually poses no major problem since we shall just estimate everything to get the approximate variance
) ≈ (
where σ̂2 is an estimate of the variance of X1, say the sample variance. Now, we know that both X̂ and σ̂2 are consistent estimators in that X̂ → µ and σ̂2 → σ2 in probability. Thus,(
)2 σ̂ →
( 1 µ
)2 σ in probability. This allows us to apply Slutsky’s Theorem to get
√ n (
X̂ − 1µ
( 1 µ
)2 σ̂ ·
√ n (
X̂ − 1µ
→ N (0, 1)
in distribution. It bears pointing out that the written form of the convergence has changed since now our parameters that were once in the limiting distribution are estimates dependent on n. It would not make much sense to have convergence of
√ n (
X̂ − 1µ
) to a distribution with variance
dependent on n. 1The statement of the Delta Method allows for great generality of sequences Yn satisfying the CLT. This is because
there are multiple forms of the CLT. Typically, the sample mean is used in these types of approximations and from elementary probability, we know the sample mean is one such sequence of random variables that satisfies the CLT.
3 Second-Order Delta Method
A natural question to ask is, in all the above work, what happens if g′(θ) = 0? To answer this, we go back to the Taylor expansion. Using the notation from the Delta Method theorem, we add the second-order term to get
g(Yn) = g(θ) + g′(θ)(Yn − θ) + g′′(θ)
2 (Yn − θ)2 + Remainder.
Since g′(θ) = 0, this gives
g(Yn)− g(θ) = g′′(θ)
2 (Yn − θ)2 + Remainder.
Now, just like √ n(Yn − θ)/σ2 → N (0, 1) in distribution, we also have
n(Yn − θ)2
σ → χ21
in distribution where χ21 is a chi-squared random variable with 1 degree of freedom. This new convergence is all very natural because we are now dealing with a second-order term. The first- order approximation converged to a Gaussian random variable so we could reasonably guess that the second-order term would converge to the square of a Gaussian, which just so happens to be a chi-squared random variable. In precise terms, we give the Second-Order Delta Method:
Theorem: (Second-Order Delta Method) Let Yn be a sequence of random variables that satisfies
√ n(Yn − θ) → N (0, σ2) in distribution. For a given function g and a specific value of θ,
suppose that g′(θ) = 0 and ′′(θ) exists and is not 0. Then
n(g(Yn)− g(θ))→ σ2 g′′(θ)
2 χ21 in distribution.
4 Multivariate Delta Method
We have actually already seen the multivariate precursor to the multivariate extension to the Delta Method. We use an example to illustrate the usage.
4.1 Moments of a Ratio Estimator
Suppose X and Y are random variables with nonsero means µX and µY , respectively. The para- metric function to be estimated is g(µX , µY ) = µX/µY . It is straightforward to calculate
∂µX g(µX , µY ) =
∂µY g(µX , µY ) = −
Using Equations (2) and (3), we have
E ( X
) ≈ µX µY
Var ( X
) ≈ 1 µ2Y
VarX + µ2X µ4Y
VarY − 2µX µ3Y
= ( µX µY
+ VarY µ2Y
− 2Cov(X,Y ) µXµY
We now have an approximation that can be built up from estimates of the means, variances, and covariances of the individual random variables X and Y .‖
We now come to the finale, the multivariate version of the Delta Method. First, we set out a few definitions. Define the random vector X = (X1, . . . , Xp) with mean µ = (µ1, . . . , µp) and covariances Cov(Xi, Xj) = σij . We shall observe n i.i.d. random ramples of the population of X and denote these samples as X(1), . . . , X(n). Furthermore, we shall call the sample means for each element of the vector X̂i =
(k) i , i = 1, . . . , p and X̂ as the vector of sample means. Lastly,
we consider the multivariate function g : R 7→ R with g(x) = g(x1, . . . , xp) and use (1) to write
g(X̂1, . . . , X̂p) ≈ g(µ1, . . . , µp) + p∑ i=1
g′i(µi)(X̂i − µi).
In vector notation, this is g(X̂) ≈ g(µ) +∇T g(µ)(X̂ − µ),
with the abuse of notation that ∇T g(µ) = (∇T g(x))|x=µ.
Theorem: (Multivariate Delta Method) Let X(1), . . . , X(n) be a random sample with EX(k)i = µi and Cov(X
(k) i , X
(k) j ) = σij. For a given function g with continuous first partial derivatives and
a specific value of µ = (µ1, . . . , µp) for which τ2 = ∑
∑ j σijg
′ j(µ) > 0,
√ n(g(X̂(1), . . . , X̂(n))− g(µ1, . . . , µp))→ N (0, τ2) in distribution.
In vector form, this is
Theorem: (Multivariate Delta Method in Vector Form) Let X(1), . . . , X(n) be a random sample with EX(i) = µ and covariance matrix E(X(i)− µ)(X(i)− µ)T = Σ. For a given function g with continuous first partial derivatives and a specific value of µ for which τ2 = ∇T g(µ)Σ∇g(µ) > 0,
√ n(g(X̂)− g(µ))→ N (0, τ2) in distribution.