Download Functional Delta Method and Bootstrap - Lecture Notes | STAT 210A and more Study notes Data Analysis & Statistical Methods in PDF only on Docsity!
Stat210B: Theoretical Statistics Lecture Date: May 1, 2007
Functional Delta Method and Bootstrap
Lecturer: Michael I. Jordan Scribe: Arash Ali Amini
1 Functional Delta Method
Example 1 (quantile function continued). Recall the definition of the quantile function F − 1 (p) = inf{x :
F (x) ≥ p}. Let’s assume for simplicity that p = F F − 1 (p). To obtain the influence function we need to
differentiate implicitly using this relation. We have done this in the previous lecture and obtained
IFF (x) = −
(^1) [x,+∞)
F −^1 (p)
− p
f
F −^1 (p)
where f (·) is the density associated with F. In deriving (1), we have also assumed differentiability of F and
positivity of f at the quantile. The graph of the influence function is given in Figure 1. It is easy to show
F −^1 (p)
p f
( F −^1 (p)
)
− 1 −p f
( F −^1 (p)
)
Figure 1: Influence function of the pth quantile
that E
IFF (X)
= 0. Let’s compute the variance:
γ 2 (F ) = Var
IFF (X)
(^1) [x,+∞)
F
− 1 (p)
− p
f
F −^1 (p)
dF (x)
f 2
F −^1 (p)
[
(^1) [x,+∞)
F
− 1 (p)
− 2 p · (^1) [x,+∞)
F
− 1 (p)
]
dF (x)
f 2
F −^1 (p)
[
F
F
− 1 (p)
− 2 p F
F
− 1 (p)
]
where the last equality follows because 1x,+∞)(y) = 1{x ≤ y} = 1(−∞,y. Reusing our assumption
p = F F −^1 (p), we get
γ 2 (F ) =
p (1 − p)
f 2
F −^1 (p)
which (correctly) suggests that
n
Fn − 1 (p) − F − 1 (p)
d −→ N
p (1 − p)
f 2
F −^1 (p)
Let’s now derive this result rigorously using Hadamard-differentiability. We will need the following lemma
which we state without proof:
Lemma 2. (van der Vaart, 1998, Lemma 21.3, p. 306) Let F be differentiable at a point ξp ∈ (a, b) such
that F (ξp) = p with F ′(ξp) > 0. Then φ(F ) = F −^1 (p) is Hadamard-differentiable at F tangentially to the
set of functions h that are continuous at ξp, with derivative
φ ′ F (h) =^ −^
h(ξp)
F ′(ξp)
Using a variation of the functional delta method, namely the second part of Thm 20.8 form van der Vaart
(1998, p. 297), we conclude that
n
Fn
− 1 (p) − F − 1 (p)
is asymptotically equivalent to φ ′ F evaluated at √ n
Fn − F
. Using the lemma, this means
n
Fn
− 1 (p) − F − 1 (p)
n
Fn − F
(ξp)
f
F −^1 (p)
) (^) + op(1).
Expanding Fn and rearranging, we get
n
Fn
− 1 (p) − F − 1 (p)
n
1 n
i 1 {^ Xi≤F^ −^1 (p)}^ −^ p
f
F −^1 (p)
) (^) + op(1)
n
i
(^1) { Xi≤F −^1 (p)} − p
f
F −^1 (p)
Note that the influence function appears again, i.e. we have got the expansion (1/
n)
i IFF^ (Xi). It only remains to apply CLT (and Slutsky’s lemma) to get (2).
2 Bootstrap
Bootstrap is a plug-in methodology, introduced by Brad Efron, for estimating performance measures asso-
ciated with statistics. The bootstrap estimate (of a performance measure) is obtained by replacing every
occurrence of the true (unknown) distribution F with the empirical distribution Fn (e.g. EF (·) is replaced
by EFn (·) and φ(F ) is replaced by φ(Fn)) and by replacing the original sample {Xi}ni=1 with bootstrap
sample {X i∗ }ni=1 obtained by resampling (with replacement) from Fn. In practice, computing expectations
w.r.t to Fn is difficult (if not impossible). Instead, the bootstrap estimate is usually obtained by computer
simulation, i.e. by generating multiple bootstrap samples and approximating EFn (·) by the (frequentist’s)
average (1/B)
∑B
b=
, where B is the number of bootstrap samples. For example, a bootstrap estimate
of the variance of the median could be
B
∑^ B
b=
θ^ ˆ∗ b −^
B
∑^ B
i=
θ^ ˆ∗ b
where θˆ ∗ b is the sample meadian of the^ b-th bootstrap sample.
We will demonstrate the idea by several examples. First, let’s consider a toy example to show that the
bootstrap estimate may be computed only based on the knowledge of the original sample, without any
resampling.
We have calculated moments of U-statistics before (cf. the final part of Thm 12.3 of van der Vaart (1998)):
λn(F ) =
4(n − 2)
n − 1
γ 2 1 +^
n − 1
γ 2 2 ,
γ 2 1 =^ EΨ(X^1 , X^2 )Ψ(X^1 , X^3 ),
γ 2 2 =^ EΨ
2 (X 1 , X 2 ).
It follows that λn(F ) → λ(F ) = 4γ 2
- We also obtain
λn(Fn) =
4(n − 2)
n − 1
γ ∗ 2 1 +^
n − 1
γ ∗ 2 2 ,
γ ∗ 2 1 =^
n^3
i
j
k
Ψ(Xi, Xj )Ψ(Xi, Xk),
γ ∗ 2 2 =^
n^2
i
j
2 (Xi, Xj ).
Note that θˆn depends on Ψ(Xi, Xj ) for i 6 = j, but λn(Fn) depends also on Ψ(Xi, Xi). The situation thus
depends on the diagonal of the kernel:
- We obtain consistency when γ^21 , γ^22 and γ^23 are all finite, where
γ 2 3 =^ EΨ
2 (Xi, Xi).
Indeed, because of finiteness assumption, we can use the law of large numbers 1 (LLN) to conclude γ ∗ 2 1 →^ γ
2 1 and^ γ
∗ 2 2 →^ γ
2 2 , which in turn imply^ λn(Fn)^ →^4 λ
2
- But what if γ 2 3 =^ ∞? Note that
γ ∗ 2 2 =^
n^2
i 6 =j
Ψ(Xi, Xj ) 2
n^2
i
2 (Xi, Xi)
The first term converges in probability to γ 2 2.^ But, the second term may diverge or converge.^ An example of a setting in which the second term diverges is:
Xi
iid ∼ Un(0, 1),
|Ψ(x, y)| < M, ∀x 6 = y,
Ψ(x, x) = e 1 /x .
References
van der Vaart, A. W. (1998). Asymptotic Statistics. Cambridge University Press, Cambridge.
(^1) We are using both the usual LLN and also the LLN for U-statistics.
