Download Sequences and Limit Theorems - Lecture Notes | ECE 6010 and more Study notes Stochastic Processes in PDF only on Docsity!
ECE 6010
Lecture 5 – Sequences and Limit Theorems
Convergent sequences of real numbers and functions
Definition 1 Let x 1 , x 2 ,... be a sequence of real numbers. This sequence converges to a point x R if for every > 0 there is an N Z
such that
xn x
< for all n N. We write xn x , or lim n xn x. 2 For real numbers (which are complete), a necessary and sufficient condition:
xn (^) n 1 converges lim n sup m > n
xm xn
The latter condition says that xn is a Cauchy sequence.
Definition 2 Suppose f 1 , f 2 ,... is a sequence of functions (^) R. This sequence converges pointwise to f : (^) R if fn ( x ) (^) f ( x ) for every x . That is, for every x and > 0, there is an N Z
such that
fn ( x ) (^) f ( x )
< for all n N. 2 (It may be necessary to choose a different N for each x .)
Definition 3 We say that fn converges uniformly to f if for each > 0 there is an N Z
such that
fn ( x ) (^) f ( x )
< for all n N and for all x . 2
Modes of convergence of sequences of r.v.s
Suppose X 1 , X 2 ,... is a sequence of random variables defined on (, F, P ). How can we define a limit of this sequence? As it turns out, there are several different (and inequivalent) ways of defining convergence.
Almost sure convergence
This is a very strong form of convergence, and usually quite difficult to prove.
Definition 4 A sequence of r.v.s Xn (^) n 1 converges almost surely (a.s.) to the r.v. X if P ( 0 ) 1, where 0 ω : Xn (ω) (^) X (ω)
This is also called convergence with probability 1. 2 One tool for showing a.s. convergence is the following fact: Xn X a.s. if and only if P ( lim n sup m > n
Xn Xm 0 )^1.
Example 1 Let [0, 1], F B[0, 1]. Let Xn (ω) ne n ω^ , ω [0, 1] and n Z
. Note that Xn (ω) (^) 0 for all ω ( 0 , 1]. Xn ( 0 ) (^) n (diverges).
So if P ( 0 ) 0 then Xn 0 a.s. But if P ( 0 ) > 0 then Xn doesn’t converge in the almost sure sense. 2
Mean-square convergence
This is a strong mode of convergence which is usually easier to show than a.s. It is widely used in engineering.
Definition 5 The sequence Xn n 1 converges to the r.v. X in the mean-square sense if
lim n E [( Xn X )^2 ] 0. 2 We write Xn X (m.s.) or Xn X (q.m.) “quadratic mode.” There is a Cauchy criterion for m.s. convergence: If E [ X (^) n^2 ] < for all n Z
, then Xn converges in mean-square if and only if lim n sup m > n
E [( Xm Xn )^2 ] 0.
Example 2 Let [0, 1], F B[0, 1], and P is uniform: P ([ a , b ])
b (^) a
. Let
Xn (ω)
n ω [0, 1 / n^3 ], n Z
0 otherwise.
Then
E [ X^2 n ] n^2 P ([0, 1 / n^3 ]) 02 P ([1/ n^3 , 1]) n^2
n^3
n
So Xn 0 m.s. What about a.s. convergence in this case? 2 Here is an interesting fact: If Xn X (m.s.) and Xn Y (a.s.), then X Y (a.s.).
Convergence in Probability
Definition 6 The sequence Xn n 1 converges to X in probability (i.p.) if
P (
Xn X
) (^) 0
as n (^) for every > 0. Equivalently, we say that P ( Xn X ) (^) 1. 2
Example 3 Let [0, 1], F B[0, 1], and P is uniform. Let
Xn
n ω [0, 1 / n ] 0 otherwise.
Note: Xn 0 (a.s.), but Xn does not converge in m.s.
P ( Xn 0
) P ([0, 1 / n ]) 1 / n (^) 0 ,
so Xn 0 (i.p.) 2
Convergence in Distribution
Definition 7 The sequence Xn n 1 converges in distribution (or in law ) to the random variable X if FXn ( x ) (^) FX ( x ) at all continuous points of FX. 2
Example 4 [0, 1], F B[0, 1], P is uniform. Let
Xn
1 ω 1 / n 0 ω < 1 / n.
Proof of Xn X (i.p.) Xn X (in distribution) Suppose Xn X (i.p.). Choose > 0 and let x be a continuity point of FX. Then
FX ( x (^) ) P ( X
x (^) ) P ( X
x (^) , Xn
x ) P ( X^
x (^) , Xn > x ).
FXn ( x ) P ( X
x (^) , Xn
x ) P ( X^ >^ x^^ ,^ Xn
x ).
Solving for the bracketed term in the second and substituting it into the first we obtain
FX ( x (^) ) FXn ( x ) P ( X x (^) , Xn > x ) (^) P ( X > x (^) , Xn x ) FXn ( x ) P ( X
x (^) , Xn > x ).
Observe that X
x (^) , Xn > x Xn X
(for example, let Xn x δ 1 and X x (^) (^) δ 2 with δ 1 > 0 and δ 2 0. Then Xn X
δ 1 δ 2
)^ so that^ P (^ X^
x (^) , Xn > x )
P (
Xn X
). Thus FX ( x (^) )
FXn ( x ) P (
Xn x
).
Similarly, FXn ( x )
FX ( x ) P (
Xn X
).
Since we have convergence in probability, lim n P (
Xn X
) 0, so FX ( x (^) ) lim n FXn ( x )
Similarly also FX ( x ) lim n FXn ( x ).
Combining these, FX ( x (^) )
lim n FXn ( x )
FX ( x ).
Since x is a continuity point of FX and is chosen arbitrarily, we can write
lim 0
FX ( x (^) ) FX ( x ) lim n FXn ( x ) FX ( x ) lim 0
FX ( x ).
So FXn ( x ) (^) FX ( x ). (Convergence in distribution.) 2
Some examples of invalid implications
To see which modes are “stronger” than others, we can consider some counterexamples.
Example 6 Let Xn X (i.p.) Can we say that Xn X (m.s.)? Let (, F, P ) ([0, 1], B[0, 1],uniform). Let
Xn
n ω [0, 1 / n ] 0 otherwise.
We’ve shown that Xn 0 (i.p.), but E [ X (^) n^2 ] (^) so Xn 0 (m.s.). Since Xn 0 (a.s.), we also see that a.s. m.s. 2
Example 7 Does i.p. imply a.s.? Define a sequence of r.v.s as follows on [0, 1]: X 1 (ω) 1. For X 2 , X 3 , divide into two parts [0, 1 / 2 ), ( 1 / 2 , 1], with X 2 (ω) 1 on the first half, and X 3 (ω) 1 on the second half. For X 4 , X 5 , X 6 , X 7 split into fourths, with X 4 (ω) 1 on the first fourth, etc.
P (
Xn 0
) P ( Xn 1 )
which decreases (at a rate approximately 1/ log n ) as n (^) . So Xn 0 (i.p.)
However, for a.s. convergence, we see that Xn alternates (non uniformly) between 0 and 1. So Xn 0 (a.s.)
Note that this example also converges in m.s. because the 2nd moment is P ( X n 1 )
1 / log 2 ( n ) (^) 0. 2
Example 8 What about convergence in distribution and convergence i.p.?
Let X N ( 0 , 1 ), and Xn ( 1 ) n^ X. Note that Xn N ( 0 , 1 ). So FXn Fx for all n.
But
P (
Xn X
)
P (
X
/ 2 ) n odd 0 n even
So it does not (^) 0 for all ; it alternates. All the other modes of convergence depend on joint distributions, but convergence in distribution depends on marginals, which don’t tell us the whole picture. 2
Some other relationships:
- If Xn X (i.p.) then there is a subsequence Xnk k 1 such that lim k Xnk X (a.s.)
- If Xn X and there is a r.v. Y with finite second moment such that Xn
Y^ (a.s.) for every n Z
, then Xn X (m.s.).
- If Xn C (in distribution), then Xn C (i.p.)
Limit Theorems
Laws of Large Numbers
Suppose X 1 , X 2 ,... , is a sequence of r.v.s. We are often interested in sums
ni 1 Xi , as n becomes large. What can we say about such sums? Suppose all Xi have the same means μ, E [ Xi ] μ, and are uncorrelated. We would expect the average^1 n
n
i 1 Xi^ to “approach”^ μ^ in some way as^ n^^ . If var( xi^ ) <^ ,
consider 1 n
^ n
i 1
Xi μ.
Let us look at m.s. convergence;
E
n
^ n
i 1
Xi μ
2 E
n
i
( Xi μ)
2
n^2
E [
i
( Xi μ)^2 ]
1 n^2
E [
i
j
( Xi μ)( X (^) j μ)]
n^2
i
j
cov( Xi , X (^) j )
n^2
^ n
i 1
cov( Xi , X (^) j )
n^2
^ n
i 1
var( Xi )
Theorem 2 Kinchine’s Strong Law of Large Numbers. Suppose Xi i 1 is an i.i.d. sequence (i.e., a sequence of i.i.d. r.v.s) with finite mean E [ Xi ]
μ
Then the sample mean converges almost surely to the ensemble mean:
1 n
^ n
i 1
Xi μ (a.s.)
Proving these types of theorems
The proofs follow from more general limit theorems.
Definition 11 Let An n 1 be a sequence of events. The limit superior (lim sup) of An is lim sup n
An n 1 k n Ak
This is the set of all points that are in An infinitely often. 2 So ω lim sup n An ω is in infinitely many of the sets An. (It keeps coming back.) Another notation is: lim sup n An An i.o. (infinitely often). We observe that if An
A^ or^ An A^ then^ An (i.o.)^ A.
Lemma 1 The Borel Cantelli lemma. [This is frequently a good problem for math qualifiers.]
1. If
i 1 P ( An ) < then P ( An (i.o.) ) 0. That is, P ( An ) 0.
2. (Conversely) If An n 1 are independent events and
i 1 P ( An ) then P ( A (i.o.) )
Proof
- n 1 k n Ak An (i.o.)
k n Ak^ for all^ n. So
P ( An (i.o.))
P ( (^) k n Ak )
k n
P ( Ak ) (^) 0
as n if
k 1 P ( Ak ) < . So P ( An (i.o.)) 0 if
k 1 P ( Ak ) < .
- Using DeMorgan’s law, [ An (i.o.)] c^^ k 1 k n Ack. Pick n and N with n < N. Consider
P ( (^) k n Ack )
^ N
k n
P ( Ack ) (by independence)
^ N
k n
(^1) P ( Ak )
N
k n
e P ( Ak^ )^ since 1 (^) x
e x
exp[ (^)
N
k n
P ( Ak )].
If
(^) k 1 P ( Ak ) diverges, then
(^) k n P ( Ak ) diverges too, and thus
lim N
exp[ (^)
N
k n
P ( Ak )] (^) 0.
So lim N
P ( (^) kN n Ack ) 0
for all n , i.e., P ( (^) k n Ack ) 0 for all n. Now, lim sup n An is just the union of all of those intersections, so
P ( (^) n 1 k n Ack )
n 1
P ( k n Ack ) 0.
so that P ( An (i.o.)) 1.
2
Kolmogorov’s Inequality
Suppose X 1 , X 2 ,... , are independent with zero means and finite variances. Define Sn to be the running sum
Sn
^ n
k 1
Xk
Then for each α > 0,
P ( max 1 k n
Sk α)^
α^2
var( Sn ).
This is a lot like the Chebyshev inequality, but instead of looking at the variance of all of the terms, we simply look at the variance of the last one.
Central Limit Theorems
Theorem 3 Central Limit Theorem Suppose Xn is a sequence of i.i.d. random variables with mean
mu < and variance σ 2 < . Then
n
^ n
i 1
( Xi μ) (^) X (in distribution)
where
X N ( 0 , σ 2 ).
That is,
P
n
^ n
i 1
( Xi μ)
x
x
e t
(^2) / 2 σ 2 dt
The main point: Sums of i.i.d. random variables tend to look Gaussian. To work our way up to this, here are a couple of lemmas:
Lemma 2 Suppose Xn is a sequence of r.v.s with characteristic functions φ n. If there exists a r.v. X with ch.f. φ such that lim n φ n ( u ) φ( u )
for all u R then Xn X (in distribution).
