Download Berry-Esseen Theorem, Lecture Slide - Computer Science and more Slides Computer Architecture and Organization in PDF only on Docsity!
Analysis of Boolean Functions (CMU 18-859S, Spring 2007)
Lecture 21: Berry-Esseen Theorem
April. 3, 2007 Lecturer: Ryan O’Donnell Scribe: Suresh Purini
1 Berry-Esseen Theorem
In this class we study a simplified version of the Berry-Esseen theorem, which quantifies how “close” are the distributions of the two random variables Q(x 1 , · · · , xn) and Q(g 1 , · · · , gn), where x 1 , · · · , xn are uniform random bits ± 1 and g 1 , · · · , gn are Gaussian random variables with mean 0 and variance 1, and Q is a multivariate polynomial of degree 1 or in other words it is an affine function. In the next class, we study a version of Berry-Esseen theorem where Q is a multi-linear polynomial. This theorem is used to prove that “Majority is Stablest”. The following is the main theorem of this class.
Theorem 1.1 Let Q(u 1 , · · · , un) =
∑n i=
αiui , αi ∈ R , be a linear polynomial over formal variables
u 1 , · · · , un_. Also assume that_
∑n i=
α^2 i = 1 and α^2 i ≤ τ, ∀i ∈ [n]. Let x 1 , · · · , xn be i.i.d uniform
random ± 1 bits. Let the random variable g be normally distributed with mean 0 and variance 1, i.e., g ∼ N(0, 1). Then the random variables Q(x 1 , · · · , xn) and g are “close” in distribution. In particular
1. ∀to ∈ R , |Pr[Q(x 1 , · · · , xn) < t 0 ] − Pr[g < t 0 ]| ≤ O(τ ). 2. |E[|Q(x 1 , · · · , xn)|] − E[|g|]| ≤ O(τ ).
We first understand some of the ideas used in the proof of theorem 1.1. Idea 1: The first idea is to replace the Gaussian random variable g by another Gaussian random variable with same mean and variance but looks very similar to the random variable Q(x 1 , · · · , xn). Let g 1 , · · · , gn be i.i.d Gaussian random variables with mean 0 and variance
- Consider the random variable Q(g 1 , · · · , gn) ∼
∑n i=
αigi. It turns out that Q(g 1 , · · · , gn) ∼
N(0,
∑n i=1 α 2 i )^ ∼^ N(0,^ 1) (^1) is also Gaussianly distributed. Now we can compare the distributions
Q(x 1 , · · · , xn) and Q(g 1 , · · · , gn) which at least appear to look alike.
(^1) If X ∼ N (0, 1), then αX ∼ N (0, α (^2) ). If X ∼ N (μ, σ (^2) ) and Y ∼ N (ν, τ 2 ), then X + Y ∼ N (μ + ν, σ^2 + τ 2 ). In other words, sum of Gaussian random variables is Gaussian whose mean and variance are respectively equal to sum of the individual means and variances of Gaussian random variables used in summation. Refer http://en.wikipedia.org/wiki/Sum of normally distributed random variables
Idea 2: We shall try to see a generic way to say that two random variables X and Y are close in distribution. For all “nice” test functions ψ : R → R,
|E[ψ(X)] − E[ψ(Y )]| ≤ “small”.
First note that if the functions ψt 0 (t) =
1 if t < t 0 0 otherwise and ψ 2 (t) = |t| fit in the definition of
“nice”, then by letting X = Q(x 1 , · · · , xn) and Y = Q(g 1 , · · · , gn) we shall get statements 1 and 2 of theorem 1.1 respectively. We shall later see that the above two functions ψ 1 and ψ 2 do not fit in our definition of “nice” but they can be approximated by “nice” functions. Let us look at some examples of “nice” functions with respect to the random variables X = Q(x 1 , · · · , xn) and Y = Q(g 1 , · · · , gn). Note that if g ∼ N(0, 1), then E[g^2 k+1] = 0 and E[g^2 k] = (2k − 1) · (2k − 3) · · · 5 · 3 · 1. In particular E[g^4 ] = 3. These facts are repeatedly used in the following examples.
- Let ψ(t) = a + bt. Then
E[ψ(X)] − E[ψ(Y )] = E[a + bX] − E[a + bY ] = b(E[X] − E[Y ])
= b(E[
∑n i=
αixi] − E[
∑n i=
αigi])
= 0
- Let ψ(t) = a + bt + ct^2. Then
E[ψ(X)] − E[ψ(Y )] = E[a + bX + cX^2 ] − E[a + bY + cY 2 ]
= c(E[(
∑n i=
αixi)^2 ] − E[(
∑n i=
αigi)^2 ])
= 0
- Let ψ(t) = t^3. Then
E[ψ(X)] − E[ψ(Y )] = E[(
∑n i=
αixi)^3 ] − E[(
∑n i=
αigi)^3
=
i,j,k
αiαj αkE[xixj xk] −
i,j,k
αiαj αkE[gigj gk]
= 0
At this point one might conjecture that all polynomials are nice functions. But it turns out it is not “quite” true as we see in the next example.
Theorem 1.4 (Taylor’s theorem) For all smooth functions f and for any r ∈ N , there exists y ∈ [x, x + ] , such that
f (x + ) = f (x) + f ′(x) +
^2
f ′′(x) + · · · +
r−^1 (r − 1)!
f (r−1)(x) +
r r!
f (r)(y)
Theorem 1.5 For all B -nice functions ψ
|E[ψ(Zi− 1 )] − E[ψ(Zi)]| ≤ O(Bτ ).
Proof: Write R = α 1 g 1 + · · · + αi− 1 gi− 1 + αi+1xi+1 + · · · + αnxn. Then Zi− 1 = R + αixi and Zi = R + αigi Note that xi, gi, R are mutually independent. We want to bound |E[ψ(R + αixi)] − E[ψ(R + αigi)]|. Since ψ is a B-nice function we have ψ′′′′(t) ≤ B, ∀t. This gives us the following
∀t, > 0 , ψ(t + ) = ψ(t) + ψ′(t) + ψ′′(t)
^2
ψ′′′(t) 6
^3 + {≤
B
^4 }
Hence
|E[ψ(X)] − E[ψ[Y ]]| = |E[ψ(R + αixi)] − E[ψ(R + αigi)]|
= |E[ψ(R) + ψ′(R)(αixi) + ψ′′(R)
(αixi)^2 2
ψ′′′(R) 6
(αixi)^3 + {≤
B
(αixi)^4 }]
− E[ψ(R) + ψ′(R)(αigi) + ψ′′(R)
(αigi)^2 2
ψ′′′(R) 6
(αigi)^3 + {≤
B
(αigi)^4 }]|
≤ E[{
B
(αixi)^4 } + {
B
(αigi)^4 }]
B
α^4 i +
B
3 α^4 i
≤ O(Bα^4 i ) ≤ O(Bτ )
Recall that we mentioned before that if the threshold function ψt 0 (t) =
1 if t < t 0 0 otherwise and the absolute value function ψ 2 (t) = |t| fit in the definition of B-nice functions we have our Berry-Esseen theorem proved. We can see that they are not B-nice functions. However, they can be approximated by B-nice functions. We use this fact prove the Berry-Esseen theorem.
Claim 1.6 ∀to ∈ R and ∀λ, 0 < λ < 12 , there exists a O( (^) λ^14 ) -nice function ψt 0 ,λ : R → R which approximates ψt 0 in the following sense: ψt 0 ,λ = 1 for t < t 0 − λ ; ψt 0 ,λ(t) = 0 for t > t 0 + λ and 0 ≤ ψt 0 ,λ(t) ≤ 1 for |t − t 0 | ≤ λ.
We give a proof sketch for the first part of the Berry-Esseen theorem, ∀t 0 ∈ R, Pr[X <
t 0 ] − Pr[g < t 0 ]| ≤ O(τ ), where X =
∑n i=
αixi, with xi’s as i.i.d uniform random ± 1 bits and
g ∼ N(0, 1). Also
∑n i=1 α 2 i = 1^ and^ α 2 i ≤^ τ^ ,^ ∀i^ ∈^ [n]. Proof Sketch: E[ψt 0 −λ,λ(X)] ≤ Pr[X < t 0 ] ≤ E[ψt 0 +λ,λ(X)]
By Berry-Essen theorem we have E[ψt 0 −λ,λ(X)] = E[ψt 0 −λ,λ(g)] ± O( (^) λτ 4 ) and E[ψt 0 +λ,λ(X)] = E[ψt 0 +λ,λ(g)] ± O( (^) λτ 4 ). But E[ψt 0 +λ,λ(g)] = Pr[g < t 0 + λ], which is within O(λ) of Pr[g < t 0 ]. Therefore we have |Pr[X < t 0 ] − Pr[g < t 0 ]| ≤ O(
τ λ^4
) + O(λ)
By taking λ = τ (^15) , we have
|Pr[X < t 0 ] − Pr[g < t 0 ]| ≤ O(τ
(^15) ).
