Talagrand's Inequality: Bounds for Distance between Random Vectors and Subspaces, Lecture notes of Mathematics

Download Talagrand's Inequality: Bounds for Distance between Random Vectors and Subspaces and more Lecture notes Mathematics in PDF only on Docsity!

Talagrand’s Inequality

Fall 2019 Reading Course: Concentration of Measure

University of Utah

December 3, 2019

References

The Probabilistic Method, N. Alon & J.H. Spencer, Third edition (John Wiley & Sons),

2008, ISBN: 978-0-470-17020-

Concentration of Measure and Isoperimetric Inequalities in Product Spaces, M. Talagrand,

Publications Mathématiques de l’Institut des Hautes Études Scientifiques, 81:1, pp 73-205,

1995 https://arxiv.org/abs/math/

https:

//terrytao.wordpress.com/2009/06/09/talagrands-concentration-inequality/

Notation + Setup

Throughout: X “ pX 1

,... , X

n q P R

n has iid components.

1

Norms and distances: Euclidean in R

n

We let the support of X be S Ă R

n , which is a product set S “ ˆ

n

j“ 1

S

1

A Ă R

n is X-measurable.

Simplified example: X j is a Rademacher RV, S “ t´ 1 , 1 u

n , A is a subspace of R

n .

Overall goal: to study the probability that X is “far” from A.

1 Identical distribution is not necessary, but we will assume it for simplicity.

Talagrand’s Inequality (1/2)

We consider a simplified setup: assume that A is convex and X has iid Rademacher

components.

Then for all t ą 0 :

P rX P As P rdistpX, Aq ě ts ď expp´ct

2

q. (babyTI)

For example, c “ 1 { 16 works.

Virtues:

dimension-independent constants

probability that X is far from A is exponentially small, if A is “large” enough

Applications (1a/2)

Concentration of Lipschitz functions: let f : S Ñ R be 1-Lipschitz:

|f pxq ´ f pyq| ď }x ´ y} 1

Assume S is bounded, so that d :“ max a,bPS 1

|a ´ b| ă 8. Then

r f :“ f {pd

nq satisfies:

f pxq ´

f pyq

ď

}x ´ y} 1

d

n

ď }rx ´ ys} 1 , αr

“ d αr

px, yq

where αr “

1 ?

n

p 1 , 1 ,... , 1 q P R

n has Euclidean norm 1.

Given some m P R, let A :“

f

´ 1 pp´8, msq. Talagrand’s inequality states:

P

r f pXq ď m

ı

P rρpX, Aq ě ts ď expp´t

2

{ 4 q

Applications (1b/2)

P

r f pXq ď m

ı

P rρpX, Aq ě ts ď expp´t

2 { 4 q (1)

Suppose now that x is such that

r f pxq ě m ` t. Then for any y P A:

m ` t ď

r f pxq ď

r f pyq ` d αr

px, yq ùñ m t ď m d αr

px, Aq ď m ` ρpx, Aq

Thus:

r f

´ 1 prm ` t, 8qq Ď tx P S | ρpx, Aq ě tu.

Then (1) implies:

P

r f pXq ď m

ı

P

r f pXq ě m ` t

ı

ď expp´t

2 { 4 q

Take m “ medpf pXqq and m “ t ´ medpf pXqq to yield:

P

r f pXq ´ medp

r f pXqq| ě t

ı

ď 4 expp´t

2

{ 4 q,

BabyTI proof (1/4)

A convex, X P t´ 1 , 1 u

n iid Rademacher sequence.

Ideas:

inequalities: Jensen, Markov, Hölder

induction on dimension n

convexity of A: conditioning for inductive step

Markov’s inequality,

P p|X| ě tq ď

φptq

Eφp|X|q, φ : R ` Ñ R monotonically increasing

applied to φprq “ exppcrq reveals that

P pX P AqE

`

exppcdist

2 pX, Aqq

ď 1 ùñ (babyTI),

so we focus on proving this.

BabyTI proof (2/4)

P pX P AqE

`

exppcdist

2 pX, Aqq

ď 1 ùñ (babyTI),

Induction: n “ 0 is easy. Now assume this is true for n ´ 1. X P t´ 1 , 1 u

n .

We compute two components: (i) P pX P Aq and (ii) E

`

exppcdist

2 pX, Aqq

(i): Decompose X “ p

r X, x n q, for x n “ ˘ 1. Fix t P R:

A

t :“ rx P R

n´ 1

px, tr q P A

Note: A t is convex since A is convex.

Then:

p :“ P pX P Aq “

P p

r X P A ´ 1 q `

P p

r X P A ` 1 q.

Symmetrize around p:

p ˘ :“ P p

r X P A ˘ 1 q “: pp 1 ˘ qq, q P r 0 , 1 s

BabyTI proof (4/4)

Exponentiating and

r X-expectation’ing,

E

Ă X

exppcdist

2 pX, Aqq ď e

4 cλ

2

E

Ă X

pe

cdist

2 p

Ă X,A Xn q q

1 ´λ pe

cdist

2 p

Ă X,A ´Xn q q

λ

ı

Recall: P p

r X P A X n

q “ p ˘ 1 , and inductive hypothesis:

P p

r X P A X n

qE Ă X

exppcdist

2 p

r X, A X n

qq ď 1 ,

Therefore:

E

Ă X

exppcdist

2 pX, Aqq ď e

4 cλ

2

E

Ă X

pe

cdist

2 p

Ă X,A Xn q q

1 ´λ pe

cdist

2 p

Ă X,A ´Xn q q

λ

ı

(Hölder)

ď e

4 cλ

2

E

Ă X

pe

cdist

2 p

Ă X,A Xn q q

ı 1 ´λ

E

Ă X

e

cdist

2 p

Ă X,A ´Xn q

ı λ

ď e

4 cλ

2

p

p X n

q

1 ´λ

p

p ´X n

q

λ

.

Then if X n “ ˘ 1 , we have

pE Ă X

exppcdist

2 pX, Aqq ď e

4 cλ

2 1

p 1 ˘ qq

1 ´λ p 1 ¯ qq

λ

?

ď 1

Final step: expectation wrt X n , taking c small enough ensures that λ can be chosen to

make the inequality work.