ECE 534: Random Pro cesses Spring 2010
Solutions to Problem Set 4
2.21 Applications of Jensen’s inequality
(a) Convex function: ϕ(u) = 1
u. Random variable X.
(b) Convex function: ϕ(u) = u2. Random variable X2.
(c) Convex function: ϕ(u) = uln u. Random variable L=f(Y)/g(Y) where Yhas prob-
ability density g. Indeed, in this case, Jensen’s inequality is E[ϕ(L)] ≥ϕ(E[L]).
But E[ϕ(L)] = RAf(y)
g(y)ln f(y)
g(y)g(y)dy =D(f|g), and E[Y] = RAf(y)
g(y)g(y)dy =
RAf(y)dy = 1 and ϕ(1) = 0, so that Jensen’s inequality becomes D(f|g)≥0.
Another solution is to use the function ϕ(u) = −ln uand the random variable Z=
g(X)/f(X), where Xhas density f. Indeed, in this case, Jensen’s inequality is E[ϕ(Z)] ≥
ϕ(E[Z]). But E[ϕ(Z)] = RA−ln g(x)
f(x)f(x)dx =D(f|g), and E[Z] = RAg(x)
f(x)f(x)dx =
RAg(x)dx = 1 and ϕ(1) = 0, so that Jensen’s inequality becomes D(f|g)≥0.
2.29 Chernoff bound for Gaussian and Poisson random variables
(a) ln MX(θ) = µθ +θ2σ2
2and l(a) = maxθaθ −(µθ +θ2σ2
2) = (a−µ)2
2σ2.Taking a=µ+cin
the optimized Chernoff inequality yields P{X≥E[X] + c} ≤ exp(−c2
2σ2) for c≥0.
(b) The log moment generating function of Yis given by lnMY(θ) = ln P∞
k=0 eθkλke−λ
k!=
ln(eλ(eθ−1)) = λ(eθ−1). Therefore,
l(a) = max
θaθ −eλ(θ−1) = aln( a
λ) + λ−a
Setting a=λ+cin the optimized Chernoff inequality yields
P{Y≥E[Y] + c} ≤ exp −(λ+c) ln λ+c
λ+c
(c) Using the definition of ψyields c2
2λψ(c
λ) = λg(1 + c
λ)=(λ+c) ln λ+c
λ−cas desired. For
more information see Shorack and Wellner, Empirical Processess, 1986.
1
