Study with the several resources on Docsity
Earn points by helping other students or get them with a premium plan
Prepare for your exams
Study with the several resources on Docsity
Earn points to download
Earn points by helping other students or get them with a premium plan
Material Type: Notes; Class: Communications Systems; Subject: Electrical and Computer Engr; University: University of Illinois - Urbana-Champaign; Term: Unknown 2005;
Typology: Study notes
1 / 8
This page cannot be seen from the preview
Don't miss anything!
Prepared by: Suneil Hosmane
Based on Lectures by Prof. Hadjicostis
University of Illinois at Urbana-Champaign
Department of Electrical and Computer Engineering
© Copyright 2005 Christoforos Hadjicostis. All rights reserved.
Just like (binary) hypothesis testing is important in digital communication systems,
estimation is important in analog communications.
Consider the following simplified “communication channel”:
X denotes the signal that we would like to transmit over a channel and is modeled as a
random variable with some known pdf f x
(x).
N represents noise in the channel and is modeled by as a random variable with normal
distribution (Gaussian random variable) with mean = μ n
(typically 0) and variance = σ n
2
(Noise is added to the signal X when it is transmitted through the channel.)
The receiver receives Y and needs to make an intelligent guess as to the actual signal X
that was sent. How do we do this? We need a suitable estimator that can map our
observation Y = y to some estimate about what was transmitted. The following sections
discuss how we can choose our estimators.
y
x = g ( y )
y is probabilistically related to x (e.g., via a known joint pdf f x,y
(x,y)).
x
is an estimate of x and is a function of y.
Consider the following example which is not a communication example but still a very
relevant application of estimation.
An exam was given at UIUC. The exam was out of a total of 80 points. We need to guess
what a particular student (student X ) scored in this exam. We only need to guess, 0-5, 6-
10, 11-15, etc…. (interval guessing). What should our guess be? This is a difficult
question because we have no prior information about how hard or easy the exam was.
Suppose that I now reveal to you the histogram for that particular exam. What would be
your guess for student’s X score?
The mean? The mode? The median?
Estimation Process
Suppose we want to choose x
so that we minimize the following quantity:
( ) ( ( ) ) [ ] [ ]
2 2
E x − x = E x y − x
, or
( ( ) ) ( ( ) )
∫∫
E [ x y − x ]= x y − x ⋅ fx , y ( x , y ) dxdy
2 2
If we make the substitution
f x , y ( x , y ) dxdy = fx | y ( x | Y = y ) , we get
( ) ( ( ) )
∫∫
min( x y ) x y − x ⋅ fx | y ( x | Y = y )⋅ fy ( y ) dxdy
min( x ( y ) ) [ ( x ( y ) x ) fx | y ( x | Y y ) dx ] fy ( y ) dy
2
∫ ∫
Since
x ( y )
is fixed for a given Y = y and since all quantities are positive, what we need
to do is:
( ) min x fx | y ( x | Y y ) dx
2
∫
α α where we substituted
x ( y )
α =
To minimize the expression we take the derivative with respect to
α .
( ) | ( | ) 0
2
∫
α x fx y x Y ydx
α
2 ⋅ ( − ) ⋅ | ( | = ) = 0
∫
α x fx yx Y ydx
f x | y ( x | Y = y ) dx = x ⋅ fx | y ( x | Y = y ) dx
∫ ∫
α
= x ⋅ fx | y ( x | Y = y ) dx
∫
α
α= E [ X | Y = y ]
∴ x ( y )= E [ X | Y = y ]
This is the MMSE estimator for X given the observation that Y = y.
Setup: X , Y are two Random Variables with joint density f x,y
(x,y). Given Y = y, find
f x,y
(x,y) = C in the shaded region.
Area of triangle = ½ bh
Since the total area of f x,y
(x,y) = 1,
that must mean that ½ *C = 1.
Therefore, C = 2.
f (y)
f (x, y)
f (x|Y y)
y
x, y
f x,y
(x,y) = 2 0 < x < 1 1-x < y < 1
Since there is a constant distribution in the shaded region,
f X|Y (x|Y=y) is a uniform
probability density from 1-y < x < 1.
To illustrate, look at the following picture:
Take any slice of the 2-dimensional space along Y = y. Since there is an even
probability distribution from the point of intersection of
the diagonal line (x = 1 – y) and the vertical line at x = 1,
f X|Y (x|Y= y) will have a uniform distribution from 1-y <
x < y.
( ) x ( , ) ( y y )
y
x
xLMMSE y x y μ
σ
σ
=μ + ρ −
Here μ x
, σ x
(μ y
, σ y
) are the mean and standard deviation of X ( Y ) and ρ ( x , y )is the
correlation coefficient between X and Y.
Setup: X , Y are two Random Variables with known f x,y
(x,y). Find
( ) x ( , ) ( y y )
y
x
xLMMSE y x y μ
σ
σ
=μ + ρ −
dy y x x
x
x
f (x) 2 2 2 2 2 2
1
1
1
1
−
−
∫
1
0
3
1
0
2
∫
x
x dx
var(X) E[X ]-(E[X]) 2
1 2 2
0
4
1
0
2 2 3
∫
x
x dx
Due to symmetry, E[X] = E[Y] and var(X) = var(Y).
x y
x y
σ σ σ σ
ρ
cov( , ) [ ] [ ] [ ]
( )
1
0
1 3 4
0
2 3
1
1
2
1
0
1
1
∫ ∫ ∫
−
−
x x
xydy dx xy x x x dx
x
2
x y
σ σ
ρ
y
xLMMSE y = − y − = −
Notice we get the same result as in the unrestricted MMSE case. In general, the MMSE
and LMMSE estimators will not be the same; However, if the MMSE estimator is linear,
then MMSE = LMMSE. The following property always holds:
2 2
E xMMSE y − x ≤ E xLMMSE y − x
