Download Outline for Minimum Mean Squared Error Estimation | STAT 322 and more Study notes Statistics in PDF only on Docsity!
MINIMUM MEAN SQUARED ERROR ESTIMATION
OUTLINE
- Motivation for the estimation problem
- Nonlinear MMSE estimation
- Estimation based on several measurement
- Linear MMSE estimation
- Orthogonality Principle
Reading: Bertsekas & Tsitsiklis, 4.
EE/STAT 322, #24 1
MOTIVATION FOR ESTIMATION
There are many examples and applications of the estimation problem:
- RADAR: estimation the location/speed of the target
- Communications: estimation of signal frequency
- Speech: recovering of the voice signal from the noisy signal
- Weather reporting: estimation of tomorrow’s temperature
GENERAL SETUP
- There are some quantity X that we would like to know or estimate.
- We are not able to make direct and accurate measurement of X
- But we can make some measurements or observations to collect some
quantity Y that is related to X.
- We want to estimate X from Y.
- There are different models for X:
− deterministic (fixed unknown constant)
− probabilistic (a RV, whose PDF/PMF is assumed known)
EE/STAT 322, #24 3
ONE EXAMPLE
Estimating the value of a signal in noise:
Y = X + W
X, W, Y are random variables.
- X is a random variable uniformly distributed between (0, 1)
- W is Gaussian RV with zero mean and variance σ
2
W
- W and X are independent
- We can observe Y , want to find a good estimate
X = g(Y ) of X
Y could be the signal picked up by a microphone in a classroom with
background noise.
MMSE ESTIMATION: WITH OBSERVATION Y
SETUP: There is a RV X whose value we don’t know. There is another
RV Y , whose value we know. Suppose we know the distribution of X|Y.
What is the best guess of X, given that Y = y, in the MMSE sense?
That is, want to find a function
X = g(y) such that we can minimize
J = MSE = E[ (X − g(y))
2
|Y = y]
Any function of y is call an estimator. Some are good, some are bad. The
estimator g(y) that minimizes J is called the MMSE estimator.
The same reasoning as in the previous slide leads to
∂J
∂g(y)
= 0 → g(y) = E[X|Y = y]
EE/STAT 322, #24 7
MMSE ESTIMATION BASED ON SEVERAL
MEASUREMENTS
SETUP: Given several measurements Y 1
, Y
2
,... , Y
n
, want to find the
MMSE estimate of another RV X.
That is, find a function g(Y 1 , Y 2 ,... , Yn) to minimize
E[(X − g(Y 1
, Y
2
,... , Y
n
2
|Y 1
= y 1
, Y
2
= y 2
,... , Y
n
= y n
]
and hence E[(X − g(Y 1
, Y
2
,... , Y
n
2
]
The answer is similar: the MMSE estimator is given by
g(Y 1
, Y
2
,... , Y
n
) = E[X|Y
1
, Y
2
,... , Y
n
]
PROPERTY: THE MMSE ESTIMATOR IS UNBIASED
Define
X = E[X|Y ] and
X = X −
X. We have
E[
X] = 0, E[
X|Y = y] = 0, for all y
Because of the iterated expectation, we only need to show E[
X|Y = y] = 0
E[
X|Y = y] = E[X −
X|Y = y]
= E[X|Y = y] − E[
X|Y = y]
= E[X|Y = y] − E[X|Y = y] = 0
E[ ˜X] = E[ E[ ˜X|Y ] ] = 0
EE/STAT 322, #24 9
LINEAR MMSE ESTIMATION
The nonlinear MMSE estimator based on several measurements performs
well, but
- It requires the conditional distribution of X given Y i
’s
- Even if we know the conditional distribution, the conditional expectation
may be difficult to compute (or is in an ugly form)
Solution: Lower our standard and look for something simpler
MATRIX FORM
The equations above can be expressed in a compact form, called Yule-Walker
equations.
[a 1
,... , a n
]R
Y
Y = R
XY
[a 1
,... , a n
]
Y + b =
X
where R Y
= E[YY
T
] is the correlation matrix of vector Y = [Y 1
,... , Y
n
]
T
,
given by
R
Y
R
Y
(0) R
Y
(1)... R
Y
(n − 1)
R
Y
(1) R
Y
. R
Y
(n − 2)
R
Y
(n − 1) R Y
(1) R
Y
R
XY
= E[XY] = [R
XY
(0),... R
XY
(n − 1)].
EE/STAT 322, #24 13
ORTHOGONALITY PRINCIPLE
- Homogeneous linear estimate of X:
X = aY ;
Nonhomogeneous linear estimate of X:
X = aY + b.
- the estimation error = X −
X. X = ˆX + .
X is the LMMSE estimate of X, then E(
X) = E(Y ) = 0, i.e.,
X
(and Y ) are uncorrelated with (orthogonal to) .
Proof: Consider a homogeneous case, and let
X = aY.
dJ
da
= 0 ⇒E[(X − aY )Y ] = 0 ⇒E[Y ] = 0.
We can also show E[
X] = E[aY ] = 0.
GENERAL LINEAR ESTIMATION
X is estimated from y = [y 1
,... , y n
]
, and the estimate is
x ˆ = a
y =
n
i=
a i
y i
MSE: J = E[(x − xˆ)
2
] = E[(x −
n
i=
a i
y i
2
].
Using the orthogonality principle,
= x − xˆ ⊥ xˆ.
Also ⊥ y i
i.e. E[y i
] = 0, for i = 1,... , n.
1
y
2
y
2 2
ay
1 1
ay
x
x ˆ
ε
0
11 2 2
x ˆ = ay + ay and x = x ˆ+ε
EE/STAT 322, #24 15
SUMMARY OF PROPERTIES
- Estimation error is uncorrelated with estimate
X
E[
X] = 0.
- Estimation error is uncorrelated with observation Y i
E[Y
i
] = 0.
- var(X) = var( ˆX) + var(),
where var(X) = σ
2
x
, var( ˆX) = ρ
2
XY
σ
2
x
, and
var() = J = E[(X −
X)
2 ] = (1 − ρ
2
XY
)σ
2
x
