Download Bayesian Inference & Hypothesis Testing: ELEC 303 Lecture 15 at Rice Univ. - Prof. Farinaz and more Study notes Electrical and Electronics Engineering in PDF only on Docsity!
ELEC 303 – Random Signals
Lecture 15 – Bayesian Statistical Inference, Hypothesis testing, MAP, LMS Dr. Farinaz Koushanfar ECE Dept., Rice University Oct 22, 2008
Lecture outline
- Reading: 8.2-8.
- Bayesian inference and the posterior distribution
- Point estimation
- Hypothesis testing
- Bayesian least mean square estimator
Bayesian inference and posterior
distribution
- Unknown quantity of interest: Θ
- Observations (or measurements, or observation vector) of X=(X 1 ,X 2 ,…,Xn)
- We assume that we know
- A prior distribution pΘ or fΘ
- A conditional distribution pX|Θ or fX|Θ
- A complete answer is described by pΘ|X(θ|x)
Observation Process
Posterior Calculation
Point estimates Error analysis, etc
Prior p ΘΘΘΘ
Conditional pX| ΘΘΘΘ
x p ΘΘΘΘ |X(.|X=x)
Four versions of Bayes rule
- Θ discrete, X discrete
- Θ discrete, X continuous
- Θ continuous, X discrete
- Θ continuous, X continuous
Θ Θ Θ θ θ θ = θ θ ' X|
|X X| p ( ')p (x| ') p ( |x) p ( )p (x| )
Θ Θ Θ θ θ θ = θ θ ' X|
|X X| p ( ')f (x| ') p ( |x) p ( )f (x| )
θ = θ θ Θ Θ
Θ Θ Θ f ( ')p (x| ')d '
f ( |x) f ( )p (x| ) X |
|X X|
θ = θ θ Θ Θ
Θ Θ Θ f ( ')f (x| ')d '
f ( |x) f ( )f (x| ) X |
|X X|
Four versions of MAP rule
- Θ discrete, X discrete
- Θ discrete, X continuous
- Θ continuous, X discrete
- Θ continuous, X continuous
p (^) Θ(θ )pX |Θ(x|θ )
p (^) Θ(θ )fX |Θ(x|θ )
f (^) Θ(θ )pX |Θ(x|θ )
f (^) Θ(θ )fX |Θ(x|θ )
Example – spam filter
- Email may be spam or legitimate
- Parameter Θ, taking values 1,2, corresponding to spam/legitimate, prob pΘ(1), PΘ(2) given
- Let ω 1 ,…, ωn be a collection of special words, whose appearance suggests a spam
- For each i, let Xi be the Bernoulli RV that denotes the appearance of ωi in the message
- Assume that the conditional prob are known
- Use the MAP rule to decide if spam or not.
Point estimation
- A point estimate is a single numerical value representing our best guess of Θ
- An estimator is assumed to be a RV of the form for some function g
- Different g’s corresponds to different estimators
- An estimate is the value of the estimator determined by the value x of observations X
- The MAP rule sets the estimate to a value that maximizes the posterior distributions
- Once values x of X observed, the conditional expectation (LMS) estimator sets the to E[Θ|X=x]
Θˆ^ =g(X )
θˆ
θˆ
θˆ
Couple of remarks on estimation
- If the posterior is symmetric around its conditional mean and unimodal , the max occurs at the mean - MAP estimate is the same as conditional expectation
- If Θ is continuous, the actual evaluation of MAP may be derivable analytically, e.g., using derivatives
Multiple hypothesis
Example – biased coin, single toss
- Two biased coins, with head prob. p 1 and p 2
- Randomly select a coin and infer its identity based on a single toss
- Θ=1 (Hypothesis 1), Θ=2 (Hypothesis 2)
- X=0 (tail), X=1(head)
- MAP compares PΘ(1)PX|Θ(x|1)? PΘ(2)PX|Θ(x|2)
- Compare PX|Θ(x|1) and PX|Θ(x|2) (WHY?)
- E.g., p 1 =.46 and p 2 =.52, and the outcome tail
Example – biased coin, multiple tosses
- Assume that we toss the selected coin n times
- Let X be the number of heads obtained -?
Example – signal detection and matched filter
- A transmitter sending two messages Θ=1,Θ=
- Massages expanded:
- If Θ=1, S=(a 1 ,a 2 ,…,an), if Θ=1, S=(b 1 ,b 2 ,…,bn)
- The receiver observes the signal with corrupted noise: Xi=Si+Wi, i=1,…,n
- Assume Wi∼N(0,1)
LMS – example 1
- Let Θ~U[4,10]
- Suppose we observe Θ with noise W: X= Θ+W
- Assume W~U[-1,+1] and independent of Θ
- Find the LMS estimate of Θ, given X
LMS – example 2
- Consider the date example, where Juliet is late by a RV X~U[0,Θ], Θ~U[0,1]
- The MAP estimate: x
- The LMS estimate:
- Find the conditional mean squared error for MAP and the LMS estimate
|log x |
1 - x .|log x|
E ( |X x)^11
Θ = = ∫x θθ =
Properties of the estimation error
- The estimation error is unbiased, i.e., it has zero conditional and unconditional mean:
- The estimation error is uncorrelated with the estimate
- The variance of Θ can be decomposed as
E [ Θ^ ~] = 0 E [ Θ^ ~|X=x]= 0 , forallx
Θˆ^ ) 0
Cov( Θˆ ,Θ =
var(Θ )=var(Θˆ)+ var(Θ
Uninformative observation
- Let us say that the observation X is uninformative if the mean squared error is the same as var(Θ), the unconditional variance of Θ
- When is this the case?
E [ Θ^ ~^2 ]= var(Θ^ ~^2 )
