Download Classical Statistical Inference - Lecture Slides | ELEC 303 and more Exams Electrical and Electronics Engineering in PDF only on Docsity!
ELEC 303 – Random Signals
Lecture 17 – Classical Statistical Inference, Dr. Farinaz Koushanfar ECE Dept., Rice University Oct 29, 2008
Lecture outline
- Reading: 9.
- Classical statistics
- Properties of estimators
- Maximum Likelihood estimation (ML)
- Confidence Intervals
Classic statistical inference
- θ is deterministic - no longer a RV
- Observations X are considered random, and its distribution pX(x;θ) or fX(x;θ) depends on θ
- Multiple candidate models instead of one
- A good estimation is one with certain properties
Observation Process
Point estimation Hypothesis selection Confidence Intervals, etc.
pX(x;θ) θ
x
Classic statistical inference
- Expectation and probabilities dependent on θ
- Eθ[h(X)] the expected value of a RV h(X)
- There are similarities between this case and the Bayesian estimation case
- Difference: the criteria must be satisfied for all possible values of the unknown parameter
Classic parameter estimation
- θ is not random, but an unknown constant
- Given observations (X 1 ,…,Xn), an estimator is a random variable of the form
- Since distribution of X depends on θ, the same is true for
Θˆ^ =g(X )
Θˆ
Terminologies for estimators
- Let be an estimator of unknown variable θ
- The estimation error is
- The bias of the estimator is the expected value of the estimation error
- We call unbiased if for all θ,
- We call asymptotically unbiased if for all θ
- We call consistent if sequence converges to the true value of θ, in probability for all θ
Θ^ ˆ n Θ~ (^) n =Θˆn−θ
bθ (Θˆ (^) n)=Eθ[Θˆn]− θ Θ^ ˆn (^) θ Θ ]=θ E [ˆn Θ^ ˆ n lim (^) n→∞ Eθ[Θˆn]=θ Θ^ ˆn Θ^ ˆn
Estimation error
- Generally the estimation error is nonzero
- The mean estimation error is zero unbiased
- Size of the estimation error is of interest:
- We have:
- The above formula is known as the bias- variance tradeoff and is really important
E (^) θ [Θ^ ~n ] E (^) θ [Θ^ ~n ]=b^2 θ(Θˆ)+varθ(Θˆn )
Maximum Likelihood (ML)
- Vector of observations X=(X 1 ,…,Xn) described by a joint PMF pX(x;θ)
- The form depends on the unknown θ
- We observe a particular value x=(x 1 ,…,xn)
- ML finds the value of the θ that maximizes the numerical function pX(x 1 ,…,xn;θ) over all θ θˆ^ n =argmaxθ pX(x 1 ,...,xn;θ ) θˆ^ n =argmaxθ fX(x 1 ,...,xn;θ ) Likelihood function
Interpretation of likelihood
- Note that pX(x;θ) is not the probability that the unknown parameter is equal to θ
- It is the probability that the observed x can arise when parameter is equal to θ
- Recall that MAP maximized pΘθpX|Θ(x|θ)
- If we view pX(x;θ) as the conditional PMF, then ML can be interpreted as MAP with flat prior
ML - examples
- Estimate the Juliet latency when she is always late by X~[0,θ], θ unknown
- Estimate the mean of a Bernoulli RV
- Estimate the parameters of an exponential RV
- Estimate the mean and variance of normal
ML estimation properties
- The realization x=x 1 ,…,xn) of a random vector X=X 1 ,…,Xn distributed as pX(x; θ), or fX(x; θ)
- Recall that ML is the value of θ that maximizes the likelihood function pX(x; θ), or fX(x; θ) for ∀ θ
- The ML estimate of a one-to-one function h(θ) of θ is
- When the Xi’s are iid, under mild assumptions, each component of ML is consistent and asymptotically normal
h (θˆn )
Estimation of mean and variance
- No assumption of normality anymore
- Assume i.i.d observations X 1 ,..,Xn with an unknown common mean
- The most natural estimator is the sample mean: Mn= (X 1 +…+Xn)/n
- This is an unbiased estimator
- The mean squared error is equal to variance υ/n, where υ is the common variance of Xi
- The mean square error does not depend on θ
Estimator of variance
- Want to find an estimator of variance
- Coincides with ML estimator derived in earlier example under the normality assumptions
- Is this unbiased?
- How can we make it unbiased?
n i 1
2 i n
2 n (^) n (X M ) S^1
