Stat 312: Lecture 21
Linear Regression II.
Moo K. Chung
Dec 7, 2004
Concepts
1. Maximum likelihood estimation. Given a linear
model
Yj=β0+β1xj+²j
with ²j∼N(0, σ2), we will estimate β0, β1using
the maximum likelihood estimation. Note that
Yj∼N(β0+β1xj, σ2).
The density function for Yjare
f(yj) = 1
√2πσ exp h(yj−β0−βjxj)2
2σ2i.
The loglikelihood is
L(β0, β1) = const −1
2σ2
n
X
j=1
(yj−β0−β1xj)2.
We maximize the loglikelihood which is equivalent
to minimizing the sum of the residuals
1
2σ2
n
X
j=1
(yj−β0−β1xj)2.
Hence MLE is LSE in linear regression.
2. The least squares estimation for β1are given by
ˆ
β1=Sxy
Sxx
where Sxy =n(xy −¯x¯y). It can be shown that
ˆ
β1=Sxy
Sxx
=
n
X
j=1
cjYj,
where
cj= (xj−¯x)/Sxx.
So LSE is a linear estimation. From Pn
j=1 cj= 0,
Pn
j=1 cjxj= 1 and Pn
j=1 c2
j=S−1
xx we can show
that
Eˆ
β1=β1
showing unbiasness. Further, Vˆ
β1=σ2/Sxx.
Since ˆ
β1is a linear combination of normals, ˆ
β1∼
N(β1, σ2/Sxx).
3. We are interested in hypothesis testing
H0:β1= 0 vs. H1:β16= 0.
Inference on the slope parameter β1is based on test
statistic
T=ˆ
β1−β1
Sˆ
β1∼tn−2,
where Sˆ
β1= ˆσ/√Sxx and ˆσ=SSE/(n−2).
It can be shown that SSE =Syy −S2
xy/Sxx , we
get Sˆ
β1=1
√n−2qSyy
Sxx −(Sxy
Sxx )2. Then we reject
H0if |t|> tα/2,n−2at 100(1 −α)% significance.
We don’t usually compute the test statistic by hand.
Use R-package.
Example. We continue Lecture 19 example.
>summary(lm(y˜x))
Call: lm(formula = y ˜ x)
Residuals:
Min 1Q Median 3Q Max
-10.908 -6.312 1.758 4.354 10.836
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 29.48 13.23 2.22 0.06
x 0.55 0.17 3.12 0.01 *
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
Residual standard error: 7.647 on 8 degrees
of freedom Multiple R-Squared: 0.5519,
Adjusted R-squared:0.4959 F-statistic:9.854on
1 and 8 DF, p-value: 0.01383