STAT 542 Fall 2008
Homework Assignment 1
Due Tuesday, September 16
1. Assume a linear regression model with Normal errors. Take the error variance σ2known.
Show that the AIC criterion and Mallow’s Cpare equivalent. Recall that
AIC = −log likelihood evaluated at MLE+dim of the model
Cp= residual sum of squares + 2σ2dim of the model
2. In a simple linear regression problem, we observe npairs of observations (y1,x1),...,(yn, xn),
where yi∈Rand xi∈Rp. The least squares estimate of the p-dimensional regression
coefficient is given by
ˆ
β= arg minβ∈Rp
n
X
j=1
(yj−xt
jβ)2.
A basic tool for examining the fit is the leave-one-out residual: consider fitting the model
omitting the ith observation with the corresponding LS estimate defined as
ˆ
β[i]= arg min
n
X
j:j6=i
(yj−xt
jβ)2,
get a prediction for the omitted observation, ˆy[i]=xt
iˆ
β[i], then the leave-one-out residual
for the ith observation is (yi−ˆy[i]).
We will show that
yi−ˆy[i]=yi−xt
iˆ
β
1−Hii
,
where Hii denotes the (i, i)th entry of the projection matrix H. That is, the leave-one-
out residual is a re-scale of the original residual, hence it is not necessary to re-fit the
model each time an observation is omitted.
We will prove this result via the following steps. Throughout, we assume all the design
matrices are of ful l rank, and all the LS estimates exist and are unique.
(a) Consider a new data set of nobservations where y∗
1= ˆy[1] =xt
1ˆ
β[1] and the others
are the same as the original data. Let ˆ
β∗denote the corresponding LS estimate
based on the new data. Show that
ˆ
β∗=ˆ
β[1].
Hint: Show that ˆ
β[1] minimizes (y∗
1−xt
1β)2+Pn
j=2(yj−xt
jβ)2.
(b) Show that
ˆy[1] =Pn
j=2 H1jyj
1−H11
.
(c) Show that
y1−ˆy[1] =y1−xt
1ˆ
β
1−H11
.
