Stat 312: Lecture 19
Linear Regression
Moo K. Chung
Nov 30, 2004
Concepts
1. Let xbe the speed of a car and ybe the distance the car
traveled in an hour hour. Then we have model
y=β0+β1x.
Suppose we have npaired measurements (xi, yi), i =
1,· · · , n. Since all measurement are supposed to be
noisy, we introduce a noise term ²in the above equation.
Our modified stochastic model is
y=β0+β1x+²,
where ²∼N(0, σ2). Since ²is a random variable, we
use Yinstead of yfor convenience:
Y=β0+β1x+².
Note that EY=β0+β1xand VY=σ2.
2. Equivalently we can write the above linear model for
each paired measurement (xi, yj):
Yj=β0+β1xj+²j,
where yjis the observed value of random variable Yj
and ²j∼². Note that EYj=β0+β1xj. Let ˆ
β0,ˆ
β1be
estimators of β0, β1. Then the predicted values or fitted
values are given by
ˆyj=ˆ
β0+ˆ
β1xj.
The differences between the observations yjand the pre-
dicted values ˆyjare called the residuals (errors), i.e.
rj=yj−ˆyj=yj−ˆ
β0−ˆ
β1xj.
3. Least squares estimation. The least squares estimation
is a method of estimating parameters β0and β1by min-
imizing the sum of the squared errors (SSE):
SS E =
n
X
j=1
r2
j=
n
X
j=1
(yj−ˆyj)2=
n
X
j=1
(yj−ˆ
β0−ˆ
β1xj)2.
Then the regression line is given by y=ˆ
β0+ˆ
β1x. By
differentiating SSE with respect to β0and β1, we get
normal equations:
β0+xβ1=y
60 70 80 90 100
60 70 80 90
x
y
xβ0+x2β1=xy
Solving these equations, we get
ˆ
β1=Sxy
Sxx
ˆ
β0=y−xSxy
Sxx
,
where the sample covariance Sxy =n(xy −¯x¯y).
Example. 10 students took two midterm exams.
Student k01 02 03 04 05 06 07 08 09 10
Midterm 1k80 75 60 90 99 60 55 85 65 70
Midterm 2k70 60 70 72 95 66 60 80 70 60
Let’s find the least squares regression line.
> x<-c(80, 75, 60, 90, 99, 60, 55,
85, 65, 70)
> y<-c(70, 60, 70, 72, 95, 66, 60,
80, 70, 60)
> plot(x,y)
> rarara <-lm(y˜x)
> rarara
Call: lm(formula = y ˜ x)
Coefficients: (Intercept) x
29.4827 0.5523
> abline(rarara)