Study with the several resources on Docsity
Earn points by helping other students or get them with a premium plan
Prepare for your exams
Study with the several resources on Docsity
Earn points to download
Earn points by helping other students or get them with a premium plan
Estimation approach in Multiple Linear Regression.
Typology: Slides
1 / 6
This page cannot be seen from the preview
Don't miss anything!
Professor
School of Industrial and Systems Engineering
Learning Objectives:
multiple linear regression
different roles the predicting factors have
in multiple linear regression
Model in Matrix Form : Y = "# + %
Design Matrix
!,!
!,#
!,$
#,!
#,#
#,$
%,!
%,#
%,$
Response
!
%
Error
!
%
Coefficients
&
!
$
Data : ' !,!
!,$
!
%,!
%,$
%
Model : * '
&
!
',!
',#
$
',$
'
To estimate - &
!
$
, find values
&
!
$
that minimize squared errors:
'(!
%
'
'
'(!
%
'
&
!
',!
$
',$
)
)
: −
)
: − ;
Or, equivalently
)
;
)
If ;
)
; is invertible,
By linear algebra, derive the system of equations in matrix form:
)
;
*!
)
:
System of Equations
!
%B
"
(chi-squared distribution with n - p - 1 degrees of freedom）
Estimating σ
Sample variance
Assuming 0 ̂ '
~ N 0 , σ
This is the sample variance estimator except we use n-p- 1 degrees of
freedom. Why?
Recall that 0 '
'
&
!
x ',!
$
x ',$
Replaced by 0 ̂ '
'
&
!
x ',!
$
x ',$
Use p +1 degrees of
freedom because
&
&
!
!
$
Thus, assuming that $
'
~ N 0 , σ
>σ
= MSE ~ W %$!
(This is called the sampling distribution of >σ
,
.)
The Least Squares estimated coefficients have specific interpretations:
&
The estimated expected value of the response variable when all
predicting variables equal zero.
'
The estimated expected change in the value of the response
variable associated with one unit of change in the value of the 1
th
predicting variable (i.e., associated with a one-unit change in X '
where 1 is any of 1 , … , F), holding all other predictors in the
model fixed (i.e., holding fixed X
à Note : The interpretation provided for
'
is for quantitative variables.
Marginal versus conditional relationship:
Marginal Simple linear regression captures the association of a predicting variable to
the response variable marginally, i.e., without consideration of other factors.
Conditional Multiple linear regression captures the association of a predicting variable to
the response variable conditionally, i.e., conditional of all other predicting
variables in the model.
The estimated regression coefficients for conditional and marginal relationships can
differ not only in magnitude but also in sign or direction of the relationship.
