Download Regression Analysis Notation and Methods and more Study notes Mathematical Statistics in PDF only on Docsity!
1
This procedure performs multiple linear regression with five methods for entry and removal of variables. It also provides extensive analysis of residual and influential cases. Caseweight (CASEWEIGHT) and regression weight (REGWGT) can be specified in the model fitting.
Notation
The following notation is used throughout this chapter unless otherwise stated:
yi (^) Dependent variable for case i with variance σ 2 g (^) i
ci Caseweight for case i; ci = 1 if CASEWEIGHT is not specified
g (^) i Regression weight for case i; g (^) i = 1 if REGWGT is not specified l Number of distinct cases wi c gi i
W (^) wi i
l
=
∑ 1 p Number of independent variables
C (^) Sum of caseweights: ci i
l
=
∑ 1 x (^) ki The^ kth independent variable for case^ i
X (^) k Sample mean for the kth independent variable: (^) X (^) k w xi ki W i
l
%
'
& &
(
0
) ) =
∑ 1
Y (^) Sample mean for the dependent variable: Y w yi i W i
l
%
'
& &
(
0
) ) =
∑ 1 hi Leverage for case^ i
hi^ g W
i (^) +hi
S (^) kj Sample covariance for X (^) k and X (^) j
S (^) yy Sample variance for^ Y
S (^) ky Sample covariance for X (^) k and Y
p∗^ Number of coefficients in the model. p ∗^ = pif the intercept is not included; otherwise p ∗^ = p+ 1 R (^) The sample correlation matrix for X 1 ,! ,X (^) pand Y
Descriptive Statistics
R =
1
3
2 2 2 2 2
4
6
5 5 5 5 5
r r r r r r
r r r
p y p y
y yp yy
11 1 1 21 2 2
1
where
r
S
S S
kj
kj kk jj
and
r r
S
S S
yk ky
ky kk yy
The sample mean X (^) i and covariance Sij are computed by a provisional means algorithm. Define
Wk wi i
k = = =
∑ 1
cumulative weight up to case k
Yi = β 0 + β 1 X (^1) i + β 2 X (^2) i + "+ βp X (^) pi +ei
sweep operations are used to compute the least squares estimates b of i and the
associated regression statistics. The sweeping starts with the correlation matrix R.
Let
R be the new matrix produced by sweeping on the kth row and column of R.
The elements of
R are
r
r
r
r
r
i k
r
r
r
j k
kk kk
ik
ik kk
kj
kj
kk
and
~r r r^ r r , , r ij i^ k^ j^ k
ij kk ik kj kk
If the above sweep operations are repeatedly applied to each row of R 11 in
R
R R
R R
% ' &
( 0 )
11 12 21 22
where R 11 contains independent variables in the equation at the current step, the
result is
R
R R R
R R R R R R
% ' &
( 0 )
− − − −
11
1 11
1 12 21 11
1 22 21 11
1 12
The last row of
R 21 R 11 −^1
contains the standardized coefficients (also called BETA), and
R 22 − R 21 R 11 −^1 R^12
can be used to obtain the partial correlations for the variables not in the equation, controlling for the variables already in the equation. Note that this routine is its own inverse; that is, exactly the same operations are performed to remove a variable as to enter a variable.
Variable Selection Criteria
Let rij be the element in the current swept matrix associated with X (^) i and X (^) j. Variables are entered or removed one at a time. X (^) k is eligible for entry if it is an independent variable not currently in the model with
rkk ≥ t (tolerance with a default of 0.0001)
and also, for each variable X (^) j that is currently in the model,
r
r r r jj jk kj t kk
% '&^
( 0 )^
The above condition is imposed so that entry of the variable does not reduce the tolerance of variables already in the model to unacceptable levels. The F-to-enter value for X (^) k is computed as
F to enter
C p V k r V
k
yy k
R W
with 1 and C − p∗^ − 1 degrees of freedom, where p∗^ is the number of coefficients currently in the model and
V
r r k r
yk ky kk
Enter (Forced Entry)
Choose X (^) k such that rkk is maximum and enter X (^) k. Repeat for all variables to be entered.
Remove (Forced Removal)
Choose X (^) k such that rkk is minimum and remove X (^) k. Repeat for all variables to be removed.
Statistics
Summary
For the summary statistics, assume p independent variables are currently entered in the equation, of which a block of q variables have been entered or removed in the current step.
Multiple R
R = 1 −ryy
R Square
R 2 = 1 −ryy
Adjusted R Square
R R
R p
C p
adj
2 2
R W
R Square Change (when a block of q independent variables was added or removed)
9 R 2 = R (^) current^2 −Rprevious^2
F Change and Significance of F Change
F
R C p
q R
q
R C p q
q R
q
current
previous
7
8
u u u
9
u u u
∗
∗
2
2 2
2
R W
R W R W
R W
for the addition of independent variables
for the removal of independent variables
the degrees of freedom for the addition are q and C − p∗^ , while the degrees of freedom for the removal are q and C − p ∗^ −q.
Residual Sum of Squares
SS (^) e = r (^) yy IC − (^1) TSyy
with degrees of freedom C − p∗^.
Sum of Squares Due to Regression
SS (^) R = R (^2) IC − (^1) TSyy
with degrees of freedom p.
Amemiya’s Prediction Criterion (PC)
PC
R C p
C p
∗
∗
R^1 2 WR W
Mallow’s Cp (CP)
CP
SS
= e+ p −C
σ 2
where σ 2 is the mean square error from fitting the model that includes all the variables in the variable list.
Schwarz Bayesian Criterion (SBC)
SBC C
SS
C
= % e p C '&^
( 0 )^
ln + ∗lnI T
Collinearity
Variance Inflation Factors
VIF
r i ii
Tolerance
Tolerance (^) i =rii
Eigenvalues, λκ
The eigenvalues of scaled and uncentered cross-product matrix for the independent variables in the equation are computed by the QL method (Wilkinson and Reinsch, 1971).
Condition Indices
η
λ k λ
j k
max
Variance-Decomposition Proportions
Let
v i = (^) Q vi 1 , !,vipV
be the eigenvector associated with eigenvalue λi. Also, let
Aij = vij^2 λ (^) i and A (^) j Aij i
p
=
∑ 1
The variance-decomposition proportion for the jth regression coefficient associated with the ith component is defined as
π (^) ij = Aij Aj
Statistics for Variables in the Equation
Regression Coefficient b (^) k
b
r S
S
k
yk yy
kk
= for k = 1, !,p
F-test for Beta (^) k
F
Beta (^) k Beta (^) k
%
'
&
(
0 σ )
2
with 1 and C − p∗^ degrees of freedom.
Part Correlation of Xk with Y
Part Corr X
r r k
yk kk
− (^) I T=
Partial Correlation of Xk with Y
Partial Corr X
r r r r r k
yk kk yy yk ky
I T
Statistics for Variables Not in the Equation
Standardized regression coefficient Beta k^ ∗^ if Xk enters the equation at the next step
Beta
r r k
yk kk
The F-test for Beta (^) k^ ∗
F
C p r
r r r
yk
kk yy yk
2
R W
with 1 and C − p∗^ degrees of freedom
Partial Correlation of Xk with Y
Partial X
r r r k
yk yy kk
I T =
Tolerance of Xk
Tolerance (^) k =rkk
Minimum tolerance among variables already in the equation if Xk enters at the next step is
min , 1
%
'
& &
(
0
) j p (^) rjj rkj rjk rkk kk)
r Q V
Residuals and Associated Statistics
There are 19 temporary variables that can be added to the active system file. These variables can be requested with the RESIDUAL subcommand.
Centered Leverage Values
For all cases, compute
h
g C
X X X X r S S
g C
X X r S S
i
i ji^ j^ ki^ k^ jk k jj^ kk
p
j
p
i ji^ ki^ jk k jj^ kk
p
j
p
7
8
u u u u
9
u u u u
= =
= =
∑∑
∑∑
1 1
1 1
H S
Q VQ V
H S
if intercept is included
otherwise
Standardized Predicted Values
ZPRED
Y Y
sd i
i
(^7) −
8
uu
9
u u
if no regression weight is specified
SYSMIS otherwise
where sd is computed as
sd
c Y Y C
i i
i
l
=
∑
R W^2
1
Studentized Residuals
SRES
e s
h g
c
e s
h g
i
i i i
i
i i i
7
8
u uu
9
u u u
R W
R W
for selected cases with
otherwise
Deleted Residuals
DRESID
e h c e i
i i i i
(^7) − > 8
u
9 u
R W for selected cases with otherwise
Studentized Deleted Residuals
SDRESID
DRESID
s
c
e
s h g
i
i i
i
i i i
7
8
u uu
9
u u u
I T
R W
for selected cases with
otherwise
where s I Ti is computed as
s C p
C p s
h i DRESID i
I T i
R W
− −
∗
∗ 1 1 1
2 2 ~
Adjusted Predicted Values
ADJPRED (^) i = Yi −DRESIDi
DfBeta
DFBETA b b i
g e h i
i i i
t
i
− I T
I X WX T^ X 1
1
where
X it^ i pi i pi
X X
X X
7 8
u
9 u^
1
Q V Q V
if intercept is included ottherwise
and W = diag (^) Iw 1 , !,wlT.
For unselected cases with ci > 0
MAHAL
C h i (^) C h
i i
7 8 9
if intercept is included I 1 T otherwise
Cook’s Distance (Cook, 1977)
For selected cases with ci > 0
COOK
DRESID h g s p
DRESID h g s p
i
i i i
i i i
(^7) + 8
u
9 u
2 2
2 2
R W I^ T
R W R W
if intercept is included
otherwise
For unselected cases with ci > 0
COOK
DRESID h W
s p
DRESID h s p
i
i i
i i
% ' & ( 0 )
% ' &
( 0 ) +
7
8
uu
9
u u
2 2
2 2
I T
R W R W
if intercept is included
otherwise
where hi′ is the leverage for unselected case i, and ~s 2 is computed as
~s C^ p^
SS e h W
C p
SS e h
e i i
e i i
2
2
2
% ' & ( 0 )
1 3
2
4 6
5
7
8
u u
9
u u
if intercept is included
I T otherwise
Standard Errors of the Mean Predicted Values
For all the cases with positive caseweight,
SEPRED
s h g s h g
i
i i i i
7 8
u
9 u
if intercept is included otherwise
95% Confidence Interval for Mean Predicted Response
LMCIN Y t SEPRED
UMCIN Y t SEPRED
i i (^) C p i
i i (^) C p i
−
−
∗
∗
. , . ,
0 025
0 025
95% Confidence Interval for a Single Observation
LICIN
Y t s h g
Y t s h g
UICIN
Y t s h g
Y t s h g
i
i (^) C p i i
i C p i i
i
i (^) C p i i
i C p i i
7 8
u
9
u
7 8
u
9
u
−
−
−
−
∗
∗
. , . , . , . ,
0 025
0 025
0 025
0 025
R W
I T
R W
I T
if intercept is included
otherwise
if intercept is included
otherwise
Durbin-Watson Statistic
DW
e e
c e
i i i
l
i i i
= l
=
=
∑
∑
1
2
2 2
1
I T
where ~e (^) i = ei gi.
