Matrix Notation OLS Estimator Variance-Covariance R-square F Test Bootstrapping
Precept 5: Simple OLS
Soc 500: Applied Social Statistics
Simone Zhang1
Princeton University
October 2016
1This draws material from Matt Blackwell.
Prepare for your exams
Study with the several resources on Docsity
Earn points to download
Earn points by helping other students or get them with a premium plan
Guidelines and tips
Prepare for your exams
Study with the several resources on Docsity
Earn points to download
Earn points by helping other students or get them with a premium plan
Matrix Notation in Statistical Models: OLS Estimator, Variance-Covariance, R-square, Lecture notes of Statistics
An introduction to matrix notation in statistical models, focusing on the Ordinary Least Squares (OLS) estimator, variance-covariance matrix, R-square, F test, and bootstrapping. It covers the intuition behind these concepts, the variance-covariance matrix of a vector, the matrix version of homoskedasticity, the sampling variance for OLS estimates, and the estimation of error variance.
Typology: Lecture notes
2021/2022
1 / 18
This page cannot be seen from the preview
Don't miss anything!
Related documents
Partial preview of the text
Download Matrix Notation in Statistical Models: OLS Estimator, Variance-Covariance, R-square and more Lecture notes Statistics in PDF only on Docsity!
Precept 5: Simple OLS
Soc 500: Applied Social Statistics
Simone Zhang
1
Princeton University
October 2016
1 This draws material from Matt Blackwell.
Todayâs Agenda
Basic matrix operations
Review matrix notation for linear regression
Notation
OLS estimation
Variance-covariance matrix
R-square
F-test
Bootstrap
OLS Estimator
β = (X
ⲠX)
â 1 X
Ⲡy
Whatâs the intuition here?
âNumeratorâ X
Ⲡy: is roughly composed of the covariances
between the columns of X and y
âDenominatorâ X
ⲠX is roughly composed of the sample
variances and covariances of variables within X
Thus, we have something like:
β â (variance of X)
â 1 (covariance of X & y)
This is a rough sketch and isnât strictly true, but it can provide
intuition.
Variance-Covariance Matrix
The homoskedasticity assumption is different: var(u|X) = Ď
2
u
I
n
In order to investigate this, we need to know what the variance
of a vector is.
The variance of a vector is actually a matrix:
var[u] = ÎŁ u
var(u 1 ) cov(u 1 , u 2 )... cov(u 1 , u n
cov(u 2 , u 1 ) var(u 2 )... cov(u 2 , u n
cov(u n , u 1 ) cov(u n , u 2 )... var(u n
This matrix is symmetric since cov(ui , uj ) = cov(uj , ui )
Sampling Variance for OLS Estimates
Under assumptions 1-5, the sampling variance of the OLS
estimator can be written in matrix form as the following:
var[
β] = Ď
2
u
(X
ⲠX)
â 1
This matrix looks like this:
β 0
β 1
β 2
β K
β 0 var[
β 0 ] cov[
β 0
β 1 ] cov[
β 0
β 2 ] ¡ ¡ ¡ cov[
β 0
β K
]
β 1 cov[
β 0
β 1 ] var[
β 1 ] cov[
β 1
β 2 ] ¡ ¡ ¡ cov[
β 1
β K
]
β 2 cov[
β 0 ,
β 2 ] cov[
β 1 ,
β 2 ] var[
β 2 ] ¡ ¡ ¡ cov[
β 2 ,
β K
]
βK cov[
β 0 ,
βK ] cov[
βK ,
β 1 ] cov[
βK ,
β 2 ] ¡ ¡ ¡ var[
βK ]
Estimating Error Variance
Note that we never observe the true error variance, Ď
2
u
. We can
estimate it with the following:
Ě Ď
2
u
Ëu
ⲠËu
n â (k + 1 )
where n-(K+1) = residual degrees of freedom and
Ě u
â˛ Ě u = (y â X
β)
Ⲡ(y â X
β)
Sum of Squares
1 2 3 4 5 6 7 8
6
7
8
9
10
11
12
Total Prediction Errors
Log Settler Mortality
Log GDP per capita growth
Sum of Squares
1 2 3 4 5 6 7 8
6
7
8
9
10
11
12
Residuals
Log Settler Mortality
Log GDP per capita growth
F Test Procedure
The F statistic can be calculated by the following procedure:
(^1) Fit the Unrestricted Model (UR) which does not impose H 0
(^2) Fit the Restricted Model (R) which does impose H 0
(^3) From the two results, compute the F Statistic:
F 0 =
(SSRr â SSRur )/q
SSRur /(n â k â 1 )
where SSR=sum of squared residuals, q=number of restrictions,
k=number of predictors in the unrestricted model, and n= # of
observations.
Intuition:
increase in prediction error
original prediction error
The Bootstrap
We see a single sample that is a draw from a population:
Thereâs a true mean loan amount; we only observe one sample
Since we cannot resample from the population, we resample from the
sample!
Idea: Within a loop, generate a bootstrapped sample:
(^1) Sample from { 1 , 2 ,... , N} with replacement
2 Re-calculate the quantity of interest on each bootstrapped sample
(^3) Resampling from the sample approximates sampling again from the
full population (giving us a sense of the sampling distribution)
(Thanks to Ted Enamorado for sharing slides on bootstrapping)
Simple Example with Sample Means
Let X i
Bootstrapped Samples:
X
boot
X
boot, 1
X
boot, 1
X
boot, 1
Bootstrapped Standard Error
Bootstrapped Standard Error
sd(
X
boot
Bootstrapped Confidence Interval:
Take the 2.5% and 97.5% quantiles of
Xboot