Model Building - Lecture Notes - Hierarchical Linear Models | STAT 587, Study notes of Statistics

Download Model Building - Lecture Notes - Hierarchical Linear Models | STAT 587 and more Study notes Statistics in PDF only on Docsity!

Model Building

Model Building

Edps/Psych/Stat/ 587

Carolyn J. Anderson

Department of Educational Psychology

I L L I N O I S^ UNIVERSITY OF ILLINOIS AT URBANA

CHAMPAIGN

Introduction Preliminary Fixed Effects Preliminary Random EffectsStructure Model Reduction Model Diagnostics Model Building

Outline n

Introduction

n

Steps in an analysis:1. Selecting Preliminary Fixed Effects Structure2. Selecting a Preliminary Random Effects Structure3. Model Reduction4. Model Diagnostics5. Interpretation

n

SAS

Major References: Verbeke & Molenbergs: Chapters 4, 9 andSnijders & Boskers chapter 9

Introduction l

Introduction l

Dependency: Mean and Covariance l

The Covariance Structure l

Two Stage Process l

General Guidelines on Model Selection l

Basic structure for Model Building l

Basic structure for Model Building Preliminary Fixed Effects Preliminary Random EffectsStructure Model Reduction Model Diagnostics Model Building

Dependency: Mean and Covariance

Mean Structure

Covariance Structure











Estimation of

Γ

,

T

,

σ

2

Covariance matrix for

Γ

,

T

,

σ

2

t

-tests and

F

-tests

Confidence intervals

EfficiencyPrediction





?

?

(Copied from Verbeke & Molenberghs)

Introduction l

Introduction l

Dependency: Mean and Covariance l

The Covariance Structure l

Two Stage Process l

General Guidelines on Model Selection l

Basic structure for Model Building l

Basic structure for Model Building Preliminary Fixed Effects Preliminary Random EffectsStructure Model Reduction Model Diagnostics Model Building

The Covariance Structure n

It explains and helps to understand the random variability inthe data; the “unexplained” variance.

n

It is highly dependent on the fixed effect structure (i.e., thesystematic part of the variability of

Y

j

).

n

An appropriate one is required for valid inference regardingthe mean structure (unless you use robust estimation).

n

Under-parameterized covariance structure invalidatesinferences.

n

Over-parameterized covariance structure leads to inefficientestimation and poor standard errors.

n

Is interesting in helping to understand the random variabilityin the data.

n

An appropriate covariance structure leads to betterpredictions.

Introduction l

Introduction l

Dependency: Mean and Covariance l

The Covariance Structure l

Two Stage Process l

General Guidelines on Model Selection l

Basic structure for Model Building l

Basic structure for Model Building Preliminary Fixed Effects Preliminary Random EffectsStructure Model Reduction Model Diagnostics Model Building

General Guidelines on Model Selection... Finding an appropriate linear mixed model for a specificdata set.The procedures presented here n

Are a combination of general modeling guidelines andpossible exploratory data analyses.

n

May not yield the most appropriate model.

n

Do not guarantee that all distributional assumptions aresatisfied.

n

Not an exhaustive set of tools.

Introduction l

Introduction l

Dependency: Mean and Covariance l

The Covariance Structure l

Two Stage Process l

General Guidelines on Model Selection l

Basic structure for Model Building l

Basic structure for Model Building Preliminary Fixed Effects Preliminary Random EffectsStructure Model Reduction Model Diagnostics Model Building

Basic structure for Model Building 1. Remove the

systematic

part from the data.

2. Study the

residuals

in an effort to get a preliminary or

reasonable random effects structure that will permit validinference regarding fixed effects.

3. Remove/revise the

fixed effects

, including testing

substantive research hypotheses.

4. Remove/revise the

random effects

, including testing

substantive research hypotheses.

5. Cycle through steps 3 and 4.6. Model diagnostics on potential final model(s).7. Interpret final model.

Introduction Preliminary Fixed Effects l

Selecting Preliminary Fixed Effects l

Justification for Using OLS l

Procedures for Preliminary Fixed l

Preliminary Fixed by Example l

Preliminary Fixed by Example l

Averaging over Sub-populations (continued) l

Question 1 l

Math Scores by HOMEW l

Math Scores by HOMEW l

Math Scores by HOMEW l

Math Scores by HOMEW l

Math Scores by HOMEW l

Math Scores by HOMEW l

All Schools: Join HOMEW l

Schools Averages w/rt HOMEW l

Schools Averages w/rt HOMEW l

Math Scores by HOMEW l

Schools Means: HOMEW & WHITE l

Another look at WHITE l

Schools Means: HOMEW & WHITE l

Schools Means: HOMEW & RACE l

Smoother Look at Homework and Race l

Caution l

Caution (continued) Model Building

Selecting Preliminary Fixed Effects n

Examine each group graphically.

n

Averaging over sub-populations and graph.

n

Exploring Group Specific Data

u

Measure each group’s goodness of fit.

u

Measure overall goodness of fit.

u

“Testing” for model extension (skip this).

Why Start with Fixed Effects? n

The covariance matrix accounts for all the variability that’snot accounted for by the systematic part of the model.

n

We start with a complex, preliminary fixed effects (i.e.,

X

j

Γ

)

and then remove it from the data.

n

We can ignore dependencies in the data and use ordinaryleast squares estimation to estimate the fixed effects. The justification for using OLS?

Introduction Preliminary Fixed Effects l

Selecting Preliminary Fixed Effects l

Justification for Using OLS l

Procedures for Preliminary Fixed l

Preliminary Fixed by Example l

Preliminary Fixed by Example l

Averaging over Sub-populations (continued) l

Question 1 l

Math Scores by HOMEW l

Math Scores by HOMEW l

Math Scores by HOMEW l

Math Scores by HOMEW l

Math Scores by HOMEW l

Math Scores by HOMEW l

All Schools: Join HOMEW l

Schools Averages w/rt HOMEW l

Schools Averages w/rt HOMEW l

Math Scores by HOMEW l

Schools Means: HOMEW & WHITE l

Another look at WHITE l

Schools Means: HOMEW & WHITE l

Schools Means: HOMEW & RACE l

Smoother Look at Homework and Race l

Caution l

Caution (continued) Model Building

Justification for Using OLS “Generalized Estimation Equation” (GEE) Theory:

The OLS estimate of

Γ

is consistent.

Therefore, we can use

r

ij

=

y

ij

−

x

′ ij

Γ

to study the dependencies in the data.

Introduction Preliminary Fixed Effects l

Selecting Preliminary Fixed Effects l

Justification for Using OLS l

Procedures for Preliminary Fixed l

Preliminary Fixed by Example l

Preliminary Fixed by Example l

Averaging over Sub-populations (continued) l

Question 1 l

Math Scores by HOMEW l

Math Scores by HOMEW l

Math Scores by HOMEW l

Math Scores by HOMEW l

Math Scores by HOMEW l

Math Scores by HOMEW l

All Schools: Join HOMEW l

Schools Averages w/rt HOMEW l

Schools Averages w/rt HOMEW l

Math Scores by HOMEW l

Schools Means: HOMEW & WHITE l

Another look at WHITE l

Schools Means: HOMEW & WHITE l

Schools Means: HOMEW & RACE l

Smoother Look at Homework and Race l

Caution l

Caution (continued) Model Building

Preliminary Fixed by Example Data: NELS88, N=23 schools.

Math:

Response variable.

Homework:

How much time a student spends doing

homework.

SES:

Student’s SES.

Race:

Whether a student is white or non-white.

Gender:

Whether a student is male or female.

Sector:

Whether the school is public or private.

Mean SES:

Average SES of students attending a school.

Introduction Preliminary Fixed Effects l

Selecting Preliminary Fixed Effects l

Justification for Using OLS l

Procedures for Preliminary Fixed l

Preliminary Fixed by Example l

Preliminary Fixed by Example l

Averaging over Sub-populations (continued) l

Question 1 l

Math Scores by HOMEW l

Math Scores by HOMEW l

Math Scores by HOMEW l

Math Scores by HOMEW l

Math Scores by HOMEW l

Math Scores by HOMEW l

All Schools: Join HOMEW l

Schools Averages w/rt HOMEW l

Schools Averages w/rt HOMEW l

Math Scores by HOMEW l

Schools Means: HOMEW & WHITE l

Another look at WHITE l

Schools Means: HOMEW & WHITE l

Schools Means: HOMEW & RACE l

Smoother Look at Homework and Race l

Caution l

Caution (continued) Model Building

Preliminary Fixed by Example Data: NELS88, N=23 schools. Goal of the analysis

: Try to account for differences between

students’ math performance in terms of student characteristicsand school characteristics.Start with some exploratory methods and use the results in ournext stage. Averaging over Sub-populations Question

: Can our response variable (math scores) be

modeled by a linear regression model?Possible graphical displays depend on whether the explanatoryvariables are n

Discrete.

n

Continuous or virtually continuous.

Introduction Preliminary Fixed Effects l

Selecting Preliminary Fixed Effects l

Justification for Using OLS l

Procedures for Preliminary Fixed l

Preliminary Fixed by Example l

Preliminary Fixed by Example l

Averaging over Sub-populations (continued) l

Question 1 l

Math Scores by HOMEW l

Math Scores by HOMEW l

Math Scores by HOMEW l

Math Scores by HOMEW l

Math Scores by HOMEW l

Math Scores by HOMEW l

All Schools: Join HOMEW l

Schools Averages w/rt HOMEW l

Schools Averages w/rt HOMEW l

Math Scores by HOMEW l

Schools Means: HOMEW & WHITE l

Another look at WHITE l

Schools Means: HOMEW & WHITE l

Schools Means: HOMEW & RACE l

Smoother Look at Homework and Race l

Caution l

Caution (continued) Model Building

Question 1 How do the math scores depend on homew?Ordinal variable that’s treated numerically: “homew,” timestudent spends on math homework.Some possibilities: n

Plot all the math scores by homew and fit a smooth curve tothe points.

n

Plot the math scores by homework for each school and fitcurve.

n

Plot the average math scores of students within schools foreach level of homework versus homew.

Introduction Preliminary Fixed Effects l

Selecting Preliminary Fixed Effects l

Justification for Using OLS l

Procedures for Preliminary Fixed l

Preliminary Fixed by Example l

Preliminary Fixed by Example l

Averaging over Sub-populations (continued) l

Question 1 l

Math Scores by HOMEW l

Math Scores by HOMEW l

Math Scores by HOMEW l

Math Scores by HOMEW l

Math Scores by HOMEW l

Math Scores by HOMEW l

All Schools: Join HOMEW l

Schools Averages w/rt HOMEW l

Schools Averages w/rt HOMEW l

Math Scores by HOMEW l

Schools Means: HOMEW & WHITE l

Another look at WHITE l

Schools Means: HOMEW & WHITE l

Schools Means: HOMEW & RACE l

Smoother Look at Homework and Race l

Caution l

Caution (continued) Model Building

Math Scores by HOMEW math

ij

= 45

.

56015 + 3

.

126375(

HOMEW

)

i

j

Introduction Preliminary Fixed Effects l

Selecting Preliminary Fixed Effects l

Justification for Using OLS l

Procedures for Preliminary Fixed l

Preliminary Fixed by Example l

Preliminary Fixed by Example l

Averaging over Sub-populations (continued) l

Question 1 l

Math Scores by HOMEW l

Math Scores by HOMEW l

Math Scores by HOMEW l

Math Scores by HOMEW l

Math Scores by HOMEW l

Math Scores by HOMEW l

All Schools: Join HOMEW l

Schools Averages w/rt HOMEW l

Schools Averages w/rt HOMEW l

Math Scores by HOMEW l

Schools Means: HOMEW & WHITE l

Another look at WHITE l

Schools Means: HOMEW & WHITE l

Schools Means: HOMEW & RACE l

Smoother Look at Homework and Race l

Caution l

Caution (continued) Model Building

Math Scores by HOMEW

Introduction Preliminary Fixed Effects l

Selecting Preliminary Fixed Effects l

Justification for Using OLS l

Procedures for Preliminary Fixed l

Preliminary Fixed by Example l

Preliminary Fixed by Example l

Averaging over Sub-populations (continued) l

Question 1 l

Math Scores by HOMEW l

Math Scores by HOMEW l

Math Scores by HOMEW l

Math Scores by HOMEW l

Math Scores by HOMEW l

Math Scores by HOMEW l

All Schools: Join HOMEW l

Schools Averages w/rt HOMEW l

Schools Averages w/rt HOMEW l

Math Scores by HOMEW l

Schools Means: HOMEW & WHITE l

Another look at WHITE l

Schools Means: HOMEW & WHITE l

Schools Means: HOMEW & RACE l

Smoother Look at Homework and Race l

Caution l

Caution (continued) Model Building

Math Scores by HOMEW