Lecture 9: Internal Validity and Threats to Validity in Research, Study notes of History

Download Lecture 9: Internal Validity and Threats to Validity in Research and more Study notes History in PDF only on Docsity!

Lecture 9 Internal Validity

Objectives

§ Internal Validity

§ Threats to Internal Validity

§ Causality

§ Bayesian Networks

Construct validity refers to the degree to which inferences can legitimately be made from your study to the theoretical constructs on which those operationalizations were based. Internal Validity is the approximate truth about inferences regarding cause-effect or causal relationships. External Validity: Assuming that there is a causal relationship in this study between the constructs of the cause and the effect, can we generalize this effect to other persons, places or times?

Validity

Statistical validity has to do with basing conclusions on proper use of statistics.

Internal validity

Statistical Validity

H0 (null hypothesis) true H1 (alternative hypothesis) false H0 (null hypothesis) false H1 (alternative hypothesis) True 1 - I (e.g., .95) THE CONFIDENCE LEVEL The probability we say there is no relationship when there is not I (e.g., .05) Type I Error The probability we say there is a relationship when there is not 1 - J (e.g., 80) THE POWER The probability we say there is a relationship when there is one We accept H We reject H We reject H We accept H In Reality We Conclude J (e.g., 20) Type II Error The probability we say there is no relationship when there is one

Construct

Validity

Challenges to Internal Validity

a. History

b. Maturation

c. Experimental mortality

d. Instrumentation

e. Testing

f. Interactions with selection

History

Any events that occur during the course of the

experiment which might effect outcome.

Example: An important event not related to

the experiment affects the measurement

of pre and post-test.

Experimental Mortality

Dropouts from the experiment; especially when the dropouts systematically bias the comparisons Example: If your include pretest subsequent dropouts in the pretest and not in the posttest you will bias the test based on the characteristics of dropouts. And, you won't necessarily solve this problem by comparing pre-post averages for only those who stayed in the study. This subsample would certainly not be representative even of the original entire sample

Instrumentation

Any way in which the instrument used for

observation or collecting data changes from the

pre-test to the post-test.

Example: Test, Interview, Measurement

Technique or Instrument.

Regression Threat

The highest and lowest scorers will regress toward the mean at a higher rate than those who scored close to the mean. There will be a higher degree of regression for unreliable measures than for more reliable ones.

• The degree of asymmetry (i.e., how far

from the overall mean of the first measure

the selected group's mean is)

• The correlation between the two

measures

The absolute amount of regression to

the mean depends on two factors:

Interactions with Selection

When there is a relationship between the treatment and the selection of subjects, this causes a systematic bias which affects causal inference Example: A selection threat is any factor other than the program that leads to posttest differences between groups. These include: history, maturation, test, instrumentation, mortality, and regression.

SIMPSON’S PARADOX

Any statistical relationship between two variables may be reversed by including additional factors in the analysis.