Reliability and Validity - Social Research Methods - Lecture Slides, Slides of Sociology

Download Reliability and Validity - Social Research Methods - Lecture Slides and more Slides Sociology in PDF only on Docsity!

Reliability and Validity

1.Reliability

2. Validity

3. Threats to internal validity

Measurement

• MEASUREMENT is any process by which a

value is assigned to the level or state of some

quality of an object of study

Measuring violence

Violence against a woman can range from beatings, to sexual

violence or torture, to broken bones and very serious injury caused by

pouring of acid or burning the victim alive

Measurement

• Measurement involves the expression of

information in quantifies (numbers) rather

than by verbal statement

• It provides a powerful means of reducing

qualitative data to a more condensed form for

summarization, manipulation, and analysis

Measurement

• The best measure should be both reliable and

valid

What is reliability?

• We often speak about “reliable cars."

• On news people talk about a "usually reliable source“

• In both cases, the word reliable usually means "dependable"

or "trustworthy."

• In research, the term "reliable" also means dependable in a

general sense, but that's not a precise enough definition

Reliability

• Reliability is the consistency of your

measurement, or the degree to which an

instrument measures the same way each

time it is used under the same condition with

the same subjects

Reliability

• A measure is considered reliable if a person's

score on the same test given twice is similar

Reliability of measuring devices

• The slightest variations in measuring devices

in Olympic track and field events (whether it is

a tape or clock) could mean the difference

between the gold and silver medals

Reliability of measuring devices

• Olympic measuring devices, then, must be

reliable from one throw or race to another and

from one competition to another

• They must also be reliable when used in

different parts of the world, as temperature, air

pressure, humidity, interpretation, or other

variables might affect their readings

• There are two ways that reliability is usually

estimated:

• test/retest
• internal consistency

Test-Retest Reliability

• We estimate test-retest reliability when we

administer the same test to the same sample

on two different occasions

•

Test/Retest

• The idea behind test/retest is that you should get the

same score on test 1 as you do on test 2.

• The three main components to this method are as

follows:

• 1) implement your measurement instrument at two

separate times for each subject;

2) compute the correlation between the two separate

measurements

3) assume there is no change in the underlying

condition between test 1 and test 2.

Internal Consistency

• Internal consistency estimates reliability by grouping questions

in a questionnaire that measure the same concept

• After collecting the responses, run a correlation between

groups of questions to determine if your instrument is reliably

measuring that concept.

• Your computer output generates one number for Cronbach's

alpha - and just like a correlation coefficient

• The closer it is to one, the higher the reliability estimate of

your instrument.

Example: Deviance scale

The offenses include the following:

“ How many times in the past year have you…

• carried a hidden weapon other than a plain pocket knife?

• attacked someone with the idea of seriously hurting or killing

them?

• been involved in gang fights?

• hit or threatened to hit a teacher or other adult at school?

• hit or threatened to hit your parents?

• hit or threatened to hit other students?

• had or tried to have sexual relations with someone against

their will

Correlations 1 .174** .755** .197** .019 .242** .000 .000 .000 .586. 869 869 868 864 869 863 .174** 1 .178** .149** .037 .220** .000 .000 .000 .271. 869 879 878 874 879 873 .755** .178** 1 .415** .019 .097** .000 .000 .000 .565. 868 878 878 873 878 872 .197** .149** .415** 1 .112** .258** .000 .000 .000 .001. 864 874 873 874 874 873 .019 .037 .019 .112** 1 .102** .586 .271 .565 .001. 869 879 878 874 879 873 .242** .220** .097** .258** .102** 1 .000 .000 .004 .000. 863 873 872 873 873 873 Pearson Correlation Sig. (2-tailed) N Pearson Correlation Sig. (2-tailed) N Pearson Correlation Sig. (2-tailed) N Pearson Correlation Sig. (2-tailed) N Pearson Correlation Sig. (2-tailed) N Pearson Correlation Sig. (2-tailed) N

Y2-270: CARRIED

HIDDN WEAP -FREQ

Y2-274: ATTACKED

SOMEONE -FREQ

Y2-280: BEEN IN

GANG FIGHTS-FREQ

Y2-292: HIT TEACHER

-FREQUENCY

Y2-294: HIT PARENT

-FREQUENCY

Y2-296: HIT OTHER

STUDENTS -FREQ

Y2-270:

CARRIED

HIDDN WEAP

-FREQ

Y2-274:

ATTACKED

SOMEONE

-FREQ

Y2-280: BEEN

IN GANG

FIGHTS-

FREQ

Y2-292: HIT

TEACHER

-FREQUENCY

Y2-294: HIT

PARENT

-FREQUENCY

Y2-296: HIT

OTHER

STUDENTS

-FREQ

**. Correlation is significant at the 0.01 level (2-tailed).

Reliability Statistics

.438 7

Cronbach's

Alpha N of Items

Validity

• Validity involves the degree to which you are

measuring what you are supposed to

• More simply, the accuracy of your

measurement

Four types of validity

• Conclusion validity

• Internal validity

• Construct validity

• External validity

Example

• Say we are studying the effect of strict attendance

policies on class participation

• Suppose, we observe that class participation did

increase after the policy was established

• Each type of validity would highlight a different

aspect of the relationship between our treatment

(strict attendance policy) and our observed outcome

(increased class participation).

Conclusion validity

• Conclusion validity asks is there a relationship

between the program and the observed

outcome?

• Or, in our example, is there a connection

between the attendance policy and the

increased participation we saw?

Internal Validity

• The key question in internal validity is whether

observed changes can be attributed to your

program or intervention (i.e., the cause) and

not to other possible causes (sometimes

described as "alternative explanations" for the

outcome)

Internal Validity

Construct validity

• It asks if there is a relationship between how I

operationalized my concepts in this study to the

actual causal relationship I'm trying to study?

• Or in our example, did our treatment (attendance

policy) reflect the construct of attendance, and did

our measured outcome - increased class participation

- reflect the construct of participation?

External validity

• External validity refers to our ability to

generalize the results of our study to other

settings.

• In our example, could we generalize our results

to other classrooms?

Reliability & Validity

• We often think of reliability and validity as separate ideas but,

in fact, they're related to each other.

• One of my favorite metaphors is the target

• Think of the center of the target as the concept that you are

trying to measure

• Imagine that for each person you are measuring, you are

taking a shot at the target. If you measure the concept

perfectly for a person, you are hitting the center of the target

• If you don't, you are missing the center. The more you are off

for that person, the further you are from the center.