Download Understanding Research Validity and Reliability: A Comprehensive Guide and more Study notes Production and Operations Management in PDF only on Docsity!
Reliability and Validity
Validity
Ā Term most often used to judge the
quality or merit of a study
Ā Found on truth or fact, capable of being
justified or defended
Ā Webster, 1997
Ā Is the study/instrument measuring what
it says it measuring?
Reliability
Ā Fit to be trusted or relied on, trustworthy,
dependable
Ā Webster, 1997
Ā Consistency throughout a series of
measurements
Ā Cronbach, 1960, p. 127
Ā If you measure a variable more than once
with the same instrument, you will obtain
similar results.
Measurement Reliability of Each Instrument
Measurement Validity of Each Instrument
Internal
Validity
Measurement Reliability and Statistics for the Study
Operations and Measurement Validity for the Study
External
Validity
Overall Research Validity of the Study
Overall validity of a research study depends on the four major aspects or dimensions of research validity and in turn on the measurement reliability and validity of the several variables.
Four Major Aspect of Research
Validity
Ā Internal validity
Ā External validity
Ā Measurement reliability and statistics
Ā Operations and measurement validity
Measurement Reliability of Each Instrument
Measurement Validity of Each Instrument
Internal
Validity
Measurement Reliability and Statistics for the Study
Operations and Measurement Validity for the Study
External
Validity
Overall Research Validity of the Study
Ā Other methods
Ā Random assignment of treatment to intact
groups
Ā Better than self assignment
Ā Example: Choosing a certain teacher in school
Ā Matching
Ā Matching participant characteristics other than
the IV
Ā Age, gender, educational level
Ā Pretest to test for equivalence
Ā Associational research
Ā One group
Ā Are participants who scored high on the IV
equivalent to those who scored low?
Control of Experience
/Environmental Variables
Ā Do extraneous variables effect on group more
than the other
Ā Ex: One groups finds out they are the control
Ā May give up
Ā May try extra hard eliminating potential
differences
Ā Associational
Ā Are the experiences for those high on the IV
different form those that are low
Ā Is something other than the IV affecting
the DV for both/all groups?
Ā Historical events, maturation
Ā Longitudinal studies
Ā Maturation IS a variable
Ā Very complex issue!
Measurement Reliability of Each Instrument
Measurement Validity of Each Instrument
Internal
Validity
Measurement Reliability and Statistics for the Study
Operations and Measurement Validity for the Study
External
Validity
Overall Research Validity of the Study
External Validity
Ā Answers the question of generalizability
Ā To what populations or settings can this
effect be generalized?
Ā Two aspects
Ā Population validity
Ā Ecological Validity
Sampling and
Internal/External Validity
Ā Internal validity not directly effected by
sample design
Ā Randomized experiments on a convenience
sample
Ā High internal validity
Ā Randomization control for extraneous
variables and group differences
Ā Careful about the word ārandomā
Ā Random selection of participants = high
external validity
Ā Random assignment of participants to
groups/treatments = high internal validity
Measurement Reliability and
Validity
Ā 2 aspects/dimensions
Ā Measurement reliability and statistics
Ā Operations and measurement validity
Measurement Reliability of Each Instrument
Measurement Validity of Each Instrument
Internal
Validity
Measurement Reliability and Statistics for the Study
Operations and Measurement Validity for the Study
External
Validity
Overall Research Validity of the Study
Measurement Reliability and
Statistics
Ā Also called āstatistical conclusion
validityā
Ā Measurement reliability is involved
(more on this later)
Ā Four important issues:
Ā Reliability of instruments/measures
Ā Where the variables measured reliably?
Ā A test or measure cannot be valid if it is
not reliable
Ā Appropriateness of power
Ā Can a statistically significant relationship be
detected, assuming one exists.
Ā This ability is calledpower
Operations and Measurement
Validity
Ā Are variables appropriately operationally
defined?
Ā Do they measure the concepts or
constructs under investigation?
Ā The key is whether operational
definitions are representative of the
intended concept or construct.
Measurement Reliability of Each Instrument
Measurement Validity of Each Instrument
Internal
Validity
Measurement Reliability and Statistics for the Study
Operations and Measurement Validity for the Study
External
Validity
Overall Research Validity of the Study
Methods to assess
Ā Test-Retest Reliability
Ā Common
Ā Same sample take test A and later take
test B
Ā Correlate the two sets of scores (Pearson
Product Moment Correlation Coefficient)
Ā .8 or higher, the test has good test-retest
reliability
Ā Considerations
Ā Do not establish reliabilityduring the study
Ā Usually already established by someone else
Ā Develop own instrument
Ā When selecting an instrument
Ā Reliability measure above. Ā Length of time between tests similar to time in study Ā As time increase, reliability decreases Ā Samples used to find reliability factor similar to sample in study
Ā Coefficients of Equivalence
Ā Parallel Forms
Ā Test-retest has a carryover effect
Ā This method eliminates that
Ā Two similar yet different forms of an instrument
Ā Eliminate the time between tests issue
Ā Reliability measure the same as test-retest
Ā Correlation Coefficient
Ā Internal Consistency Reliability
Ā Consistency among items on an instrument
Ā The instrument is measuring a single concept/construct
Ā 3 methods
Ā Split-half
Ā KR
Ā Cronbachās Alpha
Ā Percentage Agreement
Ā 2 or more judges
Ā Before the study
Ā Measure of similar observations
Ā Intra-class correlation
Ā When observation of the behavior is the DV
Ā Calculate the reliability between 2 or more judges
Ā Must be interval scaled data
Ā Kappa
Ā Intra-class correlation with nominal data
Measurement Reliability of Each Instrument
Measurement Validity of Each Instrument
Internal
Validity
Measurement Reliability and Statistics for the Study
Operations and Measurement Validity for the Study
External
Validity
Overall Research Validity of the Study
Measurement Validity
Ā āHow well does the instrument assess the
characteristic, construct, or behavior the user
desires to measure?ā
Ā Cromack (1989)
Ā An instrument may be consistent (reliable),
but not valid.
Ā There are measurements for reliability, but
not validity
Ā 4 ways to assess validity
Face Validity
Ā Not really considered to be a recognized
measure of validity
Ā Does the content appear to be
appropriate for the purpose of the
instrument?
Ā Example ā The title and table of
contents of a book.
Content Validity
Ā Refers to the actual content of the
instrument.
Ā Is the concept in the instrument
representative of the content you are trying
to measure?
Ā Questions/tasks are about the content being
measured.
Ā Questions/tasks are representative of real
life.
Ā Concurrent Validity
Ā Similar to predictive
Ā Measure instrument against an external criterion
Ā Have current graduate students take GRE and
then correlate scores
Ā Not as good as predictive
Ā Assuming no difference between undergraduate
and graduate school students
Ā Major draw back ā
Ā Finding a criterion and being able to
measure it
Ā Example ā Are the GRE a good measure of
success in graduate school?
Ā Asking the question āwhat makes a good
graduate student?ā
Construct Validity
Ā The most complex
Ā Constructs
Ā Hypothetical concepts which can not be
observed directly
Ā Intelligence, achievement, anxiety
Ā We know they exist through other
observable behaviors
Ā Cannot observe anxiety, but can observe
anxious behavior i.e. ā sweat, pacing
Ā Construct validity must be grounded in
the theory
Ā This can take time
Ā A process of several studies where the
researcher attempts to demonstrate
validity
