EDSPE 511 Study Guide White (2006) Methods for Computing Reliability
Generally
Try to develop an appreciation for: (1) the range of methods available for
computing data collection reliability estimates; (2) how different methods can lead to
huge differences in the size of the reliability estimate achieved; and (3) when each
method of computing data collection reliability might be most useful in the research
process.
Terms
Frequency ratio / overall
session reliability
Point-by-point agreement
Occurrence reliability
Non-occurrence reliability
Weighted combined
occurrence & non-
occurrence reliability
Pearson product moment
correlation reliability
Phi (nonparametric)
correlation reliability
Kappa
Questions
1. Name one approach for estimating data collection reliability that would be
particularly useful when developing behavioral definitions or coding systems.
2. When should you use “occurrence” reliability estimates rather than “non-
occurrence” reliability estimates (if, indeed, you needed to select one or the
other)?
3. Why do you think most researchers prefer to report “overall session” reliability
estimates instead of one of the others? (Note: White doesn’t really address this
issue in this text, but mentions it in class — especially if someone asks.)
