Download Sampling and Validity in Research: Representative Samples and Subpopulations - Prof. Me Gr and more Study notes Teaching method in PDF only on Docsity!
EDRM 700
GREDLER
Sampling/External Validity
Researchers typically conduct studies on a sample of individuals. The expectation is that the findings will apply to the larger group of which the research participants (sample) are members. The larger group is referred to as the population. On occasion, the entire population may be sampled for the research study. Suppose, for example, the population is the senior class in a small high school. The entire class of 60 students may be sampled for the study. Typically, however, sampling the entire population is not feasible in terms of cost and logistics. A sample from the population is studied instead.
Research problems typically identify the population of interest as indicated in the following examples.
Ten-year-old boys and girls Do 10-year-old boys and girls differ in linguistic ability?
Sixth graders taking science Does peer tutoring produce a greater understanding of science for 6th^ graders than small-group discussion?
University freshmen Do study habits, in addition to other factors predict achievement in university freshmen?
Third-year teachers in What do third-year teachers in district X district X think of the district’s mentoring program for new teachers?
Second-semester freshmen What do second-semester freshmen think of University 101 as an orientation to university life?
The population of interest, referred to as the target population , is often unavailable. Examples are ten-year-old boys and girls, sixth graders taking science, and university freshmen. The population to which the researcher can generalize the study results is the accessible population. The sample for the study is drawn from the accessible population.
Example: Sixth graders taking science (target population) Sixth graders in Richland District I (accessible population) Ten percent of the sixth graders (sample)
Because the researcher wants the results to apply to the population, the sample should be as representative as possible of accessible population. Stated another way, the intent is
for the results of the study to generalize to the population. The extent to which the study results can be generalized to the population is an issue of external validity.
Random sampling methods
Methods that have the greatest chance of producing a sample that is representative of the population are some type of random sampling. The purpose is to ensure, as much as possible, that there is no systematic variation between the characteristics of the sample and the characteristics of the population. In other words, differences between the individuals selected for the sample and the individuals in the population are the result of only chance variation.
Simple random sample. A simple random sample is one in which each and every member of the population has an equal and independent chance of being selected. This means that only chance determines whether a person is or is not selected for the sample. If the random sample is large enough, it is likely to be representative of the population. A simple way to illustrate the selection of a random sample is as follows:
(a) Put all the names of those in the population on slips of paper and place them into a large bowl. (b) Then draw one paper slip from the bowl and record the name (or number) that it shows. (c) Return that paper slip to the bowl, and shake the bowl. (d) Continue this process until the total number for the sample is reached.
The above process, however is not feasible for even relatively small populations. Therefore, a random numbers table may used. (Several websites generate sets of random numbers, e.g., www.randomizer.org). Statistical software programs can select random samples. The researcher first numbers the members of the population and enters the numbers into the computer so that a random sample can be generated.
Stratified random sampling. The population of interest for the research study may not be homogeneous; it may be composed of two or more subpopulations. To ensure that these subpopulations are adequately represented in the sample, the researcher may use stratified random sampling. The steps in stratified random sampling are as follow:
(a) Identify the two or more subpopulations in the accessible population. (The subpopulations are referred to as strata .)
(b) Randomly select research participants from each subpopulation.
The two methods of random selection from the subpopulations are equal allocation and proportional allocation.
Equal allocation. In equal allocation, equal numbers are selected from each subpopulation (stratum). For example, if the three strata are Native Americans, African-
Systematic sampling is often used in research that studies large populations and when lists of the population members are available. Directors of institutional research often use this method because it requires less work than simple random sampling.
Researchers should be aware of one potential, although unlikely, problem with systematic sampling. The problem occurs if there is a cyclical pattern of characteristics within the list(s) of population members. For example, suppose that a high school is planning to survey a sample of students on an important topic. The homeroom teachers, instead of sending the principal the alphabetical rolls of their homeroom students, send lists that rank the homeroom students from high to low in achievement. The principal selects every 10th student from each list of approximately 30 students, beginning with the 2 nd^ student on each list. From each list, the selections are the 2nd, 12th, and 22nd^ students. Systematic sampling, in this case, results in a biased (nonrepresentative) sample because the poorest students, those numbered 23 to 30 do not have a chance of being selected.
Convenience sampling. At times, obtaining a random or systematic sample is not possible and the researcher uses a convenience sample. Basically, a convenience sample is exactly what the name implies – the sample was conveniently available. For example, several grades may be available in a school because the school is implementing some new initiatives and the district is seeking some data about the initiatives. In other cases, a district, professor, or college may volunteer classes or courses for a study for various reasons.
In general, a convenience sample should not be considered to meet the requirements for generalizing the findings of the study to a defined population. The sample does not meet the requirements for assuming that the study has external validity. That is, the finding cannot be automatically extended to a defined population; the population that the convenience sample represents is not clear.
When the researcher cannot avoid relying on a convenience sample, two requirements should be met: (a) extensive demographic information and other characteristics of the sample should be documented, and (b) the study should be replicated with other samples to decrease the possibility that the research findings are only a one-time occurrence (Fraenkel & Wallen, 2003, p. 107).
The issue of sampling bias
A biased sample is one that does not represent the population. A biased sample can be due to a number of sources (Wiersma & Jurs, 2009). It is a threat when (a) nonrandom sampling is implemented, or (b) random sampling occurs with a biased source (p. 331). For example, the focus of the study may be activities for middle-school science, but the convenience sample consists of only high-ability classes. The findings are not generalizable to average or typical classes. The research lacks external validity.
Perhaps the best-known example of random sampling that resulted in a biased sample occurred in the 1936 presidential election. Literary Digest conducted a survey using random sampling to identify the sample. The survey results predicted that Alf Landon would win the presidential election, not the incumbent Franklin D. Roosevelt. However, telephone directories and automobile registration lists were used to select the sample— and these lists did not represent the voting population (p. 331).
The issue of sampling bias is that the findings from a biased sample cannot legitimately be generalized to the population because of either (a) systematic differences between the participants in the research study and the population that are known, or (b) unknown differences that limit generalization to a population. In such situations, the research study lacks external validity.
References
Fraenkel, J. R., & Wallen, N. E. (2003). How to design and evaluate research in education (5th^ ed.). Boston: McGraw-Hill.
Wiersma, W., & Jurs, S. G. (2009). Research methods in education. Boston: Allyn & Bacon.
