Download The Impact of Color and Design on Mail Questionnaire Response Rates: A Review and more Study notes Design in PDF only on Docsity!
EDITOR'S USDA Forest service Research Paper P N W - 140
FILE COPY
Questionnaires- for Research :
L An Annotated Bibliography on
L
Design, Construction, and Use
Dale R. Potter
Kathryn M.Sharpe
John C. Hendee
Roger N. Clark
PACIFIC NORTHWEST FOREST AND RANGE EXPERIMENT STATION
U.S.DEPARTMENT OF AGRICULTURE FOREST SERVICE PORTLAND,OREGON.
ABSTRACT
Questionnaires as social science tools are used increasingly to study people aspects of outdoor recreation and other natural resource fields. A n annotated bibliography including subjective evaluations of each article and a keyword list is presented for 193 references to aid researchers and managers in the design, construction, and use of mail questionnaires.
Keywords: Bibliography, questionnaires, recreation, natural resources, public opinion surveys. research.
ABOUT THE AUTHORS
Dale R. Potter i s a Social Scientist with the Outdoor Recreation Research Project of the Pacific Northwest Forest and Range Experiment Station.
Kathryn M. Sharpe was a Recreation Research Technician cooperating with the Pacific Northwest Forest and Range Experiment Station.
John C, Hendee i s Recreation Research Project Leader for the Pacific Northwest Forest and Range Experiment Station.
Roger N. Clark i s a Social Scientist with the Outdoor Recreation Research Project of the Pacific Northwest Forest and Range Experiment Station.
CONTENTS
P a g e
INTRODUCTION............................... 1
ABBREVIATIONS FOR REFERENCE SOURCES.............. 2
ANNOTATED BIBLIOGRAPHY....................... 3
AUTHOR INDEX............................... 74
KEYWORD INDEX.............................. 77
INTRODUCTION
This bibliography was compiled in response to frequent requests for information on questionnaires by outdoor recreation researchers and the increasing use and misuse of questionnaires by investigators studying public reaction in the area of natural re- sources.u Questionnaires are a long standing tool for gathering data in social science research but are used increasingly by researchers and professional managers unfamiliar with the problems associated with their use in fields such as recreation and natural resource. The objective of this annotated bibliography is to make available a summary and appraisal of literature on the construction and use of questionnaires for research purposes.
Several features are built into the bibliography to increase its usefulness:
First, the annotations summarize the content and conclusions of the article rather than merely describing what the article is about. Of course, readers are fore- warned against quoting annotations directly without first consulting the original source because our interpretation of the original paper may be different from theirs.
Second, accompanying most annotations is our subjective evaluation of the article. These evaluations may reflect different standards among the evaluators; nevertheless, they should be useful in helping readers decide which references warrant their further attention.
Third, each citation contains a list of keywords geared specifically to topics covered in the article. These keywords are the basis for the index in the back of the bibliography to guide readers to those aspects of questionnaires in which they might be interested.
In conclusion, readers contemplating the use of questionnaires are advised that the complexities of using questionnaires for research are often vastly underrated. The popularity of the method often rests on ignorance of associated problems of data analysis, bias, reliability, and validity of results.
Persons desiring publications included in this bibliography should consult the reference source cited o r write to the agency which sponsored the reference. Neither the Pacific Northwest Forest and Range Experiment Station nor the authors can provide copies of any references listed in this bibliography.
Portions of this work were conducted under cooperative agreement between the Forest Service and the University of Washington College of Forest Resources.
ANNOTATED BIBLIOGRAPHY
- Ash, Philip, and Edward^ Abrarnson
- The effect of anonymity on attitude-questionnaire response. J. Abnorm. & Soc. Behav. 47(3): 722-723. Results from three attitude scales of ethnocentrism, political-economic conservatism, and anti-Negro prejudice administered to 9 6 college sociology students showed no significant difference between half the group required to sign answer sheets and half who remained anonymous. EVALUATION: Extension of results should be made with caution--students are not necessarily representative of other sampling populations. KEYWORDS: Anonymous respondent.
- Azrin, N. H. , W. Holz, and I. Goldiamond 19 61. Response bias in questionnaire reports. J. Consult. Psychol. 2 5(4): 324- 32 6. To investigate the validity of an earlier study of military personnel con- cerning behavioral reactions indicative of fear in combat, the authors used two questionnaire forms on two groups of college students totaling 160 people. A response bias was found toward certain symptoms of combat fear regardless of whether o r not the symptom was alleged to occur. Correlations between stu- dents and fliers were high. Probably the same type of response bias influenced the combat fliers. Unless objective and direct means of measurement are avail able, the questionnaire responses may be independent of the behavior being studied. EVALUATION: Good paper reminding researchers of bias involved if certain lestionnaire answers are expected and the respondent learns of this expectation. KEYWORDS: Bias: nonresponse, stated vs. actual behavior.
- Bachrack, Stanley D., and Harry M. Scoble
- Mail questionnaire efficiency: controlled reduction of nonresponse. Publ. Opin. Quart. 31: 264-271. Contributors to the National Committee for an Effective Congress as of 1960 were sampled with a 25-page questionnaire yielding a 63.5-percent re- sponse rate. A^ two-page^ followup^ yielded an additional 19.5 percent.^ These procedures were used to reduce nonresponse: a prequestionnaire explanatory letter, a prestamped and addressed return envelope, and a post card for re- questing report of findings. Followup letters repeated relevant information, and a second fo'tlowup letter preceded the two-page questionnaire sent to all remaining no~respondents. An identical form of this schedule was sent 1 month later, and thc remaining nonrespondents were sent post cards containing 13 short, censul-type data questions. Three major reasons are suggested that account for nonresponse: infirmities of advancing age, personal disturbances in individual's life style (job change or bereavement), and hostility toward
social survey research. Since a mailed questionnaire is an imposition, resis- tance is overcome by persuasion, persistence, attention to procedural techniques and details calculated to activate a potential respondent. EVALUATION: Good, practical suggestions for increasing returns along with a case study example to support conclusions. KEYWORDS: Followup: mail, post card Incentive: postage Cover letter.
4. Bain, Read
- Stability in questionnaire response. Am. J. Sociol, 37(3): 445-453. A 61-item questionnaire given to 50 college freshmen was resubmitted to them 2-1/2 months later to measure the reliability of original results. One- fourth of the answers changed. Factual personal data changed less than factual family or subjective personal data. Girls changed their answers less often than boys. EVALUATION: The small sample and the author's occasional overenthusiastic and unsupported statements reduce the value of this article. Appears slanted to support the author's point of view. KEYWORDS: Reliability. 5. Ball, Robert Jaudon
- The correspondence method in follow-up studies of delinquent boys. J. Juvenile Res. 14: 107-113, illus. A brief questionnaire was distributed to parents to determine their satis- faction with the behavior of recently paroled sons. Frequency of meaningful response appeared correlated with child's IQ, but parents gave favorable report when opposite was known to be true, Inferences about parents1 intelli- gence were drawn from the manner in which the questionnaire was completed. Rate of return was extremely low (40-percent return with only 45 percent properly filled out). EVALUATION: Dated, superficial, weak. It does alert researchers to the possibility of gaining information from the manner in which a questionnaire is completed. KEYWORDS: Bias: nonresponse,
- Barnette, W. Leslie, Jr. 195 1. The non-respondent problem in lestionnaire research. J. Apple Psychol. 34(6): 397-398. The solution to the nonresponse bias detection when a large nonrespondent group remains after two or three mail questionnaire followups is to randomly sample these nonrespondents and make persistent attempts to contact them. An example is given.
but nonresponse and nonsense answers increased. Test results were, there- fore, consistent with the rule of thumb. EVALUATION: Insufficient information is given to allow the reader to check conclusions, but findings do support other literature. KEYWORDS: Questionnaire: design,
- Baur, E. Jackson
- Response bias in a mail survey. Publ, Opin. Quart. 11: 594- 600, illus. The Veterans Administration mailed a questionnaire to 6,000 World War I veterans; 92 percent were returned. Characteristics of nonrespondents-- interest in questionnaire topic, formal education, age, marital status, and parenthood--were analyzed. Nonresponse bias was highest from difference in interest in questionnaire topic. Forty-nine percent of the nonrespondents had not attained a high school education, whereas of the respondents, 30 percent were without high school education. Those who were married o r had little education were slower to respond. Age and parenthood did not influence res- ponse. Nonrespondents were not the same as late respondents. EVALUATION: Well-written, substantive article but a little weak in statistical analysis of results. KEYWORDS: Bias: nonresponse.
- Bender, DonaldH.
- Colored stationery in direct-mail advertising. J. Appl. P S Y C ~ O ~ ~ 41(3): 161-164.
Navy and A i r Force officers and enlisted men totaling 12,056 received variously colored questionnaires to determine if color would increase the response rates. Response rates for various combinations of color including white, blue, green, canary, pink, and goldenrod ranged from 6.4 to 11. percent. Mailings of colored questionnaires yielded higher returns as a group than white questionnaires, but the difference was statistically insig- nificant, EVALUATION: This experiment is one of few which tests color, but the return rate was so low that results are totally unreliable. KEYWORDS: Incentive: motivation Questionnaire: format.
12. Bennett,^ E.^ M.,^ R.^ L.^ Blomquist, and A.^ C.^ Goldstein 1954. Response stability in limited-response questioning. Publ. Opin. Quart. 18: 218-223.
Coefficients of stability were determined for 30 limited- response questions, for a $-week, test-retest period, on 197 subjects.^ All stabilities were
significantly greater than could be expected by chance. The questions were classified into four general categories, and the mean coefficients of stability for these classes of questions ranged from 0.713 to 0.970. Theoretical implications and polling procedures are described in detail. EVALUATION: Excellent discussion demonstrating problems of interpretation for fixed response-categories between informant and investigator. KEYWORDS: Validity Question: wording.
Benson, Lawrence E.
- Mail surveys can be valuable. Publ. Opin. Quart. 1 G : 234-241.
Six advantages and seven disadvantages of the mail questionnaire for use in public opinion research are discussed based on Gallup Poll experience. The two main disadvantages are difficulty of obtaining returns from a representative cross section of the population and the proportionately greater number of returns that come from strongly biased respondents. Often results of surveys need to be adjusted to account for predicted changes in voter behavior. Mail questionnaires can be valuable provided their limita- tions are known and their results properly understood and correctly inter- preted. Actual survey problems are used as results, and extensive explana- tion is made of factors responsible for gross inaccuracies in the 1936 Literary Digest presidential poll. EVALUATION: Excellent review of problems involved in the use of question- naires with population samples, specifically for political surveys. Good treatment of nonresponse. KEYWORDS: Advantages of questionnaires Bias: nonresponse Applications Disadvantages of questionnaires.
- Bevis, Joseph C.
- Economical incentive used for mail questionnaire. Publ. Opin. Quart. '12: 492-493.
During pretesting by Opinion Research Corporation, lo-, 2 5-, and 50-cent war stamps were distributed with a mail questionnaire. The 25- and 50-cent stamps resulted in roughly the same return rate, but the 10-cent stamp produced definitely lower returns. In the subsequent survey, only 25-cent war stamps were included. Of^ the 2,734 questionnaires,^84 percent were returned, with one followup. A number of the latter respon- dents also returned the stamp. EVALUATION: Information missing on size of pretest samples and percentage returns. Difficult to say that economic incentive o r patriotism resulted in greater returns.
sample; after a followup letter, total returns were 99 percent. Post cards were an important factor in this high percentage of return, although the type of people surveyed and their relationship with the organization had some effect on the return. EVALUATION: Rather superficial; response rate may have approached 99 percent without the post card due to nature of topic and official positions of subjects. KEYWORDS: Anonymous respondent Followup: post-card, mail.
- Bradt, Kenneth
- The usefulness of a post card technique in a mail questionnaire study. Publ. Opin. Quart. 19: 218-222.
Questionnaires mailed to 5,356 dropouts of an armed forces correspondence course included a post card to be returned separately to provide a means of followup without destroying anonymity of the respondent. Slightly more than 4 percent of those who returned a questionnaire failed to return a post card. A n 80-percent return was realized after one followup reminder. This post-card technique has broader application than is generally recognized. EVALUATION: Good case study sf the use of post cards to preserve respondent anonymity. KEYWORDS: Anonymous respondent Followup: mail.
- Brennan, Robert D.
- Trading stamps a s an incentive in mail surveys. J. Marketing 22(3): 306-307. From a census tract with characteristics approximating the average of the market area, 456 households were selected at random. In a series of three experiments, half of the households were either sent o r promised trading stamps o r 25 cents, and half were given no incentives o r had the length of their questionnaire reduced. The results of the three experiments indicate that small incentives do not increase the percent of mail returns significantly and that the additional cost does not warrant them, Such procedure merits consideration only if maximal information is desired. EVALUATION: Study supported by fact that return rates were extremely low-- 22 to 29 percent. KEYWORDS: Incentive: money, reward.
- Britton, Joseph H e , and Jean 0. Britton
- Factors^ in the return of questionnaires mailed to older persons. J. Appl. Psychol. 35(1): 57-60.
One study of 328 retired Y. M. C, A. secretaries received 68-percent returns from a preliminary questionnaire about retirement and 51.9-percent returns from a longer one on the same subject. Followup letters to the 57 secretaries who had returned neither questionnaire resulted in a 50.8-percent return. Generally the same procedure was followed for a second study of 2,853 retired teachers. A 20-percent sample of nonrespondents was interviewed. Results show that the retired men in both studies had higher return rates. Former school administrators had higher returns than school teachers, and Y. M. C. A. secretaries with the highest returns were most apt to have served 25 years o r more and to have moved more often since retirement. EVALUATION: Case study examples not tied to any broader framework with which to generalize results and conclusions.
KEYWORDS: Questionnaire: length Bias: nonresponse Followup: mail.
Brooks, Vernon
- Can you trust mail questionnaires? Printers' Ink 86, 88, 90, 92, 96, 100, Registered voters in New York City were selected in a market survey to determine if the respondents were a representative cross section of society. Their results showed that the respondents tended to be better educated, and to be in the upper income and professional and managerial occupation classes. The 20- to 29-year age group along with the 50-year-and-older group were underrepresented while the 30- to 49-year-old group was overrepresented. The authors conclude that if you are seeking opinions from a homogeneous group, mail questionnaires are fairly reliable, but if you seek opinions from a cross section of society, the mail survey is not reliable. EVALUATION: This article points to some areas that the researcher should consider. The results may be questioned due to biasing effect of a voter registra- tion sample being used as a representative cross section of people. KEYWORDS: Bias: representativeness.
Brown, Morton L.
- Use of a postcard query in mail surveys.^ Publ.^ Opin.^ Quart. 29: 635-637. A sample of 523 physicians was subdivided into two subsamples. The first received a two-page query consisting of a cover letter, two screening questions, and space for patient diagnosis. The second subsample received a cover letter and a return post card containing only the two screening questions. For the original mailing, the post-card form elicited a 15-percent greater response, but this difference disappeared at the conclusion of the second mailing. Only 5-percent fewer contacts were required for the post-card query group. Most significantly, the feasibility of the post-card query is questionable because only 80 percent of the physicians responded to the additional query sent following receipt'of the post- card form. The post-card query procedure was considered less efficient than the alternate query for eliciting the type of data needed.
Data of known validity from a sample of general hospitals were compared with data obtained from the responses of 462 persons in an interview procedure, and 465 persons in a self-enumerative procedure. Results show that when records are available or when reporting from respondents is done by proxy, the self enum- erative procedure is more accurate. In contrast, diagnosis and type of surgery are reported more accurately in an interview. This is consistent with the common observation that interviews about episodes involving threatening or embarrassing surgery is better than in self-enumerative procedures. Education of the respondent was more important in the interview. But respondent motivation was found more important in self-enumerative procedure, which stresses the need for developing special motivational techniques if this procedure is employed. EVALUATION: A^ well-written report tied directly to the data.^ Hypotheses are developed and tested, and conclusions consistent with the data are drawn. KEYWORDS: Mail questionnaire vs. interview Val idity.
- Cantril, Hadley
- Problems and techniques experiments in the wording of questions. Publ. Opin. Quart. 4: 330-332. In a national survey of 3,100 persons, alternate wording of two questions was given. The first question read as follows,^ "Do^ you approve of Sumner^ Wellesl visit to European capitals ?" and its alternate read, "Do you approve of President Roosevelt s sending Sumner Welles to visit European capitals? l t Response choices were: approve, disapprove, and no opinion.^ When^ RooseveltTsname was used, more people had opinions and although the percentage of people who approved the visit remained identical at 43 percent, more people disapproved when it was con- nected with Roosevelt. Response to the question without the President's name was 25 percent and response to the one with his name was 31 percent. This was more true of women than of men. EVALUATION: One should gain an appreciation for the wording of questions by reading this article. The point to be made is, be extremely cautious in wording questions. For example, in the first question the respondent is asked about Sumner Wellesl visit, and in the second question, about Rooseveltts decision. KEYWORDS: Question: wording.
- Cavan, Ruth Shonle
- The questionnaire in a sociological research project.^ Am. J. Sociol. 38(5): 72 1-727. The reliability of an 80-item questionnaire was determined by retesting 123 eighth-grade children after a 1-week interval. The replies of 62 pairs of siblings were compared, and 29 mothers were interviewed to check on factual questions. Factual questions about parents were completely reliable in 92 percent of the cases; facts about the home, in 95 percent of the cases; and those involving estimates of numbers, time, etc, , 62 percent. There was 38-percent disagreement in attitudinal questions, but half were minor. Concludes that questions involving attitudes o r estimates have lower reliability than factual questions. Reliability is increased by avoiding fine detail.
EVALUATION: Valuable even though dated, Poorly written with some lack in design and slight inconsistencies in reporting, KEYWORDS: Reliability Attitudes.
- Champion, DeanJ., andAlanM. Sear 1969, Questionnaire response rate: a methodological analysis. Soc. Forces 47: 335-339. A total sample of 2,290 people, from the city directories of Knoxville, Nashville, and Chattanooga, were mailed questionnaires resulting in 42-percent return after one followup, Special delivery postage yielded returns of 61.2 per- cent versus 4 1.9 percent for regular postage. Six-page and nine-page question- naires were returned more often than three-page questionnaires. High and medium socioeconomic status (SES) persons responded more than low ones. Cover letters with an egoistic appeal (benefits the respondent would get from responding) resulted in greater response than letters with an altruistic appeal (benefits the research organization would get) from their response. Higher SES respondents favored altruistic-oriented cover letters, and Iower SES respondents favored egoistic. 16 references. EVALUATION: Informative study of response rate, clearly written with good literature review. KEYWORDS: Incentive: postage Cover letter.
29. Clausen, John A. , and Robert N. Ford
- Controlling bias in mail questionnaires. Am. Statist. Assoc. J. 42(240): 497-511. Nonresponse bias can be attacked on two fronts: (1)increase response and thus minimize unknown bias from nonrespondents and (2) estimate and correct for bias existing in nonresponding groups. Presents empirical data and literature review to spur returns. Concludes that a well-written cover letter plus followups can motivate response. Airmail and special delivery letters often help. Person- alized greeting o r signature do not, Interest in the questionnaire can be increased if questions of general o r widespread interest are included. Estimating response bias may include consideration of respondent-nonrespondent homogeneity such as education or consistently different answers in successive waves of response after f ollowups. EVALUATION: Review of bias problems plaguing questionnaires but some too specific to be of general help. KEYWORDS: Incentive: postage, motivation Followup: mail, personal contact Cover letter,
30, Colley, R. H.
- Don't^ look down your nose at mail questionnaires.^ Printers' Ink 21O(March): 21, 104, 106, 108.
were rearranged each time, and irrelevant questions inserted to disguise the pur- pose of the study. Seventy-two percent of the answers were consistent between applications. Factual questions showed the least reliability, differences in class levels o r age of respondents were consistent. Questionnaire data should be accepted with caution until the methodological problems of reliability are investi- gated further. EVALUATION: A note of caution against blind acceptance of questionnaire data. Contradicts other findings that factual questions are most reliable. Does not suggest guidelines to improve reliability or validity. KEYWORDS: Reliability Validity.
- Davis, Robert A. , and Edwin L. Barrow 1935, A critical study of the questionnaire in education. Educ. Admin. & Superv. 2 1(Feb. ): 137-144. Over a period of 38 years, 500 questionnaires used in educational studies were reviewed to determine the rate of return, types of response called for, techniques used to increase returns, reliability, validity, and source of respondent information (fact, opinion, memory). Authors found that reports of many studies omit vital information such as length and number of returns of questionnaires. Reliability and validity of most questionnaires are probably so low that results are inconclusive. More appropriate statistical techniques are needed to determine reliability and validity. EVALUATION: Review article based on logical research criteria, Dated, but problems identified still exist. KEYWORDS: Bias: nonresponse.
- Deming, W. Edwards
- On errors in surveys. Am. Sociol. Rev. 9(4): 359-369. Thirteen factors affecting the usefulness of surveys are: variability in response, differences between kinds and degrees of canvas, bias and variation arising from the interviewer, bias from sponsorship, imperfections in question- naire design and tabulation plans, changes taking place prior to availability of tabulations, bias from late reports, bias from unrepresentative selection of respondents o r the period covered, sampling errors and biases, processing errors, and errors in interpretation, Errors from some of these factors are larger than commonly supposed; research must find ways of reducing errors and survey limi- tations. There is a special responsibility in presenting data to describe all diffi- culties to reduce errors of interpretation and utilization. A s knowledge about bias and variability increases, their effects can be taken into account systemati- cally and surveys will be less hindered. EVALUATION: Excellent paper summarizing major sources of error and their impact on survey research results. KE YWORDS: Bias Questionnaire: design Interpretation.
- Desing, Minerva F.
- Suggestions^ to^ the novice in the mechanics of research.^ Sch. Rev. 49(March): 206-212. A list of suggestions is given to assist students, clerical help, and novices in gathering, organizing, and treating data to guard against ambiguities, inconsis- tencies, and other irritations impeding progress (and validity) of a questionnaire study. Twelve suggestions are given for gathering data and 23 relevant to coding, analysis, and cleric a1 procedure. EVALUATION: A good list of flhowto do itf1suggestions and precautions that would be vital to the novice and useful for the expert to review. KEYWORDS: Questionnaire: design Coding Analysis.
- Deutscher, Irwin
- Physiciansf reactions to a mailed questionnaire: A study in llresistentialism.l1 Publ. Opin. Quart. 20: 599-604. A questionnaire headed llPublic Images of Female Occupations, l1 was mailed to a random sample of 379 male physicians. Returns were not received from 43 percent. The following social-psychological factors were isolated as potential reasons for nonresponse: resistance to demands to express stereotypes and broad generalizations; resistance to the limited alternatives presented in multiple-choice questions; resistance to value-laden, linguistic symbols such as llclassll;and resistance to projective- type questions whose meaning and purpose are not clear to the potential respondent, The reduction of such forms of resistance would reduce nonresponse and incomplete returns. E VALUATION: Excellent suggestions worth consideration. KEYWORDS: Questionnaire: design Bias: nonresponse.
- Words and deeds: social science and social policy. Soc, Probl. 13(3): 235-254. Social science conclusions are often based on verbal response to questions about attitudes and behaviors. But as early as 1934, evidence by LaPiere showed there may be no positive relationship between what people say and do, and there may be a high inverse relationship. However, both empirical evidence and theo- retical rationale about this problem have been largely ignored. The social science model shows uncommon concern for methodological issues of reliability, neglect- ing problems of validity. A technology inappropriate to the understanding of human behavior has been developed. We do not know under what conditions a change in attitude anticipates a change in behavior and vice versa. Ideally, we should obtain information from the same population on verbal behavior and interaction on behavior under natural social conditions. The empirical evidence can best be'summarized as reflecting wide variation in the relationships between attitudes and behaviors. Role theory can help us understand why man is constrained from acting as he "ought, l1 and much can be learned from the undeveloped field of sociolinguistics.
