Download Formulating Research Problems in Multimethod Social Research and more Schemes and Mind Maps Sociology in PDF only on Docsity!
Three
Formulating
Research Problems
R
esearch problems are questions that indicate gaps in the scope or the certainty of our knowledge. They point either to problematic phenomena, observed events that are puzzling in terms of our currently accepted ideas, or to problematic theories, current ideas that are chal- lenged by new hypotheses. This chapter first looks at the role of such questions in the research process, and especially the ongoing debate among social scientists as to when and how problems should be formu- lated. Second, we consider methodology’s effect on defining problems, and how the multimethod approach can be used to focus research more sharply upon the substance of research problems. Finally, we consider the role of theory in problem formulation, and how the multimethod approach integrates theory and research more closely in posing these research questions.
The Role of Research Problems
in the Research Process
The problems of everyday life are difficulties to be avoided, if possible. Research problems are eagerly sought after. The difference is that research problems represent opportunities as well as trouble spots. Because
39
scientific knowledge is provisional, all empirical findings and theories are in principle problematic and are, therefore, subject to further investiga- tion. But in addition to seeking more exact confirmations of existing claims to knowledge, research has the equally important goal of generat- ing new claims. Problem formulation is the logical first step toward this goal. As Northrop (1966) writes, “Inquiry starts only when something is unsatisfactory, when traditional beliefs are inadequate or in question, when the facts necessary to resolve one’s uncertainties are not known, when the likely relevant hypotheses are not even imagined. What one has at the beginning of inquiry is merely the problem” (p. 17). The formulation of research problems also has an important social func- tion. As Merton, Broom, and Cottrell (1959) suggest, researchers must jus- tify the demands for attention and other scarce resources that research makes: “In conferring upon the scientist the right to claim that a question deserves the concerted attention of others as well as himself, the social institution of science exacts the obligation that he justify the claim” (p. xix). Achieving significant research results is perhaps the most powerful justifi- cation for such claims, but this type of justification can be offered only after the fact, and only in the event that the research is successful. A compel- ling research problem, by contrast, must marshal support in advance of research and, if it is sufficiently compelling, can even sustain that support through the sometimes fruitless periods that researchers experience. However, despite research problems’ logical priority in inquiry, and their importance as a priori justifications, a problem’s formulation, as John Dewey stresses, is in fact a “progressive” matter. Dewey means that problem formulations are themselves problematic and so require contin- ual attention to assure that the questions being asked will direct research toward the desired end: “If we assume, prematurely, that the problem involved is definite and clear, subsequent inquiry proceeds on the wrong track. Hence the question arises; How is the formation of a genuine prob- lem so controlled that further inquiries will move toward a solution?” (quoted by Northrop, 1966, p. 13).
When and How to Formulate Problems: A Debate
It sometimes seems that there is little about which social scientists agree, and the most effective procedure for formulating research prob- lems is no exception. In particular, there has been considerable debate over whether or not it is important to define problems explicitly in
40 FOUNDATIONS OF MULTIMETHOD RESEARCH
view, prestated problems and hypotheses do much more than make it “easier to program action” (as Schatzman and Strauss [1973] suggest). They discipline research in the interest of testing theory, accumulating knowledge, and achieving a theoretical standpoint independent of the time and place in which researchers presently find themselves.
Overcoming Methodological
Constraints on Problem Formulation
Both sides in the foregoing debate clearly have merit. However, in practice the decision as to when and how research problems should be defined usually depends less upon the perceived merits of one or the other of these procedures than upon the research style selected. Methods differ in their abilities to predict the kinds, quantities, and quality of the data that may be available in any given instance. For example, survey researchers or experimentalists can usually say with more certainty than fieldworkers whether or not the data pertinent to a particular research problem can be readily collected. Fieldwork offers the possibility of many data sources, but it is usually hard to say in advance which data will actu- ally be obtainable. Similarly, Selltiz, Jahoda, Deutsch, and Cook (1959) note the need to take a “wait-and-see” attitude in the use of nonreactive data sources such as statistical records: “The use of such data demands a capacity to ask many different questions related to a research problem.... The guiding principle for the use of available statistics con- sists in keeping oneself flexible with respect to the form in which the research questions are asked” (p. 318). Furthermore, as we will discuss in greater detail in Chapter 4, an empirical search for problems is considerably less expensive with some methods than others. Exploratory experiments and surveys are certainly feasible, but pilot field studies and searches through archives generally cost less, except perhaps for the researcher whose personal expenditure of time and energy usually “fund” such studies. Moreover, discoveries arise in different ways for different methods. Fieldworkers and nonreactive researchers are more likely to make discoveries as a result of finding new data sources and examining new situations; while survey researchers and experimentalists are more likely to make discoveries through innova- tions in techniques of study design, sampling, or data analysis, which can generate unexpected (serendipitous) findings by more precise tests of hypotheses.
42 FOUNDATIONS OF MULTIMETHOD RESEARCH
Different research styles thus exert different constraints on formulating problems: open-ended constraints in response to the immediate research situation for fieldwork and nonreactive research or more programmed constraints for surveys and experiments. The multimethod strategy pro- vides the opportunity to overcome these methodological constraints upon problem formulation and thereby gain the advantages of each approach while compensating for its disadvantages. Sieber (1973), for example, notes Stinchcombe’s (1964) reliance upon about six months of fieldwork among the teachers and administrators in a high school to formulate the hypotheses that guided Stinchcombe’s analy- sis of survey data from the same school. Sieber (1973) concludes that “an optimal schedule for theoretical survey research would include a lengthy period of fieldwork prior to the survey” (p. 1346). He further observes that although he could find in the literature few other examples of this practice of deriving a survey’s guiding theory from fieldwork, it may be quite common, since “Often, only passing acknowledgment is made of prior personal familiarity with the situation, a familiarity that has pro- duced rather definite ideas for research (p. 1345). Sieber (1973) cites, for instance, Lipset’s (1964) autobiographical account of how the childhood experience of his father’s membership in the International Typographical Union, along with the classic works of Robert Michels and Alexis de Tocqueville, influenced the research problem that Lipset and his col- leagues formulated and tested in the classic survey study, Union Democracy (1956). If, as Dewey suggested, the correct formulation of research prob- lems is crucial to their solution, then it is critical that no source of poten- tially valid information—no matter how “unscientific” it may seem—be ignored. Furthermore, Sieber (1973) demonstrates how despite “an historical antagonism between proponents of qualitative fieldwork and survey research,” integration between these two research styles has been achieved in numerous studies (p. 1335). He shows how fieldwork has been employed to define the theoretical structure of problems later stud- ied in surveys, to define and gain greater knowledge of the problem- relevant populations for surveys, and to reformulate problems by aiding in the interpretation of surprising survey findings and statistical relation- ships between variables. He likewise shows how surveys have been used to define and pinpoint relevant cases for fieldwork, to verify and establish the generality of field observations, and to cast new light on “hitherto inexplicable or misinterpreted” observations.
Formulating Research Problems 43
Each style of social research can be employed either to generate or to verify theories. But in fact, purely generational studies tend to rely more upon fieldwork or nonreactive data sources than upon experiments or surveys, and often more upon qualitative than upon quantitative obser- vation and analysis. The transition from generational to verificational research may therefore involve a methodological shift as well as a change in the focus of problem formulation. As Chapter 2 suggested, studying a theory with different research methods provides an opportunity for fuller examination of that theory. However, employing a new or different method also creates difficulties. It may be far from obvious how, for instance, concepts and propositions developed through qualitative field studies may be measured and operationalized in terms suitable for quan- titative surveys or experiments—or vice versa, how to design a field study to test a theory deriving from surveys or experiments. There may also be questions about the appropriateness of the new method to the theory’s content, or about whether or not operational hypotheses that can be tested with that method do in fact adequately represent the theory and so provide a fair and full test. Bernstein, Kelly, and Doyle (1977) encountered these kinds of difficul- ties in formulating and testing hypotheses derived from symbolic interac- tionist theories of deviance. These were theories that had been generated largely in qualitative field studies. Bernstein et al.’s strategy was to com- bine qualitative field observation with quantitative analysis of interviews and court records collected for a larger sample of criminal defenders. This multimethod approach, which is an example of the transition study des- cribed in Chapter 2, allowed them to use the fieldwork data to aid in both the design and the interpretation of the survey and archival segment of their study. The approach also permitted them to be open and sensitive to the kinds of firsthand field observations that had prompted the initial theories. They thereby retained descriptive realism without sacrificing either the quantitative precision required for verification or the generaliz- ability provided by their larger sample.
The Empirical Unfolding of Research Problems
Once a study is published, it is in many ways irrelevant whether the research problem prompted the study or instead emerged from it. With publication, the study’s problem enters the public domain and becomes
Formulating Research Problems 45
the responsibility not only of the study’s author but of all who are professionally interested in that research area. At that point, the key issue is what to do with the problem next. Research into a problem does not end with a single study. Nor is there truly a final formulation of a problem any more than there is a final solution. All research, as Chapter 2 suggested, involves some simplification of the problem being investigated. This is unavoidable given the limitations on our resources, theories, and meth- ods. However, each of a discipline’s separate new studies, or each phase of study in an individual’s research program, reveals new aspects of the problem by addressing issues (such as those raised by the “skeptic’s ques- tions” in Chapter 2) that earlier research could not address. The two modes of formulating research problems that we have just dis- cussed differ in that one looks to past studies, while the other looks to ongoing work. But the two are similar in that both rely upon empirical inquiry rather than upon nonempirical procedures, such as speculation or the purely logical analysis of ideas. This means that whether research problems emerge from current research or instead derive from earlier work, research methods are directly implicated in the process. Every empirically based research problem has a methodological as well as a sub- stantive component, and this methodological component may equally influence our perceptions as to which particular phenomena and theories are problematic. One of the central questions to be posed, therefore, is how do the methods employed in research directly affect the formulation of research problems?
The Substantive Importance of Methodology
Deutscher (1966), for example, posed this question of methodological influence by revealing one of the major simplifications of social policy research conducted through the early 1960s. He noted the very heavy reliance upon survey research at that time, and suggested that this reli- ance upon surveys led social scientists to oversimplify research problems by assuming that verbal responses reflect behavioral tendencies. Deutscher observed that only by making this assumption were resear- chers, who were studying issues such as racial and ethnic discrimination, able to make causal inferences about behavior solely on the basis of ques- tionnaire and interview data. However, he stressed that this assumption neglected a central problem that had begun to emerge from exploratory field studies as early as the 1930s: People’s words and deeds frequently do not agree. To correct this oversimplification, Deutscher urged both that
46 FOUNDATIONS OF MULTIMETHOD RESEARCH
greater consistency between subjects’ words and deeds than did observational field studies. When such methodologically linked contradictions appear in the course of a problem’s development, the suspicion is that they may derive from theoretically irrelevant characteristics of the different meth- ods employed rather than from the substantive complexity of the prob- lem. Inconsistent findings require reformulations of research problems. When these inconsistencies reflect unanticipated substantive complexity, then concepts and propositions must be recast to take account of that complexity. But although more complicated theories are sometimes neces- sary to achieve theoretical realism, simplicity is preferable. And if, in fact, contradictory research findings are attributable to methodological influ- ences and can be shown to be consistent with existing theories, once those influences have been taken into account, so much the better. The substance of social life is certainly diverse enough to generate inconsistent findings, but the methods of social research are also diverse. Only by analyzing the methods employed to obtain research findings can it be determined which source of inconsistency any given set of findings reflects. For example, Hovland (1959) observed that textbooks summariz- ing the effects of communication on opinion-change in the 1950s often reported substantive contradictions in research findings without regard to differences in methodology, despite the fact that stronger effects were gen- erally found in experiments than in surveys. However, Hovland found that upon closer inspection these apparent contradictions might be explained in terms of the idiosyncrasies of these two different types of methods and might not require new theoretical explanations. In sum, although the exclusive use of a single type of research method can oversimplify research problems, the use of different types of research methods, without system- atic comparisons of their results and an understanding of possible method- ological influences, can make problems appear to be more complex—or complex in different ways—than they really are.
Research Questions Stemming from Multimethod Research
Multimethod research can help in sorting out substantive and method- ological issues. But not even this approach can provide totally method- free results. No research style can do that; what we know is always shaped in part by how we came to know it. Multimethod studies may be expected, therefore, to spawn as well as to aid in answering research questions. A major problem is how to proceed with inquiry once it has been discovered
48 FOUNDATIONS OF MULTIMETHOD RESEARCH
that two or more methods’ findings diverge. As we said in Chapter 1, convergent results from different methods increase confidence in each method, but contradictory results call for reanalysis of the methods, both in relation to one another and in relation to the original research problem. (We shall see in Chapters 6 and 7 that under some circumstances conver- gent findings also raise questions, but it is best to examine one source of problems at a time.) When contrasting different methods’ results, there are two general classes of potential research questions that emerge in particular. The first is whether quite different styles of research really study the same phe- nomenon in anything but name. The second is whether different variants of the same research style will yield the same results. Let us consider the types of potential research problems that emerge from multimethod research in connection with these two issues. In Chapter 1, we saw that crime data from official statistics and from criminal victimization surveys might measure quite different aspects of criminal deviance. Hindelang, Hirschi, and Weis (1979) have considered still another survey method of measuring crime (the self-report method), which has also often been found to give results different from those obtained with official statistics. They find that many of the apparent dis- crepancies may stem from a failure to recognize that these two methods may tap quite different domains of crime (trivial versus major crimes), which may have quite different social correlates. The self-report method of measuring crime, most commonly used to measure juvenile delinquency, calls upon respondents to report their own offenses rather than offenses committed against them, as in criminal vic- timization surveys. The method is essentially a technique for estimating delinquency in nondelinquent populations; that is, among juveniles not officially labeled as deviant by arrest or conviction. It provides data to study possible correlates of self-reported delinquent behavior, irrespec- tive of whether or not that behavior was previously identified in official data sources, which are often suspected of measuring official action more nearly than deviance. With self-report data it is possible to estimate how (if at all) social factors such as gender, race, or social class are related to delinquent behavior, when the possibly contaminating influences of offi- cial detection and recording practices are eliminated. Self-report studies, like criminal victimization surveys, were intended to resolve the crime-measurement problem. But the result was quite differ- ent. Instead of confirming the findings of earlier methods, the self-report
Formulating Research Problems 49
Dunaway, Cullen, Burton, and Evans (2000) have done more research specifically designed to help clarify these issues. They conducted a self- report mail survey of the general population aged 18 and over living in an urban area in the Midwest, inquiring about serious crimes (including vio- lence) as well as lesser offenses. Their study thus focused on adults rather than juveniles and a fuller range of offenses than covered in earlier studies of juvenile delinquency. They describe their effort as follows: “The goal of the current study has been to add an additional piece to solving the class- crime puzzle. In particular, we offer the conclusions that among the general adult population social class appears to be weakly related to involvement in self-reported criminality, but that evidence exists to sug- gest that that this relationship is specified by race and type of crime (vio- lence)” (p. 611). However, they recognize that no single study such as theirs will resolve the empirical issues. And they urge that in “the under researched area of adult crime, we may need carefully designed, sus- tained research on class that uses multiple methods across diverse sam- ples” (p. 611). But they suggest also that the current state of the research, unresolved as many issues may be, presents a theoretical problem that needs to be addressed: “those researchers wishing to construct class- based theories of crime must confront why class position, even if related to serious crime, is only modestly implicated in the causation of less seri- ous offenses” (p. 620). Brannon, Cyphers, Hesse, Hesselbart, Keane, Schuman, Viccaro, and Wright (1973) pick up the problem of “attitude versus action” as it stood in the early 1970s. They note that most studies at that time had reported either negative or mixed relationships between what people say and do. But they also note that these studies had not “concentrated on the valid- ity of typical survey questions in general populations.” They carefully observe that although this failure does not invalidate the evidence from the earlier studies, it does leave us “uncertain of their implications for the validity of standard cross-section attitude surveys” (p. 625). Their remedy was to pose their questions about substance and method jointly and to design a multimethod study in which a typical attitude question on the important social policy issue of “open housing” was embedded in a larger survey. The survey was then followed three months later by a field experi- ment designed to test respondents’ willingness to act in a manner consis- tent with their earlier expressed attitudes. Brannon et al. (1973) report an overall high level of consistency between the survey responses and the later experimental findings. However, they conclude not simply that
Formulating Research Problems 51
“attitudes and actions are consistent” but rather they use their findings as the basis for stating three hypotheses to explain why in this instance atti- tudes and actions were found to coincide. The study by Brannon et al. (1973), like Hindelang et al. (1979), in addi- tion to illustrating how to develop specific multimethod research ques- tions, illustrates an important general point about multimethod research. Generic labels for research methods, such as those that we introduced in Chapter 2, conceal a great deal of species variation among the actual research techniques that compose the style designated by each label (fieldwork, survey research, etc.). For example, a survey may refer either to a questionnaire study of a convenient sample of college students con- ducted by a faculty member in a college classroom or to an interview study conducted by a team of paid interviewers in the households of a cross-sectional sample of a city’s population. (Similarly, Campbell and Stanley [1963] have demonstrated the variety of “experiments.”) Broad classifications of methods are useful for purposes of general discussion, and they are important to the analysis of research when they designate groups of techniques that are thought to have common strengths and weaknesses. But if the labels are used without regard for the underlying variations in techniques, they can easily lead to mistaken inferences. In all cases, it is an empirical question whether the findings from a given form of a method correlate well or poorly with a given form of another method.
The Role of Theory in Problem Formulation
Theory plays a dual role in research. On the one hand, new theories solve research problems by accounting for unexplained phenomena and by superseding questionable older theories. On the other hand, existing theory guides researchers in formulating research problems. In determin- ing whether and in what respects a phenomenon or a theory is problem- atic, researchers consider the context of accumulated theoretical as well as empirical knowledge. And only those phenomena and theories that appear to be problematic when viewed in that context are then studied. Ideally, at least, formulating problems in this manner ensures the orderly advance of knowledge because new research is aimed at solving problems left unsolved in past work rather than being aimed at either totally new or theoretically irrelevant problems. The guiding role of theory in problem formulation is obvious in verifi- cational studies. But while less obvious, it is equally important in exploratory
52 FOUNDATIONS OF MULTIMETHOD RESEARCH
Paradigmatic Pragmatism
It has become increasingly clear that research methods cannot be assumed a priori to be neutral or atheoretical tools. For example, Walton (1966) demonstrated that the different theories of community power held by political scientists and sociologists might well be a consequence of the fact that researchers in these two disciplines have characteristically stud- ied community power with different types of methods. And Perrucci and Pilisuk (1970) have further shown that the method employed to study community power may not only determine which theory one accepts but may also determine which theories one can formulate and test. More generally, Ritzer (1980) has posited systematic links between theoretical styles (or paradigms) and research styles (or methods):
Those who accept the social facts paradigm tend to use questionnaires and/or interviews when they do empirical research.... Those who accept the social definition paradigm tend to use the observation method in their research.... The choice of methods is, of course, made necessary by the nature of the social definition paradigm.... All of the methods discussed in this book could be used by the social behaviorist... [but]... the behaviorist almost invariably uses the experimental method. (pp. 67, 125, 177–178)
However, Platt (1996) in her historical study of sociological methods has cautioned against assuming tight connections between general theo- retical orientations and particular types of method. She concludes that “the link between theory and method has been a loose one. It does not fol- low that it is insignificant, or equally loose in all cases, but it is clear that we need to look elsewhere to explain both the origins of particular meth- ods and the choice of method made in particular projects” (p. 123). To the extent that theoretical styles and research styles are systemati- cally linked, it may be expected that researchers will pose problems that are compatible with their own particular theoretical orientations and with the methods linked to those orientations, and will ignore problems that are either theoretically or methodologically incompatible. Specific theories and methods may be associated for at least two reasons. First, certain methods may be better suited for gathering data on specific types or classes of variables, and these variables may in turn suggest certain types or classes of theoretical concepts and propositions more readily than others. Second, certain theories may contain concepts and propositions that imply types or classes of variables that in turn recommend
54 FOUNDATIONS OF MULTIMETHOD RESEARCH
certain methods as being more appropriate than others. Both of these reasons imply that the form in which research problems are initially posed may be shaped by methodological or theoretical commitments, but neither implies any necessary linkage between or limitation by singular methods and singular theories. And it is also to be expected that researchers working in different theo- retical and methodological styles will frequently disagree about the relative importance of particular research problems and even as to whether particu- lar phenomena or theories are problematic. For example, survey researchers who assume a “common understanding” about questionnaire items are often criticized by phenomenologists (e.g., Cicourel, 1962) who regard such understandings as highly problematic. And human ecologists and demog- raphers who study the relationships between resources and population characteristics are criticized by conflict theorists for ignoring concepts and variables pertaining to political power and governmental structures. Such debates will and should continue until the issues are resolved. However, the pragmatism of employing multiple research methods to study the same general problem by posing different specific questions has some pragmatic implications for social theory. Rather than being wedded to a particular theoretical style, its pet problems and questions, and its most compatible method, one might instead combine methods that would encourage or even require the integration of different theoretical perspec- tives to interpret the data. If hypotheses and variables that have been previously isolated each within their own theoretical systems are instead empirically interrelated in the same study, then conceptual linkages between different theoretical systems are more likely to follow. Sandole’s (1999) multimethod and multiparadigm study using simula- tion-generated, archival, and survey data to investigate violent ethnic conflicts in the post cold war era provides a good example of research that tries “to achieve not only the ‘normal science’ type... of additive cumula- tion , whereby one refines and expands what one already knows, but the potentially ‘extraordinary science ‘ type of additive cumulation... whereby one goes beyond what one already knows, perhaps shifting or combining paradigms in the process”( p. 192).
Mixing Metaphors to Generate Research Problems
“A stitch in time gathers no moss” may make little sense as a homily, but if one struggled to make sense of it, then a new meaning or insight
Formulating Research Problems 55
their solutions, scientists work by example, analogy, or metaphor, applying exemplars from one situation to another:
The resultant ability to see a variety of situations as like each other... is, I think, the main thing a student acquires by doing exemplary problems... After he has completed a certain number, which may vary from one individual to the next, he views the situations that confront him as a scientist in the same gestalt as other members of his specialists’ group. (p. 189)
The legitimacy of the use of exemplars or analogies is ultimately based upon the community of practicing scientists accepting such models. For example, biological analogies have been widely accepted within sociol- ogy, whereas models from physics have not. Social physics was offered by St. Simon as an alternative label to sociology as a name for the discipline, but only a few analogies from physics are to be found in research. One such is Samuel Stouffer’s (1940) gravity model (inverse square law) that relates the amount of geographical mobility between cities to the distance between them. New metaphors and new concepts suggest new variables and new methods—new questions and new data to answer them. As in the parable of the blind men and the elephant, the metaphor used to describe a phe- nomenon (it is like a snake said the man holding the trunk, like a tree said the one holding a leg, etc.) depends partially upon the aspect of reality one happens to get hold of. But metaphorical depiction of reality is also determined by the method of observation one uses. Hearing an elephant, one might liken it to a trumpeter swan; or tasting a juicy, rare elephant steak, one might liken it to a cow. In short, metaphors are often (some say always) used to define reality, and metaphors are in part measurement and method specific. Mixing metaphors can suggest new questions requiring new methods, and mixing methods can generate new questions leading to new metaphors.
Formulating Research Problems 57
