Download The relationship between internal and external validity and more Study Guides, Projects, Research Economics in PDF only on Docsity!
Clarissa Busch
The relationship between internal and
external validity
Clarissa Busch∗
Abstract The connection between internal and external validity is a matter of some debate in the philosophy of science. Some authors claim that there is a tradeoff between internal and external validity; some argue that internal validity is a pre- requisite for external validity. This paper presents an attempt at resolving the tension between these two views. The main contention is that the two views refer to different components of internal and external validity. The tradeoff is reframed as a tradeoff between the degree of confidence that a researcher can attain about the internal validity of her result and the de- gree of confidence she can attain regarding its external valid- ity. This interpretation of the tradeoff is compatible with the ∗Thanks to Donal Khosrowi for his supervision of this paper and to Johanna Thoma for her comments on an earlier version.
The relationship between internal and external validity
prerequisite view. An important implication of this reinter- pretation is that the tradeoff relationship places a constraint on experimental design, while the prerequisite relation does not.
1 Introduction
What is the relationship between internal and external validity? This question is a matter of some debate in the philosophy of science literature. There are two co-existing yet apparently contradictory views of the relation between the two. One is that internal validity is a prerequisite for external validity (see e.g. Hogarth 2005). The other is that there is a tradeoff between the two (see e.g. Campbell 1957). Both views are encountered frequently in the literature on the methodology of applied econometrics; at times both views are even supported by the same author (e.g. Guala 2003). While there is widespread agreement that there is some tension between these views (e.g. Persson and Wallin 2015), there have been few explicit attempts at resolving it. One such attempt is by Jimenez-Buedo and Miller (2010), who conclude that researchers are simply mis- taking the difficulty of making predictions about the world based on experiments for a tradeoff between internal and external valid- ity. However, since the tradeoff view is so widespread in the debate about the validity of experimental results, I propose an alternative
The relationship between internal and external validity
factor (or set of factors) X, and X really is a cause of Y in E. Furthermore, it is externally valid if X causes Y not only in E but also in a set of other circumstances of interest F, G, H”
Guala’s definition highlights the importance of the researcher’s knowl- edge of the causality of the relationship between two factors. In fact, his definition seems to suggest that internal validity is a two- part concept, with an epistemic and an ontic component. On this reading, for an experimental result to be internally valid, it must fulfill an epistemic requirement, i.e. the experimenter must identify a certain factor X as a causal factor of Y, and an ontic require- ment, i.e. this factor X must really be a cause of Y. The epistemic component of internal validity is the part that is accessible to the researcher. Thus, for practical research purposes, it is not useful to distinguish between an ontic and an epistemic component of inter- nal validity. Yet, as will become clear in the subsequent discussion, this distinction can help clarify the relation between internal and external validity.
2.2 External validity
Guala’s definition highlights the importance of the researcher’s knowl- edge of the causality of the relationship between two factors. In fact, his definition seems to suggest that internal validity is a two- part concept, with an epistemic and an ontic component. On this
Clarissa Busch
reading, for an experimental result to be internally valid, it must fulfill an epistemic requirement, i.e. the experimenter must identify a certain factor X as a causal factor of Y, and an ontic require- ment, i.e. this factor X must really be a cause of Y. The epistemic component of internal validity is the part that is accessible to the researcher. Thus, for practical research purposes, it is not useful to distinguish between an ontic and an epistemic component of inter- nal validity. Yet, as will become clear in the subsequent discussion, this distinction can help clarify the relation between internal and external validity.
Broad external validity, on the other hand, is the transferability of a result to other contexts in general, not just to a particular “context of interest ”, so-labelled by the researcher. In what follows, I will explain the two types of external validity in more detail and examine whether the epistemic/ontic distinction can be applied to them as well. This will be useful in the subsequent discussion of the relation between the two types of validity.
Narrow external validity is also sometimes referred to as “ecolog- ical validity ” (see e.g. Roe and Just 2009). It refers simply to the applicability of an experimental result to another context that is of interest to the researcher. Narrow external validity can be split up into the same two components as internal validity. The on- tic component concerns the question of whether the experimental result under consideration, e.g. a qualitative causal relationship, ac- tually holds in the context of interest. The epistemic component,
Clarissa Busch
3 The conflict
3.1 The prerequisite view and the tradeoff view
Having introduced these crucial concepts, I will now sketch the two conflicting views of internal and external validity. According to Guala, “there exists a trade-off between the two dimensions of experimental validity. The stronger an experimental design is with respect to one validity issue, the weaker it is likely to be with respect to the other. ” (Guala 2005, 144). Even though Guala’s remarks seem somewhat tentative, the tradeoff between internal and exter- nal validity has come to be known as a “well-known methodological truism ” (Cartwright 2007, 220).
Interestingly, Guala also seems to hold the other common view of the relationship between external and internal validity, which is of- ten seen as conflicting with the tradeoff view. He writes: “Problems of internal validity are usually chronologically and epistemically an- tecedent to problems of external validity: it does not make much sense to ask whether a result is valid outside the experimental cir- cumstances unless we are confident that it does therein ” (Guala 2003, 1198). In other words, on this view, internal validity is (usu- ally) understood to be a prerequisite for external validity. There is a clear tension between the tradeoff and prerequisite views. This is because if there is in fact a tradeoff between internal and external validity, while at the same time there can be no external validity
The relationship between internal and external validity
without internal validity, then both relationships cannot be true in all cases. It could be the case, for example, that an experimental result must be internally valid to some degree for external validity to be achievable. But can internal validity be fulfilled “to some de- gree ” or is it a dichotomous property? In the ensuing discussion, I hope to shed some light on the relation between these two types of validity.
While the conflict between the prerequisite and tradeoff views has been documented in the literature (see e.g. Persson and Wallin 2015 and Jimenez-Buedo and Miller 2010), there have been few attempts to resolve it.
3.2 Dropping the tradeoff
Jimenez-Buedo and Miller (from here on JM) make one such at- tempt, wherein they argue that the two views cannot be supported simultaneously and conclude that the tradeoff view should be aban- doned. Their argument proceeds as follows:
The tradeoff view has two implications at least one of which must hold true if there actually is a tradeoff between external and internal validity. Specifically, if there is a tradeoff relationship, then it must be the case that either
(1) a given experimental setting can be altered such that the re- sults of the experiment become more internally valid at the
The relationship between internal and external validity
(2) must be true. This means that holding both views to be true inevitably leads to a contradiction, either between (1) and (4) or be- tween (2) and (3). JM thus conclude that the prerequisite view and the tradeoff view cannot be held simultaneously and that researchers might simply be mistaking the difficulty of making predictions about real-world phenomena based on experimental results for a tradeoff between internal and external validity.^2
4 The resolution
4.1 Reinterpreting the tradeoff
Since the tradeoff view is nevertheless ubiquitous in the literature on internal and external validity, I propose a different approach to resolving the tension between the tradeoff and prerequisite views. The intuition leading to my approach stems from a question JM pose: “Is internal validity a quality that an experiment either has or not, or can experiments be more or less internally valid? ” (p.7).
The tradeoff view implicitly presupposes that internal and external validity come in degrees. This is because even though there could be a tradeoff relation between the two if they took on binary values, this would mean that researchers would have to opt for one of the (^2) What JM describe seems to be a relationship of simple attenuation accord- ing to Matthewson and Weisberg’s taxonomy of tradeoffs. For details, see Matthewson and Weisberg 2008.
Clarissa Busch
two, completely sacrificing the other, which does not seem to align with the kind of options practitioners face. In the prerequisite view, on the other hand, internal validity might well be a binary concept. If internal validity must be given in order for external validity to be possible, it may well be the case that an experiment either is or is not internally valid. Thus, the question of whether internal validity is scalar or dichotomous promises to illuminate the relation between internal and external validity.
To help resolve this tension, consider again Guala’s definition of the two validity concepts and the distinction introduced above. Given the distinction between ontic and epistemic internal validity, it be- comes clear that internal validity may in fact be part dichotomous and part scalar. This would mean that there is no genuine conflict between the tradeoff and prerequisite views. To see this, consider whether ontic and epistemic internal and narrow external validity are dichotomous or scalar properties. Ontic internal validity is a dichotomous property. It either is or isn’t the case that there is a causal relationship between two variables. On the other hand, the fulfillment of the epistemic condition of internal validity is a mat- ter of degree, specifically degrees of belief or credences. To make this claim more plausible, consider the definition of internal validity. The epistemic condition is fulfilled if “the experimenter attributes the production of an effect Y to a factor X. ” What does it mean for the experimenter to “attribute ” an effect to a particular cause? To attribute the effect Y to a given cause X, the experimenter must be confident that X is the cause of Y. Confidence is a matter of degree.
Clarissa Busch
lidity could be a dichotomous concept. This suggests that the pre- requisite view might be compatible with the ontic components of internal and external validity. These components however, focus on a different set of issues. They ask whether there really is a causal relation between the variables under consideration and whether that same causal relation also holds in the context of interest.
This reinterpretation of the tradeoff can resolve the debate about the relationship between internal and external validity because it makes the tradeoff and prerequisite views compatible: It is entirely possible for the epistemic components of internal and narrow external valid- ity to be in a tradeoff relation while the ontic components of internal and narrow external validity are in a prerequisite relation. On this interpretation, researchers face a tradeoff between epistemic access to the narrow external validity of a result and epistemic access to the internal validity of a result. This means that the more they change their design to gain confidence that the observed relation between the experimental variables is actually causal, the more difficult it be- comes to extrapolate that result to another context under the (thus not easily verifiable) assumption that it holds in the experimental context.
At the same time, ontic internal validity must be fulfilled (whether the researcher knows it or not) for ontic external validity to be pos- sible. In other words, the relationship between the variables in the experimental context must be causal for the result to be applicable to the context of interest.
The relationship between internal and external validity
To illustrate this reading of the tradeoff, I will apply it to a classic instance of the validity tradeoff, to wit, the tradeoff researchers face when choosing a research design.
4.2 The epistemic tradeoff in “Tradeoffs between Exper- iments, Field Experiments, Natural Experiments and Field Data”
In their 2009 paper “Internal and External Validity in Economics Re- search: Tradeoffs between Experiments, Field Experiments, Natural Experiments and Field Data, ” Brian Roe and David Just describe the tradeoff between different kinds of study designs. They differen- tiate between four types of studies and order them along a spectrum according to the “degree of verifiable exogenous variation within the economic context that produces the data ” (p. 4). According to them, observational studies have the lowest internal and highest ex- ternal validity, followed by natural experiments, which have higher internal validity but somewhat lower external validity. Field ex- periments have still higher internal validity because they allow for manipulation by the researcher, which also further decreases exter- nal validity. Laboratory experiments have very high internal validity but are weak as far as external validity is concerned. There is thus a tradeoff between external and internal validity when choosing be- tween these four types of designs.
The tradeoff they describe can be explained in terms of the tradeoff
The relationship between internal and external validity
researcher can manipulate the experimental setup to gain epistemic access to the internal validity of the result. The epistemic compo- nent of internal validity is thus easily fulfilled in lab experiments. Overall, while the internal validity of the results of laboratory ex- periments is highly epistemically accessible to the researcher, the narrow external validity of the results is not and must be argued for using additional information beyond the study’s findings.
On Roe and Just’s validity spectrum, field experiments and natural experiments are located between lab experiments and observational studies. Field experiments are closest to lab experiments as far as validity is concerned. They are fairly strong on internal validity. Because the researcher can manipulate the experimental setup to some degree, she gains epistemic access to the internal validity of the results. At the same time, this very manipulation also limits the researcher’s epistemic access to the narrow external validity of the results. So while the epistemic component of narrow external validity can be fulfilled to some extent because the experiment is conducted within the context of interest, it cannot be completely fulfilled because the researcher manipulates that context to conduct the experiment, thereby making the experimental context poten- tially relevantly dissimilar from the (non-manipulated) context of interest.
Natural experiments are closer to observational studies as far as va- lidity is concerned. This is because they are set within the context of interest and involve no manipulation of the experimental context
Clarissa Busch
on the part of the researcher. This means that the researcher has high epistemic access to the narrow external validity of her results, i.e. she can be fairly confident that the observed effect holds in the context of interest. On the other hand, due to the lack of manipula- bility, the researcher also cannot be fully confident that the observed effect is really causal. The epistemic component of internal validity is thus not so easily fulfilled.
As the preceding description of the validity tradeoff between different types of studies suggests, the epistemic interpretation of the tradeoff between internal and external validity is applicable to this classic instance of the tradeoff.
5 Conclusion
My aim was to clarify the relationship between the internal and ex- ternal validity of experimental results. In doing so, I first examined the concepts of internal and external validity in detail. From this analysis I concluded that internal validity and narrow external va- lidity might be two-part concepts, each comprising an epistemic and an ontic component. This distinction, proves helpful in illuminating the relationship between the two types of validity. It inspired my reinterpretation of the tradeoff and prerequisite views as referring to different sub-concepts of internal and external validity. On this reading the ontic component of internal validity is a prerequisite for the ontic component of external validity, while at the same time,
Clarissa Busch
the relationship between internal and external validity is therefore the validity tradeoff and not the prerequisite relation.
The relationship between internal and external validity
References
[1] Guala, F., 2005. External Validity. In: The methodology of ex- perimental economics. Cambridge University Press, pp. 141-160.
[2] Jimenez-Buedo, M. and Miller, L.M., 2010. Why a trade-off? The relationship between the external and internal validity of experiments. Theoria. Revista de Teor´ıa, Historia y Fundamentos de la Ciencia, 25(3), pp.301-321.
[3] Hogarth, R.M., 2005. The challenge of representative design in psychology and economics. Journal of Economic Methodology, 12(2), pp.253-263.
[4] Campbell, D.T., 1957. Factors relevant to the validity of experi- ments in social settings. Psychological bulletin, 54(4), p.297.
[5] Guala, F., 2003. Experimental localism and external validity. Philosophy of science, 70(5), pp.1195-1205.
[6] Persson, J. and Wallin, A., 2015. The (misconceived) distinc- tion between internal and external validity. Against boredom: 17 essays on ignorance, values, creativity, metaphysics, decision- making, truth, preference, art, processes, Ramsey, ethics, ratio- nality, validity, human ills, science, and eternal life to Nils-Eric Sahlin on the occasion of his 60th birthday, pp.187-195.
[7] Cartwright, N., 2007. Hunting causes and using them: ap- proaches in philosophy and economics. Cambridge University Press.
