Download Testing: A Roadmap for Efficient Software Quality Assurance and more Exams Computer Science in PDF only on Docsity!
Testing: A Roadmap
Mary Jean Harrold
Key Research Pointers
Development of techniques and tools that will help component users integrate and test the components with their applications more efficiently and effectively Creation of techniques and tools that can use precode artifacts, such as architectural specifications, for planning and implementing testing activities. Development of techniques and tools for use in estimating, predicting, and performing testing on evolving software systems. Establishment of effective processes for analyzing and testing software systems. Investigation of methods that use testing artifacts to assist in software development.
The Author
Mary Jean Harrold received the BS and MA degrees in mathematics from Marshall University and the MS and PhD degrees in computer science from the University of Pittsburgh. She is currently an associate professor in the College of Computing at Georgia Institute of Technology. Her research interests include the development of efficient techniques and tools that will automate, or partially automate, development, testing, and maintenance tasks. Her research to date has involved program-analysis- based software engineering, with an emphasis on regression testing, analysis and testing of imperative and object-oriented software, and development of software tools. Her recent research has focused on the investigation of the scalability issues of these techniques, through algorithm development and empirical evaluation. She is a recipient of the National Science Foundation's National Young Investigator Award. Dr. Harrold serves on the editorial board of IEEE Transactions on Software engineering. She is serving as the program chair for the ACM International Symposium on Software Testing and Analysis (July 2000) and the program co-chair of the 23rd International Conference on Software Engineering (May 2001). She is a member of the Computing Research Association's Committee on the Status of Women in Computing, and she directs the committee's Distributed Mentor Project. She is a member of the IEEEComputer Society and the ACM.
Testing: A Roadmap
M a r y J e a n Harrold
C o l l e g e o f C o m p u t i n g G e o r g i a I n s t i t u t e o f T e c h n o l o g y 801 A t l a n t i c D r i v e A t l a n t a , G A 30332- h a r r o l d @ c c. g a t e c h. e d u
ABSTRACT
Testing is an important process that is performed to support quality assurance. Testing activities support quality assurance by gathering information about the nature of the software being studied. These activities consist of designing test cases, executing the software with those test cases, and examining the results pro- duced by those executions. Studies indicate that more than fifty percent of the cost of software development is devoted to testing, with the percentage for testing crit- ical software being even higher. As software becomes more pervasive and is used more often to perform crit- ical tasks, it will be required to be of higher quality. Unless we can find efficient ways to perform effective testing, the percentage of development costs devoted to testing will increase significantly. This report briefly as- sesses the state of the art in software testing, outlines some future directions in software testing, and gives some pointers to software testing resources. 1 I N T R O D U C T I O N A report by the Workshop on Strategic Directions in Software Quality posits that software quality will be- come the dominant success criterion in the software in- dustry [36]. If this occurs, the practitioner's use of pro- cesses that support software quality assurance will be- come increasingly important. One process that is per-
formed to support quality assurance is testing. Test-
ing activities support quality assurance by executing the software being studied to gather information about the nature of that software. The software is executed with
input data, or test cases, and the output data is ob-
served. The output data produced by the execution of the program with a particular test case provides a spec- ification of the actual program behavior [36]. Studies indicate that testing consumes more than fifty percent of the cost of software development. This percentage is even higher for critical software, such as that used
Permission to make digital or hard copies of all or part of this work tbr personal or classroom use is granted without lee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post oll servers or to redistribute to lists, requires prior specific permission and/or a li~e. Future of Sofware En~neering Limerick Ireland CopyrightACM 2000 1-58113-253-0/00/6...$5.
for avionics systems. As software becomes more perva- sive and is used more often to perform critical tasks, it will be required to be of higher quality. Unless we can find more efficient ways to perform effective testing, the percentage of development costs devoted to testing will increase significantly. Because testing requires the execution of the software,
it is often called dynamic analysis. A form of verifi-
cation that does not require execution of the software,
such as model checking, is called static analysis. As a
form of verification, testing has several advantages over static-analysis techniques. One advantage of testing is the relative ease with which many of the testing ac- tivities can be performed. Test-case requirements 1 can be generated from various forms of the software, such as its implementation. Often, these test-case require- ments can be generated automatically. Software can be instrumented so that it reports information about the executions with the test cases. This information can be used to measure how well the test cases satisfy the test- case requirements. Output from the executions can be compared with expected results to identify those test cases on which the software failed. A second advantage of testing is that the software being developed can be executed in its expected environment. The results of these executions with the test cases provide confidence that the software will perform as intended. A third ad- vantage of testing is that much of the process can be automated. With this automation, the test cases can be reused for testing as the software evolves. Although, as a form of verification, testing has a num- ber of advantages, it also has a number of limitations. Testing cannot show the absence of faults - - it can show only their presence. Additionally, testing cannot show that the software has certain qualities. Moreover, test execution results for specific test cases cannot usually be generalized. Despite these limitations, testing is widely used in practice to provide confidence in the quality of software. The emergence of new technologies, such as
1 Test-case requirements a r e t h o s e a s p e c t s o f t h e s o f t w a r e t h a t a r e t o b e t e s t e d a c c o r d i n g t o t h e t e s t plan; examples a r e s o f t w a r e requirements, source-code s t a t e m e n t s , a n d module interfaces.
Testing Component-Based Systems
The increased size and complexity of software systems has led to the current focus on developing distributed applications that are constructed from component- based systems. A component-based system is com- posed primarily of components: modules that encap- sulate both data and functionality and are configurable through parameters at run-time [29]. Given the increas- ing incidence of component-based systems, we require efficient, effective ways to test these systems. We can view the issues that arise in the testing of component-based systems from two perspectives: the component-provider and the component-user. The component-provider perspective addresses testing issues that are of interest to the provider (i.e., developer) of the software components. The component provider views the components independently of the context in which the components are used. The provider must therefore, effectively test all configurations of the components in a context-independent manner. The component-user per- spective, in contrast, addresses testing issues that con- cern the user (i.e., application developer) of software components. The component user views the compo- nents as context-dependent units because the compo- nent user's application provides the context in which the components are used. The user is thus concerned only with those configurations or aspects of the behavior of the components that are relevant to the component user's application. One factor that distinguishes issues that are pertinent in the two perspectives is the availability of the compo- nent's source code: the component providers have ac- cess to the source code, whereas the component users typically do not. One type of software for which the source code is usually not available is commercial off- the-shelf software (COTS). Although there are no regu- lations imposed on developers of COTS, to standardize development and reduce costs, many critical applica- tions are requiring the use of these systems [32]. The lack of availability of the source code of the components limits the testing that the component user can perform. Researchers have extended existing testing techniques for use by component providers. For example, Doong and Frankl describe techniques based on algebraic spec- ifications [14], Murphy and colleagues describe their experiences with cluster and class testing [34], and Kung and colleagues present techniques based on object states [26]. Other researchers have extended code-based approaches for use by component providers for test- ing individual components. For example, Harrold and Rothermel present a method that computes definition- use pairs for use in class testing [22]. These definition- use pairs can be contained entirely in one method or can consist of a definition in one method that reaches a use
in another method. Buy and colleagues present a simi- lar approach that uses symbolic evaluation to generate sequences of method calls that will cause the definition- use pairs to be executed [6]. Researchers have considered ways that component users can test systems that are constructed from compo- nents. Rosenblum proposes a theory for test adequacy of component-based software [45]. His work extends Weyuker's set of axioms that formalize the notion of test adequacy [52], and provides a way to test the compo- nent from each subdomain in the program that uses it. Devanbu and Stubblebine present an approach that uses cryptographic techniques to help component users ver- ify coverage of components without requiring the com- ponent developer to disclose intellectual property [13]. With additional research in these areas, we can expect efficient techniques and tools that will help component users test their applications more effectively. We need to understand and develop effective techniques for test- ing various aspects of the components, including secu- rity, dependability, and safety; these qualities are espe- cially important given the explosion of web-based sys- tems. These techniques can provide information about the testing that will increase the confidence of develop- ers who use the components in their applications. We need to identify the types of testing information about a component that a component user needs for testing applications that use the component. For exam- ple, a developer may want to measure coverage of the parts of the component that her application uses. To do this, the component must be able to react to inputs provided by the application, and record the coverage provided by those inputs. For another example, a com- ponent user may want to test only the integration of the component with her application. To do this, the com- ponent user must be able to identify couplings between her application and the component. We need to develop techniques for representing and computing the types of testing information that a com- ponent user needs. Existing component standards, such as COM and JavaBeans, supply information about a component that is packaged with the component. Like- wise, standards for representing testing information about a component, along with efficient techniques for computing and storing this information, could be de- veloped. For example, coverage information for use in code-based testing or coupling information for use in in- tegration testing could be stored with the component; or techniques for generating the information could be developed by the component provider and made acces- sible through the component interface. Finally, we need to develop techniques that use the in- formation provided with the component for testing the
application. These techniques will enable the compo- nent user to effectively and efficiently test her applica- tion with the component.
Testing Based On Precode Artifacts
Testing techniques can be based on precode artifacts, such as design, requirements, and architecture specifi- cations. In the past, many of these techniques have been based on informal specifications. Recently, how- ever, more formal approaches have been used for these specifications. Techniques that use these precode spec- ifications for tasks such as test-case planning and de- velopment can help improve the overall testing process. This section discusses the use of one type of precode artifact - - the software's architecture - - for testing.
The increased size and complexity of software systems has led to the emergence of the discipline of software architecture. Software architecture involves the descrip- tion of elements from which systems are built, interac- tions among those elements, patterns that guide their composition, and constraints on these patterns [50]. Software-architecture styles define families of systems in terms of patterns of structural organization. Given the increasing size and complexity of software systems, techniques are needed to evaluate the qualities of sys- tems early in their development. Through its abstrac- tions, software architecture provides a promising way to manage large systems. The emerging formal notations for software architec- ture specification can provide a basis on which effec- tive testing techniques can be developed. Recently, re- searchers have begun to investigate ways to use these formal architectural specifications in such a way. For example, Eickelmann and Richardson consider the ways in which architectural specification can be used to assess the testability of a software system [15]; Bertolino and colleagues consider the ways in which the architectural specification can be used in integration and unit test- ing [5]; Harrold presents approaches for using software architecture specification for effective regression testing [19]; and Richardson, Stafford, and Wolf present a com- prehensive architecture-based approach to testing that includes architecture-based coverage criteria, architec- tural testability, and architecture slicing [42]. These architecture-based testing techniques and tools can fa- cilitate dynamic analysis, and thus, detection of errors, much earlier in the development process than is cur- rently possible. To expedite research in this area, in 1998, the Italian National Research Council and the U. S. National Sci- ence Foundation sponsored the Workshop on the Role of Software Architecture in Testing and Analysis [43]. This workshop brought together researchers in software architecture, testing, and analysis to discuss research
directions. A report on the results of this workshop can be found at http://www.ics.uci.edu/-~djr/rosatea. Additional research in this area promises to provide sig- nificant savings in software testing. We need to develop techniques that can be used with the architectural speci- fication for test-case development. These techniques can provide test-case requirements for assessing various as- pects of the architecture. This approach will let various aspects of the system be assessed early in development. These techniques can also provide functional test-case requirements that can be used to develop test cases for use in testing the implementation. These techniques will facilitate the systematic development of test cases early in the development process. Finally, these techniques can provide ways for test cases to be generated auto- matically. These techniques will enable efficient gen- eration of test cases at an early stage of the software development. We also need to develop techniques that can be used to evaluate software architectures for their testability. With this information, developers can consider alterna- tive designs and select the one that suits their testability requirements.
Testing Evolving Software.
Regression testing, which attempts to validate modi- fied software and ensure that no new errors are intro- duced into previously tested code, is used extensively during software development and maintenance. Regres- sion testing is used to test software that is being devel- oped under constant evolution as the market or tech- nology changes, to test new or modified components of a system, and to test new members in a family of sim- ilar products. Despite efforts to reduce its cost, regres- sion testing remain one of the most expensive activities performed during a software system's lifetime: studies indicate that regression testing can account for as much as one-third of the total cost of a software systems [28]. Because regression testing is expensive, but important, researchers have focused on ways to make it more effi- cient and effective. Research on regression testing spans a wide variety of topics. Chen and colleagues [7], Os- trand and Weyuker [37], and Rothermel and Harrold [48] developed techniques that, given an existing test suite and information about a previous testing, select a subset of the test suite for use in testing the modified software. 2 Harrold, Gupta, and Sofia [20] and Wong and colleagues [53] present techniques to help manage the growth in size of a test suite. Leung and White [28] and Rosenblum and Weyuker [44] present techniques to assess regression testability. These techniques per- mit estimation, prior to regression test selection, of the
2Rothermel and Harrold present comprehensive comparison of regression-test selection techniques [46].
plete coverage suffice for data-flow and other testing cri- teria.
Establishing Effective Processes For Testing
An important aspect of testing is the process that we use for planning and implementing it. Beizer describes a process for testing [4]. Such a process typically consists of construction of a test plan during the requirements- gathering phase and implementation of the test plan after the software-implementation phase. To develop its software, Microsoft, Inc. uses a different model that (1) frequently synchronizes what people are doing and (2) periodically stabilizes the product in increments as a project proceeds. These activities are done continu- ally throughout the project. An important part of the model builds and tests a version of the software each night [9]. Richardson and colleagues advocate the idea of a perpetual testing process. 3 Their perpetual testing project is building the foundation for treating analysis and testing as on-going activities to improve quality. Perpetual testing is necessarily incremental and is per- formed in response to, or' in anticipation of, changes in software artifacts or associated information.
A process for regression testing is implicit in selective regression testing techniques [7, 37, 48, 53]. For these techniques to be employed, testing must be performed on one version of the software, and testing artifacts, such as input-output pairs and coverage information, must be gathered. These artifacts are used by the tech- niques to select test cases for use in testing the next version of the software. Onoma and colleagues present an explicit process for regression testing that integrates many key testing techniques into the development and maintenance of evolving software [35]. This process con- siders all aspects of development and maintenance. Additional research can validate these existing models. For example, does a nightly build and test, such as that performed by Microsoft, reduce the testing that is re- quired later? For another example, how often do testing artifacts need to be computed for effective regression- test selection? Additional research can also develop new process models for testing and validate these models. Although testing is important for assessing software qualities, it cannot show that the software possesses cer- tain qualities, and the results obtained from the testing often cannot be generalized. A process for developing high-quality software, however, could combine testing with other quality tools. Osterweil and colleagues [36] suggest that various quality techniques and tools could be integrated to provide value considerably beyond what the separate technologies can provide. We need to understand the way in which these various 3 M o r e i n f o r m a t i o n can be f o u n d a t the P e r p e t u a l T e s t i n g h o m e page: http:/ /www.ics.uci.edu/~djr/edcs/PerpTest.html.
testing and analysis techniques are related, and develop process models that incorporate them. A process that combines static analysis techniques with testing has the potential to improve quality and reduce costs.
Using Testing Artifacts
The process of testing produces many artifacts. Arti- facts from the testing include the execution traces of the software's execution with test cases. These execu- tion traces may include information about which state- ments were executed, which paths in the program were executed, or which values particular variables got during the execution. Artifacts from the testing also include re- suits of the test-case execution, such as whether a test ease passed or failed. These artifacts can be stored for use in retesting the software after it is modified. Given the magnitude and complexity of these artifacts, they can also be useful for other testing and software en- gineering tasks. Researchers have begun to investigate new ways to use these artifacts. Many techniques have been developed that use execution traces. Pan, DeMillo, and Spafford present a technique that uses dynamic pro- gram slices, 4 which are derived from execution traces, along with the pass/fail results for the executions, to identify potential faulty code [38]. They apply a num- ber of heuristics, which consider various combinations of the intersections and unions of the dynamic slices for the subset of the test suite that passed and the subset of the test suite that failed. In empirical studies on small subjects, the results of applying the heuristics helped to localize the faulty code. Ernst and colleagues present another technique that uses execution traces that contain values, at each pro- gram point, for each variable under consideration [16]. The goal of their approach is to identify program in- variants. After repeated execution of the program with many test cases, the approach provides a list of likely invariants in the program. Their empirical results show that this approach can be quite successful in identifying these invariants. These dynamically inferred invariants can be used in many applications. For example, they m a y assist in test-case generation or test-suite valida- tion. Researchers have also developed techniques that use coverage information for software engineering tasks. Rosenblum and Weyuker [44] present a technique that uses coverage information to predict the magnitude of regression testing. Their technique predicts, on average, the percentage of the test suite that must be retested af- ter changes are made to a program. Later work by Har-
4 A dynamic program slice f o r a p r o g r a m p o i n t , a v a r i a b l e , a n d a t e s t c a s e is t h e set of all s t a t e m e n t s in t h e p r o g r a m t h a t af- f e c t e d ( e i t h e r d i r e c t l y o r i n d i r e c t l y ) t h e v a l u e of t h e v a r i a b l e a t t h e p r o g r a m p o i n t w h e n t h e p r o g r a m is r u n w i t h t h e t e s t c a s e.
rold and colleagues provided additional evaluation of the work, and presented an improved model of prediction [21]. A number of researchers have developed techniques based on coverage information to select test cases from a test suite for use in regression testing [7, 37, 48, 53]. Several researchers have used testing artifacts for test- suite reduction and prioritization [20, 47, 53]. Empirical studies indicate that these techniques can be effective in reducing the time required for regression testing. Ball presented a technique that performs concept analysis on coverage information to compute relationships among executed entities in the program [2]. Comparing these dynamic relationships with their static counterparts can help testers uncover properties of their test suite. Reps and colleagues present a technique that compares path spectra 5 from different runs of a program [41]. Path spectra differences can be used to identify paths in the program along which control diverges in the two runs, and this information can be used to assist in de- bugging, testing, and maintenance tasks. Results of em- pirical studies using various types of spectra performed by Harrold and Rothermel suggest that spectra based on less expensive profiling, such branch spectra, can be as effective, in terms of their ability to distinguish program behavior, to spectra based on more expensive profiling, such as path spectra [23]. Other researchers have provided visualization tech- niques for testing artifacts. For example, Ball and Eick present a system for visualizing information, ;ncluding testing information such as coverage, for large programs [3], and Telcordia Technologies has several tools that combine analysis and visualization of testing artifacts to help software maintainers [24]. Although there have been some successes in using test- ing artifacts for software engineering tasks, this research is in its infancy. Additional research can verify that ex- isting techniques provide useful information for software engineers. For example, we can determine whether the heuristics developed by Pan and colleagues [38] help to identify faulty code when there are many faults or in- teracting faults in a program. These results can provide a starting point for other research. Additional research in this area can also provide new techniques that use testing artifacts for software engi- neering tasks. We need to identify the types of infor- mation that software engineers and managers require at various phases of the software's development. We also need techniques that will find important relation- ships that exist in the software. Techniques such as data mining may help with this task. Given these types of information, we need to develop techniques to present
5 A path spectrum is a d i s t r i b u t i o n o f p a t h s d e r i v e d f r o m a n e x e c u t i o n o f a p r o g r a m.
the information in a useful way. Techniques for effec- tive visualization of the testing information can provide effective tools for software engineers.
Other Testing Techniques
In addition to the areas for fundamental research dis- cussed in the preceding sections, there are many other areas in which techniques could help us reach our des- tination. This section briefly presents a few of them. Generating test data (inputs for test cases) is often a labor-intensive process. To date, a number of techniques have been presented that generate test data automati- cally. Most of these techniques, however, are applicable for unit testing, and may not scale to large systems. We need to develop automatic or semi-automatic test- data generation techniques that testers can use for large systems. These data could be generated using precode representations or using the code itself. Many testing techniques require some type of static analysis information. For example, data-flow analysis is useful for data-flow testing of software units and for in- tegration testing when these units are combined. How- ever, existing techniques for computing precise data- flow information are prohibitively expensive. We need to develop scalable analysis techniques that can be used to compute the required information. Existing techniques for measuring adequacy for rigorous testing criteria, such as data-flow, require expensive or intrusive instrumentation. For example, care must be taken when inserting probes into a real-time system to ensure that the probes do not cause the program to behave differently than it does without the probes. If we expect to use these more rigorous criteria, we need efficient instrumenting and recording techniques. M e t h o d s a n d Tools Ultimately, we want to develop efficient methods and tools that can be used by practitioners to test their software. Pfleeger presented reasons why software en- gineering technology requires, on average 18 years to be transfered into practice [39]. Researchers must work with industry to reduce this time for technology trans- fer. She also presented a comprehensive approach to effecting that transfer. One important aspect of this approach for technology transfer is the development of methods and tools that can be used in industrial set- tings to demonstrate the effectiveness of the techniques we create. We must develop methods and tools that im- plement the techniques and that can be used to demon- strate their effectiveness. To accomplish this, an important criterion is that these methods and tools be scalable to large systems. In- dustrial systems are large and complex, and the meth- ods and tools must function on these systems. Scalable
4 C O N C L U D I N G R E M A R K S Historically, testing has been widely used as a way to help engineers develop high-quality systems. However, pressure to produce higher-quality software at lower cost is increasing. Existing techniques used in practice are not sufficient for this purpose. W i t h f u n d a m e n t a l re- search t h a t addresses the challenging problems, devel- o p m e n t of m e t h o d s and tools, and empirical studies, we can expect significant i m p r o v e m e n t in the way we test software. Researchers will d e m o n s t r a t e the effec- tiveness of m a n y existing techniques for large i n d u s t r i a l software, thus facilitating transfer of these techniques to practice. The successful use of these techniques in in- dustrial software development will validate the results of the research and drive future research. T h e pervasive use of software and the increased cost of validating it will m o t i v a t e the creation of p a r t n e r s h i p s between in- d u s t r y and researchers to develop new techniques and facilitate their transfer to practice. Development of effi- cient testing techniques and tools t h a t will assist in the creation of high-quality software will become one of the most i m p o r t a n t research areas in the near future. 5 A C K N O W L E D G M E N T S The author is s u p p o r t e d by N S F under NYI A w a r d CCR-9696157 and ESS A w a r d CCR-9707792 to Ohio S t a t e University and by a grant from Boeing C o m m e r - cial Airplanes. A n t h o n y Finkelstein and J a m e s A. Jones m a d e m a n y helpful c o m m e n t s t h a t i m p r o v e d the presen- t a t i o n of the paper.
R E F E R E N C E S [1] D. C. Atkinson and W. G. Griswold. The design of whole-program analysis tools. In Proceedings of the 18th International Conference on Software Engineering, pages 16-27, March 1996. [2] T. Ball. The concept of dynamic analysis. In Proceed- ings of the Joint Seventh European Software Engineer- ing Conference (ESEC) and Seventh A C M SIGSOFT International Symposium on the Foundations of Soft- ware Engineering, September 1999. [3] T. Ball and S. G. Eick. Software visualization in the large. Computer, 29(4):33-43, April 1996. [4] B. ]3eizer. Software Testing Techniques. Van Nostrand Reinhold, New York, NY, 1990. [5] A. Bertolino, P. Inverardi, H. Muccini, and A. Rosetti. An approach to integration testing based on architec- tural descriptions. In Proceedings of the IEEE ICECCS- 97. [6] U. Buy, A. Orso, and Pezz~. Issues in testing dis- tributed component-based systems. In Proceedings of the First International Workshop on Testing Dis- tributed Component-Based Systems, May 1999. [7] Y. F. Chert, D. S. Rosenblum, and K. P. Vo. TestTube: A system for selective regression testing. In Proceed- ings of the 16th International Conference on Software Engineering, pages 211-222, May 1994.
[8] J. J. Chilenski and S. P. Miller. Applicability of mod- ified condition/decision coverage to software testing. Software Engineering Journal, 9(5):191-200. [9] M. A. Cusamano and R. Selby. How Microsoft builds software. Communications of the ACM, 40(6):53-61, June 1997. [10] R. A. DeMillo. Test adequacy and program mutation. In Proceedings of the 11th International on Software Engineering, pages 355-356, may 1989. [11] P. Devanbu. GENOA - A customizable, front-end retargetable source code analysis framework. A CM Transactions on Software Engineering and Methodol- ogy, 9(2):177-212, April 1999. [12] P. Devanbu and S. Stubblebine. Software engineering for security: A roadmap. In A. Finkelstein, editor, The Future of Software Engineering. ACM Press, New York,
[13] P. Devanbu and S. Stubblebine. Cryptographic verifi- cation of test coverage claims. IEEE Transactions on Software Engineering, in press. [14] R.-K. Doong and P. G. Frankl. The ASTOOT approach to testing object-oriented programs. A CM Transactions on Software Engineering and Methodology, 3(2):101- 130, April 1994. [15] N. S. Eickelmann and D. J. Richardson. What makes one software architecture more testable than another? In Proceedings of the International Software Architec- ture Symposium, October 1996. [16] M. D. Ernst, J. Cockrell, W. Griswold, and D. Notldn. Dynamically discovering likely program invariants to support program evolution. In Proceedings of the $l st International Conference on Software Engineering, pages 213-224, May 1999. [17] N. Fenton and M. Neil. Software metrics: A roadmap. In A. Finkelstein, editor, The Future of Software Engi- neering. ACM Press, New York, 2000. [18] J. ]3. Goodenough and S. L. Gerhart. Toward a theory of test data selection. IEEE Transactions of Software Engineering, pages 156-173, June 1975. [19] M. J. Haxrold. Architecture-based regression testing of evolving systems. In International Workshop on the Role of Software Architecture in Testing and Analysis, July 1998. [20] M. J. Harrold, R. Gupta, and M. L. Sofia. A method- ology for controlling the size of a test suite. A CM Transactions on Software Engineering and Methodol- ogy, 2(3):270-285, July 1993. [21] M. J. Harrold, D. Rosenblum, G. Rothermel, and E. J. Weyuker. Empirical Studies of a Prediction Model for Regression Test Selection. IEEE Transactions on Soft- ware Engineering, in press. [22] M. J. Harrold and G. Rothermel. Performing dataflow testing on classes. In Proceedings of the Second A CM SIGSOFT Symposium on Foundations of Software En- gineering, pages 154-163, December 1994.
[23] M. J. Harrold, G. Rothermel, R. Wu, and L. Yi. An empirical evaluation of program spectra. In Proceedings of ACM Workshop on Program Analysis for Software Tools and Engineering, pages 83-90, June 1998.
[24] J. R. Horgan. Mining system tests to aid software maintenance. The Telcordia Software Visualization and Analysis Research Team, Telcordia Technologies.
[25] D. Jackson and M. Rinard. Reasoning and analysis: A roadmap. In A. Finkelstein, editor, The Future of Software Engineering. ACM Press, New York, 2000.
[26] D. Kung, N. Suchak, J. Gao, P. Hsia, Y. Toyoshima, and C. Chen. On object state testing. In Proceedings of COMPSAC'94, 1994.
[27] J. W. Laski and B. Korel. A data flow oriented pro- gram testing strategy. IEEE Transactions on Software Engineering, 9(3):347-54, May 1983.
[28] H. K. N. Leung and L. White. Insights Into Regression Testing. In Proceedings of the Conference on Software Maintenance, pages 60~9, October 1989.
[29] T. Lewis. The next 10, 0002 years, part II. IEEE Com- puter, pages 78-86, May 1996.
[30] B. Littlewood and L. Strigini. Software reliability and dependability: A roadmap. In A. Finkelstein, editor, The Future of Software Engineering. ACM Press, New York, 2000.
[31] R. Lutz. Software engineering for safety: A roadmap. In A. Finkelstein, editor, The Future of Software Engi- neering. ACM Press, New York, 2000. [32] G. McGraw and 3. Viega. Why COTS software in- creases security risks. In Proceedings of the First Inter- national Workshop on Testing Distribu ted Component- Based Systems, May 1999. [33] G. Murphy and D. Notldn. Lightweight source model extraction. In Proceedings of the Third A CM SIGSOFT Symposium on the Foundations of Software Engineer- ing, pages 116-127, October 1995. [34] G. Murphy, P. Townsend, and P. Wong. Experiences with cluster and class testing. Communications of the ACM, 37(9):39-47, 1994. [35] K. Onoma, W-T. Tsai, M. Poonawala, and H. Sug- anuma. Regression testing in an industrial environ- ment. Commummications of the ACM, 41(5):81-86, May 1988. [36] L. J. Osterweil ET AL. Strategic directions in software quality. ACM Computing Surveys, (4):738-750, Decem- ber 1996. [37] T. J. Ostrand and E. J. Weyuker. Using datattow analysis for regression testing. In Sixth Annual Pacific Northwest Software Quality Conference, pages 233-247, September 1988. [38] H. Pan, R. DeMillo, and E. H. Spafford. Failure and fault analysis for software debugging. In Proceedings of COMPSAC '97, August 1997. [39] S. L. Pfieeger. Understanding and improving technology transfer in software engineering. Journal of Systems and Software, 47(2-3):111-124, July 1999.
[40] S. Rapps and E. J. Weyuker. Selecting software test data using data flow information. IEEE Transactions on Software Engineering, SE-11(4):367-375, April 1985. [41] T. Reps, T. Ball, M. Das, and J. Larus. The use of program profiling for software maintenance with ap- plications to the year 2000 problem, pages 432-439, September 1997. [42] D. Richardson, J. Stafford, and A. Woff. A formal ap- proach to architecture-based testing. Technical report, University of California, Irvine, 1998. [43] D. J. Richardson, P. Inverardi, and A. Bertolino, edi- tors. Proceedings of the CNR-NSF International Work- shop on the Role of Software Architecture in Testing and Analysis. July 1998. [44] D. Rosenblum and E. J. Weyuker. Using coverage in- formation to predict the cost-effectiveness of regression testing strategies. IEEE Transactions on Software En- gineering, 23(3):146-156, March 1997. [45] D. S. Rosenblum. Adequate testing of component-based software. Technical Report Technical Report UCI-ICS- 97-34, August 1997. [46] G. Rothermel and M. J. Harrold. Analyzing regression test selection techniques. IEEE Transactions on Soft- ware Engineering, 22(8), August 1996. [47] G. Rothermel, M. J. Harrold, J. Ostrin, and C. Hong. An empirical study of the effects of minimization on the fault-detection capabifities of test suites. In Pro- ceedings of the International Conference on Software Maintenance, pages 34-43, November 1998. [48] G. Rothermel and M.J. Harrold. A safe, efficient re- gression test selection technique. ACM Transactions on Software Engineering and Methodology, 6(~):173-210, April 1997. [49] G. Rothermel, L. Li, C. DuPnis, and M. Burnett. What you see is what you test: A methodology for test- ing form-based visual programs. In Proceedings of the 20th International Conference on Software Engineering, pages 198-207, April 1998. [50] M. Shaw and D. Garlan. Software Architecture Perspec- tives on an Emerging Discipline. Prentice Hall, New Jersey, 1996. [51] J. Stafford, D. J. Richardson, and A. L. Wolf. Chaining: A dependence analysis technique for software architec- ture. Technical Report CU-CS-845-97, University of Colorado, September 1.997. [52] E. J. Weyuker. Axiomatizing software test data ad- equacy. IEEE Transactions on Software Engineering, 12(12):1128-1138, December 1986. [53] W. E. Wong, J. R. Horgan, S. London, and H. Agrawal. A study of effective regression testing in practice. In Proceedings of the Eighth International Symposium on Software Reliability Engineering, pages 230-238, November 1997.
