Study with the several resources on Docsity
Earn points by helping other students or get them with a premium plan
Prepare for your exams
Study with the several resources on Docsity
Earn points to download
Earn points by helping other students or get them with a premium plan
Two approaches to test case prioritization and elimination through classification, proposed by srivastava and thiagarajan and cohen et al. The methods aim to increase team efficiency by prioritizing test cases based on code classification and impact, as well as reducing the number of unnecessary tests. The document also compares the greedy algorithm, hill climbing, and simulated annealing approaches.
Typology: Study Guides, Projects, Research
1 / 11
This page cannot be seen from the preview
Don't miss anything!
DCS860B: Data Analysis and Modeling
Submitted in Partial Fulfillment
of the Requirements for the Degree of
Doctor of Professional Studies in Computing
at
Pace University
School of Computer Science and Information Systems
by
James F. Kile
Donald Little
Samir Shah
October 8, 2005
Given: T , a test suite; PT , the set of permutations of T ; and f , a function from PT to
the real numbers,
Find
( T )( T PT )( T T )[ f ( T ) f ( T )]
1
It should be noted that PT represents the set of all possible prioritizations (orderings
of T ) and f is a function that, applied to any such ordering yields an award value for that
ordering that can be used to prioritize.
1.2 Test Case Classification
The use of test prioritization techniques and test elimination techniques to
approximate a solution to the test case prioritization problem discussed in the prior
section is an example of using data classification to solve a problem whereby a definitive
solution is difficult to ascertain or would be to costly in time and effort to achieve. This
paper summarizes two works which propose techniques for achieving test case
prioritization and elimination through classifying test cases into two primary
classifications: Execute and do not execute. In addition, the execute classification is
further sub-divided into test cases of high, medium, and low priority.
1
The equation should be read as follows: Find a new test suite as an element of all possible
permutations such that for all newly identified test suites each is a valid permutation of T , is not the same as
the initial test suite, and has a functional award vale less than or equal to its predecessor.
Srivastava and Thiagarajan [11] propose a method of test case prioritization and
elimination using a method of test case analysis based upon a greedy search algorithm. A
greedy algorithm attacks a problem by constructing the complete solution in a series of
steps [8]. To solve the prioritization problem, Srivastava and Thiagarajan propose a
series of steps which will qualify which cases are available for prioritization
(elimination), determine new code blocks which are covered by existing tests
(prioritization), and identify which order the selected tests should run (prioritization)
[11]. The entire process is shown in the following diagram ():
2.1 Step 1: Binary Match for Code Classification
According to Srivastava and Thiagarajan, the first step in determining whether a test
case should be executed and its execution priority requires the classification of the code
itself into the following categories: Old blocks of code, old modified blocks of code, and
new blocks of code [11]. Old modified and new blocks of code are classified as
impacted blocks [11]. Anything not residing in the impacted blocks class is discarded
and the remainder becomes input to the next step.
2.2 Step 2: New Version Coverage
The second step in the process is to identify whether new code versions are covered
by tests being run for existing code blocks that have been modified [11]. Here, the
also propose an extension of this algorithm to also take test execution time into
consideration.
Cohen et al. describe a method of test case classification for elimination and
prioritization based upon a hill climbing and simulated annealing algorithm [2, 3]. A hill
climbing method uses an iterative improvement algorithm to arrive at a result [8]. One
disadvantage is that the solution can be dependent upon the starting point generating what
is termed a local optima (rather than an overall optimum solution) [3, 8]. To account for
this phenomenon, the authors also suggest an extension which uses simulated annealing.
Simulated annealing allows the algorithm to make choices that reduce the quality of the
result by using a controlled probability [1, 3].
3.1 Hill Climbing Selection
The hill climbing portion of the authors’ test case selection and prioritization
technique begins with the setting of loose upper and lower bounds or using the size of the
entire test suite itself [3]. A solution is selected randomly and is transformed iteratively
by replacing test cases in the suite to generate a new suite. If the new test suite is of
equal or lower cost, it is accepted as the new current feasible solution. If the new test
suite has a higher cost, it is rejected and another randomly chosen transformation is made.
The trial is repeated several times with different initial random test set solutions and the
best is selected.
[1] D. M. Cohen, S. R. Dalal, M. L. Fredman and G. C. Patton, "The AETG System:
An Approach to Testing Based on Combinatorial Design", IEEE Transactions on
Software Engineering , vol. 23, no. 7, pp. 437-444, July 1997.
[2] D. M. Cohen, S. R. Dalal, M. L. Fredman and G. C. Patton, "Method and System
for Automatically Generating Efficient Test Cases for Systems Having Interacting
Elements", United States, Patent No. 5,542,043, July 30, 1996.
[3] M. B. Cohen, P. B. Gibbons, W. B. Mugridge and C. J. Colbourn, "Constructing
Test Suites for Interaction Testing", 25th International Conference on Software
Engineering (ICSE'30) , pp. 38-49, Portland, Oregon, United States, IEEE
Computer Society, 2003.
[4] S. Elbaum, A. G. Malishevsky and G. Rothermel, "Prioritizing Test Cases for
Regression Testing ", 2000 ACM SIGSOFT International Symposium on Software
Testing and Analysis , pp. 102-112, Portland, Oregon, United States, ACM Press,
[5] S. Fraser, D. Astels, K. Beck, B. Boehm, J. McGregor, J. Newkirk and C. Poole,
"Discipline and Practices of TDD: (Test Driven Development)", Conference on
Object Oriented Programming Systems Languages and Applications , pp. 268-
270, Anaheim, California, United States, ACM Press, 2003.
[6] B. George and L. Williams, "An Initial Investigation of Test-Driven Development
in Industry", Proceedings of the 2003 ACM Symposium on Applied Computing ,
pp. 1135-1139, Melbourne, Florida, UASA, ACM Press, 2003.
[7] P. Louridas, "JUnit: Unit Testing and Coding in Tandem", IEEE Software , vol.
22, no. 4, pp. 12-15, July/August 2005.
[8] Z. Michalewicz and D. B. Fogel, How to Solve It: Modern Heuristics. 2nd ed.,
Berlin: Springer-Verlag, 2004.
[9] S. Nerur, R. Mahapatra and G. Mangalaraj, "Challenges of Migrating to Agile
Methodologies", Communications of the ACM , vol. 48, no. 5, pp. 72-78, May
[10] K. J. Nurmela and P. R. J. Östergård, "Constructing Covering Designs by
Simulated Annealing", Technical Report, Report No. B10, Helsinki University of
Technology, Digital Systems Laboratory, pp. 1-29, January 1993.
[11] A. Srivastava and J. Thiagarajan, "Effectively Prioritizing Tests in Development
Environment", Proceedings of the 2002 ACM SIGSOFT International Symposium
on Software Testing and Analysis , pp. 97-106, Rome, Italy, ACM Press, 2002.
