Download Path-Oriented Random Testing: Combining Partition Testing and Random Testing and more Exams Computer Science in PDF only on Docsity!
Path-oriented random testing
Arnaud Gotlieb
IRISA - INRIA
Campus Beaulieu
35042 Rennes Cedex
Matthieu Petit
IRISA - INRIA
Campus Beaulieu
35042 Rennes Cedex
ABSTRACT
Test campaigns usually require only a restricted subset of paths in a program to be thoroughly tested. As random testing (RT) offers interesting fault-detection capacities at low cost, we face the prob- lem of building a sequence of random test data that execute only a subset of paths in a program. We address this problem with an orig- inal technique based on backward symbolic execution and constr- aint propagation to generate random test data based on an uniform distribution. Our approach derives path conditions and computes an over-approximation of their associated subdomain to find such a uniform sequence. The challenging problem consists in building efficiently a path-oriented random test data generator by minimiz- ing the number of rejects within the generated random sequence. Our first experimental results, conducted over a few academic ex- amples, clearly show a dramatic improvement of our approach over classical random testing.
Categories and Subject Descriptors
D.2.5 [ Software Engineering ]: Testing and debugging— Testing tools (e.g. data generators, coverage testing)
General Terms
Reliability
Keywords
Random testing, random test data generation, constraint solving
1. INTRODUCTION
Random Testing (RT) is the process of selecting test data at ran- dom according to an uniform probability distribution over the pro- gram’s input domain. Uniform means that every point of a domain has the same probability to be selected. Although RT has tradition- ally been considered as a blind approach of program testing [15], the results of actual random testing experiments confirmed its ef- fectiveness in revealing faults [6, 9]. Among other advantages [12], one key advantage of RT over other techniques is that it selects ob- jectively the test data by ignoring the specification or the structure
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. RT’06, July 20, 2006, Portland, ME, USA Copyright 2006 ACM 1-59593-457-X/06/0007 ...$5.00.
of the program under test [3]. For a long time, RT has been op- posed to partition testing which aims at selecting one or more test data from each subdomain of a partition of the input domain [17]. As a typical example of partition testing, path testing requires to find a test suite so that every control flow path is traversed at least once. As every feasible^1 path corresponds to a subdomain of the in- put domain, path testing consists in selecting at least one test datum from each subdomain. Test campaigns usually require only a subset of paths to be thor- oughly tested as (exhaustive) path testing is most of the time im- possible. In fact, as soon as the program contains a loop, the num- ber of paths is potentially unbounded, and deciding whether a path is feasible or not is an undecideable problem in the general case [16]. Usual white-box testing approaches require only a subset of paths to be selected to cover all statements, all decisions or other structural criteria [20]. Moreover, it is well known that some (fea- sible) paths are irrelevant for the computation of the function im- plemented by the program as they will never be activated during the operational life of the program. For example, consider a pro- gram where some robustness code has been added just to avoid its usage in unsuitable conditions. These remarks lead us naturally to the idea of performing random testing by selecting first a subset of paths, in order to benefit from the advantages of both random testing and partition testing. In this paper, we introduce a technique to generate random test data based on a uniform distribution for only a subset of paths. Our approach, called Path-oriented Random Testing (PRT), derives the path conditions of the selected paths by using a backward symbolic execution [14] and computes an over-approximation of their asso- ciated subdomain by using constraint propagation [10, 13]. The challenging problem consists in building efficiently a uniform ran- dom test data generator by minimizing the number of rejects within the generated random sequence. A reject is produced whenever the randomly generated test datum does not satisfy the path cond- itions. Our approach addresses this problem by using subtle con- straint propagation and constraint refutation combined with random test data generation. We implemented our PRT approach by using the clp(fd) constraint library of SICStus Prolog [2] and get some first experimental results showing that PRT outperforms traditional RT. In addition, we provide an algorithm that can detect some non- feasible paths in the set of selected paths. This also shows a quali- tative improvement over traditional RT. Outline of the paper. Section 2 gives an overview of our ap- proach on a motivating example. Section 3 recalls some back- ground on symbolic execution while Section 4 explains the prin- ciples on which the path-oriented random testing is based. Section 5 details our constraint propagation approach to build an uniform
(^1) A path is feasible iff it can be activated by some test data
ush foo( ush x, ush y) {
- if (x =< 100 && y =< 100 ) {
- if (y > x + 50) 3....
- if (x ∗ y < 60 ) 5....
Figure 1: Program foo (ush stands for unsigned short integers)
test data generator for a subset of paths. An algorithm to perform PRT is given and analysed. Section 6 reports on the experimental results we obtained with our implementation and Section 7 presents further work.
2. A MOTIVATING EXAMPLE
We start by giving an overview of the PRT approach over tra- ditional RT on a motivating example. Consider the C program of Fig.1 and the problem of building an uniform test data gener- ator for path 1 → 2 → 3 → 4 → 5. By looking at the decisions of the program, we can see that x and y must necessary range in 0 .. 100 to satisfy our objective. However, the other decisions cannot be tackled so easily. By using a uniform random test data genera- tor that independently pick up a value xi in 0 .. 100 and a value yi in 0 .. 100 and rejects the pairs (xi, yi) that do not satisfy the constraints yi > xi + 50 ∧ xi ∗ yi < 60 , we get a uniform ran- dom generator that solves our problem. However, this simple RT approach is highly expensive as it requires rejecting a lot of ran- domly generated pairs. In fact, by manually analyzing the pro- gram, we can determine that the average probability of rejecting a possible pair is not far from 10099 with this RT approach. Indeed, executing path 1 → 2 → 3 → 4 → 5 is an event which has a very low probability as only 58 input points over 1012 satisfy the two de- cisions. In contrast, our PRT approach exploits subtle constraint propagation and constraint refutation to minimize this probability and then to reduce the length of the generated test suite. By us- ing constraint propagation over finite domains on this example, we get immediately that any solution pair (x, y) must range over the rectangle D 1 = (x ∈ 0 .. 1 , y ∈ 51 ..100) which is a (correct) over-approximation of the 58 solutions. Building a random test data generator for D 1 is an easy task as we can select x, y inde- pendently. This would not have been true if D 1 had the shape of a triangle, for example. However, by combining domain bisection and constraint refutation, we can get an even better approximation: D 2 = (x = 0, y ∈ 51 ..101) ∪ (x = 1, y ∈ 51 ..67) where the subdomain (x = 1, y ∈ 68 ..100) has been refuted while just a single spurious pair was added (x = 0, y = 101). Note also that one can still easily build a random test data generator for D 2 by selecting y independently from x as D 2 can be divided in an union of rectangles of the same area. In fact, we design our PRT method while keeping this latter constraint in mind. Finally, we can de- termine that the average probability of rejecting a possible pair is just around 10015 for D 2 ( 58 input points over the 68 of D 2 sat- isfy the two decisions). By using the PRT approach, the size of the randomly generated test suite is dramatically decreased while the overhead introduced by constraint propagation and constraint refutation remains tractable as we will explain in the paper.
3. BACKWARD SYMBOLIC EXECUTION
In this section, we explain how to derive path conditions associ- ated to a subset of selected paths in the program under test. This process is based on backward symbolic execution that was first for- malized by Clarke and Richardson in [14]. This technique is based
on the selection of paths of the control flow graph and the compu- tation of symbolic states.
3.1 Control flow graph The control flow graph of a program P is a connected oriented graph composed of a set of vertices, a set of edges and two distin- guished nodes, e the unique entry node, and s the unique exit node. Each node represents a basic block and each edge represents a pos- sible branching between two basic blocks. Programs with multiple exits can easily been tackled by adding an additional shared exit node. A path of P is a finite sequence of edge-connected nodes of the control flow graph which starts on e. As an example, consider the program power.c given in Fig.2 along with its CFG. This pro- gram computes xy^. Note that this program contains a non-feasible
double P( ush x, ush y) { w = abs(y) ; z = 1.0 ; while ( w != 0 ) {z = z * x ; w = w - 1 ; } if ( y<0 ) z = 1.0 / z ; return (z) ; }
1 2 3 4 5 6
w = abs(y) ; z = 1.0;
z = z*x ; w = w-1 ;
Z = 1.0/z;
return(z)
while( w != 0)
if( y<0)
Figure 2: Control flow graph of program power.c
path ( 1 → 2 → 4 → 5 → 6 ), as in many imperative programs [19].
3.2 Symbolic states Symbolic execution works by computing symbolic states for a given path. A symbolic state for path e→n 1 →... →nk in P is a triple (e→n 1 →... →nk , {(v, φv )}v∈V ar(P ), P C) where φv is a sym- bolic expression associated to the variable v and P C(e→n 1 →.. .) = c 1 ∧... ∧ cn is a set of constraints, called path conditions. V ar(P ) denotes the set of variables in P. A symbolic expression is either a symbolic value (possibly undef ) or a well parenthesized expression composed over symbolic values. In fact, when computing a new symbolic expression, each internal variable reference is replaced by its previously computed symbolic expression. In the program of Fig.2, the symbolic state of path 1 → 2 → 4 → 5 → 6 can easily be obtained by computing the following sequence of symbolic states : (1, {(x, X), (y, Y ), (w, undef ), (z, undef )}, true) (1→ 2 , {(x, X), (y, Y ), (w, abs(Y )), (z, 1 .0)}, true) (1→ 2 → 4 , {(x, X), (y, Y ), (w, abs(Y )), (z, 1 .0)}, abs(Y ) = 0) (1→ 2 → 4 → 5 → 6 , {(x, X), (y, Y ), (w, abs(Y )), (z, 1 .0)}, Y < 0 ∧ abs(Y ) = 0) where X (resp. Y ) is the symbolic value of the input variable x (resp. y). Note that symbolic expressions and path conditions hold only over symbolic input values (except in the presence of floating- point computations [1]). Solving the path conditions yields either to demonstrate that a given path is non-feasible or to find a test datum on which the se- lected path is executed. In the previous example, it is trivial to see that the (non-linear) path conditions Y < 0 ∧ abs(Y ) = 0 have no
the hypercuboid domain but not for all possible domains. Consider for example a triangle domain, such as the one shown in figure 3. Suppose this triangle domain has been computed by a linear solver
x 1 x 2
y
x
x 1
x 2
(x 1 ,y 1 )
(x 2 ,y 2 )
Figure 3: The triangle domain
over integers and is defined as the intersection of the three follow- ing subspaces y ≥ 0 , x ≤ 14 , x ≥ y. Let x 1 , x 2 be two randomly selected integers such as x 2 > x 1 and y 1 , y 2 two randomly selected integers such as y 1 belongs to 1 ..x 1 and y 2 belongs to 1 ..x 2 , then the probability of the event (x 1 , y 1 ) is selected is equal to 1 /x 1 whereas the probability of the event (x 2 , y 2 ) is selected is equal to 1 /x 2. As x 1 6 = x 2 , we have lost uniformity. In fact, these two events are not independents over the triangle domain whereas they always are over an hypercuboid domain.
4.4 Two invariance principles of RT
Our PRT approach makes use of two fundamental invariance principles of uniform random number generators. The first prin- ciple just states that any uniform random generator for a given do- main D can also be employed as an uniform random generator for any of the subdomains of D. More formally:
PROPERTY 1 (FIRST INVARIANCE PRINCIPLE). Let S be a se- quence of uniformly distributed tuples of values for a domain D , then for any subset D′^ of D , it is possible to extract from S a se- quence S′^ of uniformly distributed tuples for D′^ by rejecting the tuples of S that do not belong to D′. The remaining sequence S′^ is still uniformly distributed over D′.
This first invariance principle is illustrated in Fig.4. To obtain an uniform sequence of values for D′, it suffices to examine each of the tuples of the sequence for D and to reject those tuples that do not belong to D′. Of course, the smaller D′^ is w.r.t. D, the larger the uniform sequence for D must be. By looking back at the tri- angle example, the first invariance principle just says that we get a uniform random test data generator for the triangle domain by tak- ing a uniform test data generator for the encompassing rectangle and rejecting the pairs that do not belong to the triangle. The second principle is more subtle as it states that a random test data generator can be built in a hierarchical manner:
PROPERTY 2 (SECOND INVARIANCE PRINCIPLE). Let D be a domain of n tuples, let k be a divisor of n and D 1 ,... , Dk be a partition of D such that each Di possesses the same number of
D’
D
Figure 4: The first invariance principle
tuples, then an uniform random sequence for D can be built by building first a uniform random sequence over D 1 ,... , Dk , and then picking up a single tuple in each Di at random. The resulting sequence of tuples is still uniformly distributed over D_._
The important point here is that all the domains Di have the same number of tuples. Fig. 5 illustrates the second invariance princi- ple over the triangle domain. To build an uniform sequence over the triangle domain, we can first break its encompassing rectan- gle into D 1 ,... , D 15 equivalent subdomains then build a random sequence of these subdomains and finally select a point in each subdomain.
y
x
D
D
D10 D11^ D
D13 D14 D
D
D8 D
D
D
D
D
Figure 5: The second invariance principle
By combining the two principles, we can optimize the process of building a random tuples generator for the triangle domain. It suffices to remove the subdomains that do not intersect the domain D′. For example, in the Fig. 5, we can first remove the subdomains D 1 , D 2 , D 4 , D 7 and then build a random tuples generator by first building a random sequence over D 3 , D 5 , D 6 , D 8 ,... , D 15. The algorithm we implemented is based on these same ideas. The key
point of our PRT approach is that it employs constraint refutation to remove subdomains as we do not have a geometrical view of the solution set of the constraint system. We just have the symbolic expressions of the path conditions.
5. THE PATH-ORIENTED RANDOM TEST-
ING ALGORITHM The PRT algorithm is based on constraint refutation and domain division. These two processes are first explained before giving the algorithm itself.
5.1 Constraint refutation to test domain inter- section Constraint refutation is the process of temporarily adding a con- straint to a set of constraints and testing whether there is no solu- tion by constraint propagation. When there is no solution, then the added constraint is shown to be contradictory with the rest of the constraint system. When there could be a solution, then we cannot deduce anything as constraint propagation is incomplete in gen- eral. This process can be implemented very efficiently as constraint propagation over ranges is linear w.r.t. the number of constraints. Consider again the triangle example and the constraint set y ≥ 0 , x ≤ 14 , x ≥ y that correspond to the triangle domain. Constr- aint propagation over these constraints give D = (x ∈ 0 .. 14 , y ∈ 0 ..14). Then adding the constraint corresponding to D1 = (x ∈ 0 .. 4 , y ∈ 12 ..14) and performing constraint propagation leads to exhibit a contradiction. As a result, D 1 can be removed from the list of subdomains. By repeating this process until all the subdo- mains of D be tested, we get that D 1 , D 2 , D 4 , D 7 can be removed, as expected. Using constraint refutation has another advantage as it is use- ful to detect some non-feasible paths. In fact, non-feasible paths correspond to unsatisfiable constraint systems. Hence, in the PRT approach, if all the subdomains of the partition are shown to be in- consistent, then that means the corresponding path is non-feasible. This contrasts with traditional RT which can never detect non-feasible paths. Note however that the PRT approach can miss to detect some non-feasible paths due to the incompleteness of constraint propaga- tion.
5.2 Dividing the hypercuboid into equivalent
subdomains When a path is feasible, constraint propagation always results in an hypercuboid that is a correct over-approximation of the solu- tion set of the path conditions. The PRT approach builds a random test data generator for the exact path-conditions solution domain within this hypercuboid. Special attention must be paid to the way this hypercuboid is broken into subdomains in order to preserve the uniformity of the generator. As the hypercuboid is only made of integer tuples, we need to introduce some tricks to preserve uni- formity. Let k be a given parameter, called the division parameter, our PRT algorithm is based on the division of each domain vari- able into k subdomains of equal area. When the size of a domain variable cannot be divided by k, then we enlarge its domain until its size can be divided by k. If the input domain is of dimension n, then this trick yields to partition the (augmented) hypercuboid into kn^ subdomains. For the triangle domain of Fig.3, consider a division param- eter equal to 4. Then we have to divide the rectangle domain x ∈ 0 .. 14 , y ∈ 0 .. 14 into 42 = 16 subdomains of equal area. But, 4 does not divide 15 , therefore we propose to enlarge the do- main of x and the domain of y with a value each. Finally, we
get the 16 following subdomains: D 1 = (x ∈ 0 .. 3 , y ∈ 0 ..3), D 2 = (x ∈ 4 .. 7 , y ∈ 0 ..3),..,D 16 = (x ∈ 12 .. 15 , y ∈ 12 ..15). In theory, selecting big values for k yields to maximize the de- ductions as a lot of small subdomains will be tested for satisfiability. But, the time required to check the satisfiability will increase ac- cordingly. In practice, we recommend to select a small value such as k = 2, 3 or 4 as the gain will be maximal (large subdomains could be eliminated) while the overhead will remain neglictible.
5.3 An algorithm to perform path-oriented ran- dom testing By making use of the two invariance principles described above we can then set up an algorithm able to perform path-oriented ran- dom testing (PRT). The algorithm takes as inputs a set of vari- ables along with their finite variation domain, a constraint set corre- sponding to the path conditions (under Disjunctive Normal Form), the division parameter k, an integer N that represent the length of the expected random sequence. The algorithm returns a list of N uniformly distributed random tuples that all satisfy the path cond- itions. The list is void whenever the corresponding paths are all non-feasible.
Algorithm 1: The Path-oriented Random Testing Algorithm Input : (Dom(x 1 ),... , Dom(xn)), (x 1 ,... , xn), C, k, N Output : t 1 ,... , tN
T := ∅ ; (D 1 ,... , Dkn ) := Divide({Dom(X 1 ),... , Dom(Xn)} , k); forall Di ∈ (D 1 ,... , Dkn ) do if Di is inconsistent w.r.t. C then remove Di from (D 1 ,... , Dkn ) ; end end Let D′ 1 ,... , D′ p be the remaining list of domains; if p ≥ 1 then while N > 0 do Pick up uniformly D at random from D′ 1 ,... , D′ p; Pick up uniformly t = (y 1 ,... , yn) at random from D; if C is satisfied by t then add t to T ; N := N − 1 ; end end end return T ;
The PRT algorithm makes use of the Divide function which partitions the hypercuboid in kn^ subdomains as described above. The PRT algorithm is only a semi-correct procedure, meaning that it is not guaranteed to terminate, but when it terminates, it provides the correct result. Indeed, in the second loop, N is decreased iff t satisfies C, which can happen only if C is satisfiable. In other words, if C is unsatisfiable and if this has not been detected by the constraint propagation step (p ≥ 1 ), then the PRT algorithm will surely not terminate. Note that similar problems arise whenever classical random testing with rejects is employed as nothing pre- vent an unsatisfiable goal C to be selected and in this case all the test cases will be rejected. In practice, a time out procedure is nec- essary to force termination. Note that selecting an optimal value
Expected 50 100 150 200 250 300 350 400 450 500 RT 9942 17085 25751 35246 42682 51611 60429 72385 76303 85183 PRT (k = 2) 60 133 206 253 321 376 465 509 569 630 PRT (k = 3) 57 113 179 229 288 351 411 467 529 594 PRT (k = 4) 55 110 166 226 280 330 397 440 510 553
Figure 6: Length of the generated test suite on program foo
Expected 5000 10000 15000 20000 25000 30000 35000 Compiled RT 27.7s 56.8s 83.2s 109.6s 140.3s 160.3s – PRT (k = 2) 0.3s 0.5s 0.6s 0.7s 0.9s 1.0s 1.1s PRT (k = 3) 0.3s 0.4s 0.5s 0.6s 0.7s 0.9s 1.0s PRT (k = 4) 0.3s 0.4s 0.5s 0.6s 0.7s 0.9s 1.0s
Figure 7: CPU time required for generating random test suite on program foo
probability to happen. For example, generating a sequence of three equal tuples (equilateral triangle) is a rare event. Of course, simi- lar drawbacks are expected with the PRT approach as it is mainly an extension of RT. A randomly chosen value is never propagated through the constraint network as this would bias the random test data generator! The PRT approach with k = 2 does not elimi- nate any subdomain but as it makes use of constraint propagation to eliminate some of the non-feasible paths, the complete process outperforms again traditional RT. In fact, among the 7 paths, 4 are shown to be non-feasible. For the PRT method, to start finding inconsistent subdomains, the division parameter k must be instan- tiated to 13. In this case, 469 subdomains are inconsistents over a total of 2197. As this value depends on the problem, we have not used it in our experiments, so the results are just presented with k = 2. This experiment shows that PRT is suitable not only for a single path but also when a subset of paths is given as input. Of course, in this case, constraint propagation is less efficient as dis- junctions are handled in a lazy manner in the constraint solver. In fact, it waits until one of the disjuncts is entailed by the rest of cons- traints. However, even in this case, PRT outperforms traditional RT as shown by Fig.8.
7. FURTHER WORK
In this paper, we introduced a new approach that combines both the advantage of partition testing and random testing. This ap- proach, called Path-oriented Random Testing (PRT), implements RT over only a restricted subset of the control flow paths of a pro- gram to be tested. The challenging problem is to preserve the uni- formity of the random generator only for a subdomain of the pro- gram’s input domain. We have shown that PRT outperforms tradi- tional RT by minimizing the number of rejects needed by any RT approach of this problem. Moreover, by exploiting subtle constr- aint propagation and refutation, the PRT approach also permits to detect some of the non-feasible paths, something which is out of the scope of traditional RT. Our further work will focus on tech- niques to improve the scope of current PRT methods. In particular, dealing with pointers and dynamic structures in PRT appears to be very challenging as we do not possess yet any constraint solvers on these constructions. Moreover, consider pointers as inputs of pro- grams leads to consider unbounded input domains which is chal- lenging for RT methods. More generally, for a random test data generator, maintaining uniformity on a subdomain defined by a set of constraints is a difficult and not yet solved problem. A lot of work remains to be done to understand what are the links between
this problem and other RT approaches.
Acknowledgments Many thanks to Nicky Williams who provided us with very helpful comments on an earlier draft of this paper.
8. REFERENCES [1] B. Botella, A. Gotlieb, and C. Michel. Symbolic execution of floating-point computations. The Software Testing, Verification and Reliability journal , 2006. to appear. [2] M. Carlsson, G. Ottosson, and B. Carlson. An open–ended finite domain constraint solver. In Proc. of Programming Languages: Implementations, Logics, and Programs , 1997. [3] T. Chen, T. Tse, and Y. Yu. Proportional sampling strategy: a compendium and some insights. The Journal of Systems and Software , 58:65–81, 2001. [4] R. DeMillo and J. Offut. Constraint-based automatic test data generation. IEEE Transactions on Software Engineering , 17(9):900–910, September 1991. [5] R. DeMillo and J. Offut. Experimental results from an automatic test case generator. ACM Transactions on Software Engineering Methodology , 2(2):109–127, April 1993. [6] J. Duran and S. Ntafos. An Evaluation of Random Testing. IEEE Transactions on Software Engineering , 10(4):438–444, Jul. 1984. [7] P. Ecuyer. Random Number Generation , chapter draft for a chapter of the forthcoming Handbook of Computational Statistics. Springer-Verlag, 2004. J. E. Gentle, W. Haerdle, and Y. Mori, eds. [8] A. Gotlieb, B. Botella, and M. Rueher. Automatic test data generation using constraint solving techniques. In Proceedings of the International Symposium on Software Testing and Analysis (ISSTA’98) , pages 53–62, Clearwater Beach, FL, USA, March 1998. [9] D. Hamlet and R. Taylor. Partition Testing Does Not Inspire Confidence. IEEE Transactions on Software Engineering , 16(12):1402–1411, Dec. 1990. [10] P. Hentenryck, V. Saraswat, and Y. Deville. Design, implementation, and evaluation of the constraint language cc(fd). Journal of Logic Programming , 37:139–164, 1998. Also in CS-93-02 Brown–University 1993. [11] D. Knuth. The art of computer programming, volume 2:
Expected 10000 20000 30000 40000 50000 60000 70000 80000 RT 17.6s 34.9s 52.6s 70.0s 87.7s 105.3s 123.1s 140.4s PRT with k = 2 0.8s 1.8s 1.9s 2.5s 3.2s 3.8s 4.4s 5.0s
Figure 8: CPU time required for generating random test suite on program trityp
seminumerical algorithms. Addison-Wesley, Reading, 2nd edition, 1981.
[12] S. Mankefors, R. Torkar, and A. Boklund. New quality estimations in random testing. In The 14th Int. Symp. on Software Reliability Engineering (ISSRE’03) , pages 468–478. IEEE Computer Society Press, 2003.
[13] K. Marriott and P. Stuckey. Programming with Constraints : An Introduction. The MIT Press, 1998.
[14] S. Muchnick and N. Jones. Program Flow Analysis: Theory and Applications – Chapter 9 : L. Clarke, D. Richardson. Prentice-Hall, Englewood Cliffs, New Jersey, 1981. [15] G. J. Myers. The Art of Software Testing. John Wiley, New York, 1979.
[16] E. Weyuker. Translatability and decidability questions for restricted classes of program schemas. SIAM Journal of Computing , 8(4):587–598, November 1979. [17] E. Weyuker and B. Jeng. Analyzing Partition Testing Strategies. IEEE Transactions on Software Engineering , 17(7):703–711, Jul. 1991.
[18] B. A. Wichmann and I. D. Hill. Algorithm as 183: An efficient and portable pseudo-random number generator. Applied Statistics , 31:188–190, 1982. [19] D. Yates and N. Malevris. Reducing The Effects Of Infeasible Paths In Branch Testing. In Proc. of Symposium on Software Testing, Analysis, and Verification (TAV3) , volume 14(8) of Software Engineering Notes , pages 48–54, Key West, FL, Dec. 1989. [20] H. Zhu, P. Hall, and J. May. Software unit test coverage and adequacy. ACM Computing Surveys , 29(4):366–426, Dec.
