Download Data Flow Testing as Model Checking - Research Article | CS 576 and more Papers Computer Science in PDF only on Docsity!
Data Flow Testing as Model Checking ∗
Hyoung Seok Hong, Sung Deok Cha
Department of Electrical Engineering and Computer Science and AITrc
Korea Advanced Institute of Science and Technology
{hshong,cha}@salmosa.kaist.ac.kr
Insup Lee, Oleg Sokolsky
Department of Computer and Information Science
University of Pennsylvania
{lee,sokolsky}@saul.cis.upenn.edu
Hasan Ural
School of Information Technology and Engineering
University of Ottawa
Abstract
This paper presents a model checking-based approach to data flow testing. We characterize data flow oriented cover- age criteria in temporal logic such that the problem of test generation is reduced to the problem of finding witnesses for a set of temporal logic formulas. The capability of model checkers to construct witnesses and counterexamples allows test generation to be fully automatic. We discuss complexity issues in minimal cost test generation and describe heurstic test generation algorithms. We illustrate our approach us- ing CTLas temporal logic and SMV as model checker.
1 Introduction
During the last two decades, there have been a num- ber of data flow testing methods.Included are those pro- posed by Rapps and Weyuker[26], Ntafos[24], Ural[30], and Laski and Korel[22], which are originally devel- oped for modules in procedural languages.These meth- ods have been extended for interprocedural programs in procedural languages[13], object-oriented programmming languages[14], and requirements specification languages such as SDL[29, 31] and statecharts[17].In data flow test- ∗This research was supported in part by Advanced Information Tech- nology Research Center at KAIST, NSF CCR-9988409, NSF CCR- 0086147, NSF CCR-0209024, ARO DAAD19-01-1-0473, DARPA ITO MOBIES F33615-00-C-1707, and the Natural Sciences and Engineering Research Council of Canada under grant OGP00000976.
ing, we usually model a software as a flow graph which identifies the information of control flow and data flow in the software.We then establish certain associations be- tween definitions and uses of variables required to be cov- ered in a given coverage criterion by applying conventional data flow analysis upon the flow graph.Finally we select a finite number of paths which cover the associations as a test suite. Model checking is a formal verification technique for de- termining whether a system model satisfies a property writ- ten in temporal logic and model checkers such as SMV[23] and SPIN[16] are already used on a regular basis for the verification of real-world applications.In addition to being automatic, an important feature of model checking is the ability to explain the success or failure of a temporal logic formula[5, 6, 15].If a system model satisfies a formula, model checkers are capable of supplying an execution of the model as a witness demonstrating the success of the for- mula.Conversely, a counterexample is supplied when the model fails to satisfy the formula. This paper presents a model checking-based approach to data flow testing.In our approach, the problems of data flow analysis and path selection in data flow testing are for- mulated in terms of model checking.We investigate four groups of coverage criteria in [26, 24, 30, 22] and character- ize each coverage criterion by specifying the requirements of the coverage criterion using a set of temporal logic for- mulas such that the problem of test generation is reduced to the problem of finding witnesses for the set of formu- las.The capability of model checkers to construct witnesses
and counterexamples allows test generation to be fully au- tomatic.As a by-product, the characterization enables us to discuss complexity issues in minimal cost test genera- tion.This paper illustrates our approach using CTL[4] as temporal logic and SMV[23] as model checker.The main advantages of our approach may be summarized as follows: First, the approach enables test generation from large flow graphs whose size is limited by the capabilities of current model checkers.Second, the approach allows focusing on only high-level specifications of coverage criteria written in temporal logic.All the details about test generation algo- ritms and their implementations are hidden in model check- ers.Third, the approach is language independent in that the temporal logic formulas employed in the approach are applicable with minor modifications to flow graphs con- structed from various kinds of programming languages and requirements specification languages. Connections between data flow analysis and model checking were made in [27, 28] which show that model checking can be used to solve various data flow analysis problems including the standard bit-vector problems.Our approach extends the work of [27, 28] in that data flow testing combines data flow analysis with the path selec- tion problem.Recently, connections between test genera- tion and model checking have been considered especially in specification-based testing.In [20], local and on-the-fly model checking algoritms are applied to test generation.In [32], SPIN is used for on-the-fly test generation.Test gen- eration using the capability of model checker to construct counterexamples has been applied in several contexts.In [1], the application of model checking to mutation analysis is described.In [3, 9], tests are generated by constructing counterexamples for user-supplied temporal logic formulas. In [12], the capability of SMV and SPIN to construct coun- terexamples is applied to test generation for control flow oriented coverage criteria.No consideration is given to data flow testing in the above work. In [18, 19], the authors discuss the application of model checking to test generation from requirements specifica- tions for both control flow and data flow oriented coverage criteria.The approach in [18, 19] is based on the fact that the state space of a specification is often finite and hence one can use reachability graphs instead of flow graphs for test generation.On one hand, this paper extends [18, 19] by considering more comprehensive groups of data flow ori- ented coverage criteria.On the other hand, the flow-graph approach we advocate here can be seen as complementary to the reachability-graph approach in [18, 19].In the flow- graph approach one can generate tests from programs or specifications with infinite state space because the values of variables are not expanded in flow graphs.It, however, requires posterior analysis such as symbolic execution or constraint solving to determine the executability of tests and
for the selection of variable values which make tests exe- cutable.The reachability-graph approach can handle only finite state space but has the advantage that only executable tests are generated which obviates the necessity of posterior analysis. Section 2 briefly reviews the basics of flow graph and CTL which are the model and logic employed in our ap- proach, respectively.Section 3 characterizes the coverage criteria in [26, 24, 30, 22] by associating a CTL formula, parameterized with the propositions of a given flow graph, with each entity required to be covered in a given criterion. Each formula is defined in such a way that a flow graph sat- isfies the formula if and only if the flow graph has an execu- tion covering the entity described by the formula.By find- ing witnesses for every formula in a given criterion, we gen- erate a test suite satisfying the criterion.Section 4 discusses complexity issues in minimal cost test generation.Typically a CTL formula can have several executions as its witness. By selecting the right witness for each formula, one can minimize the size of the test suite.We show that two opti- mization problems of minimal cost test generation are NP- hard and describe heuristic test generation algorithms em- ploying the capability of model checkers to construct coun- terexamples.We report the experimental results obtained by applying the heuristics to a moderate flow graph.In our experience with SMV, we were able to generate test suites from flow graphs containing dozens of variable definitions and uses in seconds.Finally, Section 5 concludes the paper with a discussion of future work.
2 Flow Graph and CTL
A flow graph G = (V ,vs,vf ,A) is a directed graph where V is a finite set of vertices; vs ∈ V is the start vertex; vf ∈ V is the final vertex; and A is a finite set of arcs.A vertex represents a statement and an arc represents possible flow of control between statements.We adopt the following convention to decorate each vertex with data flow informa- tion.Let x be a variable and v be a vertex.We say that x is defined at v, denoted by dxv , if v represents a statement assigning a value to x.We say that x is used at v, denoted by uxv , if v represents a statement referencing x.We use DEF (v) and USE (v) to denote the sets of definitions and uses at v, respectively.A sequence v 1 ...vn of vertices is a path if (vi,vi+1) ∈ A for 1 ≤ i ≤ n − 1 .A path is complete if it starts from the start vertex vs and ends at the final vertex vf. A test sequence is a complete path and a test suite is a finite set of test sequences.Figure 1 shows a program and its flow graph. We view a flow graph as a Kripke structure M = (Q,qinit ,L,R) where Q is a finite set of states; qinit ∈ Q is the initial state; L: Q → 2 AP^ is the function labelling each state with a subset of the set AP of atomic proposi-
where Π 1 ∗ Π 2 = {π | ∃i : πi ∈ Π 1 , πi^ ∈ Π 2 }, πi denotes the prefix of π ending at i, and πi^ denotes the suffix of π starting from i.Let q 0 ...qn be a witness in W(M ,f ) and M ′ be its corresponding Kripke structure defined as (Q, qinit , L, R − {(q, q′) | q = qi for some 1 ≤ i ≤ n − 1 }).It is not hard to see that M ′^ |= f and M simulates M ′. Finally we extend the notion of witnesses to a set of WCTL formulas.Let M be a Kripke structure and F be a set of WCTL formulas.A witness-set Π for F with respect to M is a set of finite paths such that, for every formula f in F with M |= f , there is a finite path π in Π that is a witness for f .It is easy to see that Π is a witness-set for F if and only if it is a witness-set for {f ∈ F | M |= f }. For example, in Figure 2 we observe that {q 0 q 1 q 3 q 4 q 0 q 2 q 3 }, {q 0 q 2 q 3 q 4 q 0 q 1 q 3 }, and {q 0 q 1 q 3 , q 0 q 2 q 3 } are witness-sets for { EF (a ∧ EFc), EF (b ∧ EFc)}.
q 0
q 1
q 2
q 3 ✒✑
✲ ✲ q 4
L(q 0 )=∅, L(q 1 )={a}, L(q 2 )={b}, L(q 3 )={c}, L(q 4 )=∅
Figure 2. An example of Kripke structures
3 Characterizing Data Flow Oriented Cover-
age Criteria
This section characterizes four groups of coverage criteria[26, 24, 30, 22] in terms of witness-sets for WCTL formulas.
3.1 Rapps and Weyuker’s Criteria
Rapps and Weyuker’s criteria require certain associa- tions between definitions and uses of the same variable be covered[26].The criteria are extended with the notion of executability by Frankl and Weyuker[11].We first adopt the following terminology.A path (v, v 1 , ..., vn, v′) is a definition-clear path from v to v′^ with respect to variable x if n = 0 or x is not defined at vi for every 1 ≤ i ≤ n. A pair (dxv , uxv′ ) is a definition-use pair (in short, du-pair) if there is a definition-clear path from v to v′^ with respect to x. For example, consider dxv 1 and uxv 3 in Figure 1.We observe that (dxv 1 , uxv 3 ) is a du-pair through a definition-clear path v 1 v 2 v 3.
3.1.1 Characterization
We first describe how to generate a test sequence covering a pair (dxv , uxv′ ).The first step is to determine whether ( dxv , uxv′ ) is a du-pair or not.For this, we associate the following WCTL formula with (dxv , uxv′ ).
wctl (dxv , uxv′ ) = EF (dxv ∧ EXE [¬ def (x) U (uxv′ ∧ EF final )]) where def (v) is the disjunction of all definitions of x.For example, in Figure 1 we have that def (x) ::= dxv 1 , def (y) ::= dyv 1 , def (z) ::= dzv 1 , and def (max) ::= dmaxv 3 ∨dmaxv 4 ∨dmaxv 6 .It is not hard to see that (dxv , uxv′ ) is a du-pair if and only if the Kripke structure M (G) of a flow graph G satisfies wctl (dxv , uxv′ ).Hence the problem of determining whether ( dxv , uxv′ ) is a du-pair is reduced to a model checking problem.After determining whether (dxv , uxv′ ) is a du-pair, we generate a test sequence covering it.It is also not hard to see that a test sequence covers a du-pair (dxv , uxv′ ) if and only if it is a witness for wctl (dxv , uxv′ ).Hence the problem of generating a test sequence covering (dxv , uxv′ ) is reduced to the problem of finding a witness for wctl (dxv , uxv′ ).For example, a test sequence covering the du-pair (dxv 1 , uxv 3 ) is shown in Fig- ure 3, which is also a witness for EF (dxv 1 ∧ EXE [¬ def (x) U (uxv 3 ∧ EF final )]).
vs
start ✲ ✒✑
v 1
dxv 1
✲ ✒✑
v 2
¬ def (x) ✲ ✒✑
v 3
uxv 3
✲ ✒✑
v 5 ✲ ✒✑
v 6 ✲ ✒✑
v 7 ✲ ✒✑
vf
final
Figure 3. A test sequence covering du-pair ( dxv 1 , uxv 3 )
Now we describe how to generate a set of test sequences for a set of pairs (dxv , uxv′ ) according to the criteria by Rapps and Weyuker.Basically we associate a formula wctl (dxv , uxv′ ) with every pair (dxv , uxv′ ) and characterize each cov- erage criterion in terms of witness-sets for the formulas wctl (dxv , uxv′ ).This reduces the problem of generating a test suite to the problem of finding a witness-set for a set of WCTL formulas. A test suite Π satisfies all-defs coverage criterion if, for every definition dxv and some use uxv′ , some definition-clear path with respect to x from v to v′^ is covered by a test se- quence in Π.Let DEF (G) and USE (G) be the sets of defi- nitions and uses in G, respectively.A test suite Π satisfies all-defs coverage criterion if and only if it is a witness-set for
{
uxv′ ∈USE (G)
wctl(dxv , uxv′ ) | dxv ∈ DEF (G)}.
A test suite Π satisfies all-uses coverage criterion if, for every definition dxv and every use uxv′ , some definition-clear
path with respect to x from v to v′^ is covered by a test se- quence in Π.A test suite Π satisfies all-uses coverage crite- rion if and only if it is a witness-set for
{wctl(dxv , uxv′ ) | dxv ∈ DEF (G), uxv′ ∈ USE (G)}.
In the worst case, the number of formulas can be quadrac- tic in the size of a flow graph since the number of pairs (dxv , uxv′ ) can be O(n^2 ) in a flow graph of size n.For example, for all-uses coverage criterion in Fig- ure 1 we associate 11 formulas with the pairs (dxv 1 ,uxv 2 ), (dxv 1 ,uxv 3 ), (dyv 1 ,uyv 2 ), (dyv 1 ,uyv 4 ), (dzv 1 ,uzv 6 ), (dmaxv 3 ,umaxv 6 ), (dmaxv 3 ,umaxv 7 ), (dmaxv 4 ,umaxv 6 ), (dmaxv 4 ,umaxv 7 ), (dmaxv 6 ,umaxv 6 ), and (dmaxv 6 ,umaxv 7 ).Among them, the formulas for (dmaxv 3 ,umaxv 7 ), (dmaxv 4 ,umaxv 7 ), and (dmaxv 6 ,umaxv 6 ) are not satis- fied in Figure 1, which means that the pairs are not du-pairs. A test suite Π satisfies all-du-paths coverage criterion if, for every definition dxv and every use uxv′ , every cycle-free definition-clear path with respect to x from v to v′^ is cov- ered by a test sequence in Π.Unlike other coverage crite- ria, all-du-paths coverage criterion cannot be characterized in terms of witness-sets.To generate test suites satisfying this criterion properly in our approach, we should be able to construct all cycle-free witnesses instead of only one for a given formula, which is beyond the capability of existing model checkers.In general, extending model checkers to construct all witnesses for a given formula or a subset of witnesses satisfying certain constraints is an open problem.
3.2 Ntafos’ Criteria
Ntafos’ criteria emphasize interactions between differ- ent variables[24].Such interactions are captured in terms of sequences of alternating definitions and uses, called k-dr interactions.A sequence [ dx v 11 ux v 21 dx v^22 ux v^23 ... dx vnn ux nn+1] is a data flow chain (df-chain) if, for every 1 ≤ i ≤ n, (dx vii , ux vii+1 ) is a du-pair[30].Note that the use ux vii+1 and definition dx vii+1+1 occur at the same vertex for every 1 ≤ i ≤ n.A path v 1 π 1 v 2 π 2 ...vn+1 is an interaction subpath of a df- chain if, for every 1 ≤ i ≤ n, viπivi+1 is a definition-clear path from vi to vi+1 with respect to xi.A df-chain consist- ing of k − 1 du-pairs, k ≥ 2 , is a k-definition/reference in- teraction (k-dr interaction) in the terminology of [24]^1 .For example, in Figure 1 we observe that [dxv 1 uxv 3 dmaxv 3 umaxv 6 ] is a 3-dr interaction which has v 1 v 2 v 3 v 5 v 6 as its interaction subpath.
3.2.1 Characterization
For a sequence κ = [dx v^11 ux v^12 dx v^22 ux v^23 ... dx vkk−−^11 ux kk −^1 ], k ≥ 2 , define wctl (κ) as follows. (^1) We do not require the variables x 1 , ..., xn and the vertices v 1 , ..., vn+1 be distinct.This definition is consistent with that of Clarke et al.[7] and Ntafos[25] and is different from the original one[24] which requires the vertices to be distinct.
- if κ is empty, then wctl (κ) = EF final ,
- if κ is [dx vii ux vii+1 ] · κ′, then
wctl (κ) = dx vii ∧ EXE [¬ def (xi) U (ux vii+1 ∧ wctl (κ′))],
By induction on the number of pairs (dx vii , ux vii+1 ) in κ, it can be shown that κ is a k-dr interaction if and only if the Kripke structure M (G) of a flow graph G satisfies wctl (κ). Moreover, a test sequence covers κ if and only if it is a witness for wctl (κ).For example, a test sequence covering the 3-dr interaction [dxv 1 uxv 3 dmaxv 3 umaxv 6 ] is shown in Fig- ure 4, which is also a witness for EF (dxv 1 ∧ EXE [¬ def (x) U (uxv 3 ∧ dmaxv 3 ∧ EXE [¬ def (max) U (umaxv 6 ∧ EF final )])]).
vs
start ✲ ✒✑
v 1
dxv 1
✲ ✒✑
v 2
¬ def (x) ✲ ✒✑
v 3
dmaxv 3 uxv 3
✲ ✒✑
v 5
¬ def (max) ✲ ✒✑
v 6
umaxv 6
✲ ✒✑
v 7 ✲ ✒✑
vf
final
Figure 4. A test sequence covering 3-dr inter- action [ dxv 1 uxv 3 dmaxv 3 umaxv 6 ]
A test suite Π satisfies required k-tuples coverage crite- rion if, for every k-dr interaction κ, some interaction sub- path of κ is covered by a test sequence in Π.A test suite Π satisfies required k-tuples coverage criterion if and only if it is a witness-set for
{wctl([dx v 11 ux v 21 dx v 22 ux v 32 ...dx vkk−−^11 ux kk −^1 ])
| dx vii ∈ DEF (G), ux vii+1 ∈ USE (G), 1 ≤ i ≤ k − 1 }.
3.3 Ural’ Criteria
Ural’s criteria also emphasize interactions between dif- ferent variables[30].While Ntafos’ criteria consider df- chains consisting of fixed number of du-pairs, Ural’s crite- ria consider df-chains consisting of an arbitrary (but finite) number of du-pairs which start with inputs and end with outputs.The rationale here is to identify the functionality of a module in terms of the interactions with its environ- ment by identifying the effects of inputs accepted from the environment on outputs offered to the environment.We say that a definition dxv affects a use ux
′ v′^ if
- either x = x′^ and (dxv , ux
′ v′^ ) is a du-pair or
- there is a use uxv′′ such that (dxv , uxv′′ ) is a du-pair and there is a definition dx
′′ v′′^ , given in terms of^ u
x v′′^ , that affects ux
′ v′^.
Theorem 1 Q ( dxv , ux
′ v′^ ) is a least fixpoint of^ τ^.
P ROOF Assume that Z 1 ⊆ Z 2 .Then τ (Z 1 ) ⊆ τ (Z 2 ) be- cause Z 1 [x′′/x, v′′/v] ⊆ Z 2 [x′′/x, v′′/v] and the modal op- erator U is monotonic.Hence τ is monotonic. Let Zf be Q(dxv , ux
′ v′^ ).It is easy to see that^ Zf^ =^ τ^ (Zf^ ) and hence Zf is a fixpoint of τ. To prove that Zf is a least fixpoint of τ , it is sufficient to show that Zf = ∪iτ i^ where τ 0 (Z) = Z and τ i+1(Z) = τ (τ i(Z)).We first prove that τ i( false ) ⊆ Zf for every i. Clearly, τ 0 ( false ) ⊆ Zf .Assume that τ i( false ) ⊆ Zf .Be- cause τ is monotonic, τ i+1( false ) ⊆ τ (Zf ).Because Zf is a fixpoint of τ , τ i+1( false ) ⊆ Zf .Hence we have the first direction ∪iτ i( false ) ⊆ Zf .The other direction, Zf ⊆ ∪iτ i( false ), is proved by induction on the number of du- pairs.Suppose that q 0 |= Zf , then there is a path q 0 q 1 ... covering a df-chain for (dxv , ux
′ v′^ ).Let^ j^ ≥^1 be the number of du-pairs of the df-chain.We show that q 0 ∈ τ j^ ( false ) for every j.For the base case j = 1, we have that x = x′^ and q 0 |= dxv ∧ EXE [¬ def (v) U (ux
′ v′^ ∧^ EF final )].Hence^ q^0 ∈ τ 1 ( false ).For the inductive step, suppose q 0 ∈ τ j^ ( false ) for j = n.Let j = n + 1 and qk be the state in q 0 q 1 ... at which the first du-pair in the df-chain ends.Then there are n du-pairs from qk and qk ∈ τ n( false ) by the induction hy- pothesis.Hence q 0 |= (dxv ∧ EXE [¬ def (v) U
uxv′′ ∈USE (G) (uxv′′ ∧ τ n( false )[x′′/x, v′′/v])]) and q 0 ∈ τ n+1( false ).
3.4 Laski and Korel’s Criteria
Laski and Korel’s criteria emphasize that a vertex may contain uses of several different variables in which each use may be reached by several different definitions[22].Such definitions constitute the definition context of the vertex. Let v be a vertex and {ux v 1 , ..., ux vn } be a subset of USE (v). An ordered definition context of v with respect to {ux v 1 , ..., ux v n} is a sequence [dx v 11 ... dx vnn ] of definitions such that there is a subpath v 1 π 1 v 2 π 2 ...πnv, called orderded con- text subpath , satisfying the following property: for every 1 ≤ i ≤ n, viπivi+1...πnv is a definition-clear path from vi to v with respect to xi. A definition context of v is a set of definitions, some permutation of which is an ordered defi- nition context of v.For example, consider the vertex v 6 in Figure 1.[ dzv 1 dmaxv 6 ] is an ordered definition context of v 6 with respect to {uzv 6 umaxv 6 } whose ordered context subpath is v 1 v 2 v 3 v 5 v 6.
3.4.1 Characterization
Let v be a vertex and {ux v 1 , ..., ux vn } be a subset of USE (v). For a sequence λ = [dx v 11 ... dx vnn ] of definitions, define wctl (λ) as follows.
wctl (λ, nodef ) = ux v^1 ∧ ... ∧ ux vn ∧ EFfinal ,
- if λ is [dx vii ] · λ′, then
wctl (λ, nodef ) = nodef ∧ dx vii ∧ EXE [ nodef ′ U wctl (λ′, nodef ′)]),
where nodef ′^ = nodef ∧ ¬ def (vi),
- wctl (λ) = EF wctl (λ, true ).
By induction on the number of definitions in λ, it can be shown that λ is an ordered definition context of v with re- spect to {ux v 1 , ..., ux v n} if and only if the Kripke structure M (G) of a flow graph G satisfies wctl (λ).Moreover, a test sequence covers λ if and only if it is a witness for wctl (λ).For example, a test sequence covering the or- dered definition context [dzv 1 dmaxv 3 ] with respect to {uxv 6 , umaxv 6 } is shown in Figure 5, which is also a witness for EF (dzv 1 ∧ EXE [¬ def (z) U (¬ def (z) ∧dmaxv 3 ∧ EXE [(¬ def (z) ∧ ¬ def (max)) U (uzv 6 ∧ umaxv 6 ∧ EF final )])]).
vs
start ✲ ✒✑
v 1
dzv 1
✲ ✒✑
v 2
¬ def (z) ✲ ✒✑
v 3
dmaxv 3 ¬ def (z) ✲ ✒✑
v 5
¬ def (max) ¬ def (z) ✲ ✒✑
v 6
umaxv 6 uzv 6
✲ ✒✑
v 7 ✲ ✒✑
vf
final
Figure 5. A test sequence covering ordered context [ dzv 1 dmaxv 3 ] with respect to {uzv 6 , umaxv 6 }
A test suite Π satisfies context coverage criterion if, for every vertex v and every definition context dc of v, an or- dered context subpath of dc is covered by a test sequence in Π.A test suite Π satisfies context coverage criterion if and only if it is a witness-set for
{wctl({dx v^11 , ..., dx vnn })
| v ∈ V, ux v i∈ USE (v), dx vii ∈ DEF (G), 1 ≤ i ≤ n}
where wctl ({dx v^11 , ..., dx vnn }) is defined as wctl (λ 1 ) ∨...∨ wctl (λn), where λ 1 , ..., λn are the permutations of {dx v 11 , ..., dx vnn }. A test suite Π satisfies ordered context coverage crite- rion if, for every vertex v and every ordered definition con- text odc of v, an ordered context subpath of odc is covered by a test sequence in Π.A test suite Π satisfies ordered context coverage criterion if and only if it is a witness-set for {wctl([dx v^11 , ..., dx vnn ])
| v ∈ V, ux vi ∈ USE (v), dx vii ∈ DEF (G), 1 ≤ i ≤ n}.
4 Generating Minimal Cost Test Suites
This section discusses complexity issues in minimal cost test generation and describes heuristic algorithms and our experience with SMV for automatic test generation. We wish to generate a minimal cost test suite Π with respect to one of the two costs: (i) the number of test se- quences in Π or (ii) the total length of test sequences in Π. After finishing the execution of a test sequence, an imple- mentation under test should be reset into its initial state from which another test sequence can be applied.It is appropri- ate to use the first cost if the reset operation is expensive, and the second one otherwise.For example, in Figure 2 we have that {q 0 q 1 q 3 q 4 q 0 q 2 q 3 } and {q 0 q 2 q 3 q 4 q 0 q 1 q 3 } are minimal in the number of test sequences, while {q 0 q 1 q 3 , q 0 q 2 q 3 } is minimal in the total length of test sequences. We first consider the Minimal Number Test Generation (MNTG) problem which is an optimization problem defined by: given a Kripke structure M and a set F of WCTL for- mulas, generate a minimal witness-set Π in the number of witnesses in Π.We show this problem to be NP-hard by considering its decision problem MNTG′: given M, F , and a positive integer k, is there a witness-set Π with |Π| ≤ k?
Theorem 2 MNTG ′^ is NP-complete.
P ROOF On input 〈〈M, F, k〉, Π〉 where Π is a set of finite paths, we determine whether Π is a witness-set with |Π| ≤ k for F with respect to M by verifying (i) |Π| ≤ k, (ii) for every π ∈ Π, π is a path of M , and (iii) for every f ∈ F , there is a witness π ∈ Π for f .This is a polynomial time verifier and hence MNTG′^ is in NP. We next show that a NP-complete problem, called the Hitting Set problem, is polynomially reducible to MNTG′. The Hitting Set problem is defined by: given a collection of subsets Ci, 1 ≤ i ≤ n, of a finite set S and a positive integer k, is there a subset S ⊆ S′, called hitting set, such that |S′| ≤ k and containing at least one element from each Ci? Given an instance of the Hitting Set problem, we construct M = (Q,qinit ,L,R) and F as follows:
Ci},
- qinit = q 0 ,
- L(q 0 ) = ∅ and, for every qc, i ∈ L(qc) if and only if qc ∈ Ci,
- R = {(q 0 , qc) | c ∈
Ci}, and
This reduction is in polynomial time.Clearly, c ∈ Ci if and only if q 0 qc is a witness for EF i.Therefore, a subset S′^ of S is a hitting set with |S′| ≤ k for the collection of Ci if and only if Π = {q 0 qs | s ∈ S′} is a witness-set with |Π| ≤ k.
Second we consider the Minimal Length Test Generation (MNTG) problem defined by: given M and F , generate a minimal witness-set Π in the total length of witnesses in Π. Its decision problem MNTG′^ is defined by: given M , F , and k, is there a witness-set Π with
π∈Π |Π| ≤^ k?
Theorem 3 MLTG ′^ is NP-complete.
P ROOF We use the same reduction as in Theorem 2.Since all paths in the Kripke structure M are of length one, the minimum total length of Π is achieved when Π contains the minimum number of witnesses.Therefore, a solution for the MLTG′^ problem in this case will yield the same witness- set which is also a solution to the MNTG′^ problem. Because of NP-hardness, we do not expect an optimal so- lution to the minimal cost test generation problems.Instead we describe a heuristic algorithm which can be applied to both MNTG′^ and MLTG′^ problems.Figure 6 describes the algorithm in a generic fashion without being specific about any coverage criteria.We directly employ the capability of model checkers to construct counterexamples by exploiting the fact that a witness for a WCTL formula is also a coun- terexample for its negation.
I NPUT : a Kripke structure M and a coverage criterion C O UTPUT : a test suite Π satisfying C
1: Π := ∅; 2: mark every entity required to be covered in C as uncovered ; 3: repeat 4: choose an entity marked as uncovered ; 5: let f be the WCTL formula for the entity; 6: model-check f against M ; 7: if M |= f 8: mark the entity as untestable ; 9: else 10: let π be a witness for f (or equivalently a counterexample for ¬f ); 11: let En(π) be the set of entities covered by π; 12: mark every entity in En(π) as covered ; 13: for every π′^ ∈ Π such that En(π′) ⊆ En(π) 14: Π := Π − {π′}; 15: Π := Π ∪ {π}; 16: until every entity is marked as covered or untestable 17: return Π;
Figure 6. A heuristic algorithm for test gener- ation
Basically, we construct a witness for every formula by finding a counterexample for its negation.The algorithm is locally optimal in the sense that model checkers such as SMV find a shortest counterexample for a given formula through breadth-first search of the state space.However, it
to generate test suites from flow graphs with 220 vertices and one hundred formulas in one minute and flow graphs with 270 vertices and one hundred formulas in one hour.Of course, further experiments are compulsory to demonstrate the feasibility of our approach when applied to data flow testing with huge state space. We are planning to extend our approach for interproce- dural programs and object-oriented programs.Data flow testing of such programs is more complicated due to proce- dure call/return, recursion, and reference parameters as well as global variables.Data flow testing methods for such pro- grams were proposed in [13, 14] which employ interproce- dural data flow analysis.Recently, the problem of interpro- cedural data flow analysis has been formulated as a model checking problem[2, 10].Combining both work together may be a starting point for developing a model checking- based approach to data flow testing of interprocedural pro- grams and object-oriented programs. We showed that a subclass of CTL, which we call WCTL, is expressive enough to characterize a number of data flow oriented coverage criteria except those by Ural[30].For Ural’s criteria, we extended WCTL with least fixpoints so that model checking of the resulting logic can be readily implemented in existing model checkers for CTL such as SMV.To characterize the criteria considered in this paper in a more uniform way, it is necessary to employ a logic more powerful than CTL.We are currently work- ing with a subclass of mu-calculus[21], more specifically alternation-free mu-calculus[8], which supports the explicit use of fixpoint operators. We cannot directly use linear time temporal logic for the characterization of data flow oriented coverage criteria, be- cause it requires existential quantification over paths.It is, however, possible to construct a witness for a WCTL for- mula using linear time model checkers by exploiting the fact that a path is a witness for a WCTL formula if and only if the path is a counterexample for its negation.For example, we can construct a witness for a WCTL formula EFEF p by finding a counterexample for ¬ EFEF p = AGAG ¬p, which is in turn equivalent to the LTL formula AGG ¬p.This opens the possibility of applying linear time model checkers such as SPIN to data flow testing.
References
[1] P.Ammann, P.Black, and W.Majurski, “Using Model Checking to Generate Tests from Specifications,” in Pro- ceedings of the 2nd IEEE International Conference on For- mal Engineering Methods , pp.46-54, 1998.
[2] T.Ball and S.K.Rajamani, “Bebop: a Symolic Model Checker for Boolean Programs,” SPIN Workshop ’00 , Vol. 1885 of LNCS, pp.113-130, Springer-Verlag, 2000.
[3] J.Callahan, F.Schneider, and S.Easterbrook, “Specification- based Testing Using Model Checking,” in Proceedings of 1996 SPIN Workshop , also Technical Report NASA-IVV-96- 022, West Virginia Univeristy, 1996. [4] E.M. Clarke, E.A. Emerson, and A.P. Sistla, “Automatic Ver- ification of Finite-State Concurrent Systems Using Temporal Logic Specifications,” ACM Transactions on Programming Languages and Systems , 8(2):244-263, Apr.1986. [5] E.M.Clarke, O.Grumberg, K.McMillan, and X.Zhao, “Efficient Generation of Counterexamples and Witnesses in Symbolic Model Checking,” in Proceedings of the 32nd De- sign Automation Conference , pp.427-432, 1995. [6] E.M.Clarke, S.Jha, Y.Lu, and H.Veith, “Tree-Like Coun- terexamples in Model Checking,” in Proceedings of the 17th Annual IEEE Symposium on Logic in Computer Science , pp. 19-29, 2002. [7] L.A. Clarke, A. Podgurski, D.J. Richardson, and S.J. Zeil, “A Formal Evaluation of Data Flow Path Selection Criteria,” IEEE Transactions on Software Engineering , 15(11):1318- 1332, Nov.1989. [8] R.Cleaveland and B.Steffen, “A Linear-Time Model- Checking Algorithm for the Alternation-Free Modal Mu- Calculus,” Formal Methods in System Design , Vol.2, pp. 121-147, 1993. [9] A.Engels, L.Feijs, and S.Mauw, “Test Generation for Intel- ligent Networks Using Model Checking,” TACAS ’97 , Vol. 1217 of LNCS, pp.384-398, Springer-Verlag, 1997. [10] J.Esparza and J.Knoop, “An Automata-Theoretical Ap- proach to Interprocedural Data-Flow Analysis,” FOSSACS ’99 , Vol.1578 of LNCS, pp.14-30, Springer-Verlag, 1999. [11] P.G. Frankl and E.J. Weyuker, “An Applicable Family of Data Flow Testing Criteria,” IEEE Transactions on Software Engineering , 14(10):1483-1498, Oct.1988. [12] A.Gargantini and C.Heitmeyer, “Using Model Checking to Generate Tests from Requirements Specifications,” in Pro- ceedings of ESEC/FSE ’99 pp.146-162, 1999. [13] M.J. Harrold and M.L. Soffa, “Interprocedural Data Flow Testing,” in Proceedings of the 3rd Symposium on Software Testing, Analysis, and Verification , pp.158-167, 1989. [14] M.J.Harrold and G.Rothermel, “Performing Data Flow Testing on Classes,” in Proceedings of the 2nd ACM SIG- SOFT Symposium on the Foundations of Software Engineer- ing , pp.154-163, 1994. [15] R.Hojati, R.K.Brayon, and R.P.Kurshan, “BDD-based De- bugging of Designs Using Language Containment and Fair CTL,” CAV ’99 , Vol.697 of LNCS, pp.41-58, Springer- Verlag, 1993. [16] G.J. Holzmann, “The Model Checker SPIN,” IEEE Transac- tions on Software Engineering , Vol.23, No.5, pp.279-295, May 1997. [17] H.S.Hong, Y.G.Kim, S.D.Cha, D.H.Bae, and H.Ural, “A Test Sequence Selection Method for Statecharts,” Journal of Software Testing, Verification, and Reliability , 10(4):203- 227, Dec.2000.
[18] H.S.Hong, I.Lee, O.Sokolsky, and S.D.Cha, “Automatic Test Generation from Statecharts Using Model Checking,” in Proceedings of the First Workshop on Formal Approaches to Testing of Software , pp.15-30, 2001. [19] H.S.Hong, I.Lee, O.Sokolsky, and H.Ural, “A Tempo- ral Logic Based Theory of Test Coverage and Generation,” TACAS ’02 , Vol.2280 of LNCS, pp.327-341, Springer- Verlag, 2002. [20] T.Jeron and P.Morel, “Test Generation Derived From Model Checking,” CAV ’99 , Vol.1633 of LNCS, pp.108-121, Springer-Verlag, 1999. [21] D.Kozen, “Results on the Propositional Mu-Calculus,” The- oretical Computer Science , 27:333-354, 1983. [22] J.W. Laski and B. Korel, “A Data Flow Oriented Program Testing Strategy,” IEEE Transactions on Software Engineer- ing , 9(5):347-354, May 1983. [23] K.L. McMillan, Symbolic Model Checking − an Approach to the State Explosion Problem , Kluwer Academic Publishers,
[24] S.C. Ntafos, “On Required Element Testing,” IEEE Transac- tions on Software Engineering , 10(11):795-803, Nov.1984. [25] S.C. Ntafos, “A Comparison of Some Structural Testing Strategies,” IEEE Transactions on Software Engineering , 14(6):868-874, June 1988. [26] S.Rapps and E.J.Weyuker, “Selecting Software Test Data Using Data Flow Information,” IEEE Transactions on Soft- ware Engineering , 11(4):367-375, Apr.1985. [27] D.A. Schmidt and B. Steffen, “Data-flow Analysis as Model Checking of Abstract Interpretations,” SAS ’98 , Vol.1503 of LNCS, pp.351-380, Springr-Verlag, 1998. [28] B.Steffen, “Generating Data-Flow Analysis Algorithms for Modal Specifications,” Science of Computer Programming , 21:115-139, 1993. [29] H.Ural and B.Yang, “A Test Sequence Generation Method for Protocol Testing,” IEEE Transactions on Communica- tions , 39(4):514-523, Apr.1991. [30] H.Ural, “IO-df-chains criterion,” ISO Working Group on Formal Methods on Conformance Testing, Draft Interna- tional Standard, Sept.1993. [31] H.Ural, K.Saleh, and A.Williams, “Test Generation Based on Control and Data Dependencies within System Specifica- tions in SDL,” Computer Communications , 23(7):609-627, Mar.2000. [32] R.de Vries and J.Tretmans, “On-the-Fly Conformance Test- ing Using SPIN,” International Journal on Software Tools for Technology Transfer , 2(4):382-393, 2000.
