Download Random Testing: Advantages, Disadvantages, and Coverage Analysis - Prof. Yashwant K. Malai and more Study notes Computer Science in PDF only on Docsity!
November 30, 2006 1
Fault Tolerant Computing Fault Tolerant Computing
CS 530 CS 530
Random Testing Random Testing
Yashwant K. Malaiya
Colorado State University
11/30/2006 FTC YKM 2
Random Testing: Outline
• RT: advantages and tradeoffs
• RT vs pseudorandom testing (PR)
• Coverage and detectability profile
• Hardware and software DPs
• C(L) for random and pseudorandom tests
• High and low testability faults during early & late
testing
• Implications of a late asymmetric profile
11/30/2006 FTC YKM 3
Random Testing
- Extensively used for both hardware and software
- Ideally each input is selected randomly. PR
(Pseudorandom) schemes approximate random.
- Generally quite effective for moderate coverage.
Coverage hard to determine a priori.
Ineffective for random-pattern-resistant faults.
Coverage tools: Random (functional) followed by Structural testing.
11/30/2006 FTC YKM 4
Random Testing: Advantage
- No test generation using structural information needed.
- Test set-up using comparison:
Random
pattern
generator
Unit
under
test
Gold
Unit
comparator
Random
pattern
generator
Unit
under
test
Stored
response
comparator
- Alternative: Is response reasonable?{software testing}
11/30/2006 FTC YKM 7
Detectability Profiles: Ex
Inputs=4 (N=16), M=
H=(h 1 ,h 2 ,h 3 ,h 4 ,h 5 ,h 6 ,h 8 )
=(1,11,2,43,21,4,8)
counterexample:
Inputs= 4 (N=16), M=
H=(h 1 ,h 2 ,h 3 ,h 14 )=(23,19,1,1)
hk
k
hk
k Hardest to test
11/30/2006 FTC YKM 8
Coverage with L random vectors
- hk out of M defects detectable by exactly k vectors: detection probability k/N
- P{a defect with dp k/N not detected by a vector} =
- P{a defect with dp k/N not detected by L vectors} =
- Of hk faults, expected number not covered is
- Expected test coverage with L vectors
C(L) 1 ( 1 )
= − −
N
k
L k
M
h N
k
( 1 ) N
k −
L
N
k ( 1 − )
k
L (^) h N
k ( 1 − )
11/30/2006 FTC YKM 9
Ex: C(L) and components for CECL Full Adder
remaining 0.28 0.76 0.03 0.14 0.01 0.00 0.
covered 0.72 10.24 1.97 42.86 20.99 4.00 8.
After 20 vectors:
L
k => 1 2 3 4 5 6 8 Coverage
Hk 1 11 2 43 21 4 8
CECL full adder N 16 M 90
11/30/2006 FTC YKM 10
Coverage of partitions
test length L
partition coverage
k=
k=
11/30/2006 FTC YKM 13
Detectability Profile: software
- Regardless of initial profile, after some initial testing, the profile will become asymmetric
- Dunham’s data based on NASA experiments for 16 faults.
error rate
faults
11/30/2006 FTC YKM 14
Detectability Profile: software
Adam's data (Product 1)
Detection rate
Defects with this detection rate
11/30/2006 FTC YKM 15
Detectability Profile: Software
- Software detectability profile
is exponential (Adam’s data, IBM).
- Justification: Early testing
will find & remove easy-to- test faults.
- Testing methods need to focus
on hard-to-find faults.
0
1
0 5 10 15 20 k
Hard to test Low hanging fruit
11/30/2006 FTC YKM 16
Implications: Fault Seeding
- A program has x defects. We want to estimate x.
- Seed j new faults.
- Do some testing. Let faults found be j 1 seeded
faults and x 1 original faults.
- Assuming j 1 /j = x 1 /x we get
- However, in reality the x faults include harder
faults to test,
1
1 j
j x = x
1
1 1 1 j
x j hencex x
x
j
j
11/30/2006 FTC YKM 19
References
- Y.K. Malaiya, A. von Mayrhauser and P. Srimani, “An Examination of Fault Exposure Ratio,” IEEE Trans. Software Engineering, Nov. 1993, pp. 1087-1094.
- S. C. Seth, V. D. Agrawal, H. Farhat, "A Statistical Theory of Digital Circuit Testability," IEEE Trans. Computers, 1990, pp. 582-586.
- K. Wagnor, C. Chin, and E. McCluskey, “Pseudorandom testing. IEEE Trans. Computer, Mar. 1987, pp. 332—343.
- J R Dunham, "Experiments in software reliability: Life-critical applications," IEEE Tran. SE, January 1986, pp. 110 - 123
- Y. K. Malaiya, S. Yang, “The Coverage Problem for Random Testing,” IEEE International Test Conference 1984, pp. 237-245.
