Download Random Testing: Advantages, Disadvantages, and Coverage Analysis - Prof. Yashwant K. Malai and more Study notes Computer Science in PDF only on Docsity!
October 21, 2008 1
Fault Tolerant ComputingFault Tolerant Computing
CS 530CS 530
Random TestingRandom Testing
Yashwant K. Malaiya
Colorado State University
10/21/2008 FTC YKM 2
Random Testing: Outline
• RT: advantages and tradeoffs
• RT vs pseudorandom testing (PR)
• Coverage and detectability profile
• Hardware and software DPs
• C(L) for random and pseudorandom tests
• High and low testability faults during early & late
testing
• Implications of a late asymmetric profile
10/21/2008 FTC YKM 3
Random Testing
- Extensively used for both hardware and software
- Ideally each input is selected randomly. PR
(Pseudorandom) schemes approximate random.
- Generally quite effective for moderate coverage.
Coverage hard to determine a priori.
Ineffective for random-pattern-resistant faults.
Coverage tools: Random (functional) followed by Structural testing.
10/21/2008 FTC YKM 4
Random Testing: Advantage
- No test generation using structural information needed.
- Test set-up using comparison:
Random
pattern
generator
Unit
under
test
Gold
Unit
comparator
Random
pattern
generator
Unit
under
test
Stored
response
comparator
- Alternative: Is response reasonable?{software testing}
10/21/2008 FTC YKM 7
Detectability Profiles: Ex
Inputs=4 (N=16), M=
H=(h 1 ,h 2 ,h 3 ,h 4 ,h 5 ,h 6 ,h 8 )
=(1,11,2,43,21,4,8)
counterexample:
Inputs= 4 (N=16), M=
H=(h 1 ,h 2 ,h 3 ,h 14 )=(23,19,1,1)
hk
k
hk
k Hardest to test
10/21/2008 FTC YKM 8
Coverage with L random vectors
- hk out of M defects detectable by exactly k vectors: detection
probability k/N
- P{a defect with dp k/N not detected by a vector} =
- P{a defect with dp k/N not detected by L vectors} =
- Of hk faults, expected number not covered is
- Expected test coverage with L vectors
C(L) 1 ( 1 )
∑
= − −
N
k
L k
M
h N
k
( 1 ) N
k −
L
N
k ( 1 − )
k
L (^) h N
k ( 1 − )
10/21/2008 FTC YKM 9
Ex: C(L) and components for CECL Full Adder
CECL full adder N 16 M 90
Hk 1 11 2 43 21 4 8 k => 1 2 3 4 5 6 8 Coverage L 0 0 0 0 0 0 0 0 0 1 0.0625 0.1250 0.1875 0.2500 0.3125 0.3750 0.5000 0. 5 0.2758 0.4871 0.6459 0.7627 0.8464 0.9046 0.9688 0. 10 0.4755 0.7369 0.8746 0.9437 0.9764 0.9909 0.9990 0. 15 0.6202 0.8651 0.9556 0.9866 0.9964 0.9991 1.0000 0. 20 0.7249 0.9308 0.9843 0.9968 0.9994 0.9999 1.0000 0.
After 20 vectors:
covered 0.72 10.24 1.97 42.86 20.99 4.00 8.
remaining 0.28 0.76 0.03 0.14 0.01 0.00 0.
10/21/2008 FTC YKM 10
Coverage of partitions
test length L
partition coverage
k=
k=
10/21/2008 FTC YKM 13
Detectability Profile: software
- Regardless of initial profile, after some initial testing, the profile will become asymmetric
- Dunham’s data based on NASA experiments for 16 faults.
error rate
faults
10/21/2008 FTC YKM 14
Detectability Profile: software
Adam's data (Product 1)
Detection rate
Defects with this detection rate
10/21/2008 FTC YKM 15
Detectability Profile: Software
- Software detectability profile
is exponential (Adam’s data, IBM).
- Justification: Early testing
will find & remove easy-to- test faults.
- Testing methods need to focus
on hard-to-find faults.
0
1
0 5 10 15 20 k
Hard to test Low hanging fruit
10/21/2008 FTC YKM 16
Implications: Fault Seeding
- A program has x defects. We want to estimate x.
- Seed j new faults.
- Do some testing. Let faults found be j 1 seeded
faults and x 1 original faults.
- Assuming j 1 /j = x 1 /x we get
- However, in reality the x faults include harder
faults to test,
1
1 j
j x = x
1
1 1 1
j
x j hencex x
x
j
j
10/21/2008 FTC YKM 19
References
- Y.K. Malaiya, A. von Mayrhauser and P. Srimani, “An Examination of Fault Exposure Ratio,” IEEE Trans. Software Engineering, Nov. 1993, pp. 1087-1094.
- S. C. Seth, V. D. Agrawal, H. Farhat, "A Statistical Theory of Digital Circuit Testability," IEEE Trans. Computers, 1990, pp. 582-586.
- K. Wagnor, C. Chin, and E. McCluskey, “Pseudorandom testing. IEEE Trans. Computer, Mar. 1987, pp. 332—343.
- J R Dunham, "Experiments in software reliability: Life-critical applications," IEEE Tran. SE, January 1986, pp. 110 - 123
- Y. K. Malaiya, S. Yang, “The Coverage Problem for Random Testing,” IEEE International Test Conference 1984, pp. 237-245.
