Download Testing in Programming Languages: Black Box, White Box, and Structural Coverage and more Study notes Programming Languages in PDF only on Docsity!
CMSC 433 – Programming Language
Technologies and Paradigms
Fall 2008
Testing
Some slides adapted from FSE’98 Tutorial by Michal Young and Mauro Pezze’
Testing
• Execute program on sample input data
- Check if output correct (acceptable)
• Goals
- Increase confidence program works correctly
- Find bugs in program
3
Simple Example
% java TestServlet HelloWorld /FooBar/Test > out HTTP/1.0 200 Content-Type: text/plain Hello /FooBar/Test % diff out expectedOutput
Limitations of Testing
• Program runs on (very small) subset of input data
- Exhaustive testing usually impossible
- Too large input space (possibly infinite)
• Many situations hard to test
- Parallel code (due to non-determinism)
- Hard-to-reach states (e.g., error states)
- Inadequate test environment (e.g., lack of hardware)
• Testing cannot prove absence of bugs
- Especially a problem in security
7
The Test Case Generation Problem
• How to choose tests that will show that my
program works?
- Must consider “operational scenarios”
- What is legitimate input?
- What is the correct action or output?
- Must consider “abnormal behaviors” as well
• How can I make sure that all of the important
behaviors of my program have been tested?
Test Cases via Specifications
// Return true if x in a, else returns false boolean contains(int[] a, int x);
• Two “paths” in specification
- Test case where x is in a
- Test case where x is not in a
9
Test Cases via Inferred Implementation
• Think about how the implementation might look
- Test by boundary condition
- What test cases are likely to exercise the same logic?
- Want to avoid redundant tests, to save time
- Test by common mistake
- What cases may be tricky to implement?
• At the same time, tests should still be
implementation-independent
Test Cases via Boundary Conditions
interface List { ... Inserts the specified element at the specified position in this list (optional operation). Shifts the element currently at that position (if any) and any subsequent elements to the right (adds one to their indices). public void add(int index, Object element) }
• Test with empty list
• Test with index at first/last element
• Others?
13
Statement Coverage
One test case (n=1, a[0]=-7, x=9) covers all statements Faults handling positive values of a[i] not revealed int select(int[] a, int n, int x) { int i=0; while (i<n && a[i] <x) { if (a[i]<0) a[i] = - a[i]; i++; } return 1; } i++ i<n and a[i] <x a[i]< a[i] = - a[i]; return 1 true false true false i=
Branch Coverage
i= i<n and a[i] <x a[i]< a[i] = - a[i]; return 1 true false true false Must add test case (n=1, a[0]=7, x=9) to cover false branch of if Faults handling positive values of a[i] revealed. Faults exiting the loop with a[i] <x not revealed int select(int[] a, int n, int x) { int i=0; while (i<n && a[i] <x) { if (a[i]<0) a[i] = - a[i]; i++; } return 1; } i++
15
Condition Coverage
i= i<n and a[i] <x a[i]< a[i] = - a[i]; return 1 true false true false Both i<n and a[i]<x must be false and true for different tests. Must add tests that cause loop to exit for a value greater than X. Faults that arise after several loop iterations not revealed. int select(int[] a, int n, int x) { int i=0; while (i<n && a[i] <x) { if (a[i]<0) a[i] = - a[i]; i++; } return 1; } (^) i++
Structural Coverage Testing
• Adequacy criteria
- If significant parts of program structure are not tested, testing is surely inadequate
• Control flow coverage criteria
- Statement (node, basic block) coverage
- Branch (edge) coverage
- Condition coverage
• Attempted compromise between the impossible
and the inadequate
19
Testing Activities
• Test case execution is only a part of the process
• Must also consider
- Test case generation
- Test result evaluation
• Planning is essential
- To achieve early and continuous visibility
- To choose appropriate techniques at each stage
- To build a testable product
- To coordinate complementary analysis and testing
The Testing Environment
• Want to create a scaffold for executing tests
- Code infrastructure to run tests and check output
• Many benefits
- Can automate testing process
- Useful for regression testing
• But, can take some time to implement
21
Testing Environment Components
• A user to generate input for tested component
• An oracle for verifying the results are correct
• These two may be combined into a single system
Unit Testing with Junit
• Testing environment for writing black-box tests
- Write special classes to test other classes
- Several ways to use/set up test cases
• Can be downloaded from
25
Junit Components
• Test cases
- Individual tests (@Test)
- Can reuse test case setup (@Before, @BeforeClass)
- Can reuse test case teardown (@After, @AfterClass)
• Test suites (@RunWith(Suite.class))
• Test runner (org.junit.runner.JUnitCore)
- Executes test suites and presents results
- Can also execute tests within Eclipse
To Execute Tests within a Class
• Invoke test runner on the test case class
java org.junit.runner.JUnitCore ListTest JUnit version 4. .. Time: 0. OK (2 tests)
27
… or run in Eclipse
• Add test case class to your project
• Right click, and do Run As … Junit test.
- Note: need to set up CLASSPATH correctly for the command-line execution of tests. Take a look at link in the class Resources page
Each Test Has Three Parts
• Code that creates test objects
• Code that executes the test
• Code that verifies the result
- E.g., use org.junit.assertTrue() to check results (throws exception is test fails)
31
More Asserts
• Junit has several different tests
- assertTrue(b) -- asserts that b is true
- assertFalse(b) -- asserts that b is false
- assertEquals(o1, o2) -- assert that o1.equals(o2)
- assertNotNull(o) -- assert o != null
- assertNull(o) -- assert o == null
- assertSame(o1, o2) -- assert o1==o
- assertNotSame(o1, o2) -- assert o1 != o
• And many others
Manually Constructing a Test Suite
import org.junit.runner.RunWith; import org.junit.runners.Suite; @RunWith(Suite.class) @Suite.SuiteClasses( {Test1.class, Test2.class} ) public class AllTests {}
Java org.junit.runner.JUnitCore AllTests
