Download Testing in Programming: Techniques and Approaches and more Study notes Programming Languages in PDF only on Docsity!
CMSC 433 – Programming Language
Technologies and Paradigms
Spring 2005
Testing
Some slides adapted from FSE’98 Tutorial by Michal Young and Mauro Pezze’
Testing
• Execute program on sample input data
- Check if output correct (acceptable)
• Goals
- Increase confidence program works correctly
- Find bugs in program
3
Simple Example
% java TestServlet HelloWorld /FooBar/Test > out HTTP/1.0 200 Content-Type: text/plain Hello /FooBar/Test % diff out expectedOutput
Limitations of Testing
• Program runs on (very small) subset of input data
- Exhaustive testing usually impossible
- Too large input space (possibly infinite)
• Many situations hard to test
- Parallel code (due to non-determinism)
- Hard-to-reach states (e.g., error states)
- Inadequate test environment (e.g., lack of hardware)
• Testing cannot prove absence of bugs
- Especially a problem in security
7
The Test Case Generation Problem
• How to choose tests that will show that my
program works?
- Must consider “operational scenarios”
- What is legitimate input?
- What is the correct action or output?
- Must consider “abnormal behaviors” as well
• How can I make sure that all of the important
behaviors of my program have been tested?
Test Cases via Specifications
// Return true if x in a, else returns false boolean contains(int[] a, int x);
• Two “paths” in specification
- Test case where x is in a
- Test case where x is not in a
9
Test Cases via Inferred Implementation
• Think about how the implementation might look
- Test by boundary condition
- What test cases are likely to exercise the same
logic?
- Want to avoid redundant tests, to save time
- Test by common mistake
- What cases my be tricky to implement?
• At the same time, tests should still be
implementation-independent
Test Cases via Boundary Conditions
interface List { ... Inserts the specified element at the specified position in this list (optional operation). Shifts the element currently at that position (if any) and any subsequent elements to the right (adds one to their indices). public void add(int index, Object element) }
• Test with empty list
• Test with index at first/last element
• Others?
13
Statement Coverage
One test case (n=1, a[0]=-7, x=9) covers all statements Faults handling positive values of a[i] not revealed int select(int[] a, int n, int x) { int i=0; while (i<n && a[i] <x) { if (a[i]<0) a[i] = - a[i]; i++; } return 1; } i++ i<n and a[i] <x a[i]< a[i] = - a[i]; return 1 true false true false i=
Branch Coverage
i= i<n and a[i] <x a[i]< a[i] = - a[i]; return 1 true false true false Must add test case (n=1, a[0]=7, x=9) to cover false branch of if Faults handling positive values of a[i] revealed. Faults exiting the loop with a[i] <x not revealed int select(int[] a, int n, int x) { int i=0; while (i<n && a[i] <x) { if (a[i]<0) a[i] = - a[i]; i++; } return 1; } i++
15
Condition Coverage
i= i<n and a[i] <x a[i]< a[i] = - a[i]; return 1 true false true false Both i<n and a[i]<x must be false and true for different tests. Must add tests that cause loop to exit for a value greater than X. Faults that arise after several loop iterations not revealed. int select(int[] a, int n, int x) { int i=0; while (i<n && a[i] <x) { if (a[i]<0) a[i] = - a[i]; i++; } return 1; } (^) i++
Structural Coverage Testing
• Adequacy criteria
- If significant parts of program structure are not tested,
testing is surely inadequate
• Control flow coverage criteria
- Statement (node, basic block) coverage
- Branch (edge) coverage
- Condition coverage
• Attempted compromise between the impossible
and the inadequate
19
Testing Activities
• Test case execution is only a part of the process
• Must also consider
- Test case generation
- Test result evaluation
• Planning is essential
- To achieve early and continuous visibility
- To choose appropriate techniques at each stage
- To build a testable product
- To coordinate complementary analysis and testing
The Testing Environment
• Want to create a scaffold for executing tests
- Code infrastructure to run tests and check output
• Many benefits
- Can automate testing process
- Useful for regression testing
• But, can take some time to implement
21
Testing Environment Components
• A user to generate input for tested component
• An oracle for verifying the results are correct
• These two may be combined into a single system
Unit Testing with Junit
• Testing environment for writing black-box tests
- Write special TestCase classes to test other classes
- Several ways to use/set up test cases
• Can be downloaded from
25
To Execute Tests within a Class
• Pick a Test Runner:
- junit.awtui.TestRunner – Graphical
- junit.swingui.TestRunner – Graphical
- junit.textui.TestRunner – Textual
• Invoke on the test case class
> java junit.textui.TestRunner ListTest .. Time: 0. OK (2 tests)
Junit Components
• Test cases (class TestCase )
- Individual tests
- Can reuse test case setup (optional)
• Test suites (class TestSuite , not used in Eclipse)
• Test runner (various classes)
- Executes test suites and presents results
- Can also execute tests within Eclipse
27
Each Test Has Three Parts
• Code that creates test objects
- Create a subclass of junit.framework.TestCase
• Code that executes the test
- Override the method runTest() (which executes the test)
• Code that verifies the result
- E.g., use junit.framework.assertTrue() to check results
(throws exception is test fails)
Setup/Teardown
• Creating objects for each test insufficient
- Setup overhead grows as number of tests grows
- Instead, group setup (and teardown) code in one place
and reuse
• junit.framework.TestCase.run() executes test case:
- public void run () { setUp(); runTest(); tearDown(); }
- Do not override this method!
- Put setup code in setUp() method
- Put cleanup code in tearDown() method
31
Manually Constructing a Test Suite
public class ListTest extends TestCase { ... public static Test suite() { TestSuite suite = new TestSuite(); suite.addTest(new ListTest() { protected void runTest() { testAdd(); } }); suite.addTest(new ListTest() { protected void runTest() { testPushPop();} }); return suite; } }
Manually Constructing a Suite (cont’d)
• You can also create test suites more easily:
public static Test suite() { TestSuite suite = new TestSuite(); suite.addTest(new ListTest(“testAdd”)); suite.addTest(new ListTest(“testPushPop”)); return suite; }
• Or simply:
public static Test suite() { return new TestSuite(ListTest.class); }
Using a Test Suite
• Test runners will use static suite() method
• If no suite() method, suite selected automatically
- Every method that is public , returns void , takes no
arguments, and begins with “test”
- This is the way to go – for project 2, use this style
• Then use junit.*.TestRunner TestClass
