Download Testing and Debugging Concurrent Software in Java: A Look at the 2007 JavaOneSM Conference and more Study notes Programming Languages in PDF only on Docsity!
TS-
Testing Concurrent Software
Bill Pugh
Professor of Computer Science, University of Maryland
Brian Goetz
Senior Staff Engineer, Sun Microsystems
Cliff Click
Distinguished Engineer, Azul Systems
The Bottom Line
Testing concurrent software is difficult,
but not impossible.
By a combination of multiple techniques
(careful design, static analysis, code
review, extensive testing), you can get
the upper hand on concurrency bugs.
Some good news, some bad news
Agenda
Introduction Creating a Test Plan Unit Testing Concurrent Failure Modes Performance Testing System Testing Summary
Like testing sequential code...
Testing Concurrent Software
(^) Test cases for sequential code...
...may test safety or performance (or both)
...exercise code and assert invariants and
postconditions
...try to explore as much of the state space as possible
One rough measure of this is code coverage
...try to find combinations of inputs and actions that are
most likely to cause failure
(^) Test cases for concurrent code do the same
So we already know how to do it, right?
More extensive testing required
Testing Concurrent Software
(^) State space is much larger due to thread interactions (^) Need more intensive tests
Run for longer periods
Look for rare probabilistic failures
Account for impact of GC, JITing, etc
(^) Must test on multiple platforms
Different CPU architectures, JVMs, number of CPUs
Some tests don't happen on some architectures
(^) Tests must be written to avoid masking bugs
Design For Testability
(^) Where possible, separate concurrency logic from business and functional logic
Concurrency is challenging enough
Even harder when mixed in with your business logic!
(^) Isolate concurrency by extracting concurrent abstractions
Such as bounded buffers, semaphores, thread pools
Use the ones from java.util.concurrent where possible
Implement your own only if the provided ones don't fit
(^) Testing a single concurrent abstraction is a lot easier than testing an entire application
Concurrent programming is hard enough
Testing is only part of it
Building a QA Plan
(^) The goal of QA is not to “find all the bugs”
Because this is impossible
(^) Goal of QA is really to increase confidence (^) QA approaches include
Education, training, careful design
Understanding the concurrent design/implementation of what
you have
Manual code review
Static analysis (automated code review)
Testing
Unit tests, load tests, performance tests, system tests
Testing is only part of it
Building a QA Plan
(^) Testing can never show the absence of errors, only their presence
Even more true with rare probabilistic failures
(^) Testing, code review, and reviewing analysis reports are all subject to diminishing returns
Luckily, also tend to find different types of problems
(^) By combining them, you buy more confidence for your QA budget than testing alone
Automated code review
Static Analysis
(^) Analyzes a program without running it (^) Can check rules/patterns
Such as “Hold a lock consistently when accessing a
field”
(^) Annotations that document concurrency design are very helpful
For both humans and automatic tools
See Java Concurrency in Practice , FindBugs, and Fluid
from SureLogic
(^) See TS-2007: Improving Software Quality with Static Analysis
Lots of reasons to test...
Concurrent Testing Scenarios
(^) Unit testing functionality
Basic tests of safety and liveness (can be sequential)
(^) Unit testing functionality under concurrent stress
Looking for rare, timing-related interactions
Attempting to explore more of the state space
(^) Component performance testing
Evaluate performance or scalability of a concurrent
abstraction under varying load
(^) System stress testing
Test a large application to see if it works
Unit Testing
(^) Start with basic unit tests
Some tests can be sequential – goal is to establish that
documented sequential functionality works at all
Easier to debug basic functionality in sequential environment
(^) But many concurrent classes have behavior that cannot be tested with just one thread
Testing blocking behavior requires at least two threads
One thread that performs an operation that blocks
Another thread that then performs an action that unblocks the
first thread
Don't forget the basics
Unit Testing
Exchanger
Inherently requires two threads to exchange
CyclicBarrier
Inherently requires N threads to reach a barrier point
Lock
If one thread holds it, does it actually block other threads?
When holding thread releases it, can another acquire it?
BlockingQueue
Threads block if they try to add too many elements
Blocked threads unblock when room is made
Threads block if they try to remove nonexistent elements
Some behaviors require multiple threads to test
Unit Testing
void testPutThenTake() throws InterruptedException {
BoundedBlockingQueue buf
= new BoundedBlockingQueue( 1 );
buf.put(42);
assertEquals(42, buf.take());
void testPutPutTakeTake() throws InterruptedException {
BoundedBlockingQueue buf
= new BoundedBlockingQueue( 1 );
buf.put(42);
buf.put(17);
assertEquals(42, buf.take());
assertEquals(17, buf.take());
More framework support needed
This blocks and
can’t get unstuck
Unit Testing
void testPutPutTakeTake() throws InterruptedException {
final BoundedBlockingQueue buf
= new BoundedBlockingQueue( 1 );
Thread t = new Thread() {
public void run() {
assertEquals(42, buf.take());
assertEquals(17, buf.take());
t.start();
buf.put(42);
buf.put(17);
t.join();
More framework support needed
Won't compile; take() throws InterruptedException
Assertion
failure won't be
noticed by JUnit
