Embedded Software Testing: Kinds, Levels, and Strategies, Study notes of Computer Science

Download Embedded Software Testing: Kinds, Levels, and Strategies and more Study notes Computer Science in PDF only on Docsity!

Stuff

‹^

Labs 1 and 2 graded ‹^

Next week is fall break ‹^

Week after break, homework 2 and lab 3 due ‹^

Week after that, exam 1 ‹^

Lab 4 assigned after the exam

Last Time

‹^

Software perspective on power and energymanagement ‹^

Mechanisms are provided by the HW people^ ¾^

Frequency scaling ¾ Voltage scaling ¾ Sleep modes ‹^

Analysis of HW + workload can give us ballparkestimate of whether there is a good match ‹^

Policies are up to software^ ¾^

But it’s often difficult to balance power, performance, andusers’ convenience

‹^

Fact: Most multithreaded Java programs run all oftheir test cases perfectly well when all locking isremoved^ ¾^

What does this mean?

Testing

‹^

Testing is the fundamental way that reliableembedded software is created ‹^

However, good testing techniques are neither easyor intuitive ‹^

Lots of basic questions:^ ¾^

When to test? ¾ Who tests? ¾ Where do test cases come from? ¾ How to evaluate the result of a test? ¾ How much testing is enough?

Kinds of Tests

‹^

Functionality – testing functional behavior ‹^

Interfaces – testing interaction with other systems ‹^

Security – test for robustness to intrusion ‹^

Standards – check for compliance ‹^

Regression^ ¾^

Testing whether everything works after the system has beenchanged ¾ Test cases derived from prior failures ‹^

Resources – measuring required resources such asCPU time, memory, network bandwidth ‹^

Load and stress – trying to overload the system

Test Levels

‹^

Hardware unit test ‹^

Hardware integration test ‹^

Software unit test ‹^

Software integration test ‹^

HW/SW integration test ‹^

System test ‹^

Acceptance test ‹^

Field test

When to Test

‹^

Every combination of kind of test and test levelshould be run as early as is feasible ‹^

Basic fact: Cost to fix a bug increases greatly asdevelopment progresses

Testing by Developers

‹^

Why?^ ¾^

Defects cheaper to fix when found earlier ¾ High-quality parts make integration possible ¾ Defects found late are hard to map back to the source code ¾ Some kinds of tests are only possible at the unit level ¾ Developers understand the implementation, which can leadto better test cases ‹^

Quality cannot be added at the end of development^ ¾^

Has to be there from the start

Integration Testing Strategies ‹^ Bottom-up

¾^ Start with low-level modules with few dependencies ¾^ Exercise them using drivers ‹^

Top-down^ ¾^

Overall control structure drives tests ¾ Stubs provided for nonexistant modules ¾ “Look and feel” of the system established early ‹^

Big-bang^ ¾^

Only works for small systems ¾ Useful for tightly coupled systems where top-down andbottom-up are difficult

Design for Test

‹^

Term most often used in context of hardware^ ¾^

Also applies to software ‹^

How to do this?^ ¾^

Lots of assertions for preconditions and postconditions ¾ Implement self-tests ¾ Provide test scaffolding along with code ¾ Expose all interfaces for testing ‹^

Examples – how would you design these for test?^ ¾^

Code to set PLL ¾ Code responding to an external interrupt source

Test Coverage

‹^

Coverage metrics try to answer the question: Howcan we know when to stop testing? ‹^

Example metrics:^ ¾^

Function coverage – are all functions executed? ¾ Statement coverage – are all statements executed? ¾ Branch coverage – is every possible decision executed atevery branch? ¾ Path coverage – is every path through the code executed? ¾ Value coverage – is the full range of every variable covered? ¾ Mutation coverage – are all variants of the programcovered? ¾ Exception coverage – are all exceptions signaled? ‹^

In most cases goal is 100% coverage

Evaluating Coverage Metrics ‹^ Coverage metric must be understood by the user ‹^ Near-complete coverage must be achievable^ ¾

Exceptions require fixing or manual review ‹^

Some action should be taken upon reaching 100%coverage

Stress Testing

‹^

Test system at the limits of (and outside) its loadparameters^ ¾^

Intuition: This exposes different kinds of problems thanregular test cases do ‹^

Examples – how would you stress test:^ ¾^

Embedded web server ¾ An RTOS ¾ A cell phone ‹^

Tricky problem: Thinking of as many sources ofstress as possible

Stress Testing for Interrupts ‹ What bugs are we trying to find? ‹ How to do it?^ ¾

What if data comes along with the interrupt? ‹^

How to tell when we’re done?