Download Error Tracking-Methods Of Software Engineering-Lecture Notes and more Study notes Software Engineering in PDF only on Docsity!
Lecture No. 26
Error Tracking
Error tracking can also be used to estimate the progress of the project. In this case we track errors in work products (requirement specifications, design documents, source code etc) to assess the status of a project. The process works as follows:
We collect error related metrics over many projects and determine our defect removal efficiency in the following manner:
Defect removal efficiency, DRE = E / (E+D), where
- E – errors found before shipment
- D – errors found during operation
It provides a strong indication of the effectiveness of the quality assurance activities.
Now let us assume that we have collected the following errors and defect data over the last 24 months:
- Errors per requirement specification page – Ereq
- Error per component – design level – Edesign
- Errors per component – code level – Ecode
- DRE – requirement analysis
- DRE – architectural design
- DRE – coding
We now record the number of errors found during each SE step and calculate current values for Ereq, Edesign, and Ecode. These values are then compared to averages of past projects. If the current results vary more than 20% from average, there may be cause for concern and there is certainly cause for investigation.
Example
- Ereq for the current project = 2.
- Organizational average = 3.
- Two possibilities
- The team has done an outstanding job
- The team has been lax in its review approach
- If the second scenario appears likely
- Build additional design time
This can also be used to better target review and/or testing resources in the following manner:
- 120 components
- 32 exhibit Edesign > 1.2 average
- Adjust code review resources accordingly
Time Boxing
Time-boxing is used in severe deadline pressure. It is a use incremental strategy where tasks associated with each increment are time-boxed in the following manner:
- Schedule for each task is adjusted by working backward from the delivery date.
- A box is put around each task
- When a task hits the boundary of the box, work stops and next task begins
The principle behind time-boxing is the 90-10 rule (similar to Pareto Principle) – rather than becoming stuck on the 10% of a task, the product proceeds towards the delivery date in 90% of the cases.
management. If data provided through QA identifies problems, the management deploys the necessary resources to fix it and hence achieves desired quality control.
Cost of quality
A very significant question is: does quality assurance add any value. That is, is worth spending a lot of money in quality assurance practices? In order to understand the impact of quality assurance practices, we have to understand the cost of quality (or lack thereof) in a system. Quality has a direct and indirect cost in the form of cost of prevention, appraisal, and failure. If we try to prevent problems, obviously we will have to incur cost. This cost includes:
- Quality planning
- Formal technical reviews
- Test equipment
- Training
We will discuss these in more detail in the later sections. The cost of appraisal includes activities to gain insight into the product condition. It involves in-process and inter-process inspection and testing. And finally, failure cost. Failure cost has two components: internal failure cost and external failure cost. Internal failure cost requires rework, repair, and failure mode analysis. On the other hand, external failure cost involves cost for complaint resolution, product return and replacement, help-line support, warranty work, and law suits. It is trivial to see that cost increases as we go from prevention to detection to internal failure to external failure. This is demonstrated with the help of the following example: Let us assume that a total of 7053 hours were spent inspecting 200,000 lines of code with the result that 3112 potential defects were prevented. Assuming a programmer cost of $ per hour, the total cost of preventing 3112 defects was $382,120, or roughly $91 per defect. Let us now compare these numbers to the cost of defect removal once the product has been shipped to the customer. Suppose that there had been no inspections, and the programmers had been extra careful and only one defect one 1000 lines escaped into the product shipment. That would mean that 200 defects would still have to be fixed in the field. As an estimated cost of $25000 per fix, the cost would be $5 Million or approximately 18 times more expensive than the total cost of defect prevention
That means, quality translates to cost savings and an improved bottom line.
SQA Activities
There are two different groups involved in SQA related activities:
- Software engineers who do the technical work
- SQA group who is responsible for QA planning, oversight, record keeping, analysis, and reporting Software engineers address quality by applying solid technical methods and measures, conducting formal and technical reviews, and performing well planned software testing. The SQA group assists the software team in achieving a high quality product.
SQA Group Activities
An SQA plan is developed for the project during project planning and is reviewed by all stake holders. The plan includes the identification of:
- Evaluations to be performed
- Audits and reviewed to be performed
- Standards that are applicable to the project
- Procedures for error reporting and tracking
- Documents to be produced by the SQA group
- Amount of feedback provided to the software project team
The group participates in the development of the project‘s software process description. The software team selects the process and SQA group reviews the process description for compliance with the organizational policies, internal software standards, externally imposed standards, and other parts of the software project plan. The SQA group also reviews software engineering activities to verify compliance with the defined software process. It identifies, documents, and tracks deviations from the process and verifies that the corrections have been made. In addition, it audits designated software work products to verify compliance with those defined as part of the software process. It, reviews selected work products, identifies, documents, and tracks deviations; verifies that corrections have been made; and reports the results of its work to the project manager. The basis purpose is to ensure that deviations in software work and work products are documented and handled according to documented procedures. These deviations may be encountered in the project plan, process description, applicable standards, or technical work products. The group records any non-compliance and reports to senior management and non-compliant items are recorded and tracked until they are resolved. Another very important role of the group is to coordinate the control and management of change and help to collect and analyze software metrics.
Quality Control
The next question that we need to ask is, once we have defined how to assess quality, how are we going to make sure that our processes deliver the product with the desired quality. That is, how are we going to control the quality of the product? The basic principle of quality control is to control the variation as variation control is the heart of quality control. It includes resource and time estimation, test coverage, variation in number of bugs, and variation in support. From one project to another we want to minimize the predicted resources needed to complete a project and calendar time. This involves a series of inspection, reviews, and tests and includes feedback loop. So quality control is a combination of measurement and feedback and combination of automated tools and manual interaction.
Importance of reviews
Technical work needs reviewing for the same reason that pencils needs erasers: To err is human. The second reason that we need technical reviews is although that people are good at catching errors, large class of errors escape the originator more easily than they escape anyone else.
Freedman defines a review – any review – as a way of using the diversity of a group of people to:
- Point out needed improvements in the product of a single person or team
- Confirm those parts of a product in which improvement is either not desired or no needed
- Achieve technical work of more uniform, or at least more predictable, quality than can be achieved without reviews, in order to make technical work more manageable.
Reviews help the development team in improving the defect removal efficiency and hence play an important role in the development of a high-quality product.
Types of Reviews
There are many types of reviews. In general they can be categorized into two main categories namely informal and formal technical reviews. Formal Technical reviews are sometimes called as walkthroughs or inspections. They are the most effective filter from QA standpoint. To understand the significance of these reviews, let us look at the defect amplification model shown below.
This model depicts that each development step inherits certain errors from the previous step. Some of these errors are just passed through to the next step while some are worked on and hence are amplified with a ratio of 1:x. In addition, each step may also generate some new errors. If each step has some mechanism for error detection, some of these errors may be detected and removed and the rest are passed on to the next step.
Let us now assume that we do not have any SQA related activities for the first two stages and we are only using testing for detection of any defects. Let us assume that the
Errors Passed Through Amplified Errors 1:x Newly generated errors
Percentage Efficiency For error detection
Errors passed To next step
Errors from previous step
Development Step
Defects Detection Errors Passed Through Amplified Errors 1:x Newly generated errors
Percentage Efficiency For error detection
Errors passed To next step
Errors from previous step
Development Step
Defects Detection
Preliminary design generated 10 defects which were passed on to Detailed design. At that phase, 6 defects were passed on to the next stages and 4 were amplified at a ration of 1:1.5. In addition, there were 25 new defects introduced at this stage. Therefore, a total of 37 defects were passed on to the next stage as shown in the diagram. In the Code and Unite test stage, we start to test our system and assuming 20% defect removal efficiency of this stage, 94 defects (80% of (10 + 27 * 3 + 25)) are passed on to the next stage. This process continues and the system is delivered with 12 defects remaining in the product.
If FTR are used in the earlier stages, the quality of the end-product is much better as shown in the following diagram. Note that in this case we have code inspection in addition to unit testing at the third stage and the defect removal efficiency of that stage is 60%.
Preliminary
Design
Detailed
Design
Code /Unit
test
Preliminary
Design
Detailed
Design
Code /Unit
test
Integration test
Validation test
System test
Integration test
Validation test
System test
Preliminary
Design
Detailed
Design
Code /Unit
test
Integration
test
Validation
test
System test
Preliminary
Design
Detailed
Design
Code /Unit
test
Integration
test
Validation
test
System test
During the FTR the recorder notes all the issues. They are summarized at the end and a review issue list is prepared. A summary report is produced that includes:
- What is reviewed
- Who reviewed it
- What were the findings and conclusions
It then becomes part of project historical record.
The review issue list
It is sometimes very useful to have a proper review issue list. It has two objectives.
- Identify problem areas within the WP
- Action item checklist
It is important to establish a follow-up procedure to ensure that items on the issue list have been properly addressed.
Review Guidelines
It is essential to note that an uncontrolled review can be worse than no review. The basis principle is that the review should focus on the product and not the producer so that it does not become personal. Remember to be sensitive to personal egos. Errors should be pointed out gently and the tone should be loose and constructive.
This can be achieved by setting an agenda and maintaining it. In order to do so, the review team should:
Avoid drift
- Limit debate and rebuttal
- Enunciate problem areas but don‘t try to solve all problems
- Take written notes
- Limit the number of participants and insist upon advanced preparation
- Develop a checklist for each product that is likely to be reviewed
- Allocate resources and schedule time for FTRs
- Conduct meaningful training for all reviewers
- Review your early reviews
- Determine what approach works best for you
