Download Software testing notes and more Study notes Software Engineering in PDF only on Docsity!
NOTES
SOFTWARE TSETING AND QUALITY ASSURANCE
UNIT-I
Testing as an engineering activity
This is an exciting time to be a software developer. Software systems are becoming more challenging to build. They are playing an increasingly important role in society. People with software development skills are in demand. New methods, techniques, and tools are becoming available to support development and maintenance tasks. Because software now has such an important role in our lives both economically and socially, there is pressure for softwa re professionals to focus on quality issues. Poor quality software that can cause loss of life or property is no longer acceptable to society. Failures can result in catastrophic losses. Conditions demand software development staffs with interest and training in the areas of software product and process quality. Highly qualified staff ensure that software products are built on time, within budget, and are of the highest quality with respect to attributes such as reliability, correctness, usability, and the ability to meet all user requirements. Using an engineering approach to software development implies that: The development process is well understood; Projects are planned; Life cycle models are defined and adhered to; sta ndards are in place for product and process; Measurements are employed to evaluate product and process quality; Components are reused; Validation and verification processes play a key role in quality determination; engineers have proper education, training, and certification.
Role of process in software quality
The need for software products of high quality has pressured those in the profession to identify and quantify quality factors such as usability, testability, maintainability, and reliability, and to identify engineering practices that support the production of quality products having these favorable attributes. Among the practices identified that contribute to the development of high-quality software are project planning, requirements management, development of formal specifications, structured design with use of information hiding and encapsulation, design and code reuse,inspections and reviews, product and process measures, education and training of software professionals, development and application of CASE tools, use of effective testing techniques, and integration of testing activities into the entire life cycle. In addition to identifying these individual best technical and managerial practices, software researchers realized that it was important to integrate them within the context of a high-quality software development process.
Process, in the software engineering domain, is the set of methods, practices, standards, documents, activities, policies, and procedures that software engineers use to develop and maintain a software system and its associated artifacts, such as project and test plans, design documents, code, and manuals.
It also was clear that adding individual practices to an existing software development process in an ad hoc way was not satisfactory. The software development process, like most engineering artifacts, must be engineered. That is, it must be designed, implemented, evaluated, and maintained. As in other engineering disciplines, a software development process must evolve in a consistent and predictable manner, and the best technical and managerial practices must be integrated in a systematic way. These models allow an organization to evaluate its current software process and to capture an understanding of its state. Strong support for incremental process improvement is provided by the models, consistent with historical process evolution and the application of quality principles. The models have received much attention from industry, and resources have been invested in process improvement efforts with many successes recorded. All the software process improvement models that have had wide acceptance in industry are highlevel models, in the sense that they focus on the software process as a whole and do not offer adequate support to evaluate and improve specific software development sub processes such as design and testing. Most software engineers would agree that testing is a vital component of a quality
process—one that reveals defects, as well as one that is used to evaluate quality attributes of the software such as reliability, security, usability, and correctness.
Also note that testing and debugging, or fault localization, are two very different activities. The debugging process begins after testing has been carried out and the tester has noted that the software is not behaving as specified.
Debugging, or fault localization is the process of (1) locating the fault or defect, (2) repairing the code, and (3) retesting the code.
Testing as a process has economic, technical and managerial aspects. Economic aspects are related to the reality that resources and time are available to the testing group on a limited basis. In fact, complete testing is in many cases not practical because of these economic constraints. An organization must structure its testing process so that it can deliver software on time and within budget, and also satisfy the client„s requirements. The technical aspects of testing relate to the techniques, methods, measurements, and tools used to insure that the software under test is as defect-free and reliable as possible for the conditions and constraints under which it must operate. Testing is a process, and as a process it must managed. Minimally that means that an organizational policy for testing must be defined and documented. Testing procedures and steps must be defined and documented. Testing must be planned, testers should be trained, the process should have associated quantifiable goals that can be measured and monitored. Testing as a process should be able to evolve to a level where there are mechanisms in place for making continuous improvements.
Software testing principles
Principles play an important role in all engineering disciplines and are usually introduced as part of an educational background in each branch of engineering. Figure 1.1 shows the role of basic principles in various engineering disciplines. Testing principle s are important to test specialists/ engineers because they provide the foundation for developing testing knowledge and acquiring testing skills. They also provide guidance for defining testing activities as performed in the practice of a test specialist.A principle can be defined as:
- a general or fundamental, law, doctrine, or assumption;
- a rule or code of conduct;
- the laws or facts of nature underlying the working of an artificial device.
Extending these three definitions to the software engineering domain we can say that software engineering principles refer to laws, rules, or doctrines that relate to software systems, how to build them, and how they behave. In the software domain, principles may also refer to rules or codes of conduct relating to professionals who design, develop, test, and maintain software systems. Testing as a component of the software engineering discipline also has a specific set of principles that serve as guidelines for the tester. They guide testers in defining how to test software systems, and provide rules of conduct for testers as professionals. Glenford Myers has outlined such a set of execution-based testing principles in
his pioneering book, The Art of Software Testing [9]. Some of these principles are described below. Principles 1-8, and 11 are derived directly from Myers„ original set. The author has reworded these principles, and also has made modifications to the original set to reflect the evolution of testing from an art, to a quality-related process within the context of an engineering discipline. Note that the principles as stated below only relate to execution-based testing. Principles relating to reviews, proof of correctness, and certification as testing activities are not covered.
Principle 1. Testing is the process of exercising a software component using a selected set of test cases, with the intent of (i) revealing defects, and (ii) evaluating quality. Software engineers have made great progress in developing methods to prevent and eliminate defects. However, defects do occur, and they have a negative impact on software quality. Testers need to detect these defects before the software becomes operational. This principle supports testing as an execution-based activity to detect defects. It also supports the separation of testing from debugging since the intent of the latter is to locate defects and repair the software. The term ―software component‖ is used in this context to represent any unit of software ranging in size and complexity from an individual procedure or method, to an entire software system. The term ―defects‖ as used in this and in subsequent principles represents any deviations in the software that have a negative impact on its functionality, performance, reliability, security, and/or any other of its specified quality attributes.
Principle 2. When the test objective is to detect defects, then a good test case is one that has a high probability of revealing a yetundetected defect(s).
Principle 2 supports careful test design and provides a criterion with which to evaluate test case design and the effectiveness of the testing effort when the objective is to detect defects. It requires the tester to consider the goal for each test case, that is, which specific type of defect is to be detected by the test case. In this way the tester approaches testing in the same way a scientist approaches an experiment. In the case of the scientist there is a hypothesis involved that he/she wants to prove or disprove by means of the experiment. In the case of the tester, the hypothesis is related to the suspected occurrence of specific types of defects. The goal for the test is to prove/disprove the hypothesis, that is, determine if the specific defect is present/absent. Based on the hypothesis, test inputs are selected, correct outputs are determined, and the test is run. Results are analyzed to prove/disprove the hypothesis. The reader should realize that many resources are invested in a test, resources for designing the test cases, running the tests, and recording and analyzing results. A tester can justify the expenditure of the resources by careful test design so that principle 2 is supported.
Principle 3. Test results should be inspected meticulously.
Testers need to carefully inspect and interpret test results. Several erroneous and costly scenarios may occur if care is not taken. For example: A failure may be overlooked, and the test may be granted a ―pass‖ status when in reality the software has failed the test. Testing may continue based on erroneous test results. The defect may be revealed at some later stage of testing, but in that case it may be more costly and difficult to locate and repair.
current version of the component and work on a redesign, or plan to expend additional testing resources on this component to insure it meets its requirements. This issue is especially important for components that implement mission or safety critical functions.
Principle 7. Testing should be carried out by a group that is independent of the development group.
This principle holds true for psychological as well as practical reasons. It is difficult for a developer to admit or conceive that software he/she has created and developed can be faulty. Testers must realize that (i) developers have a great deal of pride in their work, and (ii) on a practical level it may be difficult for them to conceptualize where defects could be found. Even when tests fail, developers often have difficulty in locating the defects since their mental model of the code may overshadow their view of code as it exists in actuality. They may also have misconceptions or misunderstandings concerning the requirements and specifications relating to the software. The requirement for an independent testing group can be interpreted by an organization in several ways. The testing group could be implemented as a completely separate functional entity in the organization. Alternatively, testers could be members of a Software Quality Assurance Group, or even be a specialized part of the development group, but in the latter case especially, they need the capability to be objective. Reporting management that is separate from development can support their objectivity and independence. As a member of any of these groups, the principal duties and training of the testers should lie in testing rather than in development. Finally, independence of the testing group does not call for an adversarial relationship between developers and testers. The testers should not play ―gotcha‖ games with developers. The groups need to cooperate so that software of the highest quality is released to the custome r.
Principle 8. Tests must be repeatable and reusable.
Principle 2 calls for a tester to view his/her work as similar to that of an experimental scientist. Principle 8 calls for experiments in the testing domain to require recording of the exact conditions of the test, any special events that occurred, equipment used, and a careful accounting of the results. This information is invaluable to the developers when the code is returned for debugging so that they can duplicate test conditions. It is also useful for tests that need to be repeated after defect repair. The repetition and reuse of tests is also necessary during regression test (the retesting of software that has been modified) in the case of a new release of the software. Scientists expect experiments to be repeatable by others, and testers should expect the same!
Principle 9. Testing should be planned.
Test plans should be developed for each level of testing, and objectives for each level should be described in the associated plan. The objectives should be stated as quantitatively as possible. Plans, with their precisely specified objectives, are necessary to ensure that adequate time and resources are allocated for testing tasks, and that testing can be monitored and managed. Test planning activities should be carried out throughout the software life cycle (Principle 10). Test planning must be coordinated with project planning. The test manager
and project manager must work together to coordinate activities. Testers cannot plan to test a component on a given date unless the developers have it available on that date. Test risks must be evaluated. For example, how probable are delays in delivery of software components, which components are likely to be complex and difficult to test, do the testers need extra training with new tools? A test plan template must be available to the test manager to guide development of the plan according to organizational policies and standards. Careful test planning avoids wasteful ―throwaway‖ tests and unproductive and unplanned ―test- patch-retest‖ cycles that often lead to poor-quality software and the inability to deliver software on time and within budget.
Principle 10. Testing activities should be integrated into the software life cycle.
It is no longer feasible to postpone testing activities until after the code has been written. Test planning activities as supported by Principle 10, should be integrated into the software life cycle starting as early as in the requirements analysis phase, and continue on throughout the software life cycle in parallel with development activities. In addition to test planning, some other types of testing activities such as usability testing can also be carried out early in the life cycle by using prototypes. These activities can continue on until the software is delivered to the users. Organizations can use process models like the V-model or any others that support the integration of test activities into the software life cycle.
Principle 11. Testing is a creative and challenging task.
Difficulties and challenges for the tester include the following:
- A tester needs to have comprehensive knowledge of the software engineering discipline.
- A tester needs to have knowledge from both experience and education as to how software is specified, designed, and developed.
- A tester needs to be able to manage many details.
- A tester needs to have knowledge of fault types and where faults of a certain type might occur in code constructs.
- A tester needs to reason like a scientist and propose hypotheses that relate to presence of specific types of defects.
- A tester needs to have a good grasp of the problem domain of the software that he/she is testing. Familiarly with a domain may come from educational, training, and work-related experiences.
- A tester needs to create and document test cases. To design the test cases the tester must select inputs often from a very wide domain.
Basic definitions
Errors
An error is a mistake, misconception, or misunderstanding on the part of a software developer.In the category of developer we include software engineers, programmers, analysts, and testers. For example, a developer may misunderstand a design notation, or a programmer might type a variable name incorrectly.
Faults (Defects)
A fault (defect) is introduced into the software as the result of an error. It is an anomaly in the software that may cause it to behave incorrectly, and not according to its specification.
Faults or defects are sometimes called ―bugs.‖ Use of the latter term trivializes the impact faults have on software quality. Use of the term ―defect‖ is also associated with software artifacts such as requirements and design documents. Defects occurring in these artifacts are also caused by errors and are usually detected in the review process.
Failures
A failure is the inability of a software system or component to perform its required functions within specified performance requirements. During execution of a software component or system, a tester, developer, or user observes that it does not produce the expected results. In some cases a particular type of misbehavior indicates a certain type of fault.
Test case
A test case in a practical sense is a test-related item which contains the following information:
- A set of test inputs. These are data items received from an external source by the code under test. The external source can be hardware, software, or human.
- Execution conditions. These are conditions required for running the test, for example, a certain state of a database, or a configuration of a hardware device.
- Expected outputs. These are the specified results to be produced by the code under test.
Test
A test is a group of related test cases, or a group of related test cases and test procedures.
Test Oracle
A test oracle is a document, or piece of software that allows testers to determine whether a test has been passed or failed.
Limitations of Testing:
- Testing can show presence of errors-not their absence.
- No matter how hard you try, you would never find the last bug in an application.
- The domain of possible inputs is too large to test.
- There are too many possible paths through the programs to test.
- In short, maximum coverage through minimum test-cases. That is the challenge of testing.
- Various testing techniques are complementary in nature and it is only through their combined use that one can hope to detect most errors.
Testing and Debugging:
Testing activity is carried down by a team of testers, in order to find the defect in the software. Test engineers run their tests on the piece of software and if they encounter any defect (i.e. actual results don't match expected results), they report it to the development team. Along with the nature of defect, testers also have to report at what point the defect occurred and what happened due the occurrence of that defect. All this information will be used by development team to DEBUG the defect. (Read more about Defect Life Cycle)
Debugging is the activity which is carried out by the development team (or developer), after getting the test report from the testing team about defect(s) (you may note defects can also be reports by the client). The developer then tries to find the cause of the defect, in this quest he may need to go through lines of code and find which part of code in causing that defect. After finding out the bug, he tries to modify that portion of code and then he rechecks if the defect has been finally removed. After fixing the bug, developers send the software back to testers.
It uses methods like
inspections, walk
throughs, Desk-
checking etc.
It uses methods like black box,
gray box, white box testing etc.
It, generally, comes
first- done before
validation.
It generally follows verification.
It answers to the
question- Are we
building the product
right?
It answers to the question- Are
we building the right product?
It can catch errors that
validation cannot catch.
It can catch errors that
verification cannot catch.
Test Levels:
Testing levels are basically to identify missing areas and prevent overlap and repetition between the development life cycle phases. In software development life cycle models there are defined phases like requirement gathering and analysis, design, coding or implementation, testing and deployment. Each phase goes through the testing. Hence there are various levels of testing. The various levels of testing are:
- Unit testing: It is basically done by the developers to make sure that their code is working fine and meet the user specifications. They test their piece of code which they have written like classes, functions, interfaces and procedures.
- Component testing: It is also called as module testing. The basic difference between the unit testing and component testing is in unit testing the developers test their piece of code but in component testing the whole component is tested. For example, in a student record application there are two modules one which will save the records of the students and other module is to upload the results of the students. Both the modules are developed separately and when they are tested one by one then we call this as a component or module testing.
- Integration testing: Integration testing is done when two modules are integrated, in order to test the behavior and functionality of both the modules after integration. Below are few types of integration testing:
Big bang integration testing
Top down
Bottom up
Functional incremental
- Component integration testing: In the example above when both the modules or components are integrated then the testing done is called as Component integration testing. This testing is basically done to ensure that the code should not break after integrating the two modules.
- System integration testing: System integration testing (SIT) is a testing where testers basically test that in the same environment all the related systems should maintain data integrity and can operate in coordination with other systems.
- System testing: In system testing the testers basically test the compatibility of the application with the system.
- Acceptance testing: Acceptance testing are basically done to ensure that the requirements of the specification are met.
- Alpha testing: Alpha testing is done at the developers site. It is done at the end of the development process
- Beta testing: Beta testing is done at the customers site. It is done just before the launch of the product.
Software Quality
- Quality relates to the degree to which a system, system component, or process meets specified requirements.
- Quality relates to the degree to which a system, system component, or process meets customer or user needs, or expectations.
In order to determine whether a system, system component, or process is of high quality we use
what are called quality attributes. the degree to which they possess a given quality attribute with quality metrics.
Quality Control and Quality Assurance:
- Quality Assurance: The planned and systematic activities, implemented in a quality system so that the quality requirements for a product or service will be fulfilled, is known as quality assurance.
- Quality Control: The observation techniques and activities used to fulfill requirements for quality is known as quality control.
Quality Assurance (QA) Quality Control (QC)
It is process related. It is product related.
It focuses on the process use d to develop a product.
It focuses on testing of a product developed or a product under development.
It involves the quality of the processes It involves the quality of the products
It is a preventive control It is a detective control.
Allegiance is to development Allegiance is not to development.
Quality Assuurance Analyst:
A software quality analyst is responsible for applying the principles and practices of software quality assurance throughout the software development life cycle. Though often referred to as "quality assurance", software testing is considered to be only one part of the larger process of reducing errors.[1]^ Testing is used to detect errors in a product; software quality assurance also fixes the processes that resulted in those errors.
Roles in Software Quality Assurance are designated as Software Quality Assurance Engineer, Software Quality Analyst, and Process Consultant.[ citation needed ]
Some of the tasks of software quality analysts are:[ citation needed ]
Participate in QMS management review meetings.
Approves the QMS documents of which QA manager is the main author.
Undertake internal quality audits.
Plan and conduct quality audits of subcontractors.
Maintain and update training and audit databases to provide reports / indicators for discussion during QMS management review meetings
Identify problems or deficiencies in products and QMS.
Resolve problems QMS implementation issues.
Perform Internal Quality Audit.
Participate in External Quality Audit.
Track all the software related metrics in terms of schedule,effort,defects etc.
Review the model followed by project.
Software Quality Factors
A software quality factor is a non-functional requirement for a software program which is not called up by the customer's contract, but nevertheless is a desirable requirement which enhances the quality of the software program.
Some software quality factors are listed here:
- Understandability is possessed by a software product if the purpose of the product is clear. This goes further than just a statement of purpose - all of the design and user documentation must be clearly written so that it is easily understandable. This is obviously subjective in that the user context must be taken into account, i.e. if the software product is to be used by software engineers it is not required to be understandable to the layman.
- A software product possesses the characteristic completeness to the extent that all of its parts are present and each of its parts is fully developed. This means that if the code calls a sub-routine from an external library, the software package must provide reference to that library and all required parameters must be passed. All required input data must be available.
- A software product possesses the characteristic conciseness to the extent that no excessive information is present. This is important where memory capacity is limited, and it is important to reduce lines of code to a minimum. It can be improved by replacing repeated functionality by one sub-routine or function which achieves that functionality. It also applies to documents.
Activities of Software Quality Management:
Quality Assurance - QA aims at developing Organizational procedures and standards for quality at Organizational level.
Quality Planning - Select applicable procedures and standards for a particular project and modify as required to develop a quality plan.
Quality Control - Ensure that best practices and standards are followed by the software development team to produce quality products.
Components of Quality:
Most software quality assurance activities can be categorized into software testing (that is, verification and validation), software configuration management, and quality control. However, the success of a software quality assurance program also depends on a coherent collection of standards, practices, conventions, and specifications, as shown in Figure below.
Software testing is a popular risk management strategy. It is used to verify that functional requirements were met. The limitation of this approach, however, is that by the time testing occurs, it is too late to build quality into the product. Tests are only as good as the test cases, but they can be inspected to ensure that all the requirements are tested across all possible combinations of inputs and system states. However, not all defects are discovered during testing. Software testing includes the activities outlined in this text, including verification and validation activities. In many organizations, these activities, or their supervision, are included within the charter for the software quality assurance function. The extent to which personnel independent of design and coding should participate in software quality assurance activities is a matter of institutional, organizational, and project policy.
The major purpose of verification and validation activities is to ensure that software design, code, and documentation meet all the requirements imposed on them. Examples of requirements include user requirements; specifications derived from and designed to meet user requirements; code review and inspection criteria; test requirements at the modular, subsystem, and integrated software levels; and acceptance testing of the code after it has been fully integrated with hardware.
During software design and implementation, verification helps determine whether the products of one phase of the software development life cycle fulfill the requirements established during the previous phase. The verification effort takes less time and is less complex when conducted throughout the development process.
Cost Aspect of Quality:
DEFINITION
Cost of Quality (COQ) is a measure that quantifies the cost of control/conformance and the cost of failure of control/non-conformance. In other words, it sums up the costs related to prevention and detection of defects and the costs due to occurrences of defects.
Definition by ISTQB: cost of quality: The total costs incurred on quality activities and issues and often split into prevention costs, appraisal costs, internal failure costs and external failure costs.
Definition by QAI: Money spent beyond expected production costs (labor, materials, equipment) to ensure that the product the customer receives is a quality (defect free) product. The Cost of Quality includes prevention, appraisal, and correction or repair costs.
EXPLANATION
Cost of Control (Also known as Cost of Conformance)
o Prevention Cost
The cost arises from efforts to prevent defects.
Example: Quality Assurance costs
o Appraisal Cost
The cost arises from efforts to detect defects.
Example: Quality Control costs
Cost of Failure of Control (Also known as Cost of Non-Conformance)
o Internal Failure Cost
