Testing Policy: Types, Levels, and Documentation, Schemes and Mind Maps of Designs and Groups

Download Testing Policy: Types, Levels, and Documentation and more Schemes and Mind Maps Designs and Groups in PDF only on Docsity!

Wouter Van Caneghem Functional Analyst - Fedict

Dirk Dussart Architect - Fedict

Contents

Date Version Description Author 11/10/2011 1.0 Initial draft version Wouter Van Caneghem 17/10/2011 1.1 Addition of chapter entitled “Tools” Wouter Van Caneghem 21/10/2011 1.2 Addition of roles and responsibilities Wouter Van Caneghem 17/11/2011 1.3 Addition of unit testing Diederik Delen 12/12/2011 1.4 Addition of test matrix Wouter Van Caneghem 13/12/2011 1.5 Addition of staging criteria Wouter Van Caneghem 27/02/2012 1.6 Update of staging criteria and addition of annex

Wouter Van Caneghem

05/03/2012 1.7 Update of test matrix Dirk Dussart 13 /04/2012 1.8 Review of Coralius nv Coralius nv 14/05/2012 1.9 Amendment to the severities Wouter Van Caneghem

1. Introduction

1.1. PURPOSE OF THIS DOCUMENT

The purpose of this document is to provide a clear statement as to Fedict’s requirements with regard to the testing process for software and to outline who is responsible for which parts of that process. The requirements stated in this document shall apply to all projects. In the specific contract documents or requirement documents, however they may be adapted in line with the needs that apply to the particular project or product concerned.

An additional purpose of this document is to help the supplier to provide a product of the highest possible quality from the first delivery onwards, thereby creating a win-win situation for both parties. In doing this, our aim is to ensure that the effort expended by Fedict for the purpose of testing and retesting and the effort expended by the supplier for the purpose of reworking the product is kept to a minimum.

Chapter 2 contains an overall summary of the testing process, based on the V-model. In that part of the document, we will indicate which test levels lie within the scope of responsibility of Fedict and which ones are the responsibility of the software supplier.

In Chapter 3, we will provide a summary of the deliverables to be provided, together with details of the requirements with which these are required to comply.

In Chapter 4, we will provide additional recommendations regarding the way in which a supplier will be able to fulfil the requirements set, without seeking to impose a specific development cycle or methodology.

In Annex 1, we will provide a list of templates that form part of this policy and will explain how these can be used.

1.2. WHY IS TESTING REQUIRED?

Testing forms an essential precondition to ensure the successful construction and implementation of information systems. The complexity of modern-day software is such that it is almost impossible to implement it correctly the first time around, without any form of verification.

Testing is needed in order to detect potential problems within the software as early as possible, so that they can be corrected at minimum cost.

A second reason to carry out tests is to develop trust in and a knowledge of the product provided.

Defects that exist within a software product can have severe consequences for the “business” and the users alike. Whilst providing a means of avoiding faults as much as possible, testing is also a useful way of demonstrating to management and users that the product supplied fulfils their requirements (is “fit for purpose”).

• Principle 4: Clustering of defects.

A small number of the modules contain the largest number of defects discovered during the pre-release tests and/or are responsible for the most operational errors.

• Principle 5: The pesticides paradox.

If the same set of test cases are carried out once again each time, there will come a time when they no longer reveal any defects. That is the reason why the test cases need to be re-examined on a regular basis. New tests must be written in order to verify different parts of the software, so that new defects may be discovered.

• Principle 6: Testing is context-dependent.

The test method employed will depend on the context in which the product will ultimately be used. For example, security software will be tested in a different way to an e-commerce website.

• Principle 7: The absence-of-errors fallacy.

Tracing and rectifying defects will be of no benefit if the system is unusable and/or does not fulfil the needs and expectations of the end-users.

2. The testing process and overall

responsibilities

2.1. THE FEDICT V MODEL

The Fedict V model illustrates the levels of testing used, in addition to the overall responsibilities of Fedict and of the supplier.

2.2. LEVELS OF TESTING

Levels of testing take the form of groups of test activities that are collectively managed and directly related to the responsibilities defined as part of a project.

Fedict applies the testing levels as described in the rest of this section.

If a supplier uses different names or different testing levels, he must be able to demonstrate how the terminology he himself uses relates to the terminology used by Fedict. He must also be able to demonstrate that his approach is, at the very least, equivalent to the approach described by Fedict.

Functional

documents

Design

documents

Acceptance

test

System

integration test

System test

Component

integration test

Unit/Component test

Requirements, Business

Analysis & Contract

Document

SUPPLIER

Tests must be clearly named. The method or test name must describe what the test sets out to do. Names such as test1 or test2 are unacceptable. It is also important to remember that Fedict must be able to verify the unit tests.

Unit tests must also be independent of one another and it must be possible for every one of them to be conducted separately.

2.2.3. Integration test level

As far as the integration tests are concerned, these set out to test the interfaces between the various components and the interactions with different parts of a system (such as the control system, file system, hardware etc.). Integration tests can exist in more than one level and they can be carried out on test items of various sizes.

We are able to distinguish between 2 separate levels of integration test:

• Component integration tests: The testing of the interaction between software components. These are carried out following the unit tests.
• System integration tests: The testing of the interaction between various systems. These are carried out following the system tests.

The greater the scope of integration, the more difficult it becomes to isolate defects that exist in a specific component or system.

At any point in the integration, the tests will concentrate solely on integration itself. For example: A new module A is integrated with a module B. During the integration tests, the primary focus will lie on the communication between the modules and not on the functionality of the modules themselves.

2.2.4. System test level

In the case of system tests, our aim is to test the behaviour of an entire system, as defined within the project. For the system tests, no knowledge is required, a priori of the internal design or logic of the system; the majority of techniques used are primarily black box techniques. In the case of the system tests, the environment must correspond as closely as possible with the ultimate production environment, in order to minimise the risk of environment-specific defects. System tests are derived from risk specifications, requirement specifications, business processes, use-cases or other high-level definitions and are carried out within the context of functional requirements. What is more, system tests will also investigate the non-functional requirements (quality attributes).

System tests typically include the following types of test: GUI tests, usability tests, regression tests, performance tests, error-handling tests, maintenance tests, compatibility tests, load tests, security tests, scalability tests, etc.

The intention is that once the system test has been carried out, the supplier is satisfied that the system complies with all of the necessary requirements and is ready to be handed on to the customer (possibly with the exception of interaction with peer systems).

2.2.5. Acceptance test level

Acceptance tests are often the responsibility of the customers or (end-)users of a system. Other stakeholders can also be involved, however. The purpose of acceptance testing is to gain confidence in the system, parts of the system or specific non-functional properties. Locating defects is not the primary aim of acceptance tests. Acceptance tests are a means of checking whether a system is ready for use.

Acceptance tests can occur at more than just one test level. For example:

• Acceptance testing of the usability of a component can take place during unit testing.
• Acceptance testing of a new functional improvement can take place before the system tests.

Acceptance tests typically involve the following elements, which may also be regarded as a separate test level:

• User acceptance tests: Verifying the readiness of a system for use by business users
• Operational acceptance tests: Testing the acceptance of a system by system administrators. These consist of tests of backup/restore, disaster recovery, user management, maintenance tasks etc.
• Contractual and guideline tests (conformance test): The acceptance criteria must be agreed during the contract negotiations. Acceptance then takes place on the basis of those criteria relating to the production of software. These also include standards (government standards, legal standards, security standards, etc.).
• In the absence of any specific agreements, the system must satisfy all of the applicable requirements, as described in the (signed) contract document and associated documents, in addition to the requirements described in all of the more technical documents approved by Fedict.
• Alpha and beta-testing: This involves obtaining feedback from potential or existing customers, before the product is launched on the market. Alpha-testing takes place within the developer’s own organisation. Beta-testing (field-testing) is carried out by individuals in their own working environment.

2.3. TYPES OF TEST

2.3.1. Functional tests

Functional tests set out to verify the functions that a system, sub-system or component is required to perform. Functional tests can be described in what are known as work products, such as requirement specifications, use-cases, functional specifications, etc.

These describe “what” the system actually does.

The functional tests are based on the functionalities and their interoperability with specific systems. Functional tests can be carried out at any test level. For example, tests upon components may be based upon component specifications.

The black-box tests are based on the functional specifications and quality requirements. In these tests, the system is examined in the form in which it will operate when it finally comes into use. The software is examined as a “black box”, without any knowledge of the internal implementation or of its internal structure. Typically, a number of test-cases are drawn up, based upon the functional specifications and requirements.

Though both groups of test techniques can be used at all levels, the white-box techniques are mainly used at the lower levels of testing, whilst the black-box techniques are used at the higher levels.

The acceptance tests carried out by Fedict will mainly make use of black-box techniques.

2.5. TESTING PROCESS

The supplier is free to select a testing process that corresponds to his development methodology, on condition that he fulfils the requirements set out in Section 3.

In Section 4, we set out a number of guidelines in this regard, without however imposing a specific process.

2.6. RISK-BASED

Risk-based testing implies that a (product) risk analysis is carried out and that testing activities focus upon those parts or aspects of an application that pose the greatest risk.

It is regarded as good practice for the testing strategy to be based, at least in part, on the outcomes of a risk analysis.

Two possibilities exist in that regard:

• Fedict has added a risk analysis to the contract documents or associated documents.
• Fedict has not added a risk analysis to the contract documents or associated documents.

In the first instance, we would advise the supplier to make use of that analysis and, if necessary, to refine it, once the technical aspects of the system being tested have been more clearly defined.

In the second instance, the supplier may carry out an analysis of its own. In that case, Fedict may provide input regarding the impact of the risks involved.

If the supplier is expecting Fedict to take some sort of action in connection with the risk analysis, the supplier must include in the tender:

• A description of its approach with regard to risk analysis and the role to be played by Fedict in that regard
• An estimate of the time that Fedict will require in order to fulfil that role

2.7. RESPONSIBILITIES

In this section, we will indicate where the responsibilities of the suppler and of Fedict lie.

2.7.1. General

We assume that the supplier is a professional in the field of software development and possesses the necessary specialist knowledge.

If, by virtue of that knowledge, the supplier were to discover shortcomings in the specifications that could lead to the system not being fit-for-purpose, we will assume that the supplier will inform Fedict of this without delay, so that a solution may be found in good time. It is clear that the previous assertion may sometimes extend beyond the contractual stipulations, but it is always in the interests of both parties that a usable system is delivered.

The intention is that the supplier will deliver to Fedict a system that has been fully tested and that is operating correctly. This also includes the delivery of the test documentation, as described in the previous section, and possibly specified in more detail in a contract document or a requirements document.

Fedict will then carry out acceptance tests on the system, in order to establish whether all of the requirements have been fulfilled effectively.

2.7.2. Lower test levels

The test levels involving unit tests (component tests), integration tests and system tests and the corresponding levels, as used by the supplier, shall lie entirely within the responsibility of the supplier.

Fedict will carry out random checks in order to establish whether the work products fulfil the stated requirements.

1.1.1.1. System Integration tests

As far as the system integration tests are concerned, it is possible that the supplier’s systems will need to be integrated with systems from other suppliers or from the government. Collaboration will therefore be possible in that regard.

Fedict will act as a facilitator in that regard, but the final responsibility will lie with the supplier of the system being tested.

Fedict will carry out random checks in order to establish whether the work products fulfil the stated requirements.

by Fedict. In the event that various specifications are not consistent with one another, the interpretation that most favourable to Fedict will be used.

[End of prescriptive section]

3. Requirements for the supplier

3.1. GENERAL

This section contains the work products that must be delivered as standard, together with the associated quality criteria and the requirements to install the system being tested in the various environments.

This section is prescriptive, unless explicitly stated that that is not the case.

The requirements stated here may be different to those contained in the contract documents and associated requirements documents, or in the contracts between the supplier and Fedict. The differences may take the form of more or more stringent requirements, or may involve less stringent requirements being imposed.

This section must be regarded as constituting the test requirements with regard to all aspects of the testing process that are not explicitly transposed into contractually relevant documents.

3.2. CONTENT OF THE WORK PRODUCTS TO BE SUPPLIED

The work products to be supplied must, in principle, contain the details referred to in the templates described in Annex 1. Individual contract documents, requirements documents or contracts may contain additional requirements with regard to content.

If the supplier deviates from these, it will need to document the reason for any such deviation, either in the test plans or in the documents containing the deviation, giving the reason for the deviation.

If the supplier uses its own templates, it must ensure that these contain the necessary information. If so requested by Fedict, the supplier must also provide a document mapping the structure it has used, compared to the ones contained in the attached templates.

The fact that these templates have been provided does NOT imply that Fedict is asking that those templates be used for the actual purpose of generating documents.

[non-prescriptive] It is important to note these are primarily intended as a checklist. In a number of cases, it is certainly advisable to store the requested information in a tool, as opposed to in MS Office documents. [end of non-prescriptive section]

A test summary report (or automated test dashboard) will be included in the delivery.

[non-prescriptive] We would advise the supplier to apply a system of continual integration or a daily build with integrated test-cases. [end of non-prescriptive section]

Fedict will carry out random checks in order to determine whether the unit tests supplied fulfil the requirements of this policy, together with any additional requirements or standards imposed in the contract document, the requirements document or any other contractual agreements.

[non-prescriptive] In the case of complex or critical projects, the statement coverage will be increased, or a switch may be made to a more stringent form of coverage, such as decision or condition average. [end of non-prescriptive section]

Input (^) • Functional documents

• Technical design documents
• Code of individual components

Actions (^) • Testing of individual components

Output (deliverables)

• Test-cases (test-case specification), including an automated test suite
• Test results
• Test summary report (or dashboard)

Role played by Fedict

• Random checks

3.3.3. Integration tests

With each delivery of software to Fedict, the supplier shall also include the component integration test-cases. These may take the form of code or of manual test-cases.

With each delivery of software to Fedict, the supplier shall also include the system integration test- cases, insofar as the system integration has already been carried out or is applicable. These may take the form of automated or manual test-cases.

The supplier shall demonstrate (for example, by submitting a traceability matrix) that:

• All internal components have been integrated
• Each external interface has been tested in relation the following aspects: o Communication via the interface concerned is operational. o The interpretation of the data exchanged is the same for both systems. If this is relevant and as soon as the system integration has taken place, the supplier shall also demonstrate that the end-to-end flow of the application has undergone thorough functional testing (end-to-end tests, based upon the business analysis, chain-testing).

[non-prescriptive] We would advise the supplier to include at least part of the component integration tests in a system of continuous integration or a daily build. [end of non-prescriptive section]

Fedict will carry out random checks in order to determine whether the integration tests supplied fulfil the requirements of this policy, together with any additional requirements or standards imposed in the contract document, the requirements document or any other contractual agreements.

Input • Business analysis (in relation to the system integration tests)

• Functional documents
• Design documents
• Source code of the components to be integrated
• Artefacts of the components to be integrated

Actions (^) • Testing of the interfaces between the components

• Testing of the interfaces between the system and its peer-systems
• End-to-end testing of functional and non-functional quality attributes

Output (^) • Test-cases (test case design and test case specification)

• Test results
• Test summary report
• Traceability matrices

Role played by Fedict

• Random checks

3.3.4. System tests

With each delivery of software to Fedict, the supplier shall also include the system test-cases, insofar as these are applicable at that point in time. These may take the form of automated or manual test-cases.

The supplier shall demonstrate (for example, by supplying a traceability matrix) that the full extent of the functionality described in functional documents approved by Fedict has been covered and the risk analysis has been observed effectively.

The supplier shall demonstrate (for example, by supplying a traceability matrix) that the non- functional quality attributes allocated for testing at system test level within the relevant test plan have been covered effectively.

Fedict will carry out random checks in order to determine whether the system tests supplied fulfil the requirements of this policy, together with any additional requirements or standards imposed in the contract document, the requirements document or any other contractual agreements.

Input (^) • Functional documentation

Actions (^) • Testing of the functionality of the integrated system

• Testing of the non-functional quality attributes of the integrated system

Output (deliverables)

• Test cases (test case design and test case specification)
• Test results, including a list of all known outstanding defects
• Test summary report
• Traceability matrices