Download Checklist for Requirements Definition: Ensuring Comprehensive Design Specifications and more Summaries Design in PDF only on Docsity!
REQUIREMENTS CHECKLISTS: BENCHMARKING THE
COMPREHENSIVENESS OF THE DESIGN SPECIFICATION
Becattini, Niccolo (1); Cascini, Gaetano (1); Rotini, Federico (2) 1: Politecnico di Milano, Italy; 2: Università degli Studi di Firenze, Italy
Abstract
It is commonly recognized that the definition of product requirements is an essential step of any design process. Many techniques have been proposed for building a suitable design specification, i.e. for defining a set of requirements characterized by validity, completeness, operationality, non- redundancy, conciseness and practicability. Among them, several methods and tools primarily aim at populating the design specification: some of them focus on very specific objectives but are applicable in many different domains (e.g., Design for X). Others are domain specific, but try to cover the entire scope of the specification (e.g., checklists and standards). This paper describes an abstract-level checklist for requirements definition, suitable for any field of application, aiming at producing exhaustive lists of requirements. A previous experimental application with Mechanical Engineering students clearly showed that the proposed multi-purpose checklist allows populating design specifications more complete than those defined without any support. This paper follows up demonstrating the capability of the novel checklist against the checklist for conceptual design by Pahl and Beitz.
Keywords : Requirements, Design methods, Design education, Design specification, Requirements checklists
Contact : Dr.-Ing. Niccolo Becattini Politecnico di Milano Dipartimento di Meccanica Italy [email protected]
INTERNATIONAL CONFERENCE ON ENGINEERING DESIGN, ICED 15 27 -30 JULY 2015, POLITECNICO DI MILANO, ITALY
Please cite this paper as: Surnames, Initials: Title of paper. In: Proceedings of the 20th International Conference on Engineering Design (ICED15), Vol. nn: Title of Volume, Milan, Italy, 27.-30.07.
1 INTRODUCTION
A proper definition of requirements is the key for an effective and efficient innovation and development process. Such a process requires the designer to search solutions to problems in a space of alternatives that, in turn, has to be limited by the goals to be achieved and ruled by the laws governing the situation at hand (Simon, 1973). In this reference, both problem state and goal state can be characterized by means of requirements: in the former, they get unsatisfactory values, while in the latter they change so as to make the problem solved or not relevant anymore. In other words, they represent the target conditions to be satisfied and therefore they both support the development of design proposals as well as the evaluation of solutions, for which they can be considered as benchmarking criteria for the related decision-making (e.g. Cross, 2008). Roozenburg and Eekels (1995) explained the relationship that requirements hold between system characteristics and user’s needs and expectations, through an explicit hierarchy that ranges from overall goals, to more specific objectives till the definition of requirements. To this purpose, they also defined the characteristics that a design specification (i.e. the set of requirements) should have in order to be really useful for the development processes:
- Validity – requirements should define the criteria for assessing the satisfaction of the design objectives;
- Completeness – requirements should cover all the potentially relevant objectives;
- Operationality – requirements should be measurable with reference to the objectives;
- Non-redundancy – requirements shouldn’t be duplicated, if they have the same meaning;
- Conciseness – requirements should take into account all the relevant aspects, overlooking the ones that are not relevant;
- Practicability - requirements satisfaction should be testable with available information.
Other authors have concentrated their efforts on the development of appropriate design methods and tools for requirement-related purposes. Quality Function Deployment (QFD), for instance, is a largely diffused approach to support the translation of customer requirements into technical requirements, e.g. (Akao, 2004). It is usually presented as part of the House of Quality (HoQ), a graphical and textual model that integrates the analysis of those requirements so that priorities and potential issues can be formulated in a more structured way. Different contributions, moreover, have also aimed at defining appropriate classes in order to characterize requirements and treat them in a more efficient or user- oriented perspective, e.g. the classification of customer requirements by Kano (1984). Other studies, in turn, try to combine the contributions about requirements classification with the existing tools to manage them like QFD, as, e.g., proposed by Matzler et al. (1998). Among the different methods and tools for dealing with requirements, a relevant role is played by those that primarily aim at populating the design specification. Roughly, they can be organized into two main categories. Those applicable in a great variety of situations and domains of technique but focusing on very specific objectives (e.g. Design for X methods) and domain-specific checklists (sometimes also formalized into standards), which address a wide range of requirements but that present some limitations in the applicability outside the domain they have been developed for. With the overall goal to contribute to the overcoming of this dichotomy, the authors have proposed an abstract-level checklist for requirements definition, suitable for any field of application, but also to produce exhaustive lists of requirements. So far, it has been proficiently used with different purposes in industrial contexts, to start checking its effectiveness, e.g., in order to support benchmarking activities along a project focusing on the substitution of manufacturing technologies (Becattini, 2011); in order to properly define accurate technology descriptions for manufacturing machines using the fluid-bed principle (Becattini, 2013); in order to assess the opportunities of integration with a computer-aided innovation prototype for the prioritization of design problems on the basis of network analysis and conflicting requirements (Becattini and Cascini, 2013). Moreover, the authors have also recently started a testing activity to statistically evaluate the performance of their proposal and retrieve experimental data so as to also plan further improvements. The experiments carried out so far aimed at checking the performances in populating a design specification with the novel multi-purpose checklist. These tests have been carried out with Mechanical Engineering students and the results have been compared with those of a control group having the same background but with no support by any checklist or other instruments. Two products
Second, none of the testers had already got in touch with requirements’ checklists before the test (this enables the participants to be fresh to both approaches). The checklist has to be used before starting the conceptual design phase when clarifying the task, so as to drive the exploration of solutions and the generation of ideas and concepts accordingly. Actually, Pahl and Beitz, about their checklist just say: “ The checklist […] is a generic one based on ideas described in [a previous] Section. The items in this list are checked against the existing task and its requirements in order to obtain further requirements ” (p. 151, 3rd^ ed.). The checklist (Figure 1) is globally organized in 17 classes ranging different structural, behavioural and functional features along the lifecycle phases of the product to be developed, with an overall set of 92 items to be checked.
2.2 An abstract-level multi-purpose checklist
As briefly mentioned in the introduction, this checklist has been developed at an abstract level so as to overcome the dichotomy between approaches that are capable of mapping a wide set of requirements in a very specific field of investigation and those focused on a specific target but can be applied in any field. Furthermore, the multi-purpose checklist is tailored to more effectively support conceptual design tasks. Indeed, it starts defining the detail level of the analysis by identifying the functional outcome that is expected (the product of the function) and the initial conditions of the product before the system to be developed carries out its function. Both the initial and the final condition of the product of the function are represented by means of the Energy-Material-Signal model (Pahl et al., 2004). Three main axes of investigation are considered along the checklist. They have been defined in accordance with the main drivers defined by the 4th^ Law of Engineering Systems Evolution, from TRIZ theory. The law says that systems evolve towards the increase of their ideality, where ideality is conceived as the sum of useful functions (UF) the system delivers divided by the sum of harmful functions (HF) it produces or undergoes and the costs (C) to make it work (Equation 1). In other words, technical systems evolve towards the improvement of their performances, the reduction of undesired effects they trigger or suffer and the reduction of any kind of expenses (Altshuller, 1984).
This checklist is also conceived as an instrument to enhance thinking instead of purely “checking” the requirements. It aims at stimulating reflections that can support the generation of non-obvious requirements, since trivial ones do not need any methodological tool for their identification. The three axes concerning performances (related to useful functions), side effects (related to harmful functions) and resources consumptions (related to costs) are, therefore, organized so that the designer needs to contextualize the abstract criteria to the specific situation she/he is dealing with, potentially enlightening unexplored alternatives. The dimension of performances as a result of useful functions is explored in order to check both the quantitative and the qualitative aspects of the function, according to the following sub-dimensions:
- Threshold achievement: requirements describing the capability to impact the object of the function with the expected extent (see Figure 2) ;
- Versatility: requirements characterizing the capability to adapt the behaviour of the technical system according to different operating conditions;
- Robustness: requirements accounting the capability of the technical system to obtain the same (stable in values) desired outcome under varying inputs;
- Sensitivity to external conditions: requirements concerning the capability of the technical system in carrying out its function regardless of the conditions of the environment in which it is immersed;
- Controllability: requirements about the capability to set system characteristics and parameters so as to obtain a desired result according to user’s will. The side effects generated by harmful functions are considered from three different viewpoints:
- Impact on the object of the Main Useful Function (e.g. an undesired side effect caused by the same mechanism adopted to deliver the function or as its consequence);
- Impact on the system and subsystems integrity (e.g. an undesired side effect on the technical system as a whole or on its parts);
- Impact on the external environment (e.g. an undesired side effect that compromises some environmental conditions or damages some of the elements that pertains to the world in which the technology/technical system is immersed in).
Figure 2. An example of abstract criteria for the identification of system requirements (Threshold values for the main function of the system). It is provided as a visual representation, a textual description and an example.
Finally, the costs due to the consumption of resources are further sub-classified into five categories: Resources of Time, Information/Knowledge, Material, Energy and Space Overall, the abstract-level multi-purpose checklist is organized into three main classes, which are further organized into 13 subclasses for which a set of 53 examples are provided so that one can be facilitated in the contextualization and more effectively inspired for requirements definition. The complete description of the checklist cannot be presented here for space reasons, but it is completely available in Becattini (2013).
The next Section presents the approach followed to benchmark the two checklists against each other in order to evaluate their performance in producing more complete design specifications.
3 TESTING METHOD
The testing methodology has been developed consistently with the objective of comparing the performances of two checklists to populate a complete design specification. In order to properly design the experiment, the authors started identifying the method for data analysis. Indeed, it has to be chosen consistently with the objectives of the research in order to meaningfully identify the test dynamic and the profile of participants. The approach to carry out and analyse the results of the test allows further benchmarking studies to be carried out on a similar basis, so as to also compare results coming from different testers with uniform criteria. This test exclusively focuses on the amount of requirements generated by means of the competing checklists, as a first attempt to evaluate the completeness of a design specification. The authors are planning to extend and refine this test in order to cover a wider range of facets concerning the design specification (e.g.: presence of duplicates, capability to formulate requirements so as to make them measurable and operational, etc.). However, the test dynamic presented in Section 3.2 allows experimenters to extend its applicability to other characteristics of the design specification (e.g. benchmarking non-redundancy, conciseness, etc.) after a small adaptation.
participants are required to define a design specification. Account one hour per each round. (2 rounds were established for this study: during the first one, the system to be considered was an iron; during the second round it was a laser printer).
- During each round, the groups focus the attention on the same technical system as the subject for defining requirements. Each of the groups work with a requirements checklist that is different from the others. (The group A first works with the Pahl and Beitz checklist, while B with the checklist proposed by the authors. For the second round, the two groups switch the checklist to be used).
- Considering the perspective of a benchmarking analysis, provide the testers with the checklist as the original reference sources in literature propose to use them. The media, e.g. paper, text file, spreadsheet,etc, on which the contributions are produced have to be the same in the different rounds, in order to avoid biases due to usability issues depending on the media. (The test was done simply with pen and paper for both the checklists. Pahl and Beitz’s checklist have been proposed as a single printed sheet with an additional sheet where students could write down requirements and, not compulsorily, the category they had been identified from. The checklist proposed by the authors appears as a more consistent document of 3 pages printed on both sides. The criteria were presented with textual and visual explanation for each class of requirements and specific subclasses and sufficient room to write down one or more requirement, upon their identification. Examples in the field of washing machines aim at clarifying the meaning of the proposed criteria.)
- Gather the outputs from all the participants before switching to the next round, if any, or conclude the testing session. (The second round has started immediately after the first one).
- Count the requirements per each of the gathered outputs, without introducing any adjustment or element of evaluation in order to avoid biases due the perspective of the investigation. Each output specification gets a score that corresponds to the amount of requirements that the tester has defined. (The results have been manually counted; the related data have been organized in spreadsheets and double-checked to verify the correct transcriptions of numbers).
- Organize the results dividing the contribution according to the specific theme on which to focus the attention, i.e. by round of the testing session, and by the specific checklist used to identify system requirements, i.e. by the specific group that participated the investigation. (Results are presented as aggregated data in Section 4).
- Run the statistical analysis described in Section 3.1, so as to verify the hypothesis on an appropriate couple of samples at a time. The two samples should come from different groups, but collected during the same round of investigation. (Results are presented in Section 4)
- Define conclusions out of the results of the investigation, considering the degree of statistical significance. (The results are discussed in Section 4 and summarized in Section 5).
3.3 Participants
The test has been carried out in an educational environment so as to gather a sufficient number of participants and carry out a significant test from which to draw more robust fact-based evidences. The mechanical engineering MS (mechanical design major) students, 29 volunteers that participated the test, attended a course on “Product Development and Engineering” (Italian name: “Sviluppo e Ingegnerizzazione del prodotto”) held at the University of Florence - Italy. Their background is largely homogenous considering that they all hold a BS in mechanical engineering. Along the course, the students get introduced to a systematic approach to design, which is fundamentally grounded on the work by Pahl and Beitz (2004). Different tools and techniques for the different phases of design (Product Planning, Conceptual and Embodiment Design) are taught along the course, in order to facilitate the students in learning both the overall structure of a systematic approach and the tools to deal with the different issues popping up as the design deliverable get defined and refined. This sample is a convenient and relevant one, because it is composed by people that already have higher education knowledge in mechanical design, even if to be further improved and applied on the field, thus representing right candidates for using these instruments in their design activities. Moreover, at
the moment the test was carried out, students were just exposed to the overall structure of Pahl and Beitz’s design methodology and the test worked as a way to introduce them to the application of design specification. Being a candidate user of the checklist should represent the discriminating factor for determining the suitability of a person to participate the study.
4 RESULTS OF THE TEST
In order to avoid language related biases, the test has been submitted to students with checklist translated from English to Italian without altering the content of the original sources. The results of the two rounds of testing have been collected, the requirements counted and the related aggregated data as descriptive statistics are collected in Table 1 (Pahl and Beitz’s checklist) and Table 2 (authors’ checklist). It is also worth recalling that the test aims at understanding if the author’s checklist is capable to overcome the performances of Pahl and Beitz’s checklist to identify design requirements.
Table 1. Summary of the results obtained with the Pahl and Beitz checklist Theme MAX min Average St. Dev. Sample size Round Iron 41 13 23,53 8,9 2 14 1 Laser Printer
Total 51 13 26,86 9,85 29 -
Table 2. Summary of the results obtained with the author’s checklist Theme MAX min Average St. Dev. Sample size Round Iron 64 32 50,21 8,63 15 1 Laser Printer
Total 64 17 45,5 2 12,50 29 -
To this purpose, the following analysis just considers the results from a quantitative perspective, taking into account a more complete design specification, which is populated by a larger number of requirements. A quick analysis of this descriptive statistics shows that, in general, the authors’ checklist has performed better in both the rounds, regardless of the specific theme of its application. The average value for authors’ checklist generated requirements (50,21) doubles the ones obtained with Pahl and Beitz’s checklist (23,53 requirements) when the tester considered an iron as the focus of the investigation. The test with the laser printer, in turn, shows a similar but less marked behaviour (41, requirements and 30,43 respectively with the authors’ and Pahl and Beitz’s checklist). Moreover, also in terms of variability the two cases present some differences. While the standard deviations get analogous values for the results obtained with the iron, the situation with the laser printer radically changes. The variability of the sample using the authors’ checklist is almost 50% bigger than the variability obtained with its own competitor. In order to verify if and to which extent the current results are significant from a statistical point of view, the hypothesis test becomes necessary. It has been carried out as detailed in Section 3.1.
Table 3. Summary of the test of hypothesis: t-test statics and related p-values.
Theme Checklist Sample size Degrees offreedom S^2 p μ 1 -μ (^2) value^ t-test p-value
Laser Printer
Authors’ 14 27 76,93 26,68 8,19 0,000 (***) Pahl and Beitz’s 15
Iron
Authors’ 15 27 146,93 11,70 2,38 0,012 (**) Pahl and Beitz’s 14
Total
Author’s 29 56 126,69 18,66 6,31 0,000 (***) Pahl and Beitz’s 29
Moreover, t-paired test will be carried out in order to see if significant shift appears considering the same tester using two different approaches for the same technical system, in order to detect a potential superposition of the two approaches and the related effects. This investigation might be also enriched with benchmark against different requirements checklists as the one mentioned in the second chapter.
REFERENCES
Akao Y.; (2004); Quality Function Deployment: Integrating Customer Requirements Into Product Design; Productivity Press. Altshuller, G.S.; (1984); “Creativity as an Exact Science: The Theory of the Solution of Inventive Problems”. Gordon and Breach Science Publishers, ISBN 0-677-21230-5, (original publication in Russian - 1979). Becattini N., Cascini G., Petrali P., Pucciarini A.; (2011); “Production processes modeling for identifying technology substitution opportunities”- Published on the proceedings of the 11th TRIZ Future Conference 2011 – Dublin– Ireland (2nd-4th Nov). Becattini, N., & Cascini, G. (2013). Mapping Causal Relationships and Conflicts among Design Parameters and System Requirements. Computer-Aided Design and Applications, 10(4). Becattini N. (2013) Requirements Identification and Characterization in Innovation Process. PhD Dissertation, Politecnico di Milano - Italy Becattini N., Cascini G., (2014) General-purpose requirements checklist for improving the completeness of a design specification. DESIGN 2014 Conference - Cavtat- Dubrovnik - Croatia May 19-22nd. Cross, N. Engineering Design Methods: Strategies for Product Design. Fourth edition. Chichester: John Wiley and Sons Ltd.; 2008 Gero, J.S.; (1990); ‘Design prototypes: a knowledge representation schema for design’ AI Magazine Vol 11 No 4 p.26– Hales C., Gooch S., (2004) Managing Engineering Design, Springer Science & Business Media Hubka, V. and Eder, W.E. (1988) Theory of Technical Systems: A Total Concept Theory for Engineering Design, Berlin: Springer. Kano N.; Seraku N., Takahashi F., Tsuji S.; (1984); "Attractive quality and must-be quality" (in Japanese). Journal of the Japanese Society for Quality Control 14 (2): 39–48. Matzler K.; Hintertuber H.H.; (1998); How to make product development projects more successful by integrating Kano’s model of customer satisfaction into quality function deployment; Technovation, 18(1) 25–38; Elsevier Science Ltd Pahl G., Beitz W., Feldhusen J., Grote K.H; (2007) Engineering Design: a systematic Approach, 3rd Edition; Translation from German by Wallace K., Blessing L. (eds.) ; Springer-Verlag, London. Pugh, S. (1990) Total Design: Integrated Methods for Successful Product Engineering, Wokingham: Addison Wesley. Roozenburg, N.F.M. & Eekels, J.; (1995); Product Design: Fundamentals and Methods; Wiley; Chicester, New York; ISBN:0471943517. Simon, H. A.; (1973); The structure of ill structured problems; Artificial Intelligence; Elsevier; Volume 4, Issues 3-4, Winter, Pages 181-201.
