Metrics - Applying Systems Engineering - Lecture Slides, Slides of Systems Engineering

Download Metrics - Applying Systems Engineering - Lecture Slides and more Slides Systems Engineering in PDF only on Docsity!

Metrics

Objectives

• Metrics

• What they are

• How they are used

• Variations

• Value of metrics

Key Terminology

• Metrics

– Process Metrics

• Earned Value Measurement System (EVMS) ANSI

Standard (ANSI/EIA-

– Technical Performance Measurement

• Weight, throughput, speed, capacity,

• Any good measure of what your system is trying to

accomplish.

• Tied to requirements

Scientific Method

• Metrics are just measurements

• Recall the Scientific Method

– Define the Problem

– Form a Hypothesis

– Collect Data

– Analyze the Data

– Prove (or disprove) the hypothesis

• Don’t forget it!

Our Text book author

• Stevens states:

– System Engineering is concerned with creating

effective solutions to problems, and managing the

technical complexity of the resulting developments.

• The objective of SE is to ensure the quality of the

product by having rational and structured

processes.

• It is therefore vital that the outputs of an instance

of the process be measured and compared.

• These measurements are “metrics”

INCOSE-Handbook

• Definition: A composite (or calculated value) of

one or more measures (e.g., software

productivity = source lines of code/labor

months). This term also describes a set of values

that includes both metrics and measures.

• Requirements provide metrics for how well the

functions must be performed.

• These functions may be the program processes or

the product functions.

INCOSE-Handbook

STEP 10 - MONITORING TEAM PROGRESS

- (^) METRICS

SCHEDULE STATUS BUDGET vs. EXPENDITURES vs. COMPLETION

DESIGN EFFICIENCY METRICS PRODUCT PERFORMANCE & COST METRICS

TASK/ Month Ja Fe Ma Ap Ma Ju Jy Au Sep^ Oct Nov Dec

**1. Sys Analy

2. Reqts Der
3. Funct Anl
4. Arch Con
5. Reqts Alloc
6. Conc. Des.
7. Sys. Eval.**

Deliverables:

Final^20 Rpt.

April 12 1 2

Month Ja Fe Ma Ap Ma Ju^ Jy Au Sep Oct Nov Dec

Milestones:

April 12

4 of 5 Met (5 Days Behind)

1,000s

PLAN

ACTUAL

MONTHS Ja^ Fe Ma^ Ap Ma^ Ju^ Jy Au Sep Oct Nov^ Dec^ MONTHS Ja^ Fe Ma^ Ap Ma^ Ju^ Jy Au Sep Oct Nov^ Dec

W E I G H T

Kg

Requirement

Goal

Perf. Reqt.

Cost Reqt.

Cost of First Production Unit

Actual

P E R F O R M A N C E

Example Technical Performance Measurement (TPM)

INCOSE-Handbook – Example Metrics

1. Mission Analysis, Percent Completion
2. Mission Needs Statement, Percent Completion
3. Alternative System Architectures Definition and Evaluation,

Percent Completion

4. System Requirements Document, Percent Completion
5. Requirements Stability and Growth Metrics, such as, Number

of Requirements Added, Modified, or Deleted during the

preceding time interval (month, quarter, etc.).

6. Percent Completion of contract requirements documentation:

SOW, RFP, CDRL, etc. (each)

INCOSE-Handbook – Metrics in

Risk Management

• In practice, a variety of semi-quantitative and

qualitative metrics, including technical

performance indicators, are used to support risk

management (see Section 4.2.4.2.).

• The most common quantitative measure of risk is

the risk profile, which is defined as probability

times consequence, which we discussed in the

last class. The overall metric is the measure of

residual risk.

INCOSE-Metrics Premier

• Copy has been provided to you

GQM From INCOSE-Metrics Premier

• A method that can be used to identify appropriate and useful measures from identified

project goals is the Goal/Question/Metric (GQM) approach. The four basic steps of GQM

are:

• 1. State the information Goal. Identify the information consumer groups (stakeholders)

want to know and determine what they want to do with the information. Work top-down,

including both organizational and project goals as appropriate.

• 2. Ask the Question. What questions should be asked to determine whether the goal is

being met?

• 3. Determine the Measure. Identify the specific parameters that must be measured to

answer the question(s) posed in Step 2. What measures are needed (directly or indirectly)

and what must be measured (collected) to obtain it?

• 4. Do and evaluate. Apply the measures selected and evaluate their usefulness. Return to

Step 1 or 2, if measures are inadequate for their intended purpose.

• Is this any different from the Scientific Method?

INCOSE-Metrics Premier

• See Pages 28, 29, 30

Sahra Sedigh-Ali, Arif Ghafoor, Raymond A. Paul,

Authors

• Handling the growth of these COTS-based systems requires new approaches to

quality and risk management. The authors offer software-engineering metrics to

aid developers and managers in analyzing their quality-improvement initiatives'

return on investment and to facilitate the modeling of cost and quality. They assert

that large-scale component reuse or COTS component acquisition can generate

savings in development resources, which can be applied to improve quality and

enhance reliability, availability, and maintenance. Further, metrics play a critical

role in identifying risks that involve performance, reliability, adaptability,

scheduling, and product evaluation. COTS products change rapidly, and research

on COTS-based- system development is still in the early stages. Given that cost-

effectiveness and quality provide the major factors in deciding for or against

component acquisition, the authors see an urgent need for empirical and

analytical research that will lead to more accurate cost and quality models for

these systems.

Role interactions –

Sarah A. Sheard,

Dividing any complex body of knowledge into pieces is something of an art, and specifying systems engineering

roles is no exception. Although the boundaries between roles have been chosen for cohesion and low communication, significant interactions still occur. Some of the more noteworthy are as follows:

• Risk. Risk analysis has been included here as part of the SA (System Analyst) role, risk identification as part

of the G (Glue) role, and the management of risk (workarounds, mitigation plans) has been included in the TM (Technical Manager) role.

• Design Reviews. As part of the TM role, systems engineers set up the reviews and track action items. For

externally-required design reviews, the CI (Customer Interface) role ensures appropriate customer attention, and the G (Glue) role is involved to ensure that the design is thoroughly reviewed by appropriate parties.

• Metrics. The PE (Process Engineer) role identifies needed process metrics, establishes the measurements

that need to be taken, and defines the metrics algorithms. Systems engineers performing life-cycle tasks take the measurements, and technical managers use the metrics for decision making.

• Customer Interaction. The LO (Logistics and Operations) role is very customer-focused, attempting to

ensure usability and operational suitability. ROs (Requirements Owners) need to ensure the specified, designed, and built system will be what the customer wanted, and the CI (Customer Interface) role encompasses the various interfacing that should be done with the customer throughout the program.

• http://www.software.org/pub/externalpapers/12ROLES.html