Failure Confinement: A Graduate Course on Fault-Tolerant Computing - Prof. B. Parhami, Exams of Electrical and Electronics Engineering

Download Failure Confinement: A Graduate Course on Fault-Tolerant Computing - Prof. B. Parhami and more Exams Electrical and Electronics Engineering in PDF only on Docsity!

Nov. 2007

Failure Confinement

Fault-Tolerant Computing^ Dealing withHigh-LevelImpairments

Nov. 2007

Failure Confinement

About This Presentation

Edition

Released

Revised

Revised

First

Nov. 2006

Nov. 2007

This presentation has been prepared for the graduatecourse ECE 257A (Fault-Tolerant Computing) byBehrooz Parhami, Professor of Electrical and ComputerEngineering at University of California, Santa Barbara.The material contained herein can be used freely inclassroom teaching or any other educational setting.Unauthorized uses are prohibited. © Behrooz Parhami

Nov. 2007

Failure Confinement

Nov. 2007

Failure Confinement

Slide 5

Multilevel Model of Dependable Computing

Component

Logic

Service

Result

Information

System

Level

→

Low-Level Impaired

Mid-Level Impaired

High-Level Impaired

Unimpaired

Entry Legend:

Deviation

Remedy

Tolerance

Ideal

Defective

Faulty

Erroneous

Malfunctioning

Degraded

Failed

Nov. 2007

Failure Confinement

Slide 7

Importance of Experimental Failure Data • Indicate where effort is most needed • Help with verification of analytic models System outage stats (%)*

Hardware

Software

Operations Environment

Bellcore [Ali86]

Tandem [Gray87]

Northern Telecom

Japanese Commercial

Mainframe users

Overall average

*Excluding scheduled maintenance Tandem unscheduled outages^ Power

Communication lines

Application software

File system

Hardware

Tandem outages due to hardware^ Disk storage

Communications

Processors

Wiring

Spare units

Nov. 2007

Failure Confinement

Slide 8

Failure Data Used to Validate or Tune Models^ Example:

Two disks, each with MTTF = 50,000 hr, MTTR = 5 hr Disk pair failure rate

≈^2 λ

2 /μ

Disk pair MTTF

≈ μ

/(2λ

×^10

8 hr = 285 centuries

In 48,000 years of observation (2 years

×^ 6000 systems

×^ 4 disk pairs),

35 double disk failures were reported

MTTF

≈^ 14 centuries 2

1

0

2 λ^

λ μ

-^ Indicate accuracy of model predictions (compare multiple models?) •^ Help in fine-tuning of models to better match the observed behavior Problems with experimental failure data: •^ Difficult to collect, while ensuring uniform operating conditions •^ Logs may not be complete or accurate (the embarrassment factor) •^ Assigning a cause to each failure not an easy task •^ Even after collection, vendors may not be willing to share data •^ Impossible to do for one-of-a-kind or very limited systems

Nov. 2007

Failure Confinement

Preparing for Failure

-^ Minimum requirement: accurate estimation of failure probability •^ Putting in place procedures for dealing with failures when they occur Failure probability = Unreliability Reliability models are by nature pessimistic (provide lower bounds) However, we do not want them to be too pessimistic

Risk

=^

Frequency

×^

Magnitude

Consequence / Unit time

Events / Unit time

Consequence / Event

Frequency may be equated with unreliability or failure probability Magnitude is estimated via economic analysis (next slide) Failure handling is often the most neglected part of the process An important beginning: clean, unambiguous messages to operator/user Listing the options and urgency of various actions is a good idea Two way system-user communication (adding user feedback) helpful Quality of failure handling affects the “Magnitude” term in risk equation

Nov. 2007

Failure Confinement

How Much Is Your Life Worth to You?

Thought experiment: You are told that you have a 1/10,000 chance of dying todayHow much would you be willing to pay to buy out this risk, assumingthat you’re not limited by current assets (can use future earnings too)? If your answer is $1000, then your life is worth $10M to you

Risk

=^

Frequency

×^

Magnitude

Consequence / Unit time

Events / Unit time

Consequence / Event

Can visualize the risk by imagining that 10,000 people in a stadium aretold that one will be killed unless they collectively pay a certain sum Consciously made tradeoffs in the face of well-understood risks (salarydemanded for certain types of work, willingness to buy smoke detector)has been used to quantify the worth of a “statistical human life”

Nov. 2007

Failure Confinement

Believability and Helpfulness of Failure Warning

“No warning system will function effectively if its messages,however logically arrived at, are ignored, disbelieved, or lead toinappropriate actions.”

Foster, H.D., “Disaster Warning Systems,” 1987

Unbelievable failure warnings: Failure event after numerous false alarms Real failure occurring in the proximity of a scheduled test run Users or operators inadequately trained (May 1960 Tsunami in Hilo,^ Hawaii, killed 61, despite 10-hour advance warning via sirens) Unhelpful failure warnings: Autos – “Check engine” Computer systems – “Fatal error”

Nov. 2007

Failure Confinement

Engineering Ethics

Risks must be evaluated thoroughly and truthfully IEEE Code of Ethics:

As IEEE members, we agree to

1. accept responsibility in making decisions consistent with the safety,^ health and welfare of the public, and to disclose promptly factors^ that might endanger the public or the environment 6. maintain and improve our technical competence and to undertake^ technological tasks for others only if qualified by training or^ experience, or after full disclosure of pertinent limitations; 7. seek, accept, and offer honest criticism of technical work,^ to acknowledge and correct errors, and to credit properly the^ contributions of others ACM Code of Ethics:

Computing professionals must

minimize malfunctions by following generally accepted standards forsystem design and testing give comprehensive and thorough evaluations of computer systemsand their impacts, including analysis of possible risks

Nov. 2007

Failure Confinement

Fail-Safe Systems

Fail-safe:

Produces one of a predetermined set of safe outputs when it fails as a result of “undesirable events” that it cannot tolerate Fail-safe traffic light: Will remain stuck on red Fail-safe gas range/furnace pilot: Cooling off of the pilot assembly dueto the flame going out will shut off the gas intake valve A fail-safe digital system must have at least two binary output lines,together representing the normal outputs and the safe failure condition Reason: If we have a single output line, then even if one value (say, 0)is inherently safe, the output stuck at the other value would be unsafe Two-rail encoding is a possible choice:

^0 : 01,

^1 : 10,

F : 00, 11, or both

Totally fail-safe:

Only safe erroneous outputs are produced, provided

another failure does not occur before detection of the current one Ultimate fail-safe:

Only safe erroneous output is produced, forever

Nov. 2007

Failure Confinement

Slide 17

Fail-Safe System Specification

Is the specification above consistent and complete?

Correctoutput Safe outputsUnsafeoutputs

Input spaceInput

Output space

Amusement park train safety system Signal

s when asserted indicates thatB^ the train is at beginning of its track (can move forward, but should not beallowed to go back) Signal

s when asserted indicates thatE^ the train is at end of its track (can goback, but should not move forward) No, because it does not say what happens if

s =B^

s = 1; this would notE^

occur under normal conditions, but because such sensors are oftendesigned to fail in the safe mode, the combination is not impossible Why is this a problem, though? (Train simply cannot be moved at all) Completeness will prevent potential implementation or safety problems

Nov. 2007

Failure Confinement

Fail-Safe State Machines

Use an error code to encode states Implement the next-state logic so that the machine is forced to an errorstate when something goes wrong Possible design methodology: Use Berger code for states, avoiding the all 0s state with all-1s check,and vice versa Implement next-state logic equations in sum-of-products form for themain state bits and in product-of-sums form for the check state bits The resulting state machine will be fail-safe under unidirectional errors

Input State

x =^

x =

A^

E^

B

B^

C^

D

C^

A^

D

D^

E^

D

E^

A^

D

State

Encoding A

B^

C^

D^

E^

Hardware overhead for n -state machine consistsof O(log log

n ) additional

state bits and associatednext-state logic, and aBerger code checkerconnected to state FFs

Nov. 2007

Failure Confinement

Principles of Safety Engineering

Principles for designing a safe system

(J. Goldberg, 1987)

1. Use barriers and interlocks to constrain access to critical system^ resources or states 2. Perform critical actions incrementally, rather than in a single step 3. Dynamically modify system goals to avoid or mitigate damages 4. Manage the resources needed to deal with a safety crisis, so that^ enough will be available in an emergency 5. Exercise all critical functions and safety features regularly^ to assess and maintain their viability 6. Design the operator interface to provide the information and power^ needed to deal with exceptions 7. Defend the system against malicious attacks