Performance Evaluation II, Lecture Slide - Computer Science, Slides of Introduction to Computers

Download Performance Evaluation II, Lecture Slide - Computer Science and more Slides Introduction to Computers in PDF only on Docsity!

Performance Evaluation II

November 10, 1998

Topics

Amdahl’s Law

Benchmarking (lying with numbers)

class23.ppt

• 2 –

class23.ppt

Amdahl’s law

You plan to visit a friend in Normandy France and must

Normandy.it will take 4 hours Pgh to NY and 4 hours Paris to($3,100) or a 747 ($1,021) from NY to Paris, assumingdecide whether it is worth it to take the Concorde SST

time NY->Paris

total trip time

speedup over 747

8.5 hours

16.5 hours

SST

3.75 hours

11.75 hours

Taking the SST (which is 2.2 times faster) speeds up

the overall trip by only a factor of 1.4!

CS 213 F’

• 4 –

class23.ppt

Amdahl’s law (cont)

Two key parameters:

F

enhanced

= T

2

/ T

(fraction of original time that can be improved)

S

enhanced

= T

2

/ T

2

(speedup of enhanced part)

T’ = T

1

’ + T

2

’ = T

1

+ T

2

’ = T(1-F

enhanced

) + T

2

’

= T(1-F

enhanced

) + (T

2

/S

enhanced

[by def of S

enhanced

]

= T(1-F

enhanced

) + T(F

enhanced

/S

enhanced

[by def of F

enhanced

]

= T((1-F

enhanced

) + F

enhanced

/S

enhanced

Amdahl’s Law:

S

overall

= T / T’ = 1/((1-F

enhanced

) + F

enhanced

/S

enhanced

just computer programs.of diminishing returns. It applies to any activity, notKey idea: Amdahl’s law quantifies the general notion

• 5 –

class23.ppt

Amdahl’s law (cont)

Trip example: Suppose that for the New York to Paris

of space-time (0 minutes):rocket ship (15 minutes) or a handy rip in the fabricleg, we now consider the possibility of taking a

time NY->Paris

total trip time

speedup over 747

8.5 hours

16.5 hours

SST

3.75 hours

11.75 hours

rocket

0.25 hours

8.25 hours

rip

0.0 hours

8 hours

• 7 –

class23.ppt

Characterizing computer performance

Computer buyers want a single number that predicts

performance of real applications.

Computer makers have resisted measures that would

allow meaninful direct comparisons.

lack of operating system and language standards

difficult to develop portable and realistic applications

1970’s and 1980’s:

era of meaningless rates (e.g, MIPS)

1980’s:

age of meaningless benchmarks (e.g., Whetstone)

1990’s:

dawn of semi-realistic benchmarks (e.g., SPEC CPU95)

• 8 –

class23.ppt

Meaningless rate #1: MHz

MHz doesn’t predict running time:MHz = millions of clock cycles/sec

T secs = I inst x (C cycles/I inst) x 1/(MHz x 10^6) cycles/sec

CPU

MHz

System

SPECfp95 time (secs)

Pentium Pro

Alder

POWER

RS/6000 591

CS 213 F’

• 10 –

class23.ppt

Meaningless rate #3: peak MFLOPS

peak MFLOPS = MFLOPS for some optimal instructionMFLOPS = millions of floating operations /sec

stream.

MFLOPS doesn’t predict execution time:

floating point operations do not predict running time

even if the did, the ideal instruction stream is usually unrealistic

ProgramMeasured MFLOPS on Intel i860 (peak MFLOPS = 80):

1d fft

sasum

saxpy

sdot

sgemm sgemv

spvma

MFLOPS

%peak

• 11 –

class23.ppt

Benchmarking

Goal: Measure a set of programs (benchmarks) that

predict the running time of those applications.represent the workload of real applications and that

Steps in the benchmarking process:

(1) Choose representative benchmark programs.

• difficult to find realistic AND portable programs.

(2) Choose an individual performance measure (for each benchmark)

• time, normalized time, rate?

(3) Choose an aggregate performance measure (for all benchmarks)

• sum, normalized sum, mean, normalized mean?

• 13 –

class23.ppt

Benchmark examples

(Toy) Benchmarks

10-100 line

e.g.,: sieve, puzzle, quicksort

Synthetic Benchmarks

attempt to match average frequencies of real workloads

e.g., Whetstone, Dhrystone

Kernels

Time critical excerpts of REAL programs

element models.compression, sparse matrix vector product from unstructured finitee.g., 8x8 Discrete Cosine Transform (DCT) from JPEG and MPEG

• 14 –

class23.ppt

Successful Benchmark Suite: SPEC

www.specbench.org/osg/

1987: RISC industry mired in “bench marketing”:

“Egads! That is an 8 MIPS machine, but they claim 10 MIPS!”

1988 : EE Times + 5 companies band together to

(SPEC) in 1988perform Systems Performance Evaluation Committee

Sun, MIPS, HP, Apollo, DEC

Create standard list of programs, inputs, reporting:

some real programs, includes OS calls, some I/O

Currently SPEC is more than 40 computer companies:

SGI, SunCompaq, Cray, DEC, HP, Hitachi, IBM, Intel, Motorola, Netscape,

• 16 –

class23.ppt

SPEC95 integer benchmarks

benchmark

description

go

plays a game of go

m88ksim

Motorola 88k chip simulator

gcc

Gnu C compiler

compress

in-memory LZW file compression

li

Lisp interpreter

jpeg

spectral based image compression/decompression

perl

Perl program that manipulates strings and primes

vortex

database program

• 17 –

class23.ppt

SPEC95 floating point benchmarks

benchmark

description

tomcatv

mesh generation program

swim

513x513 shallow water finite difference model

su2cor

Monte Carlo simulation

hydro2d

2D Navier-Stokes solver

mgrid

3D multigrid solver

applu

parabolic/elliptic PDE solver

turb3d

turbulence model

apu

air pollution model

fppp

quantum chemistry model

wave

electromagnetic particle model

CS 213 F’

• 19 –

class23.ppt

Lying with means and ratios

frames

sys A

sys B

sys C

prog 1

20 secs

10 secs

40 secs

prog 2

40 secs

80 secs

20 secs

total

60 secs

90 secs

60 secs

Total running time is the ultimate performance measure.

CS 213 F’

• 20 –

class23.ppt

Lying with means and ratios (cont)

seconds

frames

sys A

sys B

sys C

prog 1

prog 2

total

normalized to A

normalized to B

normalized to C

Normalized total running time is OK too. It tracks with total running time.