Understanding Parallelism, Concurrency, and Scalability in Extreme Computing, Study notes of Computer Science

Download Understanding Parallelism, Concurrency, and Scalability in Extreme Computing and more Study notes Computer Science in PDF only on Docsity!

Extreme Computing: Parallelism

8/12/

Parallelism

Part 1

Extreme Computing: Parallelism

8/12/

Parallel

• Webster: side by side• Web definitions:
  • Two lines that are equidistant apart at all points– Simultaneous performance of multiple operations– Simultaneous and essentially independent processing in

different central processing units

• Our definitions:
  • “

Executed in parallel

”: executed at exactly the same time

• “

Parallel hardware

:” multiple, usually identical,

hardware, each capable of independent computation

• “

Parallel program

:” Program which divides computation

into steps that might in fact be executed “in parallel” on“parallel hardware”

Extreme Computing: Parallelism

8/12/

Concurrency

• Webster
  • to occur at the same time– to act together
    • Concurrency (web definitions)
      • Execution of two processes or operations simultaneously– Doing several things at once; includes time slicing of

processes as well as parallelism

• More than one process competing for processor time at the

same time, and potentially interacting with each other

• Other
  • “Action occurs

simultaneously

everywhere at same time”

(Lester, The Art of Parallel Programming, 2006, p. 29)

• Our Definition: overlap of operations as seen during

program execution

Extreme Computing: Parallelism

8/12/

Concurrency

• Web definitions
  • Execution of two processes or operations simultaneously– Doing several things at once; includes time slicing of

processes as well as parallelism

• More than one process competing for processor time at the

same time, and potentially interacting with each other

• Our definition: How many different actions can be

in state of computation at the same time

• Related:

degree of concurrency

= # of distinct

actions that are in an active state of execution atsame time

Extreme Computing: Parallelism

8/12/

Scalability

• Merriam-Webster’s Online: capable of being

easily upgraded or upgraded on demand

• The ease with which a system or component can

be modified to fit the problem area.

• Wikipedia:
  • desirable property of a system, a network, or a process,

which indicates its ability to either handle growing amountsof work in a graceful manner, or to be readily enlarged

• property of natural and economic systems … a property

scalable when it is subject to extremes

Extreme Computing: Parallelism

8/12/

Scalability

(Still from Wikipedia)

• A system whose performance improves after

adding hardware, proportionally to the capacityadded, is said to be a

scalable system

• An algorithm, design, networking protocol,

program, or other system is said to scale if– it is suitably efficient and practical when applied to large

situations (e.g. a large input data set or large number ofparticipating nodes in the case of a distributed system).

• If the design fails when the quantity increases

then it does not scale.

Extreme Computing: Parallelism

8/12/

To Scale

• From Wikipedia:
  • To scale up/vertically: add resources to a single node

in a system, typically involving the addition of CPUsor memory to a single computer

• To scale out/horizontally: to add more nodes to a

system, such as adding a new computer to adistributed software application

• Scaling has been discussed in terms of
  • Application performance– Hardware resources– Software

Extreme Computing: Parallelism

8/12/

Work and Performance Metrics

Work

: total number of basic operations

that need to be executed in a program

Performance metric

: measure of work

performed per unit time–

Peak performance metric

: absolute maximum

work per unit time that can be done by a processorfor any algorithm

• Processor specific but not algorithm specific

Sustained performance metric

: work per second

observed when running a specific algorithm

• algorithm and processor specific

Extreme Computing: Parallelism

8/12/

Precise Definitions

• T

A,X

(N,P) = time to solution

• for some algorithm A– when executed on a processor system with

architecture X

• where the size of the problem is N– and the number of processors available is P
  • We term A,X as “the system”
    • And drop subscripts from now on

Extreme Computing: Parallelism

8/12/

Application Scaling Definitions

• Define: T(N,P) = solution time for problem of size

N using P processors

Strong Scaling

: T(N,P) ~ T(N,1)/P

• Given fixed sized problem– Time decreases as P increases–^

Perfect strong scaling

if time decrease = 1/P

• Perfect linear speedup• Problem piece solved per processor also drops as 1/P

Weak Scaling

: T(1,1) ~ T(P,P)

• For a fixed solution time– Solvable problem size increases with processor count–^

Perfect weak scaling

if problem size proportional to P

• Problem piece solved per processor is constant

Extreme Computing: Parallelism

8/12/

Time to Solution Surface:

Single Processor Curve (1,1,1)

T(N,P): Time to Solution

P: # of Processors

N: Problem Size T(N,1)

Sequential time complexity curve

Note: assume log axes,with time normalizedso origin is T(1,1) =

Extreme Computing: Parallelism

8/12/

Perfect Strong Scaling:

T(N,P) = T(N,1)/P

Curve in NxP plane is

reflection of curve

in NxT plane

Lines are 1/P

(1,1,1)

T(N,P): Time to Solution

P: # of Processors

N: Problem Size T(N,1)

When P = T(N,1) T(1,P) = T(P,1)

then

T(N,P) = T(N,1)/P= T(N,1)/T(N,1)

= 1

Enough processorsare used to keep

problem solution time = 1

Less than perfect strong scaling is

in front of

surface

Extreme Computing: Parallelism

8/12/

Weak Scaling

Lines of ConstantExecution Time

(1,1,1)

T(N,P): Time to Solution

P: # of Processors

N: Problem Size T(N,1)

T(X,X)=

Lines of ConstantExecution Time

(1,1,1)

T(N,P): Time to Solution

P: # of Processors

N: Problem Size T(N,1)

T(X,X)=

Extreme Computing: Parallelism

8/12/

Some Weak Scaling Lemmas

If work delivered by a processor to a solution for a

particular system is independent of N and P:

then system exhibits perfect weak scaling if

T(N,1) = N*T(1,1)

i.e. the sequential time complexity is linearin problem size

then if system exhibits perfect weak scalingthen it also exhibits perfect strong scaling

then the scaling surface is represented byT(N,P) = N*T(1,1)