Beyond Low Level Parallelism - Computer Architecture | ECE 511, Study notes of Computer Architecture and Organization

Download Beyond Low Level Parallelism - Computer Architecture | ECE 511 and more Study notes Computer Architecture and Organization in PDF only on Docsity!

Lecture 19

Beyond Low-level Parallelism

Outline

• Models for exploiting large grained

parallelism

• Issues in parallelization: communication,

synchronization, Amdahl’s Law

• Basic Single-Bus Shared Memory Machines

Dimension 2: Flynn’s Classification

Single Instruction, Single Data (SISD)

• simple CPU (1-wide pipeline)

  Single Instruction, Multiple Data (SIMD) - vector computer, multimedia processors - Multiple Instruction, Single Data (MISD) - ???, systolic arrays perhaps? - Multiple Instruction, Multiple Data (MIMD) - multiprocessor (now also, superscalar CPU) – Also: Single Program Multiple Data (SPMD)

Dimension 3: Parallel Programming

Models

Data Parallel Model:

• The same operation is applied to a large data set. All such applications can be done in parallel. - E.g. Matrix multiplication: all elements of the product array can be generated in parallel. - Function Parallel Model: - A program is partitioned into parallel modules. – E.g.“ Compiler: phases of a compiler can be made into parallel modules: parser, semantic analyzer, optimizer1, optimizer 2, and code generator. They can beorganized into a high-level pipeline.

Memory Architecture Models CPU CPU CPU CPU Interconnect Mem Mem Mem Shared Memory Model Message Passing Model (amenable to single address space) (amenable to multiple address spaces)

Memory Architecture (Cont.)

Message passing (or Distributed Memory)

• Each processor has a local memory. For a processor to access a piece of data not present in its own localmemory, the processor has to exchange message withthe processor whose local memory contains the data. - For each data access, the programmer mustdetermine if a thread is executed by the processorwhose local memory contains the data. - In general, the programmer has to partition the data and assign them explicitly to the processor local memories. - The programmer then parallelizes the program and assign the threads to processors. Using the assignmentdecisions, each access is then determined to be either alocal memory access or a message exchange.

Simple Problem, Data Parallelized

pstart = p# * 100000; min[p#] = D[0]; for (i = pstart; i < pstart+100000; i++) { if (D[i] < min[p#]) min[p#] = D[i]; } barrier(); if (p# == 0) { real_min = min[0]; for (i = 1; i < pMax; i++) { if (min[i] < real_min) real_min = min[i]; } cout << real_min; }

Loop Level Parallelization

• DOALL: There is no dependence between

loop iterations. All can be executed inparallel.

– DOALL 30 J=1,J

• X(II1+J) = X(II1+J) * SC1 • Y(II1+J) = Y(II1+J) * SC1 • Z(II1+J) = Z(II1+J) * SC
  • 30

CONTINUE

DOACROSS Example

• DOACROSS 40 I=4,IL

– AWAIT(1, 3) – X(I) = Y(I) + X(I-3) – ADVANCE (1)

• 40

CONTINUE

• The current iteration depends on synch

point 1 of three iterations ago.

Issue 1: Communication

Objective is to write parallel programs that can use asmany processors as are available.

Parallelization is limited by communication. - In terms of latency – In terms of amount of communication

Mutual Exclusion

Atomic update to data structures so that theactions from different processors are serialized.

• This is especially important for data structures that need to be updated in multiple steps. Each processor shouldbe able to lock out other processors, update the datastructure, leave the data structure in a consistent state,and unlock. - This is usually realized by using atomic Read Modify Write instructions. - Park space example, file editing example

Issue 3 : Serial Component

Also, no matter how much you parallelize, theperformance is ultimately limited by the serialcomponent.

Better known as Amdahl’s Law

Jet Service to Chicago

Message Passing Machines

• Network characteristic are of primary

importance

• Latency, bisection bandwidths, node

bandwidth, occupancy of communication

Latency = sender overhead + transport latency + receiver overhead

© Wen-mei Hwu and S. J. Patel, 2002ECE 412, University of Illinois

Synchronization support for

shared memory machines

Example: two processors trying to increment a sharedvariable

Requires an atomic memory update instr - example: test and set, or compare and exchange - Using cmpxchg, implement barrier(); - Scalability issues with traditional synchronization