Basic Communication Operations 2-Parallel Processing-Lecture Slides, Slides of Parallel Computing and Programming

Download Basic Communication Operations 2-Parallel Processing-Lecture Slides and more Slides Parallel Computing and Programming in PDF only on Docsity!

Basic Communication Operations

Communication Patterns in Different

Topologies

• One-to-All Broadcast and All-to-One

Reduction

• All-to-All Broadcast and Reduction

& All-Reduce Operations

• Scatter and Gather (one to all & all to

one personalized comm)

All-to-All Broadcast and Reduction

All-to-all broadcast and all-to-all reduction.

All-to-All Broadcast & Reduction

on a Ring

•^

Simplest approach: perform

p

one-to-all broadcasts

This is not the most efficient way, though.

-^

Each node first sends to one of its neighbors thedata it needs to broadcast.

-^

In subsequent steps, it forwards the data receivedfrom one of its neighbors to its other neighbor.

-^

The algorithm terminates in

p-

steps.

All-to-all broadcast on a Hypercube

-^

Generalization of the mesh algorithm to

log p

dimensions.

-^

Message size doubles at each of the

log p

steps.

All to All Broadcast Cost Analysis

-^

On a ring, the time is given by:

(t

s^

+ t

w

m)(p-1)

-^

On a Torus, the time is given by:

2t

(s^

p – 1) + t

w

m(p-1)

-^

On a hypercube, we have:

Scatter and Gather Operations

Scatter and gather operations.

Scatter and Gather

•^

In the

scatter

operation, a single node sends a unique

message of size

m

to every other node (also called a

one-to-all personalized communication).

-^

In the

gather

operation, a single node collects a unique

message from each node.

-^

While the scatter operation is fundamentally differentfrom broadcast, the algorithmic structure is similar,except for differences in message sizes (messages getsmaller in scatter and stay constant in broadcast).

-^

The gather operation is exactly the inverse of the scatteroperation and can be executed as such.