Chapter 5: CS621 1
Parallel and Distributed Computing
Chapter 5: Basic Communications
Operations
Jun Zhang
Laboratory for High Performance Computing & Computer Simulation
Department of Computer Science
University of Kentucky
Lexington, KY 40506
Prepare for your exams
Study with the several resources on Docsity
Earn points to download
Earn points by helping other students or get them with a premium plan
Guidelines and tips
Prepare for your exams
Study with the several resources on Docsity
Earn points to download
Earn points by helping other students or get them with a premium plan
Parallel and Distributed Computing: Basic Communications Operations in Parallel Systems - , Study notes of Computer Science
A chapter from a computer science textbook focusing on the fundamental concepts of communication operations in parallel systems. It covers topics such as nearest neighbor and remote node communication, cut-through routing, one-to-one, one-to-all, all-to-all, and all-to-one communication, as well as commonly used interconnections like linear array, two-dimensional mesh, and hypercube. The document emphasizes the importance of efficiently implementing these communication operations on various parallel architectures.
Typology: Study notes
Pre 2010
1 / 46
This page cannot be seen from the preview
Don't miss anything!
Related documents
Partial preview of the text
Download Parallel and Distributed Computing: Basic Communications Operations in Parallel Systems - and more Study notes Computer Science in PDF only on Docsity!
Chapter 5: CS
Parallel and Distributed ComputingChapter 5: Basic CommunicationsOperations
Jun Zhang
Laboratory for High Performance Computing & Computer Simulation
Department of Computer Science
University of KentuckyLexington, KY 40506
Chapter 5: CS
5.1a: Communication in Parallel System
Nearest neighbor communication:^ Communication between two directly linknodes
Remote node communication
: With more
than one links between the communicatingnodes1.) Store-and-forward routing2.) Cut-through routing
Chapter 5: CS
5.1c: Commonly Used Interconnections
Linear array
Two-dimensional mesh
Hypercube
Chapter 5: CS
5.1d: Mesh Topology ^
A large number of processors can be connectedrelatively inexpensively with mesh topology ^
Many applications map naturally onto a meshnetwork ^
The disadvantage of high diameter of meshtopology can be diminished for networks with cut-through routing ^
Several commercially available parallel computersare based on mesh network ^
T3D, SGI, IBM Blue Gene
Chapter 5: CS
5.2a: Basic Assumptions
Network supports store-and-forward routingand cut-through routing
The communication links are bidirectional
Single-port communication modelOne node can only send one message at atimeIt can only receive one message at a timeSend and receive can be donesimultaneously
Chapter 5: CS
5.2b: Dual Operations
A
dual
of a communication is the opposite of
the original operation
It can be performed by reversing the directionand sequence of messages in the originaloperationE.g., All-to-one communication (reduction) isthe dual of one-to-all broadcast.
Chapter 5: CS
5.3b: Ring Network
0
1
2
4 3
5
6
7
Message of size
m
at node 0, to be sentto all other nodes inthe networkNaïve algorithmBetter algorithmFast algorithmRecursive doubling
Chapter 5: CS
5.3c: Mesh Network
0
1
2
3
4
7
6
5
8
9
10
15 11
14
13
12
One
-^
to-^
all broadcast in
a 2D mesh can beperformed in two steps,each step is a one
-^
to- a
ll
broadcast using the ringalgorithm
Chapter 5: CS
5.3e: Balanced Binary Tree
Only the root nodes areprocessing nodes,map the hypercubealgorithm directly
Chapter 5: CS
5.3f: Communication Cost^
If we assume cut-through routing and ignore the perhop time, all one-to-all broadcast communicationscan be viewed as log
p
steps of point-to-point
communications. ^
The communication cost for all networks is the same:
) log
comm
s^
w
T
t
t m
p
Chapter 5: CS
5.4b: Ring and Linear Array ^
All-to-all broadcast is achieved by a pipelined point-to-point nearest neighbor communication ^
For linear array, bi-directional link is necessary ^
For all-to-all reduction, the procedure is reversed,each node needs to perform the operation at eachstep ^
The total communication cost is:
(
)(
ring
s^
w
T
t^
t m
p
=
−
Chapter 5: CS
5.4c: All-to-All on a Ring Network
0
1
2
4 3
5
6
7
Chapter 5: CS
5.4e: Illustration of 2D Mesh All-to-All
0
1
2
3
4
7
6
5
8
9
10
15 11
14
13
12
Chapter 5: CS
5.4f: Cost of 2D Mesh All-to-All ^
In the first phase, the message size is
m
, the
number of links is ^
In the second phase, the message size is
and the number of links is ^
The total communication cost is the sum of bothphases:
(^
)(
(^
)(
2 (
(^
comm
s^
w^
s^
w
s^
w
t^
t^
t m
p
t^
t m
p
p
t^
p
t m p
=
−
−
=
−
−
m
p
(^
p^
−
(^
p^
−