U
UNIVERSITY OF
NIVERSITY OF M
MASSACHUSETTS
ASSACHUSETTS A
AMHERST
MHERST •
•Department of Computer Science
Department of Computer Science
Parallel & Concurrent
Programming:
Message-Passing
Emery Berger
CMPSCI 691W
Spring 2006
Prepare for your exams
Study with the several resources on Docsity
Earn points to download
Earn points by helping other students or get them with a premium plan
Guidelines and tips
Prepare for your exams
Study with the several resources on Docsity
Earn points to download
Earn points by helping other students or get them with a premium plan
Parallel and Concurrent Programming: Message-Passing | CMPSCI 691, Study notes of Computer Science
Material Type: Notes; Class: S-Gen Purpose Cmptn-GPU; Subject: Computer Science; University: University of Massachusetts - Amherst; Term: Spring 2006;
Typology: Study notes
Pre 2010
1 / 28
This page cannot be seen from the preview
Don't miss anything!
Related documents
Partial preview of the text
Download Parallel and Concurrent Programming: Message-Passing | CMPSCI 691 and more Study notes Computer Science in PDF only on Docsity!
U^ NIVERSITY OFUNIVERSITY OF
MM
ASSACHUSETTSASSACHUSETTS
AA
MHERST •MHERST
-^ Department of Computer ScienceDepartment of Computer Science
Parallel & ConcurrentProgramming:Message-Passing
Emery BergerCMPSCI 691W
Spring 2006
U^ NIVERSITY OFUNIVERSITY OF
MM
ASSACHUSETTSASSACHUSETTS
AA
MHERST •MHERST
-^ Department of Computer ScienceDepartment of Computer Science
Outline
Today:^
Distributed parallel programming via^ message-passing
MPI
library
approach
some material adapted from slides by Kathy Yelick
U^ NIVERSITY OFUNIVERSITY OF
MM
ASSACHUSETTSASSACHUSETTS
AA
MHERST •MHERST
-^ Department of Computer ScienceDepartment of Computer Science
From Pat Worley, ORNL
Scaling Limits ^
Kernel
used in
atmosphericmodels^
99% floatingpoint ops;multiplies/adds
^
Sweeps throughmemory withlittle reuse
^
One “copy” ofcode runningindependently onvarying numbersof procs
U^ NIVERSITY OFUNIVERSITY OF
MM
ASSACHUSETTSASSACHUSETTS
AA
MHERST •MHERST
-^ Department of Computer ScienceDepartment of Computer Science
Distributed Memory
Distributed memory machines: local memory
but
no global memory
Individual
nodes
often SMPs
Network interface
for all interprocessor
communication
interconnect
P
memory
NI
...
P
memory
NI
Pn memory
NI
U^ NIVERSITY OFUNIVERSITY OF
MM
ASSACHUSETTSASSACHUSETTS
AA
MHERST •MHERST
-^ Department of Computer ScienceDepartment of Computer Science
Message Passing
Pros:
efficient
Makes data sharing explicit
Can communicate only what is strictlynecessary for computation
^
No coherence protocols, etc.
Cons:
difficult
Requires
manual partitioning
Unnatural model
Deadlock-prone
Not portable (previously)
U^ NIVERSITY OFUNIVERSITY OF
MM
ASSACHUSETTSASSACHUSETTS
AA
MHERST •MHERST
-^ Department of Computer ScienceDepartment of Computer Science
Portability
Bad old days:
each vendor
had own message-passingsolution = no portability^
Tied to different
network
topologies
^
Bus
,^ star
,^ hypercube
, ...
^
Vastly different
platforms
^
SMP
boxes
^
Beowulf
clusters
^
Supercomputers
Goal: write once, runeverywhere
U^ NIVERSITY OFUNIVERSITY OF
MM
ASSACHUSETTSASSACHUSETTS
AA
MHERST •MHERST
-^ Department of Computer ScienceDepartment of Computer Science
MPI execution model
Spawns multiple copies of
same
program (
SPMD
Each = different “process”(different local memory)
Can act differently by determiningwhich processor “self” corresponds to
U^ NIVERSITY OFUNIVERSITY OF
MM
ASSACHUSETTSASSACHUSETTS
AA
MHERST •MHERST
-^ Department of Computer ScienceDepartment of Computer Science
An Example %
mpirun –np 10
exampleProgram
#include <stdio.h>#include <mpi.h>int main(int argc, char * argv[]) {
int rank, size; MPI_Init
(&argc, &argv );
MPI_Comm_size
( MPI_COMM_WORLD
, &size);
MPI_Comm_rank
( MPI_COMM_WORLD
, &rank);
printf("Hello world from process %d of %d\n",
rank, size);
MPI_Finalize
();
return 0; }
U^ NIVERSITY OFUNIVERSITY OF
MM
ASSACHUSETTSASSACHUSETTS
AA
MHERST •MHERST
-^ Department of Computer ScienceDepartment of Computer Science
An Example %
mpirun –np 10
exampleProgram
#include <stdio.h>#include <mpi.h>int main(int argc, char * argv[]) {
int rank, size; MPI_Init
(&argc, &argv );
MPI_Comm_size
( MPI_COMM_WORLD
, &size);
MPI_Comm_rank
( MPI_COMM_WORLD
, &rank);
printf("Hello world from process %d of %d\n",
rank, size);
MPI_Finalize
();
return 0; }
returns # ofprocessors in
“world”
U^ NIVERSITY OFUNIVERSITY OF
MM
ASSACHUSETTSASSACHUSETTS
AA
MHERST •MHERST
-^ Department of Computer ScienceDepartment of Computer Science
An Example %
mpirun –np 10
exampleProgram
#include <stdio.h>#include <mpi.h>int main(int argc, char * argv[]) {
int rank, size; MPI_Init
(&argc, &argv );
MPI_Comm_size
( MPI_COMM_WORLD
, &size);
MPI_Comm_rank
( MPI_COMM_WORLD
, &rank);
printf("Hello world from process %d of %d\n",
rank, size);
MPI_Finalize
();
return 0; }
which processor am
I?
U^ NIVERSITY OFUNIVERSITY OF
MM
ASSACHUSETTSASSACHUSETTS
AA
MHERST •MHERST
-^ Department of Computer ScienceDepartment of Computer Science
Message Passing
Messages can be sent directly toanother processor^
MPI_Send
,^
MPI_Recv
Or to all processors^
MPI_Bcast
(does send
or
receive)
U^ NIVERSITY OFUNIVERSITY OF
MM
ASSACHUSETTSASSACHUSETTS
AA
MHERST •MHERST
-^ Department of Computer ScienceDepartment of Computer Science
Broadcast ^
Repeatedly broadcast input (one integer) to all
#include <stdio.h>#include <mpi.h>int main(int argc, char * argv[]) {
int rank, value; MPI_Init
( &argc, &argv );
MPI_Comm_rank
(
MPI_COMM_WORLD
, &rank );
do {
if (rank == 0)
scanf( "%d", &value ); MPI_Bcast
( &value, 1,
MPI_INT
, ^0
,
MPI_COMM_WORLD
);
printf( "Process %d got %d\n", rank, value ); } while (value >= 0); MPI_Finalize
( );
return 0; }
U^ NIVERSITY OFUNIVERSITY OF
MM
ASSACHUSETTSASSACHUSETTS
AA
MHERST •MHERST
-^ Department of Computer ScienceDepartment of Computer Science
Broadcast ^
Repeatedly broadcast input (one integer) to all
#include <stdio.h>#include <mpi.h>int main(int argc, char * argv[]) {
int rank, value; MPI_Init
( &argc, &argv );
MPI_Comm_rank
(
MPI_COMM_WORLD
, &rank );
do {
if (rank == 0)
scanf( "%d", &value ); MPI_Bcast
( &value, 1,
MPI_INT
,
0 ,^
MPI_COMM_WORLD
);
printf( "Process %d got %d\n", rank, value );
} while (value >= 0); MPI_Finalize
( );
return 0; }
how many tosend/receive?
U^ NIVERSITY OFUNIVERSITY OF
MM
ASSACHUSETTSASSACHUSETTS
AA
MHERST •MHERST
-^ Department of Computer ScienceDepartment of Computer Science
Broadcast ^
Repeatedly broadcast input (one integer) to all
#include <stdio.h>#include <mpi.h>int main(int argc, char * argv[]) {
int rank, value; MPI_Init
( &argc, &argv );
MPI_Comm_rank
(
MPI_COMM_WORLD
, &rank );
do {
if (rank == 0)
scanf( "%d", &value ); MPI_Bcast
( &value, 1,
MPI_INT
,
0 ,^
MPI_COMM_WORLD
);
printf( "Process %d got %d\n", rank, value );
} while (value >= 0); MPI_Finalize
( );
return 0; }
what’s thedatatype?