High Performance Computing Lecture 40: Message Passing Interface (MPI) with Docsity.com, Slides of Computer Science

Download High Performance Computing Lecture 40: Message Passing Interface (MPI) with Docsity.com and more Slides Computer Science in PDF only on Docsity!

High Performance Computing

Lecture 40

3 Message Passing Interface (MPI)

Standard API

 Hides software/hardware details  Portable, flexible

Implemented as a library

Your program MPI Library Custom software Standard TCP/IP Standard network HW Custom hardware

5 Making MPI Programs

 Executable must be built by compiling program

and linking with MPI library

 Header files (mpi.h) provide definitions and

declarations

 MPI commonly used in SPMD mode

 One executable file

 Multiple instances of it executed in parallel

 Implementations provide a command to initiate

execution of MPI processes (mpirun)

 Options: number of processes, which

processors they are to run on

6 MPI Communicators

 Defines communication domain of a

communication operation: set of processes

that are allowed to communicate among

themselves

 Initially all processes are in the communicator

MPI_COMM_WORLD

 Processes have unique ranks associated with

communicator, numbered from 0 to n- 1

 Other communicators can be established for

groups of processes

8 Example MPI_Comm_rank(MPI_COMM_WORLD,&myrank); if (myrank == 0) { int x; MPI_Send(&x, 1, MPI_INT, 1, msgtag, MPI_COMM_WORLD); } else if (myrank == 1) { int x; MPI_Recv(&x, 1, MPI_INT, 0,msgtag,MPI_COMM_WORLD,status); }

9 MPI Message Tag

 Cooperating processes may need to send

several messages between each other

 Message tag: Used to differentiate between

different types of messages being sent

 The message tag is carried within the

message and used in both send and receive

calls

11 MPI Message Tag

 Cooperating processes may need to send

several messages between each other

 Message tag: Used to differentiate between

different types of messages being sent

 Message tag is carried within the message

and used in both send and receive calls

 If special matching is not required, a wild card

message tag is used so that the receive will

match with any send

 MPI_ANY_TAG

12 MPI: Matching Sends and Recvs

 Sender always specifies destination and tag

 Receiver can specify for exact match or using

wild cards

 MPI_ANY_SOURCE

 MPI_ANY_TAG

Flavours of Sends/Receives

Synchronous

Asynchronous

14 Asynchronous Message Passing

 Send/receive do not wait for actions to

complete before returning

 Usually require local storage for messages

 In general, they do not synchronize

processes but allow processes to move

forward sooner

15 Parameters of Send

MPI_Send ( buf, count, datatype, dest, tag, comm)

Address of send buffer Number of items to send Datatype of each item Rank of destination process Message tag Communicator

17 Non-blocking Routines

 MPI_Isend (buf, count, datatype, dest, tag,

comm, request)

 MPI_Irecv (buf, count, datatype, source, tag,

comm, request)

 Completion detected by MPI_Wait() and

MPI_Test()

 MPI_Wait() waits until operation completed and then returns  MPI_Test() returns with flag set indicating whether or not operation has completed

18 MPI Group Communication

 Until now we have looked at what are called

point-to-point messages

 MPI also provides routines that sends messages

to a group of processes or receive messages

from a group of processes

 Not absolutely necessary for programming  More efficient than separate point-to-point routines

 Examples: broadcast, gather, scatter, reduce,

barrier

 MPI_Bcast, MPI_Reduce, MPI_Allreduce, MPI_Alltoall, MPI_Scatter, MPI_Gather, MPI_Barrier

20 MPI Broadcast

MPI_Bcast (void *buf,

int count,

MPI_Datatype datatype,

int root,

MPI_Comm Comm )

21

Scatter

Process 0 (^) Process 1 Process n- 1 data buf MPI_scatter(..); data MPI_scatter(..); data MPI_scattert(..);