MPI: comunicazioni collettive

Transcript

1 - Gruppo Supercalcolo - Dipartimento Sistemi e Tecnologie 29 settembre 5 ottobre 2008 Collective Communications -Communications involving a group of process -Called by all processes in a communicator Barrier Synchronization Broadcast Gather/Scatter Reduction (sum, max, prod, ) 1

2 Characteristics Collective communications do not interfere with point-to-point communication and vice-versa All processes must call the collective routine No non-blocking collective communication No tags Receive buffers must be exactly the right size Safest communication mode 2 MPI_Barrier Stop processes until all processes within a communicator reach the barrier Fortran: CALL MPI_BARRIER(comm, ierr) C: int MPI_Barrier(MPI_Comm comm) 3

3 Barrier t 1 t 2 t 3 P 2 P 1 P 3 P 4 P 1 P 2 P 3 P 4 barrier barrier P 1 P 2 P 3 P 4 t 4 Broadcast (MPI_BCAST) One-to-all communication: same data sent from root process to all others in the communicator Fortran: INTEGER count, type, root, comm, ierr CALL MPI_BCAST(buf, count, type, root, comm, ierr) Buf array of type type C: int MPI_Bcast(void *buf, int count, MPI_Datatype, datatypem int root, MPI_Comm comm) All processes must specify same root, rank and comm 5

4 Broadcast PROGRAM broad_cast INCLUDE 'mpif.h' INTEGER ierr, myid, nproc, root INTEGER status(mpi_status_size) REAL A(2) CALL MPI_INIT(ierr) CALL MPI_COMM_SIZE(MPI_COMM_WORLD, nproc, ierr) CALL MPI_COMM_RANK(MPI_COMM_WORLD, myid, ierr) root = 0 IF( myid.eq. 0 ) THEN P 1 a(1) = 2.0 a(2) = 4.0 END IF CALL MPI_BCAST(a, 2, MPI_REAL, 0, MPI_COMM_WORLD, ierr) WRITE(6,*) myid, ': a(1)=', a(1), 'a(2)=', a(2) CALL MPI_FINALIZE(ierr) END P 2 P 3 6 Scatter / Gather Scatter Gather sndbuf sndbuf sndbuf sndbuf sndbuf P a a a 3 a 4 P P 1P P 1 2P P a 0 1 a 2 a 2 3 3P a 3 4 P 1 a 2 P 2 a 3 P 3 a 4 rcvbuf rcvbuf rcvbuf rcvbuf a 2 a 3 a 4 rcvbuf 7

5 MPI_Scatter One-to-all communication: different data sent from root process to all others in the communicator Fortran: CALL MPI_SCATTER(sndbuf, sndcount, sndtype, rcvbuf, rcvcount, rcvtype, root, comm, ierr) Arguments definition are like other MPI subroutine sndcount is the number of elements sent to each process, not the size of sndbuf, that should be sndcount times the number of process in the communicator The sender arguments are meaningful only for root 8 9

6 MPI_SCATTERV Usage int MPI_Scatterv( void* sendbuf, /* in */ int* sendcounts, /* in */ int* displs, /* in */ MPI_Datatype sendtype, /* in */ void* recvbuf, /* in */ int recvcount, /* in */ MPI_Datatype recvtype, /* in */ int root, /* in */ MPI_Comm comm); /* in */ Description Distributes individual messages from root to each process in communicator Messages can have different sizes and displacements 10 11

7 MPI_Gather One-to-all communication: different data collected by the root process, from all others processes in the communicator. Is the opposite of Scatter Fortran: CALL MPI_GATHER(sndbuf, sndcount, sndtype, rcvbuf, rcvcount, rcvtype, root, comm, ierr) Arguments definition are like other MPI subroutine rcvcount is the number of elements collected from each process, not the size of rcvbuf, that should be rcvcount times the number of process in the communicator The receiver arguments are meaningful only for root 12 13

8 MPI_GATHERV Usage int MPI_Gatherv( void* sendbuf, /* in */ int sendcount, /* in */ MPI_Datatype sendtype, /* in */ void* recvbuf, /* out */ int* recvcount, /* in */ int* displs, /* in */ MPI_Datatype recvtype, /* in */ int root, /* in */ MPI_Comm comm ); /* in */ Description Collects individual messages from each process in communicator to the root process and store them in rank order Messages can have different sizes and displacements 14 15

9 Scatter example PROGRAM scatter INCLUDE 'mpif.h' INTEGER ierr, myid, nproc, nsnd, I, root INTEGER status(mpi_status_size) REAL A(16), B(2) CALL MPI_INIT(ierr) CALL MPI_COMM_SIZE(MPI_COMM_WORLD, nproc, ierr) CALL MPI_COMM_RANK(MPI_COMM_WORLD, myid, ierr) root = 0 IF( myid.eq. root ) THEN DO i = 1, 16 a(i) = REAL(i) END DO END IF nsnd = 2 CALL MPI_SCATTER(a, nsnd, MPI_REAL, b, nsnd, & MPI_REAL, root, MPI_COMM_WORLD, ierr) WRITE(6,*) myid, ': b(1)=', b(1), 'b(2)=', b(2) CALL MPI_FINALIZE(ierr) 16 Gather example PROGRAM gather INCLUDE 'mpif.h' INTEGER ierr, myid, nproc, nsnd, I, root INTEGER status(mpi_status_size) REAL A(16), B(2) CALL MPI_INIT(ierr) CALL MPI_COMM_SIZE(MPI_COMM_WORLD, nproc, ierr) CALL MPI_COMM_RANK(MPI_COMM_WORLD, myid, ierr) root = 0 b(1) = REAL( myid ) b(2) = REAL( myid ) nsnd = 2 CALL MPI_GATHER(b, nsnd, MPI_REAL, a, nsnd, & MPI_REAL, root MPI_COMM_WORLD, ierr) IF( myid.eq. root ) THEN DO i = 1, (nsnd*nproc) WRITE(6,*) myid, ': a(i)=', a(i) END DO END IF CALL MPI_FINALIZE(ierr) 17

10 MPI_Alltoall Fortran: CALL MPI_ALLTOALL(sndbuf, sndcount, sndtype, rcvbuf, rcvcount, rcvtype, comm, ierr) a 2 a 3 a 4 b 1 c 1 d 1 sndbuf b 1 c 1 b 2 c 2 b 3 c 3 b 4 c 4 a 2 b 2 c 2 d 2 a 3 b 3 c 3 d 3 rcvbuf d 1 d 2 d 3 d 4 P a 0 4 b 4 c 4 d 4 Very useful to implement data transposition 18 Reduction The reduction operation allow to: Collect data from each process Reduce the data to a single value Store the result on the root processes Store the result on all processes Overlap of communication and computing 19

11 Reduce, Parallel Sum b 1 S a S b P 1 a 2 b 2 P 1 S a S b S a = +a 2 +a 3 +a 4 P 2 a 3 b 3 S b =b 1 +b 2 +b 3 +b 4 P 2 S a S b P 3 a 4 b 4 P 3 S a S b Reduction function works with arrays other operation: product, min, max, and,. 20 MPI_REDUCE and MPI_ALLREDUCE Fortran: MPI_REDUCE( snd_buf, rcv_buf, count, type, op, root, comm, ierr) snd_buf rcv_buf Count type op root comm ierr input array of type type containing local values. output array of type type containing global results (INTEGER) number of element of snd_buf and rcv_buf (INTEGER) MPI type of snd_buf and rcv_buf (INTEGER) parallel operation to be performed (INTEGER) MPI id of the process storing the result (INTEGER) communicator of processes involved in the operation (INTEGER) output, error code (if ierr=0 no error occours) MPI_ALLREDUCE( snd_buf, rcv_buf, count, type, op, comm, ierr) The argument root is missing, the result is stored to all processes. 21

12 Predefined Reduction Operations MPI op MPI_MAX MPI_MIN MPI_SUM MPI_PROD MPI_LAND MPI_BAND MPI_LOR MPI_BOR MPI_LXOR MPI_BXOR MPI_MAXLOC MPI_MINLOC Function Maximum Minimum Sum Product Logical AND Bitwise AND Logical OR Bitwise OR Logical exclusive OR Bitwise exclusive OR Maximum and location Minimum and location 22 Reduce / 1 C: int MPI_Reduce(void * snd_buf, void * rcv_buf, int count, MPI_Datatype type, MPI_Op op, int root, MPI_Comm comm) int MPI_Allreduce(void * snd_buf, void * rcv_buf, int count, MPI_Datatype type, MPI_Op op, MPI_Comm comm) 23

13 Reduce, example PROGRAM reduce INCLUDE 'mpif.h' INTEGER ierr, myid, nproc, root INTEGER status(mpi_status_size) REAL A(2), res(2) CALL MPI_INIT(ierr) CALL MPI_COMM_SIZE(MPI_COMM_WORLD, nproc, ierr) CALL MPI_COMM_RANK(MPI_COMM_WORLD, myid, ierr) root = 0 a(1) = 2.0*myid a(2) = 4.0+myid CALL MPI_REDUCE(a, res, 2, MPI_REAL, MPI_SUM, root, & MPI_COMM_WORLD, ierr) IF( myid.eq. 0 ) THEN WRITE(6,*) myid, ': res(1)=', res(1), 'res(2)=', res(2) END IF CALL MPI_FINALIZE(ierr) END 24