Title: COMP309/509 - Lecture 23

COMP309/509 - Parallel and Distributed Computing

Lecture 23 - Advanced MPI

By Mitchell Welch

###University of New England

Reading

A users guide to MPI (This respurce is a bit dataed, however the examples are still current)
Assignment 4
Project (COMP509)

Summary

Collective Communication
I/O On The Cluster
Derived Datatypes

Collective Communication

Last time we looked at three forms of collective communication.
Other than the usual sending, receiving, and barriers.
They were:
- MPI_Bcast used to distribute a value.
- MPI_Reduce used to accumulate (via an MPI_OP) a result at the root.
- The idea being that it reduces the results to s single value.
- MPI_Scatter used to distribute blocks of data to the nodes.
- MPI_Gather accumulates blocks of data in a buffer at the root.

Collective Communication

There are versions that accumulate the results at all nodes, not just the root. These are called:
- MPI_Allreduce
- MPI_Allgather

Collective Communication


int MPI_Allreduce(void *sndbuf, 
    void *rcvbuf, 
    int count, 
    MPI_Datatype datatyp, 
    MPI_Op op, 
    MPI_Comm comm 
);

This procedure combines the values on all processes to a single value, and then distributes that value to all processes.
This is a more efficient way of collecting results, since it uses tree shaped communication.
Note that it differs from MPI_Reduce in that has no root argument.

Collective Communication

sndbuf is the send buffer
rcvbuf address of receive buffer significant at all nodes
datatyp is data type of elements of send buffer
op is the reduce operation.
comm is the communicator

Collective Communication

int MPI_Allgather(void *sndbuf,  
    int sndcnt, 
    MPI_Datatype sndtyp, 
    void *rcvbuf,  
    int rcvcnt, 
    MPI_Datatype rcvtyp, 
    MPI_Comm comm
);

Gathers together values from a group of processes, then distributes the results to all processes.

Collective Communication

sndbuf is the address of send buffer.
sndcnt is the number of elements in send buffer.
sndtyp is the type of send buffer items.
rcvbuf is the address of receive buffer.
- significant at all processes
rcvcnt is the number of elements for any single receive
- significant at all processes
comm is the communicator of the processes participating in the gathering.

I/O On The Cluster

Not all MPI processes on the cluster can do IO.
At least not input, output is funnelled back to the console node.
To find out which can do IO you use MPI_Comm_get_attr.
You need to know the value of the MPI_IO attribute.
This will be either:
- The rank of a clone that can accept input
- MPI_PROC_NULL if no clone can do input.
- MPI_ANY_SOURCE if all clones can do input.

I/O On The Cluster

Review simple_io.c

I/O On The Cluster

From the previous example we distill out a subroutine.
This subroutine carries out an election to determine the clone that can do IO.
It is vital that this procedure returns the same value in all clones!
This is because the chosen clone will usually be root in subsequent collective communications.
I.e. the root must be the same in all calls to the collective communication primitive being called.


#include <stdio.h>
#include <unistd.h>
#include <limits.h>
#include "mpi.h"
#define VERBOSE 0
int get_io_clone(){
  int *iop, iorank, flag;
  MPI_Comm_get_attr(MPI_COMM_WORLD, MPI_IO, &iop, &flag);
  if(!flag){
    if(VERBOSE)fprintf(stderr, "MPI_Attr_get failed, exiting\n");
    return -1;
  }
  if(*iop == MPI_PROC_NULL){
    if(VERBOSE)fprintf(stderr, "No one can do IO!\n");
    return -1;
  }
  if(*iop == MPI_ANY_SOURCE){
    if(VERBOSE)fprintf(stderr, "Anyone can do IO!\n");
    return 0;
  }
  MPI_Allreduce(iop, &iorank, 1, MPI_INT, MPI_MIN, MPI_COMM_WORLD);
  return iorank;
}

I/O On The Cluster

Review io_test.c

Derived Datatypes

Sends are expensive!
A general rule of thumb: The fewer messages the better!
Recall versions 1 and 2 of the parallel trapezoidal rule from lecture 21.
To distribute three numbers:

float a; //start of interval
float b; //end of interval
int n;   //number of trapezoids

We had to do:
- three sends and three receives in version one.
three broadcasts in version two.
Today we'll see how to send structured data (i.e structs).
As well as use MPI's versions of PVM's pack and unpack.

Derived Datatypes

First declare our new type:

typedef struct {
  float a;
  float b;
  int   n;
} ParTrapData;

Create the appropriate data:

printf("Enter a, b, and n\n");  
scanf("%f %f %d", a_ptr, b_ptr, n_ptr);
ParTrapData  ptd = { *a_ptr, *b_ptr, *n_ptr };

Then send it:

MPI_Bcast(&ptd, 1, ParTrapData, 0, MPI_COMM_WORLD);

Derived Datatypes

The Problem?
ParTrapData is not the correct type!
It is not MPI_Datatype.
MPI is a library of precompiled functions.
Thus MPI only knows about pre-existing datatypes.

Derived Datatypes

Firstly it isn't trivial.
We only need to follow a simple recipe
We use:
- MPI_Type_create_struct
- MPI_Get_address and
- MPI_Type_commit

Derived Datatypes

#include "mpi.h"
int MPI_Type_create_struct(int count, 
                    int blocklens[], 
               MPI_Aint indices[], 
           MPI_Datatype old_types[], 
           MPI_Datatype *newtype )

Creates a struct datatype of type
MPI_Datatype placed in newtype
The input parameters describe the type of struct making up the datatype.
count is the number of items in the struct.
count is also the length of the next three arrays.
blocklens is the number of elements in each block (element of the struct).
indices is the byte displacement of each block.
old_types is the type of elements in each block.

Derived Datatypes

Make space for our newly created type:

MPI_Datatype mpi_ptdatatype;

Create the appropriate arguments:

int count = 3;
MPI_Aint blocks[count]  = {1, 1, 1};
MPI_Aint indices[count];
MPI_Datatype old_types[count] = {MPI_Float, MPI_Float, MPI_Int};

Computing the indices requires an actual ParTrapData object:

printf("Enter a, b, and n\n");  
scanf("%f %f %d", a_ptr, b_ptr, n_ptr);
ParTrapData  ptd = { *a_ptr, *b_ptr, *n_ptr };

Derived Datatypes

Find the addresses of things:

MPI_Aint addresses[count + 1];
MPI_Get_address(&ptd, &addresses[0]);
MPI_Get_address(&(ptd.a) , &addresses[1]);
MPI_Get_address(&(ptd.b) , &addresses[2]);
MPI_Get_address(&(ptd.n) , &addresses[3]);

Compute the displacements:

indices[0] = addresses[1] - addresses[0];
indices[1] = addresses[2] - addresses[0]; 
indices[2] = addresses[3] - addresses[0];

Make the Datatype:

MPI_Type_create_struct(count, blocks, indices, old_types, &mpi_ptdatatype);

Commit it, so it can be used: MPI_Type_commit(&mpi_ptdatatype);

Derived Datatypes

#include "mpi.h"
int MPI_Type_commit(MPI_Datatype *datatype);

Commits the datatype
*datatype is the datatype to be commited

Derived Datatypes

Review parallel_trap3.c

Derived Datatypes

There are three other derived datatype constructors
- MPI_Type_contiguous
- MPI_Type_vector
- MPI_Type_indexed
However we won't deal with them.

Derived Datatypes

Just like PVM one can pack data into a buffer.
Just like PVM one can unpack data out of a buffer.
To pack the data needed for the parallel trapezoidal rule:

#define BUFF 100
char buff[BUFF]
int position = 0;

MPI_Pack(a_ptr, 1, MPI_FLOAT, buff, BUFF, &position, MPI_COMM_WORLD);
MPI_Pack(b_ptr, 1, MPI_FLOAT, buff, BUFF, &position, MPI_COMM_WORLD);
MPI_Pack(n_ptr, 1, MPI_INT, buff, BUFF, &position, MPI_COMM_WORLD);

To broadcast the buffer:

MPI_Bcast(buffer, BUFF, MPI_PACKED, root, MPI_COMM_WORLD);

To unpack it:

MPI_Unpack(buff, BUFF, &position, a_ptr, 1, MPI_FLOAT,  MPI_COMM_WORLD);
MPI_Unpack(buff, BUFF, &position, b_ptr, 1, MPI_FLOAT,  MPI_COMM_WORLD);
MPI_Unpack(buff, BUFF, &position,n_ptr, 1, MPI_INT,  MPI_COMM_WORLD);

Derived Datatypes

Review:

parallel_trap4.c
parallel_trap5.c

Summary

Collective Communication
I/O On The Cluster
Derived Datatypes

Questions?

Reading

A users guide to MPI (This respurce is a bit dataed, however the examples are still current)
Assignment 4
Project (COMP509)