Title: COMP309/509 - Lecture 20 class: middle, center, inverse

COMP309/509 - Parallel and Distributed Computing

Lecture 20 - Massage Passing Interface

By Mitchell Welch

University of New England

Reading

• MPI: A Message-Passing Interface Standard Chapters 1 and 2
• man mpirun
• Examples
• Assignment 3
• Project (COMP509)

Summary

• Introduction
• Helloworld in MPI
• The MPI Paradigm
• MPI Elementary Functions

Introduction

• We will spend the next four lectures on MPI.
• Just enough to:
  • Compare it with PVM
  • Multiply some matrices on a Torus!
• MPI = Message Passing Interface.
• It is a specification, not an implementation.
• MPI programs can be SPMD style-wise: Single Program Multiple Data or MPMD style-wise: Multiple Program Multiple Data

Introduction

• Open MPI is the implementation we will use.
• Open MPI:
  • Is an open source implementation that runs on COWS (not the ones that moo).
  • COWS = Cluster Of Work Stations

• Open MPI reference: http://www.open-mpi.org

• See the online manual page: man mpirun

• The MPI Standard: http://www.mpi-forum.org

Helloworld in MPI

#include <stdio.h>
#include "mpi.h"
int main(int argc, char** argv) {
  MPI_Init(&argc, &argv);
  printf("\tHello World!\n");
  MPI_Finalize();
  return 0;
} 

Helloworld in MPI

• Make sure you have this in your .bashrc file:

# Source global definitions
if [ -f /etc/bashrc ]; then
        . /etc/bashrc
fi

• Compile using mpicc:

Examples> mpicc -Wall helloworld.c -o helloworld
Examples>
Run using mpirun:
Examples> mpirun -np 1 helloworld
        Hello World!
Examples>  mpirun -np 2 helloworld
        Hello World!
        Hello World!
Examples>

Helloworld in MPI

• In the second example, 2 clones of the same program were executed. An example of SPMD programming.
• mpiexec has the same functionality as mpirun:

Examples> mpiexec -np 2 helloworld
        Hello World!
        Hello World!
Examples>

Helloworld in MPI

all: helloworld whoAmI whereAmI greetings ring

helloworld:  helloworld.c
        mpicc -Wall helloworld.c -o helloworld

whereAmI:  whereAmI.c
        mpicc -Wall whereAmI.c -o whereAmI

whoAmI:  whoAmI.c
        mpicc -Wall whoAmI.c -o whoAmI

greetings:  greetings.c
        mpicc -Wall greetings.c -o greetings

ring:  ring.c
        mpicc -Wall ring.c -o ring

clean: 
        rm -f helloworld whereAmI whoAmI greetings ring

The MPI Paradigm

• Note that the number of processes is a command line argument!
• Thus we never explicitly write forks, spawns, creates or whatever.
• We will cover MPI in two stages:
  • Elementary MPI
  • Advanced MPI
• Elementary MPI is small - just six (6) functions. Advanced MPI is large: 372 functions for MPI-3

The MPI Paradigm

• One can master elementary MPI using just 6 functions:
  • MPI_Init – initializing
  • MPI_Comm_Size – number of peers
  • MPI_Comm_Rank – rank amongst peers
  • MPI_Send – sending
  • MPI_Recv – receiving
  • MPI_Finalize – exiting

MPI Elementary Functions

#include "mpi.h"
int MPI_Init(int *argc, char ***argv)

• Initializes the MPI execution environment
• On exit from this routine, all processes will have a copy of the argument list.
• Compared to main there is an extra * in both arguments!
• Thus a typical call looks like:

MPI_Init(&argc, &argv);

MPI Elementary Functions

• MPI specifies no command-line arguments but does allow an MPI implementation to make use of them.
• This is not required by the MPI standard.
• Portable programs should not rely on it.
• This is provided as a service by this implementation.
• An MPI implementation is allowed to distribute the command line arguments but is not required to.
• It is actually mpirun that distributes argc and argv.
• All MPI routines return an error value, or MPI_SUCCESS MPI_SUCCESS indicates success.
• I’ll omit error checking from my code, you shouldn’t!

MPI Elementary Functions

#include "mpi.h"
int MPI_Finalize()

• Terminates MPI execution environment
• All processes must call this routine before exiting.
• The MPI standard does not say what a program can do before an MPI_INIT or after an MPI_FINALIZE.
• You should do as little as possible:
  • It is best not to perform much more than a return after calling MPI_Finalize.
• In particular, avoid anything that changes the external state of the program, such as opening files, reading standard input or writing to standard output.
• The number of processes running after this routine is called is undefined.

MPI Elementary Functions

• Self-Discovery in MPI is ust like PVM, two of the first questions asked in an MPI program are:
  • How many processes are there? size
  • Which one of them am I? rank
• How many?
  • Is answered using: MPI_Comm_size.
• Which one?
  • Is answered using: MPI_Comm_rank.
• As in PVM groups:
  • The rank ranges from 0 upto size - 1.

MPI Elementary Functions

• whoami.c

#include <stdio.h>
#include "mpi.h"
int main(int argc, char** argv) {
  int me, nproc; 
  MPI_Init(&argc, &argv);
  MPI_Comm_rank(MPI_COMM_WORLD, &me);
  MPI_Comm_size(MPI_COMM_WORLD, &nproc);
  printf("I'm clone %d out of %d total!\n", me, nproc);
  MPI_Finalize();
  return 0;
}

MPI Elementary Functions

• Process numbers bigger than 20 cause noticeable load spikes on turing!
• Process numbers bigger than 50 might make you very unpopular!

Examples> mpirun -np 1 whoAmI
I'm clone 0 out of 1 total!
Examples> mpirun -np 10 whoAmI
I'm clone 0 out of 10 total!
I'm clone 8 out of 10 total!
I'm clone 5 out of 10 total!
I'm clone 6 out of 10 total!
I'm clone 9 out of 10 total!
I'm clone 4 out of 10 total!
I'm clone 2 out of 10 total!
I'm clone 1 out of 10 total!
I'm clone 7 out of 10 total!
I'm clone 3 out of 10 total!
Examples>

MPI Elementary Functions

• whereAmI.c

#include <stdio.h>
#include <unistd.h>
#include <limits.h>
#include "mpi.h"
int main(int argc, char** argv) {
  int me, nproc;                
  char hostname[PATH_MAX];  
  MPI_Init(&argc, &argv);
  MPI_Comm_rank(MPI_COMM_WORLD, &me);
  MPI_Comm_size(MPI_COMM_WORLD, &nproc);
  if(gethostname(hostname, PATH_MAX) == 0){
    printf("I'm clone %d out of %d total located @ %s\n", 
           me, nproc, hostname);
    } else {
      printf("I'm a lost clone %d out of %d total!\n", 
             me, nproc);
    }
  MPI_Finalize();
  return 0;
}

MPI Elementary Functions

• You can use a hostfile of node names for MPI to use when executing clones to distribute the processess.

$ cat setup-b1to8
b1
b2
b3
b4
b5
b6
b7
b8
$

MPI Elementary Functions

• Then ask MPI to distribute the clones among the nodes using the bynode option:

$ mpirun -np 5 --map-by node --hostfile setup-b1to8 whereAmI
I'm clone 0 out of 5 total located @ b1
I'm clone 2 out of 5 total located @ b3
I'm clone 4 out of 5 total located @ b5
I'm clone 1 out of 5 total located @ b2
I'm clone 3 out of 5 total located @ b4
$

MPI Elementary Functions

#include "mpi.h"
int MPI_Comm_rank ( MPI_Comm comm, int *rank )

• Determines the rank of the calling process in the communicator comm, placed in the location rank.
• A communicator is essentially a collection of processes that can send each other messages.
• The rank is a number between 0 and one less than the size size - 1.
• The rank of a process (in the communicator) is also used as its address, when sending or receiving messages.

MPI Elementary Functions

• We’ll mostly use MPI_COMM_WORLD as the first argument, for a while.
• It consists of all the processes running when program execution begins (i.e. the <num> used in the mpirun -np <num> <prog>). Typical call:

MPI_Comm_rank(MPI_COMM_WORLD,&me);

MPI Elementary Functions

#include "mpi.h"
int MPI_Comm_size ( MPI_Comm comm, int *size )

• Puts the number of processes in the group of comm into size
• We’ll mostly use MPI_COMM_WORLD as the first argument, for a while.
• Typical call:

MPI_Comm_size(MPI_COMM_WORLD,&nproc);

MPI Elementary Functions

• greetings.c

#include <stdio.h>
#include <string.h>
#include "mpi.h"
int main(int argc, char** argv) {
    int my_rank, nproc,  src, dst, tag = 50;
    char msg[100];
    MPI_Status status;
    MPI_Init(&argc, &argv);
    MPI_Comm_rank(MPI_COMM_WORLD, &my_rank);
    MPI_Comm_size(MPI_COMM_WORLD, &nproc);
    if (my_rank != 0) {
      sprintf(msg, "Greetings from process %d!",
              my_rank);
      dst = 0;
      MPI_Send(msg,strlen(msg)+1,MPI_CHAR,dst,tag,MPI_COMM_WORLD);
    } else {
      for (src = 1; src < nproc; src++) {
        MPI_Recv(msg,100,MPI_CHAR,src,tag,MPI_COMM_WORLD,&status);
        printf("%s\n", msg);
      }
    }
    MPI_Finalize();
    return 0;
}

MPI Elementary Functions

• For MPMD Programming in Open MPI, clones of different programs can be run on specific nodes.
• In this example, 2 clones of whoAmI and 3 clones of whereAmI are run on different nodes.

$ mpirun -np 2 --host b2 whoAmI : -np 3 --host b3 whereAmI
I'm clone 1 out of 5 total!
I'm clone 0 out of 5 total!
I'm clone 2 out of 5 total located @ b3
I'm clone 4 out of 5 total located @ b3
I'm clone 3 out of 5 total located @ b3

MPI Elementary Functions

• To send messages in MPI one needs to:
  • Construct the message.
  • Specify:
    • the recipient.
    • the type of data being sent.
    • the amount of data being sent.
    • an identification tag
• Similar to PVM groups, a recipient is specified by two things:
  • The rank of the recipient in a Communicator.
  • The Communicator.
• In elementary MPI we’ll stick with one Communicator: MPI_COMM_WORLD

MPI Elementary Functions

#include "mpi.h"
int MPI_Send(void *buf, 
              int count, 
     MPI_Datatype datatype, 
              int dest, 
              int tag,
         MPI_Comm comm )

• Performs a basic message send.
• This routine may block until the message is received.
• buf is the address of send buffer.
• count is the number of elements in send buffer.
• datatype is the datatype of each send buffer element
• dest is the rank of destination
• tag is the message tag.
• comm is the communicator.

MPI Elementary Functions

• MPI_Send may block until the message is received. Whether or not MPI_Sendblocks depends on factors such as:
  • How large the message is
  • How many messages are pending to the specific destination
  • Whether LAMD or C2C communication is being used.
  • Read man mpirun for more details on this.

MPI Elementary Functions

#include "mpi.h"
int MPI_Recv(void *buf, 
              int count, 
     MPI_Datatype datatype, 
              int source, 
              int tag, 
         MPI_Comm comm, 
       MPI_Status *status )

• Performs a basic message receive
• The count argument indicates the maximum length of a message.
• The actual number can be determined with MPI_Get_count.

MPI Elementary Functions

• buf is the address of receive buffer.
• status is where the status object will be placed.
• datatype is the datatype of each buffer element.
• source is the rank of the source.
• tag is the message tag.
• comm is the communicator.

MPI Elementary Functions

• Recall that in sending and receiving we need to specify the data types being sent.
• For this purpose we have the following built in datatypes.

• There are also unsigned versions.
• There are also two odd ones called MPI_BYTE and MPI_PACKED.

MPI Elementary Functions

#include "mpi.h"
int MPI_Get_count(MPI_Status *status, 
                  MPI_Datatype datatype, 
                  int *count)

• Gets the number of elements
• The number of received elements is placed in count.
• If the size of the datatype is zero, this routine will return a count of zero.
• If the amount of data in status is not an exact multiple of the size of datatype (so that count would not be an integer), a count of MPI_UNDEFINED is returned instead.

MPI Elementary Functions

*The Token Ring Revisited

#include <stdio.h>
#include "mpi.h"
#define TOKTAG 5
#define VERBOSE 1
int main(int argc, char** argv) {
  int me, nproc, src, dst, token;
  MPI_Status status;  
  MPI_Init(&argc, &argv);
  MPI_Comm_rank(MPI_COMM_WORLD, &me);
  MPI_Comm_size(MPI_COMM_WORLD, &nproc);
  src = (me == nproc - 1) ? 0 : me + 1;
  dst = (me == 0) ? nproc - 1 : me - 1;

  ....

MPI Elementary Functions

  ....

  if (me == 0) {
    token = 0;
    fprintf(stderr, "Token = %i  sent\n", token);
    MPI_Send(&token, 1, MPI_INT, dst, TOKTAG, MPI_COMM_WORLD);
    MPI_Recv(&token, 1, MPI_INT, src, TOKTAG, MPI_COMM_WORLD, &status);
    fprintf(stderr, "Token = %i  arrived back\n", token);
  } else {
    MPI_Recv(&token, 1, MPI_INT, src, TOKTAG, MPI_COMM_WORLD, &status);
    if(VERBOSE)fprintf(stderr, "Node %d passing token = %i\n", me, token);
    token++;
    MPI_Send(&token, 1, MPI_INT, dst, TOKTAG, MPI_COMM_WORLD);
  }
  MPI_Finalize();
  return 0;
}

class: middle, center, inverse

Questions?

Summary

• Introduction
• Helloworld in MPI
• The MPI Paradigm
• MPI Elementary Functions

Reading

• MPI: A Message-Passing Interface Standard Chapters 1 and 2
• man mpirun
• Examples
• Assignment 3
• Project (COMP509)