COMP309/509 - Parallel and Distributed Computing

Lecture 4 - Interprocess Communication

By Mitchell Welch

University of New England

Reading

• Chapter 3, 8 and 15 from Advanced Programming in the UNIX environment
• Assignment 1 Description

Summary

• File I/O Revision
• Interprocess Communication with Buffers (Pipes)
• Building a Ring-of-Processes for Computation

File I/O Revision

• Recall from lecture 1, the read(...) and write(...) functions provide raw (unbuffered I/O to storage via file descriptors)

#include  <fcntl.h>

// read(...) returns the number of bytes read or -1 on error
// errno is also set upon on error
int  read(  int  handle,  void  *buffer,  size_t  nbyte );

// write returns the number of bytes written or -1 on error
//errno is also set upon error
int  write(  int  handle,  void  *buffer,  size_t  nbyte  );

File I/O Revision

• handle is a file descriptor open(...)ed in the appropriate mode
• nbyte is the number of bytes to read/write to/from the buffer (needs to be smaller than SSIZE_MAX, in POSIX >= 32767)

• buffer is a pointer to the memory location where the data will be copied to/from

• The read(...) function attempts to read nbytes from the file associated with handle, and places the characters read into buffer.

• The write() function attempts to write nbytes from buffer to the file associated with handle.

File I/O Revision

• The contents of errno should be displayed using the perror(...)

// The contents of the string pointed to by str are appended to the 
// beginning of the errno message
void perror ( const char * str );

#include <stdio.h>

int main ()
{
  FILE * pFile;
  pFile=fopen ("unexist.ent","rb");
  if (pFile==NULL)
    perror ("The following error occurred");
  else
    fclose (pFile);
  return 0;
}

File I/O Revision

• Reading one byte at a time

int i, buffsize, bytes;
char buff[buffsize];
...
for(i = 0; i < buffsize; i++){
    bytes = read(STDIN_FILENO, &buff[i], sizeof(char));
    ...
}

File I/O Revision

• Why is this a bad idea? Where should you not use this approach?

//Example 1
int integer, bytes;
...
bytes = read(STDIN_FILENO, &integer, sizeof(int));

Example 2
typedef struct token_header {
  ....
} token_hdr_t;  
token_hdr_t thp;
bytes =  read(fd,&thp,sizeof(token_hdr_t));

• Inter-machine communication - integer precision can differ between platforms!

File I/O Revision

• The number of bytes actually read in by the read(...) function is limmited by:
  • The end of the file
  • When reading from the terminal device, usually only one line is returned at a time
  • Network buffering (when reading from a socket)
  • The content in a pipe or FIFO (i.e. if there are fewer bytes in the pipe/FIFO than nbyte)
  • Signal interrupts

File I/O Revision

• Errors from the write(...) function include:
  • Exceeding the file size limit for a process or file system
  • Incorrect oflags set by the open(...) call ( O_RDONLY, O_WRONLY, ORDWR and O_APPEND)
  • System issues (e.g. writing to a file system that only supports read)

Interprocess Communication with Buffers

• Pipes are interprocess communication buffers (of course they are files like everything in UNIX).

#include <unistd.h>
int pipe(int filedes[2]);

• The pipe call creates a (uni-directional) communication buffer which the caller can access through the file descriptors fildes[0] and fildes[1]
• The data written (using write) to fildes[1] is read (using read) from fildes[0] on a first in first out basis (i.e. a fifo buffer).
• 0 for input - Recall that 0 is STDIN_FILENO which is standard input.
• 1 for output - Recall that 1 is STDOUT_FILENO which is standard output.

Interprocess Communication with Buffers

• The structure of a pipe:

.center[]

Interprocess Communication with Buffers

• Pipes are private. A pipe has no external or permanent name and a program can only access it through the two file descriptors.
• Pipes can only be used between processes that have a common ancestor.
• In a standard scenario, a pipe is created in a process that then calls fork():
  1. The pipe is copied to the child process (i.e. it exists as part of the parents memory image)
  2. The pipe can then be used between the parent and child for communication

Interprocess Communication with Buffers

• The pipe structure straight after a process calls fork(): .center[ ]

Interprocess Communication with Buffers

• In the example below, the parent closes the read-end of the pipe (fd[0]) and the child closes the write-end of the pipe (fd[1]) to create a pipe from the parent to the child:

.center[]

Interprocess Communication with Buffers

• In summary, the usual scenario is that a process that calls pipe then forks.
• What happens then depends on which direction of data flow we want:
  1. for a pipe from parent to child, the parent closes fd[0] and the child closes fd[1]
  2. for a pipe from the child to the parent, the child closes fd[0] and the parent closes fd[1].

Interprocess Communication with Buffers

• A simple example:

int main(void) {
    int     n;
    int     fd[2];
    pid_t   pid;
    char    line[MAXLINE];

    if (pipe(fd) < 0)
        perror("pipe error");
    if ((pid = fork()) < 0) {
        perror("fork error");
    } else if (pid > 0) {       /* parent */
        close(fd[0]);
        write(fd[1], "hello world\n", 12);
    } else {                    /* child */
        close(fd[1]);
        n = read(fd[0], line, MAXLINE);
        write(STDOUT_FILENO, line, n);
    }
    exit(0);
}

Interprocess Communication with Buffers

• The dup(...) and dup2(...) allow for the manipulation of file descriptors.

#include <unistd.h>

int dup(int filedes)
int dup2(int filedes, int filedes2);

• dup2(...) closes the entry filedes2 of the file descriptor table, and then copies the pointer of entry filedes into filedes2.
• In other words it changes the pointer in filedes2 to the pointer in filedes.
• dup2(...) returns the value of the second parameter (fildes2) upon success. A negative return value means that an error occured. errno is also set on error.

Interprocess Communication with Buffers

• The dup(...) function duplicates the file descriptor, returning the lowest file descriptor available in the table. (i.e. the file descriptor returned by dup(...) points to the same entry in the system file table

A Ring of Processes

• Using pipes (and other IPC approaches) it is possible to link processes together in different ways or topologies
• In this context, the topology refers to the formation in which the processes are connected.
• There are is any number of difference configurations possible:
  • 2D or 3D Grid - Each node is connected to multiple neighbours
  • Star - Each process is connected only to a central process
  • ring - Each process connected to at most 2 neighbours

Building a Ring-of-Processes for Computation

• Pipes can be used to connect processes in different ways, using different topologies, to produce multi-process programs that efficiently solve large problems.
• The Ring topology is an example of one such arrangement that provides an efficient communication structure for processes to work together.
• The Ring of processes consists of n processes (0,1,2…n)
  • such that for each process i in (0,1,2…n), STDOUT_FILE of i is piped to STDIN_FILE of the process ( i + 1 ) mod n

Building a Ring-of-Processes for Computation

• The ring topology has the following basic properties:
  • The interconnection cost scale linearly as processes are added.
  • Communication latency also scales linearly
  • Communication Bandwidth is independent of the number of processes

Building a Ring-of-Processes for Computation

• The result in pictures:

.center[]

Building a Ring-of-Processes for Computation

• Making a ring of 1 processes is simple.
• Lets give it a name and formally define it (without error code to make it more legible):

void make_trivial_ring(){
  int  fd[2];     
  pipe (fd);
  dup2(fd[0], STDIN_FILENO);
  dup2(fd[1], STDOUT_FILENO);
  close(fd[0]);
  close(fd[1]); 
}

Building a Ring-of-Processes for Computation

• Our trivial ring:

.center[]

Building a Ring-of-Processes for Computation

• So we know how to make a trivial ring.
• Suppose we have an n ring already constructed (n > 0).
• To add a new node to the ring all we have to do is get one of the nodes in the ring to execute the following function (again leaving out error code):

Building a Ring-of-Processes for Computation

void add_new_node(int *pid){
  int   fd[2];
  pipe (fd);
  *pid = fork();
  if (*pid > 0)  dup2(fd[1], STDOUT_FILENO);
  else           dup2(fd[0], STDIN_FILENO);
  close(fd[0]); 
  close(fd[1]); 
}

• The PID of the new node is placed in the pointer to int passed in.

Building a Ring-of-Processes for Computation

• The ring structure prior to the inductive step:

.center[]

Building a Ring-of-Processes for Computation

• The ring structure after the inductive step:

.center[]

Building a Ring-of-Processes for Computation

• The ring structure after the child calls dup2(fd[0],STDIN_FILENO);

.center[]

Building a Ring-of-Processes for Computation

• After both close fd[0] and fd[1]

.center[]

Building a Ring-of-Processes for Computation

• After a quick re-arrangement

.center[]

Building a Ring-of-Processes for Computation

• Now we can easily write code to illustrate how a ring is made:

int main(int argc,  char *argv[ ]){
  int  i, childpid;

  make_trivial_ring();
  for (i = 1; i < atoi(argv[1]); i++){
    add_new_node(&childpid);
    if (childpid) break;   }
   fprintf(stderr,
       "I'm %d ID = %d parent ID = %d\n",
           i, (int)getpid(), (int)getppid());
   sleep(5);
   exit(0); 
}

Note the way the ring grows, with the parent breaking out of the loop. We could just as easily do it the other way, but this way is more convenient.

Ring-of-Processes full Listing

/* written by ian a. mason @ une march '98 */
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <string.h>
#include <errno.h>
int parse_args(int argc,  char *argv[ ], int *np){
  if ( (argc != 2) || ((*np = atoi (argv[1])) <= 0) ) {
    fprintf (stderr, "Usage: %s nprocs\n", argv[0]);
    return(-1); };
  return(0); 
}
...

Ring-of-Processes full Listing

int make_trivial_ring(){   
  int   fd[2];
  if (pipe (fd) == -1) 
    return(-1); 
  if ((dup2(fd[0], STDIN_FILENO) == -1) ||
      (dup2(fd[1], STDOUT_FILENO) == -1)) 
    return(-2); 
  if ((close(fd[0]) == -1) || (close(fd[1]) == -1))   
    return(-3); 
  return(0); }

Ring-of-Processes full Listing

int add_new_node(int *pid){
  int fd[2];
  if (pipe(fd) == -1) 
    return(-1); 
  if ((*pid = fork()) == -1)
    return(-2); 
  if(*pid > 0 && dup2(fd[1], STDOUT_FILENO) < 0)
    return(-3); 
  if (*pid == 0 && dup2(fd[0], STDIN_FILENO) < 0)
    return(-4); 
  if ((close(fd[0]) == -1) || (close(fd[1]) == -1)) 
    return(-5);
  return(0);
}

Ring-of-Processes full Listing

int main(int argc,  char *argv[ ]){
   int   i;             /* number of this process (starting with 1)   */
   int   childpid;      /* indicates process should spawn another     */ 
   int   nprocs;        /* total number of processes in ring          */ 
   if(parse_args(argc,argv,&nprocs) < 0) exit(EXIT_FAILURE);
   if(make_trivial_ring() < 0){
     perror("Could not make trivial ring");
     exit(EXIT_FAILURE); };
   for (i = 1; i < nprocs;  i++) {
     if(add_new_node(&childpid) < 0){
       perror("Could not add new node to ring");
       exit(EXIT_FAILURE); };
     if (childpid) break; };
   /* ring process code  */

   fprintf(stderr, "node %d of %d\n", i, nprocs);
   exit(EXIT_SUCCESS);
}     /* end of main program here */

Summary

• File I/O Revision
• Interprocess Communication with Buffers (Pipes)
• Building a Ring-of-Processes for Computation

Reading

• Chapter 3, 8 and 15 from Advanced Programming in the UNIX environment
• Assignment 1 Description

Next Lecture

• Using the ring-o-processes for computation
• Debugging with DDD
• A Look at other forms of IPC - File Sharing, Memory Mapping etc.