Alternative to MPI_Sendrecv_replace - c

Question

I am trying to get an alternative code to
if(rank %2==0 && rightNeighbour != MPI_PROC_NULL)
MPI_Sendrecv_replace(&bufferRight[0], len, MPI_DOUBLE, rightNeighbour, 1,
rightNeighbour, 1, MPI_COMM_WORLD, &status);
else if(rank%2 ==1 && leftNeighbour != MPI_PROC_NULL)
MPI_Sendrecv_replace(&bufferLeft[0], len, MPI_DOUBLE, leftNeighbour, 1,
leftNeighbour, 1, MPI_COMM_WORLD, &status);
if (rank % 2 == 1 && rightNeighbour != MPI_PROC_NULL)
MPI_Sendrecv_replace(&bufferRight[0], len, MPI_DOUBLE, rightNeighbour, 1,
rightNeighbour, 1, MPI_COMM_WORLD, &status);
else if (rank % 2 == 0 && leftNeighbour != MPI_PROC_NULL)
MPI_Sendrecv_replace(&bufferLeft[0], len, MPI_DOUBLE, leftNeighbour, 1,
leftNeighbour, 1, MPI_COMM_WORLD, &status);
using MPI_Send and MPI_Recv but it seems it's deadlocking. Any easy way of doing the same with MPI_Send and MPI_Recv ?
I have tried using
if(rank %2==0 && rightNeighbour != MPI_PROC_NULL){
MPI_Recv(&bufferRight[0], len, MPI_DOUBLE, rightNeighbour, 1,
MPI_COMM_WORLD, &status);
MPI_Send(&bufferRight[0], len, MPI_DOUBLE, rightNeighbour, 1,
MPI_COMM_WORLD);
}

Answer 1

This code:
if(rank % 2 == 0 && rightNeighbour != MPI_PROC_NULL)
MPI_Sendrecv_replace(&bufferRight[0], len, MPI_DOUBLE, rightNeighbour, 1,
rightNeighbour, 1, MPI_COMM_WORLD, &status);
else if(rank % 2 == 1 && leftNeighbour != MPI_PROC_NULL)
MPI_Sendrecv_replace(&bufferLeft[0], len, MPI_DOUBLE, leftNeighbour, 1,
leftNeighbour, 1, MPI_COMM_WORLD, &status);
is equivalent to:
if(rank % 2 == 0)
{
MPI_Send(&bufferRight[0], len, MPI_DOUBLE, rightNeighbour, 1,
MPI_COMM_WORLD);
MPI_Recv(&bufferRight[0], len, MPI_DOUBLE, rightNeighbour, 1,
MPI_COMM_WORLD, &status);
}
else if(rank % 2 == 1)
{
double *temp = malloc(len * sizeof(double));
MPI_Recv(temp, len, MPI_DOUBLE, leftNeighbour, 1,
MPI_COMM_WORLD, &status);
MPI_Send(&bufferLeft[0], len, MPI_DOUBLE, leftNeighbour, 1,
MPI_COMM_WORLD, &status);
memcpy(&bufferLeft[0], temp, len * sizeof(double));
free(temp);
}
Note that order of the send and receive calls is reversed in the odd ranks. Also, the receive uses a temporary buffer in order to implement the semantics of MPI_Sendrecv_replace, which guarantees that the data in the buffer is first sent and only then overwritten with the received one.
Note that the check whether a rank is not MPI_PROC_NULL is pointless since a send to/receive from MPI_PROC_NULL is essentially a no-op and will always succeed. One of the key ideas of the semantics of MPI_PROC_NULL is to facilitate the writing of symmetric code that doesn't contain such if's.

Answer 2

You better use MPI_Irecv and MPI_Isend rather than the blocking calls (MPI_Recv and MPI_Send). Then, after issuing the communication routines, simply wait for the requests using the MPI_Waitall (or two calls to MPI_Wait). However, to do this you cannot use the same buffer (i.e. the replace) -- you need to have two separate buffers -- because otherwise they would get corrupted as the buffers might get the content replaced before the actual send.
Let A be the incoming buffer and B the outgoing buffesr, your code should look something like
if(rank %2==0 && rightNeighbour != MPI_PROC_NULL){
MPI_Request req[2];
MPI_Status status[2];
MPI_Irecv (&A, len, MPI_DOUBLE, rightNeighbour, 1,
MPI_COMM_WORLD, &req[0]);
MPI_Isend (&B, len, MPI_DOUBLE, rightNeighbour, 1,
MPI_COMM_WORLD, &req[1]);
/* A */
MPI_Waitall (2, req, status);
}
Note that in /* A */ you can take advantage to do some computation while the communication is flying. Also, the error checking is omitted in the code -- you better check all return codes for the MPI calls.

Answer 3

Although probably the best way to proceed is to follow Harald's suggestion on using MPI_Isend and MPI_Irecv, there's one alternative which might not work depending on the case.
For small buffer sizes, some MPI implementations follow what is known as eager mode. In this mode, the message is sent regardless the receiver is already waiting for the message or not. The send buffer data is copied to a temporary buffer and, since the user's send buffer is available after the copy is done, MPI_Send returns before the communication has actually been completed.
This is a bit risky, since for large messages MPI usually works in rendezvous mode, which actually synchronizes both ends, so the following code would produce a deadlock as well:
if(rank %2==0 && rightNeighbour != MPI_PROC_NULL){
MPI_Send(&bufferRight[0], len, MPI_DOUBLE, rightNeighbour, 1,
MPI_COMM_WORLD);
MPI_Recv(&bufferRight[0], len, MPI_DOUBLE, rightNeighbour, 1,
MPI_COMM_WORLD, &status);
}
In MPI_Recv, however, the reception buffer is (obviously) not available until the data has already been received.
Additional info on MPI performance tips.