Mpirun hangs when mpi send and recieve is put in a loop - c

Question

I was trying to run the given program on a 4 node cluster using mpirun.
Node0 is distributing the data to node 1, 2 and 3.
In the program, computation has to be done for different values of variable 'dir',
ranging from -90 to 90.
So Node0 is distributing the data and collecting the result in a looped fashion(for different values of var 'dir').
When the do {*******}while(dir<=90); loop is given, mpirun hangs, and gets no output.
But when I comment the do {*******}while(dir<=90); loop output is obtained for initialized value of the variable dir,(dir=-90), and that output is correct. The problem occurs when given in loop.
Could anyone please help me solve this issue.
#include "mpi.h"
int main(int argc,char *argv[])
float dir=-90;
int rank,numprocs;
MPI_Status status;
MPI_Init(&argc,&argv);
MPI_Comm_rank(MPI_COMM_WORLD,&rank);
MPI_Comm_size(MPI_COMM_WORLD,&numprocs);
if(rank==0)
{
do{
/*initializing data*/
for(dest=1;dest<numprocs;dest++)
{
MPI_Send(&offset,1,MPI_INT,dest,FROM_MASTER,MPI_COMM_WORLD);
MPI_Send(&s_psi[offset],count,MPI_FLOAT,dest,FROM_MASTER,MPI_COMM_WORLD);
}
gettimeofday(&start,NULL);
for (dest=1; dest<numprocs; dest++)
{
MPI_Recv(&offset,1,MPI_INT,dest,FROM_WORKER,MPI_COMM_WORLD,&status);
MPI_Recv(&P[offset],count,MPI_FLOAT,dest,FROM_WORKER,MPI_COMM_WORLD,&status);
}
gettimeofday(&end,NULL);
timersub(&end,&start,&total);
printf("time consumed=%ds %dus\n",total.tv_sec,total.tv_usec);
dir++;
}while(dir<=90);
}
if(rank>0)
{
MPI_Recv(&offset,1,MPI_INT,0,FROM_MASTER,MPI_COMM_WORLD,&status);
MPI_Recv(&s_psi[offset],count,MPI_FLOAT,0,FROM_MASTER,MPI_COMM_WORLD,&status);
//Does the computation
}
MPI_Send(&offset,1,MPI_INT,0,FROM_WORKER,MPI_COMM_WORLD);
MPI_Send(&P[offset],count,MPI_FLOAT,0,FROM_WORKER,MPI_COMM_WORLD);
}
MPI_Finalize();
return 0;
}

Answer 1

The part where rank > 0 should be enclosed in a loop.
Each MPI_Send should have its corresponding MPI_Recv.
if(rank>0) {
do {
MPI_Recv(&offset,1,MPI_INT,0,FROM_MASTER,MPI_COMM_WORLD,&status);
MPI_Recv(&s_psi[offset],count,MPI_FLOAT,0,FROM_MASTER,MPI_COMM_WORLD,&status);
// Computation
MPI_Send(&offset,1,MPI_INT,0,FROM_WORKER,MPI_COMM_WORLD);
MPI_Send(&P[offset],count,MPI_FLOAT,0,FROM_WORKER,MPI_COMM_WORLD);
dir++;
} while(dir <= 90);
}
But you probably don't know dir in you worker nodes. Usually, we node0 send a magic packet to end the workers.
at the end of node0 :
for(r = 1; r < numprocs; r++)
MPI_Send(&dummy, 1, MPI_INT, r, STOP, COMM);
for the woker nodes :
if(rank>0) {
while(true) {
MPI_Recv(&offset,1,MPI_INT,0,FROM_MASTER,MPI_COMM_WORLD,&status);
MPI_Recv(&s_psi[offset],count,MPI_FLOAT,0,FROM_MASTER,MPI_COMM_WORLD,&status);
// Computation
MPI_Send(&offset,1,MPI_INT,0,FROM_WORKER,MPI_COMM_WORLD);
MPI_Send(&P[offset],count,MPI_FLOAT,0,FROM_WORKER,MPI_COMM_WORLD);
if(MPI_Iprobe(ANY_SOURCE, STOP, COMM, &flag, &status)) {
MPI_Recv(&dummy, 1, MPI_INT, ANY_SOURCE, STOP, COMM, NO_STATUS);
break;
}
};
}
And you can finally MPI_finalize
By the way, you might want to look at blocking and not bloking Send/Recv.