MPI_Wait: Request pending due to failure - c

Question

I'm having trouble trying to implement a non-blocking send and receive in my code below and am getting this error:
16 Reading <edge192x128.pgm>
17 Rank 2 [Sat Apr 28 11:24:58 2018] [c6-0c0s13n1] Fatal error in PMPI_Wait: Request pending due to failure, error stack:
18 PMPI_Wait(207): MPI_Wait(request=0x7ffffff95534, status=0x7fffffff74b0) failed
19 PMPI_Wait(158): Invalid MPI_Request
20 Rank 3 [Sat Apr 28 11:24:58 2018] [c6-0c0s13n1] Fatal error in PMPI_Wait: Request pending due to failure, error stack:
21 PMPI_Wait(207): MPI_Wait(request=0x7ffffff95534, status=0x7fffffff74b0) failed
22 PMPI_Wait(158): Invalid MPI_Request
23 _pmiu_daemon(SIGCHLD): [NID 01205] [c6-0c0s13n1] [Sat Apr 28 11:24:58 2018] PE RANK 2 exit signal Aborted
24 [NID 01205] 2018-04-28 11:24:58 Apid 30656034: initiated application termination
25 Application 30656034 exit codes: 134
26 Application 30656034 resources: utime ~0s, stime ~0s, Rss ~7452, inblocks ~7926, outblocks ~19640
My program attempts to perform the following (assuming 4 processes for this example):
Root process reads in an image file as a two-dimensional array PM x PN into masterbuf;
Root process uses MPI_Issend to transfer subsections of masterbuf (PM/2 x PN/2) to all 4 processes (which includes itself). I have used a strided datatype to split the original array into 4 sections.
All processes use MPI_Irecv to store PM/2 x PN/2 subsection in their own copy of buf.
MPI_Wait is called to prevent program continuing until distribution of data is complete (I understand I could have used MPI_Waitall here, which I intend to do after I've got this working).
I've been playing with the code for hours now and just can't fix this issue so any help would be appreciated. Code is below. I've removed some non-relevant blocks.
1 #include <stdio.h>
2 #include <stdlib.h>
3 #include <mpi.h>
4 #include <math.h>
5 #include "pgmio.h"
6
7 #define M 192
8 #define N 128
9
10 #define PX 2 // number of processes in X dimension
11 #define PY 2 // number of processes in Y dimension
12 #define MP M/PX
13 #define NP N/PY
14
15
16 #define FILEIN "edge192x128.pgm"
17 #define FILEOUT "ex7_0_192x128.pgm"
18
19 int main(int argc, char **argv)
20 {
21 double buf[MP][NP];
22 double old[MP + 2][NP + 2];
23 double new[MP + 2][NP + 2];
24 double edge[MP + 2][NP + 2];
25 double masterbuf[M][N];
26 double delta, delta_max, master_delta;
27
28 int rank, cart_rank, size, left, right, up, down, iter;
29 int dims[] = {2, 2};
30 int periods[] = {0, 0};
31 int reorder = 0;
32 int tag = 0;
33
34 MPI_Status status;
35 MPI_Comm comm = MPI_COMM_WORLD;
36 MPI_Comm cart_comm;
37
38 /* initialise MPI */
39 MPI_Init(&argc, &argv);
40 MPI_Comm_size(comm, &size);
41 MPI_Comm_rank(comm, &rank);
42 MPI_Request request[2 * size];
43 int coords[size][2];
44
45 /* initialise cartesian topology */
46 MPI_Cart_create(comm, 2, dims, periods, reorder, &cart_comm);
47 MPI_Comm_rank(cart_comm, &cart_rank);
48 MPI_Cart_shift(cart_comm, 1, 1, &left, &right);
49 MPI_Cart_shift(cart_comm, 0, 1, &up, &down);
50 printf("cart_rank: %d\n", cart_rank);
51
56
57 /* create block datatype for allocation of subsections of image to processes */
58 MPI_Datatype MPI_block;
59 MPI_Type_vector(M / PX, N / PY, N, MPI_DOUBLE, &MPI_block);
60 MPI_Type_commit(&MPI_block);
61
73
74 /* master process: read edges data file into masterbuff and distribute */
75 if (rank == 0)
76 {
77 printf("Reading <%s>\n", FILEIN);
78 pgmread(FILEIN, masterbuf, M, N);
79
80 printf("Distributing data to processes...\n");
81 for (int i = 0; i < size; i++)
82 {
83 /* send chunk to each process: i refers to cart_rank */
84 MPI_Cart_coords(cart_comm, i, 2, &coords[i][0]);
85 printf("coords = (%d, %d), rank = %d\n", coords[i][0], coords[i][1], \
86 cart_rank);
87 MPI_Issend(&masterbuf[coords[i][0] * MP][coords[i][1] * NP], MP * NP, \
88 MPI_block, i, tag, cart_comm, &request[i]);
89 }
90
91 MPI_Wait(&request[0], &status);
92 MPI_Wait(&request[1], &status);
93 MPI_Wait(&request[2], &status);
94 MPI_Wait(&request[3], &status);
95 }
96
97 /* all processes: receive data sent by master process */
98 MPI_Irecv(buf, MP * NP, MPI_block, cart_rank, tag, cart_comm, \
99 &request[cart_rank + size]);
100
101 /* Could change this to MPI_Waitall */
102 MPI_Wait(&request[5], &status);
103 MPI_Wait(&request[4], &status);
104 MPI_Wait(&request[7], &status);
105 MPI_Wait(&request[6], &status);
106
107 if (rank == 0)
108 {
109 printf("...complete.\n");
110 }

Answer 1

Your application deadlock when rank 0 send to itself and no receive was yet posted.
Also, there are 4 MPI_Wait() but a single MPI_Recv().
As a side note, you can MPI_Waitall() instead of calling several consecutive MPI_Wait().