MPI: Efficient exchange of different amounts of data between all nodes - c

Question

I'm totally new to MPI and have absolutely no idea as to what is efficient.
I have x (x = 2, 4, 8, 16, ...) nodes, each of them with large text files of more than 4 million lines.
I want to sort these lines using bucket sort, and for that, I want to send the correct lines to each of the nodes (= buckets), meaning I have to send (and receive) from every node to every other node.
Currently my idea is like this:
With my root, I calculate bucket limits that every node has to know so they can send the right lines to the right nodes. I broadcast these limits using MPI_Bcast.
After that, since the count of every set of lines to be send is different, I send the count of lines to be received from every node to every other node using MPI_Isend, MPI_Irecv and MPI_Waitall.
When every node knows how much data it's going to receive from each node, I do the same for the data itself.
The sends and receives are done alternating for each node (process[i] wants to send to process[i+1] first and then receive from process[i-1]).
My code for the send and receive part looks like this:
// fill the buffer for the current rank (because data doesn't have to be sent)
buffers[rank] = buckets[rank];
int i, amount, sendIndex, receiveIndex;
// create MPI_Requests
int reqsIndex = 0;
MPI_Request dummyRequest;
MPI_Request reqs[numtasks - 1];
for (i = 0; i < numtasks - 1; i++)
reqs[i] = MPI_REQUEST_NULL;
// amounts[i]: Which process receives how much data from the current rank
int* amounts = malloc(sizeof(int) * (numtasks-1));
sendIndex = (rank + 1) % numtasks; // always contains the index of the next process to send data to (starting with [rank + 1])
receiveIndex = (rank + numtasks - 1) % numtasks; // always contains the index of the next process to receive data from (starting with [rank - 1])
int isSending = 1; // determines if the current process sends or receives data (always sends first)
// alternating send and receive for each process
for (i = 0; i < (numtasks - 1) * 2; i++)
{
if (isSending) {
MPI_Isend(&buckets[sendIndex]->n, 1, MPI_INT, sendIndex, TAG_NUMBER_OF_SENDED_LINES, MPI_COMM_WORLD, &dummyRequest);
sendIndex = (sendIndex + 1) % numtasks;
if (sendIndex == rank)
sendIndex = (sendIndex + 1) % numtasks;
}
else {
MPI_Irecv(&amounts[receiveIndex], 1, MPI_INT, receiveIndex, TAG_NUMBER_OF_SENDED_LINES, MPI_COMM_WORLD, &reqs[reqsIndex]);
receiveIndex = (receiveIndex + 1) % numtasks;
if (receiveIndex == rank)
receiveIndex = (receiveIndex + 1) % numtasks;
reqsIndex++;
}
isSending = (isSending + 1) % 2; // switch isSending from 1 to 0 or 0 to 1
}
MPI_Waitall(numtasks - 1, reqs, MPI_STATUSES_IGNORE); // waits for all receives to be finished
// requests reset
reqsIndex = 0;
for (i = 0; i < numtasks - 1; i++)
reqs[i] = MPI_REQUEST_NULL;
sendIndex = (rank + 1) % numtasks; // always contains the index of the next process to send data to (starting with [rank + 1])
receiveIndex = (rank + numtasks - 1) % numtasks; // always contains the index of the next process to receive data from (starting with [rank - 1])
isSending = 1; // determines if the current process sends or receives data (always sends first)
for (i = 0; i < (numtasks - 1) * 2; i++)
{
if (isSending) {
MPI_Isend(buckets[sendIndex]->data, buckets[sendIndex]->n * LINE_LENGTH, MPI_BYTE, sendIndex, TAG_LINES, MPI_COMM_WORLD, &dummyRequest);
sendIndex = (sendIndex + 1) % numtasks;
if (sendIndex == rank)
sendIndex = (sendIndex + 1) % numtasks;
}
else {
lineBuffer* lines = allocLines(amounts[receiveIndex]);
MPI_Irecv(lines->data, amounts[receiveIndex] * LINE_LENGTH, MPI_BYTE, receiveIndex, TAG_LINES, MPI_COMM_WORLD, &reqs[reqsIndex]);
buffers[receiveIndex] = lines;
receiveIndex = (receiveIndex + 1) % numtasks;
if (receiveIndex == rank)
receiveIndex = (receiveIndex + 1) % numtasks;
reqsIndex++;
}
isSending = (isSending + 1) % 2; // switch isSending from 1 to 0 or 0 to 1
}
MPI_Waitall(numtasks - 1, reqs, MPI_STATUSES_IGNORE); // waits for all receives to be finished
Currently, the send/receive times for each node are very different, some processes taking a much longer time to go through this code than other nodes. Is there something better to use than isend/irecv/waitall, especially considering that everything that is being sent is of different size?
Cheers. :)