Related
The following code is a implementation of MPI. There is a message with increasing length that is being sent en returned. With each iteration the number of elements of the message increase.
There are 2 statements in the code that I don't understand.
MPI_Probe(MPI_ANY_SOURCE, MPI_ANY_TAG, MPI_COMM_WORLD, &status);
MPI_Get_count(&status, MPI_INT, &numberOfElementsReceived);
On the first sight it looked not necessary in order to pass the message forward and back, but when I deteted it and compiled my code, it gave an error.
I also checked this post: what is the difference between MPI_Probe and MPI_Get_count in mpi
""While MPI_Probe may be used to find the size of a message you have to use MPI_Get_count to get that size. MPI_Probe returns a status which is a data structure providing information about the message, including its source, tag and size. But to get that size you call MPI_Get_count with the status as an argument.""
Why is it important to know the size and length of the message with the functions MPI_Probe and MPI_Get_Count? This is confusing me because you already describe the number of elements you send and receive in the MPI_Send and MPI_Recv functions.
for (message_size = 1; message_size <= MAX_ARRAY_SIZE; message_size <<= 1)
{
// Use a loop to vary the message size
if (myRank == 0)
{
double startTime, endTime;
numberOfElementsToSend = message_size;
printf("Rank %2.1i: Sending %i elements\n", myRank, numberOfElementsToSend);
// Measure the time spent in MPI communication
// (use the variables startTime and endTime)
startTime = MPI_Wtime();
MPI_Send(myArray, numberOfElementsToSend, MPI_INT, 1, 0,
MPI_COMM_WORLD);
MPI_Probe(MPI_ANY_SOURCE, MPI_ANY_TAG, MPI_COMM_WORLD, &status);
MPI_Get_count(&status, MPI_INT, &numberOfElementsReceived);
MPI_Recv(myArray, numberOfElementsReceived, MPI_INT, 1, 0,
MPI_COMM_WORLD, MPI_STATUS_IGNORE);
endTime = MPI_Wtime();
printf("Rank %2.1i: Received %i elements\n",
myRank, numberOfElementsReceived);
printf("Ping Pong took %f seconds\n", endTime - startTime);
}
else if (myRank == 1)
{
// Probe message in order to obtain the amount of data
MPI_Probe(MPI_ANY_SOURCE, MPI_ANY_TAG, MPI_COMM_WORLD, &status);
MPI_Get_count(&status, MPI_INT, &numberOfElementsReceived);
MPI_Recv(myArray, numberOfElementsReceived, MPI_INT, 0, 0,
MPI_COMM_WORLD, MPI_STATUS_IGNORE);
printf("Rank %2.1i: Received %i elements\n",
myRank, numberOfElementsReceived);
numberOfElementsToSend = numberOfElementsReceived;
printf("Rank %2.1i: Sending back %i elements\n",
myRank, numberOfElementsToSend);
MPI_Send(myArray, numberOfElementsToSend, MPI_INT, 0, 0,
MPI_COMM_WORLD);
}
}
#include "mpi.h"
#include <stdio.h>
int main(argc,argv)
int argc;
char *argv[]; {
int numtasks, rank, dest, source, rc, count, tag=1;
MPI_Status Stat;
MPI_Init(&argc,&argv);
MPI_Comm_size(MPI_COMM_WORLD, &numtasks);
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
if (rank%2 == 1) {
dest = (rank+1)%numtasks;
source = (rank-1+numtasks)%numtasks;
rc = MPI_Send(&outmsg, 1, MPI_CHAR, dest, tag, MPI_COMM_WORLD);
rc = MPI_Recv(&inmsg, 1, MPI_CHAR, source, tag,MPICOMM_WORLD, &Stat);
}
else {
dest = (rank-1+numtasks)%numtasks;
source = (rank+1)%numtasks;
rc = MPI_Recv( &inmsg, 1, MPI_CHAR, source, tag, MPI_COMM_WORLD, &Stat );
rc = MPI_Send( &outmsg, 1, MPI_CHAR, dest, tag, MPI_COMM_WORLD );
}
rc = MPI_Get_count(&Stat, MPI_CHAR, &count);
printf("Task %d: Received %d char(s) from task %d with tag %d\n",
rank, count, Stat.MPI_SOURCE, Stat.MPI_TAG);
MPI_Finalize();
}
Results:
Task 2: Received 1 char(s) from task 3 with a tag 1
Task 0: Received 1 char(s) from task 1 with a tag 1
Why the task can't identify more than 2 processors?
I want run it on more than 2 processors.
I've updated the mpi program with the ring pattern, i think maybe wrong at the line if(rank%2==1)
May try to better investigate the MPI mechanics:
Do not hesitate to experiment with the code, as it will help you grasp the concept of numtasks, rank-ID and associated run-time operations. Compilation errors will report shortcuts in the above posted code. For more MPI-related practices and insights, one ought also assign and evaluate rc return codes, from the respective MPI_<fun>()-calls
Q: Why the task can't identify more than 2 processors?
A: It can, but the if(){..}else if(){..} code-blocks did not allow others to produce any visible output.
Q: I want to change that 4197005 result into 2, 3 - how?
A: One cannot change an MPI-reported number, but you can change the behaviour of your code ( see below ) and make some more outputs, where feasible about who is doing what, when, how and where it is actually going on. This way we learn to understand the MPI-concepts and validate the actual MPI-code execution.
Feel free to ask more.
#include "mpi.h"
#include <stdio.h>
int main( int argc,
char *argv[]
) {
int numtasks, rank, dest, source, rc, count, tag = 1;
MPI_Status Stat;
printf( "(INF) will MPI_Init()...\n" );
MPI_Init( &argc, &argv );
printf( "(INF) MPI_Init() done.\n" );
MPI_Comm_size( MPI_COMM_WORLD, &numtasks );
MPI_Comm_rank( MPI_COMM_WORLD, &rank );
printf( "(INF) I am MPI[%d of %d]\n",
rank,
numtasks
);
if ( rank == 0 ) {
dest = 1;
source = 1;
rc = MPI_Send( &outmsg, 1, MPI_CHAR, dest, tag, MPI_COMM_WORLD );
rc = MPI_Recv( &inmsg, 1, MPI_CHAR, source, tag, MPI_COMM_WORLD, &Stat );
printf( "(INF) I am MPI[%d of %d] having just completed an MPI_Send( ->MPI[%d] ) + MPI_Recv( <-MPI[%d] ). Last RetCODE == %d.\n",
rank,
numtasks,
dest,
source,
rc
);
}
else
if ( rank == 1 ) {
dest = 0;
source = 0;
rc = MPI_Recv( &inmsg, 1, MPI_CHAR, source, tag, MPI_COMM_WORLD, &Stat );
rc = MPI_Send( &outmsg, 1, MPI_CHAR, dest, tag, MPI_COMM_WORLD );
printf( "(INF) I am MPI[%d of %d] having just completed an MPI_Recv( <-MPI[%d] ) + MPI_Send( ->MPI[%d] ). Last RetCODE == %d\n",
rank,
numtasks,
source,
dest,
rc
);
}
else
if ( rank == 2 ) {
dest = 3;
source = 3;
rc = MPI_Recv( &inmsg, 1, MPI_CHAR, source, tag, MPI_COMM_WORLD, &Stat );
rc = MPI_Send( &outmsg, 1, MPI_CHAR, dest, tag, MPI_COMM_WORLD );
printf( "(INF) I am MPI[%d of %d] having just completed an MPI_Recv( <-MPI[%d] ) + MPI_Send( ->MPI[%d] ). Last RetCODE == %d\n",
rank,
numtasks,
source,
dest,
rc
);
}
else
if ( rank == 3 ) {
dest = 2;
source = 2;
rc = MPI_Send( &outmsg, 1, MPI_CHAR, dest, tag, MPI_COMM_WORLD );
rc = MPI_Recv( &inmsg, 1, MPI_CHAR, source, tag, MPI_COMM_WORLD, &Stat );
printf( "(INF) I am MPI[%d of %d] having just completed an MPI_Send( ->MPI[%d] ) + MPI_Recv( <-MPI[%d] ). Last RetCODE == %d\n",
rank,
numtasks,
dest,
source,
rc
);
}
else {
printf( "(INF) I am MPI[%d of %d] will NOP...\n",
rank,
numtasks
);
}
rc = MPI_Get_count( &Stat, MPI_CHAR, &count );
/*
Task 3: Received 1 char(s) from task 4197005 with a tag 0
Task 2: Received 1 char(s) from task 4197005 with a tag 0
Task 1: Received 1 char(s) from task 0 with a tag 1
Task 0: Received 1 char(s) from task 1 with a tag 1 */
printf( "Task %d: Received %d char(s) from task %d with tag %d\n",
rank,
count,
Stat.MPI_SOURCE,
Stat.MPI_TAG
);
printf( "(INF) I am MPI[%d of %d]: will MPI_Finalize()...\n",
rank,
numtasks
);
MPI_Finalize();
printf( "(INF) I am MPI[%d of %d]: MPI_Finalize() done, will exit()...\n",
rank,
numtasks
);
}
count and Stat are only initialized on ranks 0 and 1, so you are basically printing garbage on ranks 2 and 3.
Your program is hard-coded to run on 2 tasks (it will crash on 1 task, and print garbage with 3 tasks or more).
I suggest you investigate using MPI_Sendrecv() in order to send and receive deadlock-free in one MPI call.
Not sure what you are trying to achieve, please elaborate and update your question. That being said, you might consider a ring pattern so your program can run on any number of MPI tasks (e.g. send to (rank+1)%numtasks and receive from (rank+numtasks-1)%numtasks)
Per your edited question, the communication pattern can the same for all the MPI ranks as follow
dest = (rank+1)%numtasks;
source = (rank-1+numtasks)%numtasks;
MPI_Sendrecv(&outmsg, 1, MPI_CHAR, dest, tag,
&inmsg, 1, MPI_CHAR, source, tag,
MPI_COMM_WORLD, &Stat);
#include "mpi.h"
#include <stdio.h>
int main (argc,argv)
int argc;
char *argv[]; {
int numtasks, rank, dest, source, rc, count, tag=1;
char inmsg, outmsg='x';
MPI_Status Stat;
MPI_Init (&argc, &argv);
MPI_Comm_size (MPI_COMM_WORLD, &numtasks);
MPI_Comm_rank (MPI_COMM_WORLD, &rank);
if (rank%2 == 0)
{
dest=(rank+1)%numtasks;
source=(rank+1)%numtasks;
rc = MPI_Send(&outmsg, 1, MPI_CHAR, source, tag, MPI_COMM_WORLD);
rc = MPI_Recv(&inmsg, 1, MPI_CHAR, source, tag, MPI_COMM_WORLD, &Stat);
}
else {
dest = (rank-1+numtasks)%numtasks;
source = (rank-1+numtasks)%numtasks;
rc = MPI_Recv (&inmsg, 1, MPI_CHAR, source, tag, MPI_COMM_WORLD, &Stat);
rc = MPI_Send (&outmsg, 1, MPI_CHAR, dest, tag, MPI_COMM_WORLD);
}
{
rc = MPI_Get_count (&Stat, MPI_CHAR, &count);
printf ("Task %d: Received %d char(s) from task %d with a tag %d \n", rank, count, Stat.MPI_SOURCE, Stat.MPI_TAG);
}
MPI_Finalize();
I have a strange issue where the values I am packing into a buffer for an MPI Bcast are different from those that have been unpacked.
I am not sure what this may have resulted from. The first value packed (MTD) is always correct but those afterward are very off.
I require one of these values (dimensions) for dynamic memory allocation, the result of which then causes my program to crash.
If anyone familiar with MPI can cast there eyes over I would be eternally grateful!.
void recv_sharedData(int me, SharedData *d) {
fprintf(stderr, "%d: RECEIVING SHARED DATA\n", me);
int buff_size = sizeof(int) * 4 + sizeof(float);
char * buffer = malloc(buff_size);
int position = 0;
if (MPI_Bcast(
buffer, 5, MPI_PACKED, 0, MPI_COMM_WORLD) != MPI_SUCCESS) {
fprintf(stderr, "ID %d: Could not receive SharedData", me);
MPI_Abort(MPI_COMM_WORLD, -1);
}
MPI_Unpack(buffer, buff_size, &position, &d->max_tree_depth, 1,
MPI_INT, MPI_COMM_WORLD);
MPI_Unpack(buffer, buff_size, &position, &d->bit_depth, 1,
MPI_INT, MPI_COMM_WORLD);
MPI_Unpack(buffer, buff_size, &position, &d->dimensions, 1,
MPI_INT, MPI_COMM_WORLD);
MPI_Unpack(buffer, buff_size, &position, &d->origin, 1,
MPI_INT, MPI_COMM_WORLD);
MPI_Unpack(buffer, buff_size, &position, &d->colour_max, 1,
MPI_FLOAT, MPI_COMM_WORLD);
fprintf(stderr, "%d: MTD: %d\n", me, d->max_tree_depth);
fprintf(stderr, "%d: BitDepth: %d\n", me, d->bit_depth);
fprintf(stderr, "%d: DIM: %d\n", me, d->dimensions);
fprintf(stderr, "%d: ORIGIN: %d\n", me, d->origin);
....
void bcast_sharedData(SharedData *d) {
fprintf(stderr, "Broadcasting Shared Data\n");
int position = 0;
int buff_size = sizeof(int) * 4 + sizeof(float);
char * buffer = malloc(buff_size);
int result = MPI_SUCCESS;
while(result == MPI_SUCCESS) {
result = MPI_Pack(&d->max_tree_depth, 1, MPI_INT, buffer,
buff_size, &position, MPI_COMM_WORLD);
result = MPI_Pack(&d->bit_depth, 1, MPI_INT, buffer, buff_size,
&position, MPI_COMM_WORLD);
result = MPI_Pack(&d->dimensions, 1, MPI_INT, buffer, buff_size,
&position, MPI_COMM_WORLD);
result = MPI_Pack(&d->origin, 1, MPI_INT, buffer, buff_size,
&position, MPI_COMM_WORLD);
result = MPI_Pack(&d->colour_max, 1, MPI_FLOAT, buffer,
buff_size, &position, MPI_COMM_WORLD);
break;
}
if (result != MPI_SUCCESS) {
fprintf(stderr,
"ID 0: Could not Pack Shared Data for Broadcasting");
MPI_Abort(MPI_COMM_WORLD, -1);
}
fprintf(stderr, "%d: MTD: %d\n", 0, d->max_tree_depth);
fprintf(stderr, "%d: BitDepth: %d\n", 0, d->bit_depth);
fprintf(stderr, "%d: DIM: %d\n", 0, d->dimensions);
fprintf(stderr, "%d: ORIGIN: %d\n", 0, d->origin);
The related values to the above are output below. The root values (0:) are what are being packed:
Broadcasting Shared Data
0: MTD: 2
0: BitDepth: 24
0: DIM: 19
0: ORIGIN: 9
3: MTD: 2
3: BitDepth: 32536
3: DIM: 11816240
3: ORIGIN: 0
3: Done Recieving SharedData
1: MTD: 2
1: BitDepth: 32536
1: DIM: 30698800
1: ORIGIN: 0
1: Done Recieving SharedData
1: ALLOCATING KERNEL
2: MTD: 2
2: BitDepth: 32536
2: DIM: 27622704
2: ORIGIN: 0
2: Done Recieving SharedData
I am trying to get an alternative code to
if(rank %2==0 && rightNeighbour != MPI_PROC_NULL)
MPI_Sendrecv_replace(&bufferRight[0], len, MPI_DOUBLE, rightNeighbour, 1,
rightNeighbour, 1, MPI_COMM_WORLD, &status);
else if(rank%2 ==1 && leftNeighbour != MPI_PROC_NULL)
MPI_Sendrecv_replace(&bufferLeft[0], len, MPI_DOUBLE, leftNeighbour, 1,
leftNeighbour, 1, MPI_COMM_WORLD, &status);
if (rank % 2 == 1 && rightNeighbour != MPI_PROC_NULL)
MPI_Sendrecv_replace(&bufferRight[0], len, MPI_DOUBLE, rightNeighbour, 1,
rightNeighbour, 1, MPI_COMM_WORLD, &status);
else if (rank % 2 == 0 && leftNeighbour != MPI_PROC_NULL)
MPI_Sendrecv_replace(&bufferLeft[0], len, MPI_DOUBLE, leftNeighbour, 1,
leftNeighbour, 1, MPI_COMM_WORLD, &status);
using MPI_Send and MPI_Recv but it seems it's deadlocking. Any easy way of doing the same with MPI_Send and MPI_Recv ?
I have tried using
if(rank %2==0 && rightNeighbour != MPI_PROC_NULL){
MPI_Recv(&bufferRight[0], len, MPI_DOUBLE, rightNeighbour, 1,
MPI_COMM_WORLD, &status);
MPI_Send(&bufferRight[0], len, MPI_DOUBLE, rightNeighbour, 1,
MPI_COMM_WORLD);
}
This code:
if(rank % 2 == 0 && rightNeighbour != MPI_PROC_NULL)
MPI_Sendrecv_replace(&bufferRight[0], len, MPI_DOUBLE, rightNeighbour, 1,
rightNeighbour, 1, MPI_COMM_WORLD, &status);
else if(rank % 2 == 1 && leftNeighbour != MPI_PROC_NULL)
MPI_Sendrecv_replace(&bufferLeft[0], len, MPI_DOUBLE, leftNeighbour, 1,
leftNeighbour, 1, MPI_COMM_WORLD, &status);
is equivalent to:
if(rank % 2 == 0)
{
MPI_Send(&bufferRight[0], len, MPI_DOUBLE, rightNeighbour, 1,
MPI_COMM_WORLD);
MPI_Recv(&bufferRight[0], len, MPI_DOUBLE, rightNeighbour, 1,
MPI_COMM_WORLD, &status);
}
else if(rank % 2 == 1)
{
double *temp = malloc(len * sizeof(double));
MPI_Recv(temp, len, MPI_DOUBLE, leftNeighbour, 1,
MPI_COMM_WORLD, &status);
MPI_Send(&bufferLeft[0], len, MPI_DOUBLE, leftNeighbour, 1,
MPI_COMM_WORLD, &status);
memcpy(&bufferLeft[0], temp, len * sizeof(double));
free(temp);
}
Note that order of the send and receive calls is reversed in the odd ranks. Also, the receive uses a temporary buffer in order to implement the semantics of MPI_Sendrecv_replace, which guarantees that the data in the buffer is first sent and only then overwritten with the received one.
Note that the check whether a rank is not MPI_PROC_NULL is pointless since a send to/receive from MPI_PROC_NULL is essentially a no-op and will always succeed. One of the key ideas of the semantics of MPI_PROC_NULL is to facilitate the writing of symmetric code that doesn't contain such if's.
You better use MPI_Irecv and MPI_Isend rather than the blocking calls (MPI_Recv and MPI_Send). Then, after issuing the communication routines, simply wait for the requests using the MPI_Waitall (or two calls to MPI_Wait). However, to do this you cannot use the same buffer (i.e. the replace) -- you need to have two separate buffers -- because otherwise they would get corrupted as the buffers might get the content replaced before the actual send.
Let A be the incoming buffer and B the outgoing buffesr, your code should look something like
if(rank %2==0 && rightNeighbour != MPI_PROC_NULL){
MPI_Request req[2];
MPI_Status status[2];
MPI_Irecv (&A, len, MPI_DOUBLE, rightNeighbour, 1,
MPI_COMM_WORLD, &req[0]);
MPI_Isend (&B, len, MPI_DOUBLE, rightNeighbour, 1,
MPI_COMM_WORLD, &req[1]);
/* A */
MPI_Waitall (2, req, status);
}
Note that in /* A */ you can take advantage to do some computation while the communication is flying. Also, the error checking is omitted in the code -- you better check all return codes for the MPI calls.
Although probably the best way to proceed is to follow Harald's suggestion on using MPI_Isend and MPI_Irecv, there's one alternative which might not work depending on the case.
For small buffer sizes, some MPI implementations follow what is known as eager mode. In this mode, the message is sent regardless the receiver is already waiting for the message or not. The send buffer data is copied to a temporary buffer and, since the user's send buffer is available after the copy is done, MPI_Send returns before the communication has actually been completed.
This is a bit risky, since for large messages MPI usually works in rendezvous mode, which actually synchronizes both ends, so the following code would produce a deadlock as well:
if(rank %2==0 && rightNeighbour != MPI_PROC_NULL){
MPI_Send(&bufferRight[0], len, MPI_DOUBLE, rightNeighbour, 1,
MPI_COMM_WORLD);
MPI_Recv(&bufferRight[0], len, MPI_DOUBLE, rightNeighbour, 1,
MPI_COMM_WORLD, &status);
}
In MPI_Recv, however, the reception buffer is (obviously) not available until the data has already been received.
Additional info on MPI performance tips.
I have developed a given simple MPI program such that process 0 sends message to process 1 and receives message from process p-1. Following is the code :
In the skeleton given to me ,
char *message;
message= (char*)malloc(msg_size);
is confusing me. To check the correctness of program, I am trying to look value of message that been sent or received. So should it be hexadecimal value?
int main(int argc, char **argv)
{
double startwtime, endwtime;
float elapsed_time, bandwidth;
int my_id, next_id; /* process id-s */
int p; /* number of processes */
char* message; /* storage for the message */
int i, k, max_msgs, msg_size, v;
MPI_Status status; /* return status for receive */
MPI_Init( &argc, &argv );
MPI_Comm_rank( MPI_COMM_WORLD, &my_id );
MPI_Comm_size( MPI_COMM_WORLD, &p );
if (argc < 3)
{
fprintf (stderr, "need msg count and msg size as params\n");
goto EXIT;
}
if ((sscanf (argv[1], "%d", &max_msgs) < 1) ||
(sscanf (argv[2], "%d", &msg_size) < 1))
{
fprintf (stderr, "need msg count and msg size as params\n");
goto EXIT;
}
**message = (char*)malloc (msg_size);**
if (argc > 3) v=1; else v=0; /*are we in verbose mode*/
/* don't start timer until everybody is ok */
MPI_Barrier(MPI_COMM_WORLD);
int t=0;
if( my_id == 0 ) {
startwtime = MPI_Wtime();
// do max_msgs times:
// send message of size msg_size chars to process 1
// receive message of size msg_size chars from process p-1
while(t<max_msgs) {
MPI_Send((char *) message, msg_size, MPI_CHAR, 1 , 0, MPI_COMM_WORLD);
MPI_Recv((char *) message, msg_size, MPI_CHAR, p-1, 0, MPI_COMM_WORLD, &status);
t++;
}
MPI_Barrier(MPI_COMM_WORLD);
endwtime = MPI_Wtime();
elapsed_time = endwtime-startwtime;
bandwidth = 2.0 * max_msgs * msg_size / (elapsed_time);
printf("Number, size of messages: %3d , %3d \n", max_msgs, msg_size);
fflush(stdout);
printf("Wallclock time = %f seconds\n", elapsed_time );
fflush(stdout);
printf("Bandwidth = %f bytes per second\n", bandwidth);
fflush(stdout);
} else if( my_id == p-1 ) {
// do max_msgs times:
// receive message of size msg_size from process to the left
// send message of size msg_size to process to the right (p-1 sends to 0)
while(t<max_msgs) {
MPI_Send((char *) message, msg_size, MPI_CHAR, 0, 0, MPI_COMM_WORLD);
MPI_Recv((char *) message, msg_size, MPI_CHAR, my_id-1, 0, MPI_COMM_WORLD, &status);
t++;
}
} else {
while(t<max_msgs) {
MPI_Send((char *) message, msg_size, MPI_CHAR, my_id+1, 0, MPI_COMM_WORLD);
MPI_Recv((char *) message, msg_size, MPI_CHAR, my_id-1, 0, MPI_COMM_WORLD, &status);
t++;
}
}
MPI_Barrier(MPI_COMM_WORLD);
EXIT:
MPI_Finalize();
return 0;
}
I am not completely sure if this is what you mean, but I will try.
For what I understand, you want to know what is the message being sent. Well, for the code you provide, memory is assign to the message but any real "readable" message is specify. In this line.
message = (char*)malloc (msg_size);
malloc reserves the memory for the messages, so anyone can write it, however, it doesn't provide any initial value. Sometimes, the memory contains other information previously stored and freed. Then, the message being sent is that "garbage" that is before. This is probably what you call hexadecimal (I hope I understand this right).
The type of value in this case is char (defined as MPI_CHAR in the MPI_Send and MPI_Recv functions). Here you can find more data types for MPI.
I will suggest to assign a value to the message with the with my_id and next_id. So you know who is sending to whom.