I am tring to use GDB to see what's going on about low level OPEN MPI calls. However, I couldn't use GDB to jump into low level library functions in OpenMPI.
I first compiled OpenMPI with tag "--enable-debug --enable-debug-symbols", so library should be compiled with "-g" tag. Then I compile my test code using "mpicc /path/to/my/code -g -o /dest/code". Code is simple shown as below:
#include "mpi.h"
#include <stdio.h>
main(int argc, char *argv[]) {
int numtasks, rank, dest, source, rc, count, tag=1;
int buf;
const int root=0;
MPI_Status Stat;
// sleep still GDB attached
int i = 0;
char hostname[256];
gethostname(hostname, sizeof(hostname));
printf("PID %d on %s ready for attach\n", getpid(), hostname);
fflush(stdout);
while (0 == i)
sleep(5);
MPI_Init(&argc,&argv);
MPI_Comm_size(MPI_COMM_WORLD, &numtasks);
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
if(rank == root) {
buf = 777;
}
printf("[%d]: Before Bcast, buf is %d\n", rank, buf);
MPI_Bcast(&buf, 1, MPI_INT, root, MPI_COMM_WORLD);
printf("[%d]: After Bcast, buf is %d\n", rank, buf);
MPI_Finalize();
}
This code will firstly printout PID for every process, then keep sleep wait for attach. Then I use GDB attach method to attach one of the process. Set break point & reset the value of i will jump out of waiting while loop.
Then, I continue use "s" to step next my code. It supposed to jump into low level library functions like MPI_Init(), MPI_Bcast(). However, GDB will just going to next line and ignore jumping into any function. Neither "bt" could print out backtrace information. But the whole code could run without error.
Could someone found any mistake I made in this whole process? Thanks ahead.
Related
The following code has non-deterministic behaviour on my machine (already with using only two processes).
#include <mpi.h>
#include <stdio.h>
#include <stdlib.h>
int main(int argc, char *argv[]) {
int rank, size;
MPI_Init(&argc, &argv);
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
MPI_Comm_size(MPI_COMM_WORLD, &size);
if (rank == size - 1) {
printf("I am the last rank\n");
MPI_Barrier(MPI_COMM_WORLD);
} else {
MPI_Barrier(MPI_COMM_WORLD);
printf("I am rank %d\n", rank);
}
MPI_Finalize();
return 0;
}
Sometimes the output from the last rank appears first on the terminal but sometimes it does appear later, even though a barrier is used.
I assume the reason for this is that printf does internal buffering and that MPI respectively mpirun/mpiexec and printf do not really cooperate with each other. Is there a more valid source to read up on this topic?
Output redirection is strongly platform-dependent and is only tangentially mentioned in the MPI standard. There is absolutely no guarantee as to what order output from different ranks will be displayed in. There isn't even a guarantee that you can get that output - some implementations only redirect the output from rank 0, some redirect no output, and both cases are compliant with the MPI standard.
If you want strongly ordered output, you should only do output in a single rank.
I'm trying to write a elaborate hello world program using MPI. The idea is that the root process sends a message to the other processes, which in turn sends a message to the root containing their process id. So I want to send several character arrays to the root process and eventually print it.
The problem is that I don't manage to collect all the messages into a variable in the root process. I only get compile errors or unwanted output. I feel like I'm making some basic error in manipulating the char arrays ... I'd be glad if you could help me out!
The code so far:
// Hello world using MPI
#include <stdio.h>
#include <math.h>
#include <mpi.h>
#include <string.h>
int main(int argc, char **argv){
// Define useful global constants
int my_id, root_process, num_procs, rc;
MPI_Status status;
char[] final_message; // <---- How should I define final message? Can I do this in another way?
// Let master process be process 0
root_process = 0;
// Now, replicate this process to create parallel processes:
rc = MPI_Init(&argc, &argv);
// Find out MY process ID and how many processes were started:
rc = MPI_Comm_rank(MPI_COMM_WORLD, &my_id);
rc = MPI_Comm_size(MPI_COMM_WORLD, &num_procs);
// Broadcast master's message to slaves
char message1[20];
strcpy(message1, "Hello, World!");
rc = MPI_Bcast(&message1, 14, MPI_CHAR, root_process, MPI_COMM_WORLD);
// TEST: Each slave now displays message1 along with which process it is.
//printf("I'm process %i, the message is: %.13s \n",my_id,message1);
// Now the master collects a return message from each slave, including the process ID.
char message2[31];
char message22[2];
strcpy(message2, "Master, I'm process number ");// %i \n",&my_id ); //28
sprintf(message22,"%d\n",my_id);
strcat(message2,message22);
rc = MPI_Gather(message2, 1, MPI_CHAR, final_message, 1, MPI_CHAR, root_process, MPI_COMM_WORLD);
// Display the message collected by the master.
printf(final_message);
// Close down this process
rc = MPI_Finalize();
}
I am trying to run the example code at the following url. I compiled the program with "mpicc twoGroups.c" and tried to run it as "./a.out", but got the following message: Must specify MP_PROCS= 8. Terminating.
My question is how do you set MP_PROCS=8 ?
Group and Communication Routine examples at here https://computing.llnl.gov/tutorials/mpi/
#include "mpi.h"
#include <stdio.h>
#define NPROCS 8
main(int argc, char *argv[]) {
int rank, new_rank, sendbuf, recvbuf, numtasks,
ranks1[4]={0,1,2,3}, ranks2[4]={4,5,6,7};
MPI_Group orig_group, new_group;
MPI_Comm new_comm;
MPI_Init(&argc,&argv);
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
MPI_Comm_size(MPI_COMM_WORLD, &numtasks);
if (numtasks != NPROCS) {
printf("Must specify MP_PROCS= %d. Terminating.\n",NPROCS);
MPI_Finalize();
exit(0);
}
sendbuf = rank;
/* Extract the original group handle */
MPI_Comm_group(MPI_COMM_WORLD, &orig_group);
/* Divide tasks into two distinct groups based upon rank */
if (rank < NPROCS/2) {
MPI_Group_incl(orig_group, NPROCS/2, ranks1, &new_group);
}
else {
MPI_Group_incl(orig_group, NPROCS/2, ranks2, &new_group);
}
/* Create new new communicator and then perform collective communications */
MPI_Comm_create(MPI_COMM_WORLD, new_group, &new_comm);
MPI_Allreduce(&sendbuf, &recvbuf, 1, MPI_INT, MPI_SUM, new_comm);
MPI_Group_rank (new_group, &new_rank);
printf("rank= %d newrank= %d recvbuf= %d\n",rank,new_rank,recvbuf);
MPI_Finalize();
}
When you execute an MPI program, you need to use the appropriate wrappers. Most of the time it looks like this:
mpiexec -n <number_of_processes> <executable_name> <executable_args>
So for your simple example:
mpiexec -n 8 ./a.out
You will also see mpirun used instead of mpiexec or -np instead of -n. Both are fine most of the time.
If you're just starting out, it would also be a good idea to make sure you're using a recent version of MPI so you don't get old bugs or weird execution environments. MPICH and Open MPI are the two most popular implementations. MPICH just released version 3.1 available here while Open MPI has version 1.7.4 available here. You can also usually get either of them via your friendly neighborhood package manager.
I'm studying a bit of MPI, and decided to do a test by making a program that calls objects, eg main.c -> main program, function.c -> any function
function.c that will only use the MPI. compiling I as follows:
gcc-c main.c
to create main.o, mpicc-c to create function.c function.o, of course I create the file function.h too.
I compile with mpicc-o program main.o function.o
Here is main.c
#include <stdio.h>
#include "function.h"
void main(int argc, char *argv[])
{
printf("Hello\n");
function();
printf("Bye\n");
}
just function has the MPI code, but when I'm running the program mpiexe -np 2 I get
Hello
Hello
----- function job here -----
Bye
Bye
But I wanted it to be
Hello
------ function job -----
Bye
What can I do?
Your whole program is run on both of the two processors you set with the -np 2. A common way to prevent duplicates of printouts, final results, etc., is have one thread do those things just by checking the thread id first. Like:
int id;
MPI_Comm_rank(MPI_COMM_WORLD, &id);
if (id == 0) {
printf("only process %d does this one\n", id);
}
printf("hello from process %d\n", id); // all processes do this one
When starting out in MPI I found it helpful to print out those id numbers along with whatever partial results or data each thread was dealing with. Helped me make more sense out of what was happening.
Basically mpirun -np 2 starts 2 identical processes and you have to use MPI_Comm_rank function to check process rank.
Here is a quick snippet:
int main(int argc, char **argv)
{
int myrank;
MPI_Init(&argc, &argv);
MPI_Comm_rank(MPI_COMM_WORLD, &myrank);
if (myrank == 0) {
printf("Hello\n");
function();
MPI_Barrier(MPI_COMM_WORLD);
printf("Done\n");
} else {
function();
MPI_Barrier(MPI_COMM_WORLD);
}
MPI_Finalize();
return 0;
}
I generally prefer this method for printing data.
It involves barriers. So you must be careful while using it.
if(1)
do
for(i = 0 to num_threads)
do
if(i==my_rank)
do
do_printf
done
******* barrier ********
end for
done
If the set of threads printing the value does not include all threads, just add the relevant threads to barrier.
Another method is for every thread to write its output in a dedicated file. This way :
you don't have to access a barrier
you do not lose printfs of any thread
you output is explicit. so there is no cluttering while debugging programs.
Code :
sprintf(my_op_file_str, "output%d", myThreadID);
close(1)
open(my_op_file_str)
Now use printf's anywhere you may like.
I having some synchronization issues using the OpenMPI implementation of MPI_Barrier:
int rank;
int nprocs;
int rc = MPI_Init(&argc, &argv);
if(rc != MPI_SUCCESS) {
fprintf(stderr, "Unable to set up MPI");
MPI_Abort(MPI_COMM_WORLD, rc);
}
MPI_Comm_size(MPI_COMM_WORLD, &nprocs);
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
printf("P%d\n", rank);
fflush(stdout);
MPI_Barrier(MPI_COMM_WORLD);
printf("P%d again\n", rank);
MPI_Finalize();
for mpirun -n 2 ./a.out
output should be:
P0
P1
...
output is sometimes:
P0
P0 again
P1
P1 again
what's going on?
The order in which your print out lines appear on your terminal is not necessarily the order in which things are printed. You are using a shared resource (stdout) for that so there always must be an ordering problem. (And fflush doesn't help here, stdout is line buffered anyhow.)
You could try to prefix your output with a timestamp and save all of this to different files, one per MPI process.
Then to inspect your log you could merge the two files together and sort according to the timestamp.
Your problem should disappear, then.
There is nothing wrong with MPI_Barrier().
As Jens mentioned, the reason why you are not seeing the output you expected is because stdout is buffered on each processes. There is no guarantee that prints from multiple processes will be displayed on the calling process in order. (If stdout from each process is be transferred to the main process for printing in real time, that will lead to lots of unnecessary communication!)
If you want to convince yourself that the barrier works, you could try writing to a file instead. Having multiple processes writing to a single file may lead to extra complications, so you could have each proc writing to one file, then after the barrier, swap the files they write to. For example:
Proc-0 Proc-1
| |
f0.write(..) f1.write(...)
| |
x ~~ barrier ~~ x
| |
f1.write(..) f0.write(...)
| |
END END
Sample implementation:
#include "mpi.h"
#include <stdio.h>
#include <stdlib.h>
int main(int argc, char **argv) {
char filename[20];
int rank, size;
FILE *fp;
MPI_Init(&argc, &argv);
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
MPI_Comm_size(MPI_COMM_WORLD, &size);
if (rank < 2) { /* proc 0 and 1 only */
sprintf(filename, "file_%d.out", rank);
fp = fopen(filename, "w");
fprintf(fp, "P%d: before Barrier\n", rank);
fclose(fp);
}
MPI_Barrier(MPI_COMM_WORLD);
if (rank < 2) { /* proc 0 and 1 only */
sprintf(filename, "file_%d.out", (rank==0)?1:0 );
fp = fopen(filename, "a");
fprintf(fp, "P%d: after Barrier\n", rank);
fclose(fp);
}
MPI_Finalize();
return 0;
}
After running the code, you should get the following results:
[me#home]$ cat file_0.out
P0: before Barrier
P1: after Barrier
[me#home]$ cat file_1.out
P1: before Barrier
P0: after Barrier
For all files, the "after Barrier" statements will always appear later.
Output ordering is not guaranteed in MPI programs.
This is not related to MPI_Barrier at all.
Also, I would not spend too much time on worrying about output ordering with MPI programs.
The most elegant way to achieve this, if you really want to, is to let the processes send their messages to one rank, say, rank 0, and let rank 0 print the output in the order it received them or ordered by ranks.
Again, dont spend too much time on trying to order the output from MPI programs. It is not practical and is of little use.
Adding to the previous answers here, your MPI_BARRIER works fine.
Though, if you just intend to see it working, you can force pause the execution (SLEEP(1)) for a moment to let the output catch up.