I am learning MPI for parallel programming in C and I am using a processor with 4 cores. I am trying to do an example from a tutorial in which the output should be:
Hello world! I'm process 0 out of 4 processes
Hello world! I'm process 2 out of 4 processes
Hello world! I'm process 1 out of 4 processes
Hello world! I'm process 3 out of 4 processes
In whatever order.
Here is my code:
#include <stdio.h>
#include <mpi.h>
int main(int argc, char** argv)
{
int ierr, num_procs, my_id;
ierr = MPI_Init(&argc, &argv);
ierr = MPI_Comm_rank(MPI_COMM_WORLD, &my_id);
ierr = MPI_Comm_size(MPI_COMM_WORLD, &num_procs);
printf("Hello world! I'm process %i out of %i processes\n", my_id, num_procs);
ierr = MPI_Finalize();
}
I compile it using:
mpicc helloworld.c -o helloworld
And I run it using:
mpirun -np 4 helloworld
This is what is outputted:
Hello world! I'm process 0 out of 1 processes
Hello world! I'm process 0 out of 1 processes
Hello world! I'm process 0 out of 1 processes
Hello world! I'm process 0 out of 1 processes
It's outputting it 4 times which is relatively good news I guess but the program isn't recognising the number of threads and each thread ID.
Is it even running in parallel or is it just running 4 times in serial?
How can I get the program to recognise the amount of threads and the thread ID properly?
Thanks in advance!
mpicc helloworld.c -o helloworld
mpirun -np 4 helloworld
Hello world! I'm process 0 out of 1 processes
Hello world! I'm process 0 out of 1 processes
Hello world! I'm process 0 out of 1 processes
Hello world! I'm process 0 out of 1 processes
This sequence clearly shows us that your MPI runtime was unable to detect parallel start, it probably from misconfiguration: your mpicc is from one MPI implementation and your mpirun is from other. For example, both MPICH and OpenMPI have mpicc scripts for compiling MPI programs, but their mpiexec/mpirun programs are incompatible. Compile with MPICH, start with OpenMPI starter and MPICH runtime will not receive needed environment variables to figure out parallel run and its parameters.
You should revisit list of installed packages (dpkg -l|egrep 'mpich|openmpi') and check which file is from which library (dpkg -L mpich, dpkg -L openmpi-bin; dpkg -L libmpich-dev, dpkg -L libopenmpi-dev). Ubuntu/debian also have system of "alternatives" which will install symbolic links mpicc and mpirun to actual scripts (do ls -l /usr/bin/mpicc /usr/bin/mpirun to see current state of the links). Check update-alternatives tool, its man page and docs to learn how to reset all mpi-named scripts to one implementation (and there is galternatives GUI for it).
According to file lists in packages, mpich and openmpi have variants of mpirun/mpiexec with suffixes http://packages.ubuntu.com/yakkety/amd64/openmpi-bin/filelist http://packages.ubuntu.com/yakkety/amd64/mpich/filelist:
/usr/bin/mpiexec.openmpi
/usr/bin/mpirun.openmpi
/usr/bin/mpiexec.hydra
/usr/bin/mpiexec.mpich
/usr/bin/mpirun.mpich
Same situation for mpicc scripts: http://packages.ubuntu.com/yakkety/amd64/libopenmpi-dev/filelist http://packages.ubuntu.com/yakkety/amd64/libmpich-dev/filelist
/usr/bin/mpicc.openmpi
/usr/bin/mpicc.mpich
Always use mpicc and mpirun (or mpiexec) from the same implementation. You may also use variants with suffix to be sure: mpicc.openmpi & mpiexec.openmpi pair or mpicc.mpich & mpiexec.mpich pair.
And to use some MPI implementation, you should have it fully installed, both for bin, lib and dev packages.
Related
mpirun (and mpiexec) do not seem to be passing command line arguments to my c code.
Running my exeutable "gecko" directly from the command line gives expected:
$ ./gecko -np 2
main:There are 3 arguments:
arg=./gecko
arg=-np
arg=2
But running the same through mpirun is different:
$ mpirun -np 2 ./gecko
main:There are 1 arguments:
arg=./gecko
which means MPI_init(argc,argv) doesn't have any arguments to work with.
I'm on Ubuntu 12.04 with MPICH 2.
Can anyone see a reason why this is not happening?
Thanks.
--------------------------- EDIT ---------------------------------
There are many examples on the net that say the way to initialize MPI is via the command line arguments, eg.:
#include <stdio.h>
#include “mpi.h”
int main(int argc, char* argv[])
{
int size, rank;
MPI_Init(&argc, &argv);
MPI_Comm_size(MPI_COMM_WORLD, &size);
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
printf(“Greetings from process %i\n”, rank);
MPI_Finalize();
return 0;
}
and that the way to execute the mpi code is to use:
mpirun -np 2 ./code_name
So, if mpirun does not pass the arguments np and 2 to the c code, how does the c code ever get to know how many processors it should be run on?
The -np argument is not meant to be interpreted by your executable, but by mpirun. If you want to pass extra arguments to your executable, you'd have to list them after the executable name as usual.
If you need the number of MPI tasks, you should use the appropriate MPI function that provide you with that number.
Edit:
MPI may pass startup information through the arguments given to MPI_Init or by any other means that pleases the implementor. Depending on the MPI implementation that you use, you may see that MPI passes a lot more arguments to your program. MPI_Init then rips all stuff that it needs and leaves only those things that it doesn't understand.
You should not rely on any such argument passing convention in your application. MPI is meant to be portable and you should only use MPI functions to access the runtime parameters.
Ever since circa 1997 MPI_Init() also works if both its arguments are NULL. This means that practically all MPI implementations use a different mechanism to pass configuration information, namely environment variables and files. For example, Open MPI passes the following variables:
$ cat print_ompi_vars
#!/bin/bash
printenv | grep ^OMPI | sort
$ mpiexec -np 1 ./print_ompi_vars
OMPI_COMM_WORLD_LOCAL_RANK=0
OMPI_COMM_WORLD_LOCAL_SIZE=1
OMPI_COMM_WORLD_NODE_RANK=0
OMPI_COMM_WORLD_RANK=0
OMPI_COMM_WORLD_SIZE=1
OMPI_MCA_btl=^tcp
OMPI_MCA_btl_openib_ib_timeout=24
OMPI_MCA_btl_openib_warn_default_gid_prefix=0
OMPI_MCA_btl_tcp_if_include=ib0
OMPI_MCA_carto=file
OMPI_MCA_carto_file_path=/opt/MPI/openmpi/carto/carto_2K-64C.txt
OMPI_MCA_ess=env
OMPI_MCA_mpi_yield_when_idle=0
OMPI_MCA_oob_tcp_if_include=ib0
OMPI_MCA_orte_app_num=0
OMPI_MCA_orte_cpu_model=Intel(R) Xeon(R) CPU X5675 # 3.07GHz
OMPI_MCA_orte_cpu_type=GenuineIntel
OMPI_MCA_orte_daemonize=1
OMPI_MCA_orte_ess_jobid=4114219009
OMPI_MCA_orte_ess_node_rank=0
OMPI_MCA_orte_ess_num_procs=1
OMPI_MCA_orte_ess_vpid=0
OMPI_MCA_orte_hnp_uri=4114219008.0;tcp://1.2.3.4:48206
OMPI_MCA_orte_local_daemon_uri=4114219008.1;tcp://1.2.3.4:59277
OMPI_MCA_orte_num_nodes=1
OMPI_MCA_orte_num_restarts=0
OMPI_MCA_orte_precondition_transports=1f4c3cf87403b137-a8e3173542efb9c3
OMPI_MCA_plm=rsh
OMPI_MCA_shmem_RUNTIME_QUERY_hint=mmap
OMPI_UNIVERSE_SIZE=1
It works similarly with other MPI implementations.
Anyway, if you wish to pass arguments to your MPI executable, put them as usual after the name of the executable:
$ mpiexec -n 16 <other mpiexec args> ./gecko -np 2 <other program args>
I was writing a code to understand MPI_Comm_spawn. I found some examples in the Internet, all compiled and executed with sucess but when I put the follow statement: mpirun -np 2 ./program I received a error. So, MPI_Comm_spawn only work with one process (-np 1)?
mpirun -np 1 ./executable - ok
mpirun -np 2 ./executable - don't run with values greater than one
I use the following code:
code used
You can definitely spawn processes when you have more than 1 process. That example is just designed to only work with one process. Here is an example of some code that will work with more than 1 process.
I'm currently making a command line app in C in which many child process is created. So I need to debug this children code. I have created a child process on Xcode as follow. (see the break point and running cursor.)
After executing some statements, I get xcode to attach GBN(885) to the xcode debugger as shown in below figure.
It doesn't work. How can I attach the child process to xcode debugger? Thank you in advance.
Google and Apple developer page are really silent on this issue and finally I've found a good workaround. This error message appears when we get Xcode debugger to attach to the process that is a child process of process being debugged or that is a process already being debugged by Xcode debugger, gdb, or lldb.
In order to avoid this irritating message, First put kill(0, SIGSTOP) statement where you want to stop the child process. In the following figure, I have put the kill statement right after the child is created.
Second, use gcc to compile your source codes with -g option.
$ ls
GBN.1 gbn.c gbn.h type.h util.c util.h
$ gcc *.c -g
$ ./a.out
[1]+ Stopped ./a.out
$ ps -ef | grep a.out
501 628 600 0 9:33AM ttys000 0:00.00 ./a.out
501 629 628 0 9:33AM ttys000 0:00.00 ./a.out
Now, all we have to do is to tell Xcode debugger to attach to the process by pid. Click the Debug menu and then find the Attach to process option to click By Process Identifier (PID) or name option. Then type the pid for child process (or parent process) and click attach button.
Enjoy Debugging!
Suppose I have a generic binary without source and I want to determine whether it is running serially or spawns multiple threads.
Is there a way I can do this from the linux command line?
First install strace.
$ yum install strace
Run the program with strace, and look for clone or fork system calls. Here's a quick example with a program I wrote that just calls fork and returns.
$ strace ./a.out
execve("./a.out", ["./a.out"], [/* 43 vars */]) = 0
brk(0) = 0x74f000
...
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7fb22b16da10) = 6567
exit_group(1) = ?
+++ exited with 1 +++
You can use ps for that. From man ps:
-L Show threads, possibly with LWP and NLWP columns.
So you can do:
$ ps -L <pid>
and it will show you something like this:
PID LWP TTY STAT TIME COMMAND
4112 4112 ? Sl 65:35 /usr/lib/firefox/firefox
4112 4116 ? Sl 0:04 /usr/lib/firefox/firefox
Each line of the output corresponds to one thread. This of course, only works for a certain
moment in time. To track the spawning of threads, use strace, as suggested by Jonathon Reinhart.
An alternative to strace is, of course, gdb. See this question for details on managing threads in gdb. You may also read the thread section of the gdb manual. Quick introduction:
$ gdb /usr/lib/firefox/firefox <pid>
[... initialization output ...]
> info threads # lists threads
> thread <nr> # switch to thread <nr>
Your comment:
How can I figure out where to set an instruction-level breakpoint if the program only takes seconds to run?
This answer might help you here, as it shows how to break on thread creation (with pthread_create) using gdb. So every time a thread is created, execution stops and you might investigate.
Just run: cat /proc/<pid>/stat | awk '{print $20}' to get the number of threads of a running process.
proc manpage
I am trying to run a simple MPI program on 4 nodes. I am using OpenMPI 1.4.3 running on Centos 5.5. When I submit the MPIRUN Command with the hostfile/machinefile, I get no output, receive a blank screen. Hence, I have to kill the job. .
I use the following run command: : mpirun --hostfile hostfile -np 4 new46
OUTPUT ON KILLING JOB:
mpirun: killing job...
--------------------------------------------------------------------------
mpirun noticed that the job aborted, but has no info as to the process that caused
that situation.
--------------------------------------------------------------------------
mpirun was unable to cleanly terminate the daemons on the nodes shown
below. Additional manual cleanup may be required - please refer to
the "orte-clean" tool for assistance.
--------------------------------------------------------------------------
myocyte46 - daemon did not report back when launched
myocyte47 - daemon did not report back when launched
myocyte49 - daemon did not report back when launched
Here is the MPI program I am trying to execute on 4 nodes
**************************
if (my_rank != 0)
{
sprintf(message, "Greetings from the process %d!", my_rank);
dest = 0;
MPI_Send(message, strlen(message)+1, MPI_CHAR, dest, tag, MPI_COMM_WORLD);
}
else
{
for (source = 1;source < p; source++)
{
MPI_Recv(message, 100, MPI_CHAR, source, tag, MPI_COMM_WORLD, &status);
printf("%s\n", message);
}
****************************
My hostfile looks like this:
[amohan#myocyte48 ~]$ cat hostfile
myocyte46
myocyte47
myocyte48
myocyte49
*******************************
I ran the above MPI program independently on each of the nodes and it compiled and ran just fine. I have this issue of "Daemon did not report back when launched" when I use the hostfile. I am trying to figure out what could be the issue.
Thanks!
I think these lines
myocyte46 - daemon did not report back when launched
are pretty clear -- you're having trouble either launching the mpi daemons or communicating with them afterwards. So you need to start looking at networking. Can you ssh without password into these nodes? Can you ssh back? Leaving aside the MPI program, can you
mpirun -np 4 hostname
and get anything?