Segfault when running openmpi - c

Question

I am currently working on a project where I need to implement a parallel fft algorithm using openmpi. I have a compiling piece of code, but when I run it over the cluster I get segmentation faults.
I have my hunches about where things are going wrong, but I don't think I have enough of an understanding about pointers and references to be able to make a efficient fix.
The first chunk that could be going wrong is in the passing of the arrays to the helper functions. I believe that either my looping is inconsistent, or I am not understanding how the to pass these pointers and get back the things I need.
The second possible spot would be within the actual mpi_Send/Recv commands. I am sending a type that is not supported by the openmpi c datatypes, so I am using the mpi_byte type to send the raw data instead. Is this a viable option? Or should I be looking into an alternative to this method.
/* function declarations */
double complex get_block(double complex c[], int start, int stop);
double complex put_block(double complex from[], double complex to[],
int start, int stop);
void main(int argc, char **argv)
{
/* Initialize MPI */
MPI_Init(&argc, &argv);
double complex c[N/p];
int myid;
MPI_Comm_rank(MPI_COMM_WORLD, &myid);
//printf("My id is %d\n",myid);
MPI_Status status;
int i;
for(i=0;i<N/p;i++){
c[i] = 1.0 + 1.0*I;
}
int j = log(p)/log(2) + 1;
double q;
double complex z;
double complex w = exp(-2*PI*I/N);
double complex block[N/(2*p)]; // half the size of chunk c
int e,l,t,k,m,rank,plus,minus;
int temp = (log(N)-log(p))/log(2);
//printf("temp = %d", temp);
for(e = 0; e < (log(p)/log(2)); e++){
/* loop constants */
t = pow(2,e); l = pow(2,e+temp);
q = n/2*l; z = cpow(w,(complex)q);
j = j-1; int v = pow(2,j);
if(e != 0){
plus = (myid + p/v)%p;
minus = (myid - p/v)%p;
} else {
plus = myid + p/v;
minus = myid - p/v;
}
if(myid%t == myid%(2*t)){
MPI_Recv((char*)&c,
sizeof(c),
MPI_BYTE,
plus,
MPI_ANY_TAG,
MPI_COMM_WORLD,
&status);
/* transform */
for(k = 0; k < N/p; k++){
m = (myid * N/p + k)%l;
c[k] = c[k] + c[k+N/v] * cpow(z,m);
c[k+N/v] = c[k] - c[k + N/v] * cpow(z,m);
printf("(k,k+N/v) = (%d,%d)\n",k,k+N/v);
}*/
printf("\n\n");
/* end transform */
*block = get_block(c, N/v, N/v + N/p + 1);
MPI_Send((char*)&block,
sizeof(block),
MPI_BYTE,
plus,
1,
MPI_COMM_WORLD);
} else {
// send data of this PE to the (i- p/v)th PE
MPI_Send((char*)&c,
sizeof(c),
MPI_BYTE,
minus,
1,
MPI_COMM_WORLD);
// after the transformation, receive data from (i-p/v)th PE
// and store them in c:
MPI_Recv((char*)&block,
sizeof(block),
MPI_BYTE,
minus,
MPI_ANY_TAG,
MPI_COMM_WORLD,
&status);
*c = put_block(block, c, N/v, N/v + N/p - 1);
//printf("Process %d send/receive %d\n",myid, plus);
}
}
/* shut down MPI */
MPI_Finalize();
}
/* helper functions */
double complex get_block(double complex *c, int start, int stop)
{
double complex block[stop - start + 1];
//printf("%d = %d\n",sizeof(block)/sizeof(double complex), sizeof(&c)/sizeof(double complex));
int j = 0;
int i;
for(i = start; i < stop+1; i++){
block[j] = c[i];
j = j+1;
}
return *block;
}
double complex put_block(double complex from[], double complex to[], int start, int stop)
{
int j = 0;
int i;
for(i = start; i<stop+1; i++){
to[i] = from[j];
j = j+1;
}
return *to;
}
I really appreciate the feedback!

Answer 1

You are using arrays / pointers to arrays in the wrong way. For example you declare an array as double complex block[N], which is fine (although uncommon, in most cases it is better to use malloc) and then you receive into it via MPI_Recv(&block). However "block" is already a pointer to that array, so by writing "&block" you are passing the pointer of the pointer to MPI_Recv. That's not what it expects. If you want to use the "&" notation you have to write &block[0], which would give you the pointer to the first element of the block-array.

Answer 2

Have you tried debugging your code? This can be a pain in a parallel setting, but it can tell you exactly where it is failing and usually also why.
If you're using Linux or OS X, you could run your code as follows on the command line:
mpirun -np 4 xterm -e gdb -ex run --args ./yourprog yourargs
where I'm assuming yourprog is the name of your program and yourargs are any command-line arguments you want to pass.
What this command will do is launch four xterm windows. Each xterm will in turn launch gdb as specified by the option -e. gdb will then execute the command run as specified by the option -ex and launch your executable with the given options, as specified by --args.
What you get are four xterm windows running four instances of your program in parallel with MPI. If any of the instances crashes, gdb will tell you where and why.