C MPI - Crash for no apparent reason - Program Received signal SIGSEGV - c

Question

I am running a program in a linux cluster using MPICH2 1.0.6 (i can't actually update it to MPICH3, so i am stuck to that version) and for no apparent reason the program doesn't execute.
I compile it using mpicc -o prog prog.c -lm and execute with mpiexec
The program is an implementation of the hierarchical agglomerative clustering algorithm using the vector space model. The data collection is an n*m array (in the program DOC*MAXWORDS) which are divided into the nodes of the cluster like PARTS=DOC/procs so every node is responsible for PARTS*MAXWORDS data.
While debugging in a serial machine using gdb and ddd i get that the program has a segmentation error in a specific line of the code, where i can't find what is wrong with it. Take a look.
while(iterations != DOC-k){//bigLoop
iterations++;
x=y=-1;
pos1=pos2=node1=node2=-1;
for(i=0;i<PARTS;i++){//ELEGXOS MEGISTOU TOPIKA
if(max_array[i]>x){
x=max_array[i];
pos1=i;
}
else if(max_array[i]==x){
pos2=i;
} //ELEGXOS META TO LOOP GIA OMOIOTHTES
}
if(max_array[pos1]!=max_array[pos2]){
for(i=0;i<PARTS;i++){
if(max_array[i]>max_array[pos2] && i!=pos1)
pos2=1;
}
}
if(MPI_Allgather(&x,1,MPI_DOUBLE,
n_max,1,MPI_DOUBLE,MPI_COMM_WORLD) != MPI_SUCCESS) {
printf("Allgather high valuer - error");
exit(1);
}
for(i=0;i<procs;i++){
if(n_max[i]>y){
y=n_max[i];
node1=i;
}
else if(n_max[i]==y){
node2=i;
}
}
for(i=0;i<MAXWORDS;i++){
merger_one[i]=merger_two[i]=0;
}
if(n_max[node1]==n_max[node2]){
if(id==node1){
for(i=0;i<MAXWORDS;i++){
merger_one[i]=vector[node1*PARTS+pos1][i];
last_one[i]=vector[(node1*PARTS)+texts_vectors[node1]][i];
}
size_one=size_of[pos1];
nn_array[pos1]=nn_array[texts_vectors[node1]];
max_array[pos1]=max_array[texts_vectors[node1]];
size_of[pos1]=size_of[texts_vectors[node1]];
texts_vectors[node1]--;
}
if(id==node2){
for(i=0;i<MAXWORDS;i++){
merger_two[i]=vector[node2*PARTS+pos2][i];
last_two[i]=vector[(node2*PARTS)+texts_vectors[node2]][i];
}
j=pos2;
pos2=pos1;
pos1=j;
size_two=size_of[pos2];
nn_array[pos2]=nn_array[texts_vectors[node2]];
max_array[pos2]=max_array[texts_vectors[node2]];
size_of[pos2]=size_of[texts_vectors[node2]];
texts_vectors[node2]--;
}
}
else{
node2=node1;
if(id==node1){
for(i=0;i<MAXWORDS;i++){
merger_one[i]=vector[node1*PARTS+pos1][i];
merger_two[i]=vector[node2*PARTS+pos2][i];
last_one[i]=vector[(node1*PARTS)+texts_vectors[node1]][i];/*SIGSEV ERROR*/
last_two[i]=vector[(node2*PARTS)+texts_vectors[node2]-1][i];
}
size_one=size_of[pos1];
size_two=size_of[pos2];
nn_array[pos1]=nn_array[texts_vectors[node1]];
max_array[pos1]=max_array[texts_vectors[node1]];
size_of[pos1]=size_of[texts_vectors[node1]];
nn_array[pos2]=nn_array[texts_vectors[node2]-1];
max_array[pos2]=max_array[texts_vectors[node2]-1];
size_of[pos2]=size_of[texts_vectors[node2]-1];
texts_vectors[node1]=texts_vectors[node1]-2;
}
}
MPI_Bcast(&pos1, 1, MPI_INT,node1, MPI_COMM_WORLD);
MPI_Bcast(&pos2, 1, MPI_INT,node2, MPI_COMM_WORLD);
MPI_Bcast(&size_one, 1, MPI_INT,node1, MPI_COMM_WORLD);
MPI_Bcast(&size_two, 1, MPI_INT,node2, MPI_COMM_WORLD);
MPI_Bcast(merger_one, MAXWORDS, MPI_INT,node1, MPI_COMM_WORLD);
MPI_Bcast(merger_two, MAXWORDS, MPI_INT,node2, MPI_COMM_WORLD);
MPI_Bcast(last_one, MAXWORDS, MPI_INT,node1, MPI_COMM_WORLD);
MPI_Bcast(last_two, MAXWORDS, MPI_INT,node2, MPI_COMM_WORLD);
MPI_Allgather(&texts_vectors,1,MPI_INT,texts_vectors,1,MPI_INT,MPI_COMM_WORLD);
for(i=0;i<MAXWORDS;i++){
vector[node1*PARTS+pos1][i]=last_one[i];
vector[node2*PARTS+pos2][i]=last_two[i];
}
Pmanager=PARTS+1;
for(i=0;i<procs;i++){
if(texts_vectors[i]<Pmanager)
Pmanager=i;
}
texts_vectors[Pmanager]++;
for(i=0;i<MAXWORDS;i++){
x=merger_one[i]*size_one;
y=merger_two[i]*size_two;
vector[Pmanager*PARTS+texts_vectors[Pmanager]][i]=(x+y)/(size_one + size_two);
}
for(i=id*PARTS; i< (id+1)*texts_vectors[id]; i++){
for(j=0;j<procs;j++){
for(m=j*PARTS;m<j*PARTS+texts_vectors[j];m++){
x=0;y=0;z=0;
for(l=0; l < MAXWORDS; l++){
x+=vector[i][l]*vector[m][l];
y+=vector[i][l]*vector[i][l];
z+=vector[m][l]*vector[m][l];
}
if(i!=m){
if(y!=0 && z!=0){
sim_matrix[i-(PARTS*id)][m] = x / (sqrt(y) * sqrt(z) );
}
else{
sim_matrix[i-(PARTS*id)][m] = 0.0;
}
}
}
}
}
for(i=0; i<texts_vectors[id]; i++){
x=0.0;
for(j=0;j<DOC;j++){
if(sim_matrix[i][j]>x){
nn_array[i]=j;
max_array[i]=x=sim_matrix[i][j];
}
}
}
}
Prior to this there is the creation of the arrays and the data input into the vector[i][j]
I created the arrays using malloc :
int **vector = malloc(DOC * sizeof *vector);
for (i = 0; i < DOC; i++){
vector[i] = malloc(MAXWORDS * sizeof **vector);
}
double **sim_matrix = malloc(PARTS * sizeof *sim_matrix);
for (i = 0; i < PARTS; i++)
sim_matrix[i] = malloc(DOC * sizeof **sim_matrix);
int *list = malloc(WHOLE * sizeof(int));
int *nn_array = malloc(PARTS * sizeof(int));
double *max_array = malloc(PARTS * sizeof(double));
int *size_of = malloc(PARTS * sizeof(int));
double *n_max = malloc(procs * sizeof(double));
int *texts_vectors = malloc(procs * sizeof(int));
int *merger_one = malloc(MAXWORDS * sizeof(int));
int *merger_two = malloc(MAXWORDS * sizeof(int));
int *last_one = malloc(MAXWORDS * sizeof(int));
int *last_two = malloc(MAXWORDS * sizeof(int));
The line where the problem persists: last_one[i]=vector[(node1*PARTS)+texts_vectors[node1]][i];/*SIGSEV ERROR*/ is also executed in the first part of the if-loop if(n_max[node1]==n_max[node2]){ but in that case there is no error.
The only thing that feels a little suspicious about this problem is the texts_vectors[i] array which keeps counting the number of vector[i][j] type data that are currently inside the nodes. But even that i think i got it covered.
I really hope that somebody could have a look at this cause it is really frustrating and it needs to be done.
If you have a better idea of what is going on and want to take a look at the whole code, i pasted it into a pastezone. Cheers and thanks in advance.
EDIT:
As it turns out the value that i passed with the array text_vectors where exceeding the boundaries of the array. Since the value was giving the maximum value, for the actual last position in the array i should subtract 1. So that was it, no segmentation fault in the serial gdb and ddd. However this program now it doesn't run for more than 2 nodes. If i execute it in 4> nodes it crashes.

Answer 1

This line has multiple errors in it:
MPI_Allgather(&texts_vectors,1,MPI_INT,texts_vectors,1,MPI_INT,MPI_COMM_WORLD);
First, you are providing a pointer to a pointer to the data as the first argument of the gather-to-all operation. Therefore the value transmitted by each rank is not the first element of text_vectors but rather the memory address of the data (or the lower half of the address on 64-bit little-endian LP64 systems).
Second, if you fix that by removing the address-off operator & from the beginning of the first argument, you will run into another problem. The MPI standard does not allow the source and destination buffers in MPI_Allgather to overlap. Some MPI implementations do not enforce that requirement and silently do The Right Thing (TM). Some other MPI implementations will try to copy the data with memcpy and run into problems with the C library (memcpy does not allow overlapping buffers). And finally, some MPI implementations will give you a nice error message about overlapping buffers and terminate your program.
Since you are sending a single integer element, simply copy the value into a temporary variable and use its address as the first argument.