MPI Error: (Segmentation fault: 11) - c

I have seen many similar threads here, but the problem with me, is, that my program actually runs for different settings.
For example: when my matrix is 1024x1024
For 2 cores : Error 11
For 4, 8, 16 etc works fine.
Matrix 2048x2048:
For any core setting : Error 11.
I don't understand why this happens, each process is taking a 2048/(total processes) X 2028 matrix to calculate. And it should be working correctly.
This is how i declare my matrix:
int temp[wp][hp];
For receive:
rc = MPI_Recv(&temp[0][0], wp*hp, MPI_INT, i, tag, MPI_COMM_WORLD, &status);
And for send:
rc = MPI_Send(&temp[0][0], wp*hp, MPI_INT, 0, tag, MPI_COMM_WORLD);
I don't get it, it should be working. Do you think it is perhaps a memory issue and not pointer related?

I would create the array with malloc
int *temp =(int*)malloc(wp * hp * sizeof(int));
then I would change the other lines to
rc = MPI_Recv(temp, wp*hp, MPI_INT, i, tag, MPI_COMM_WORLD, &status);
and
rc = MPI_Send(temp, wp*hp, MPI_INT, 0, tag, MPI_COMM_WORLD);
and when I'm done with the array, I free it.
free(temp)
Like one of the commenters already stated, allocating the array your way is not legal c++.
Edit:
if you want to access to array twodimensional, here is the pattern for that:
temp[rowToAccess * numberOfColumns + columnToAcess]

The answer given is basically a rundown of the issue -- you are more than likely using up the stack space to store the array, and thus exhausting the memory reserved for the stack.
Thus the solution is to create the array dynamically.
Here is an example of creating a 2D array dynamically:
#include <stdlib.h>
//...
int** create2DArray(unsigned nrows, unsigned ncols)
{
unsigned i;
int** ptr = malloc(nrows * sizeof(int*)); // allocate pointers to rows
if ( ptr != NULL )
{
int* pool = malloc(nrows * ncols * sizeof(int)); // allocate pool of memory
if ( pool != NULL )
{
for (i = 0; i < nrows; ++i, pool += ncols )
ptr[i] = pool; // point the row pointers into the pool
}
else
{
free(ptr);
ptr = NULL;
}
}
return ptr;
}
void delete2DArray(int** arr)
{
free(arr[0]); // remove the pool
free(arr); // remove the pointers
}
int main()
{
int **temp = create2DArray(2048, 2048);
if ( temp != NULL )
{
temp[0][0] = 10; // for example you can use it like a 2D array
delete2DArray(temp); // free the memory
}
}
This will create in essence, a contiguous 2D array, similar to the one you attempted to create using int temp[2048][2048], but this time, the memory is obtained from the heap, not the stack.
Note you can use the [][] syntax, just like a 2 dimensional array.
I won't go into detail to how it works, however it is simple enough to follow the logic.

So these were the problems:
1) As indicated by #PaulMcKenzie and #Alex, I had to dynamically allocate memory. Now the problem was that If i used
rc = MPI_Send(temp, wp*hp, MPI_INT, 0, tag, MPI_COMM_WORLD);
or
rc = MPI_Recv(temp, wp*hp, MPI_INT, i, tag, MPI_COMM_WORLD, &status);
as #Alex suggested, then, my program would crash for some reason. So my initial &temp[0][0] was correct.
2) The second problem, was, that I was saving this array using fwrite but when u have dynamical arrays you need to do it like this:
for (int i = 0; i < w; i++){
for (int j = 0; j < h; j++){
fwrite(&im[i][j], sizeof(int), 1, yourfilename);
}
}
Cheers everyone

Related

How to ensure a 2D array is allocated contigously in memory

I am sending a 2D array over MPI and for this to work correctly the array needs to be contiguously allocated in memory.
I am allocating it as follows:
int **array;
array = malloc(size1 * sizeof( *(array) );
for (int k = 0; k < size1; k++)
array[k] = malloc(size2 * sizeof(**(array));
Then I would like to use:
MPI_Send(array, size1*size2, MPI_INT, target_pe, tag, MPI_COMM_WORLD);
How can I ensure the array is allocated contiguously?
Currently I am trying this:
for (int k = 0; k < size1; k++)
MPI_Send(array[k], size2, MPI_INT, target_pe, tag, MPI_COMM_WORLD);
which leads to a seg fault later in an unrelated part of the program. However if I send the elements 1 by 1 it works.
How to ensure a 2D array is allocated contiguously in memory
Allocate in one step.
Example uses C99 code which supports variable length arrays. Here a pointer to a VLA is used.
// ptr is a pointer to a 2D array
int (*ptr)[size1][size2] = malloc(sizeof *ptr);
(*ptr)[0][0] = 1; // One corner of the 2D array
(*ptr)[size-1][size-1] = 2; // Opposite corner of the 2D array
I'll look into sample MPI() code later.
Use:
int (*array)[size2] = malloc(size1 * sizeof *array);
if (!array) Handle error…
If size2 is not a constant and your C implementation does not support variable length arrays, then use:
int *array = malloc(size1 * size2 * sizeof *array);
if (!array) Handle error…
In this case, you will have to use manual address arithmetic to access array elements. Instead of array[i][j], use array[i*size2 + j].
You can not ensure that by using sequential calls to malloc(). What you can do instead, you can allocate from the beginning malloc(size1 * size2) space in the memory, then you can split your allocated buffer how you desire. You can now say that array[i] is in fact array[i * size2] in the memory.
The correct way is to allocate the memory for the 2D array in on single operation. Then as arrays are not really first class citizens in C, not speaking of multi-dimensional arrays, you will have to allocate memory for an array of pointers and make its elements points to the beginning of each row:
int *mem = malloc(sizeof(*mem) * size1 * size2);
int **array = malloc(size1 * sizeof(*array));
for (int k = 0; k < size1; k++) {
array[k] = mem + k * size2 * sizeof(*mem);
}
This will ensure contiguous allocation when it matters for passing the array to routine expecting that, and still allow to access the elements as array[i][j];

Issue Creating and Freeing C Dynamic Arrays

I am writing a C program that involves passing 2D arrays between functions changing their size and entries. I decided to use dynamic arrays with pointers for this.
Whenever I free the pointers to an array, I find that I wipe the values held in other arrays. I can successfully change which array a pointer points to. I believe this is an issue with the way I'm freeing my pointers or declaring them. Below is code I'm using to create and free pointers to my arrays.
int** create_array(int m, int n)
{
int i;
int* values = calloc(m * n, sizeof(int));
int** rows = malloc(n * sizeof(int*));
for(i = 0; i < n; i++) {
rows[i] = values + (i * m);
}
return rows;
}
void destroy_array(int** arr)
{
free(arr[0]);
free(arr);
}
OLD CODE to Create and Free Pointers
int** create_array(int m, int n)
{
int i;
int* values = calloc(m * n, sizeof(int));
int** rows = malloc(n * sizeof(int*));
for(i = 0; i < n; i++) {
rows[i] = values + (i * m * sizeof(int));
}
return rows;
}
void destroy_array(int** arr, int m, int n)
{
int i;
for(i = 0; i < n; i++) {
free(arr[i]);
}
free(arr);
}
My program gets a segfault after I destroy the pointers to an array and try to read values from another array. Below is the code where I destroy my pointers to these arrays. positions_last and positions are both arrays that I can read from properly before this point.
positions_last = positions;
printf("- %d %d %d - ", positions_last[0][1], positions_last[1][1], positions_last[2][1]);
fflush(stdout); // this prints fine
destroy_array(positions);
printf("- %d %d %d - ", positions_last[0][1], positions_last[1][1], positions_last[2][1]);
fflush(stdout); // this does not print, I get a segfault at this point
I just did an elementary test which suggests that the issue lies in my current code for creating or destroying arrays (so far as I know).
int** positions2 = create_array(10, 3);
int** positions3 = create_array(10, 3);
printf("%d %d %d", positions3[0][1], positions3[1][1], positions3[2][1]);
fflush(stdout); // This line prints fine
positions3 = positions2;
destroy_array(positions2);
printf("%d %d %d", positions3[0][1], positions3[1][1], positions3[2][1]);
fflush(stdout); // This line triggers a segfault
Anyone have an idea what the issue may be?
You called calloc once and malloc once, but then you're calling free n+1 times (and of course you're freeing the same value, arr[1] n times). There should be exactly one free for each malloc or calloc.
void destroy_array(int** arr)
{
free(arr[0]);
free(arr);
}
This line
rows[i] = values + (i * m * sizeof(int));
should be
rows[i] = values + (i * m);
The reason behind this is that values is a typed pointer, namely pointing to int. Adding 1 to it increases it by 1 * sizeof (int). Your code assumes it would be increased by 1 only.
This are basic pointer arithmetics: http://cslibrary.stanford.edu/104/ ;-)
So you are running into undefined behaviuor even before the 1st call to free().
Also malloc/calloc and free follow the pattern:
One Allocation --> One Freeing
So freeing might look like this:
free(*arr); /* Free what had been allocated to "values". */
free(arr); /* Free what had been allocated to "rows". */
The code you show does differently, as pointed out by zindorsky's answer.
Regarding your edit:
This
positions_last = positions;
does not copy the array, its elements, but just the reference to the array 1st member. SO if you deallocate positions also positions_last points to freed, that is then invalif memory. Accessing it provokes UB, as this lines does:
printf("- %d %d %d - ", positions_last[0][1], positions_last[1][1], positions_last[2][1]);
Lesson learned: In C one cannot copy an array by a simple assignment

double free() error after swapping rows of a matrix

I'm getting the double free or corruption error upon executing my code. Essentially, I am just creating a matrix in C (of any RxC dimension, which is why pointers are used), swapping two rows, printing the results, and then attempting to free the memory. When I do not swap the rows, the freeing works perfectly. When I do, it crashes. I've attempted to alter the way the swapping works to no avail. I think it has something to do with the temporary pointer for swapping going out of scope, but I'm not sure if this is the problem and how I would fix it.
MatElement is just a double.
typedef double MatElement;
main:
int main(int argc, char *argv[]) {
MatElement** matrix = matrixAlloc(3,3);
int i;
int j;
for(i = 0; i < 3; i++) {
for(j = 0; j < 3; j++) {
matrix[i][j] = i+j;
}
}
matrixPrint(matrix, "%5.1f", 3, 3);
swapRows(matrix, 0, 2);
matrixPrint(matrix, "%5.1f", 3, 3);
matrixFree(matrix);
return 0;
}
The way the matrices are allocated:
MatElement **matrixAlloc(int nr, int nc) {
int i;
MatElement *ptr;
MatElement **A;
A = malloc(nr * sizeof(MatElement *)); /* array of ptrs */
ptr = calloc(nr * nc, sizeof(MatElement)); /* matrix elements */
for (i = 0; i < nr; i++) /* set row pointers properly */
A[i] = ptr + nc * i;
return A;
}
The way they are freed:
void matrixFree(MatElement **A) {
free(A[0]);
free(A);
}
The way their rows are swapped:
void swapRows(MatElement** G, int pivotRow, int rowExamined) {
MatElement* temp;
temp = G[rowExamined];
G[rowExamined] = G[pivotRow];
G[pivotRow] = temp;
}
Does anyone have any idea about what would be causing this double free() / invalid free()?
At some point, you are swapping the first row of the matrix into another position, so the free(A[0]) in matrixFree() is attempting to free a pointer into the middle of the array, instead of the pointer returned by calloc(). You will need to save that original pointer somewhere so that you can pass it, unmolested, to free().
Your matrix looks like this way :
A has all the pointers to the first element of all the rows. So initially, A[0] points to 0th row, A[1] to 1st row and so on.
So, when you are trying to free the matrix, you know that A[0] points to first row. But at that time you are assuming that your matrix elements are in continuous position and A[0] will always point to initial pointer returned by calloc. But when you swap some rows (specially the 0th with any of other row), then A[0] doesnt point to pointer returned by calloc.
possible solution involves like allocating one more memory slot in A like :
A = malloc( ((nr+1) * sizeof(MatElement )); / array of ptrs */
and then storing original pointer returned by calloc to A[nr].
So, A[nr] will also point to calloc returned pointer.

Segmentation fault after scatterv

/**
* BLOCK_LOW
* Returns the offset of a local array
* with regards to block decomposition
* of a global array.
*
* #param (int) process rank
* #param (int) total number of processes
* #param (int) size of global array
* #return (int) offset of local array in global array
*/
#define BLOCK_LOW(id, p, n) ((id)*(n)/(p))
/**
* BLOCK_HIGH
* Returns the index immediately after the
* end of a local array with regards to
* block decomposition of a global array.
*
* #param (int) process rank
* #param (int) total number of processes
* #param (int) size of global array
* #return (int) offset after end of local array
*/
#define BLOCK_HIGH(id, p, n) (BLOCK_LOW((id)+1, (p), (n)))
/**
* BLOCK_SIZE
* Returns the size of a local array
* with regards to block decomposition
* of a global array.
*
* #param (int) process rank
* #param (int) total number of processes
* #param (int) size of global array
* #return (int) size of local array
*/
#define BLOCK_SIZE(id, p, n) ((BLOCK_HIGH((id), (p), (n))) - (BLOCK_LOW((id), (p), (n))))
/**
* BLOCK_OWNER
* Returns the rank of the process that
* handles a certain local array with
* regards to block decomposition of a
* global array.
*
* #param (int) index in global array
* #param (int) total number of processes
* #param (int) size of global array
* #return (int) rank of process that handles index
*/
#define BLOCK_OWNER(i, p, n) (((p)*((i)+1)-1)/(n))
/*Matricefilenames:
small matrix A.bin of dimension 100 × 50
small matrix B.bin of dimension 50 × 100
large matrix A.bin of dimension 1000 × 500
large matrix B.bin of dimension 500 × 1000
An MPI program should be implemented such that it can
• accept two file names at run-time,
• let process 0 read the A and B matrices from the two data files,
• let process 0 distribute the pieces of A and B to all the other processes,
• involve all the processes to carry out the the chosen parallel algorithm
for matrix multiplication C = A * B ,
• let process 0 gather, from all the other processes, the different pieces
of C ,
• let process 0 write out the entire C matrix to a data file.
*/
#include <stdio.h>
#include <stdlib.h>
#include <mpi.h>
#include "mpi-utils.c"
void read_matrix_binaryformat (char*, double***, int*, int*);
void write_matrix_binaryformat (char*, double**, int, int);
void create_matrix (double***,int,int);
void matrix_multiplication (double ***, double ***, double ***,int,int, int);
int main(int argc, char *argv[]) {
int id,p; // Process rank and total amount of processes
int rowsA, colsA, rowsB, colsB; // Matrix dimensions
double **A; // Matrix A
double **B; // Matrix B
double **C; // Result matrix C : AB
int local_rows; // Local row dimension of the matrix A
double **local_A; // The local A matrix
double **local_C; // The local C matrix
MPI_Init (&argc, &argv);
MPI_Comm_rank (MPI_COMM_WORLD, &id);
MPI_Comm_size (MPI_COMM_WORLD, &p);
if(argc != 3) {
if(id == 0) {
printf("Usage:\n>> %s matrix_A matrix_B\n",argv[0]);
}
MPI_Finalize();
exit(1);
}
if (id == 0) {
read_matrix_binaryformat (argv[1], &A, &rowsA, &colsA);
read_matrix_binaryformat (argv[2], &B, &rowsB, &colsB);
}
if (p == 1) {
create_matrix(&C,rowsA,colsB);
matrix_multiplication (&A,&B,&C,rowsA,colsB,colsA);
char* filename = "matrix_C.bin";
write_matrix_binaryformat (filename, C, rowsA, colsB);
free(A);
free(B);
free(C);
MPI_Finalize();
return 0;
}
// For this assignment we have chosen to bcast the whole matrix B:
MPI_Bcast (&B, 1, MPI_DOUBLE, 0, MPI_COMM_WORLD);
MPI_Bcast (&colsA, 1, MPI_INT, 0, MPI_COMM_WORLD);
MPI_Bcast (&colsB, 1, MPI_INT, 0, MPI_COMM_WORLD);
MPI_Bcast (&rowsA, 1, MPI_INT, 0, MPI_COMM_WORLD);
MPI_Bcast (&rowsB, 1, MPI_INT, 0, MPI_COMM_WORLD);
local_rows = BLOCK_SIZE(id, p, rowsA);
/* SCATTER VALUES */
int *proc_elements = (int*)malloc(p*sizeof(int)); // amount of elements for each processor
int *displace = (int*)malloc(p*sizeof(int)); // displacement of elements for each processor
int i;
for (i = 0; i<p; i++) {
proc_elements[i] = BLOCK_SIZE(i, p, rowsA)*colsA;
displace[i] = BLOCK_LOW(i, p, rowsA)*colsA;
}
create_matrix(&local_A,local_rows,colsA);
MPI_Scatterv(&A[0],&proc_elements[0],&displace[0],MPI_DOUBLE,&local_A[0],
local_rows*colsA,MPI_DOUBLE,0,MPI_COMM_WORLD);
/* END SCATTER VALUES */
create_matrix (&local_C,local_rows,colsB);
matrix_multiplication (&local_A,&B,&local_C,local_rows,colsB,colsA);
/* GATHER VALUES */
MPI_Gatherv(&local_C[0], rowsA*colsB, MPI_DOUBLE,&C[0],
&proc_elements[0],&displace[0],MPI_DOUBLE,0, MPI_COMM_WORLD);
/* END GATHER VALUES */
char* filename = "matrix_C.bin";
write_matrix_binaryformat (filename, C, rowsA, colsB);
free (proc_elements);
free (displace);
free (local_A);
free (local_C);
free (A);
free (B);
free (C);
MPI_Finalize ();
return 0;
}
void create_matrix (double ***C,int rows,int cols) {
*C = (double**)malloc(rows*sizeof(double*));
(*C)[0] = (double*)malloc(rows*cols*sizeof(double));
int i;
for (i=1; i<rows; i++)
(*C)[i] = (*C)[i-1] + cols;
}
void matrix_multiplication (double ***A, double ***B, double ***C, int rowsC,int colsC,int colsA) {
double sum;
int i,j,k;
for (i = 0; i < rowsC; i++) {
for (j = 0; j < colsC; j++) {
sum = 0.0;
for (k = 0; k < colsA; k++) {
sum = sum + (*A)[i][k]*(*B)[k][j];
}
(*C)[i][j] = sum;
}
}
}
/* Reads a 2D array from a binary file*/
void read_matrix_binaryformat (char* filename, double*** matrix, int* num_rows, int* num_cols) {
int i;
FILE* fp = fopen (filename,"rb");
fread (num_rows, sizeof(int), 1, fp);
fread (num_cols, sizeof(int), 1, fp);
/* storage allocation of the matrix */
*matrix = (double**)malloc((*num_rows)*sizeof(double*));
(*matrix)[0] = (double*)malloc((*num_rows)*(*num_cols)*sizeof(double));
for (i=1; i<(*num_rows); i++)
(*matrix)[i] = (*matrix)[i-1]+(*num_cols);
/* read in the entire matrix */
fread ((*matrix)[0], sizeof(double), (*num_rows)*(*num_cols), fp);
fclose (fp);
}
/* Writes a 2D array in a binary file */
void write_matrix_binaryformat (char* filename, double** matrix, int num_rows, int num_cols) {
FILE *fp = fopen (filename,"wb");
fwrite (&num_rows, sizeof(int), 1, fp);
fwrite (&num_cols, sizeof(int), 1, fp);
fwrite (matrix[0], sizeof(double), num_rows*num_cols, fp);
fclose (fp);
}
My task is to do a parallel matrix multiplication of matrix A and B and gather the results in matrix C.
I am doing this by dividing matrix A in rowwise pieces and each process is going to use its piece to multiply matrix B, and get back its piece from the multiplication. Then I am going to gather all the pieces from the processes and put them together to matrix C.
I allready posted a similiar question, but this code is improved and I have progressed but I am still getting a segmentation fault after the scatterv call.
So I see a few problems right away:
MPI_Bcast (&B, 1, MPI_DOUBLE, 0, MPI_COMM_WORLD);
Here, you're passing not a pointer to doubles, but a pointer to a pointer to a pointer to a double (B is defined as double **B) and you're telling MPI to follow that pointer and send 1 double from there. That is not going to work.
You might think that what you're accomplishing here is sending the pointer to the matrix, from which all tasks can read the array -- that doesn't work. The processes don't share a common memory space (that's why MPI is called distributed memory programming) and the pointer doesn't go anywhere. You're actually going to have to send the contents of the matrix,
MPI_Bcast (&(B[0][0]), rowsB*colsB, MPI_DOUBLE, 0, MPI_COMM_WORLD);
and you're going to have to make sure the other processes have correctly allocated memory for the B matrix ahead of time.
There's similar pointer problems elsewhere:
MPI_Scatterv(&A[0], ..., &local_A[0]
Again, A is a pointer to a pointer to doubles (double **A) as is local_A, and you need to be pointing MPI to pointer to doubles for this to work, something like
MPI_Scatterv(&(A[0][0]), ..., &(local_A[0][0])
that error seems to be present in all the communications routines.
Remember that anything that looks like (buffer, count, TYPE) in MPI means that the MPI routines follow the pointer buffer and send the next count pieces of data of type TYPE there. MPI can't follow pointers within the buffer you sent becaue in general it doens't know they're there. It just takes the next (count * sizeof(TYPE)) bytes from pointer buffer and does whatever communications is appropriate with them. So you have to pass it a pointer to a stream of data of type TYPE.
Having said all that, it would be a lot easier to work with you on this if you had narrowed things down a bit; right now the program you've posted includes a lot of I/O stuff that's irrelevant, and it means that no one can just run your program to see what happens without first figuring out the matrix format and then generating two matrices on their own. When posting a question about source code, you really want to post a (a) small bit of source which (b) reproduces the problem and (c) is completely self-contained.
Consider this an extended comment as Jonathan Dursi has already given a fairly elaborate answer. You matrices are really represented in a weird way but at least you followed the advice given to your other question and allocate space for them as contiguous blocks and not separately for each row.
Given that, you should replace:
MPI_Scatterv(&A[0],&proc_elements[0],&displace[0],MPI_DOUBLE,&local_A[0],
local_rows*colsA,MPI_DOUBLE,0,MPI_COMM_WORLD);
with
MPI_Scatterv(A[0],&proc_elements[0],&displace[0],MPI_DOUBLE,local_A[0],
local_rows*colsA,MPI_DOUBLE,0,MPI_COMM_WORLD);
A[0] already points to the beginning of the matrix data and there is no need to make a pointer to it. The same goes for local_A[0] as well as for the parameters to the MPI_Gatherv() call.
It has been said many times already - MPI doesn't do pointer chasing and only works with flat buffers.
I've also noticed another mistake in your code - memory for your matrices is not freed correctly. You are only freeing the array of pointers and not the matrix data itself:
free(A);
should really become
free(A[0]); free(A);

MPI_Bcast a dynamic 2d array of structs

I followed Jonathan's code from here ( MPI_Bcast a dynamic 2d array ) to MPI_Bcast a dynamically allocated 2d array of structs. The struct is as follows:
typedef struct {
char num[MAX];
int neg;
} bignum;
Since this is not a native datatype, I used MPI_Type_create_struct to commit this datatype.
I omit the code here because it works on another project I was doing.
The method I used to dynamically create ("contiguous?") the array is by calling:
bignum **matrix = creatematrix(DIMMAX,DIMMAX);
which is implemented as:
bignum ** creatematrix (int num_rows, int num_cols){
int i;
bignum *one_d = (bignum*)malloc (num_rows * num_cols * sizeof (bignum));
bignum **two_d = (bignum**)malloc (num_rows * sizeof (bignum *));
for (i = 0; i < num_rows; i++)
two_d[i] = &(one_d[i*num_cols]);//one_d + (i * num_cols);
return two_d;
}
Now I'd get input and store it inside matrix and call MPI_Bcast as:
MPI_Bcast(&(matrix[0][0]), DIMMAX*DIMMAX, Bignum, 0, MPI_COMM_WORLD);
There seems to be no segmentation fault, but the problem is that only the first row (matrix[0]) is broadcasted. All other ranks beside the root have data only for the first row. Other rows remain untouched.
It looks like there's something more than allocating a contiguous memory block. Is there anything I missed causing broadcast to be unsuccessful?
EDIT:
I fumbled on a weird workaround, using nested structs. Can anyone explain why this method works while the one above doesn't?
typedef struct {
char num[MAX];
int neg;
} bignum;
typedef struct {
bignum elements[DIMMAX];
} row;
int main(/*int argc,char *argv[]*/){
int dim, my_rank, comm_sz;
int i,j,k; //iterators
bignum global_sum = {{0},0};
row *matrix = (row*)malloc(sizeof(row)*DIMMAX);
row *tempmatrix = (row*)malloc(sizeof(row)*DIMMAX);
MPI_Init(NULL,NULL);
MPI_Comm_size(MPI_COMM_WORLD, &comm_sz);
MPI_Comm_rank(MPI_COMM_WORLD, &my_rank);
//CREATE DERIVED DATATYPE Bignum
MPI_Datatype Bignum;
MPI_Datatype type[4] = {MPI_LB, MPI_CHAR, MPI_INT, MPI_UB};
int blocklen[4] = {1,MAX,1,1};
MPI_Aint disp[4];
//get offsets
MPI_Get_address(&global_sum, disp);
MPI_Get_address(&global_sum.num, disp+1);
MPI_Get_address(&global_sum.neg, disp+2);
//MPI_Get_address(&local_sum+1, disp+3);
int base;
base = disp[0];
for (i=0;i<3;i++){
disp[i] -= base;
}
disp[3] = disp[2]+4; //int
MPI_Type_create_struct(4,blocklen,disp,type,&Bignum);
MPI_Type_commit(&Bignum);
//CREATE DERIVED DATATYPE BignumRow
MPI_Datatype BignumRow;
MPI_Datatype type2[3] = {MPI_LB, Bignum, MPI_UB};
int blocklen2[3] = {1,DIMMAX,1};
MPI_Aint disp2[3] = {0,0,DIMMAX*(MAX+4)};
MPI_Type_create_struct(3,blocklen2,disp2,type2,&BignumRow);
MPI_Type_commit(&BignumRow);
MPI_Bcast(&dim, 1, MPI_INTEGER, 0, MPI_COMM_WORLD);
MPI_Bcast(matrix, dim, BignumRow, 0, MPI_COMM_WORLD);
This seems to come up a lot: If you pass around multi-dimensional arrays with MPI in C, you sort of have to think of a multi-dimensional array as an array of arrays. I think in your first example you've gotten lucky that matrix[0][0] (due to the consecutive mallocs?) sits at the beginning of a large block of memory (and hence, no seg fault).
In the second (working) example, you've described the memory addresses of not just the beginning of the memory region (matrix[0][0]), but also all the internal pointers as well.

Resources