Related
I've been trying to learn MPI and I've this code snippet from C which should be formatted to MPI to make it parallizable;
// this is a dot product
int A[100000];
int B[100000];
int C=0;
for (int i=0 ; i<100000; i++){
c += A[i] * B[i];
}
Im kinda confused of how to deal with for-loops when using MPI but this is my answer to the exercise;
// this is a dot product
int A[100000];
int B[100000];
int C=0;
int rootId = 0;
int numtasks, taskId, len, partner, message;
MPI_Status status;
MPI_Init(&argc, &argv);
MPI_Comm_rank(MPI_COMM_WORLD, &taskid);
for (int i=0 ; i<100000; i++){
c = A[i] * B[i];
if(rootId == taskid){
MPI_Send(&A[100000], &B[100000], MPI_INT, 1, 0, MPP_COMM_WORLD, MPI_Status)
}else if (rootId < taskId){
MPI_Recv(C, 100000, MPI_INT, 0, 0, MPP_COMM_WORLD, MPI_Status)
}
MPI_Finalize(); return 0;
}
Im not sure if this even correct, but I believe im in the right direction...
The problem about MPI is that I can't really find good examples of how to rewrite or structe it with loops - There are some examples in fortran but this is nothing im familiar with...
I've only seen simple examples of "Hello world" C in MPI.... But nothing useful to get from there.
All help would be appreciated..
was there a typo and did you mean instead ?
for (int i=0 ; i<100000; i++){
c += A[i] * B[i];
}
parallelization heavily depends on how your arrays A and B are distributed.
the simplest case, which is also the one with the biggest memory footprint, is to have the full arrays A and B on all MPI tasks.
based on a task rank and the total number of tasks, each task can compute a part of the dot product e.g.
for (int i=start; i<end; i++) {
c += A[i] * B[i];
}
and then you can MPI_Reduce()/MPI_Allreduce() with MPI_SUM the partial dot products in order to get the final result.
My assignment is to parallelize some provided sequential code that takes an array of numbers and merges it with another array in order to come up with a sorted list. The first step is to initialize the array on each processor, then fill it with values on the root process. Next, I'm supposed to scatter the array out to the other processors so each processor has a chunk of the data for sorting, then sort these small lists locally and then gather them back up with the root process. My problem here is that no matter what I try the scatter won't actually send any data to the other processors. The array in these processors is always full of 0's, whereas the root processor does contain a list of numbers. Anybody care to take a look at my code and tell me what I'm missing?
The original sequential code
#include <mpi.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
/**
* Prints a vector without decimal places...
*/
void print_vector(int n, double vector[]) {
int i;
printf("[%.0f", vector[0]);
for (i = 1; i < n; i++) {
printf(", %.0f", vector[i]);
}
printf("]\n");
}
/**
* Just checks sequentially if everything is in ascending order.
*
* Note: we don't care about stability in this sort since we
* have no data attached to the double value, see
* http://en.wikipedia.org/wiki/Sorting_algorithm#Stability
* for more detail.
*/
void test_correctness(int n, double v[]) {
int i;
for (i = 1; i < n; i++) {
if (v[i] < v[i-1]) {
printf("Correctness test found error at %d: %.4f is not < %.4f but appears before it\n", i, v[i-1], v[i]);
}
}
}
/**
* Initialize random vector.
*
* You may not parallelize this (even though it could be done).
*/
void init_random_vector(int n, double v[]) {
int i, j;
for (i = 0; i < n; i++) {
v[i] = rand() % n;
}
}
double *R;
/**
* Merges two arrays, left and right, and leaves result in R
*/
void merge(double *left_array, double *right_array, int leftCount, int rightCount) {
int i,j,k;
// i - to mark the index of left aubarray (left_array)
// j - to mark the index of right sub-raay (right_array)
// k - to mark the index of merged subarray (R)
i = 0; j = 0; k =0;
while (i < leftCount && j < rightCount) {
if(left_array[i] < right_array[j])
R[k++] = left_array[i++];
else
R[k++] = right_array[j++];
}
while (i < leftCount)
R[k++] = left_array[i++];
while (j < rightCount)
R[k++] = right_array[j++];
}
/**
* Recursively merges an array of n values using group sizes of s.
* For example, given an array of 128 values and starting s value of 16
* will result in 8 groups of 16 merging into 4 groups of 32, then recursively
* calling merge_all which merges them into 2 groups of 64, then once more
* recursively into 1 group of 128.
*/
void merge_all(double *v, int n, int s) {
if (s < n) {
int i;
for (i = 0; i < n; i += 2*s) {
merge(v+i, v+i+s, s, s);
// result is in R starting at index 0
memcpy(v+i, R, 2*s*sizeof(double));
}
merge_all(v, n, 2*s);
}
}
void merge_sort(int n, double *v) {
merge_all(v, n, 1);
}
int main(int argc, char *argv[]) {
if (argc < 2) {
// () means optional
printf("usage: %s n (seed)\n", argv[0]);
return 0;
}
int n = atoi(argv[1]);
int seed = 0;
if (argc > 2) {
seed = atoi(argv[2]);
}
srand(seed);
int mpi_p, mpi_rank;
MPI_Init(&argc, &argv);
MPI_Comm_size(MPI_COMM_WORLD, &mpi_p);
MPI_Comm_rank(MPI_COMM_WORLD, &mpi_rank);
// init the temporary array used later for merge
R = malloc(sizeof(double)*n);
// all MPI processes will allocate a vector of length n, but only
// process 0 will initialize the values. You must distribute the
// values to processes. You will also likely need to allocate
// additional memory in your processes, make sure to clean it up
// by adding a correct free at the end of each process.
double *v = malloc(sizeof(double)*n);
if (mpi_rank == 0) {
init_random_vector(n, v);
}
double start = MPI_Wtime();
// do all the work ourselves! (you should make a better algorithm here!)
if (mpi_rank == 0) {
merge_sort(n, v);
}
double end = MPI_Wtime();
if (mpi_rank == 0) {
printf("Total time to solve with %d MPI Processes was %.6f\n", mpi_p, (end-start));
test_correctness(n, v);
}
MPI_Finalize();
free(v);
free(R);
return 0;
}
The parts I edited
double start = MPI_Wtime();
// do all the work ourselves! (you should make a better algorithm here!)
if (mpi_rank == 0)
{
//Scatter(sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype, root, comm)
MPI_Scatter(&v, sizeOf, MPI_DOUBLE, &v, sizeOf, MPI_DOUBLE, 0, MPI_COMM_WORLD);
}
if (mpi_rank > 0)
{
//merge_sort(sizeOf, R);
printf("%f :v[0]", v[0]); printf("%s", "\n");
printf("%f :v[1]", v[1]); printf("%s", "\n");
printf("%f :v[2]", v[0]); printf("%s", "\n");
}
if (mpi_rank == 0)
{
//Gather(sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype, root, comm)
//MPI_Gather(&v, sizeOf, MPI_DOUBLE, test, sizeOf, MPI_DOUBLE, 0, MPI_COMM_WORLD);
//merge(v, test, n, sizeOf);
merge_sort(n, v);
}
double end = MPI_Wtime();
The code should output three values from each processor that isn't the root, but it just gives me 0's. I tried many different parameters inside the scatter function call, to no avail. Almost everything I tried produces the same output found below. Sample output:
0.000000 :v[0]
0.000000 :v[1]
0.000000 :v[2]
Total time to solve with 2 MPI Processes was 0.000052
EDIT: Calling scatter from every process is also something I tried, as it sounds like this is the way you're supposed to call it instead of just calling it from the root. However, then I get a bunch of errors instead of any output at all. Errors look like this:
*** Process received signal ***
Signal: Segmentation fault (11)
Signal code: (128)
Failing at address: (nil)
*** Process received signal ***
Signal: Segmentation fault (11)
Signal code: (128)
Failing at address: (nil)
Something is clearly wrong here. Not sure what I'm doing wrong though.
The following code creates a Matrix [m][n] using double pointer malloc method and sends equal number of chunks of the matrix to each one of n-1 processors using non-blocking MPI functions. Processor P=0 is responsible for generating the matrix and sending them such that each one of P != 0 processors will receive a set of rows and process them.
The code does not work even though I have spent days to make sure every line is correct but I don't know where the bugs come from :( I appreciate any help.
#include <stdio.h>
#include <string.h>
#include <time.h>
#include "mpi.h"
int main (int argc, char* argv[]) {
const int RANK_0 = 0; // Rank 0
const int ROWS = 24; // Row size
const int COLS = 12; // Column size
const int TAG_0 = 0; // Message ID
const int TAG_0 = 0; // Message ID
int rank; // The process ID
int P; // Number of Processors
/* MPI Initialisation */
MPI_Init(&argc, &argv);
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
MPI_Comm_size(MPI_COMM_WORLD, &P);
/* Each client processor receives ROWS/P set of arrays */
if(rank != RANK_0){
int i,j;
int chunckSize= ROWS/P;
MPI_Request *req[chunckSize]; // Requests
MPI_Request *req1[chunckSize]; // Requests
MPI_Status status[chunckSize];
int ptr[chunckSize];
int **buffRecv= malloc(chunckSize * sizeof(int *));
for (i = 0; i < chunckSize ; i++) {
buffRecv[i] = malloc(COLS * sizeof(int));
MPI_Irecv(&ptr[i], 1, MPI_INT, RANK_0, TAG_1, MPI_COMM_WORLD, req1[i]);
MPI_Irecv(buffRecv[i], COLS, MPI_INT, RANK_0, TAG_0, MPI_COMM_WORLD, req[i]);
MPI_Wait(req1[i], MPI_STATUSES_IGNORE);
MPI_Wait(req[i], MPI_STATUSES_IGNORE);
}
printf("\n ===> Processor %d has recieved his set of rows, now start calculation: \n", rank);
for(i = 0; i< chunckSize; i++){
// print arrays row by row or do something
}
printf("\n Rank %d has done its tasks \n", rank);
}
else
{
/* MASTER PROCESS*/
int n=0;
int k,i,j,dest,offset;
int inc=1;
MPI_Request *req[ROWS]; // Requests
MPI_Request *req1[ROWS]; // Requests
int chunkSize= ROWS/P;
int **buf= malloc(ROWS * sizeof(int *));
offset = chunkSize;
for(dest = P; dest >= 0; dest--){
// ROWS/P rows to each destination
for (i = n; i < offset; i++)
{
buf[i] = malloc(COLS * sizeof(int));
for (j = 0; j < COLS; j++)
{
buf[i][j]=1;
}
if(dest == 0)
{
// rank_0 chunk will be handled here
}
else
{
MPI_Isend(&i, 1, MPI_INT, dest, TAG_1, MPI_COMM_WORLD, req1[i]);
MPI_Isend(buf[i], COLS, MPI_INT, dest, TAG_0, MPI_COMM_WORLD, req[i]);
}
}
// Print the result after each ROWS/P rows is sent
if(dest != 0){
printf("Row[%d] to Row[%d] is sent to rank# %d\n", n, k, dest);
}
n=offset;
offset= offset + chunkSize;
}
}
MPI_Finalize();
}
There are many issues in this code, which I'll try to enumerate later. But the most important one I believe is that the sending requested are never waited for, and re-utilised from one destination to the next. This is very wrong and since there is no testing or waiting point, the sending actions are likely to never happen.
I'll leave you with that for now and edit my answer slowly.
Edit:
Ok, now let's progress step by step:
The memory management: since you plan to distribute chunks of data to your processes, it is better to maximise the size of each transfer, and therefore to minimise the number of transfers. But to transfer several rows of your matrix inn one go, you need the data to be stored contiguously in memory. To achieve that while keeping the [i][j] double bracket access simplicity, you need to: first allocate the whole storage you need for your data, and second, to allocate a pointer of pointers to this data, which you will make point on each starting index of each row... This will look like this:
int **matrix = malloc( ROWS * sizeof( int* ) );
matrix[0] = malloc( COLS * ROWS * sizeof( int ) );
for ( int i = 1; i < ROWS; i++ ) {
matrix[i] = matrix[i-1] + COLS;
}
This is far from being the main issue but that's a good trick for another time.
The request issue: as already mentioned, your sending requests are not waited for and that is wrong. No MPI transaction is completed until you either waited for it with a MPI_Wait() or MPI_Waitall(), or after you checked it sufficiently with one of the MPI_Testxxx() functions. The simplest is here to use a MPI_Waitall()
What about process #0? It sends to itself, but never will it receive what was sent...
I didn't check the chunk sizes and offsets, but I'm pretty sure that if the number of processes doesn't divide the number of rows, you'll be in trouble.
Finally (hopefully), what you tried to do here corresponds very much to a MPI_Scatter() or possibly a MPI_Scatterv(). Now that your memory is stored linearly, have a look at it and that should just solve your problem.
Hope this helps.
I am currently working on a project where I need to implement a parallel fft algorithm using openmpi. I have a compiling piece of code, but when I run it over the cluster I get segmentation faults.
I have my hunches about where things are going wrong, but I don't think I have enough of an understanding about pointers and references to be able to make a efficient fix.
The first chunk that could be going wrong is in the passing of the arrays to the helper functions. I believe that either my looping is inconsistent, or I am not understanding how the to pass these pointers and get back the things I need.
The second possible spot would be within the actual mpi_Send/Recv commands. I am sending a type that is not supported by the openmpi c datatypes, so I am using the mpi_byte type to send the raw data instead. Is this a viable option? Or should I be looking into an alternative to this method.
/* function declarations */
double complex get_block(double complex c[], int start, int stop);
double complex put_block(double complex from[], double complex to[],
int start, int stop);
void main(int argc, char **argv)
{
/* Initialize MPI */
MPI_Init(&argc, &argv);
double complex c[N/p];
int myid;
MPI_Comm_rank(MPI_COMM_WORLD, &myid);
//printf("My id is %d\n",myid);
MPI_Status status;
int i;
for(i=0;i<N/p;i++){
c[i] = 1.0 + 1.0*I;
}
int j = log(p)/log(2) + 1;
double q;
double complex z;
double complex w = exp(-2*PI*I/N);
double complex block[N/(2*p)]; // half the size of chunk c
int e,l,t,k,m,rank,plus,minus;
int temp = (log(N)-log(p))/log(2);
//printf("temp = %d", temp);
for(e = 0; e < (log(p)/log(2)); e++){
/* loop constants */
t = pow(2,e); l = pow(2,e+temp);
q = n/2*l; z = cpow(w,(complex)q);
j = j-1; int v = pow(2,j);
if(e != 0){
plus = (myid + p/v)%p;
minus = (myid - p/v)%p;
} else {
plus = myid + p/v;
minus = myid - p/v;
}
if(myid%t == myid%(2*t)){
MPI_Recv((char*)&c,
sizeof(c),
MPI_BYTE,
plus,
MPI_ANY_TAG,
MPI_COMM_WORLD,
&status);
/* transform */
for(k = 0; k < N/p; k++){
m = (myid * N/p + k)%l;
c[k] = c[k] + c[k+N/v] * cpow(z,m);
c[k+N/v] = c[k] - c[k + N/v] * cpow(z,m);
printf("(k,k+N/v) = (%d,%d)\n",k,k+N/v);
}*/
printf("\n\n");
/* end transform */
*block = get_block(c, N/v, N/v + N/p + 1);
MPI_Send((char*)&block,
sizeof(block),
MPI_BYTE,
plus,
1,
MPI_COMM_WORLD);
} else {
// send data of this PE to the (i- p/v)th PE
MPI_Send((char*)&c,
sizeof(c),
MPI_BYTE,
minus,
1,
MPI_COMM_WORLD);
// after the transformation, receive data from (i-p/v)th PE
// and store them in c:
MPI_Recv((char*)&block,
sizeof(block),
MPI_BYTE,
minus,
MPI_ANY_TAG,
MPI_COMM_WORLD,
&status);
*c = put_block(block, c, N/v, N/v + N/p - 1);
//printf("Process %d send/receive %d\n",myid, plus);
}
}
/* shut down MPI */
MPI_Finalize();
}
/* helper functions */
double complex get_block(double complex *c, int start, int stop)
{
double complex block[stop - start + 1];
//printf("%d = %d\n",sizeof(block)/sizeof(double complex), sizeof(&c)/sizeof(double complex));
int j = 0;
int i;
for(i = start; i < stop+1; i++){
block[j] = c[i];
j = j+1;
}
return *block;
}
double complex put_block(double complex from[], double complex to[], int start, int stop)
{
int j = 0;
int i;
for(i = start; i<stop+1; i++){
to[i] = from[j];
j = j+1;
}
return *to;
}
I really appreciate the feedback!
You are using arrays / pointers to arrays in the wrong way. For example you declare an array as double complex block[N], which is fine (although uncommon, in most cases it is better to use malloc) and then you receive into it via MPI_Recv(&block). However "block" is already a pointer to that array, so by writing "&block" you are passing the pointer of the pointer to MPI_Recv. That's not what it expects. If you want to use the "&" notation you have to write &block[0], which would give you the pointer to the first element of the block-array.
Have you tried debugging your code? This can be a pain in a parallel setting, but it can tell you exactly where it is failing and usually also why.
If you're using Linux or OS X, you could run your code as follows on the command line:
mpirun -np 4 xterm -e gdb -ex run --args ./yourprog yourargs
where I'm assuming yourprog is the name of your program and yourargs are any command-line arguments you want to pass.
What this command will do is launch four xterm windows. Each xterm will in turn launch gdb as specified by the option -e. gdb will then execute the command run as specified by the option -ex and launch your executable with the given options, as specified by --args.
What you get are four xterm windows running four instances of your program in parallel with MPI. If any of the instances crashes, gdb will tell you where and why.
I have a 2D array which is distributed across a MPI process grid (3 x 2 processes in this example). The values of the array are generated within the process which that chunk of the array is distributed to, and I want to gather all of those chunks together at the root process to display them.
So far, I have the code below. This generates a cartesian communicator, finds out the co-ordinates of the MPI process and works out how much of the array it should get based on that (as the array need not be a multiple of the cartesian grid size). I then create a new MPI derived datatype which will send the whole of that processes subarray as one item (that is, the stride, blocklength and count are different for each process, as each process has different sized arrays). However, when I come to gather the data together with MPI_Gather, I get a segmentation fault.
I think this is because I shouldn't be using the same datatype for sending and receiving in the MPI_Gather call. The data type is fine for sending the data, as it has the right count, stride and blocklength, but when it gets to the other end it'll need a very different derived datatype. I'm not sure how to calculate the parameters for this datatype - does anyone have any ideas?
Also, if I'm approaching this from completely the wrong angle then please let me know!
#include<stdio.h>
#include<array_alloc.h>
#include<math.h>
#include<mpi.h>
int main(int argc, char ** argv)
{
int size, rank;
int dim_size[2];
int periods[2];
int A = 2;
int B = 3;
MPI_Comm cart_comm;
MPI_Datatype block_type;
int coords[2];
float **array;
float **whole_array;
int n = 10;
int rows_per_core;
int cols_per_core;
int i, j;
int x_start, x_finish;
int y_start, y_finish;
/* Initialise MPI */
MPI_Init(&argc, &argv);
/* Get the rank for this process, and the number of processes */
MPI_Comm_size(MPI_COMM_WORLD, &size);
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
if (rank == 0)
{
/* If we're the master process */
whole_array = alloc_2d_float(n, n);
/* Initialise whole array to silly values */
for (i = 0; i < n; i++)
{
for (j = 0; j < n; j++)
{
whole_array[i][j] = 9999.99;
}
}
for (j = 0; j < n; j ++)
{
for (i = 0; i < n; i++)
{
printf("%f ", whole_array[j][i]);
}
printf("\n");
}
}
/* Create the cartesian communicator */
dim_size[0] = B;
dim_size[1] = A;
periods[0] = 1;
periods[1] = 1;
MPI_Cart_create(MPI_COMM_WORLD, 2, dim_size, periods, 1, &cart_comm);
/* Get our co-ordinates within that communicator */
MPI_Cart_coords(cart_comm, rank, 2, coords);
rows_per_core = ceil(n / (float) A);
cols_per_core = ceil(n / (float) B);
if (coords[0] == (B - 1))
{
/* We're at the far end of a row */
cols_per_core = n - (cols_per_core * (B - 1));
}
if (coords[1] == (A - 1))
{
/* We're at the bottom of a col */
rows_per_core = n - (rows_per_core * (A - 1));
}
printf("X: %d, Y: %d, RpC: %d, CpC: %d\n", coords[0], coords[1], rows_per_core, cols_per_core);
MPI_Type_vector(rows_per_core, cols_per_core, cols_per_core + 1, MPI_FLOAT, &block_type);
MPI_Type_commit(&block_type);
array = alloc_2d_float(rows_per_core, cols_per_core);
if (array == NULL)
{
printf("Problem with array allocation.\nExiting\n");
return 1;
}
for (j = 0; j < rows_per_core; j++)
{
for (i = 0; i < cols_per_core; i++)
{
array[j][i] = (float) (i + 1);
}
}
MPI_Barrier(MPI_COMM_WORLD);
MPI_Gather(array, 1, block_type, whole_array, 1, block_type, 0, MPI_COMM_WORLD);
/*
if (rank == 0)
{
for (j = 0; j < n; j ++)
{
for (i = 0; i < n; i++)
{
printf("%f ", whole_array[j][i]);
}
printf("\n");
}
}
*/
/* Close down the MPI environment */
MPI_Finalize();
}
The 2D array allocation routine I have used above is implemented as:
float **alloc_2d_float( int ndim1, int ndim2 ) {
float **array2 = malloc( ndim1 * sizeof( float * ) );
int i;
if( array2 != NULL ){
array2[0] = malloc( ndim1 * ndim2 * sizeof( float ) );
if( array2[ 0 ] != NULL ) {
for( i = 1; i < ndim1; i++ )
array2[i] = array2[0] + i * ndim2;
}
else {
free( array2 );
array2 = NULL;
}
}
return array2;
}
This is a tricky one. You're on the right track, and yes, you will need different types for sending and receiving.
The sending part is easy -- if you're sending the whole subarray array, then you don't even need the vector type; you can send the entire (rows_per_core)*(cols_per_core) contiguous floats starting at &(array[0][0]) (or array[0], if you prefer).
It's the receiving that's the tricky part, as you've gathered. Let's start with the simplest case -- assuming that everything divides evenly so all the blocks have the same size. Then you can use the very helfpul MPI_Type_create_subarray (you could always cobble this together with vector types, but for higher-dimensional arrays this becomes tedious, as you need to create 1 intermediate type for each dimension of the array except the last...
Also, rather than hardcoding the decomposition, you can use the also-helpful MPI_Dims_create to create an as-square-as-possible decomposition of your ranks. Note
that this doesn't necessarily have anything to do with MPI_Cart_create, although you can use it for the requested dimensions. I'm going to skip the cart_create stuff here, not because it's not useful, but because I want to focus on the gather stuff.
So if everyone has the same size of array, then root is receiving the same data type from everyone, and one can use a very simple subarray type to get their data:
MPI_Type_create_subarray(2, whole_array_size, sub_array_size, starts,
MPI_ORDER_C, MPI_FLOAT, &block_type);
MPI_Type_commit(&block_type);
where sub_array_size[] = {rows_per_core, cols_per_core}, whole_array_size[] = {n,n}, and for here, starts[]={0,0} - eg, we'll just assume that everything starts the start.
The reason for this is that we can then use Gatherv to explicitly set the displacements into the array:
for (int i=0; i<size; i++) {
counts[i] = 1; /* one block_type per rank */
int row = (i % A);
int col = (i / A);
/* displacement into the whole_array */
disps[i] = (col*cols_per_core + row*(rows_per_core)*n);
}
MPI_Gatherv(array[0], rows_per_core*cols_per_core, MPI_FLOAT,
recvptr, counts, disps, resized_type, 0, MPI_COMM_WORLD);
So now everyone sends their data in one chunk, and it's received into the type into the right part of the array. For this to work, I've resized the type so that it's extent is just one float, so the displacements can be calculated in that unit:
MPI_Type_create_resized(block_type, 0, 1*sizeof(float), &resized_type);
MPI_Type_commit(&resized_type);
The whole code is below:
#include<stdio.h>
#include<stdlib.h>
#include<math.h>
#include<mpi.h>
float **alloc_2d_float( int ndim1, int ndim2 ) {
float **array2 = malloc( ndim1 * sizeof( float * ) );
int i;
if( array2 != NULL ){
array2[0] = malloc( ndim1 * ndim2 * sizeof( float ) );
if( array2[ 0 ] != NULL ) {
for( i = 1; i < ndim1; i++ )
array2[i] = array2[0] + i * ndim2;
}
else {
free( array2 );
array2 = NULL;
}
}
return array2;
}
void free_2d_float( float **array ) {
if (array != NULL) {
free(array[0]);
free(array);
}
return;
}
void init_array2d(float **array, int ndim1, int ndim2, float data) {
for (int i=0; i<ndim1; i++)
for (int j=0; j<ndim2; j++)
array[i][j] = data;
return;
}
void print_array2d(float **array, int ndim1, int ndim2) {
for (int i=0; i<ndim1; i++) {
for (int j=0; j<ndim2; j++) {
printf("%6.2f ", array[i][j]);
}
printf("\n");
}
return;
}
int main(int argc, char ** argv)
{
int size, rank;
int dim_size[2];
int periods[2];
MPI_Datatype block_type, resized_type;
float **array;
float **whole_array;
float *recvptr;
int *counts, *disps;
int n = 10;
int rows_per_core;
int cols_per_core;
int i, j;
int whole_array_size[2];
int sub_array_size[2];
int starts[2];
int A, B;
/* Initialise MPI */
MPI_Init(&argc, &argv);
/* Get the rank for this process, and the number of processes */
MPI_Comm_size(MPI_COMM_WORLD, &size);
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
if (rank == 0)
{
/* If we're the master process */
whole_array = alloc_2d_float(n, n);
recvptr = &(whole_array[0][0]);
/* Initialise whole array to silly values */
for (i = 0; i < n; i++)
{
for (j = 0; j < n; j++)
{
whole_array[i][j] = 9999.99;
}
}
print_array2d(whole_array, n, n);
puts("\n\n");
}
/* Create the cartesian communicator */
MPI_Dims_create(size, 2, dim_size);
A = dim_size[1];
B = dim_size[0];
periods[0] = 1;
periods[1] = 1;
rows_per_core = ceil(n / (float) A);
cols_per_core = ceil(n / (float) B);
if (rows_per_core*A != n) {
if (rank == 0) fprintf(stderr,"Aborting: rows %d don't divide by %d evenly\n", n, A);
MPI_Abort(MPI_COMM_WORLD,1);
}
if (cols_per_core*B != n) {
if (rank == 0) fprintf(stderr,"Aborting: cols %d don't divide by %d evenly\n", n, B);
MPI_Abort(MPI_COMM_WORLD,2);
}
array = alloc_2d_float(rows_per_core, cols_per_core);
printf("%d, RpC: %d, CpC: %d\n", rank, rows_per_core, cols_per_core);
whole_array_size[0] = n;
sub_array_size [0] = rows_per_core;
whole_array_size[1] = n;
sub_array_size [1] = cols_per_core;
starts[0] = 0; starts[1] = 0;
MPI_Type_create_subarray(2, whole_array_size, sub_array_size, starts,
MPI_ORDER_C, MPI_FLOAT, &block_type);
MPI_Type_commit(&block_type);
MPI_Type_create_resized(block_type, 0, 1*sizeof(float), &resized_type);
MPI_Type_commit(&resized_type);
if (array == NULL)
{
printf("Problem with array allocation.\nExiting\n");
MPI_Abort(MPI_COMM_WORLD,3);
}
init_array2d(array,rows_per_core,cols_per_core,(float)rank);
counts = (int *)malloc(size * sizeof(int));
disps = (int *)malloc(size * sizeof(int));
/* note -- we're just using MPI_COMM_WORLD rank here to
* determine location, not the cart_comm for now... */
for (int i=0; i<size; i++) {
counts[i] = 1; /* one block_type per rank */
int row = (i % A);
int col = (i / A);
/* displacement into the whole_array */
disps[i] = (col*cols_per_core + row*(rows_per_core)*n);
}
MPI_Gatherv(array[0], rows_per_core*cols_per_core, MPI_FLOAT,
recvptr, counts, disps, resized_type, 0, MPI_COMM_WORLD);
free_2d_float(array);
if (rank == 0) print_array2d(whole_array, n, n);
if (rank == 0) free_2d_float(whole_array);
MPI_Finalize();
}
Minor thing -- you don't need the barrier before the gather. In fact, you hardly ever really need a barrier, and they're expensive operations for a few reasons, and can hide problems -- my rule of thumb is to never, ever, use barriers unless you know exactly why the rule needs to be broken in this case. In this case in particular, the collective gather routine does exactly the same syncronization as the barrier, so just use that.
Now, moving onto the harder stuff. If things don't divide evenly, you have a few options. The simplest, though not necessarily the best, is just to pad the array so that it does divide evenly, even if just for this operation.
If you can arrange it so that the number of columns does divide evenly, even if the number of rows doesn't, then you can still use the gatherv and create a vector type for each part of the row, and gatherv that the appropriate number of rows from each processor. That would work fine.
If you definately have the case where neither can be counted on to divide, and you can't pad data for sending, then there are three sub-options I can see:
As susterpatt suggests, do point-to-point. For small numbers of tasks, this is fine, but as it gets larger, this will be significantly less efficient than the collective operations.
Create a communicator consisting of all the processors not on the outer edges, and use exactly the code above to gather their code; and then point-to-point the edge tasks' data.
Don't gather to process 0 at all; use the Distributed array type to describe the layout of the array, and use MPI-IO to write all the data to a file; once that's done, you can have process zero display the data in some way if you like.
It looks like the first argument to you MPI_Gather call should probably be array[0], and not array.
Also, if you need to get different amounts of data from each rank, you might be better off using MPI_Gatherv.
Finally, not that gathering all your data in once place to do output is not scalable in many circumstances. As the amount of data grows, eventually, it will exceed the memory available to rank 0. You might be much better off distributing the output work (if you are writing to a file, using MPI IO or other library calls) or doing point-to-point sends to rank 0 one at a time, to limit the total memory consumption.
On the other hand, I would not recommend coordinating each of your ranks printing to standard output, one after another, because some major MPI implementations don't guarantee that standard output will be produced in order. Cray's MPI, in particular, jumbles up standard output pretty thoroughly if multiple ranks print.
Accordding to this (emphasis by me):
The type-matching conditions for the collective operations are more strict than the corresponding conditions between sender and receiver in point-to-point. Namely, for collective operations, the amount of data sent must exactly match the amount of data specified by the receiver. Distinct type maps between sender and receiver are still allowed.
Sounds to me like you have two options:
Pad smaller submatrices so that all processes send the same amount of data, then crop the matrix back to its original size after the Gather. If you're feeling adventurous, you might try defining the receiving typemap so that paddings are automatically overwritten during the Gather operation, thus eliminating the need for the crop afterwards. This could get a bit complicated though.
Fall back to point-to-point communication. Much more straightforward, but possibly higher communication costs.
Personally, I'd go with option 2.