MPI Ring Leader Election returns segmentation fault - c

This is what I am trying to achieve.
Blue is the message.
Yellow is when the specific node changes the leader known to it.
Green is the final election of each node.
The code seems correct to me but it's always stuck inside the while loop no matter what I tried. For a small number of nodes during runtime it returns a segmentation fault after a while.
election_status=0;
firstmsg[0]=world_rank; // self rank
firstmsg[1]=0; // counter for hops
chief=world_rank; // each node declares himself as leader
counter=0; // message counter for each node
// each node sends the first message to the next one
MPI_Send(&firstmsg, 2, MPI_INT, (world_rank+1)%world_size, 1, MPI_COMM_WORLD);
printf("Sent ID with counter to the right node [%d -> %d]\n",world_rank, (world_rank+1)%world_size);
while (election_status==0){
// EDIT: Split MPI_Recv for rank 0 and rest
if (world_rank==0){
MPI_Recv(&incoming, 2, MPI_INT, world_size-1, 1, MPI_COMM_WORLD, &status);
}
else {
MPI_Recv(&incoming, 2, MPI_INT, (world_rank-1)%world_size, 1, MPI_COMM_WORLD, &status);
}
counter=counter+1;
if (incoming[0]<chief){
chief=incoming[0];
}
incoming[1]=incoming[1]+1;
// if message is my own and hopped same times as counter
if (incoming[0]==world_rank && incoming[1]==counter) {
printf("Node %d declares node %d a leader.\n", world_rank, chief);
election_status=1;
}
//sends the incremented message to the next node
MPI_Send(&incoming, 2, MPI_INT, (world_rank+1)%world_size, 1, MPI_COMM_WORLD);
}
MPI_Finalize();

In order to determine some minimum number among a number of ranks for all ranks, use MPI_Allreduce!
MPI_Send is blocking. It can block forever until a matching receive is posted. Your program deadlocks on the first call to MPI_Send - and any successive once should it complete by coincidence. To avoid that specifically use MPI_Sendrecv.
(world_rank-1)%world_size will produce -1 for world_rank == 0. Using -1 as rank number is not valid. It might coincidentially be MPI_ANY_SOURCE.

Related

MPI dynamic communication between 2 Processes

I have a process that manages a list of 2d arrays and passes different 2d arrays to each process. There is a possibility that I have not enough processes for all 2d arrays So I need to ask process 0 which manages the list of 2d arrays if there are any arrays left after any process which is not 0 finishes working with the first received array. I dont know how to implement this.
(...)
if(rank == 0)
//check if we have elements in 2d array left
while (ptr != NULL)
{
MPI_Status status;
int sig;
// Wait for a process to ask for a 2d array
MPI_Recv(&sig, 1, MPI_INT, MPI_ANY_SOURCE, 0, MPI_COMM_WORLD, &status);
// send it to them
MPI_Send(&ptr->grid.sudoku, N * N, MPI_INT, status.MPI_SOURCE, 0, MPI_COMM_WORLD);
ptr = ptr->next;
}
// free memory
delete_grids(list);
elementsAvailable = 0;
}
// rank != 0
else
{
lookForSolution(recvGrid(), rank); // recvGrid calls MPI_Recv and passes
//the given array to a function to calculate something
}
MPI_Bcast(&sudokusAvailable, 1, MPI_INT, 0, MPI_COMM_WORLD); // Here I thought I make a MPI_Bcast
//to tell other processes that process 0 has no arrays left but If I put it here
//the next if statement will never be called in the first place
if (rank != 0 && elementsAvailable == 1)
{
MPI_Status status;
// Send process 0 that ready for new 2d array
MPI_Send(1, 1, MPI_INT, 0, 0, MPI_COMM_WORLD);
lookForSolution(recvGrid(), rank);
}
(...)
The simplest solution is to let process zero send data to another process, and then post an MPI_Irecv for the result. When all the processes have work, the manager does an MPI_Waitany to see if any process is returning results, if so, accept, and send new work. If the manager runs out of work, it sends a special message to the workers, and they quit.
From here on it's an exercise to the reader.
If you want the manager also to participate in the work, it becomes a little more complicated, and you could solve the whole thing with one-sided communication, creating in effect a shared object with all the work.

LU Factorization with OpenMPI and send-ahead

I am trying to implement the LU factorization of the vandermonde matrix with OpenMPI.
Therefore, the matrix is split onto the processors in a cycle-wise manner, e.g. processor 0 gets rows 0, 3, 6, 9 and so on.
In iteration k, every process needs to use row k. The corresponding processor should send it already at the end of iteration k-1 with a non-blocking broadcast, and any other processor should receive it at the beginning of iteration k.
What I did:
Row 0 is send before iterating over k by Process 0.
At the beginning of iteration k:
int source = k%size;
if (source != rank)
{
printf("Process %d waits for row %d from process %d\n", rank, k, source);
MPI_Irecv(bcast_buffer, n, MPI_DOUBLE, source, k, MPI_COMM_WORLD, &request);
MPI_Wait(&request, &status);
printf("Process %d received row %d from process %d and continues\n", rank, k, source);
}
And the end of iteration k (l2gl is local to global for row indices)
// if current row (i) is row k+1
if((l2gl[i]==k+1) && (l2gl[i]!=n-1))
{
// Safe row to broadcast buffer and send it to any other process
for(tmp=0; tmp<n; tmp++)
{
bcast_buffer[tmp] = A[i][tmp];
}
for(int r=0; r<size; r++)
{
printf("Process %d sends row %d to process %d\n", rank, k+1, r);
MPI_Isend(bcast_buffer, n, MPI_DOUBLE, r, k+1, MPI_COMM_WORLD, &request);
}
}
In the output I find the following:
Process 3 received row 6 from Process 2 and continues
Process 2 received row 5 from Process 1 and continues
Process 2 sends row 6 to Process 0
Process 2 sends row 6 to Process 1
Process 2 sends row 6 to Process 2
Process 2 sends row 6 to Process 3
Does it definitely mean that I have an error in communication or may the prints be in a 'wrong' order due to delays of the print output or whatever?
As the result of the factorization is wrong, there has to be a mistake, either in the communication or the calculation.
I appreciate any help.

Trying to receive a vector with MPI_Recv

I'm implementing the Chan and Dehne sorting algorithm using MPI and the CGM realistic parallel model. So far each process receives N/p numbers from the original vector, each process then order their numbers sequentially using quick sort, each process then creates a sample from it's local vector (the sample has size p), each process then sends their sample over to P0; P0 should receive all samples in a bigger vector of size p*p so it can accommodate data from all processors. This is where I'm stuck, it seems to be working but for some reason after P0 receives all the data it exits with Signal: Segmentation fault (11). Thank you.
Here is the relevant part of the code:
// Step 2. Each process calculates it's local sample with size comm_sz
local_sample = create_local_sample(sub_vec, n_over_p, comm_sz);
// Step 3. Each process sends it's local sample to P0
if (my_rank == 0) {
global_sample_receiver = (int*)malloc(pow(comm_sz,2)*sizeof(int));
global_sample_receiver = local_sample;
for (i = 1; i < comm_sz; i++) {
MPI_Recv(global_sample_receiver+(i*comm_sz), comm_sz, MPI_INT,
i, 0, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
}
} else {
MPI_Send(local_sample, comm_sz, MPI_INT, 0, 0, MPI_COMM_WORLD);
}
printf("P%d got here\n", my_rank);
MPI_Finalize();
What is funny is that every process reachs the command printf("P%d got here\n", my_rank); and therefor prints to the terminal. Also global_sample_receiver does contain the data it is supposed to contain at the end, but the program still finished with a segmentation fault.
Here is the output:
P2 got here
P0 got here
P3 got here
P1 got here
[Krabbe-Ubuntu:05969] *** Process received signal ***
[Krabbe-Ubuntu:05969] Signal: Segmentation fault (11)
[Krabbe-Ubuntu:05969] Signal code: Address not mapped (1)
[Krabbe-Ubuntu:05969] Failing at address: 0x18000003e7
--------------------------------------------------------------------------
mpiexec noticed that process rank 0 with PID 5969 on node Krabbe-Ubuntu
exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------
Edit: I found the problem, turns out local_sample also needed a malloc.
The issue is you overwrite global_sample_receiver (which is a pointer) with local_sample (which is an other pointer) on rank zero.
If you want to set the first comm_sz elements of global_sample_receiver with the first comm_sz elements from local_sample, then you have to copy the data (e.g. not the pointer) manually.
memcpy(global_sample_receiver, local_sample, comm_sz * sizeof(int));
That being said, the natural MPI way of doing this is via MPI_Gather().
Here is what step 3 would look like :
// Step 3. Each process sends it's local sample to P0
if (my_rank == 0) {
global_sample_receiver = (int*)malloc(pow(comm_sz,2)*sizeof(int));
}
MPI_Gather(global_sample_receiver,comm_sz, MPI_INT, local_sample, comm_sz, MPI_INT, 0, MPI_COMM_WORLD);

C - MPI child processes receiving incorrect string

I am making a MPI password cracker, that uses brute force approach to crack a SHA512 hash key, I have code that works fine with 1 password and multiple processes, or multiple passwords and 1 process, but when I do multiple passwords & multiple processes I get the following error:
[ubuntu:2341] *** An error occurred in MPI_Recv
[ubuntu:2341] *** reported by process [191954945,1]
[ubuntu:2341] *** on communicator MPI_COMM_WORLD
[ubuntu:2341] *** MPI_ERR_TRUNCATE: message truncated
[ubuntu:2341] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
[ubuntu:2341] *** and potentially your MPI job)
I believe this is caused by process rank #1 receiving a string "/" instead of the password hash.
The issue is I am not sure why.
I have also noticed something strange with my code, I have the following loop in process rank 0:
while(!done){
MPI_Iprobe(MPI_ANY_SOURCE, MPI_ANY_TAG, MPI_COMM_WORLD, &done, &status);
if(done==1) {
for(i=1;i<size;i++){
if(i!=status.MPI_SOURCE){
printf("sending done to process %d\n", i);
MPI_Isend(&done, 1, MPI_INT, i, 0, MPI_COMM_WORLD, &request[i]);
}
}
}
}
Which keeps looping waiting for one of the child processes to alert it that it has found the password. Lets say I am running 2 processes (excluding the base process) and process 2 finds the password, the output will then be:
sending done to process 1
sending done to process 1
When it should only be sending that once, or at the least if it is sending it twice surely the one of those values should be 2, not both of them being 1?
The main bulk of my code is as follows:
Process 0 :
while(!feof(f)) {
fscanf(f, "%s\n", buffer);
int done = 0;
int i, sm;
// lengths of the word (we know it should be 92 though)
length = strlen(buffer);
// Send the password to every process except process 0
for (sm=1;sm<size;sm++) {
MPI_Send(&length, 1, MPI_INT, sm, 0, MPI_COMM_WORLD);
MPI_Send(buffer, length+1, MPI_CHAR, sm, 0, MPI_COMM_WORLD);
}
// While the passwords are busy cracking - Keep probing.
while(!done){
MPI_Iprobe(MPI_ANY_SOURCE, MPI_ANY_TAG, MPI_COMM_WORLD, &done, &status);
if(done==1) {
for(i=1;i<size;i++){
if(i!=status.MPI_SOURCE){
printf("sending done to process %d\n", i);
MPI_Isend(&done, 1, MPI_INT, i, 0, MPI_COMM_WORLD, &request[i]);
}
}
}
}
}
Which loops through the file, grabs a new password, sends the string to the child processes, at which point they receive it:
MPI_Recv(&length, 1, MPI_INT, 0, 0, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
printf("string to be recieived has %d characters\n", length);
MPI_Recv(buffer, length+1, MPI_INT, 0, 0, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
printf("Process %d received %s\n", rank, buffer);
The child processes crack the password, then repeat with the next one. (assuming there is one, currently it's an infinite loop but I want to sort it out with 2 passwords before I fix that).
All processes recieve the correct password for the first time, it's when they grab the second password that only 1 process has the correct one and the rest receive a "/" character.
Alright, typical case of me getting worked up and over looking something simple.
I'll leave this question just in case anyone else happens to have the same issue though.
I was forgetting to also receive the solution after probing it.
Was never fully clear how probe differed from receive but I guess probe just flags something changes, but to actually take it out of the "queue" you need to then collect it with receive.

how do I print the log in order in MPI

What I have
I have a C-program using MPI, and it uses 4 processes: 1 vehicle(taskid=0) and 3 passengers.
Vehicle can accommodate 2 passengers at a time.
3 customers keep coming back to get a ride.
For Vehicle, I have:
int passengers[C] = {0};
while(1)
MPI_Recv(&pid, 1, MPI_INT, MPI_ANY_SOURCE, 1, MPI_COMM_WORLD, &status);
//put pid in passengers[totalNumberArrived]
if(totalNumberArrived == 2)
printf("vehicle left...");
sleep(5);
printf("vehicle came back...");
for (i=0; i<2; i++)
MPI_Send(&passengers[i], 1, MPI_INT, passengers[i], 1, MPI_COMM_WORLD);
totalNumberArrived = 0;
if(done)//omitting the details here
break;
And, for each passengers, I have:
for (i to NumOfRound)
sleep(X);
printf("%d is sending a msg", tasked)
MPI_Send(&taskid, 1, MPI_INT, 0, 1, MPI_COMM_WORLD);
MPI_Recv(&pid, 1, MPI_INT, 0, 1, MPI_COMM_WORLD, &pstatus);
printf("%d received from %d\n", tasked, pid, pstatus.MPI_SOURCE);
issue
If the vehicle left for a ride with taskid 1 and 3, I expect to see this kind of output:
vehicle left...
vehicle came back...
1 is sending a msg
3 is sending a msg (this could be before 1's msg though)
but I sometimes get
vehicle left...
1 is sending a msg
3 is sending a msg
vehicle came back...
which looks like the passenger is not blocked until the vehicle comes back.
I thought that MPI_Recv blocks the task until it gets a msg from the vehicle, so I researched and read that MPI_Recv does block but this kind of issues occur because the printf is not necessarily printing in order. I also read that some recommends to use flush but in some cases flush doesn't work.
I'm not sure what I should do in my case. Is it really just the matter of printf order?
I've also read this: Ordering Output in MPI
and wonder if I should add a master thread and let it be the central controller for vehicle and passengers??
You can't rely on printing output to be ordered between processes. The only thing you can count on is that output will be ordered per processes. Therefore, if for some reason it's critical that you can print things to STDOUT/STDERR in order, you need to aggregate it to one process first.

Resources