How to robustly write to a slow & unreliable NFS - c

Question

I'm not an expert in C and I'm looking for some advice to to make my program more robust and reliable. Just to give some context: I've written a program to do some scientific computation that takes quite a long time (about 20h) that I'm executing on a large university HPC linux cluster using a SLRUM scheduling system and NFS mounted file systems. What seems to happen is that some time during the 20h the connection to the file system goes stale (on the entire machine; independent of my program) and the first attempt to open & write a file takes a really long time and that results in a segfault cored dumped error that I have so far not been able to precisely track down. Below is a minimal file that at least conceptually reproduces the error: The program starts, opens a file and everything works. The program does some long computation (simulated by sleep()), tries to open & write to the same file again, and it fails. What are some conventions to make my code more robust and reliably write my results to file without crashing?
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
int main(int argc, char **argv) {
// Declare variables
FILE *outfile;
char outname[150] = "result.csv";
// Open file for writing
printf("CHECKING if output file '%s' is writable?", outname);
outfile=fopen(outname, "w");
if (outfile == NULL) {
perror("Failed: ");
exit(EXIT_FAILURE);
}
fclose(outfile);
printf(" PASSED.\n");
// Do some computation that takes really long (around 19h)
sleep(3);
// Open file again and Write results
printf("Writing results to %s ...", outname);
outfile=fopen(outname, "w");
if (outfile == NULL) {
perror("Failed writing in tabulate_vector_new: ");
exit(EXIT_FAILURE);
}
fprintf( outfile, "This is the important result.\n");
fclose(outfile);
printf(" DONE.\n");
return 0;
}

Answer 1

It seems odd that your program would segfault due to an NFS issue. I would expect it to hang indefinitely, not crash. That having been said, I would suggest forking a new process to check whether the NFS mount is working. That way, your important code won't be directly involved in testing the problematic file system. Something like the following approach may be useful:
pid_t pid = fork();
if (pid == -1)
{
// error, failed to fork(). should probably give up now. something is really wrong.
}
else if (pid > 0)
{
// if the child exits, it has successfully interacted with the NFS file system
wait(NULL);
// proceed with attempting to write important data
}
else
{
// we are the child; fork df in order to test the NFS file system
execlp("df", "df", "/mnt", (char *)NULL)
// the child has been replaced by df, which will try to statfs(2) /mnt for us
}
The general concept here is that we utilize the df command to check whether the NFS file system (which I assume is at /mnt) is working. If it's temporarily not working, df should hang until it starts working again, and then exit, returning control to your program. If you suspect df might hang forever, you could enhance my example by using alarm(2) to wait a certain period of time, probably at least a few minutes, after which you could retry running df. Note that this could result in zombie df processes sticking around.
In the end, the correct solution is to try to get a more reliable NFS server, but until you can do that, I hope this is helpful.