I'm writing a MPI application that takes a filename as an argument and tries to read from the file using regular C functions. I run this application on several nodes of a cluster by using qsub, which in turn uses mpiexec.
The application runs just fine on a local node where the file is. For this I just call mpiexec directly:
mpiexec -n 4 ~/my_app ~/input_file.txt
But when I submit it with qsub to be run on other nodes of the cluster, the file reading part fails. The application errors at fopen call -- it can't open the file (likely because it's not present).
The question is, how do I make the file available to all nodes? I have looked over qsub manpage and couldn't fine anything relevant.
I guess Vanilla Gorilla doesn't need an answer any more? However, let's consider the case of a pathological system with no parallel file system and a file system available only at one node. There is a way in ROMIO (a very common MPI-IO implementation) to achieve your goal:
how can i transfer file from one proccess to all other with mpi?
Related
I am confused with the following code from tcl wiki 1089,
#define TEMPBUFSIZE 256 /* usually enough space! */
char buf[[TEMPBUFSIZE]];
I was curious and I tried to compile the above syntax in gcc & armcc, both fails. I was looking to understand how tcl handle to file pointer mechanism works to solve the chaos on data logging by multiple jobs running in a same folder [log files unique to jobs].
I have multiple tcl scripts running in parallel as LSF Jobs each using a log file.
For example,
Job1 -> log1.txt
Job2 -> log2.txt
(file write in both case is "intermittent" over the entire job execution)
Some of the text which I expect to be part of log1.txt is written to log2.txt and vice versa randomly. I have tried with "fconfigure $fp -buffering none", the behaviour still persists. One important note, all the LSF jobs are submitted from the same folder and if I submit the jobs from individual folder, the log files dont have text write from other job. I would like the jobs to be executed from same folder to reduce space consumption from repeating the resource in different folder.
Question1:
Can anyone advice me how the tcl "handles" is interpreted to a pointer to the memory allocated for the log file? I have mentioned intermitent because of the following, "Tcl maps this string internally to an open file pointer when it is time for the interpreter to do some file I/O against that particular file - wiki 1089"
Question2:
Is there a possiblity that two different "open" can end up having same "file"?
Somewhere along the line, the code has been mangled; it looks like it happened when I converted the syntax from one type of highlighting scheme to another in 2011. Oops! My original content used:
char buf[TEMPBUFSIZE];
and that's what you should use. (I've updated the wiki page to fix this.)
I am writing output from a simulation to a file using the following code
sprintf(filename, "time_data.dat");
FILE *fp = fopen(filename,"w");
for(i=0;i<ntime;i++){
compute_data();
fprintf(fp, "%d %lf %lf \n", step, time_val ,rho_rms);
}
return;
On my desktop, I get to see the file time_data.dat update every few hours (compute_data() takes a few hundred seconds per time step, with OpenMP on an i7 machine). I have now submitted the job to a cluster node (E5 2650 processor running ubuntu server). I have been waiting for 5 days now, and not a line line has appeared in the file yet. I do
tail -f time_data.dat
to check the output. The simulation will take another couple of weeks to complete. I can't wait for that long to see if my output is good. Is there a way I can probe the OS in the node to flush its buffer without disturbing the computation? If I cancel the job now, I am sure there won't be any output.
Please note that the hard disk to which the output file is being written is one shared using NFS over multiple nodes and the master node. Is this causing any trouble? Is there a temporary file place were the output is actually being written?
PS: I did du -h to find the file showing size 0. I also tried ls -l proc/$ID/fd to confirm the file did open.
You might use lsof or simply ls -l /proc/$(pidof yoursimulation)/fd to check (on the cluster node) that indeed time_data.dat has been opened.
For such long-running programs, I would believe it is worthwhile to consider using:
application checkpointing techniques
persistency of your application data e.g. in some database
design some way to query your app's state (eg use some HTTP server library such as libonion, or at least have some JSONRPC or other service to query something about the state)
This is a very beginner-level question in C.
Don't know where to start looking/searching.
So, if I have a program continuously running in C, what is the best way to accept input through the command line into the program?
EX, mysql is already running, but you can process a command call
mysql SELECT * FROM *
Do I need a different program to write to file/stdin?enter code here
Clarification:
So, mysql seems to be able to take in commands while it is already running... is that possible in C?
Goal:
I have some hooks into open gl es, and I want to run a continuous draw loop in the background, while having the ability to call commands such as
glhookprogram make "object1" model "triangle" program "default"
glhookprogram attr "object1" position "1.0, 1.0, 0.0" scale "2.0" rotation "45, 0, 0"
this way, I can have a node server run hw-accelerated animations in javascript on the rpi.
Looks like this is what you need (and I'm sorry - I won't be going into too much details as there are plenty of sources on the Web about that):
A "server" - that would be your background process that stays running in memory and can accept and process commands (requests)
A "client" - a (short-running?) process that can accept commands from user (GUI, command-line. Network? Other process?) and send requests to your "server"
This is not a trivial task for a beginner. I would suggest googling for "server-client" and for "inter-process communications" first and go from there.
The range of options to "accept input" into your server includes (but is not limited to) the following:
(Windows) messages
Shared memory and a command queue (producer-consumer)
Shared file (just listing it here for completeness, I'd advise against this particular one for your case)
Named pipes
Sockets (thanks for reminding me of those in the comments, can't believe I missed that!)
I have three processes designed to run constantly in both Linux and Mac OS X environments. One process (the Downloader) downloads and stores a local copy of a large XML file every 30 seconds. Two other processes (the Workers) use the stored XML file for input. Each Worker starts and runs at random times. Since the XML file is big, it takes a long time to download. The Workers also take a long time to read and parse it.
What is the safest way to setup the processes so the Downloader doesn't clobber the stored file while the Workers are trying to read it?
For Linux and Mac OS X machines that use inode based file systems, use temporary files to store the data while its being downloaded (and is an incomplete state). Once the download is complete, move the temporary file into its final location with an atomic action.
For a little more detail, there are two main things to watch out for when one process (e.g. Downloader) writes a file that's actively read by other processes (e.g. Workers):
Make sure the Workers don't try to read the file before the Downloader has finished writing it.
Make sure the Downloader doesn't alter the file while the Workers are reading it.
Using temporary files accommodates both of these points.
For a more specific example, when the Downloader is actively pulling the XML file, have it write to a temporary location (e.g. 'data-storage.tmp') on the same device/disk* where the final file will be stored. Once the file is completely downloaded and written, have the Downloader move it to its final location (e.g. 'data-storage.xml') via an atomic (aka linearizable) rename command like bash's mv.
* Note that the reason the temporary file needs to be on the same device as the final file location is to ensure the inode number stays the same and the rename can be done atomically.
This methodology ensures that while the file is being downloaded/written the Workers won't see it since it's in the .tmp location. Because of the way renaming works with inodes, it also make sure that any Worker that opened the file continues to see the old content even if a new version of the data-storage file is put in place.
Downloader will point 'data-storage.xml' to a new inode number when it does the rename, but the Worker will continue to access 'data-storage.xml' from the previous inode number thereby continuing to work with the file in that state. At the same time, any Worker that opens a new copy 'data-storage.xml' after Downloader has done the rename will see contents from the new inode number since it's now what is referenced directly in the file system. So, two Workers can be reading from the same filename (data-storage.xml) but each will see a different (and complete) version of the contents of the file based on which inode the filename was pointed to when the file was first opened.
To see this in action, I created a simple set of example scripts that demonstrate this functionality on github. They can also be used to test/verify that using a temporary file solution works in your environment.
An important note is that it's the file system on the particular device that matters. If you are using a Linux or Mac machine but working with a FAT file system (for example, a usb thumb drive), this method won't work.
I'm intending to create a programme that can permanently follow a large dynamic set of log files to copy their entries over to a database for easier near-realtime statistics. The log files are written by diverse daemons and applications, but the format of them is known so they can be parsed. Some of the daemons write logs into one file per day, like Apache's cronolog that creates files like access.20100928. Those files appear with each new day and may disappear when they're gzipped away the next day.
The target platform is an Ubuntu Server, 64 bit.
What would be the best approach to efficiently reading those log files?
I could think of scripting languages like PHP that either open the files theirselves and read new data or use system tools like tail -f to follow the logs, or other runtimes like Mono. Bash shell scripts probably aren't so well suited for parsing the log lines and inserting them to a database server (MySQL), not to mention an easy configuration of my app.
If my programme will read the log files, I'd think it should stat() the file once in a second or so to get its size and open the file when it's grown. After reading the file (which should hopefully only return complete lines) it could call tell() to get the current position and next time directly seek() to the saved position to continue reading. (These are C function names, but actually I wouldn't want to do that in C. And Mono/.NET or PHP offer similar functions as well.)
Is that constant stat()ing of the files and subsequent opening and closing a problem? How would tail -f do that? Can I keep the files open and be notified about new data with something like select()? Or does it always return at the end of the file?
In case I'm blocked in some kind of select() or external tail, I'd need to interrupt that every 1, 2 minutes to scan for new or deleted files that shall (no longer) be followed. Resuming with tail -f then is probably not very reliable. That should work better with my own saved file positions.
Could I use some kind of inotify (file system notification) for that?
If you want to know how tail -f works, why not look at the source? In a nutshell, you don't need to periodically interrupt or constantly stat() to scan for changes to files or directories. That's what inotify does.