I am programming win32 using fopen fread fwrite in C.
How can I force windows to write data through to the disk? Is than an API call for this?
I have a program that must absolutely make sure that data is saved to disk before it saves to a different indexing file, otherwise if there are crashes the index file can sometimes update but the other file doesn't causing a bad inconsistency.
fflush() does what you want. If you decide to use the Win32 APIs to do your file IO instead of the C STDIO lib, then supply the FILE_FLAG_WRITE_THROUGH flag to CreateFile(), which does exactly what you want.
Use fflush function after each fwrite
See the _commit function. Also at this other MSDN page. This only works with _open and other functions, so see this article - you can link your program with commode.obj to cause fflush to commit to the disk.
Beware that whatever function you use this will not save you against a power outage or hard reset. Hard disks play smart and can report that write has been finished even if they have just put the data into cache and have not actually written it onto surface.
See this archived KB article.
When you initially open your file using fopen, include the "c" mode option as the LAST OPTION:
fopen( path, "wc") // w - write mode, c - allow immediate commit to disk
Then when you want to force a flush to disk, call
_flushall()
We made this call before calling
fclose()
The issue of writes not being persisted to disk caused us real problems, until we discovered this method.
From that above site:
"Microsoft C/C++ version 7.0 introduces the "c" mode option for the fopen()
function. When an application opens a file and specifies the "c" mode, the
run-time library writes the contents of the file buffer to disk when the
application calls the fflush() or _flushall() function. "
Related
One way to write into a file is by using fprintf(). However, this function does not write the results into a file immediately. It rather seems to write everything at once when the program is terminated or finished.
My question is the following: I have a program that takes very long time to run (4-5 hours for big dataset). During this time, I want to see the intermediate results so that I don't have to wait for 5 hours. My university uses Sun Grid Engine for job submission. As most of you know, you have to wait until your job finishes to see your final results. Thus, I want to be able to write the intermediate results into a text file and see the updated results as the program is processing (Similarly if I am using printf).
How can I modify fprintf() to write anything I want immediately to the target file?
You can use the fflush function after each write to flush the output buffer to disk.
fprintf(fileptr, "writing to file\n");
fflush(fileptr);
If you're on a POSIX system (i.e. Linux, BSD, etc), and you really want to be sure the file is written to disk, i.e. you want to flush the kernel buffers as well as the userspace buffers, also use fsync:
fsync(fileno(fileptr));
But fflush should be sufficient. Don't bother with fsync unless you find that you need to.
Maybe you can set FILE pointer _IONBF mode. Then you cloud not use fflush or fsync.
FILE *pFilePointor = fopen(...);
setvbuf(pFilePointor, NULL, _IONBF, 0);
fprintf(...)
fprintf(...)
fflush
This works on FILE *. For your case it looks more appropriate. Please note fflush(NULL) will update all opened files / streams and my be CPU intensive. You may like to use/avoid fflush(NULL) for performance reason.
fsync
This works on int descriptor. It not only updates file/stream, also metadata. It can work even in system crash / reboot cases as well. You can check man page for more details.
Personally I use fflush, and it works fine for me (in Ubuntu / Linux).
I understand the general process of writing and reading from a file, but I was curious as to what is happening under the hood during file writing. For instance, I have written a program that writes a series of numbers, line by line, to a .txt file. One thing that bothers me however is that I don't see the information written until after my c program is finished running. Is there a way to see the information written while the program is running rather than after? Is this even possible to do? This is a hard question to phrase in one line, so please forgive me if it's already been answered elsewhere.
The reason I ask this is because I'm writing to a file and was hoping that I could scan the file for the highest and lowest values (the program would optimally be able to run for hours).
Research buffering and caching.
There are a number of layers of optimisation performed by:
your application,
your OS, and
your disk driver,
in order to extend the life of your disk and increase performance.
With the careful use of flushing commands, you can generally make things happen "quite quickly" when you really need them to, though you should generally do so sparingly.
Flushing can be particularly useful when debugging.
The GNU C Library documentation has a good page on the subject of file flushing, listing functions such as fflush which may do what you want.
You observe an effect solely caused by the C standard I/O (stdio) buffers. I claim that any OS or disk driver buffering has nothing to do with it.
In stdio, I/O happens in one of three modes:
Fully buffered, data is written once BUFSIZ (from <stdio.h>) characters were accumulated. This is the default when I/0 is redirected to a file or pipe. This is what you observe. Typically BUFSIZ is anywhere from 1k to several kBytes.
Line buffered, data is written once a newline is seen (or BUFSIZ is reached). This is the default when i/o is to a terminal.
Unbuffered, data is written immediately.
You can use the setvbuf() (<stdio.h>) function to change the default, using the _IOFBF, _IOLBF or _IONBF macros, respectively. See your friendly setvbuf man page.
In your case, you can set your output stream (stdout or the FILE * returned by fopen) to line buffered.
Alternatively, you can call fflush() on the output stream whenever you want I/O to happen, regardless of buffering.
Indeed, there are several layers between the writing commands resp. functions and the actual file.
First, you open the file for writing. This causes the file to be either created or emptied. If you write then, the write doesn't actually occur immediately, but the data are cached until the buffer is full or the file is flushed or closed.
You can call fflush() for writing each portion of data, or you can actually wait until the file is closed.
Yes, it is possible to see whats written in the file(s). If you programm under Linux you can open a new Terminal and watch the progress with for example "less Filename".
I write Server for client-server application in C. I have to save logs to file.
I write it into a file using fprintf, but when the server go down I lost the data in the file, cause I don't close filedescriptor, is there any function which tell my program save the data?
Thx
If you fflush after every fprintf it helps.
fflush should do what you want — it ensures all output is explicitly written to the file rather than e.g. being cached for later writing. So that moves the data out of user space.
sync can then be used — it causes all buffered file changes to be physically written (though per the spec it needn't block until the writes are complete, so you can be certain they've started but not that they've finished).
I do most of my programming on embedded processors, or on linux. When i need to sync data to my persistant store, i usually use the sync(2) system call. Is there an equivelent for Windows?
http://msdn.microsoft.com/en-us/library/aa364439(VS.85).aspx
http://www.codeproject.com/KB/system/EjectMediaByLetter.aspx
FlushFileBuffers with a handle to a volume. You have to do this for every volume :(
Use FlushFileBuffers, but you need a handle to the file you need flushed.
If you are using posix file functions (fopen() etc.), you can use _flushall:
http://msdn.microsoft.com/en-us/library/s9xk9ehd%28v=vs.80%29.aspx
See here: https://jeffpar.github.io/kbarchive/kb/066/Q66052/
When you initially open your file using fopen, include the "c" mode option as the LAST OPTION:
fopen( path, "wc") // w - write mode, c - allow immediate commit to disk
Then when you want to force a flush to disk, call
_flushall()
We made this call before calling
fclose()
Note that this approach does NOT required Administrative rights, which FlushFileBuffers does require.
From that above site:
"Microsoft C/C++ version 7.0 introduces the "c" mode option for the fopen()
function. When an application opens a file and specifies the "c" mode, the
run-time library writes the contents of the file buffer to disk when the
application calls the fflush() or _flushall() function. "
I am currently writing a plugin in C++. For my functionality I ask the API to save out a file. The API gives me a return value when the file is written... or so it seemd. The problem is, that this return value is returned too early so that I can not be sure, that the file is written completely.
Is there a possibility of checking the write completeness of the file independent of the api?
That's because the system does not write data to disk as soon as it's requested, but still returns. In C, you could use int fflush (FILE *stream), but I don't know how you'd do that in C++.
Not really, even if we re-read the file, to 'verify' that the write had taken place, you could still be looking at a kernel buffer.
You could compare the original buffer and the file you have written to byte for byte, but I think it is better to trust your operating system with the according fflush and fclose operations.
As mentioned you can use fflush(). you can call sync() / fsync() based upon whether you are using stream class or descriptor.