I need to write all logs in a single file by using a buffer.
This buffer can be accessed by different threads and logs can be written,
This is protected under a critical section
Sometimes I observed a delay of 1 second in writing the buffer into file (This is observed in time stamp in logs)
How much time taken for fflush to write the data into the files.
Related
Lets say I open up a binary file (or text as binary) and read 16K bytes. Another file opens the process and truncates it, appends, or changes bytes I haven't read (or already read)
Would there be a way to have read fail? Currently when I get to the end of the file I do a stat and check if the modification time matches the time I started with but I rather fail sooner than later
You could use a critical section. I think that's what Nate Eldredge Suggested.
Enter critical section
Read file
Leave critical section
Now you can write
Then your "read" will not fail, but there will be no more reason to do so.
I am trying to benchmark file read times (sequential access) for NTFS. My code gets a start time, performs a read of size equal to 4096 bytes (cluster size of NTFS on the system) and records the end time. The difference between the two times is then stored and the process is repeated until end of file is reached. The file size I currently use is 40K, so I get 10 time difference values.
When accessing the file opened (using CreateFile) without FILE_FLAG_NO_BUFFERING, access time for the first block is close to 30 micro-seconds and drops to about 7 micro-seconds for subsequent accesses (due to caching).
When using FILE_FLAG_NO_BUFFERING, the first block's access time is close to 21 milli-seconds and drops to about 175 micro-seconds for subsequent accesses.
Shouldn't the first block access time be same with or without the flag, since it's not buffered? Also, why do access times drop after the first when the flag is used? I was expecting them to remain constant since we've specified we don't want buffering.
amongst other things, the access time include several other (longish) factors beside the actual data transfer time.
Such times include searching the directory structure (the first time only) to find the actual file (this include a 'head seek' time (which is very long as it requires the physical movement of the heads),
then the rotation time to get over the correct sector on the disk,
then the actual data transfer time.
This is followed by a 'head seek' time to the actual beginning of the file cylinder,
followed by a 'sector seek' time to get over the correct sector,
followed by the actual data transfer time.
subsequent reads will not include the accessing of the directory info.
Any access can (but not always) include some 'head seek' time (which varies in length and depends on where the heads are currently located and where the desire data is currently located.
With buffering, the subsequent access times are greatly reduced (on most reads) because the actual transfer will include multiple sectors so only occasionally needs to actually access the disk.
When not buffered, a lot depends on if the disk, itself, performs any buffering (these days, most do perform local buffering) Accessing data that is already in the disk buffer eliminates all seek times (head and sector) making the transfer much faster.
My understanding of FUSE's multithreaded read cycle is something like this:
....
.-> read --.
/ \
open ---> read ----+-> release
\ /
`-> read --'
....
i.e., Once a file is open'd, multiple read threads are spawned to read different chunks of the file. Then, when everything that was wanted has been read, there is a final, single release. All these are per ones definition of open, read and release as FUSE operations.
I'm creating an overlay filesystem which converts one file type to another. Clearly, random access without any kind of indexing is a problem; so for the time being, I'm resorting to streaming. That is, in the above model, each read thread would begin the conversion process from the start, until it arrives at the correct bit of converted data to push out into the read buffer.
This is obviously very inefficient! To resolve this, a single file conversion process can start at the open stage and use a mutex and read cursor (i.e., "I've consumed this much, so far") that the reader threads can use to force sequential access. That is, the mutex gets locked by the thread that requests the data from the current cursor position and all other reader threads have to wait until it's their turn.
I don't see why this wouldn't work for streaming a file out. However, if any random/non-sequential access occurs we'll have a deadlock: if the requested offset is beyond or before the current cursor position, the mutex will never unlock for the appropriate reader to be able to reach that point. (Presumably we have no access to FUSE's threads, so to act as a supervisor. Also, I can't get around the problem by forcing the file to be a FIFO, as FUSE doesn't support writing to them.)
Indeed, I would only be able to detect when this happens if the mutex is locked and the cursor is beyond the requested offset (i.e., the "can't rewind" situation). In that case, I can return EDEADLK, but there's no way to detect "fast-forward" requests that can't be satisfied.
At the risk of the slightly philosophical question... What do I do?
I understand the general process of writing and reading from a file, but I was curious as to what is happening under the hood during file writing. For instance, I have written a program that writes a series of numbers, line by line, to a .txt file. One thing that bothers me however is that I don't see the information written until after my c program is finished running. Is there a way to see the information written while the program is running rather than after? Is this even possible to do? This is a hard question to phrase in one line, so please forgive me if it's already been answered elsewhere.
The reason I ask this is because I'm writing to a file and was hoping that I could scan the file for the highest and lowest values (the program would optimally be able to run for hours).
Research buffering and caching.
There are a number of layers of optimisation performed by:
your application,
your OS, and
your disk driver,
in order to extend the life of your disk and increase performance.
With the careful use of flushing commands, you can generally make things happen "quite quickly" when you really need them to, though you should generally do so sparingly.
Flushing can be particularly useful when debugging.
The GNU C Library documentation has a good page on the subject of file flushing, listing functions such as fflush which may do what you want.
You observe an effect solely caused by the C standard I/O (stdio) buffers. I claim that any OS or disk driver buffering has nothing to do with it.
In stdio, I/O happens in one of three modes:
Fully buffered, data is written once BUFSIZ (from <stdio.h>) characters were accumulated. This is the default when I/0 is redirected to a file or pipe. This is what you observe. Typically BUFSIZ is anywhere from 1k to several kBytes.
Line buffered, data is written once a newline is seen (or BUFSIZ is reached). This is the default when i/o is to a terminal.
Unbuffered, data is written immediately.
You can use the setvbuf() (<stdio.h>) function to change the default, using the _IOFBF, _IOLBF or _IONBF macros, respectively. See your friendly setvbuf man page.
In your case, you can set your output stream (stdout or the FILE * returned by fopen) to line buffered.
Alternatively, you can call fflush() on the output stream whenever you want I/O to happen, regardless of buffering.
Indeed, there are several layers between the writing commands resp. functions and the actual file.
First, you open the file for writing. This causes the file to be either created or emptied. If you write then, the write doesn't actually occur immediately, but the data are cached until the buffer is full or the file is flushed or closed.
You can call fflush() for writing each portion of data, or you can actually wait until the file is closed.
Yes, it is possible to see whats written in the file(s). If you programm under Linux you can open a new Terminal and watch the progress with for example "less Filename".
I write Server for client-server application in C. I have to save logs to file.
I write it into a file using fprintf, but when the server go down I lost the data in the file, cause I don't close filedescriptor, is there any function which tell my program save the data?
Thx
If you fflush after every fprintf it helps.
fflush should do what you want — it ensures all output is explicitly written to the file rather than e.g. being cached for later writing. So that moves the data out of user space.
sync can then be used — it causes all buffered file changes to be physically written (though per the spec it needn't block until the writes are complete, so you can be certain they've started but not that they've finished).