I have some embedded code, that is writing data to a file in one task and reading data from the same file in another task. The writer is faster than the reader, so the reader will first reach end-of-file after the writer has completed.
WRITER TASK:
fseek(f, 0, SEEK_END);
fwrite(...)
READER TASK:
fseek(f, rdpos, SEEK_SET);
rdpos += fread(...)
Currently the code logic works, but it is too slow.
The fseek calls is taking too long time, especially when the file gets large.
Is there another way of writing and reading a file simultaneously ?
EDIT: I can see this question is probably needs some more info.
The system is a ESP32, so the memory is relative limited, and the file is located on an sdcard and the file size is up to 100Mbyte.
I have tried to open the file two times, like:
fwr = fopen(filename, "w+");
...
frd = fopen(filename, "r");
Then fwrite(..., fwr) and fread(..., frd), but the read data is bogus.
EDIT2:
The data being stored is audio, we want the audio to be stored fast, but simultaneously being able to start playing the audio.
The audio output codec has a limited buffer, so to prevent this buffer from running empty, we need a low read time. The isolated write and read times are ok, but when during seek, the time gets too high.
EDIT3:
The write speed is about 250Kbyte/s, and the read speed is 16Kbyte/s.
Related
I ran my code through a profiler and saw most of the time (60%) is spent reading files. It only takes a few milliseconds to run but I was wondering if I can make it any faster. My code has a list of 20 files (from 1k to 1M). It opens one, reads the entire file into ram, process it (sequential, reads everything once), then repeats open/read/process/close for the rest of the files reusing the same buffer
I was wondering if there's a way to make anything faster? I tried using posix_fadvise with POSIX_FADV_SEQUENTIAL and POSIX_FADV_WILLNEED, using the file offset len as 0, 0 and 0, st_size. It didn't seem to make a difference. I haven't yet written code to open all the files before reading. Would that make a difference? Should I be using posix_fadvise on all of them? Should I be using POSIX_FADV_SEQUENTIAL or POSIX_FADV_WILLNEED?
fadvise only really helps if:
It's issued well before you begin reading, or
You're reading the file piecemeal (ideally with some processing gaps between the reads).
If you're just slurping the whole file into RAM up front immediately after opening it, there's not much to be optimized; the file has to be read from beginning to end, and you haven't given the OS enough warning to cache it. Things to consider:
Opening and fadviseing file n+1 just before you begin reading from file n (so the OS is caching the next file in while you're processing the current file)
Using mmap+madvise(WILLNEED) to avoid the need to copy the file from kernel to user buffers all at once before you can begin processing; if the processing of the file is expensive enough, the subsequent pages may be read in by the time you've finished processing the early pages in the file.
Given these are small files, I'd just stick with WILLNEED; SEQUENTIAL enlarges the read-ahead buffer, but you're going to read the whole file anyway (possibly in bulk, where SEQUENTIAL won't help much) so you may as well cache the whole thing in as quickly as possible.
In my opinion, this code, extracted from the first edition of "The C programming language" its quite difficult to superate:
#include <stdio.h>
main()
{
int c;
while((c = getchar()) != EOF)
putchar(c);
}
Is there a way to open a file and write any string to the end of the file without using the O_APPEND (append) option when opening the file?
I'm coding in C in a Unix environment for the first time for class.
I know I could use lseek(fd, 0, SEEK_END) to seek the end of the file and fstat() to get the file size, but overall, I'm not sure what my code should be like.
What i have is
int fwrite = open (“abc.txt”, O_RDWR);
int fread = open
Also, this is my first time on Stack Overflow, please guide me
Yes, but no. O_APPEND has a special attribute that permits multiple processes to write to the file in non-interfering mode. So, if N independent processes open a file O_APPEND, the writes might be interleaved, but will be coherent. This is exploited in log files as an example.
If you open the file, lseek to the end, by the time you write(), the end point might have changed and you are over-writing valid data.
In short, if you need to append, use O_APPEND; if you want random access don't. The same program can open the same file in different modes.
Consider the following code:
lseek(fd, 100, 0); /* Seek to the 100th byte in the file fd. */
write(fd, buf, n); /* Write from that position. */
lseek(fd, 0, 0); /* Is this necessary? Will it trigger a actual disk movement? */
I'd like to lseek back to the beginning of the file, in case another line of code continues writing from that position thinking that it starts at the beginning of the file. First, is this good practice? Second...
I'd like to know if an lseek does trigger an actual disk movement. Or, is the disk movement triggered only in the event of an actual reading or writing.
Disk seeking is a huge performance hit, and I'd like to know the tradeoffs between such defensive coding practices and performance.
Assuming that this is a Windows or Unix type system, a regular file and you did nothing fancy with file open flags, none of those functions will trigger a disk seek.
It is likely that in 5 seconds or so, the buffer containing that new file data will be written to disk, along with everything else that happened.
Also, the file position that lseek sets is an entirely imaginary property of a file. It controls where data will read or write to in the file by default, but there are many functions that simply override file position.
As to if it is good practice I don't think it matters much. However, I've gotten out of the habit of using seek functions when writing to files because of multithreading. You might want to use pread and pwrite by preference.
I use my C program to stream binary data to ImageMagick:
inbuf = popen(string, "wb");
setbuf(inbuf, NULL); /// !!! ///
fwrite(buffer, frame, 1, inbuf);
pclose(inbuf);
And ImageMagick doesn't always receive all data on windows (mingw). Without the setbuf command (disabling bufferization) it receives even less data and problem appears on Linux (gcc) as well.
When I dump just the same buffer to file everything works fine and all data is written to disk and I don't even have to disable buffering:
outbuf = fopen("temp\\tune.gray", "wb");
fwrite(buffer, frame, 1, outbuf);
fclose(outbuf);
I discovered that problem occurs when I send odd number of bytes :) When I send even number everything works fine. I tried to write data not in bulk mode but splitting buffer to smaller portions, tried even sending data byte-by-byte - it doesn't help. Any ideas?
fwrite returns the total number of bytes successfully written. You can write a loop which tests how many bytes are actually written, and proceed to next write operation (from the location which was successfully written) until the entire buffer is successfully written.
My application (C program) opens two file handles to the same file (one in write and one in read mode). Two separate threads in the app read from and write to the file. This works fine.
Since my app runs on embedded device with a limited ram disk size, I would like write FileHandle to wrap to beginning of file on reaching max size and the read FileHandle to follow like a circular buffer. I understand from answers to this question that this should work. However as soon as I do fseek of write FileHandle to beginning of file, fread returns error. Will the EOF get reset on doing fseek to beginning of file? If so, which function should be used to cause write file position to get set to 0 without causing EOF to be reset.
EDIT/UPDATE:
I tried couple of things:
Based on #neodelphi I used pipes this works. However my usecase requires I write to a file. I receive multiple channels of live video surveilance stream that needs to be stored to harddisk and also read back decoded and displayed on monitor.
Thanks to #Clement suggestions on doing ftell I fixed a couple of bugs in my code and wrap works for the reader however, the data read appears to be stale data since write are still buffered but reader reads stale content from hard disk. I cant avoid buffering due to performance considerations (I get 32Mbps of live data that needs to be written to harddisk). I have tried things like flushing writes only in the interval from when write wraps to when read wraps and truncating the file (ftruncate) after read wraps but this doesnt solve the stale data problem.
I am trying to use two files in ping-pong fashion to see if this solves the issue but want to know if there is a better solution
You should have something like that :
// Write
if(ftell(WriteHandle)>BUFFER_MAX) rewind (WriteHandle);
fwrite(WriteHandle,/* ... */);
// Read (assuming binary)
readSize = fread (buffer,1,READ_CHUNK_SIZE,ReadHandle);
if(readSize!=READ_CHUNK_SIZE){
rewind (ReadHandle);
if(fread (buffer+readSize,1,READ_CHUNK_SIZE-readSize,ReadHandle)!=READ_CHUNK_SIZE-readSize)
;// ERROR !
}
Not tested, but it gives an idea. The write should also handle the case BUFFER_MAX is not modulo WRITE_CHUNK_SIZE.
Also, you may read only if you are sure that the data has already been written. But I guess you already do that.
You could mmap the file into you're virtual memory and then just create a normal circular buffer with the pointer returned.
int fd = open(path, O_RDWR);
volatile void * mem = mmap(NULL, max_size, PROT_WRITE, MAP_SHARED, fd, 0);
volatile char * c_mem = (volatile char *)mem;
c_mem[index % max_size] = 'a'; // This line will now write to the offset index in the file
// Now doing
Can also probably be stricter on permissions depending on on exact case.