I want to create a non-sparse file of a given length (i.e. 2GB), but I want to check if that is possible before actually writing stuff to disk.
In other words I want to avoid getting ENOSPC (No space left on device) while writing. I'd prefer not to create a "test file" of size 2GB or things like that just to check that there is enough space left.
Is that possible?
Use posix_fallocate(3).
From the description:
The function posix_fallocate() ensures that disk space is allocated
for the file referred to by the descriptor fd for the bytes in the
range starting at offset and continuing for len bytes. After a
successful call to posix_fallocate(), subsequent writes to bytes in
the specified range are guaranteed not to fail because of lack of
disk space
You can use the statvfs function to determine how much free bytes (and inodes) a given filesystem has.
That should be enough for a quick check, but do remember that it's not a guarantee that you'll be able to write as much (or, for that matter, that writing more than that would have failed) - other applications could also be writing to (or deleting from) the same filesystem. So do continue to check for various write errors.
fallocate or posix_fallocate can be used to allocate (and deallocate) a large chunk. Probably a better option for your use-case. (Check the man page, there's a lot of options for space management that you might find interesting.)
Related
Consider SetFilePointer. The documentation on MSDN (nor learn.microsoft.com) does not explain if a forward seek constitutes sequential access or not - this has implications for applications' IO performance.
For example, if you use CreateFile with FILE_FLAG_RANDOM_ACCESS then Win32 will use a different buffering and caching strategy compared to FILE_FLAG_SEQUENTIAL_SCAN - if you're reading a file from start to finish then you can expect better performance than with the random-access option.
However, supposing the file format you're reading does not necessitate that every byte (or even buffer-page) be read into memory, such as a flag in the file's header indicates that the first 100 bytes - or 100 kilobytes - contains no useful data. Is it wise to call ReadFile to read the next 100 bytes (or 100 kilobytes - or more?) - or will it always be faster to call SetFilePointer( file, 100, NULL, FILE_CURRENT ) to skip-over those 100 bytes?
If it is generally faster to use SetFilePointer, does the random-access vs sequential option make a difference? I would think that seeking forward constitutes a form of random-access because you could seek forward beyond the currently cached buffer (and any future buffers that the OS might have pre-loaded for you behind the scenes) but in that case will Windows always discard the cached buffers and re-read from disk? Is there a way to find out the maximum amount one can seek-forward without triggering a buffer reload?
(I would try to profile and benchmark to test my hypothesis, but all my computers at-hand have NVMe SSDs - obviously things will be very different on platter drives).
at first about SetFilePointer.
SetFilePointer internally called ZwSetInformationFile with FilePositionInformation. it full handled by I/O manager - the file system is not even called. all what is done on this call : CurrentByteOffset from FILE_OBJECT is set to given position.
so this call absolute independent from file buffering and caching strategy. more - this is absolute senseless call, which only waste time - we always can set direct offset in call to ReadFile or WriteFile - look in OVERLAPPED Offset and OffsetHigh. SetEndOfFile ? but much more better and effective call ZwSetInformationFile with FileEndOfFileInformation or SetFileInformationByHandle with FileEndOfFileInfo (SetEndOfFile of course internally call ZwSetInformationFile with FileEndOfFileInformation and before it call ZwQueryInformationFile with FilePositionInformation for read CurrentByteOffset from FILE_OBJECT - so you simply do 2-3 unnecessary extra calls to kernel in case SetEndOfFile). not exist situation when call to SetFilePointer really need.
so file position - is only software variable (CurrentByteOffset in FILE_OBJECT) which used primary by I/O manager -
filesystem always get read/write request with explicit offset - in FastIoRead as in argument or in IO_STACK_LOCATION.Parameters.Read.ByteOffset
the I/O manager get this offset or from explicit ByteOffset value to NtReadFile or from CurrentByteOffset in FILE_OBJECT if ByteOffset not present (NULL pointer for ByteOffset)
ReadFile use NULL pointer for ByteOffset if NULL pointer for OVERLAPPED, otherwise use pointer to OVERLAPPED.Offset
about question - are exist sense sequential read all bytes or just read from needed offset ?
in case we open file without caching ( FILE_NO_INTERMEDIATE_BUFFERING) - we have no choice Offset and Length passed to ReadFile or WriteFile must be a multiple of the sector size
in case using cache - we anyway nothing gain if read some additional (and not needed to us bytes) before read actual needed bytes. in any case file system will be need read this bytes from disk, if it yet not read - reading another bytes does not accelerate this process.
with FILE_FLAG_SEQUENTIAL_SCAN cache manager read more sectors from disk than need for complete current request and next reading at sequential offset - will (as minimum partially) fall cache - so count of direct read from disk (most expensive operation) will be less. but when you need read file at specific offset - sequential read bytes before this offset not help in any way - anyway will be need read this bytes
in other words - you anyway need read required bytes (at specific offset) from file - and if you before this read some another bytes - this not increase performance. only diminishes
so if you need read file at some offset - just read at this offset. and not use SetFilePointer. use explicit offset on OVERLAPPED
I am looking at man page of fallocate and I do not understand the difference between these two. One seems to allocate blocks but without writing them, other seems to deallocate blocks without overwriting them. Either way, effect seems to be indistinguishable from user perspective. Please explain this me.
Zeroing file space
Specifying the FALLOC_FL_ZERO_RANGE flag (available since Linux 3.15)
in mode zeroes space in the byte range starting at offset and
continuing for len bytes. Within the specified range, blocks are
preallocated for the regions that span the holes in the file. After
a successful call, subsequent reads from this range will return
zeroes.
Zeroing is done within the filesystem preferably by converting the
range into unwritten extents. This approach means that the specified
range will not be physically zeroed out on the device (except for
partial blocks at the either end of the range), and I/O is
(otherwise) required only to update metadata.
If the FALLOC_FL_KEEP_SIZE flag is additionally specified in mode,
the behavior of the call is similar, but the file size will not be
changed even if offset+len is greater than the file size. This
behavior is the same as when preallocating space with
FALLOC_FL_KEEP_SIZE specified.
Not all filesystems support FALLOC_FL_ZERO_RANGE; if a filesystem
doesn't support the operation, an error is returned. The operation
is supported on at least the following filesystems:
* XFS (since Linux 3.15)
* ext4, for extent-based files (since Linux 3.15)
* SMB3 (since Linux 3.17)
Increasing file space
Specifying the FALLOC_FL_INSERT_RANGE flag (available since Linux
4.1) in mode increases the file space by inserting a hole within the
file size without overwriting any existing data. The hole will start
at offset and continue for len bytes. When inserting the hole inside
file, the contents of the file starting at offset will be shifted
upward (i.e., to a higher file offset) by len bytes. Inserting a
hole inside a file increases the file size by len bytes.
This mode has the same limitations as FALLOC_FL_COLLAPSE_RANGE
regarding the granularity of the operation. If the granularity
requirements are not met, fallocate() will fail with the error
EINVAL. If the offset is equal to or greater than the end of file,
an error is returned. For such operations (i.e., inserting a hole at
the end of file), ftruncate(2) should be used.
No other flags may be specified in mode in conjunction with
FALLOC_FL_INSERT_RANGE.
FALLOC_FL_INSERT_RANGE requires filesystem support. Filesystems that
support this operation include XFS (since Linux 4.1) and ext4 (since
Linux 4.2).
It depends on what the application wants w.r.t the disk space consumed by a file as a result of using either.
The FALLOC_FL_PUNCH_HOLE flag deallocates blocks. Since it has to be ORed with FALLOC_FL_KEEP_SIZE, what this means is you end up in a sparse file.
FALLOC_FL_ZERO_RANGE on the other hand, allocates blocks for the (offset, length) if not already present and zeroes it out. So in effect you are losing some of its sparseness if the file had holes to begin with. Also, it is a method of zeroing out regions of a file without the application manually having to write(2) zeroes.
All these flags to fallocate(2) are typically used by virtualization software like qemu.
I would like to be able to zero out a range of a file memory-mapping without invoking any io (in order to efficiently sequentially overwrite huge files without incurring any disk read io).
Doing std::memset(ptr, 0, length) will cause pages to be read from disk if they are not already in memory even if the entire pages are overwritten thus totally trashing disk performance.
I would like to be able to do something like madvise(ptr, length, MADV_ZERO) which would zero out the range (similar to FALLOC_FL_ZERO_RANGE) in order to cause zero fill page faults instead of regular io page faults when accessing the specified range.
Unfortunately MADV_ZERO does not exists. Even though the corresponding flag FALLOC_FL_ZERO_RANGE does exists in fallocate and can be used with fwrite to achieve a similar effect, though without instant cross process coherency.
One possible alternative I would guess is to use MADV_REMOVE. However, that can from my understanding cause file fragmentation and also blocks other operations while completing which makes me unsure of its long term performance implications. My experience with Windows is that the similar FSCTL_SET_ZERO_DATA command can incur significant performance spikes when invoked.
My question is how one could implement or emulate MADV_ZERO for shared mappings, preferably in user mode?
1. /dev/zero/
I have read it being suggested to simply read /dev/zero into the selected range. Though I am not quite sure what "reading into the range" means and how to do it. Is it like a fread from /dev/zero into the memory range? Not sure how that would avoid a regular page fault on access?
For Linux, simply read /dev/zero into the selected range. The
kernel already optimises this case for anonymous mappings.
If doing it in general turns out to be too hard to implement, I
propose MADV_ZERO should have this effect: exactly like reading
/dev/zero into the range, but always efficient.
EDIT: Following the thread a bit further it turns out that it will actually not work.
It does not do tricks when you are dealing with a shared mapping.
2. MADV_REMOVE
One guess of implementing it in Linux (i.e. not in user application which is what I would prefer) could be by simply copying and modifying MADV_REMOVE, i.e. madvise_remove to use FALLOC_FL_ZERO_RANGE instead of FALLOC_FL_PUNCH_HOLE. Though I am bit over my head in guessing this, especially as I don't quite understand what the code around the vfs_allocate is doing:
// madvice.c
static long madvise_remove(...)
...
/*
* Filesystem's fallocate may need to take i_mutex. We need to
* explicitly grab a reference because the vma (and hence the
* vma's reference to the file) can go away as soon as we drop
* mmap_sem.
*/
get_file(f); // Increment ref count.
up_read(¤t->mm->mmap_sem); // Release a read lock? Why?
error = vfs_fallocate(f,
FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE, // FALLOC_FL_ZERO_RANGE?
offset, end - start);
fput(f); // Decrement ref count.
down_read(¤t->mm->mmap_sem); // Acquire read lock. Why?
return error;
}
You probably cannot do what you want (in user space, without hacking the kernel). Notice that writing zero pages might not incur physical disk IO because of the page cache.
You might want to replace a file segment by a file hole (but this is not exactly what you want) in a sparse file, but some file systems (e.g. VFAT) don't have holes or sparse files. See lseek(2) with SEEK_HOLE, ftruncate(2)
I have to do a program (for Linux) where there's an extremely large index file and I have to search and interpret the data from the file. Now the catch is, I'm only allowed to have x-bytes of the file cached at any time (determined by argument) so I have to remove certain data from the cache if it's not what I'm looking for.
If my understanding is correct, fopen (r) doesn't put anything in the cache, only when I call getc or fread(specifying size) does it get cached.
So my question is, lets say I use fread and read 100 bytes but after checking it, only 20 of the 100 bytes contains the data I need; how would I remove the useless 80 bytes from cache (or overwrite it) in order to read more from the file.
EDIT By caching I mean data stored in memory, which makes the problem easier
fread's first argument is a pointer to a block of memory. So the way to go about this is to set that pointer to the stuff you want to over write. For example lets say you want to keep bytes 20-40 and overwrite everything else. You could either a) invoke fread on start with a length of 20 then invoke it again on buffer[40] with a size of 60. or b) You could start by defragmenting (ie copy the bytes you want to keep to the start) then invoke fread with a pointer to the next section.
Why do you want to micromanage the cache? Secondly, what makes you think you can? No argument specified on the command line of your program can control what the cache manager does internally - it may decide to read an entire file into RAM, it may decide to read none of it, or it may decide to throw a party. Any control you have over it would use low-level APIs/syscalls and would not very granular.
I think you might be confused about the requirements, or maybe the person who gave them to you. You seem to be referring to the cache managed by the operating system, which there is no need for an application to ever have to worry about. The operating system will make sure it doesn't grow too large automatically.
The other meaning of "cache" is the one you create yourself, the char* buffer or whatever you create to temporarily hold the data in memory while you process it. This one should be fairly easy to manage yourself simply by not allocating too much memory for that buffer.
To discard the read buffer of a file opened with fopen(), you can use fflush(). Also note that you can control the buffer size with setvbuf().
You should consider using open/read (instead of fopen/fread) if you must have exact control over buffering, though.
I am using the low-level I/O function 'write' to write some data to disk in my code (C language on Linux). First, I accumulate the data in a memory buffer, and then I use 'write' to write the data to disk when the buffer is full. So what's the best buffer size for 'write'? According to my tests it isn't the bigger the faster, so I am here to look for the answer.
There is probably some advantage in doing writes which are multiples of the filesystem block size, especially if you are updating a file in place. If you write less than a partial block to a file, the OS has to read the old block, combine in the new contents and then write it out. This doesn't necessarily happen if you rapidly write small pieces in sequence because the updates will be done on buffers in memory which are flushed later. Still, once in a while you could be triggering some inefficiency if you are not filling a block (and a properly aligned one: multiple of block size at an offset which is a multiple of the block size) with each write operation.
This issue of transfer size does not necessarily go away with mmap. If you map a file, and then memcpy some data into the map, you are making a page dirty. That page has to be flushed at some later time: it is indeterminate when. If you make another memcpy which touches the same page, that page could be clean now and you're making it dirty again. So it gets written twice. Page-aligned copies of multiples-of a page size will be the way to go.
You'll want it to be a multiple of the CPU page size, in order to use memory as efficiently as possible.
But ideally you want to use mmap instead, so that you never have to deal with buffers yourself.
You could use BUFSIZ defined in <stdio.h>
Otherwise, use a small multiple of the page size sysconf(_SC_PAGESIZE) (e.g. twice that value). Most Linux systems have 4Kbytes pages (which is often the same as or a small multiple of the filesystem block size).
As other replied, using the mmap(2) system call could help. GNU systems (e.g. Linux) have an extension: the second mode string of fopen may contain the latter m and when that happens, the GNU libc try to mmap.
If you deal with data nearly as large as your RAM (or half of it), you might want to also use madvise(2) to fine-tune performance of mmap.
See also this answer to a question quite similar to yours. (You could use 64Kbytes as a reasonable buffer size).
The "best" size depends a great deal on the underlying file system.
The stat and fstat calls fill in a data structure, struct stat, that includes the following field:
blksize_t st_blksize; /* blocksize for file system I/O */
The OS is responsible for filling this field with a "good size" for write() blocks. However, it's also important to call write() with memory that is "well aligned" (e.g., the result of malloc calls). The easiest way to get this to happen is to use the provided <stdio.h> stream interface (with FILE * objects).
Using mmap, as in other answers here, can also be very fast for many cases. Note that it's not well suited to some kinds of streams (e.g., sockets and pipes) though.
It depends on the amount of RAM, VM, etc. as well as the amount of data being written. The more general answer is to benchmark what buffer works best for the load you're dealing with, and use what works the best.