There's no way in any operating system I'm aware of for a program to prepend data to a file efficiently.
And yet, this doesn't seem difficult -- the file system can add another extent to a file descriptor if needed.
So the question is, why don't operating systems implement this (rather trivial) operation?
I don't think it's as easy as you suggest. It's true that the file-system could allocate a new block, store the prepended data in it, change the file pointer to point to that block and then chain the rest of the file from that block. Just like adding a node to the front of a linked list, right?
But what happens when (as is probably the case) the prepended data doesn't fill the assigned block. I don't imagine that many filesystems would have a machanism for chaining partial blocks, but even if they do it would result in huge inefficiencies. You'd end up with a file consisting of mostly empty blocks, and you have to have to read and write the entire file to defragment it. Might as well just do the read-and-write operation up front when you're prepending in the first place.
In prepending or appending data to a file, there is always an issue of allocating space. Appending additional space to a file is much easier than prepending because of file descriptors pointing to the beginning of a file's stream. If you want to append to a file, the file descriptor need not be changed, just the size of the file and the allocated memory. If you want to prepend to a file, a new file descriptor must be immediately instantiated, either to write the prepended data to first or to store the location of the data being prepended while the original is updated.
Making a new file descriptor can be tricky, as you must also update any references pointing to it. This is why it is easy for an OS to implement appending data and slightly harder to implement prepending.
Related
I'm learning about in-kernel data transferring between two file descriptors in Linux and came across something I cannot understand. Here is the quote from copy_file_range manpage
copy_file_range() gives filesystems an opportunity to implement "copy
acceleration" techniques, such as the use of reflinks (i.e., two or
more i-nodes that share pointers to the same copy-on-write disk
blocks) or server-side-copy
I used to think of index nodes as something that is returned by stat/statx syscall. The st_ino type is typedefed here as
typedef unsigned long __kernel_ulong_t;
So what does it ever mean "two or more i-nodes that share pointers to the same copy-on-write disk blocks"?
According to my understanding the fact that copy_file_range do not need to pass the data through the user-mode means the kernel doesn't have to load the data from the disk at all (it still might but it doesn't have to) and this allows further optimization by pushing the operation down the file-system stack. This covers the case of the server-side-copy over NFS.
The actual answers about the other optimization starts with an intro into how files are stored, you may skip it if you already know that.
There are 3 layers in how files are stored in a typical Linux FS:
The file entry in some directory (which is itself a file containing a list of such entries). Such entry essentially maps file name to some inode. It is done by storing the inode-number aka st_ino which is effectively a pointer to the inode in some table.
The inode that contains some shared (see further) metadata (as the one returned by stat) and some pointer(s) to data block(s) that store the actual file contents.
The actual data blocks
So for example a hard-link is a record in some directory that points to the same inode as the "original" file (and incrementing the "link counter" inside the inode). This means that only file names (and possibly directories) are different, all the rest of the data and meta-data is shared between hard-links. Note that creating a hard link is a very fast way to copy a file. The only drawback is that both files now are bound to share their contents forever so this is not a true copy. But if we used some copy-on-write method to fix the "write" part, it would work very nice. This is what some FSes (such as Btrfs) support via reflinks.
The idea of this copy-on-wrote trick is that you can create a new inode with new appropriate metadata but still share the same data blocks. You also add cross-references between the two inodes in the "invisible" part of the inode metadata so they know they share the data blocks. Obviously this operation is very fast comparing to the real copying. And again as long as the files are only read, everything works perfectly. But unlike hard-link we can deal with writes treating them as independent as well. When some write is performed, the FS checks if the file (or rather the inode) is really the only owner of the data blocks and else copies the data before writing to it. Depending on the FS implementation it can copy the whole file on the first write or it can store some more detailed metadata and only copy the blocks that have to be modified and still share the rest between the files. In the later case blocks might not need to be copied at all if the write size is more than a block.
So the simplest trick copy_file_range() can do is to check if the whole file is actually being copied and if so, to perform the reflink trick described above (obviously if the FS supports it).
Some more advanced optimizations are also possible if the FS supports more detailed meta-data on data blocks. Assume you copy first N bytes from the start of the file into a new file. Then the FS can just share the starting blocks and probably has to copy only the last one that is not fully copied.
Why, in C, do you need a separate buffer to read a FILE *? When you declare a FILE * and assign to it with fopen, does the file then not exist in contiguous memory starting at the address of said pointer? I'm struggling to make the connection as to why you need need to read via fread() into a separate buffer. If someone could explain how theFIlE *file = fopen(filename, "r") and the subsequent fread(&buffer,...) work in conjunction it would help my understanding tremendously. Thanks in advance.
The FILE * returned by fopen is an unnecessary, but useful, layer of indirection.
Theoretically, fopen could have been designed to read the whole file into a buffer in memory, and just return you that buffer.
The issue approach is that it's not flexible at all. It forces you to read the entire file for all file IO operations, which is very undesirable. For example, here are some problems that would come about:
How could you read a file that's too big to fit in RAM?
What if you just want to append a new line at the end of a file (such as for logging). You would have to read the whole file, append the line at the end, then rewrite the entire file back. Expensive!
What if you're only interested in reading a small part of a file, such as reading the magic number to identify the file's type, without regard for its actual content?
What if you wanted to simultaneously edit the file from multiple programs. Each program need to constantly reread the whole file into memory, to ensure it kept up-to-date.
fopen returns a file handle that identifies a file still on disk. How much you read out of this file into memory is entirely up to you.
The explanation given above is pretty much self explanatory , still I would try to make it simple (in case anyone has problem understanding it)
In Short,consider this example and you yourself would know 'why?'
1) your files might be too large and stored in your hard drive , then if you try reading it frequently, don't you think this is an overhead for loading whole file again and again.
2) And more worse say the file is huge then if you load whole of your file it consumes your RAM even if you don't need whole file at once.
Why, in C, do you need a separate buffer to read a FILE *?
First thing, Because reading into buffers and then using it is always faster.
does the file then not exist in contiguous memory starting at the
address of said pointer?
May or May not be ,depending on its size.
If someone could explain how theFIlE *file = fopen(filename, "r") and
the subsequent fread(&buffer,...) work in conjunction
The fopen() function is used to open a file and associates an I/O stream with it. This function takes two arguments. The first argument is a pointer to a string containing name of the file to be opened while the second argument is the mode in which the file is to be opened.
Various modes can be like r, r+, w, w+, a, a+ .
The fopen() function returns a FILE stream pointer on success while it returns NULL in case of a failure.
Look here for detailed info.
Why, in C, do you need a separate buffer to read a FILE *?
No buffer aren't necessary while they are usually present to accelerate I/Os.
When you declare a FILE * and assign to it with fopen, does the file
then not exist in contiguous memory starting at the address of said
pointer?
Certainly not, 'cause this would be at least inefficient (why read a entire file huge if it is not needed at the end?) and at worst impossible at all (RAM size is usually much less than DISK size).
If someone could explain how the FILE *file = fopen(filename, "r") and
the subsequent fread(&buffer,...) work in conjunction it would help my
understanding tremendously.
Then FILE * is not an handle to a memory object that contains the file data, but is a memory object that contains data to help accessing file data on disk. That opaque object (opaque means don't try to look inside details are hidden) contains for example the current offset (remember when you read or write this is done at a given offset and this would modify the offset accordingly), or the open mode (this way writing into an opened for read file will correctly fails), or some buffer (that may contains part of the file and sometimes the whole file!), etc. A FILE * is handle as a handle for a door. Don't confuse file and FILE*, the first is a generic term to embrace what you already know (data on disk), then second is a type to represent an opened file which is a dynamic object to represent manipulation of a given file.
I'm struggling to make the connection as to why you need need to read
via fread() into a separate buffer.
If you don't have/can't have the file in memory, then you need to ask for reading the part you are interested in.
Prepending to a large file is difficult, since it requires pushing all
other characters forward. However, could it be done by manipulating
the inode as follows?:
Allocate a new block on disk and fill with your prepend data.
Tweak the inode to tell it your new block is now the first
block, and to bump the former first block to the second block
position, former second block to the third position, and so on.
I realize this still requires bumping blocks forward, but it should be
more efficient than having to use a temp file.
I also realize the new first block will be a "short" block (not all the data in the block is part of the file), since your prepend data is unlikely to be exactly the same size as a block.
Or, if inode blocks are simply linked, it would require very little
work to do the above.
NOTE: my last experience directly manipulating disk data was with a
Commodore 1541, so my knowledge may be a bit out of date...
Modern-day operating systems should not allow a user to do that, as inode data structures are specific to the underlying file system.
If your file system/operating system supports it, you could make your file a sparse file by prepending empty data at the beginning, and then writing to the sparse blocks. In theory, this should give you what you want.
YMMV, I'm just throwing around ideas. ;)
This could work! Yes, userland programs should not be mucking around with inodes. Yes, it necessarily depends on whatever scheme used to track blocks by whatever file systems implement this function. None of this is a reason to reject this proposal out of hand.
Here is how it could work.
For the sake of illustration, suppose we have an inode that tracks blocks by an array of direct pointers to data blocks. Further suppose that the inode carries a starting-offset and and ending-offset that apply to the first and last blocks respectively, so you can have less-than-full blocks both at the beginning and end of a file.
Now, suppose you want to prepend data. It would go something like this.
IF (new data will fit into unused space in first data block)
write the new data to the beginning of the first data block
update the starting-offset
return success indication to caller
try to allocate a new data block
IF (block allocation failed)
return failure indication to caller
shift all existing data block pointers down by one
write the ID of the newly-allocated data block into the first slot of the array
write as much data as will fit into the second block (the old first block)
write the rest of data into the newly-allocated data block, shifted to the end
starting-offset := (data block size - length of data in first block)
return success indication to caller
I need two file pointers (FILE *) to operate alongside each other. One is to apply append operations and another is for reading and overwriting.
I need appends to the file from one pointer to be recognised by the other file pointer so that the other file pointer can both correctly read and overwrite this appended data.
To synchronise the data, it appears that using fflush() on the appending file pointer works (at least for reading it does), but is this the correct way to achieve what I want and is it portable?
Thank you.
You should be able to do that with one pointer (and thus not having to sync unnecessarily). Just use fseek(f, SEEK_END, 0); when you want to add at the end. Use "rb+" to make the file readable and writeable.
As long as you don't use multiple threads to access the file, this should work just fine.
In C, when opening a file with
FILE *fin;
fin=fopen("file.bin","rb");
I only have a pointer to a structure of FILE. Where is the actual FILE struct allocated on Windows machine? And does it contain all the necessary information for accessing the file?
My aim is to dump the whole data segment to disk and then to reload the dumped file back to the beginning of the data segment. The code that reloads the dumped file is placed in a separate function. This way, the fin pointer is local and is on the stack, thus is not being overwritten on reload. But the FILE struct itself is not local. I take care not to overwrite the memory region of size sizeof(FILE) that starts at the address fin.
The
fread(DataSegStart,1,szTillFin,fin);
fread(dummy,1,sizeof(FILE),fin);
fread(DataSegAfterFin,1,szFinTillEnd,fin);
operations completes successfully, but I get an assertion failure on
fclose(fin)
Do I overwrite some other necessary file data other than in the FILE struct?
The actual instance of the FILE structure exists within the standard library. Typically the standard library allocates some number of FILE structures, which may or may not be a fixed number of them. When you call fopen(), it returns a pointer to one of those structures.
The data within the FILE structure likely contains pointers to other things such as buffers. You're unlikely to be able to save and restore those structures to disk without some really deep integration with your standard library implementation.
You may be interested in something like CryoPID which does process save and restore at a different level.
It seems like you're trying to do something dangerous, unlikely to work.
fopen allocates a FILE structure and initializes it. fclose releases it. How it allocates it and what it puts in it is implementation dependent. It could contain a pointer to another piece of memory, which is also allocated somewhere (since it's buffered I/O, I guess it does allocate a buffer somewhere).
Writing code that relies on the internals of fopen is dangerous, most likely won't work, and surely won't be stable and portable.
Well, you have a pointer to a FILE object, so technically you know where it is but you should be aware that FILE is deliberately an opaque type. You shouldn't need to know what it contains, you just need to know that you can pass it to functions that know about it to perform certain actions. Additionally, FILE may not be a complete type so sizeof(FILE) might not be correct and, additionally, the object might contain pointers to other structures. Simply avoiding overwriting the FILE object is not likely to be sufficient for you to avoid corrupting the program by writing over most of its memory.
FILE is defined in stdio.h. It contains all the information about the file but, looking at the code you show, I think you don't understand its purpose. It is created and run through the operating system with the C library which fills FILE with information about the file but it is not contained in the file itself.