If I create a file and my OS starts that file at some memory address on some disk, but I end up writing so much data to my file that the file descriptor is moved into the disk space of the next file on the same disk, does the OS have a way of protecting the file that I am at risk of overwriting? And if so what does UNIX/Linux do in particular?
Typically the OS prevents contents of different files from overlapping by allowing the contents of a file to be stored in non-contiguous regions of disk. Growing a file past the free region will simply cause the file to include a non-adjacent region of disk. When the disk is sufficiently full, this causes the infamous fragmentation.
Note that user-space code is never allowed to directly access on-disk addresses of file contents. The file descriptor offset as returned by lseek is not an on-disk address, but a virtual offset that the OS can translated to the actual address. The fact that addresses are hidden from user code means that file contents may also be moved to other locations on disk to optimize storage.
Read up on filesystems for more information on this topic.
Related
I've been working with assembly and have been working with file IO. From what I've learned, the process goes as follows. CPU makes a system call to the kernel to open a file ie "hello.txt". The kernel then finds that location in the filesystem (persistent memory), makes it accessible for read and/or write, and returns a file descriptor that uniquely identifies that file. From my understanding the file descriptor is an index for a table that stores file data. My question is: what data is stored? presumably storing the entire file data would get grossly memory expense for large files. Does it store file metadata like mime-type, encoding, etc? Or does it actually store the whole contents?
I've always tought of pointers as being RAM adresses, pointing towards bytes of memory that can be random accessed. However, when we create a file in C, we use the pointer FILE*, that points towards the file but, after i close the program, isn't the created file saved in my HD? So, i see two possibilities here:
1) A pointer can points towards a HDD file
2) The file is saved in RAM (that doesn't make much sense to me)
Which one of it is true? Or, if there is a third possibility, what is it?
Thanks in advance.
As mention in How exactly does fopen(), fclose() work
When called, fopen allocates a FILE object on the heap. Note that the data in a FILE object is undocumented - FILE is an opaque struct, you can only use pointers-to-FILE from your code.
The FILE object gets initialized. For example, something like fillLevel = 0 where fillLevel is the amount of buffered data that hasn't been flushed yet.
A call to the filesystem driver (FS driver) opens the file and provides a handle to it, which is put somewhere in the FILE struct.
To do this, the FS driver figures out the HDD address corresponding to the requested path, and internally remembers this HDD address, so it can later fulfill calls to fread etc.
The FS driver uses a sort of indexing table (stored on the HDD) to figure out the HDD address corresponding to the requested path. This will differ a lot depending on the filesystem type - FAT32, NTFS and so on.
The FS driver relies on the HDD driver to perform the actual reads and writes to the HDD.
A cache might be allocated in RAM for the file. This way, if the user requests 1 byte to be read, C++ may read a KB just in case, so later reads will be instantaneous.
A pointer to the allocated FILE gets returned from fopen.
I am unable to understand how files are managed in memory mapped I/O. As normal If we open a file using open or fopen, it returns fd or
file pointer respectively. After this open where the file resides for processing. It is in memory(copy of the file which is in hard disk) or not? If it
is not in memory where the data is fetch by consequent read or write system call or It fetchs data from the hard disk for each time of calling read or write.
Otherwise the copy of the file is stored in memory and the file is accessed by process for furthur manipulation and once the process is completed the file is copied to hard disk. In the above concepts
which scenario is worked ?
The following is the definition given for memory mapped i/o in Advanced Programming in Unix Environment(2nd Edition) book:
Memory-mapped I/O lets us map a file on disk into a buffer in memory so that, when we fetch bytes from the buffer, the corresponding bytes of the file are read. Similarly, when we store data in the buffer, the corresponding bytes are automatically written to the file. This lets us perform I/O without using read or write.
what is mapping a file into memory? And here, they defined the memory is placed in between stack and heap. In this memory, what
type of data is present after mapping a file. It contains copy of the file or the address of the file which resides in hard disk. And
how the above scenario becomes true.
Does anyone explain the working mechanism of memory mapped I/O and mmap functionality?
Normally when you open a file, the system sets up some bookkeeping structures (metadata) but does not need to read any part of the actual data of the file. When you call read(), the system loads a chunk of the file into (virtual) memory which you allocated for the purpose.
When you memory-map a file, the system again sets up bookkeeping, and also sets up a (virtual) memory "mapping" which means a range of valid addresses which, if used, will reflect reads (or writes) of the underlying file. It does not mean the entire file needs to be read at once, because it can be "paged in" on demand, i.e. the system can give you an address range to use, then wait for you to actually use it before loading any data there. This "page faulting" is supported by a hardware device called the Memory Management Unit, or MMU. The same system is used when you run an executable file--the system can simply map it into virtual memory and read pages (chunks) from disk only as needed.
It is in memory(copy of the file which is in hard disk) or not?
According to Computer Programming and Utilization, When you open file with fopen its content are loaded into memory. (Partially or wholly).
If it is not in memory where the data is fetch by consequent read or
write system call
When you fwrite some data, it is eventually copied into the kernel which will then write it to disk (or wherever) after buffering. In general, no part of a file needs to be loaded in order to write.
what is mapping a file into memory?
For more refer here
In this memory, what type of data is present after mapping a file. It
contains copy of the file or the address of the file which resides in
hard disk.
A memory-mapped file is a segment of virtual memory which has been assigned a direct byte-for-byte correlation with some portion of a file or file-like resource.Refer this
It is possible to mmap a file to a region of memory. When this is done, the file can be accessed just like an array in the program.This is more efficient than read or write, as only the regions of the file that a program actually accesses are loaded. Accesses to not-yet-loaded parts of the mmapped region are handled in the same way as swapped out pages.
After this open where the file resides for processing. It is in memory(copy of the file which is in hard disk) or not?
On the disk. It may also be partly or completely in memory if the operating system does a read-ahead, but that isn't detectable by you. You still have to issue reads to get data from the file.
If it is not in memory where the data is fetch by consequent read or write system call
From the disk.
or It fetchs data from the hard disk for each time of calling read or write.
In effect, but you also have to consider the effect of any caching.
Otherwise the copy of the file is stored in memory and the file is accessed by process for furthur manipulation and once the process is completed the file is copied to hard disk.
No. The file behaves as though it is all on the disk.
And here, they defined the memory is placed in between stack and heap.
Not in what you quoted.
In this memory, what type of data is present after mapping a file.
The data in the file. The question 'what type of data' doesn't make sense. Data is data.
It contains copy of the file or the address of the file which resides in hard disk.
It effectively contains a copy of the file.
And how the above scenario becomes true.
Via virtual memory. Too broad to cover here.
I am working on a program which needs to load up to a few hundred images into memory at once. Each file takes up 100mb so I don't really want to be storing all of them in memory. I want to memory map the files so the operating system will swap them out when necessary to save physical memory. Here is what I am wondering. If I already have the data I want in the file in malloced memory should I open a file descriptor, write the data to the file using write() and then map the file. Or can I memory map a new file and then copy the data using memcpy. If I were to create a new file and when I call mmap give it a length large than the file size will it just increase the size of the file on the disk?
From the POSIX standard: “The mmap() function can be used to map a region of memory that is larger than the current size of the object. Memory access within the mapping but beyond the current end of the underlying objects may result in SIGBUS signals being sent to the process.” (http://pubs.opengroup.org/onlinepubs/9699919799/)
That said, you could try mmap() with MAP_FIXED over the same memory region you just wrote from, if you got it from a page-aligned aligned_alloc() rather than malloc(), or free and then mmap(). But note that the OS will page memory you haven’t used for a while to swap anyway, and you can help it out with posix_madvise().
I lack formal knowledge in Operating systems and C. My questions are as follows.
When I try to read first single byte of a file using fread in C, does the entire disk block containing that byte is brought into memory or just the byte?
If entire block is brought into memory, what happens on reading
second byte since the block containing that byte is already in
memory?.
Is there significance in reading the file in size of disk blocks?
Where is the read file block kept in memory?
Here's my answers
More than 1 block, default caching is 64k. setvbuffer can change that.
On the second read, there's no I/O. The data is read from the disk cache.
No, a file is ussuly smaller than it's disk space. You'll get an error reading past the file size even if you're within the actual disk space size.
It's part of the FILE structure. This is implementation (compiler) specific so don't touch it.
The above caching is used by the C runtime library not the OS. The OS may or may not have disk caching and is a separate mechanism.