I am unable to understand how files are managed in memory mapped I/O. As normal If we open a file using open or fopen, it returns fd or
file pointer respectively. After this open where the file resides for processing. It is in memory(copy of the file which is in hard disk) or not? If it
is not in memory where the data is fetch by consequent read or write system call or It fetchs data from the hard disk for each time of calling read or write.
Otherwise the copy of the file is stored in memory and the file is accessed by process for furthur manipulation and once the process is completed the file is copied to hard disk. In the above concepts
which scenario is worked ?
The following is the definition given for memory mapped i/o in Advanced Programming in Unix Environment(2nd Edition) book:
Memory-mapped I/O lets us map a file on disk into a buffer in memory so that, when we fetch bytes from the buffer, the corresponding bytes of the file are read. Similarly, when we store data in the buffer, the corresponding bytes are automatically written to the file. This lets us perform I/O without using read or write.
what is mapping a file into memory? And here, they defined the memory is placed in between stack and heap. In this memory, what
type of data is present after mapping a file. It contains copy of the file or the address of the file which resides in hard disk. And
how the above scenario becomes true.
Does anyone explain the working mechanism of memory mapped I/O and mmap functionality?
Normally when you open a file, the system sets up some bookkeeping structures (metadata) but does not need to read any part of the actual data of the file. When you call read(), the system loads a chunk of the file into (virtual) memory which you allocated for the purpose.
When you memory-map a file, the system again sets up bookkeeping, and also sets up a (virtual) memory "mapping" which means a range of valid addresses which, if used, will reflect reads (or writes) of the underlying file. It does not mean the entire file needs to be read at once, because it can be "paged in" on demand, i.e. the system can give you an address range to use, then wait for you to actually use it before loading any data there. This "page faulting" is supported by a hardware device called the Memory Management Unit, or MMU. The same system is used when you run an executable file--the system can simply map it into virtual memory and read pages (chunks) from disk only as needed.
It is in memory(copy of the file which is in hard disk) or not?
According to Computer Programming and Utilization, When you open file with fopen its content are loaded into memory. (Partially or wholly).
If it is not in memory where the data is fetch by consequent read or
write system call
When you fwrite some data, it is eventually copied into the kernel which will then write it to disk (or wherever) after buffering. In general, no part of a file needs to be loaded in order to write.
what is mapping a file into memory?
For more refer here
In this memory, what type of data is present after mapping a file. It
contains copy of the file or the address of the file which resides in
hard disk.
A memory-mapped file is a segment of virtual memory which has been assigned a direct byte-for-byte correlation with some portion of a file or file-like resource.Refer this
It is possible to mmap a file to a region of memory. When this is done, the file can be accessed just like an array in the program.This is more efficient than read or write, as only the regions of the file that a program actually accesses are loaded. Accesses to not-yet-loaded parts of the mmapped region are handled in the same way as swapped out pages.
After this open where the file resides for processing. It is in memory(copy of the file which is in hard disk) or not?
On the disk. It may also be partly or completely in memory if the operating system does a read-ahead, but that isn't detectable by you. You still have to issue reads to get data from the file.
If it is not in memory where the data is fetch by consequent read or write system call
From the disk.
or It fetchs data from the hard disk for each time of calling read or write.
In effect, but you also have to consider the effect of any caching.
Otherwise the copy of the file is stored in memory and the file is accessed by process for furthur manipulation and once the process is completed the file is copied to hard disk.
No. The file behaves as though it is all on the disk.
And here, they defined the memory is placed in between stack and heap.
Not in what you quoted.
In this memory, what type of data is present after mapping a file.
The data in the file. The question 'what type of data' doesn't make sense. Data is data.
It contains copy of the file or the address of the file which resides in hard disk.
It effectively contains a copy of the file.
And how the above scenario becomes true.
Via virtual memory. Too broad to cover here.
Related
I've been working with assembly and have been working with file IO. From what I've learned, the process goes as follows. CPU makes a system call to the kernel to open a file ie "hello.txt". The kernel then finds that location in the filesystem (persistent memory), makes it accessible for read and/or write, and returns a file descriptor that uniquely identifies that file. From my understanding the file descriptor is an index for a table that stores file data. My question is: what data is stored? presumably storing the entire file data would get grossly memory expense for large files. Does it store file metadata like mime-type, encoding, etc? Or does it actually store the whole contents?
I've always tought of pointers as being RAM adresses, pointing towards bytes of memory that can be random accessed. However, when we create a file in C, we use the pointer FILE*, that points towards the file but, after i close the program, isn't the created file saved in my HD? So, i see two possibilities here:
1) A pointer can points towards a HDD file
2) The file is saved in RAM (that doesn't make much sense to me)
Which one of it is true? Or, if there is a third possibility, what is it?
Thanks in advance.
As mention in How exactly does fopen(), fclose() work
When called, fopen allocates a FILE object on the heap. Note that the data in a FILE object is undocumented - FILE is an opaque struct, you can only use pointers-to-FILE from your code.
The FILE object gets initialized. For example, something like fillLevel = 0 where fillLevel is the amount of buffered data that hasn't been flushed yet.
A call to the filesystem driver (FS driver) opens the file and provides a handle to it, which is put somewhere in the FILE struct.
To do this, the FS driver figures out the HDD address corresponding to the requested path, and internally remembers this HDD address, so it can later fulfill calls to fread etc.
The FS driver uses a sort of indexing table (stored on the HDD) to figure out the HDD address corresponding to the requested path. This will differ a lot depending on the filesystem type - FAT32, NTFS and so on.
The FS driver relies on the HDD driver to perform the actual reads and writes to the HDD.
A cache might be allocated in RAM for the file. This way, if the user requests 1 byte to be read, C++ may read a KB just in case, so later reads will be instantaneous.
A pointer to the allocated FILE gets returned from fopen.
If I create a file and my OS starts that file at some memory address on some disk, but I end up writing so much data to my file that the file descriptor is moved into the disk space of the next file on the same disk, does the OS have a way of protecting the file that I am at risk of overwriting? And if so what does UNIX/Linux do in particular?
Typically the OS prevents contents of different files from overlapping by allowing the contents of a file to be stored in non-contiguous regions of disk. Growing a file past the free region will simply cause the file to include a non-adjacent region of disk. When the disk is sufficiently full, this causes the infamous fragmentation.
Note that user-space code is never allowed to directly access on-disk addresses of file contents. The file descriptor offset as returned by lseek is not an on-disk address, but a virtual offset that the OS can translated to the actual address. The fact that addresses are hidden from user code means that file contents may also be moved to other locations on disk to optimize storage.
Read up on filesystems for more information on this topic.
I Understand the usage of the mmap. Considering simple read/write operation on the file, involves, opening the file, and allocating the buffer, read [ which requires context switch, ], and then the data available to the user in the buffer, and changes in the buffer will not reflect into the file unless it is written explictly.
Instead , if we use mmap, writting directly to the buffer is nothing but writting into the file.
The Question:
1) File is in the hard disk, mmaped into the process, Each time i write into mmaped memory, is it written directly to the file?. In this case, does not it require any context switch, because, the changes are done directly into the file itself. If mmap is faster than legacy file accessing, where we see the time saving?
Kindly explain. correct me if i m wrong also.
Updates to the file are not immediately visible in the disk, but are visible after an unmap or following an msync call. Hence, there is no system call during the updates, and the kernel is not involved. However, since the file is lazily read page by page, as needed, OS may need to read-in portions of the file as you cross page boundaries. Most obvious advantage of memory mapping is that it eliminates kernel-space to user-space data copies. There is also no need for system calls to seek to a specific position in a file.
I am reading about Memory Mapped Files, the souce says it is faster than traditional methods to open a file or read a file such as an open system call and read system call respectively without giving the description that how open or read system call works.
So here's my question how the open system call works?
As far i know it will load the file into the memory, whereas by using mapped file only their addresses will be saved in the memory and when needed the requested page may be brought into the memory.
I expect clarification over my so far understanding.
EDIT
My previous understanding written above is almost wrong, for coorrect explanation refer to the accepted answer by Pawel.
Since you gave no details I'm assuming you are interested in behavior of Unix-like systems.
Actually open() system call only creates a file descriptor which then may be used by either mmap() or read().
Both memory mapped I/O and standard I/O internally access files on disk through page cache, a buffer in which files are cached in order to reduce number of I/O operations.
Standard I/O approach (using write() and read()) involves performing a system call which then copies data from (or to if you are writing) page cache to a buffer chosen by application. In addition to that non-sequential access requires another system call lseek(). System calls are expensive and so is copying data.
When a file is memory mapped usually a memory region in process address space is mapped directly to page cache, so that all reads and writes of already loaded data can be performed without any additional delay (no system calls, no data copying). Only when an application attempts to access file region that is not already loaded a page fault occurs and the kernel loads required data (whole page) from disk.
EDIT:
I see that I also have to explain memory paging. On most modern architectures there is physical memory which is a real piece of hardware and virtual memory which creates address spaces for processes. Kernel decides how addresses in virtual memory are mapped to addresses in physical memory. The smallest unit is a memory page (usually, but not always 4K). It does not have to be 1:1 mapping, for example all virtual memory pages may be mapped to the same physical address.
In memory mapped I/O part of application address space and kernel's page cache are mapped to the same physical memory region, hence program is able to directly access page cache.
Pawel has beautifully explained how read/writes are performed. Let me explain the original question: How does fopen(3) works:
when user space process encounters fopen(defined in libc or any user space library), it translates it into open(2) system call. First, it takes arguments from fopen, writes them into architecture specific registers along with open() syscall number. This number tells kernel the system call user space program wants to run. After loading these register, user space process interrupts kernel(via softirq, traditionally INT 80H on x86) and blocks.
Kernel verifies the arguments provided and access permissions etc, and then either returns error or invokes actual system call which is vfs_open() in this case. vfs_open() checks for available file descriptor in fd array and allocates struct file. The ref counts of accessed file is increased and fd is returned to user program. That's completes the working of open, and of most of the system calls in general.
open() together with read()/write(), followed by close() is undoubtedly much lengthy process than having memory mapped file in buffer cache.
For a lucid explanation of how open and read work on Linux, you can read this. The code snippets are from an older version of the kernel but the theory still holds.
You would still need to use the open() system call to get a valid file descriptor, which you would pass on to mmap(). As to why mmaped IO is faster, it is because there is no copy of data from (to) user space to (from) kernel space buffers which is what happens with read and write system calls.