I am working on a program which needs to load up to a few hundred images into memory at once. Each file takes up 100mb so I don't really want to be storing all of them in memory. I want to memory map the files so the operating system will swap them out when necessary to save physical memory. Here is what I am wondering. If I already have the data I want in the file in malloced memory should I open a file descriptor, write the data to the file using write() and then map the file. Or can I memory map a new file and then copy the data using memcpy. If I were to create a new file and when I call mmap give it a length large than the file size will it just increase the size of the file on the disk?
From the POSIX standard: “The mmap() function can be used to map a region of memory that is larger than the current size of the object. Memory access within the mapping but beyond the current end of the underlying objects may result in SIGBUS signals being sent to the process.” (http://pubs.opengroup.org/onlinepubs/9699919799/)
That said, you could try mmap() with MAP_FIXED over the same memory region you just wrote from, if you got it from a page-aligned aligned_alloc() rather than malloc(), or free and then mmap(). But note that the OS will page memory you haven’t used for a while to swap anyway, and you can help it out with posix_madvise().
Related
I've always tought of pointers as being RAM adresses, pointing towards bytes of memory that can be random accessed. However, when we create a file in C, we use the pointer FILE*, that points towards the file but, after i close the program, isn't the created file saved in my HD? So, i see two possibilities here:
1) A pointer can points towards a HDD file
2) The file is saved in RAM (that doesn't make much sense to me)
Which one of it is true? Or, if there is a third possibility, what is it?
Thanks in advance.
As mention in How exactly does fopen(), fclose() work
When called, fopen allocates a FILE object on the heap. Note that the data in a FILE object is undocumented - FILE is an opaque struct, you can only use pointers-to-FILE from your code.
The FILE object gets initialized. For example, something like fillLevel = 0 where fillLevel is the amount of buffered data that hasn't been flushed yet.
A call to the filesystem driver (FS driver) opens the file and provides a handle to it, which is put somewhere in the FILE struct.
To do this, the FS driver figures out the HDD address corresponding to the requested path, and internally remembers this HDD address, so it can later fulfill calls to fread etc.
The FS driver uses a sort of indexing table (stored on the HDD) to figure out the HDD address corresponding to the requested path. This will differ a lot depending on the filesystem type - FAT32, NTFS and so on.
The FS driver relies on the HDD driver to perform the actual reads and writes to the HDD.
A cache might be allocated in RAM for the file. This way, if the user requests 1 byte to be read, C++ may read a KB just in case, so later reads will be instantaneous.
A pointer to the allocated FILE gets returned from fopen.
I am unable to understand how files are managed in memory mapped I/O. As normal If we open a file using open or fopen, it returns fd or
file pointer respectively. After this open where the file resides for processing. It is in memory(copy of the file which is in hard disk) or not? If it
is not in memory where the data is fetch by consequent read or write system call or It fetchs data from the hard disk for each time of calling read or write.
Otherwise the copy of the file is stored in memory and the file is accessed by process for furthur manipulation and once the process is completed the file is copied to hard disk. In the above concepts
which scenario is worked ?
The following is the definition given for memory mapped i/o in Advanced Programming in Unix Environment(2nd Edition) book:
Memory-mapped I/O lets us map a file on disk into a buffer in memory so that, when we fetch bytes from the buffer, the corresponding bytes of the file are read. Similarly, when we store data in the buffer, the corresponding bytes are automatically written to the file. This lets us perform I/O without using read or write.
what is mapping a file into memory? And here, they defined the memory is placed in between stack and heap. In this memory, what
type of data is present after mapping a file. It contains copy of the file or the address of the file which resides in hard disk. And
how the above scenario becomes true.
Does anyone explain the working mechanism of memory mapped I/O and mmap functionality?
Normally when you open a file, the system sets up some bookkeeping structures (metadata) but does not need to read any part of the actual data of the file. When you call read(), the system loads a chunk of the file into (virtual) memory which you allocated for the purpose.
When you memory-map a file, the system again sets up bookkeeping, and also sets up a (virtual) memory "mapping" which means a range of valid addresses which, if used, will reflect reads (or writes) of the underlying file. It does not mean the entire file needs to be read at once, because it can be "paged in" on demand, i.e. the system can give you an address range to use, then wait for you to actually use it before loading any data there. This "page faulting" is supported by a hardware device called the Memory Management Unit, or MMU. The same system is used when you run an executable file--the system can simply map it into virtual memory and read pages (chunks) from disk only as needed.
It is in memory(copy of the file which is in hard disk) or not?
According to Computer Programming and Utilization, When you open file with fopen its content are loaded into memory. (Partially or wholly).
If it is not in memory where the data is fetch by consequent read or
write system call
When you fwrite some data, it is eventually copied into the kernel which will then write it to disk (or wherever) after buffering. In general, no part of a file needs to be loaded in order to write.
what is mapping a file into memory?
For more refer here
In this memory, what type of data is present after mapping a file. It
contains copy of the file or the address of the file which resides in
hard disk.
A memory-mapped file is a segment of virtual memory which has been assigned a direct byte-for-byte correlation with some portion of a file or file-like resource.Refer this
It is possible to mmap a file to a region of memory. When this is done, the file can be accessed just like an array in the program.This is more efficient than read or write, as only the regions of the file that a program actually accesses are loaded. Accesses to not-yet-loaded parts of the mmapped region are handled in the same way as swapped out pages.
After this open where the file resides for processing. It is in memory(copy of the file which is in hard disk) or not?
On the disk. It may also be partly or completely in memory if the operating system does a read-ahead, but that isn't detectable by you. You still have to issue reads to get data from the file.
If it is not in memory where the data is fetch by consequent read or write system call
From the disk.
or It fetchs data from the hard disk for each time of calling read or write.
In effect, but you also have to consider the effect of any caching.
Otherwise the copy of the file is stored in memory and the file is accessed by process for furthur manipulation and once the process is completed the file is copied to hard disk.
No. The file behaves as though it is all on the disk.
And here, they defined the memory is placed in between stack and heap.
Not in what you quoted.
In this memory, what type of data is present after mapping a file.
The data in the file. The question 'what type of data' doesn't make sense. Data is data.
It contains copy of the file or the address of the file which resides in hard disk.
It effectively contains a copy of the file.
And how the above scenario becomes true.
Via virtual memory. Too broad to cover here.
Currently I use shm_open to get a file descriptor and then use ftruncate and mmap whenever I want to add a new buffer to the shared memory. Each buffer is used individually for its own purposes.
Now what I need to do is arbitrarily resize buffers.
And also munmap buffers and reuse the free space again later.
The only solution I can come up with for the first problem is: ftuncate(file_size + old_buffer_size + extra_size), mmap, copy data accross into the new buffer and then munmap the original data. This looks very expensive to me and there is probably a better way. It also entails removing the original buffer every time.
For the second problem I don't even have a bad solution, I clearly can't move memory around everytime a buffer is removed. And if I keep track of free memory and use it whenever possible it will slow down allocation process as well as leave me with bits and pieces in between that are unused.
I hope this is not too confusing.
Thanks
As best as I understand you need to grow (or shrink) the existing memory mapping.
Under linux shared memory implemented as a file, located in /dev/shm memory filesystem. All operations in this file is the same as on the regular files (and file descriptors).
if you want to grow the existing mapping first expand the file size with ftruncate (as you wrote) then use mremap to expand the mapping to the requested size.
If you store pointers points to this region you maybe have to update these, but first try to call with 0 flag. In this case the system tries to grow the existing mapping to the requested size (if there is no collision with other preserved memory region) and pointers remains valid.
If previous option not available use MREMAP_MAYMOVE flag. In this case the system remaps to another locations, but mostly it's done effectively (no copy applied by the system.) then update the pointers.
Shrinking is the same but the reverse order.
I wrote an open source library for just this purpose:
rszshm - resizable pointer-safe shared memory
To quote from the description page:
To accommodate resizing, rszshm first maps a large, private, noreserve
map. This serves to claim a span of addresses. The shared file mapping
then overlays the beginning of the span. Later calls to extend the
mapping overlay more of the span. Attempts to extend beyond the end of
the span return an error.
I extend a mapping by calling mmap with MAP_FIXED at the original address, and with the new size.
I have seen a similar question on this site, but there is no helpful answer.
Scenario:
Following is the data transmission process ,
embedded devices-------->buffer-------->AWS(Cloud Storage)
Conditions:
Owing to the limit of embedded device, there is not enough memory to store the data.
My idea:
Using mmap() to allocate "memory" on disk, and manage the data relay on another lib, which is a opensource lib on github.
Problem:
However, I discover it just now that the it will occupy memory in the real memory. This method seems cannot solve my condition.
What's your idea ? Buddy...
All mmmap(2) does is to avoid an extra data copy operation between the user-space application's buffer and a kernel holding buffer. The portion of the real file which is mapped becomes part of the application's virtual address space and occupies physical memory in the block cache, even if you are using an anonymous map (a map without a backing file, the fd arg is set to -1).
So, by moving the mmap(2) window you can gain direct access to the kernel's buffer cache holding the file data. Use a 4K map window to correspond to the virtual memory map hardware feature and your file can be arbitrary size but only use a 4K map window into the file.
The good thing about mmap(2) is that you can open the file, create the mmap(2) window, and then close the file. Now you can access the file data using loads/stores treating the mapped window as a data array object.
So, I understand that if you need some dynamically allocated memory, you can use malloc(). For example, your program reads a variable length file into a char[]. You don't know in advance how big to make your array, so you allocate the memory in runtime.
I'm trying to understand when you would use mmap(). I have read the man page and to be honest, I don't understand what the use case is.
Can somebody explain a use case to me in simple terms? Thanks in advance.
mmap can be used for a few things. First, a file-backed mapping. Instead of allocating memory with malloc and reading the file, you map the whole file into memory without explicitly reading it. Now when you read from (or write to) that memory area, the operations act on the file, transparently. Why would you want to do this? It lets you easily process files that are larger than the available memory using the OS-provided paging mechanism. Even for smaller files, mmapping reduces the number of memory copies.
mmap can also be used for an anonymous mapping. This mapping is not backed by a file, and is basically a request for a chunk of memory. If that sounds similar to malloc, you are right. In fact, most implementations of malloc will internally use an anonymous mmap to provide a large memory area.
Another common use case is to have multiple processes map the same file as a shared mapping to obtain a shared memory region. The file doesn't have to be actually written to disk. shm_open is a convenient way to make this happen.
Whenever you need to read/write blocks of data of a fixed size it's much simpler (and faster) to simply map the data file on disk to memory using mmap and acess it directly rather than allocate memory, read the file, access the data, potentially write the data back to disk, and free the memory.
consider the famous producer-consumer problem, the producer creates a shared memory object using shm_open(), and since our goal is to make the producer and consumer share data, we use the mmap syscall to map that shared memory region to the process' address space. Now, the consumer can open that shared memory object (shared memory objects are referred to by a "name") and read from it, after a call to mmap to map the address space as done for the producer.