I know that, in general, a named memory block is shared memory which you can assign and access by a name.
What I want to know is what are the advantages of using a named block of memory and when should it be used in terms of memory management ?
What you are describing has different names depending upon the operating system.
It is a range of pages that can be mapped to the address space of multiple processes. It really has two components:
1) The storage in the page file
2) The physical memory--with paging, there might not be physical memory associated with it all the time.
The name serves as the way of identifying the shared memory so that it can be mapped to the process address space.
It is used for sharing data between processes. They were very commonly used with database systems. They are the fastest method of interprocess communication but require some kind of locking mechanism that the application has to implement. Often they are used with on writer and multiple readers.
If processes A&B map to the shared memory block, and process A writes to the block, B immediately sees the change.
Related
I have a massive file ie 1TiB owned as 'filehandler', permitted rwx------. I mmap(2)-ed it into the 64bit address space, and all works successfully. This file handled by a process running as user 'filehandler'.
Other processes request services from this handler process running as other user than the filehandler. They login into handler through unix socket. They communicate by IPC rules, all is ok.
The entire file must not be shared to requesters due to security reasons. In the file only some parts are allowed to access for requester processes.
The best performance will be given if share of the memory, just the allowed parts of the file with the requesting processes.
For example the shm gives the key to access the segment for other processes, it is a practical targeting to requester.
Is there any way to share only the allowed parts of a mmap(2)-ed space to any processes identified like shm technology?
Is there any way to share only the allowed parts of a mmap(2)-ed space to any processes identified like shm technology?
TL;DR: No.
In more detail,
How to remap a file mmap(2)-ed in memory like shmget
mmap() and shmget()are not really comparable. A better comparison would be between the combination of shm_open() / ftruncate() / mmap() on one hand and the combination of shmget() / shmat() on the other. These are the main alternatives in POSIX for creating labeled shared-memory segments and mapping them into the process's address space. You should recognize there that the analog of shmget() is shm_open(), and the analog of mmap() in this context is shmat().
Now, returning to
Is there any way to share only the allowed parts of a mmap(2)-ed space to any processes identified like shm technology?
Note well that in both cases above, it is the object being mapped (a shared memory segment) that provides for sharing between unrelated processes, not anything to do with mmap() itself. The same applies when mmap() maps any other kind of object, such as a regular file. It is always the mapped object through which any shared access is mediated. It has to be this way, because a memory mapping is a property of one process -- it is not itself share-able.
Your design calls for a filehandler process to serve as gatekeeper to the data, rather than allowing clients to access it directly. That's fine, but it precludes the clients mapping the file into memory. You could probably arrange for client to access the data through a shared memory segment of either flavor, but that would require the server copying the right data out of the big file into the client's shared memory segment. That might indeed be something to consider, but you can forget about the server providing clients direct memory-mapped access to the file.
There's no connection between implementations of shmget system call (a System V AT&T derived implementation) and mmap (a berkeley's BSD system derived implementation) It's true that in BSD systems, AT&T shared memory is implemented by using mmaped private segments with no file attached, but that's of no use also, because you need the shared segment to be associated with a file.
As you need, the only possibility to share memory segments related to a file's contents are by using mmap system call, because System V shared memory segments have no means to associate a file with them.
All of these resources (either SysV or BSD) have a set of permissions bits associated with them that allow them to be used with some security, but as happens with files, only in a global (the entire resource) way, making you able to access the whole thing or nothing at all.
BTW, you can implement what you want by means of copying segment contents to a different, private, segment (only the size you want the client to be allowed to see) only the segments it is allowed access, and this way you can have finer control over whom and what the clients are allowed to do.
And last, remember that this approach requires copying of segments of shared memory, so you need to remember to copy back the exported segment for a customer if you don't want the modifications made by that client to be lost when the client finishes using them.
From my point of view, you are complicating things a little, but you better know how your application is designed than me.
I am trying to understand how mmap works while looking at man mmap.
As I understand it, it adds a mapping to the page table that maps between the file and the virtual address (which is the address that is given void *addr)
So, what happens when 2 programs map the same file?
Are there 2 entries in the page table, one for each program?
So, what happens when 2 programs map the same file? Are there 2 entries in the page table, one for each program?
In modern operating systems, each process has its own page table for its memory, that may point to pages of physical memory shared with other user and kernel processes.
With MAP_SHARED, this mapping is shared: updates to the mapping are visible to other processes that map this file, and are carried through to the underlying file. The file may not actually be updated until msync(2) or munmap() is called.
This seems very interesting, but there are numerous caveats:
The actual pages mmapped by both processes for the same file may reside at the same address or at a different address in each process, storing pointers into this shared memory may not allow the other process to use them as they might point to inconsistent addresses.
The implementation may use the same physical memory pages for both mappings or not: for subtile reasons (cache strategies, out of sync reading...), even if it is the same physical memory, modifications done by one process to its memory may not be immediately reflected in the memory of the other process.
So the modification may or may not be visible to the other processes mmapping the file nor reading it via read or the FILE* stream API.
If one of the processes calls msync(), the modifications should be visible in all maps and for all yet unread portions of the file, bearing in mind that the FILE* streaming APIs may have buffered some data in internal unshared buffers: modifications in this area will not be reflected.
Conclusion: it is risky and unreliable to use these mechanisms to implement inter process communication. The behavior may depend on system specific characteristics such as the OS strategies, the CPU and cache architectures, the type of RAM in use, the clock speed, and who knows what else. It is safer to rely on proven APIs that may indeed be implemented using mmapped memory, but only if it is know to provide the correct semantics.
The actual system implementation is different. At the risk of over simplification (and omitting paging here):
A mmap will map physical page frames to a file.
So, what happens when 2 programs map the same file? Are there 2 entries in the page table, one for each program?
If two processes (P and Q) map to the same file, then P and Q will each have there own page table; each page table will have entry mapping to the same physical page frame (which could be mapped to different addresses within P and Q).
I am writing a C program that makes use of mmap system call, running on Linux 3.12 64-bit.
If I have two processes mmaping the same region of a disk file with read/write access, and then modify the region content from both processes at the same time...
Can one process see (read) changed data made by the other process, before or after msync?
If update can be seen by the other process - is there a happens-before guarantee made by Linux mmap implementation?
Yes, that's one of the purposes of memory mapping: as a form of "instantaneous IPC".
You must set the MAP_SHARED flag:
http://man7.org/linux/man-pages/man2/mmap.2.html
If you wish to use shared memory for this purpose, I would consider the shminit()/shmat() APIs instead:
http://linux.die.net/man/2/shmat
Suggestion: check out Beej's Guide to *nix Interprocess Communication:
http://beej.us/guide/bgipc/
And no, if you use the raw mmap() APIs, there's no "before/after guarantee", and you must use some kind of locking (e.g. semaphores) if you wish to read/write data concurrently.
Also, from Understanding Memory Mapping:
Both the mmap and shmat services provide the capability for multiple
processes to map the same region of an object such that they share
addressability to that object. However, the mmap subroutine extends
this capability beyond that provided by the shmat subroutine by
allowing a relatively unlimited number of such mappings to be
established. While this capability increases the number of mappings
supported per file object or memory segment, it can prove inefficient
for applications in which many processes map the same file data into
their address space.
The mmap subroutine provides a unique object address for each process
that maps to an object. The software accomplishes this by providing
each process with a unique virtual address, known as an alias. The
shmat subroutine allows processes to share the addresses of the mapped
objects.
The above is true of any *nix variant, including both Linux and AIX.
Here is the situation:
A process has two pages vp1 and vp2. These two pages are mapped to 2 physical pages or 2 pages in the swap. Let's call these physical (or in swap) pages pp1 and pp2. The mapping is:
vp1->pp1
vp2->pp2
Now, if I want to change the mapping to:
vp1->pp2
vp2->pp1
That means, reading from vp2 by the process will get the content originally in vp1. Is there a method to do this without changing the kernel on Linux?
Yes, but you have to do some work first. One way to accomplish this is to create two shared memory objects. Then you can map and unmap the shared memory objects in the process address space. See the system calls shmat, shmdt, shmget, and shmctl for details.
Mapping and unmapping is likely to take considerable time, so it may not save time over using some pointer scheme to choose which addresses a process uses to access data.
No. Not in the general case if you want to keep your system working. But if you control how the mappings are created you can create them with mmap of a file or an object from shm_open and when you need to swap them just overwrite them with mmap(... MAP_FIXED ...).
What I actually want to do is to redirect writes in a certain memory area to a separate memory area which is shared between two processes. Can this be done at user level? For example, for some page X. What I want to do is to change its (virtual to physical) mapping to some shared mapping when it's written. Is this achievable? I need to do it transparently too, that is the program still uses the variables in page X by their names or pointers, but behind the scenes, we are using a different page.
Yes, it is possible to replace memory mappings in Linux, though it is not advisable to do it since it is highly non-portable.
First, you should find out in what page the X variable is located by taking its address and masking out the last several bits - query the system page size with sysconf(_SC_PAGE_SIZE) in order to know how many bits to mask out. Then you can create a shared memory mapping that overlaps this page using the MAP_FIXED | MAP_SHARED flag to mmap(2) or mmap2(2). You should copy the initial content of the page and restore it after the new mapping. Since other variables may reside in the same page, you should be very careful about memory layout and better use a dedicated shared memory object.
What you're trying to do isn't entirely possible, because, at least on x86, memory cannot be remapped on that fine-grained of a scale. The smallest quantum that you can remap memory on is a 4k page, and the page containing any given variable (e.g, X) is likely to contain other variables or program data.
That being said, you can share memory between processes using the mmap() system call.