I am bit confused about the following sentence which appears in the manual of mlockall.
mlockall() locks all pages mapped into the address space of the calling process. This includes the pages of the code, data and stack segment, as well as shared libraries, user space kernel data, shared memory, and memory-mapped files.
Does this mean that mlockall(MCL_CURRENT | MCL_FUTURE) will read in advance all the pages of a shared library and lock them into memory when the function is invoked? Or when those pages are actually needed by the process?
No. Just when the pages are needed by the process. The only thing mlockall does is stop pages from being loaded into swap space (swapped).
Related
In Understanding the Linux Kernel, 3rd edition, it says:
Shared libraries are especially convenient on systems that provide file memory mapping, because they reduce the amount of main memory requested for executing a
program. When the dynamic linker must link a shared library to a process, it does not copy the object code, but performs only a memory mapping of the relevant portion of the library file into the process’s address space. This allows the page frames containing the machine code of the library to be shared among all processes that are using the same code. Clearly, sharing is not possible if the program has been linked statically. (page 817)
I am interested in this, want to write a small program in C to verify, given two pids as input such as two gedit processes, and then get the address information from page frames to be shared. Does anyone know how to do it? From that book, I think the bss segment and text segment address from two or more gedit processes are same, is that correct?
It is not the text and bss sections of your gedit (or whatever) that have the same address, but the content of the libc.so shared library - and all other shared libraries used by the two gedit processes.
This, as the quoted text says, allows the shared library to be ONE copy, and this is the main benefit of the shared library in general.
bss is generally not shared - since that is per process data. text sections of two processes running the same executable, in Linux, will share the same code.
Unfortunately, the proof of this would be to look at the physical mapping of pages (page at address X in process A is at physical address Y, and page for address X in process B is also at physical address Y) within the processes, and that's, as far as I know, not easily available without groking about inside the OS kernel.
Look at the contents of /proc/*/maps.
When we in Linux use function mmap (,,, MAP_ANON | MAP_SHARED);, then for the same region of fragmented physically memory (which allocated) between processes are allocating virtual memory pages (PTEs). Ie these PTEs are copied from page table of one process to the page table of another process (with the same sequence of fragments of physical addresses allocated memory), is this true?
But mmap () needs to be done before fork (). And if we already have two working process (ie after fork ()), then we need to use a file for the mmap(). Which functions used to copying mechanism of PTEs between the two already established processes to create a shared memory?
Can I with PTEs/SGL(scatter-gather-list) which indicate to fragments of physical memory which have been allocated to create appropriate PTEs in other process by using linux-kernel, and how to do it?
I want to understand how it mmap() works at a lower level .
When we in Linux use function mmap (,,, MAP_ANON | MAP_SHARED);, then
for the same region of fragmented physically memory (which allocated)
between processes are allocating virtual memory pages (PTEs).
Restate the question/statement, please, the above does not make sense.
Ie these PTEs are copied from page table of one process to the page
table of another process (with the same sequence of fragments of
physical addresses allocated memory), is this true?
No, it is not true.
When you establish a new mapping, a kernel first looks
for a sufficiently large unused range of addresses in the virtual address space of the process. Then it modifies the corresponding page table entries to indicate that that address range is valid, but physical pages there are not present.
When you attempt to access an address in that range, a page fault is generated. The kernel looks in its data structures and determines that the access is valid. Then it allocates a
fresh physical page, modifies the page entry to establish the mapping between the
virtual address and the physical address and marks the page as present. Upon return from
the page fault exception, the offending instruction is restarted and this time executes successfully.
But mmap () needs to be done before fork (). And if we already have
two working process (ie after fork ()), then we need to use a file for
the mmap(). Which functions used to copying mechanism of PTEs between
the two already established processes to create a shared memory?
If you do a mmap after the fork, the two processes will create and initialize
page table entries entirely independent of each other. However, when you mmap a file,
the kernel will not allocate simply a free physical page - it will allocate a page,
fill it with data from the file and put the page in the page/buffer cache. When a second
process mmaps the same file, the kernel looks in the page cache, finds there the physical
page, which corresponds to the same file and the required file offset and points the PTE
to that page. Now, there will be two completely independently created PTE, which just point to the same physical page.
Can I with PTEs/SGL(scatter-gather-list) which indicate to fragments
of physical memory which have been allocated to create appropriate
PTEs in other process by using linux-kernel, and how to do it?
Restate this too, it's not clear what you are asking.
I want to understand how it mmap() works at a lower level .
I would recommend an operating systems book, a chapter on virtual memory management,
something like Operating System Concepts by Silberschatz el al.
http://www.amazon.co.uk/Operating-System-Concepts-Abraham-Silberschatz/dp/1118112733/ref=sr_1_5?ie=UTF8&qid=1386065707&sr=8-5&keywords=Operating+System+Concepts%2C+by+Silberschatz%2C+Galvin%2C+and+Gagne
I've been googling this a little bit and haven't been able to find a clear answer so I'm hoping someone has some insight into shared memory segments in windows VS linux.
In Linux there are 2 ways of creating shared memory for IPC: shared memory segments (shmget, et al) and memory mapped files (mmap). From my brief understanding mmap requires you to have an actual file somewhere in the OS to map whereas shared memory segments are just name based segments in memory that can be attached to by processes.
In Windows there only seems to be the equivalent of memory mapped files where you have to have an actual file floating around somewhere.
My question: Is this actually the only kind of shared memory in Windows or does it have an api for creating non file based shared memory segments.
The Unix mmap() API is practically equivalent to the CreateFileMapping/MapViewOfFile Windows API. Both can map files and/or can create shared (anonymous) maps that are backed by the swap device (if any). As a matter of fact, glibc uses anonymous mmap() to implement malloc() when the requested memory size is sufficiently large.
Windows supports one additional mechanism - shared data sections in the executable file, something that is not implemented in Linux. First you create a named data section using the #pragma data_seg(".somename") and put the shared variables inside. Then you tell the linker to mark the section as read/write/shared with the following option: /SECTION:.somename,RWS. The whole process is described in MSDN. This only works for copies of the same module, either EXE or DLL file. Same module means same file in the same file system location: processes created from different copies of the same executable but located in different places won't see each others named sections as shared.
Yes, you can use non file-based shared memory segments in Windows.
#pragma comment(linker, "/SECTION:.shared,RWS")
#pragma data_seg(".shared")
int g_iShared = 0;
#pragma data_seg()
This might be a bit late.
Windows shared memory is not the same as Linux only some things are similar.
The biggest difference is the memory allocation granularity size. Linux is 4K and Windows is 64K. If it's important to have say arbitrary 8K pages mapped into specific 8K destinations well you are stuck on Windows and it just can't be done. (If someone figures this out then please let me know).
Another difference is you can mmap a new page over the top of an existing page effectively replacing the first page mapping. In windows you can't do this but instead must destroy the entire view and rebuild the entire view in what ever new layout that is required. So if the "view" contains 1024 pages and 1 page changes then in Linux you can just change that one page. In Windows you must drop all 1024 pages and re-view the same 1023 pages + the one new page!
On Linux you can share memory without a backing file by using ANONYMOUS memory but then it becomes hard to share or you can use shm_opn to create a shared memory file descriptor and pass that to mmap. Here is a good link.
http://nullprogram.com/blog/2016/04/10/
I've used this and it works.
Is there a way in FreeBSD by which we can restrict few contiguous instruction pages to read and write to only to few data segment pages? Basically I am trying to restrict dynamically linked libraries to read and write only pages linked due to dynamically linked library and only few pages of the global data in the rest of the process address space.
Can this be done using linker directives or by modifying page level bits within the kernel?
Regards,
Mahesh
You can set the protection level of pages (e.g. make them read-only) within the virtual address space of a process with mprotect(2).
Here is the situation:
A process has two pages vp1 and vp2. These two pages are mapped to 2 physical pages or 2 pages in the swap. Let's call these physical (or in swap) pages pp1 and pp2. The mapping is:
vp1->pp1
vp2->pp2
Now, if I want to change the mapping to:
vp1->pp2
vp2->pp1
That means, reading from vp2 by the process will get the content originally in vp1. Is there a method to do this without changing the kernel on Linux?
Yes, but you have to do some work first. One way to accomplish this is to create two shared memory objects. Then you can map and unmap the shared memory objects in the process address space. See the system calls shmat, shmdt, shmget, and shmctl for details.
Mapping and unmapping is likely to take considerable time, so it may not save time over using some pointer scheme to choose which addresses a process uses to access data.
No. Not in the general case if you want to keep your system working. But if you control how the mappings are created you can create them with mmap of a file or an object from shm_open and when you need to swap them just overwrite them with mmap(... MAP_FIXED ...).