I am using C on Linux, and allocating 2 shared memory segments.
The first segment is created in a main program, then I call a subprocess and create the second segment.
In the subprocess, I place the address of the second segment in a pointer I set aside in the first segment.
Upon returning to the main program, when I attach to the second segment and compare the pointers (the one returned from shmat, and the one previously stored by the subprocess) I find they are different.
Is this expected?
Thanks, Mark.
Yes, this is expected. Mapping to a common address in the virtual space of the two processes would be a very constraining limitation. Among others, the memory manager would have to know simultaneously which processes are willing to map, so that it finds a common free area. This would defeat the very principle of virtual memory (every process sees a blank address space), and cause configurations impossible to arbitrate.
Sharing at common addresses is indeed possible, but only makes sense when the mapping is to some reserved section of the address space, so that it doesn't get mapped elsehow.
Related
I am currently taking a course in Operating Systems and I came across address virtualization. I will give a brief about what I know and follow that with my question.
Basically, the CPU(modern microprocessors) generates virtual addresses and then an MMU(memory management unit) takes care of translating those virtual address to their corresponding physical addresses in the RAM. The example that was given by the professor is there is a need for virtualization because say for example: You compile a C program. You run it. And then you compile another C program. You try to run it but the resident running program in memory prevents loading a newer program even when space is available.
From my understanding, I think having no virtualization, if the compiler generates two physical addresses that are the same, the second won't run because it thinks there isn't enough space for it. When we virtualize this, as in the CPU generates only virtual addresses, the MMU will deal with this "collision" and find a spot for the other program in RAM.(Our professor gave the example of the MMU being a mapping table, that takes a virtual address and maps it to a physical address). I thought of that idea to be very similar to say resolving collisions in a hash table.
Could I please get some input on my understanding and any further clarification is appreciated.
Could I please get some input on my understanding and any further clarification is appreciated.
Your understanding is roughly correct.
Clarifications:
The data structures are nothing like a hash table.
If anything, the data structures are closer to a BTree, but even there are important differences with that as well. It is really closest to a (Java) N-dimensional array which has been sparsely allocated.
It is mapping pages rather than complete virtual / physical addresses. (A complete address is a page address + an offset within the page.).
There is no issue with collision. At any point in time, the virtual -> physical mappings for all users / processes give a one-to-one mapping from (process id + virtual page) to a either a physical RAM page or a disk page (or both).
The reasons we use virtual memory are:
process isolation; i.e. one process can't see or interfere with another processes memory
simplifying application writing; i.e. each process thinks it has a contiguous set off memory addresses, and the same set each time. (To a first approximation ...)
simplifying compilation, linking, loading; i.e. the compilers, etc there is no need to "relocate" code at compile time or run time to take into account other.
to allow the system to accommodate more processes than it has physical RAM for ... though this comes with potential risks and performance penalties.
I think you have a fundamental misconception about what goes on in an operating system in regard to memory.
(1) You are describing logical memory, not virtual memory. Virtual memory refers to the use of disk storage to simulate memory. Unmapped pages of logical memory get mapped to disk space.
Sadly, the terms logical memory and virtual memory get conflated but they are distinct concepts the the distinction is becoming increasingly important.
(2) Programs run in a PROCESS. A process only runs one program at a time (in unix each process generally only runs one program (two if you count the cloned caller) in its life.
In modern systems each process process gets a logical address space (sequential addresses) that can be mapped to physical locations or no location at all. Generally, part of that logical address space is mapped to a kernel area that is shared by all processes. The logical address space is create with the process. No address space—no process.
In a 32-bit system, addresses 0-7FFFFFFF might be user address that are (generally) mapped to unique physical locations while 80000000-FFFFFFFFmight be mapped to a system address space that is the same for all processes.
(3) Logical memory management primarily serves as a means of security; not as a means for program loading (although it does help in that regard).
(4) This example makes no sense to me:
You compile a C program. You run it. And then you compile another C program. You try to run it but the resident running program in memory prevents loading a newer program even when space is available.
You are ignoring the concept of a PROCESS. A process can only have one program running at a time. In systems that do permit serial running of programs with the same process (e.g., VMS) the executing program prevents loading another program (or the loading of another program causes the termination of the running program). It is not a memory issue.
(5) This is not correct at all:
From my understanding, I think having no virtualization, if the compiler generates two physical addresses that are the same, the second won't run because it thinks there isn't enough space for it. When we virtualize this, as in the CPU generates only virtual addresses, the MMU will deal with this "collision" and find a spot for the other program in RAM.
The MMU does not deal with collisions. The operating system sets up tables that define the logical address space when the process start. Logical memory has nothing to do with hash tables.
When a program accesses logical memory the rough sequence is:
Break down the address into a page and an offset within the page.
Does the page have an corresponding entry in the page table? If not FAULT.
Is the entry in the page table valid? If not FAULT.
Does the page table entry allow the type of access (read/write/execute) requested in the current operating mode (kernel/user/...)? If not FAULT.
Does the entry map to a physical page? If not PAGE FAULT (go load the page from disk--virtual memory––and try again).
Access the physical memory referenced by the page table.
I have to scan the memory space of a calling process in C. This is for homework. My problem is that I don't fully understand virtual memory addressing.
I'm scanning the memory space by attempting to read and write to a memory address. I can not use proc files or any other method.
So my problem is setting the pointers.
From what I understand the "User Mode Space" begins at address 0x0, however, if I set my starting point to 0x0 for my function, then am I not scanning the address space for my current process? How would you recommend adjusting the pointer -- if at all -- to address the parent process address space?
edit: Ok sorry for the confusion and I appreciate the help. We can not use proc file system, because the assignment is intended for us to learn about signals.
So, basically I'm going to be trying to read and then write to an address in each page of memory to test if it is R, RW or not accessible. To see if I was successful I will be listening for certain signals -- I'm not sure how to go about that part yet. I will be creating a linked list of structure to represent the accessibility of the memory. The program will be compiled as a 32 bit program.
With respect to parent process and child process: the exact text states
When called, the function will scan the entire memory area of the calling process...
Perhaps I am mistaken about the child and parent interaction, due to the fact we've been covering this (fork function etc.) in class, so I assumed that my function would be scanning a parent process. I'm going to be asking for clarification from the prof.
So, judging from this picture I'm just going to start from 0x0.
From a userland process's perspective, its address space starts at address 0x0, but not every address in that space is valid or accessible for the process. In particular, address 0x0 itself is never a valid address. If a process attempts to access memory (in its address space) that is not actually assigned to that process then a segmentation results.
You could actually use the segmentation fault behavior to help you map out what parts of the address space are in fact assigned to the process. Install a signal handler for SIGSEGV, and skip through the whole space, attempting to read something from somewhere in each page. Each time you trap a SIGSEGV you know that page is not mapped for your process. Go back afterward and scan each accessible page.
Do only read, however. Do not attempt to write to random memory, because much of the memory accessible to your programs is the binary code of the program itself and of the shared libraries it uses. Not only do you not want to crash the program, but also much of that memory is probably marked read-only for the process.
EDIT: Generally speaking, a process can only access its own (virtual) address space. As #cmaster observed, however, there is a syscall (ptrace()) that allows some processes access to some other processes' memory in the context of the observed process's address space. This is how general-purpose debuggers usually work.
You could read (from your program) the /proc/self/maps file. Try first the following two commands in a terminal
cat /proc/self/maps
cat /proc/$$/maps
(at least to understand what are the address space)
Then read proc(5), mmap(2) and of course wikipages about processes, address space, virtual memory, MMU, shared memory, VDSO.
If you want to share memory between two processes, read first shm_overview(7)
If you can't use /proc/ (which is a pity) consider mincore(2)
You could also non-portably try reading from (and perhaps rewriting the same value using volatile int* into) some address and catching SIGSEGV signal (with a sigsetjmp(3) in the signal handler), and do that in a -dichotomical- loop (in multiple of 4Kbytes) - from some sane start and end addresses (certainly not from 0, but probably from (void*)0x10000 and up to (void*)0xffffffffff600000)
See signal(7).
You could also use the Linux (Gnu libc) specific dladdr(3). Look also into ptrace(2) (which should be often used from some other process).
Also, you could study elf(5) and read your own executable ELF file. Canonically it is /proc/self/exe (a symlink) but you should be able to get its from the argv[0] of your main (perhaps with the convention that your program should be started with its full path name).
Be aware of ASLR and disable it if your teacher permits that.
PS. I cannot figure out what your teacher is expecting from you.
It is a bit more difficult than it seems at the first sight. In Linux every process has its own memory space. Using any arbitrary memory address points to the memory space of this process only. However there are mechanisms which allow one process to access memory regions of another process. There are certain Linux functions which allow this shared memory feature. For example take a look at
this link which gives some examples of using shared memory under Linux using shmget, shmctl and other system calls. Also you can search for mmap system call, which is used to map a file into a process' memory, but can also be used for the purpose of accessing memory of another process.
When we in Linux use function mmap (,,, MAP_ANON | MAP_SHARED);, then for the same region of fragmented physically memory (which allocated) between processes are allocating virtual memory pages (PTEs). Ie these PTEs are copied from page table of one process to the page table of another process (with the same sequence of fragments of physical addresses allocated memory), is this true?
But mmap () needs to be done before fork (). And if we already have two working process (ie after fork ()), then we need to use a file for the mmap(). Which functions used to copying mechanism of PTEs between the two already established processes to create a shared memory?
Can I with PTEs/SGL(scatter-gather-list) which indicate to fragments of physical memory which have been allocated to create appropriate PTEs in other process by using linux-kernel, and how to do it?
I want to understand how it mmap() works at a lower level .
When we in Linux use function mmap (,,, MAP_ANON | MAP_SHARED);, then
for the same region of fragmented physically memory (which allocated)
between processes are allocating virtual memory pages (PTEs).
Restate the question/statement, please, the above does not make sense.
Ie these PTEs are copied from page table of one process to the page
table of another process (with the same sequence of fragments of
physical addresses allocated memory), is this true?
No, it is not true.
When you establish a new mapping, a kernel first looks
for a sufficiently large unused range of addresses in the virtual address space of the process. Then it modifies the corresponding page table entries to indicate that that address range is valid, but physical pages there are not present.
When you attempt to access an address in that range, a page fault is generated. The kernel looks in its data structures and determines that the access is valid. Then it allocates a
fresh physical page, modifies the page entry to establish the mapping between the
virtual address and the physical address and marks the page as present. Upon return from
the page fault exception, the offending instruction is restarted and this time executes successfully.
But mmap () needs to be done before fork (). And if we already have
two working process (ie after fork ()), then we need to use a file for
the mmap(). Which functions used to copying mechanism of PTEs between
the two already established processes to create a shared memory?
If you do a mmap after the fork, the two processes will create and initialize
page table entries entirely independent of each other. However, when you mmap a file,
the kernel will not allocate simply a free physical page - it will allocate a page,
fill it with data from the file and put the page in the page/buffer cache. When a second
process mmaps the same file, the kernel looks in the page cache, finds there the physical
page, which corresponds to the same file and the required file offset and points the PTE
to that page. Now, there will be two completely independently created PTE, which just point to the same physical page.
Can I with PTEs/SGL(scatter-gather-list) which indicate to fragments
of physical memory which have been allocated to create appropriate
PTEs in other process by using linux-kernel, and how to do it?
Restate this too, it's not clear what you are asking.
I want to understand how it mmap() works at a lower level .
I would recommend an operating systems book, a chapter on virtual memory management,
something like Operating System Concepts by Silberschatz el al.
http://www.amazon.co.uk/Operating-System-Concepts-Abraham-Silberschatz/dp/1118112733/ref=sr_1_5?ie=UTF8&qid=1386065707&sr=8-5&keywords=Operating+System+Concepts%2C+by+Silberschatz%2C+Galvin%2C+and+Gagne
Suppose I have two process a and b on Linux. and in both process I use malloc() to allocate a memory,
Is there any chances that malloc() returns the same starting address in two processes?
If no, then who is going to take care of this.
If yes, then both process can access the same data at this address.
Is there any chances that malloc() return same starting address in two process.
Yes, but this is not a problem.
What you're not understanding is that operating systems firstly handle your physical space for you - programs etc only see virtual addresses. There is only one virtual address space, however, the operating system (let's stick with 32-bit for now) divides that up. On Windows, the top half (0xA0000000+) belongs to the kernel and the lower half to user mode processes. This is referred to as the 2GB/2GB split. On Linux, the divide is 3GB/1GB - see this article:
Kernel memory is defined to start at PAGE_OFFSET,which in x86 is 0XC0000000, or 3 gigabytes. (This is where the 3gig/1gig split is defined.) Every virtual address above PAGE_OFFSET is the kernel, any address below PAGE_OFFSET is a user address.
Now, when a process switch (as opposed to a context switch) occurs, all of the pages belonging to the current process are unmapped from virtual memory (not necessarily paging them) and all of the pages belonging to the to-be-run process are copied in (disclaimer: this might not exactly be true; one could mark pages dirty etc and copy on access instead, theoretically).
The reason for the split is that, for performance reasons, the upper half of the virtual memory space can remained mapped to the operating system kernel.
So, although malloc might return the same value in two given processes, that doesn't matter because:
physically, they're not the same address.
the processes don't share virtual memory anywhere.
For 64-bit systems, since we're currently only using 48 of those bits there is a gulf between the bottom of user mode and kernel mode which is not addressable (yet).
Yes, malloc() can return the same pointer value in separate processes, if the processes run in separate address spaces, which is achieved via virtual memory. But they won't access the same physical memory location in that case and the data at the address need not be the same, obviously.
Process is a collection of threads plus an address-space. This address-space is referred as virtual because every byte of it is not necessarily backed by physical memory. Segments of a virtual address-space will be eventually backed by physical memory if the application in the process ends up by using effectively this memory.
So, malloc() may return an identical address for two process, but it is no problem since these malloced memories will be backed by different segments of physical memory.
Moreover malloc() implementation is moslty not reentrant, therefore calling malloc() in differents threads sharing the same address-space hopefully won't result in returning the same virtual address.
I was stracing some of the common commands in the linux kernel, and saw mprotect() was used a lot many times. I'm just wondering, what is the deciding factor that mprotect() uses to find out that the memory address it is setting a protection value for, is in its own address space?
On architectures with an MMU1, the address that mprotect() takes as an argument is a virtual address. Each process has its own independent virtual address space, so there's only two possibilities:
The requested address is within the process's own address range; or
The requested address is within the kernel's address range (which is mapped into every process).
mprotect() works internally by altering the flags attached to a VMA2. The first thing it must do is look up the VMA corresponding to the address that was passed - if the passed address was within the kernel's address range, then there is no VMA, and so this search will fail. This is exactly the same thing happens if you try to change the protections on an area of the address space that is not mapped.
You can see a representation of the VMAs in a process's address space by examining /proc/<pid>/smaps or /proc/<pid>/maps.
1. Memory Management Unit
2. Virtual Memory Area, a kernel data structure describing a contiguous section of a process's memory.
This is about virtual memory. And about dynamic linker/loader. Most mprotect(2) syscalls you see in the trace are probably related to bringing in library dependencies, though malloc(3) implementation might call it too.
Edit:
To answer your question in comments - the MMU and the code inside the kernel protect one process from the other. Each process has an illusion of a full 32-bit or 64-bit address space. The addresses you operate on are virtual and belong to a given process. Kernel, with the help of the hardware, maps those to physical memory pages. These pages could be shared between processes implicitly as code, or explicitly for interprocess communications.
The kernel looks up the address you pass mprotect in the current process's page table. If it is not in there then it fails. If it is in there the kernel may attempt to mark the page with new access rights. I'm not sure, but it may still be possible that the kernel would return an error here if there were some special reason that the access could not be granted (such as trying to change the permissions of a memory mapped shared file area to writable when the file was actually read only).
Keep in mind that the page table that the processor uses to determine if an area of memory is accessible is not the one that the kernel used to look up that address. The processor's table may have holes in it for things like pages that are swapped out to disk. The tables are related, but not the same.