I'm a beginner in processes, still struggling to understand the purpose of reaping child processes and memory allocation for them, so my questions are:
Q1. Why Linux doesn't have a mechanism of auto reaping child processes, I mean all child processes are terminated and removed once they finished just like garbage collection, so that users don't need to use waitpid to manually reap child processes?
Q2. my textbooks says that when users use fork() to create a new child process, the child gets an identical (but separate) copy of the parent’s text, data, and bss segments, heap, and user stack. So does it mean that same size of memory address of parents is allocated to the child as well and the memory content of child is exact then same as its parent? if so, lets say we create a huge number of child processes, isn't that the stack will overflow very easily?
Q3. Lets say the picture below is a parent processes
and you can see the user stack highlighted in red is the stack of the parent. So if the parent program uses fork() and when it executes fork() function, how does child processes' stack get allocated? is the child stack next to current stack
Q1. Why Linux doesn't have a mechanism of auto reaping child processes, I mean all child processes are terminated and removed once they finished just like garbage collection, so that users don't need to use waitpid to manually reap child processes?
It does. Just set SIGCHLD to be ignored and your children will be reaped for you.
Q2. my textbooks says that when users use fork() to create a new child process, the child gets an identical (but separate) copy of the parent’s text, data, and bss segments, heap, and user stack. So does it mean that same size of memory address of parents is allocated to the child as well and the memory content of child is exact then same as its parent?
That's an implementation detail. Most modern systems just share the memory between the two processes.
if so, lets say we create a huge number of child processes, isn't that the stack will overflow very easily?
No. The stack overflows when you run out of address space in a particular process' view of memory. Since each fork creates a new process, you aren't going to run out of address space in any particular process.
Q3. Lets say the picture below is a parent processes enter image description here
... and you can see the user stack highlighted in red is the stack of the parent. So if the parent program uses fork() and when it executes fork() function, how does child processes' stack get allocated? is the child stack next to current stack?
The child process' stack is in another process. So if that's the address space for the parent process' segments, the child process' stack isn't in it at all. The child process begins with a copy of the parent's address space -- typically with the actual memory pages shared, at least until either process tries to modify them (which unshares them).
Although David Schwartz already answered the stated questions, I'd like to talk about the underlying picture. So, do not consider this an answer, but an extended comment.
The problem is, the image shown is not a good representation of the address space a normal userspace application sees on current computers and operating systems.
Each process has their own address space. This is implemented using virtual memory; a mapping from virtual addresses to actual hardware accesses that is for all intents and purposes invisible to the userspace process. (Virtual memory is not per-address, but uses smallish chunks called pages. On all current architectures, each page is a power of two bytes in size. 212 = 4096 is common, but other sizes like 216 = 65536 and 221 = 2097152 are also used.)
Even when you use shared memory, it can reside at different virtual addresses for different processes. (This is also the reason you cannot really use pointers in shared memory.)
When a process is fork()ed, the virtual memory is cloned. (It is not really copied per se, as that would waste resources. Usually a technique called copy-on-write is used by the OS kernel, so that the actual same physical RAM is used for memory used by both/all processes, until one of them modifies their "copy"; at which point the OS kernel detaches the affected page duplicating its contents, before allowing the modifiaction to proceed.)
This means that after a fork(), the parent and child processes will have their stacks at exactly the same virtual addresses.
The only limit is how much actual RAM there is available. Actually, the OS kernel can also move currently unused files to swap or paging file; but if those pages end up being needed soon, it slows down the operation of the machine. On Linux at least, binaries and libraries also get directly mapped to their respective files -- that being the reason why you can't modify executable files when they are in use --, so unless the RAM copy of the code is modified, they tend to not use swap/paging file.
In most cases, some of the virtual memory range is reserved for the OS kernel. It does not mean that the kernel memory is visible to userspace, or accessible in any way; it is just a way to ensure that when transferring data to or from userspace processes, the OS kernel can use the userspace virtual memory addresses, and not mix them with its own internal addresses. Essentially, the OS kernel just won't create any virtual memory mappings to addresses it uses itself, for any userspace process, to make its own job simpler.
An interesting detail on Linux is that typically, the default stack size for new threads is rather large, 8 MiB (8,388,608 bytes) on 32-bit x86. Unless you set a smaller stack, the number of threads a process can create is limited by the available virtual memory. Each userspace process can use the lower 3 GiB, or virtual memory addresses below 3,221,225,472 on 32-bit x86; and you can fit at most 384 8 MiB stacks in that. If you account for the standard libraries and so on, typically on those systems a process can create about 300 threads, before it runs out virtual memory. If you use a much smaller stack, say 65536 bytes, a process can create thousands of threads even on 32-bit x86. Just remember that the issue there is running out of usable virtual memory address space, not memory per se.
Related
I was reading a textbook which talks about how fork() works with virtual memory:
When the fork function is called by the current process, the kernel creates various data structures for the new process and assigns it a unique PID. To create the virtual memory for the new process, it creates exact copies of the current process’s mm_struct, area structs, and page tables. It flags each page in both processes as read-only [emphasis added], and flags each area struct in both processes as private copy-on-write.
Source: Computer Systems: A Programmer's Perspective, Chapter 3, Section 9.8.2 - The fork Function Revisited.
I don't understand why it needs to flag each page in both processes as read-only. If each page in the parent process is read-only then the parent process will never be able to modify some uninitialised global variables (.bss section). How then can the program work?
If each page in the parent process is read-only then the parent process will never be able to modify some uninitialised global variables
That would only be true if the pages stay read only. But they don't as it says in the next part of the sentence:
and flags each area struct in both processes as private copy-on-write
Each page starts off as read-only so that a single copy can be shared by both parent and child. If either process tries to modify such a page only at that point will a writeable copy be made (if the page is indeed meant to be writeable). After the copy the writing process can make any changes it likes without affecting the other process's original (still read-only) page.
This can save memory for pages that neither parent nor child actually ends up changing.
From user space point of view (that is from syscalls(2) used after the fork(2) in your application code...), the memory pages (managed by the MMU) are not all read-only. That abstraction is provided by the kernel.
And after a successful fork(2) you could call mprotect(2), mmap(2), munmap(2), sbrk(2) (perhaps used by malloc(3) or dlopen(3)...) and execve(2) to change the address space of your process.
Read Advanced Linux Programming and a good textbook on Operating Systems. See of course LinuxAteMyRAM
From inside the Linux kernel, things are of course very different. Refer to kernelnewbies and OSDev websites.
Both the Linux kernel, the GNU libc or musl-libc, and most applications (e.g. GNU bash) in major Linux distributions such as Debian are open source. You are allowed to download and study their source code.
Consider reading proc(5) and elf(5), and using pmap(1), objdump(1), readelf(1). Try cat /proc/$$/maps in a terminal.
Fork() executes the same program and has copy same variables of the father at the moment of the fork, how do the OS keep both process in memory, safeguarding each process only access his variables?
When the kernel creates a new process, it also creates a new memory mapping. Initially all pages in the new mapping are shared with the parent process, but once pages in the map are modified by the child process those are copied into their own pages.
Useful terms to search for: Virtual memory, on demand paging, memory mapping, shared memory, copy on write.
The OS copies virtual memory space of the forking process (with possible optimizations like copy-on-write).
Fork is a technique that in general makes a separate address space for the child. The child has the same memory of the parent, but they have different PID. So you can distinguish them: specifically fork() returns 0 in the child process and a non zero value (child's PID) in the parent process.
When a process forks, would the child process have the customized shared library (.so file) in its address space?
If so, is the address of the shared library be same or different from its parent process (due to ASLR) ?
Would the function running before the main function __attribute__ ((constructor)) constructor be executed again in all the child process? What about thread?
Yes, the child will retain the parent's mappings. Ordinarily, Linux's virtual memory system will actually share the page between the two processes, up until either one tries to write new data. At that point, a copy will be made and each process will have its own unique version - at a different physical address but retaining the same virtual address. This is referred to as "copy on write" and is a substantial efficiency and resources advantage over systems which cannot support this, particularly running code which forks frequently.
Address Space Layout Randomization (ASLR) can't apply for libraries or objects which are already allocated virtual addresses, as to do so would break any pointers held anywhere in the code - something that a system running non-managed code can't know enough about to account for.
Since all previously constructed objects already exist in memory, constructors are not called again just because of the fork. Any objects which need to be duplicated because they are being uniquely modified have this done invisibly by the VM system behind the scenes - they don't really know that they are being cloned, and you could very well end up having a pair of objects where part of the implementation continues to share a physical page with identical contents while another part has been invisibly bifurcated into distinct physical pages with differing contents for each process.
You also asked about threads, and that is an area where things get complicated. Normally, only the thread which called fork() will exist in live form in the child (though data belonging to the others will exist in shared mappings, since it can't be known what might be shared with the forked thread). If you need to try to fork a multithreaded program, you will need to see the documentation of your threading implementation. For the common pthreads implementation on Linux, particularly pay attention to pthread_atfork()
This is sort of a technical question, maybe you can help me if you know about C and UNIX (or maybe it is a really newbie question!)
A question came up today while analizing some code in our Operative Systems course. We are learning what it means to "fork" a process in UNIX, we already know it creates a copy of the current process parallel to it and they have separate data sections.
But then I thought that maybe, if one creates a variable and a pointer pointing at it before doing fork(), because the pointer stores the memory address of the variable, one could try to modify the value of that variable from the child process by using that pointer.
We tried a code similar to this in class:
#include <stdio.h>
#include <sys/types.h>
#include <stdlib.h>
int main (){
int value = 0;
int * pointer = &value;
int status;
pid_t pid;
printf("Parent: Initial value is %d\n",value);
pid = fork();
switch(pid){
case -1: //Error (maybe?)
printf("Fork error, WTF?\n");
exit(-1);
case 0: //Child process
printf("\tChild: I'll try to change the value\n\tChild: The pointer value is %p\n",pointer);
(*pointer) = 1;
printf("\tChild: I've set the value to %d\n",(*pointer));
exit(EXIT_SUCCESS);
break;
}
while(pid != wait(&status)); //Wait for the child process
printf("Parent: the pointer value is %p\nParent: The value is %d\n",pointer,value);
return 0;
}
If you run it, you'll get something like this:
Parent: Initial value is 0
Child: I'll try to change the value
Child: The pointer value is 0x7fff733b0c6c
Child: I've set the value to 1
Parent: the pointer value is 0x7fff733b0c6c
Parent: The value is 0
It's obvious that the child process didn't affect at all the parent process. Frankly, I was expecting some "segmentation fault" error, because of accessing a not permitted memory address. But what really happened?
Remember, I'm not looking for a way to communicate processes, that's not the point. What I want to know is what did the code do. Inside the child process, the change is visible, so it DID something.
My main hypothesis is that pointers are not absolute to memory, they are relative to the process' stack. But I haven't been able to find an answer (no one in class knew, and googling I just found some questions about process communication) so I'd like to know from you, hopefully someone will know.
Thanks for taking your time reading!
The key here is the concept of a virtual address space.
Modern processors (Say anything newer then a 80386) have a memory management unit which maps from a per process virtual address space to physical memory pages under control of the kernel.
When the kernel sets up a process it creates a set of page table entries for that process that define the physical memory pages to virtual address space mapping, and it is in this virtual address space that the program executes.
Conceptually when you fork, the kernel copies the existing process pages to a new set of physical pages and sets up the new processes page tables so that as far as the new process is concerned it appears to be running in the same virtual memory layout as the original one had, while actually addressing entirely different physical memory.
The detail is more subtle as nobody wants to waste time copying hundreds of MB of data unless such is necessary.
When the process calls fork() the kernel sets up a second set of page table entries (for the new process), but points them at the same physical pages as the original process, it then sets the flag in both sets of pages to make the mmu consider them read only.....
As soon as either process writes to a page, the memory management unit generates a page fault (due to the PTE entry having the read only flag set), and the page fault handler then allocates a new page from physical memory, copies the data over, updates the page table entry and sets the pages back to read/write.
In this way, pages are only actually copied the first time either process tries to make a change to a copy on write page, and the slight of hand goes completely unnoticed by either process.
Regards, Dan.
Logically, the fork()ed process gets its own, independent copy of more or less the whole state of the parent process. That couldn't work if pointers in the child referred to memory belonging to the parent.
The details of how a particular UNIX-like kernel makes that work can vary. Linux implements the child process's memory via copy-on-write pages, which makes fork()ing comparatively cheap relative to other possible implementations. In that case, the child's pointers really do point to the parent process's memory, up until such time that either child or parent tries to modify that memory, at which time a copy is made for the child to use. That all relies on the underlying virtual memory system. Other UNIX and UNIX-like systems can and have done it differently.
The child modified a pointer that is perfectly legal in its address space because it is a copy of its parent. There was no effect on the parent because the memory is not logically shared. Each process gets to go its separate way after the fork.
UNIX has a number of ways of creating shared memory (where one process can modify memory and have that modification seen by another process), but fork is not one of them. And it's a good thing because otherwise, synchronization between the parent and child would be almost impossible.
How is the implementation of threads done in a system?
I know that child processes are created using the fork() call
and a thread is a light weight. How does the creation of a thread differ from that of a child process?
Threads are created using the clone() system call that can make a new process that shares memory space and some of the kernel control structures with its parent. These processes are called LWPs (light-weight processes) and are also known as kernel-level threads.
fork() creates a new process that initially shares memory with its parent but pages are copy-on-write, which means that separate memory pages are created when the content of the original one is altered. Thus both parent and child processes can no longer change each other's memory and effectively they run as separate processes. Also the newely forked child is a full-blown processes with its separate kernel control structures.
Each process has its own address space aka range of virtual addresses that the process can access. When a new process is forked a duplicate copy of all the resources involved has to be made. After the forking is complete the child and the parent have their own distinct address space and all the resources involved within it.Naturally, this is an performance intensive operation.
While all threads in the same process share the same address space, So when a new thread is spawned each thread only needs its own stack and there is no duplication of all resources as in case of processes.Hence spawning of an thread is considerably less performance intensive.
Ofcourse the two operations cannot and should not be compared because both provide essentially different features for different requirements.
Well it differs very much, first of all child process is in some way copy of parent program and have all variables duplicated, and you differ child from parent by its PID. Threads are like new programs , they run at the same time as main program (it looks like at the same time, due to slicing time of cpu by os ). Threads could use global variables in program, but they don't make duplicate as processes. So it`s much cheaper to use threads then new processes.
Well you've read the important parts, now here's something behind the curtains:
In current implementations(where current means the last few decades), the process memory isn't technically copied immediately upon forking. Read-only sections are just shared between the two processes (as they can't change anyway), as well as the read-only parts of shared libraries, of course. But most importantly, everything writeable is initially also just shared. However, it is shared in a write-protected manner, and as soon as you write to the child process memory (e.g. by incrementing a variable), a page fault is generated in the kernel, which only then causes the kernel to actually copy the respective page (where the modification then occurs).
This great optimization, which is called "copy on write", results in child processes usually not really consuming exactly as much (physical) memory as their parent processes. To the program developer (and user), however, it's completely transparent.