I programming on a parallel implementation of an algorithm, which uses non thread-safe operations. Therefore I use fork() and POSIX-Shared Memory, which works fine. Now the questions. What happens with the dynamicaly allocated memory of the parent, when the child exits?
The code looks like this
int compute(....) {
// prepare shared memory
pid_t child = fork();
if ( child == 0 ) {
// process the child code
int *workspace = malloc(worksize*sizeof(int));
// copy the result to Shared memory
free(workspace);
exit(0);
}
// do parents work
wait(&status);
// retrieve results from shared memory
return 0;
}
The problem is that I do not know from where compute is called and which memory is allocated their. The only thing I hope I can grantee is the memory that is allocated by the parent is only used read only in the child. Dynammically allocated memory which is alloced by the child is freed by the child.
Does this cause a memory leak or not? Valgrind says yes and I dont have an idea to avoid this. Tracking all memory allocations is unfortunately not possible.
If the code looks exactly as shown, no memory leak is produced. As you say, the child frees all its memory before exit. valgrind may have some problems accounting forked processes.
Related
There are similar question all over and one which is closes is from this stack exchange site. But even tho I learned a lot reading them none of them a exactly answer my question.
Here is the question, say I have this program (it is a very simplified version)
inline void execute(char *cmd)
{
int exitstat = 0;
//lots of other thins happed
exitstat = execve(cmd, cmdarg, environ);
free(cmd), free(other passed adress); ## DO I NEED THIS, since I am already about to exit
_exit (exitstat);
}
int main(void)
{
int childstat;
char *str = malloc(6); //I also have many more allocated space in heap in the parent process
strcpy(str, "Hello");
pid_t childid = fork() //creat a child process, which will also get a copy of all the heap memories even tho it is CoW.
if (childid < 0)
exit(-1);
if (child == 0)
{
execute(str);
}
else
{
wait(&childstat);
free(str);
}
//do somethign else with str with other functions and the rest of the program
}
In this program, the parent process does its thing for a while, allocates a lot of process in the heap, free some, keep other for later and at some point it wants to execute some command, but it doesn't want to terminate, so it creates a child to do its biting.
The child then calls another function which will do some task and in the end use execve to execute the command passed to it. If this was successful there would be no problem, since the executed program will handle all the allocated spaces, but if it failed the child exits with a status code. The parent waits for the child to answer, and when it does it moves on to its next routine. The problem here is that when the child fails to execute all the heap data remains allocated, but does that matter? since,
it is about to be exited int he next line,
Even though I am getting started with this, I have learned that a new process is created during fork, so the memory leak shouldn't affect the parent since the child is about to die.
If my assumption is correct and this leaks doesn't actually matter, why does valgrind lament about them?
Would there be a better way to free all the memories in the child heap with out actually passing all the memores (str, and the others in this example) to the execute function and laboriously call free each time? does kill() have a mechanism for this?
Edit
Here is working code
#include <unistd.h>
#include <sys/wait.h>
#include <stdlib.h>
#include <string.h>
inline void execute(char *cmd)
{
extern char **environ;
int exitstat = 0;
char *cmdarg[2];
cmdarg[0] = cmd;
cmdarg[1] = NULL;
exitstat = execve(cmd, cmdarg, environ);
free(cmd);
_exit (exitstat);
}
int main(void)
{
int childstat;
char *str = malloc(6);
pid_t childid;
strcpy(str, "Hello");
childid = fork();
if (childid < 0)
exit(-1);
if (childid == 0)
{
execute(str);
}
else
{
wait(&childstat);
free(str);
}
return (0);
}
When there is a free inside the child process here is valgrinds result.
When run with the free inside the child process is commented.(#free(cmd) in the execute function).
As you might see there is no overall error () since this programm is short lived. But in my real program which has an infinite loop every problem in the child () matters. So I was asking if I be worried by this leaked memories in this child process just before exiting and hunt them down or just leave them as they are in another virtual space and they are cleaned as their processes exit (but in that case why is valgrind still complaining about them if their corresponding process clean them during exit)
exitstat = execve(cmd, cmdarg, environ);
free(cmd), free(other passed adress);
If successful, execve does not return. No code is executed after it, the execution is terminated and the cmd process is started. All memory hold by current process is freed anyway by the operating system. The code free(cmd), free(other passed adress); will not execute if execve call is successful.
The only case where those liens could be relevant are in case of execve error. And in case execve returns with an error, yes, you should free that memory. I much doubt most programs with execve actually do that and I believe they would just call abort() after failed execve call.
inline void execute(char *cmd)
Please read about inline. inline is tricky. I recommend not use it, forget it exists. I doubt the intention here is to make an inline function - there is no other function the compiler can choose from anyway. After fixing typos in code, I could not compile the code because of undefined reference to execute - inline has to be removed.
The problem here is that when the child fails to execute all the heap data remains allocated, but does that matter?
Ultimately it depends on what do you want to do after the child fails. If you want to just call exit and you do not care about memory leaks in such case, then just don't care and it will not matter.
I see glibc exec*.c functions try to avoid dynamic allocation as much as possible and use variable length arrays or alloca().
If my assumption is correct and this leaks doesn't actually matter, why does valgrind lament about them?
It doesn't. If you correct all the mistakes in your code, then valgrind will not "lament" about memory leaks before calling execve (well, unless the execve call fails).
Would there be a better way to free all the memories in the child heap with out actually passing all the memores (str, and the others in this example) to the execute function and laboriously call free each time?
"Better" will tend to be opinion based. Subjectively: no. But writing/using own wrapper around dynamic allocation with garbage collector to free all memory is also an option.
does kill() have a mechanism for this?
No, sending a signal seems unrelated to freeing memory.
There is no simple solution.
On termination, the OS will free all memory for a process. That means that you don't have to free everything. However, for memory that is no longer needed it is usually best to free it as soon as possible. That leaves things like buffers that could potentially be used right up to the last moment before termination.
Reasons to free memory.
There's only one that I can think of. If you have a large number of blocks in use at the end it can be difficult to sort the wheat from the chaff. Either you end up with enormous logs that no-one looks at, or you have to maintain a suppression file. Either way you risk overlooking genuine issues.
Reasons not to free memory
Small performance gain
It can be quite difficult sometimes to free everything. For instance if you use putenv() then knowing whether the string was added (and needs freeing) or replaced (must not be freed) is a bit tricky.
If you need to use atexit() to free things, then there is a limit on the number of functions that you can call via atexit(). That might be a problem if you have many things being freed in many modules.
My advice is try to free most things but don't sweat it or you'll get bitten by the law of diminishing returns. Use a suppression file judiciously for the few percent that are tricky to free.
What i want to achieve is the following:
Spawn a new childprocess (pchild) that DOES NOT use its own, but the memoryblock from its parentprocess (pparent).
Why i want to achieve this behavior:
Think of multiple tests where the first one leads to segvault.
Normally your process would stop here due to segfault, all the other tests wouldn't be executed anymore. Therefore i want to encapsule each test in its own process.
Main Problem:
Once i spawn a process it gets its own memory copy (well, i'm
aware of the fact that this is not totally true for all OS, due to 'copy on write' technique). Think of e.g. testing tree functionallity where i have a node structure that has two pointers to other nodes. Once i retrieve a node by e.g. using a pipe or some shared memory block those pointers point to an address which is part of the memory block of the pchild and therefore i get a segvault when i try from pparent to get the childnode by following the pointers inside the node structure.
A Thread isn´t usefull, due to the main behavior some OS have once a segfault happens. (Killing child and father due to 'unclear state' ).
What i have so far (only fork testing part):
int main (void) {
// forking
pid_t pid = fork();
if (pid < 0 ) {
// somewhat went wrong
printf("An error occured!");
} else if (pid != 0) { // inside parent
// closing writing end, as not needed
if(wait(NULL)!=0){
printf("Segfault in Child\n");
} else {
printf("Everyone is done!\n");
}
} else {
printf("Child forked");
char *s = (char *)0xDEADBEEF;
*s = 'a';
printf("this probally is never executed due to segfault\n");
}
return 0;
}
Now my idea is to try to let the pchild access the memory segment of pparent only.
I'd welcome any ideas on how to do so.
Greetings,
Lars
Your question isn't very clear, but I think you have an XY problem.
My understanding is that you want to run a series of tests, where each test sees the results of previous tests that succeeded, but not tests that crashed/failed. If this is the case, one approach would be:
fork before each test
Execute the test in the child.
If the test succeeded, have the parent exit or simply wait while the child forks again and executes the next test in its child. If the test failed, have the parent fork again to do the next test.
Another approach might be to keep your data structures only in shared memory allocated by mmap with MAP_SHARED|MAP_ANON, but then if one test left them in an inconsistent state, all future test results would be junk.
Your idea of sharing all memory between processes is technically possible, but it will immediately blow up because they clobber each other's state.
Every process has it's own virtual memory adress space so there is no way to give a RAM adress to another process (well you can send a number, but in another virtual memory address space it will point to different 'real' place).
All you need is another thread. Threads of same process share one virtual memory space.
try to use vfork(); instead of fork();systemcall.
that shares the Resources of parent and child process.
I quote "when a process creates a new process using fork() call, Only the shared memory segments are shared between the parent process and the newly forked child process. Copies of the stack and the heap are made for the newly created process" from "operating system concepts" solutions by Silberschatz.
But when I tried this program out
#include <stdio.h>
#include <sys/types.h>
#define MAX_COUNT 200
void ChildProcess(void); /* child process prototype */
void ParentProcess(void); /* parent process prototype */
void main(void)
{
pid_t pid;
char * x=(char *)malloc(10);
pid = fork();
if (pid == 0)
ChildProcess();
else
ParentProcess();
printf("the address is %p\n",x);
}
void ChildProcess(void)
{
printf(" *** Child process ***\n");
}
void ParentProcess(void)
{
printf("*** Parent*****\n");
}
the result is like:
*** Parent*****
the address is 0x1370010
*** Child process ***
the address is 0x1370010
both parent and child printing the same address which is in heap.
can someone explain me the contradiction here. please clearly state what are all the things shared by the parent and child in memory space.
Quoting myself from another thread.
When a fork() system call is issued, a copy of all the pages
corresponding to the parent process is created, loaded into a separate
memory location by the OS for the child process. But this is not
needed in certain cases. Consider the case when a child executes an
"exec" system call or exits very soon after the fork(). When the
child is needed just to execute a command for the parent process,
there is no need for copying the parent process' pages, since exec
replaces the address space of the process which invoked it with the
command to be executed.
In such cases, a technique called copy-on-write (COW) is used. With
this technique, when a fork occurs, the parent process's pages are not
copied for the child process. Instead, the pages are shared between
the child and the parent process. Whenever a process (parent or child)
modifies a page, a separate copy of that particular page alone is made
for that process (parent or child) which performed the modification.
This process will then use the newly copied page rather than the
shared one in all future references. The other process (the one which
did not modify the shared page) continues to use the original copy of
the page (which is now no longer shared). This technique is called
copy-on-write since the page is copied when some process writes to it.
Also, to understand why these programs appear to be using the same space of memory (which is not the case), I would like to quote a part of the book "Operating Systems: Principles and Practice".
Most modern processors introduce a level of indirection, called
virtual addresses. With virtual addresses, every process's memory
starts at the "same" place, e.g., zero.
Each process thinks that it has the entire machine to itself, although
obviously that is not the case in reality.
So these virtual addresses are translations of physical addresses and doesn't represent the same physical memory space, to leave a more practical example we can do a test, if we compile and run multiple times a program that displays the direction of a static variable, such as this program.
#include <stdio.h>
int main() {
static int a = 0;
printf("%p\n", &a);
getchar();
return 0;
}
It would be impossible to obtain the same memory address in two
different programs if we deal with the physical memory directly.
And the results obtained from running the program several times are...
Yes, both processes are using the same address for this variable, but these addresses are used by different processes, and therefore aren't in the same virtual address space.
This means that the addresses are the same, but they aren't pointing to the same physical memory. You should read more about virtual memory to understand this.
The address is the same, but the address space is not. Each process has its own address space, so parent's 0x1370010 is not the same as child's 0x1370010.
You're probably running your program on an operating system with virtual memory. After the fork() call, the parent and child have separate address spaces, so the address 0x1370010 is not pointing to the same place. If one process wrote to *x, the other process would not see the change. (In fact those may be the same page of memory, or even the same block in a swap-file, until it's changed, but the OS makes sure that the page is copied as soon as either the parent or the child writes to it, so as far as the program can tell it's dealing with its own copy.)
When the kernel fork()s the process, the copied memory information inherits the same address information since the heap is effectively copied as-is. If addresses were different, how would you update pointers inside of custom structs? The kernel knows nothing about that information so those pointers would then be invalidated. Therefore, the physical address may change (and in fact often will change even during the lifetime of your executable even without fork()ing, but the logical address remains the same.
Yes address in both the case is same. But if you assign different value for x in child process and parent process and then also prints the value of x along with address of x, You will get your answer.
#include <stdio.h>
#include <sys/types.h>
#include <stdlib.h>
#include <unistd.h>
#define MAX_COUNT 200
void ChildProcess(void); /* child process prototype */
void ParentProcess(void); /* parent process prototype */
void main(void)
{
pid_t pid;
int * x = (int *)malloc(10);
pid = fork();
if (pid == 0) {
*x = 100;
ChildProcess();
}
else {
*x = 200;
ParentProcess();
}
printf("the address is %p and value is %d\n", x, *x);
}
void ChildProcess(void)
{
printf(" *** Child process ***\n");
}
void ParentProcess(void)
{
printf("*** Parent*****\n");
}
Output of this will be:
*** Parent*****
the address is 0xf70260 and value is 200
*** Child process ***
the address is 0xf70260 and value is 100
Now, You can see that value is different but address is same. So The address space for both the process is different. These addresses are not actual address but logical address so these could be same for different processes.
I want to know how copy-on-write happens in fork().
Assuming we have a process A that has a dynamical int array:
int *array = malloc(1000000*sizeof(int));
Elements in array are initialized to some meaningful values.
Then, we use fork() to create a child process, namely B.
B will iterate the array and do some calculations:
for(a in array){
a = a+1;
}
I know B will not copy the entire array immediately, but when does the child B allocate memory for array? during fork()?
Does it allocate the entire array all at once, or only a single integer for a = a+1?
a = a+1; how does this happen? Does B read data from A and write new data to its own array?
I wrote some code to explore how COW works. My environment: ubuntu 14.04, gcc4.8.2
#include <stdlib.h>
#include <stdio.h>
#include <sys/sysinfo.h>
void printMemStat(){
struct sysinfo si;
sysinfo(&si);
printf("===\n");
printf("Total: %llu\n", si.totalram);
printf("Free: %llu\n", si.freeram);
}
int main(){
long len = 200000000;
long *array = malloc(len*sizeof(long));
long i = 0;
for(; i<len; i++){
array[i] = i;
}
printMemStat();
if(fork()==0){
/*child*/
printMemStat();
i = 0;
for(; i<len/2; i++){
array[i] = i+1;
}
printMemStat();
i = 0;
for(; i<len; i++){
array[i] = i+1;
}
printMemStat();
}else{
/*parent*/
int times=10;
while(times-- > 0){
sleep(1);
}
}
return 0;
}
After fork(), the child process modifies a half of numbers in array, and then modifies the entire array. The outputs are:
===
Total: 16694571008
Free: 2129162240
===
Total: 16694571008
Free: 2126106624
===
Total: 16694571008
Free: 1325101056
===
Total: 16694571008
Free: 533794816
It seems that the array is not allocated as a whole. If I slightly change the first modification phase to:
i = 0;
for(; i<len/2; i++){
array[i*2] = i+1;
}
The outputs will be:
===
Total: 16694571008
Free: 2129924096
===
Total: 16694571008
Free: 2126868480
===
Total: 16694571008
Free: 526987264
===
Total: 16694571008
Free: 526987264
Depends on the Operating System, hardware architecture and libc. But yes in case of recent Linux with MMU the fork(2) will work with copy-on-write. It will only (allocate and) copy a few system structures and the page table, but the heap pages actually point to the ones of the parent until written.
More control over this can be exercised with the clone(2) call. And vfork(2) beeing a special variant which does not expect the pages to be used. This is typically used before exec().
As for the allocation: the malloc() has meta information over requested memory blocks (address and size) and the C variable is a pointer (both in process memory heap and stacks). Those two look the same for the child (same values because same underlying memory page seen in the address space of both processes). So from a C program point of view the array is already allocated and the variable initialized when the process comes into existence. The underlying memory pages are however pointing to the original physical ones of the parent process, so no extra memory pages are needed until they are modified.
If the child allocates a new array it depends if it fits into the already existing heap pages or if the brk of the process needs to be increased. In both cases only the modified pages get copied and the new pages get allocated only for the child.
This also means that the physical memory might run out after malloc(). (Which is bad as the program cannot check the error return code of "a operation in a random code line"). Some operating systems will not allow this form of overcommit: So if you fork a process it will not allocate the pages, but it requires them to be available at that moment (kind of reserves them) just in case. In Linux this is configurable and called overcommit-accounting.
Some systems have a system call vfork(), which was originally
designed as a lower-overhead version of fork(). Since
fork() involved copying the entire address space of the process,
and was therefore quite expensive, the vfork() function was
introduced (in 3.0BSD).
However, since vfork() was introduced, the
implementation of fork() has improved drastically, most notably
with the introduction of 'copy-on-write', where the copying of the
process address space is transparently faked by allowing both processes
to refer to the same physical memory until either of them modify
it. This largely removes the justification for vfork(); indeed, a
large proportion of systems now lack the original functionality of
vfork() completely. For compatibility, though, there may still be
a vfork() call present, that simply calls fork() without
attempting to emulate all of the vfork() semantics.
As a result, it is very unwise to actually make use of any of the
differences between fork() and vfork(). Indeed, it is
probably unwise to use vfork() at all, unless you know exactly
why you want to.
The basic difference between the two is that when a new process is created with vfork(), the parent process is temporarily suspended, and the child process might borrow the parent's address space. This strange state of affairs continues until the child process either exits, or calls execve(), at which point the parent process continues.
This means that the child process of a vfork() must be careful to
avoid unexpectedly modifying variables of the parent process. In
particular, the child process must not return from the function
containing the vfork() call, and it must not call
exit() (if it needs to exit, it should use _exit();
actually, this is also true for the child of a normal fork()).
I am confused about something. I have read that when a child is created by a parent process, the child gets a copy of its parent's address space. What does it mean by copy?
If I use the code below, then it prints the same value for variable 'a' which is on the heap in both tthe child and parent. So what is happening here?
int main ()
{
pid_t pid;
int *a = (int *)malloc(4);
printf ("heap pointer %p\n", a);
pid = fork();
if (pid < 0) {
fprintf (stderr, "Fork Failed");
exit(-1);
}
else if (pid == 0) {
printf ("Child\n");
printf ("in child heap pointer %p\n", a);
}
else {
wait (NULL);
printf ("Child Complete\n");
printf ("in parent heap pointer %p\n", a);
exit(0);
}
}
The child gets an exact copy of the parents address space, which in many cases is likely to be laid out in the same format as the parent address space. I have to point out that each one will have it's own virtual address space for its memory, such that each could have the same data at the same address, yet in different address spaces. Also, Linux uses copy on write when creating child processes. This means that the parent and child will share the parent address space until one of them does a write, at which point the memory will be physically copied to the child. This eliminates unneeded copies when execing a new process. Since you're just going to overwrite the memory with a new executable, why bother copying it?
Yes, you will get the same virtual address, but remember each one has it's own process virtual address spaces.
Till there is a Copy-On-Write operation done everything is shared.
So when you try to strcpy or any write operation the Copy-On-Write takes place which means the child process virtual address of pointer a will be updated for the child process, but not so for the parent process.
A copy means exactly that, a bit-identical copy of the virtual address space. For all intents and purposes, the two copies are indistinguishable, until you start writing to one (the changes are not visible in the other copy).
With fork() the child process receives a new address space where all the contents of the parent address space are copied (actually, modern kernels use copy-on-write).
This means that if you modify a or the value pointed by it in a process, the other process still sees the old value.
You get two heaps, and since the memory addresses are translated to different parts of physical memory, both of them have the same virtual memory address.