I'm new to linux programming and i want to know is it possible to increase the heap size of a running process. If it is possible, please help me how to do it right. Thanks anyone for helping.
Heap is just memory. There is nothing special about it. Any memory can become heap. Diagrams showing a heap area are pedagogical, rather than real.
"Heap" is is "Heap" only because the memory is allocated by a heap manager. While most programs only have on heap manager, it is possible to have multiple heap managers.
Thus heap size is controlled by a heap manager. Most simple heap managers give the user no control over the heap size. The heap manager allocates more memory when it needs memory to respond to allocation calls.
Some heap managers give the user function calls to allow him to allocate an expand the heap size.
Just use a function like malloc() or calloc() to allocate memory dynamically. To deallocate the memory and return it to the heap, use free(). These functions will manage the size of the heap by expanding or shrinking it as needed.
Example:
Everything in heap is anonymous. You can't access the memory directly. Every access is indirect. So store the address of allocated memory returned by malloc() in a pointer.
int *ptr = malloc(sizeof(int));
We can use *ptr to access the memory's contents.
*ptr = 3;
printf("%d", *ptr);
Once you are done using the memory. You deallocate it with
free(ptr);
According to Peter van der Linden's book on C programming,
The end of the heap is marked by a pointer known as the "break". When the heap manager needs more memory, it can push the break further away using the system calls brk and sbrk.
You typically don't call brk yourself explicitly, but if you malloc enough memory, brk will eventually be called for you.
Your program may not call both malloc() and brk(). If you use malloc, malloc expects to have sole control over when brk and sbrk are called.
The limit on the maximum size of heap is determined by the size of virtual memory of the system.
Here's a crude reproduction of the image:
Related
From my book:
Recall from our first discussion that modern dynamic memory managers
not only use sbrk() but also mmap(). This process helps reduce the
negative effects of memory fragmentation when large blocks of memory
are freed but locked by smaller, more recently allocated blocks lying
between them and the end of the allocated space. In this case, had the
block been allocated with sbrk(), it would have probably remained
unused by the system for some time (or at least most of it).
Can someone kindly explain how using mmap reduces the negative effects of memory fragmentation? The given example didn't make any sense to me and wasn't clear at all.
it would have probably remained unused by the system for some time
Why this claim was made, when we free it the system can use it later. Maybe the OS keeps list of freed blocks in heap to use them when possible instead of using more space in heap.
Please Relate to both questions.
Advantages of mmap() over sbrk()?
brk/sbrk is LIFO. Let's say you increase the segment size by X number of bytes to make room for allocation A and X number of bytes to make allocation B, and then free A. You cannot reduce the allocated memory because B is still allocated. And since the segment is shared across the entire program, if multiple parts of the program use it directly, you will have no way of knowing whether particular part is still in use or not. And if one part of the program (let's say malloc) assumes entire control over the use of brk/sbrk, then calling them elsewhere will break the program.
By contrast, mmap can be unmapped in any order and allocation by one part of the program doesn't conflict with other parts of the program.
brk/sbrk are not part of the POSIX standard and thus not portable.
By contrast, mmap is standard and portable.
mmap can also do things like map files into memory which is not possible using brk/sbrk.
it would have probably remained unused by the system for some time
Why this claim was made
See 1.
Maybe the OS keeps list of freed block
There are no "blocks". There is one (virtual) block called the data segment. brk/sbrk sets the size of that block.
But doesn't mmap allocate on heap
No. "Heap" is at the end of the data segment and heap is what grows using brk/sbrk. mmap does not allocate in the area of memory that has been allocated using brk/sbrk.
mmap creates a new segment elsewhere in the address space.
does malloc actually save the free blocks that were allocated with sbrk for later usage?
If it is allocated using brk/sbrk in the first place, and if malloc hasn't reduced the size of the "heap" (in case that was possible), then malloc may reuse a free "slot" that has been previously freed. It would be a useful thing to do.
"then calling them elsewhere will break the program." can you give an example
malloc(42);
sbrk(42);
malloc(42); // maybe kaboom, who knows?
In conclusion: Just don't use brk/sbrk to set the segment size. Perhaps there's little reason to use (anonymous) mmap either. Use malloc in C.
When sbrk() is used, the heap is just one, large block of memory. If your pattern of allocating and freeing doesn't leave large, contiguous blocks of memory, every large allocation will need to grow the heap. This can result in inefficient memory use, because of all the unused gaps that are left in the heap.
With mmap(), you can have a bunch of independent blocks of mapped memory. So you could use the sbrk() heap for your small allocations, which can be packed neatly, and use mmap() for large allocations. When you're done with one of these large blocks, you can just remove the entire mapping.
I wish to put pointers allocated via GC_MALLOC in memory allocated with the standard malloc(). The manual says to not do this, but this memory is allocated in a library I do not have control over. Is it possible to designate the memory allocated by malloc() as uncollectable in the same way that GC_MALLOC_UNCOLLECTABLE() does for memory that it allocates? That is, so that it will know to scan that memory for pointers, but not attempt to free it.
The only alternative I can think of is to allocate some memory via GC_MALLOC_UNCOLLECTABLE() in addition to the malloc-allocated memory, just to place the pointer in both places, but I am hoping for a less awkward solution.
The VirtualAlloc() will allocate a virtual memory page that have some attributes (by the parameter "fdwProtect").
What about the memory that allocated by malloc() ?Is that have same attributes?
Is the memory by malloc() have attributes that "commintting" or "reserving" ?
Further more, what about other C/C++ lib function?
VirtualAlloc
This function allows you to specify additional options for memory allocation. But it allocates memory in large page with a minimum indicated by GetLargePageMinimum, you can commit, reserve with it. It's not for general use. Memory allocated by this function is automatically initialized to zero.
malloc
The standard C version to allocate memory. Prefer it if you are writing in C rather than C++, and your code needs to work on on other platforms, or someone specifically says that you need to use it. It's quite possible that, on Windows, malloc would be implemented on top of HeapAlloc. malloc can allocate any chunk of memory, it doesn't have any concept to commit and reserve by current standard. Memory allocated by this function is not initialized.
I recoded malloc() free() and realloc().
I have a linked list with the pointers returned by malloc().
The question is : what does free() really do ?
Currently, I did a memset() to have the same behavior of free().
But was it better just to set a flag in my list as 'is free' rather than doing a memset() in order to make it faster ?
Free : Call will unlink / unallocate the memory pointed by pointer so that other process can use it.
Memset : Call will set a memory / fill a memory location. It won't unlink / unallocate a memory and memory remain allocated / occupied till the program exist. Which can cause memory leaks.
You can use valgrind tool to check memory leaks.
And it is a better practice to unlink / unallocate a memory if its not required.
Usually free(3) does not do anything to the memory itself. (If security or privacy is a concern, you should clear memory before freeing.)
If you want to implement malloc, you need to have some database of free memory blocks. When memory is freed you should join it with adjoint free memory, if there is any. If a complete page ends up unused, you should tell the kernel, that you don't need it anymore (depending on how you got that memory in the first place)
The C library function void free(void *ptr) deallocates the memory previously allocated by a call to calloc, malloc, or realloc.
You should use it to prevent memory leaks.
What exactly is heap memory?
Whenever a call to malloc is made, memory is assigned from something called as heap. Where exactly is heap. I know that a program in main memory is divided into instruction segment where program statements are presents, Data segment where global data resides and stack segment where local variables and corresponding function parameters are stored. Now, what about heap?
The heap is part of your process's address space. The heap can be grown or shrunk; you manipulate it by calling brk(2) or sbrk(2). This is in fact what malloc(3) does.
Allocating from the heap is more convenient than allocating memory on the stack because it persists after the calling routine returns; thus, you can call a routine, say funcA(), to allocate a bunch of memory and fill it with something; that memory will still be valid after funcA() returns. If funcA() allocates a local array (on the stack) then when funcA() returns, the on-stack array is gone.
A drawback of using the heap is that if you forget to release heap-allocated memory, you may exhaust it. The failure to release heap-allocated memory (e.g., failing to free() memory gotten from malloc()) is sometimes called a memory leak.
Another nice feature of the heap, vs. just allocating a local array/struct/whatever on the stack, is that you get a return value saying whether your allocation succeeded; if you try to allocate a local array on the stack and you run out, you don't get an error code; typically your thread will simply be aborted.
The heap is the diametrical opposite of the stack. The heap is a large pool of memory that can be used dynamically – it is also known as the “free store”. This is memory that is not automatically managed – you have to explicitly allocate (using functions such as malloc), and deallocate (e.g. free) the memory. Failure to free the memory when you are finished with it will result in what is known as a memory leak – memory that is still “being used”, and not available to other processes. Unlike the stack, there are generally no restrictions on the size of the heap (or the variables it creates), other than the physical size of memory in the machine. Variables created on the heap are accessible anywhere in the program.
Oh, and heap memory requires you to use pointers.
A summary of the heap:
the heap is managed by the programmer, the ability to modify it is
somewhat boundless
in C, variables are allocated and freed using functions like malloc() and free()
the heap is large, and is usually limited by the physical memory available
the heap requires pointers to access it
credit to craftofcoding
Basically, after memory is consumed by the needs of programs, what is left is the heap. In C that will be the memory available for the computer, for virtual machines it will be less than that.
But, this is the memory that can be used at run-time as your program needs memory dynamically.
You may want to look at this for more info:
http://computer.howstuffworks.com/c28.htm
Reading through this, this is actually beyond the realms of C. C doesn't specify that there's a heap behind malloc; it could just as easily be called a linked list; you're just calling it a heap by convention.
What the standard guarantees is that malloc will either return a pointer to an object that has dynamic storage duration, and your heap is just one type of data structure which facilitates the provision of such a storage duration. It's the common choice. Nonetheless, the very developers who wrote your heap have recognised that it might not be a heap, and so you'll see no reference of the term heap in the POSIX malloc manual for example.
Other things that are beyond the realms of standard C include such details of the machine code binary which is no longer C source code following compilation. The layout details, though typical, are all implementation-specific as opposed to C-specific.
The heap, or whichever book-keeping data structure is used to account for allocations, is generated during runtime; as malloc is called, new entries are (presumably) added to it and as free is called, new entries are (again, presumably) removed from it.
As a result, there's generally no need to have a section in the machine code binary for objects allocated using malloc, however there are cases where applications are shipped standalone baked into microprocessors, and in some of these cases you might find that flash or otherwise non-volatile memory might be reserved for that use.