Is there anything that:
Allocates/deallocates/reallocates many variable sized objects without fragmentation (basically what malloc does) and
keeps track of all memory pages used in for these allocations so that I can
later mprotect all of these pages to make them read-only
preferably without any locking - all access will be single-threaded
that works on Linux and OS X, preferably with something equivalent on Windows?
I can't think of a way of doing this with standard memory allocation functions. The only strategy that comes to mind is using a custom memory pool instead of malloc. So my question is: is there a way to do this without a custom malloc or (if there isn't) suggestions on what to use?
I could wrap malloc and keep track of all pages it has used pretty easily how do I guarantee that once I have called mprotect on these pages malloc doesn't try to use memory that is "caught" either before the start or after the end of an allocated block within one of the affected pages?
The open source Memory Pool System will allocate memory in operating system page-sized chunks which the MPS does not itself touch. You can mprotect these pages if you like and be certain that they won't be touched by the allocator itself (which keeps all its data structures elsewhere) or by any other memory pool. If you use the MVT pool class you can also take advantage on inline lockless allocation as well. Linux, Mac OS X, and Windows are supported.
Disclaimer: I'm the architect of the MPS.
Related
In other words, why doesn't free() just return the memory to the operating system, and malloc simply request memory from the operating system?
This unpacks to three closely related questions:
Why does C need to manage its own heap? (Is it because the OS will only allow you to allocate and free contiguous memory of a minimum size?)
Assuming what I wrote in parentheses is true, why is it?
Can this problem affect the operating system itself, so that it's unable to allocate blocks of memory to any running processes?
Why does C need to manage its own heap?
It's not actually specified that it needs to, but it needs to implement malloc() and friends the way they are described in the standard. So if there was an OS already providing such an interface, a C implementation could just provide a tiny wrapper.
Is it because the OS will only allow you to allocate and free contiguous memory of a minimum size? And if that's true, what's the reason?
Yes. A typical OS will manage paged memory and map or unmap processes whole pages of memory. The unit of memory that can be "paged" depends on the hardware architecture. You might want to read some details on how memory management units (MMU) work. On architectures without MMU, the operating system might not do anything and a C implementation would just fullfill malloc() requests from a fixed location in physical address space.
malloc is a C method itself. You are using a standard library that provides it for you, but in the end, it is C code just like yours is.
In some operating systems, you can only get memory from the OS in the size of pages (using mmap). This is too big for your regular data structure.
Doing a system call every time you need memory is way too expensive.
Actually it is left for the implementation. The question :
why doesn't free() just return the memory to the operating system, and
malloc simply request memory from the operating system?
is wrong as nothing stops the implementation from doing it. So there is no answer for this question - every implementation can be potentially different (it only has to be standard compliant)
Why does C need to manage its own heap? (Is it because the OS will only allow you to allocate and free contiguous memory of a minimum size?)
Operating systems manage memory in pages. Allocating and freeing pages has a high overhead. Most allocations in C tend to be much smaller than a page size.
From the other answers, here's what I gathered the answer to be.
Most operating systems only allocate memory to processes in fixed sizes, called pages. When a process returns memory to the OS, it can only do so in pagefuls. A page is a sequence of memory of a fixed size. The start and end points of a page are fixed, so even if you've got a large enough amount of free memory, you won't be able to return it to the OS unless it's between the start and end points of a page.
On the other hand, you could imagine that there is no operating system (or that the operating system allocates memory to programs from its own heap). This helps me understand things better, because the operating system was getting in the way of my intuition, because it looked as if the freed memory was being dropped into a black hole, and the allocated memory was coming out of a similar black hole. Without any OS, all the memory in a computer could be pictured as belonging to a large sequence of cells. If you allocate all available memory in a computer, and then start freeing some memory, then you might not be able to find a large enough contiguous chunk of memory to fulfill a malloc request.
When we malloc memory, only virtual memory is available, and it actually pointed to zero-page. The real physical memory will be allocated when we try to write to the malloced memory, at this moment, there will be copy-on-wright that copy zeros from zero-page to physical memory which mapped by page-fault. My problem is, how/where zero-fill-on demand is implemented in linux source code, I want to disable this functionality to do some test. I guess it may happened in page-fault procedure, rather than brk() or mmap().
Similar topics related to zero-fill-on-demand. ZFOD and COW.
You want to use the MAP_UNINITIALIZED parameter to mmap and enable CONFIG_MMAP_ALLOW_UNINITIALIZED in your kernel compilation.
MAP_UNINITIALIZED (since Linux 2.6.33)
Don't clear anonymous pages. This flag is intended to improve
performance on embedded devices. This flag is honored only if
the kernel was configured with the CONFIG_MMAP_ALLOW_UNINITIAL‐
IZED option. Because of the security implications, that option
is normally enabled only on embedded devices (i.e., devices
where one has complete control of the contents of user memory).
If you want your userspace process to allocate real memory every *alloc call, I think in the next options:
If it is for performance reasons, you can replace all calloc calls for malloc+memset so processes will always have a real memory page. However, the kernel could still be able to merge some memory pages.
Disable memory overcommit so that every malloc will return the page at the moment. This way, your program will not be able to allocate more memory than available (RAM + swap). See https://www.kernel.org/doc/Documentation/vm/overcommit-accounting
How does actually malloc get the present free memory space available in the microcontroller.
Does it keep a list of areas unallocated continously in the runtime?
How does it get information of a previous malloc assignment memory allocation if there are two malloc statements in the code
How can one know which memory is free and which one is not at runtime. At compilation time we can know which all locations in RAM is assigned by the compiler for the variable. Does malloc uses this information to do this.
As the commentators said above, there are multiple implementations of malloc and the algorithm may vastly vary for each of these implementations. This is a vast and complicated area and you should read up on the memory management to get a complete idea on the topic.
In simple words, all the malloc implementations are backed up by the kernel's memory management schemes. The kernel see the whole system memory as pages for fixed size (4k, 8k etc) and all the allocations and frees are done on the pages. There will be a memory management subsystem exists for all the kernel implementations and which does the accounting of whole memory allocations and frees happening on the system. When you call a malloc, it will eventually reaches this memory management subsystem, and looks for the next available free page from the pool and allocates for the requesting process. Before giving the page to the requester, he will make sure to mark it as used and same way when you free up the memory it will add it back to the free pool and unmark used. There exists so many implementations on how the kernel does all these effectively (read up on memory manager implementations in linux)
In common implementations, there exists a minimal memory manager functionality in the userspace itself. The user space process itself maintains a free pool and when a malloc requests memory, before breaking in to kernel, it will look in its own free pool if memory is available. If available it will mark it up and satisfies the request without the help of kernel. Similarly, when you free up the memory, the freed up chunk of memory will not immediately go back to kernel's free pool instead it will stay with the process's free pool so that next malloc can use this.
As I said in the beginning, this is a huge and complicated topic and you can find a lot of documentations available in the internet about this.
Is it possible to 'reserve' memory before a malloc() call? In other words, can I do something (perhaps OS-specific) which ensures there is a certain amount of free memory available, so that you know that your next malloc() (or realloc() etc.) call won't return NULL due to lack of memory?
The 'reservation' or 'pre-allocation' can fail just like a malloc, but if it succeeds, I want to be sure my next malloc() succeeds.
Notes:
Yes, I know, I want to allocate memory before allocating memory. That's exactly right. The thing is the later allocations are not really under my control and I want to be able to assume they succeed.
Bonus points for an answer regarding multi-threaded code as well.
My motivation: I was considering adopting the use of glib for my C development, but apparently it abort()s when it fails to allocate memory, and that's not acceptable to me.
Perhaps a solution which dynamically replaces the malloc symbol with something else? Or the symbol for the function wrapping the sbrk system call?
With glibc you can hook the allocation functions:
https://www.gnu.org/software/libc/manual/html_node/Hooks-for-Malloc.html
Now that you control memory allocation in your program, you can do what you like, including writing a function to reserve a (possibly thread-local, since you asked about multi-threading) chunk of memory from the system that future calls to your malloc and realloc hooks will use to return memory.
Obviously, you need to somehow know in advance an upper bound how much memory will be required by the series of malloc calls that you need to not fail.
Back in the old Mac Toolbox days it was extremely common to use a chunk of memory called a "rainy-day fund." You'd allocate enough memory such that if you freed it, there'd be enough free memory to throw up a dialog box explaining that the app had run out of memory, save your work, and exit. Then you'd keep that pointer around until malloc() returned null, and at least you'd be guaranteed to be able to deal with it gracefully.
That was on a 100% real-memory system, though, and things these days are very different. Still, if we're talking about those small and simple real-memory systems that still exist, then a similar strategy still makes sense.
I realize the following does not directly answer your question with respect to malloc(). It is instead an attempt to offer up another avenue that might be applicable to your situation.
For a few years I was dealing with certified embedded systems. Two of the constraints were that 1) we were forbidden to free memory and 2) we were forbidden from allocating memory beyond a certain point during the initialization process. This was because fragmentation that could result from dynamic memory allocations and deallocations made it too costly to certify (and guarantee that allocations would succeed).
Our solution was to allocate pools of memory during the early initialization process. The blocks of memory handled by a given pool would all be the same size, thereby avoiding the fragmentation issue. Different pools would handle differently sized memory blocks for a different purpose. This meant that we had to allocate enough memory up front for our worst case memory consumption scenario as well as manage those pools ourselves.
Hope this helps.
Obviously there's no magic way for your program to ensure your system has an arbitrary amount of memory, but you can get the memory as soon as your process starts, so that it won't fail unexpectedly part way through the work/day when it'll be a right pain.
On some OSes, simply doing a big malloc then freeing the memory immediately will still have called sbrk or similar OS function to grow your process memory, but even that's not a great solution because getting virtual address space is still a ways short of getting physical memory to back it when needed, so you'd want to write through that memory with some noise values, then even if it's swapped out to disk while unused you can expect that the virtual memory of the system is committed to your memory needs and will instead deny other programs (or smaller new/malloc requests you make later ;-P) memory should the system be running short.
Another approach is to seek an OS specific function to insist on locking memory pages in physical memory, such as mlock(2) on Linux.
(These kind of "I'm the most important thing on the server" assumptions tend to make for a fragile system once a few running programs have all taken that attitude....)
I have made a program in c and wanted to see, how much memory it uses and noticed, that the memory usage grows while normally using it (at launch time it uses about 250k and now it's at 1.5mb). afaik, I freed all the unused memory and after some time hours, the app uses less memory. Could it be possible, that the freed memory just goes from the 'active' memory to the 'wired' or something, so it's released when free space is needed?
btw. my machine runs on mac os x, if this is important.
How do you determine the memory usage? Have you tried using valgrind to locate potential memory leaks? It's really easy. Just start your application with valgrind, run it, and look at the well-structured output.
If you're looking at the memory usage from the OS, you are likely to see this behavior. Freed memory is not automatically returned to the OS, but normally stays with the process, and can be malloced later. What you see is usually the high-water mark of memory use.
As Konrad Rudolph suggested, use something that examines the memory from inside the process to look for memory links.
The C library does not usually return "small" allocations to the OS. Instead it keeps the memory around for the next time you use malloc.
However, many C libraries will release large blocks, so you could try doing a malloc of several megabytes and then freeing it.
On OSX you should be able to use MallocDebug.app if you have installed the Developer Tools from OSX (as you might have trouble finding a port of valgrind for OSX).
/Developer/Applications/PerformanceTools/MallocDebug.app
I agree with what everyone has already said, but I do want to add just a few clarifying remarks specific to os x:
First, the operating system actually allocates memory using vm_allocate which allocates entire pages at a time. Because there is a cost associated with this, like others have stated, the C library does not just deallocate the page when you return memory via free(3). Specifically, if there are other allocations within the memory page, it will not be released. Currently memory pages are 4096 bytes in mac os x. The number of bytes in a page can be determined programatically with sysctl(2) or, more easily, with getpagesize(2). You can use this information to optimize your memory usage.
Secondly, user-space applications do not wire memory. Generally the kernel wires memory for critical data structures. Wired memory is basically memory that can never be swapped out and will never generate a page fault. If, for some reason, a page fault is generated in a wired memory page, the kernel will panic and your computer will crash. If your application is increasing your computer's wired memory by a noticeable amount, it is a very bad sign. It generally means that your application is doing something that significantly grows kernel data structures, like allocating and not reaping hundreds of threads of child processes. (of course, this is a general statement... in some cases, this growth is expected, like when developing a virtual host or something like that).
In addition to what the others have already written:
malloc() allocates bigger chunks from the OS and spits it out in smaller pieces as you malloc() it. When free()ing, the piece first goes into a free-list, for quick reuse by another malloc if the size fits. It may at this time be merged with another free item, to form bigger free blocks, to avoid fragmentation (a whole bunch of different algorithms exist there, from freeLists to binary-sized-fragments to hashing and what not else).
When freed pieces arrive so that multiple fragments can be joined, free() usually does this, but sometimes, fragments remain, depending on size and orderof malloc() and free(). Also, only when a big such free block has been created will it be (sometimes) returned to the OS as a block. But usually, malloc() keeps things in its pocket, dependig on the allocated/free ratio (many heuristics and sometimes compile or flag options are often available).
Notice, that there is not ONE malloc/free algotrithm. There is a whole bunch of different implementations (and literature). Highly system, OS and library dependent.