I have a couple of questions.
Suppose a program is compiled using 2 object files. Each uses malloc and free in most of their functions. But these object files were generated at different times and happen to be using different malloc implementations. Let's say the implementations share variable names and function names. Will the program work fine or not? Why?
If a program has object file 1 and 2, code from object file 1 call malloc and allocates some memory then frees it. Now code from object file 2 calls malloc. Can it use the memory that was freed? How does it work underneath?
Trying to provide a useful answer, even though it's far from complete.
Part 1.
First, it's hard enough to link the program with two implementations of malloc sharing function names: duplicate definitions usually cause linker errors. I can see how we manage to do it using GNU binutils, and there probably are some equivalent tricks for other toolchains. For the rest of the answer, let's assume we managed to link two implementations successfully. (It's usually a good thing that you get linker errors instead of mixing two implementations, possibly even introducing malloc/free asymmetry which has almost no chance to work).
Let's also assume that memory allocated with one particular implementation is always freed using free from the same implementation. Otherwise, it's virtually guaranteed to fail.
Two implementations may work together, or they may interfere, depending on how they request more memory from the OS when their local heaps run out of space. MS Windows has a system interface for managing heaps, and two different mallocs are likely to be built on top of it; then nothing prevents them from working together. Implementations requesting memory with sbrk-like call will work together if they're both ready that someone else will request sbrk increase independently of malloc. I'd expect that malloc from glibc won't fail here, but I'm not really sure.
Part 2.
If the implementation used by object 1 is able to return memory to OS, memory can be reused by the implementation called by object 2. That is, memory reuse may happen but it's less likely than when a single implementation is used.
The possibility of returning memory to OS depends on malloc/free implementation, and may also depend on allocated chunk size and various system settings. For example, glibc uses anonymous mmap for large chunks of memory, and these chunks are unmapped when freed.
Related
(This question concerns only the logical addresses)
I was experimenting with some code where I print out the addresses of different types/scopes of variables to better visualize the process image.
My confusion arose when I printed the addresses a few variables that have been allocated on the heap by malloc, and then also printed the address of the printf function out of curiosity.
What I discovered is that printf was being stored at a much higher address (ie closer to the stack) on the heap than my malloc allocated variables. This doesn't make sense to me because I assumed that the library functions would be loaded on the heap first thing at runtime before any other instructions are executed. I even put a printf statement before any malloc statements, in case library functions were loaded on the fly as they were needed, but it didn't change anything.
Thanks.
(This answer concerns Unix only. I don't know how it is on Windows.)
Most shared libraries are loaded into RAM before control reaches main, and the library containing printf definitely will be. The functions in dlfcn.h can be used to load more shared libraries during execution of the program, that's the most important exception.
Shared libraries have never been loaded as part of the "heap", if by that you mean the region of memory used to satisfy malloc requests. They are loaded using the system primitive mmap, and could be placed anywhere at all in memory. As user3386109 pointed out in a comment on the question, on modern systems, their locations are intentionally randomized as a countermeasure for various exploits.
I'm wrapping malloc() for some reason. I would like to have some (system-specific, at-run-time) information beyond what I can get by merely calling it. For example:
What's the minimum alignment malloc() is using for allocation?
When allocating a specific stretch of memory, how much did it actually allocate (this could theoretically be more than the amount requested)?
Whether (assuming no other concurrent operations) a realloc() will succeed with the same original address or require a move.
Note: I'd like as portable as answer as possible, but a platform-specific answer is still relevant: Linux, Windows, MacOs, Un*x.
glibc implements malloc_usable_size, which returns the actual allocation size available for application use. Some alternative mallocs implement it as well. Note that glibc can perform a non-moving realloc even if the requested new size is larger than the one malloc_usable_size, so it is not useful for this purpose.
For the other things you are asking, there are no clear answers. Theoretically, malloc should provide memory at least aligned to _Alignof (max_align_t), but many implementations do not do this for various reasons:
max_align_t comes from a compiler such as GCC and thus reflects the compiler's view of the world, and not what malloc provides (see glibc malloc is incompatible with GCC 7 for an example). The C standard assumes a uniform implementation, but in practice, the compiler, the C run-time library, and even malloc are separate components, built from different sources, and on different release cycles, so they can fall out of sync, and a compile-time constant such as _Alignof (max_align_t) will rarely accurately reflect what malloc does at run-time.
Providing the ABI-mandated 16 byte alignment on x86-64 for allocations of 8 or 4 bytes is wasteful.
A malloc implementation may have internal constraints which result in larger alignment then what is required by the architecture specification. Applications obviously cannot rely on that, but it is still observable.
Your question about non-moving realloc does not even have a proper answer: For a multi-threaded program, another thread might place an allocation which blocks the enlargement of the current allocation between the determination of the resize limit and the actual realloc call. A non-moving version of realloc could be a useful addition, but the interface would be quite different (probably something along the lines of please resize to this maximum value if possible without moving the block, otherwise return the largest possible size or something like that).
If you want a portable answer to these questions, your best bet is to implement your own allocation scheme. It would be safer to not use the names malloc(), calloc(), realloc(), free(), strdup(), etc. because you might run into problems with dynamic name resolution, even if your reimplementation of the standard functions is conformant.
Any source code you control could be made to call your allocator by defining a set of macros at the head of every module (via a common header file).
Using system specific tricks to retrieve information from the allocator's metadata is risky because your program will bind to the C library dynamically, so it is possible that such structures change from one system to another, even for the same operating system.
Re-implementing malloc() in terms of lower level systems calls such as mmap() or other system specific stuff seems deceptively simple, but it is a lot of work to make it correct, stable, efficient and reliable. You should look at available proven alternative implementations of malloc and try and tailor them to your needs.
My process which is linked to multiple libraries is causing a memory leak. The memory leak is coming from one of the libraries. I am trying to see if there is a way to identify the memory allocated from the functions residing in these libraries. what size each library is using?
Would memory allocator follow any specific way while allocating based on where malloc is called from. Like, if it is called from Lib A, allocation will happen from address starts from 0xA, for lib B, 0xB etc.
Basically, I 'm trying to see if there is a way to identify the leaking library and leaked memory and to dump that.
This is going to be a bit hard if you do it without the help of external tools. You have to be aware that there's nothing like a "gauge" telling your process how much memory it's actually using, and which library function allocated that memory. This has basically to do with two things:
Your OS, which is handing your process the memory, doesn't care or know which library asked for memory -- getting a new page mapped into your process' memory is just a syscall like any other.
Usually, libc's are the ones providing functionality like malloc() and free() to programs/libraries/programmers. These functions wrap your OS'es memory assignment/unassignment (in fact, mapping and unmapping) functionalities; this allows you to allocate and free memory in units that are not multiples of page sizes (4kB, usually). However, this also means that you can't really rely on your OS to tell you how much of the memory your process got really is in use, how much has been properly cleaned up, and how much is leaking.
Therefore, you'll need some mechanism to deal with libc and your OS, that allows you to inspect what your process is doing inside. A typical tool is Valgrind. It's not overly complex, so I encourage you to try it.
malloc is not guaranteed to return 0'ed memory. The conventional wisdom is not only that, but that the contents of the memory malloc returns are actually non-deterministic, e.g. openssl used them for extra randomness.
However, as far as I know, malloc is built on top of brk/sbrk, which do "return" 0'ed memory. I can see why the contents of what malloc returns may be non-0, e.g. from previously free'd memory, but why would they be non-deterministic in "normal" single-threaded software?
Is the conventional wisdom really true (assuming the same binary and libraries)
If so, Why?
Edit Several people answered explaining why the memory can be non-0, which I already explained in the question above. What I'm asking is why the program using the contents of what malloc returns may be non-deterministic, i.e. why it could have different behavior every time it's run (assuming the same binary and libraries). Non-deterministic behavior is not implied by non-0's. To put it differently: why it could have different contents every time the binary is run.
Malloc does not guarantee unpredictability... it just doesn't guarantee predictability.
E.g. Consider that
return 0;
Is a valid implementation of malloc.
The initial values of memory returned by malloc are unspecified, which means that the specifications of the C and C++ languages put no restrictions on what values can be handed back. This makes the language easier to implement on a variety of platforms. While it might be true that in Linux malloc is implemented with brk and sbrk and the memory should be zeroed (I'm not even sure that this is necessarily true, by the way), on other platforms, perhaps an embedded platform, there's no reason that this would have to be the case. For example, an embedded device might not want to zero the memory, since doing so costs CPU cycles and thus power and time. Also, in the interest of efficiency, for example, the memory allocator could recycle blocks that had previously been freed without zeroing them out first. This means that even if the memory from the OS is initially zeroed out, the memory from malloc needn't be.
The conventional wisdom that the values are nondeterministic is probably a good one because it forces you to realize that any memory you get back might have garbage data in it that could crash your program. That said, you should not assume that the values are truly random. You should, however, realize that the values handed back are not magically going to be what you want. You are responsible for setting them up correctly. Assuming the values are truly random is a Really Bad Idea, since there is nothing at all to suggest that they would be.
If you want memory that is guaranteed to be zeroed out, use calloc instead.
Hope this helps!
malloc is defined on many systems that can be programmed in C/C++, including many non-UNIX systems, and many systems that lack operating system altogether. Requiring malloc to zero out the memory goes against C's philosophy of saving CPU as much as possible.
The standard provides a zeroing cal calloc that can be used if you need to zero out the memory. But in cases when you are planning to initialize the memory yourself as soon as you get it, the CPU cycles spent making sure the block is zeroed out are a waste; C standard aims to avoid this waste as much as possible, often at the expense of predictability.
Memory returned by mallocis not zeroed (or rather, is not guaranteed to be zeroed) because it does not need to. There is no security risk in reusing uninitialized memory pulled from your own process' address space or page pool. You already know it's there, and you already know the contents. There is also no issue with the contents in a practical sense, because you're going to overwrite it anyway.
Incidentially, the memory returned by malloc is zeroed upon first allocation, because an operating system kernel cannot afford the risk of giving one process data that another process owned previously. Therefore, when the OS faults in a new page, it only ever provides one that has been zeroed. However, this is totally unrelated to malloc.
(Slightly off-topic: The Debian security thing you mentioned had a few more implications than using uninitialized memory for randomness. A packager who was not familiar with the inner workings of the code and did not know the precise implications patched out a couple of places that Valgrind had reported, presumably with good intent but to desastrous effect. Among these was the "random from uninitilized memory", but it was by far not the most severe one.)
I think that the assumption that it is non-deterministic is plain wrong, particularly as you ask for a non-threaded context. (In a threaded context due to scheduling alea you could have some non-determinism).
Just try it out. Create a sequential, deterministic application that
does a whole bunch of allocations
fills the memory with some pattern, eg fill it with the value of a counter
free every second of these allocations
newly allocate the same amount
run through these new allocations and register the value of the first byte in a file (as textual numbers one per line)
run this program twice and register the result in two different files. My idea is that these files will be identical.
Even in "normal" single-threaded programs, memory is freed and reallocated many times. Malloc will return to you memory that you had used before.
Even single-threaded code may do malloc then free then malloc and get back previously used, non-zero memory.
There is no guarantee that brk/sbrk return 0ed-out data; this is an implementation detail. It is generally a good idea for an OS to do that to reduce the possibility that sensitive information from one process finds its way into another process, but nothing in the specification says that it will be the case.
Also, the fact that malloc is implemented on top of brk/sbrk is also implementation-dependent, and can even vary based on the size of the allocation; for example, large allocations on Linux have traditionally used mmap on /dev/zero instead.
Basically, you can neither rely on malloc()ed regions containing garbage nor on it being all-0, and no program should assume one way or the other about it.
The simplest way I can think of putting the answer is like this:
If I am looking for wall space to paint a mural, I don't care whether it is white or covered with old graffiti, since I'm going to prime it and paint over it. I only care whether I have enough square footage to accommodate the picture, and I care that I'm not painting over an area that belongs to someone else.
That is how malloc thinks. Zeroing memory every time a process ends would be wasted computational effort. It would be like re-priming the wall every time you finish painting.
There is an whole ecosystem of programs living inside a computer memmory and you cannot control the order in which mallocs and frees are happening.
Imagine that the first time you run your application and malloc() something, it gives you an address with some garbage. Then your program shuts down, your OS marks that area as free. Another program takes it with another malloc(), writes a lot of stuff and then leaves. You run your program again, it might happen that malloc() gives you the same address, but now there's different garbage there, that the previous program might have written.
I don't actually know the implementation of malloc() in any system and I don't know if it implements any kind of security measure (like randomizing the returned address), but I don't think so.
It is very deterministic.
This query is regarding allocation of memory using malloc.
Generally what we say is malloc allocates memory from heap.
Now say I have a plain embedded system(No operating system), I have normal program loaded where I do malloc in my program.
In this case where is the memory allocated from ?
malloc() is a function that is usually implemented by the runtime-library. You are right, if you are running on top of an operating system, then malloc will sometimes (but not every time) trigger a system-call that makes the OS map some memory into your program's address space.
If your program runs without an operating system, then you can think of your program as being the operating system. You have access to all addresses, meaning you can just assign an address to a pointer, then de-reference that pointer to read/write.
Of course you have to make sure that not other parts of your program just use the same memory, so you write your own memory-manager:
To put it simply you can set-aside a range of addresses which your "memory-manager" uses to store which address-ranges are already in use (the datastructures stored in there can be as easy as a linked list or much much more complex). Then you will write a function and call it e.g. malloc() which forms the functional part of your memory-manager. It looks into the mentioned datastructure to find an address of ranges that is as long as the argument specifies and return a pointer to it.
Now, if every function in your program calls your malloc() instead of randomly writing into custom addresses you've done the first step. You can write a free()-function which will look for the pointer it is given in the mentioned datastructure, and adapts the datastructure (in the naive linked-list it would merge two links).
The only real answer is "Wherever your compiler/library-implementation puts it".
In the embedded system I use, there is no heap, since we haven't written one.
From the heap as you say. The difference is that the heap is not provided by the OS. Your application's linker script will no doubt include an allocation for the heap. The run-time library will manage this.
In the case of the Newlib C library often used in GCC based embedded systems not running an OS or at least not running Linux, the library has a stub syscall function called sbrk(). It is the respnsibility of the developer to implement sbrk(), which must provide more memory the the heap manager on request. Typically it merely increments a pointer and returns a pointer to the start of the new block, thereafter the library's heap manager manages and maintains the new block which may or may not be contiguous with previous blocks. The previous link includes an example implementation.