Just for educational purposes, I'm writing a C program without any malloc, and I'm checking that there's no heap usage by using mallinfo().uordblks. I've noticed that the function opendir triggers a huge spike in malloc'd memory according to mallinfo, and I'm not sure why. I'm wondering if there's a way to give opendir a stack-allocated buffer in order to do what it needs so that I can avoid this (similar to setvbuf, which I used to avoid buffering on the heap for stdout/stderr). Bascially, how do I read the contents of a directory without using heap-allocated memory?. If it makes a difference, I'm on a Linux machine.
You can't, any more than you could use stdio without the possibility that it calls malloc, or likewise many other components in libc. Fundamentally there's no reason that any of the standard library functions can't use malloc internally, although for many it would have to be conditional with fallback paths (because they're not allowed to fail, or because they need to be async-signal-safe, etc.) and for lots it would make no sense whatsoever for them to do so in a reasonable implementation.
In any case, since unlike with stdio (where you can do low-level fd operations instead) there is no portable directory-access API that's not normally implemented with a userspace buffer object (DIR), you either have to accept that it uses malloc or go with a non-portable lower-level interface (on Linux, the SYS_getdents64 syscall).
One option on systems that let you define your own malloc would be doing that, and having it allocate from a fixed pool or direct mmap or similar, if there's a reason you need to avoid whatever malloc normally does on your system.
Related
I'm writing a C library which needs to often move around various sensitive data. I want to have benefits of realloc (extending allocated block instead copying when memory is available) while having some way to erase content of old block if copying is necessary.
Is there some lightweight implementation of malloc/realloc/free which could be used for mingw-gcc or some other trick to it, or I must overallocate and just allocate-and-copy without relying on realloc?
On Linux, mmap the block, mlock it, and then do mremap instead of using realloc.
Protecting against hidden copies isn't enough. You also need to make sure the memory never ever gets swapped to disk before you get a chance to zero it.
Can I use functions malloc and delete in embedded C? For example, I have one function, where was created pointer on structure with function malloc. This function return address in ram and I can use this . After exit from my function, where memory was allocated, this pointer will be deleted or this memory reserved for this, while not will be function delete terminated ?
Typedef struct {
Char varA;
Char varB
} myStruct ;
Void myfunc ( void)
{
myStruct * ptrStruct = ( myStruct *) malloc ( sizeof (myStruct)) ;
// Code here
//........
return ;
}
Generally, you shouldn't be using malloc in embedded systems, because doing so doesn't make any sense as explained here. In particular, it doesn't make any sense what-so-ever to use it on bare metal systems.
The only place where it makes sense to use dynamic memory allocation is large hosted, multi-process systems where multiple processes share the same RAM. If your definition of an embedded system is an Android smart phone or a portable PC, then yes it is fine to use malloc.
If you find yourself using it anywhere else, it almost certainly means that your program design is fundamentally flawed and also that you don't know how a heap works.
In addition, almost every embedded systems programming standard bans dynamic memory allocation.
There is nothing specific about embedded systems that prevent the use of dynamic memory.
However you may need to provide support for it in a number of ways, for example:
You need to ensure that the linker allocates sufficient space for the dynamic heap. Some linker scripts may already automatically allocate all remaining memory to the heap after stack and any other reserved allocations.
You may need to implement low level stubs to allow the library to access heap memory - for example in the newlib library, you need to implement sbrk_r() for malloc() etc. to work correctly.
In a multi-threaded system you may need to implement mutex stubs to ensure safe heap allocation. If the library does not provide such stubs, then malloc()/free() etc. will not be safe to use in such an environment, and you should write wrapper functions that assert the locks externally.
There are a number of reasons however why you might choose to avoid using dynamic memory (or at least standard library implemented dynamic memory) in an embedded system however:
Standard allocation schemes have non-deterministic timing unsuited to hard-real-time systems.
You need to handle the possibility of allocation failure gracefully for every allocation. Handling a potential non-deterministic run-time error safely is more complex than simply having the compiler tell you have insufficient memory at build time.
You need to guard against memory leaks; true of any system, but with no OS to manage memory exhaustion and kill a leaking process how will your system behave?
The standard library heap management may not be thread-safe without mutex stubs or wrapper functions.
Bugs that corrupt the heap are unlikely to affect execution immediately, often only causing an observable failure when a new heap operation is performed, resulting in non-deterministic behaviour at a time and location unrelated to the actual cause - making them very hard to diagnose. Again this is true of any system, but the debug facilities in a cross-hosted embedded system are often less sophisticated that on a self-hosted system.
Yes, you can use malloc in embedded C. Some embedded systems have its own encapsulated memory allocation APIs. malloc() is the C lib API.
The memory is allocated from heap, a dedicated memory range defined by system designer. If you did not free the allocated memory after your function exits, the allocated memory is reserved and other processes cannot use it. Typically, it is memory leak. If you free the allocated memory but you still use the pointer after that, it is a wild pointer and will cause unknown behaviour.
I heard "malloc is thread-safe because it provide a synchronization primitive so that simultaneous to malloc will not corrupt the heap".
But when I look at the source code of malloc function in visual studio crt, it turns out that the malloc function just pass the request to syscall HeapAlloc. So I think it is the opearting system itself provide some kind of synchronization to protect application from corrupted heap rather than malloc.
Then what about linux? Does malloc itself provide some kind of synchronization?
The only standard that speaks about this is C11 (since there was no notion of multithreading before), which says (7.22.3/2):
For purposes of determining the existence of a data race, memory allocation functions
behave as though they accessed only memory locations accessible through their
arguments and not other static duration storage. These functions may, however, visibly
modify the storage that they allocate or deallocate. A call to free or realloc that
deallocates a region p of memory synchronizes with any allocation call that allocates all
or part of the region p. This synchronization occurs after any access of p by the
deallocating function, and before any such access by the allocating function.
In short, "it's all fine".
However, specific implementations like Linux will surely have been providing their own, strong guarantees for a long time (since ptmalloc2 I think), and it's basically always been fine. [Update, thanks to #ArjunShankar: Posix does indeed require that malloc be thread-safe.]
(Note, though, that other implementations such as Google's tcmalloc may have better performance in multithreaded applications.)
(For C++, see C++11: 18.6.1.4.)
I want to know which C standard library functions use malloc and free under the hood. It looked to me as if printf would be using malloc, but when I tested a program with valgrind, I noticed that printf calls didn't allocate any memory using malloc. How come? How does it manage the memory then?
Usually, the only routines in the C99 standard that might use malloc() are the standard I/O functions (in <stdio.h> where the file structure and the buffer used by it is often allocated as if by malloc(). Some of the locale handling may use dynamic memory. All the other routines have no need for dynamic memory allocation in general.
Now, is any of that formally documented? No, I don't think it is. There is no blanket restriction 'the functions in the library shall not use malloc()'. (There are, however, restrictions on other functions - such as strtok() and srand() and rand(); they may not be used by the implementation, and the implementation may not use any of the other functions that may return a pointer to a static memory location.) However, one of the reasons why the extremely useful strdup() function is not in the standard C library is (reportedly) because it does memory allocation. It also isn't completely clear whether this was a factor in the routines such as asprintf() and vasprintf() in TR 24731-2 not making it into C1x, but it could have been a factor.
The standard doesn't place any requirements on the implementation, AFAIK.
I don't know exactly how printf is implemented, but of the top of my head, I can't think of a reason why it would need to dynamically allocate memory. You could always look at the source for your platform.
It depends on which libc you are using. There should be no restriction on the C spec and up to the implementation.
For instance, newlib's printf usually done with using memory on stack frame, but when it really needs to, it calls an internal function _malloc_r() directly.
I have not used valgrind, I'm not sure if it can detect use of _malloc_r().
Neither the C nor the POSIX standard force implementors to make use of malloc(), so there's no general answer to your question.
However, every sane standard library implementation that uses malloc() in one of its functions will set errno to ENOMEM if malloc() fails. Hence, you can derive from the documentation whether a library function uses malloc() or not. Point in case: on my system, mmap() may use malloc(), since mmap() may set errno to ENOMEM.
That having said, using valgrind is a poor way to find out whether a particular function calls malloc() or not. Consider the following piece of code:
void foo(int x)
{
if (!x) malloc(1);
}
If you call this function with an argument other than 0, valgrind won't notice that it may actually call malloc(). Think of valgrind as a virtual machine (since that's what it is): it doesn't look at your code, it only sees what the machine would actually execute.
printf doesn't need to form the entire output string in one shot, it can send it to output piece by piece, and when it encounters a format specifier, it can output that piece of data as it is formed, and continue on with the rest of the string.
At most it would need a locally defined array of characters (on the stack) large enough to hold the largest integer or floating point number it can handle, which isn't very large.
Is there a way that you can assign memory from heap without a call to malloc?
Can the following call be affective for it?
void* loc = (void*) &heap[end_of_heap];
No. The C language itself provides no such functionality. If you only care about Unix systems conforming to a deprecated feature of an old version of the Unix standard (SUSv2 or earlier, if I remember correctly), the brk and sbrk functions provide this functionality. But you really should not use it unless you're writing very low-level code that will never need to be portable.
There is no portable way besides malloc and friends, but if you're willing to get platform-specific sbrk (and brk) in old-fashioned Unix (not in current Posix), used to be the underlying syscalls. Now their manpage says
Avoid using brk() and sbrk(): the
malloc(3) memory allocation package is
the
portable and comfortable way of allocating memory.
and that advice is surely good (no real advantage in using the old-fashioned syscalls even in platforms that supply them). mmap of some /dev/ is a totally different way for some modern Unix versions, Windows has its own totally different "win32 API calls" for the purpose, and so on.
There is no way to get a pointer to new and valid heap memory other than using a heap allocating function. You cannot simply add a pointer into the heap at the end of an existing pointer and expect to reliably access it.
The Standard does not say anything about heap (search it, if you don't believe this). An implementation is not even required to have a heap (as we commonly know it).
However, the short answer to your question is, no in Standard C. Unless of course you use a platform specific API. Typically, OS APIs sometimes do give you some leeway as to accessing memory.
You cannot access heap reliably without malloc, but there are alternatives for memory allocation.
If you're trying to get finer control over memory allocations, you can use other memory managers like bget memory allocator. Here you grab a huge chunk of heap (the maximum memory requiredment anticipated + some overhead) using malloc and pass it to the bget using bpool. From there on, call bget instead of malloc to allocate memory and brel to free it. bget is reportedly better in avoiding memory fragmentation.