Related
Is there a way in C to find out the size of dynamically allocated memory?
For example, after
char* p = malloc (100);
Is there a way to find out the size of memory associated with p?
There is no standard way to find this information. However, some implementations provide functions like msize to do this. For example:
_msize on Windows
malloc_size on MacOS
malloc_usable_size on systems with glibc
Keep in mind though, that malloc will allocate a minimum of the size requested, so you should check if msize variant for your implementation actually returns the size of the object or the memory actually allocated on the heap.
comp.lang.c FAQ list · Question 7.27 -
Q. So can I query the malloc package to find out how big an
allocated block is?
A. Unfortunately, there is no standard or portable way. (Some
compilers provide nonstandard extensions.) If you need to know, you'll
have to keep track of it yourself. (See also question 7.28.)
The C mentality is to provide the programmer with tools to help him with his job, not to provide abstractions which change the nature of his job. C also tries to avoid making things easier/safer if this happens at the expense of the performance limit.
Certain things you might like to do with a region of memory only require the location of the start of the region. Such things include working with null-terminated strings, manipulating the first n bytes of the region (if the region is known to be at least this large), and so forth.
Basically, keeping track of the length of a region is extra work, and if C did it automatically, it would sometimes be doing it unnecessarily.
Many library functions (for instance fread()) require a pointer to the start of a region, and also the size of this region. If you need the size of a region, you must keep track of it.
Yes, malloc() implementations usually keep track of a region's size, but they may do this indirectly, or round it up to some value, or not keep it at all. Even if they support it, finding the size this way might be slow compared with keeping track of it yourself.
If you need a data structure that knows how big each region is, C can do that for you. Just use a struct that keeps track of how large the region is as well as a pointer to the region.
Here's the best way I've seen to create a tagged pointer to store the size with the address. All pointer functions would still work as expected:
Stolen from: https://stackoverflow.com/a/35326444/638848
You could also implement a wrapper for malloc and free to add tags
(like allocated size and other meta information) before the pointer
returned by malloc. This is in fact the method that a c++ compiler
tags objects with references to virtual classes. Here is one working
example:
#include <stdlib.h>
#include <stdio.h>
void * my_malloc(size_t s)
{
size_t * ret = malloc(sizeof(size_t) + s);
*ret = s;
return &ret[1];
}
void my_free(void * ptr)
{
free( (size_t*)ptr - 1);
}
size_t allocated_size(void * ptr)
{
return ((size_t*)ptr)[-1];
}
int main(int argc, const char ** argv) {
int * array = my_malloc(sizeof(int) * 3);
printf("%u\n", allocated_size(array));
my_free(array);
return 0;
}
The advantage of this method over a structure with size and pointer
struct pointer
{
size_t size;
void *p;
};
is that you only need to replace the malloc and free calls. All
other pointer operations require no refactoring.
No, the C runtime library does not provide such a function.
Some libraries may provide platform- or compiler-specific functions that can get this information, but generally the way to keep track of this information is in another integer variable.
Everyone telling you it's impossible is technically correct (the best kind of correct).
For engineering reasons, it is a bad idea to rely on the malloc subsystem to tell you the size of an allocated block accurately. To convince yourself of this, imagine that you were writing a large application, with several different memory allocators — maybe you use raw libc malloc in one part, but C++ operator new in another part, and then some specific Windows API in yet another part. So you've got all kinds of void* flying around. Writing a function that can work on any of these void*s impossible, unless you can somehow tell from the pointer's value which of your heaps it came from.
So you might want to wrap up each pointer in your program with some convention that indicates where the pointer came from (and where it needs to be returned to). For example, in C++ we call that std::unique_ptr<void> (for pointers that need to be operator delete'd) or std::unique_ptr<void, D> (for pointers that need to be returned via some other mechanism D). You could do the same kind of thing in C if you wanted to. And once you're wrapping up pointers in bigger safer objects anyway, it's just a small step to struct SizedPtr { void *ptr; size_t size; } and then you never need to worry about the size of an allocation again.
However.
There are also good reasons why you might legitimately want to know the actual underlying size of an allocation. For example, maybe you're writing a profiling tool for your app that will report the actual amount of memory used by each subsystem, not just the amount of memory that the programmer thought he was using. If each of your 10-byte allocations is secretly using 16 bytes under the hood, that's good to know! (Of course there will be other overhead as well, which you're not measuring this way. But there are yet other tools for that job.) Or maybe you're just investigating the behavior of realloc on your platform. Or maybe you'd like to "round up" the capacity of a growing allocation to avoid premature reallocations in the future. Example:
SizedPtr round_up(void *p) {
size_t sz = portable_ish_malloced_size(p);
void *q = realloc(p, sz); // for sanitizer-cleanliness
assert(q != NULL && portable_ish_malloced_size(q) == sz);
return (SizedPtr){q, sz};
}
bool reserve(VectorOfChar *v, size_t newcap) {
if (v->sizedptr.size >= newcap) return true;
char *newdata = realloc(v->sizedptr.ptr, newcap);
if (newdata == NULL) return false;
v->sizedptr = round_up(newdata);
return true;
}
To get the size of the allocation behind a non-null pointer which has been returned directly from libc malloc — not from a custom heap, and not pointing into the middle of an object — you can use the following OS-specific APIs, which I have bundled up into a "portable-ish" wrapper function for convenience. If you find a common system where this code doesn't work, please leave a comment and I'll try to fix it!
#if defined(__linux__)
// https://linux.die.net/man/3/malloc_usable_size
#include <malloc.h>
size_t portable_ish_malloced_size(const void *p) {
return malloc_usable_size((void*)p);
}
#elif defined(__APPLE__)
// https://www.unix.com/man-page/osx/3/malloc_size/
#include <malloc/malloc.h>
size_t portable_ish_malloced_size(const void *p) {
return malloc_size(p);
}
#elif defined(_WIN32)
// https://learn.microsoft.com/en-us/cpp/c-runtime-library/reference/msize
#include <malloc.h>
size_t portable_ish_malloced_size(const void *p) {
return _msize((void *)p);
}
#else
#error "oops, I don't know this system"
#endif
#include <stdio.h>
#include <stdlib.h> // for malloc itself
int main() {
void *p = malloc(42);
size_t true_length = portable_ish_malloced_size(p);
printf("%zu\n", true_length);
}
Tested on:
Visual Studio, Win64 — _msize
GCC/Clang, glibc, Linux — malloc_usable_size
Clang, libc, Mac OS X — malloc_size
Clang, jemalloc, Mac OS X — works in practice but I wouldn't trust it (silently mixes jemalloc's malloc and the native libc's malloc_size)
Should work fine with jemalloc on Linux
Should work fine with dlmalloc on Linux if compiled without USE_DL_PREFIX
Should work fine with tcmalloc everywhere
Like everyone else already said: No there isn't.
Also, I would always avoid all the vendor-specific functions here, because when you find that you really need to use them, that's generally a sign that you're doing it wrong. You should either store the size separately, or not have to know it at all. Using vendor functions is the quickest way to lose one of the main benefits of writing in C, portability.
I would expect this to be implementation dependent.
If you got the header data structure, you could cast it back on the pointer and get the size.
If you use malloc then you can not get the size.
In the other hand, if you use OS API to dynamically allocate memory, like Windows heap functions, then it's possible to do that.
Well now I know this is not answering your specific question, however thinking outside of the box as it were... It occurs to me you probably do not need to know. Ok, ok, no I don't mean your have a bad or un-orthodox implementation... I mean is that you probably (without looking at your code I am only guessing) you prbably only want to know if your data can fit in the allocated memory, if that is the case then this solution might be better. It should not offer too much overhead and will solve your "fitting" problem if that is indeed what you are handling:
if ( p != (tmp = realloc(p, required_size)) ) p = tmp;
or if you need to maintain the old contents:
if ( p != (tmp = realloc(p, required_size)) ) memcpy(tmp, p = tmp, required_size);
of course you could just use:
p = realloc(p, required_size);
and be done with it.
Quuxplusone wrote: "Writing a function that can work on any of these void*s impossible, unless you can somehow tell from the pointer's value which of your heaps it came from."
Determine size of dynamically allocated memory in C"
Actually in Windows _msize gives you the allocated memory size from the value of the pointer. If there is no allocated memory at the address an error is thrown.
int main()
{
char* ptr1 = NULL, * ptr2 = NULL;
size_t bsz;
ptr1 = (char*)malloc(10);
ptr2 = ptr1;
bsz = _msize(ptr2);
ptr1++;
//bsz = _msize(ptr1); /* error */
free(ptr2);
return 0;
}
Thanks for the #define collection. Here is the macro version.
#define MALLOC(bsz) malloc(bsz)
#define FREE(ptr) do { free(ptr); ptr = NULL; } while(0)
#ifdef __linux__
#include <malloc.h>
#define MSIZE(ptr) malloc_usable_size((void*)ptr)
#elif defined __APPLE__
#include <malloc/malloc.h>
#define MSIZE(ptr) malloc_size(const void *ptr)
#elif defined _WIN32
#include <malloc.h>
#define MSIZE(ptr) _msize(ptr)
#else
#error "unknown system"
#endif
Note: using _msize only works for memory allocated with calloc, malloc, etc. As stated on the Microsoft Documentation
The _msize function returns the size, in bytes, of the memory block
allocated by a call to calloc, malloc, or realloc.
And will throw an exception otherwise.
https://learn.microsoft.com/en-us/cpp/c-runtime-library/reference/msize?view=vs-2019
This code will probably work on most Windows installations:
template <class T>
int get_allocated_bytes(T* ptr)
{
return *((int*)ptr-4);
}
template <class T>
int get_allocated_elements(T* ptr)
{
return get_allocated_bytes(ptr)/sizeof(T);
}
I was struggling recently with visualizing the memory that was available to write to (i.e using strcat or strcpy type functions immediately after malloc).
This is not meant to be a very technical answer, but it could help you while debugging, as much as it helped me.
You can use the size you mallocd in a memset, set an arbitrary value for the second parameter (so you can recognize it) and use the pointer that you obtained from malloc.
Like so:
char* my_string = (char*) malloc(custom_size * sizeof(char));
if(my_string) { memset(my_string, 1, custom_size); }
You can then visualize in the debugger how your allocated memory looks like:
This may work, a small update in your code:
void* inc = (void*) (++p)
size=p-inc;
But this will result 1, that is, memory associated with p if it is char*. If it is int* then result will be 4.
There is no way to find out total allocation.
I have a variable where the size is determined in run-time.
Is it generally better to realloc it every time a new element is added like:
array_type * someArray;
int counter = 0;
//before a new element is added:
counter ++;
someArray = realloc(someArray, counter * sizeof(array_type));
Or allocate more memory than probably needed using malloc once:
array_type * someArray = malloc(ENOUG_MEMORY * sizeof(array_type));
What is best in terms of efficiency(speed), readability and memory-management? And why?
realloc could occasionally be useful in the past, with certain allocation patterns in single-threaded code. Most modern memory managers are optimized for multi-threaded programs and to minimize fragmentation, so, when growing an allocation, a realloc will almost certainly just allocate a new block, copy the existing data, and then free the old block. So there's no real advantage to trying to use realloc.
Increasing the size one element at a time can create an O(n^2) situation. In the worst case, the existing data has to be copied each time. It would be better to increase the size in chunks (which is still O(n^2) but with a smaller constant factor) or to grow the allocation geometrically (which gives an O(n) amortized cost).
Furthermore, it's difficult to use realloc correctly.
someArray = realloc(someArray, counter * sizeof(array_type));
If the realloc fails, someArray is set to NULL. If that was your only copy of the pointer to the previously allocated memory, you've just lost it.
You won't be able to access the data you had already placed, and you can't free the original allocation, so you'll have a memory leak.
What is best in terms of efficiency(speed), readability and memory-management? And why?
There is no general best. Specific best depends on your specific application and use case and environment. You can throw wars over perfect realloc ratio and decide if you need realloc at all.
Remember about rules of optimization. You do not optimize. Then, you do not optimize, without measuring first. The best in any terms can be measured for your specific setup, your specific environment, your specific application that uses specific operating system and *alloc implementation.
what is the best practice?
Allocating a constant amount of memory (if it's small enough) is static. It's just an array. Refactor you application to just:
array_type someArray[ENOUGH_MEMORY];
If you do not want to over allocate (or ENOUGH_MEMORY is big enough), then use realloc to add one element, as presented.
If you want, "optimize" by not calling realloc that often and over allocating - it seems that ratio 1.5 is the most preferred from the linked thread above. Still, it's highly application specific - I would over allocate on Linux, I would not when working on STM32 or other bare metal.
I would use realloc with caution.
Calling realloc in general leads to:
allocating completely new block
copying all data from old to new location
releasing (freeing) the initial block.
All combined could be questionable from performance perspective, depending on the app, volume of data, response requirements.
In addition, in case of realloc failure, the return value is NULL which means that allocation to new block is not straightforward (indirection is required). E.g.
int *p = malloc(100 * sizeof *p);
if (NULL == p)
{
perror("malloc() failed");
return EXIT_FAILURE;
}
do_something_with_p(p);
/* Reallocate array to a new size
* Using temporary pointer in case realloc() fails. */
{
int *temp = realloc(p, 100000 * sizeof *temp);
if (NULL == temp)
{
perror("realloc() failed");
free(p);
return EXIT_FAILURE;
}
p = temp;
}
malloc vs realloc - what is the best practice?
Helper functions
When writing robust code, I avoid using library *alloc() functions directly. Instead I form helper functions to handle various use cases and take care of edge cases, parameter validation, etc.
Within these helper functions, I use malloc(), realloc(), calloc() as building blocks, perhaps steered by implementation macros, to form good code per the use case.
This pushes the "what is best" to a narrower set of conditions where it can be better assessed - per function. In the growing by 2x case, realloc() is fine.
Example:
// Optimize for a growing allocation
// Return new pointer.
// Allocate per 2x *nmemb * size.
// Update *nmemb_new as needed.
// A return of NULL implies failure, old not deallocated.
void *my_alloc_grow(void *ptr, size_t *nmemb, size_t size) {
if (nmemb == NULL) {
return NULL;
}
size_t nmemb_old = *nmemb;
if (size == 0) { // Consider array elements of size 0 as error
return NULL;
}
if (nmemb_old > SIZE_MAX/2/size)) {
return NULL;
}
size_t nmemb_new = nmemb_old ? (nmemb_old * 2) : 1;
unsigned char *ptr_new = realloc(ptr, nmemb_new * size);
if (ptr_new == NULL) {
return NULL;
}
// Maybe zero fill new memory portion.
memset(ptr_new + nmemb_old * size, 0, (nmemb_new - nmemb_old) * size);
*nmemb = nmemb_new;
return ptr_new;
}
Other use cases.
/ General new memory
void *my_alloc(size_t *nmemb, size_t size); // General new memory
void *my_calloc(size_t *nmemb, size_t size); // General new memory with zeroing
// General reallocation, maybe more or less.
// Act like free() on nmemb_new == 0.
void *my_alloc_resize(void *ptr, size_t *nmemb, size_t nmemb_new, size_t size);
Is there a way in C to find out the size of dynamically allocated memory?
For example, after
char* p = malloc (100);
Is there a way to find out the size of memory associated with p?
There is no standard way to find this information. However, some implementations provide functions like msize to do this. For example:
_msize on Windows
malloc_size on MacOS
malloc_usable_size on systems with glibc
Keep in mind though, that malloc will allocate a minimum of the size requested, so you should check if msize variant for your implementation actually returns the size of the object or the memory actually allocated on the heap.
comp.lang.c FAQ list · Question 7.27 -
Q. So can I query the malloc package to find out how big an
allocated block is?
A. Unfortunately, there is no standard or portable way. (Some
compilers provide nonstandard extensions.) If you need to know, you'll
have to keep track of it yourself. (See also question 7.28.)
The C mentality is to provide the programmer with tools to help him with his job, not to provide abstractions which change the nature of his job. C also tries to avoid making things easier/safer if this happens at the expense of the performance limit.
Certain things you might like to do with a region of memory only require the location of the start of the region. Such things include working with null-terminated strings, manipulating the first n bytes of the region (if the region is known to be at least this large), and so forth.
Basically, keeping track of the length of a region is extra work, and if C did it automatically, it would sometimes be doing it unnecessarily.
Many library functions (for instance fread()) require a pointer to the start of a region, and also the size of this region. If you need the size of a region, you must keep track of it.
Yes, malloc() implementations usually keep track of a region's size, but they may do this indirectly, or round it up to some value, or not keep it at all. Even if they support it, finding the size this way might be slow compared with keeping track of it yourself.
If you need a data structure that knows how big each region is, C can do that for you. Just use a struct that keeps track of how large the region is as well as a pointer to the region.
Here's the best way I've seen to create a tagged pointer to store the size with the address. All pointer functions would still work as expected:
Stolen from: https://stackoverflow.com/a/35326444/638848
You could also implement a wrapper for malloc and free to add tags
(like allocated size and other meta information) before the pointer
returned by malloc. This is in fact the method that a c++ compiler
tags objects with references to virtual classes. Here is one working
example:
#include <stdlib.h>
#include <stdio.h>
void * my_malloc(size_t s)
{
size_t * ret = malloc(sizeof(size_t) + s);
*ret = s;
return &ret[1];
}
void my_free(void * ptr)
{
free( (size_t*)ptr - 1);
}
size_t allocated_size(void * ptr)
{
return ((size_t*)ptr)[-1];
}
int main(int argc, const char ** argv) {
int * array = my_malloc(sizeof(int) * 3);
printf("%u\n", allocated_size(array));
my_free(array);
return 0;
}
The advantage of this method over a structure with size and pointer
struct pointer
{
size_t size;
void *p;
};
is that you only need to replace the malloc and free calls. All
other pointer operations require no refactoring.
No, the C runtime library does not provide such a function.
Some libraries may provide platform- or compiler-specific functions that can get this information, but generally the way to keep track of this information is in another integer variable.
Everyone telling you it's impossible is technically correct (the best kind of correct).
For engineering reasons, it is a bad idea to rely on the malloc subsystem to tell you the size of an allocated block accurately. To convince yourself of this, imagine that you were writing a large application, with several different memory allocators — maybe you use raw libc malloc in one part, but C++ operator new in another part, and then some specific Windows API in yet another part. So you've got all kinds of void* flying around. Writing a function that can work on any of these void*s impossible, unless you can somehow tell from the pointer's value which of your heaps it came from.
So you might want to wrap up each pointer in your program with some convention that indicates where the pointer came from (and where it needs to be returned to). For example, in C++ we call that std::unique_ptr<void> (for pointers that need to be operator delete'd) or std::unique_ptr<void, D> (for pointers that need to be returned via some other mechanism D). You could do the same kind of thing in C if you wanted to. And once you're wrapping up pointers in bigger safer objects anyway, it's just a small step to struct SizedPtr { void *ptr; size_t size; } and then you never need to worry about the size of an allocation again.
However.
There are also good reasons why you might legitimately want to know the actual underlying size of an allocation. For example, maybe you're writing a profiling tool for your app that will report the actual amount of memory used by each subsystem, not just the amount of memory that the programmer thought he was using. If each of your 10-byte allocations is secretly using 16 bytes under the hood, that's good to know! (Of course there will be other overhead as well, which you're not measuring this way. But there are yet other tools for that job.) Or maybe you're just investigating the behavior of realloc on your platform. Or maybe you'd like to "round up" the capacity of a growing allocation to avoid premature reallocations in the future. Example:
SizedPtr round_up(void *p) {
size_t sz = portable_ish_malloced_size(p);
void *q = realloc(p, sz); // for sanitizer-cleanliness
assert(q != NULL && portable_ish_malloced_size(q) == sz);
return (SizedPtr){q, sz};
}
bool reserve(VectorOfChar *v, size_t newcap) {
if (v->sizedptr.size >= newcap) return true;
char *newdata = realloc(v->sizedptr.ptr, newcap);
if (newdata == NULL) return false;
v->sizedptr = round_up(newdata);
return true;
}
To get the size of the allocation behind a non-null pointer which has been returned directly from libc malloc — not from a custom heap, and not pointing into the middle of an object — you can use the following OS-specific APIs, which I have bundled up into a "portable-ish" wrapper function for convenience. If you find a common system where this code doesn't work, please leave a comment and I'll try to fix it!
#if defined(__linux__)
// https://linux.die.net/man/3/malloc_usable_size
#include <malloc.h>
size_t portable_ish_malloced_size(const void *p) {
return malloc_usable_size((void*)p);
}
#elif defined(__APPLE__)
// https://www.unix.com/man-page/osx/3/malloc_size/
#include <malloc/malloc.h>
size_t portable_ish_malloced_size(const void *p) {
return malloc_size(p);
}
#elif defined(_WIN32)
// https://learn.microsoft.com/en-us/cpp/c-runtime-library/reference/msize
#include <malloc.h>
size_t portable_ish_malloced_size(const void *p) {
return _msize((void *)p);
}
#else
#error "oops, I don't know this system"
#endif
#include <stdio.h>
#include <stdlib.h> // for malloc itself
int main() {
void *p = malloc(42);
size_t true_length = portable_ish_malloced_size(p);
printf("%zu\n", true_length);
}
Tested on:
Visual Studio, Win64 — _msize
GCC/Clang, glibc, Linux — malloc_usable_size
Clang, libc, Mac OS X — malloc_size
Clang, jemalloc, Mac OS X — works in practice but I wouldn't trust it (silently mixes jemalloc's malloc and the native libc's malloc_size)
Should work fine with jemalloc on Linux
Should work fine with dlmalloc on Linux if compiled without USE_DL_PREFIX
Should work fine with tcmalloc everywhere
Like everyone else already said: No there isn't.
Also, I would always avoid all the vendor-specific functions here, because when you find that you really need to use them, that's generally a sign that you're doing it wrong. You should either store the size separately, or not have to know it at all. Using vendor functions is the quickest way to lose one of the main benefits of writing in C, portability.
I would expect this to be implementation dependent.
If you got the header data structure, you could cast it back on the pointer and get the size.
If you use malloc then you can not get the size.
In the other hand, if you use OS API to dynamically allocate memory, like Windows heap functions, then it's possible to do that.
Well now I know this is not answering your specific question, however thinking outside of the box as it were... It occurs to me you probably do not need to know. Ok, ok, no I don't mean your have a bad or un-orthodox implementation... I mean is that you probably (without looking at your code I am only guessing) you prbably only want to know if your data can fit in the allocated memory, if that is the case then this solution might be better. It should not offer too much overhead and will solve your "fitting" problem if that is indeed what you are handling:
if ( p != (tmp = realloc(p, required_size)) ) p = tmp;
or if you need to maintain the old contents:
if ( p != (tmp = realloc(p, required_size)) ) memcpy(tmp, p = tmp, required_size);
of course you could just use:
p = realloc(p, required_size);
and be done with it.
Quuxplusone wrote: "Writing a function that can work on any of these void*s impossible, unless you can somehow tell from the pointer's value which of your heaps it came from."
Determine size of dynamically allocated memory in C"
Actually in Windows _msize gives you the allocated memory size from the value of the pointer. If there is no allocated memory at the address an error is thrown.
int main()
{
char* ptr1 = NULL, * ptr2 = NULL;
size_t bsz;
ptr1 = (char*)malloc(10);
ptr2 = ptr1;
bsz = _msize(ptr2);
ptr1++;
//bsz = _msize(ptr1); /* error */
free(ptr2);
return 0;
}
Thanks for the #define collection. Here is the macro version.
#define MALLOC(bsz) malloc(bsz)
#define FREE(ptr) do { free(ptr); ptr = NULL; } while(0)
#ifdef __linux__
#include <malloc.h>
#define MSIZE(ptr) malloc_usable_size((void*)ptr)
#elif defined __APPLE__
#include <malloc/malloc.h>
#define MSIZE(ptr) malloc_size(const void *ptr)
#elif defined _WIN32
#include <malloc.h>
#define MSIZE(ptr) _msize(ptr)
#else
#error "unknown system"
#endif
Note: using _msize only works for memory allocated with calloc, malloc, etc. As stated on the Microsoft Documentation
The _msize function returns the size, in bytes, of the memory block
allocated by a call to calloc, malloc, or realloc.
And will throw an exception otherwise.
https://learn.microsoft.com/en-us/cpp/c-runtime-library/reference/msize?view=vs-2019
This code will probably work on most Windows installations:
template <class T>
int get_allocated_bytes(T* ptr)
{
return *((int*)ptr-4);
}
template <class T>
int get_allocated_elements(T* ptr)
{
return get_allocated_bytes(ptr)/sizeof(T);
}
I was struggling recently with visualizing the memory that was available to write to (i.e using strcat or strcpy type functions immediately after malloc).
This is not meant to be a very technical answer, but it could help you while debugging, as much as it helped me.
You can use the size you mallocd in a memset, set an arbitrary value for the second parameter (so you can recognize it) and use the pointer that you obtained from malloc.
Like so:
char* my_string = (char*) malloc(custom_size * sizeof(char));
if(my_string) { memset(my_string, 1, custom_size); }
You can then visualize in the debugger how your allocated memory looks like:
This may work, a small update in your code:
void* inc = (void*) (++p)
size=p-inc;
But this will result 1, that is, memory associated with p if it is char*. If it is int* then result will be 4.
There is no way to find out total allocation.
I am sure someone must have implemented something like this already!
What I am looking for is the ability to "checkpoint" the heap state and then clear all allocations that have happened since the last checkpoint.
Basically what I am looking for is a natural corollary of the _CrtMemCheck Apis.
Something like(preferably cross-platform)
//we save the heap state here in s1
_CrtMemCheckpoint( &s1 );
//allocs and frees
//Get rid of all allocs since checkpoint s1 that have not been freed!
_CrtMemClearAllObjectsSince(&s1);
There is no standard way to use mark/release memory allocation in C. If you know for a fact that all malloc/free calls will be used in a LIFO fashion, you may be able to link in your ownmalloc/free` functions using something like the following:
#define MY_HEAP_SIZE 12345678
unsigned char my_mem[MY_HEAP_SIZE];
unsigned char *my_alloc_ptr = my_mem;
void *malloc(size_t size)
{
void *ret = my_alloc_ptr;
if (size <= MY_HEAP_SIZE && ((my_alloc_ptr - my_mem)+size) <= MY_HEAP_SIZE)
{
my_alloc_ptr += size;
return (void*)ret;
}
else
return (void*)0;
}
void free(void *ptr)
{
if (ptr)
my_alloc_ptr = ptr;
}
This approach requires zero bytes of overhead per allocation block, but calling free() on any block will also free all blocks that were allocated later. An alternative approach which could be used if the external code doesn't use malloc/free in LIFO order, but it would be okay if blocks don't freed until your code does so, would be to make free() do nothing, but have some other function which behaves like free above. More sophisticated variations are possible as well, but in cases where the first approach will suffice, there's no beating its efficiency. Very nice for embedded systems (though I'd usually call it something other than malloc).
You can modify malloc()/free() using hooks to remember allocated memory (for example, suppose that your record the new pointer in an array of pointers). Then your can have two functions:
int get_checkpoint(), that returns the next free array index,
void free_until(int checkpoint), that frees memory from the current stored pointer in the array backwards, until checkpoint is reached.
This way, you can do:
int cpoint = get_checkpoint();
LibraryDoSomething();
free_until(cpoint);
Of course, this technique is still dangerous; calling a C library function can have side effects that you can easily affect. The best advice is still that of Amardeep.
Another possible and interesting solution could be the use of LD_PRELOAD. As the man page for LD_PRELOAD states "This can be used to selectively override functions in other shared libraries."
Thus, you can have your own implementations of malloc and free wherein you can implement the required checks and then call the default malloc or free.
You can check the details here: http://somethingswhichidintknow.blogspot.com/2009/10/dll-injection.html
Is there a way in C to find out the size of dynamically allocated memory?
For example, after
char* p = malloc (100);
Is there a way to find out the size of memory associated with p?
There is no standard way to find this information. However, some implementations provide functions like msize to do this. For example:
_msize on Windows
malloc_size on MacOS
malloc_usable_size on systems with glibc
Keep in mind though, that malloc will allocate a minimum of the size requested, so you should check if msize variant for your implementation actually returns the size of the object or the memory actually allocated on the heap.
comp.lang.c FAQ list · Question 7.27 -
Q. So can I query the malloc package to find out how big an
allocated block is?
A. Unfortunately, there is no standard or portable way. (Some
compilers provide nonstandard extensions.) If you need to know, you'll
have to keep track of it yourself. (See also question 7.28.)
The C mentality is to provide the programmer with tools to help him with his job, not to provide abstractions which change the nature of his job. C also tries to avoid making things easier/safer if this happens at the expense of the performance limit.
Certain things you might like to do with a region of memory only require the location of the start of the region. Such things include working with null-terminated strings, manipulating the first n bytes of the region (if the region is known to be at least this large), and so forth.
Basically, keeping track of the length of a region is extra work, and if C did it automatically, it would sometimes be doing it unnecessarily.
Many library functions (for instance fread()) require a pointer to the start of a region, and also the size of this region. If you need the size of a region, you must keep track of it.
Yes, malloc() implementations usually keep track of a region's size, but they may do this indirectly, or round it up to some value, or not keep it at all. Even if they support it, finding the size this way might be slow compared with keeping track of it yourself.
If you need a data structure that knows how big each region is, C can do that for you. Just use a struct that keeps track of how large the region is as well as a pointer to the region.
Here's the best way I've seen to create a tagged pointer to store the size with the address. All pointer functions would still work as expected:
Stolen from: https://stackoverflow.com/a/35326444/638848
You could also implement a wrapper for malloc and free to add tags
(like allocated size and other meta information) before the pointer
returned by malloc. This is in fact the method that a c++ compiler
tags objects with references to virtual classes. Here is one working
example:
#include <stdlib.h>
#include <stdio.h>
void * my_malloc(size_t s)
{
size_t * ret = malloc(sizeof(size_t) + s);
*ret = s;
return &ret[1];
}
void my_free(void * ptr)
{
free( (size_t*)ptr - 1);
}
size_t allocated_size(void * ptr)
{
return ((size_t*)ptr)[-1];
}
int main(int argc, const char ** argv) {
int * array = my_malloc(sizeof(int) * 3);
printf("%u\n", allocated_size(array));
my_free(array);
return 0;
}
The advantage of this method over a structure with size and pointer
struct pointer
{
size_t size;
void *p;
};
is that you only need to replace the malloc and free calls. All
other pointer operations require no refactoring.
No, the C runtime library does not provide such a function.
Some libraries may provide platform- or compiler-specific functions that can get this information, but generally the way to keep track of this information is in another integer variable.
Everyone telling you it's impossible is technically correct (the best kind of correct).
For engineering reasons, it is a bad idea to rely on the malloc subsystem to tell you the size of an allocated block accurately. To convince yourself of this, imagine that you were writing a large application, with several different memory allocators — maybe you use raw libc malloc in one part, but C++ operator new in another part, and then some specific Windows API in yet another part. So you've got all kinds of void* flying around. Writing a function that can work on any of these void*s impossible, unless you can somehow tell from the pointer's value which of your heaps it came from.
So you might want to wrap up each pointer in your program with some convention that indicates where the pointer came from (and where it needs to be returned to). For example, in C++ we call that std::unique_ptr<void> (for pointers that need to be operator delete'd) or std::unique_ptr<void, D> (for pointers that need to be returned via some other mechanism D). You could do the same kind of thing in C if you wanted to. And once you're wrapping up pointers in bigger safer objects anyway, it's just a small step to struct SizedPtr { void *ptr; size_t size; } and then you never need to worry about the size of an allocation again.
However.
There are also good reasons why you might legitimately want to know the actual underlying size of an allocation. For example, maybe you're writing a profiling tool for your app that will report the actual amount of memory used by each subsystem, not just the amount of memory that the programmer thought he was using. If each of your 10-byte allocations is secretly using 16 bytes under the hood, that's good to know! (Of course there will be other overhead as well, which you're not measuring this way. But there are yet other tools for that job.) Or maybe you're just investigating the behavior of realloc on your platform. Or maybe you'd like to "round up" the capacity of a growing allocation to avoid premature reallocations in the future. Example:
SizedPtr round_up(void *p) {
size_t sz = portable_ish_malloced_size(p);
void *q = realloc(p, sz); // for sanitizer-cleanliness
assert(q != NULL && portable_ish_malloced_size(q) == sz);
return (SizedPtr){q, sz};
}
bool reserve(VectorOfChar *v, size_t newcap) {
if (v->sizedptr.size >= newcap) return true;
char *newdata = realloc(v->sizedptr.ptr, newcap);
if (newdata == NULL) return false;
v->sizedptr = round_up(newdata);
return true;
}
To get the size of the allocation behind a non-null pointer which has been returned directly from libc malloc — not from a custom heap, and not pointing into the middle of an object — you can use the following OS-specific APIs, which I have bundled up into a "portable-ish" wrapper function for convenience. If you find a common system where this code doesn't work, please leave a comment and I'll try to fix it!
#if defined(__linux__)
// https://linux.die.net/man/3/malloc_usable_size
#include <malloc.h>
size_t portable_ish_malloced_size(const void *p) {
return malloc_usable_size((void*)p);
}
#elif defined(__APPLE__)
// https://www.unix.com/man-page/osx/3/malloc_size/
#include <malloc/malloc.h>
size_t portable_ish_malloced_size(const void *p) {
return malloc_size(p);
}
#elif defined(_WIN32)
// https://learn.microsoft.com/en-us/cpp/c-runtime-library/reference/msize
#include <malloc.h>
size_t portable_ish_malloced_size(const void *p) {
return _msize((void *)p);
}
#else
#error "oops, I don't know this system"
#endif
#include <stdio.h>
#include <stdlib.h> // for malloc itself
int main() {
void *p = malloc(42);
size_t true_length = portable_ish_malloced_size(p);
printf("%zu\n", true_length);
}
Tested on:
Visual Studio, Win64 — _msize
GCC/Clang, glibc, Linux — malloc_usable_size
Clang, libc, Mac OS X — malloc_size
Clang, jemalloc, Mac OS X — works in practice but I wouldn't trust it (silently mixes jemalloc's malloc and the native libc's malloc_size)
Should work fine with jemalloc on Linux
Should work fine with dlmalloc on Linux if compiled without USE_DL_PREFIX
Should work fine with tcmalloc everywhere
Like everyone else already said: No there isn't.
Also, I would always avoid all the vendor-specific functions here, because when you find that you really need to use them, that's generally a sign that you're doing it wrong. You should either store the size separately, or not have to know it at all. Using vendor functions is the quickest way to lose one of the main benefits of writing in C, portability.
I would expect this to be implementation dependent.
If you got the header data structure, you could cast it back on the pointer and get the size.
If you use malloc then you can not get the size.
In the other hand, if you use OS API to dynamically allocate memory, like Windows heap functions, then it's possible to do that.
Well now I know this is not answering your specific question, however thinking outside of the box as it were... It occurs to me you probably do not need to know. Ok, ok, no I don't mean your have a bad or un-orthodox implementation... I mean is that you probably (without looking at your code I am only guessing) you prbably only want to know if your data can fit in the allocated memory, if that is the case then this solution might be better. It should not offer too much overhead and will solve your "fitting" problem if that is indeed what you are handling:
if ( p != (tmp = realloc(p, required_size)) ) p = tmp;
or if you need to maintain the old contents:
if ( p != (tmp = realloc(p, required_size)) ) memcpy(tmp, p = tmp, required_size);
of course you could just use:
p = realloc(p, required_size);
and be done with it.
Quuxplusone wrote: "Writing a function that can work on any of these void*s impossible, unless you can somehow tell from the pointer's value which of your heaps it came from."
Determine size of dynamically allocated memory in C"
Actually in Windows _msize gives you the allocated memory size from the value of the pointer. If there is no allocated memory at the address an error is thrown.
int main()
{
char* ptr1 = NULL, * ptr2 = NULL;
size_t bsz;
ptr1 = (char*)malloc(10);
ptr2 = ptr1;
bsz = _msize(ptr2);
ptr1++;
//bsz = _msize(ptr1); /* error */
free(ptr2);
return 0;
}
Thanks for the #define collection. Here is the macro version.
#define MALLOC(bsz) malloc(bsz)
#define FREE(ptr) do { free(ptr); ptr = NULL; } while(0)
#ifdef __linux__
#include <malloc.h>
#define MSIZE(ptr) malloc_usable_size((void*)ptr)
#elif defined __APPLE__
#include <malloc/malloc.h>
#define MSIZE(ptr) malloc_size(const void *ptr)
#elif defined _WIN32
#include <malloc.h>
#define MSIZE(ptr) _msize(ptr)
#else
#error "unknown system"
#endif
Note: using _msize only works for memory allocated with calloc, malloc, etc. As stated on the Microsoft Documentation
The _msize function returns the size, in bytes, of the memory block
allocated by a call to calloc, malloc, or realloc.
And will throw an exception otherwise.
https://learn.microsoft.com/en-us/cpp/c-runtime-library/reference/msize?view=vs-2019
This code will probably work on most Windows installations:
template <class T>
int get_allocated_bytes(T* ptr)
{
return *((int*)ptr-4);
}
template <class T>
int get_allocated_elements(T* ptr)
{
return get_allocated_bytes(ptr)/sizeof(T);
}
I was struggling recently with visualizing the memory that was available to write to (i.e using strcat or strcpy type functions immediately after malloc).
This is not meant to be a very technical answer, but it could help you while debugging, as much as it helped me.
You can use the size you mallocd in a memset, set an arbitrary value for the second parameter (so you can recognize it) and use the pointer that you obtained from malloc.
Like so:
char* my_string = (char*) malloc(custom_size * sizeof(char));
if(my_string) { memset(my_string, 1, custom_size); }
You can then visualize in the debugger how your allocated memory looks like:
This may work, a small update in your code:
void* inc = (void*) (++p)
size=p-inc;
But this will result 1, that is, memory associated with p if it is char*. If it is int* then result will be 4.
There is no way to find out total allocation.