After a quick scan of related questions on SO, I have deduced that there's no function that would check the amount of memory that malloc has allocated to a pointer. I'm trying to replicate some of std::string basic functionality (mainly dynamic size) using simple char*'s in C and don't want to call realloc all the time. I guess I'll need to keep track of how much memory has been allocated. In order to do that, I'm considering creating a typedef that will contain the string itself and an integer with the amount of memory currently allocated, something like this:
typedef struct {
char * str;
int mem;
} my_string_t;
Is that an optimal solution, or perhaps you can suggest something that will bear better results? Thanks in advance for your help.
You will want to allocate the space for both the length and the string in the same block of memory. This may be what you intended with your struct, but you have reserved space for only a pointer to the string.
There must be space allocated to contain the characters of the string.
For example:
typedef struct
{
int num_chars;
char string[];
} my_string_t;
my_string_t * alloc_my_string(char *src)
{
my_string_t * p = NULL;
int N_chars = strlen(src) + 1;
p = malloc( N_chars + sizeof(my_string_t));
if (p)
{
p->num_chars = N_chars;
strcpy(p->string, src);
}
return p;
}
In my example, to access the pointer to your string, you address the string member of the my_string_t:
my_string_t * p = alloc_my_string("hello free store.");
printf("String of %d bytes is '%s'\n", p->num_chars, p->string);
Be careful to realize that you are obtaining the pointer for the string as a consequence of allocating space to store the characters. The resource you are allocating is the storage for the characters, the pointer obtained is a reference to the allocated storage.
In my example, the memory allocated is laid out sequentially as follows:
+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+
| 00 | 00 | 00 | 11 | 'h'| 'e'| 'l'| 'l'| 'o'| 20 | 'f'| 'r'| 'e'| 'e'| 20 | 's'| 't'| 'o'| 'r'| 'e'| '.'| 00 |
+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+
^^ ^
|| |
p| |
p->num_chars p->string
Notice that the value of p->string is not stored in the allocated memory, it is four bytes from the beginning of the allocated memory, immediately subsequent to the (presumed 32-bit, four-byte) integer.
Your compiler may require that you declare the flexible C array as:
typedef struct
{
int num_chars;
char string[0];
} my_string_t;
but the version lacking the zero is supposedly C99-compliant.
You can accomplish the equivalent thing with no array member as follows:
typedef struct
{
int num_chars;
} mystr2;
char * str_of_mystr2(mystr2 * ms)
{
return (char *)(ms + 1);
}
mystr2 * alloc_mystr2(char *src)
{
mystr2* p = NULL;
size_t N_chars = strlen(src) + 1;
if (N_chars num_chars = (int)N_chars;
strcpy(str_of_mystr2(p), src);
}
return p;
}
printf("String of %d bytes is '%s'\n", p->num_chars, str_of_mystr2 (p));
In this second example, the value equivalent to p->string is calculated by str_of_mystr2(). It will have approximately the same value as the first example, depending on how the end of structs are packed by your compiler settings.
While some would suggest tracking the length in a size_t I would look up some old Dr. Dobb's article on why I disagree. Supporting values greater than INT_MAX is of doubtful value to your program's correctness. By using an int, you can write assert(p->num_chars >= 0); and have that test something. With an unsigned, you would write the equivalent test something like assert(p->num_chars < UINT_MAX / 2); As long as you write code which contains checks on run-time data, using a signed type can be useful.
On the other hand, if you are writing a library which handles strings in excess of UINT_MAX / 2 characters, I salute you.
This is the obvious solution. And while you are at it, you might want to have a struct member that maintains the amount of allocated memory actually in use. This will avoid having to call strlen() all the time, and would enable you to support non null-terminated strings, as the C++ std::string class does.
That is how it was done in the Pleistocene, and that's how you should do it today. You are dead on the money that malloc does not offer any portable, supported, mechanism to query the size of an allocated block.
A more common way is to wrap malloc (and realloc) and keep a list of sizes and pointers
That way you don't need to change any string functions.
write wrapper functions. If you are using malloc then you should do that anyway.
For an example look in "writing solid code"
I think you could use malloc_usable_size.
Related
I am rewriting this post because I managed to figure out the problem. The problem with my extremely broken output was due to an improper dynamic memory allocation.
Basically I needed to allocate memory for an array of pointers that pointed to structs, but the array itself was nested inside of another struct and the nesting confused me slightly and I ended up over complicating it.
So I had a struct named Catalog, that my array was in and that array pointed to another struct named Books.
When I originally allocated memory for it I was only allocated memory for an array, not an array of pointers:
catalog->arrayP = malloc(INITIAL_CAPACITY * sizeof( Books );
// But I should have done this:
catalog->arrayP = (Books *) malloc(INITIAL_CAPACITY * sizeof( Books );
// That first (Books *) was extremely important
The second issue I was having was that when I was trying to update the memory to allow for more books I was actually decreasing it:
catalog->arrayP = realloc(catalog->arrayP, 2 * sizeof( catalog->arrayP));
// I did this thinking it would just increase the memory to twice that of what it currently was, but it didn't
cataloc->capacity = catalog->capacity * 2;
catalog->arrayP = realloc(catalog->arrayP, catalog->capacity * sizeof( catalog->arrayP));
So whenever I needed to grow my array of pointers I ended up just allocating enough memory for 2 books rather than double the current.
Frankenstein; Or, The Modern Prometh..Shelley, Mary Woll.
Your printing results kind of give away the answer. You forgot the null terminator on your strings and printf invaded the next field until reached the null terminator.
In the following fields it couldn't find and invaded even more stuff.
Here's a minimal example
#include <stdio.h>
#include <string.h>
struct test{
char test[37]; // space for 36 chars + null
char test2[16]; // space for 15 chars + null
};
int main(void) {
struct test Test;
strcpy(Test.test, "randomrandomrandomrandomrandomrandom"); // Copy the 37 bytes
strcpy(Test.test2, "notnotnotnotnot"); // Copy the 16 bytes
//Replace null terminator with trash for demonstration purposes
Test.test[36] = '1'; // replaces 37th byte containing the terminator (\0) with trash
printf("%38s", Test.test); // should print randomrandomrandomrandomrandomrandom1notnotnotnotnot
return 0;
}
I have a struct that has the following memory layout:
uint32_t
variable length array of type uint16_t
variable length array of type uint16_t
Because of the variable length of the arrays I have pointers to these arrays, effectively:
struct struct1 {
uint32_t n;
uint16_t *array1;
uint16_t *array2;
};
typedef struct struct1 struct1;
Now, when allocation these structs I see two options:
A) malloc the struct itself, then malloc space for the arrays individually and set the pointers in the struct to point to the correct memory location:
uint32_t n1 = 10;
uint32_t n2 = 20;
struct1 *s1 = malloc(sizeof(struct1));
uint16 *array1 = malloc(sizeof(uint16) * n1));
uint16 *array2 = malloc(sizeof(uint16) * n2));
s1->n = n1;
s1->array1 = array1;
s1->array2 = array2;
B) malloc memory for everything combined, then "distribute" the memory over the struct:
struct1 *s1 = malloc(sizeof(struct1) + (n1 + n2) * sizeof(uint16_t));
s1->n = n1;
s1->array1 = s1 + sizeof(struct1);
s1->array2 = s1 + sizeof(struct1) + n1 * sizeof(uint16_t);
Note that array1 and array2 are not bigger than a few KB and usually not a lot of struct1s are needed. However, cache efficiency is a concern as numeric data crunching is done with this struct.
Is approach B) possible and if so better (faster) than A in terms of memory locality?
I am not very familiar with C, is there a better way of doing B (or A), ie. using memcpy or realloc or something?
Anything else to be mindful about in this situation?
Note, that right now I'm using gcc (C89?) on linux but could use C99/C11 if necessary. Thanks in advance.
EDIT: To clarify further: The size of the arrays will never change after creation. Multiple struct1s will not be always be allocated at once but rather occasionally during the program's runtime.
I think your option A is much cleaner and would scale in a more sensible way. Imagine having to realloc space when the array in one of the structures becomes larger: in option A, you can realloc that memory since it isn't logically attached to anything else. In option B, you need to add in additional logic to ensure you don't break your other array.
I also think (even in C89, but I could be wrong) that there is nothing wrong with this:
struct1 *s1 = malloc(sizeof(struct1));
s1->array1 = malloc(sizeof(uint16) * n1));
s1->array2 = malloc(sizeof(uint16) * n2));
s1->n = n1;
The above takes out the middle-man arrays. I think it is cleaner because you immediately see that you are allocating space for a pointer in a structure.
I have used your option B before for 2D arrays, where I just allocate a single space and use logical rules in my code to use it as a 2D space. This is useful when I want it to be a rectangular 2D space, so when I increase it, I always increase each row or column. In other words, I never want to have heterogeneous array sizes.
Update: 'Arrays will never change in size'
Because you clarified that your structures/arrays will never need to be reallocated, I think option B is less bad. It still seems to be a worse solution for this application than option A, and here are my reasons for thinking this:
malloc is optimized such that there wouldn't be much optimization from allocating a single space compared to allocating the spaces individually.
The ability of other engineers to look at and immediately understand your code would be reduced. To be clear, any competent software engineer should be able to look at option B and figure out what the writer of the code was doing, but it very well could waste that engineers' brain-cycles and could cause a junior engineer to misunderstand the code and create a bug.
So, if you comment the code thoroughly, and your application absolutely requires you to optimize everything you possibly can, at the expense of clean and logically sensible code (where memory space and data structures are logically separated in a similar way), and you know that this optimization is better than what a good compiler (like Clang) can do, then option B could be a better option.
Update: Testing
In the spirit of self-criticism I wanted to see if I could evaluate the difference. So I wrote two programs (one for option A and one for option B) and compiled them with optimizations off. I used a FreeBSD virtual machine to get as clean of an environment as possible, and I used gcc.
Here are the programs that I used to test the two methods:
optionA.c:
#include <stdlib.h>
#include <stdio.h>
#include <time.h>
#define NSIZE 100000
#define NTESTS 10000000
struct test_struct {
int n;
int *array1;
int *array2;
};
void freeA(struct test_struct *input) {
free(input->array1);
free(input->array2);
free(input);
return;
}
void optionA() {
struct test_struct *s1 = malloc(sizeof(*s1));
s1->array1 = malloc(sizeof(*(s1->array1)) * NSIZE);
s1->array2 = malloc(sizeof(*(s1->array1)) * NSIZE);
s1->n = NSIZE;
freeA(s1);
s1 = 0;
return;
}
int main() {
clock_t beginA = clock();
int i;
for (i=0; i<NTESTS; i++) {
optionA();
}
clock_t endA = clock();
int time_spent_A = (endA - beginA);
printf("Time spent for option A: %d\n", time_spent_A);
return 0;
}
optionB.c:
#include <stdlib.h>
#include <stdio.h>
#include <time.h>
#define NSIZE 100000
#define NTESTS 10000000
struct test_struct {
int n;
int *array1;
int *array2;
};
void freeB(struct test_struct *input) {
free(input);
return;
}
void optionB() {
struct test_struct *s1 = malloc(sizeof(*s1) + 2*NSIZE*sizeof(*(s1->array1)));
s1->array1 = s1 + sizeof(*s1);
s1->array2 = s1 + sizeof(*s1) + NSIZE*sizeof(*(s1->array1));
s1->n = NSIZE;
freeB(s1);
s1 = 0;
return;
}
int main() {
clock_t beginB = clock();
int i;
for (i=0; i<NTESTS; i++) {
optionB();
}
clock_t endB = clock();
int time_spent_B = (endB - beginB);
printf("Time spent for option B: %d\n", time_spent_B);
return 0;
}
Results for these tests are given in clocks (see clock(3) for more information).
Series | Option A | Option B
------------------------------
1 | 332 | 158
------------------------------
2 | 334 | 155
------------------------------
3 | 334 | 156
------------------------------
4 | 333 | 154
------------------------------
5 | 339 | 156
------------------------------
6 | 334 | 155
------------------------------
avg | 336.0 | 155.7
------------------------------
Note that these speeds are still incredibly fast and translate to milliseconds over millions of tests. I have also found that Clang (cc) is better than gcc at optimizing. On my machine, even after writing a method that writes data to the arrays (to ensure they don't get optimized out of existence) I got no differential between the two methods when compiling with cc.
I would advice a hybrid of the two:
allocate the structs in one call (it is now an array of structs);
allocate the arrays in one call, and make sure the size includes any padding for the allignment required by your compiler/platform;
distribute the arrays over the structs, taking the allignment into acount.
However, malloc is already optimized, so your first solution would still be prefered.
Note: as user Greg Schmit's solution points out, allocating all the arrays in one time, will cause difficulties if the array size needs to be changed in run-time
Because the two arrays have the same type, there are much more options than that, based on creative use of the C99 flexible array member. I'd recommend you use a pointer only for the second array,
struct foo {
uint16_t *array2;
uint32_t field;
uint16_t array1[];
};
and allocate memory for both at the same time:
struct foo *foo_new(const size_t length1, const size_t length2)
{
struct foo *result;
result = malloc( sizeof (struct foo)
+ length1 * sizeof (uint16_t)
+ length2 * sizeof (uint16_t) );
if (!result)
return NULL;
result->array2 = result->array1 + length1;
return result;
}
Note that with struct foo *bar, accessing element i in the two arrays uses the same notation, bar->array1[i] and bar->array2[i], respectively.
In the context of scientific computing, I would consider completely other options, depending on the access patterns. For example, if the two arrays are accessed in lockstep (in any direction), I would use
typedef uint16_t pair16[2];
struct bar {
uint32_t field;
pair16 array[];
};
If the arrays were large, then copying them into temporary buffers (arrays of pair16 above, if accessed in lockstep) would possibly help, but with at most a few thousand entries, it is likely not going to give a significant speed boost.
In cases where the access pattern depends on the other, but you still do enough of computation on each entry, it may be useful to compute the address of the next entry early, and use __builtin_prefetch() GCC built-in to tell the CPU you'll need it soon, before doing the computation on the current entry. It may reduce the data access latencies (although the access predictors are pretty darn good on current processors already).
With GCC (and to a lesser extent on other common compilers like Intel Compiler Collection, Portland Group, and Pathscale C compilers), I've noticed that code that manipulates pointers (instead of array pointers and array indexing) compiles to better machine code on x86 and x86-64. (The reason is actually quite simple: with array pointers and array indexing, you need at least two separate registers, and x86 has relatively few of those. Even x86-64 doesn't have that many of them. GCC in particular is not very strong at optimizing register usage -- it's much better now than in the version 3 era --, so this seems to help a lot in some cases). For example, if you were to access the first array in a struct foo sequentially, then
void do_something(struct foo *ref)
{
uint16_t *array1 = ref->array1;
uint16_t *const limit1 = ref->array1 + (number of elements in array1);
for (; array1 < limit1; array1++) {
/* ... */
}
}
Approach B is possible, (why don't you just try it?) and it is better, not so much because of memory locality, but because malloc() costs, so the fewer times you call it, the better off you are. (Assuming that 'better' means 'faster', which admittedly, is not necessarily the case.)
Memory locality is only marginally improved, since all memory blocks would most likely be continuous (one after the other) in memory, so if you went with approach A your arrays would only be separated by block headers, which are not very big. (Of the order of 32 bytes each, maybe a bit larger, but not by much.) The only situation in which your blocks would not be continuous is if you had previously been doing malloc() and free(), so your memory would be fragmented.
I've allocated an "array" of mystruct of size n like this:
if (NULL == (p = calloc(sizeof(struct mystruct) * n,1))) {
/* handle error */
}
Later on, I only have access to p, and no longer have n. Is there a way to determine the length of the array given just the pointer p?
I figure it must be possible, since free(p) does just that. I know malloc() keeps track of how much memory it has allocated, and that's why it knows the length; perhaps there is a way to query for this information? Something like...
int length = askMallocLibraryHowMuchMemoryWasAlloced(p) / sizeof(mystruct)
I know I should just rework the code so that I know n, but I'd rather not if possible. Any ideas?
No, there is no way to get this information without depending strongly on the implementation details of malloc. In particular, malloc may allocate more bytes than you request (e.g. for efficiency in a particular memory architecture). It would be much better to redesign your code so that you keep track of n explicitly. The alternative is at least as much redesign and a much more dangerous approach (given that it's non-standard, abuses the semantics of pointers, and will be a maintenance nightmare for those that come after you): store the lengthn at the malloc'd address, followed by the array. Allocation would then be:
void *p = calloc(sizeof(struct mystruct) * n + sizeof(unsigned long int),1));
*((unsigned long int*)p) = n;
n is now stored at *((unsigned long int*)p) and the start of your array is now
void *arr = p+sizeof(unsigned long int);
Edit: Just to play devil's advocate... I know that these "solutions" all require redesigns, but let's play it out.
Of course, the solution presented above is just a hacky implementation of a (well-packed) struct. You might as well define:
typedef struct {
unsigned int n;
void *arr;
} arrInfo;
and pass around arrInfos rather than raw pointers.
Now we're cooking. But as long as you're redesigning, why stop here? What you really want is an abstract data type (ADT). Any introductory text for an algorithms and data structures class would do it. An ADT defines the public interface of a data type but hides the implementation of that data type. Thus, publicly an ADT for an array might look like
typedef void* arrayInfo;
(arrayInfo)newArrayInfo(unsignd int n, unsigned int itemSize);
(void)deleteArrayInfo(arrayInfo);
(unsigned int)arrayLength(arrayInfo);
(void*)arrayPtr(arrayInfo);
...
In other words, an ADT is a form of data and behavior encapsulation... in other words, it's about as close as you can get to Object-Oriented Programming using straight C. Unless you're stuck on a platform that doesn't have a C++ compiler, you might as well go whole hog and just use an STL std::vector.
There, we've taken a simple question about C and ended up at C++. God help us all.
keep track of the array size yourself; free uses the malloc chain to free the block that was allocated, which does not necessarily have the same size as the array you requested
Just to confirm the previous answers: There is no way to know, just by studying a pointer, how much memory was allocated by a malloc which returned this pointer.
What if it worked?
One example of why this is not possible. Let's imagine the code with an hypothetic function called get_size(void *) which returns the memory allocated for a pointer:
typedef struct MyStructTag
{ /* etc. */ } MyStruct ;
void doSomething(MyStruct * p)
{
/* well... extract the memory allocated? */
size_t i = get_size(p) ;
initializeMyStructArray(p, i) ;
}
void doSomethingElse()
{
MyStruct * s = malloc(sizeof(MyStruct) * 10) ; /* Allocate 10 items */
doSomething(s) ;
}
Why even if it worked, it would not work anyway?
But the problem of this approach is that, in C, you can play with pointer arithmetics. Let's rewrite doSomethingElse():
void doSomethingElse()
{
MyStruct * s = malloc(sizeof(MyStruct) * 10) ; /* Allocate 10 items */
MyStruct * s2 = s + 5 ; /* s2 points to the 5th item */
doSomething(s2) ; /* Oops */
}
How get_size is supposed to work, as you sent the function a valid pointer, but not the one returned by malloc. And even if get_size went through all the trouble to find the size (i.e. in an inefficient way), it would return, in this case, a value that would be wrong in your context.
Conclusion
There are always ways to avoid this problem, and in C, you can always write your own allocator, but again, it is perhaps too much trouble when all you need is to remember how much memory was allocated.
Some compilers provide msize() or similar functions (_msize() etc), that let you do exactly that
May I recommend a terrible way to do it?
Allocate all your arrays as follows:
void *blockOfMem = malloc(sizeof(mystruct)*n + sizeof(int));
((int *)blockofMem)[0] = n;
mystruct *structs = (mystruct *)(((int *)blockOfMem) + 1);
Then you can always cast your arrays to int * and access the -1st element.
Be sure to free that pointer, and not the array pointer itself!
Also, this will likely cause terrible bugs that will leave you tearing your hair out. Maybe you can wrap the alloc funcs in API calls or something.
malloc will return a block of memory at least as big as you requested, but possibly bigger. So even if you could query the block size, this would not reliably give you your array size. So you'll just have to modify your code to keep track of it yourself.
For an array of pointers you can use a NULL-terminated array. The length can then determinate like it is done with strings. In your example you can maybe use an structure attribute to mark then end. Of course that depends if there is a member that cannot be NULL. So lets say you have an attribute name, that needs to be set for every struct in your array you can then query the size by:
int size;
struct mystruct *cur;
for (cur = myarray; cur->name != NULL; cur++)
;
size = cur - myarray;
Btw it should be calloc(n, sizeof(struct mystruct)) in your example.
Other have discussed the limits of plain c pointers and the stdlib.h implementations of malloc(). Some implementations provide extensions which return the allocated block size which may be larger than the requested size.
If you must have this behavior you can use or write a specialized memory allocator. This simplest thing to do would be implementing a wrapper around the stdlib.h functions. Some thing like:
void* my_malloc(size_t s); /* Calls malloc(s), and if successful stores
(p,s) in a list of handled blocks */
void my_free(void* p); /* Removes list entry and calls free(p) */
size_t my_block_size(void* p); /* Looks up p, and returns the stored size */
...
really your question is - "can I find out the size of a malloc'd (or calloc'd) data block". And as others have said: no, not in a standard way.
However there are custom malloc implementations that do it - for example http://dmalloc.com/
I'm not aware of a way, but I would imagine it would deal with mucking around in malloc's internals which is generally a very, very bad idea.
Why is it that you can't store the size of memory you allocated?
EDIT: If you know that you should rework the code so you know n, well, do it. Yes it might be quick and easy to try to poll malloc but knowing n for sure would minimize confusion and strengthen the design.
One of the reasons that you can't ask the malloc library how big a block is, is that the allocator will usually round up the size of your request to meet some minimum granularity requirement (for example, 16 bytes). So if you ask for 5 bytes, you'll get a block of size 16 back. If you were to take 16 and divide by 5, you would get three elements when you really only allocated one. It would take extra space for the malloc library to keep track of how many bytes you asked for in the first place, so it's best for you to keep track of that yourself.
This is a test of my sort routine. It sets up 7 variables to hold float values, then assigns them to an array, which is used to find the max value.
The magic is in the call to myMax:
float mmax = myMax((float *)&arr,(int) sizeof(arr)/sizeof(arr[0]));
And that was magical, wasn't it?
myMax expects a float array pointer (float *) so I use &arr to get the address of the array, and cast it as a float pointer.
myMax also expects the number of elements in the array as an int. I get that value by using sizeof() to give me byte sizes of the array and the first element of the array, then divide the total bytes by the number of bytes in each element. (we should not guess or hard code the size of an int because it's 2 bytes on some system and 4 on some like my OS X Mac, and could be something else on others).
NOTE:All this is important when your data may have a varying number of samples.
Here's the test code:
#include <stdio.h>
float a, b, c, d, e, f, g;
float myMax(float *apa,int soa){
int i;
float max = apa[0];
for(i=0; i< soa; i++){
if (apa[i]>max){max=apa[i];}
printf("on i=%d val is %0.2f max is %0.2f, soa=%d\n",i,apa[i],max,soa);
}
return max;
}
int main(void)
{
a = 2.0;
b = 1.0;
c = 4.0;
d = 3.0;
e = 7.0;
f = 9.0;
g = 5.0;
float arr[] = {a,b,c,d,e,f,g};
float mmax = myMax((float *)&arr,(int) sizeof(arr)/sizeof(arr[0]));
printf("mmax = %0.2f\n",mmax);
return 0;
}
In uClibc, there is a MALLOC_SIZE macro in malloc.h:
/* The size of a malloc allocation is stored in a size_t word
MALLOC_HEADER_SIZE bytes prior to the start address of the allocation:
+--------+---------+-------------------+
| SIZE |(unused) | allocation ... |
+--------+---------+-------------------+
^ BASE ^ ADDR
^ ADDR - MALLOC_HEADER_SIZE
*/
/* The amount of extra space used by the malloc header. */
#define MALLOC_HEADER_SIZE \
(MALLOC_ALIGNMENT < sizeof (size_t) \
? sizeof (size_t) \
: MALLOC_ALIGNMENT)
/* Set up the malloc header, and return the user address of a malloc block. */
#define MALLOC_SETUP(base, size) \
(MALLOC_SET_SIZE (base, size), (void *)((char *)base + MALLOC_HEADER_SIZE))
/* Set the size of a malloc allocation, given the base address. */
#define MALLOC_SET_SIZE(base, size) (*(size_t *)(base) = (size))
/* Return base-address of a malloc allocation, given the user address. */
#define MALLOC_BASE(addr) ((void *)((char *)addr - MALLOC_HEADER_SIZE))
/* Return the size of a malloc allocation, given the user address. */
#define MALLOC_SIZE(addr) (*(size_t *)MALLOC_BASE(addr))
malloc() stores metadata regarding space allocation before 8 bytes from space actually allocated. This could be used to determine space of buffer. And on my x86-64 this always return multiple of 16. So if allocated space is multiple of 16 (which is in most cases) then this could be used:
Code
#include <stdio.h>
#include <malloc.h>
int size_of_buff(void *buff) {
return ( *( ( int * ) buff - 2 ) - 17 ); // 32 bit system: ( *( ( int * ) buff - 1 ) - 17 )
}
void main() {
char *buff = malloc(1024);
printf("Size of Buffer: %d\n", size_of_buff(buff));
}
Output
Size of Buffer: 1024
This is my approach:
#include <stdio.h>
#include <stdlib.h>
typedef struct _int_array
{
int *number;
int size;
} int_array;
int int_array_append(int_array *a, int n)
{
static char c = 0;
if(!c)
{
a->number = NULL;
a->size = 0;
c++;
}
int *more_numbers = NULL;
a->size++;
more_numbers = (int *)realloc(a->number, a->size * sizeof(int));
if(more_numbers != NULL)
{
a->number = more_numbers;
a->number[a->size - 1] = n;
}
else
{
free(a->number);
printf("Error (re)allocating memory.\n");
return 1;
}
return 0;
}
int main()
{
int_array a;
int_array_append(&a, 10);
int_array_append(&a, 20);
int_array_append(&a, 30);
int_array_append(&a, 40);
int i;
for(i = 0; i < a.size; i++)
printf("%d\n", a.number[i]);
printf("\nLen: %d\nSize: %d\n", a.size, a.size * sizeof(int));
free(a.number);
return 0;
}
Output:
10
20
30
40
Len: 4
Size: 16
If your compiler supports VLA (variable length array), you can embed the array length into the pointer type.
int n = 10;
int (*p)[n] = malloc(n * sizeof(int));
n = 3;
printf("%d\n", sizeof(*p)/sizeof(**p));
The output is 10.
You could also choose to embed the information into the allocated memory yourself with a structure including a flexible array member.
struct myarray {
int n;
struct mystruct a[];
};
struct myarray *ma =
malloc(sizeof(*ma) + n * sizeof(struct mystruct));
ma->n = n;
struct mystruct *p = ma->a;
Then to recover the size, you would subtract the offset of the flexible member.
int get_size (struct mystruct *p) {
struct myarray *ma;
char *x = (char *)p;
ma = (void *)(x - offsetof(struct myarray, a));
return ma->n;
}
The problem with trying to peek into heap structures is that the layout might change from platform to platform or from release to release, and so the information may not be reliably obtainable.
Even if you knew exactly how to peek into the meta information maintained by your allocator, the information stored there may have nothing to do with the size of the array. The allocator simply returned memory that could be used to fit the requested size, but the actual size of the memory may be larger (perhaps even much larger) than the requested amount.
The only reliable way to know the information is to find a way to track it yourself.
I have to create a very inexpensive algorithm (processor and memory) to remove the first char from a string (char array) in C.
I'm currently using:
char *newvalue = strdup(value+1);
free(value);
value = newvalue;
But I want to know if there is some less expensive way to do that. The string value is dynamically allocated.
value+1 is a char* that represent the string with the first character removed. It is the less expensive way to obtain such a string..
You'll have to be careful when freeing memory though to make sure to free the original pointer and not the shifted one.
Reuse the original array. May or may not be faster, depend on the relative speed of memory (de)allocation and copy.
int size = strlen(value);
if (size > 0) memmove(value, value+1, size);
Since heap calls will be quite expensive, the obvious optimization is to avoid them.
If you need to do this often, you could probably come up with some simple wrapper around the bare pointer that can express this.
Something like:
typedef struct {
const char *chars;
size_t offset;
} mystring;
Then you'd need to devise an API to convert a mystring * into a character pointer, by adding the offset:
const char * mystring_get(const mystring *ms)
{
return ms->chars + ms->offset;
}
and of course a function to create a suffix where the 1st character is removed:
mystring mystring_tail(const mystring *ms)
{
const mystring suffix = { ms->chars, ms->offset + 1};
return suffix;
}
note that mystring_tail() returns the new string structure by value, to avoid heap allocations.
This is really strange... and I can't debug it (tried for about two hours, debugger starts going haywire after a while...). Anyway, I'm trying to do something really simple:
Free an array of strings. The array is in the form:
char **myStrings. The array elements are initialized as:
myString[index] = malloc(strlen(word));
myString[index] = word;
and I'm calling a function like this:
free_memory(myStrings, size); where size is the length of the array (I know this is not the problem, I tested it extensively and everything except this function is working).
free_memory looks like this:
void free_memory(char **list, int size) {
for (int i = 0; i < size; i ++) {
free(list[i]);
}
free(list);
}
Now here comes the weird part. if (size> strlen(list[i])) then the program crashes. For example, imagine that I have a list of strings that looks something like this:
myStrings[0] = "Some";
myStrings[1] = "random";
myStrings[2] = "strings";
And thus the length of this array is 3.
If I pass this to my free_memory function, strlen(myStrings[0]) > 3 (4 > 3), and the program crashes.
However, if I change myStrings[0] to be "So" instead, then strlen(myStrings[0]) < 3 (2 < 3) and the program does not crash.
So it seems to me that free(list[i]) is actually going through the char[] that is at that location and trying to free each character, which I imagine is undefined behavior.
The only reason I say this is because I can play around with the size of the first element of myStrings and make the program crash whenever I feel like it, so I'm assuming that this is the problem area.
Note: I did try to debug this by stepping through the function that calls free_memory, noting any weird values and such, but the moment I step into the free_memory function, the debugger crashes, so I'm not really sure what is going on. Nothing is out of the ordinary until I enter the function, then the world explodes.
Another note: I also posted the shortened version of the source for this program (not too long; Pastebin) here. I am compiling on MinGW with the c99 flag on.
PS - I just thought of this. I am indeed passing numUniqueWords to the free function, and I know that this does not actually free the entire piece of memory that I allocated. I've called it both ways, that's not the issue. And I left it how I did because that is the way that I will be calling it after I get it to work in the first place, I need to revise some of my logic in that function.
Source, as per request (on-site):
#include <stdio.h>
#include <string.h>
#include <ctype.h>
#include <stdlib.h>
#include "words.h"
int getNumUniqueWords(char text[], int size);
int main(int argc, char* argv[]) {
setvbuf(stdout, NULL, 4, _IONBF); // For Eclipse... stupid bug. --> does NOT affect the program, just the output to console!
int nbr_words;
char text[] = "Some - \"text, a stdin\". We'll have! also repeat? We'll also have a repeat!";
int length = sizeof(text);
nbr_words = getNumUniqueWords(text, length);
return 0;
}
void free_memory(char **list, int size) {
for (int i = 0; i < size; i ++) {
// You can see that printing the values is fine, as long as free is not called.
// When free is called, the program will crash if (size > strlen(list[i]))
//printf("Wanna free value %d w/len of %d: %s\n", i, strlen(list[i]), list[i]);
free(list[i]);
}
free(list);
}
int getNumUniqueWords(char text[], int length) {
int numTotalWords = 0;
char *word;
printf("Length: %d characters\n", length);
char totalWords[length];
strcpy(totalWords, text);
word = strtok(totalWords, " ,.-!?()\"0123456789");
while (word != NULL) {
numTotalWords ++;
printf("%s\n", word);
word = strtok(NULL, " ,.-!?()\"0123456789");
}
printf("Looks like we counted %d total words\n\n", numTotalWords);
char *uniqueWords[numTotalWords];
char *tempWord;
int wordAlreadyExists = 0;
int numUniqueWords = 0;
char totalWordsCopy[length];
strcpy(totalWordsCopy, text);
for (int i = 0; i < numTotalWords; i++) {
uniqueWords[i] = NULL;
}
// Tokenize until all the text is consumed.
word = strtok(totalWordsCopy, " ,.-!?()\"0123456789");
while (word != NULL) {
// Look through the word list for the current token.
for (int j = 0; j < numTotalWords; j ++) {
// Just for clarity, no real meaning.
tempWord = uniqueWords[j];
// The word list is either empty or the current token is not in the list.
if (tempWord == NULL) {
break;
}
//printf("Comparing (%s) with (%s)\n", tempWord, word);
// If the current token is the same as the current element in the word list, mark and break
if (strcmp(tempWord, word) == 0) {
printf("\nDuplicate: (%s)\n\n", word);
wordAlreadyExists = 1;
break;
}
}
// Word does not exist, add it to the array.
if (!wordAlreadyExists) {
uniqueWords[numUniqueWords] = malloc(strlen(word));
uniqueWords[numUniqueWords] = word;
numUniqueWords ++;
printf("Unique: %s\n", word);
}
// Reset flags and continue.
wordAlreadyExists = 0;
word = strtok(NULL, " ,.-!?()\"0123456789");
}
// Print out the array just for funsies - make sure it's working properly.
for (int x = 0; x <numUniqueWords; x++) {
printf("Unique list %d: %s\n", x, uniqueWords[x]);
}
printf("\nNumber of unique words: %d\n\n", numUniqueWords);
// Right below is where things start to suck.
free_memory(uniqueWords, numUniqueWords);
return numUniqueWords;
}
You've got an answer to this question, so let me instead answer a different question:
I had multiple easy-to-make mistakes -- allocating a wrong-sized buffer and freeing non-malloc'd memory. I debugged it for hours and got nowhere. How could I have spent that time more effectively?
You could have spent those hours writing your own memory allocators that would find the bug automatically.
When I was writing a lot of C and C++ code I made helper methods for my program that turned all mallocs and frees into calls that did more than just allocate memory. (Note that methods like strdup are malloc in disguise.) If the user asked for, say, 32 bytes, then my helper method would add 24 to that and actually allocate 56 bytes. (This was on a system with 4-byte integers and pointers.) I kept a static counter and a static head and tail of a doubly-linked list. I would then fill in the memory I allocated as follows:
Bytes 0-3: the counter
Bytes 4-7: the prev pointer of a doubly-linked list
Bytes 8-11: the next pointer of a doubly-linked list
Bytes 12-15: The size that was actually passed in to the allocator
Bytes 16-19: 01 23 45 67
Bytes 20-51: 33 33 33 33 33 33 ...
Bytes 52-55: 89 AB CD EF
And return a pointer to byte 20.
The free code would take the pointer passed in and subtract four, and verify that bytes 16-19 were still 01 23 45 67. If they were not then either you are freeing a block you did not allocate with this allocator, or you've written before the pointer somehow. Either way, it would assert.
If that check succeeded then it would go back four more and read the size. Now we know where the end of the block is and we can verify that bytes 52 through 55 are still 89 AB CD EF. If they are not then you are writing over the end of a block somewhere. Again, assert.
Now that we know that the block is not corrupt we remove it from the linked list, set ALL the memory of the block to CC CC CC CC ... and free the block. We use CC because that is the "break into the debugger" instruction on x86. If somehow we end up with the instruction pointer pointing into such a block it is nice if it breaks!
If there is a problem then you also know which allocation it was, because you have the allocation count in the block.
Now we have a system that finds your bugs for you. In the release version of your product, simply turn it off so that your allocator just calls malloc normally.
Moreover you can use this system to find other bugs. If for example you believe that you've got a memory leak somewhere all you have to do is look at the linked list; you have a complete list of all the outstanding allocations and can figure out which ones are being kept around unnecessarily. If you think you're allocating too much memory for a given block then you can have your free code check to see if there are a lot of 33 in the block that is about to be freed; that's a sign that you're allocating your blocks too big. And so on.
And finally: this is just a starting point. When I was using this debug allocator professionally I extended it so that it was threadsafe, so that it could tell me what kind of allocator was doing the allocation (malloc, strdup, new, IMalloc, etc.), whether there was a mismatch between the alloc and free functions, what source file contained the allocation, what the call stack was at the time of the allocation, what the average, minimum and maximum block sizes were, what subsystems were responsible for what memory usage...
C requires that you manage your own memory; this definitely has its pros and cons. My opinion is that the cons outweigh the pros; I much prefer to work in automatic storage languages. But the nice thing about having to manage your own storage is that you are free to build a storage management system that meets your needs, and that includes your debugging needs. If you must use a language that requires you to manage storage, use that power to your advantage and build a really powerful subsystem that you can use to solve professional-grade problems.
The problem is not how you're freeing, but how you're creating the array. Consider this:
uniqueWords[numUniqueWords] = malloc(strlen(word));
uniqueWords[numUniqueWords] = word;
...
word = strtok(NULL, " ,.-!?()\"0123456789");
There are several issues here:
word = strtok(): what strtok returns is not something that you can free, because it has not been malloc'ed. ie it is not a copy, it just points to somewhere inside the underlying large string (the thing you called strtok with first).
uniqueWords[numUniqueWords] = word: this is not a copy; it just assigns the pointer. the pointer which is there before (which you malloc'ed) is overwritten.
malloc(strlen(word)): this allocates too little memory, should be strlen(word)+1
How to fix:
Option A: copy properly
// no malloc
uniqueWords[numUniqueWords] = strdup(word); // what strdup returns can be free'd
Option B: copy properly, slightly more verbose
uniqueWords[numUniqueWords] = malloc(strlen(word)+1);
strcpy(uniqueWords[numUniqueWords], word); // use the malloc'ed memory to copy to
Option C: don't copy, don't free
// no malloc
uniqueWords[numUniqueWords] = word; // not a copy, this still points to the big string
// don't free this, ie don't free(list[i]) in free_memory
EDIT As other have pointed out, this is also problematic:
char *uniqueWords[numTotalWords];
I believe this is a GNU99 extension (not even C99), and indeed you cannot (should not) free it. Try char **uniqueWords = (char**)malloc(sizeof(char*) * numTotalWords). Again the problem is not the free() but the way you allocate. You are on the right track with the free, just need to match every free with a malloc, or with something that says it is equivalent to a malloc (like strdup).
You are using this code in an attempt to allocate the memory:
uniqueWords[numUniqueWords] = malloc(strlen(word));
uniqueWords[numUniqueWords] = word;
numUniqueWords++;
This is wrong on many levels.
You need to allocate strlen(word)+1 bytes of memory.
You need to strcpy() the string over the allocated memory; at the moment, you simply throw the allocated memory away.
Your array uniqueWords is itself not allocated, and the word values you have stored are from the original string which has been mutilated by strtok().
As it stands, you cannot free any memory because you've already lost the pointers to the memory that was allocated and the memory you are trying to free was never in fact allocated by malloc() et al.
And you should be error checking the memory allocations too. Consider using strdup() to duplicate strings.
You are trying to free char *uniqueWords[numTotalWords];, which is not allowed in C.
Since uniqueWords is allocated on the stack and you can't call free on stack memory.
Just remove the last free call, like this:
void free_memory(char **list, int size) {
for (int i = 0; i < size; i ++) {
free(list[i]);
}
}
Proper way of allocating and deallocating char array.
char **foo = (char **) malloc(row* sizeof(char *));
*foo = malloc(row * col * sizeof(char));
for (int i = 1; i < row; i++) {
foo[i] = *foo + i*col;
}
free(*foo);
free(foo);
Note that you don't need to go through each & every element of the array for deallocation of memory. Arrays are contiguous so call free on the name of the array.