So, to start off I've already looked at a few questions including this one and none of them seem to help.
I'm simply trying to write a function that extends the size of an array using realloc().
My code currently looks like this:
unsigned char *xtnd = malloc(4);
xtndc(&xtnd, 4);
// sizeof(*xtnd) should now be 8
void xtndc ( unsigned char ** bytesRef , uint8_t count ) {
*bytesRef = realloc(*bytesRef, (sizeof(**bytesRef)) + count);
}
But no matter what I do it seems that the size of xtnd is always 4. After running xtndc() on it it should now be 8 bytes long.
Any suggestions?
The type of **bytesRef is unsigned char, so sizeof(**bytesRef) is 1. sizeof doesn't keep track of dynamic allocations, it's a compile time tool that gives you the size of a type, in this case unsigned char.
You have to keep track of the array size manually to calculate the new required size.
Your program does in fact change the size of the memory block. It changes the size of your original memory block from 4 bytes to 5 bytes. It changes to 5 bytes because you are essentially doing sizeof(unsigned char) + 4 which 1 + 4 = 5. If you want to double the size instead, do count*sizeof(unsigned char) + count. There are two points to be noted here:
The sizeof function returns the size of the data type, not the size of the allocated bytes. There is no way to know the size of the dynamically allocated memory.
The function realloc (and malloc and calloc as well) is not always guaranteed to return the requested reallocation. It may or may not succeed all the time.
I fixed the problem with the following code.
typedef struct CArrPtr {
unsigned char* ptr;
size_t size;
} CArrPtr;
void xtndc ( CArrPtr *bytesRef, uint8_t count );
. . .
CArrPtr xtnd = { .ptr = malloc(4), .size = 4 };
xtndc( &xtnd, 4 );
// xtnd.size is now 8 bytes
. . .
void xtndc ( CArrPtr *bytesRef, uint8_t count ) {
unsigned char *nptr;
if((nptr = realloc(bytesRef->ptr, bytesRef->size + count)) != 0)
{
bytesRef->ptr = nptr;
bytesRef->size = bytesRef->size + count;
}
}
As I am somewhat new to C, what I learned from this is that malloc specifically creates a pointer to a memory block, but you have no direct access to information about the memory block. Instead, you must store the size of the array that you created with malloc somewhere as well.
Since in the past I'd been initializing arrays with unsigned char arr[size]; and then using sizeof on it, I was under the impression that sizeof returned the size of the array, which is of course wrong as it gives you the size of a type.
Glad I could learn something from this.
sizeof is used to calculate size of data type or array. Pointer and array are very similar, but they are different things. For int *ap, sizeof(ap) will return 4 on x86, sizeof(*ap) will return 4; for int a[10], sizeof(a) will return 40.
sizeof expression is processed at compile time, so it will be a constant written into the executable file before you run the program.
malloc and realloc don't maintain size.
If realloc succeeds, it will reallocate the requested size. So you don't need to check the size after realloc returns, but you should check the return value of realloc to ensure that realloc succeeds.
I've allocated an "array" of mystruct of size n like this:
if (NULL == (p = calloc(sizeof(struct mystruct) * n,1))) {
/* handle error */
}
Later on, I only have access to p, and no longer have n. Is there a way to determine the length of the array given just the pointer p?
I figure it must be possible, since free(p) does just that. I know malloc() keeps track of how much memory it has allocated, and that's why it knows the length; perhaps there is a way to query for this information? Something like...
int length = askMallocLibraryHowMuchMemoryWasAlloced(p) / sizeof(mystruct)
I know I should just rework the code so that I know n, but I'd rather not if possible. Any ideas?
No, there is no way to get this information without depending strongly on the implementation details of malloc. In particular, malloc may allocate more bytes than you request (e.g. for efficiency in a particular memory architecture). It would be much better to redesign your code so that you keep track of n explicitly. The alternative is at least as much redesign and a much more dangerous approach (given that it's non-standard, abuses the semantics of pointers, and will be a maintenance nightmare for those that come after you): store the lengthn at the malloc'd address, followed by the array. Allocation would then be:
void *p = calloc(sizeof(struct mystruct) * n + sizeof(unsigned long int),1));
*((unsigned long int*)p) = n;
n is now stored at *((unsigned long int*)p) and the start of your array is now
void *arr = p+sizeof(unsigned long int);
Edit: Just to play devil's advocate... I know that these "solutions" all require redesigns, but let's play it out.
Of course, the solution presented above is just a hacky implementation of a (well-packed) struct. You might as well define:
typedef struct {
unsigned int n;
void *arr;
} arrInfo;
and pass around arrInfos rather than raw pointers.
Now we're cooking. But as long as you're redesigning, why stop here? What you really want is an abstract data type (ADT). Any introductory text for an algorithms and data structures class would do it. An ADT defines the public interface of a data type but hides the implementation of that data type. Thus, publicly an ADT for an array might look like
typedef void* arrayInfo;
(arrayInfo)newArrayInfo(unsignd int n, unsigned int itemSize);
(void)deleteArrayInfo(arrayInfo);
(unsigned int)arrayLength(arrayInfo);
(void*)arrayPtr(arrayInfo);
...
In other words, an ADT is a form of data and behavior encapsulation... in other words, it's about as close as you can get to Object-Oriented Programming using straight C. Unless you're stuck on a platform that doesn't have a C++ compiler, you might as well go whole hog and just use an STL std::vector.
There, we've taken a simple question about C and ended up at C++. God help us all.
keep track of the array size yourself; free uses the malloc chain to free the block that was allocated, which does not necessarily have the same size as the array you requested
Just to confirm the previous answers: There is no way to know, just by studying a pointer, how much memory was allocated by a malloc which returned this pointer.
What if it worked?
One example of why this is not possible. Let's imagine the code with an hypothetic function called get_size(void *) which returns the memory allocated for a pointer:
typedef struct MyStructTag
{ /* etc. */ } MyStruct ;
void doSomething(MyStruct * p)
{
/* well... extract the memory allocated? */
size_t i = get_size(p) ;
initializeMyStructArray(p, i) ;
}
void doSomethingElse()
{
MyStruct * s = malloc(sizeof(MyStruct) * 10) ; /* Allocate 10 items */
doSomething(s) ;
}
Why even if it worked, it would not work anyway?
But the problem of this approach is that, in C, you can play with pointer arithmetics. Let's rewrite doSomethingElse():
void doSomethingElse()
{
MyStruct * s = malloc(sizeof(MyStruct) * 10) ; /* Allocate 10 items */
MyStruct * s2 = s + 5 ; /* s2 points to the 5th item */
doSomething(s2) ; /* Oops */
}
How get_size is supposed to work, as you sent the function a valid pointer, but not the one returned by malloc. And even if get_size went through all the trouble to find the size (i.e. in an inefficient way), it would return, in this case, a value that would be wrong in your context.
Conclusion
There are always ways to avoid this problem, and in C, you can always write your own allocator, but again, it is perhaps too much trouble when all you need is to remember how much memory was allocated.
Some compilers provide msize() or similar functions (_msize() etc), that let you do exactly that
May I recommend a terrible way to do it?
Allocate all your arrays as follows:
void *blockOfMem = malloc(sizeof(mystruct)*n + sizeof(int));
((int *)blockofMem)[0] = n;
mystruct *structs = (mystruct *)(((int *)blockOfMem) + 1);
Then you can always cast your arrays to int * and access the -1st element.
Be sure to free that pointer, and not the array pointer itself!
Also, this will likely cause terrible bugs that will leave you tearing your hair out. Maybe you can wrap the alloc funcs in API calls or something.
malloc will return a block of memory at least as big as you requested, but possibly bigger. So even if you could query the block size, this would not reliably give you your array size. So you'll just have to modify your code to keep track of it yourself.
For an array of pointers you can use a NULL-terminated array. The length can then determinate like it is done with strings. In your example you can maybe use an structure attribute to mark then end. Of course that depends if there is a member that cannot be NULL. So lets say you have an attribute name, that needs to be set for every struct in your array you can then query the size by:
int size;
struct mystruct *cur;
for (cur = myarray; cur->name != NULL; cur++)
;
size = cur - myarray;
Btw it should be calloc(n, sizeof(struct mystruct)) in your example.
Other have discussed the limits of plain c pointers and the stdlib.h implementations of malloc(). Some implementations provide extensions which return the allocated block size which may be larger than the requested size.
If you must have this behavior you can use or write a specialized memory allocator. This simplest thing to do would be implementing a wrapper around the stdlib.h functions. Some thing like:
void* my_malloc(size_t s); /* Calls malloc(s), and if successful stores
(p,s) in a list of handled blocks */
void my_free(void* p); /* Removes list entry and calls free(p) */
size_t my_block_size(void* p); /* Looks up p, and returns the stored size */
...
really your question is - "can I find out the size of a malloc'd (or calloc'd) data block". And as others have said: no, not in a standard way.
However there are custom malloc implementations that do it - for example http://dmalloc.com/
I'm not aware of a way, but I would imagine it would deal with mucking around in malloc's internals which is generally a very, very bad idea.
Why is it that you can't store the size of memory you allocated?
EDIT: If you know that you should rework the code so you know n, well, do it. Yes it might be quick and easy to try to poll malloc but knowing n for sure would minimize confusion and strengthen the design.
One of the reasons that you can't ask the malloc library how big a block is, is that the allocator will usually round up the size of your request to meet some minimum granularity requirement (for example, 16 bytes). So if you ask for 5 bytes, you'll get a block of size 16 back. If you were to take 16 and divide by 5, you would get three elements when you really only allocated one. It would take extra space for the malloc library to keep track of how many bytes you asked for in the first place, so it's best for you to keep track of that yourself.
This is a test of my sort routine. It sets up 7 variables to hold float values, then assigns them to an array, which is used to find the max value.
The magic is in the call to myMax:
float mmax = myMax((float *)&arr,(int) sizeof(arr)/sizeof(arr[0]));
And that was magical, wasn't it?
myMax expects a float array pointer (float *) so I use &arr to get the address of the array, and cast it as a float pointer.
myMax also expects the number of elements in the array as an int. I get that value by using sizeof() to give me byte sizes of the array and the first element of the array, then divide the total bytes by the number of bytes in each element. (we should not guess or hard code the size of an int because it's 2 bytes on some system and 4 on some like my OS X Mac, and could be something else on others).
NOTE:All this is important when your data may have a varying number of samples.
Here's the test code:
#include <stdio.h>
float a, b, c, d, e, f, g;
float myMax(float *apa,int soa){
int i;
float max = apa[0];
for(i=0; i< soa; i++){
if (apa[i]>max){max=apa[i];}
printf("on i=%d val is %0.2f max is %0.2f, soa=%d\n",i,apa[i],max,soa);
}
return max;
}
int main(void)
{
a = 2.0;
b = 1.0;
c = 4.0;
d = 3.0;
e = 7.0;
f = 9.0;
g = 5.0;
float arr[] = {a,b,c,d,e,f,g};
float mmax = myMax((float *)&arr,(int) sizeof(arr)/sizeof(arr[0]));
printf("mmax = %0.2f\n",mmax);
return 0;
}
In uClibc, there is a MALLOC_SIZE macro in malloc.h:
/* The size of a malloc allocation is stored in a size_t word
MALLOC_HEADER_SIZE bytes prior to the start address of the allocation:
+--------+---------+-------------------+
| SIZE |(unused) | allocation ... |
+--------+---------+-------------------+
^ BASE ^ ADDR
^ ADDR - MALLOC_HEADER_SIZE
*/
/* The amount of extra space used by the malloc header. */
#define MALLOC_HEADER_SIZE \
(MALLOC_ALIGNMENT < sizeof (size_t) \
? sizeof (size_t) \
: MALLOC_ALIGNMENT)
/* Set up the malloc header, and return the user address of a malloc block. */
#define MALLOC_SETUP(base, size) \
(MALLOC_SET_SIZE (base, size), (void *)((char *)base + MALLOC_HEADER_SIZE))
/* Set the size of a malloc allocation, given the base address. */
#define MALLOC_SET_SIZE(base, size) (*(size_t *)(base) = (size))
/* Return base-address of a malloc allocation, given the user address. */
#define MALLOC_BASE(addr) ((void *)((char *)addr - MALLOC_HEADER_SIZE))
/* Return the size of a malloc allocation, given the user address. */
#define MALLOC_SIZE(addr) (*(size_t *)MALLOC_BASE(addr))
malloc() stores metadata regarding space allocation before 8 bytes from space actually allocated. This could be used to determine space of buffer. And on my x86-64 this always return multiple of 16. So if allocated space is multiple of 16 (which is in most cases) then this could be used:
Code
#include <stdio.h>
#include <malloc.h>
int size_of_buff(void *buff) {
return ( *( ( int * ) buff - 2 ) - 17 ); // 32 bit system: ( *( ( int * ) buff - 1 ) - 17 )
}
void main() {
char *buff = malloc(1024);
printf("Size of Buffer: %d\n", size_of_buff(buff));
}
Output
Size of Buffer: 1024
This is my approach:
#include <stdio.h>
#include <stdlib.h>
typedef struct _int_array
{
int *number;
int size;
} int_array;
int int_array_append(int_array *a, int n)
{
static char c = 0;
if(!c)
{
a->number = NULL;
a->size = 0;
c++;
}
int *more_numbers = NULL;
a->size++;
more_numbers = (int *)realloc(a->number, a->size * sizeof(int));
if(more_numbers != NULL)
{
a->number = more_numbers;
a->number[a->size - 1] = n;
}
else
{
free(a->number);
printf("Error (re)allocating memory.\n");
return 1;
}
return 0;
}
int main()
{
int_array a;
int_array_append(&a, 10);
int_array_append(&a, 20);
int_array_append(&a, 30);
int_array_append(&a, 40);
int i;
for(i = 0; i < a.size; i++)
printf("%d\n", a.number[i]);
printf("\nLen: %d\nSize: %d\n", a.size, a.size * sizeof(int));
free(a.number);
return 0;
}
Output:
10
20
30
40
Len: 4
Size: 16
If your compiler supports VLA (variable length array), you can embed the array length into the pointer type.
int n = 10;
int (*p)[n] = malloc(n * sizeof(int));
n = 3;
printf("%d\n", sizeof(*p)/sizeof(**p));
The output is 10.
You could also choose to embed the information into the allocated memory yourself with a structure including a flexible array member.
struct myarray {
int n;
struct mystruct a[];
};
struct myarray *ma =
malloc(sizeof(*ma) + n * sizeof(struct mystruct));
ma->n = n;
struct mystruct *p = ma->a;
Then to recover the size, you would subtract the offset of the flexible member.
int get_size (struct mystruct *p) {
struct myarray *ma;
char *x = (char *)p;
ma = (void *)(x - offsetof(struct myarray, a));
return ma->n;
}
The problem with trying to peek into heap structures is that the layout might change from platform to platform or from release to release, and so the information may not be reliably obtainable.
Even if you knew exactly how to peek into the meta information maintained by your allocator, the information stored there may have nothing to do with the size of the array. The allocator simply returned memory that could be used to fit the requested size, but the actual size of the memory may be larger (perhaps even much larger) than the requested amount.
The only reliable way to know the information is to find a way to track it yourself.
How can i know the size of the array using a pointer that is allocated using malloc?
#include <stdio.h>
int main(){
int *ptr = (int *)malloc(sizeof(int * 10));
printf("Size:%d",sizeof(ptr));
free(ptr_one);
return 0;
}
I get only the size of the pointer in this case which is 8.How to modify the code to get the size of array which will be 40.
You cannot.
You will need to do the bookkeeping and keep track of it yourself. With new you allocate dynamic memory and while deallocating the memory you just call delete, which knows how much memory it has deallocate because the language takes care of it internally so that users do not need to bother about the bookkeeping. If you still need it explicitly then you need to track it through separate variable.
if your machine is 32 bit you will get pointer size always 4 bytes of any data type
if your machine is 64 bit you will get pointer size always 8 bytes of any data type
if you declare static array you will get size by using sizeof
int a[10];
printf("Size:%lu",sizeof(a));
But you did not get the size of array which is blocked by pointer. where the memory to the block is allocated dynamically using malloc .
see this below code:
#include <stdio.h>
#include<stdlib.h>
int main()
{
int i;
int *ptr = (int *)malloc(sizeof(int) * 10);
// printf("Size:%lu",sizeof(ptr));
// In the above case sizeof operater returns size of pointer only.
for(i=1;ptr && i<13 ;i++,ptr++)
{
printf("Size:%d %p\n",((int)sizeof(i))*i,ptr);
}
return 0;
}
output:
Size:4 0x8ec010
Size:8 0x8ec014
Size:12 0x8ec018
Size:16 0x8ec01c
Size:20 0x8ec020
Size:24 0x8ec024
Size:28 0x8ec028
Size:32 0x8ec02c
Size:36 0x8ec030
Size:40 0x8ec034 //after this there is no indication that block ends.
Size:44 0x8ec038
Size:48 0x8ec03c
About C structs and pointers...
Yesterday I wrote sort of the following code (try to memorize parts of it out of my memory):
typedef struct {
unsigned short int iFrames;
unsigned short int* iTime; // array with elements [0..x] holding the timing for each frame
} Tile;
Tile* loadTile(char* sFile)
{
// expecting to declare enough space for one complete Tile structure, of which the base memory address is stored in the tmpResult pointer
Tile* tmpResult = malloc(sizeof(Tile));
// do things that set values to the Tile entity
// ...
// return the pointer for further use
return tmpResult;
}
void main()
{
// define a tile pointer and set its value to the returned pointer (this should also be allowed in one row)
// Expected to receive the VALUE of the pointer - i.e. the base memory address at where malloc made space available
Tile* tmpTile;
tmpTile = loadTile("tile1.dat");
// get/set elements of the tile
// ...
// free the tile
free(tmpTile);
}
What I see: I cán use the malloced Tile structure inside the function, but once I try to access it in Main, I get an error from Visual Studio about the heap (which tells me that something is freed after the call is returned).
If I change it so that I malloc space in Main, and pass the pointer to this space to the loadTile function as an argument (so that the function does no longer return anything) then it does work but I am confident that I should also be able do let the loadTile function malloc the space and return a pointer to that space right?!
Thanks!!
There's nothing wrong with what you're trying to do, or at least not from the code here. However, I'm concerned about this line:
unsigned short int* iTime; // array with elements [0..x] holding the timing for each frame
That isn't true unless you're also mallocing iTime somewhere:
Tile* tmpResult = malloc(sizeof(Tile));
tmpResult->iTime = malloc(sizeof(short) * n);
You will need to free it when you clean up:
free(tmpTile->iTime);
free(tmpTile);
You are probably writing over memory you don't own. I guess that in this section:
// do things that set values to the Tile entity
you're doing this:
tmpResult->iFrames = n;
for (i = 0 ; i < n ; ++n)
{
tmpResult->iTime [i] = <some value>;
}
which is wrong, you need to allocate separate memory for the array:
tmpResult->iTime = malloc (sizeof (short int) * n);
before writing to it. This make freeing the object more complex:
free (tile->iTime);
free (tile);
Alternatively, do this:
typedef struct {
unsigned short int iFrames;
unsigned short int iTime [1]; // array with elements [0..x] holding the timing for each frame
} Tile;
and malloc like this:
tile = malloc (sizeof (Tile) + sizeof (short int) * (n - 1)); // -1 since Tile already has one int defined.
and the for loop remains the same:
for (i = 0 ; i < n ; ++n)
{
tmpResult->iTime [i] = <some value>;
}
but freeing the tile is then just:
free (tile);
as you've only allocated one chunk of memory, not two. This works because C (and C++) does not do range checking on arrays.
You code, with as little changes as I could live with, works for me:
#include <stdio.h>
#include <stdlib.h>
typedef struct {
unsigned short int iFrames;
unsigned short int* iTime;
} Tile;
Tile *loadTile(char* sFile) {
Tile *tmpResult = malloc(sizeof *tmpResult);
if (!tmpResult) return NULL;
/* do things that set values to the Tile entity */
/* note that iTime is uninitialized */
tmpResult->iFrames = 42;
(void)sFile; /* used parameter */
return tmpResult;
}
int main(void) {
Tile* tmpTile;
tmpTile = loadTile("tile1.dat");
if (!tmpTile) return 1;
printf("value: %d\n", tmpTile->iFrames);
free(tmpTile);
return 0;
}
The code you showed looks OK, the error must be in the elided code.
Whatever problem you are having, it is not in the code shown in this question. Make sure you are not clobbering the pointer before returning it.
This should work fine... could just be a warning from VisualStudio that you are freeing a pointer in a different function than it was malloced in.
Technically, your code will work on a C compiler. However, allocating dynamically inside functions and returning pointers to the allocated data is an excellent way of creating memory leaks - therefore it is very bad programming practice. A better way is to allocate the memory in the caller (main in this case). The code unit allocating the memory should be the same one that frees it.
Btw if this is a Windows program, main() must be declared to return int, or the code will not compile on a C compiler.
I am writing a light weight serialization function and need to include two variable sized arrays within this.
How should I track the size of each?
How should I define the struct?
Am I going about this all wrong?
EDIT: the result must be a contiguous block of memory
This resolves to something like
typedef struct
{
size_t arr_size_1, arr_size_2;
char arr_1[0/*arr_size_1 + arr_size_2*/];
} ...;
The size(s) should be in the front of the dynamic sized data, so that it doesn't move when expanding your array.
You cannot have 2 unknown sized arrays in your struct, so you must collapse them into one and then access the data relative from the first pointer.
typedef struct MyStruct_s
{
int variable_one_size;
void* variable_one_buf;
int variable_two_size;
void* variable_two_buf;
} MyStruct;
MyStruct* CreateMyStruct (int size_one, int size_two)
{
MyStruct* s = (MyStruct*)malloc (sizeof (MyStruct));
s->variable_one_size = size_one;
s->variable_one_buf = malloc (size_one);
s->variable_two_size = size_two;
s->variable_two_buf = malloc (size_two);
}
void FreeMyStruct (MyStruct* s)
{
free (s->variable_one_buf);
free (s->variable_two_buf);
free (s);
}
Since the data should be continuous in memory it is necessary to malloc a chunk of memory of the right size and manage it's contents more or less manually. You probably best create a struct that contains the "static" information and related management functions that do the memory management and give access to the "dynamic" members of the struct:
typedef struct _serial {
size_t sz_a;
size_t sz_b;
char data[1]; // "dummy" array as pointer to space at end of the struct
} serial;
serial* malloc_serial(size_t a, size_t b) {
serial *result;
// malloc more memory than just sizeof(serial), so that there
// is enough space "in" the data member for both of the variable arrays
result = malloc(sizeof(serial) - 1 + a + b);
if (result) {
result->sz_a = a;
result->sz_b = b;
}
return result;
}
// access the "arrays" in the struct:
char* access_a(serial *s) {
return &s->data[0];
}
char* access_b(serial *s) {
return &s->data[s->sz_a];
}
Then you could do things like this:
serial *s = ...;
memcpy(access_a(s), "hallo", 6);
access_a(s)[1] = 'e';
Also note that you can't just assign one serial to another one, you need to make sure that the sizes are compatible and copy the data manually.
In order to serialize variably-sized data, you have to have a boundary tag of some sort. The boundary tag can be either a size written right before the data, or it can be a special value that is not allowed to appear in the data stream and is written right after the data.
Which you choose depends on how much data you are storing, and if you are optimizing for size in the output stream. It is often easier to store a size before-hand, because you know how big to make the receiving buffer. If you don't then you have to gradually resize your buffer on load.
In some ways, I'd do things like Dan Olson. However:
1) I'd create the final struct by having two instances of a simpler struct that has just one variable array.
2) I'd declare the array with byte* and use size_t for its length.
Having said this, I'm still not entirely clear on what you're trying to do.
edit
If you want it contiguous in memory, just define a struct with two lengths. Then allocate a block big enough for both blocks that you want to pass, plus the struct itself. Set the two lengths and copy the two blocks immediately after. I think it should be clear how the lengths suffice to make the struct self-describing.