As this answer on another question covers, using an aggregate initialization
struct foo {
size_t a;
size_t b;
};
struct foo bar = {0};
results in built-in types being initialized to zero.
Is there any difference between using the above and using
struct foo * bar2 = calloc(1, sizeof(struct foo));
leaving aside the fact that one variable is a pointer.
Looking at the debugger we can see that both a and b are indeed set to zero for both of the above examples.
What's the difference between two above examples, are there any gotchas or hidden issues?
Yes, there is a crucial difference (aside from storage-class of your object of type struct foo):
struct foo bar = {0};
struct foo * bar2 = calloc(1, sizeof *bar2);
Every member of bar is zero-initialized (and the padding is zeroed out for sub-object without initializer, or if bar is of static or thread_local storage-class),
while all of *bar2 is zeroed out, which might have completely different results:
Neither null-pointers (T*)0 nor floating-point-numbers with value 0 are guaranteed to be all-bits-0.
(Actually, only for char, unsigned char and signed char (as well as some of the optional exact-size-types from <stdint.h>) it is guaranteed that all-bits-0 matches value-0 till some time after C99. A later technical corrigenda guaranteed it for all integral types.)
The floating-point-format might not be IEEE754.
(On most modern systems you can ignore that possibility though.)
Cite from c-faq (Thanks to Jim Balter for linking it):
The Prime 50 series used segment 07777, offset 0 for the null pointer, at least for PL/I.
struct foo bar = {0};
This defines an object of type struct foo named bar, and initializes it to zero.
"Zero" is defined recursively. All integer subobjects are initialized to 0, all floating-point subobjects to 0.0, and all pointers to NULL.
struct foo * bar2 = calloc(1, sizeof(struct foo));
IMHO this is better (but equivalently) written as:
struct foo *bar2 = calloc(1, sizeof *bar2);
By not repeating the type name, we avoid the risk of a mismatch when the code is changed later on.
This dynamically allocates an object of type struct foo (on the heap), initializes that object to all-bits-zero, and initializes bar2 to point to it.
calloc can fail to allocate memory. If it does, it returns a null pointer. You should always check for that. (The declaration of bar also allocates memory, but if it fails it's a stack overflow, and there's no good way to handle it.)
And all-bits-zero is not guaranteed to be the same as "zero". For integer types (including size_t), it's very nearly guaranteed. For floating-point and pointer types, it's entirely legal for 0.0 or NULL to have some internal representation other than all-bits-zero. You're unlikely to run into this, and since all the members of your structure are integer you probably don't need to worry about it.
calloc gives you a heap dynamically allocated zeroed memory zone (into your bar2). But an automatic variable (like bar, assuming its declaration is inside a function) is allocated on the call stack. See also calloc(3)
In C, you need to explicitly free heap allocated memory zone. But stack allocated data is popped when its function is returning.
Rerad also wikipage on C dynamic memory allocation, and on garbage collection. Reference counting is a widely used technique in C and in C++, and could be viewed as a form of GC. Think about circular references, they are hard to handle.
The Boehm conservative GC can be used in C programs.
Notice that the liveness of a memory zone is a global program-wide property. You generally cannot claim that a give zone belongs to a particular function (or library). But you could adopt conventions about that.
When you code a function returning a heap-allocated pointer (i.e. some pointer to dynamic storage) you should document that fact and decide who is in charge of freeing it.
About initialization: a calloc pointer is zeroed (when calloc succeeds). An automatic variable initialized as {0} is also zeroed. In practice, some implementations may calloc differently big objects (by asking whole zeroed pages from the kernel for them, e.g. with mmap(2)) and small objects (by reusing, if available, a previously free-d zone and zeroing it). zero-ing a zone is using a fast equivalent of memset(3)
PS. I am ignoring the weird machines on which an all zero-bit memory zone is not a cleared data for the C standard, i.e. like {0}. I don't know such machines in practice, even if I know they are in principle possible (and in theory the NULL pointer might not be an all-zero-bit word)
BTW, the compiler may optimize an all-zero local structure (and perhaps not allocate it at all on the stack, since it would fit in registers).
(This answer focuses on the differences in initialization, in the case of a struct only containing integral types)
Both forms set a and b to 0. This is because the Standard defines that all-bits-zero for an integral type must represent a value of 0.
If there is structure padding, then the calloc version sets that but the zero-initialization may not. For example:
struct foo a = { 0 }, b = { 0 };
struct foo c, d; memset(&c, 0, sizeof c); memset(&d, 0, sizeof d);
if ( memcmp(&a, &b, sizeof a) )
printf("This line may appear.\n");
if ( memcmp(&c, &d, sizeof c) )
printf("This line must not appear.\n");
A technique you will sometimes see (especially in code designed to fit on systems with small amounts of storage) is that of using memcmp to compare two structs for equality. When there is padding between structure members, this is unreliable as the padding may be different even though the structure members are the same.
The programmer didn't want to compare structure members individually as that is too much code size, so instead, he will copy structs around using memcpy, initialize them using memset; in order to preserve the ability to use memcmp to check for equality.
In modern programming I'd strongly advise to not do this; and to always use the { 0 } form of initailization. Another benefit of the latter is that there is no chance of making a mistake with the size argument and accidentally setting too much memory or too little memory.
There is a serious difference: allocation of automatic variables is done at compile-time and comes for free (when the stack frame is reserved, the room is there.) On the opposite, dynamic allocation is done at run-time and has an unpredictible and non neglectible cost.
As regards initialization, the compiler has opportunities for optimization with automatic variables (for instance by not clearing if unnecessary); this is not possible with a call of calloc.
If you like the calloc style, you also have the option of performing memset on the automatic variable.
memset(&bar, 0, sizeof bar);
UPDATE: allocation of automatic variables is quasi-done at compile-time.
Related
The setup
Let's say I have a struct father which has member variables such as an int, and another struct(so father is a nested struct). This is an example code:
struct mystruct {
int n;
};
struct father {
int test;
struct mystruct M;
struct mystruct N;
};
In the main function, we allocate memory with malloc() to create a new struct of type struct father, then we fill it's member variables and those of it's children:
struct father* F = (struct father*) malloc(sizeof(struct father));
F->test = 42;
F->M.n = 23;
F->N.n = 11;
We then get pointers to those member variables from outside the structs:
int* p = &F->M.n;
int* q = &F->N.n;
After that, we print the values before and after the execution of free(F), then exit:
printf("test: %d, M.n: %d, N.n: %d\n", F->test, *p, *q);
free(F);
printf("test: %d, M.n: %d, N.n: %d\n", F->test, *p, *q);
return 0;
This is a sample output(*):
test: 42, M.n: 23, N.n: 11
test: 0, M.n: 0, N.n: 1025191952
*: Using gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
Full code on pastebin: https://pastebin.com/khzyNPY1
The question
That was the test program that I used to test how memory is deallocated using free(). My idea(from reading K&R "8.7 Example - A Storage Allocator", in which a version of free() is implemented and explained) is that, when you free() the struct, you're pretty much just telling the operating system or the rest of the program that you won't be using that particular space in memory that was previously allocated with malloc(). So, after freeing those memory blocks, there should be garbage values in the member variables, right? I can see that happening with N.n in the test program, but, as I ran more and more samples, it was clear that in the overwhelming majority of cases, these member variables are "reset" to 0 more than any other "random" value. My question is: why is that? Is it because the stack/heap is filled with zeroes more frequently than any other value?
As a last note, here are a few links to related questions but which do not answer my particular question:
C - freeing structs
What REALLY happens when you don't free after malloc?
After calling free, the pointers F, p and q no longer point to valid memory. Attempting to dereference those pointers invokes undefined behavior. In fact, the values of those pointers become indeterminate after the call to free, so you may also invoke UB just by reading those pointer values.
Because dereferencing those pointers is undefined behavior, the compiler can assume it will never happen and make optimizations based on that assumption.
That being said, there's nothing that states that the malloc/free implementation has to leave values that were stored in freed memory unchanged or set them to specific values. It might write part of its internal bookkeeping state to the memory you just freed, or it might not. You'd have to look at the source for glibc to see exactly what it's doing.
Apart from undefined behavior and whatever else the standard might dictate, since the dynamic allocator is a program, fixed a specific implementation, assuming it does not make decisions based on external factors (which it does not) the behavior is completely deterministic.
Real answer: what you are seeing here is the effect of the internal workings of glibc's allocator (glibc is the default C library on Ubuntu).
The internal structure of an allocated chunk is the following (source):
struct malloc_chunk {
INTERNAL_SIZE_T mchunk_prev_size; /* Size of previous chunk (if free). */
INTERNAL_SIZE_T mchunk_size; /* Size in bytes, including overhead. */
struct malloc_chunk* fd; /* double links -- used only if free. */
struct malloc_chunk* bk;
/* Only used for large blocks: pointer to next larger size. */
struct malloc_chunk* fd_nextsize; /* double links -- used only if free. */
struct malloc_chunk* bk_nextsize;
};
In memory, when the chunk is in use (not free), it looks like this:
chunk-> +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Size of previous chunk, if unallocated (P clear) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Size of chunk, in bytes |A|M|P| flags
mem-> +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| User data starts here... |
Every field except mchunk_prev_size and mchunk_size is only populated if the chunk is free. Those two fields are right before the user usable buffer. User data begins right after mchunk_size (i.e. at the offset of fd), and can be arbitrarily large. The mchunk_prev_size field holds the size of the previous chunk if it's free, while the mchunk_size field holds the real size of the chunk (which is at least 16 bytes more than the requested size).
A more thorough explanation is provided as comments in the library itself here (highly suggested read if you want to know more).
When you free() a chunk, there are a lot of decisions to be made as to where to "store" that chunk for bookkeeping purposes. In general, freed chunks are sorted into double linked lists based on their size, in order to optimize subsequent allocations (that can get already available chunks of the right size from these lists). You can see this as a sort of caching mechanism.
Now, depending on your glibc version, they could be handled slightly differently, and the internal implementation is quite complex, but what is happening in your case is something like this:
struct malloc_chunk *victim = addr; // address passed to free()
// Add chunk at the head of the free list
victim->fd = NULL;
victim->bk = head;
head->fd = victim;
Since your structure is basically equivalent to:
struct x {
int a;
int b;
int c;
}
And since on your machine sizeof(struct malloc_chunk *) == 2 * sizeof(int), the first operation (victim->fd = NULL) is effectively wiping out the contents of the first two fields of your structure (remember, user data begins exactly at fd), while the second one (victim->bk = head) is altering the third value.
The Standard specifies nothing about the behavior of a program that uses a pointer to allocated storage after it has been freed. Implementations are free to extend the language by specifying the behavior of more programs than required by the Standard, and the authors of the Standard intended to encourage variety among implementations which would support popular extensions on a quality-of-implementation basis directed by the marketplace. Some operations with pointers to dead objects are widely supported (e.g. given char *x,*y; the Standard would allow conforming implementations to behave in arbitrary fashion if a program executes free(x); y=x; in cases where x had been non-null, without regard for whether anything ever does anything with y after its initialization, but most implementations would extend the language to guarantee that such code would have no effect if y is never used) but dereferencing of such pointers generally isn't.
Note that if one were to pass two copies of the same pointer to a freed object to:
int test(char *p1, char *p2)
{
char *q;
if (*p1)
{
q = malloc(0):
free(q);
return *p1+*p2;
}
else
return 0;
}
it is entirely possible that the act of allocating and freeing q would disturb the bit patterns in the storage that had been allocated to *p1 (and also *p2), but a compiler would not be required to allow for that possibility. A compiler might plausibly return the sum of the value that was read from *p1 before the malloc/free, and a value that was read from *p2 after it; this sum could be an odd number even though if p1 and p2 are equal, *p1+*p2 should always be even.
Two things happen when you call free:
In the C model of computing, any pointer values that point to the freed memory (either its beginning, such as your F, or things within it, such as your p and q) are no longer valid. The C standard does not define what happens when you attempt to use these pointer values, and optimization by the compiler may have unexpected effects on how your program behaves if you attempt to use them.
The freed memory is released for other purposes. One of the most common other purposes for which it is used is tracking memory that is available for allocation. In other words, the software that implements malloc and free needs data structures to record which blocks of memory have been freed and other information. When you free memory, that software often uses some of the memory for this purpose. That can result in the changes you saw.
The freed memory may also be used by other things in your program. In a single-threaded program without signal handlers or similar things, generally no software would run between the free and the preparation of the arguments to the printf you show, so nothing else would reuse the memory so quickly—reuse by the malloc software is the most likely explanation for what you observed. However, in a multithreaded program, the memory might be reused immediately by another thread. (In practice, this may be a bit unlikely, as the malloc software may keep preferentially separate pools of memory for separate threads, to reduce the amount of inter-thread synchronization that is necessary.)
When a dynamically allocated object is freed, it no longer exists. Any subsequent attempt to access it has undefined behavior. The question is therefore nonsense: the members of an allocated struct cease to exist at the end of the host struct's lifetime, so they cannot be set or reset to anything at that point. There is no valid way to attempt to determine any values for such no-longer-existing objects.
Why is the value of
int array[10];
undefined when declared in a function and is 0-initialized when declared as static?
I have been reading the answer of this question and it is clear that
[the expression int array[10];] in a function means: take the ownership of 10-int-size area of memory without doing any initialization. If the array is declared as a global one or as static in a function, then all elements are initialized to zero if they aren't initialized already.
Question: why this behaviour? Do the compiler programmers decide that (for a particular reason)? Can a particular compiler used do the things differently?
Why I am asking this: I am asking this question because I would like to make my code portable among architectures/compilers. In order to ensure it, I know I can always initialize the declared array. But this means that I will lose precious time only for this operation. So, which is the right decision?
An automatic int array[10]; isn't implicitly zeroed because the zeroing takes time and you might not need it zeroed. Additionally, you'd pay the cost not just once but each time control ran past the initialized variable.
A static/global int array[10]; is implicitly zeroed because statics/globals are allocated at load time. The memory will be fresh from the OS and if the OS is security conscious at all, the memory will have been zeroed already. Otherwise the loading code (the OS or a dynamic linker) will have to zero them (because the C standard requires it), but it should be able to do it in one call to memset for all globals/statics, which is considerably more efficient than zeroing each static/global variable at a time.
This initialization is done once. Even statics inside of functions are initialized just once, even if they have nonzero initializers (e.g., static int x = 42;. This is why C requires that the initializer of a static be a constant expression).
Since the loadtime zeroing of all globals/statics is either OS-guaranteed or efficiently implementable, it might as well be standard-guaranteed and thereby make programmers' lives easier.
The values are not undefined but indeterminate, and it behaves this way because the standard says so.
Section 6.7.9p10 of the C standard regarding initialization states:
If an object that has automatic storage duration is not
initialized explicitly, its value is indeterminate. If an
object that has static or thread storage duration is not
initialized explicitly, then:
if it has pointer type, it is initialized to a null pointer;
if it has arithmetic type, it is initialized to (positive or unsigned) zero;
if it is an aggregate, every member is initialized (recursively) according to these rules,and any padding is initialized to zero bits;
if it is a union, the first named member is initialized (recursively) according to theserules, and any padding is initialized
to zero bits;
So for any variable defined either at file scope or static you can safely assume the values are zero-initialized. For variables declared inside of a function or scope, you cannot make any assumptions about uninitialized variables.
As for why, global/static variables are initialized at program startup or even at compile time, while locals have to be initialized each time they come into scope and doing so would take time.
The reason for not defining the initial value of the variables in stack-allocated/local variables is efficiency. The C Standard expects your program to allocate your array and later fill it:
int array[10];
for (i = 0; i < 10; ++i)
array[i] = i * 42;
In this case, any initialization would be pointless, so the C Standard wants to avoid it.
If your program needs these values initialized to zero, you can do it explicitly:
int array[10] = {0}; // initialize to zero so the accumulation below works
while (condition)
{
... // some code
for (i = 0; i < 10; ++i)
array[i] += other_array[i];
}
It is your decision whether to initialize or not, because you are supposed to know how your program behaves. This decision will be different for different arrays.
However, this decision will not depend on a compiler - they are all standard-compliant. One little detail regarding portability - if you don't initialize your array and still see all zeros in it when you use a particular compiler - don't be fooled; the values are still undefined; you cannot rely on them being 0.
Some other languages decided that zero initialization is cheap enough to do even if it's superfluous, and its advantage (safety) outweighs its disadvantage (performance). In C, performance is more important, so it decided otherwise.
The C philosophy is to a) always trust the programmer and b) prioritize execution speed over programmer convenience. C assumes that the programmer is in the best position to know whether an array (or any other auto variable) needs to be initialized to a specific value, and if so, is smart enough to write the code to do it themselves. Otherwise it won't waste the CPU cycles.
Same thing for bounds checking on array accesses, same thing for NULL checks on pointer dereferences, etc.
This is simultaneously C's greatest strength (fast code with a small footprint) and greatest weakness (lots of manual labor to make code safe and secure).
This is actually a much more concise, much more clear question than the one I had asked here before(for any who cares): C Language: Why does malloc() return a pointer, and not the value? (Sorry for those who initially think I'm spamming... I hope it's not construed as the same question since I think the way I phrased it there made it unintentionally misleading)
-> Basically what I'm trying to ask is: Why does a C programmer need a pointer to a dynamically-allocated variable/object? (whatever the difference is between variable/object...)
If a C programmer has the option of creating just 'int x' or just 'int *x' (both statically allocated), then why can't he also have the option to JUST initialize his dynamically-allocated variable/object as a variable (and NOT returning a pointer through malloc())?
*If there are some obscure ways to do what I explained above, then, well, why does malloc() seem the way that most textbooks go about dynamic-allocation?
Note: in the following, byte refers to sizeof(char)
Well, for one, malloc returns a void *. It simply can't return a value: that wouldn't be feasible with C's lack of generics. In C, the compiler must know the size of every object at compile time; since the size of the memory being allocated will not be known until run time, then a type that could represent any value must be returned. Since void * can represent any pointer, it is the best choice.
malloc also cannot initialize the block: it has no knowledge of what's being allocated. This is in contrast with C++'s operator new, which does both the allocation and the initialization, as well as being type safe (it still returns a pointer instead of a reference, probably for historical reasons).
Also, malloc allocates a block of memory of a specific size, then returns a pointer to that memory (that's what malloc stands for: memory allocation). You're getting a pointer because that's what you get: an unitialized block of raw memory. When you do, say, malloc(sizeof(int)), you're not creating a int, you're allocating sizeof(int) bytes and getting the address of those bytes. You can then decide to use that block as an int, but you could also technically use that as an array of sizeof(int) chars.
The various alternatives (calloc, realloc) work roughly the same way (calloc is easier to use when dealing with arrays, and zero-fills the data, while realloc is useful when you need to resize a block of memory).
Suppose you create an integer array in a function and want to return it. Said array is a local variable to the function. You can't return a pointer to a local variable.
However, if you use malloc, you create an object on the heap whose scope exceeds the function body. You can return a pointer to that. You just have to destroy it later or you will have a memory leak.
It's because objects allocated with malloc() don't have names, so the only way to reference that object in code is to use a pointer to it.
When you say int x;, that creates an object with the name x, and it is referenceable through that name. When I want to set x to 10, I can just use x = 10;.
I can also set a pointer variable to point to that object with int *p = &x;, and then I can alternatively set the value of x using *p = 10;. Note that this time we can talk about x without specifically naming it (beyond the point where we acquire the reference to it).
When I say malloc(sizeof(int)), that creates an object that has no name. I cannot directly set the value of that object by name, since it just doesn't have one. However, I can set it by using a pointer variable that points at it, since that method doesn't require naming the object: int *p = malloc(sizeof(int)); followed by *p = 10;.
You might now ask: "So, why can't I tell malloc to give the object a name?" - something like malloc(sizeof(int), "x"). The answer to this is twofold:
Firstly, C just doesn't allow variable names to be introduced at runtime. It's just a basic restriction of the language;
Secondly, given the first restriction the name would have to be fixed at compile-time: if this is the case, C already has syntax that does what you want: int x;.
You are thinking about things wrong. It is not that int x is statically allocated and malloc(sizeof(int)) is dynamic. Both are allocated dynamically. That is, they are both allocated at execution time. There is no space reserved for them at the time you compile. The size may be static in one case and dynamic in the other, but the allocation is always dynamic.
Rather, it is that int x allocates the memory on the stack and malloc(sizeof(int)) allocates the memory on the heap. Memory on the heap requires that you have a pointer in order to access it. Memory on the stack can be referenced directly or with a pointer. Usually you do it directly, but sometimes you want to iterate over it with pointer arithmetic or pass it to a function that needs a pointer to it.
Everything works using pointers. "int x" is just a convenience - someone, somewhere got tired of juggling memory addresses and that's how programming languages with human-readable variable names were born.
Dynamic allocation is... dynamic. You don't have to know how much space you are going to need when the program runs - before the program runs. You choose when to do it and when to undo it. It may fail. It's hard to handle all this using the simple syntax of static allocation.
C was designed with simplicity in mind and compiler simplicity is a part of this. That's why you're exposed to the quirks of the underlying implementations. All systems have storage for statically-sized, local, temporary variables (registers, stack); this is what static allocation uses. Most systems have storage for dynamic, custom-lifetime objects and system calls to manage them; this is what dynamic allocation uses and exposes.
There is a way to do what you're asking and it's called C++. There, "MyInt x = 42;" is a function call or two.
I think your question comes down to this:
If a C programmer has the option of creating just int x or just int *x (both statically allocated)
The first statement allocates memory for an integer. Depending upon the placement of the statement, it might allocate the memory on the stack of a currently executing function or it might allocate memory in the .data or .bss sections of the program (if it is a global variable or static variable, at either file scope or function scope).
The second statement allocates memory for a pointer to an integer -- it hasn't actually allocated memory for the integer itself. If you tried to assign a value using the pointer *x=1, you would either receive a very quick SIGSEGV segmentation violation or corrupt some random piece of memory. C doesn't pre-zero memory allocated on the stack:
$ cat stack.c
#include <stdio.h>
int main(int argc, char *argv[]) {
int i;
int j;
int k;
int *l;
int *m;
int *n;
printf("i: %d\n", i);
printf("j: %d\n", j);
printf("k: %d\n", k);
printf("l: %p\n", l);
printf("m: %p\n", m);
printf("n: %p\n", n);
return 0;
}
$ make stack
cc stack.c -o stack
$ ./stack
i: 0
j: 0
k: 32767
l: 0x400410
m: (nil)
n: 0x4005a0
l and n point to something in memory -- but those values are just garbage, and probably don't belong to the address space of the executable. If we store anything into those pointers, the program would probably die. It might corrupt unrelated structures, though, if they are mapped into the program's address space.
m at least is a NULL pointer -- if you tried to write to it, the program would certainly die on modern hardware.
None of those three pointers actually point to an integer yet. The memory for those integers doesn't exist. The memory for the pointers does exist -- and is initially filled with garbage values, in this case.
The Wikipedia article on L-values -- mostly too obtuse to fully recommend -- makes one point that represented a pretty significant hurdle for me when I was first learning C: In languages with assignable variables it becomes necessary to distinguish between the R-value (or contents) and the L-value (or location) of a variable.
For example, you can write:
int a;
a = 3;
This stores the integer value 3 into whatever memory was allocated to store the contents of variable a.
If you later write:
int b;
b = a;
This takes the value stored in the memory referenced by a and stores it into the memory location allocated for b.
The same operations with pointers might look like this:
int *ap;
ap=malloc(sizeof int);
*ap=3;
The first ap= assignment stores a memory location into the ap pointer. Now ap actually points at some memory. The second assignment, *ap=, stores a value into that memory location. It doesn't update the ap pointer at all; it reads the value stored in the variable named ap to find the memory location for the assignment.
When you later use the pointer, you can choose which of the two values associated with the pointer to use: either the actual contents of the pointer or the value pointed to by the pointer:
int *bp;
bp = ap; /* bp points to the same memory cell as ap */
int *bp;
bp = malloc(sizeof int);
*bp = *ap; /* bp points to new memory and we copy
the value pointed to by ap into the
memory pointed to by bp */
I found assembly far easier than C for years because I found the difference between foo = malloc(); and *foo = value; confusing. I hope I found what was confusing you and seriously hope I didn't make it worse.
Perhaps you misunderstand the difference between declaring 'int x' and 'int *x'. The first allocates storage for an int value; the second doesn't - it just allocates storage for the pointer.
If you were to "dynamically allocate" a variable, there would be no point in the dynamic allocation anyway (unless you then took its address, which would of course yield a pointer) - you may as well declare it statically. Think about how the code would look - why would you bother with:
int x = malloc(sizeof(int)); *x = 0;
When you can just do:
int x = 0;
even after reading quite a bit about the strict-aliasing rules I am still confused. As far as I have understood this, it is impossible to implement a sane memory allocator that follows these rules, because malloc can never reuse freed memory, as the memory could be used to store different types at each allocation.
Clearly this cannot be right. What am I missing? How do you implement an allocator (or a memory pool) that follows strict-aliasing?
Thanks.
Edit:
Let me clarify my question with a stupid simple example:
// s == 0 frees the pool
void *my_custom_allocator(size_t s) {
static void *pool = malloc(1000);
static int in_use = FALSE;
if( in_use || s > 1000 ) return NULL;
if( s == 0 ) {
in_use = FALSE;
return NULL;
}
in_use = TRUE;
return pool;
}
main() {
int *i = my_custom_allocator(sizeof(int));
//use int
my_custom_allocator(0);
float *f = my_custom_allocator(sizeof(float)); //not allowed...
}
I don't think you're right. Even the strictest of strict aliasing rules would only count when the memory is actually allocated for a purpose. Once an allocated block has been released back to the heap with free, there should be no references to it and it can be given out again by malloc.
And the void* returned by malloc is not subject to the strict aliasing rule since the standard explicitly states that a void pointer can be cast into any other sort of pointer (and back again). C99 section 7.20.3 states:
The pointer returned if the allocation succeeds is suitably aligned so that it may be assigned to a pointer to any type of object and then used to access such an object or an array of such objects in the space allocated (until the space is explicitly deallocated).
In terms of your update (the example) where you don't actually return the memory back to the heap, I think your confusion arises because allocated object are treated specially. If you refer to 6.5/6 of C99, you see:
The effective type of an object for an access to its stored value is the declared type of the object, if any (footnote 75: Allocated objects have no declared type).
Re-read that footnote, it's important.
If a value is stored into an object having no declared type through an lvalue having a type that is not a character type, then the type of the lvalue becomes the effective type of the object for that access and for subsequent accesses that do not modify the stored value.
If a value is copied into an object having no declared type using memcpy or memmove, or is copied as an array of character type, then the effective type of the modified object for that access and for subsequent accesses that do not modify the value is the effective type of the object from which the value is copied, if it has one.
For all other accesses to an object having no declared type, the effective type of the object is simply the type of the lvalue used for the access.
In other words, the allocated block contents will become the type of the data item that you put in there.
If you put a float in there, you should only access it as a float (or compatible type). If you put in an int, you should only process it as an int (or compatible type).
The one thing you shouldn't do is to put a specific type of variable into that memory and then try to treat it as a different type - one reason for this being that objects are allowed to have trap representations (which cause undefined behaviour) and these representations may occur due to treating the same object as different types.
So, if you were to store an int in there before the deallocation in your code, then reallocate it as a float pointer, you should not try to use the float until you've actually put one in there. Up until that point, the type of the allocated is not yet float.
I post this answer to test my understanding of strict aliasing:
Strict aliasing matters only when actual reads and writes happen. Just as using multiple members of different type of an union simultaneously is undefined behavior, the same applies to pointers as well: you cannot use pointers of different type to access the same memory for the same reason you cannot do it with an union.
If you consider only one of the pointers as live, then it's not a problem.
So if you write through an int* and read through an int*, it is OK.
If you write using an int* and read through an float*, it is bad.
If you write using an int* and later you write again using float*, then read it out using a float*, then it's OK.
In case of non-trivial allocators you have a large buffer, which you typically store it in a char*. Then you make some sort of pointer arithmetic to calculate the address you want to allocate and then dereference it through the allocator's header structs. It doesn't matter what pointers do you use to do the pointer arithmetic only the pointer you dereference the area through matters. Since in an allocator you always do that via the allocator's header struct, you won't trigger undefined behavior by that.
Standard C does not define any efficient means by which a user-written memory allocator can safely take a region of memory that has been used as one type and make it safely available as another. Structures in C are guaranteed not to trap representations--a guarantee which would have little purpose if it didn't make it safe to copy structures with fields containing Indeterminate Value.
The difficulty is that given a structure and function like:
struct someStruct {unsigned char count; unsigned char dat[7]; }
void useStruct(struct someStruct s); // Pass by value
it should be possible to invoke it like:
someStruct *p = malloc(sizeof *p);
p->count = 1;
p->dat[0] = 42;
useStruct(*p);
without having to write all of the fields of the allocated structure first.
Although malloc will guarantee that the allocation block it returns may
be used by any type, there is no way for user-written memory-management
functions to enable such reuse of storage without either clearing it in
bytewise fashion (using a loop or memset) or else using free() and malloc()
to recycle the storage.
Within the allocator itself, only refer to your memory buffers as (void *). when it is optimized, the strict-aliasing optimizations shouldn't be applied by the compiler (because that module has no idea what types are stored there). when that object gets linked into the rest of the system, it should be left well-enough alone.
Hope this helps!
I have a piece of code written by a very old school programmer :-) . it goes something like this
typedef struct ts_request
{
ts_request_buffer_header_def header;
char package[1];
} ts_request_def;
ts_request_def* request_buffer =
malloc(sizeof(ts_request_def) + (2 * 1024 * 1024));
the programmer basically is working on a buffer overflow concept. I know the code looks dodgy. so my questions are:
Does malloc always allocate contiguous block of memory? because in this code if the blocks are not contiguous, the code will fail big time
Doing free(request_buffer) , will it free all the bytes allocated by malloc i.e sizeof(ts_request_def) + (2 * 1024 * 1024),
or only the bytes of the size of the structure sizeof(ts_request_def)
Do you see any evident problems with this approach, I need to discuss this with my boss and would like to point out any loopholes with this approach
To answer your numbered points.
Yes.
All the bytes. Malloc/free doesn't know or care about the type of the object, just the size.
It is strictly speaking undefined behaviour, but a common trick supported by many implementations. See below for other alternatives.
The latest C standard, ISO/IEC 9899:1999 (informally C99), allows flexible array members.
An example of this would be:
int main(void)
{
struct { size_t x; char a[]; } *p;
p = malloc(sizeof *p + 100);
if (p)
{
/* You can now access up to p->a[99] safely */
}
}
This now standardized feature allowed you to avoid using the common, but non-standard, implementation extension that you describe in your question. Strictly speaking, using a non-flexible array member and accessing beyond its bounds is undefined behaviour, but many implementations document and encourage it.
Furthermore, gcc allows zero-length arrays as an extension. Zero-length arrays are illegal in standard C, but gcc introduced this feature before C99 gave us flexible array members.
In a response to a comment, I will explain why the snippet below is technically undefined behaviour. Section numbers I quote refer to C99 (ISO/IEC 9899:1999)
struct {
char arr[1];
} *x;
x = malloc(sizeof *x + 1024);
x->arr[23] = 42;
Firstly, 6.5.2.1#2 shows a[i] is identical to (*((a)+(i))), so x->arr[23] is equivalent to (*((x->arr)+(23))). Now, 6.5.6#8 (on the addition of a pointer and an integer) says:
"If both the pointer operand and the result point to elements of the same array object, or one past the last element of the array object, the evaluation shall not produce an overflow; otherwise, the behavior is undefined."
For this reason, because x->arr[23] is not within the array, the behaviour is undefined. You might still think that it's okay because the malloc() implies the array has now been extended, but this is not strictly the case. Informative Annex J.2 (which lists examples of undefined behaviour) provides further clarification with an example:
An array subscript is out of range, even if an object is apparently accessible with the
given subscript (as in the lvalue expression a[1][7] given the declaration int
a[4][5]) (6.5.6).
3 - That's a pretty common C trick to allocate a dynamic array at the end of a struct. The alternative would be to put a pointer into the struct and then allocate the array separately, and not forgetting to free it too. That the size is fixed to 2mb seems a bit unusual though.
This is a standard C trick, and isn't more dangerous that any other buffer.
If you are trying to show to your boss that you are smarter than "very old school programmer", this code isn't a case for you. Old school not necessarily bad. Seems the "old school" guy knows enough about memory management ;)
1) Yes it does, or malloc will fail if there isn't a large enough contiguous block available. (A failure with malloc will return a NULL pointer)
2) Yes it will. The internal memory allocation will keep track of the amount of memory allocated with that pointer value and free all of it.
3)It's a bit of a language hack, and a bit dubious about it's use. It's still subject to buffer overflows as well, just may take attackers slightly longer to find a payload that will cause it. The cost of the 'protection' is also pretty hefty (do you really need >2mb per request buffer?). It's also very ugly, although your boss may not appreciate that argument :)
I don't think the existing answers quite get to the essence of this issue. You say the old-school programmer is doing something like this;
typedef struct ts_request
{
ts_request_buffer_header_def header;
char package[1];
} ts_request_def;
ts_request_buffer_def* request_buffer =
malloc(sizeof(ts_request_def) + (2 * 1024 * 1024));
I think it's unlikely he's doing exactly that, because if that's what he wanted to do he could do it with simplified equivalent code that doesn't need any tricks;
typedef struct ts_request
{
ts_request_buffer_header_def header;
char package[2*1024*1024 + 1];
} ts_request_def;
ts_request_buffer_def* request_buffer =
malloc(sizeof(ts_request_def));
I'll bet that what he's really doing is something like this;
typedef struct ts_request
{
ts_request_buffer_header_def header;
char package[1]; // effectively package[x]
} ts_request_def;
ts_request_buffer_def* request_buffer =
malloc( sizeof(ts_request_def) + x );
What he wants to achieve is allocation of a request with a variable package size x. It is of course illegal to declare the array's size with a variable, so he is getting around this with a trick. It looks as if he knows what he's doing to me, the trick is well towards the respectable and practical end of the C trickery scale.
As for #3, without more code it's hard to answer. I don't see anything wrong with it, unless its happening a lot. I mean, you don't want to allocate 2mb chunks of memory all the time. You also don't want to do it needlessly, e.g. if you only ever use 2k.
The fact that you don't like it for some reason isn't sufficient to object to it, or justify completely re-writing it. I would look at the usage closely, try to understand what the original programmer was thinking, look closely for buffer overflows (as workmad3 pointed out) in the code that uses this memory.
There are lots of common mistakes that you may find. For example, does the code check to make sure malloc() succeeded?
The exploit (question 3) is really up to the interface towards this structure of yours. In context this allocation might make sense, and without further information it is impossible to say if it's secure or not.
But if you mean problems with allocating memory bigger than the structure, this is by no means a bad C design (I wouldn't even say it's THAT old school... ;) )
Just a final note here - the point with having a char[1] is that the terminating NULL will always be in the declared struct, meaning there can be 2 * 1024 * 1024 characters in the buffer, and you don't have to account for the NULL by a "+1". Might look like a small feat, but I just wanted to point out.
I've seen and used this pattern frequently.
Its benefit is to simplify memory management and thus avoid risk of memory leaks. All it takes is to free the malloc'ed block. With a secondary buffer, you'll need two free. However one should define and use a destructor function to encapsulate this operation so you can always change its behavior, like switching to secondary buffer or add additional operations to be performed when deleting the structure.
Access to array elements is also slightly more efficient but that is less and less significant with modern computers.
The code will also correctly work if memory alignment changes in the structure with different compilers as it is quite frequent.
The only potential problem I see is if the compiler permutes the order of storage of the member variables because this trick requires that the package field remains last in the storage. I don't know if the C standard prohibits permutation.
Note also that the size of the allocated buffer will most probably be bigger than required, at least by one byte with the additional padding bytes if any.
Yes. malloc returns only a single pointer - how could it possibly tell a requester that it had allocated multiple discontiguous blocks to satisfy a request?
Would like to add that not is it common but I might also called it a standard practice because Windows API is full of such use.
Check the very common BITMAP header structure for example.
http://msdn.microsoft.com/en-us/library/aa921550.aspx
The last RBG quad is an array of 1 size, which depends on exactly this technique.
This common C trick is also explained in this StackOverflow question (Can someone explain this definition of the dirent struct in solaris?).
In response to your third question.
free always releases all the memory allocated at a single shot.
int* i = (int*) malloc(1024*2);
free(i+1024); // gives error because the pointer 'i' is offset
free(i); // releases all the 2KB memory
The answer to question 1 and 2 is Yes
About ugliness (ie question 3) what is the programmer trying to do with that allocated memory?
the thing to realize here is that malloc does not see the calculation being made in this
malloc(sizeof(ts_request_def) + (2 * 1024 * 1024));
Its the same as
int sz = sizeof(ts_request_def) + (2 * 1024 * 1024);
malloc(sz);
YOu might think that its allocating 2 chunks of memory , and in yr mind they are "the struct", "some buffers". But malloc doesnt see that at all.