Context:
I was reviewing some code that receives data from an IO descriptor into a character buffer, does some control on it and then use part of the received buffer to populate a struct, and suddenly wondered whether a strict aliasing rule violation could be involved.
Here is a simplified version
#define BFSZ 1024
struct Elt {
int id;
...
};
unsigned char buffer[BFSZ];
int sz = read(fd, buffer, sizeof(buffer)); // correctness control omitted for brievety
// search the beginning of struct data in the buffer, and process crc control
unsigned char *addr = locate_and_valid(buffer, sz);
struct Elt elt;
memcpy(&elt, addr, sizeof(elt)); // populates the struct
// and use it
int id = elt.id;
...
So far, so good. Provide the buffer did contain a valid representation of the struct - say it has been produced on same platform, so without endianness or padding problem - the memcpy call has populated the struct and it can safely be used.
Problem:
If the struct is dynamically allocated, it has no declared type. Let us replace last lines with:
struct Elt *elt = malloc(sizeof(struct Element)); // no declared type here
memcpy(elt, addr, sizeof(*elt)); // populates the newly allocated memory and copies the effective type
// and use it
int id = elt->id; // strict aliasing rule violation?
...
Draft n1570 for C language says in 6.5 Expressions §6
The effective type of an object for an access to its stored value is the declared type of the
object, if any.87) If a value is stored into an object having no declared type through an
lvalue having a type that is not a character type, then the type of the lvalue becomes the
effective type of the object for that access and for subsequent accesses that do not modify
the stored value. If a value is copied into an object having no declared type using
memcpy or memmove, or is copied as an array of character type, then the effective type
of the modified object for that access and for subsequent accesses that do not modify the
value is the effective type of the object from which the value is copied, if it has one.
buffer does have an effective type and even a declared type: it is an array of unsigned char. That is the reason why the code uses a memcpy instead of a mere aliasing like:
struct Elt *elt = (struct Elt *) addr;
which would indeed be a strict aliasing rule violation (and could additionaly come with alignment problems). But if memcpy has given an effective type of an unsigned char array to the zone pointed by elt, everything is lost.
Question:
Does memcpy from an array of character type to a object with no declared type give an effective type of array of character?
Disclaimer:
I know that it works without a warning with all common compilers. I just want to know whether my understanding of standard is correct
In order to better show my problem, let us considere a different structure Elt2 with sizeof(struct Elt2)<= sizeof(struct Elt), and
struct Elt2 actual_elt2 = {...};
For static or automatic storage, I cannot reuse object memory:
struct Elt elt;
struct Elt2 *elt2 = &elt;
memcpy(elt2, &actual_elt2, sizeof(*elt2));
elt2->member = ... // strict aliasing violation!
While it is fine for dynamic one (question about it there):
struct Elt *elt = malloc(sizeof(*elt));
// use elt
...
struct Elt2 *elt2 = elt;
memcpy(elt2, &actual_elt2, sizeof(*elt2));
// ok, memory now have struct Elt2 effective type, and using elt would violate strict aliasing rule
elt2->member = ...; // fine
elt->id = ...; // strict aliasing rule violation!
What could make copying from a char array different?
The code is fine, no strict aliasing violation. The pointed-at data has an effective type, so the bold cited text does not apply. What applies here is the part you left out, last sentence of 6.5/6:
For all other accesses to an object having no declared type, the effective type of the object is simply the type of the lvalue used for the access.
So the effective type of the pointed-at object becomes struct Elt. The returned pointer of malloc does indeed point to an object with no delcared type, but as soon as you point at it, the effective type becomes that of the struct pointer. Otherwise C programs would not be able to use malloc at all.
What makes the code safe is also that you are copying data into that struct. Had you instead just assigned a struct Elt* to point at the same memory location as addr, then you would have a strict aliasing violation and UB.
Lundin's answer is correct; what you are doing is fine (so long as the data is aligned and of same endianness).
I want to note this is not so much a result of the C language specification as it is a result of how the hardware works. As such, there's not a single authoritative answer. The C language specification defines how the language works, not how the language is compiled or implemented on different systems.
Here is an interesting article about memory alignment and strict aliasing on a SPARC versus Intel processor (notice the exact same C code performs differently, and gives errors on one platform while working on another):
https://askldjd.com/2009/12/07/memory-alignment-problems/
Fundamentally, two identical structs, on the same system with the same endian and memory alignment, must work via memcpy. If it didn't then the computer wouldn't be able to do much of anything.
Finally, the following question explains more about memory alignment on systems, and the answer by joshperry should help explain why this is a hardware issue, not a language issue:
Purpose of memory alignment
Related
I've encountered a similiar problem as described in another thread (perf_event_open - how to monitoring multiple events). I was able to solve it and the code is working, but I want to understand why this part actually works and how this is not a violation of any kind:
char buf[4096];
struct read_format* rf = (struct read_format*) buf;
struct read_format is defined as followed:
struct read_format {
uint64_t nr;
struct {
uint64_t value;
uint64_t id;
} values[/*2*/]; };
How does the compiler know to which value uint64_t nr should be initialized? Or how to initialize the inner struct right?
This code is incorrect in Standard C:
char buf[4096];
read(fd1, buf, 4096); // Assume error handling, omitted for brevity
struct read_format* rf = (struct read_format*) buf;
printf("%llu\n", rf->nr);
There are two issues -- and these are distinct issues which should not be conflated -- :
buf might not be correctly aligned for struct read_format. If it isn't, the behaviour is undefined.
Accessing rf->nr violates the strict aliasing rule and the behaviour is undefined. An object with declared type char cannot be read of written by an expression of type . unsigned long long. Note that the converse is not true.
Why does it appear to work? Well, "undefined" does not mean "must explode". It means the C Standard no longer specifies the program's behaviour. This sort of code is somewhat common in real code bases. The major compiler vendors -- for now -- include logic so that this code will behave as "expected", otherwise too many people would complain.
The "expected" behaviour is that accessing *rf should behave as if there exists a struct read_format object at the address, and the bytes of that object are the same as the bytes of buf . Similar to if the two were in a union.
The code could be made compliant with a union:
union
{
char buf[4096];
struct read_format rf;
} u;
read(fd1, u.buf, sizeof u.buf);
printf("%llu\n", u.rf->nr);
The strict aliasing rule is "disabled" for union members accessed by name; and this also addresses the alignment problem since the union will be aligned for all members.
It's up to you whether to be compliant, or trust that compilers will continue put practicality ahead of maximal optimization within the constraints permitted by the Standard.
It doesn't The buffer is zero-initialized and the struct pointer is initialized with a pointer to the buffer.
It looks completely whack; however it really isn't. The read function is going to read as many structures into the buffer as fit.
The outer structure is variable-length. The advance loop looks like this:
struct read_format *current = rf;
if (readstructs(..., ¤t, 4096)) {
for (;current;current=current->nr?((struct read_format *)((char *)current + current->nr)):NULL) {
}
}
These things appear in system-level OS calls to decrease the complexity of copying memory across security boundaries. The read side is easy and well-taught. The writer performs the operations necessary in filling the buffer to ensure this simple reader does not violate any system-level constraints. The code will work despite looking like it violates types left and right because the writer has set it up to work. In particular, the pointer will be aligned.
I've seen a similar method used in old file formats. Unfortunately that only follows the rules of the platform that wrote it (usually something ancient and far more permissive than a modern system) and leads to having to write a byte-at-a-time reader because the host doing the reading doesn't correspond.
Consider the following struct:
struct s {
int a, b;
};
Typically1, this struct will have size 8 and alignment 4.
What if we create two struct s objects (more precisely, we write into allocated storage two such objects), with the second object overlapping the first?
char *storage = malloc(3 * sizeof(struct s));
struct s *o1 = (struct s *)storage; // offset 0
struct s *o2 = (struct s *)(storage + alignof(struct s)); // offset 4
// now, o2 points half way into o1
*o1 = (struct s){1, 2};
*o2 = (struct s){3, 4};
printf("o2.a=%d\n", o2->a);
printf("o2.b=%d\n", o2->b);
printf("o1.a=%d\n", o1->a);
printf("o1.b=%d\n", o1->b);
Is anything about this program undefined behavior? If so, where does it become undefined? If it is not UB, is it guaranteed to always print the following:
o2.a=3
o2.b=4
o1.a=1
o1.b=3
In particular, I want to know what happens to the object pointed to by o1 when o2, which overlaps it, is written. Is it still allowed to access the unclobbered part (o1->a)? Is accessing the clobbered part o1->b simply the same as accessing o2->a?
How does effective type apply here? The rules are clear enough when you are talking about non-overlapping objects and pointers that point to the same location as the last store, but when you start talking about the effective type of portions of objects or overlapping objects it is less clear.
Would anything change if the the second write was of a different type? If the members were say int and short rather than two ints?
Here's a godbolt if you want to play with it there.
1 This answer applies to platforms where this isn't the case too: e.g., some might have size 4 and alignment 2. On a platform where the size and alignment were the same, this question wouldn't apply since aligned, overlapping objects would be impossible, but I'm not sure if there is any platform like that.
Basically this is all grey area in the standard; the strict aliasing rule specifies basic cases and leaves the reader (and compiler vendors) to fill in the details.
There have been efforts to write a better rule but so far they haven't resulted in any normative text and I'm not sure what the status of this is for C2x.
As mentioned in my answer to your previous question, the most common interpretation is that p->q means (*p).q and the effective type applies to all of *p, even though we then go on to apply .q .
Under this interpretation, printf("o1.a=%d\n", o1->a); would cause undefined behaviour as the effective type of the location *o1 is not s (since part of it has been overwritten).
The rationale for this interpretation can be seen in a function like:
void f(s* s1, s* s2)
{
s2->a = 5;
s1->b = 6;
printf("%d\n", s2->a);
}
With this interpretation the last line could be optimised to puts("5"); , but without it, the compiler would have to consider that the function call may have been f(o1, o2); and therefore lose all benefits that are purportedly provided by the strict aliasing rule.
A similar argument applies to two unrelated struct types that both happen to have an int member at different offset.
Why do we have pointer types? eg
int *ptr;
I know its for type safety, eg to dereference 'ptr', the compiler needs to know that its dereferencing the ptr to type int, not to char or long, etc, but as others outlined here Why to specify a pointer type? , its also because "we should know how many bytes to read. Dereferencing a char pointer would imply taking one byte from memory while for int it could be 4 bytes." That makes sense.
But what if I have something like this:
typedef struct _IP_ADAPTER_INFO {
struct _IP_ADAPTER_INFO* Next;
DWORD ComboIndex;
char AdapterName[MAX_ADAPTER_NAME_LENGTH + 4];
char Description[MAX_ADAPTER_DESCRIPTION_LENGTH + 4];
UINT AddressLength;
BYTE Address[MAX_ADAPTER_ADDRESS_LENGTH];
DWORD Index;
UINT Type;
UINT DhcpEnabled;
PIP_ADDR_STRING CurrentIpAddress;
IP_ADDR_STRING IpAddressList;
IP_ADDR_STRING GatewayList;
IP_ADDR_STRING DhcpServer;
BOOL HaveWins;
IP_ADDR_STRING PrimaryWinsServer;
IP_ADDR_STRING SecondaryWinsServer;
time_t LeaseObtained;
time_t LeaseExpires;
} IP_ADAPTER_INFO, *PIP_ADAPTER_INFO;
PIP_ADAPTER_INFO pAdapterInfo = (IP_ADAPTER_INFO *)malloc(sizeof(IP_ADAPTER_INFO));
What would be the point of declaring the type PIP_ADAPTER_INFO here? After all, unlike the previous example, we've already allocated enough memory for the pointer to point at (using malloc), so isn't defining the type here redundant? We will be reading as much data from memory as there has been allocated.
Also, side note: Is there any difference between the following 4 declarations or is there a best practice?
PIP_ADAPTER_INFO pAdapterInfo = (IP_ADAPTER_INFO *)malloc(sizeof(IP_ADAPTER_INFO));
or
PIP_ADAPTER_INFO pAdapterInfo = (PIP_ADAPTER_INFO)malloc(sizeof(IP_ADAPTER_INFO));
or
IP_ADAPTER_INFO *pAdapterInfo = (IP_ADAPTER_INFO *)malloc(sizeof(IP_ADAPTER_INFO));
or
IP_ADAPTER_INFO *pAdapterInfo = (PIP_ADAPTER_INFO)malloc(sizeof(IP_ADAPTER_INFO));
You’re kind of asking two different questions here - why have different pointer types, and why hide pointers behind typedefs?
The primary reason for distinct pointer types comes from pointer arithmetic - if p points to an object of type T, then the expression p + 1 points to the next object of that type. If p points to an 4-byte int, then p + 1 points to the next int. If p points to a 128-byte struct, then p + 1 points to the next 128-byte struct, and so on. Pointers are abstractions of memory addresses with additional type semantics.
As for hiding pointers behind typedefs...
A number of us (including myself) consider hiding pointers behind typedefs to be bad style if the user of the type still has to be aware of the type’s “pointer-ness” (i.e., if you ever have to dereference it, or if you ever assign the result of malloc/calloc/realloc to it, etc.). If you’re trying to abstract away the “pointer-ness” of something, you need to do it in more than just the declaration - you need to provide a full API that hides all the pointer operations as well.
As for your last question, best practice in C is to not cast the result of malloc. Best practice in C++ is to not use malloc at all.
I think this is more a question of type definition style than of dynamic memory allocation.
Old-school C practice is to describe structs by their tags. You say
struct foo {
...
};
and then
struct foo foovar;
or
struct foo *foopointer = malloc(sizeof(struct foo));
But a lot of people don't like having to type that keyword struct all the time. (I guess I can't fault then; C has always favored terseness, sometimes seemingly just to reduce typing.) So a form using typedef became quite popular (and it either influenced, or was influenced by, C++):
typedef struct {
...
} Foo;
and then
Foo foovar;
or
Foo *foopointer = malloc(sizeof(Foo));
But then, for reasons that are less clear, it became popular to throw the pointerness into the typedef, too, like this:
typedef struct {
...
} Foo, *Foop;
Foop foopointer = malloc(sizeof(*Foop));
But this is all a matter of style and personal preference, in the service of what someone imagines to be clarity or convenience or usefulness. (But of course opinions on clarity and convenience, like opinions on style, can legitimately vary.) I've seen the pointer typedefs disparaged as being a misleading or Microsoftian practice, but I'm not sure I can fault them right now.
You also asked about the casts, and we could also dissect various options for the sizeof call as the argument to malloc.
It doesn't really matter whether you say
Foop foopointer = (Foop)malloc(sizeof(*Foop));
or
Foop foopointer = (Foo *)malloc(sizeof(*Foop));
The first one may be clearer, in that you don't have to go back and check that Foop and Foo * are the same thing. But they're both poor practice in C, and in at least some circles they've been deprecated since the 1990's. Those casts are are considered distracting and unnecessary in straight C -- although of course they're necessary in C++, or I suppose if you're using a C++ compiler to compile C code. (If you were writing straight C++ code, of course, you'd typically use new instead of malloc.)
But then what should you put in the sizeof()? Which is better,
Foop foopointer = malloc(sizeof(*Foop));
or
Foop foopointer = malloc(sizeof(Foo));
Again, the first one can be easier to read, since you don't have to go back and check that Foop and Foo * are the same thing. But by the same token, there's a third form that can be even clearer:
Foop foopointer = malloc(sizeof(*foopointer));
Now you know that, whatever type foopointer points at, you're allocating the right amount of space for it. This idiom works best, though, if it's maximally clear that foopiinter is in fact a pointer that points at some type, meaning that the variants
Foo *foopointer = malloc(sizeof(*foopointer));
or even
struct foo *foopointer = malloc(sizeof(*foopointer));
can be considered clearer still -- and this may be one of the reasons people consider the pointer typedef to be less than perfectly useful.
Bottom line, if you're still with me: If you don't find PIP_ADAPTER_INFO useful, don't use it -- use IP_ADAPTER_INFO (along with explicit *'s when you need them) instead. Someone thought PIP_ADAPTER_INFO might be useful, which is why it's there, but the arguments in favor of its use aren't too compelling.
What is the point of “pointer types” when you dynamically allocate memory?
At least for the example you show there is none.
So the follow up question would be if there were situations where typedefing a pointer made sense.
And the answer is: Yes.
It definitely makes sense if one is in the need of an opaque data type.
A nice example is the pthread_t type which defines a handle to a POSIX thread.
Depending on the implementation it is defined as
typedef struct bla pthread_t;
typedef struct foo * pthread_t;
typedef long pthread_t;
and with this abstracts away the kind of implementation, as it is of no interest to the user, which probably is not the intention with the struct you show in your question.
Why do we have pointer types?
To accommodate architectures where the size and encoding may differ for various types. C ports well to many platforms, even novel ones.
It is not unusual today that pointers to functions have a different size than pointers to objects. An object pointer coverts to a void *, yet a function pointer may not.
A pointer to char need not be the same size as a pointer to an int or union or struct. This is uncommon today. The spec details follow (my emphasis):
A pointer to void shall have the same representation and alignment requirements as a
pointer to a character type. Similarly, pointers to qualified or unqualified versions of compatible types shall have the same representation and alignment requirements. All
pointers to structure types shall have the same representation and alignment requirements
as each other. All pointers to union types shall have the same representation and
alignment requirements as each other. Pointers to other types need not have the same
representation or alignment requirements. C11dr §6.2.5 28
I have a large buffer that represents a 3D model file loaded and decompressed from a HDD, the file has a header and some vertex, index and subset data after it. At first i thought i could calculate byte offsets where each vertex/index/subset data begins and simply cast it to a compatible pointer type and use it, but this would break strict-aliasing rules. So an solution would be to memcpy bytes to separate vertex/index/subset data arrays (each array of different type ofc)?
unsigned char *buf = NULL;
size_t offset = 0;
/* ... */
/* now #buf points to data immediately following the file header */
/* copy mesh subsets list */
memcpy(out->subsets, buf, sizeof(*out->subsets) * header.num_subsets);
/* copy vertex indices list */
offset = sizeof(*out->subsets) * header.num_subsets;
memcpy(out->indices, &buf[offset], sizeof(*out->indices) *
header.num_indices);
/* copy mesh vertices list */
offset += sizeof(*out->indices) * header.num_indices;
memcpy(out->vertices, &buf[offset], sizeof(*out->vertices) *
header.num_vertices);
You are attacking the strict aliasing rules from the wrong angle. Casting an array of char to your structure would in fact be UB. This is not only because of aliasing but also because alignment properties can be different. Don't do that.
You'd have to do it the other way round: declare your structure of the real type that you want to have, and then use a void* or char* pointer to that structure to read or copy your data into it.
This is always guaranteed to work:
character types are exempted from the strict aliasing rules
passing a pointer of an object to a function (memcpy or other) always ensures that the compiler can't make any assumptions about the state of that object after the call, so he has to reload the entire object.
Edit: Perhaps some the confusion comes from the strange gcc warning about "aliasing rules". This is only one facet of the problems that come from type-punning through pointer casts. Generally accessing an object through a pointer of a wrong type other than character types can have undefined behavior. Aliasing is only one of several things that can go wrong with that. Just avoid it.
The problem with casting your char* to a pointer to your structure is not strict aliasing rules: char types are exempted from strict aliasing rules. That is, you may read data written as char data as any other type, and you may conversely read any data as char data.
The problem with the cast is alignment. Unless you have obtained your char* directly from a memory allocation function (which is guaranteed to produce a pointer sufficiently aligned for any data), you risk misalignment, which can crash your program. Using memcpy() works around this. However, if you can be certain that your char* is perfectly aligned, there is no need for the copy.
To avoid confusion, this is perfectly legal code:
typedef struct Foo {
...
} Foo;
void bar() {
char* buffer = malloc(sizeof(Foo));
fillBuffer(buffer);
Foo* header = (Foo*)buffer; //Ok, buffer is a perfectly aligned pointer.
readHeader(header); //Ok, reading data written as char data does not violate strict aliasing rules.
}
even after reading quite a bit about the strict-aliasing rules I am still confused. As far as I have understood this, it is impossible to implement a sane memory allocator that follows these rules, because malloc can never reuse freed memory, as the memory could be used to store different types at each allocation.
Clearly this cannot be right. What am I missing? How do you implement an allocator (or a memory pool) that follows strict-aliasing?
Thanks.
Edit:
Let me clarify my question with a stupid simple example:
// s == 0 frees the pool
void *my_custom_allocator(size_t s) {
static void *pool = malloc(1000);
static int in_use = FALSE;
if( in_use || s > 1000 ) return NULL;
if( s == 0 ) {
in_use = FALSE;
return NULL;
}
in_use = TRUE;
return pool;
}
main() {
int *i = my_custom_allocator(sizeof(int));
//use int
my_custom_allocator(0);
float *f = my_custom_allocator(sizeof(float)); //not allowed...
}
I don't think you're right. Even the strictest of strict aliasing rules would only count when the memory is actually allocated for a purpose. Once an allocated block has been released back to the heap with free, there should be no references to it and it can be given out again by malloc.
And the void* returned by malloc is not subject to the strict aliasing rule since the standard explicitly states that a void pointer can be cast into any other sort of pointer (and back again). C99 section 7.20.3 states:
The pointer returned if the allocation succeeds is suitably aligned so that it may be assigned to a pointer to any type of object and then used to access such an object or an array of such objects in the space allocated (until the space is explicitly deallocated).
In terms of your update (the example) where you don't actually return the memory back to the heap, I think your confusion arises because allocated object are treated specially. If you refer to 6.5/6 of C99, you see:
The effective type of an object for an access to its stored value is the declared type of the object, if any (footnote 75: Allocated objects have no declared type).
Re-read that footnote, it's important.
If a value is stored into an object having no declared type through an lvalue having a type that is not a character type, then the type of the lvalue becomes the effective type of the object for that access and for subsequent accesses that do not modify the stored value.
If a value is copied into an object having no declared type using memcpy or memmove, or is copied as an array of character type, then the effective type of the modified object for that access and for subsequent accesses that do not modify the value is the effective type of the object from which the value is copied, if it has one.
For all other accesses to an object having no declared type, the effective type of the object is simply the type of the lvalue used for the access.
In other words, the allocated block contents will become the type of the data item that you put in there.
If you put a float in there, you should only access it as a float (or compatible type). If you put in an int, you should only process it as an int (or compatible type).
The one thing you shouldn't do is to put a specific type of variable into that memory and then try to treat it as a different type - one reason for this being that objects are allowed to have trap representations (which cause undefined behaviour) and these representations may occur due to treating the same object as different types.
So, if you were to store an int in there before the deallocation in your code, then reallocate it as a float pointer, you should not try to use the float until you've actually put one in there. Up until that point, the type of the allocated is not yet float.
I post this answer to test my understanding of strict aliasing:
Strict aliasing matters only when actual reads and writes happen. Just as using multiple members of different type of an union simultaneously is undefined behavior, the same applies to pointers as well: you cannot use pointers of different type to access the same memory for the same reason you cannot do it with an union.
If you consider only one of the pointers as live, then it's not a problem.
So if you write through an int* and read through an int*, it is OK.
If you write using an int* and read through an float*, it is bad.
If you write using an int* and later you write again using float*, then read it out using a float*, then it's OK.
In case of non-trivial allocators you have a large buffer, which you typically store it in a char*. Then you make some sort of pointer arithmetic to calculate the address you want to allocate and then dereference it through the allocator's header structs. It doesn't matter what pointers do you use to do the pointer arithmetic only the pointer you dereference the area through matters. Since in an allocator you always do that via the allocator's header struct, you won't trigger undefined behavior by that.
Standard C does not define any efficient means by which a user-written memory allocator can safely take a region of memory that has been used as one type and make it safely available as another. Structures in C are guaranteed not to trap representations--a guarantee which would have little purpose if it didn't make it safe to copy structures with fields containing Indeterminate Value.
The difficulty is that given a structure and function like:
struct someStruct {unsigned char count; unsigned char dat[7]; }
void useStruct(struct someStruct s); // Pass by value
it should be possible to invoke it like:
someStruct *p = malloc(sizeof *p);
p->count = 1;
p->dat[0] = 42;
useStruct(*p);
without having to write all of the fields of the allocated structure first.
Although malloc will guarantee that the allocation block it returns may
be used by any type, there is no way for user-written memory-management
functions to enable such reuse of storage without either clearing it in
bytewise fashion (using a loop or memset) or else using free() and malloc()
to recycle the storage.
Within the allocator itself, only refer to your memory buffers as (void *). when it is optimized, the strict-aliasing optimizations shouldn't be applied by the compiler (because that module has no idea what types are stored there). when that object gets linked into the rest of the system, it should be left well-enough alone.
Hope this helps!