The C99 standard states that:
When two pointers are subtracted, both shall point to elements of the same array object, or one past the last element of the array object
Consider the following code:
struct test {
int x[5];
char something;
short y[5];
};
...
struct test s = { ... };
char *p = (char *) s.x;
char *q = (char *) s.y;
printf("%td\n", q - p);
This obviously breaks the above rule, since the p and q pointers are pointing to different "array objects", and, according to the rule, the q - p difference is undefined.
But in practice, why should such a thing ever result in undefined behaviour? After all, the struct members are laid out sequentially (just as array elements are), with any potential padding between the members. True, the amount of padding will vary across implementations and that would affect the outcome of the calculations, but why should that result be "undefined"?
My question is, can we suppose that the standard is just "ignorant" of this issue, or is there a good reason for not broadening this rule? Couldn't the above rule be rephrased to "both shall point to elements of the same array object or members of the same struct"?
My only suspicion are segmented memory architectures where the members might end up in different segments. Is that the case?
I also suspect that this is the reason why GCC defines its own __builtin_offsetof, in order to have a "standards compliant" definition of the offsetof macro.
EDIT:
As already pointed out, arithmetic on void pointers is not allowed by the standard. It is a GNU extension that fires a warning only when GCC is passed -std=c99 -pedantic. I'm replacing the void * pointers with char * pointers.
Subtraction and relational operators (on type char*) between addresses of member of the same struct are well defined.
Any object can be treated as an array of unsigned char.
Quoting N1570 6.2.6.1 paragraph 4:
Values stored in non-bit-field objects of any other object type
consist of n × CHAR_BIT bits, where n is the size of an object of that
type, in bytes. The value may be copied into an object of type
unsigned char [ n ] (e.g., by memcpy); the resulting set of bytes is
called the object representation of the value.
...
My only suspicion are segmented memory architectures where the members
might end up in different segments. Is that the case?
No. For a system with a segmented memory architecture, normally the compiler will impose a restriction that each object must fit into a single segment. Or it can permit objects that occupy multiple segments, but it still has to ensure that pointer arithmetic and comparisons work correctly.
Pointer arithmetic requires that the two pointers being added or subtracted to be part of the same object because it doesn't make sense otherwise.
The quoted section of standard specifically refers to two unrelated objects such as int a[b]; and int b[5]. The pointer arithmetic requires to know the type of the object that the pointers pointing to (I am sure you are aware of this already).
i.e.
int a[5];
int *p = &a[1]+1;
Here p is calculated by knowing the that the &a[1] refers to an int object and hence incremented to 4 bytes (assuming sizeof(int) is 4).
Coming to the struct example, I don't think it can possibly be defined in a way to make pointer arithmetic between struct members legal.
Let's take the example,
struct test {
int x[5];
char something;
short y[5];
};
Pointer arithmatic is not allowed with void pointers by C standard (Compiling with gcc -Wall -pedantic test.c would catch that). I think you are using gcc which assumes void* is similar to char* and allows it.
So,
printf("%zu\n", q - p);
is equivalent to
printf("%zu", (char*)q - (char*)p);
as pointer arithmetic is well defined if the pointers point to within the same object and are character pointers (char* or unsigned char*).
Using correct types, it would be:
struct test s = { ... };
int *p = s.x;
short *q = s.y;
printf("%td\n", q - p);
Now, how can q-p be performed? based on sizeof(int) or sizeof(short) ? How can the size of char something; that's in the middle of these two arrays be calculated?
That should explain it's not possible to perform pointer arithmetic on objects of different types.
Even if all members are of same type (thus no type issue as stated above), then it's better to use the standard macro offsetof (from <stddef.h>) to get the difference between struct members which has the similar effect as pointer arithmetic between members:
printf("%zu\n", offsetof(struct test, y) - offsetof(struct test, x));
So I see no necessity to define pointer arithmetic between struct members by the C standard.
Yes, you are allowed to perform pointer arithmetric on structure bytes:
N1570 - 6.3.2.3 Pointers p7:
... When a pointer to an object is converted to a pointer to a character type,
the result points to the lowest addressed byte of the object. Successive increments of the
result, up to the size of the object, yield pointers to the remaining bytes of the object.
This means that for the programmer, bytes of the stucture shall be seen as a continuous area, regardless how it may have been implemented in the hardware.
Not with void* pointers though, that is non-standard compiler extension. As mentioned on paragraph from the standard, it applies only to character type pointers.
Edit:
As mafso pointed out in comments, above is only true as long as type of substraction result ptrdiff_t, has enough range for the result. Since range of size_t can be larger than ptrdiff_t, and if structure is big enough, it's possible that addresses are too far apart.
Because of this it's preferable to use offsetof macro on structure members and calculate result from those.
I believe the answer to this question is simpler than it appears, the OP asks:
but why should that result be "undefined"?
Well, let's see that the definition of undefined behavior is in the draft C99 standard section 3.4.3:
behavior, upon use of a nonportable or erroneous program construct or
of erroneous data, for which this International Standard imposes no
requirements
it is simply behavior for which the standard does not impose a requirement, which perfectly fits this situation, the results are going to vary depending on the architecture and attempting to specify the results would have probably been difficult if not impossible in a portable manner. This leaves the question, why would they choose undefined behavior as opposed to let's say implementation of unspecified behavior?
Most likely it was made undefined behavior to limit the number of ways an invalid pointer could be created, this is consistent with the fact that we are provided with offsetof to remove the one potential need for pointer subtraction of unrelated objects.
Although the standard does not really define the term invalid pointer, we get a good description in Rationale for International Standard—Programming Languages—C which in section 6.3.2.3 Pointers says (emphasis mine):
Implicit in the Standard is the notion of invalid pointers. In
discussing pointers, the Standard typically refers to “a pointer to an
object” or “a pointer to a function” or “a null pointer.” A special
case in address arithmetic allows for a pointer to just past the end
of an array. Any other pointer is invalid.
The C99 rationale further adds:
Regardless how an invalid pointer is created, any use of it yields
undefined behavior. Even assignment, comparison with a null pointer
constant, or comparison with itself, might on some systems result in
an exception.
This strongly suggests to us that a pointer to padding would be an invalid pointer, although it is difficult to prove that padding is not an object, the definition of object says:
region of data storage in the execution environment, the contents of
which can represent values
and notes:
When referenced, an object may be interpreted as having a particular
type; see 6.3.2.1.
I don't see how we can reason about the type or the value of padding between elements of a struct and therefore they are not objects or at least is strongly indicates padding is not meant to be considered an object.
I should point out the following:
from the C99 standard, section 6.7.2.1:
Within a structure object, the non-bit-field members and the units in which bit-fields
reside have addresses that increase in the order in which they are declared. A pointer to a
structure object, suitably converted, points to its initial member (or if that member is a
bit-field, then to the unit in which it resides), and vice versa. There may be unnamed
padding within a structure object, but not at its beginning.
It isn't so much that the result of pointer subtraction between members is undefined so much as it is unreliable (i.e. not guaranteed to be the same between different instances of the same struct type when the same arithmetic is applied).
Related
Are explicit casts from char * to other pointer types fully defined behavior according to ANSI C89 if the pointer is guaranteed to meet the alignment requirements of the type you're casting to? Here's an example of what I mean:
/* process.c */
void *process(size_t elem_size, size_t cap) {
void *arr;
assert(cap > 5);
arr = malloc(elem_size * cap);
/* set id of element 5 to 0xffffff */
*(long *)((char *)arr + elem_size*5) = 0xffffff;
/* rest of the code omitted */
return arr;
}
/* main.c */
struct some_struct { long id; /* other members omitted */ };
struct other_struct { long id; /* other members omitted */ };
int main(int argc, char **argv) {
struct some_struct *s = process(sizeof(struct some_struct), 40);
printf("%lx\n", s[5].id);
return 0;
}
This code compiles without warnings and works as expected on my machine but I'm not fully sure if these kinds of casts are defined behavior.
C89 draft, section 4.10.3 (Memory management functions):
The pointer returned if the allocation succeeds is suitably aligned so that it may be assigned to a pointer to any type of object and then used to access such an object in the space allocated (until the space is explicitly freed or reallocated).
C89 draft, section 3.3.4 (Cast operators):
A pointer to an object or incomplete type may be converted to a pointer to a different object type or a different incomplete type. The resulting pointer might not be valid if it is improperly aligned for the type pointed to. It is guaranteed, however, that a pointer to an object of a given alignment may be converted to a pointer to an object of the same alignment or a less strict alignment and back again; the result shall compare equal to the original pointer.
This clearly specifies what happens if you cast from struct some_struct * to char * and back to struct some_struct * but in my case the code responsible for allocation doesn't have access to the full struct definition so it can't initially specify the pointer type to be struct some_struct * so I'm not sure if the rule still applies.
If the code I posted is technically UB, is there another standards-compliant way to modify array elements without knowing their full type? Are there real-world impementations where you would expect it to do something else than ((struct some_struct *)arr)[5].id = 0xffffff;?
This code compiles without warnings and works as expected on my machine but I'm not fully sure if these kinds of casts are defined behavior.
In general, the casts have defined behavior, but that behavior can be that the result is not a valid pointer. Thus dereferencing the result of the cast may produce UB.
Considering only function process(), then, it is possible that the result of its evaluation of (long *)((char *)arr + elem_size*5) is an invalid pointer. If and only if it is invalid, the attempt to use it to assign a value to the object it hypothetically points to produces UB.
However, main()s particular usage of that function is fine:
In process(), the pointer returned by malloc and stored in arr is suitably aligned for a struct some_struct (and for any other type).
The compiler must choose a size and layout for struct some_struct such that every element and every member of every element of an array of such structures is properly aligned.
Arrays are composed of contiguous objects of the array's element type, without gaps (though the element types themselves may contain padding if they are structures or unions).
Therefore, (char *)arr + n * sizeof(struct some_struct) must be suitably aligned for a struct some_struct, for any integer n such that the result points within or just past the end of the allocated region. This computation is closely related to the computations involved in accessing an array of struct some_struct via the indexing operator.
struct some_struct has a long as its first member, and that member must appear at offset 0 from the beginning of the struct. Therefore, every pointer that is suitably aligned for a struct some_struct must also be suitably aligned for a long.
Determining whether an operation upholds or violates "type aliasing" constraints requires being able to answer two questions:
For purposes of the aliasing rules, when does a region of storage hold an "object" of a particular type.
For purposes of the aliasing rules, is a particular access performed "by" an lvalue expression of a particular type.
For your question, the second issue above is the most the relevant: if a pointer or lvalue of type T1 is used to derive a value of type T2* which is dereferenced to access storage, the access may sometimes need to be regarded for purposes of aliasing as though performed by an lvalue of type T1, and sometimes as though performed by one of type T2, but the Standard fails to offer any guidance as to which interpretation should apply when. Constructs like yours would be processed predictably by implementation that didn't abuse the Standard as an excuse to behave nonsensically, but could be processed nonsensically by conforming but obtuse implementations that do abuse the Standard in such fashion.
The authors of C89 didn't expect anyone to care about the precise boundaries between constructs whose behavior was defined by the Standard, versus those which all implementations were expected to process identically but which the Standard didn't actually define, and thus saw no need to define the terms "object" and "by" with sufficient precision to unambiguously answer the above questions in ways that would yield defined program behavior in all sensible cases.
Let's say I want to move a void* pointer by 4 bytes. Are the following equivalent:
A:
void* new_address(void* in_ptr) {
intptr_t tmp = (intptr_t)in_ptr;
intptr_t new_address = tmp + 4;
return (void*)new_address;
}
B:
void* new_address(void* in_ptr) {
char* tmp = (char*)in_ptr;
char* new_address = tmp + 4;
return (void*)new_address;
}
Are both defined behavior? Is one more popular/accepted convention? Any other reason to use one over the other?.
Let's only consider 64bit systems. If intptr_t is not available we can use int64_t instead.
The context is a custom memory allocator which needs to move the pointer before allocating new block of memory to a specific address (for alignment purposes). We don't know what object the resulting pointer is going to point to yet but we know we need to move it to a specific location which in the examples above is 4 bytes.
Michael Kerrisk says on page 1415 that,
The C standards make one exception to the rule that pointers of
different types need not have the same representation: pointers of the
types char * and void * are required to have the same internal
representation.
All the C standard guarantees (7.18.1.4) is that you can convert void* values to intptr_t (or uintptr_t) and back again and end up with an equal value for the pointer.
The nuance is here that we cannot apply mathematical operations (including ==) if void* is in use.
Is casting a pointer to intptr_t [...] defined behavior?
Converting a pointer to any integer type is defined and the result is implementation defined, except when result can't be represented in integer type, then it's undefined behavior. See C11 6.3.2.3p6. But intptr_t has to be able to represent void* - the behavior is defined.
, doing arithmetic on it and then casting back, defined behavior?
Any integer may be converted to any pointer type. The resulting pointer is implementation defined - there is no guarantee that adding 4 to intptr_t will increment the pointer value by 4. See C11 6.3.2.3p5.
Are both defined behavior?
Yes, however the result is implementation defined.
Is one more popular/accepted convention?
Subjective: I say using uintptr_t is more popular then intptr_t. Converting a pointer to uintptr_t or to char* to do some arithmetic happens in some code, I can't say which is more popular.
Any other reason to use one over the other?.
Not really, but I think go with char*.
When it comes to actually accessing the data behind the resulting pointer - it depends. If the resulting pointer points within the same object then you're fine (remember, conversion is implementation defined). If the resulting pointer does not point to the same object, I believe the best interpretation would be from reading c2263 Clarifying Pointer Provenance v4 2.2.3Q5 and I think that's: the current C11 standard does not clearly specify that, which would make the behavior not defined.
Because you tagged gcc, both code snippets should compile to equivalent code - I believe on all architectures pointers are converted 1:1 to (u)intptr_t on gcc. Gcc docs implementation defined behavior 4.7 arrays and pointers states casting from pointer to integer and back again, the resulting pointer must reference the same object as the original pointer, otherwise the behavior is undefined - so you're safe as long as the resulting pointer points to the same object.
The context is a custom memory allocator
See implementations of container_of and offsetof macros. Do not hardcode + 4 in your code, and if you do, do not depend on alignment requirements on accessing the resulting pointers - remember to use memcpy to safely copy the context or handle alignment properly. Do not reinvent the wheel - when in doubt see other implementations like glibc malloc.c or newlib malloc.c - they both calculate on char* in mem2chunk macro, but also happen to do calculations on uintptr_t integers.
No 'strictly conforming program uses A. Using the result may be Undefined Behaviour as there is no requirement for addition against intptr_t to be reflected in a pointer value if that intptr_ is converted back to a pointer.
It is both unspecified behaviour and implementation-defined.
If the optional type intptr_t is defined all you are guaranteed is that you can convert void * to intptr_t and then convert that value back to void * and the two values will compare equal (==).
The strictly conforming way to perform pointer arithmetic is B. B is guaranteed to work if and only if the pointer int_ptr is valid and for the largest enclosing object there are 3 or more bytes in that object beyond that value. It's 3 because it's valid to point to (but not dereference) to the address that is (logically) one byte beyond the end of an object.
Object includes a declared object (including array) or block of memory such as returned by malloc().
All good practice is to prefer to write 'strictly conforming' programs where possible. So all good practice is to prefer B over A.
According to the standard the use of the pointer (as a pointer) may result in Undefined Behaviour because it may be (implementation defined) to be a trap representation.
A strictly conforming program is defined as "A strictly conforming program shall use only those features of the language and library specified in this International Standard.3) It shall not produce output dependent on any unspecified, undefined, or implementation-defined behavior, and shall not exceed any minimum implementation limit.
There's some disagreement about whether the code offered for A is unspecified or implementation defined. The standard says both because implementation-defined behaviour is a sub-category of unspecified. However because the implementation may document it as a trap representation using the value may result in Undefined Behaviour.
But I hope that is swept aside by the fact that 'strictly conforming programs' don't depend on unspecified, undefined or implementation defined behaviour.
So good practice here is certainly B.
Consider a secure environment that encrypts pointer values to deliberately confound the de-referencing of arbitrary pointer values. In principle it could provide intptr_t and be conformant.
Though I still maintain that if A doesn't work then intptr_t being an optional type it would be better to not provide it. Whether it is defined is unspecified and implementation dependent. That's because no 'strictly conforming program' uses it and it has no practical use other than to manipulate a pointer as an arithmetic type in a way not supported by pointer arithmetic on a compatible pointer type char *. The snippet in A falls into that category.
To store a void * declare a void * or char[sizeof(void*)] or malloc() or similar. To overlay a void * over an arithmetic type, declare a union and benefit that the union will be aligned for a void *.
But according to the specification it is unspecified, implementation-defined no 'strictly conforming program' can rely on it and may result in Undefined Behaviour.
A very long winded way of saying the answer, here, is B.
Consider the following code:
int main()
{
typedef struct { int first; float second; } type;
type whole = { 1, 2.0 };
void * vp = &whole;
struct { int first; } * shorn = vp;
printf("values: %d, %d\n", ((type *)vp)->first, shorn->first);
if (vp == shorn)
printf("ptrs compare the same\n");
return 0;
}
Two questions:
Is the pointer equality comparison UB?
Regarding the "shearing" away of the second member on the line that initializes shorn: is it valid C to cast away struct members like this and then dereference the manipulated pointer to access the remaining member?
Comparing two pointers with == when one is a void * is well defined.
Section 6.5.9 of the C standard regarding the equality operator == says the following:
2 One of the following shall hold:
both operands have arithmetic type;
both operands are pointers to qualified or unqualified versions of compatible types;
one operand is a pointer to an object type and the other is a pointer to a qualified or unqualified version of void; or
one operand is a pointer and the other is a null pointer constant
...
5 Otherwise, at least one operand is a pointer. If one operand is a pointer and the other is a null pointer constant, the null pointer
constant is converted to the type of the pointer. If one operand
is a pointer to an object type and the other is a pointer
to a qualified or unqualified version of void, the former is
converted to the type of the latter.
The usage of shorn->first works because a pointer to a struct can be converted to a pointer to its first member. For both type and the unnamed struct type their first member is an int so it works out.
Section 6.2.5 Types paragraph 28 of the C standard says:
[...] All pointers to structure types shall have the same representation and alignment requirements as each other. [...]
Section 6.3.2.3 Pointers paragraph 1 says:
A pointer to void may be converted to or from a pointer to any object type. A pointer to any object type may be converted to a pointer to void and back again; the result shall compare equal to the original pointer.
And paragraph 7 says:
A pointer to an object type may be converted to a pointer to a different object type. If the resulting pointer is not correctly aligned68) for the referenced type, the behavior is undefined. Otherwise, when converted back again, the result shall compare equal to the original pointer. [...]
And footnote 68 says:
In general, the concept "correctly aligned" is transitive: if a pointer to type A is correctly aligned for a pointer to type B, which in turn is correctly aligned for a pointer to type C, then a pointer to type A is correctly aligned for a pointer to type C.
Because all pointers to structure types have the same representation, the conversions between pointers to void and pointers to structure types must be the same for all pointers to structure types. So it seems that a pointer to structure type A could be converted by a cast operator directly to a pointer to structure type B without an intermediate conversion to a pointer to void as long as the pointer is "correctly aligned" for structure type B. (This may be a weak argument.)
The question remains when, in the case of two structure types A and B where the initial sequence of structure type A consists of all the members of structure type B, a pointer to structure type A is guaranteed to be correctly aligned for structure type B (the reverse is obviously not guaranteed). As far as I can tell, the C standard makes no such guarantee. So strictly speaking, a pointer to the larger structure type A might not be correctly aligned for the smaller structure type B, and if it is not, the behavior is undefined. For a "sane" compiler, the larger structure type A would not have weaker alignment than the smaller structure type B, but for an "insane" compiler, that might not be the case.
Regarding the second question about accessing members of the truncated (shorter) structure using the pointer derived from the full (longer) structure, then as long as the pointer is correctly aligned for the shorter structure (see above for why that might not be true for an "insane" compiler), and as long as strict aliasing rules are avoided (for example, by going through an intermediate pointer to void in an intermediate external function call across compilation unit boundaries), then accessing the members through the pointer to the shorter structure type should be perfectly fine. There is a special guarantee for that when objects of both structure types appear as members of the same union type. Section 6.3.2.3 Structure and union members paragraph 6 says:
One special guarantee is made in order to simplify the use of unions: if a union contains several structures that share a common initial sequence (see below), and if the union
object currently contains one of these structures, it is permitted to inspect the common initial part of any of them anywhere that a declaration of the completed type of the union
is visible. Two structures share a common initial sequence if corresponding members have compatible types (and, for bit-fields, the same widths) for a sequence of one or more initial members.
However, since the offsets of members within a structure type does not depend on whether an object of the structure type appears in a union type or not, the above implies that any structures with a common initial sequence of members will have those common members at the same offsets within their respective structure types.
In the language the C89 Standard was written to describe, it was well established that if two structures share a common initial sequence, a pointer to either may be cast to the other and used to inspect members of that common initial sequence. Code which relied upon this was commonplace and not considered even remotely controversial.
In the interest of optimization, the authors of the C99 Standard deliberately allowed compilers to assume that structures of different types won't alias in cases where such assumption would be useful for their customers. Because there are many good means by which implementations could recognize cases where such assumptions would be needlessly break what had been perfectly good code, and because they the authors of the Standard expected that compiler writers would make a bona fide effort to behave in ways useful to the programmers using their products, the Standard doesn't mandate any particular means of making such distinctions. Instead, it regards the ability to support constructs that had been universally supported as a "quality of implementation" issue, which would be reasonable if compiler writers made a bona fide effort to treat it as such.
Unfortunately, some compiler writers who aren't interested in selling their products to paying customers have taken the Standard's failure to mandate useful behavior as an invitation to behave in needlessly useless fashion. Code which relies upon Common Initial Sequence guarantees can thus not be meaningfully processed by clang or gcc without using either non-standard syntax or disabling type-based aliasing entirely.
I've been trying to understand a particular aspect of strict aliasing recently, and I think I have made the smallest possible interesting piece of code. (Interesting for me, that is!)
Update: Based on the answers so far, it's clear I need to clarify the question. The first listing here is "obviously" defined behaviour, from a certain point of view. The real issue is to follow this logic through to custom allocators and custom memory pools. If I malloc a large block of memory at the start, and then write my own my_malloc and my_free that uses that single large block, then is it UB on the grounds that it doesn't use the official free?
I'll stick with C, somewhat arbitrarily. I get the impression it is easier to talk about, that the C standard is a bit clearer.
int main() {
uint32_t *p32 = malloc(4);
*p32 = 0;
free(p32);
uint16_t *p16 = malloc(4);
p16[0] = 7;
p16[1] = 7;
free(p16);
}
It is possible that the second malloc will return the same address as the first malloc (because it was freed in between). That means that it is accessing the same memory with two different types, which violates strict aliasing. So surely the above is undefined behaviour (UB)?
(For simplicity, let's assume the malloc always succeeds. I could add in checks for the return value of malloc, but that would just clutter the question)
If it's not UB, why? Is there an explicit exception in the standard, which says that malloc and free (and calloc/realloc/...) are allowed to "delete" the type associated with a particular address, allowing further accesses to "imprint" a new type on the address?
If malloc/free are special, then does that mean I cannot legally write my own allocator which clones the behaviour of malloc? I'm sure there are plenty of projects out there with custom allocators - are they all UB?
Custom allocators
If we decide, therefore, that such custom allocators must be defined behaviour, then it means the strict aliasing rule is essentially "incorrect". I would update it to say that it is possible to write (not read) through a pointer of a different ('new') type as long as you don't use pointers of the old type any more. This wording could be quietly-ish changed if it was confirmed that all compilers have essentially obeyed this new rule anyway.
I get the impression that gcc and clang essentially respect my (aggressive) reinterpretation. If so, perhaps the standards should be edited accordingly? My 'evidence' regarding gcc and clang is difficult to describe, it uses memmove with an identical source and destination (which is therefore optimized out) in such a way that it blocks any undesirable optimizations because it tells the compiler that future reads through the destination pointer will alias the bit pattern that was previously written through the source pointer. I was able to block the undesirable interpretations accordingly. But I guess this isn't really 'evidence', maybe I was just lucky. UB clearly means that the compiler is also allowed to give me misleading results!
( ... unless, of course, there is another rule that makes memcpy and memmove special in the same way that malloc may be special. That they are allowed to change the type to the type of the destination pointer. That would be consistent with my 'evidence'. )
Anyway, I'm rambling. I guess a very short answer would be: "Yes, malloc (and friends) are special. Custom allocators are not special and are therefore UB, unless they maintain separate memory pools for each type. And, further, see example X for an extreme piece of code where compiler Y does undesirable stuff precisely because compiler Y is very strict in this regard and is contradicting this reinterpretation."
Follow up: what about non-malloced memory? Does the same thing apply. (Local variables, static variables, ...)
Here are the C99 strict aliasing rules in (what I hope is) their entirety:
6.5
(6) The effective type of an object for an access to its stored value is the declared type of the object, if any. If a value is stored into an object having no declared type through an lvalue having a type that is not a character type, then the type of the lvalue becomes the effective type of the object for that access and for subsequent accesses that do not modify the stored value. If a value is copied into an object having no declared type using
memcpy or memmove, or is copied as an array of character type, then the effective type
of the modified object for that access and for subsequent accesses that do not modify the
value is the effective type of the object from which the value is copied, if it has one. For
all other accesses to an object having no declared type, the effective type of the object is
simply the type of the lvalue used for the access.
(7) An object shall have its stored value accessed only by an lvalue expression that has one of
the following types:
— a type compatible with the effective type of the object,
— a qualified version of a type compatible with the effective type of the object,
— a type that is the signed or unsigned type corresponding to the effective type of the object,
— a type that is the signed or unsigned type corresponding to a qualified version of the effective type of the object,
— an aggregate or union type that includes one of the aforementioned types among its members (including, recursively, a member of a subaggregate or contained union), or
— a character type.
These two clauses together prohibit one specific case, storing a value via an lvalue of type X and then subsequently retrieving a value via an lvalue of type Y incompatible with X.
So, as I read the standard, even this usage is perfectly OK (assuming 4 bytes are enough to store either an uint32_t or two uint16_t).
int main() {
uint32_t *p32 = malloc(4);
*p32 = 0;
/* do not do this: free(p32); */
/* do not do this: uint16_t *p16 = malloc(4); */
/* do this instead: */
uint16_t *p16 = (uint16_t *)p32;
p16[0] = 7;
p16[1] = 7;
free(p16);
}
There's no rule that prohibits storing an uint32_t and then subsequently storing an uint16_t at the same address, so we're perfectly OK.
Thus there's nothing that would prohibit writing a fully compliant pool allocator.
Your code is correct C and does not invoke undefined behaviour (except that you do not test malloc return value) because :
you allocate a bloc of memory, use it and free it
you allocate another bloc of memory, use it and free it.
What is undefined is whether p16 will receive same value as p32 had at a different time
What would be undefined behaviour, even if value was the same would be to access p32 after it has been freed. Examples :
int main() {
uint32_t *p32 = malloc(4);
*p32 = 0;
free(p32);
uint16_t *p16 = malloc(4);
p16[0] = 7;
p16[1] = 7;
if (p16 == p32) { // whether p16 and p32 are equal is undefined
uint32_t x = *p32; // accessing *p32 is explicitely UB
}
free(p16);
}
It is UB because you try to access a memory block after it has been freed. And even when it does point to a memory block, that memory block has been initialized as an array of uint16_t, using it as a pointer to another type is formally undefined behaviour.
Custom allocation (assuming a C99 conformant compiler) :
So you have a big chunk of memory and want to write custom free and malloc functions without UB. It is possible. Here I will not go to far into the hard part of management of allocated and free blocs, and just give hints.
you will need to know what it the strictest alignement for the implementation. stdlib malloc knows it because 7.20.3 §1 of C99 language specification (draft n1256) says : The pointer returned if the allocation
succeeds is suitably aligned so that it may be assigned to a pointer to any type of object. It is generally 4 on 32 bits systems and 8 on 64 bits systems, but might be greater or lesser ...
you memory pool must be a char array because 6.3.2.3 §7 says : A pointer to an object or incomplete type may be converted to a pointer to a different
object or incomplete type. If the resulting pointer is not correctly aligned for the pointed-to type, the behavior is undefined. Otherwise, when converted back again, the result shall compare equal to the original pointer. When a pointer to an object is converted to a pointer to a character type, the result points to the lowest addressed byte of
the object. Successive increments of the result, up to the size of the object, yield pointers to the remaining bytes of the object. : that means that provided you can deal with the alignement, a character array of correct size can be converted to a pointer to an arbitrary type (and is the base of malloc implementation)
You must make your memory pool start at an address compatible with the system alignement :
intptr_t orig_addr = chunk;
int delta = orig_addr % alignment;
char *pool = chunk + alignement - delta; /* pool in now aligned */
You now only have to return from your own pool addresses of blocs got as pool + n * alignement and converted to void * : 6.3.2.3 §1 says : A pointer to void may be converted to or from a pointer to any incomplete or object type. A pointer to any incomplete or object type may be converted to a pointer to void and back again; the result shall compare equal to the original pointer.
It would be cleaner with C11, because C11 explicitely added _Alignas and alignof keywords to explictely deal with it and it would be better than the current hack. But it should work nonetheless
Limits :
I must admit that my interpretation of 6.3.2.3 §7 is that a pointer to a correctly aligned char array can be converted to a pointer of another type is not really neat and clear. Some may argue that what is said is just that if it originally pointed to the other type, it can be used as a char pointer. But as I start from a char pointer it is not explicitely allowed. That's true, but it is the best that can be done, it is not explicely marked as undefined behaviour ... and it is what malloc does under the hood.
As alignement is explicitely implementation dependant, you cannot create a general library usable on any implementation.
The actual rules regarding aliasing are laid out in standard section 6.5, paragraph 7. Note the wording:
An object shall have its stored value accessed only by an lvalue expression that has one of the following types:
(emphasis mine)
Aliasing includes the notion of objects, not just general memory. For malloc to have returned the same address on a second use requires the original object to have been deallocated. Even if it has the same address, it is not considered the same object. Any attempts to access the first object through dangling pointers leftover after free are UB for completely different reasons, so there's no aliasing because any continued use of the first pointer p32 is invalid anyway.
I can't seem to wrap my head around certain parts of the C standard, so I'm coming here to clear up that foggy, anxious uncertainty that comes when I have to think about what such tricks are defined behaviour and what are undefined or violate the standard. I don't care whether or not it will WORK, I care if the C standard considers it legal, defined behaviour.
Such as this, which I am fairly certain is UB:
struct One
{
int Hurr;
char Durr[2];
float Nrrr;
} One;
struct Two
{
int Hurr;
char Durr[2];
float Nrrr;
double Wibble;
} Two;
One = *(struct One*)&Two;
This is not all I am talking about. Such as casting the pointer to One to int*, and dereferencing it, etc. I want to get a good understanding of what such things are defined so I can sleep at night. Cite places in the standard if you can, but be sure to specify whether it's C89 or C99. C11 is too new to be trusted with such questions IMHO.
I think that technically that example is UB, too. But it will almost certainly work, and neither gcc nor clang complain about it with -pedantic.
To start with, the following is well-defined in C99 (§6.5.2.3/6): [1]
union OneTwo {
struct One one;
struct Two two;
};
OneTwo tmp = {.two = {3, {'a', 'b'}, 3.14f, 3.14159} };
One one = tmp.one;
The fact that accessing the "punned" struct One through union must work implies that the layout of the prefix of struct Two is identical to struct One. This cannot be contingent on the existence of a union because the a given composite type can only have one storage layout, and its layout cannot be contingent on its use in a union because the union does not need to be visible to every translation unit in which the struct is used.
Furthermore, in C all types are no more than a sequence of bytes (unlike, for example, C++) (§6.2.6.1/4) [2]. Consequently, the following is also guaranteed to work:
struct One one;
struct Two two = ...;
unsigned char tmp[sizeof one];
memcpy(tmp, two, sizeof one);
memcpy(one, tmp, sizeof one);
Given the above and the convertibility of any pointer type to a void*, I think it is reasonable to conclude that the temporary storage above is unnecessary, and it could have been written directly as:
struct One one;
struct Two two = ...;
unsigned char tmp[sizeof one];
memcpy(one, two, sizeof one);
From there to the direct assignment through an aliased pointer as in the OP is not a very big leap, but there is an additional problem for the aliased pointer: it is theoretically possible for the pointer conversion to create an invalid pointer, because it's possible that the bit format of a struct Two* differs from a struct One*. Although it is legal to cast one pointer type to another pointer type with looser alignment (§6.3.2.3/7) [3] and then convert it back again, it is not guaranteed that the converted pointer is actually usable, unless the conversion is to a character type. In particular, it is possible that the alignment of struct Two is different from (more strict than) the alignment of struct One, and that the bit format of the more strongly-aligned pointer is not directly usable as a pointer to the less strongly-aligned struct. However, it is hard to see an argument against the almost equivalent:
one = *(struct One*)(void*)&two;
although this may not be explicitly guaranteed by the standard.
In comments, various people have raised the spectre of aliasing optimizations. The above discussion does not touch on aliasing at all because I believe that it is irrelevant to a simple assignment. The assignment must be sequenced after any preceding expressions and before any succeeding ones; it clearly modifies one and almost as clearly references two. An optimization which made a preceding legal mutation of two invisible to the assignment, would be highly suspect.
But aliasing optimizations are, in general, possible. Consequently, even though all of the above pointer casts should be acceptable in the context of a single assignment expression, it would certainly not be legal behaviour to retain the converted pointer of type struct One* which actually points into an object of type struct Two and expect it to be usable either to mutate a member of its target or to access a member of its target which has otherwise been mutated. The only context in which you could get away with using a pointer to struct One as though it were a pointer to the prefix of struct Two is when the two objects are overlaid in a union.
--- Standard references:
[1] "if a union contains several structures that share a common initial sequence (see below), and if the union object currently contains one of these structures, it is permitted to inspect the common initial part of any of them anywhere that a declaration of the completed type of the union is visible."
[2] "Values stored in non-bit-field objects of any other object type consist of n × CHAR_BIT
bits, where n is the size of an object of that type, in bytes. The value may be copied into
an object of type unsigned char [n] (e.g., by memcpy)…"
[3] "A pointer to an object type may be converted to a pointer to a different object type… When a pointer to an object is converted to a pointer to a character type, the result points to the lowest addressed byte of the object. Successive increments of the result, up to the size of the object, yield pointers to the remaining bytes of the object."
C99 6.7.2.1 says:
Para 5
As discussed in 6.2.5, a structure is a type consisting of a sequence
of members, whose storage is allocated in an ordered sequence
Para 12
Each non-bit-field member of a structure or union object is aligned in
an implementation-defined manner appropriate to its type.
Para 13
Within a structure object, the non-bit-field members and the units in
which bit-fields reside have addresses that increase in the order in
which they are declared. A pointer to a structure object, suitably
converted, points to its initial member (or if that member is a
bit-field, then to the unit in which it resides), and vice versa. There
may be unnamed padding within a structure object, but not at its
beginning
That last paragraph covers your second question (casting the pointer to One to int*, and dereferencing it).
The first point - whether it is valid to "Downcast" a Two* to a One* - I could not find specifically addressed. It boils down to whether the other rules ensure that the memory layout of the fields of One and the initial fields of Two are identical in all cases.
The members have to be packed in ordered sequence, no padding is allowed at the beginning, and they have to be aligned according to type, but the standard does not actually say that the layout needs to be the same (even though in most compilers I am sure it is).
There is, however, a better way to define these structures so that you can guarantee it:
struct One
{
int Hurr;
char Durr[2];
float Nrrr;
} One;
struct Two
{
struct One one;
double Wibble;
} Two;
You might think you can now safely cast a Two* to a One* - Para 13 says so. However strict aliasing might bite you somewhere unpleasant. But with the example above you don't need to anyway:
One = Two.one;
A1. Undefined behaviour, because of Wibble.
A2. Defined.
S9.2 in N3337.
Two standard-layout struct (Clause 9) types are layout-compatible if
they have the same number of non-static data members and corresponding
non-static data members (in declaration order) have layout-compatible
types
Your structs would be layout compatible and thus interchangeable but for Wibble. There is a good reason too: Wibble might cause different padding in struct Two.
A pointer to a standard-layout struct object, suitably converted using
a reinterpret_cast, points to its initial member (or if that member is
a bit-field, then to the unit in which it resides) and vice versa.
I think that guarantees that you can dereference the initial int.