I'm interested in the technique used by Sean Barrett to make a dynamic array in C for any type. Comments in the current version claims the code is safe to use with strict-aliasing optimizations:
https://github.com/nothings/stb/blob/master/stb_ds.h#L332
You use it like:
int *array = NULL;
arrput(array, 2);
arrput(array, 3);
The allocation it does holds both the array data + a header struct:
typedef struct
{
size_t length;
size_t capacity;
void * hash_table;
ptrdiff_t temp;
} stbds_array_header;
The macros/functions all take a void* to the array and access the header by casting the void* array and moving back one:
#define stbds_header(t) ((stbds_array_header *) (t) - 1)
I'm sure Sean Barrett is far more knowledgeable than the average programmer. I'm just having trouble following how this type of code is not undefined behavior because of the strict aliasing rules in modern C. If this does avoid problems I'd love to understand why it does so I can incorporate it myself (maybe with a few less macros).
Lets follow the expansions of arrput in https://github.com/nothings/stb/blob/master/stb_ds.h :
#define STBDS_REALLOC(c,p,s) realloc(p,s)
#define arrput stbds_arrput
#define stbds_header(t) ((stbds_array_header *) (t) - 1)
#define stbds_arrput(a,v) (stbds_arrmaybegrow(a,1), (a)[stbds_header(a)->length++] = (v))
#define stbds_arrmaybegrow(a,n) ((!(a) || stbds_header(a)->length + (n) > stbds_header(a)->capacity) \
? (stbds_arrgrow(a,n,0),0) : 0)
#define stbds_arrgrow(a,b,c) ((a) = stbds_arrgrowf_wrapper((a), sizeof *(a), (b), (c)))
#define stbds_arrgrowf_wrapper stbds_arrgrowf
void *stbds_arrgrowf(void *a, size_t elemsize, size_t addlen, size_t min_cap)
{
...
b = STBDS_REALLOC(NULL, (a) ? stbds_header(a) : 0, elemsize * min_cap + sizeof(stbds_array_header));
//if (num_prev < 65536) prev_allocs[num_prev++] = (int *) (char *) b;
b = (char *) b + sizeof(stbds_array_header);
if (a == NULL) {
stbds_header(b)->length = 0;
stbds_header(b)->hash_table = 0;
stbds_header(b)->temp = 0;
} else {
STBDS_STATS(++stbds_array_grow);
}
stbds_header(b)->capacity = min_cap;
return b;
}
how this type of code is not undefined behavior because of the strict aliasing
Strict aliasing is about accessing data that has different effective type than data stored there. I would argue that the data stored in the memory region pointed to by stbds_header(array) has the effective type of the stbds_array_header structure, so accessing it is fine. The structure members are allocated by realloc and initialized one by one inside stbds_arrgrowf by stbds_header(b)->length = 0; lines.
how this type of code is not undefined behavior
I think the pointer arithmetic is fine. You can say that the result of realloc points to an array of one stbds_array_header structure. In other words, when doing the first stbds_header(b)->length = inside stbds_arrgrowf function the memory returned by realloc "becomes" an array of one element of stbds_array_header structures, as If a value is stored into an object having no declared type through an lvalue having a type that is not a character type, then the type of the lvalue becomes the effective type of the object for that access from https://port70.net/~nsz/c/c11/n1570.html#6.5p6 .
int *array is assigned inside stbds_arrgrow to point to "one past the last element of an array" of one stbds_array_header structure. (Well, this is also the same place where an int array starts). ((stbds_array_header *) (array) - 1) calculates the address of the last array element by subtracting one from "one past the last element of an array". I would rewrite it as (char *)(void *)t - sizeof(stbds_array_header) anyway, as (stbds_array_header *) (array) sounds like it would generate a compiler warning.
Assigning to int *array in expansion of stbds_arrgrow a pointer to (char *)result_of_realloc + sizeof(stbds_array_header) may very theoretically potentially be not properly aligned to int array type, breaking If the resulting pointer is not correctly aligned for the referenced type, the behavior is undefined from https://port70.net/~nsz/c/c11/n1570.html#6.3.2.3p7 . This is very theoretical, as stbds_array_header structure has size_t and void * and ptrdiff_t members, in any normal architecture it will have good alignment to access int (or any other normal type) after it.
I have only inspected the code in expansions of arrput. This is a 2000 lines of code, there may be other undefined behavior anywhere.
Related
The code seen below is typical of what is seen in some arena implementations. One such
example can be found here (blog article on an example impl.).
#include<stdio.h>
#include<stdint.h>
#include<stdalign.h>
struct thing {
int a;
int b;
};
char buffer[128];
int main ()
{
uintptr_t p1 = (uintptr_t)buffer;
if (p1 % alignof(struct thing)) return 1;
struct thing *t1 = (void*)buffer;
t1->a = 10;
t1->b = 20;
uintptr_t p2 = (uintptr_t)(buffer + sizeof(struct thing));
if (p2 % alignof(struct thing)) return 1;
struct thing *t2 = (void*)(buffer + sizeof(struct thing));
t2->a = 30;
t2->b = 40;
printf("%d\n",t1->a);
printf("%d\n",t2->a);
return 0;
}
edited code: Made the program return 1 if any pointer lacks proper alignment
Is this an instance of a strict aliasing violation, and ...
Is the only way to place such structures in a buffer and to retrieve a safe to use pointer to the structure to do for example:
struct thing *t1 = memcpy(buffer,&((struct thing){10,20}),sizeof(struct thing));
Is this an instance of a strict aliasing violation
Yes. t1->a etc access the character array through a different type than the "effective type" (char).
Is the only way to place such structures in a buffer and to retrieve a safe to use pointer to the structure to do for example:
You can also create a union of a raw character array and the type you wish to convert to. Example:
typedef union
{
struct thing t;
char buf[128];
} strict_aliasing_hack;
...
strict_aliasing_hack thing t1 = *(strict_aliasing_hack*)buffer;
This is ok because strict_aliasing_hack is "an aggregate or union type that includes one a type compatible with the effective type of the object among its members" (C17 6.5/7).
Naturally, it is best to stay clear of fishy conversions like this entirely. For example the chunk of data returned from malloc has no effective type. So the original code is much better written as:
struct thing *t1 = malloc(128);
And now you can lvalue access *t1 in any way you like.
The problem is that per standard only dynamic memory can be used that way.
Clause 5 Expressions says (ref n1570 for C11):
§6 The effective type of an object for an access to its stored value is the declared type of the object, if any.
If you use:
void * buffer = malloc(128);
then buffer is guaranteed to have an alignment compatible with any standard type and has no declared type.
In that case you can safely store a thing object in buffer without triggering any strict aliasing violation. But in your example code, buffer has a declared type which is char. Whatever the alignment, using a different type is then a strict aliasing violation.
IMO memcpy is always the safest way. It will produce the optimized enough output for the particular platform.
typedef struct thing {
int a;
int b;
}thing;
int geta(const void *buff, const size_t offset)
{
const unsigned char *chbuff = buff;
thing t;
memcpy(&t, chbuff + offset, sizeof(t));
return t.a;
}
int geta1(const void *buff, const size_t offset)
{
const unsigned char *chbuff = buff;
int a;
memcpy(&a, chbuff + offset + offsetof(thing, a), sizeof(a));
return a;
}
https://godbolt.org/z/x8e96ezW9
I am working on a set of C code to optimize it. I came across a warning while fixing a broken code.
The environment is Linux, C99, compiling with -Wall -O2 flags.
Initially a struct text is defined like this:
struct text {
char count[2];
char head[5];
char textdata[5];
}
The code is to return pointer T1 and T2 to expected head and textdata strings:
int main(void) {
struct text *T1;
char *T2;
char data[] = "02abcdeabcde";
T1 = (struct text *)data;
T2 = T1->textdata;
gettextptr((char *)T1, T2);
printf("\nT1 = %s\nT2 = %s\n", (char *)T1, T2);
return (0);
}
void gettextptr(char *T1, char *T2) {
struct text *p;
int count;
p = (struct text *)T1;
count = (p->count[0] - '0') * 10 + (p->count[1] - '0');
while (count--) {
if (memcmp(T2, T1, 2) == 0) {
T1 += 2;
T2 += 2;
}
}
}
This wasn't working as expected. It was expected to return the addresses of first 'c' and last 'e'. Through GDB, I found that, once execution pointer returns from gettextptr() to parent function, it doesn't keep the address of T1 and T2. Then I tried another approach to 'Call by reference' by using double pointer:
int main(void) {
struct text *T1;
char *T2;
char data[] = "02abcdeabcde";
T1 = (struct text *)data;
T2 = T1->textdata;
gettextptr((char **)&T1, &T2);
printf("\nT1 = %s\nT2 = %s\n", (char *)T1, T2);
return (0);
}
void gettextptr(char **T1, char **T2) {
struct text *p;
int count;
p = (struct text *)(*T1);
count = (p->count[0] - '0') * 10 + (p->count[1] - '0');
while (count--) {
if (memcmp(*T2, *T1, 2) == 0) {
*T1 += 2;
*T2 += 2;
}
}
}
When I compile this code with -Wall -O2, I am getting the following GCC warning:
pointer.c: In function ‘main’:
pointer.c:23: warning: dereferencing type-punned pointer will break strict-aliasing rules
So:
Was the code calling by value in first case?
Isn't (char **) permitted for casting while keeping strict aliasing rules?
What am I missing to resolve this warning?
The strict aliasing rule is paragraph 6.5/7 of the Standard. It says basically that you may access an object only through an lvalue of compatible type, possibly with additional qualifiers; the corresponding signed / unsigned type; an array, structure, or union type with one of those among its members, or a character type. The diagnostic you received is saying that your code violates that rule, and it does, multiple times.
You get yourself in trouble very early with:
T1 = (struct text *)data;
That conversion is allowed, though the resulting pointer is not guaranteed to be correctly aligned, but there's not much you can then do with T1 without violating the strict aliasing rule. In particular, if you dereference it with * or -> -- which is in fact the very next thing you do -- then you access a char array as if it were a struct text. That is not allowed, though the reverse would be a different story.
Converting T1 to a char * and accessing the pointed to array through that pointer, as you do later, are some of the few things you may do with it.
gettextexpr() is the same (both versions). It performs the same kind of conversion described above, and dereferences the converted pointer when it accesses p->count. The resulting behavior violates the strict aliasing rule, and is therefore undefined. What GCC is actually complaining about in the second case, however, is probably accessing *T1 as if it were a char *, when it is really a struct text * -- another, separate, strict aliasing violation.
So, in response to your specific questions:
Was the code calling by value in first case?
C has only pass by value, so yes. In the first case, you pass two char pointers by value, which you could then use to modify the caller's char data. In the second case, you pass two char * pointers by value, which you can and do use to modify the caller's char * variables.
Isn't (char **) permitted for casting while keeping strict aliasing rules?
No, absolutely not. Casting to char * (not char **) can allow you to access an object's representation through the resulting pointer, because dereferencing a char * produces an lvalue of character type, but there is no type that can generically be converted from without strict-aliasing implications.
What am I missing to resolve this warning?
You are missing that what you are trying to do is fundamentally disallowed. C does not permit access a char array as if it were a struct text, period. Compilers may nevertheless accept code that does so, but its behavior is undefined.
Resolve the warning by abandoning the cast-to-structure approach, which is providing only a dusting of syntactic sugar, anyway. It's actually simpler and clearer to get rid of all the casting and write:
count = ((*T1)[0] - '0') * 10 + ((*T1)[1] - '0');
It's perhaps clearer still to get rid of all the casting use sscanf:
sscanf(*T1, "%2d", &count);
Note also that even if it were allowed, your specific access pattern seems to make assumptions about the layout of the structure members that are not justified by the language. Implementations may use arbitrary padding between members and after the last member, and your code cannot accommodate that.
typedef struct {
int num;
char arr[64];
} A;
typedef struct {
int num;
char arr[];
} B;
I declared A* a; and then put some data into it. Now I want to cast it to a B*.
A* a;
a->num = 1;
strcpy(a->arr, "Hi");
B* b = (B*)a;
Is this right?
I get a segmentation fault sometimes (not always), and I wonder if this could be the cause of the problem.
I got a segmentation fault even though I didn't try to access to char arr[] after casting.
This defines a pointer variable
A* a;
There is nothing it is cleanly pointing to, the pointer is non-initialised.
This accesses whatever it is pointing to
a->num = 1;
strcpy(a->arr, "Hi");
Without allocating anything to the pointer beforehand (by e.g. using malloc()) this is asking for segfaults as one possible consequence of the undefined behaviour it invokes.
This is an addendum to Yunnosch's answer, which identifies the problem correctly. Let's assume you do it correctly and either write just
A a;
which gives you an object of automatic storage duration when declared inside a function, or you dynamically allocated an instance of A like this:
A *a = malloc(sizeof *a);
if (!a) return -1; // or whatever else to do in case of allocation error
Then, the next thing is your cast:
B* b = (B*)a;
This is not correct, types A and B are not compatible. Here, it will probably work in practice because the struct members are compatible, but beware that strange things can happen because the compiler is allowed to assume a and b point to different objects because their types are not compatible. For more information, read on the topic of what's commonly called the strict aliasing rule.
You should also know that an incomplete array type (without a size) is only allowed as the very last member of a struct. With a definition like yours:
typedef struct {
int num;
char arr[];
} B;
the member arr is allowed to have any size, but it's your responsibility to allocate it correctly. The size of B (sizeof(B)) doesn't include this member. So if you just write
B b;
you can't store anything in b.arr, it has a size of 0. This last member is called a flexible array member and can only be used correctly with dynamic allocation, adding the size manually, like this:
B *b = malloc(sizeof *b + 64);
This gives you an instance *b with an arr of size 64. If the array doesn't have the type char, you must multiply manually with the size of your member type -- it's not necessary for char because sizeof(char) is by definition 1. So if you change the type of your array to something different, e.g. int, you'd write this to allocate it with 64 elements:
B *b = malloc(sizeof *b + 64 * sizeof *(b->arr));
It appears that you are confusing two different topics. In C99/C11 char arr[]; as the last member of a structure is a Flexible Array Member (FAM) and it allows you to allocate for the structure itself and N number of elements for the flexible array. However -- you must allocate storage for it. The FAM provides the benefit of allowing one-allocation and one-free where there would normally be two required. (In C89 a similar implementation went by the name struct hack, but it was slightly different).
For example, B *b = malloc (sizeof *b + 64 * sizeof *b->arr); would allocate storage for b plus 64-characters of storage for b->arr. You could then copy the members of a to b using the proper '.' and '->' syntax.
A short example can illustrate:
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#define NCHAR 64 /* if you need a constant, #define one (or more) */
typedef struct {
int num;
char arr[NCHAR];
} A;
typedef struct {
int num;
char arr[]; /* flexible array member */
} B;
int main (void) {
A a = { 1, "Hi" };
B *b = malloc (sizeof *b + NCHAR * sizeof *b->arr);
if (!b) {
perror ("malloc-b");
return 1;
}
b->num = a.num;
strcpy (b->arr, a.arr);
printf ("b->num: %d\nb->arr: %s\n", b->num, b->arr);
free (b);
return 0;
}
Example Use/Output
$ ./bin/struct_fam
b->num: 1
b->arr: Hi
Look things over and let me know if that helps clear things up. Also let me know if you were asking something different. It is a little unclear exactly where you confusion lies.
I'm trying to understanding the pointer casting in this case.
# https://github.com/udp/json-parser/blob/master/json.c#L408
#define json_char char
typedef struct _json_object_entry
{
json_char * name;
unsigned int name_length;
struct _json_value * value;
} json_object_entry;
typedef struct _json_value
{
struct
{
unsigned int length;
json_object_entry * values;
#if defined(__cplusplus) && __cplusplus >= 201103L
decltype(values) begin () const
{ return values;
}
decltype(values) end () const
{ return values + length;
}
#endif
} object;
}
(*(json_char **) &top->u.object.values) += string_length + 1;
Due to what I see top->u.object.values has the address of the first element of values ( type : json_object_entry ), and then we get the address of values, casting it to char, .. And from here I'm lost. I don't really understand the purpose of this.
// Notes : This is two pass parser for those who wonders what is this.
Thanks
_json_value::values is a pointer to the beginning of (or into) an array of json_object_entrys. The code adjusts its value by a few bytes, e.g in order to skip a header or such before the actual data. Because the pointer is typed one can without casting only change its value in quants of sizeof(_json_object_entry), but apparently the offset can have any value, depending on some string_length. So the address of the pointer is taken, cast to the address of a char pointer (a char pointer can be changed in 1 increments), dereferenced so the result is a pointer to char residing at the same place as the real u.object.values, and then assigned to.
One should add that such code may break at run time if the architecture demands a minimal alignment for structures (possibly depending on their first element, here a pointer) and the string length can have a value which is not a multiple of that alignment. That would make the code UB. I'm not exactly sure whether the code is nominally UB if the alignment is preserved.
Author here (guilty as charged...)
In the first pass, values hasn't yet been allocated, so the parser cheats by using the same field to store the amount of memory (length) that's going to be required when it's actually allocated in the second pass.
if (state.first_pass)
(*(json_char **) &top->u.object.values) += string_length + 1;
The cast to json_char is so that we add multiples of char to the length, rather than multiples of json_object_entry.
It is a bit (...OK, more than a bit...) of a dirty hack re-using the field like that, but it was to save adding another field to json_value or using a union (C89 unions can't be anonymous, so it would have made the structure of json_value a bit weird).
There's no UB here, because we're not actually using values as an array of structs at this point, just subverting the type system and using it as an integer.
json_object_entry * values;
...
}
(*(json_char **) &top->u.object.values) += string_length + 1;
forgetting type correctness, you can collapse the & and *:
((json_char **) top->u.object.values) += string_length + 1;
top->u.object.values is indeed the pointer to first element of values array. It is typecasted to a pointer to a pointer to json_char, and then advanced string_length + 1 characters. The net result is that top->u.object.values now points (string_length + 1) json_chars ahead of what it used to.
I have a piece of code bellow,and what's the difference of them?
The first one,the address of buf element of the struct is 4 bigger than that of the struct while the second one is not.
First
#include <stdio.h>
typedef struct A
{
int i;
char buf[]; //Here
}A;
int main()
{
A *pa = malloc(sizeof(A));
char *p = malloc(13);
memcpy(p, "helloworld", 10);
memcpy(pa->buf, p, 13);
printf("%x %x %d %s\n", pa->buf, pa, (char *)pa->buf - (char *)pa, pa->buf);
}
Second
typedef struct A
{
int i;
char *buf; //Here
}A;
The first is a C99 'flexible array member'. The second is the reliable fallback for when you don't have C99 or later.
With a flexible array member, you allocate the space you need for the array along with the main structure:
A *pa = malloc(sizeof(A) + strlen(string) + 1);
pa->i = index;
strcpy(pa->buf, string);
...use pa...
free(pa);
As far as the memory allocation goes, the buf member has no size (so sizeof(A) == sizeof(int) unless there are padding issues because of array alignment — eg if you had a flexible array of double).
The alternative requires either two allocations (and two releases), or some care in the setup:
typedef struct A2
{
int i;
char *buf;
} A2;
A2 *pa2 = malloc(sizeof(A2));
pa2->buff = strdup(string);
...use pa2...
free(pa2->buff);
free(pa2);
Or:
A2 *pa2 = malloc(sizeof(A2) + strlen(string) + 1);
pa2->buff = (char *)pa2 + sizeof(A2);
...use pa2...
free(pa2);
Note that using A2 requires more memory, either by the size of the pointer (single allocation), or by the size of the pointer and the overhead for the second memory allocation (double allocation).
You will sometimes see something known as the 'struct hack' in use; this predates the C99 standard and is obsoleted by flexible array members. The code for this looks like:
typedef struct A3
{
int i;
char buf[1];
} A3;
A3 *pa3 = malloc(sizeof(A3) + strlen(string) + 1);
strcpy(pa3->buf, string);
This is almost the same as a flexible array member, but the structure is bigger. In the example, on most machines, the structure A3 would be 8 bytes long (instead of 4 bytes for A).
GCC has some support for zero length arrays; you might see the struct hack with an array dimension of 0. That is not portable to any compiler that is not mimicking GCC.
It's called the 'struct hack' because it is not guaranteed to be portable by the language standard (because you are accessing outside the bounds of the declared array). However, empirically, it has 'always worked' and probably will continue to do so. Nevertheless, you should use flexible array members in preference to the struct hack.
ISO/IEC 9899:2011 §6.7.2.1 Structure and union specifiers
¶3 A structure or union shall not contain a member with incomplete or function type (hence,
a structure shall not contain an instance of itself, but may contain a pointer to an instance
of itself), except that the last member of a structure with more than one named member
may have incomplete array type; such a structure (and any union containing, possibly
recursively, a member that is such a structure) shall not be a member of a structure or an
element of an array.
¶18 As a special case, the last element of a structure with more than one named member may
have an incomplete array type; this is called a flexible array member. In most situations,
the flexible array member is ignored. In particular, the size of the structure is as if the
flexible array member were omitted except that it may have more trailing padding than
the omission would imply. However, when a . (or ->) operator has a left operand that is
(a pointer to) a structure with a flexible array member and the right operand names that
member, it behaves as if that member were replaced with the longest array (with the same
element type) that would not make the structure larger than the object being accessed; the
offset of the array shall remain that of the flexible array member, even if this would differ
from that of the replacement array. If this array would have no elements, it behaves as if
it had one element but the behavior is undefined if any attempt is made to access that
element or to generate a pointer one past it.
struct A {
int i;
char buf[];
};
does not reserve any space for the array, or for a pointer to an array. What this says is that an array can directly follow the body of A and be accessed via buf, like so:
struct A *a = malloc(sizeof(*a) + 6);
strcpy(a->buf, "hello");
assert(a->buf[0] == 'h');
assert(a->buf[5] == '\0';
Note I reserved 6 bytes following a for "hello" and the nul terminator.
The pointer form uses an indirection (the memory could be contiguous, but this is neither depended on nor required)
struct B {
int i;
char *buf;
};
/* requiring two allocations: */
struct B *b1 = malloc(sizeof(*b1));
b1->buf = strdup("hello");
/* or some pointer arithmetic */
struct B *b2 = malloc(sizeof(*b2) + 6);
b2->buf = (char *)((&b2->buf)+1);
The second is now laid out the same as a above, except with a pointer between the integer and the char array.