Suppose I have structure in C like this
struct A {
int len;
char s[1];
}
I want to have an array of above structure but char s[1] member of struct A can be of variable length. How can we do this? Even struct hack trick in C99 doesn't seem to work here. One solution is to have char * as a last member and do dynamic memory allocation, but I want all data of struct to be in contiguous locations as my implementation needs to be cache oblivious.
You can't have an array of variable size objects, so you can't have an array of structures using the struct hack. All the objects in an array must be the same size. And if they're all the same size, the size must be implied by the structure, so you won't be using the struct hack after all; there'll be a size other than 1 in the dimension of the array s in your structure (unless 1 is big enough for everything). The reason is that the storage location for a[i] (where a is the name of an array and i is an index into the array) must be computable as 'byte address of a plus (i times size of one object in array)'. So the size of the objects in the array (in this case, structures) must be known and fixed.
As an alternative, you can have an array of pointers to variable size objects; you simply have to arrange to allocate each object separately with the appropriate size, and save the pointer to that in the array.
Note that C99 does away with the 'struct hack' (which was never officially portable, though in practice it was) and introduces 'flexible array members' instead:
struct A {
int len;
char data[];
};
However, the advice above still applies.
If there is a maximum size for "s", you could use that instead of [1]. That keeps everything contiguous.
If you really don't want to use dynamic memory, then you can't do it with an array. You need your own "manager" that will use the struct hack trick on each member individually - but that means you can't do indexed lookups - you have to look at each element to see how big it is and jump the right number of bytes to the next element.
In C, array indexing involves multiplying the base address by the compile-time-constant size of an individual element. For that reason, you can't use inbuilt array support directly with the "struct hack", as each s element will be allocated exactly the 1 byte you request, and indices further past the struct will access following S elements in the array (or go off the end completely, possibly crashing).
If you really need contiguous data for cache-access speed, you can pack it yourself, you can solve this (like most things) with an indirection... have a contiguous array of S*, and manually pack your data into another contiguous buffer (malloc() or stack-allocate enough memory for all your S objects including the real data size of all s[] members). Your performance may suffer (or your OS crash) if the int len elements aren't optimally (properly) aligned for your architecture, so you may need to manually pad between S instances.
S* index[100] char data[10000];
(S*)(data) --------------> S with 14-byte s[] using data[0]..[17]
(S*)(data + 20) -----\ 2 byte padding so next S is 4-byte aligned
(S*)(data + 32) --\ \---> S with 7-byte s[] using data[20]..[30]
\ 1 byte padding...
\-----> ...
Unfortunately, this is quite an inflexible data layout - you can't just grow the amount of data in an element's s member without schuffling all the other data out of the way and patching the index, but that's normal for arrays so if you were already considering using them then perhaps this will suit you. Another hassle is calculating the total size of S structs (including s[] and any padding) up front....
Related
I'm quit confused with the difference between flexible arrays and pointer as struct members. Someone suggested, struct with pointers need malloc twice. However, consider the following code:
struct Vector {
size_t size;
double *data;
};
int len = 20;
struct Vector* newVector = malloc(sizeof *newVector + len * sizeof*newVector->data);
printf("%p\n",newVector->data);//print 0x0
newVector->data =(double*)((char*)newVector + sizeof*newVector);
// do sth
free(newVector);
I find a difference is that the address of data member of Vector is not defined. The programmer need to convert to "find" the exactly address. However, if defined Vector as:
struct Vector {
size_t size;
double data[];
};
Then the address of data is defined.
I am wondering whether it is safe and able to malloc struct with pointers like this, and what is the exactly reason programmers malloc twice when using struct with pointers.
The difference is how the struct is stored. In the first example you over-allocate memory but that doesn't magically mean that the data pointer gets set to point at that memory. Its value after malloc is in fact indeterminate, so you can't reliably print it.
Sure, you can set that pointer to point beyond the part allocated by the struct itself, but that means potentially slower access since you need to go through the pointer each time. Also you allocate the pointer itself as extra space (and potentially extra padding because of it), whereas in a flexible array member sizeof doesn't count the flexible array member. Your first design is overall much more cumbersome than the flexible version, but other than that well-defined.
The reason why people malloc twice when using a struct with pointers could either be that they aren't aware of flexible array members or using C90, or alternatively that the code isn't performance-critical and they just don't care about the overhead caused by fragmented allocation.
I am wondering whether it is safe and able to malloc struct with pointers like this, and what is the exactly reason programmers malloc twice when using struct with pointers.
If you use pointer method and malloc only once, there is one extra thing you need to care of in the calculation: alignment.
Let's add one extra field to the structure:
struct Vector {
size_t size;
uint32_t extra;
double *data;
};
Let's assume that we are on system where each field is 4 bytes, there is no trailing padding on struct and total size is 12 bytes. Let's also assume that double is 8 bytes and requires alignment to 8 bytes.
Now there is a problem: expression (char*)newVector + sizeof*newVector no longer gives address that is divisible by 8. There needs to be manual padding of 4 bytes between structure and data. This complicates the malloc size calculation and data pointer offset calculation.
So the main reason you see 1 malloc pointer version less, is that it is harder to get right. With pointer and 2 mallocs, or flexible array member, compiler takes care of necessary alignment calculation and padding so you don't have to.
What is the difference between using flexible array member (FAM) or pointer member ? In the two cases, a malloc and an affectation element by element must be done. But with FAM, a memory allocation is done for the whole structure and with ptr member, a memory allocation is done for the ptr member only (see code). What are the pros ans the cons of these two methods ?
#include <stdio.h>
#include <stdlib.h>
typedef struct farr_mb {
int lg;
int arr[];
} Farr_mb;
typedef struct ptr_mb {
int lg;
int * ptr;
} Ptr_mb;
int main() {
int lg=5;
Farr_mb *a=malloc(sizeof(Farr_mb)+lg*sizeof(int));
Ptr_mb b; b.ptr=malloc(lg*sizeof(int));
for (int i=0;i<lg;i++) (a->arr)[i]=i;
for (int i=0;i<lg;i++) (b.ptr)[i]=i;
for (int i=0;i<lg;i++) printf("%d \t",(a->arr)[i]=i);
printf("\n");
for (int i=0;i<lg;i++) printf("%d \t",(b.ptr)[i]=i);
return 0;
}
Before we get to the pros and cons, let's look at some real-world examples.
Let's say we wish to implement a hash table, where each entry is a dynamically managed array of elements:
struct hash_entry {
size_t allocated;
size_t used;
element array[];
};
struct hash_table {
size_t size;
struct hash_entry **entry;
};
#define HASH_TABLE_INITIALIZER { 0, NULL }
This in fact uses both. The hash table itself is a structure with two members. The size member indicates the size of the hash table, and the entry member is a pointer to an array of hash table entry pointers. This way, each unused entry is just a NULL pointer. When adding elements to a hash table entry, the entire struct entry can be reallocated (for sizeof (struct entry) + allocates * sizeof (element) or freed, as long as the corresponding pointer in the entry member in the struct hash_table is updated accordingly.
If we used element *array instead, we would need use struct hash_entry *entry: in the struct hash_table; or allocate the struct hash_entry separately from the array; or allocate both struct hash_entry and array in the single chunk, with the array pointer pointing just after the same struct hash_entry.
The cost of that would be two extra size_ts worth of memory used for each unused hash table slot, as well as an extra pointer dereference when accessing elements. (Or, to get the address of the array, two consecutive pointer dereferences, instead of one pointer dereference plus offset.) If this is a key structure heavily used in an implementation, that cost can be visible in profiling, and negatively affect cache performance. For random accesses, the larger the element array is, the less difference there is, however; the cost is largest when the arrays are small, and fit within the same cacheline (or a few cachelines) as the allocated and used members.
We do not usually want to make the entry member in the struct hash_table a flexible array member, because that would mean you no longer can declare a hash table statically, using struct hash_table my_table = HASH_TABLE_INITIALIZER;; you would need to use a pointer to a table, and an initializer function: struct hash_table *my_table; my_table = hash_table_init(); or similar.
I do have another example of related data structures using both pointer members and flexible array members. It allows one to use variables of type matrix to represent any 2D matrix with double entries, even when a matrix is a view to another (say, a transpose, a block, a row or column vector, or even a diagonal vector); these views are all equal (unlike in e.g. GNU Scientific Library, where matrix views are represented by a separate data type). This matrix representation approach makes writing robust numerical linear algebra code easy, and the ensuing code is much more readable than when using GSL or BLAS+LAPACK. In my opinion, that is.
So, let's look at the pros and cons, from the point of view of how to choose which approach to use. (For that reason, I will not designate any feature as "pro" or "con", as the determination depends on the context, on each particular use case.)
Structures with flexible array members cannot be initialized statically. You can only refer to them via pointers.
You can declare and initialize structures with pointer members. As shown in above example, using a preprocessor initializer macro can mean you do not need an initializer function. For example, a function accepting a struct hash_table *table parameter can always resize the array of pointers using realloc(table->entry, newsize * sizeof table->entry[0]), even when table->entry is NULL. This reduces the number of functions needed, and simplifies their implementation.
Accessing an array via a pointer member can require an extra pointer dereference.
If we compare the accesses to arrays in statically initialized structures with pointer to the array, to a structure with a flexible array member referred via a static pointer, the same number of dereferences are made.
If we have a function that gets the address of a structure as a parameter, then accessing an array element via a pointer member requires two pointer dereferences, whereas accessing a flexible array element requires only one pointer dereference and one offset. If the array elements are small enough and the array index small enough, so that the accessed array element is in the same cacheline, the flexible array member access is often significantly faster. For larger arrays, the difference in performance tends to be insignificant. This does vary between hardware architectures, however.
Reallocating an array via a pointer member hides the complexity from those using the structure as an opaque variable.
This means that if we have a function that receives a pointer to a structure as a parameter, and that structure has a pointer to a dynamically allocated array, the function can reallocate that array without the caller seeing any change in the structure address itself (only structure contents change).
However, if we have a function that receives a pointer to a structure with a flexible array member, reallocating the array means reallocating the entire structure. That potentially modifies the address of the structure. Because the pointer is passed by value, the modification is not visible to the caller. Thus, a function that may resize a flexible array member, must receive a pointer to a pointer to the structure with a flexible array member.
If the function only examines the contents of a structure with a flexible array member, say counts the number of elements that fulfill some criteria, then a pointer to the structure suffices; and both the pointer and the pointed-to data can be marked const. This might help the compiler produce better code. Furthermore, all the data accessed is linear in memory, which helps more complex processors manage caching more efficiently. (To do the same with an array having a pointer member, one would need to pass the pointer to the array, as well as the size field at least, as parameters to the counting function, instead of a pointer to the structure containing those values.)
An unused/empty structure with a flexible array member can be represented by a NULL pointer (to such structure). This can be important when you have an array of arrays.
With structures with flexible array members, the outer array is just an array of pointers. With structures with pointer members, the outer array can be either an array of structures, or an array of pointers to structures.
Both can support different types of sub-arrays, if the structures have a common type tag as the first member, and you use an union of those structures. (What 'use' means in this context, is unfortunately debatable. Some claim you need to access the array via the union, I claim the visibility of such an union is sufficient because anything else will break a huge amount of existing POSIX C code; basically all server-side C code using sockets.)
Those are the major ones I can think of right now. Both forms are ubiquitous in my own code, and I have had no issues with either. (In particular, I prefer using a structure free helper function that poisons the structure to help detect use-after-free bugs in early testing; and my programs do not often have any memory-related issues.)
I will edit the above list, if I find I've missed important facets. Therefore, if you have a suggestion or think I've overlooked something above, please let me know in a comment, so I can verify and edit as appropriate.
I want to create a struct Ring with nestd zero lenght array:
typedef struct Data_Block
{
size_t Data_Len;
char Buf[0];
}Block;
typedef struct Block_Ring
{
int head;
int tail;
int full;
int block_num;
Block blk[0];
}Ring;
How can I correctly allocate memory for a Ring which contains 32 Block, and one Block contains Buf of size 16? Because if I malloc with the right size, the number of Block will become just one.
Fundamental Problem
There is a fundamental problem you must deal with before addressing this task: Constructing an array requires having elements of a fixed and known size.
That is because array element i is located by adding i times the size of an element to the base address of the array. One can perform that calculation only if the size of an element exists (elements have a fixed size) and you know it (the size is known).
Although you define Block to contain a member of zero size (Buf is an array with zero elements), you intend to use it as if that member were 16 bytes (an array of 16 char). However, there is no way to tell the compiler that the Block objects you will allocate and use are actually Block objects with 16 extra bytes. You certainly can allocate space for them, and I will show you how, but then how do you intend to use them? If x is a Ring object, and you write x.blk[i], the compiler will generate code that multiplies i by what it thinks the size of a Block is, and that will be wrong because the compiler thinks a Block has zero bytes for Buf, but your Block objects are bigger.
Standard C Versus An Extension
Declaring a structure member as an array with zero elements is an extension (notably available in GCC). The 1999 C standard introduced a similar feature called flexible array members. With standard C, a flexible array member is declared with no dimension, rather than a zero dimension.
A flexible array member is an incomplete type (C 2018 6.7.2.1 18). In other words, the type is not fully specified. The number of members of the array is unknown, and so the total size of the array is unknown.
Then, in defining Ring, we cannot define the blk member to be a flexible array member that is an array of Block, because standard C requires that the element type of an array be a complete type (C 2018 6.7.6.2 1, “The element type shall not be an incomplete or function type”).
Therefore, this code cannot be made into standard C. This is actually an advantage: The C standard is preventing you from making the fundamental mistake above of creating an array that cannot work because the size of its elements is not known.
Oddly, GCC 8.1 for x86-64 fails to give a diagnostic for this. It should give a diagnostic for the constraint violation. Apple LLVM version 9.1.0 (clang-902.0.39.2) does issue a diagnostic.
However, we will proceed to consider the code as you have written it, using the language extension.
How Big Are The Elements?
When a C implementation lays out a structure, it must ensure that each member in the structure is correctly aligned. (What alignments are correct is defined by the implementation, so they vary. However, whatever they are, the compiler must lay out the structure accordingly.) Since structures can be used as elements of an array, the size of the laid-out structure must be such that when one structure follows another in the array, all the members in the following structure are also correctly aligned.
Satisfying this constraint requires that the size of the structure be a multiple of the alignment requirements of all members. For example, if there are members with alignment requirements of 4 bytes and 8 bytes, the size of the structure must be a multiple of 8 bytes, since that is the least common multiple of 4 bytes and 8 bytes. In fact, all alignment requirements are powers of two, so the least common multiple of all the alignment requirements is simply the largest (most restrictive) alignment requirement.
What this means is that, when allocating space for an array of your Block objects, you cannot simply use an arbitrary number of bytes for the extra Buf elements. You must ensure the total size of each Block object is a multiple of the alignment requirement of its members.
C provides a way to know the alignment requirement of the structure. The expression _Alignof(Block) is the alignment requirement. So, if you want each Block to have x elements in Buf, the size you need for each Block is the size of the base structure (sizeof(Block)) plus the size you need for the actual array elements (x * sizeof(char)) plus enough padding to round the total up to a multiple of the alignment requirement. You can calculate this with:
// Calculate desired space.
size_t S = sizeof(Block) + x * sizeof(char);
// Note the alignment requirement.
static const size_t A = _Alignof(Block);
// Round up to multiple of alignment requirement.
S = (S-1) / A * A + 1;
(This is a well-known expression for rounding up to a multiple of A. You can tinker with some examples to see why it works.)
Once you have calculated the space needed for one Block using the above code (with 16 for x), you can allocate space for one Ring with an array of 32 of these Block using:
Ring *R = malloc(sizeof(Ring) + 32 * S);
Accessing Array Elements
Now that you have the space, how do you access members of blk? As discussed above, the compiler does not know how to do this. Unfortunately, C does not provide any assistance. You will have to calculate addresses manually. Since you know the size of each of your Block objects, S, you can calculate the address of the Block with index i with:
Block *B = (Block *) ((char *) R->blk + S*i);
Discussion
This is cumbersome and error-prone. The address calculation could be wrapped into a helper function to make it a little better. However, it is generally not a good idea to use complicated code like this. You ought to consider alternative solutions.
What is the benefit of declaring a C structure member as in array of size 1 instead of a pointer :
struct {
a_struct_t a_member[1];
...
}b_struct;
Thanks in advance
In a typical case, a structure with a member that's declared as an array of one item will have that member as the last item in the struct. The intent is that the struct will be allocated dynamically. When it is allocated, the code will allocate space for as many items as you really want/need in that array:
struct X {
time_t birthday;
char name[1];
};
struct X *x = malloc(sizeof(*x) + 35);
x->birthday = mktime(&t);
strcpy(x->name, "no more than 35 characters");
This works particularly well for strings -- the character you've allocated in the struct gives you space for the NUL terminator, so when you do the allocation, the number of characters you allocate is exactly the strlen() of the string you're going to put there. For most other kinds of items, you normally want to subtract one from the allocation size (or just live with the allocated space being one item larger than is strictly necessary).
You can do (sort of) the same thing with a pointer, but it results in allocating the body of the struct separately from the item you refer to via the pointer. The good point is that (unlike the method above) more than one item can be allocated dynamically, where the method above only works for the last member of the struct.
What you describe are two different things entirely. If you have a pointer as a member:
a_struct_t* a_member;
then it is simply a pointer. There is no memory allocated inside of the struct to hold an a_struct_t. If, on the other hand, you have an array of size 1:
a_struct_t a_member[1];
then your struct actually has an object of type a_struct_t inside of it. From a memory standpoint, it isn't much different from just putting an object of that type inside the struct:
a_struct_t a_member;
From a usage standpoint, an array requires indirection to access the one element (i.e., you need to use *a_member instead of a_member).
"Array of size 1 instead of a pointer"? Sorry, but I don't see how this quiestion can possibly make sense. I would understand if you asked about "array of size 1 instead of an ordinary member (non-array)". But "instead of a pointer"? What does pointer have to do with this? How is it interchangeable with an array, to justify the question?
If what you really wanted to ask is why it is declared as an array of size 1 instead of non-array as in
struct {
a_struct_t a_member;
} b_struct;
then one possible explanation is the well-known idiom called "struct hack". You might see a declaration like
struct {
...
a_struct_t a_member[1];
} b_struct;
used to implement an array of flexible size as the last member of the struct object. The actual struct object is later created within a memory block that is large enough to accomodate as many array elements as necessary. But in this case the array has to be the last member of the struct, not the first one as in your example.
P.S. From time to time you might see "struct hack" implemented through an array of size 0, which is actually a constraint violation in C (i.e. a compile error).
So I think it's been stated that the main difference between pointers and arrays is that you have to allocate memory for pointers.
The tricky part about your question is that even as you allocate space for your struct, if your struct contains a pointer you have to allocate a SECOND time for the pointer, but the pointer itself would be allocated as part of the struct's allocaiton.
If your struct contained an array of 1 you would not have to allocate any additional memory, it would be stored in the struct (which you still have to allocate).
These are different things.
Such member's name is an address of allocated memory, allocated inside the struct instance itself.
I heard a rumor that, in C, arrays that are contained inside structs may have padding added in between elements of the array. Now obviously, the amount of padding could not vary between any pair of elements or calculating the next element in an array is not possible with simple pointer arithmetic.
This rumor also stated that arrays which are not contained in structures are guaranteed to contain no padding. I know at least that part is true.
So, in code, the rumor is:
{
// Given this:
struct { int values[20]; } foo;
int values[20];
// This may be true:
sizeof(values) != sizeof(foo.values);
}
I'm pretty certain that sizeof(values) will always equal sizeof(foo.values). However, I have not been able to find anything in the C standard (specifically C99) that explicitly confirms or denies this.
Does anyone know if this rumor is addressed in any C standard?
edit: I understand that there may be padding between the end of the array foo.values and the end of the struct foo and that the standard states that there will be no padding between the start of foo and the start of foo.values. However, does anyone have a quote from or reference to the standard where it says there is no padding between the elements of foo.values?
No, there will never be padding in between elements of an array. That is specifically not allowed. The C99 standard calls array types "An array type describes a contiguously allocated nonempty set of objects...". For contrast, a structure is "sequentially", not "contiguously" allocated.
There might be padding before or after an array within a structure; that is another animal entirely. The compiler might do that to aid alignment of the structure, but the C standard doesn't say anything about that.
Careful here. Padding may be added at the end of the struct, but will not be added between the elements of the array as you state in your question. Arrays will always reference contiguous memory, though an array of structures may have padding added to each element as part of the struct itself.
In your example, the values and foo.values arrays will have the same size. Any padding will be part of the struct foo instead.
Here's the explanation as to why a structure may need padding between its members or even after its last member, and why an array doesn't:
Different types might have different alignment requirements. Some types need to be aligned on word boundaries, others on double or even quad word boundaries. To accomplish this, a structure may contain padding bytes between its members. Trailing padding bytes might be needed because the memory location directly ofter a structure must also conform to the structure's alignment requirements, ie if bar is of type struct foo *, then
(struct foo *)((char *)bar + sizeof(struct foo))
yields a valid pointer to struct foo (ie doesn't fail due to mis-alignment).
As each 'member' of an array has the same alignment requirement, there's no reason to introduce padding. This holds true for arrays contained in structures as well: If an array's first elment is correctly aligned, so are all following elements.
Yes, sort of. Variables are often aligned to some boundry, depending on the variable. Take the following, for instance:
typedef struct
{
double d;
char c;
} a_type_t;
double and char are 8 and 1 bytes, on my system, respectively. Total of 9. That structure, however, will be 16 bytes, so that the doubles will always be 8-byte aligned. If I had just used ints, chars, etc, then the alignment might be 1, 2, 4, or 8.
For some type T, sizeof(T) may or may not equal sizeof(T.a) + sizeof(T.b) + sizeof(T.c) ... etc.
Generally, this is entirely compiler and architecture dependent. In practice, it never matters.
Consider:
struct {
short s;
int i;
} s;
Assuming shorts are 16 bits and you're on 32 bits, the size will probably be 8 bytes as each struct members tends to be aligned a word (32 bit in this case) boundary. I say "probably" because it is implementation specific behaviour that can be varied by compiler flags and the like.
It's worth stressing that this is implementation behaviour not necessarily defined by the C standard. Much like the size of shorts, ints and longs (the C standard simply says shorts won't be larger than ints and longs won't be smaller than ints, which can end up as 16/32/32, 16/32/64, 32/32/64 or a number of other configurations).