It is clear that arrays in C cannot insert padding between their elements. However, is there any rule saying that they can't add trailing padding at the end of the array as a whole?
i.e. is this program guaranteed to give the same results everywhere?
#include <stdio.h>
int main(void) {
typedef char a[3];
typedef a b[3];
printf("%zu %zu\n", sizeof(a), sizeof(b)); // -> 3 9
}
As far as I can work out, adding a trailing byte or five to the size of a, perhaps in a misguided optimization attempt, wouldn't break the array access rules (b[1][1] still maps exactly to *(&b + sizeof(a) * 1 + 1) regardless of the size of its contained a objects, and accessing beyond the length of a contained a is UB anyway).
I can't find anywhere in the C standard where it actually says outright that the size of an array is the size of the element type multiplied by the number of elements. 6.5.3.4 only says that sizeof returns the "number of bytes" in the array (it does give sizeof array / sizeof array[0] as a code example, but it's only an example - it doesn't say it has to work, and it doesn't give any details).
The implicit guarantee is useful for writing portable code that depends on exact data layouts, e.g. passing packed RGB values:
typedef uint8_t RGB[3];
RGB * data = ...;
glColorPointer(3, GL_UNSIGNED_BYTE, 0, data);
(OK so OpenGL can accept stride values so this is a bad example, but you get the point)
For that matter, I assume from the widespread notion (even to the example in the standard) that you can get the number of elements of an array with sizeof, that this is likely to hold true everywhere anyway - are there any known situations where it isn't?
I believe it was never considered necessary for the standard to actually spell out that arrays don't have padding, for the simple reason that there is absolutely no reason why such padding might ever be useful on any implementation.
That said, I do believe the standard forbids such padding, through the description of the == operator.
6.5.9 Equality operators
Semantics
6 Two pointers compare equal if and only if [...] or one is a pointer
to one past the end of one array object and the other is a pointer to the start of a different array object that happens to immediately follow the first array object in the address space.
Given
int array[2][2];
the expression &array[0][2] points is a pointer one past the end of the first array subobject. &array[1][0] is a pointer to the second array subobject, which immediately follows the first array in memory. These pointers are required to compare equal. If int[2] had trailing padding, if sizeof(int[2]) > 2 * sizeof(int), I cannot imagine how any implementation could make the two pointers compare as equal.
Related
Take a look at the following code, taken from an older version of ffmpeg:
#include <stdio.h>
#include <stdint.h>
#include <stdlib.h>
struct foo
{
int16_t (*ac_val_base)[16];
int16_t (*ac_val[3])[16];
};
int main(int argc, char *argv[])
{
struct foo bar;
int16_t *ac_val, *ac_val1;
bar.ac_val_base = malloc(4639 * 16 * sizeof(int16_t));
bar.ac_val[0] = bar.ac_val_base + 66;
ac_val = bar.ac_val[0][0] + 3780 * 16;
ac_val1 = ac_val;
printf("Result: %d\n", (int) (((char *) ac_val1) - ((char *) bar.ac_val[0][0])));
return 0;
}
When compiling this with established compilers like gcc or Visual C, the result is 120960. This makes sense to me because I'm adding 3780 * 16 to an int16_t array pointer so I'd expect the resulting pointer to be 120960 bytes above the source pointer.
When compiling the code using vbcc, however, the result is -8000 because the compiler performs some optimizations. The author of the vbcc compiler is convinced that the optimization is covered by 6.5.6/8 of the C99 standard which says that the behaviour is undefined in that case, quote:
If both the pointer operand and the result point to elements of the
same array object, or one past the last element of the array object,
the evaluation shall not produce an overflow; otherwise, the behavior
is undefined.
So is the code above really relying on undefined behaviour? I'm a bit skeptical because the code works on all compilers except vbcc.
The short answer is that the type of the expression bar.ac_val[0][0] is "array of 16 int16_t". Although this array object is located within a larger malloc block, and the expression evaluates to a pointer within the block, the pointer has provenance from an array.
A pointer obtained from an array expression, where the array dimension is N, can be displaced by at most N (one byte past the end of the array), while staying within defined behavior. (If displaced all the way to N, the pointer must not be dereferenced.)
A simpler example is something like:
struct obj {
int arr[32];
int other_member;
};
Suppose you have a malloc-ed pointer to this, but use ptr->arr[32] to access other_member, this is not well-defined, even though everything is in the malloc-ed object.
One possible optimization the compiler can perform is to use some addressing mode which only works for that size of array. Say that ptr->arr[i] translates to some instruction which has a five-bit field to encode a scaled displacement value from 0 to 31. The compiler is free to ignore that the displacement [32] cannot fit into that instruction, and just truncate it to the lowest five bits, which are zero, effectively changing the meaning to ptr->arr[0].
Alternatively, the rules can enable useful diagnostic tools. The compiler may be able to warn you at compile time that there is an array overrun, and because it's undefined behavior, it can fail the translation, while remaining conforming. There can be tooling whereby the code is compiled in such a way that you get detailed array bound checking at run time (not just checking for overrun of the malloc-ed block). Accessing past the end of the array can be an accident, resulting in a hard-to-find bug, particularly if the access doesn't go past the allocation.
In
ac_val = bar.ac_val[0][0] + 3780 * 16;
bar.ac_val[0][0] is int16_t[16], so that adding anything other than values in range [0, 16) to it results in undefined behaviour.
The reason for the undefined behaviour is segmented memory model (as opposed to modern flat/linear memory model) which C is still compatible with, when pointer values are composed of segment descriptor and byte offset within the segment. In such a model, distinct arrays may reside in different segments. Segment descriptor units are not byte offsets, so that subtracting segment descriptor values doesn't produce distance in bytes. The difference between pointers to different arrays residing in different segments ends up subtracting segment descriptors resulting in undefined behaviour.
Your particular array is allocated using malloc. It cannot possibly span multiple memory segments. As long as your pointers, including expression temporaries, don't point outside this heap allocated array, these pointers are valid and well-defined.
It is the array element type int16_t[16] and indexing outside its bounds what causes the undefined behaviour. This array element type is essentially a red herring for a C compiler.
If you switch your array element type to plain int16_t and convert your 2d array indexes into 1d, e.g. [row][column] to [row * n_columns + column], this problem ceases to exist.
You can also side-step the undefined behaviour arising from pointer arithmetic with integer arithmetic:
uintptr_t ac_val = (uintptr_t)bar.ac_val[0][0] + 3780 * 16 * sizeof(int16_t);
printf("Result: %zu\n", (size_t) ((ac_val - ((uintptr_t) bar.ac_val[0][0])));
This relies on the facts that:
Converting a pointer to uintptr_t and back is well-defined.
Unsigned integer addition and subtraction is well-defined.
Some code flattens multidimensional arrays like this:
int array[10][10];
int* flattened_array = (int*)array;
for (int i = 0; i < 10*10; ++i)
flattened_array[i] = 42;
This is, as far as I know, undefined behaviour.
I am trying to detect cases like this with gcc sanitizers, however, neither -fsanitize=address nor -fsanitize=undefined work.
Is there a sanitizer option that I'm missing, or perhaps a different way to detect this at run time? Or maybe I am mistaken and the code is legal?
Edit: the sanitizers detect this access as an error:
array[0][11] = 42;
but do not detect this:
int* first_element = array[0];
first_element[11] = 42;
Furthermore, clang detects the first access statically, and gives out a warning
warning: array index 11 is past the end of the array (which contains 10 elements) [-Warray-bounds]
Edit: the above does not change if int in the declaration is replaced with char.
Edit: There are two potential sources of UB.
Accessing an object (of type int[10]) through an lvalue of an incompatible type (int).
Out-of-bounds access with a pointer of type int* and an index >=10 where the size of the underlying array is 10 (rather than 100).
Sanitizers don't seem to detect the first kind of violation. There's a debate whether this is a violation at all. After all, there's also an object of type int at the same address.
As for the second potential UB, the UB sanitizer does detect such access, but only if it is done directly via the 2D array itself and not via another variable that points to its first element, as shown above. I don't think the two accesses should differ in legality. They should be either both legal (and then ubsan has a false positive) or both illegal (and then ubsan has a false negative).
Edit: Appendix J2 says array[0][11] should be UB, even though it is only informative.
From a language lawyer point of view, this is generally seen as invalid code because the integers arrays are only of size 10 and the code does access past the declared array size. Yet it used to be a common idiom, and I know no compiler that would not accept it. Still with all real world compilers I know, the resulting program will have the expected behaviour.
After a second (in reality much more) reading of the C11 standard draft (n1570) the intent of the standard is still not clear. 6.2.5 Types § 20 says:
An array type describes a contiguously allocated nonempty set of objects with a
particular member object type, called the element type.
It makes clear that an array contains contiguously allocated objects. But IMHO is unclear about whether a contiguously allocated set of objects is an array.
If you answer no, then the shown code does invoke UB by accessing an array past it last element
But if you answer yes, then a set of 10 contiguous sets of 10 contiguous integers gives 100 contiguous integers and can be seen as an array of 100 integers. Then the shown code would be legal.
That latter acception seems to be common in the real word because it is consistent with dynamic array allocation: you allocate enough memory for a number of objects, and you can access that as if it had been declared as an array - and the allocation function ensures no alignment problem.
My conclusion so far is:
is it nice and clean code: certainly not and I would avoid it in production code
does it invokes UB: I really do not know and my personal opinion is probably no
Let us look at the code added in the edit:
array[0][11] = 42;
The compiler knows that array is declared as int[10][10]. So it knows that both indexes must be less than 10, and it can raise a warning.
int* first_element = array[0];
first_element[11] = 42;
first_element is declared as a mere pointer. Statically, the compiler has to assume that it can point inside an array of unknown size, so outside of a specific context, it is much harder to raise a warning. Of course for a human programmer it is evident that both way should be seen the same, but as a compiler is not required to emit any diagnostic for out of bounds array, efforts to detect them are left to the minimum and only trivial cases are detected.
In addition, when a compiler internally codes pointer arithmetics on common platforms, it just computes a memory address which is the original address and a byte offset. So it could emit the same code as:
char *addr = (char *) first_element; // (1)
addr += 11 * sizeof(int); // (2)
*((int *) addr) = 42; // (3)
(1) is legal because a pointer to any objet (here an int) can be converter to a pointer to char, which is required to point to the first byte of the representation of the object
(2) the trick here is that (char *) first_element is the same as (char *) array because the first byte of the 10*10 array is the first byte of the first int of the first row, and an single byte can only have one single address. As the size of array is 10 * 10 * sizeof(int), 11 * sizeof(int) is a valid offset in it.
(3) for the very same reason, (char *) &array[1][1] is addr because elements in an array are contiguous so their byte representation are also contiguous. And as a forth and back conversion between 2 types is legal and required to give back the original pointer, (int *) addr is (int*) ((char*) &array[1][1]). That means that dereferencing (int *) addr is legal and shall have the same effect as array[1][1] = 42.
This does not mean that first_element[11] does not involve UB. array[0] has a declared size which is 10. It just explains why all known compilers accepts it (in addition to not wanting to break legacy code).
The sanitizers are not especially good at catching out-of-bounds access unless the array in question is a complete object.
For example, they do not catch out-of-bounds access in this case:
struct {
int inner[10];
char tail[sizeof(int)];
} outer;
int* p = outer.inner;
p[10] = 42;
which is clearly illegal. But they do catch access to p[11].
Array flattening is not really different in spirit from this kind of access. Code generated by the compiler, and the way it is instrumented by sanitizers, should be pretty similar. So there's little hope that array flattening can be detected by these tools.
Multidimensional arrays are required to be contiguously allocated (C uses row-major). And there can't be any padding between elements of an array - though not stated explicitly in the standard, this can be inferred with array definition that says "contiguously allocated nonempty set of objects" and the definition of sizeof operator.
So the "flattening" should be legal.
Re. accessing array[0][11]: although, Annex J2 directly gives an example, what exactly is the violation in the normative isn't obvious. Nevertheless, it's still possible to make it legal an intermediate cast to char*:
*((int*)((char*)array + 11 * sizeof(int))) = 42;
(writing such code is obviously not advised ;)
The problem here is that there Standard describes as equivalent two operations, one of which clearly should be defined and one of which the Standard expressly says is not defined.
The cleanest way to resolve this, which seems to coincide with what clang and gcc already do, which is to say that applying [] operator to an array lvalue or non-l value does not cause it to decay, but instead looks up an element directly, yielding an lvalue if the array operand was an lvalue, and a non-l value otherwise.
Recognizing the use of [] with an array as being a distinct operator would clean up a number of corner cases in the semantics, including accessing an array within a structure returned by a function, register-qualified arrays, arrays of bitfields, etc. It would also make clear what the inner-array-subscript limitations are supposed to mean. Given foo[x][y], a compiler would be entitled to assume that y would be within the bounds of the inner array, but given *(foo[x]+y) it would not be entitled to make such an assumption.
Is subtraction of pointers not pointing to different elements of same array valid in C?
Is something such as below guaranteed to work according to C Standards? I vaguely remember reading that this is not valid?
int * a;
int * b;
a = (int*) 100;
b = (int*) 200;
printf("%d\n", b-a);
Will this give me 25.
From the C spec, Appendix J.2 Undefined behaviour:
Pointers that do not point into, or just beyond, the same array object are subtracted (6.5.6).
6.5.6 Additive operators, paragraph 9 says:
When two pointers are subtracted, both shall point to elements of the same array object, or one past the last element of the array object; the result is the difference of the subscripts of the two array elements.
There you have it - your example causes undefined behaviour. That said, on most systems it will work just fine. You probably do want to change your printf format to %td to indicate that you're printing a ptrdiff_t type.
This is undefined behavior.
For one, those pointers don't point to memory that you own.
You can only substract pointers that point inside the same array (or one position after the end of the array).
Of course, it will most likely work on most compilers, and you get 25 because sizeof(int) == 4 on your platform. If they were char *, you'd get 100. (possibly, or it could crash, that's the beauty of UB).
Even the standard does not promise a defined behavior, the result is correct.
An integer needs in your architecture 4 bytes. Thus, the difference give you the number of integer values the both pointer are apart.
You can use the difference as an index or offset in an array.
For the same reason
int *p = (int*) 100;
p++;
will result in p=104.
Of course it's undefined. Subtracting two arbitrary pointers (viewed as integers) is not even guaranteed to be a multiple of your object's size.
Following an hot comment thread in another question, I came to debate of what is and what is not defined in C99 standard about C arrays.
Basically when I define a 2D array like int a[5][5], does the standard C99 garantee or not that it will be a contiguous block of ints, can I cast it to (int *)a and be sure I will have a valid 1D array of 25 ints.
As I understand the standard the above property is implicit in the sizeof definition and in pointer arithmetic, but others seems to disagree and says casting to (int*) the above structure give an undefined behavior (even if they agree that all existing implementations actually allocate contiguous values).
More specifically, if we think an implementation that would instrument arrays to check array boundaries for all dimensions and return some kind of error when accessing 1D array, or does not give correct access to elements above 1st row. Could such implementation be standard compilant ? And in this case what parts of the C99 standard are relevant.
We should begin with inspecting what int a[5][5] really is. The types involved are:
int
array[5] of ints
array[5] of arrays
There is no array[25] of ints involved.
It is correct that the sizeof semantics imply that the array as a whole is contiguous. The array[5] of ints must have 5*sizeof(int), and recursively applied, a[5][5] must have 5*5*sizeof(int). There is no room for additional padding.
Additionally, the array as a whole must be working when given to memset, memmove or memcpy with the sizeof. It must also be possible to iterate over the whole array with a (char *). So a valid iteration is:
int a[5][5], i, *pi;
char *pc;
pc = (char *)(&a[0][0]);
for (i = 0; i < 25; i++)
{
pi = (int *)pc;
DoSomething(pi);
pc += sizeof(int);
}
Doing the same with an (int *) would be undefined behaviour, because, as said, there is no array[25] of int involved. Using a union as in Christoph's answer should be valid, too. But there is another point complicating this further, the equality operator:
6.5.9.6
Two pointers compare equal if and only if both are null pointers, both are pointers to the same object (including a pointer to an object and a subobject at its beginning) or function, both are pointers to one past the last element of the same array object, or one is a pointer to one past the end of one array object and the other is a pointer to the start of a different array object that happens to immediately follow the first array object in the address space. 91)
91) Two objects may be adjacent in memory because they are adjacent elements of a larger array or adjacent members of a structure with no padding between them, or because the implementation chose to place them so, even though they are unrelated. If prior invalid pointer operations (such as accesses outside array bounds) produced undefined behavior, subsequent comparisons also produce undefined behavior.
This means for this:
int a[5][5], *i1, *i2;
i1 = &a[0][0] + 5;
i2 = &a[1][0];
i1 compares as equal to i2. But when iterating over the array with an (int *), it is still undefined behaviour, because it is originally derived from the first subarray. It doesn't magically convert to a pointer into the second subarray.
Even when doing this
char *c = (char *)(&a[0][0]) + 5*sizeof(int);
int *i3 = (int *)c;
won't help. It compares equal to i1 and i2, but it isn't derived from any of the subarrays; it is a pointer to a single int or an array[1] of int at best.
I don't consider this a bug in the standard. It is the other way around: Allowing this would introduce a special case that violates either the type system for arrays or the rules for pointer arithmetic or both. It may be considered a missing definition, but not a bug.
So even if the memory layout for a[5][5] is identical to the layout of a[25], and the very same loop using a (char *) can be used to iterate over both, an implementation is allowed to blow up if one is used as the other. I don't know why it should or know any implementation that would, and maybe there is a single fact in the Standard not mentioned till now that makes it well defined behaviour. Until then, I would consider it to be undefined and stay on the safe side.
I've added some more comments to our original discussion.
sizeof semantics imply that int a[5][5] is contiguous, but visiting all 25 integers via incrementing a pointer like int *p = *a is undefined behaviour: pointer arithmetics is only defined as long as all pointers invoved lie within (or one element past the last element of) the same array, as eg &a[2][1] and &a[3][1] do not (see C99 section 6.5.6).
In principle, you can work around this by casting &a - which has type int (*)[5][5] - to int (*)[25]. This is legal according to 6.3.2.3 §7, as it doesn't violate any alignment requirements. The problem is that accessing the integers through this new pointer is illegal as it violates the aliasing rules in 6.5 §7. You can work around this by using a union for type punning (see footnote 82 in TC3):
int *p = ((union { int multi[5][5]; int flat[25]; } *)&a)->flat;
This is, as far as I can tell, standards compliant C99.
If the array is static, like your int a[5][5] array, it's guaranteed to be contiguous.
I'd like to find out how C will allocate a the data items of a multidimensional array, and if their allocation is consistent across machines.
I know that, at the lowest level, the data items are neighbours, but I don't know how they're arranged further up.
For example, if I allocate a 3D array as int threeD[10][5][6], can I assume that &(threeD[4][2][5]) + 1 == &(threeD[4][3][0])? On all machines?
Thanks in advance for your help.
Yes, arrays are stored in row major order across all implementations of C compilers.
The Standard says (I applied some reformatting):
6.5.2.1 Array subscripting
Constraints
3 Successive subscript operators designate an element of a multidimensional
array object.
If E is an n-dimensional array (n >= 2) with dimensions i * j * . . . * k,
then E (used a s other than an lvalue) is converted to a pointer to an
(n - 1)-dimensional array with dimensions j * . . . * k.
If the unary * operator is applied to this pointer explicitly, or
implicitly as a result of subscripting, the result is the pointed-to
(n - 1)-dimensional array, which itself is converted into a pointer if
used as other than an lvalue. It follows from this that arrays are stored
in row-major order (last subscript varies fastest).
The C standard is very specific in equating array subscripting with pointer arithmetic, and specifies that arrays are stored in row major order.
Consider the array object defined by the declaration
int x[3][5];
Here x is a 3 x 5 array of ints; more precisely, x is an array of three element objects, each of which is an array of five ints. In the expression x[i], which is equivalent to
(*((x)+(i))), x is first converted to a pointer to the initial array of five ints. Then
i is adjusted according to the type of x, which conceptually entails multiplying i by the size of the object to which the pointer points, namely an array of five int objects. The results are added and indirection is applied to yield an array of five ints. When used in the expression x[i][j], that array is in turn converted to a pointer to the first of the ints, so x[i][j] yields an int.
The elements are stored in Row Major order. So Elements along the last dimension are contiguous. However, elements between rows (as indicated by your example) aren't guaranteed to be contiguous. It depends on how the initial memory has been allocated.
#include <malloc.h>
#include <stdio.h>
#include <stdlib.h>
// only elements in a single row are guaranteed to be
// contiguous because of the multiple mallocs
void main(void)
{
// 3 rows, 4 columns
int *a[3];
for ( int row = 0; row < 3; row++ )
a[row] = (int *)malloc(4*sizeof(int));
}
// all elements are guaranteed to be contiguous
// in a row major order.
void main(void)
{
// 3 rows, 4 columns
int *a[3];
int *buf = (int *)malloc(3*4*sizeof(int));
for ( int row = 0; row < 3; row++ )
a[row] = buf+4*row;
assert( (&a[1][3] + 1) == &a[2][0] );
}
Firstly, In C language address arithmetic is only defined within the boundaries of a given array. (I wanted to say "single-dimensional (SD) array", but technically all arrays in C are SD. Multi-dimensional arrays are built as SD arrays of SD arrays. And this view of arrays is the most appropriate for this topic). In C you can start from the pointer to the beginning of an array and move back and forth within that array using additive operations. You are not allowed to cross the boundaries of the array you started from, except that it is legal to form a pointer to an imaginary element that follows the last element. However, when it comes to accessing elements (reading and writing), you are only allowed to access the real, existing elements of the array you started from.
Secondly, in your example '&threeD[4][2][5] + 1' you are forming a pointer to the imaginary "past-the-last" element of array 'threeD[4][2]'. This by itself is legal. However, the language specification does not guarantee that this pointer is equal to the address of '&threeD[4][3][0]'. The only thing that it says is that it might be equal to it. It is true, that the other requirements imposed on arrays by the language specification pretty much "force" this relationship to hold. But it is not formally guaranteed. Some pedantic (to the point of being malicious) implementation is perfectly allowed to use some kind of compiler magic to break this relationship.
Thirdly, actually accessing '*(threeD[4][2][5] + 1)' is always illegal. Even if the pointer is pointing into the next array, the compiler is allowed to perform the necessary run-time checks and generate a segmentation fault, since you are using pointer arithmetic on 'threeD[4][2]' array and trying to access something outside its boundaries.
Fourthly, doing 'threeD[4][2][5] + 2', '...+ 3' etc. is always illegal for similar reasons (remember: one past the end is OK, but 2, 3 or more is illegal).
And finally, fifthly: yes I know that in many (if not most) (if not all) practical cases interpreting a 'T A[2][3][4]' array as a flat 'T A[2*3*4]' array will work. But, again, from the formal language point of view this is illegal. And don't be surprised if this perfectly working code will one day trigger a huge amount of warnings from some static or dynamic code analysis tool, if not from the compiler itself.