I've been at this for a while now and it really puzzles me. This is a very distilled code fragment that reproduces the problem:
uint8_t dataz[] = { 1, 2, 3, 4, 5, 6 };
struct mystruct {
uint8_t dummy1[1];
uint16_t very_important_data;
uint8_t dummy2[3];
} *mystruct = (void *) dataz;
printf("%x\n", mystruct -> very_important_data);
What do you expect should be the output ? I'd say x302, but nope. It gives me x403. The same as if using this structure:
struct mystruct {
uint8_t dummy1[2];
uint16_t very_important_data;
uint8_t dummy2[2];
} *mystruct = (void *) dataz;
How would you explain that?
As others have mentioned, unless your compiler alignment is byte-aligned, your structure is likely to have "holes" in it. The compiler does this because it speeds up memory access.
If you're using gcc, there is a "packed" attribute which will cause the struct to be byte-aligned, and so remove the "holes":
struct __attribute((__packed__)) mystruct {
uint8_t dummy1[1];
uint16_t very_important_data;
uint8_t dummy2[3];
} *mystruct = (void *) dataz;
However, this will not necessarily fix the problem. The 16-bit value may not be set to what you think it should be, depending on the endianness of your machine. You will have to swap the bytes in any multi-byte integers in the struct. There is no general function to do this, as it would require information on the layout of the structure at run-time, which C does not provide.
Mapping structures to binary data is generally non-portable, even if you get it to work on your machine, right now.
Packing. The is no guarantee how members of a struct are physically located inside the struct. They may be word-aligned, leaving gaps.
There are pragmas in some versions of C to explictly control packing.
Most likely, the compiler has added a byte of padding between dummy1 and very_important_data to align very_important_data on a 16-bit boundary.
In general, the alignment and padding of fields in a struct is implementation-dependent, so you shouldn't rely on it. If you absolutely need a particular behavior, many compilers offer #pragma or other directives to control this. Check your compiler's documentation.
It depends on the compiler, but usually a compiler aligns each member to its natural alignment. In the case you ran into, very_important_data is a uint16_t which probably has a natural alignment of 2 bytes.
Related
In a C program, having an array that is meant to work as a buffer FooBuffer[] for storing the contents of a data member of a struct like this:
struct Foo {
uint64_t data;
};
I was told that this line might cause unaligned access:
uint8_t FooBuffer[10] = {0U};
I have some knowledge that unaligned access depends on the alignment offset of the processor and in general, it consumes more read/write cycles. Under what circumstances would this cause unaligned memory access and how could I prevent it?
Edit:
A variable of type struct Foo would be stored in the buffer. Particularly, its member data would be split up into eight bytes that would be stored in the array FooBuffer. See attached code with some options for this.
#include <stdio.h>
#include <string.h>
typedef unsigned long uint64;
typedef unsigned char uint8;
struct Foo
{
uint64 data;
};
int main()
{
struct Foo foo1 = {0x0123456789001122};
uint8 FooBuffer[10] = {0U};
FooBuffer[0] = (uint8)(foo1.data);
FooBuffer[1] = (uint8)(foo1.data >> 8);
FooBuffer[2] = (uint8)(foo1.data >> 16);
FooBuffer[3] = (uint8)(foo1.data >> 24);
FooBuffer[4] = (uint8)(foo1.data >> 32);
FooBuffer[5] = (uint8)(foo1.data >> 40);
FooBuffer[6] = (uint8)(foo1.data >> 48);
FooBuffer[7] = (uint8)(foo1.data >> 56);
struct Foo foo2 = {0x9876543210112233};
uint8 FooBuffer2[10] = {0U};
memcpy(FooBuffer2, &foo2, sizeof(foo2));
return 0;
}
However, it is not clear how this process is done since a piece of privative software performs the operation. What would be the scenarios that could result in unaligned memory access after the "conversion"?
Defining either a structure such as struct Foo { uint64_t data; } or an array such as uint8_t FooBuffer[10]; and using them in normal ways will not cause an unaligned access. (Why did you use 10 for FooBuffer? Only 8 bytes are needed for this example?)
A method that novices sometimes attempt that can cause unaligned accesses is attempting to reinterpret an array of bytes as a data structure. For example, consider:
// Get raw bytes from network or somewhere.
uint8_t FooBuffer[10];
CallRoutineToReadBytes(FooBuffer,...);
// Reinterpret bytes as original type.
struct Foo x = * (struct Foo *) FooBuffer; // Never do this!
The problem here is that struct Foo has some alignment requirement, but FooBuffer does not. So FooBuffer could be at any address, but the cast to struct Foo * attempts to force it to an address for a struct Foo. If the alignment is not correct, the behavior is not defined by the C standard. Even if the system allows it and the program “works,” it may be accessing a struct Foo at an improperly aligned address and suffering performance problems.
To avoid this, a proper way to reinterpret bytes is to copy them into a new object:
struct Foo x;
memcpy(&x, FooBuffer, sizeof x);
Often a compiler will recognize what is happening here and, especially if struct Foo is not large, implement the memcpy in an efficient way, perhaps as two load-four-byte instructions or one load-eight-byte instruction.
Something you can do to help that along is ask the compiler to align FooBuffer by declaring it with the _Alignas keyword:
uint8_t _Alignas(Struct Foo) FooBuffer[10];
Note that that might not help if you need to take bytes from the middle of a buffer, such as from a network message that includes preceding protocol bytes and other data. And, even if it does give the desired alignment, never use the * (struct Foo *) FooBuffer shown above. It has more problems than just alignment, one of which is that the C standard does not guarantee the behavior of reinterpreting data like this. (A supported way to do it in C is through unions, but memcpy is a fine solution.)
In the code you show, bytes are copied from foo1.data to FooBuffer using bit shifts. This also will not cause alignment problems; expressions that manipulate data like this work just fine. But there are two issues with it. One is that it nominally manipulates individual bytes one by one. That is perfectly legal in C, but it can be slow. A compiler might optimize it, and there might be built-ins or library functions to assist with it, depending on your platform.
The other issue is that it puts the bytes in an order according to their position values: The low-position-value bytes are put into the buffer first. In contrast, the memcpy method copies the bytes in the order they are stored in memory. Which method you want to use depends on the problem you are trying to solve. To store data on one system and read it back later on the same system, the memcpy method is fine. To send data between two systems using the same byte ordering, the memcpy method is fine. However, if you want to send data from one system on the Internet to another, and the two systems do not use the same byte order in memory, you need to agree on an order to use in the network packages. In this case, it is common to use the arrange-bytes-by-position-value method. Again, your platform may have builtins or library routines to assist with this. For example, the htonl and ntohl routines are BSD routines that take a normal 32-bit unsigned integer and return it with its bytes arranged for network order or vice-versa.
struct {
uint8_t foo;
uint8_t bar;
uint8_t baz;
uint8_t foos[252];
uint8_t somethingOrOther;
} A;
struct {
uint8_t foo;
uint8_t bar;
uint8_t baz;
uint8_t somethingOrOther;
uint8_t foos[252];
} B;
Does it matter that I've put foos on byte 3 in the first example, vs on byte 4 in B?
Does an array in C have to start aligned?
Is the size of this struct 256 bytes exactly?
Given that the data type is uint8_t (equivalent to unsigned char), there is no need for padding in the structure, regardless of how it is ordered. So, you can reasonably assume in this case that every compiler will make that structure into 256 bytes, regardless of the order of the elements.
If there were data elements of different sizes, then you might well get padding added and the size of the structure might vary depending on the order of the elements.
As the good book says (C11 section 6.7.2.1 paragraph 14):
Each non-bit-field member of a structure or union object is aligned in
an implementation- defined manner appropriate to its type... There
may be unnamed padding within a structure object
You didn't "put foos on byte 3" - apart from the fact that the first element is always on byte 0, you have no real control over what byte an element will be put on. The compiler can give each field a full machine word or more if it thinks that will provide the most efficient accesses. If so, it will allocate "padding" bytes in between - unused space.
Note that this only applies to structs; arrays themselves do not add any padding bytes (if they did, the pointer arithmetic/indexing rule wouldn't work), so you can be sure that the size of foos itself is exactly the declared size, and know the exact alignment of each numbered element.
No, it doesn't matter.
Pretty much every compiler out there will align struct fields to a natural boundary appropriate for the target architecture, by inserting hidden "padding bytes" between fields where necessary.
Specifically, since you're using only uint8_t, there will be no padding bytes inserted - every field already falls on a naturally-aligned boundary. (Everything is a multiple of one.)
Both of the structs you've shown are exactly 256 bytes in size. You can confirm this like so:
int main(void)
{
printf("sizeof(struct A)=%zu \n", sizeof(struct A));
printf("sizeof(struct B)=%zu \n", sizeof(struct B));
}
You can prevent this padding from being added by "packing" the structure:
Microsoft Visual C uses #pragma pack(1)
GCC uses __attribute__((packed))
Do note that doing this on a structure with unaligned members can have a serious impact on the performance of your program.
I have 2 structures defined like below.
#include<stdint.h>
typedef struct
{
uint32_t a;
uint8_t b;
uint8_t pad[3]; //padding here is added intentionally.
uint32_t c;
}A;
typedef struct
{
uint32_t a;
uint8_t b;
uint8_t pad[3];
uint32_t c;
}__attribute__((__packed__)) B;
Are these 2 structs guaranteed to have exactly same memory layout on all the hardware platforms? It can be assumed that the compiler is always gcc.
No. There could still be padding in the unpacked version of this struct. Even if there is no such implementation today, there could be an architecture in the future where-by all of it's types are most optimal when aligned to offsets evenly divisible by 128, and any of the members in the unpacked version could be followed by between 120 and 96 bits of padding on such a system. A compiler might take advantage of this. Stackoverflow is forever.
The answer would be "possibly" but not "guaranteed" since all architectures is a very broad range of coverage. I can envision some systems that have memory architectures whose read/write performance would prefer keeping reads and writes on boundaries larger than uint8_t.
Note that when packed, compilers will use read/write code that is often suboptimal for the particular architecture, but correct with respect to accessing the structure member.
I have a structure like this
struct packet
{
int seqnum;
char type[1];
float time1;
float pri;
float time2;
unsigned char data[512];
}
I am receiving packet in an array
char buf[529];
I want to take the seqnum,data everything separately.Does the following typecast work.. It is giving junk value for me.
struct packet *pkt;
pkt=(struct packet *)buf;
printf(" %d",pkt->seqnum)
No, that likely won't work and is generally a bad and broken way of doing this.
You must use compiler-specific extensions to make sure there's no invisible padding between your struct members, for something like that to work. With gcc, for instance, you do this using the __attribute__() syntax.
It is, thus, not a portable idea.
It's much better to be explicit about it, and unpack each field. This also gives you a chance to have a well-defined endianness in your network protocol, which is generally a good idea for interoperability's sake.
No, that isn't generally valid code. You should make the struct first and then memcopy stuff into it:
packet p;
memcpy(&p.seqnum, buf + 0, 4);
memcpy(&p.type[0], buf + 4, 1);
memcpy(&p.time1, buf + 5, 4);
And so forth.
You must take great care to get the type sizes and endianness right.
First of all, you cannot know in advance where the compiler will insert padding bytes in your structure for performance optimization (cache line alignment, integer alignment etc) since this is platform-dependent. Except, of course, if you are considering building the app only on your platform.
Anyway, in your case it seems like you are getting data from somewhere (network ?) and it is highly probable that the data has been compacted (no padding bytes between fields).
If you really want to typecast your array to a struct pointer, you can still tell the compiler to remove the padding bytes it might add. Note that this depends on the compiler you use and is not a standard C implementation. With gcc, you might add this statement at the end of your structure definition :
struct my_struct {
int blah;
/* Blah ... */
} __attribute__((packed));
Note that it will affect the performance for member access, copy etc ...
Unless you have a very good reason to do so, don't ever use the __attribute__((packed)) thing !
The other solution, which is much more advisable is to make the parsing on your own. You just allocate an appropriate structure and fill its fields by seeking the good information from your buffer. A sequence of memcpy instructions is likely to do the trick here (see Kerrek's answer)
Can we predict how a C struct will be implemented by the compiler?
If I write the (very badly aligned) struct:
struct {
uint16_t a;
uint32_t b;
uint8_t c;
} s;
char *p = (char*)&s;
can I guarantee that p[6] is the same as s.c? Are the struct fields allocated in this most obvious and canonical way, so we can predict where each field will be in memory?
Edit: Will struct __attribute__ ((__packed__)) {...} s; get me this behavior in GCC?
No you cannot. Don't do that.
You are guaranteed only the order and the same compiler will always do the same layout.
If you need such a thing consult your compiler's documentation for how to enable byte packing (always available) and pad yourself.
The fields have to be allocated in ascending order, but the compiler is free to insert padding between fields as it sees fit, so there's no guarantee of what value of n in p[n] will refer to s.c. OTOH, you can obtain the correct offset using offsetof(s,c).
Even with the __packed__ attribute, it may be impossible to get this alignment due to architectural restrictions. For example, if uint32_t requires 4-byte alignment, it will be at offset 4 even with __packed__.
If you need to assume a particular alignment, put in a static check that will prevent the code compiling with a different alignment.
For a given version of a compiler on a given version of the operating system - and with the same build options = yes
But don't !
See #pragma pack(packed) and #pragma pack(reset). It has the same impact as the GCC attribute __packed__ you mentioned.