In GCC C, how can I get an array of packed structs, where each entry in the array is aligned?
(FWIW, this is on a PIC32, using the MIPS4000 architecture).
I have this (simplified):
typedef struct __attribute__((packed))
{
uint8_t frameLength;
unsigned frameType :3;
uint8_t payloadBytes;
uint8_t payload[RADIO_RX_PAYLOAD_WORST];
} RADIO_PACKET;
RADIO_PACKETs are packed internally.
Then I have RADIO_PACKET_QUEUE, which is a queue of RADIO_PACKETs:
typedef struct
{
short read; // next buffer to read
short write; // next buffer to write
short count; // number of packets stored in q
short buffers; // number of buffers allocated in q
RADIO_PACKET q[RADIO_RX_PACKET_BUFFERS];
} RADIO_PACKET_QUEUE;
I want each RADIO_PACKET in the array q[] to start at an aligned address (modulo 4 address).
But right now GCC doesn't align them, so I get address exceptions when trying to read q[n] as a word. For example, this gives an exception:
RADIO_PACKET_QUEUE rpq;
int foo = *(int*) &(rpq.q[1]);
Perhaps this is because of the way I declared RADIO_PACKET as packed.
I want each RADIO_PACKET to remain packed internally, but want GCC to add padding as needed after each array element so each RADIO_PACKET starts at an aligned address.
How can I do this?
Since you specify that you are using GCC, you should be looking at type attributes. In particular, if you want RADIO_PACKETs to be aligned on 4-byte (or wider) boundaries, then you would use __attribute__((aligned (4))) on the type. When applied to a struct, it describes the alignment of instances of the overall struct, not (directly) the alignment of any individual members, so it is possible to use it together with attribute packed:
typedef struct __attribute__((aligned(4), packed))
{
uint8_t frameLength;
unsigned frameType :3;
uint8_t payloadBytes;
uint8_t payload[RADIO_RX_PAYLOAD_WORST];
} RADIO_PACKET;
The packed attribute prevents padding between structure elements, but it does not prevent trailing padding in the structure representation, exactly as is necessary for ensuring the required alignment for every element of an array of the specified type. You should not then need to do anything special in the declaration of RADIO_PACKET_QUEUE.
This is a lot cleaner and clearer than the alternative you came up with, but it is GCC-specific. Since you were GCC-specific already, I don't see that being a problem.
You can wrap your packed structure, inside another unaligned structure. Then you do array from this unaligned structures.
Solution 2 could be, to add dummy member char[] at the end of the packed structure. In this case you will need to calculate it somehow, probably manually.
I also will suggest you to rearrange your structure, by placing longer members first and placing uint8_t members last (assuming you have 16/32 bit members and not doing some HW mapping).
I think I've solved this, based on the hint from #Nick in the question comments.
I added a RADIO_PACKET_ALIGNED wrapper around RADIO_PACKET. This includes calculated padding.
Then I substituted RADIO_PACKET_ALIGNED for RADIO_PACKET in the RADIO_PACKET_QUEUE structure.
Seems to work:
typedef struct
{
RADIO_PACKET packet;
uint8_t padding[3 - (sizeof(RADIO_PACKET) + 3) % 4];
} RADIO_PACKET_ALIGNED;
typedef struct
{
short read; // next buffer to read
short write; // next buffer to write
short count; // number of packets stored in q
short buffers; // number of buffers allocated in q
RADIO_PACKET_ALIGNED q[RADIO_RX_PACKET_BUFFERS];
} RADIO_PACKET_QUEUE;
Thanks to all the commenters!
Edit: A more portable version of the wrapper would use:
uint8_t padding[(sizeof(int) - 1) - (sizeof(RADIO_PACKET) + (sizeof(int) - 1)) % sizeof(int)];
Related
I am using a for loop to copy an array from a UART RX buffer to memory.
This looks like as follows:
UART2_readTimeout(uartR2, rxBuf3, 54, NULL, 500);
GPIO_toggle(CONFIG_GPIO_LED_3);
if ((rxBuf3[4] == 0x8C) && (rxBuf3[10] != 0x8C)) {
int i;
for (i = 0; i < 47; i++) {
sioR2[i]=rxBuf3[i];
}
I want to then use a struct such as the following to make it possible to use dot notation when working with and organizing the data:
typedef struct
{
uint16_t voltage;
uint16_t current;
uint16_t outTemp;
uint16_t inTemp;
uint16_t status;
uint32_t FaultA;
uint32_t FaultB;
uint32_t FaultC;
uint32_t FaultD;
uint8_t softwareMode;
uint8_t logicLoad;
uint8_t outputBits;
uint16_t powerOut;
uint32_t runHours;
uint16_t unitAddresses[6];
} unitValues;
Assuming the total length of these are the same, is it possible to perform a memcpy on the entire array to a single instance of the struct?
Array : 001110101....110001
||||||||||||||||||| <- memcpy
vvvvvvvvvvvvvvvvvvv
Struct : 001110101....110001
Provided that your C implementation offers a way to ensure that the layout of your structure is the same as the layout that the driver in question uses for writing the buffer, a pretty good way to go about this would be to have the driver write directly into the structure. I'm inferring the signature of the driver function here, but that would probably be something like:
UART2_readTimeout(uartR2, (uint8_t *) &values, 54, NULL, 500);
Assuming that uint8_t is an alias for unsigned char or maybe char, it is valid to write into the representation of the structure via a pointer of type uint8_t *. Thus, this avoids you having to make a copy.
The trick, however, is the structure layout. Supposing that you expect the data to be laid out as the structure members given, in the order given, with no gaps, such a structure layout would prevent structure instances being positioned in memory so that all members are aligned on addresses that are multiples of their sizes. Depending on the alignment rules of your hardware, this might be perfectly fine, but probably either it would slow accesses to some of the members, or it would make attempts to access some of the members crash the program.
if you still want to proceed then you will need to check your compiler's documentation for information about how to get the wanted layout of your structure. You might look for references to structure "packing", structure layout, or structure member alignment. There is no standard way to do this -- if your C implementation supports it at all then that constitutes an extension, with implementation-specific details.
All the same issues and caveats would apply to using memcpy to copy the buffer contents onto an instance of your structure type, so if you don't multiple copies of the data and you can arrange to make bulk copy onto the structure work, then you're better off writing directly onto the structure than writing into a separate buffer and then copying.
On the other hand, the safe and standard alternative would be to allow your implementation to lay out the structure however it thinks is best, And to copy the data out of your buffer into the structure in member-by-member fashion, with per-member memcpy()s. Yes, the code will be a bit tedious, but it will not be sensitive to alignment-related issues, nor even to reordering structure members or adding new ones.
You have to change the packing align to 1 byte for the structure.
#pragma pack(1) /* change */
typedef struct {
...
}
#pragma pack() /* restore */
In theory, you can use memcpy() to set the member fields of a struct from the elements of a byte array. However, you will need to be very careful to prevent your compiler from adding 'empty' fields to your struct (see: Structure padding and packing) unless those empty fields are taken into account when loading the data into the source array. (The elements of the source array will be packed into contiguous memory.)
Different compilers use different command-line and/or #pragma options to control structure packing but, for the MSVC compiler, you can use the #pragma pack(n) directives in your source code or the /Zp command-line switch.
Using the MSVC compiler, the structure you have provided will have a total size of 47 bytes only if you have single-byte packing; for default packing, the size will be 52 bytes.
The following code block shows where these 'extra' bytes will be inserted for different packing sizes.
#pragma pack(push, 1) // This saves the current packing level then sets it to "n" (1, here)
typedef struct {
uint16_t voltage;
uint16_t current;
uint16_t outTemp;
uint16_t inTemp;
uint16_t status;
// 4+ byte packing will insert two bytes here
uint32_t FaultA;
uint32_t FaultB;
uint32_t FaultC;
uint32_t FaultD;
uint8_t softwareMode;
uint8_t logicLoad;
uint8_t outputBits;
// 2+ byte packing will insert one byte here
uint16_t powerOut;
// 4+ byte packing will insert two bytes here
uint32_t runHours;
uint16_t unitAddresses[6];
} unitValues;
#pragma pack(pop) // This restores the previous packing level
So, the sizeof(unitValues) will be:
47 bytes when using #pragma pack(1)
48 bytes when using #pragma pack(2)
52 bytes when using #pragma pack(4) (or any higher/default value)
I'm trying to create a C client for dalmatinerdb but having trouble to understand how to combine the variables, write it to a buffer and send it to the database. The fact that dalmatinerdb is written in Erlang makes it more difficult. However, by looking at a python client for dalmatinerdb i have (probably) found the necessary variable sizes and order.
The erlang client has a function called "encode", see below:
encode({stream, Bucket, Delay}) when
is_binary(Bucket), byte_size(Bucket) > 0,
is_integer(Delay), Delay > 0, Delay < 256->
<<?STREAM,
Delay:?DELAY_SIZE/?SIZE_TYPE,
(byte_size(Bucket)):?BUCKET_SS/?SIZE_TYPE, Bucket/binary>>;
According to the official dalmatinerdb protocol we can see the following:
-define(STREAM, 4).
-define(DELAY_SIZE, 8). /bits
-define(BUCKET_SS, 8). /bits
Let's say i would like to create this kind of structure in C,
would it look something like the following:
struct package {
unsigned char[1] mode; // = "4"
unsigned char[1] delay; // = for example "5"
unsigned char[1] bucketNameSize; // = "5"
unsigned char[1] bucketName; // for example "Test1"
};
Update:
I realized that the dalmatinerdb frontend (web interface) only reacts and updates when values have been sent to the bucket. With other words just sending the first struct won't give me any clue if it's right or wrong. Therefore I will try to create a secondary struct with the actual values.
The erland code snippet which encodes values looks like this:
encode({stream, Metric, Time, Points}) when
is_binary(Metric), byte_size(Metric) > 0,
is_binary(Points), byte_size(Points) rem ?DATA_SIZE == 0,
is_integer(Time), Time >= 0->
<<?SENTRY,
Time:?TIME_SIZE/?SIZE_TYPE,
(byte_size(Metric)):?METRIC_SS/?SIZE_TYPE, Metric/binary,
(byte_size(Points)):?DATA_SS/?SIZE_TYPE, Points/binary>>;
The different sizes:
-define(SENTRY, 5)
-define(TIME_SIZE, 64)
-define(METRIC_SS, 16)
-define(DATA_SS, 32)
Which gives me this gives me:
<<?5,
Time:?64/?SIZE_TYPE,
(byte_size(Metric)):?16/?SIZE_TYPE, Metric/binary,
(byte_size(Points)):?32/?SIZE_TYPE, Points/binary>>;
My guess is that my struct containing a value should look like this:
struct Package {
unsigned char sentry;
uint64_t time;
unsigned char metricSize;
uint16_t metric;
unsigned char pointSize;
uint32_t point;
};
Any comments on this structure?
The binary created by the encode function has this form:
<<?STREAM, Delay:?DELAY_SIZE/?SIZE_TYPE,
(byte_size(Bucket)):?BUCKET_SS/?SIZE_TYPE, Bucket/binary>>
First let's replace all the preprocessor macros with their actual values:
<<4, Delay:8/unsigned-integer,
(byte_size(Bucket):8/unsigned-integer, Bucket/binary>>
Now we can more easily see that this binary contains:
a byte of value 4
the value of Delay as a byte
the size of the Bucket binary as a byte
the value of the Bucket binary
Because of the Bucket binary at the end, the overall binary is variable-sized.
A C99 struct that resembles this value can be defined as follows:
struct EncodedStream {
unsigned char mode;
unsigned char delay;
unsigned char bucket_size;
unsigned char bucket[];
};
This approach uses a C99 flexible array member for the bucket field, since its actual size depends on the value set in the bucket_size field, and you are presumably using this structure by allocating memory large enough to hold the fixed-size fields together with the variable-sized bucket field, where bucket itself is allocated to hold bucket_size bytes. You could also replace all uses of unsigned char with uint8_t if you #include <stdint.h>. In traditional C, bucket would be defined as a 0- or 1-sized array.
Update: the OP extended the question with another struct, so I've extended my answer below to cover it too.
The obvious-but-wrong way to write a struct corresponding to the metric/time/points binary is:
struct Wrong {
unsigned char sentry;
uint64_t time;
uint16_t metric_size;
unsigned char metric[];
uint32_t points_size;
unsigned char points[];
};
There are two problems with the Wrong struct:
Padding and alignment: Normally, fields are aligned on natural boundaries corresponding to their sizes. Here, the C compiler will align the time field on an 8-byte boundary, which means there will be padding of 7 bytes following the sentry field. But the Erlang binary contains no such padding.
Illegal flexible array field in the middle: The metric field size can vary, but we can't use the flexible array approach for it as we did in the earlier example because such arrays can only be used for the final field of a struct. The fact that the size of metric can vary means that it's impossible to write a single C struct that matches the Erlang binary.
Solving the padding and alignment issue requires using a packed struct, which you can achieve with compiler support such as the gcc and clang __packed__ attribute (other compilers might have other ways of achieving this). The variable-sized metric field in the middle of the struct can be solved by using two structs instead:
typedef struct __attribute((__packed__)) {
unsigned char sentry;
uint64_t time;
uint16_t size;
unsigned char metric[];
} Metric;
typedef struct __attribute((__packed__)) {
uint32_t size;
unsigned char points[];
} Points;
Packing both structs means their layouts will match the layouts of the corresponding data in the Erlang binary.
There's still a remaining problem, though: endianness. By default, fields in an Erlang binary are big-endian. If you happen to be running your C code on a big-endian machine, then things will just work, but if not — and it's likely you're not — the data values your C code reads and writes won't match Erlang.
Fortunately, endianness is easily handled: you can use byte swapping to write C code that can portably read and write big-endian data regardless of the endianness of the host.
To use the two structs together, you'd first have to allocate enough memory to hold both structs and both the metric and the points variable-length fields. Cast the pointer to the allocated memory — let's call it p — to a Metric*, then use the Metric pointer to store appropriate values in the struct fields. Just make sure you convert the time and size values to big-endian as you store them. You can then calculate a pointer to where the Points struct is in the allocated memory as shown below, assuming p is a pointer to char or unsigned char:
Points* points = (Points*)(p + sizeof(Metric) + <length of Metric.metric>);
Note that you can't just use the size field of your Metric instance for the final addend here since you stored its value as big-endian. Then, once you fill in the fields of the Points struct, again being sure to store the size value as big-endian, you can send p over to Erlang, where it should match what the Erlang system expects.
My code has a structure type-defined as follows:
typedef struct
{
Structure_2 a[4];
UCHAR b;
UCHAR c;
}Structure_1;
where the definition of Structure_2 is as follows:
typedef struct
{
ULONG x;
USHORT y;
UCHAR z;
}Structure_2;
There are also two functions in the code. The first one (named setter) declares a structure of type “Structure_1” and fills it with the data:
void setter (void)
{
Structure_1 data_to_send ;
data_to_send.a[0].x = 0x12345678;
data_to_send.a[0].y = 0x1234;
data_to_send.a[0].z = 0x12;
data_to_send.a[1].x = 0x12345678;
data_to_send.a[1].y = 0x1234;
data_to_send.a[1].z = 0x12;
data_to_send.a[2].x = 0x12345678;
data_to_send.a[2].y = 0x1234;
data_to_send.a[2].z = 0x12;
data_to_send.a[3].x = 0x12345678;
data_to_send.a[3].y = 0xAABB;
data_to_send.a[3].z = 0x12;
data_to_send.b =0;
data_to_send.c = 0;
getter(&data_to_send);
}
The compiler saves data_to_send in memory like that:
The second one named getter:
void getter (Structure_1 * ptr_to_data)
{
UCHAR R_1 = ptr_to_data -> b;
UCHAR R_2 = ptr_to_data -> c;
/* The remaining bytes are received */
}
I expect that R_1 will have the value “00”, and R_2 will have the value “00”.
But what happen is the compiler translates the following two lines like that:
/* Get the data at the address ptr_to_data -> b,
which equals the start address of structure + 28 which contains the
value “AA”, and hence R_1 will have “AA” */
UCHAR R_1 = ptr_to_data -> b;
/* Get the data at the address ptr_to_data -> c,
which equals the start *address of structure + 29 which contains the
value “BB”, and hence R_2 will *have “BB” */
UCHAR R_2 = ptr_to_data -> c;
The compiler adds padding b/yte while saving the structure in stack, However when it starts reading it, it forget what it did (and includes the padding bytes in reading).
How could I inform the compiler that you should skip the padding byte while reading the elements of structure ?
I don't want a work around to solve this problem, I am curious to know why the compiler behaves like that ?
My compiler is GreenHills and My target is 32-bit
How could I inform the compiler that you should skip the padding byte while reading the elements of structure ?
Short answer: You cannot.
The compiler will not dis-regard contents contained in your struct. However you can control how it will treat the contents in your struct.
I am curious to know why the compiler behaves like that ?
Short answer: data alignment.
Two issues to consider: data alignment boundaries and data structure padding. You have some control over each:
Data alignment Is the reason your compiler sees what it sees. Data alignment means putting the data at a memory address equal to some multiple of the word size (4 bytes for a 32 bit environment) Even if you do not use explicit padding, the data is stored such that these boundaries are observed, and the size of the struct will indicate padding in the total byte space used.
Structure padding - meaningless bytes placed into a structure to help align the size to be a multiple of word size. You have this in your example code.
You can use pragma macros that cause compiler to pre-process (resolve before compile) packing of a struct a certain way: example #pragma pack(n) simply sets the new alignment. Or, #pragma pack() sets the alignment to the one that was in effect when compilation started.
Example:
#pragma pack(push) /* push current alignment to stack */
#pragma pack(1) /* set alignment to 1 byte boundary */
struct MyPackedData
{
char Data1;
long Data2;
char Data3;
};
#pragma pack(pop) /* restore original alignment from stack */
Note:
The unit of n for pack(n) is byte. Values for n are compiler specific, for MSVC for example are typically 1, 2, 4, 8, and 16.
Question: If you are using prama pack macros, do they use consistent pack values between the getter()/setter() functions? (credit to #alain)
But again, this will not cause the compiler to disregard the contents of your struct, only process it a different way.
See information here and here for more information on root cause of your observations.
The longer version of my comment to #ryykers good answer:
The code you have shown in your question is perfectly valid, there is absolutely no reason why you would get the wrong values when reading the struct members in getter, provided
there is no casting
the same packing rules are in effect
Otherwise the compiler you are using would be severly broken.
The way to set the packing rules differ from compiler to compiler, they are not standardized, so maybe it's not named #pragma pack.
"Normally", there is no reason to interfere with structure packing, but one reason is sending data over a network or to a file. When the structs are packed with no padding at all, you can cast them to a void * or char * and pass the structs directly to a "send" function, for example:
send((void *)&data_to_send, sizeof(data_to_send));
The variable name data_to_send in your question is a hint that this could be what happens in this code. I'm not saying this is good practice, but it's quite common, because you don't have to write serializing code.
struct {
uint8_t foo;
uint8_t bar;
uint8_t baz;
uint8_t foos[252];
uint8_t somethingOrOther;
} A;
struct {
uint8_t foo;
uint8_t bar;
uint8_t baz;
uint8_t somethingOrOther;
uint8_t foos[252];
} B;
Does it matter that I've put foos on byte 3 in the first example, vs on byte 4 in B?
Does an array in C have to start aligned?
Is the size of this struct 256 bytes exactly?
Given that the data type is uint8_t (equivalent to unsigned char), there is no need for padding in the structure, regardless of how it is ordered. So, you can reasonably assume in this case that every compiler will make that structure into 256 bytes, regardless of the order of the elements.
If there were data elements of different sizes, then you might well get padding added and the size of the structure might vary depending on the order of the elements.
As the good book says (C11 section 6.7.2.1 paragraph 14):
Each non-bit-field member of a structure or union object is aligned in
an implementation- defined manner appropriate to its type... There
may be unnamed padding within a structure object
You didn't "put foos on byte 3" - apart from the fact that the first element is always on byte 0, you have no real control over what byte an element will be put on. The compiler can give each field a full machine word or more if it thinks that will provide the most efficient accesses. If so, it will allocate "padding" bytes in between - unused space.
Note that this only applies to structs; arrays themselves do not add any padding bytes (if they did, the pointer arithmetic/indexing rule wouldn't work), so you can be sure that the size of foos itself is exactly the declared size, and know the exact alignment of each numbered element.
No, it doesn't matter.
Pretty much every compiler out there will align struct fields to a natural boundary appropriate for the target architecture, by inserting hidden "padding bytes" between fields where necessary.
Specifically, since you're using only uint8_t, there will be no padding bytes inserted - every field already falls on a naturally-aligned boundary. (Everything is a multiple of one.)
Both of the structs you've shown are exactly 256 bytes in size. You can confirm this like so:
int main(void)
{
printf("sizeof(struct A)=%zu \n", sizeof(struct A));
printf("sizeof(struct B)=%zu \n", sizeof(struct B));
}
You can prevent this padding from being added by "packing" the structure:
Microsoft Visual C uses #pragma pack(1)
GCC uses __attribute__((packed))
Do note that doing this on a structure with unaligned members can have a serious impact on the performance of your program.
What is the best practice of using variable length array inside a structure?
Say
typedef struct foo_s {
uint32_t data_type;
uint16_t data_len;
uint8_t data[];
} foo_t;
On a x86_64 machine with Gcc 4.8, i got
sizeof(foo_t) == 8, but
offsetof(foo_t, data) == 6
looks like there is a difference there, no padding after data_len, but there is padding for the structure.
Should I keep always keep the largest member last to avoid this? i.e.
typedef struct foo_s {
uint16_t data_len;
uint32_t data_type;
uint8_t data[];
} foo_t;
What's the best practice for using var len array?
Unless you have a particular reason for wanting data to be 4-byte aligned (and if so, why is it a uint8?), the first one is mildly preferable because it'll save you a couple of bytes. For a variable-length structure like this, the value reported by sizeof is not really relevant, for exactly this reason. If you decide to allocate sizeof(foo_t) + data_len bytes for it then you'll be wasting a couple of bytes, but you'd waste them in padding in your second structure definition anyway.
If you want to pack your structs without sacrificing alignment then yes: the best option is to order elements in decreasing or increasing order. The array must be the last element, so here your best option is decreasing order (note that the size win is small, and it only matters if you have a large array of structs, but with a flexible array member you can't have an array of structs).