I have a struct like this:
typedef struct _HEADER_IO
{
uint8_t field1 : 2;
uint8_t field2 : 4;
uint8_t field3 : 1;
uint8_t field4 : 1;
uint16_t field5;
uint8_t field6;
} HEADER_IO;
It's basicly a message header that will be sent over tcp. The server reads this so that it knows what data follows in the buffer. However for some reason intead of the size being 4 bytes (2+4+1+1 first byte + 2 bytes from field 5 + 1 byte field 6) the size is 6 bytes.
Looking it up in memory view it is:
XX AA XX XX XX AA
Instead of:
XX XX XX XX
Where AA are never set no matter what I do. This is a problem because I am planning for the header to be send() to a server and the extra bytes are included making the server interpret the header wrong. What am I doing wrong?
In general, it's a bad idea to use bitfields for things like these. Since you can't know beforehand exactly which byte the bits will end up in, and since there are padding and alignment issues.
In my opinion, it's better to "own up" to the fact that you need more control over the external representation than what C structures give you, and do it manually. You can of course keep the struct as the in-memory (internal) representation.
Basically, you would write a function like:
size_t header_serialize(unsigned char *buf, size_t max, const HEADER_IO *header);
whose job it would be to, in the memory at buf, build the proper byte sequence that represents header.
To clarify (based on comments), the intent is to read the fields from header, not just do e.g.
memcpy(buf, header, sizeof *header); /* DON'T DO THIS! */
Instead, you're supposed to assemble the expected external representation, byte by byte, from the fields of header. That way, you always get the same external representation regardless of what the compiler does to the in-memory format of header.
In standard C you can't help the fact that struct members can have padding inserted between them. You have to write a function to decode the data and store it in your struct before processing. This is because on some architectures unaligned memory access (reading from a pointer not aligned to, for example, 4 bytes) is very expensive and C will automatically pad your structures to avoid the cost. There's no standard way to turn the feature on or off.
For example in GCC you can add __attribute__((packed)) after the struct definition and Visual Studio has some #pragma commands (see http://gcc.gnu.org/onlinedocs/gcc/Structure_002dPacking-Pragmas.html) that are also supported by GCC but beware that overall this is non-standard.
Since your comments mentioned it's a Windows program, probably it would work if you add this before the struct definition:
#pragma pack(push,1)
And this after it:
#pragma pack(pop)
While it would be more portable to write code to more manually decode the header, the above approach should be faster.
Related
In a C program, having an array that is meant to work as a buffer FooBuffer[] for storing the contents of a data member of a struct like this:
struct Foo {
uint64_t data;
};
I was told that this line might cause unaligned access:
uint8_t FooBuffer[10] = {0U};
I have some knowledge that unaligned access depends on the alignment offset of the processor and in general, it consumes more read/write cycles. Under what circumstances would this cause unaligned memory access and how could I prevent it?
Edit:
A variable of type struct Foo would be stored in the buffer. Particularly, its member data would be split up into eight bytes that would be stored in the array FooBuffer. See attached code with some options for this.
#include <stdio.h>
#include <string.h>
typedef unsigned long uint64;
typedef unsigned char uint8;
struct Foo
{
uint64 data;
};
int main()
{
struct Foo foo1 = {0x0123456789001122};
uint8 FooBuffer[10] = {0U};
FooBuffer[0] = (uint8)(foo1.data);
FooBuffer[1] = (uint8)(foo1.data >> 8);
FooBuffer[2] = (uint8)(foo1.data >> 16);
FooBuffer[3] = (uint8)(foo1.data >> 24);
FooBuffer[4] = (uint8)(foo1.data >> 32);
FooBuffer[5] = (uint8)(foo1.data >> 40);
FooBuffer[6] = (uint8)(foo1.data >> 48);
FooBuffer[7] = (uint8)(foo1.data >> 56);
struct Foo foo2 = {0x9876543210112233};
uint8 FooBuffer2[10] = {0U};
memcpy(FooBuffer2, &foo2, sizeof(foo2));
return 0;
}
However, it is not clear how this process is done since a piece of privative software performs the operation. What would be the scenarios that could result in unaligned memory access after the "conversion"?
Defining either a structure such as struct Foo { uint64_t data; } or an array such as uint8_t FooBuffer[10]; and using them in normal ways will not cause an unaligned access. (Why did you use 10 for FooBuffer? Only 8 bytes are needed for this example?)
A method that novices sometimes attempt that can cause unaligned accesses is attempting to reinterpret an array of bytes as a data structure. For example, consider:
// Get raw bytes from network or somewhere.
uint8_t FooBuffer[10];
CallRoutineToReadBytes(FooBuffer,...);
// Reinterpret bytes as original type.
struct Foo x = * (struct Foo *) FooBuffer; // Never do this!
The problem here is that struct Foo has some alignment requirement, but FooBuffer does not. So FooBuffer could be at any address, but the cast to struct Foo * attempts to force it to an address for a struct Foo. If the alignment is not correct, the behavior is not defined by the C standard. Even if the system allows it and the program “works,” it may be accessing a struct Foo at an improperly aligned address and suffering performance problems.
To avoid this, a proper way to reinterpret bytes is to copy them into a new object:
struct Foo x;
memcpy(&x, FooBuffer, sizeof x);
Often a compiler will recognize what is happening here and, especially if struct Foo is not large, implement the memcpy in an efficient way, perhaps as two load-four-byte instructions or one load-eight-byte instruction.
Something you can do to help that along is ask the compiler to align FooBuffer by declaring it with the _Alignas keyword:
uint8_t _Alignas(Struct Foo) FooBuffer[10];
Note that that might not help if you need to take bytes from the middle of a buffer, such as from a network message that includes preceding protocol bytes and other data. And, even if it does give the desired alignment, never use the * (struct Foo *) FooBuffer shown above. It has more problems than just alignment, one of which is that the C standard does not guarantee the behavior of reinterpreting data like this. (A supported way to do it in C is through unions, but memcpy is a fine solution.)
In the code you show, bytes are copied from foo1.data to FooBuffer using bit shifts. This also will not cause alignment problems; expressions that manipulate data like this work just fine. But there are two issues with it. One is that it nominally manipulates individual bytes one by one. That is perfectly legal in C, but it can be slow. A compiler might optimize it, and there might be built-ins or library functions to assist with it, depending on your platform.
The other issue is that it puts the bytes in an order according to their position values: The low-position-value bytes are put into the buffer first. In contrast, the memcpy method copies the bytes in the order they are stored in memory. Which method you want to use depends on the problem you are trying to solve. To store data on one system and read it back later on the same system, the memcpy method is fine. To send data between two systems using the same byte ordering, the memcpy method is fine. However, if you want to send data from one system on the Internet to another, and the two systems do not use the same byte order in memory, you need to agree on an order to use in the network packages. In this case, it is common to use the arrange-bytes-by-position-value method. Again, your platform may have builtins or library routines to assist with this. For example, the htonl and ntohl routines are BSD routines that take a normal 32-bit unsigned integer and return it with its bytes arranged for network order or vice-versa.
I know that some structs, may, or may not, add padding between elements.
My current project is reading input from /dev/input files. The binary layout of these files is defined in <linux/input.h>:
struct input_event {
struct timeval time;
__u16 type;
__u16 code;
__s32 value;
};
I wonder though that this struct is not marked with a packed attribute. Meaning that the /dev/input files (which are packed bit by bit) are not guaranteed to match the same package as the struct. Thus the logic
struct input_event event;
read(fd, &event, sizeof(event));
Is not defined to work across all archs.
Do I have a fallacy in my logic? Or is it safe to assume that somethings are not going to be packed?
Packing by Layout
In the current case, you are safe. Your struct input_event is allready layed out packed.
struct timeval time; /* 8 bytes */
__u16 type; /* 2 bytes */
__u16 code; /* 2 bytes */
__s32 value; /* 4 bytes */
This means, the members form clean 32Bit blocks and so there is no padding required. This article explains how the size of struct members (especially chars) and their layout affect the padding and thus also the final size of a struct.
Packing by the Preprocessor
Packing structs via the preprocessor seems to be a good solution at a first sight. Looking a little closer, there show up several downsides and one of those hits you at the point you are caring about (see also #pragma pack effect)
Performance
Padding assures that your struct members are accessible without searching inside memoryblocks (4byte blocks on a 32bit machine and 8byte blocks on a 64bit machine respectively). In consequence, packing such structures leads to members spanning over multiple memoryblocks and therefore require the machine to search for them.
Different Platforms (Archs, Compilers)
Preprocessor instructions are heavily vendor and architecture specific. Thus, using them leads to lesser or in worst case non-portability of your code.
Conclusion
As the author of this article (already mentioned above) states, even NTP directly reads data from the network into structures.
So, carefully laying out your structures and maybe padding them by hand might be the safest and also most portable solution.
If you insist on directly loading structures into memory from binary images, then C is not your friend. Padding is allowed, and the basic types can have different widths and endiannness. You are not even guaranteed 8 bits in a byte. However packing the structures and sticking to int32_t and so on will help a great deal, it's effectively portable bar endiannness.
But the best solution is to load the structure from the stream portably. This is even possible with real numbers, though a bit fiddly.
This is how to read a 16 bit integer portably. See my github project for the rest of the functions (similar logic)
https://github.com/MalcolmMcLean/ieee754
/**
Get a 16-bit big-endian signed integer from a stream.
Does not break, regardless of host integer representation.
#param[in] fp - pointer to a stream opened for reading in binary mode
# returns the 16 bit value as an integer
*/
int fget16be(FILE *fp)
{
int c1, c2;
c2 = fgetc(fp);
c1 = fgetc(fp);
return ((c2 ^ 128) - 128) * 256 + c1;
}
I'm trying to receive a record in binary from a PIC programmed in C.
The data sent is structured so:
typedef struct{int32 num1;
float num2,num3,num4,num5;
...
}RecordStructure;
typedef union{RecordStructure Record; char Array[48];} My_Rec_Structure;
My question is this:
Do the first 4 bytes/chars belong to the int32 num1, the next 4 to float num2 and so on?
If so I'm having major issues with transmission; if not, how do I discern where the data ended up?
Yes on the first part, but with a caveat.
The C standard requries that the first struct member is placed at the beginning of the structure. So the int32 will be the first four bytes transferred. However, the PIC might have a different byte ordering, so you might end up sending 0x12345678 from the PIC, and reading the value 0x78563412 at the receiving end.
As for the adressing of the remaining struct members, your compiler are free to add padding between each member. Typically this is something that's done in order to optimize memory access.
If you have the posibility to get some text output from your PIC, you can get the offsets for the structure members by using the offsetof() macro from stddef.h:
fprintf(stderr, "num2 offset=%d\n", offsetof(RecordStructure, num2));
fprintf(stderr, "num3 offset=%d\n", offsetof(RecordStructure, num3));
How data is aligned within a struct can vary between compilers as well as the pragmas used to force a particular alignment.
If this compiler aligns items on a double word boundary then the first 4 bytes, num1, should line up with the first 4 bytes of the char Array, the second four bytes, num2, with the second four bytes of the char Array, etc.
If possible what you can do is to use a debugger to examine the data when it is received. And in the data that is sent you put a specific hex digit sequence so that you can know whether things are lining up as you think or not.
So on the PIC the data sent in the struct might be:
RecordStructure myRecord;
myRecord.num1 = 01F2E3D4C;
myRecord.num5 = myRecord.num4 = myRecord.num3 = myRecord.num2 = 0;
Then send that across to see what it looks like. and whether the num1 value is the same.
If you then modify the data for instance if myRecord.num3 = 1.0 you will be able to see if things line up or not.
Yes. The first four bytes will belong to num1, then the next four bytes will belong to num2, etc. The byte ordering within the ints and floats, however, are platform and implementation-specific.
There are two major things that will affect this, however. One is alignment and the other is padding. Make sure to set padding to 1 byte to ensure that you get no memory gaps between the variables and set alignment to 1 if possible as well. But depending on your compiler and hardware, alignment may have to be a multiple of 4 or 8 bytes for example. In this case you may need to add some padding variables. You can use the align keyword in MSVC and the alignment attribute in GCC to change the default alignment per-structure/variable:
__declspec(align(1)) //MSVC
__attribute__((aligned(1))) //GCC
Example:
#pragma pack(1)
typedef struct{int32 num1;
float num2,num3,num4,num5;
...
}RecordStructure;
typedef union{RecordStructure Record; char Array[48];} My_Rec_Structure;
This example shows how the same definition of ~RecordStructure~ can be used on both sides, PC (Linux/x86/gcc) and PIC(18F). The trick is to use #ifdef / #define to support both compilers.
#ifdef __cplusplus
# define __PACKED
# define __PACKED2 __attribute__((packed))
#else
# define __PACKED2
#endif
typedef struct __PACKED {
opcodes_t opcode : 8;
union {
osci_config_t set_config_args;
READ_SINGLE_ARGS read_single_args;
SAMPLE_SINGLE_ARGS sample_single_args;
SAMPLE_INTERLEAVED_ARGS sample_interleaved_args;
} args;
} __PACKED2 opcode_decoder_t;
I'm coding a network layer protocol and it is required to find a size of packed a structure defined in C. Since compilers may add extra padding bytes which makes sizeof function useless in my case. I looked up Google and find that we could use ___attribute(packed)___ something like this to prevent compiler from adding extra padding bytes. But I believe this is not portable approach, my code needs to support both windows and linux environment.
Currently, I've defined a macro to map packed sizes of every structure defined in my code. Consider code below:
typedef struct {
...
} a_t;
typedef struct {
...
} b_t;
#define SIZE_a_t 8;
#define SIZE_b_t 10;
#define SIZEOF(XX) SIZE_##XX;
and then in main function, I can use above macro definition as below:-
int size = SIZEOF(a_t);
This approach does work, but I believe it may not be best approach. Any suggestions or ideas on how to efficiently solve this problem in C?
Example
Consider the C structure below:-
typedef struct {
uint8_t a;
uint16_t b;
} e_t;
Under Linux, sizeof function return 4 bytes instead of 3 bytes. To prevent this I'm currently doing this:-
typedef struct {
uint8_t a;
uint16_t b;
} e_t;
#define SIZE_e_t 3
#define SIZEOF(XX) SIZE_##e_t
Now, when I call SIZEOF(e_t) in my functin, it should return 3 not 4.
sizeof is the portable way to find the size of a struct, or of any other C data type.
The problem you're facing is how to ensure that your struct has the size and layout that you need.
#pragma pack or __attribute__((packed)) may well do the job for you. It's not 100% portable (there's no mention of packing in the C standard), but it may be portable enough for your current purposes, but consider whether your code might need to be ported to some other platform in the future. It's also potentially unsafe; see this question and this answer.
The only 100% portable approach is to use arrays of unsigned char and keep track of which fields occupy which ranges of bytes. This is a lot more cumbersome, of course.
Your macro tells you the size that you think the struct should have, if it has been laid out as you intend.
If that's not equal to sizeof(a_t), then whatever code you write that thinks it is packed isn't going to work anyway. Assuming they're equal, you might as well just use sizeof(a_t) for all purposes. If they're not equal then you should be using it only for some kind of check that SIZEOF(a_t) == sizeof(a_t), which will fail and prevent your non-working code from compiling.
So it follows that you might as well just put the check in the header file that sizeof(a_t) == 8, and not bother defining SIZEOF.
That's all aside from the fact that SIZEOF doesn't really behave like sizeof. For example consider typedef a_t foo; sizeof(foo);, which obviously won't work with SIZEOF.
I don't think, that specifying size manually is more portable, than using sizeof.
If size is changed your const-specified size will be wrong.
Attribute packed is portable. In Visual Studio it is #pragma pack.
I would recommend against trying to read/write data by overlaying it on a struct. I would suggest instead writing a family of routines which are conceptually like printf/scanf, but which use format specifiers that specify binary data formats. Rather than using percent-sign-based tags, I would suggest simply using a binary encoding of the data format.
There are a few approaches one could take, involving trade-off between the size of the serialization/deserialization routines themselves, the size of the code necessary to use them, and the ability to handle a variety of deserialization formats. The simplest (and most easily portable) approach would be to have routines which, instead of using a format string, process items individually by taking a double-indirect pointer, read some data type from it, and increment it suitably. Thus:
uint32_t read_uint32_bigendian(uint8_t const ** src)
{
uint8_t *p;
uint32_t tmp;
p = *src;
tmp = (*p++) << 24;
tmp |= (*p++) << 16;
tmp |= (*p++) << 8;
tmp |= (*p++);
*src = p;
}
...
char buff[256];
...
uint8_t *buffptr = buff;
first_word = read_uint32_bigendian(&buffptr);
next_word = read_uint32_bigendian(&buffptr);
This approach is simple, but has the disadvantage of having lots of redundancy in the packing and unpacking code. Adding a format string could simplify it:
#define BIGEND_INT32 "\x43" // Or whatever the appropriate token would be
uint8_t *buffptr = buff;
read_data(&buffptr, BIGEND_INT32 BIGEND_INT32, &first_word, &second_word);
This approach could read any number of data items with a single function call, passing buffptr only once, rather than once per data item. On some systems, it might still be a bit slow. An alternative approach would be to pass in a string indicating what sort of data should be received from the source, and then also pass in a string or structure indicating where the data should go. This could allow any amount of data to be parsed by a single call giving a double-indirect pointer for the source, a string pointer indicating the format of data at the source, a pointer to a struct indicating how the data should be unpacked, and a a pointer to a struct to hold the target data.
I am porting an application to an ARM platform in C, the application also runs on an x86 processor, and must be backward compatible.
I am now having some issues with variable alignment. I have read the gcc manual for
__attribute__((aligned(4),packed)) I interpret what is being said as the start of the struct is aligned to the 4 byte boundry and the inside remains untouched because of the packed statement.
originally I had this but occasionally it gets placed unaligned with the 4 byte boundary.
typedef struct
{
unsigned int code;
unsigned int length;
unsigned int seq;
unsigned int request;
unsigned char nonce[16];
unsigned short crc;
} __attribute__((packed)) CHALLENGE;
so I change it to this.
typedef struct
{
unsigned int code;
unsigned int length;
unsigned int seq;
unsigned int request;
unsigned char nonce[16];
unsigned short crc;
} __attribute__((aligned(4),packed)) CHALLENGE;
The understand I stated earlier seems to be incorrect as both the struct is now aligned to a 4 byte boundary, and and the inside data is now aligned to a four byte boundary, but because of the endianess, the size of the struct has increased in size from 42 to 44 bytes. This size is critical as we have other applications that depend on the struct being 42 bytes.
Could some describe to me how to perform the operation that I require. Any help is much appreciated.
If you're depending on sizeof(yourstruct) being 42 bytes, you're about to be bitten by a world of non-portable assumptions. You haven't said what this is for, but it seems likely that the endianness of the struct contents matters as well, so you may also have a mismatch with the x86 there too.
In this situation I think the only sure-fire way to cope is to use unsigned char[42] in the parts where it matters. Start by writing a precise specification of exactly what fields are where in this 42-byte block, and what endian, then use that definition to write some code to translate between that and a struct you can interact with. The code will likely be either all-at-once serialisation code (aka marshalling), or a bunch of getters and setters.
This is one reason why reading whole structs instead of memberwise fails, and should be avoided.
In this case, packing plus aligning at 4 means there will be two bytes of padding. This happens because the size must be compatible for storing the type in an array with all items still aligned at 4.
I imagine you have something like:
read(fd, &obj, sizeof obj)
Because you don't want to read those 2 padding bytes which belong to different data, you have to specify the size explicitly:
read(fd, &obj, 42)
Which you can keep maintainable:
typedef struct {
//...
enum { read_size = 42 };
} __attribute__((aligned(4),packed)) CHALLENGE;
// ...
read(fd, &obj, obj.read_size)
Or, if you can't use some features of C++ in your C:
typedef struct {
//...
} __attribute__((aligned(4),packed)) CHALLENGE;
enum { CHALLENGE_read_size = 42 };
// ...
read(fd, &obj, CHALLENGE_read_size)
At the next refactoring opportunity, I would strongly suggest you start reading each member individually, which can easily be encapsulated within a function.
I've been moving structures back and forth from Linux, Windows, Mac, C, Swift, Assembly, etc.
The problem is NOT that it can't be done, the problem is that you can't be lazy and must understand your tools.
I don't see why you can't use:
typedef struct
{
unsigned int code;
unsigned int length;
unsigned int seq;
unsigned int request;
unsigned char nonce[16];
unsigned short crc;
} __attribute__((packed)) CHALLENGE;
You can use it and it doesn't require any special or clever code. I write a LOT of code that communicates to ARM. Structures are what make things work. __attribute__ ((packed)) is my friend.
The odds of being in a "world of hurt" are nil if you understand what is going on with both.
Finally, I can't for the life make out how you get 42 or 44. Int is either 4 or 8 bytes (depending on the compiler). That puts the number at either 16+16+2=34 or 32+16+2=50 -- assuming it is truly packed.
As I say, knowing your tools is part of your problem.
What is your true goal?
If it's to deal with data that's in a file or on the wire in a particular format what you should do is write up some marshaling/serialization routines that move the data between the compiler struct that represents how you want to deal with the data inside the program and a char array that deals with how the data looks on the wire/file.
Then all that needs to be dealt with carefully and possibly have platform specific code is the marshaling routines. And you can write some nice-n-nasty unit tests to ensure that the marshaled data gets to and from the struct properly no matter what platform you might have to port to today and in the future.
I would guess that the problem is that 42 isn't divisible by 4, and so they get out of alignment if you put several of these structs back to back (e.g. allocate memory for several of them, determining the size with sizeof). Having the size as 44 forces the alignment in these cases as you requested. However, if the internal offset of each struct member remains the same, you can treat the 44 byte struct as though it was 42 bytes (as long as you take care to align any following data at the correct boundary).
One trick to try might be putting both of these structs inside a single union type and only use 42-byte version from within each such union.
As I am using linux, I have found that by echo 3 > /proc/cpu/alignment it will issue me with a warning, and fix the alignment issue. This is a work around but it is very helpful with locating where the structures are failing to be misaligned.