I am learning structure padding and packing in C.
I have this doubt, as I have read padding will depend on architecture, so does it affect inter machine communication?, ie. if data created on one machine is getting read on other machine.
How this problem is avoided in this scenario.
Yes, you cannot send the binary data of a structure between platforms and expect it to look the same on the other side.
The way you solve it is you create a marshaller/demarshaller for your construct and pass it through on the way out of one system, and on the way in to the other system. This lets the compiler take care of the buffering for you on each system.
Each side knows how to take the data, as you've specified it will be sent, and deal with it for the local platform.
Platforms such as java handle this for you by creating serialization mechanisms for your classes. In C, you'll need to do this for yourself. How you do it depends on how you want to send your data. You could serialize to binary, XML, or anything else.
#pragma pack is supported by most compilers that I know of. This can allow the programmer to specify their desired padding method for structs.
http://msdn.microsoft.com/en-us/library/2e70t5y1%28v=vs.80%29.aspx
http://gcc.gnu.org/onlinedocs/gcc/Structure_002dPacking-Pragmas.html
http://clang.llvm.org/docs/UsersManual.html#microsoft-extensions
In C/C++ a structures are used as data pack. It doesn't provide any data encapsulation or data hiding features (C++ case is an exception due to its semantic similarity with classes).
Because of the alignment requirements of various data types, every member of structure should be naturally aligned. The members of structure allocated sequentially increasing order.
It will only be affected if the code you have compiled for some other architecture uses a different padding scheme.
To help alleviate problems, I recommend that you pack structures with no padding. Where padding is required, use place-holders in (eg char reserved[2]). Also, don't use bitfields!! They are not portable.
You should also be aware of other architecture-related problems. Specifically endianness, and datatype sizes. If you need better portability, you may want to serialise and de-serialise a byte stream instead of casting it as a struct.
You can use #pragma pack(1) before the struct declaration and #pragma pack() before to disable architecture based packing; this will solve half of the problem 'cause some data types are architecture based too, to solve the second half I usually use specific data type like int_16 for 16 bits integers, u_int_32 for 32 bits integers and so on.
Take a look at http://freebsd.active-venture.com/FreeBSD-srctree/newsrc/netinet/ip_icmp.h.html ; this include describe some architecture independent network data packets.
Related
I long knew there are bit-fields in C and occasionally I use them for defining densely packed structs:
typedef struct Message_s {
unsigned int flag : 1;
unsigned int channel : 4;
unsigned int signal : 11;
} Message;
When I read open source code, I instead often find bit-masks and bit-shifting operations to store and retrieve such information in hand-rolled bit-fields. This is so common that I do not think the authors were not aware of the bit-field syntax, so I wonder if there are reasons to roll bit-fields via bit-masks and shifting operations your own instead of relying on the compiler to generate code for getting and setting such bit-fields.
Why other programmers use hand-coded bit manipulations instead of bitfields to pack multiple fields into a single word?
This answer is opinion based as the question is quite open:
Many programmers are unaware of the availability of bitfields or unsure about their portability and precise semantics. Some even distrust the compiler's ability to produce correct code. They prefer to write explicit code that they understand.
As commented by Cornstalks, this attitude is rooted in real life experience as explained in this article.
Bitfield's actual memory layout is implementation defined: if the memory layout must follow a precise specification, bitfields should not be used and hand-coded bit manipulations may be required.
The handing of signed values in signed typed bitfields is implementation defined. If signed values are packed into a range of bits, it may be more reliable to hand-code the access functions.
Are there reasons to avoid bitfield-structs?
bitfield-structs come with some limitations:
Bit fields result in non-portable code. Also, the bit field length has a high dependency on word size.
Reading (using scanf()) and using pointers on bit fields is not possible due to non-addressability.
Bit fields are used to pack more variables into a smaller data space, but cause the compiler to generate additional code to manipulate these variables. This results in an increase in both space as well as time complexities.
The sizeof() operator cannot be applied to the bit fields, since sizeof() yields the result in bytes and not in bits.
Source
So whether you should use them or not depends. Read more in Why bit endianness is an issue in bitfields?
PS: When to use bit-fields in C?
There is no reason for it. Bitfields are useful and convenient. They are in the common use in the embedded projects. Some architectures (like ARM) have even special instructions to manipulate bitfields.
Just compare the code (and write the rest of the function foo1)
https://godbolt.org/g/72b3vY
In many cases, it is useful to be able to address individual groups of bits within a word, or to operate on a word as a unit. The Standard presently does not provide
any practical and portable way to achieve such functionality. If code is written to use bitfields and it later becomes necessary to access multiple groups as a word, there would be no nice way to accommodate that without reworking all the code using the bit fields or disabling type-based aliasing optimizations, using type punning, and hoping everything gets laid out as expected.
Using shifts and masks may be inelegant, but until C provides a means of treating an explicitly-designated sequence of bits within one lvalue as another lvalue, it is often the best way to ensure that code will be adaptable to meet needs.
We use the below structure in code running on a 32-bit machine. If we have to transfer this stucture to a 64-bit machine, is there any change required?
struct test
{
int num;
char a;
double dd;
};
i have two machine on network and i have two transfer data stored in above mention structure from 32 bit machine to 64 bit machine so how to make the above mention structure in generic structure so that data will not loose... this is my question.
The layout of such a structure is completely platform-dependent and you can't even use it to transfer data between two instances of a 32 bit application compiled using different compilers, or different compile settings under the same compiler.
The only safe use for such a structure in data transfer is between multiple instances of the same executable. Same as in: same build. You can't even generally guarantee that some later build will have the same structure.
To transfer binary data in a binary-compatible fashion, you need to use some kind of a binary stream that maintains a fixed binary structure, independent of the platform. Google Protocol Buffers are one example of such, another is Qt's QDataStream.
Generally the struct is not really adequate to use for network or persistency purposes, as it relies in too many ways on the C implementation (compiler + platform).
"Transferring" depends on what you're doing with the struct and contained elements.
These items should be on you checklist:
Check elements for value ranges. All used types may change in width. char may change in signedness.
Check the whole structure's size. This might be important for code relying on a specific size or some arbitrary bounds.
When leaving the process's address space (network or persistently storing) make sure that the struct's are properly migrated, incl. endings, size, alignment.
Everything depends heavily on the used C implementations on the different platforms.
On our ASIC we have two processors, which means two different compilers that behave slightly differently. We pass structs full of data between the two for communication, this happens quite often per second so we don't have much time to burn here.
The problem is that both compilers treat padding differently. So when we go to copy data from one domain to the other, we get values that don't align correctly. Our initial solution was to put attribute((packed)) on everything inside the struct. While this seems to work most of the time, it is by no means portable. As we are moving the code to different platforms, I'm noticing that not all compilers understand attribute((packed)) and I'd like to keep our code portable.
Has anyone here dealt with this kind of issue? What would you recommend?
Thanks in advance!
I would manually pack such structures, so that there is no padding issue with any compiler, present or future.
It is tedious, but it is a portable future-proof solution, worth the effort.
Fundamentally struct arrangement in C isn't portable, and so __attribute__((packed)) or similar is generally the typical way to impose a fixed layout on a struct.
The other option is to manually add pad fields in appropriate places and be aware of each platforms' alignment constraints to ensure that across your two platforms the two structures match - but this is essentially manual attribute((packed)).
Note that the pahole utility from dwarves (announcement, paper) is a great tool to check and view the alignment of structures, assuming your compiler emits ELF files.
This is the reason why you shouldn't use structs for data protocols. It might seem like a harsh thing to say, but structs are unfortunately non-portable and therefore dangerous. You could consider doing something like this (pseudo code):
typedef struct
{
uint32_t x;
uint16_t y;
...
} my_struct_t; // some custom struct
#define PROTOCOL_SIZE ( sizeof(uint32_t) + sizeof(uint16_t) + ... )
void pack (uint8_t raw_data[PROTOCOL_SIZE],
const my_struct_t* ms)
{
uint16_t i=0;
memcpy(&raw_data[i], ms->x, sizeof(ms->x));
i += sizeof(ms->x);
memcpy(&raw_data[i], ms->y, sizeof(ms->y));
i += sizeof(ms->y);
...
}
And then make a similar unpack() function which copies raw data into the struct.
Advantages of this is: 100% portable. And if the protocol specifies a particular endianess, this function could also handle that conversion (which you would have to do anyhow.)
Disadvantages are one extra memory buffer and some extra data copying.
I have a question about structure padding and memory alignment optimizations regarding structures in C language. I am sending a structure over the network, I know that, for run-time optimizations purposes, the memory inside a structure is not contiguous. I've run some tests on my local computer and indeed, sizeof(my_structure) was different than the sum of all my structure members. I ran some research to find out two things :
First, the sizeof() operator retrieves the padded size of the structure (i.e the real size that would be stored in memory).
When specifying __attribute__((__packed__)) in the declaration of the structure this optimization is disabled by the compiler, so sizeof(my_structure) will be exactly the same as the sum of the fields of my structure.
That being said, i am wondering if the sizeof operator was getting the padded size on every compilers implementation and on every architecture, in other words, is it always safe to copy a structure with memcpy for example using the sizeof operator such as :
memcpy(struct_dest, struct_src, sizeof(struct_src));
I am also wondering what is the real purpose of __attribute__((__packed__)), is it used to send a less important amount the data on a network when submitting a structure or is it, in fact, used to avoid some unspecified and platform-dependant sizeof operator behaviour ?
Thanks by advance.
Different compilers on different architectures can and do use different padding. So for wire transmission it is not uncommon to pack structs to achieve a consistent binary layout. This can then cater for the code at each end of the wire running on different architecture.
However you also need to make sure that your data types are the same size if you use this approach. For example, on 64 bit systems, long is 4 bytes on Windows and 8 bytes almost everywhere else. And you also need to deal with endianness issues. The standard is to transmit over the wire in network byte order. In practice you would be better using a dedicated serialization library rather than trying to reinvent solutions to all these issues.
I am sending a structure over the network
Stop there. Perhaps some would disagree with me on this (in practice you do see a lot of projects doing this), but struct is a way of laying out things in memory - it's not a serialization mechanism. By using this tool for the job, you're already tying yourself to a bunch of non-portable assumptions.
Sure, you may be able to fake it with things like structure padding pragmas and attributes, but - can you really? Even with those non-portable mechanisms you never know what quirks might show up. I recall working in a code base where "packed" structures were used, then suddenly taking it to a platform where access had to be word aligned... even though it was nominally the same compiler (thus supported the same proprietary extensions) it produced binaries which crashed. Any pain you get from this path is probably deserved, and I would say only take it if you can be 100% sure it will only run in a given compiler and environment, and that will never change. I'd say the safer bet is to write a proper serialization mechanism that doesn't allow writing structures around across process boundaries.
Is it always safe to copy a structure with memcpy for example using the sizeof operator
Yes, it is and that is the purpose of providing the sizeof operator.
Usually __attribute__((__packed__)) is used not for size considerations but when you want want to to make sure of the layout of a structure is exactly as you want it to be.
For ex:
If a structure is to be used to match hardware or be sent on a wire then it needs to have the exact same layout without any padding.This is because different architectures usually implement different kinds & amounts of padding and alignment and the only way to ensure common ground is to remove padding out out of the picture by using packing.
I've lots of different structs containing enum members that I have to transmit via TCP/IP. While the communication endpoints are on different operating systems (Windows XP and Linux) meaning different compilers (gcc 4.x.x and MSVC 2008) both program parts share the same header files with type declarations.
For performance reasons, the structures should be transmitted directly (see code sample below) without expensively serializing or streaming the members inside.
So the question is how to ensure that both compilers use the same internal memory representation for the enumeration members (i.e. both use 32-bit unsigned integers). Or if there is a better way to solve this problem...
//type and enum declaration
typedef enum
{
A = 1,
B = 2,
C = 3
} eParameter;
typedef enum
{
READY = 400,
RUNNING = 401,
BLOCKED = 402
FINISHED = 403
} eState;
#pragma pack(push,1)
typedef struct
{
eParameter mParameter;
eState mState;
int32_t miSomeValue;
uint8_t miAnotherValue;
...
} tStateMessage;
#pragma pack(pop)
//... send via socket
tStateMessage msg;
send(iSocketFD,(void*)(&msg),sizeof(tStateMessage));
//... receive message on the other side
tStateMessage msg_received;
recv(iSocketFD,(void*)(&msg_received),sizeof(tStateMessage));
Additionally...
Since both endpoints are little endian maschines, endianess is not a problem here.
And the pack #pragma solves alignment issues satisfactorily.
Thx for your answers,
Axel
I'll answer your question pragmatically because you've chosen a relatively risky path after weighing the performance gains against the possible downsides (at least I hope you have!).
If portability and robustness against future changes to those compilers have also been considered then an empirical approach would be the best guard against problems.
Ensure you are using initializers for the enums (your examples do this) in all cases.
Do empirical testing to see how the data is interpreted on the receiving side.
Record the version numbers of the build tools on both sides and archive them with the source code. Preferably archive the tools as well.
Document everything you did so unforeseen maintenance in the future is not handicapped.
Pray for the best! ;-)
I would advise you to use one of the serialization libraries specially designed for such problems, like:
Apache Avro (tutorial)
Facebook's Thrift (tutorial)
Google's Protocol Buffers (tutorial)
What you will get is maximum platform portability, an easy way of changing the interface and the type of messages transmitted plus a lot more useful features.
Note that only Avro has an officially supported C API. For Thrift and Protocol Buffers you either make a thin C wrapper over the C++ API or use one of the C APIs, like protobuf-c.
This is premature optimization. You have made two costly assumptions without measurements.
The first assumption is that this part of the code is a performance bottleneck in the first place. Is it? Very unlikely. If one is going to make assumptions about performance, then the safe assumption is that the network speed will be the bottleneck, not the code which sends and receives the network messages. This alone should prevent you from ever considering the second assumption.
The second assumption is that serializing the struct portably will be noticeably slower than writing the raw bits of the struct. This assumption is nearly always false.
Skeptical? Measure it! :)
If you don't want to go through serialization, one method I've seen used is to eschew enums and simply use 32-bit unsigned ints and #DEFINEs to emulate enums. You trade away some type safety for some assurances about data format.
Otherwise, you are relying on behaviour that isn't guarenteed in the language specification to be implemented the same way on all your compilers. If you aren't worried about general portability and just want to ensure the same effect on two compilers, it should be possible through trial and error and a lot of testing to get the two to do the same thing. I believe the C99 spec allows enums to internally be the size of int or smaller, but not larger than int. So one thing I've seen done to supposedly hint the compiler in the right direction is:
typedef enum
{
READY = 400,
RUNNING = 401,
BLOCKED = 402,
FINISHED = 403,
MAX = MAX_INT
} eState;
This should limit the compiler's choices for how to store the enum. Note that compilers can violate the standard, however, I know gcc has a non-standard feature where it will allow 64-bit enums if necessary.
Also, check out:
What is the size of an enum in C?
It is strongly recommended to serialize the data in some way or at least use an indicator about the hardware architecture. Even if you use the same compiler, you can have problems with internal data representations (little endian, big endian etc).