I want to synchronize two Raspberry Pi's with a C program. It is working fine, if the program only is running on the Pi's, but for development I want to use my PC (where its also easier to debug), but I send the timespec struct directly as binary over the wire. A raspberry is using 4bytes for long and time_t, my PC is using 8byte each... So they do not come together.
Is it possible to set long and time_t to 4byte each, only for this C script?
I know that the size of long, short, etc. is defined by the system.
Important: I only want to define it once in the script and not transforming it to uintXX or int each time.
In programming, it is not uncommon to need to treat network transmissions as separate from in-memory handling. In fact, it is pretty much the norm. So converting it to a network format of the proper byte order and size is really recommended and while help with the abstractions for your interfaces.
You might as well consider transforming to plain text, if that is not a time-critical piece of data exchange. It makes for a lot easier debugging.
C is probably not the best tool for the job here. It's much too low level to provide automatic data serialization like JavaScript, Python or similar more abstract languages.
You cannot assume the definitions of timespec will be identical on different platforms. For one thing the size of an int will be different depending on the 32/64 bits architecture, and you can have endianing problems too.
When you want to exchange data structures between heterogeneous platforms, you need to define your own protocol with unambiguous data and a clear endianing convention.
One solution would be to send the numbers as ASCII. Not terribly efficient, but if it's just a couple of values, who cares?
Another would be to create an exchange structure with (u)intXX_t fields.
You can assume a standard raspberry kernel will be little endian like your PC, but if you're writing a small exchange protocol, you might as well add a couple of htonl/ntohl for good measure.
Related
We use the below structure in code running on a 32-bit machine. If we have to transfer this stucture to a 64-bit machine, is there any change required?
struct test
{
int num;
char a;
double dd;
};
i have two machine on network and i have two transfer data stored in above mention structure from 32 bit machine to 64 bit machine so how to make the above mention structure in generic structure so that data will not loose... this is my question.
The layout of such a structure is completely platform-dependent and you can't even use it to transfer data between two instances of a 32 bit application compiled using different compilers, or different compile settings under the same compiler.
The only safe use for such a structure in data transfer is between multiple instances of the same executable. Same as in: same build. You can't even generally guarantee that some later build will have the same structure.
To transfer binary data in a binary-compatible fashion, you need to use some kind of a binary stream that maintains a fixed binary structure, independent of the platform. Google Protocol Buffers are one example of such, another is Qt's QDataStream.
Generally the struct is not really adequate to use for network or persistency purposes, as it relies in too many ways on the C implementation (compiler + platform).
"Transferring" depends on what you're doing with the struct and contained elements.
These items should be on you checklist:
Check elements for value ranges. All used types may change in width. char may change in signedness.
Check the whole structure's size. This might be important for code relying on a specific size or some arbitrary bounds.
When leaving the process's address space (network or persistently storing) make sure that the struct's are properly migrated, incl. endings, size, alignment.
Everything depends heavily on the used C implementations on the different platforms.
I am working on a project that is using Unix domain socket (AF_UNIX) as a choice of IPC between different processes.
When I want to pass a data structure from one process to another, do I need to do serialization on the data structure as mentioned in this question (Passing a structure through Sockets in C)?
Since these processes are compiled with same compiler and running on the same machine, there should be no endianness , nor different padding issue. So I am not sure if serialization is necessary.
You need only ensure that the received structure is intelligible.
If the structure is composed of self-contained types then no processing is required, you can just call write() or send() to push the data into the socket.
Serialisation is needed where the structure is not self-contained ( eg if it contains pointers, or platform-specific data types)
If there a chance that the two processes could have different bit-ness
(eg 32 bit vis 64 bit) or different endian-ness you'll want to take care that the struct is well-defined such that it comes out with the same binary representation in both forms.
Serialization is not necessary in this case. Every operating system and CPU architecture combination will have a quite well defined ABI which says how structs and such are laid out in memory. This severely limits the compiler in how much it is allowed to change things around and for a good reason - change the ABI and all precompiled libraries stop working. So if you compile stuff with the same compiler targeting the same architecture the in-memory layout of structs will be the same.
To be sure, just remember to rebuild both sides on major operating system updates in case the ABI changes (which it never does, but it could happen some day).
Okay I hate to ask this question ... but here goes. I am writing some code in C on an x86 machine. I want to send a struct over the network and and I want to convert the struct to Network Byte Order ... I understand all the drama about packing and the the gcc packing pragmas ... what I want to know is HOW do I convert a struct (or an array or any such arbitrary memory blob) to network byte order.
Is there a standard (Unix/Linux/Posix) function call I can use or must I roll my own.
x
In principle, you can go through the struct and call htonl or htons on each uint32_t or uint16_t field, respectively, assigning the results back or to a copy of the struct. However, I would not recommend this sort of approach. It's very fragile and subject to struct alignment issues, etc.
Unless transmitting and receiving data is extremely performance-critical, I would simply implement proper serialize and deserialize methods for your structures. You can write numeric values out one byte at a time in binary format, choosing whether you want to write to least significant or most significant part first. But really, I would recommend choosing a modern text-based serialization format like json or (uhg, I hate to say this) xml. The cost of serializing and deserializing text is quite small, and the advantages in terms of debugging ease and extensibility are significant.
Finally, if you want to use text but find json or xml too distasteful, too heavy, or too much of a learning curve, you can always just use printf and scanf formatting to read and write structures as text in a fixed order. Writing all numeric values, including floats, in hex rather than decimal will probably improve performance a bit and ensure round-trip accuracy of floating point values. If you don't have C99, another option for floats could be to decompose them to mantissa/exponent form and recompose them using frexp and ldexp.
If you want to send data over the network, you will want to look at using the network-to-host and host-to-network byte ordering functions.
http://beej.us/guide/bgnet/output/html/multipage/htonsman.html
htons
http://www.gnu.org/s/libc/manual/html_node/Byte-Order.html
RESPONSE: Thank you for the responses. I guess the CORRECT answer is that for structs, arrays and such memory blobs you MUST implement your own serialization function ... this is fine ... I will look into this. I wanted to get a good feel before I attempted such a thing ...
x
Also look into frameworks that have been implemented to tackle this exact problem, that allow you to marshall / demarshall arbitrarily complex data structures. If you are going to do this on a scale any larger than a few types, then use a framework.
rpcgen / XDR : Don't let all the talk about RPC / client / server scare you away. You can use rpcgen to generate XDR marshalling / demarshalling routines for your data, that you can transport whichever way you like.
Flick IDL Compiler Kit
CORBA : Most CORBA frameworks have IDL compilers e.g. ACE TAO.
ASN.1 : If you enjoy some pain. Very exotic though.
I've lots of different structs containing enum members that I have to transmit via TCP/IP. While the communication endpoints are on different operating systems (Windows XP and Linux) meaning different compilers (gcc 4.x.x and MSVC 2008) both program parts share the same header files with type declarations.
For performance reasons, the structures should be transmitted directly (see code sample below) without expensively serializing or streaming the members inside.
So the question is how to ensure that both compilers use the same internal memory representation for the enumeration members (i.e. both use 32-bit unsigned integers). Or if there is a better way to solve this problem...
//type and enum declaration
typedef enum
{
A = 1,
B = 2,
C = 3
} eParameter;
typedef enum
{
READY = 400,
RUNNING = 401,
BLOCKED = 402
FINISHED = 403
} eState;
#pragma pack(push,1)
typedef struct
{
eParameter mParameter;
eState mState;
int32_t miSomeValue;
uint8_t miAnotherValue;
...
} tStateMessage;
#pragma pack(pop)
//... send via socket
tStateMessage msg;
send(iSocketFD,(void*)(&msg),sizeof(tStateMessage));
//... receive message on the other side
tStateMessage msg_received;
recv(iSocketFD,(void*)(&msg_received),sizeof(tStateMessage));
Additionally...
Since both endpoints are little endian maschines, endianess is not a problem here.
And the pack #pragma solves alignment issues satisfactorily.
Thx for your answers,
Axel
I'll answer your question pragmatically because you've chosen a relatively risky path after weighing the performance gains against the possible downsides (at least I hope you have!).
If portability and robustness against future changes to those compilers have also been considered then an empirical approach would be the best guard against problems.
Ensure you are using initializers for the enums (your examples do this) in all cases.
Do empirical testing to see how the data is interpreted on the receiving side.
Record the version numbers of the build tools on both sides and archive them with the source code. Preferably archive the tools as well.
Document everything you did so unforeseen maintenance in the future is not handicapped.
Pray for the best! ;-)
I would advise you to use one of the serialization libraries specially designed for such problems, like:
Apache Avro (tutorial)
Facebook's Thrift (tutorial)
Google's Protocol Buffers (tutorial)
What you will get is maximum platform portability, an easy way of changing the interface and the type of messages transmitted plus a lot more useful features.
Note that only Avro has an officially supported C API. For Thrift and Protocol Buffers you either make a thin C wrapper over the C++ API or use one of the C APIs, like protobuf-c.
This is premature optimization. You have made two costly assumptions without measurements.
The first assumption is that this part of the code is a performance bottleneck in the first place. Is it? Very unlikely. If one is going to make assumptions about performance, then the safe assumption is that the network speed will be the bottleneck, not the code which sends and receives the network messages. This alone should prevent you from ever considering the second assumption.
The second assumption is that serializing the struct portably will be noticeably slower than writing the raw bits of the struct. This assumption is nearly always false.
Skeptical? Measure it! :)
If you don't want to go through serialization, one method I've seen used is to eschew enums and simply use 32-bit unsigned ints and #DEFINEs to emulate enums. You trade away some type safety for some assurances about data format.
Otherwise, you are relying on behaviour that isn't guarenteed in the language specification to be implemented the same way on all your compilers. If you aren't worried about general portability and just want to ensure the same effect on two compilers, it should be possible through trial and error and a lot of testing to get the two to do the same thing. I believe the C99 spec allows enums to internally be the size of int or smaller, but not larger than int. So one thing I've seen done to supposedly hint the compiler in the right direction is:
typedef enum
{
READY = 400,
RUNNING = 401,
BLOCKED = 402,
FINISHED = 403,
MAX = MAX_INT
} eState;
This should limit the compiler's choices for how to store the enum. Note that compilers can violate the standard, however, I know gcc has a non-standard feature where it will allow 64-bit enums if necessary.
Also, check out:
What is the size of an enum in C?
It is strongly recommended to serialize the data in some way or at least use an indicator about the hardware architecture. Even if you use the same compiler, you can have problems with internal data representations (little endian, big endian etc).
I have seen countless references about endianness and what it means. I got no problems about that...
However, my coding project is a simple game to run on linux and windows, on standard "gamer" hardware.
Do I need to worry about endianness in this case? When should I need to worry about it?
My code is simple C and SDL+GL, the only complex data are basic media files (png+wav+xm) and the game data is mostly strings, integer booleans (for flags and such) and static-sized arrays. So far no user has had issues, so I am wondering if adding checks is necessary (will be done later, but there are more urgent issues IMO).
The times when you need to worry about endianess:
you are sending binary data between machines or processes (using a network or file). If the machines may have different byte order or the protocol used specifies a particular byte order (which it should), you'll need to deal with endianess.
you have code that access memory though pointers of different types (say you access a unsigned int variable through a char*).
If you do these things you're dealing with byte order whether you know it or not - it might be that you're dealing with it by assuming it's one way or the other, which may work fine as long as your code doesn't have to deal with a different platform.
In a similar vein, you generally need to deal with alignment issues in those same cases and for similar reasons. Once again, you might be dealing with it by doing nothing and having everything work fine because you don't have to cross platform boundaries (which may come back to bite you down the road if that does become a requirement).
If you mean a PC by "standard gamer hardware", then you don't have to worry about endianness as it will always be little endian on x86/x64. But if you want to port the project to other architectures, then you should design it endianness-independently.
Whenever you recieve/transmit data from a network, remeber to convert to/from network and host byte order. The C functions htons, htonl etc, or equivalients in your language, should be used here.
Whenever you read multi-byte values (like UTF-16 characters or 32 bit ints) from a file, since that file might have originated on a system with different endianness. If the file is UTF 16 or 32 it probably has a BOM (byte-order mark). Otherwise, the file format will have to specify endianness in some way.
You only need to worry about it if your game needs to run on different hardware architectures. If you are positive that it will always run on Intel hardware then you can forget about it. If it will run on Linux though many people use different architectures than Intel and you may end up having to think about it.
Are you distributing you game in source code form?
Because if you are distributing you game as a binary only, then you know exactly which processor families your game will run on. Also, the media files, are they user generated (possibly via a level editor) or are they really only ment to be supplied by yourself?
If this is a truly closed environment (your distribute binaries and the game assets are not intended to be customized) then you know your own risks to endians and I personally wouldn't fool with it.
However, if you are either distributing source and/or hoping people will customize their game, then you have a potential for concern. However, with most of the desktop/laptop computers around these days moving to x86 I would think this is a diminishing concern.
The problem occurs with networking and how the data is sent and when you are doing bit fiddling on different processors since different processors may store the data differently in memory.
I believe Power PC has the opposite endianness of the Intel boards. Might be able to have a routine that sets the endianness dependant on the architecture? I'm not sure if you can actually tell what the hardware architecture is in code...maybe someone smarter then me does know the answer to that question.
Now in reference to your statement "standard" Gamer H/W, I would say typically you're going to look at Consumer Off the Shelf solutions are really what most any Standard Gamer is using, so you're almost going to for sure get the same endian across the board. I'm sure someone will disagree with me but that's my $.02
Ha...I just noticed on the right there is a link that is showing up related to the suggestion I had above.
Find Endianness through a c program