(Excuse me if I am not able to put the question correctly. English in not my primary language.)
I am trying to parse SyncE ESMC packet. It is Ethernet slow protocol packet.
Approach 1:
For parsing this packet, I used byte by byte approach similar to what has been done here.
Approach 2:
Other way to parse the packet would be to define a 'structure' to represent the whole packet and access individual fields to retrieve the value at particular offset.
However in this approach structure padding and alignment may come into picture(which I am not sure) but on Linux various packet headers are defined in form of structure, e.g. iphdr in ip.h. IP packets (buffer) can be type casted to 'iphdr' to retrieve ip header fields, so it must be working.
Which approach is better to parse a network packet in C?
Does structure padding and aligment make any difference while parsing a packet via approach 2? If yes, how did Linux headers overcome this problem?
Approach 1 is the best for portability. It allows you to safely avoid misaligned accesses, for example. In particular, if your machine is little-endian, this approach lets you take care of the byte-swapping very easily.
Approach 2 is sometimes convenient and often is faster code to write. If structure padding gets in your way, your compiler probably provides a flag or attribute (like __attribute__((__packed__)) or #pragma pack) to work around it. If you have a little-endian machine, however, you're still going to have to byteswap fields all over the place.
Related
I want to send some stuff over TCP, including int8_t, uint32_t, char, .... and I need to send it at once, so I was thinking about using one big byte array - especially since the receiver will treat the incoming packet as a byte array.
What's the proper way to do this in C? Since there's no "byte" datatype, I'm thinking about just using a int8_t array, filling it and sending it via FreeRTOS. What's stuff I should be aware of? Should I cast every value to int8_t (while splitting up larger ones with bit operators)? Anything I should take care of when trying to put unsigned data types in there?
Thanks for any help, haven't found much online surprisingly.
I'm working on a project where I must send data to server and this client will run on different os's. I knwo the problem with endians on different machines, so I'm converting 'everything' (almost) to network byte order, using htonl and vice-versa on the other end.
Also, I know that for a single byte, I don't need to convert anything. But, what should I do for char*? ex:
send(sock,(char*)text,sizeof(text));
What's the best approach to solve this? Should I create an 'intermediate function' do intercept this 'send', then really send char-by-char of this char array? If so, do I need to convert every char to network byte order? I think no, since every char is only one byte.
Thinking of this, if I create this 'intermediate functions', I don't have to convert nothing more to network byte order, since this function will send char by char, thus don't need conversion of endians.
I any advice on this.
I am presuming from your question that the application layer protocol (more specifically everything above level 4) is under your design control. For single byte-wide (octet-wide in networking parlance) data there is no issue with endian ordering and you need do nothing special to accommodate that. Now if the character data is prepended with a length specifier that is, say 2 octets, then the ordering of the bytes must be treated consistently.
Going with network byte ordering (big-endian) will certainly fill the bill, but so would consistently using little-endian. Consistency of byte ordering on each end of a connection is the crucial issue.
If the protocol is not under your design control, then the protocol specification should offer guidance on the issue of byte ordering for multi-byte integers and you should follow that.
Suppose I have a C struct defined as follows :
typedef struct servData {
char max_word[MAX_WORD];
char min_word[MAX_WORD];
int word_count ;
} servSendData ;
where 'MAX_WORD' could be any value.
Now if I have an instance of this structure :
servSendData myData ;
And if I populate this instance and then send it over the network, will there be any portability issues here considering that I want my server as well as the client to be running on either a 64-bit system or a 32-bit system.
I am going to send and receive data as follows :
//server side
strcpy(myData.max_word, "some large word") ;
strcpy(myData.min_word, "small") ;
myData.word_count=100 ;
send(sockFd, (char*)&myData, sizeof(myData);
//client side
recv(sockFd, (char*)&myData, sizeof(myData);
printf("large word is %s\n", myData.max_word) ;
printf("small word is %s\n", myData.min_word) ;
printf("total words is %d\n", myData.word_count) ;
Yes, there definitely will be portability issues.
Alignment of structure members can be different even among different compilers on the same platform, let alone different platforms. And that's all assuming that sizeof(int) is the same across all of them (though granted, it usually is --- but do you really want to rely on "usually" and hope for the best?).
This holds even if MAX_WORD is the same on both computers (I'll assume they are from here on out; if they're not, then you're in trouble here).
What you need to do is send (and receive) each field separately. There is also a problem with sizeof(int) and endianness, so I've added a call to htonl() to convert from system to network byte order (the inverse function is ntohl()). They both return uint32_t which has a fixed, known, size.
send(sockFd, myData.max_word, sizeof(myData.max_word)); // or just MAX_WORD
send(sockFd, myData.min_word, sizeof(myData.min_word));
uint32_t count = htonl(myData.word_count); // convert to network byte order
send(sockFd, &count, sizeof(count));
// error handling!
if((ret = recv(sockFd, myData.max_word, sizeof(myData.max_word))) != sizeof(myData.max_word))
{
// handle error or read more data
}
... // and so on
// remember to convert back from network byte order on recv!
// also keep in mind the third field is now `uint32_t`, and not `int` in the stream
As other relies have stated there are real problems in copying a C structure between different machines with different compilers/word size/and endian structure. One common way to resolve this issue is to transform your data into a machine independent format, transfer it across the network and then transform it back into a structure on the receiver. This is such a common requirement that multiple technologies already exist to do this - the two that spring to my mind initially are gsoap and rpcgen although there are probably many other options.
I've mostly used gsoap and after you get past the initial learning curve you can develop robust solutions that scale well (with multiple threads) and which handles both the networking and data translations for you.
If you don't want to go down this route then the safest approach is to write routines that convert your data to/from a standard string format (if you have issues with Unicode you'll need to take that into account as well) and then send that across the network.
You have to take care about the endians.
May you should use hton() or ntoh() functions, to convert between little and big endian.
You can use structure packing. With most C compilers, you can enforce a specific structure alignment. It is sometimes used for what you need it - to transfer a struct over a network.
Note that this still leaves endianness issues, so this is not a universal solution.
If you are not writing embedded software, sending data between applications without serializing properly is rarely a good idea.
The same goes for using raw sockets, which is not very convenient, and feels a bit like "reinventing the wheel".
Many libraries can help you with both! Of course, you don't have to use them, but reading their documentation, and understanding how they work will help you make better choices. Things you have not yet planned can come out of the box (like, what happens when you want to update your system, and the message format changes?)
For serialization, have a read on those general purpose formats:
Human readable: JSON, XML, YAML, others...
Binary: Protobuf, TPL, Avro, BSON, MessagePack, and many others
For socket abstraction, look up
Boost ASIO
ZeroMQ
nanomsg
Many others
In my program I have a few structs and a char array that I want to send as a single entity over UDP.
I am struggling to think of a good way to do this.
My first thought was to create a structure which contains everything I want to send but it would be of the wrong type for using sendto()
How would I store the two structs and a char array in another array so that it will be received in the way I intended?
Thanks
Since C allows you to cast to your heart's content, there's no such thing as a wrong type for sendto(). You simply cast the address of your struct to a void * and pass that as the argument to sendto().
However, a lot of people will impress on you that it's not advisable to send structs this way in the first place:
If the programs on either side of the connection are compiled by different compilers or in different environments, chances are your structs will not have the same packing.
If the two hosts involved in the transfer don't have the same endinanness, part of your data will end up backwards.
If the host architectures differ (e.g. 32 bit vs. 64 bits) then sizes of structs may be off as well. Certainly there will be size discrepancies if the sizes of your basic data types (int, char, long, double, etc.) differ.
So... Please take the advice of the first paragraph only if you're sure your two hosts are identical twins, or close enough to it.
In other cases, consider converting your data to some kind of neutral text representation, which could be XML but doesn't need to be anything that complicated. Strings are sent as a sequence of bytes, and there's much less that can go wrong. Since you control the format, you should be able to parse that stuff with little trouble on the receiving side.
Update
You mention that you're transferring mostly bit fields. That means that your data essentially consists of a bunch of integers, all of them less than (I'm assuming) 32 bits.
My suggestion for a "clean" solution, therefore, would be to write a function to unpack all those bit fields, and to ship the whole works as an array of (perhaps unsigned) integers. Assuming that sizeof(int) is the same across machines, htons() will work successfully on the elements (each individually!) of those arrays, and you can then wrap them back into a structure on the other side.
You can send multiple pieces of data as one with writev. Just create the array of struct iovec that it needs, with one element for each data structure you want to send.
Are there any libraries or guides for how to read and parse binary data in C?
I am looking at some functionality that will receive TCP packets on a network socket and then parse that binary data according to a specification, turning the information into a more useable form by the code.
Are there any libraries out there that do this, or even a primer on performing this type of thing?
I have to disagree with many of the responses here. I strongly suggest you avoid the temptation to cast a struct onto the incoming data. It seems compelling and might even work on your current target, but if the code is ever ported to another target/environment/compiler, you'll run into trouble. A few reasons:
Endianness: The architecture you're using right now might be big-endian, but your next target might be little-endian. Or vice-versa. You can overcome this with macros (ntoh and hton, for example), but it's extra work and you have make sure you call those macros every time you reference the field.
Alignment: The architecture you're using might be capable of loading a mutli-byte word at an odd-addressed offset, but many architectures cannot. If a 4-byte word straddles a 4-byte alignment boundary, the load may pull garbage. Even if the protocol itself doesn't have misaligned words, sometimes the byte stream itself is misaligned. (For example, although the IP header definition puts all 4-byte words on 4-byte boundaries, often the ethernet header pushes the IP header itself onto a 2-byte boundary.)
Padding: Your compiler might choose to pack your struct tightly with no padding, or it might insert padding to deal with the target's alignment constraints. I've seen this change between two versions of the same compiler. You could use #pragmas to force the issue, but #pragmas are, of course, compiler-specific.
Bit Ordering: The ordering of bits inside C bitfields is compiler-specific. Plus, the bits are hard to "get at" for your runtime code. Every time you reference a bitfield inside a struct, the compiler has to use a set of mask/shift operations. Of course, you're going to have to do that masking/shifting at some point, but best not to do it at every reference if speed is a concern. (If space is the overriding concern, then use bitfields, but tread carefully.)
All this is not to say "don't use structs." My favorite approach is to declare a friendly native-endian struct of all the relevant protocol data without any bitfields and without concern for the issues, then write a set of symmetric pack/parse routines that use the struct as a go-between.
typedef struct _MyProtocolData
{
Bool myBitA; // Using a "Bool" type wastes a lot of space, but it's fast.
Bool myBitB;
Word32 myWord; // You have a list of base types like Word32, right?
} MyProtocolData;
Void myProtocolParse(const Byte *pProtocol, MyProtocolData *pData)
{
// Somewhere, your code has to pick out the bits. Best to just do it one place.
pData->myBitA = *(pProtocol + MY_BITS_OFFSET) & MY_BIT_A_MASK >> MY_BIT_A_SHIFT;
pData->myBitB = *(pProtocol + MY_BITS_OFFSET) & MY_BIT_B_MASK >> MY_BIT_B_SHIFT;
// Endianness and Alignment issues go away when you fetch byte-at-a-time.
// Here, I'm assuming the protocol is big-endian.
// You could also write a library of "word fetchers" for different sizes and endiannesses.
pData->myWord = *(pProtocol + MY_WORD_OFFSET + 0) << 24;
pData->myWord += *(pProtocol + MY_WORD_OFFSET + 1) << 16;
pData->myWord += *(pProtocol + MY_WORD_OFFSET + 2) << 8;
pData->myWord += *(pProtocol + MY_WORD_OFFSET + 3);
// You could return something useful, like the end of the protocol or an error code.
}
Void myProtocolPack(const MyProtocolData *pData, Byte *pProtocol)
{
// Exercise for the reader! :)
}
Now, the rest of your code just manipulates data inside the friendly, fast struct objects and only calls the pack/parse when you have to interface with a byte stream. There's no need for ntoh or hton, and no bitfields to slow down your code.
The standard way to do this in C/C++ is really casting to structs as 'gwaredd' suggested
It is not as unsafe as one would think. You first cast to the struct that you expected, as in his/her example, then you test that struct for validity. You have to test for max/min values, termination sequences, etc.
What ever platform you are on you must read Unix Network Programming, Volume 1: The Sockets Networking API. Buy it, borrow it, steal it ( the victim will understand, it's like stealing food or something... ), but do read it.
After reading the Stevens, most of this will make a lot more sense.
Let me restate your question to see if I understood properly. You are
looking for software that will take a formal description of a packet
and then will produce a "decoder" to parse such packets?
If so, the reference in that field is PADS. A good article
introducing it is PADS: A Domain-Specific Language for Processing Ad
Hoc Data. PADS is very complete but unfortunately under a non-free licence.
There are possible alternatives (I did not mention non-C
solutions). Apparently, none can be regarded as completely production-ready:
binpac
PacketTypes
DataScript
If you read French, I summarized these issues in Génération de
décodeurs de formats binaires.
In my experience, the best way is to first write a set of primitives, to read/write a single value of some type from a binary buffer. This gives you high visibility, and a very simple way to handle any endianness-issues: just make the functions do it right.
Then, you can for instance define structs for each of your protocol messages, and write pack/unpack (some people call them serialize/deserialize) functions for each.
As a base case, a primitive to extract a single 8-bit integer could look like this (assuming an 8-bit char on the host machine, you could add a layer of custom types to ensure that too, if needed):
const void * read_uint8(const void *buffer, unsigned char *value)
{
const unsigned char *vptr = buffer;
*value = *buffer++;
return buffer;
}
Here, I chose to return the value by reference, and return an updated pointer. This is a matter of taste, you could of course return the value and update the pointer by reference. It is a crucial part of the design that the read-function updates the pointer, to make these chainable.
Now, we can write a similar function to read a 16-bit unsigned quantity:
const void * read_uint16(const void *buffer, unsigned short *value)
{
unsigned char lo, hi;
buffer = read_uint8(buffer, &hi);
buffer = read_uint8(buffer, &lo);
*value = (hi << 8) | lo;
return buffer;
}
Here I assumed incoming data is big-endian, this is common in networking protocols (mainly for historical reasons). You could of course get clever and do some pointer arithmetic and remove the need for a temporary, but I find this way makes it clearer and easier to understand. Having maximal transparency in this kind of primitive can be a good thing when debugging.
The next step would be to start defining your protocol-specific messages, and write read/write primitives to match. At that level, think about code generation; if your protocol is described in some general, machine-readable format, you can generate the read/write functions from that, which saves a lot of grief. This is harder if the protocol format is clever enough, but often doable and highly recommended.
You might be interested in Google Protocol Buffers, which is basically a serialization framework. It's primarily for C++/Java/Python (those are the languages supported by Google) but there are ongoing efforts to port it to other languages, including C. (I haven't used the C port at all, but I'm responsible for one of the C# ports.)
You don't really need to parse binary data in C, just cast some pointer to whatever you think it should be.
struct SomeDataFormat
{
....
}
SomeDataFormat* pParsedData = (SomeDataFormat*) pBuffer;
Just be wary of endian issues, type sizes, reading off the end of buffers, etc etc
Parsing/formatting binary structures is one of the very few things that is easier to do in C than in higher-level/managed languages. You simply define a struct that corresponds to the format you want to handle and the struct is the parser/formatter. This works because a struct in C represents a precise memory layout (which is, of course, already binary). See also kervin's and gwaredd's replies.
I'm not really understand what kind of library you are looking for ? Generic library that will take any binary input and will parse it to unknown format?
I'm not sure there is such library can ever exist in any language.
I think you need elaborate your question a little bit.
Edit:
Ok, so after reading Jon's answer seems there is a library, well kind of library it's more like code generation tool. But as many stated just casting the data to the appropriate data structure, with appropriate carefulness i.e using packed structures and taking care of endian issues you are good. Using such tool with C it's just an overkill.
Basically suggestions about casting to struct work but please be aware that numbers can be represented differently on different architectures.
To deal with endian issues network byte order was introduced - common practice is to convert numbers from host byte order to network byte order before sending the data and to convert back to host order on receipt. See functions htonl, htons, ntohl and ntohs.
And really consider kervin's advice - read UNP. You won't regret it!