I'm making my own client-server application in C that implements the TFTP protocol. After reading the TFTP's RFC and making working a simple socket client-server app, now I'm a little confused on how to create the specific packets that have to be created for the TFTP protocol.
For example, the WRQ packet has to be this way:
2 bytes string 1 byte string 1 byte
------------------------------------------------
| Opcode | Filename | 0 | Mode | 0 |
------------------------------------------------
which is extracted from the official RFC.
I have a .h in which I define all the structures for the packets, but I'm not sure if I'm doing correctly and I'm not being lucky finding information on the web.
The struct I created for this packet is:
struct WRQ {
signed short int opcode; //2 bytes
char * filename; // N bytes
char zero_0; // 1 byte
char * mode; // N Bytes
char zero_1; // 1 byte
};
I have two doubts:
a) when I make a sizeof(struct WRQ) it returns 20 bytes. Which is not the size I want to get. Why does this happens?
b) How do I have to define the strings? because I want the server to recieve the string itself, and, I think, this way, It will recieve the pointer to the string in the client machine.
I hope that all is clear and you could help me because I'm stuck at the moment!
The following code (with no error checking) is one possible way of building up the packet and sending it. Note that it assumes that both filename and mode are not NULL. I don't know if that is a valid assumption or not. Even if it is, it would be wise to have a check for NULL before using them in real code:
struct WRQ *p;
int packetLen;
char *buf;
char *pos;
int ret;
// compute packet length. Start with fixed size data
packetLen = sizeof( p->opcode ) + sizeof( p->zero_0 ) + sizeof( p->zero_1 );
// This assumes (possibly incorrectly) that filename and mode are not null
packetLen += strlen( p->filename ) + 1;
packetLen += strlen( p->mode ) + 1;
// allocate the buffer
buf = malloc( packetLen );
pos = buf;
// and start filling it in
// I am assuming (but didn't study the RFC that it should be network byte order)
*(signed short int*)pos = htons( p->opcode );
pos += sizeof( p->opcode );
strcpy( pos, p->filename );
pos += strlen( p->filename ) + 1;
*pos = p->zero_0;
strcpy( pos, p->mode );
pos += strlen( p->mode ) + 1;
*pos = p->zero_1;
ret = send( s, buf, packetLen, flags );
free( buf );
In your code, char * filename; is a pointer to a char. This means that filename only occupies 4 bytes. Even if your string is 1000 bytes long, since filename is simply a pointer, it just contains the 4-byte memory address of the first character of your string.
So you have two solutions: use char filename[MAX_LENGTH] to declare a string of size MAX_LENGTH and always pass that. Or, you can include another field, say, "filename_length", which tells you how many bytes to expect when you read the filename field.
(The above is actually false. filename may not be 4 bytes long. filename will be sizeof(char*) bytes long. This is probably 4 bytes on your computer. But pointers are not always 4 bytes, and nowadays people are getting into a lot of trouble for assuming a 4-byte pointer on 64-bit architectures. So I'm just sayin'. Don't downvote me.)
To answer your second question - why is sizeof() 20 bytes? The compiler will pad pieces of the struct with extra bytes so that each member fits inside a 4-byte boundary. The compiler can work efficiently with "words" rather than weird-sized structures. (Again, the "4" just depends on the architecture. Each architecture has its own "word" length.) This known as alignment. There is an excellent SO thread which gives more detail: Why isn't sizeof for a struct equal to the sum of sizeof of each member?
You can't just put a char* in there and expect it to work, because that's a pointer and you need the actual character data to appear in the packet (a pointer passed between two programs will almost never work!). Since the filename portion of the packet is variable-length, you can't represent the whole packet as a struct as you are trying to do. Instead, you probably should dynamically generate the packet by concatenating the pieces on demand, such as with a function having this signature:
vector<char> makePacket(uint16_t opcode, const char* filename, const char* mode);
C structures can't have a variable size. You'll have to define "filename" and "mode" as char field_name [preset size]; or construct the structure by hand in a memory buffer at run time.
Finally, if what you need is a TFTP implementation in C, you can bet someone has written if already. E.g. there's an BSD'ed tftp-hpa used in a variety of Linux distributions.
Related
I am working with sending data to and from a client/server application. I have created my own header scheme. This is the flow I am in question about:
char *data = returnmydata();
int one = 1;
int two = 2;
int three = 3;
So what happens is I have a char buffer that contains all my data. But I need to append to the front of this data my one, two, and three integers which is used to let the other side know how to handle the data.
Instead of allocating enough at the start for the 3 header items, is it possible to reallocate an addition 12 bytes and move the start of the data to data+12?
I currently just memcpy the 3 integers at the start then offset the data +12. However I think my code would be better if I could just pass a struct that contains the 3 ints and a pointer to the data. Then I can use this function to create the full data buffer with the headers in it.
Another option is to alloc another buffer memcpy the 3 integers to the front, and then memcpy the data to the new buffer. However I would like to keep memory usage low.
Use functions realloc and memmove.
If the size of the initially allocated buffer is equal to some value N then you can write
char *tmp = realloc( data, N + 12 );
if ( tmp != NULL )
{
data = tmp;
memmove( data + 12, data, N );
// then you can insert some values in the first 12 bytes of the allocated buffer
}
I'm writing an application which reads data from a UART interface. The data is sent in packets. Each packet has a channel associated with it. My application multiplexes received packets into virtual channels (threads) so that every channel can work independently of one another. When I receive a packet I have to do something depending on it's contents and produce a response. The response is sent back using the same UART interface.
The data sent is mostly binary. When I'm reading from the UART interface, I know the size of the packet beforehand, so I can preallocate memory with no problem.
The problem for me is producing a response. I know the maximum size of a packet, so I can create a static buffer when I'm constructing a response. If I we're to work with ASCII characters, instead of binary data, I could rely on NULL terminator to determine how long the data stored in the buffer is. However, I'm working with binary data, so using a NULL byte does not work. Instead, I have to keep a variable storing how many bytes of the buffer is used up already. I was thinking of using a custom data type for storing binary data:
typedef struct {
unsigned char buff[2048];
size_t buff_used;
} binary_data_t;
What would be a standart way of handling this?
Since you know the number of bytes you need to hold a packet, just use a flexible array member:
typedef struct
{
size_t bytes;
unsigned char data[];
} binary_data_t;
(Note that identifiers ending in _t are reserved by POSIX, and you really shouldn't be using them.)
Allocation and reading data (assumes you read() from a file descriptor):
binary_data_t *p = malloc( sizeof( *p ) + numDataBytes );
p->bytes = numDataBytes;
ssize_t bytes_read = read( uartFD, p->data, numDataBytes );
one way of doing it could be to store a pointer to where in your array next byte should be placed.
typedef struct {
unsigned char buff[2048];
char* pData;
} binary_data_t;
// at init
binary_data_t rspMsg;
rspMsg.pData = &rspMsg.buff[0];
// at entering data
*(rspMsg.pData) = data;
rspMsg.pData++;
// at sending data you know the length via
length = rspMsg.pData - &rspMsg.buff[0];
This is one way of solving this.
Can be done in many ways.
I have a struct object that comprises of several primitive data types, pointers and struct pointers. I want to send it over a socket so that it can be used at the other end. As I want to pay the serialization cost upfront, how do I initialize an object of that struct so that it can be sent immediately without marshalling? For example
struct A {
int i;
struct B *p;
};
struct B {
long l;
char *s[0];
};
struct A *obj;
// can do I initialize obj?
int len = sizeof(struct A) + sizeof(struct B) + sizeof(?);
obj = (struct A *) malloc(len);
...
write(socket, obj, len);
// on the receiver end, I want to do this
char buf[len];
read(socket, buf, len);
struct A *obj = (struct A *)buf;
int i = obj->i;
char *s = obj->p->s[0];
int i obj.i=1; obj.p.
Thank you.
The simplest way to do this may be to allocate a chunk of memory to hold everything. For instance, consider a struct as follows:
typedef struct A {
int v;
char* str;
} our_struct_t;
Now, the simplest way to do this is to create a defined format and pack it into an array of bytes. I will try to show an example:
int sLen = 0;
int tLen = 0;
char* serialized = 0;
char* metadata = 0;
char* xval = 0;
char* xstr = 0;
our_struct_t x;
x.v = 10;
x.str = "Our String";
sLen = strlen(x.str); // Assuming null-terminated (which ours is)
tLen = sizeof(int) + sLen; // Our struct has an int and a string - we want the whole string not a mem addr
serialized = malloc(sizeof(char) * (tLen + sizeof(int)); // We have an additional sizeof(int) for metadata - this will hold our string length
metadata = serialized;
xval = serialized + sizeof(int);
xstr = xval + sizeof(int);
*((int*)metadata) = sLen; // Pack our metadata
*((int*)xval) = x.v; // Our "v" value (1 int)
strncpy(xstr, x.str, sLen); // A full copy of our string
So this example copies the data into an array of size 2 * sizeof(int) + sLen which allows us a single integer of metadata (i.e. string length) and the extracted values from the struct. To deserialize, you could imagine something as follows:
char* serialized = // Assume we have this
char* metadata = serialized;
char* yval = metadata + sizeof(int);
char* ystr = yval + sizeof(int);
our_struct_t y;
int sLen = *((int*)metadata);
y.v = *((int*)yval);
y.str = malloc((sLen + 1) * sizeof(char)); // +1 to null-terminate
strncpy(y.str, ystr, sLen);
y.str[sLen] = '\0';
As you can see, our array of bytes is well-defined. Below I have detailed the structure:
Bytes 0-3 : Meta-data (string length)
Bytes 4-7 : X.v (value)
Bytes 8 - sLen : X.str (value)
This kind of well-defined structure allows you to recreate the struct on any environment if you follow the defined convention. To send this structure over the socket, now, depends on how you develop your protocol. You can first send an integer packet containing the total length of the packet which you just constructed, or you can expect that the metadata is sent first/separately (logically separately, this technically can still all be sent at the same time) and then you know how much data to receive on the client-side. For instance, if I receive metadata value of 10 then I can expect sizeof(int) + 10 bytes to follow to complete the struct. In general, this is probably 14 bytes.
EDIT
I will list some clarifications as requested in the comments.
I do a full copy of the string so it is in (logically) contiguous memory. That is, all the data in my serialized packet is actually full data - there are no pointers. This way, we can send a single buffer (we call is serialized) over the socket. If simply send the pointer, the user receiving the pointer would expect that pointer to be a valid memory address. However, it is unlikely that your memory addresses will be exactly the same. Even if they are, however, he will not have the same data at that address as you do (except in very limited and specialized circumstances).
Hopefully this point is made more clear by looking at the deserialization process (this is on the receiver's side). Notice how I allocate a struct to hold the information sent by the sender. If the sender did not send me the full string but instead only the memory address, I could not actually reconstruct the data which was sent (even on the same machine we have two distinct virtual memory spaces which are not the same). So in essence, a pointer is only a good mapping for the originator.
Finally, as far as "structs within structs" go, you will need to have several functions for each struct. That said, it is possible that you can reuse the functions. For instance, if I have two structs A and B where A contains B, I can have two serialize methods:
char* serializeB()
{
// ... Do serialization
}
char* serializeA()
{
char* B = serializeB();
// ... Either add on to serialized version of B or do some other modifications to combine the structures
}
So you should be able to get away with a single serialization method for each struct.
This answer is besides the problems with your malloc.
Unfortunately, you cannot find a nice trick that would still be compatible with the standard. The only way of properly serializing a structure is to separately dissect each element into bytes, write them to an unsigned char array, send them over the network and put the pieces back together on the other end. In short, you would need a lot of shifting and bitwise operations.
In certain cases you would need to define a kind of protocol. In your case for example, you need to be sure you always put the object p is pointing to right after struct A, so once recovered, you can set the pointer properly. Did everyone say enough already that you can't send pointers through network?
Another protocolish thing you may want to do is to write the size allocated for the flexible array member s in struct B. Whatever layout for your serialized data you choose, obviously both sides should respect.
It is important to note that you cannot rely on anything machine specific such as order of bytes, structure paddings or size of basic types. This means that you should serialize each field of the element separately and assign them fixed number of bytes.
You should serialize the data in a platform independent way.
Here is an example using the Binn library (my creation):
binn *obj;
// create a new object
obj = binn_object();
// add values to it
binn_object_set_int32(obj, "id", 123);
binn_object_set_str(obj, "name", "Samsung Galaxy Charger");
binn_object_set_double(obj, "price", 12.50);
binn_object_set_blob(obj, "picture", picptr, piclen);
// send over the network
send(sock, binn_ptr(obj), binn_size(obj));
// release the buffer
binn_free(obj);
If you don't want to use strings as keys you can use a binn_map which uses integers as keys. There is also support for lists. And you can insert a structure inside another (nested structures). eg:
binn *list;
// create a new list
list = binn_list();
// add values to it
binn_list_add_int32(list, 123);
binn_list_add_double(list, 2.50);
// add the list to the object
binn_object_set_list(obj, "items", list);
// or add the object to the list
binn_list_add_object(list, obj);
Interpret your data and understand what you want to serialize. You want to serialize an integer and a structure of type B (recursivelly, you want to serialize an int, a long, and an array of strings). Then serialize them. The length you need it sizeof(int) + sizeof(long) + ∑strlen(s[i])+1.
On the other hand, serialization is a solved problem (multiple times actually). Are you sure you need to hand write a serialization routine ? Why don't you use D-Bus or a simple RPC call ? Please consider using them.
I tried the method provided by #RageD but it didn't work.
The int value I got from deserialization was not the original one.
For me, memcpy() works for non-string variables. (You can still use strcpy() for char *)
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
typedef struct A {
int a;
char *str;
} test_struct_t;
char *serialize(test_struct_t t) {
int str_len = strlen(t.str);
int size = 2 * sizeof(int) + str_len;
char *buf = malloc(sizeof(char) * (size+1));
memcpy(buf, &t.a, sizeof(int));
memcpy(buf + sizeof(int), &str_len, sizeof(int));
memcpy(buf + sizeof(int) * 2, t.str, str_len);
buf[size] = '\0';
return buf;
}
test_struct_t deserialize(char *buf) {
test_struct_t t;
memcpy(&t.a, buf, sizeof(int));
int str_len;
memcpy(&str_len, buf+sizeof(int), sizeof(int));
t.str = malloc(sizeof(char) * (str_len+1));
memcpy(t.str, buf+2*sizeof(int), str_len);
t.str[str_len] = '\0';
return t;
}
int main() {
char str[15] = "Hello, world!";
test_struct_t t;
t.a = 123;
t.str = malloc(strlen(str) + 1);
strcpy(t.str, str);
printf("original values: %d %s\n", t.a, t.str);
char *buf = serialize(t);
test_struct_t new_t = deserialize(buf);
printf("new values: %d %s\n", new_t.a, new_t.str);
return 0;
}
And the output of the code above is:
original values: 123 Hello, world!
new values: 123 Hello, world!
#Shahbaz is right I would think you actually want this
int len = sizeof(struct A);
obj = (struct A *) malloc(len);
But also you will run into problems when sending a pointer to another machine as the address the pointer points to means nothing on the other machine.
I have the following structs in C:
typedef struct sUser {
char name[nameSize];
char nickname[nicknameSize];
char mail[mailSize];
char address[addressSize];
char password[passwordSize];
int totalPoints;
PlacesHistory history;
DynamicArray requests;
}User;
typedef struct sPlacesHistory {
HistoryElement array[HistorySize];
int occupied;
int last;
}PlacesHistory;
and the functions:
void serializeUser( User * user, FILE * fp ) {
fwrite( user, nameSize + nicknameSize + mailSize + addressSize + passwordSize + sizeof( int ) + sizeof( PlacesHistory ), 1, fp );
serializeDynamicArray( user -> requests, fp );
}
User * loadUser( FILE * fp ) {
User * user = malloc( sizeof( User ) );
fread( user, nameSize + nicknameSize + mailSize + addressSize + passwordSize + sizeof( int ) + sizeof( PlacesHistory ), 1, fp );
user -> requests = loadDynamicArray( fp );
return user;
}
When I load the struct User, and I print that user (loaded from file), the field "last" of placesHistory has the value of 255 or -1, depending on the order of the fields of the PlacesHistory structure. But The User I saved had -1 on that member.
So when i get 255, it is obviously wrong..
I suspect this has to do about struct padding.
How can I do this in such a way that the order of fields in the structure doesn't matter?
Or which criteria do I need to follow to make things work right?
Do I need to fwrite/fread one member at a time? ( I would like to avoid this for efficiency matters )
Do I need to serialize to an array first instead of a file? (I hope not .. because this implicates to know the size of all my structures beforehand because of the mallocated array- which means extra work creating a function for every non simple structure to know it's size)
Note: *Size are defined constants
Note2: DynamicArray is a pointer to another structure.
Yes, it probably has to do with padding in front of either totalPoints or history.
You can just write out sizeof(User) - sizeof(DynamicArray) and read back in the same. Of course this will only be compatible as long as your struct definitions and compiler don't change. If you don't need serialized data from one version of your program to be compatible with another version, then the above should work.
Why are you adding up all elements individually? That's just adding a lot of room for error. Whenever you change your structure, your code might break if you forgot to change all the places where you add the size up (in fact, why do you add it up each time?).
And, as you suspected, your code doesn't account for structure padding either, so you may be missing up to three bytes at the end of your data block (if your largest element is 4 bytes).
Why not sizeof(User) to get the size of the amount of data you're reading/writing? If you don't want parts of it saved (like requests), then use a struct inside a struct. (EDIT: Or, like rlibby suggested, just subtract the sizeof of the part you don't want to read.)
My guess is that your strings sizes are not divisible by 4, so you are 3 bytes short, and as such, it's possible that you were supposed to read "0xffffffff" (=-1) but ended up just reading "0xff000000" (=255 when using little endian, and assuming that your structure was zeroed out initially).
padding may be your problem because
nameSize + nicknameSize + mailSize + addressSize + passwordSize + sizeof( int ) + sizeof( PlacesHistory ) != sizeof( User )
so la last member (and last in struct) remain unitialized. To check this do a memset(,0,sizeof(User)) before reading from file.
To fix this use #pragma pack(push,1) before and #pragma pack(pop) after
I have a packet from a server which is parsed in an embedded system. I need to parse it in a very efficient way, avoiding memory issues, like overlapping, corrupting my memory and others variables.
The packet has this structure "String A:String B:String C".
As example, here the packet received is compounded of three parts separated using a separator ":", all these parts must be accesibles from an structure.
Which is the most efficient and safe way to do this.
A.- Creating an structure with attributes (partA, PartB PartC) sized with a criteria based on avoid exceed this sized from the source of the packet, and attaching also an index with the length of each part in a way to avoid extracting garbage, this part length indicator could be less or equal to 300 (ie: part B).
typedef struct parsedPacket_struct {
char partA[2];int len_partA;
char partB[300];int len_partB;
char partC[2];int len_partC;
}parsedPacket;
The problem here is that I am wasting memory, because each structure should copy the packet content to each the structure, is there a way to only save the base address of each part and still using the len_partX.
How about replacing the (:) with a 0, and add a null to the end - then you have three char * to pass around. You will need to deal with 0 length strings, but that might solve it
To avoid corrupting memory and other variables, you generally declare large data buffers as statics and place them at file scope, then allocate a separate RAM segment for them. Having them sitting on the stack is a bad idea in any embedded system.
You need to consider whether there is an alignment requirement for the CPU and whether the code should be portable or not. The compiler is free to add any number of padding bytes anywhere in that struct, meaning you may not be able to do this:
parsedPacket pp;
memcpy(&pp, raw_data, sizeof(parsedPacket )) ;
For this reason, structs are generally a bad choise for storing data packages. The safest solution is this:
/* packet.h */
typedef struct parsedPacket_struct {
uint8_t* partA;
uint8_t* partB;
uint8_t* partC;
uint16_t len_partA;
uint16_t len_partB;
uint16_t len_partC;
}parsedPacket;
#define MAX_PART_A 2
#define MAX_PART_B 300
#define MAX_PART_C 2
void packet_receive (parsedPacket* packet);
/* packet.c */
static uint8 partA[MAX_PART_A];
static uint8 partB[MAX_PART_B];
static uint8 partC[MAX_PART_C];
void packet_receive (parsedPacket* packet)
{
/* receive data from server */
...
packet->len_partA = ...;
packet->len_partB = ...;
packet->len_partC = ...;
packet->partA = partA;
packet->partB = partB;
packet->partC = partC;
memcpy(partA, A_from_server, packet->len_partA);
memcpy(partB, B_from_server, packet->len_partB);
memcpy(partC, C_from_server, packet->len_partC);
}
This can be extended to contain several static buffers if needed, ie a static array of arrays for each buffer. As you are dealing with large amounts of data in an embedded system, you can never allow the program to stack the buffers at a whim. The maximum amount of copies of a received packet must be determined during program design.
I'm not sure why you think your approach is wasting memory, but here's what I would do if I were feeling especially hacky:
typedef struct {
char *a, *b, *c;
char data[1]; // or 0 if your compiler lets you, or nothing in C99
} parsedPacket;
This is called a flexible array member. Basically, when you allocate memory for your struct, you do this:
parsedPacket *p = malloc(offsetof(parsedPacket, data[N]));
N above becomes the amount of data your array needs, i.e. how long the string you read is. This allocates the struct so that the data member has enough size for your entire string of data. Then, copy the string you recieve into this member, replace ':' characters with '\0', and set a to the first string (i.e. p->a = p->data), b to the second (p->b = p->data + strlen(p->a) + 1) and c to the third. Of course, you can make this process easier by doing it all at once:
size_t current = 0;
p->a = p->data;
p->b = p->c = NULL;
while(1)
{
int i = getc();
if(i == '\n' || i == EOF) break; // or whatever end conditions you expect
if(i == ':')
{
p->data[current] = '\0';
++current;
if(p->b == NULL) p->b = &p->data[current];
else if(p->c == NULL) p->c = &p->data[current];
else /* error */;
}
else
{
p->data[current] = i;
}
}
The type of each len_partN should be a type that can count up to the length of partN. E.g.:
typedef struct parsedPacket_struct {
char partA[300];unsigned short len_partA; // unsigned shorts have < 32k distinct values
char partB[300];unsigned short len_partB;
char partC[300];unsigned short len_partC;
}parsedPacket;
This seems like a design decision. If you want the struct to be easy to create, use the above approach, but beware its drawbacks (like "what if B has more than 300 chars?").