I have a packet from a server which is parsed in an embedded system. I need to parse it in a very efficient way, avoiding memory issues, like overlapping, corrupting my memory and others variables.
The packet has this structure "String A:String B:String C".
As example, here the packet received is compounded of three parts separated using a separator ":", all these parts must be accesibles from an structure.
Which is the most efficient and safe way to do this.
A.- Creating an structure with attributes (partA, PartB PartC) sized with a criteria based on avoid exceed this sized from the source of the packet, and attaching also an index with the length of each part in a way to avoid extracting garbage, this part length indicator could be less or equal to 300 (ie: part B).
typedef struct parsedPacket_struct {
char partA[2];int len_partA;
char partB[300];int len_partB;
char partC[2];int len_partC;
}parsedPacket;
The problem here is that I am wasting memory, because each structure should copy the packet content to each the structure, is there a way to only save the base address of each part and still using the len_partX.
How about replacing the (:) with a 0, and add a null to the end - then you have three char * to pass around. You will need to deal with 0 length strings, but that might solve it
To avoid corrupting memory and other variables, you generally declare large data buffers as statics and place them at file scope, then allocate a separate RAM segment for them. Having them sitting on the stack is a bad idea in any embedded system.
You need to consider whether there is an alignment requirement for the CPU and whether the code should be portable or not. The compiler is free to add any number of padding bytes anywhere in that struct, meaning you may not be able to do this:
parsedPacket pp;
memcpy(&pp, raw_data, sizeof(parsedPacket )) ;
For this reason, structs are generally a bad choise for storing data packages. The safest solution is this:
/* packet.h */
typedef struct parsedPacket_struct {
uint8_t* partA;
uint8_t* partB;
uint8_t* partC;
uint16_t len_partA;
uint16_t len_partB;
uint16_t len_partC;
}parsedPacket;
#define MAX_PART_A 2
#define MAX_PART_B 300
#define MAX_PART_C 2
void packet_receive (parsedPacket* packet);
/* packet.c */
static uint8 partA[MAX_PART_A];
static uint8 partB[MAX_PART_B];
static uint8 partC[MAX_PART_C];
void packet_receive (parsedPacket* packet)
{
/* receive data from server */
...
packet->len_partA = ...;
packet->len_partB = ...;
packet->len_partC = ...;
packet->partA = partA;
packet->partB = partB;
packet->partC = partC;
memcpy(partA, A_from_server, packet->len_partA);
memcpy(partB, B_from_server, packet->len_partB);
memcpy(partC, C_from_server, packet->len_partC);
}
This can be extended to contain several static buffers if needed, ie a static array of arrays for each buffer. As you are dealing with large amounts of data in an embedded system, you can never allow the program to stack the buffers at a whim. The maximum amount of copies of a received packet must be determined during program design.
I'm not sure why you think your approach is wasting memory, but here's what I would do if I were feeling especially hacky:
typedef struct {
char *a, *b, *c;
char data[1]; // or 0 if your compiler lets you, or nothing in C99
} parsedPacket;
This is called a flexible array member. Basically, when you allocate memory for your struct, you do this:
parsedPacket *p = malloc(offsetof(parsedPacket, data[N]));
N above becomes the amount of data your array needs, i.e. how long the string you read is. This allocates the struct so that the data member has enough size for your entire string of data. Then, copy the string you recieve into this member, replace ':' characters with '\0', and set a to the first string (i.e. p->a = p->data), b to the second (p->b = p->data + strlen(p->a) + 1) and c to the third. Of course, you can make this process easier by doing it all at once:
size_t current = 0;
p->a = p->data;
p->b = p->c = NULL;
while(1)
{
int i = getc();
if(i == '\n' || i == EOF) break; // or whatever end conditions you expect
if(i == ':')
{
p->data[current] = '\0';
++current;
if(p->b == NULL) p->b = &p->data[current];
else if(p->c == NULL) p->c = &p->data[current];
else /* error */;
}
else
{
p->data[current] = i;
}
}
The type of each len_partN should be a type that can count up to the length of partN. E.g.:
typedef struct parsedPacket_struct {
char partA[300];unsigned short len_partA; // unsigned shorts have < 32k distinct values
char partB[300];unsigned short len_partB;
char partC[300];unsigned short len_partC;
}parsedPacket;
This seems like a design decision. If you want the struct to be easy to create, use the above approach, but beware its drawbacks (like "what if B has more than 300 chars?").
Related
In build_uart_frame() , I call calcFCS() which calculates an XOR of all the bytes in the struct members(len, cmd0, cmd1 and data).
I do not think the struct is padded therefore will calling calcFCS() be an issue? Could somebody explain what is the issue in relation to struct padding as I don't understand its role here and secondly how can I do this operation correctly?
Thank you
typedef struct uart_frame {
uint8_t sof; /* 1 byte */
uint8_t len; /* 1 bytes */
uint8_t cmd0; /* 1 byte */
uint8_t cmd1;
char data[11]; /* 0 -250 byte */
unsigned char fcs; /* 1 byte */
} uart_frame_t;
//-------------------------------------------------------------------------
// Global uart frame
uart_frame_t rdata;
//-------------------------------------------------------------------------
unsigned char calcFCS(unsigned char *pMsg, unsigned char len) {
unsigned char result = 0;
while(len--) {
result ^= *pMsg++;
}
return(result);
}
//-------------------------------------------------------------------------
// Worker code to populate the frame
int build_uart_frame() {
uart_frame_t *rd = &rdata; //pointer variable 'rd' of type uart_frame
// common header codes
rd->sof = 0xFE;
rd->len = 11;
rd->cmd0 = 0x22;
rd->cmd0 = 0x05;
snprintf(rd->data, sizeof(rd->data), "%s", "Hello World");
rd->fcs = calcFCS((unsigned char *)rd, sizeof(uart_frame_t) - 1); //issue with struct padding
return 0;
}
Given your very specific example, it is unlikely that padding will be an issue, since all data types are bytes. Padding is mostly an issue when you use larger data types, because those should typically not be allocated at misaligned addresses.
Yet that is no guarantee: the compiler could in theory decide to replace a char with an int if it thinks that will get faster code. It is free to insert any amount of padding anywhere in a struct, except at the very top.
This is why structs are unsuitable to describe memory maps or data protocols. You will have to ensure that no padding is present and preferably do so portably. The best way to ensure this is a standard C compile-time assert:
_Static_assert(sizeof(uart_frame_t) == offsetof(uart_frame_t, fcs)+sizeof(unsigned char),
"Padding detected");
Here the size of the whole struct is checked against the byte position of the last struct member + the size of that member. If they are the same, there was no padding.
Now of course this only prevents your code from compiling and misbehaving, it doesn't solve the actual problem. Unfortunately there is no portable way to block padding. #pragma pack(1) is common but non-standard. __attribute__((packed)) is another compiler-specific command for this.
Ensuring that no packing is present on the given system where the code is compiled is usually enough.
Also, some of the more exotic systems (MIPS, SPARC etc) don't even support misaligned reads, meaning that misaligned access will not just mean slower code, but a run-time bus error crash.
The only way to safely ensure maximum portability of code using structs, is to write serialize/de-serialize routines that manually copies every member to/from a raw byte array:
void uart_serialize (const uart_frame_t* frame, uint8_t* raw)
{
raw[0] = frame->sof;
raw[1] = frame->len;
...
memcpy(&raw[4], frame->data, 11);
...
}
The downside of such methods is that they obviously adds some execution time, so I would only use them for code that I know needs to be ported to all kinds of different systems.
If I want to reduce malloc()s (espacially if the data is small and allocated often) I would like to allocate the pointer and pointee at once.
If you assume something like the following:
struct entry {
size_t buf_len;
char *buf;
int something;
};
I would like to allocate memory in the following way (don't care about error checking here):
size_t buf_len = 4; // size of the buffer
struct entry *e = NULL;
e = malloc( sizeof(*e) + buf_len ); // allocate struct and buffer
e->buf_len = buf_len; // set buffer size
e->buf = e + 1; // the buffer lies behind the struct
This could even be extende, so that a whole array is allocated at once.
How would you assess such a technuique with regard to:
Portability
Maintainability / Extendability
Performance
Readability
Is this reasonable? If it is ok to use, are there any ideas on how to design a possible interface for that?
You could use a flexible array member instead of a pointer:
struct entry {
size_t buf_len;
int something;
char buf[];
};
// ...
struct entry *e = malloc(sizeof *e + buf_len);
e->buf_len = buf_len;
Portability and performance are fine. Readability: not perfect but good enough.
Extendability: you can't use this for more than one member at a time, you'd have to fall back to your explicit pointer version. Also, the explicit pointer version means that you have to muck around to ensure correct alignment if you use it with a type that doesn't have an alignment of 1.
If you are seriously thinking about this I'd consider revisiting your entire data structure's design to see if there is another way of doing it. (Maybe this way is actually the best way, but have a good think about it first).
As to portability, I am unaware of any issues, as long as the sizes are found via suitable calls to sizeof(), as in your code.
Regarding maintainability, extendability and readability, you should certainly wrap allocation and de-allocation in a well-commented function. Calls to...
entry *allocate_entry_with_buffer();
void deallocate_entry_with_buffer(entry **entry_with_buffer);
...do not need to know implementation details of how the memory actually gets handled. People use stranger things like custom allocators and memory pools quite frequently.
As for speed, this is certainly faster than making lots of small allocations. I used to allocate whole 2D matrices with a similar strategy...
It should work, but in fact you are using a pointer for a useless indirection. Windows API (for example) uses another method for variable size structs : the variable size buffer is last in struct and is declared to be char buf[1].
Your struct would become :
struct entry {
size_t buf_len;
int something;
char buf[1];
};
The allocation is (still no error checking) :
size_t buf_len = 4; // size of the buffer
struct entry *e;
e = malloc( sizeof(*e) + buf_len - 1); // struct already has room for 1 char
e->buf_len = buf_len; // set buffer size
That's all e.buf is guaranteed to be a char array of size buf_len.
That way ensures that even if the variable part was not a character array but a int, long, or anything array, the alignement would be given by the last element being a array of proper type and size 1.
For starters, the line:
e->buf = e + sizeof(*e); // the buffer lies behind the struct
Should be:
e->buf = e + 1; // the buffer lies behind the struct
This is because e + 1 will be equal to the address at the end of the structure. As you have it, it will only be the number of bytes into the structure equal to the number of bytes in a pointer.
And, yes, it's reasonable. However, I prefer this approach:
struct entry {
size_t buf_len;
int something;
char buf[1];
};
This way, you don't mess with the pointers. Just append as many bytes as needed, and they will grow the size of your buf array.
Note: I wrote a text editor using an approach similar to this but used a Microsoft c++ extension that allowed me to declare the last member as char buf[]. So it was an empty array that was exactly as long as the number of extra bytes I allocated.
seems fine to me - put comments in though
Or you could do this - which is quite common
struct entry {
size_t buf_len;
int something;
char buf;
};
ie make the struct itself variable length. and do
size_t buf_len = 4; // size of the buffer
struct entry *e = NULL;
// check that it packs right
e = malloc(sizeof(size_t) + sizeof(int) + buf_len ); // allocate struct and buffer
e->buf_len = buf_len; // set buffer size
...... later
printf(&e.buf);
Let's consider following piece of code:
int len = 100;
char *buf = (char*)malloc(sizeof(char)*len);
printf("Appended: %s\n",struct_to_string(some_struct,buf,len));
Someone allocated amount of memory in order to get it filled with string data. The problem is that string data taken from some_struct could be ANY length. So what i want to achieve is to make struct_to_string function do the following:
Do not allocate any memory that goes outside (so, buf has to be allocated outside of the function, and passed)
Inside the struct_to_string I want to do something like:
char* struct_to_string(const struct type* some_struct, char* buf, int len) {
//it will be more like pseudo code to show the idea :)
char var1_name[] = "int l1";
buf += var1_name + " = " + some_struct->l1;
//when l1 is a int or some non char, I need to cast it
char var2_name[] = "bool t1";
buf += var2_name + " = " + some_struct->t1;
// buf+= (I mean appending function) should check if there is a place in a buf,
//if there is not it should fill buf with
//as many characters as possible (without writting to memory) and stop
//etc.
return buf;
}
Output should be like:
Appended: int l1 = 10 bool t1 = 20 //if there was good amount of memory allocated or
ex: Appended: int l1 = 10 bo //if there was not enough memory allocated
To sum up:
I need a function (or couple of functions) that adds given strings to the base string without overwritting base string;
do nothing when base string memory is full
I can not use C++ libraries
Another things that I could ask but are not so important right now:
Is there a way (in C) iterate through structure variable list to get their names, or at least to get their values without their names? (for example iterate through structure like through array ;d)
I do not normally use C, but for now I'm obligated to do, so I have very basic knowledge.
(sorry for my English)
Edit:
Good way to solve that problem is shown in post below: stackoverflow.com/a/2674354/2630520
I'd say all you need is the standard strncat function defined in the string.h header.
About the 'iterate through structure variable list' part, I'm not exactly sure what you mean. If your talking about iterating over the structure's members, a short answer would be : you can't introspect C structs for free.
You need to know beforehand what structure type you're using so that the compiler know at what offset in the memory it can find each member of your struct. Otherwise it's just an array of bytes like any other.
Don't mind asking if I wasn't clear enough or if you want more details.
Good luck.
So basically I did it like here: stackoverflow.com/a/2674354/2630520
int struct_to_string(const struct struct_type* struct_var, char* buf, const int len)
{
unsigned int length = 0;
unsigned int i;
length += snprintf(buf+length, len-length, "v0[%d]", struct_var->v0);
length += other_struct_to_string(struct_var->sub, buf+length, len-length);
length += snprintf(buf+length, len-length, "v2[%d]", struct_var->v2);
length += snprintf(buf+length, len-length, "v3[%d]", struct_var->v3);
....
return length;
}
snprintf writes as much as possible and discards everything left, so it was exactly what I was looking for.
Disclosure: I'm fairly new to C. If you could explain any answers verbosely, I would appreciate it.
I am writing a linux kernel module, and in one of the functions I am writing I need to copy a structure to userspace that looks like this:
typedef struct
{
uint32_t someProperty;
uint32_t numOfFruits;
uint32_t *arrayOfFruits;
} ObjectCapabilities;
The API I'm implementing has documentation that describes the arrayOfFruits member as "an array of size numOfFruits where each element is a FRUIT_TYPE constant." I am confused how to do this, given that the arrayOfFruits is a pointer. When I copy_to_user the ObjectCapabilities structure, it will only copy the pointer arrayOfFruits to userspace.
How can userspace continuously access the elements of the array? Here is my attempt:
ObjectCapabilities caps;
caps.someProperty = 1024;
caps.numOfFruits = 3;
uint32_t localArray[] = {
FRUIT_TYPE_APPLE,
FRUIT_TYPE_ORANGE,
FRUIT_TYPE_BANANA
};
caps.arrayOfFruits = localArray;
And then for the copy... can I just do this?
copy_to_user((void *)destination, &caps, (sizeof(caps) + (sizeof(localArray) / sizeof((localArray)[0]))));
The user needs to provide enough space for all the data being copied out. Ideally he'll tell you how much space he provided, and you check that everything fits.
The copied-out data should (in general) not include any pointers, since they're "local" to a different "process" (the kernel can be viewed as a separate process, as it were, and kernel / user interactions involve process-to-process IPC, similar to sending stuff over local or even Internet-connected sockets).
Since the kernel has pretty intimate knowledge of a process, you can skirt these rules somewhat, e.g., you could compute what the user's pointer will be, and copy out a copy of the original data, with the pointer modified appropriately. But that's kind of wasteful. Or, you can copy a kernel pointer and just not use it in the user code, but now you're "leaking data" that "bad guys" can sometimes leverage in various ways. In security-people-speak you've left a wide-open "covert channel".
In the end, then, the "right" way to do this tends to be something like this:
struct user_interface_version_of_struct {
int property;
int count;
int data[]; /* of size "count" */
};
The user code mallocs (or otherwise arranges to have sufficient space) the "user interface version" and makes some system call to the kernel (read, receive, rcvmsg, ioctl, whatever, as long as it involves doing a "read"-type operation) and tells the kernel: "here's the memory holding the struct, and here's how big it is" (in bytes, or the maximum count value, or whatever: user and kernel simply need to agree on the protocol). The kernel-side code then verifies the user's values in some appropriate manner, and either does the copy-out however is most convenient, or returns an error.
"Most convenient" is sometimes two separate copy ops, or some put_user calls, e.g., if the kernel side has the data structure you showed, you might do:
/* let's say ulen is the user supplied length in bytes,
and uaddr is the user-supplied address */
struct user_interface_version_of_struct *p;
needed = sizeof(*p) + 3 * sizeof(int);
if (needed > ulen)
return -ENOMEM; /* user did not supply enough space */
p = uaddr;
error = put_user(1024, &p->property);
if (error == 0)
error = put_user(3, &p->count);
if (error == 0 && copy_to_user(&p->data, localArray, 3 * sizeof(int))
error = -EFAULT;
You may have a situation where you must conform to some not-very-nice interface, though.
Edit: if you're adding your own system call (rather than tying in to read or ioctl for instance), you can separate the header and data, as in Adam Rosenfield's answer.
You can't copy raw pointers, since a pointer into kernel space is meaningless to userspace (and will segfault if dereferenced).
The typical way of doing something like this is to ask the userspace code to allocate the memory and pass in a pointer to that memory into a system call. If the program doesn't pass in a large enough buffer, then fail with an error (e.g. EFAULT). If there's no way for the program to know in advance a priori how much memory it will need, then typically you'd return the amount of data needed when passed a NULL pointer.
Example usage from userspace:
// Fixed-size data
typedef struct
{
uint32_t someProperty;
uint32_t numOfFruits;
} ObjectCapabilities;
// First query the number of fruits we need
ObjectCapabilities caps;
int r = sys_get_fruit(&caps, NULL, 0);
if (r != 0) { /* Handle error */ }
// Now allocate memory and query the fruit
uint32_t *arrayOfFruits = malloc(caps.numOfFruits * sizeof(uint32_t));
r = sys_get_fruit(&caps, arrayOfFruits, caps.numOfFruits);
if (r != 0) { /* Handle error */ }
And here's how the corresponding code would look in kernel space on the other side of the system call:
int sys_get_fruit(ObjectCapabilities __user *userCaps, uint32_t __user *userFruit, uint32_t numFruits)
{
ObjectCapabilities caps;
caps.someProperty = 1024;
caps.numOfFruits = 3;
// Copy out fixed-size data
int r = copy_to_user(userCaps, &caps, sizeof(caps));
if (r != 0)
return r;
uint32_t localArray[] = {
FRUIT_TYPE_APPLE,
FRUIT_TYPE_ORANGE,
FRUIT_TYPE_BANANA
};
// Attempt to copy variable-sized data. Check the size first.
if (numFruits * sizeof(uint32_t) < sizeof(localArray))
return -EFAULT;
return copy_to_user(userFruit, localArray, sizeof(localArray));
}
With copy_to_user you would do two copy to users.
//copy the struct
copy_to_user((void *)destination, &caps, sizeof(caps));
//copy the array.
copy_to_user((void *)destination->array, localArray, sizeof(localArray);
I am writing a light weight serialization function and need to include two variable sized arrays within this.
How should I track the size of each?
How should I define the struct?
Am I going about this all wrong?
EDIT: the result must be a contiguous block of memory
This resolves to something like
typedef struct
{
size_t arr_size_1, arr_size_2;
char arr_1[0/*arr_size_1 + arr_size_2*/];
} ...;
The size(s) should be in the front of the dynamic sized data, so that it doesn't move when expanding your array.
You cannot have 2 unknown sized arrays in your struct, so you must collapse them into one and then access the data relative from the first pointer.
typedef struct MyStruct_s
{
int variable_one_size;
void* variable_one_buf;
int variable_two_size;
void* variable_two_buf;
} MyStruct;
MyStruct* CreateMyStruct (int size_one, int size_two)
{
MyStruct* s = (MyStruct*)malloc (sizeof (MyStruct));
s->variable_one_size = size_one;
s->variable_one_buf = malloc (size_one);
s->variable_two_size = size_two;
s->variable_two_buf = malloc (size_two);
}
void FreeMyStruct (MyStruct* s)
{
free (s->variable_one_buf);
free (s->variable_two_buf);
free (s);
}
Since the data should be continuous in memory it is necessary to malloc a chunk of memory of the right size and manage it's contents more or less manually. You probably best create a struct that contains the "static" information and related management functions that do the memory management and give access to the "dynamic" members of the struct:
typedef struct _serial {
size_t sz_a;
size_t sz_b;
char data[1]; // "dummy" array as pointer to space at end of the struct
} serial;
serial* malloc_serial(size_t a, size_t b) {
serial *result;
// malloc more memory than just sizeof(serial), so that there
// is enough space "in" the data member for both of the variable arrays
result = malloc(sizeof(serial) - 1 + a + b);
if (result) {
result->sz_a = a;
result->sz_b = b;
}
return result;
}
// access the "arrays" in the struct:
char* access_a(serial *s) {
return &s->data[0];
}
char* access_b(serial *s) {
return &s->data[s->sz_a];
}
Then you could do things like this:
serial *s = ...;
memcpy(access_a(s), "hallo", 6);
access_a(s)[1] = 'e';
Also note that you can't just assign one serial to another one, you need to make sure that the sizes are compatible and copy the data manually.
In order to serialize variably-sized data, you have to have a boundary tag of some sort. The boundary tag can be either a size written right before the data, or it can be a special value that is not allowed to appear in the data stream and is written right after the data.
Which you choose depends on how much data you are storing, and if you are optimizing for size in the output stream. It is often easier to store a size before-hand, because you know how big to make the receiving buffer. If you don't then you have to gradually resize your buffer on load.
In some ways, I'd do things like Dan Olson. However:
1) I'd create the final struct by having two instances of a simpler struct that has just one variable array.
2) I'd declare the array with byte* and use size_t for its length.
Having said this, I'm still not entirely clear on what you're trying to do.
edit
If you want it contiguous in memory, just define a struct with two lengths. Then allocate a block big enough for both blocks that you want to pass, plus the struct itself. Set the two lengths and copy the two blocks immediately after. I think it should be clear how the lengths suffice to make the struct self-describing.