When designing structures to contain textual data, I have been using two basic approaches illustrated below:
typedef struct {
STRING address1;
STRING address2;
STRING city;
STRING state;
STRING zip;
} ADDRESS;
typedef struct {
STRING* address1;
STRING* address2;
STRING* city;
STRING* state;
STRING* zip;
} ADDRESS;
where STRING is some variable length string-storing type. The advantage of the pointer version is that I can store NULL indicating that data is missing. For example, address2 might be not provided for some addresses. In the type with embedded STRINGs, I have to use a "blank" string, meaning one that has 0 length.
With the pointers there is (possibly) more code burden because I have to check every member for NULL before using. The advantage is not that great, however, because usually the embedded version has to be checked too. For example, if I am printing an address, I have to check for a zero-length string and skip that line. With pointers the user can actually indicate they want a "blank" versus a missing value, although it is hard to see a use for this.
When creating or freeing the structure, pointers add a bunch of additional steps. My instinct is to standardize on the embedded style to save these steps, but I am concerned that there might be a hidden gotcha. Is this an unwarranted fear, or should I be using the pointers for some compelling reason?
Note that memory use is an issue, but it is pretty minor. The pointer version takes a little bit more memory because I am storing pointers to the structs in addition to the structs. But each string struct takes maybe 40 bytes on average, so if I am storing 4 byte pointers, then the pointer version costs maybe 10% more memory which is not significant. Having null pointers possible does not save significant memory because most fields are populated.
Question is About ADDRESS not STRING
Some of the respondents seem to be confused and think I am asking about global tradeoffs, like how to minimize my total work. That is not the case. I am asking about how to design ADDRESS, not STRING. The members of address could have fixed arrays, or in other cases not. For the purposes of my question, I am not concerned about are the consequences for the container.
I have already stated that the only issue I can see is that it costs more time to use pointers, but I get the benefit of being able to store a NULL. However, as I already said, that benefit does not seem to be significant, but maybe it is for some reason. That is the essence of my question: is there some hidden benefit of having this flexibility that I am not seeing and will wish I had later on.
If you don't understand the question, please read the preliminary answer I have written myself below (after some additional thought) to see the kind of answer I am looking for.
Tradeoffs over memory usage and reduction in mallocs
Seems like the tradeoffs center around two questions: 1) How precious is memory? and 2) Does it matter that a fixed amount of memory is allocated for the strings, limiting the lengths to be stored in each field?
If memory is more important than anything else, then the pointer version probably wins. If predictability of storage usage and avoidance of mallocs is preferred, and limiting the length of the names to some fixed amount is acceptabe, then the fixed length version may be the winner.
One problem with the embedded style is that STRING needs to be defined as something like char[MAX_CHAR + 1] where MAX_CHAR is a pessimistic maximum length for the given fields. The pointer style allows one to allocate the correct amount of memory. The downside as you mention is a much higher cognitive overhead managing your struct.
I have been considering this more deeply and I think that in most cases pointers are necessary because it is important to discriminate between blank and missing. The reason for this is that the missing data is needed when an input is invalid or corrupt or left out. For example, let's imagine that when reading from a file, the file is corrupted so a field like zip code is unreadable. In that case the data is "missing" and the pointer should be NULL. On the other hand, let's imagine that the place has no zip code, then it is "blank". So, NULL means that the user has not yet provided information, but blank means the user has provided the information and there is none of the type in question.
So, to further illustrate the importance of using the pointer, imagine that a complex structure is populated over time in different, asynchronous steps. Here we need to know what fields have been read, and which ones have not. Unless we are using pointers (or adding additional metadata), we have no way of telling the difference between a field that has been answered and one for which the answer is "none". Imagine the system prompting the user "what is the zip code?". User says, "this place has no zip code". Then 5 minutes later the system asks again, "what is the zip code?". That use case makes it clear that we need pointers in most cases.
In this light, the only situation where I should use embedded structs is when the container structure is guaranteed to have a full set of data whenever it is created.
Related
This document called "Good Practices in Library Design, Implementation, and Maintenance" by Ulrich Drepper says (bottom of page 5):
[...] the type definition should always create at least a minimal
amount of padding to allow future growth
[...]
Second, a structure should contain at the end a certain number of fill bytes.
struct the_struct
{
int foo;
// ...and more fields
uintptr_t filler[8];
};
[...]
If at a later time a field has to be added to the structure the type definition can be changed to this:
struct the_struct
{
int foo;
// ...and more fields
union
{
some_type_t new_field;
uintptr_t filler[8];
} u;
};
I don't see the point of adding this filler at the end of the structure. Yes it means that when adding a new field (new_field) to the structure, it doesn't actually grow. But isn't the whole point of adding new fields to a structure that you didn't knew you were going to need them? In this example, what if you want to add not one field but 20? Should you then use a filler of 1k bytes just in case? Also, why does is it important that the size of a struct doesn't change in subsequent versions of a library? If the library provides clean abstractions, that shouldn't matter right? Finally, using a 64 bytes filler (8 uintpr_t (yes, it's not necessarily 64 bytes)) sounds like a waste of memory...
The document doesn't go into the details of this at all. Would you have any explanations to why this advice "adding fillers at the end of struct to plan for future growth" is a good one?
Depending on circumstances, yes, the size of the structure can be important for binary compatibility.
Consider stat(). It's typically called like this:
struct stat stbuf;
int r = stat(filename, &stbuf);
With this setup, if the size of the stat structure ever changes, every caller becomes invalid, and will need to be recompiled. If both the called and the calling code are part of the same project, that may not be a problem. But if (as in the case of stat(), which is a system call into the Unix/Linux kernel) there are lots and lots of callers out there, it's practically impossible to force them all to recompile, so the implication is that the size of the stat structure can never be changed.
This sort of problem mainly arises when the caller allocates (or inspects/manipulates) actual instances of the structure. If, on the other hand, the insides of the structure are only ever allocated and manipulated by library code -- if calling code deals only with pointers to the struct, and doesn't try to interpret the pointed-to structures -- it may not matter if the structure changes.
(Now, with all of that said, there are various other things that can be done to mitigate the issues if a struct has to change size. There are libraries where the caller allocates instances of a structure, but then passes both a pointer to the structure, and the size of the structure as the caller knows it, down into the library code. Newer library code can then detect a mismatch, and avoid setting or using newer fields which an older caller didn't allocate space for. And I believe gcc, at least, implements special hooks so that glibc can implement multiple versions of the same structure, and multiple versions of the library functions that use them, so that the correct library function can be used corresponding to the version of the structure that a particular caller is using. Going back to stat(), for example, under Linux there are at least two different versions of the stat structure, one which allocates 32 bits for the file size and one which allocates 64.)
But isn't the whole point of adding new fields to a structure that you
didn't knew you were going to need them?
Well yes, if you knew all along that you would need those members, then it would be counter-productive to intentionally omit them. But sometimes you indeed discover only later that you need some additional fields. Drepper's recommendations speak to ways to design your code -- specifically your structure definitions -- so that you can add members with the minimum possible side effects.
In this example, what if you
want to add not one field but 20?
You don't start out saying "I'm going to want to add 20 members". Rather, you start out saying "I may later discover a need for some more members." That's a prudent position to take.
Should you then use a filler of 1k
bytes just in case?
That's a judgment call. I recon that a KB of extra space in the structure definition is probably overkill in most cases, but there might be a context where that's reasonable.
Also, why does is it important that the size of a
struct doesn't change in subsequent versions of a library? If the
library provides clean abstractions, that shouldn't matter right?
How important it is that the size remains constant is a subjective question, but the size is indeed relevant to binary compatibility for shared libraries. Specifically, the question is whether I can drop a new version of the shared lib in place of the old one, and expect existing programs to work with the new one without recompilation.
Technically, if the definition of the structure changes, even without its size changing, then the new definition is incompatible with the old one as far as the C language is concerned. In practice, however, with most C implementations, if the structure size is the same and the layout does not change except possibly within previously-unused space, then existing users will not notice the difference in many operations.
If the size does change, however, then
dynamic allocation of instances of the structure will not allocate the correct amount of space.
arrays of the structure will not be laid out correctly.
copying from one instance to another via memcpy() will not work correctly.
binary I/O involving instances of the structure will not transfer the correct number of bytes.
There are likely other things that could go wrong with a size change that would (again, in practice) be ok under conversion of some trailing padding into meaningful members.
Do note: one thing that might still be a problem if the structure members change without the overall size changing is passing structures to functions by value and (somewhat less so) receiving them as return values. A library making use of this approach to provide for binary compatibility would do well to avoid providing functions that do those things.
Finally, using a 64 bytes filler (8 uintpr_t (yes, it's not
necessarily 64 bytes)) sounds like a waste of memory...
In a situation in which those 64 bytes per structure is in fact a legitimate concern, then that might trump binary compatibility concerns. That would be the case if you anticipate a very large number of those structures to be in use at the same time, or if you are extremely memory-constrained. In many cases, however, the extra space is inconsequential, whereas the extra scope for binary compatibility afforded by including padding is quite valuable.
The document doesn't go into the details of this at all. Would you
have any explanations to why this advice "adding fillers at the end of
struct to plan for future growth" is a good one?
Like most things, the recommendation needs to be evaluated relative to your particular context. In the foregoing, I've touched on most of the points you would want to consider in such an evaluation.
An interesting feature of realloc() is that it somehow knows how long your data is when it is copying or extending your allocated memory.
I read that what happens is that behind the scenes there is some meta information stored about a pointer (which contains it's allocated memory size), usually immediately before the address the pointer is pointing at (but of course, subject to implementation).
So my question is, if there is such data stored why isn't it exposed via an API, so things like the C string for example, won't have to look for a \0 to know where the end of a string is.
There could be hundreds of other uses as well.
As you said yourself, it's subject to the implementation of the standard libraries' memory manager.
C doesn't have a standard for this type of functionality, probably because generally speaking C was designed to be simple (but also provide a lot of capabilities).
With that said, there isn't really much use for this type of functionality outside of features for debugging memory allocation.
Some compilers do provide this type of functionality (specifically memory block size), but it's made pretty clear that it's only for the purpose of debugging.
It's not uncommon to write your own memory manager, allowing you to have full control over allocations, and even make assumptions for other library components. As you mentioned, your implementation can store the size of an allocation somewhere in a header, and a string implementation can reference that value rather than walking the bytes for \0 termination.
This information is not always accurate.
On Windows, the size is rounded up to a multiple of at least 8.
realloc does fine with this because it cares only about the allocated size, but you won't get the requested size.
Here it's worth understanding how allocators work a bit. For example, if you take a buddy allocator, it allocates chunks in predetermined sizes. For simplicity, let's say powers of 2 (real world allocators usually use tighter sizes).
In this kind of simplified case, if you call malloc and request 19 bytes of memory for your personal use, then the allocator is actually going to give you a 32-byte chunk worth of memory. It doesn't care that it gave you more than you needed, as that still satisfies your basic request by giving you something as large or larger than what you needed.
These chunk sizes are usually stored somewhere, somehow, for a general-purpose allocator that can handle variable-sized requests to be able to free chunks and do things like merge free chunks together and split them apart. Yet they're identifying the chunk size, not the size of your request. So you can't use that chunk size to see where a string should end, e.g., since according to you, you wanted one with 19 bytes, not 32 bytes.
realloc also doesn't care about your requested 19-byte size. It's working at the level of bits and bytes, and so it copies the whole chunk's worth of memory to a new, larger chunk if it has to.
So these kinds of chunk sizes generally aren't useful for implementing things like data structures. For these, you want to be working at the data size proportional to the amount of memory you requested which is something many allocators don't even bother to store (they don't need to in order to work given their efficiency needs). So it's often up to you to keep track of that size which is in tune with the logic of your software in some form (a sentinel like a null terminator is one form).
As for why querying these sizes is not available in the standard, it's probably because the need to know it would be a rather obscure, low-level kind of need given that it has nothing to do with the amount of memory you actually requested to work with. I have often wished for it for some very low-level debugging needs, but there are platform/compiler-specific alternatives that I have used instead, and I found they weren't quite as handy as I thought they would be (even for just low-level debugging).
You asked:
if there is such data stored why isn't it exposed via an API, so things like the C string for example, won't have to look for a \0 to know where the end of a string is.
You can allocate enough memory with the ability to hold 100 characters but you may use that memory to hold a string that is only 5 characters long. If you relied on the pointer metadata, you will get the wrong result.
char* cp = malloc(100);
strcpy(cp, "hello");
The second reason you would need the terminating null character is when you use stack memory to create a string.
char str[100] = "hello";
Without the terminating null character, you won't be able determine the length of the string held in str.
You said:
There could be hundreds of other uses as well.
I don't have a good response to that other than to say that custom memory allocators often provide access to the metadata. The reason for not including such APIs in the standard library is not obvious to me.
IMO all code that returns structure directly can be modified to return pointer to structure.
When is returning a structure directly a good practice?
Modified how? Returning a pointer to a static instance of the structure within the function, thus making the function non-reentrant; or by returning a pointer to a heap allocated structure that the caller has to make sure to free and do so appropiately? I would consider returning a structure being the good practice in the general case.
The biggest advantage to returning a complete structure instead of a pointer is that you don't have to mess with pointers. By avoiding the risks inherent with pointers, especially if you're allocating and freeing your own memory, both coding and debugging can be significantly simplified.
In many cases, the advantages of passing the structure directly outweigh the downsides (time/memory) of copying the entire structure to the stack. Unless you know that optimization is necessary, no reason not to take the easier path.
I see the following cases as the ones I would most commonly opt for the passing structs directly approach:
"Functional programming" style code. Lots of stuff is passed around and having pointers would complicate the code a lot (and that is not even counting if you need to start using malloc+free)
Small structs, like for example
struct Point{ int x, y; };
aren't worth the trouble of passing stuff around by reference.
And lastly, lets not forget that pass-by-value and pass-by-reference are actually very different so some classes of programs will be more suited to one style and will end up looking ugly if the other style is used instead.
These other answers are good, but I think missingno comes closest to "answering the question" by mentioning small structs. To be more concrete, if the struct itself is only a few machine words long, then both the "space" objection and the "time" objection are overcome. If a pointer is one word, and the struct is two words, how much slower is the struct copy operation vs the pointer copy? On a cached architecture, I suspect the answer is "none aat all". And as for space, 2 words on stack < 1 word on stack + 2 words (+overhead) on heap.
But thes considerations are only appropriate for specific cases: THIS porion of THIS program on THIS architecture.
For the level of writing C programs, you should use whichever is easier to read.
If you're trying to make your function side-effect free, returning a struct directly would help, because it would effectively be pass-by-value. Is it more efficient? No, passing by reference is quicker. But having no side effects can really simplify working with threads (a notoriously difficult task).
There are a few cases where returning a structure by value is contra-indicated:
1) A library function that returns 'token' data that is to be re-used later in other calls, eg. a file or socket stream descriptor. Returning a complete structure would break encapsulation of the library.
2) Structs containing data buffers of variable length where the struct has been sized to accommodate the absolute maximum size of the data but where the average data size is much less, eg. a network buffer struct that has a 'dataLen' int and a 'char data[65536]' at its end.
3) Large structs of any typedef where the cost of copying the data becomes significant, eg:
a) When the struct has to be returned through several function calls - multiple copying of the same data.
b) Where the struct is subsequently queued off to other threads - wide queues means longer lock times during the copy-in/copy-out and so increased chance of contention. That, and the size of the struct is inflicted on both producer and consumer thread stacks.
c) Where the struct is often moved around between layers, eg. protocol stack.
4) Where structs of varying def. are to be stored in any array/list/queue/stack/whateverContainer.
I suspect that I am so corrupted by c++ and other OO languages that I tend to malloc/new almost anything that cannot be stored in a native type
Rgds,
Martin
I have a function and i'm accessing a struct's members a lot of times in it.
What I was wondering about is what is the good practice to go about this?
For example:
struct s
{
int x;
int y;
}
and I have allocated memory for 10 objects of that struct using malloc.
So, whenever I need to use only one of the object in a function, I usually create (or is passed as argument) pointer and point it to the required object (My superior told me to avoid array indexing because it adds a calculation when accessing any member of the struct)
But is this the right way? I understand that dereferencing is not as expensive as creating a copy, but what if I'm dereferencing a number of times (like 20 to 30) in the function.
Would it be better if i created temporary variables for the struct variables (only the ones I need, I certainly don't use all the members) and copy over the value and then set the actual struct's value before returning?
Also, is this unnecessary micro optimization? Please note that this is for embedded devices.
This is for an embedded system. So, I can't make any assumptions about what the compiler will do. I can't make any assumptions about word size, or the number of registers, or the cost of accessing off the stack, because you didn't tell me what the architecture is. I used to do embedded code on 8080s when they were new...
OK, so what to do?
Pick a real section of code and code it up. Code it up each of the different ways you have listed above. Compile it. Find the compiler option that forces it to print out the assembly code that is produced. Compile each piece of code with every different set of optimization options. Grab the reference manual for the processor and count the cycles used by each case.
Now you will have real data on which to base a decision. Real data is much better that the opinions of a million highly experience expert programmers. Sit down with your lead programmer and show him the code and the data. He may well show you better ways to code it. If so, recode it his way, compile it, and count the cycles used by his code. Show him how his way worked out.
At the very worst you will have spent a weekend learning something very important about the way your compiler works. You will have examined N ways to code things times M different sets of optimization options. You will have learned a lot about the instruction set of the machine. You will have learned how good, or bad, the compiler is. You will have had a chance to get to know your lead programmer better. And, you will have real data.
Real data is the kind of data that you must have to answer this question. With out that data nothing anyone tells you is anything but an ego based guess. Data answers the question.
Bob Pendleton
First of all, indexing an array is not very expensive (only like one operation more expensive than a pointer dereference, or sometimes none, depending on the situation).
Secondly, most compilers will perform what is called RVO or return value optimisation when returning structs by value. This is where the caller allocates space for the return value of the function it calls, and secretly passes the address of that memory to the function for it to use, and the effect is that no copies are made. It does this automatically, so
struct mystruct blah = func();
Only constructs one object, passes it to func for it to use transparently to the programmer, and no copying need be done.
What I do not know is if you assign an array index the return value of the function, like this:
someArray[0] = func();
will the compiler pass the address of someArray[0] and do RVO that way, or will it just not do that optimisation? You'll have to get a more experienced programmer to answer that. I would guess that the compiler is smart enough to do it though, but it's just a guess.
And yes, I would call it micro optimisation. But we're C programmers. And that's how we roll.
Generally, the case in which you want to make a copy of a passed struct in C is if you want to manipulate the data in place. That is to say, have your changes not be reflected in the struct it self but rather only in the return value. As for which is more expensive, it depends on a lot of things. Many of which change implementation to implementation so I would need more specific information to be more helpful. Though, I would expect, that in an embedded environment you memory is at a greater premium than your processing power. Really this reads like needless micro optimization, your compiler should handle it.
In this case creating temp variable on the stack will be faster. But if your structure is much bigger then you might be better with dereferencing.
I will begin to use C for an Operating Systems course soon and I'm reading up on best practices on using C so that headaches are reduced later on.
This has always been one my first questions regarding arrays as they are easy to screw up.
Is it a common practice out there to bundle an array and its associated variable containing it's length in a struct?
I've never seen it in books and usually they always keep the two separate or use something like sizeof(array[]/array[1]) kind of deal.
But with wrapping the two into a struct, you'd be able to pass the struct both by value and by reference which you can't really do with arrays unless using pointers, in which case you have to again keep track of the array length.
I am beginning to use C so the above could be horribly wrong, I am still a student.
Cheers,
Kai.
Yes this is a great practice to have in C. It's completely logical to wrap related values into a containing structure.
I would go ever further. It would also serve your purpose to never modify these values directly. Instead write functions which act on these values as a pair inside your struct to change length and alter data. That way you can add invariant checks and also make it very easy to test.
Sure, you can do that. Not sure if I'd call it a best practice, but it's certainly a good idea to make C's rather rudimentary arrays a bit more manageable. If you need dynamic arrays, it's almost a requirement to group the various fields needed to do the bookkeeping together.
Sometimes you have two sizes in that case: one current, and one allocated. This is a tradeoff where you trade fewer allocations for some speed, paying with a bit of memory overhead.
Many times arrays are only used locally, and are of static size, which is why the sizeof operator is so handy to determine the number of elements. Your syntax is slightly off with that, by the way, here's how it usually looks:
int array[4711];
int i;
for(i = 0; i < sizeof array / sizeof *array; i++)
{
/* Do stuff with each element. */
}
Remember that sizeof is not a function, the parenthesis are not always needed.
EDIT: One real-world example of a wrapping exactly as that which you describe is the GArray type provided by glib. The user-visible part of the declaration is exactly what you describe:
typedef struct {
gchar *data;
guint len;
} GArray;
Programs are expected to use the provided API to access the array whenever possible, not poke these fields directly.
There are three ways.
For static array (not dynamically allocated and not passed as pointer) size is knows at compile time so you can used sizeof operator, like this: sizeof(array)/sizeof(array[0])
Use terminator (special value for last array element which cannot be used as regular array value), like null-terminated strings
Use separate value, either as a struct member or independent variable. It doesn't really matter because all the standard functions that work with arrays take separate size variable, however joining the array pointer and size into one struct will increase code readability. I suggest to use to have a cleaner interface for your own functions. Please note that if you pass your struct by value, called function will be able to change the array, but not the size variable, so passing struct pointer would be a better option.
For public API I'd go with the array and the size value separated. That's how it is handled in most (if not all) c library I know. How you handle it internally it's completely up to you. So using a structure plus some helper functions/macros that do the tricky parts for you is a good idea. It's always making me head-ache to re-think how to insert an item or to remove one, so it's a good source of problems. Solving that once and generic, helps you getting bugs from the beginning low. A nice implementation for dynamic and generic arrays is kvec.
I'd say it's a good practice. In fact, it's good enough that in C++ they've put it into the standard library and called it vector. Whenever you talk about arrays in a C++ forum, you'll get inundated with responses that say to use vector instead.
I don't see anything wrong with doing that but I think the reason that that is not usually done is because of the overhead incurred by such a structure. Most C is bare-metal code for performance reasons and as such, abstractions are often avoided.
I haven't seen it done in books much either, but I've been doing the same the same thing for a while now. It just seems to make sense to "package" those things together. I find it especially useful if you need to return an allocated array from a method for instance.
If you use static arrays you have access to the size of array using sizeof operator. If you'll put it into struct, you can pass it to function by value, reference and pointer. Passing argument by reference and by pointer is the same on assembly level (I'm almost sure of it).
But if you use dynamic arrays, you don't know the size of array at compile time. So you can store this value in struct, but you will also store only a pointer to array in structure:
struct Foo {
int *myarray;
int size;
};
So you can pass this structure by value, but what you realy do is passing pointer to int (pointer to array) and int (size of array).
In my opinion it won't help you much. The only thing that is in plus, is that you store the size and the array in one place and it is easy to get the size of the array. If you will use a lot of dynamic arrays you can do it this way. But if you will use few arrays, easier will be not to use structures.
I've never seen it done that way, but I haven't done OS level work in over a decade... :-) Seems like a reasonable approach at first glance. Only concern would be to make sure that the size somehow stays accurate... Calculating as needed doesn't have that concern.
considering you can calculate the length of the array (in bytes, that is, not # of elements) and the compiler will replace the sizeof() calls with the actual value (its not a function, calls to it are replaced by the compiler with the value it 'returns'), then the only reason you'd want to wrap it in a struct is for readability.
It isn't a common practice, if you did it, someone looking at your code would assume the length field was some special value, and not just the array size. That's a danger lurking in your proposal, so you'd have to be careful to comment it properly.
I think this part of your question is backwards:
"But with wrapping the two into a struct, you'd be able to pass the struct both by value and by reference which you can't really do with arrays unless using pointers, in which case you have to again keep track of the array length."
Using pointers when passing arrays around is the default behavior; that doesn't buy you anything in terms of passing the entire array by value. If you want to copy the entire array instead of having it decay to a pointer you NEED to wrap the array in a struct. See this answer for more information:
Pass array by value to recursive function possible?
And here is more on the special behavior of arrays:
http://denniskubes.com/2012/08/20/is-c-pass-by-value-or-reference/