I'm writing a library which allows to build Bayesian nets. The structure of a net is encapsulated and the user cannot access its fields; however they can get and set some of them. Say you want to write the accessor function for the field table (which is basically a double array) for instance. Between the following options, which would be more appropriate?
First option:
int getTable(Net *net, double *res)
Second option:
double *getTable(Net *net)
In the first option, the user provides a pointer to the array where the table values will be written. The function copies the table values on res, leaving the user with no direct access to the internal structure. Modifying res leaves the table of the net unchanged. Of course, another function (say getTableSize()) is provided to correctly allocate res. This seems safe (the internal structure stay coherent) and has the advantage that you can return a code value if something goes wrong. The downside is that this approach can be slower than the next one, since it involves a copy. Typically, the number of values of table may vary from just 1 to a couple of hundreds.
In the second option, the function returns directly the pointer to the internal values. In the doc, I will specify that user cannot try to free the pointer or modify the values. Freeing the pointer would likely result in a segmentation fault and memory leaks if other operations on the net are performed. Modifying the table won't result in any apparent error, but the internal coherence would be broken and the results of subsequent calculations might be very wrong and very hard to debug for the user.
Which option do you prefer? There other stuff to consider? Is there another approach to prefer?
Personally I would go for the first option because of the ability to return an error code. This would also solve your problem regarding the user freeing wrongly the returned value. And passing a pointer on a variable declared on the stack is easy.
Note that you can also make things more clear, i.e. the returned value cannot be modified or freed, with the second option by returning a const pointer like this
const double * getTable(Net *net);
In such way, the caller cannot modify the value, unless he cast it to a non const but I think this would be going a bit too far since the caller break intentionally your interface.
More info on constness can be found on wikipedia
I think a good habit is to always demand a return code from functions that can fail for some reason.
Error handling is much more efficient when working with return codes.
I'd go with option one.
Also, I don't know if this is a mistake or not but option two returns a double pointer - if this is the correct behavior, then function one should have the signature:
int getTable(Net *net, double **res)
Additionally, as Eugene Sh. mentioned in the comments, some environments might not even support malloc (some embedded devices firmware comes to mind), so giving the user a choice whether to pass in a malloc'd variable or a stack allocated variable is also a good selling point for option one.
I have two points for you to consider:
You want memory deallocation to be in the same place where allocation is. If you allocate memory in a function and return pointer to it, then the caller of the function has to know how to free it and has to remember to free it. Maintaining that kind of code would be a nightmare.
Premature optimization is the root of all evil. Do not optimize your code until you are certain (i.e. you measured) that exactly that part of code is causing problems.
That said, first option is the only option.
Related
I have a function that reads from a socket, it returns a char** where packets are stored and my intention is to use a NULL unsigned int pointer where I store the length of single packet.
char** readPackets(int numToRead,unsigned int **lens,int socket){
char** packets=(char**)malloc(numToRead);
int *len=(int*)malloc(sizeof(int)*numToRead);
*(lens)=len;
for(int i=0;i<numToRead;i++){
//read
packets[i]=(char*)malloc(MAX_ETH_LEN);
register int pack_len=read(socket,packets[i],MAX_ETH_LEN);
//TODO handler error in case of freezing
if(pack_len<=0){
i--;
continue;
}
len[i]=pack_len;
}
return packets;
}
I use it in this way:
unsigned int *lens_out=NULL;
char **packets=readPackets(N_PACK,&lens,sniff_sock[handler]);
where N_PACK is a constant defined previously.
Now the problem is that when I am inside the function everything works, in fact *(lens) points to the same memory area of len and outside the function lens_out points to the same area too. Inside the function len[i] equals to *(lens[i]) (I checked it with gdb).
The problem is that outside the function even if lens_out points to the same area of len elements with same index are different for example
len[0]=46
lens_out[0]=4026546640
Can anyone explain where I made the mistake?
Your statement char** packets=(char**)malloc(numToRead) for sure does not reserve enough memory. Note that an element of packets-array is of type char*, and that sizeof(char*) is probably 8 (eventually 4), but very very unlikely 1. So you should write
char** packets = malloc(sizeof(char*) * numToRead)
Otherwise, you write out of the bounds of reserved memory, thereby yielding undefined behaviour (probably the one you explained).
Note further that with i--; continue;, you get memory leaks since you assign a new memory block to the ith element, but you lose reference to the memory reserved right before. Write free(packets[i]);i--;continue; instead.
Further, len[0] is an integral type, whereas lens[0] refers to a pointer to int. Comparing these two does not make sense.
Firstly, I want to put it out there that you should write clear code for the sake of future maintenance, and not for what you think is optimal (yet). This entire function should merely be replaced with read. That's the crux of my post.
Now the problem is that when I am inside the function everything works
I disagree. On a slightly broader topic, the biggest problem here is that you've posted a question containing code which doesn't compile when copied and pasted unmodified, and the question isn't about the error messages, so we can't answer the question without guessing.
My guess is that you haven't noticed these error messages; you're running a stale binary which we don't have the source code for, we can't reproduce the issue and we can't see the old source code, so we can't help you. It is as valid as any other guess. For example, there's another answer which speculates:
Your statement char** packets=(char**)malloc(numToRead) for sure does not reserve enough memory.
The malloc manual doesn't guarantee that precisely numToRead bytes will be allocated; in fact, allocations to processes tend to be performed in pages and just as the sleep manual doesn't guarantee a precise number of milliseconds/microseconds, it may allocate more or it may allocate less; in the latter case, malloc must return NULL which your code needs to check.
It's extremely common for implementations to seem to behave correctly when a buffer is overflowed anyway. Nonetheless, it'd be best if you fixed that buffer overflow. malloc doesn't know about the type you're allocating; you need to tell it everything about the size, not just the number of elements.
P.S. You probably want select or sleep within your loop, you know, to "handle error in case of freezing" or whatever. Generally, OSes will switch context to another program when you call one of those, and only switch back when there's data ready to process. By calling sleep after sending or receiving, you give the OS a heads up that it needs to perform some I/O. The ability to choose that timing can be beneficial, when you're optimising. Not at the moment, though.
Inside the function len[i] equals to *(lens[i]) (I checked it with gdb).
I'm fairly sure you've misunderstood that. Perhaps gdb is implicitly dereferencing your pointers, for you; that's really irrelevant to C (so don't confuse anything you learn from gdb with anything C-related).
In fact, I strongly recommend learning a little bit less about gdb and a lot more about assert, because the former won't help you document your code for future maintenance from other people, including us, those who you ask questions to, where-as the latter will. If you include assert in your code, you're almost certainly strengthening your question (and code) much more than including gdb into your question would.
The types of len[i] and *(len[i]) are different, and their values are affected by the way types are interpreted. These values can only be considered equal When they're converted to the same type. We can see this through C11/3.19p1 (the definition of "value", where the standard establishes it is dependant upon type). len[i] is an int * value, where-as *(len[i]) is an int value. The two categories of values might have different alignment, representation and... well, they have different semantics entirely. One is for integral data, and the other is a reference to an object or array. You shouldn't be comparing them, no matter how equal they may seem; the information you obtain from such a comparison is virtually useless.
You can't use len[i] in a multiplication expression, for example. They're certainly not equal in that respect. They might compare equal (as a side-effect of comparison introducing implicit conversions), which is useless information for you to have, and that is a different story.
memcmp((int[]){0}, (unsigned char[]){ [sizeof int] = 42 }, sizeof int) may return 0 indicating that they're equal, but you know that array of characters contains an extra byte, right? Yeh... they're equal...
You must check the return value of malloc (and don't cast the return value), if you're using it, though I really think you should reconsider your options with that regard.
The fact that you use malloc means everyone who uses your function must then use free; it's locking down-stream programmers into an anti-pattern that can tear the architecture of software apart. You should separate categories of allocation logic and user interface logic from processing logic.
For example, you use read which gives you the opportunity to choose whatever storage duration you like. This means you have an immense number of optimisation opportunities. It gives you, the downstream programmer, the opportunity to write flexible code which assigns whatever storage duration you like to the memory used. Imagine if, on the other hand, you had to free every return value from every function... That's the mess you're encouraging.
This is especially a poor, inefficient design when constants are involved (i.e. your usecase), because you could just use an automatic array and get rid of the calls to malloc and free altogether... Your downstream programmers code could be:
char packet[size][count];
int n_read = read(fd, packet, size * count);
Perhaps you think using malloc to allocate (and later read) n spaces for packets is faster than using something else to allocate n spaces. You should test that theory, because from my experience computers tend to be optimised for simpler, shorter, more concise logic.
In anticipation:
But I can't return packet; like that!
True. You can't return packet; to your downstream programmer, so you modify an object pointed at by an argument. That doesn't mean you should use malloc, though.
Unfortunately, too many programs are adopting this "use malloc everywhere" mentality. It's reminiscent of the "don't use goto" crap that we've been fed. Rather than listening to cargo cult propaganda, I recommend thinking critically about what you hear, because your peers are in the same position as you; they don't necessarily know what they're talking about.
Although the subject is discussed many times, I haven't found any satisfying answer so far. When to return data from a function by return or to pass a reference to change the data on address? The classic answer is to pass a variable as reference to a function when it becomes large (to avoid stack copying). This looks true for anything like a structure or array. However returning a pointer from a function is not uncommon. In fact some functions from the C library to the exact thing. For example:
char *strcat(char *dst, const char *src);
Always returns a pointer to destination even in case of an error. In this case we can just use the passed variable and leave the return for what it is (as most do).
When looking at structures I see the same thing happening. I often return pointers when functions only need to be used in variable initialization.
char *p = func(int i, const char *s);
Then there is the argument that stack coping variables is expensive, and so to use pointers instead. But as mentioned here some compilers are able to decide this themselves (assuming this goes for C as well). Is there a general rule, or at least some unwritten convention when to use one or the other? I value performance above design.
Start by deciding which approach makes the most sense at the logical level, irrespective of what you think the performance implications might be. If returning a struct by value most clearly conveys the intent of the code, then do that.
This isn't the 1980s anymore. Compilers have gotten a lot smarter since then and do a really good job of optimizing code, especially code that's written in a clear, straightforward manner. Similarly, parameter passing and value return conventions have become fairly sophisticated as well. The simplistic stack-based model doesn't really reflect the reality of modern hardware.
If the resulting application doesn't meet your performance criteria, then run it through a profiler to find the bottlenecks. If it turns out that returning that struct by value is causing a problem, then you can experiment with passing by reference to the function.
Unless you're working in a highly constrained, embedded environment, you really don't have to count every byte and CPU cycle. You don't want to be needlessly wasteful, but by that same token you don't want to obsess over how things work at the low level unless a) you have really strict performance requirements and b) you are intimately familiar with the details of your particular platform (meaning that you not only know your platform's function calling conventions inside and out, you know how your compiler uses those conventions as well). Otherwise, you're just guessing. Let the compiler do the hard work for you. That's what it's there for.
Rules of thumb:
If sizeof(return type) is bigger than sizeof(int), you should probably pass it by pointer to avoid the copy overhead. This is a performance issue. There's some penalty for dereferencing the pointer, so there are some exceptions to this rule.
If the return type is complex (containing pointer members), pass it by pointer. Copying the local return value to the stack will not copy dynamic memory, for example.
If you want the function to allocate the memory, it should return a pointer to the newly allocated memory. It's called the factory design pattern.
If you have more than one thing you want to return from a function - return one by value, and pass the rest by pointers.
If you have a complex/big data type which is both input and output, pass it by pointer.
For example, in Linux, I have a pointer pointing to a task_struct. Later, the task_struct might migrate or deleted. How do I know whether the pointer still points to a task_struct or not?
It's not possible.
Pointers only contain addresses, and generally it's not possible to determine whether or not a given address is "valid".
Sometimes you can ask the entity that gave you the pointer to begin with if it's still valid, but that of course depends on the exact details of the entity. The language itself cannot do this.
You don't know, because:
a pointer just contains the address of the object it points to;
the type information is lost at compile time.
So, C provides no facilities for dealing with this kind of problems, you have to track what happens to stuff you point to on your own.
The most you can ask (and it is alreay OS-specific) is to check if the memory page where the structure would reside is still accessible, but usually it's not a particularly useful information.
Depending on your allocation pattern/luck, you might get a segmentation fault (which of course kills your program)...but that at least would tell you the reference is no longer valid.
However, as previously stated, the best way is to track the validity yourself.
If you need to keep moving a struct around in memory (rather than just blanking it and reinitializing it at its current location), you could consider using a pointer to a pointer to make the tracking easier.
"ie. Everything gets a reference to the pointer to the struct, and then when you move or delete the struct you just set that pointer to NULL or to the new memory location."
Also, in general, if you want to do checks on your program for this kind of weirdness, I would recommend looking into valgrind.
It is your responsibility in C to write your code so that you keep track of it. You can use the special value of NULL (representing not pointing to anything), setting the pointer to NULL when you remove (or haven't yet set) whatever it was pointing to & testing for NULL before using it. You might also design your code in a way that the question never comes up.
There is no way to query a random pointer value to see if it represents something, just like there is no way to query an int variable to check if the value in it is uninitialized, junk, or the correct result of a computation.
It is all a matter of software design and, when necessary, using the value of NULL to designate not set.
read about it here.
I need to implement a variation of such an interface, say we are given a large memory space to manage there should be getmem(size) and free(pointer to block) functions that has to make sure free(pointer to block) can actually free the memory if and only if all processes using that block are done using it.
What I was thinking about doing is to define a Collectable struct as pointer to block, size of it, and process using it count. then whenever a process using a Collectable struct instance for the first time it has to explicitly increment the count, and whenever the process free()'s it, the count is decremented.
The problem with this approach is that all processes must respond to that interface and make it explicitly work : whenever assigning collectable pointer to an instance the process must explicitly inc that counter, which does not satisfy me, I was thinking maybe there is a way to create a macro for this to happen implicitly in every assignment?
I'm seeking of ways to approach this problem for a while, so other approaches and ideas would be great...
EDIT : the above approach doesn't satisfy me not only because it doesn't look nice but mostly because I cant assume a running process's code would care for updating my count. I need a way to make sure its done without changing the process's code...
An early problem with reference counting is that it is relatively easy to count the initial reference by putting code in a custom malloc / free implementation, but it is quite a bit harder to determine if the initial recipient passes that address around to others.
Since C lacks the ability to override the assignment operator (to count the new reference), basically you are left with a limited number of options. The only one that can possibly override the assignment is macrodef, as it has the ability to rewrite the assignment into something that inlines the increment of the reference count value.
So you need to "expand" a macro that looks like
a = b;
into
if (b is a pointer) { // this might be optional, if lookupReference does this work
struct ref_record* ref_r = lookupReference(b);
if (ref_r) {
ref_r->count++;
} else {
// error
}
}
a = b;
The real trick will be in writing a macro that can identify the assignment, and insert the code cleanly without introducing other unwanted side-effects. Since macrodef is not a complete language, you might run into issues where the matching becomes impossible.
(jokes about seeing nails where you learn how to use a hammer have an interesting parallel here, except that when you only have a hammer, you had better learn how to make everything a nail).
Other options (perhaps more sane, perhaps not) is to keep track of all address values assigned by malloc, and then scan the program's stack and heap for matching addresses. If you match, you might have found a valid pointer, or you might have found a string with a luck encoding; however, if you don't match, you certainly can free the address; provided they aren't storing an address + offset calculated from the original address. (perhaps you can macrodef to detect such offsets, and add the offset as multiple addresses in the scan for the same block)
In the end, there isn't going to be a foolproof solution without building a referencing system, where you pass back references (pretend addresses); hiding the real addresses. The down side to such a solution is that you must use the library interface every time you want to deal with an address. This includes the "next" element in the array, etc. Not very C-like, but a pretty good approximation of what Java does with its references.
Semi-serious answer
#include "Python.h"
Python has a great reference counting memory manager. If I had to do this for real in production code, not homework, I'd consider embedding the python object system in my C program which would then make my C program scriptable in python too. See the Python C API documentation if you are interested!
Such a system in C requires some discipline on the part of the programmer but ...
You need to think in terms of ownership. All things that hold references are owners and must keep track of the objects to which it holds references, e.g. through lists. When a reference holding thing is destroyed it must loop its list of referred objects and decrement their reference counters and if zero destroy them in turn.
Functions are also owners and should keep track of referenced objects, e.g. by setting up a list at the start of the function and looping through it when returning.
So you need to determine in which situations objects should be transferred or shared with new owners and wrap the corresponding situations in macros/functions that add or remove owned objects to owning objects' lists of referenced objects (and adjust the reference counter accordingly).
Finally you need to deal with circular references somehow by checking for objects that are no longer reachable from objects/pointers on the stack. That could be done with some mark and sweep garbage collection mechanism.
I don't think you can do it automatically without overridable destructors/constructors.
You can look at HDF5 ref counting but those require explicit calls in C:
http://www.hdfgroup.org/HDF5/doc/RM/RM_H5I.html
I have a function and i'm accessing a struct's members a lot of times in it.
What I was wondering about is what is the good practice to go about this?
For example:
struct s
{
int x;
int y;
}
and I have allocated memory for 10 objects of that struct using malloc.
So, whenever I need to use only one of the object in a function, I usually create (or is passed as argument) pointer and point it to the required object (My superior told me to avoid array indexing because it adds a calculation when accessing any member of the struct)
But is this the right way? I understand that dereferencing is not as expensive as creating a copy, but what if I'm dereferencing a number of times (like 20 to 30) in the function.
Would it be better if i created temporary variables for the struct variables (only the ones I need, I certainly don't use all the members) and copy over the value and then set the actual struct's value before returning?
Also, is this unnecessary micro optimization? Please note that this is for embedded devices.
This is for an embedded system. So, I can't make any assumptions about what the compiler will do. I can't make any assumptions about word size, or the number of registers, or the cost of accessing off the stack, because you didn't tell me what the architecture is. I used to do embedded code on 8080s when they were new...
OK, so what to do?
Pick a real section of code and code it up. Code it up each of the different ways you have listed above. Compile it. Find the compiler option that forces it to print out the assembly code that is produced. Compile each piece of code with every different set of optimization options. Grab the reference manual for the processor and count the cycles used by each case.
Now you will have real data on which to base a decision. Real data is much better that the opinions of a million highly experience expert programmers. Sit down with your lead programmer and show him the code and the data. He may well show you better ways to code it. If so, recode it his way, compile it, and count the cycles used by his code. Show him how his way worked out.
At the very worst you will have spent a weekend learning something very important about the way your compiler works. You will have examined N ways to code things times M different sets of optimization options. You will have learned a lot about the instruction set of the machine. You will have learned how good, or bad, the compiler is. You will have had a chance to get to know your lead programmer better. And, you will have real data.
Real data is the kind of data that you must have to answer this question. With out that data nothing anyone tells you is anything but an ego based guess. Data answers the question.
Bob Pendleton
First of all, indexing an array is not very expensive (only like one operation more expensive than a pointer dereference, or sometimes none, depending on the situation).
Secondly, most compilers will perform what is called RVO or return value optimisation when returning structs by value. This is where the caller allocates space for the return value of the function it calls, and secretly passes the address of that memory to the function for it to use, and the effect is that no copies are made. It does this automatically, so
struct mystruct blah = func();
Only constructs one object, passes it to func for it to use transparently to the programmer, and no copying need be done.
What I do not know is if you assign an array index the return value of the function, like this:
someArray[0] = func();
will the compiler pass the address of someArray[0] and do RVO that way, or will it just not do that optimisation? You'll have to get a more experienced programmer to answer that. I would guess that the compiler is smart enough to do it though, but it's just a guess.
And yes, I would call it micro optimisation. But we're C programmers. And that's how we roll.
Generally, the case in which you want to make a copy of a passed struct in C is if you want to manipulate the data in place. That is to say, have your changes not be reflected in the struct it self but rather only in the return value. As for which is more expensive, it depends on a lot of things. Many of which change implementation to implementation so I would need more specific information to be more helpful. Though, I would expect, that in an embedded environment you memory is at a greater premium than your processing power. Really this reads like needless micro optimization, your compiler should handle it.
In this case creating temp variable on the stack will be faster. But if your structure is much bigger then you might be better with dereferencing.