I've written an API that requires a context to be initialized and thereafter passed into every API call. The caller allocates the memory for the context, and then passes it to the init function with other parameters that describe how they want later API calls to behave. The context is opaque, so the client can't really muck around in there; it's only intended for the internal use of the API functions.
The problem I'm running into is that callers are allocating the context, but not initializing it. As a result, subsequent API functions are referring to meaningless garbage as if it was a real context.
I'm looking for a way to verify that the context passed into an API function has actually been initialized. I'm not sure if this is possible. Two ideas I've thought of are:
Use a pre-defined constant and store it in a "magic" field of the context to be verified at API invocation time.
Use a checksum of the contents of the context, storing this in the "magic" field and verifying it at invocation time.
Unfortunately I know that either one of these options could result in a false positive verification, either because random crap in memory matches the "magic" number, or because the context happens to occupy the same space as a previously initialized context. I think the latter scenario is more likely.
Does this simply boil down to a question of probability? That I can avoid false positives in most cases, but not all? Is it worth using a system that merely gives me a reasonable probability of accuracy, or would this just make debugging other problems more difficult?
Best solution, I think, is add create()/delete() functions to your API and use create to allocate and initialize the structure. You can put a signature at the start of the structure to verify that the pointer you are passed points to memory allocated with create() and use delete() to overwrite the signature (or entire buffer) before freeing the memory.
You can't actually avoid false positives in C because the caller malloc'd memory that "happened" to start with your signature; but make you signature reasonably long (say 8 bytes) and the odds are low. Taking allocation out of the hands of the caller by providing a create() function will go a long way, though.
And, yeah, your biggest risk is that an initialized buffer is free'd without using delete(), and a subsequent malloc happens to reuse that memory block.
Your context variable is probably at the moment some kind of pointer to allocated memory. Instead of this, make it a token or handle that can be explicitly verified. Every time a context is initialised, you return a new token (not the actual context object) and store that token in an internal list. Then, when a client gives you a context later on, you check it is valid by looking in the list. If it is, the token can then be converted to the actual context and used, otherwise an error is returned.
typedef Context long;
typedef std::map<Context, InternalContext> Contexts;
Contexts _contexts;
Context nextContext()
{
static Context next=0;
return next++;
}
Context initialise()
{
Context c=nextContext();
_contexts.insert(make_pair(c, new InternalContext));
return c;
}
void doSomethingWithContext(Context c)
{
Contexts::iterator it=_ _contexts.find(c);
if (it==_contexts.end())
throw "invalid context";
// otherwise do stuff with the valid context variable
InternalContext *internalContext=*it.second;
}
With this method, there is no risk of an invalid memory access as you will only correctly use valid context references.
Look at the paper by Matt Bishop on Robust Programming. The use of tickets or tokens (similar to file handles in some respects, but also including a nonce - number used once) allows your library code ensure that the token it is using is valid. In fact, you allocate the data structure on behalf of the user, and pass back to the user a ticket which must be provided for each call to the API you define.
I have some code based closely on that system. The header includes the comments:
/*
** Based on the tickets in qlib.c by Matt Bishop (bishop#ucdavis.edu) in
** Robust Programming. Google terms: Bishop Robust Nonce.
** http://nob.cs.ucdavis.edu/~bishop/secprog/robust.pdf
** http://nob.cs.ucdavis.edu/classes/ecs153-1998-04/robust.html
*/
I also built an arena-based memory allocation system using tickets to identify different arenas.
You could define a new API call that takes uninitialised memory and initialises it in whatever way you need. Then, part of the client API is that the client must call the context initialisation function, otherwise undefined behaviour will result.
To sidestep the issue of a memory location of a previous context being reused, you could, in addition to freeing the context, reset it and remove the "magic" number, assuming of course that the user frees the context using your API. That way when the system returns that same block of memory for the next context request, the magic number check will fail.
see what your system does with uninitialzed menmory. m$ does: Uninitialized memory blocks in VC++
Related
I am updating a piece of Tk-based third-party software for use with Tcl/Tk 8.6, and I have run across a statement of this form:
interp->result = Tk_PathName(tkwin);
, where interp is a Tcl_Interp * and tkwin is a Tk_Window. It no longer being allowed (by default) in Tcl 8.6 to access the members of a Tcl_Interp directly, I want to convert that to a call to Tcl_SetResult(), along these lines:
Tcl_SetResult(interp, Tk_PathName(tkwin), /* what goes here? */);
But I'm having trouble finding any documentation of the lifetime of the string returned by Tk_PathName(), and I need to know that to specify the correct free function.
I suspect that the right thing to do would be to specify TCL_VOLATILE as the free function, so that Tcl makes and subsequently manages its own copy of the string, but that will produce a memory leak if the caller is responsible for freeing the string returned by Tk_PathName().
If the caller has responsibility to free, then I suppose that TCL_DYNAMIC should be specified, though that does assume (I think reasonably) that Tk would have allocated the path name via Tcl_Alloc.
So which is it? Or do I need something else, instead?
These days, Tcl always immediately copies the string you pass into Tcl_SetResult(), regardless of the third argument, so that it can adopt the string into the managed Tcl_Obj infrastructure it uses internally. TCL_VOLATILE would be right… except that it simply doesn't matter any more!
There was a point where Tcl internally tried to support both ways of doing things; the code to support it was horrible and weird and it was genuinely hard to be sure by inspection that it was bug free so we switched to early copy, which was absolutely definitely the right semantics and obviously so. It probably also only meant that the copy was brought forward a tiny bit anyway, since nothing in the rest of Tcl could make use of non-Tcl_Obj results at that point.
That said, the lifetime of the string result from Tk_PathName is the lifetime of the widget and always has been for as far back as I've used Tk; the string is allocated when the widget is created, stored inside the widget's internal data structure, and deleted when the widget is destroyed. As such, TCL_STATIC is another candidate if you're not about to destroy the widget.
I'm writing generic data structure implementations on C for learning purposes (using void pointers), however I'm a bit confused about the role of the "free function" that virtually all generic implementations allow the user to pass to the initializer function.
Let's say I have a stack. Should I call the user provided free function when the client calls the "pop" operation, for instance? From one side, if we free the element after accessing it, and before returning it to the caller, by the time the caller receives the value accessing that memory will be undefined behaviour, however if we don't delete it, the user will be responsible for doing so, defeating the purpose of passing a custom free function in the first place.
What is the best practice here?
Let's say I have a stack. Should I call the user provided free
function when the client calls the "pop" operation, for instance?
No, you should not free the element when it is pop'ed from the stack. The user expects to get the object back that he earlier push'ed on the stack. And then it must be valid, of course.
During a pop operation you only should free your own data that you needed to store the user's object.
I'm a bit confused about the role of the "free function" that
virtually all generic implementations allow the user to pass to the
initializer function.
The custom free functions I am aware of, are only used when the whole management data structure is destroyed. Then in addition to your management data like list nodes etc., the user data also has to be freed.
As you do not know if there are dynamically allocated objects enclosed in that user data which require some additional handling, you can only leave this at user's hand.
For this purpose the user has to pass a free function.
you always need to know how to destroy externally created objects to avoid memory leaks.
For your stack example, what happens if a user pushes an object and then destroys the stack? The stack destructor should call the destructors of all the elements it still contains.
To be safe, you should always have a function dedicated for freeing everything you malloc-ed during your program. You can do it at the end, or just when you're sure you won't ever need it after that.
I'm writing a library which allows to build Bayesian nets. The structure of a net is encapsulated and the user cannot access its fields; however they can get and set some of them. Say you want to write the accessor function for the field table (which is basically a double array) for instance. Between the following options, which would be more appropriate?
First option:
int getTable(Net *net, double *res)
Second option:
double *getTable(Net *net)
In the first option, the user provides a pointer to the array where the table values will be written. The function copies the table values on res, leaving the user with no direct access to the internal structure. Modifying res leaves the table of the net unchanged. Of course, another function (say getTableSize()) is provided to correctly allocate res. This seems safe (the internal structure stay coherent) and has the advantage that you can return a code value if something goes wrong. The downside is that this approach can be slower than the next one, since it involves a copy. Typically, the number of values of table may vary from just 1 to a couple of hundreds.
In the second option, the function returns directly the pointer to the internal values. In the doc, I will specify that user cannot try to free the pointer or modify the values. Freeing the pointer would likely result in a segmentation fault and memory leaks if other operations on the net are performed. Modifying the table won't result in any apparent error, but the internal coherence would be broken and the results of subsequent calculations might be very wrong and very hard to debug for the user.
Which option do you prefer? There other stuff to consider? Is there another approach to prefer?
Personally I would go for the first option because of the ability to return an error code. This would also solve your problem regarding the user freeing wrongly the returned value. And passing a pointer on a variable declared on the stack is easy.
Note that you can also make things more clear, i.e. the returned value cannot be modified or freed, with the second option by returning a const pointer like this
const double * getTable(Net *net);
In such way, the caller cannot modify the value, unless he cast it to a non const but I think this would be going a bit too far since the caller break intentionally your interface.
More info on constness can be found on wikipedia
I think a good habit is to always demand a return code from functions that can fail for some reason.
Error handling is much more efficient when working with return codes.
I'd go with option one.
Also, I don't know if this is a mistake or not but option two returns a double pointer - if this is the correct behavior, then function one should have the signature:
int getTable(Net *net, double **res)
Additionally, as Eugene Sh. mentioned in the comments, some environments might not even support malloc (some embedded devices firmware comes to mind), so giving the user a choice whether to pass in a malloc'd variable or a stack allocated variable is also a good selling point for option one.
I have two points for you to consider:
You want memory deallocation to be in the same place where allocation is. If you allocate memory in a function and return pointer to it, then the caller of the function has to know how to free it and has to remember to free it. Maintaining that kind of code would be a nightmare.
Premature optimization is the root of all evil. Do not optimize your code until you are certain (i.e. you measured) that exactly that part of code is causing problems.
That said, first option is the only option.
In some functions (such as *scanf variants) there is a argument that takes a memory space for the result. You could also write the code where it returns an address. What are the advantages, why design the function in such a weird way?
Example
void process_settings(char* data)
{
.... // open file and put the contents in the data memory
return;
}
vs
char* process_settings()
{
char* data = malloc(some_size);
.... // open file and load it into data memory
return data;
}
The benefit is that you can reserve the return value of the function for error checking, status indicators, etc, and actually send back data using the output parameter. In fact, with this pattern, you can send back any amount of data along with the return value, which could be immensely useful. And, of course, with multiple calls to the function (for example, calling scanf in a loop to validate user input), you don't have to malloc every time.
One of the best examples of this pattern being used effectively is the function strtol, which converts a string to a long.
The function accepts a pointer to a character as one of its parameters. It's common to declare this char locally as endptr and pass in its address to the function. The function will return the converted number if it was able to, but if not, it'll return 0 to indicate failure but also set the character pointer passed in to the non-digit character it encountered that caused the failure.
You can then report that the conversion failed on that particular character.
This is better design than using global error indicators; consider multithreaded programs. It likely isn't reasonable to use global error indicators if you'll be calling functions that could fail in several threads.
You mention that a function should be responsible for its own memory. Well, scanf doesn't exist to create the memory to store the scanned value. It exists to scan a value from an input buffer. The responsibilities of that function are very clear and don't include allocating the space.
It's also not unreasonable to return a malloc'd pointer. The programmer should be prudent, though, and free the returned pointer when they're done using it.
The decision of using one method instead of another depends on what you intend to do.
Example
If you want to modify an array inside a function an maintain the modification in the original array, you should use your first example.
If you are creating your own data structure, you have to deal with all the operations. And if you want to create a new struct you should allocate memory inside the function and return the pointer. The second example.
If you want to "return" two values from a function, like a vector and the length of the vector, and you don't want to create a struct for this, you could return the pointer of the vector and pass an int pointer as an argument of the function. That way you could modify the value of the int inside the function and you use it outside too.
char* return_vector_and_length(int* length);
Let’s say that, for example, you wanted to store process settings in a specific place in memory. With the first version, you can write this as, process_settings(output_buffer + offset);. How would you have to do it in you only had the second version? What would happen to performance if it were a really big array? Or what if, let’s say, you’re writing a multithreaded application where having all the threads call malloc() all the time would make them fight over the heap and serialize the program, so you want to preallocate all your buffers?
Your intuition is correct in some cases, though: on modern OSes that can memory-map files, it does turn out to be more efficient to return a pointer to the file contents than the way the standard library was historically written, and this is how glib does it. Sometimes allocating all your buffers on the heap helps avoid buffer overflows that smash the stack.
An important point is that, if you have the first version, you can trivially get the second one by calling malloc and then passing the buffer as the dest argument. But, if you have only the second, you can’t implement the first without copying the whole array.
In a language like C, for example, if a routine receives a pointer, is there any system call or other test that can be applied to the pointer to check that it is a valid memory location, other than catching SIGSEGV or equivalent?
No, you can't for sure check whether the address is invalid. Even if you used some operating system function to test if teh address is mapped into the address space you still can't be sure if the address is of some service data that you should not read or modify.
One good example. If your program uses Microsoft RPC to accept calls from another program you have to implement a set of callback functions to server the requests. Those callback functions will be run on separated threads started by RPC. You don't know when those thereads start and what their stack size is, so you can't detect whether a buffer overrun occurs if you write through an address that is meant to be of a stack variable but accidentially is to the stack of another thread.
Well, if you knew where the memory being pointed to was being stored (on the stack, for instance), you could check to see if it's in a certain 'range' that is the approximate address range of the stack. That could also work for something on the heap, if you have an idea of how big your heap "should" be. It's definitely not a fail-safe approach, but I'm unaware of any sure-fire methods for checking the 'validity' of a pointer.
If you mean purely within your own application you can establish a convention that any memory allocated by your code is initialized in a way you can recognize. E.g. in one project I saw they wrote an eyecatcher in the first few bytes. In some products I know they write a unique id at the start and end and each time it's accessed they check the 2 ids still match to show it's not been corrupted. E.g CICS on z/Series does the latter.