I apologize in advance if this is an incredibly dumb question...
Currently I have a circular linked list. The number of nodes is normally held static. When I want to add to it, I malloc a number of nodes (ex. 100000 or so) and splice it in. This part works fine when I malloc the nodes one by one.
I want to attempt to allocate by blocks:
NODE *temp_node = node->next;
NODE *free_nodes = malloc( size_block * sizeof( NODE ) );
node->next = free_nodes;
for ( i = 0; i < size_block - 1; i++ ) {
free_nodes[i].src = 1;
free_nodes[i].dst = 0;
free_nodes[i].next = &free_nodes[i+1];
}
free_nodes[size_block - 1].next = temp_node;
The list works as long as I don't attempt to free anything ('glibc detected: double free or corruption' error). Intuitively, I think that is because freeing it doesn't free the single node, and looping through the normal way is attempting to free it multiple times (plus freeing the entire block probably screws up all the other pointers from the nodes that still exist?), but:
Could somebody please explain to me explicitly what is happening?
Is there a way to allocate the nodes by blocks and not break things?
The purpose of this is because I am calling malloc hundreds of thousands of times, and it would be nice if things were faster. If there is a better way around this, or I can't expect it to get faster, I would appreciate hearing that too. :)
Could somebody please explain to me explicitly what is happening?
Exactly what you said. You are allocating a single space of contiguous memory for all blocks. Then if you free it, all memory will be released.
Is there a way to allocate the nodes by blocks and not break things?
Allocate different memory segments for each block. In your code (that isn't complete) should be something like:
for ( i = 0; i < size_block ; i++ ) {
free_nodes[i] = malloc (sizeof( NODE ));
}
First, the way you allocated your nodes in blocks, you always have to free the whole block with exactly the same start address as you got from malloc. There is no way around this, malloc is designed like this.
Putting up your own ways around this is complicated and usually not worth it. Modern run-times have quite efficient garbage collection behind malloc/free (for its buffers, not for user allocations) and it will be hard for you to achieve something better, better meaning more efficient but still guaranteeing the consistency of your data.
Before losing yourself in such a project measure where the real bottlenecks of your program are. If the allocation part is a problem there is still another possibility that is more likely to be the cause, namely bad design. If you are using so many elements in your linked list such that allocation dominates, probably a linked list is just not the appropriate data structure. Think of using an array with a moving cursor or something like that.
When you free a node you free the entire allocation that the node was allocated with. You must somehow arrange to free the entire group of nodes at once.
Probably your best bet is to keep a list of "free" nodes and reuse those rather than allocating/freeing each node. And with some effort you can arrange to keep the nodes in blocks and allocate from the "most used" block first such that if an entire block goes empty you can free it.
Related
gSoap usefully creates stubs to help with memory management etc. One of these
commands is soap_malloc but there doesn't seem to be a corresponding soap_realloc.
Before I start to write my own push and pop methods I just want to ensure I'm not missing anything obvious.
//example.h generated with wsdl2h
struct ns1___Customer
{
int __sizeProduct;
ns1__Product *Product;
int customerid;
}
struct ns1__Product
{
int productid;
}
I am currently using soap_malloc and then realloc for dynamically growing the array.
//I could use soap_new_ns1__Product(&soap,XXX) and allocate mem for XXX
//number of ns1__Product structures but this is wasteful and doesn't solve
//anything
struct ns1__Customter cust;
soap_default_ns1__Product(soap, &cust);
struct ns1__Product *prod_array = NULL;
//allocate mem for 1 product
prod_array = soap_new_ns1__Product(soap,1) ;
soap_default_ns1__Product(soap, &prod_array[0]);
prod_array[0].productid=111;
//Need to add product therefore need to realloc mem.
//IS THIS THE BEST WAY IN gsoap?
prod_array = realloc( prod_array, 2 * sizeof(struct ns1__Product)) ;
soap_default_ns1__Product(soap, &emp_array[1]);
prod_array[1].product=222;
//assigning array ptr to Customer
cust.Product=prod_array;
// Remember to adjust sizeProduct
cust.__sizeProduct=2;
This seems wrong and clumsy, does gsoap suggest a better way? I can't find a clear example in the documentation or by searching online.
Before I start to write my own push and pop methods I just want to ensure I'm not missing anything obvious.
I suspect you're missing that soap_malloc() allocates memory that is automatically freed under at least some circumstances. As such, using realloc() to resize the allocated memory is begging for trouble. There's a fair chance that the reallocation as such will succeed, but you're likely at minimum to end up with a nasty mess when gSOAP's automatic freeing tries to kick in in soap_end().
On the other hand, I don't think you're overlooking any reallocation function. The docs indeed do not seem to describe any. You can always implement your own reallocation wrapper that allocates fresh memory with soap_malloc(), copies the contents of the original space (whose size you'll need to know somehow), and releases the original space with soap_dealloc().
The bottom line appears to be that soap_malloc() is not intended to be a general-purpose allocator, and it is not particularly well suited to your use case. It's primary objective appears to be internal, to relieve library users of any need to manually free the temporary objects that the library allocates. I take exposing it to library users for their direct use to be intended as a convenience.
If you want the ability to reallocate blocks, then I suggest you obtain them in the first place via regular malloc(). You'll want to read the docs carefully if you're going to mix malloc()ed data with soap_malloc()ed data, but it is likely possible. Alternatively, consider approaches that do not require reallocation, such as storing your data in a linked list instead of a dynamic array.
What you're doing there is indeed wrong. After using soap_new_T() (where T in your case is ns1__Product), the soap context now manages that memory by holding onto the ns1__Product* pointer internally. Later, when you call soap_destroy() to free all soap_new_T()-allocated objects managed by the soap context, the context will be trying to free a pointer that no longer points to valid memory since you called realloc().
As John Bollinger pointed out, there's no built-in way in gSOAP to do something similar to a realloc. You'd instead just need to do the reallocation manually, e.g.:
// allocate memory for 1 product (as you already do above)
prod_array = soap_new_ns1__Product(soap, 1);
// ... do some stuff, realize you need to "realloc" to add another product ...
// allocate a new array, managed by the soap context
struct ns1__Product* new_array = soap_new_ns1__Product(soap, 2);
// copy the old array into the new one (assuming old_size is however many elements are in prod_array)
for(std::size_t i = 0; i < old_size; ++i)
{
new_array[i] = prod_array[i];
}
// tell the soap context to destroy the old array
soap_dealloc(soap, prod_array);
Aside:
It seems like it should be possible to use an std::vector<ns1__Product> rather than an array, which would solve your problem in an arguably better manner, but that question was already asked here to no avail. Unfortunately I don't know the answer to that at this time.
I have the following linked list that has several nodes which have been attached to it. Is there away to free all the notes at once instead of iterating through each node?
This is the struct I have:
struct Courses{
char *courseName;
int creditValue;
Courses *next;
};Courses;
You can write your own, intermediate allocator that allocates a big block of (say) 1000 node structures. They you can build your list by "allocating" nodes out of your big chunk one at a time, with a simple variable to keep track of how many you've used, and some code to catch the case that you've used them all up.
Then when it's time to free your list, you can just free that one block in one go.
In your example, you may also have to worry about memory dynamically allocated for the courseName pointer. You can handle that, too, although your intermediate allocator gets more complicated, because it ends up being more of a general-purpose malloc replacement, not just a special-purpose Courses node allocator.
No. There's no way other than iterating over the list.
On modern operating systems, allocated memory will be reclaimed by the operating system once the process exits. So you can avoid free()'ing yourself. But this is not a good alternative especially if the program is a long running one or nor is it a universal approach to free memory.
My question is regarding when using free() is appropriate in C. I'm using gcc 4.3.2.
Suppose, if had to deallocate a bunch of memory, in a linked list, the ideal way to go about it is(I'm guessing):
int freeLL(node *a)
{
if(a->next != NULL)
freeLL(a->next);
free(a);
return 0;
}
Now, suppose I were to do a similar thing on the following ADT:
A pointer "VertexNode" which has 2 more pointers: "Vertex" and "Edge"(say). Equivalent to saying:
struct vertexnode
{
vertex *v;
edge *e;
}
typedef struct vertexnode* VertexNode;
Later, when initializing an instance, I'll do something like -
VertexNode V = malloc(sizeof(struct vertexnode));
V->v = malloc(sizeof(vertex));
So, ultimately while freeing: I used the same analogy as I used for the linked list.
free(V->v);
free(V);
This gave a runtime error, and when I commented out "free(V->v)", the program worked fine.
My questions are:
a) Is simply doing free(V) sufficient? I mean, does free() work recursively on all pointers inside a given pointer?
b) If not, is there a memory leak in this case? And how do I ideally prevent that?
c) Lastly, is there a way I can keep track of how many bytes were allocated by malloc() and
how many of them were freed by free()?
I'm very sorry for the long question. Thank you in advance for your time and patience.
No, free does not work recursively, therefore you would indeed have memory leaks. The runtime error you are occuring is probably realated to a logic error (maybe V->v is NULL or you did not alloc it before freeing).
If you are working with linux, using valgrind can help you profiling your program and mention leak errors. Compile using cc *.c -ggdb and then run valgrind --leakcheck=full ./a.out will output leak errors.
To answer your question:
a) No. free() doesn't recursively free struct's member pointers.
b) Yes, there is memory leak in that case. You have to free all allocated memory with your code.
c) You may use tool to check memory leak, such as valgrind, that's easy. And I know some projects implements their own memory management, they wrap malloc and free in their own APIs, so that you can track memory usage in their APIs.
For a complete memory management I would to have a memory pool. This would take away all the pain of having to call a number of free's for each malloc that is called. I use Apache Portable runtime (APR) for doing memory pooling. Just allocate a chunk of memory at the start to initialize. The allocate as much as you want for each pointer. Then at the end just make one call to free all the memory. This is much more efficient then doing lots of mallocs and frees that leads to memory leaks.
As a side note. I suggest you use valgrind to test you application if you don't use the memory pool. In fact you should always use valgrind.
The thing is: you need to make sure that the pointer you're trying to free is actually initialized.
I'd also check whether V->v is not NULL before calling free on it (see my comment on your question).
free is not "recursive". You're leaking memory if you don't free e and v (in your example) as well.
A: free() does not work recursively.
- If you're on the ADT concept, then I suggest you perform the allocation inside the type by creating a createVertexNode() function, and do the checking and deallocation inside a freeVertexNode() function
B: If you wasn't able to free it, then it will be a memory leak
- you can avoid it by making sure your ADT functions check and free the memory it allocates, or ...
C: Use a memory leak checker. Visual Studio has a built-in one, or use other such as valgrind or Rational Purify. I'm sure there are more free open source libraries that, at the simplest, override the malloc() and free() calls
Basically the thing to remember is calls to malloc()/free() should be 1:1. If I call malloc() to get some memory, I need to call free() to return it when I'm done.
This answers question a), no it's not recursive. (or else one call to free() would be good enough).
b) You bet there'd be a memory leak! Think about some simple code:
typedef struct pointless {
struct pointless * next;
}pl;
pl * head = malloc(sizeof(pl)); // head gets 4 bytes
pl->next = malloc(sizeof(pl)); // head->next gets 4 bytes
free(head); // head's allocated 4 bytes are deleted
Obviously this is a pointless list, since it just points to the next empty element, but it illustrates the point. My total list has 8 bytes allocated to it (4 for head, 4 for head's next). When I call free() it works on head's 4 bytes, but that's it.
This is a good thing it works this way! Think about deleting a single item from the middle of a linked list, you want to free the memory for that node, but not the whole thing! However given my above example, you'd have a memory leak as head->next wasn't free'd; and once you call free(head) there's no guarantee that you can access head->next anymore. It's possible of course... but UB at this point.
c) memory management at that level is done by the OS. If you want to keep track of how much you allocated/freed you'd have to do something like:
int total_mem = 0;
head = malloc((total_mem += sizeof(pl)));
free(head);
total_mem -= sizeof(head);
Of course you could put wrappers around the malloc() and free() calls to do the math for you which would be much more manageable, but you get the idea.
I'm writing a malloc function for a college assignment. Here's a basic layout of my idea:
1)Define a node struct with pointers to previous node, next node, as well as a char for size and vacancy. Each region in the heap will contain a hidden node with this information.
2)Malloc function. Starting with first node loop through each node checking for vacancy. If a node is vacant and is large enough return a ptr to the beginning of the region not including the node. If no space is available use sbrk to allocate requested space PLUS space for a node.
3)Free function. Go to pointer passed as parameter-sizeof(struct node) and set the vacancy to vacant. Then starting with the beginning of the list, traverse the list merging adjacent free spaces.
How does this approach sound? My main concern is with actually starting the linked list. For instance, should I create a node with sbrk before I start to do any allocations and store a ptr to it as a global variable? If so how do I initialize a first node before I allow the malloc function to be called by a driver program?
Thanks in advance. I'm not asking for someone to write my code, only to provide some insight and suggestions regarding my ideas.
I would avoid keeping all the bookkeeping information on nodes while they're allocated. I'd have the bare minimum of information (usually just the block size) at the beginning of the block, but nothing more.
I'd track free blocks and allocated blocks separately, so when you're searching for a free block, you don't waste time on blocks that are already in use.
I'd separate the free list into two pieces, and coalesce blocks lazily. In other words, have one free list you're allocating from, and a second that's just a holding area. When the user calls free, just link the block into the holding area, nothing more. When the list you're using for allocations starts to run low, sort the blocks in the holding area by address, then merge with the allocation free list. Then walk the list and merge adjacent blocks.
When you do need to call sbrk (or whatever) to allocate more space from the system, do not just allocate enough space to satisfy the current allocation request. Instead, allocate a fairly large block (e.g., a megabyte) and then split that to get satisfy the request, and add the rest as block to the free list. If you're running low enough on memory that you have to go to sbrk once, chances are the next few calls will do the same, so you might as well be greedy, and grab enough memory immediately to stand a decent chance of satisfying more requests as well.
The basic idea of the third is to avoid doing coalescing as long as possible to increase the chances of finding adjacent blocks, so when you do coalesce you'll probably do some real good, and avoid wasting time trying to coalesce when there are only a few adjacent blocks free.
It's difficult to tell what is being asked here. This question is ambiguous, vague, incomplete, overly broad, or rhetorical and cannot be reasonably answered in its current form. For help clarifying this question so that it can be reopened, visit the help center.
Closed 11 years ago.
A chapter out of the book I have been reading has focused on memory management allocating space using malloc linux functions.
Before I read this I would make relatively small programs without allocating space.
Is it acceptable to not do anything in the way of memory allocation for applications whose memory footprint remains under 50MB? What are the repercussions of not doing so?
I think the answers are missing an important point. The size of memory is a relatively specific technical detail which isn't of primary interest. The crucial difference is that between automatic and dynamic storage, and the associated lifetime:
Automatic storage ends at the end of the scope.
Dynamic storage begins with malloc() and ends with free(), entirely at the discretion (and responsibility) of the user.
If you can and if it makes sense, everything should be automatic. This entails locality and well-defined interfaces. However, in C (not so much in C++) there comes a time when you need to talk about objects that aren't local to the scope. That's when we need dynamic allocation.
The prime example is your typical linked list. The list consists of nodes:
typedef struct node_tmp
{
int data;
struct node_tmp * next;
struct node_tmp * prev;
} node;
Now to talk about such a list boils down to talking about any of its nodes and brachiate along the prev/next pointers. However, the actual nodes cannot sensibly be part of any local scope, so they are usually dynamically allocated:
node * create_list()
{
node * p = malloc(sizeof node); // [1]
p->prev = p->next = 0;
return p;
}
void free_list(node * p) // call with head node
{
while (p->next)
{
node * tmp = p;
p = p->next;
free(tmp); // [2a]
}
free(p); // [2b]
}
void append_to_end(node * p, int data); // etc.
Here the list nodes exist outside any scope, and you have to bring them to life manually using malloc(), and clean them up when you're done.
You can use linked lists even in the tiniest of programs, but there's no real way around the manual allocation.
Edit: I thought of another example that should really convince you: You might think that you can just make the list with automatically allocated nodes:
node n1, n2, n3; // an automatic linked list
n1.prev = n3.next = 0;
n1.next = &n2; n2.prev = &n1; n2.next = &n3; n3.prev = &n2;
But note that you cannot do this dynamically! "Dynamic" means "at runtime", but automatic variables have to be determined entirely at compile time.
Suppose you wanted a program that reads integers from the user. If it's even, you add it to the list, if it's odd you ignore it, and if it's zero you stop. You cannot possibly realize such a program with automatic allocation, because the allocation needs are only determined at runtime.
It is in such a scenario that you require malloc().
If you can do without malloc for small applications, you're probably just not needing to use any heap space. Little utility programs or toy programs often don't. The things you might be doing wrong though to get by when you should be using the heap are:
Arrays. If you find yourself allocating large arrays 'just to make sure everything fits' then you should perhaps be using malloc. At the least, handle the error condition that everything overflows to check they really are big enough. With dynamically allocated arrays, you can make bigger ones on the fly if you find you need more space.
Doing too much recursion. C benefits from flattening out recursion sometimes into loops over arrays, because unlike function languages it can't optimise things properly. If you are getting your storage space by calling function lots to create it, that's pretty dangerous (the program might crash on you one day).
Using static pools of objects (structs, classes). Perhaps you have a ring buffer, and 15 objects that could be in it, and you have them statically allocated because you know that your buffer will never have more than 15 entries. That's kind of OK, but allowing the buffer to grow more by adding in more structs, created with malloc, might be nice.
Probably plenty more situations where programmes which don't need malloc could benefit from having it added.
The size of an application and the use of malloc() are two independant things. malloc() is used to allocate memory at runtime, when sizes are not known at compilation time.
Anyway, if you do know the maximum size of the structures that you want to play with, you can statically allocate them and build an application without using malloc(). Space critical software is an example of such applications.
You were probably allocating memory statically at compile time, but not dynamically.
The possible issue when allocating everything statically are :
you are wasting memory because you are always allocating an upper limit with a margin.
in some case, your application will run out of memory (because your estimation was wrong for example), and and since you can not add new memory ressources at runtime it can potentially be lethal.
That being said, in some cases like real-time embedded system, it is a requirement to not allocate any memory dynamically at runtime. (because you have hard memory constraints, or because allocating memory can break real time)