You may think that this is a coincidence that the topic of my question is similar to the name of the forum but I actually got here by googling the term "stack overflow".
I use the OPNET network simulator in which I program using C. I think I am having a problem with big array sizes. It seems that I am hitting some sort of memory allocation limitation. It may have to do with OPNET, Windows, my laptop memory or most likely C language. The problem is caused when I try to use nested arrays with a total number of elements coming to several thousand integers. I think I am exceeding an overall memory allocation limit and I am wondering if there is a way to increase this cap.
Here's the exact problem description:
I basically have a routing table. Let's call it routing_tbl[n], meaning I am supporting 30 nodes (routers). Now, for each node in this table, I keep info. about many (hundreds) available paths, in an array called paths[p]. Again, for each path in this array, I keep the list of nodes that belong to it in an array called hops[h]. So, I am using at least nph integers worth of memory but this table contains other information as well. In the same function, I am also using another nested array that consumes almost 40,000 integers as well.
As soon as I run my simulation, it quits complaining about stack overflow. It works when I reduce the total size of the routing table.
What do you think causes the problem and how can it be solved?
Much appreciated
Ali
It may help if you post some code. Edit the question to include the problem function and the error.
Meanwhile, here's a very generic answer:
The two principal causes of a stack overflow are 1) a recursive function, or 2) the allocation of a large number of local variables.
Recursion
if your function calls itself, like this:
int recurse(int number) {
return (recurse(number));
}
Since local variables and function arguments are stored on the stack, then it will in fill the stack and cause a stack overflow.
Large local variables
If you try to allocate a large array of local variables then you can overflow the stack in one easy go. A function like this may cause the issue:
void hugeStack (void) {
unsigned long long reallyBig[100000000][1000000000];
...
}
There is quite a detailed answer to this similar question.
Somehow you are using a lot of stack. Possible causes include that you're creating the routing table on the stack, you're passing it on the stack, or else you're generating lots of calls (eg by recursively processing the whole thing).
In the first two cases you should create it on the heap and pass around a pointer to it. In the third case you'll need to rewrite your algorithm in an iterative form.
Stack overflows can happen in C when the number of embedded recursive calls is too high. Perhaps you are calling a function from itself too many times?
This error may also be due to allocating too much memory in static declarations. You can switch to dynamic allocations through malloc() to fix this type of problem.
Is there a reason why you cannot use the debugger on this program?
It depends on where you have declared the variable.
A local variable (i.e. one declared on the stack is limited by the maximum frame size) This is a limit of the compiler you are using (and can usually be adjusted with compiler flags).
A dynamically allocated object (i.e. one that is on the heap) is limited by the amount of available memory. This is a property of the OS (and can technically by larger the physical memory if you have a smart OS).
Many operating systems dynamically expand the stack as you use more of it. When you start writing to a memory address that's just beyond the stack, the OS assumes your stack has just grown a bit more and allocates it an extra page (usually 4096Kib on x86 - exactly 1024 ints).
The problem is, on the x86 (and some other architectures) the stack grows downwards but C arrays grow upwards. This means if you access the start of a large array, you'll be accessing memory that's more than a page away from the edge of the stack.
If you initialise your array to 0 starting from the end of the array (that's right, make a for loop to do it), the errors might go away. If they do, this is indeed the problem.
You might be able to find some OS API functions to force stack allocation, or compiler pragmas/flags. I'm not sure about how this can be done portably, except of course for using malloc() and free()!
You are unlikely to run into a stack overflow with unthreaded compiled C unless you do something particularly egregious like have runaway recursion or a cosmic memory leak. However, your simulator probably has a threading package which will impose stack size limits. When you start a new thread it will allocate a chunk of memory for the stack for that thread. Likely, there is a parameter you can set somewhere that establishes the the default stack size, or there may be a way to grow the stack dynamically. For example, pthreads has a function pthread_attr_setstacksize() which you call prior to starting a new thread to set its size. Your simulator may or may not be using pthreads. Consult your simulator reference documentation.
Related
Like we do with macros:
#undef SOMEMACRO
Can we also undeclare or delete the variables in C, so that we can save a lot of memory?
I know about malloc() and free(), but I want to delete the variables completely so that if I use printf("%d", a); I should get error
test.c:4:14: error: ‘a’ undeclared (first use in this function)
No, but you can create small minimum scopes to achieve this since all scope local variables are destroyed when the scope is exit. Something like this:
void foo() {
// some codes
// ...
{ // create an extra minimum scope where a is needed
int a;
}
// a doesn't exist here
}
It's not a direct answer to the question, but it might bring some order and understanding on why this question has no proper answer and why "deleting" variables is impossible in C.
Point #1 What are variables?
Variables are a way for a programmer to assign a name to a memory space. This is important, because this means that a variable doesn't have to occupy any actual space! As long as the compiler has a way to keep track of the memory in question, a defined variable could be translated in many ways to occupy no space at all.
Consider: const int i = 10; A compiler could easily choose to substitute all instances of i into an immediate value. i would occupy 0 data memory in this case (depending on architecture it could increase code size). Alternatively, the compiler could store the value in a register and again, no stack nor heap space will be used. There's no point in "undefining" a label that exists mostly in the code and not necessarily in runtime.
Point #2 Where are variables stored?
After point #1 you already understand that this is not an easy question to answer as the compiler could do anything it wants without breaking your logic, but generally speaking, variables are stored on the stack. How the stack works is quite important for your question.
When a function is being called the machine takes the current location of the CPU's instruction pointer and the current stack pointer and pushes them into the stack, replacing the stack pointer to the next location on stack. It then jumps into the code of the function being called.
That function knows how many variables it has and how much space they need, so it moves the frame pointer to capture a frame that could occupy all the function's variables and then just uses stack. To simplify things, the function captures enough space for all it's variables right from the start and each variable has a well defined offset from the beginning of the function's stack frame*. The variables are also stored one after the other.
While you could manipulate the frame pointer after this action, it'll be too costly and mostly pointless - The running code only uses the last stack frame and could occupy all remaining stack if needed (stack is allocated at thread start) so "releasing" variables gives little benefit. Releasing a variable from the middle of the stack frame would require a defrag operation which would be very CPU costly and pointless to recover few bytes of memory.
Point #3: Let the compiler do its job
The last issue here is the simple fact that a compiler could do a much better job at optimizing your program than you probably could. Given the need, the compiler could detect variable scopes and overlap memory which can't be accessed simultaneously to reduce the programs memory consumption (-O3 compile flag).
There's no need for you to "release" variables since the compiler could do that without your knowledge anyway.
This is to complement all said before me about the variables being too small to matter and the fact that there's no mechanism to achieve what you asked.
* Languages that support dynamic-sized arrays could alter the stack frame to allocate space for that array only after the size of the array was calculated.
There is no way to do that in C nor in the vast majority of programming languages, certainly in all programming languages that I know.
And you would not save "a lot of memory". The amount of memory you would save if you did such a thing would be minuscule. Tiny. Not worth talking about.
The mechanism that would facilitate the purging of variables in such a way would probably occupy more memory than the variables you would purge.
The invocation of the code that would reclaim the code of individual variables would also occupy more space than the variables themselves.
So if there was a magic method purge() that purges variables, not only the implementation of purge() would be larger than any amount of memory you would ever hope to reclaim by purging variables in your program, but also, in int a; purge(a); the call to purge() would occupy more space than a itself.
That's because the variables that you are talking about are very small. The printf("%d", a); example that you provided shows that you are thinking of somehow reclaiming the memory occupied by individual int variables. Even if there was a way to do that, you would be saving something of the order of 4 bytes. The total amount of memory occupied by such variables is extremely small, because it is a direct function of how many variables you, as a programmer, declare by hand-typing their declarations. It would take years of typing on a keyboard doing nothing but mindlessly declaring variables before you would declare a number of int variables occupying an amount of memory worth speaking of.
Well, you can use blocks ({ }) and defining a variable as late as possible to limit the scope where it exists.
But unless the variable's address is taken, doing so has no influence on the generated code at all, as the compiler's determination of the scope where it has to keep the variable's value is not significantly impacted.
If the variable's address is taken, failure of escape-analysis, mostly due to inlining-barriers like separate compilation or allowing semantic interpositioning, can make the compiler assume it has to keep it alive till later in the block than strictly neccessary. That's rarely significant (don't worry about a handful of ints, and most often a few lines of code longer keeping it alive are insignificant), but best to keep it in mind for the rare case where it might matter.
If you are that concerned about the tiny amount of memory that is on the stack, then you're probably going to be interested in understanding the specifics of your compiler as well. You'll need to find out what it does when it compiles. The actual shape of the stack-frame is not specified by the C language. It is left to the compiler to figure out. To take an example from the currently accepted answer:
void foo() {
// some codes
// ...
{ // create an extra minimum scope where a is needed
int a;
}
// a doesn't exist here
}
This may or may not affect the memory usage of the function. If you were to do this in a mainstream compiler like gcc or Visual Studio, you would find that they optimize for speed rather than stack size, so they pre-allocate all of the stack space they need at the start of the function. They will do analysis to figure out the minimum pre-allocation needed, using your scoping and variable-usage analysis, but those algorithms literally wont' be affected by extra scoping. They're already smarter than that.
Other compilers, especially those for embedded platforms, may allocate the stack frame differently. On these platforms, such scoping may be the trick you needed. How do you tell the difference? The only options are:
Read the documentation
Try it, and see what works
Also, make sure you understand the exact nature of your problem. I worked on a particular embedded project which eschewed the stack for everything except return values and a few ints. When I pressed the senior developers about this silliness, they explained that on this particular application, stack space was at more of a premium than space for globally allocated variables. They had a process they had to go through to prove that the system would operate as intended, and this process was much easier for them if they allocated everything up front and avoided recursion. I guarantee you would never arrive at such a convoluted solution unless you first knew the exact nature of what you were solving.
As another solution you could look at, you could always build your own stack frames. Make a union of structs, where each struct contains the variables for one stack frame. Then keep track of them yourself. You could also look at functions like alloca, which can allow for growing the stack frame during the function call, if your compiler supports it.
Would a union of structs work? Try it. The answer is compiler dependent. If all variables are stored in memory on your particular device, then this approach will likely minimize stack usage. However, it could also substantially confuse register coloring algorithms, and result in an increase in stack usage! Try and see how it goes for you!
This is my problem in essence. In the life of a function, I generate some integers, then use the array of integers in an algorithm that is also part of the same function. The array of integers will only be used within the function, so naturally it makes sense to store the array on the stack.
The problem is I don't know the size of the array until I'm finished generating all the integers.
I know how to allocate a fixed size and variable sized array on the stack. However, I do not know how to grow an array on the stack, and that seems like the best way to solve my problem. I'm fairly certain this is possible to do in assembly, you just increment stack pointer and store an int for each int generated, so the array of ints would be at the end of the stack frame. Is this possible to do in C though?
I would disagree with your assertion that "so naturally it makes sense to store the array on the stack". Stack memory is really designed for when you know the size at compile time. I would argue that dynamic memory is the way to go here
C doesn't define what the "stack" is. It only has static, automatic and dynamic allocations. Static and automatic allocations are handled by the compiler, and only dynamic allocation puts the controls in your hands. Thus, if you want to manually deallocate an object and allocate a bigger one, you must use dynamic allocation.
Don't use dynamic arrays on the stack (compare Why is the use of alloca() not considered good practice?), better allocate memory from the heap using malloc and resize it using realloc.
Never Use alloca()
IMHO this point hasn't been made well enough in the standard references.
One rule of thumb is:
If you're not prepared to statically allocate the maximum possible size as a
fixed length C array then you shouldn't do it dynamically with alloca() either.
Why? The reason you're trying to avoid malloc() is performance.
alloca() will be slower and won't work in any circumstance static allocation will fail. It's generally less likely to succeed than malloc() too.
One thing is sure. Statically allocating the maximum will outdo both malloc() and alloca().
Static allocation is typically damn near a no-op. Most systems will advance the stack pointer for the function call anyway. There's no appreciable difference for how far.
So what you're telling me is you care about performance but want to hold back on a no-op solution? Think about why you feel like that.
The overwhelming likelihood is you're concerned about the size allocated.
But as explained it's free and it gets taken back. What's the worry?
If the worry is "I don't have a maximum or don't know if it will overflow the stack" then you shouldn't be using alloca() because you don't have a maximum and know it if it will overflow the stack.
If you do have a maximum and know it isn't going to blow the stack then statically allocate the maximum and go home. It's a free lunch - remember?
That makes alloca() either wrong or sub-optimal.
Every time you use alloca() you're either wasting your time or coding in one of the difficult-to-test-for arbitrary scaling ceilings that sleep quietly until things really matter then f**k up someone's day.
Don't.
PS: If you need a big 'workspace' but the malloc()/free() overhead is a bottle-neck for example called repeatedly in a big loop, then consider allocating the workspace outside the loop and carrying it from iteration to iteration. You may need to reallocate the workspace if you find a 'big' case but it's often possible to divide the number of allocations by 100 or even 1000.
Footnote:
There must be some theoretical algorithm where a() calls b() and if a() requires a massive environment b() doesn't and vice versa.
In that event there could be some kind of freaky play-off where the stack overflow is prevented by alloca(). I have never heard of or seen such an algorithm. Plausible specimens will be gratefully received!
The innards of the C compiler requires stack sizes to be fixed or calculable at compile time. It's been a while since I used C (now a C++ convert) and I don't know exactly why this is. http://gribblelab.org/CBootcamp/7_Memory_Stack_vs_Heap.html provides a useful comparison of the pros and cons of the two approaches.
I appreciate your assembly code analogy but C is largely managed, if that makes any sense, by the Operating System, which imposes/provides the task, process and stack notations.
In order to address your issue dynamic memory allocation looks ideal.
int *a = malloc(sizeof(int));
and dereference it to store the value .
Each time a new integer needs to be added to the existing list of integers
int *temp = realloc(a,sizeof(int) * (n+1)); /* n = number of new elements */
if(temp != NULL)
a = temp;
Once done using this memory free() it.
Is there an upper limit on the size? If you can impose one, so the size is at most a few tens of KiB, then yes alloca is appropriate (especially if this is a leaf function, not one calling other functions that might also allocate non-tiny arrays this way).
Or since this is C, not C++, use a variable-length array like int foo[n];.
But always sanity-check your size, otherwise it's a stack-clash vulnerability waiting to happen. (Where a huge allocation moves the stack pointer so far that it ends up in the middle of another memory region, where other things get overwritten by local variables and return addresses.) Some distros enable hardening options that make GCC generate code to touch every page in between when moving the stack pointer by more than a page.
It's usually not worth it to check the size and use alloc for small, malloc for large, since you also need another check at the end of your function to call free if the size was large. It might give a speedup, but this makes your code more complicated and more likely to get broken during maintenance if future editors don't notice that the memory is only sometimes malloced. So only consider a dual strategy if profiling shows this is actually important, and you care about performance more than simplicity / human-readability / maintainability for this particular project.
A size check for an upper limit (else log an error and exit) is more reasonable, but then you have to choose an upper limit beyond which your program will intentionally bail out, even though there's plenty of RAM you're choosing not to use. If there is a reasonable limit where you can be pretty sure something's gone wrong, like the input being intentionally malicious from an exploit, then great, if(size>limit) error(); int arr[size];.
If neither of those conditions can be satisfied, your use case is not appropriate for C automatic storage (stack memory) because it might need to be large. Just use dynamic allocation autom don't want malloc.
Windows x86/x64 the default user-space stack size is 1MiB, I think. On x86-64 Linux it's 8MiB. (ulimit -s). Thread stacks are allocated with the same size. But remember, your function will be part of a chain of function calls (so if every function used a large fraction of the total size, you'd have a problem if they called each other). And any stack memory you dirty won't get handed back to the OS even after the function returns, unlike malloc/free where a large allocation can give back the memory instead of leaving it on the free list.
Kernel thread stack are much smaller, like 16 KiB total for x86-64 Linux, so you never want VLAs or alloca in kernel code, except maybe for a tiny max size, like up to 16 or maybe 32 bytes, not large compared to the size of a pointer that would be needed to store a kmalloc return value.
While taking my first steps with C, I quickly noticed that int array[big number] causes my programs to crash when called inside a function. Not quite as quickly, I discovered that I can prevent this from happening by defining the array with global scope (outside the functions) or using malloc.
My question is:
Starting at which size is it necessary to use one of the above methods to make sure my programs won't crash?
I mean, is it safe to use just, e.g., int i; for counters and int chars[256]; for small arrays or should I just use malloc for all local variables?
You should understand what the difference is between int chars[256] in a function and using malloc().
In short, the former places the entire array on the stack. The latter allocates the memory you requested from the heap. Generally speaking, the heap is much larger than the stack, but the size of each can be adjusted.
Another key difference is that a variable allocated on the stack will technically be gone after you return from the method. (Oh, your program may function as though it's not gone for a bit if you continue to access that array, but ho ho ho danger lurks.) A hunk of memory allocated with malloc will remain allocated until you explicitly free it or the program exits.
You should use malloc for your dynamic memory allocation. For statically sized arrays (or any other object) within functions, if the memory required is to big you will quickly get a segmentation fault. I don't think a 'safe limit' can be defined, its probably implementation specific and other factors come in play too, like the current stack and objects within it created by callers to the current function. I would be tempted to say that anything below the page size (usually 4kb) should be safe as long as there is no recursion involved, but I do not think there are such guarantees.
It depends. If you have some guarantee that a line will never be longer than 100 ... 1000 characters you can get away with a fixed size buffer. If you don't: you don't. There is a difference between reading in a neat x KB config file and a x GB XML file (with no CR/LF). It depends.
The next choice is: do you want your program to die gracefully? It is only a design choice.
I am writing a C program that uses lots of recursive functions. I am also using a dynamic list to store some data while recursing. I implemented a Push function to push data into the list.
After several calls for the push function " > 17,000 times" i am getting the following error:
Unhandled exception at 0x77963c47 in Prob - Cap CE.exe: 0xC00000FD: Stack overflow.
at return HeapAlloc(_crtheap, 0, size ? size : 1); that is called from stack->listNode = malloc(sizeof(struct Node)); in the Push function.
I opened task manager and identified that I still have lots of free memory. So I guess it is not a memory leak issue.
Is there any limitation on how much I can add to the list, or how many times I can call a function?
The size of the stack allocated for a process is fixed. So even if the system has plenty of memory, you can overflow the stack.
What's more, your process itself usually has plenty of memory. It's just the stack that's quite small.
">17,000 times" is not "several". It's bloody loads.
You can't expect your stack to hold 17,000 frames (and how much it can hold is implementation-dependent, and also depends on how much data is in each frame).
Use iteration instead.
Well, there will be limitations, for instance recursion can only go as far as the stack memory allocated for your app. Since you're getting a stack overflow, this seems like it is probably the problem.
See Change stack size for a C++ application in Linux during compilation with GNU compiler for some details on how to modify the stack size. If you can't get it to work right even with a huge stack, then it's likely you'll have to do some optimizations to reduce the amount of memory you're using, or to limit your recursions.
You could try moving some of your stack variables into the heap by allocating them rather than just doing local declarations, just as a first idea. If you could post some code as to what your recursive functions look like, we might be able to give suggestions. A likely solution to your ills is just to convert your functions from using recursion to using loops.
This question already has answers here:
What's the point of using malloc when you can use pointer? [duplicate]
(3 answers)
Closed 5 years ago.
#include<stio.h>
main()
{
int *p,i;
p = (int*)malloc(sizeof(int));
printf("Enter:");
scanf("%d",p);
for(i=1;i<3;i++)
{
printf("Enter");
scanf("%d",p+i);
}
for(i=0;i<3;i++)
{
printf("No:%d\n",*(p+i));
}
getch();
return 0;
}
In this C program memory is accessed without allocation.The program works.Will any problem arise by accessing memory without allocation?If yes then what is the solution for storing a collection of integer data which the size is not known in advance?
Yes, it leads to undefined behavior. The problem is working here purely becuase of luck and may crash any time. The solution is to allocate the memory using malloc For example if you want to allocate memory for count number of elements then you can use int* p = (int*)malloc(sizeof(int)*count);. From here on you can access p as an array of count elements.
It likely works because the memory immediately after *p is both accessible (allocated in the VM system and has the right bits set), and not in use for anything else. This could all change if malloc finds you some bytes immediately before an inaccessible page; or if you move to a malloc implementation that uses the trailing space for bookkeeping.
So it's not really safe.
Accessing unallocated memory leads to undefined behavior. Exactly what happens will depends on a variety of conditions. It may "work" now but you could see problems when you extend your program.
If you don't know how many items you want to read, there are a couple of strategies to use.
Use realloc to grow the buffer as you need more space.
Use a linked list instead of an array
Most definitely yes. Its just pure luck that you can access without allocating. malloc does not what memory you are using and that could result in serious problems.
Hence its a compulsion (i don't want to use the word better here) to allocate memory according to your needs and then use it.
Some problems which could result are:
Segmentation fault
Memory corruption
and it may result in giving you headache for hours when the behavior is undefined.
For eg: the location of a crash may not be the exact place of origin of the problem.
The reason this code works is that the kernel never gives you a fraction of the system page size (which should be 4k). This means the memory after the first sizeof(int) bytes is actually owned by the process you run, but not allocated to you by the second layer of abstraction which is malloc.
"Segmentation fault" happens when you try and access memory outside the pages allocated to you by the kernel. You won't see it until you step out of your page.
The problem that may arise here is that you use malloc again and you will receive a pointer to a memory you used without malloc being aware of it. This will cause hellish bugs since you will change data used in different contexts without knowing.
As for your second question, the right way is very program dependent.
If the number of elements can be bounded reasonably, it might be OK to always allocate the same size using a constant defined in your program. This is ALWAYS the secure way (you need to make sure you don't let the user give you more than what you allocated).
If you really have a broad range of array sizes here, you might want to use a linked list which is built for that exactly.
There are two levels of memory allocation that take usually take place. At operating system level, you map memory pages to your address space. A page is the basic unit of memory management and is usually something like 1K or 4K bytes (but can be much larger or as small as 512 bytes, depending upon the system). It is possible to do that mapping yourself by making the appropriate system calls. However, applications generally only do that when they need large blocks of memory.
Standard libraries generally maintain a pool of pages. When you call malloc, the library looks to see if there is available memory in the pool. If so, it returns a block of memory from pages already mapped by the operating system. If not, the library make the system call to map more pages to the process and adds them to the managed pool.
Mapping and unmapping pages is a rather time consuming process. By using pooling, the library can speed things up significantly.
Invariable, the standard library functions allocate a few bytes in front of the memory returned by malloc and the like so that they can know how much memory is in the block when it is free'd. Many will also add memory add the end of the block as well for error checking.
When you are doing what you are doing, you could be reading this extra data or you could be reading some data that was mapped to the memory pool by the library.
What you are doing is bad.
IF you do not know the number of items in advance, you can use a data structure, such a linked list where new entries are created with each new number.