I invoke a function 147 times in recursive and when it invokes for 147. times, program exe stops(codeblocks).
Before invokin function again, it assigned 1 int global variable to local, 1 int 2 dimensional global array to local and 1 string global variable to local variable. So, 146 of those maybe became a very huge load for program?
The function is:
It seems your stack is overflowing by recursive calls.
Quoting from above wiki page
In software, a stack overflow occurs when the stack pointer exceeds
the stack bound. The call stack may consist of a limited amount of
address space, often determined at the start of the program. The size
of the call stack depends on many factors, including the programming
language, machine architecture, multi-threading, and amount of
available memory. When a program attempts to use more space than is
available on the call stack (that is, when it attempts to access
memory beyond the call stack's bounds, which is essentially a buffer
overflow), the stack is said to overflow, typically resulting in a
program crash
Very deep recursion and large stack variables along with recursion are some easy to fall reasons of stack overflow.
You may want to write a smarter code to get away from recursions.
Below links may help you get there.
Way to go from recursion to iteration
Replace Recursion with Iteration
Each time you invoke your function, you allocate:
int visitedS[2416] = 2416 * 32 bits = 9.4KB
char pathS[4500] = 4500 * 8 bits = 4.4KB
So that's almost 14KB that gets placed on the stack every time you recurse.
After 147 recursions, you've put 1.98MB on the stack. That's not so huge - a typical Linux stack limit is 8MB.
I would check - through using a debugger or even adding debug print statements - your assumption that this is truly happening after 147 recursions. Perhaps there is a bug causing more invocations than you believed.
Even so, it may well be worth thinking about ways to reduce the memory footprint of each invocation. You seem to be creating local arrays which are copies of a global. Why not just use the data in the global. If your function must make changes to that data, keep a small set of deltas locally.
Related
This question already has answers here:
How to determine maximum stack usage in embedded system with gcc?
(7 answers)
Closed 1 year ago.
I'm writing an embedded program that uses a static limited stack area of a known size (in other words, I have X bytes for the stack, and there's no overlaying OS that can allocate more stack on demand for me). I want to avoid errors during runtime, and catch them in build time instead - to have some indication if I mistakenly declared too much variables in some function block that won't fit in the stack during the runtime.
Given that I don't use recursive calls in my program, can I somehow know during compilation time how much space on stack all my local variables will take on the deepest function call path? Or at least know how much space my variables will take in a single block (function) if the compiler is not smart enough to analyze it on all the nested calls?
Given that I don't use recursive calls in my program, can I somehow know during compilation time how much space on stack all my local variables will take on the deepest function call path?
Only if you don't use interrupts. Which is extremely likely in any embedded system. So you'll have to find out stack use with dynamic analysis.
The old school way is to set your whole stack area to a value like 0xAA upon reset from a debugger, then let the program run for a while, make sure to provoke all use-cases. Then halt and inspect how far down you still have 0xAA in memory. It isn't a 100% scientific, fool-proof method but works just fine in practice, in the vast majority of cases.
Other methods involve setting write breakpoints at certain stack locations where you don't expect the program to end up, sort of like a "hardware stack canary". Run the program and ensure that the breakpoint never triggers. If it does, then investigate from there, move the breakpoint further down the memory map to see exactly where.
Another good practice is to always memory map your stack so that it can only overflow into forbidden memory or at least into read-only flash etc - ideally you'd get a hardware exception for stack overflow. You definitely want to avoid the stack overflowing into other RAM sections like .data/.bss, as that will cause severe and extremely subtle error scenarios.
I recently learned about stacks, so I was experimenting to see what the stack size is and what happens when it overflows. I found out that on Unix the default stack size is 8 MiB, and that supports my findings since I cannot declare a string having size greater than or equal to 8 MiB in my main function. However, when I declare a variable in main() it affects other functions. For example:
#include <stdio.h>
void foo(void)
{
long int size = 1024*1024*2;
char str[size];
str[size - 1] = 'a';
printf("%c\n", str[size - 1]);
}
int main(int argc, char** argv)
{
long int size = 1024*1024*6;
char str[size];
str[size - 1] = 'a';
printf("%c\n", str[size - 1]);
foo();
return 0;
}
This code results in segmentation fault but if I make the string size 5 MiB in main() then there is no segmentation fault. Does that mean my C program cannot allocate more than 8 MiB of RAM for local variables (of all functions)? If so, what IS the point of stacks?
No, each function doesn't get its own independent stack space. There's only one stack in your program and there's an limited finite amount of stack space available to you.
How Stack works
This LIFO behavior is exactly what a function does when returning
to the function that called it.
Flow in the Stack
The caller pushes the return address onto the stack
When the called function finishes its execution, it pops the return
address off the call stack (this popped element is also known as
stack frame) and transfers control to that address.
If a called function calls on to yet another function, it will push
another return address onto the top of the same call stack, and
so on, with the information stacking up and unstacking as the
program dictates.
All of the above process happens in the same stack memory. Each function does have its own space in the stack but every function gets its space allocated in the same stack. This is called the Global Call Stack of your program.
It is used to store local variables which are used inside the function.
However, dynamically allocated space is stored on the heap. Heap is used to store dynamic variables. It is a region of process’ memory. malloc(), calloc(), resize() all these inbuilt functions are generally used to store dynamic variables.
As for the stack overflow issue, the call stack size is limited. Only a certain amount of memory can be used. If many function calls happen, the stack space would eventually run out which would give you a stack overflow error which would most likely cause your program to crash.
If there are a lot of variables in your function or some variables which needs a huge amount of space in your program, then the stack space will eventually run out and it would cause a stack overflow. E.g. the following would probably give stack overflow in most cases and cause your program to crash:
int main() {
int A[100000][100000];
}
Hope this clears your doubt !
NOTE:
In an multi-threaded environment, each thread gets its own call stack space separately instead of having the same Global Call Stack. So, in an multi-threaded environment, the answer to your question will be YES.
Does that mean my c program cannot allocate more than 8MB of ram for local variables (of all functions) ?
Yes and no. Yes, your program can't use more space for local variables than the available stack space, whatever that is. But no, you're not limited to 8MB for all functions, you're only limited to that much total stack space for functions that are currently executing. A program might contain thousands of functions, but only a relative handful of those will be invoked at any given moment.
When your program calls a function, space is reserved on the stack for the function's return value and it's local variables. If that function calls another function, space will then be reserved for that next function's return value and local variables. When each function returns, the return value is read and the local variables and return value are popped off the stack. So functions only use stack space while they're executing.
If so, what's the point of stacks ?
The point is to provide the space needed for local variables, to facilitate returning a value to the caller, and to make allocating that space fast and efficient. Functions don't typically need huge amounts of storage for local variables, so 8MB is typically more than enough.
If you find that you need to allocate a large amount of memory, there are memory allocation functions that make that easy. Let's say you need to create a multi-megabyte string as in your example. You'd typically use a function like malloc() or calloc() to create that object on the heap instead of on the stack, and the only local variable you'd need is a pointer to the allocated memory.
The "stack" is one, shared space in memory, and true to its name, every nested function invocation "pushes" a new "frame" (set of space for local variables) onto that shared stack. Yes, the total size of the stack's space in memory is shared between all functions which are (currently) executing, and if the total used space during the run of you program exceeds what the OS has set aside for it, you will cause an (ahem) "stack overflow" crash.
The point is to provide work space for each function's invocation. Typically, the amount of space used by any particular function on the stack is quite small-- perhaps some integers or a couple smallish arrays, etc. Think tens or hundreds of bytes, not usually kilobytes or certainly megabytes. This is mostly just idiomatic and you get used to what makes sense to have on the stack and what doesn't when you've worked with enough code of your own and others'. It would be exceptionally unusual in production code to have something megabytes large as an actual local variable.
In practice, the primary cause of stack overflow errors in the real world is accidental infinite recursion-- when you end up calling through into the same functions over and over without a recursive base case. Those stack frames may each be small, but if the call chain is unbounded eventually you'll overflow.
When you want to use actual larger pieces of memory, large string buffers, etc, you'll typically allocate them from a different shared chunk of memory referred to as "the heap". You can allocate (with malloc and its cousins) what you need and then free that when done. The heap's memory space is global to your program, and is not constrained or related to particular function invocations.
C language standard does not know anything about the stack. How function are called, how parameter are passed and where automatic storage objects are stored is up to implementation.
Most of the implementation will actually have only one stack but I will give you some very common exeptions.
RTOSes. Many RTOS-es implement tasks as normal functions. Functions which are separate tasks will have separate stacks.
Many multitasking libraries (like pthread) will give threads (which are functions) separate stacks.
Many hardware designs have more than one stack - for example very popular ARM Cortex uCs - having two separate hardware stacks.
etc etc.
This simple C program rarely terminates at the same call depth:
#include <stdio.h>
#include <stdlib.h>
void recursive(unsigned int rec);
int main(void)
{
recursive(1);
return 0;
}
void recursive(unsigned int rec) {
printf("%u\n", rec);
recursive(rec + 1);
}
What could be the reasons behind this chaotic behavior?
I am using fedora (16GiB ram, stack size of 8192), and compiled using cc without any options.
EDIT
I am aware that this program will throw a stackoverflow
I know that enabling some compiler optimizations will change the behavior and that the program will reach integer overflow.
I am aware that this is undefined behavior, the purpose of this question is to understand/get an overview of the implementation specific internal behaviors that might explain what we observe there.
The question is more, given that on Linux the thread stack size is fixed and given by ulimit -s, what would influence the available stack size so that the stackoverflow does not always occur at the same call depth?
EDIT 2
#BlueMoon always sees the same output on his CentOS, while on my Fedora, with a stack of 8M, I see different outputs (last printed integer 261892 or 261845, or 261826, or ...)
Change the printf call to:
printf("%u %p\n", rec, &rec);
This forces gcc to put rec on the stack and gives you its address which is a good indication of what's going on with the stack pointer.
Run your program a few times and note what's going on with the address that's being printed at the end. A few runs on my machine shows this:
261958 0x7fff82d2878c
261778 0x7fffc85f379c
261816 0x7fff4139c78c
261926 0x7fff192bb79c
First thing to note is that the stack address always ends in 78c or 79c. Why is that? We should crash when crossing a page boundary, pages are 0x1000 bytes long and each function eats 0x20 bytes of stack so the address should end with 00X or 01X. But looking at this closer, we crash in libc. So the stack overflow happens somewhere inside libc, from this we can conclude that calling printf and everything else it calls needs at least 0x78c = 1932 (possibly plus X*4096) bytes of stack to work.
The second question is why does it take a different number of iterations to reach the end of the stack? A hint is in the fact that the addresses we get are different on every run of the program.
1 0x7fff8c4c13ac
1 0x7fff0a88f33c
1 0x7fff8d02fc2c
1 0x7fffbc74fd9c
The position of the stack in memory is randomized. This is done to prevent a whole family of buffer overflow exploits. But since memory allocations, especially at this level, can only be done in multiple of pages (4096 bytes) all initial stack pointers would be aligned at 0x1000. This would reduce the number of random bits in the randomized stack address, so additional randomness is added by just wasting a random amount of bytes at the top of the stack.
The operating system can only account the amount of memory you use, including the limit on the stack, in whole pages. So even though the stack starts at a random address, the last accessible address on the stack will always be an address ending in 0xfff.
The short answer is: to increase the amount of randomness in the randomized memory layout a bunch of bytes on the top of the stack are deliberately wasted, but the end of the stack has to end on a page boundary.
You won't have the same behaviour between executions because it depends on the current memory available. The more memory you have available, the further you'll go in this recursive function.
Your program runs infinitely as there is no base condition in your recursive function. Stack will grow continuously by each function call and will result in stack overflow.
If it would be the case of tail-recursion optimization (with option -O2), then stack overflow will occur for sure. Its invoke undefined behavior.
what would influence the available stack size so that the stackoverflow does not always occur at the same call depth?
When stack overflow occurs it invokes undefined behavior. Nothing can be said about the result in this case.
Your recursive call is not necessarily going to cause undefined behaviour due to stackoverflow (but will due to integer overflow) in practice. An optimizing compiler could simply turn your compiler into an infinite "loop" with a jump instruction:
void recursive(int rec) {
loop:
printf("%i\n", rec);
rec++;
goto loop;
}
Note that this is going to cause undefined behaviour since it's going to overflow rec (signed int overflow is UB). For example, if rec is of an unsigned int, for example, then the code is valid and in theory, should run forever.
The above code can cause two issue:
Stack Overflow.
Integer overflow.
Stack Overflow: When a recursive function is called, all its variable is pushed onto the call stack including its return address. As there is no base condition which will terminate the recursion and the stack memory is limited, the stack will exhausted resulting Stack Overflow exception. The call stack may consist of a limited amount of address space, often determined at the start of the program. The size of the call stack depends on many factors, including the programming language, machine architecture, multi-threading, and amount of available memory. When a program attempts to use more space than is available on the call stack (that is, when it attempts to access memory beyond the call stack's bounds, which is essentially a buffer overflow), the stack is said to overflow, typically resulting in a program crash.
Note that, every time a function exits/return, all of the variables pushed onto the stack by that function, are freed (that is to say, they are deleted). Once a stack variable is freed, that region of memory becomes available for other stack variables. But for recursive function, the return address are still on the stack until the recursion terminates. Moreover, automatic local variables are allocated as a single block and stack pointer advanced far enough to account for the sum of their sizes. You maybe interested at Recursive Stack in C.
Integer overflow: As every recursive call of recursive() increments rec by 1, there is a chance that Integer Overflow can occur. For that, you machine must have a huge stack memory as the range of unsigned integer is: 0 to 4,294,967,295. See reference here.
There is a gap between the stack segment and the heap segment. Now because the size of heap is variable( keeps on changing during execution), therefore the extent to which your stack will grow before stackoverflow occurs is also variable and this is the reason why your program rarely terminates at the same call depth.
When a process loads a program from an executable, typically it allocates areas of memory for the code, the stack, the heap, initialised and uninitialised data.
The stack space allocated is typically not that large, (10s of megabytes probably) and so you would imagine that physical RAM exhaustion would not be an issue on a modern system and the stack overflow would always happen at the same depth of recursion.
However, for security reasons, the stack isn't always in the same place. Address Space Layout Randomisation ensures that the base of the stack's location varies between invocations of the program. This means that the program may be able to do more (or fewer) recursions before the top of the stack hits something inaccessible like the program code.
That's my guess as to what is happening, anyway.
On Linux, using C, assume I have an dynamically determined n naming the number of elements I have to store in an array (int my_array[n]) just for a short period of time, say, one function call, whereby the called function only uses little memory (some hundred bytes).
Mostly n is little, some tenths. But sometimes n may be big, as much as 1000 or 1'000'000.
How do I calculate, whether my stack can hold n*o + p bytes without overflowing?
Basically: How much bytes are there left on my stack?
Indeed, the checking available stack question gives good answer.
But a more pragmatic answer is: don't allocate big data on the call stack.
In your case, you could handle differently the case when n<100 (and then allocating on the stack, perhaps thru alloca, makes sense) and the case when n>=100 (then, allocate on the heap with malloc (or calloc) and don't forget to free it). Make the threshold 100 a #define-d constant.
A typical call frame on the call stack should be, on current laptops or desktops, a few kilobytes at most (and preferably less if you have recursion or threads). The total stack space is ordinarily at most a few megabytes (and sometimes much less: inside the kernel, stacks are typically 4Kbytes each!).
If you are not using threads, or if you know that your code executes on the main stack, then
Record current stack pointer when entering main
In your routine, get current stack limit (see man getrlimit)
Compare difference between current stack pointer and the one recorded in step 1 with the limit from step 2.
If you are using threads and could be executing on a thread other than main, see man pthread_getattr_np
You may think that this is a coincidence that the topic of my question is similar to the name of the forum but I actually got here by googling the term "stack overflow".
I use the OPNET network simulator in which I program using C. I think I am having a problem with big array sizes. It seems that I am hitting some sort of memory allocation limitation. It may have to do with OPNET, Windows, my laptop memory or most likely C language. The problem is caused when I try to use nested arrays with a total number of elements coming to several thousand integers. I think I am exceeding an overall memory allocation limit and I am wondering if there is a way to increase this cap.
Here's the exact problem description:
I basically have a routing table. Let's call it routing_tbl[n], meaning I am supporting 30 nodes (routers). Now, for each node in this table, I keep info. about many (hundreds) available paths, in an array called paths[p]. Again, for each path in this array, I keep the list of nodes that belong to it in an array called hops[h]. So, I am using at least nph integers worth of memory but this table contains other information as well. In the same function, I am also using another nested array that consumes almost 40,000 integers as well.
As soon as I run my simulation, it quits complaining about stack overflow. It works when I reduce the total size of the routing table.
What do you think causes the problem and how can it be solved?
Much appreciated
Ali
It may help if you post some code. Edit the question to include the problem function and the error.
Meanwhile, here's a very generic answer:
The two principal causes of a stack overflow are 1) a recursive function, or 2) the allocation of a large number of local variables.
Recursion
if your function calls itself, like this:
int recurse(int number) {
return (recurse(number));
}
Since local variables and function arguments are stored on the stack, then it will in fill the stack and cause a stack overflow.
Large local variables
If you try to allocate a large array of local variables then you can overflow the stack in one easy go. A function like this may cause the issue:
void hugeStack (void) {
unsigned long long reallyBig[100000000][1000000000];
...
}
There is quite a detailed answer to this similar question.
Somehow you are using a lot of stack. Possible causes include that you're creating the routing table on the stack, you're passing it on the stack, or else you're generating lots of calls (eg by recursively processing the whole thing).
In the first two cases you should create it on the heap and pass around a pointer to it. In the third case you'll need to rewrite your algorithm in an iterative form.
Stack overflows can happen in C when the number of embedded recursive calls is too high. Perhaps you are calling a function from itself too many times?
This error may also be due to allocating too much memory in static declarations. You can switch to dynamic allocations through malloc() to fix this type of problem.
Is there a reason why you cannot use the debugger on this program?
It depends on where you have declared the variable.
A local variable (i.e. one declared on the stack is limited by the maximum frame size) This is a limit of the compiler you are using (and can usually be adjusted with compiler flags).
A dynamically allocated object (i.e. one that is on the heap) is limited by the amount of available memory. This is a property of the OS (and can technically by larger the physical memory if you have a smart OS).
Many operating systems dynamically expand the stack as you use more of it. When you start writing to a memory address that's just beyond the stack, the OS assumes your stack has just grown a bit more and allocates it an extra page (usually 4096Kib on x86 - exactly 1024 ints).
The problem is, on the x86 (and some other architectures) the stack grows downwards but C arrays grow upwards. This means if you access the start of a large array, you'll be accessing memory that's more than a page away from the edge of the stack.
If you initialise your array to 0 starting from the end of the array (that's right, make a for loop to do it), the errors might go away. If they do, this is indeed the problem.
You might be able to find some OS API functions to force stack allocation, or compiler pragmas/flags. I'm not sure about how this can be done portably, except of course for using malloc() and free()!
You are unlikely to run into a stack overflow with unthreaded compiled C unless you do something particularly egregious like have runaway recursion or a cosmic memory leak. However, your simulator probably has a threading package which will impose stack size limits. When you start a new thread it will allocate a chunk of memory for the stack for that thread. Likely, there is a parameter you can set somewhere that establishes the the default stack size, or there may be a way to grow the stack dynamically. For example, pthreads has a function pthread_attr_setstacksize() which you call prior to starting a new thread to set its size. Your simulator may or may not be using pthreads. Consult your simulator reference documentation.