I have the following C program compiled in Microsoft Visual Studio Express 2012:
int main() {
int a[300000];
return 0;
}
This crashes with a stack overflow in msvcr110d.dll!__crtFlsGetValue().
If I change the array size from 300,000 to 200,000 it works fine (in so much as this simple program can be said to 'work' since it doesn't do anything).
I'm running on Windows 7 and have also tried this with gcc under Cygwin and it produces the same behaviour (in this case a seg fault).
What the heck?
There are platform-specific limits on the size of the space used by automatic objects in C (the "stack size"). Objects that are larger than that size (which may be a few kilobytes on an embedded platform and a few megabytes on a desktop machine) cannot be declared as automatic objects. Make them static or dynamic instead.
In a similar vein, there are limits on the depth of function calls, and in particular on recursion.
Check your compiler and/or platform documentation for details on what the actual size is, and on how you might be able to change it. (E.g. on Linux check out ulimit.)
Because it's being allocated on the stack and the stack has a limited size, obviously not large enough to hold 300000 ints.
Use heap allocation a la malloc:
int* a = malloc(sizeof(int) * 300000);
// ...
free(a);
The heap can hold a lot more than the stack.
The size of thread stacks is traditionally limited by operating systems because of a finite limit to the amount of virtual address space available to each process.
As the virtual address space allocated to a thread stack can't be changed once it is allocated, there is no strategy other than to allocate a fairly large, but limited, chunk to each thread - even when most threads will use very little of it.
Similarly, there is a finite limit also on the number of threads a process is allowed to spawn.
At a guess, the limit here is 1MB, and Windows then limits the number of threads to - say - 256, this means that 256MB of the 3GB virtual address space available to a 32-bit process is allocated to thread stacks - or put another way, 1/12th.
On 64-bit systems, there is obviously a lot more virtual space to play with, but having a limit is still sensible in order to quickly detect - and terminate - infinite recursion.
Local variables claim space from the stack. So, if you allocate something large enough, the stack will inevitably overflow.
Related
so, i was making this program that let people know the number of contiguous subarray which sum is equal to a certain value.
i have written the code , but when i try to run this code in vcexpress 2010, it says these error
Unhandled exception at 0x010018e7 in test 9.exe: 0xC00000FD: Stack overflow.
i have tried to search for the solution in this website and other webisites, but i can't seem to find any solution which could help me fix the error in this code(they are using recursion while i'm not).
i would be really grateful if you would kindly explain what cause this error in my code, and how to fix this error. any help would be appreciated. Thank you.
here is my code :
#include <stdio.h>
int main ()
{
int n,k,a=0,t=0;
unsigned long int i[1000000];
int v1,v2=0,v3;
scanf("%d %d",&n,&k);
for(v3=0;v3<n;v3++)
{
scanf("%d",&i[v3]);
}
do
{
for(v1=v2;v1<n;v1++)
{
t=i[v1]+t;
if(t==k)
{
a++;
break;
}
}
t=0;
v2++;
}while(v2!=n);
printf("%lu",a);
return 0;
}
Either move
unsigned long int i[1000000];
outside of main, thus making it a global variable (not an automatic one), or better yet, use some C dynamic heap allocation:
// inside main
unsigned long int *i = calloc(1000000, sizeof(unsigned long int));
if (!i) { perror("calloc"); exit(EXIT_FAILURE); };
BTW, for such a pointer, I would use (for readability reasons) some other name than i. And near the end of main you'll better free(i); to avoid memory leaks.
Also, you could move these 2 lines after the read of n and use calloc(n, sizeof(unsigned long int)) instead of calloc(1000000, sizeof(unsigned long int)) ; then you can handle arrays bigger than a million elements if your computer and system provides enough resources for that.
Your initial code is declaring an automatic variable which goes into the call frame of main on your call stack (which has a limited size, typically a megabyte or a few of them). On some operating systems there is a way to increase the size of that call stack (in an OS-specific way). BTW each thread has its own call stack.
As a rule of thumb, your C functions (including main) should avoid having call frames bigger than a few kilobytes. With the GCC compiler, you could invoke it with gcc -Wall -Wextra -Wframe-larger-than=1024 -g to get useful warnings and debug information.
Read the virtual address space wikipage. It has a nice picture worth many words. Later, find the way to query, on your operating system, the virtual address space of your process (on Linux, use proc(5) like cat /proc/$$/maps etc...). In practice, your virtual address space is likely to contain many segments (perhaps a dozen, sometimes thousands). Often, the dynamic linker or some other part of your program (or of your C standard library) uses memory-mapped files. The standard C heap (managed by malloc etc) may be organized in several segments.
If you want to understand more about virtual address space, take time to read a good book, like: Operating systems, three easy pieces (freely downloadable).
If you want to query the organization of the virtual address space in some process, you need to find an operating-system specific way to do that (on Linux, for a process of pid 1234, use /proc/1234/maps or /proc/self/maps from inside the process).
Memory is laid out much more differently than simply 4 segments(which was done long ago). The answer to the question can be generalized this way - the global or dynamically allocated memory space is handled differently than that of local variables by the system, where as the memory for local variable is limited in size, memory for dynamic allocation or global variables doesn't put a lower constraint like this.
In modern system the concept of virtual address space is there. The process from your program gets a chunk of it. That portion of memory is now responsible for holding the required memory.
Now for dynamic allocation and so on, the process can request more memory and depending on the other processes and so on, new memory request is serviced. For dynamic or global array there is no limit process wise (of course system wise there is- it cant haverequest all memory). That's why dynamic allocation or using global variable won't cause the process to run out of it's allocated memory unlike the automatic lifetime memory that it originally had for local variables.
Basically you can check your stack size
for example in Linux : ulimit -s (Kbytes)
and then decide how you manipulate your code regarding that.
As a concept I would never allocate big piece of memory on the stack because unless you know exactly the depth of your function call and the stack use, it's hard to control the precised allocated memory on stack during run time
This is a follow-up to my previous question about why size_t is necessary.
Given that size_t is guaranteed to be big enough to represent the largest size of a block of memory you can allocate (meaning there can still be some integers bigger than size_t), my question is...
What determines how much you can allocate at once?
The architecture of your machine, the operating system (but the two are intertwined) and your compiler/set of libraries determines how much memory you can allocate at once.
malloc doesn't need to be able to use all the memory the OS could give him. The OS doesn't need to make available all the memory present in the machine (and various versions of Windows Server for example have different maximum memory for licensing reasons)
But note that the OS can make available more memory than the one present in the machine, and even more memory than the one permitted by the motherboard (let's say the motherboard has a single memory slot that accepts only 1gb memory stick, Windows could still let a program allocate 2gb of memory). This is done throught the use of Virtual Memory, Paging (you know, the swap file, your old and slow friend :-) Or, for example, through the use of NUMA.
I can think of three constraints, in actual code:
The biggest unsigned int size_t is able to allocate. size_t should be the same type (same size, etc.) the OS' memory allocation mechanism is using.
The biggest block the operating system is able to handle in RAM (how are block's size represented? how this representation affects the maximum block size?).
Memory fragmentation (largest free block) and the total available free RAM.
for example I can do
int *arr;
arr = (int *)malloc(sizeof(int) * 1048575);
but I cannot do this without the program crashing:
int arr[1048575];
why is this so?
Assuming arr is a local variable, declaring it as an array uses memory from the (relatively limited) stack, while malloc() uses memory from the (comparatively limitless) heap.
If you're allocating these as local variables in functions (which is the only place you could have the pointer declaration immediately followed by a malloc call), then the difference is that malloc will allocate a chunk of memory from the heap and give you its address, while directly doing int arr[1048575]; will attempt to allocate the memory on the stack. The stack has much less space available to it.
The stack is limited in size for two main reasons that I'm aware of:
Traditional imperative programming makes very little use of recursion, so deep recursion (and heavy stack growth) is "probably" a sign of infinite recursion, and hence a bug that's going to kill the process. It's therefore best if it is caught before the process consumes the gigabytes of virtual memory (on a 32 bit architecture) that will cause the process to exhaust its address space (at which point the machine is probably using far more virtual memory than it actually has RAM, and is therefore operating extremely slowly).
Multi-threaded programs need multiple stacks. Therefore, the runtime system needs know that the stack will never grow beyond a certain bound, so it can put another stack after that bound if a new thread is created.
When you declare an array, you are placing it on the stack.
When you call malloc(), the memory is taken from the heap.
The stack is usually more limited compared to the heap, and is usually transient (but it depends on how often you enter and exit the function that this array is declared in.
For such a large (maybe not by today's standards?) memory, it is good practice to malloc it, assuming you want the array to last around for a bit.
i declared a struct variable in C of size greater than 1024bytes. On running Coverity (a static code analyzer application) it reports that this stack variable is greater than 1024 bytes and therefore a cause of error.
I'd like to know if I need to worry about this warning? Is there really a maximum limit to the size of a single stack variable?
thanks,
che
The maximum size of a variable is the limited by the maximum size of the stack (specifically, how much of the stack is left over from any current use including variables and parameters from functions higher on the stack as well as process frame overhead).
On Windows, the stacksize of the first thread is a property of the executable set during linking while the stacksize of a thread can be specified during thread creation.
On Unix, the stacksize of the first thread is usually only limited only by how much room there is for it to grow. Depending on how the particular Linux lays out memory and your use of shared objects, that can vary. The stacksize of a thread can also be specified during thread creation.
The problem it is trying to protect you from is stack overflow, because of different execution paths, it is very hard to find in testing. Mostly for this reason - it is considered bad form to allocate a large amount of data on the stack. You are only really likely to run into a real problem on an embedded system though.
In other words, it sets an arbitrary limit to what it considers too much data on the stack.
Yes. Of course it's limited by the address space of your system. It's also limited by the amount of space allocated to the stack by your OS, which usually can't be changed after your program starts but can be changed beforehand (either by the launching process, or by the properties of the executable). At a quick glance, the maximum stack size on my OS X system is 8 MiB and on Linux it's 10 MiB. On some systems, you can even allocate a different amount of stack to each different thread you start, although this is of limited usefulness. Most compilers also have another limit to how much they'll allow in a single stack frame.
On a modern desktop, I wouldn't worry about a 1k stack allocation unless the function were recursive. If you're writing embedded code or code for use inside an OS kernel, it would be a problem. Code in the Linux kernel is only permitted 64 KiB stacks or less, depending on configuration options.
This article is pretty interesting regarding stack size http://www.embedded.com/columns/technicalinsights/47101892?_requestid=27362
Yes is it OS dependent and also other things dependent. Sorry to be so vague. You may also be able to dig up some code in the gcc collection for testing stack size.
If your function was involved (directly or indirectly) in recursion, then allocating a large amount on the stack would limit the depth of recursion and might well blow the stack. Under Windows this stack reserve defaults to 1MB, though you can increase it statically with linker commands. The stack will grow as it is used, but the operating system sometimes cannot extend it. I discuss this in a little more detail on my website here.
As I have seen, a C compiler(turbo) provides a maximum size of 64000k for a variable. If we need more size, then it is declared as "huge".
It's not a good idea to try to use a massive amount of stack space.
Here is a link to the default gcc stack size: http://www.cs.nyu.edu/exact/core/doc/stackOverflow.txt
Also, you could specify --stack,xxxxx to customize the stack size, so it's best to assume xxxxx is a small number and stick with heap allocation.
Stack, heap, low, high VM -- Nuts, for the first thread, the stack at the top pf 64 bit VM, there should be no limit, so it seems like a gcc/c compiler bug that for local automatic "int x[2621440];" I get SIGSEGV. The compiler should be letting the first thread stack grow until it hits the heap, which in a 16 billion billion byte VM is pretty unlikely for now. The kindest thing is to call it a compiler "limitation". (In testing some while back, probably on a Solaris SPARC, it seemed that local variables processed faster than global ones. Go figure!)
What's the advantage of using malloc (besides the NULL return on failure) over static arrays? The following program will eat up all my ram and start filling swap only if the loops are uncommented. It does not crash.
...
#include <stdio.h>
unsigned int bigint[ 1u << 29 - 1 ];
unsigned char bigchar[ 1u << 31 - 1 ];
int main (int argc, char **argv) {
int i;
/* for (i = 0; i < 1u << 29 - 1; i++) bigint[i] = i; */
/* for (i = 0; i < 1u << 31 - 1; i++) bigchar[i] = i & 0xFF; */
getchar();
return 0;
}
...
After some trial and error I found the above is the largest static array allowed on my 32-bit Intel machine with GCC 4.3. Is this a standard limit, a compiler limit, or a machine limit? Apparently I can have as many of of them as I want. It will segfault, but only if I ask for (and try to use) more than malloc would give me anyway.
Is there a way to determine if a static array was actually allocated and safe to use?
EDIT: I'm interested in why malloc is used to manage the heap instead of letting the virtual memory system handle it. Apparently I can size an array to many times the size I think I'll need and the virtual memory system will only keep in ram what is needed. If I never write to e.g. the end (or beginning) of these huge arrays then the program doesn't use the physical memory. Furthermore, if I can write to every location then what does malloc do besides increment a pointer in the heap or search around previous allocations in the same process?
Editor's note: 1 << 31 causes undefined behaviour if int is 32-bit, so I have modified the question to read 1u. The intent of the question is to ask about allocating large static buffers.
Well, for two reasons really:
Because of portability, since some systems won't do the virtual memory management for you.
You'll inevitably need to divide this array into smaller chunks for it to be useful, then to keep track of all the chunks, then eventually as you start "freeing" some of the chunks of the array you no longer require you'll hit the problem of memory fragmentation.
All in all you'll end up implementing a lot of memory management functionality (actually pretty much reimplementing the malloc) without the benefit of portability.
Hence the reasons:
Code portability via memory management encapsulation and standardisation.
Personal productivity enhancement by the way of code re-use.
Please see:
malloc() and the C/C++ heap
Should a list of objects be stored on the heap or stack?
C++ Which is faster: Stack allocation or Heap allocation
Proper stack and heap usage in C++?
About C/C++ stack allocation
Stack,Static and Heap in C++
Of Memory Management, Heap Corruption, and C++
new on stack instead of heap (like alloca vs malloc)
with malloc you can grow and shrink your array: it becomes dynamic, so you can allocate exactly for what you need.
This is called custom memory management, I guess.
You can do that, but you'll have to manage that chunk of memory yourself.
You'd end up writing your own malloc() woring over this chunk.
Regarding:
After some trial and error I found the
above is the largest static array
allowed on my 32-bit Intel machine
with GCC 4.3. Is this a standard
limit, a compiler limit, or a machine
limit?
One upper bound will depend on how the 4GB (32-bit) virtual address space is partitioned between user-space and kernel-space. For Linux, I believe the most common partitioning scheme has a 3 GB range of addresses for user-space and a 1 GB range of addresses for kernel-space. The partitioning is configurable at kernel build-time, 2GB/2GB and 1GB/3GB splits are also in use. When the executable is loaded, virtual address space must be allocated for every object regardless of whether real memory is allocated to back it up.
You may be able to allocate that gigantic array in one context, but not others. For example, if your array is a member of a struct and you wish to pass the struct around. Some environments have a 32K limit on struct size.
As previously mentioned, you can also resize your memory to use exactly what you need. It's important in performance-critical contexts to not be paging out to virtual memory if it can be avoided.
There is no way to free stack allocation other than going out of scope. So when you actually use global allocation and VM has to alloc you real hard memory, it is allocated and will stay there until your program runs out. This means that any process will only grow in it's virtual memory use (functions have local stack allocations and those will be "freed").
You cannot "keep" the stack memory once it goes out of scope of function, it is always freed. So you must know how much memory you will use at compile time.
Which then boils down to how many int foo[1<<29]'s you can have. Since first one takes up whole memory (on 32bit) and will be (lets lie: 0x000000) the second will resolve to 0xffffffff or thereaobout. Then the third one would resolve to what? Something that 32bit pointers cannot express. (remember that stack reservations are resolved partially at compiletime, partially runtime, via offsets, how far the stack offset is pushed when you alloc this or that variable).
So the answer is pretty much that once you have int foo [1<<29] you cant have any reasonable depth of functions with other local stack variables anymore.
You really should avoid doing this unless you know what you're doing. Try to only request as much memory as you need. Even if it's not being used or getting in the way of other programs it can mess up the process its self. There are two reasons for this. First, on certain systems, particularly 32bit ones it can cause address space to be exhausted prematurely in rare circumstances. Additionally many kernels have some kind of per process limit on reserved/virtual/not in use memory. If your program asks for memory at points in run time the kernel can kill the process if it asks for memory to be reserved that exceeds this limit. I've seen programs that have either crashed or exited due to a failed malloc because they are reserving GBs of memory while only using a few MB.