This code gives me segmentation fault about 1/2 of the time:
int main(int argc, char **argv) {
float test[2619560];
int i;
for(i = 0; i < 2619560; i++)
test[i] = 1.0f;
}
I actually need to allocate a much larger array, is there some way of allowing the operating system to allow me get more memory?
I am using Linux Ubuntu 9.10
You are overflowing the default maximum stack size, which is 8 MB.
You can either increase the stack size - eg. for 32 MB:
ulimit -s 32767
... or you can switch to allocation with malloc:
float *test = malloc(2619560 * sizeof test[0]);
Right now you're allocating (or at least trying to) 2619560*sizeof(float) bytes on the stack. At least in most typical cases, the stack can use only a limited amount of memory. You might try defining it static instead:
static float test[2619560];
This gets it out of the stack, so it can typically use any available memory instead. In other functions, defining something as static changes the semantics, but in the case of main it makes little difference (other than the mostly theoretical possibility of a recursive main).
Don't put such a large object on the stack. Instead, consider storing it in the heap, by allocation with malloc() or its friends.
2.6M floats isn't that many, and even on a 32-bit system you should be ok for address space.
If you need to allocate a very large array, be sure to use a 64-bit system (assuming you have enough memory!). 32-bit systems can only address about 3G per process, and even then you can't allocate it all as a single contigous block.
It is the stack overflower.
You'd better to use malloc function to get memory larger than stack size which you can get it from "ulimit -s".
Related
I am having a C program where I declare an 2d array with the size of 8,388,608 * 23. When I run the program I get the following error:
[1] 12142 segmentation fault (core dumped)
I think that the size of the array is to big.
Here is my code:
int a[8388608][23];
a[0][0] = 10;
You likely declared int a[8388608][23]; inside a function, and the C implementation attempted to allocate space on the stack.
In common C implementations on macOS, Linux, and Windows, the space designated for the stack by default ranges from 1 MiB to 8 MiB (8,388,608 bytes), depending on the operating system and whether it is a main thread or a spawned thread. Since your array exceeded the space for the stack, using it accessed memory not mapped for your process and generated a segmentation fault.
The C standard requires an implementation to have sufficient memory to execute at least some programs (C 2018 5.2.4.1) but allows there to be a limit on the memory available and does not require an implementation to provide any warning or error handling when a program exceeds the limit. It allows a program to fail and abort.
The stack size for a program can be set through linker options. However, it is generally best not to use the stack for large amounts of data. If a program needs an array throughout its entire execution, it can allocated statically by defining it outside of any function. The amount of memory needed will then be computed during link time and reserved when the program is loaded.
When a function needs a large amount of memory temporarily, it should be allocated dynamically. You can do this with malloc:
int (*a)[23] = malloc(8388608 * sizeof *a);
if (!a)
{
fprintf(stderr, "Error, unable to allocate memory.\n");
exit(EXIT_FAILURE);
}
When the function is done with the memory, it should release it with free(a);.
Actually there are no limit except computers or servers RAM from the memory perspective.
From another side compiler will not give permission to set size of each field more than long.
int a[(1<<31)] // ok
int a[(1<<63ll)] // ok
int a[(1<<70ll)] // not ok, since in c there is no value greater than long long from integer types
I'm trying to calculate stack memory used in my program.
should i add 4 for each integer i have defined?
what about something like char str[128], shoudld i add 128 or 129?
#define ARRAY1_LIMIT 200
#define ARRAY2_LIMIT 100
char* array1[ARRAY1_LIMIT];
char* array2[ARRAY2_LIMIT];
int i = 0;
int j = 0
array1[i] = (char *)malloc(sizeof(char)*5);
array2[j] = (char *)malloc(sizeof(char)*10);
I know that the heap memory is 5+15 = 15, but i don't know how to calculate the stack memory? is it 200 + 100?
If you're doing anything more than simply trying to learn about stack usage and how variable allocations affect it, you'll want to use a stack depth analysis tool. Such a tool can help you determine if your program could possibly overflow its stack under any possible sequence of events (excepting unexpected or unbounded recursion). You can write your own (I have, in C#, for embedded programs compiled in C for M16C and MIPS targets using GCC and IAR compilers), but it's really complex and not something for beginners to attempt.
Look for a "stack usage analyzer" or "stack usage analysis tool" for your particular processor and toolchain (e.g. x86/x64/ARM/etc and GCC/VisualStudio/IAR/etc).
If you're using GCC, you may be able to use the -fstack-usage option, but that only gives you the maximum stack usage on a per-function basis. By itself that's not terribly helpful, since to verify that a program won't blow its stack, you have to recursively walk the calltree to see what the maximum stack depth could be at any level of the call tree. If you also use the -Wstack-usage option, you can get a warning if any subprogram's stack usage can possibly exceed a specified stack depth, which is more useful than the information you get with merely the -fstack-usage option.
If you're trying to work out the amount of 'space allocated to those variables in a function, use the sizeof() operator.
Notice that the arrays are arrays of char * which are pointers to characters not characters.
#include <stdio.h>
int main(void) {
size_t total=0;
size_t first_array_size=sizeof(char *[200]);
printf("first array: %zu\n",first_array_size);
total+=first_array_size;
size_t second_array_size=sizeof(char *[100]);
printf("second array: %zu\n",second_array_size);
total+=second_array_size;
size_t int_size=sizeof(int);
printf("int size: %zu * 2 = %zu\n",int_size,int_size*2);
total+=int_size*2;
printf("total=%zu\n",total);
return 0;
}
Typical output (on a 64-bit architecture):
first array: 1600
second array: 800
int size: 4 * 2 = 8
total=2408
Results may vary.
Footnote: It's also worth understanding that the amount of space allocated to the stack frame may be greater.
For example arguments passed in are typically copied to the stack as is the return value along with a pointer representing the execution point to return to after the function is called.
There's also the complexity of alignment. For example on many modern machines space may be left between variables to make sure they're aligned. However optimizations may take some of that space back depending on how it re-orders the variables. It's also possible that values (particularly integer values) may be allocated to registers and not take up space on the stack.
Finally an implementation could in principle allocate the arrays on the heap. That's certainly a possible implementation for variable length arrays but I'm not aware of it being strictly disallowed for fixed size arrays.
#include <stdio.h>
#include <stdlib.h>
int main(void)
{
int *arr = (int*)malloc(10);
int i;
for(i=0;i<100;i++)
{
arr[i]=i;
printf("%d", arr[i]);
}
return 0;
}
I am running above program and a call to malloc will allocate 10 bytes of memory and since each int variable takes up 2 bytes so in a way I can store 5 int variables of 2 bytes each thus making up my total 10 bytes which I dynamically allocated.
But on making a call to for-loop it is allowing me to enter values even till 99th index and storing all these values as well. So in a way if I am storing 100 int values it means 200 bytes of memory whereas I allocated only 10 bytes.
So where is the flaw with this code or how does malloc behave? If the behaviour of malloc is non-deterministic in such a manner then how do we achieve proper dynamic memory handling?
The flaw is in your expectations. You lied to the compiler: "I only need 10 bytes" when you actually wrote 100*sizeof(int) bytes. Writing beyond an allocated area is undefined behavior and anything may happen, ranging from nothing to what you expect to crashes.
If you do silly things expect silly behaviour.
That said malloc is usually implemented to ask the OS for chunks of memory that the OS prefers (like a page) and then manages that memory. This speeds up future mallocs especially if you are using lots of mallocs with small sizes. It reduces the number of context switches that are quite expensive.
First of all, in the most Operating Systems the size of int is 4 bytes. You can check that with:
printf("the size of int is %d\n", sizeof(int));
When you call the malloc function you allocate size at heap memory. The heap is a set aside for dynamic allocation. There's no enforced pattern to the allocation and deallocation of blocks from the heap; you can allocate a block at any time and free it at any time. This makes it much more complex to keep track of which parts of the heap are allocated or free at any given time. Because your program is small and you have no collision in the heap you can run this for with more values that 100 and it runs too.
When you know what are you doing with malloc then you build programs with proper dynamic memory handling. When your code has improper malloc allocation then the behaviour of the program is "unknown". But you can use gdb debugger to find where the segmentation will be revealed and how the things are in heap.
malloc behaves exactly as it states, allocates n number bytes of memory, nothing more. Your code might run on your PC, but operating on non-allocated memory is undefined behavior.
A small note...
Int might not be 2 bytes, it varies on different architectures/SDKs. When you want to allocate memory for n integer elements, you should use malloc( n * sizeof( int ) ).
All in short, you manage dynamic memory with other tools that the language provides ( sizeof, realloc, free, etc. ).
C doesn't do any bounds-checking on array accesses; if you define an array of 10 elements, and attempt to write to a[99], the compiler won't do anything to stop you. The behavior is undefined, meaning the compiler isn't required to do anything in particular about that situation. It may "work" in the sense that it won't crash, but you've just clobbered something that may cause problems later on.
When doing a malloc, don't think in terms of bytes, think in terms of elements. If you want to allocate space for N integers, write
int *arr = malloc( N * sizeof *arr );
and let the compiler figure out the number of bytes.
In my algorithm I know work with static arrays, no dynamic ones. But I sometimes
reach the limit of the stack. Am I right, that static arrays are stored to the stack?
Which parameters affect my maximum stack size for one C programm?
Are there many system parameters which affect the maximal array size? Does the maximunm no. of elements depend of the array type? Does it depend on the total system RAM? Or does every C programm have a static maximum stack size?
Am I right, that static arrays are stored to the stack?
No, static arrays are stored in the static storage area. The automatic ones (i.e. ones declared inside functions, with no static storage specifier) are allocated on the stack.
Which parameters affect my maximum stack size for one C program?
This is system-dependent. On some operating systems you can change stack size programmatically.
Running out of stack space due to automatic storage allocation is a clear sign that you need to reconsider your memory strategy: you should either allocate the buffer in the static storage area if re-entrancy is not an issue, or use dynamic allocation for the largest of your arrays.
Actually, it depends on the C compiler for the platform you use.
As an example, there are even systems which don't have a real stack so recursion won't work.
A static array is compiled as a continuous memory area with pointers. The pointers might be two or four bytes in size (or maybe even only one on exotic platforms).
There are platforms which use memory pages which have "near" and "far" pointers which differ in size (and speed, of course). So it could be the case that the pointers representing the array and the objects need to fit into the same memory page.
On embedded systems, static data usually is collected in the memory area which will later be represented by the read-only memory. So your array will have to fit in there.
On platforms which run arbitrary applications, RAM is the limiting factor if none of the above applies.
Most of your questions have been answered, but just to give an answer that made my life a lot easier:
Qualitatively the maximum size of the non-dynamically allocated array depends on the amount of RAM that you have. Also it depends on the type of the array, e.g. an int may be 4 bytes while a double may be 8 bytes (they are also system dependent), thus you will be able to have an array that is double in number of elements if you use int instead of double.
Having said that and keeping in mind that sometimes numbers are indeed important, here is a very noobish code snippet to help you extract the maximum number in your system.
#include <stdio.h>
#include <stdlib.h>
#define UPPER_LIMIT 10000000000000 // a very big number
int main (int argc, const char * argv[])
{
long int_size = sizeof(int);
for (int i = 1; i < UPPER_LIMIT; i++)
{
int c[i];
for (int j = 0; j < i; j++)
{
c[j] = j;
}
printf("You can set the array size at %d, which means %ld bytes. \n", c[i-1], int_size*c[i-1]);
}
}
P.S.: It may take a while until you reach your system's maximum and produce the expected Segmentation Fault, so you may want to change the initial value of i to something closer to your system's RAM, expressed in bytes.
I have a question regarding memory allocation order.
In the following code I allocate in a loop 4 strings.
But when I print the addresses they don't seem to be allocated one after the other... Am I doing something wrong or is it some sort of defense mechanism implemented by the OS to prevent possible buffer overflows? (I use Windows Vista).
Thank you.
char **stringArr;
int size=4, i;
stringArr=(char**)malloc(size*sizeof(char*));
for (i=0; i<size; i++)
stringArr[i]=(char*)malloc(10*sizeof(char));
strcpy(stringArr[0], "abcdefgh");
strcpy(stringArr[1], "good-luck");
strcpy(stringArr[2], "mully");
strcpy(stringArr[3], "stam");
for (i=0; i<size; i++) {
printf("%s\n", stringArr[i]);
printf("%d %u\n\n", &(stringArr[i]), stringArr[i]);
}
Output:
abcdefgh
9650064 9650128
good-luck
9650068 9638624
mully
9650072 9638680
stam
9650076 9638736
Typically when you request memory through malloc(), the C runtime library will round the size of your request up to some minimum allocation size. This makes sure that:
the runtime library has room for its bookkeeping information
it's more efficient for the runtime library to manage allocated blocks that are all multiples of some size (such as 16 bytes)
However, these are implementation details and you can't really rely on any particular behaviour of malloc().
But when I print the addresses they don't seem to be allocated one after the other...
So?
Am I doing something wrong or is it some sort of defense mechanism implemented by the OS to prevent possible buffer overflows?
Probably "neither".
Just out of interest, what addresses do you get?
You shouldn't depend on any particular ordering or spacing of values returned by malloc. It behaves in mysterious and unpredictable ways.
Typically it is reasonable to expect that a series of chronological allocations will result in a memory addresses that are somehow related, but as others have pointed out, it is certainly not a requirement of the heap manager. In this particular case, though, it is possible that you are seeing results of the low fragmentation heap. Windows keeps lists of small chunks of memory that can quickly satisfy a request. These could be in any order.
You can't depend on malloc to give you contiguous addresses. It's entirely up to the implementation and presumably the current state of the heap; some implementations may, many won't.
If you need the addresses to be contiguous, allocate one large block of memory and set up your pointers to point to different areas within it.
As others have mentioned, there is no standard to specify in which order the memory blocks allocated by malloc() should be located in the memory. For example, freed blocks can be scattered all around the heap and they may be re-used in any order.
But even if the blocks happen to be one after each other, they most likely do not form a contiguous block. In order to reduce fragmentation, the heap manager only allocates blocks of specific size, for example power of two (64, 128, 256, 512 etc. bytes). So, if you reserve 10 bytes for a string, there may be perhaps 22 or 54 un-used bytes after that.
The memory overhead is another reason why it is not good idea to use dynamic memory allocation unless really necessary. It is much easier and safer just to use a static array.
Since you are interesting in knowing the addresses returned by malloc(), you should make sure you are printing them properly. By "properly", I mean that you should use the right format specifier for printf() to print addresses. Why are you using "%u" for one and "%d" for another?
You should use "%p" for printing pointers. This is also one of the rare cases where you need a cast in C: because printf() is a variadic function, the compiler can't tell that the pointers you're passing as an argument to it need to be of the type void * or not.
Also, you shouldn't cast the return value of malloc().
Having fixed the above, the program is:
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
int main(void)
{
char **stringArr;
int size=4, i;
stringArr = malloc(size * sizeof *stringArr);
for (i=0; i < size; i++)
stringArr[i] = malloc(10 * sizeof *stringArr[i]);
strcpy(stringArr[0], "abcdefgh");
strcpy(stringArr[1], "good-luck");
strcpy(stringArr[2], "mully");
strcpy(stringArr[3], "stam");
for (i=0; i<size; i++) {
printf("%s\n", stringArr[i]);
printf("%p %p\n", (void *)(&stringArr[i]), (void *)(stringArr[i]));
}
return 0;
}
and I get the following output when I run it:
abcdefgh
0x100100080 0x1001000a0
good-luck
0x100100088 0x1001000b0
mully
0x100100090 0x1001000c0
stam
0x100100098 0x1001000d0
On my computer, char ** pointers are 8 bytes long, so &stringArr[i+1] is at 8 bytes greater than &stringArr[i]. This is guaranteed by the standard: If you malloc() some space, that space is contiguous. You allocated space for 4 pointers, and the addresses of those four pointers are next to each other. You can see that more clearly by doing:
printf("%d\n", (int)(&stringArr[1] - &stringArr[0]));
This should print 1.
About the subsequent malloc()s, since each stringArr[i] is obtained from a separate malloc(), the implementation is free to assign any suitable addresses to them. On my implementation, with that particular run, the addresses are all 0x10 bytes apart.
For your implementation, it seems like char ** pointers are 4 bytes long.
About your individual string's addresses, it does look like malloc() is doing some sort of randomization (which it is allowed to do).