I have a programs like this,
Program 1
#include <stdio.h>
#include <string.h>
#define ARRAY_SIZE 0x320204
int main ()
{
unsigned long long_array[ARRAY_SIZE];
memset (long_array, 0, ARRAY_SIZE);
return 0;
}
Program 2
#include <stdio.h>
#include <string.h>
#define ARRAY_SIZE 0x320204
int main ()
{
unsigned long long_array[ARRAY_SIZE] = {0};
//memset (long_array, 0, ARRAY_SIZE);
return 0;
}
Both Programs are giving Segmentation Fault.
Please clarify how to initialize this long size array. Thanks in Advance.
The segmentation fault is because you try to allocate on the stack 0x320204 longs, which may be too much. In decimal, this is 3277316 * sizeof(unsigned long), which on my machine is 3277316 * 8 = 26218528 bytes, i.e. around 25 Mbytes.
The second way of initializing it is the most common, it will zero all elements. The first method will only zero the first ARRAY_SIZE bytes, but remember that sizeof(unsigned long) is usually 8 bytes, so you need to zero sizeof(unsigned long) * ARRAY_SIZE bytes.
What you could do is use calloc, which is only one call, clearer to a lot of people, though not really a big deal, and uses the heap, which is the biggest benefit. Once you want to make really big array (or objects, or whatever), the heap is the place for you. Eventually, using the stack for this kind of stuff isn't practical, and will likely cause a stackoverflow error. Plus, you can pass this pointer returned from calloc to anything with having to worry what will happen to that memory allocated. This is unlike the stack, where it will probably be overwritten once that function returns, something that has happened to me plenty of times, creating bugs that are super hard to fix. There are downsides, however, to calloc like the fact that its pretty slow, but just as fast as memsetin this case. But if you don't really NEED all the bits to be set to 0, you could use malloc (much faster and more popular). The second downside to calloc or malloc is that you must call free on the pointers, otherwise you leak memory. Here's how to use calloc:
#include <stdlib.h> // or "#include <cstdlib>" in C++
/* Allocate space for array, set all bytes to 0,
and return pointer. Then assign to long_array.
*/
unsigned long* long_array = (unsigned long*) calloc(ARRAY_SIZE);
// When you're done with this variable:
free(long_array);
Easy as that. Both calloc and malloc are made for this kind of stuff. I hope this helped!
Note: Though these links are to a C++ website, you can still use these functions in pure C by including <stdlib.h>
Related
I'm implementing a sequential program for sorting like quicksort. I would like to test the performance of my program in a huge array of 1 or 10 billions of integers.
But the problem is that I obtain a segmentation error due to the size of the array.
A sample code of declaration of this array:
#include <stdio.h>
#include <stdlib.h>
#include <time.h>
#define N 1000000000
int main(int argc, char **argv)
{
int list[N], i;
srand(time(NULL));
for(i=0; i<N; i++)
list[i] = rand()%1000;
return 0;
}
I got a proposition to use mmap function. But I don't know how to use it ? can anybody help me to use it ?
I'm working on Ubuntu 10.04 64-bit, gcc version 4.4.3.
Thanks for your replies.
Michael is right, you can't fit that much on the stack. However, you can make it global (or static) if you don't want to malloc it.
#include <stdlib.h>
#include <time.h>
#define N 1000000000
static int list[N];
int main(int argc, char **argv)
{
size_t i;
srand(time(NULL));
for(i=0; i<N; i++)
list[i] = rand()%1000;
return 0;
}
You must use malloc for this sort of allocation. That much on the stack will fail nearly every time.
int *list;
list = malloc(N * sizeof(int));
This puts the allocation on the heap where there is a lot more memory available.
You probably don't create so large an array and if you do you certainly don't create it on the stack; the stack just isn't that big.
If you have a 32-bit address space and a 4-byte int, then you can't create an array with a billion ints; there just won't be enough contiguous space in memory for that large an object (there probably won't be enough contiguous space for an object a fraction of that size). If you have a 64-bit address space, you might get away with allocating that much space.
If you really want to try, you'll need either to create it statically (i.e., declare the array at file scope or with the static qualifier in the function) or dynamically (using malloc).
On linux systems malloc of very large chunks just does a mmap under the hood, so it is perhaps too tedious to look into that.
Be careful that you don't have neither overflow (signed integers) nor silent wrap (unsigned integers) for your array bounds and indices. Use size_t as a type for that, since you are on a 64bit machine, this then should work.
But as a habit you should definitively check your bounds against SIZE_MAX, something like assert(N*sizeof(data[0]) <= SIZE_MAX), to be sure.
The stack allocations makes it break. N=1Gig ints => 4Gig of memory (both with a 32-bit and a 64-bit compiler). But
if you want to measure the performance of quicksort, or a similar algorithm of yours, this is not the way to go about it.
Try instead to use multiple quicksorts in sequence on prepared samples with a large size.
-create a large random sample not more than half your available memory.
make sure it doesn''t fill your ram!
If it does all measuring efforts are in vain.
500 M elements is more than enough on a 4 gig system.
-decide on a test size ( e.g. N = 100 000 elements)
-start timer
--- do the algoritm for ( *start # i*N, *end # (i+1)*N)
(rinse repeat for next i until the large random sample is depleted)
-end timer
Now you have a very precise answer to how much time your algorithm has consumed. Run it a few times to get a feel of "how precise" (use a new srand(seed) seed each time). And change the N for more inspection.
Another option is to dynamically allocate a linked list of smaller arrays. You'll have to wrap them with accessor functions, but it's far more likely that you can grab 16 256 MB chunks of memory than a single 4 GB chunk.
typedef struct node_s node, *node_ptr;
struct node_s
{
int data[N/NUM_NODES];
node_ptr next;
};
I would like to ask you a question. I have the following code:
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#define XXX 1024*1024
int main()
{
int *p;
unsigned long x=0;
while (1)
{
//p = (int *) calloc (1,XXX);
p = (int *) malloc (XXX);
memset (p,0,XXX);
x++;
printf ("%lu MB allocated.\n",x);
sleep (1);
}
return 0;
}
If I run this code, everything runs as normal. Every second, a new MB is allocated in the memory. The problem I encounter is if I uncomment the calloc() line and comment the malloc() and memset() lines. From what I know, calloc() should initialize all bytes to zero in the allocated memory; the same thing that malloc() and memset() do.
When I run the code with calloc() (without the malloc() and memset()), an initial 1 MB is allocated (as it is normal), and then after some seconds (~10) another MB is allocated.
Why is this behaviour?
Thanks in advance!
From what I know, calloc() should initialize all bytes to zero in the allocated memory.
This is partly true based on my understanding of the calloc call.
It reserves the space but doesn't initialise all memory to zero. It will often or generally initialise one section to zero, and point all others to that; when memory is then modified or accessed within this block, it will initialise it to zero before using. It means that a calloc call of very large size doesn't set all of that memory to zero multiple times, but only when it actually needs to.
tl;dr: it's an OS theory trick where kernels will cheat. There's a longer description here: https://stackoverflow.com/a/2688522/2441252.
I am really new to C, so I am sorry if this is a absolute beginner question, but I am getting a segmentation error when I am building large array, relevant bits of what I am doing is:
unsigned long long ust_limit;
unsigned long long arr_size;
/* ust_limit gets value around here ... */
arr_size = ((ust_limit + 1) / 2) - 1;
unsigned long long numbs[(int)arr_size];
This works for some values of ust_limit, but when it gets above approximately 4.000.000 a segmentation fault occurs. What I want is to detect a possible segfault and fail gracefully. How can I know which values would cause a segmentation fault. And, is this something platform dependant?
You are most likely getting a stack overflow, since you are creating a very large array on the stack. To avoid this, allocate the memory dynamically:
unsigned long long *numbs = malloc(arr_size * sizeof(unsigned long long));
Later, when you are finished with the array, free it again:
free(numbs);
You array stores on stack frame, which has a limit to its size, use malloc instead.
unsigned long long *numbs = malloc(arr_size * sizeof(long long));
// don't forget to free after use
free(numbs)
You're consuming too much stack. The limit is platform dependent.
The exact limit depends on the OS. These limits can be changed to some extent on some operating systems .
For large amounts of memory, you should be using the head with malloc and/or calloc (and free).
numbs is a Variable Length Array (VLA).
VLAs can be created only in block scope (i.e., inside a function). They're allocated like any other local variable, typically on the stack.
Unfortunately, the language doesn't provide a way of detecting or handling allocation failures for local variables. If you allocate too much memory, your program will crash if you're lucky.
Large objects whose size isn't known at compile time should be allocated via malloc() (which means you need to keep track of the allocation and release them with free()).
Incidentally, there's no need to cast arr_size to int. And both ust_limit and arr_size should be of type size_t (defined in <stddef.h>).
Example:
unsigned long long numbs = malloc(arr_size * sizeof *numbs);
/* And later, when you're done with it */
free(numbs);
I am trying to create an array of size 2^25 in c and then perform some elementary operations on it (memsweep function). The c code is
#include <stdio.h>
#include <time.h>
#define S (8191*4096)
main()
{
clock_t start = clock();
unsigned i;
volatile char large[S];
for (i = 0; i < 10*S; i++)
large[(4096*i+i)%S]=1+large[i%S];
printf("%f\n",((double)clock()-start)/CLOCKS_PER_SEC);
}
I am able to compile it but on execution it gives segmentation fault.
That might be bigger than your stack. You can
Make large global
Use malloc
The array is too big to fit on your stack. Use the heap with char *large = malloc(S) instead.
You don't have that much stack space to allocate an array that big ... on Linux for instance, the stack-size is typically 8192 bytes. You've definitely exceeded that.
The best option would be to allocate the memory on the heap using malloc(). So you would write char* large = malloc(S);. You can still access the array using the [] notation.
Optionally, if you're on Linux, you could on the commandline call sudo ulimit -s X, where X is some number large enough for your array to fit on the stack ... but I'd generally discourage that solution.
Large is being allocated on the stack and you are overflowing it.
Try using char *large = malloc(S)
I'm implementing a sequential program for sorting like quicksort. I would like to test the performance of my program in a huge array of 1 or 10 billions of integers.
But the problem is that I obtain a segmentation error due to the size of the array.
A sample code of declaration of this array:
#include <stdio.h>
#include <stdlib.h>
#include <time.h>
#define N 1000000000
int main(int argc, char **argv)
{
int list[N], i;
srand(time(NULL));
for(i=0; i<N; i++)
list[i] = rand()%1000;
return 0;
}
I got a proposition to use mmap function. But I don't know how to use it ? can anybody help me to use it ?
I'm working on Ubuntu 10.04 64-bit, gcc version 4.4.3.
Thanks for your replies.
Michael is right, you can't fit that much on the stack. However, you can make it global (or static) if you don't want to malloc it.
#include <stdlib.h>
#include <time.h>
#define N 1000000000
static int list[N];
int main(int argc, char **argv)
{
size_t i;
srand(time(NULL));
for(i=0; i<N; i++)
list[i] = rand()%1000;
return 0;
}
You must use malloc for this sort of allocation. That much on the stack will fail nearly every time.
int *list;
list = malloc(N * sizeof(int));
This puts the allocation on the heap where there is a lot more memory available.
You probably don't create so large an array and if you do you certainly don't create it on the stack; the stack just isn't that big.
If you have a 32-bit address space and a 4-byte int, then you can't create an array with a billion ints; there just won't be enough contiguous space in memory for that large an object (there probably won't be enough contiguous space for an object a fraction of that size). If you have a 64-bit address space, you might get away with allocating that much space.
If you really want to try, you'll need either to create it statically (i.e., declare the array at file scope or with the static qualifier in the function) or dynamically (using malloc).
On linux systems malloc of very large chunks just does a mmap under the hood, so it is perhaps too tedious to look into that.
Be careful that you don't have neither overflow (signed integers) nor silent wrap (unsigned integers) for your array bounds and indices. Use size_t as a type for that, since you are on a 64bit machine, this then should work.
But as a habit you should definitively check your bounds against SIZE_MAX, something like assert(N*sizeof(data[0]) <= SIZE_MAX), to be sure.
The stack allocations makes it break. N=1Gig ints => 4Gig of memory (both with a 32-bit and a 64-bit compiler). But
if you want to measure the performance of quicksort, or a similar algorithm of yours, this is not the way to go about it.
Try instead to use multiple quicksorts in sequence on prepared samples with a large size.
-create a large random sample not more than half your available memory.
make sure it doesn''t fill your ram!
If it does all measuring efforts are in vain.
500 M elements is more than enough on a 4 gig system.
-decide on a test size ( e.g. N = 100 000 elements)
-start timer
--- do the algoritm for ( *start # i*N, *end # (i+1)*N)
(rinse repeat for next i until the large random sample is depleted)
-end timer
Now you have a very precise answer to how much time your algorithm has consumed. Run it a few times to get a feel of "how precise" (use a new srand(seed) seed each time). And change the N for more inspection.
Another option is to dynamically allocate a linked list of smaller arrays. You'll have to wrap them with accessor functions, but it's far more likely that you can grab 16 256 MB chunks of memory than a single 4 GB chunk.
typedef struct node_s node, *node_ptr;
struct node_s
{
int data[N/NUM_NODES];
node_ptr next;
};