Big array gives segmentation error in C - c

I am really new to C, so I am sorry if this is a absolute beginner question, but I am getting a segmentation error when I am building large array, relevant bits of what I am doing is:
unsigned long long ust_limit;
unsigned long long arr_size;
/* ust_limit gets value around here ... */
arr_size = ((ust_limit + 1) / 2) - 1;
unsigned long long numbs[(int)arr_size];
This works for some values of ust_limit, but when it gets above approximately 4.000.000 a segmentation fault occurs. What I want is to detect a possible segfault and fail gracefully. How can I know which values would cause a segmentation fault. And, is this something platform dependant?

You are most likely getting a stack overflow, since you are creating a very large array on the stack. To avoid this, allocate the memory dynamically:
unsigned long long *numbs = malloc(arr_size * sizeof(unsigned long long));
Later, when you are finished with the array, free it again:
free(numbs);

You array stores on stack frame, which has a limit to its size, use malloc instead.
unsigned long long *numbs = malloc(arr_size * sizeof(long long));
// don't forget to free after use
free(numbs)

You're consuming too much stack. The limit is platform dependent.
The exact limit depends on the OS. These limits can be changed to some extent on some operating systems .
For large amounts of memory, you should be using the head with malloc and/or calloc (and free).

numbs is a Variable Length Array (VLA).
VLAs can be created only in block scope (i.e., inside a function). They're allocated like any other local variable, typically on the stack.
Unfortunately, the language doesn't provide a way of detecting or handling allocation failures for local variables. If you allocate too much memory, your program will crash if you're lucky.
Large objects whose size isn't known at compile time should be allocated via malloc() (which means you need to keep track of the allocation and release them with free()).
Incidentally, there's no need to cast arr_size to int. And both ust_limit and arr_size should be of type size_t (defined in <stddef.h>).
Example:
unsigned long long numbs = malloc(arr_size * sizeof *numbs);
/* And later, when you're done with it */
free(numbs);

Related

structure cast into unsigned char array without malloc/str methods

I've read all other questions but response aren't helping me so much.
Once again, I'm not a developer but a wood worker trying to make something far too complicated for his own brain.
I work on a PIC its alignment is 4 bytes, all structures are define with attribute((packed)).
I've found a way to do it but it uses malloc and str/mem-cpy, methods that aren't safe for interrupt, or for malloc that I shouldn't be using at all (cf. my previous question)
Said structure contains 16 unsigned char, one s16 (2 bytes) et three s32 (4 bytes), so its 30 bytes long, with 4bytes alignement making it 32bytes long.
(char are coded on 8bits).
1) Am I wrong until here?
int len =sizeof(data_out.point[i]);
unsigned char* raw;
raw = malloc(len);
memcpy(raw, &data_out.point[i], len);
Data_res[count].pData = malloc(len);
for (int k = 0; k<len; k++)
{
Data_res[count].pData[k] = (uint8_t)(raw[k]);
}
Data_res[count].DataLen=len;
count++;
}
data_out is a stucture (members are not relevant except following one)
point is the structure described before.
Data_res is a structure that store an uchar buffer (pData) and contains lenght and status (locked, writing, etc to avoid multi-access).
1) its not working 100%, sometimes result are really strange.
2) since yesterday I understand why its bad (malloc on not shared memory, casting malloc, interruption safety, etc).
How to do the same thing without mallocs/memcpy ?
note: I need this for debug output, I was going to just let it go, but I don't like to keep things unfinished...
Don't cast the result of malloc in C. If you have to cast, you're using a C++ compiler, and you should be using new instead of malloc.
sizeof is an operator (not a function) that determines the size of objects of that type (as a size_t, not an int), so if data.point[i] is a char * for example, sizeof data.point[i] is the same value as sizeof (char *). It is a common mistake to assume sizeof determines the size of the object pointed at by a pointer; that hold true for array types, but not pointer types.
You're leaking memory pointed at by raw when your function returns. You should be using automatic storage duration rather than dynamic allocation there. i.e. unsigned char raw[len];.
You should be performing machine-independent serialisation, e.g. translating each field to a value that doesn't depend upon machine architecture, if you're planning on transporting this data to other machines.
Finally, to answer your question, the only way you can substitute Data_res[count].pData = malloc(len); seems that you need some other object (see automatic storage duration above) to point to instead, i.e. Data_res[count].pData = raw;... You might need to declare raw in the caller function, to avoid it being destroyed when the function you provided returns.

Questions about dynamic memory allocation in C

What is the difference between
int size;
int *arr;
scanf("%i", &size);
arr = malloc(size * sizeof(*arr));
and
int size;
scanf("%i", &size);
int arr[size];
When I want to allocate memory for 2 big numbers i would use the
following code:
unsigned long *big_nums;
big_nums = malloc(2 * sizeof(*big_nums));
I would access the first big bumber using big_nums[0] an the seond
one with big_nums[1]. Let's say unsigned long is 4 bytes big,
then the code would allocate 2 * 4 = 8 bytes. Let's say I do
something like this:
unsigned long *big_nums;
big_nums = malloc(7);
Using big_nums[0] is clear for me, but how about big_nums[1]? Will
it cause some kind of segmentation fault error or what?
There are two places to get memory from: the stack and the heap. The stack is where you allocate short lived things, and the heap is for allocating long term things.
malloc() allocates from the heap, and int arr[size] allocates from the stack.
When your function exits, arr[size] will be disposed of automatically, but malloc() will not. This leads to what's called "memory leaks".
big_nums = malloc(7);
will indeed be an error if you access big_nums[1]. In general the standard says behavior is "undefined" which means it could work, or might not.
For Q#1: The second version will (try to) allocate a variable-length array on the stack. In C99 onwards, this is possible; but in traditional C variable-length arrays don't exist, and you must roll them yourself using malloc.
For Q#2: You will be allowed to make that error. And when you write to the second element of the array, you will overwrite one byte that does not "belong" to you.
My guess is that in most cases, nothing bad will happen because malloc(7) will secretly be equivalent to malloc(8). But there is NO GUARANTEE of this. Anything could happen, including a segfault or something worse.
By the way, if you have two separate questions, it would be best to write them up as two separate questions. You get more points way.

Basic array usage in C?

Is this how you guys get size of an array in ANSI-C99? Seems kind of, um clunky coming from higher language.
int tests[7];
for (int i=0; i<sizeof(tests)/sizeof(int); i++) {
tests[i] = rand();
}
Also this Segmentation faults.
int r = 10000000;
printf ("r: %i\n", r);
int tests[r];
run it:
r: 10000000
Segmentation fault
10000000 seg faults, but 1000000 works.
How do I get more info out of this? What should I be checking and how would I debug something like this? Is there a limit on C arrays? What's a segmentation fault?
Getting size of an array in C is easy. This will give you the size of array in bytes.
sizeof(x)
But I guess what you require is number of elements, in that case it would be:
sizeof(x) / sizeof(x[0])
You can write a simple macro for this:
#define NumElements(x) (sizeof(x) / sizeof(x[0]))
For example:
int a[10];
int size_a = sizeof(a); /* size in bytes */
int numElm = NumElements(a); /* number of elements, here 10 */
Why calculate the size?
Define a constant containing the size and use that when declaring the array. Reference the constant whenever you want the size of the array.
As a primarily C++ programmer, I'll say that historically the constant was often defined as an enum value or a #define. In C, that may be current rather than historic, though - I don't know how current C handles "const".
If you really want to calculate the size, define a macro to do it. There may even be a standard one.
The reason for the segfault is most likely because the array you're trying to declare is about 40 megabytes worth, and is declared as a local variable. Most operating systems limit the size of the stack. Keep your array on the heap or in global memory, and 40 megabytes for one variable will probably be OK for most systems, though some embedded systems may still cry foul. In a language like Java, all objects are on the heap, and only references are kept on the stack. This is a simple and flexible system, but often much less efficient than storing data on the stack (heap allocation overheads, avoidable heap fragmentation, indirect access overheads...).
Arrays in C don't know how big they are, so yes, you have to do the sizeof array / sizeof array[0] trick to get the number of elements in an array.
As for the segfault issue, I'm guessing that you exceeded your stack size by attempting to allocate 10000000 * sizeof int bytes. A rule of thumb is that if you need more than a few hundred bytes, allocate it dynamically using malloc or calloc instead of trying to create a large auto variable:
int r = 10000000;
int *tests = malloc(sizeof *test * r);
Note that you can treat tests as though it were an array type in most circumstances (i.e., you can subscript it, you can pass it to any function that expects an array, etc.), but it is not an array type; it is a pointer type, so the sizeof tests / sizeof tests[0] trick won't work.
Traditionally, an array has a static size. So we can do
#define LEN 10
int arr[LEN];
but not
int len;
scanf("%d", &len);
int arr[len]; // bad!
Since we know the size of an array at compile time, getting the size of an array tends to trivial. We don't need sizeof because we can figure out the size by looking at our declaration.
C++ provides heap arrays, as in
int len;
scanf("%d", &len);
int *arr = new int[len];
but since this involves pointers instead of stack arrays, we have to store the size in a variable which we pass around manually.
I suspect that it is because of integer overflow. Try printing the value using a printf:
printf("%d", 10000000);
If it prints a negative number - that is the issue.
Stack Overflow! Try allocating on the heap instead of the stack.

How to get memory block length after malloc?

I thought that I couldn't retrieve the length of an allocated memory block like the simple .length function in Java. However, I now know that when malloc() allocates the block, it allocates extra bytes to hold an integer containing the size of the block. This integer is located at the beginning of the block; the address actually returned to the caller points to the location just past this length value. The problem is, I can't access that address to retrieve the block length.
#include <stdlib.h>
#include <stdio.h>
int main(void)
{
char *str;
str = (char*) malloc(sizeof(char)*1000);
int *length;
length = str-4; /*because on 32 bit system, an int is 4 bytes long*/
printf("Length of str:%d\n", *length);
free(str);
}
**Edit:
I finally did it. The problem is, it keeps giving 0 as the length instead of the size on my system is because my Ubuntu is 64 bit. I changed str-4 to str-8, and it works now.
If I change the size to 2000, it produces 2017 as the length. However, when I change to 3000, it gives 3009. I am using GCC.
You don't have to track it by your self!
size_t malloc_usable_size (void *ptr);
But it returns the real size of the allocated memory block!
Not the size you passed to malloc!
What you're doing is definitely wrong. While it's almost certain that the word just before the allocated block is related to the size, even so it probably contains some additional flags or information in the unused bits. Depending on the implementation, this data might even be in the high bits, which would cause you to read the entirely wrong length. Also it's possible that small allocations (e.g. 1 to 32 bytes) are packed into special small-block pages with no headers, in which case the word before the allocated block is just part of another block and has no meaning whatsoever in relation to the size of the block you're examining.
Just stop this misguided and dangerous pursuit. If you need to know the size of a block obtained by malloc, you're doing something wrong.
I would suggest you create your own malloc wrapper by compiling and linking a file which defines my_malloc() and then overwiting the default as follows:
// my_malloc.c
#define malloc(sz) my_malloc(sz)
typedef struct {
size_t size;
} Metadata;
void *my_malloc(size_t sz) {
size_t size_with_header = sz + sizeof(Metadata);
void* pointer = malloc(size_with_header);
// cast the header into a Metadata struct
Metadata* header = (Metadata*)pointer;
header->size = sz;
// return the address starting after the header
// since this is what the user needs
return pointer + sizeof(Metadata);
}
then you can always retrieve the size allocated by subtracting sizeof(Metadata), casting that pointer to Metadata and doing metadata->size:
Metadata* header = (Metadata*)(ptr - sizeof(Metadata));
printf("Size allocated is:%lu", header->size); // don't quote me on the %lu ;-)
You're not supposed to do that. If you want to know how much memory you've allocated, you need to keep track of it yourself.
Looking outside the block of memory returned to you (before the pointer returned by malloc, or after that pointer + the number of bytes you asked for) will result in undefined behavior. It might work in practice for a given malloc implementation, but it's not a good idea to depend on that.
This is not Standard C. However, it is supposed to work on Windows operatings systems and might to be available on other operating systems such as Linux (msize?) or Mac (alloc_size?), as well.
size_t _msize( void *memblock );
_msize() returns the size of a memory block allocated in the heap.
See this link:
http://msdn.microsoft.com/en-us/library/z2s077bc.aspx
This is implementation dependent
Every block you're allocating is precedeed by a block descriptor. Problem is, it dependends on system architecture.
Try to find the block descriptor size for you own system. Try take a look at you system malloc man page.

Segmentation fault due to lack of memory in C

This code gives me segmentation fault about 1/2 of the time:
int main(int argc, char **argv) {
float test[2619560];
int i;
for(i = 0; i < 2619560; i++)
test[i] = 1.0f;
}
I actually need to allocate a much larger array, is there some way of allowing the operating system to allow me get more memory?
I am using Linux Ubuntu 9.10
You are overflowing the default maximum stack size, which is 8 MB.
You can either increase the stack size - eg. for 32 MB:
ulimit -s 32767
... or you can switch to allocation with malloc:
float *test = malloc(2619560 * sizeof test[0]);
Right now you're allocating (or at least trying to) 2619560*sizeof(float) bytes on the stack. At least in most typical cases, the stack can use only a limited amount of memory. You might try defining it static instead:
static float test[2619560];
This gets it out of the stack, so it can typically use any available memory instead. In other functions, defining something as static changes the semantics, but in the case of main it makes little difference (other than the mostly theoretical possibility of a recursive main).
Don't put such a large object on the stack. Instead, consider storing it in the heap, by allocation with malloc() or its friends.
2.6M floats isn't that many, and even on a 32-bit system you should be ok for address space.
If you need to allocate a very large array, be sure to use a 64-bit system (assuming you have enough memory!). 32-bit systems can only address about 3G per process, and even then you can't allocate it all as a single contigous block.
It is the stack overflower.
You'd better to use malloc function to get memory larger than stack size which you can get it from "ulimit -s".

Resources