How to allocate a large memory in C [closed] - c

Question

Closed. This question needs debugging details. It is not currently accepting answers.
Edit the question to include desired behavior, a specific problem or error, and the shortest code necessary to reproduce the problem. This will help others answer the question.
Closed 5 years ago.
Improve this question
I'm trying to write a program that can obtain a 1GB memory from the system by malloc(1024*1024*1024).
After I got the start address of the memory, In my limited understanding, if I want to initialize it, just using memset() to achieve. But the truth is there will trigger a segfault after a while.
And I tried using gdb to find where cause it, finally found if I do some operate of memory more than 128 MB will lead to this fault.
Is there has any rule that limits program just can access memory less than 128 MB? Or I used the wrong way to allocate and initialize it?
If there is a need for additional information, please tell me.
Any suggestion will be appreciated.
[Platform]
Linux 4.10.1 with gcc 5.4.0
Build program with gcc test.c -o test
CPU: Intel i7-6700
RAM: 16GB
[Code]
size_t mem_size = 1024 * 1024 * 1024;
...
void *based = malloc(mem_size); //mem_size = 1024^3
int stage = 65536;
int initialized = 0;
if (based) {
printf("Allocated %zu Bytes from %lx to %lx\n", mem_size, based, based + mem_size);
} else {
printf("Error in allocation.\n");
return 1;
}
int n = 0;
while (initialized < mem_size) { //initialize it in batches
printf("%6d %lx-%lx\n", n++, based+initialized, based+initialized+stage);
memset(based + initialized, '$', stage);
initialized += stage;
}
[Result]
Allocated 1073741824 Bytes from 7f74c9e66010 to 7f76c9e66010
...
2045 7f7509ce6010-7f7509d66010
2046 7f7509d66010-7f7509de6010
2047 7f7509de6010-7f7509e66010
2048 7f7509e66010-7f7509ee6010 //2048*65536(B)=128(MB)
Segmentation fault (core dumped)

Answer 1

There are two possible issues here. The first is that you're not using malloc() correctly. You need to check if it returns NULL, or a non-NULL value.
The other issue could be that the OS is over-committing memory, and the out-of-memory (OOM) killer is terminating your process. You can disable over-committing of memory and getting dumps to detect via these instructions.
Edit
Two major problems:
Don't do operations with side effects (ie: n++) inside a logging statement. VERY BAD practice, as logging calls are often removed at compile time in large projects, and now the program behaves differently.
Cast based to a (char *).
This should help with your problem.
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
int main(void)
{
size_t mem_size = 1024 * 1024 * 1024;
printf("MEMSIZE: %lu\n", mem_size);
printf("SIZE OF: void*:%lu\n", sizeof(void*));
printf("SIZE OF: char*:%lu\n", sizeof(char*));
void *based = malloc(mem_size); //mem_size = 1024^3
int stage = 65536;
int initialized = 0;
if (based) {
printf("Allocated %zu Bytes from %p to %p\n", mem_size, based, based + mem_size);
} else {
printf("Error in allocation.\n");
return 1;
}
int n = 0;
while (initialized < mem_size) { //initialize it in batches
//printf("%6d %p-%p\n", n, based+initialized, based+initialized+stage);
n++;
memset((char *)based + initialized, '$', stage);
initialized += stage;
}
free(based);
return 0;
}

Answer 2

Holy, I found the problem - pointer type goes wrong.
Here is the complete code
int main(int argc, char *argv[]) {
/*Allocate 1GB memory*/
size_t mem_size = 1024 * 1024 * 1024;
// the problem is here, I used to use pointer as long long type
char* based = malloc(mem_size);
// and it misleading system to calculate incorrect offset
if (based) {
printf("Allocated %zu Bytes from %lx to %lx\n", mem_size, based, based + mem_size);
} else {
printf("Allocation Error.\n");
return 1;
}
/*Initialize the memory in batches*/
size_t stage = 65536;
size_t initialized = 0;
while (initialized < mem_size) {
memset(based + initialized, '$', stage);
initialized += stage;
}
/*And then I set the breakpoint, check the memory content with gdb*/
...
return 0;
Thank you for the people who have given me advice or comments :)

Answer 3

It is very unusual for a process to need such a large chunk of continuous memory and yes, the kernel does impose such memory limitations. You should probably know that malloc() when dealing with a memory request larger than 128 Kb it calls mmap() behind the curtains. You should try to use that directly.
You should also know that the default policy for the kernel when allocating is to allocate more memory than it has.
The logic is that most allocated memory is not actually used so it relatively safe to allow allocations that exceed the actual memory of the system.
EDIT: As some people have it pointed out, when your process does start to use the memory allocated successfully by the kernel it will get killed by the OOM Killer. This code has produced the following output:
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
int main(int arc, char**argv)
{
char *c = malloc(sizeof(char) * 1024 * 1024 * 1024 * 5);
if(c)
{
printf("allocated memory\n");
memset(c, 1, sizeof(char) * 1024 * 1024 * 1024 * 5);
free(c);
}
else
{
printf("Out of memory\n");
}
return 0;
}
Output:
$ ./a.out
allocated memory
Killed
But after you change the limits of the system:
# echo 2 > /proc/sys/vm/overcommit_memory
# exit
exit
$ ./a.out
Out of memory
As you can see, the memory allocation was successful on the system and the problem appeared only after the memory was used
:EDIT
There are limits that the kernel imposes on how much memory you can allocate and you can check them out with these commands:
grep -e MemTotal -e CommitLimit -e Committed_AS /proc/meminfo
ulimit -a
The first command will print the total memory and the second will display the limit that the kernel imposes on allocations (CommitLimit). The limit is based on your system memory and the over-commit ratio defined on your system that you can check with this command cat /proc/sys/vm/overcommit_ratio.
The Committed_AS is the memory that is already allocated to the system at the moment. You will notice that this can exceed the Total Memory without causing a crash.
You can change the default behavior of your kernel to never overcommit by writing echo 2 > /proc/sys/vm/overcommit_memory You can check the man pages for more info on this.
I recommend checking the limits on your system and then disabling the default overcommit behavior of the kernel. Then try to see if your system can actually allocated that much memory by checking to see if malloc() of mmap() fail when allocating.
sources: LSFMM: Improving the out-of-memory killer
and Mastering Embedded Linux Programming by Chris Simmonds