I'm writing a program that requires a lot of memory (large graph analysis).
Currently there are two main data structures in my program (taking up most of the memory). These are:
a n*n matrix of type int **
and array of length n, type Node *
Node, in this case, is a struct containing two ints (sizeof(Node) = 8)
The biggest value for n that I can run my code on is 22900, doing a bit of calculation I get:
22900*22900 * sizeof(int) * 8 + 22900 * sizeof(Node) = 16782591360 bits
This is 1.95375077 Gigabytes.
So question 1: am I calculating the memory usage for these two data structures properly?
and 2: Is there a 2GB memory allocation limit on windows. If so, how can I get around it?
For further information, I am on a 64bit Windows 7 machine compiling with GCC, 4GB RAM with ~3GB of free RAM at time of running.
Thanks.
You aren't calculating it correctly. First, there is no reason to multiply anything by 8. The quantum of allocation in C is byte, not bit. Second, you neglect the pointer array which implements the first dimension of your matrix. So:
22900 * sizeof(int*) + 22900*22900*sizeof(int) + 22900*sizeof(Node) = 2097914800 bytes
As for useful advice, I'll leave that to the (already posted) other answer.
You are most likely compiling for 32-bits; on windows, 32-bit processes are limited to 2G of addressable space (with a 64-bit OS and the IMAGE_FILE_LARGE_ADDRESS_AWARE flag set, 4GB). Compile for 64-bit and you should see your memory limit rise substantially. However, you will likely want more physical RAM before doing so; you're using half of it already and hitting swap will kill your performance.
32bit processes are limited to 2G of user-adressable memory (on most releases of windows with default settings). 64bit processes have much larger address spaces. See this note Performance and Memory Consumption Under WOW64 for a way to give your 32bit app a 4G address space (not sure if/how GCC can build executable images with that flag set though).
Compile your code as a 64bit application and that limit should vanish (try MinGW-w64).
To get around the memory limitation, you have to compile the program in 64-bit mode; note that pointers are then 8 byte in size. The total memory usage of the matrix would be then doubled.
Related
I want to allocate a specific size of memory on the C heap. As a test to do it I used the following piece of code (to check the maximum allowable value):
int main(void) {
int *val;
val=(int*)malloc(4096);
if(!val)
break;
return 0;
}
The problem is that when trying a range of values with malloc(), at about a size of 1900MB it fails. Yet I have 16 GB of installed RAM with about 12GB free.
So I can't even think to allocate a higher value of memory.
Is there something that I'm doing wrong?
Is there something that I should know about malloc()?
I see many programs(like Virtual Machines)that use big amount of memory so i already ruled out the idea that is a security features of the OS.
The pages allocated to your process is doesn't constitute the whole RAM. What do you think? The whole user program will not be allowed to access full RAM. The OS decided how much virtual memory your process will be allocated and then the program starts. That is why you didn't show using the full RAM of your machine.
Long story short - your program is not given the whole RAM for use. If it did - then it would trigger a whole bigger problem than you think it is posing now. Also your idea of malloc use is not clear. When it is NULL which is returned by malloc it mostly calls for an error condition. Your code somehow abstracts those accept and introduce a redundant dummy if block which does nothing.
As a user program you will request memory - and in case it fails then it will return NULL and handle the condition accordingly.
malloc() is a C and C++ run time function that is part of the Standard Library. The memory addressing capabilities depends on what compiler is being used as well as the build settings for that compiler.
I did the following test with Windows 7 x64 using Microsoft Visual 2013 with C++, when I am using the 32 bit build target settings (x86), then when I try to use malloc() to allocate a 4GB block of memory, malloc() returns a NULL pointer indicating the allocation failed.
This is expected because with Windows 32 bit OS, the maximum amount of RAM that can be addressed is a 32 bit pointer but in reality less than 4GB, somewhere around 3.5 GB of usable RAM I think due to way that Windows 32 manages physical memory.
Testing with two other sizes, 2GB (failed with a NULL pointer returned) and 1GB (succeeded with a valid pointer returned) indicates that the maximum that the 32 bit C++ run time allows is somewhere between 1GB and 2GB.
Then I changed the build settings from x86 to x64 to generate a 64 bit executable with 64 bit pointers.
With the change in build settings a call to malloc() with 4GB succeeded with a valid pointer returned.
Task Manager shows the following:
See also StackOverFlow How can I allocate all the availble memory in visual studio for my application?
with the accepted answer that mentions the following:
Getting a larger virtual memory address space requires a pretty
fundamental overhaul. Albeit that it is easy today, just target x64 as
the platform target. A 64-bit process has massive amounts of address
space available, limited only by the maximum size of the paging file.
You could limp along in 32-bit mode, as long as you can count on
actually running on a 64-bit operating system, by using the
/LARGEADDRESSAWARE linker option. Which increases the VM size from 2
GB to 4 GB on a 64-bit operating system.
I continue experimenting with C. I have this program that allows you to decide how much RAM you want to eat.
char * eatRAM()
{
unsigned long long toEat;
unsigned long long i = 0;
float input;
char * pMemory = NULL;
int megaByte = 1048576;
puts("How much RAM do you want to eat? (in Mega Bytes)");
puts("NOTE: If you want to eat more RAM than you have available\nin your system, the program will crash");
printf("\n>> MB: ");
scanf("%f", &input);
toEat = (unsigned long long)(input * megaByte);
pMemory = malloc(toEat);
printf("\n\nEating in total: %llu Bytes\n", toEat);
puts("Check your task manager!\n");
if(pMemory != NULL)
{
printf("\n\nEating in total: %llu Bytes\n", toEat);
puts("Check your task manager!\n");
for(i; i < toEat; i++)
{
pMemory[i] = 'x';
}
}
else
{
puts("\nSeems like that amount of memory couldn't be allocated :( \n");
}
return pMemory;
}
UPDATED QUESTION:
The thing is that... if I enter for example 1024MB it works, I can see in the task manager it is using 1GB of RAM. Even if I enter 1500MB it works..
But if I enter 2048MB it says
Seems like that amount of memory couldn't be allocated :(
or even if I enter 1756MB
Remember I'm new to C, maybe I'm omitting something important related to how OS allows me to access memory, what could it be?
A 32-bit process on Windows has a 2 gigabyte address space available by default. The bottom half of the full pow(2, 32) address space, the top 2 GB is used by the operating system. Since just about nobody uses a 32-bit OS anymore, you can get 4 GB when you link your program with /LARGEADDRESSAWARE.
That 2 GB VM space needs to be shared by code and data. Your program typically loads at 0x00400000, any operating system DLLs you use (like kernel32.dll and ntdll.dll) have high load addresses (beyond 0x7F000000). And at least the startup thread's stack and the default process heap are created before your program starts running, their addresses are generally unpredictable.
Your program will be subjected to shrink-wrapped viral attacks in most any OS install, you'll have DLLs injected that provide "services" like anti-malware and cloud storage. The load address of those DLLs are unpredictable. Also any DLLs that you linked with yourself and are implicitly loaded when your program starts. Few programmers pay attention to their preferred base address and leave it at the default, 0x1000000. You can see these DLLs from the debugger's Modules window. Such DLLs often have their own CRT and tend to create their own heaps.
Allocations you make yourself, particularly very large ones that won't come from the low-fragmentation heap, need to find address space in the holes that are left between existing code and data allocations. If you get 1500 MB then your VM is pretty clean. In general you'll get into trouble beyond 650 MB, quickly getting less when the program has been running for a while and fragmented the VM space. Allocation failures are almost always caused because the OS can't find a big enough hole, not because you don't have enough VM left. The sum of the holes can be considerably larger than your failed allocation request.
These details are rapidly becoming a folk tale, there are very few remaining reasons to still target x86. Target x64 and address space fragmentation won't be a problem for the next 20 years, very hard to fragment 8 terabytes of VM. With lots of headroom to grow beyond that.
So it should be obvious why you can't get 2048 MB, you can't get it all. Get further insight from SysInternals' VMMap utility, it shows you how the VM is carved up. And Mark Russinovich' blog post and book give lots of background.
It is an OS limit rather then a C limit.
To address more than 4Gb system wide you need to be running a 64 bit OS and for a single process to address more than 4Gb it must be built as a 64 bit app. Win32 has a 2Gb per process memory limit. 5Gb of physical RAM is largely irrelevant since the memory is virtualised.
Quite apart from the theoretical limits of 32 and 64 bit systems and applications, an OS may still impose limits. Different versions and editions (Home, Pro, Server etc.) of Windows for example impose specific limits for commercial reasons.
A specific answer in your case would require information about your system, toolchain and build options applied. If you are using Windows and VC++ you need to consider the /LARGEADDRESAWARE option; it is not enabled by default in the 32 bit compiler, but Win32 has a 2Gb default limit in any case unless physical address extension is enabled.
I believe that a 32 bit process running on Win64 can address the full 4Gb 32 bit address space, but you will certainly need to build with /LARGEADDRESAWARE in that case. Even then not quite all that space will be available to the heap, and any single allocation must be contiguous, so may be limited by previous allocations and heap fragmentation.
Allocation will never work if the amount of remaining free memory is less than the amount you're trying to allocate.
Also, right after this line:
pMemory = (char *) malloc(toEat);
Add the following:
if (!pMemory){
printf("Can't allocate memory\n");
return NULL;
}
That way, instead of receiving "segmentation fault" related messages, you'll see one "Can't allocate memory" message instead and your function will return NULL.
Make sure you do similar value checking in functions that call your eatRam function or you'll receive "segmentation fault" messages. and also, use debuggers like gdb.
I - not a professional software engineer - am currently extending a quite large scientific software.
At runtime I get an error stating "insufficient virtual memory".
At this point during runtime, the used working memory is about 550mb and the error accurs when a rather big threedimensional array is dynamically allocated. The array - if it would be allocated - would be about a size of 170mb. Adding this to the already used 550mb the program would still be way below the 2gb boundary that is set for 32bit applications. Also there is more than enough working memory available on the system.
Visual Studio is currently set that it allocates arrays on the stack. Allocating them on the heap does not make any difference anyway.
Splitting the array into smaller arrays (being the size of the one big array in sum) results in the program running just fine. So I guess that the dynamically allocated memory has to be available in one adjacent block.
So there I am and I have no clue how to solve this. I can not deallocate some of the already used 550mb as the data is still required. I also can not change very much of the configuration (e.g. the compiler).
Is there a solution for my problem?
Thank you some much in advance and best regards
phroth248
The virtual memory is the memory your program can address. It is usually the sum of the physical memory and the swap space. For example, if you have 16GB of physical memory and 4GB of swap space, the virtual memory will be 20GB. If your Fortran program tries to allocate more than those 20 addressable GB, you will get an "insufficient virtual memory" error.
To get an idea of the required memory of your 3D array:
allocate (A(nx,ny,nz))
You have nx*ny*nz elements and each element takes 8 bytes in double precision or 4 bytes in single precision. I let you do the math.
Some things:
1. It is usually preferable to to allocate huge arrays using operating system services rather than language facilities. That will circumvent any underlying library problems.
You may have a problem with 550MB in a 32-bit system. Usually there is some division of the 4GB address space into dedicated regions.
You need to make sure you have enough virtual memory.
a) Make sure your page file space is large enough.
b) Make sure that your system is not configured to limit processes address space sizes to smaller than what you need.
c) Make sure that your accounts settings are not limiting your process address space to smaller than allowed by the system.
I have a 4GB Ram installed on Coure2Duo PC with a 32bit Windows 7 Operating system. I have increased the paging size up to 106110MB. But after doing all this I am not able to significantly increase the maximum array size.
Following are the specs
memory
Maximum possible array: 129 MB (1.348e+08 bytes) *
Memory available for all arrays: 732 MB (7.673e+08 bytes) **
Memory used by MATLAB: 563 MB (5.899e+08 bytes)
Physical Memory (RAM): 3549 MB (3.722e+09 bytes)
* Limited by contiguous virtual address space available.
** Limited by virtual address space available.
Kindly help me on your earliest. I am not even able to read a file of 48+MB size in double format.
There are two things you can do to clear up memory for MATLAB. Since you're using a 32-bit version of the program, you're normally limited to 2GB of memory. Using the /3GB switch while opening the program makes an additional 1GB of RAM available to that program.
Second, you should consider using the pack() function, which rearranges variables in RAM to free up more contiguous memory space. This, more than anything, is affecting your ability to open individual arrays.
Remember: you can figure out how many items an array will hold by dividing the memory amount available by the size of the variable type. Double variables take up 8 bytes each. Your 129MB of space available should allow around 16.85 million double values in a single array.
You can view information about memory usage using the memory functions included in MATLAB.
memory shows the memory information
inmem will show you the variables and functions stored in memory
clear will allow you to clear the memory of specific variables or functions.
You may try to set the 3GB switch, maybe this increases the possible memory. Otherwise: Switch to a 64 bit os. Your system wastes 547MB of RAM simply because there are no addresses for it.
I have the following C program compiled in Microsoft Visual Studio Express 2012:
int main() {
int a[300000];
return 0;
}
This crashes with a stack overflow in msvcr110d.dll!__crtFlsGetValue().
If I change the array size from 300,000 to 200,000 it works fine (in so much as this simple program can be said to 'work' since it doesn't do anything).
I'm running on Windows 7 and have also tried this with gcc under Cygwin and it produces the same behaviour (in this case a seg fault).
What the heck?
There are platform-specific limits on the size of the space used by automatic objects in C (the "stack size"). Objects that are larger than that size (which may be a few kilobytes on an embedded platform and a few megabytes on a desktop machine) cannot be declared as automatic objects. Make them static or dynamic instead.
In a similar vein, there are limits on the depth of function calls, and in particular on recursion.
Check your compiler and/or platform documentation for details on what the actual size is, and on how you might be able to change it. (E.g. on Linux check out ulimit.)
Because it's being allocated on the stack and the stack has a limited size, obviously not large enough to hold 300000 ints.
Use heap allocation a la malloc:
int* a = malloc(sizeof(int) * 300000);
// ...
free(a);
The heap can hold a lot more than the stack.
The size of thread stacks is traditionally limited by operating systems because of a finite limit to the amount of virtual address space available to each process.
As the virtual address space allocated to a thread stack can't be changed once it is allocated, there is no strategy other than to allocate a fairly large, but limited, chunk to each thread - even when most threads will use very little of it.
Similarly, there is a finite limit also on the number of threads a process is allowed to spawn.
At a guess, the limit here is 1MB, and Windows then limits the number of threads to - say - 256, this means that 256MB of the 3GB virtual address space available to a 32-bit process is allocated to thread stacks - or put another way, 1/12th.
On 64-bit systems, there is obviously a lot more virtual space to play with, but having a limit is still sensible in order to quickly detect - and terminate - infinite recursion.
Local variables claim space from the stack. So, if you allocate something large enough, the stack will inevitably overflow.