I want to keep some applications to run with a 4 GB address space within a 64 bit OS running on a 64 bit processor (x86 xeon, 8 core). I know there is an option to compile with -m32 option, but at this moment the computer I'm working with, doesn't have the required support for compiling with -m32, so I can't use it, neither I can install anything on that computer as I don't have any rights.
Now my question is if there is any possibility to restrict address space to 4 GB. Please don't ask me why I want to do so, just tell me how if that is possible. Thanks.
The ulimit/setrlimit mechanism, accessible in bash via ulimit, can do this with minor drawbacks.
ulimit -v 4000000
should limit the memory available to the current process group to 4 GB.
However, it limits the total size of your address space mappings, and does not limit your mappings' offset
- you still may have pointers larger than 2^32.
You could try using setrlimit to impose a virtual memory limit on the process during its initialization, via the RLIMIT_AS attribute. I have used setrlimit to increase resource allocations to a user space process before, but I don't see why it wouldn't work in the other direction as well.
Related
I want to allocate a specific size of memory on the C heap. As a test to do it I used the following piece of code (to check the maximum allowable value):
int main(void) {
int *val;
val=(int*)malloc(4096);
if(!val)
break;
return 0;
}
The problem is that when trying a range of values with malloc(), at about a size of 1900MB it fails. Yet I have 16 GB of installed RAM with about 12GB free.
So I can't even think to allocate a higher value of memory.
Is there something that I'm doing wrong?
Is there something that I should know about malloc()?
I see many programs(like Virtual Machines)that use big amount of memory so i already ruled out the idea that is a security features of the OS.
The pages allocated to your process is doesn't constitute the whole RAM. What do you think? The whole user program will not be allowed to access full RAM. The OS decided how much virtual memory your process will be allocated and then the program starts. That is why you didn't show using the full RAM of your machine.
Long story short - your program is not given the whole RAM for use. If it did - then it would trigger a whole bigger problem than you think it is posing now. Also your idea of malloc use is not clear. When it is NULL which is returned by malloc it mostly calls for an error condition. Your code somehow abstracts those accept and introduce a redundant dummy if block which does nothing.
As a user program you will request memory - and in case it fails then it will return NULL and handle the condition accordingly.
malloc() is a C and C++ run time function that is part of the Standard Library. The memory addressing capabilities depends on what compiler is being used as well as the build settings for that compiler.
I did the following test with Windows 7 x64 using Microsoft Visual 2013 with C++, when I am using the 32 bit build target settings (x86), then when I try to use malloc() to allocate a 4GB block of memory, malloc() returns a NULL pointer indicating the allocation failed.
This is expected because with Windows 32 bit OS, the maximum amount of RAM that can be addressed is a 32 bit pointer but in reality less than 4GB, somewhere around 3.5 GB of usable RAM I think due to way that Windows 32 manages physical memory.
Testing with two other sizes, 2GB (failed with a NULL pointer returned) and 1GB (succeeded with a valid pointer returned) indicates that the maximum that the 32 bit C++ run time allows is somewhere between 1GB and 2GB.
Then I changed the build settings from x86 to x64 to generate a 64 bit executable with 64 bit pointers.
With the change in build settings a call to malloc() with 4GB succeeded with a valid pointer returned.
Task Manager shows the following:
See also StackOverFlow How can I allocate all the availble memory in visual studio for my application?
with the accepted answer that mentions the following:
Getting a larger virtual memory address space requires a pretty
fundamental overhaul. Albeit that it is easy today, just target x64 as
the platform target. A 64-bit process has massive amounts of address
space available, limited only by the maximum size of the paging file.
You could limp along in 32-bit mode, as long as you can count on
actually running on a 64-bit operating system, by using the
/LARGEADDRESSAWARE linker option. Which increases the VM size from 2
GB to 4 GB on a 64-bit operating system.
I continue experimenting with C. I have this program that allows you to decide how much RAM you want to eat.
char * eatRAM()
{
unsigned long long toEat;
unsigned long long i = 0;
float input;
char * pMemory = NULL;
int megaByte = 1048576;
puts("How much RAM do you want to eat? (in Mega Bytes)");
puts("NOTE: If you want to eat more RAM than you have available\nin your system, the program will crash");
printf("\n>> MB: ");
scanf("%f", &input);
toEat = (unsigned long long)(input * megaByte);
pMemory = malloc(toEat);
printf("\n\nEating in total: %llu Bytes\n", toEat);
puts("Check your task manager!\n");
if(pMemory != NULL)
{
printf("\n\nEating in total: %llu Bytes\n", toEat);
puts("Check your task manager!\n");
for(i; i < toEat; i++)
{
pMemory[i] = 'x';
}
}
else
{
puts("\nSeems like that amount of memory couldn't be allocated :( \n");
}
return pMemory;
}
UPDATED QUESTION:
The thing is that... if I enter for example 1024MB it works, I can see in the task manager it is using 1GB of RAM. Even if I enter 1500MB it works..
But if I enter 2048MB it says
Seems like that amount of memory couldn't be allocated :(
or even if I enter 1756MB
Remember I'm new to C, maybe I'm omitting something important related to how OS allows me to access memory, what could it be?
A 32-bit process on Windows has a 2 gigabyte address space available by default. The bottom half of the full pow(2, 32) address space, the top 2 GB is used by the operating system. Since just about nobody uses a 32-bit OS anymore, you can get 4 GB when you link your program with /LARGEADDRESSAWARE.
That 2 GB VM space needs to be shared by code and data. Your program typically loads at 0x00400000, any operating system DLLs you use (like kernel32.dll and ntdll.dll) have high load addresses (beyond 0x7F000000). And at least the startup thread's stack and the default process heap are created before your program starts running, their addresses are generally unpredictable.
Your program will be subjected to shrink-wrapped viral attacks in most any OS install, you'll have DLLs injected that provide "services" like anti-malware and cloud storage. The load address of those DLLs are unpredictable. Also any DLLs that you linked with yourself and are implicitly loaded when your program starts. Few programmers pay attention to their preferred base address and leave it at the default, 0x1000000. You can see these DLLs from the debugger's Modules window. Such DLLs often have their own CRT and tend to create their own heaps.
Allocations you make yourself, particularly very large ones that won't come from the low-fragmentation heap, need to find address space in the holes that are left between existing code and data allocations. If you get 1500 MB then your VM is pretty clean. In general you'll get into trouble beyond 650 MB, quickly getting less when the program has been running for a while and fragmented the VM space. Allocation failures are almost always caused because the OS can't find a big enough hole, not because you don't have enough VM left. The sum of the holes can be considerably larger than your failed allocation request.
These details are rapidly becoming a folk tale, there are very few remaining reasons to still target x86. Target x64 and address space fragmentation won't be a problem for the next 20 years, very hard to fragment 8 terabytes of VM. With lots of headroom to grow beyond that.
So it should be obvious why you can't get 2048 MB, you can't get it all. Get further insight from SysInternals' VMMap utility, it shows you how the VM is carved up. And Mark Russinovich' blog post and book give lots of background.
It is an OS limit rather then a C limit.
To address more than 4Gb system wide you need to be running a 64 bit OS and for a single process to address more than 4Gb it must be built as a 64 bit app. Win32 has a 2Gb per process memory limit. 5Gb of physical RAM is largely irrelevant since the memory is virtualised.
Quite apart from the theoretical limits of 32 and 64 bit systems and applications, an OS may still impose limits. Different versions and editions (Home, Pro, Server etc.) of Windows for example impose specific limits for commercial reasons.
A specific answer in your case would require information about your system, toolchain and build options applied. If you are using Windows and VC++ you need to consider the /LARGEADDRESAWARE option; it is not enabled by default in the 32 bit compiler, but Win32 has a 2Gb default limit in any case unless physical address extension is enabled.
I believe that a 32 bit process running on Win64 can address the full 4Gb 32 bit address space, but you will certainly need to build with /LARGEADDRESAWARE in that case. Even then not quite all that space will be available to the heap, and any single allocation must be contiguous, so may be limited by previous allocations and heap fragmentation.
Allocation will never work if the amount of remaining free memory is less than the amount you're trying to allocate.
Also, right after this line:
pMemory = (char *) malloc(toEat);
Add the following:
if (!pMemory){
printf("Can't allocate memory\n");
return NULL;
}
That way, instead of receiving "segmentation fault" related messages, you'll see one "Can't allocate memory" message instead and your function will return NULL.
Make sure you do similar value checking in functions that call your eatRam function or you'll receive "segmentation fault" messages. and also, use debuggers like gdb.
I have c code with embedded SQL for Oracle through Pro*C.
Whenever I do an insert or update (below given an update example),
update TBL1 set COL1 = :v, . . . where rowid = :v
To manage bulk insertions and updates, I have allocated several memory chunks to insert as bulk and commit once. There are other memory allocations too going on as and when necessary. How do I better manage the memory (heap) for dynamic memory allocations? One option is to have the heap size configurable during the GNU linking time. I'm using g++ version 2.95, I know it's quite an old version, but have to use this for legacy. Since the executable (runs on solaris 10), obce built, could run on several production environments with varied resources, one-size-fit-all for heap size allocation may not be appropriate. As an alternative, need some mechanism where heaps may elastically grow as and when needed. Unlike Linux, Solaris, I think, does not have the concept of over-allocated memory. So, memory allocations could fail with ENOMEM if there is no more space left. What could be better strategy to know that we could be crossing the danger level and now we should either deallocate chunks that we are storing in case these are done using or transfer memory chunks to oracle DB in case these are still pending to be loaded and finally deallocate. Any strategy that you could suggest?
C is not java where the heap size is fixed at startup.
The heap and the stack of a C compiled application both share the same virtual memory space and adjust dynamically.
The size of this space depends on whether you are compiling a 32 bit or a 64 bit binary, and also whether your kernel is a 32 bit or a 64 bit one (on SPARC hardware, it's always 64 bit).
If you have not enough RAM and want Solaris to accept large memory reservations anyway, a similar way Linux over commits memory, you can just add enough swap for the reservation to be backed by actual storage.
If for some reason, you are unhappy with the Solaris libc memory allocator, you can evaluate the bundled alternative ones like libumem, mtmalloc or the third party hoard. See http://www.oracle.com/technetwork/articles/servers-storage-dev/mem-alloc-1557798.html for details.
One solution would be to employ soft limits in your code for various purposes, f.e. that only 100 transactions at a time are handled and other transactions have to wait until the previous ones are deallocated. This guarantees predictable behavior, as no code part can use up more memory than allowed.
The question is:
Do you really run out of memory in your application or do you fragment your memory and fail to get a sufficient large contiguous block of memory? The strategies to handle each case are different.
I'm writing a program that requires a lot of memory (large graph analysis).
Currently there are two main data structures in my program (taking up most of the memory). These are:
a n*n matrix of type int **
and array of length n, type Node *
Node, in this case, is a struct containing two ints (sizeof(Node) = 8)
The biggest value for n that I can run my code on is 22900, doing a bit of calculation I get:
22900*22900 * sizeof(int) * 8 + 22900 * sizeof(Node) = 16782591360 bits
This is 1.95375077 Gigabytes.
So question 1: am I calculating the memory usage for these two data structures properly?
and 2: Is there a 2GB memory allocation limit on windows. If so, how can I get around it?
For further information, I am on a 64bit Windows 7 machine compiling with GCC, 4GB RAM with ~3GB of free RAM at time of running.
Thanks.
You aren't calculating it correctly. First, there is no reason to multiply anything by 8. The quantum of allocation in C is byte, not bit. Second, you neglect the pointer array which implements the first dimension of your matrix. So:
22900 * sizeof(int*) + 22900*22900*sizeof(int) + 22900*sizeof(Node) = 2097914800 bytes
As for useful advice, I'll leave that to the (already posted) other answer.
You are most likely compiling for 32-bits; on windows, 32-bit processes are limited to 2G of addressable space (with a 64-bit OS and the IMAGE_FILE_LARGE_ADDRESS_AWARE flag set, 4GB). Compile for 64-bit and you should see your memory limit rise substantially. However, you will likely want more physical RAM before doing so; you're using half of it already and hitting swap will kill your performance.
32bit processes are limited to 2G of user-adressable memory (on most releases of windows with default settings). 64bit processes have much larger address spaces. See this note Performance and Memory Consumption Under WOW64 for a way to give your 32bit app a 4G address space (not sure if/how GCC can build executable images with that flag set though).
Compile your code as a 64bit application and that limit should vanish (try MinGW-w64).
To get around the memory limitation, you have to compile the program in 64-bit mode; note that pointers are then 8 byte in size. The total memory usage of the matrix would be then doubled.
i declared a struct variable in C of size greater than 1024bytes. On running Coverity (a static code analyzer application) it reports that this stack variable is greater than 1024 bytes and therefore a cause of error.
I'd like to know if I need to worry about this warning? Is there really a maximum limit to the size of a single stack variable?
thanks,
che
The maximum size of a variable is the limited by the maximum size of the stack (specifically, how much of the stack is left over from any current use including variables and parameters from functions higher on the stack as well as process frame overhead).
On Windows, the stacksize of the first thread is a property of the executable set during linking while the stacksize of a thread can be specified during thread creation.
On Unix, the stacksize of the first thread is usually only limited only by how much room there is for it to grow. Depending on how the particular Linux lays out memory and your use of shared objects, that can vary. The stacksize of a thread can also be specified during thread creation.
The problem it is trying to protect you from is stack overflow, because of different execution paths, it is very hard to find in testing. Mostly for this reason - it is considered bad form to allocate a large amount of data on the stack. You are only really likely to run into a real problem on an embedded system though.
In other words, it sets an arbitrary limit to what it considers too much data on the stack.
Yes. Of course it's limited by the address space of your system. It's also limited by the amount of space allocated to the stack by your OS, which usually can't be changed after your program starts but can be changed beforehand (either by the launching process, or by the properties of the executable). At a quick glance, the maximum stack size on my OS X system is 8 MiB and on Linux it's 10 MiB. On some systems, you can even allocate a different amount of stack to each different thread you start, although this is of limited usefulness. Most compilers also have another limit to how much they'll allow in a single stack frame.
On a modern desktop, I wouldn't worry about a 1k stack allocation unless the function were recursive. If you're writing embedded code or code for use inside an OS kernel, it would be a problem. Code in the Linux kernel is only permitted 64 KiB stacks or less, depending on configuration options.
This article is pretty interesting regarding stack size http://www.embedded.com/columns/technicalinsights/47101892?_requestid=27362
Yes is it OS dependent and also other things dependent. Sorry to be so vague. You may also be able to dig up some code in the gcc collection for testing stack size.
If your function was involved (directly or indirectly) in recursion, then allocating a large amount on the stack would limit the depth of recursion and might well blow the stack. Under Windows this stack reserve defaults to 1MB, though you can increase it statically with linker commands. The stack will grow as it is used, but the operating system sometimes cannot extend it. I discuss this in a little more detail on my website here.
As I have seen, a C compiler(turbo) provides a maximum size of 64000k for a variable. If we need more size, then it is declared as "huge".
It's not a good idea to try to use a massive amount of stack space.
Here is a link to the default gcc stack size: http://www.cs.nyu.edu/exact/core/doc/stackOverflow.txt
Also, you could specify --stack,xxxxx to customize the stack size, so it's best to assume xxxxx is a small number and stick with heap allocation.
Stack, heap, low, high VM -- Nuts, for the first thread, the stack at the top pf 64 bit VM, there should be no limit, so it seems like a gcc/c compiler bug that for local automatic "int x[2621440];" I get SIGSEGV. The compiler should be letting the first thread stack grow until it hits the heap, which in a 16 billion billion byte VM is pretty unlikely for now. The kindest thing is to call it a compiler "limitation". (In testing some while back, probably on a Solaris SPARC, it seemed that local variables processed faster than global ones. Go figure!)