I read the manual on malloc() in Solaris, and find that malloc() can set EAGAIN error in Solaris.
The manual writes:
EAGAIN There is not enough memory available to allocate size bytes of memory; but the application could try again later.
Personally, I think if the malloc() returns NULL, there must be a memory leak or some other persistent problem. If that happens how would trying again later help?
So I want to know, in what conditions can malloc() set EAGAIN errno? Has anyone encountered such situation?
Standard malloc() does not set errno to EAGAIN on failure.
Under Unix, malloc() will most probably set errno to ENOMEM.
In general errno EAGAIN means Resource temporarily unavailable. Which means that the operating system may have the resource available in some time.
This is just a way of saying right now I haven't enough memory, but I will try to free some in the nearest future and then I can give it to you.
This may be related to the way operating systems usually allocate memory to processes - even if the memory is free()'d it does not return to the operating system, but is still reserved for that process.
I am only speculating, but perhaps in case of EAGAIN the system will try to reallocate the unused memory assigned to the other processes. This may take time, hence the EAGAIN return code.
I would suggest using sleep() after receiving EAGAIN and then trying it again. After the second call either memory will be allocated or another error returned. If it's ENOMEM, then the case is clear, there's no memory. If it's EAGAIN again... It's up to you.
Standard malloc function doesn't set errno on failure. So it's only specific to the implementation of malloc on Solaris that has this additional feature.
Note that malloc still returns NULL on failure.
So you can still check for return value of malloc() and not bother checking about errno which is the standard malloc's behaviour and should be enough on all occasions. Just that errno provides additional information about failure which may be helpful on certain occasions.
Generally speacking, checking errno makes sense only along with the return code. Relying errno may or may not indicate any failures.
opengroup.org (POSIX) says:
Upon successful completion with size not equal to 0, malloc() shall return a pointer to the allocated space. If size is 0, either a null pointer or a unique pointer that can be successfully passed to free() shall be returned. Otherwise, it shall return a null pointer [CX] and set errno to indicate the error.
ERRORS
The malloc() function shall fail if:
[ENOMEM]
[CX] Insufficient storage space is available.
POSIX malloc description
Solaris comes from a different UNIX family and you will encounter lots of differences between POSIX and base Solaris - i.e., The most glaring thing for new users is normally awk. Solaris has an ancient awk : /usr/bin/awk, /usr/xpg/bin/awk is more "modern", /usr/bin/nawk is what you use when porting shell scripts to Solaris. These anachronisms come from way back and are there so old utilities and syscalls will remain functional on new versions of Solaris.
Related
I new to this can any one tell the exact difference between brk and sbrk with a brief example ? is there any efficiency factor to choose from any of the two?
malloc and new internally call brk or sbrk.
int brk(void *addr);
brk() sets the end of the data segment to the value specified by
addr, when that value is reasonable, the system has enough memory,
and the process does not exceed its maximum data size.
On success, brk() returns zero. On error, -1 is returned, and errno
is set to ENOMEM.
void *sbrk(intptr_t increment);
sbrk() increments the program's data space by increment bytes.
Calling sbrk() with an increment of 0 can be used to find the current
location of the program break.
On success, sbrk() returns the previous program break. (If the break
was increased, then this value is a pointer to the start of the newly
allocated memory). On error, (void *) -1 is returned, and errno is
set to ENOMEM.
From linux manual page
brk sets the upper limit of the data segment, sbrk increments it. In ancient Unixes malloc/free used sbrk. On modern ones things could be very different, for example, OSX does not use brk/sbrk to manage heap allocations but mmap, brk/sbrk exist but are just emulation in a small segment of memory. This is almost the same on Linux (source code of mention the history of transition from brk/sbrk to mmap).
How could this happen when reading a socket with a error ENOMEM return ?
Man 2 read, I can't see any detail about this error, can someone help me. Thanks.
My linux kernel 2.6.26-1-686.
ENOMEM
is an OS error code , as defined in kern/include/kern/errno.h ,which is returned due to insufficient memory.
The name ENOMEM stands for Error NO MEMory. Its one of the error codes returned by the fork() call which means no more storage space available.In connection with sockets they are raised when there isn’t enough resources available to create a socket. The value of the error code is 12.
You can handle this error by instantly
and to release all the allocated resources as soon as possible,
avoiding operations requiring allocating new resources.
EDIT:
Reading from socket cause allocation for receive data buffer and they needs to be free after your requirement. Also make sure enough memory is there. It will return this error when fail in allocation of memory. See link
Is the maximum heap size of a program in C fixed or if I keep malloc-ing it will at some point start to overflow?
Code:
while(connectionOK) //connectionOK is the connection with server which might be forever
{
if(userlookup_IDNotFound(userID))
user_struct* newuser = malloc(getsize(user_struct));
setupUserAccount(newuser);
}
I am using gcc in ubuntu/ linux if that matters.
I know something like getrlimit but not sure if it gives heap size. Although it does give the default stack size for one of the options in the input argument.
Also valgrind is probably a good tool as suggested here how to get Heap size of a program but I want to dynamically print an error message if there is a heap overflow.
My understanding was the process address space being allocated by the OS (which is literally allowed to use the whole memory if it wants to) at the beginning of the process creation but I am not sure if it is dynamically given more physical memory once it requests for additional memory.
The heap never overflows it just runs out of memory at a certain point (usually when malloc() returns NULL) So to detect out of memory just check the return value of the malloc() call.
if (newuser == NULL)
{
printf("OOM\n");
exit(1); /* exit if you want or can't handle being OOM */
}
malloc() internally will request more memory from the OS so it expands dynamically so it's not really fixed size as it will give back pages to the OS that it no longer needs as well as requesting more at any given time that it requires them.
Technically what malloc allocates on most systems is not memory, but address space. On a modern system you can easily allocate several petabytes of address space with malloc and malloc will probably always return a non null pointer. The reason behind this is, that most OS actually perform memory allocation only when a piece of address space is actively modified. As long as it sits there untouched, the OS will just make a note that a certain area of a process address space has been validly reserved for future use.
This kind of bahavior is called "memory overcommitment" and is of importance when maintaining Linux systems. If can happen, that theres more memory allocated than available for some time, and then some program will actually write to some of the overcommited memory. What then happens is, that the so called "Out Of Memory Killer" (OOM killer) will go on a rampage and kills those processes it sees most apropriate for; unfortunately it usually are those processes you don't want to loose under any circumstances. Databases are known to be among the prime targets of the OOM killer.
Because of this, it's strongly recommended to switch of memory overcommitment on high availability Linux boxes. With disabled memory overcommitment disabled, each request for address space must be backed by memory. In that case malloc will actually return 0 if the request can not be fullfilled.
at some point, malloc() will return NULL, when system will run out of memory. then when you try to dereference that, your program will abort executing.
See what happens when you do malloc(SIZE_MAX) a few times :-)
I have a program whose total memory footprint is about 100 MiB (VM size, in top, while stopped in gdb) that's trying to open a new (not-yet-existent) compressed log file using gzopen. This fails, with errno set to ENOMEM, despite the fact that the system has 6GB memory completely free (not even holding caches), and lots of space on the filesystem (that would be ENOSPC, I know). Are there more obscure issues that could cause this? Is something in the library incidentally allocating gigabytes upon gigabytes of memory for no good reason?
For note, my ulimits are set unlimited.
No, there is nothing in zlib that would allocate more than a MiB or two. zlib will only set errno to zero. On its own, it never sets errno to ENOMEM. The library functions it calls may. What version of zlib are you using?
Turns out zlib was not returning ENOMEM. It was bailing out because we had passed it a mode argument w+, which is invalid because it can't read and write a given gzip file at the same time. The ENOMEM came from what happened to be sitting in errno from previous library/system calls.
Recently, I work on a video player program on Windows for a CCTV program. As the program has to decode and play many videos streams at the same time, I think it might meet the situation that malloc will fail and I add checking after every malloc.
But genrally speaking, in these code of open source programs that I've read in open source projects, I seldom find any checking of result of malloc. So when malloc fails, most program will just crash. Isn't that unacceptalbe?
My colleagues who write server programs on linux will alloc a enough memory for 100 client connections. So although his program might refuse the 101 client, it will never met a failure of malloc. Is his approach also suitable for desktop applications?
On Linux, malloc() will never fail -- instead, the OOM killer will be triggered and begin killing random processes until the system falls over. Since Linux is the most popular UNIX derivative in use today, many developers have learned to just never check the result of malloc(). That's probably why your colleagues ignore malloc() failures.
On OSes which support failures, I've seen two general patterns:
Write a custom procedure which checks the result of malloc(), and calls abort() if allocation failed. For example, the GLib and GTK+ libraries use this approach.
Store a global list of "purge-able" allocations, such as caches, which can be cleared in the event of allocation failure. Then, try the allocation again, and if it still fails, report it via the standard error reporting mechanisms (which do not perform dynamic allocation).
Follow the standardized API
Even on Linux, ulimit can be used to get a prompt malloc error return. It's just that it defaults to unlimited.
There is a definite pressure to conform to published standards. On most systems, in the long run, and eventually even on Linux, malloc(3) will return a correct indication of failure. It is true that desktop systems have virtual memory and demand paging, but even then not checking malloc(3) only works in a debugged program with no memory leaks. If anything goes wrong, someone will want to set a ulimit and track it down. Suddenly, the malloc check makes sense.
To use the result of malloc without checking for null is unacceptable in code that might be open to use on platforms where malloc can fail, on those it will tend to result in crashes and unpredicatable behaviour. I can't forsee the future, don't know where my code will go, so I would write code with checks for malloc returning null - better to die than behave unpredicatbly!
Strategies for what to do if malloc fails depend upon the kind of applciation and how much confidence you have in the libraries you are using. It some situations the only safe thing to do is halt the whole program.
The idea of preallocating a known quota of memory and parcelling out in some chunks, hence steering clear of actually exhausting the memeory is a good one, if your application's memory usage is predicatable. You can extend this to writing your own memory management routines for use by your code.
It depends on the type of application that you are working on. If the application does work that is divided into discrete tasks where an individual task can be allowed to fail, then checking memory allocations can be recovered from gracefully.
But in many cases, the only reasonable way to respond to a malloc failure is by terminating the program. Allowing your code to just crash on the inevitable null dereference will achieve that. It would certainly always be better to dump a log entry or error message explaining the error, but in the real world we work on limited schedules. Sometimes the return on investment of pedantic error handling isn't there.
Always check, and pre-allocate a buffer that can be freed in this case so you can warn the user to save his data and shut down the application.
Depends on the app you write. Of course you always need to check the return value of malloc(). However, handling OOM gracefully only makes sense in very cases, such as low-level crucial system services, or when writing a library that might be used be them. Having a malloc wrapper that aborts on OOM is hence very common in many apps and frameworks. Often those wrappers are named xmalloc() or similar.
GLib's g_malloc() is aborting, too.
If you are going to handle huge amounts of memory, and want to make statements to Linux like "now I have memory area ABC and I don't need the B piece, do as you wish", have a look to mmap()/madvise() family of functions available in stock GNU C library. Depending on your usage patterns, the code can end up even simpler than using malloc. This API can also be used to help Linux not waste memory by caching files you are going to read/write only once.
They are nicely documented in GNU libc info documentation.
It is usually impossible for a program to handle running out of memory. What are you going to do? Open a file and log something? If you try to allocate a large block and it fails, you may have a fallback and try again with a smaller buffer, but if you fail to allocate 10 bytes, there is not much you can do. And checking for null constantly convolutes the code. For that reason I usually add a custom function that does checking and aborts on fail:
static void* xmalloc(size_t sz) {
void* p = malloc(sz);
if (!p) abort();
return p;
}