I've researched code of some library and noticed that calls to calloc there are followed by memset for block allocated by calloc.
I've found this question with quite comprehensive answer on differences between calloc and malloc + memset and calling memset for just before allocated storage:
Why malloc+memset is slower than calloc?
What i still can't understand is why one would want to do so.
What are benefits of this operations?
The code example from mentioned above library:
light_pcapng_file_info *light_create_default_file_info()
{
light_pcapng_file_info *default_file_info = calloc(1, sizeof(light_pcapng_file_info));
memset(default_file_info, 0, sizeof(light_pcapng_file_info));
default_file_info->major_version = 1;
return default_file_info;
}
The code of allocated structure(each array contains 32 elements):
typedef struct _light_pcapng_file_info {
uint16_t major_version;
uint16_t minor_version;
char *file_comment;
size_t file_comment_size;
char *hardware_desc;
size_t hardware_desc_size;
char *os_desc;
size_t os_desc_size;
char *user_app_desc;
size_t user_app_desc_size;
size_t interface_block_count;
uint16_t link_types[MAX_SUPPORTED_INTERFACE_BLOCKS];
double timestamp_resolution[MAX_SUPPORTED_INTERFACE_BLOCKS];
} light_pcapng_file_info;
EDIT:
In addition to accepted answer i'd like to provide some info my colleague pointed me to. There was a bug in glibc which, sometimes, prevented calloc from zeroing out memory. Here's the link:
https://bugzilla.redhat.com/show_bug.cgi?id=1293976
Actual bug report text in case link gets moved:
glibc: calloc() returns non-zero'ed memory
Description of problem:
At Facebook we had an app that started hanging and crashing weirdly when going from glibc-2.12-1.149.el6.x86_64 to glibc-2.12-1.163.el6.x86_64. Turns out this patch
glibc-rh1066724.patch
Introduced the problem.
You added the following bit to _int_malloc()
/* There are no usable arenas. Fall back to sysmalloc to get a chunk from
mmap. */
if (__glibc_unlikely (av == NULL))
{
void *p = sYSMALLOc (nb, av);
if (p != NULL)
alloc_perturb (p, bytes);
return p;
}
But this isn't ok, alloc_perturb unconditionally memset's the front byte to 0xf, unlike upstream where it checks to see if perturb_byte is set. This needs to be changed to
if (p != NULL && && __builtin_expect(perturb_byte, 0))
alloc_perturb (p, bytes);
return p;
The patch I've attached fixes the problem for me.
This problem is exacerbated by the fact that any sort of lock contention on the arena's results in us falling back on mmap()'ing a new chunk. This is because we check to see if the uncontended arena we check is corrupt, and if it is we loop through, and if we loop to the beginning we know we didn't find anything. Except if our initial arena isn't actually corrupt we'll still return NULL, so we fall back on this mmap() thing more often, which really makes things unstable.
Please get this fixed as soon as possible, I'd even go so far as to call it a possible security issue.
Calling memset() ensures that the OS actually does the virtual memory mapping. As noted in the answers on the question you linked, calloc() can be optimized so that the actual memory mapping is deferred.
The application might have reasons to not defer the actual creation of the virtual memory mapping - such as using the buffer to read from a very high-speed device, although in the case of using memset() to zero out the memory, using calloc() instead of malloc() does seem redundant.
Nobody is perfect, that's all. Yes, memset to zero following a calloc is extravagant. If you want an explicit memset in order to guarantee you are in possession of the memory that you've asked for, then it should follow a malloc instead.
New code has about 20 bugs per 1,000 lines and the laws of probability imply that not all of them are weeded out. Plus this is not really a bug, as there's no ill-behaviour to be observed.
I would call it a bug, since it's doing pointless work, calloc() is specified to return already-cleared memory so why clear it again? Perhaps a failed refactoring, when someone switched from malloc().
If the code is open source and/or in a repository you have access to, I would check the commit history for those lines and see what has been going on. With a bit of luck, there's a commit message that tells the motivation ...
Related
When I have a pointer which should repeatedly be used as argument to realloc and to save it's return value, I understand that realloc won't touch the old object if no allocation could take place, returning NULL. Should I still be worried about the old object in a construction like:
int *p = (int *)malloc(sizeof(int *));
if (p = (int *)realloc(p, 2 * sizeof(int *)))
etc...
Now if realloc succeeds, I need to free(p) when I'm done with it.
When realloc fails, I have assigned it's return NULL to p and free(p) doesn't do anything (as it is a free(NULL)). At the same time (according to the standard) the old object is not deallocated.
So should I have a copy of p (e.g. int *last_p = p;) for keeping track of the old object, so that I can free(last_p) when realloc fails?
So should I have a copy of p (e.g. int *last_p = p;) for keeping track of the old object, so that I can free(last_p) when realloc fails?
Basically: yes, of course.
It is generally not suggested to use the pattern you showed:
p = (int *)realloc(p, ...)
This is considered bad practice for the reason you found out.
Use this instead:
void *new = realloc(p, ...);
if (new != NULL)
p = new;
else
...
It is also considered bad practice to cast the return value of malloc, realloc etc. in C. That is not required.
Should you save a temporary pointer?
It can be a good thing in certain situations, as have been pointed out in previous answers. Provided that you have a plan for how to continue execution after the failure.
The most important thing is not how you handle the error. The important thing is that you do something and not just assume there's no error. Exiting is a perfectly valid way of handling an error.
Don't do it, unless you plan a sensible recovery
However, do note that in most situations, a failure from realloc is pretty hard to recover from. Often is exiting the only sensible option. If you cannot acquire enough memory for your task, what are you going to do? I have encountered a situation where recovering was sensible only once. I had an algorithm for a problem, and I realized that I could make significant improvement to performance if I allocated a few gigabytes of ram. It worked fine with just a few kilobytes, but it got noticeably faster with the extra ram usage. So that code was basically like this:
int *huge_buffer = malloc(1000*1000*1000*sizeof *hugebuffer);
if(!huge_buffer)
slow_version();
else
fast_version();
In those cases, just do this:
p = realloc(p, 2 * sizeof *p)
if(!p) {
fprintf(stderr, "Error allocating memory");
exit(EXIT_FAILURE);
}
Do note both changes to the call. I removed casting and changed the sizeof. Read more about that here: Do I cast the result of malloc?
Or even better, if you don't care in general. Write a wrapper.
void *my_realloc(void *p, size_t size) {
void *tmp = realloc(p, size);
if(tmp) return tmp;
fprintf(stderr, "Error reallocating\n");
free(p);
exit(EXIT_FAILURE);
return NULL; // Will never be executed, but to avoid warnings
}
Note that this might contradict what I'm writing below, where I'm writing that it's not always necessary to free before exiting. The reason is that since proper error handling is so easy to do when I have abstracted it all out to a single function, I might as well do it right. It only took one extra line in this case. For the whole program.
Related: What REALLY happens when you don't free after malloc?
About backwards compatibility in general
Some would say that it's good practice to free before exiting, just because it d actually does matter in certain circumstances. My opinion is that these circumstances are quite specialized, like when coding embedded systems with an OS that does not free memory automatically when they terminate. If you're coding in such an environment, you should know that. And if you're coding for an OS that does this for you, then why not utilize it to keep down code complexity?
In general, I think some C coders focuses too much on backwards compatibility with ancient computers from the 1970th that you only encounter on museums today. In most cases, it's pretty fair to assume ascii, two complement, char size of 8 bits and such things.
A comparison would be to still code web pages so that they are possible to view in Netscape Navigator, Mosaic and Lynx. Only spend time on that if there really is a need.
Even if you skip backwards compatibility, use some guards
However, whenever you make assumptions it can be a good thing to include some meta code that makes the compilation fail with wrong target. For instance with this, if your program relies on 8 bit chars:
_Static_assert(CHAR_BITS == 8, "Char bits");
That will make your program crash upon compilation. If you're doing cross compiling, this might possibly be more complicated. I don't know how to do it properly then.
So should I have a copy of p (e.g. int *last_p = p;) for keeping track of the old object, so that I can free(last_p) when realloc fails?
Yes, exactly. Usually I see the new pointer is assigned:
void *pnew = realloc(p, 2 * sizeof(int *)))
if (pnew == NULL) {
free(p);
/* handle error */
return ... ;
}
p = pnew;
If do the next:
int* array = malloc(10 * sizeof(int));
and them I use realloc:
array = realloc(array, 5 * sizeof(int));
On the second line (and only it), can it return NULL?
Yes, it can. There are no implementation guarantees on realloc(), and it can return a different pointer even when shrinking.
For example, if a particular implementation uses different pools for different object sizes, realloc() may actually allocate a new block in the pool for smaller objects and free the block in the pool for larger objects. Thus, if the pool for smaller objects is full, it will fail and return NULL.
Or it may simply decide it's better to move the block
I just used the following program to get size of actually allocated memory with glibc:
#include <stdlib.h>
#include <stdio.h>
int main()
{
int n;
for (n = 0; n <= 10; ++n)
{
void* array = malloc(n * sizeof(int));
size_t* a2 = (size_t*) array;
printf("%d -> %zu\n", n, a2[-1]);
}
}
and for n <= 6, it allocates 32 bytes, and for 7-10 it is 48.
So, if it shrank int[10] to int[5], the allocated size would shrink from 48 to 32, effectively giving 16 free bytes. Since (as it just has been noted) it won't allocate anything less than 32 bytes, those 16 bytes are lost.
If it moved the block elsewhere, the whole 48 bytes will be freed, and something could actually be put in there. Of course, that's just a science-fiction story and not a real implementation ;).
The most relevant quote from the C99 standard (7.20.3.4 The realloc function):
Returns
4 The realloc function returns a pointer to the new object (which may have the same value as a pointer to the old object), or a null pointer if the new object could not be allocated.
'May' is the key-word here. It doesn't mention any specific circumstances when that can happen, so you can't rely on any of them, even if they sound obvious at a first glance.
By the way, I think you could consider realloc() somewhat deprecated. If you'd take a look at C++, the newer memory allocation interfaces (new / delete and allocators) don't even support such a thing. They always expect you to allocate a new block. But that's just a loose comment.
The other answers have already nailed the question, but assuming you know the realloc call is a "trimming", you can wrap it with:
void *safe_trim(void *p, size_t n) {
void *p2 = realloc(p, n);
return p2 ? p2 : p;
}
and the return value will always point to an object of size n.
In any case, since the implementation of realloc knows the size of the object and can therefore determine that it's "trimming", it would be pathologically bad from a quality-of-implementation standpoint not to perform the above logic internally. But since realloc is not required to do this, you should do it yourself, either with the above wrapper or with analogous inline logic when you call realloc.
The language (and library) specification makes no such guarantee, just like it does not guarantee that a "trimming" realloc will preserve the pointer value.
An implementation might decide to implement realloc in the most "primitive" way: by doing an unconditional malloc for a new memory block, copying the data and free-ing the old block. Obviously, such implementation can fail in low-memory situations.
Don't count on it. The standard makes no such provision; it merely states "or a null pointer if the new object could not be allocated".
You'd be hard-pressed to find such an implementation, but according to the standard it would still be compliant.
I suspect there may be a theoretical possibility for failure in the scenario you describe.
Depending on the heap implementation, there may be no such a thing as trimming an existing allocation block. Instead a smaller block is allocated first, then the data is copied from the old one, and then it's freed.
For instance this may be the case with bucket-heap strategy (used by some popular heaps, such as tcmalloc).
A bit late, but there is at least one popular implementation which realloc() with a smaler size can fail: TCMalloc. (At least as far as i understand the code)
If you read the file tcmalloc.cc, in the function do_realloc_with_callback(), you will see that if you shrink enough (50% of alloced memory, otherwise it will be ignored), TCMalloc will alloc the new memory first (and possible fail) and then copy it and remove the old memory.
I do not copy the source code, because i am not sure if the copyrights (of TCMalloc and Stackoverflow) will allow that, but here is a link to the source (revision as at May 17, 2019).
realloc will not fails in shrinking the existing memory, so it will not return NULL. It can return NULL only if fails during expansion.
But shrinking can fail in some architecture, where realloc can be implemented in a different manner like allocating a smaller size memory separately and freeing the old memory to avoid fragmentation. In that case shrinking memory can return NULL. But its very rare implementation.
But its better to be in a safer side, to keep NULL checks after shrinking the memory also.
I wrote a code to test to stress test the memory management of Linux and Windows OS. Just for further tests I went ahead and checked what values are present in the memory returned by malloc().
The values that are being return are all 0 (zero). I have read the man page of malloc, checked on both Windows and Linux, but I am not able to find the reason for this behavior. According to the manpage the
The malloc() function allocates size bytes and returns a pointer to the allocated memory. The memory is not initialized.
To clear the memory segment, one has to manually use memset().
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <ctype.h>
#include <stdbool.h>
int eat(long total,int chunk){
long i;
for(i=0;i<total;i+=chunk){
short *buffer=malloc(sizeof(char)*chunk);
if(buffer==NULL){
return -1;
}
printf("\nDATA=%d",*buffer);
memset(buffer,0,chunk);
}
return 0;
}
int main(int argc, char *argv[]){
int i,chunk=1024;
long size=10000;
printf("Got %ld bytes in chunks of %d...\n",size,chunk);
if(eat(size,chunk)==0){
printf("Done, press any key to free the memory\n");
getchar();
}else{
printf("ERROR: Could not allocate the memory");
}
}
Maybe I am missing something.
The code is adapted from here
EDIT: The problem has been been answered here for the GCC specific output. I believe Windows operating system would be also following the same procedures.
The memory returned by malloc() is not initialized, which means it may be anything. It may be zero, and it may not be; 'not initialized' means it could be anything (zero included). To get a guaranteed zeroed page use calloc().
The reason you are seeing zeroed pages (on Linux anyway) is that if an application requests new pages, these pages are zeroed by the OS (or more precisely they are copy-on-write images of a fixed page of zeroes known as the 'global zero page'). But if malloc() happens to use memory already allocated to the application which has since been freed (rather than expanding the heap) you may well see non-zero data. Note the zeroing of pages provided by the OS is an OS specific trait (primarily there for security so that one process doesn't end up with pages that happen to have data from another process), and is not mandated by the C standard.
You asked for a source for get_free_page zeroing the page: that says 'get_free_page() takes one parameter, a priority. ... It takes a page off of the free_page_list, updates mem_map, zeroes the page and returns the physical address of the page.' Here's another post that explains it well, and also explains why using calloc() is better than malloc()+memset().
Note that you aren't checking the entire allocated chunk for zero. You want something like this (untested):
int n;
char nonzero=0;
char *buffer=malloc(sizeof(char)*chunk);
if(buffer==NULL){
return -1;
}
for (n = 0; n<chunk; n++)
nonzero = nonzero || buffer[n];
printf("\nDATA=%s\n",nonzero?"nonzero":"zero");
You're absolutely correct; this behaviour is not guaranteed by the C language standard.
What you're observing could just be chance (you're only checking a couple of bytes in each allocation), or it could be an artifact of how your OS and C runtime library are allocating memory.
With this statement:
printf("\nDATA=%d",*buffer);
You only check the first sizeof(short) amount of bytes that have just been malloc()'ed (typically two (2) bytes).
Furthermore, the first time you may get lucky of getting all zeroes but after having had your program execute (and use) the heap memory then the contents-after-malloc() will be undefined.
the memory allocation function: calloc() will return a pointer to the 'new area and set all the bytes to zero.
The memory allocation function: realloc() will return a pointer to a (possibly new) area and have copied the bytes from the old area. The new area will be the 'new' requested length
The memory allocation function malloc will return a pointer to the new area but will not set the bytes to any specific value
The values that are being return are all 0 (zero).
But that's not guaranteed. It's because you're just running your program. If you malloc, random fill, and free a lot, you'll start noticing the previously freed memory is being reused, so you'll start to get non-zero chunks in your mallocs.
Yes you are right malloc() doesn't zero-initialize values. It arbitrarily pulls the amount of memory it's told to allocate from the heap, which essentially means there could be anything stored already within. You should therefore use malloc() only, where you're certain, that you are going to set it to a value. If you're going to do arithmetic with it right out of the box you might get some fishy results (I have already several times personally experienced this; you're going to have functional code with sometimes crazy output).
So set stuff you're not setting to a value to zero with memset(). Or my advise is to use calloc(). Calloc, other than malloc, does zero-initialize values. And is as far as I know faster than the combination of malloc() and memset() on the other hand malloc alone is faster than calloc. So try to find the fastest version possible at point of issue by keeping you're memory in form.
Look also at this post here: MPI matrix-vector-multiplication returns sometimes correct sometimes weird values. The question was a different one, but the cause the same.
If do the next:
int* array = malloc(10 * sizeof(int));
and them I use realloc:
array = realloc(array, 5 * sizeof(int));
On the second line (and only it), can it return NULL?
Yes, it can. There are no implementation guarantees on realloc(), and it can return a different pointer even when shrinking.
For example, if a particular implementation uses different pools for different object sizes, realloc() may actually allocate a new block in the pool for smaller objects and free the block in the pool for larger objects. Thus, if the pool for smaller objects is full, it will fail and return NULL.
Or it may simply decide it's better to move the block
I just used the following program to get size of actually allocated memory with glibc:
#include <stdlib.h>
#include <stdio.h>
int main()
{
int n;
for (n = 0; n <= 10; ++n)
{
void* array = malloc(n * sizeof(int));
size_t* a2 = (size_t*) array;
printf("%d -> %zu\n", n, a2[-1]);
}
}
and for n <= 6, it allocates 32 bytes, and for 7-10 it is 48.
So, if it shrank int[10] to int[5], the allocated size would shrink from 48 to 32, effectively giving 16 free bytes. Since (as it just has been noted) it won't allocate anything less than 32 bytes, those 16 bytes are lost.
If it moved the block elsewhere, the whole 48 bytes will be freed, and something could actually be put in there. Of course, that's just a science-fiction story and not a real implementation ;).
The most relevant quote from the C99 standard (7.20.3.4 The realloc function):
Returns
4 The realloc function returns a pointer to the new object (which may have the same value as a pointer to the old object), or a null pointer if the new object could not be allocated.
'May' is the key-word here. It doesn't mention any specific circumstances when that can happen, so you can't rely on any of them, even if they sound obvious at a first glance.
By the way, I think you could consider realloc() somewhat deprecated. If you'd take a look at C++, the newer memory allocation interfaces (new / delete and allocators) don't even support such a thing. They always expect you to allocate a new block. But that's just a loose comment.
The other answers have already nailed the question, but assuming you know the realloc call is a "trimming", you can wrap it with:
void *safe_trim(void *p, size_t n) {
void *p2 = realloc(p, n);
return p2 ? p2 : p;
}
and the return value will always point to an object of size n.
In any case, since the implementation of realloc knows the size of the object and can therefore determine that it's "trimming", it would be pathologically bad from a quality-of-implementation standpoint not to perform the above logic internally. But since realloc is not required to do this, you should do it yourself, either with the above wrapper or with analogous inline logic when you call realloc.
The language (and library) specification makes no such guarantee, just like it does not guarantee that a "trimming" realloc will preserve the pointer value.
An implementation might decide to implement realloc in the most "primitive" way: by doing an unconditional malloc for a new memory block, copying the data and free-ing the old block. Obviously, such implementation can fail in low-memory situations.
Don't count on it. The standard makes no such provision; it merely states "or a null pointer if the new object could not be allocated".
You'd be hard-pressed to find such an implementation, but according to the standard it would still be compliant.
I suspect there may be a theoretical possibility for failure in the scenario you describe.
Depending on the heap implementation, there may be no such a thing as trimming an existing allocation block. Instead a smaller block is allocated first, then the data is copied from the old one, and then it's freed.
For instance this may be the case with bucket-heap strategy (used by some popular heaps, such as tcmalloc).
A bit late, but there is at least one popular implementation which realloc() with a smaler size can fail: TCMalloc. (At least as far as i understand the code)
If you read the file tcmalloc.cc, in the function do_realloc_with_callback(), you will see that if you shrink enough (50% of alloced memory, otherwise it will be ignored), TCMalloc will alloc the new memory first (and possible fail) and then copy it and remove the old memory.
I do not copy the source code, because i am not sure if the copyrights (of TCMalloc and Stackoverflow) will allow that, but here is a link to the source (revision as at May 17, 2019).
realloc will not fails in shrinking the existing memory, so it will not return NULL. It can return NULL only if fails during expansion.
But shrinking can fail in some architecture, where realloc can be implemented in a different manner like allocating a smaller size memory separately and freeing the old memory to avoid fragmentation. In that case shrinking memory can return NULL. But its very rare implementation.
But its better to be in a safer side, to keep NULL checks after shrinking the memory also.
I've seen a lot of code that checks for NULL pointers whenever an allocation is made. This makes the code verbose, and if it's not done consistently, only when the programmer felt like it, doesn't even ensure that the program won't crash when the address space runs out. Besides, if the program can't make more allocations, it wouldn't be able to do its function anyway, right?
So my question is, isn't it better for most programs not to check at all and just let the program crash if memory runs out? At least the code is more readable that way.
Note
I'm talking about desktop apps that run on modern computers (at least 2 GB address space), and that most definitely don't operate space shuttles, life support systems, or BP's oil platforms. Most importantly I'm talking about programs that use malloc but never really go above 5 MB of memory usage.
Always check the return value, but for clarity, it's common to wrap malloc() in a function which never returns NULL:
void *
emalloc(size_t amt){
void *v = malloc(amt);
if(!v){
fprintf(stderr, "out of mem\n");
exit(EXIT_FAILURE);
}
return v;
}
Then, later you can use
char *foo = emalloc(56);
foo[12] = 'A';
With no guilty conscience.
Yes, you should check for a null return value from malloc. Even if you can't recover from the failure of memory allocation you should explicitly exit. Carrying on as though memory allocation had succeeded leaves your application in an inconsistent state and is likely to cause "undefined behavior" which should be avoided.
For example, you may end up writing inconsistent data to external storage which may hinder the ability of the next run of the application to recover. It's much safer to exit swiftly in a more controlled fashion.
Many applications that want to exit on allocation failure wrap malloc in a function that checks the return value and explicitly aborts on failure.
Arguably, this is one advantage of the C++ default new approach to throw an exception on allocation failure. It requires no effort to exit on memory allocation failure.
Similar to Dave's approach above, but adds a macro that automatically passes
the file name and line number to our allocation routine so that we can report
that information in the event of a failure.
#include <stdio.h>
#include <stdlib.h>
#define ZMALLOC(theSize) zmalloc(__FILE__, __LINE__, theSize)
static void *zmalloc(const char *file, int line, int size)
{
void *ptr = malloc(size);
if(!ptr)
{
printf("Could not allocate: %d bytes (%s:%d)\n", size, file, line);
exit(1);
}
return(ptr);
}
int main()
{
/* -- Set 'forceFailure' to a non-zero value in order to observe
how 'zmalloc' behaves when it cannot allocate the
requested memory -- */
int bytes = 10 * sizeof(int);
int forceFailure = 0;
int *anArray = NULL;
if(forceFailure)
bytes = -1;
anArray = ZMALLOC(bytes);
free(anArray);
return(0);
}
but it is much more difficult to troubleshoot if you don't log where the malloc failed.
failed to allocate memory in line XX is to prefer than just to crash.
You should definitely check the return value for malloc. Helpful in debugging and the code becomes robust.
Always check malloc'ed memory?
In a hosted environment error checking the return of malloc makes not much sense nowadays. Most machines have a virtual address space of 64 bit. You'd need a lot of time to exhaust that. Your program will most likely fail at a completely different place, namely when your physical+swap memory is exhausted. It will have shown completely ridiculous performance before that, because it only was swapping and the user will have triggered Cntrl-C long before you ever come there.
Segfaulting "nicely" on a null pointer reference would be a clear point to see where things fail in a debugger. But in my practice I have never seen a failed malloc as a cause.
When programming for embedded systems the picture changes completely. There you definitively should check for failed malloc.
Edit: To clarify that after the edit of the question. The kind of programs/systems described there are clearly not "embedded". I have never seen malloc fail under the circumstances described there.
I'd like to add that edge cases should always be checked even if you think they are safe or cannot lead to other issues than a crash. Null pointer dereference can potentially be exploited (http://uninformed.org/?v=4&a=5&t=sumry).