Using the Google Test framework, I want to be able to determine that a program crashes if I accidentally free() allocated memory twice. I expected something the following to catch the crash:
TEST(DEATHTEST, InvalidFree)
{
::testing::FLAGS_gtest_death_test_style = "threadsafe";
int *data = (int *)malloc(sizeof(int));
free(data);
EXPECT_DEATH(free(data), ".*");
}
If I actually free() the memory twice outside of the test framework, the program crashes. However, the test case fails, reporting:
Death test: free(data)
Result: failed to die.
Error msg:
[ DEATH ]
[ FAILED ] DEATHTEST.InvalidFree (3 ms)
I have also tried to:
EXPECT_EXIT(free(data), ::testing::KilledBySignal(SIGABRT), ".*");
and I've tried with other signals like SIGSEGV.
What am I doing wrong? Is it simply not possible to catch this kind of crash? (I realize that the behavior of a double free is undefined but I'd expect that a crash should nonetheless be detectable.)
Note: I don't actually want to test that a double free crashes. Instead, I want to verify that a function frees some allocated memory. Since I can't readily mock the free() function, instead I thought I could verify that trying to free the memory after the function has executed would indirectly test the function's memory deallocation.
Related
I'm trying to call malloc() in an infinite loop. This might sound weird, but I actually wanted to experiment with what would happen if I did this.
My main method:
void main() {
while (true) {
int64_t* ptr = malloc(sizeof(int64_t));
if (ptr == NULL) {
printf("Out of memory\n");
break;
}
}
}
malloc() is supposed to return NULL if OS refuses to allocate more memory. When I run this program, my PC shows that memory usage is increasing even maximum allowed page file memory is committed. But the program doesn't show that "Out of memory" on the console and the operating system crashes with the Blue Screen showing the error CRITICAL PROCESS DIED.
I understand that this crash is caused by my program. But my question is why malloc didn't return NULL if the OS is running out of memory...
I compiled this program using MSVC, which comes with VS 2022.
I am trying to create a multithreading library in c. Here is the link to whole project (because pasting the code here would be too much text).
In the file tests/MultithreadingTests.c I am testing to the functionality of lib/systems/multithreading/src/ThreadPool.c. The function add_work adds any routine function the the work queue which utilises the functionality of lib/sds/lists/src/Queue.c and lib/sds/lists/src/LinkedList.c. In MultithreadingTests.c, NUM_TESTS defines the number of jobs I am adding to the work queue to be performed by NUM_THREADS
I am facing a weird issue with the code. If NUM_TESTS any number is less than 349,261, the code works perfectly fine but any number higher than or equal to 349,261 results in segmentation fault. I tried to check where exactly the segmentation fault is happening and found that it happens in the lib/sds/lists/src/Node.c at line number 29 at memcpy(node->data, data, size);
The flow of code for the error is
tests/MultiThreadingTests.c line 95 at pool->add_work(pool, new_thread_job(routine, &arguments[i]));
lib/systems/multithreading/src/ThreadPool.c line 150 thread_pool->work.push(&thread_pool->work, &job, sizeof(job));
lib/sds/lists/src/Queue.c line 54 return q->list.insert(&q->list, q->list.length, data, size);
lib/sds/lists/src/LinkedLists.c line 107 Node *node_to_insert = new_node(data, size);
lib/sds/lists/src/Node.c line 29 memcpy(node->data, data, size);
I am not sure why this issue is happening only when the number of jobs is higher than or equal to 349,261 but not when its smaller.
In function new_thread_pool(), you neither
test for allocation failure in thread_pool.pool = malloc(sizeof(pthread_t) * num_threads); nor
test for thread creation failure in pthread_create(&thread_pool.pool[i], NULL, generic_thread_function, &thread_pool);
Trying to create 349261 or more threads on any system looks more like a stress test than a real life purpose. Test for errors and report them in a usable way.
new_node does not check for allocation failure either. Unless you instrument your code for this, you should use a wrapper around malloc() calls to detect allocation failure and abort the program with an error message.
The issue in your code is in the function mt_test_add_work(): you define an array of arguments with automatic storage:
Arguments arguments[NUM_TESTS];
This object is allocated on the stack, using 8382264 bytes of stack space. This is too much for your system and causes undefined behavior down the call chain where further stack usage actually cause a segmentation fault: a typical case of Stack Overflow.
You should allocate this object from the heap and free it before exiting the function:
Arguments *arguments = malloc(sizeof(*arguments) * NUM_TESTS);
I'm writing a multithreaded server in C for a uni project and I'm having a hard time figuring out how to do error handling in a nice, readable and standard way.
Right now, if the program terminates successfully, I free every bit of allocated memory at the end of it. But what if a fatal error occurs during the execution (e.g. malloc returns NULL)?
For example, let's say I have a custom data type mydata_t and a constructor function mydata_t *mydata_init() that is used by several modules of my program. After seeing some code on the internet, this is how I would have written it:
mydata_t *mydata_init() {
mydata_t *mydata = malloc(sizeof(mydata_t));
if (!mydata) return NULL;
mydata->field1 = malloc(sizeof(mydata2_t));
if (!mydata->field1) return NULL;
mydata->field2 = malloc(sizeof(mydata3_t));
if (!mydata->field2) return NULL;
/*
Initialization of other fields
*/
return mydata;
}
It does seem nice and clean, but is this the "standard" way to do it?
In particular, what if one of the mallocs returns NULL? Is it necessary to free all the previously allocated memory? Is it reasonable to change the code to something like this?
mydata_t *mydata_init() {
mydata_t *mydata = malloc(sizeof(mydata_t));
if (!mydata) goto err_1;
mydata->field1 = malloc(sizeof(mydata2_t));
if (!mydata->field1) goto err_2;
mydata->field2 = malloc(sizeof(mydata3_t));
if (!mydata->field2) goto err_3;
/*
Initialization of other fields
*/
return mydata;
/*
Other tags
*/
err_3:
free(mydata->field1);
err_2:
free(mydata);
err_1:
return NULL;
}
Assuming a non-trivial OS, that's one with 'kill-9' or a Task Manger with 'End Process' entry on a GUI context menu, please bear in mind the following points before embarking on a lenghty and expensive campaign to write specific user code to free all malloced/whatever memory upon a fatal error:
1) Freeing all memory with user code requires more user code. That extra code must be designed, coded tested and maintained, often repeatedly after changes and/or new versions. With a complex, multithreaded app, maybe with pools of objects that are shared and communicated between threads, it's not going to be a remotely trivial exercise to even try to shut it down with user code.
2) Freeing all memory with user code after a fatal error can make things worse. If the error is a result of a corrupted heap manager, then you will raise even more faults as you try to free it. An app 'diasppearing' on a customer with an error log entry is bad enough, a screen full of AV error boxes and a stuck app is much worse.
3) Safely freeing all memory with user code can only be done by a thread if all other threads have been stopped so that they cannot possibly access any of that memory. Reliably stopping all process threads can only be done by the OS at process termination. User code cannot be guaranteed to do it - if a thread is stuck executing a lengthy operation in a library, you cannot reliably stop it. If you try, it may, for instance, leave the memory-manager locked. Just unblocking threads stuck on an I/O operation is difficult enough, often requiring bodges like opening a connection on the local network stack to force an accept() call to return early.
4) The OS 'terminate process' API, and all that is involved in it, has been tested a LOT. It works, and it comes free with your OS. Your user code that tries to stop threads and free memory will never accumulate as much testing.
5) User code that tries to stop threads and free memory is redundant - the OS does the same job, only better, quicker and more reliably. You are trying to clean up memory from a sub-allocator that the OS is going to soon destroy and deallocate anyway.
6) Many OS and other commercial libraries have already given way to the inevitable and accept that they cannot safely have all their memory freed at shutdown without causing problems, especially with multithreaded apps. The library authors cannot do it reliably, neither can you.
Sure, manage your memory during a run in a controlled and sensible manner, freeing what you malloc as required, so as not to leak memory during the lifetime of the process.
If you encounter a fatal error, however, maybe try to write the details to a log file or make some other debugging/logging action, if you can, and then call your OS 'terminate process' API.
There is nothing else you can safely do.
Your app is near death, let the OS euthanize it.
One possible alternative.
mydata_t *mydata_init()
{
mydata_t *mydata = malloc(sizeof(mydata_t));
if (mydata == NULL)
{
/* Handle error */
return NULL;
}
mydata->field1 = malloc(sizeof(mydata2_t));
mydata->field2 = malloc(sizeof(mydata3_t));
...
if (mydata->field1 != NULL &&
mydata->field2 != NULL &&
...)
{
/* success */
/*
* Initialize everything
*/
return mydata;
}
free(mydata->field1);
free(mydata->field2);
...
free(mydata);
return NULL;
}
Note that you do not need to check for NULL before calling free() on the error path. As noted in the first answer to this question
Quoting the C standard, 7.20.3.2/2 from ISO-IEC 9899:
If ptr is a null pointer, no action occurs.
Is it necessary to free all the previously allocated memory?
No, but non-leaky (good) code does.
You will find various opinions on how to do this (free up things), but the end goal is to do it, somehow. Free unused resources.
Code for clarity.
Note: goto form is an acceptable use of goto.
I'll offer another approach and use a mydata_uninit() companion function as possible.
mydata_t *mydata_uninit(mydata_t *mydata) {
if (mydata) {
free(mydate->field1);
mydate->field1 = NULL; // example discussed below **
free(mydate->field2);
// free other resources and perform other clean up/accounting.
free(mydate);
}
return NULL;
}
I'd also allocate to the size of the de-referenced pointer, not the type.
mydata_t *mydata_init(void) {
mydata_t *mydata = calloc(1, sizeof *mydata);
if (mydata == NULL) {
return NULL;
}
mydata->field1 = calloc(1, sizeof *(mydata->field1));
mydata->field2 = calloc(1, sizeof *(mydata->field2));
// Other 1st time assignments
// If any failed
if (mydata->field1 == NULL || mydata->field2 == NULL) {
mydata = mydata_uninit(mydata);
}
return mydata;
}
** Setting pointer struct members to NULL (mydata->field1) and then later freeing the struct pointer (mydata) I find aids in debug as errant code that de-references a NULL pointer typically errors faster than a free'd pointer.
The existing answers covers the various malloc/free scenarios, so I will not add to that.
What I would like to point out is that if you fail a malloc, it is pretty much game-over for the program, and it is often better to just exit() than to try to recover the situation.
Even if you manage to clean up partially allocated structs, other code, and libraries used by that code, will also fail to allocate memory and may (and often will) fail in mysterious ways.
malloc() usually only fails if you run in severely restricted environments, like embedded micro controllers), or if you are leaking memory like a sieve.
In C, I know it is good practice to always check if a newly malloced variable is null right after allocating. If so, I can output an error e.g. perror and exit the program.
But what about in more complicated programs? E.g. I have main call a function f1(returns an int), which calls a function f2(returns a char*), which calls a function f3(returns a double), and I fail to malloc inside f3.
In this case, I can't seem to just output an error and exit(and may even have memory leaks if possible) since f3 will still force me to first return a double. Then f2 will force me to return a char*, etc. In this case, it seems very painful to keep track of the errors and exit appropriately. What is the proper way to efficiently cover these sort of errors accross functions?
The obvious solution is to design your program with care, so that every function that does dynamic allocation has some means to report errors. Most often the return value of the function is used for this purpose.
In well-designed programs, errors bounce back all the way up the call stack, so that they are dealt with at the application level.
In the specific case of dynamic memory allocation, it is always best to leave the allocation to the caller whenever possible.
It's always a problem. You need a disiplined approach.
Firstly, every dynamic pointer must be "owned" by someone. C won't help you here, you just have to specify. Generally the three patterns are
a) Function calls malloc(), then calls free():
b) We have two matching functions, one which returns a buffer or dynamic
structure, one which destroys it. The function that calls create also calls the destroy.
c) We have a set of nodes we are inserting into a graph, at random throughout the program. It needs to be managed like b, one function creates the root, then calls the delete which destroys the entire graph.
The rule is owner holds and frees.
If you return a pointer, return 0 on out of memory. If you return an integer, return -1. Errors get propagated up until some high level code knows what user-level operation has failed and aborts it.
The other answers are correct that the correct way to handle this is to make sure that every function that can allocate memory can report failure to its caller, and every caller handles the possibility. And, of course, you have a test malloc shim that arranges to test every possible allocation failure.
But in large C programs, this becomes intractable — the number of cases that need testing increases exponentially with the number of malloc callsites, for starters — so it is very common to see a function like this in a utils.c file:
void *
xmalloc(size_t n)
{
void *rv = malloc(n);
if (!rv) {
fprintf(stderr, "%s: memory exhausted\n", program_name);
exit(1);
}
return rv;
}
All other code in the program always calls xmalloc, never malloc, and can assume it always succeeds. (And you also have xcalloc, xrealloc, xstrdup, etc.)
Libraries cannot get away with this, but applications can.
The one way to switch across the functions is exception handling.
When an exception is thrown, it return the scope the catch part.
But make sure of the memory allocation across the functions, Since it directly moves to the catch blok.
The sample code for reference,
// Example program
#include <iostream>
#include <string>
using namespace std ;
int f1()
{
int *p = (int*) malloc(sizeof(int)) ;
if(p == NULL)
{
throw(1) ;
}
//Code flow continues.
return 0 ;
}
char *g()
{
char *p ;
f1() ;
cout << "Inside fun g*" << endl ;
return p;
}
int f2()
{
g() ;
cout << "Inside fun f2" << endl ;
return 0 ;
}
int main()
{
try
{
f2() ;
}
catch(int a)
{
cout << "Caught memory exception" << endl ;
}
return 0 ;
}
The Matasano blog calls “Checking the return value of malloc()” a “C Programming Anti-Idiom.” Instead malloc() should automatically call abort() for you if it fails. The argument is that, since you’ll usually want to abort the program if malloc() fails, that should be the default behaviour instead of something you have to laboriously type—or maybe forget to type!—every time.
Without getting into the merits of the idea, what’s the easiest way to set this up? I’m looking for something that would automatically detect memory allocation failures by other library functions such as asprintf() too. A portable solution would be wonderful, but I’d also be happy with something mac-specific.
Summarizing the best answers below:
Mac run-time solution
Set the MallocErrorAbort=1 environment variable before running your program. Automatically works for all memory allocation functions.
Mac/linux run-time solution
Use a dynamic library shim to load a custom malloc() wrapper at runtime with LD_PRELOAD or DYLD_INSERT_LIBRARIES. You will likely want to wrap calloc(), realloc(), &c. as well.
Mac/linux compiled solution
Define your own malloc() and free() functions, and access the system versions using dyld(RTLD_NEXT, "malloc") as shown here. Again, you will likely want to wrap calloc(), realloc(), &c. as well.
#include <dlfcn.h>
#include <stdio.h>
#include <stdlib.h>
void *(*system_malloc)(size_t) = NULL;
void* malloc(size_t bytes) {
if (system_malloc == NULL) {
system_malloc = dlsym(RTLD_NEXT, "malloc");
}
void* ret = system_malloc(bytes);
if (ret == NULL) {
perror("malloc failed, aborting");
abort();
}
return ret;
}
int main() {
void* m = malloc(10000000000000000l);
if (m == NULL) {
perror("malloc failed, program still running");
}
return 0;
}
Linux compiled solution
Use __malloc_hook and __realloc_hook as described in the glibc manual.
Mac compiled solution
Use the malloc_default_zone() function to access the heap’s data structure, unprotect the memory page, and install a hook in zone->malloc:
#include <malloc/malloc.h>
#include <stdio.h>
#include <stdlib.h>
#include <sys/mman.h>
#include <unistd.h>
static void* (*system_malloc)(struct _malloc_zone_t *zone, size_t size);
static void* my_malloc(struct _malloc_zone_t *zone, size_t size) {
void* ret = system_malloc(zone, size);
if (ret == NULL) {
perror("malloc failed, aborting");
abort();
}
return ret;
}
int main() {
malloc_zone_t *zone = malloc_default_zone();
if (zone->version != 8) {
fprintf(stderr, "Unknown malloc zone version %d\n", zone->version);
abort();
}
system_malloc = zone->malloc;
if (mprotect(zone, getpagesize(), PROT_READ | PROT_WRITE) != 0) {
perror("munprotect failed");
abort();
}
zone->malloc = my_malloc;
if (mprotect(zone, getpagesize(), PROT_READ) != 0) {
perror("mprotect failed");
abort();
}
void* m = malloc(10000000000000000l);
if (m == NULL) {
perror("malloc failed, program still running");
}
return 0;
}
For completeness you will likely want to wrap calloc(), realloc(), and the other functions defined in malloc_zone_t in /usr/include/malloc/malloc.h as well.
Just wrap malloc() in some my_malloc() function that does this instead. In a lot of cases it's actually possible to handle not being able to allocate memory so this type of behaviour would be undesirable. It's easy to add functionality to malloc() but not to remove it, which is probably why it behaves this way.
Another thing to keep in mind is that this is a library you're calling into. Would you like to make a library call and have the library kill your application without you being able to have a say in it?
I guess I missed the part about asprintf but libc exports some hooks you can use (what valgrind does essentially) that let you override the malloc behavior. Here's a reference to the hooks themselves, if you know C well enough it's not hard to do.
http://www.gnu.org/savannah-checkouts/gnu/libc/manual/html_node/Hooks-for-Malloc.html
man malloc on my Mac gives the following information. It looks like you want MallocErrorAbort.
ENVIRONMENT
The following environment variables change the behavior of the allocation-related functions.
MallocLogFile <f>
Create/append messages to the given file path <f> instead of writing to the standard error.
MallocGuardEdges
If set, add a guard page before and after each large block.
MallocDoNotProtectPrelude
If set, do not add a guard page before large blocks, even if the MallocGuardEdges environment variable is set.
MallocDoNotProtectPostlude
If set, do not add a guard page after large blocks, even if the MallocGuardEdges environment variable is set.
MallocStackLogging
If set, record all stacks, so that tools like leaks can be used.
MallocStackLoggingNoCompact
If set, record all stacks in a manner that is compatible with the malloc_history program.
MallocStackLoggingDirectory
If set, records stack logs to the directory specified instead of saving them to the default location (/tmp).
MallocScribble
If set, fill memory that has been allocated with 0xaa bytes. This increases the likelihood that a program making assumptions about the contents of freshly allocated memory will fail. Also if set, fill memory that has been deallocated with 0x55 bytes. This increases the likelihood that a program will fail due to accessing memory that is no longer allocated.
MallocCheckHeapStart <s>
If set, specifies the number of allocations <s> to wait before beginning periodic heap checks every <n> as specified by MallocCheckHeapEach. If MallocCheckHeapStart is set but MallocCheckHeapEach is not specified, the default check repetition is 1000.
MallocCheckHeapEach <n>
If set, run a consistency check on the heap every <n> operations. MallocCheckHeapEach is only meaningful if MallocCheckHeapStart is also set.
MallocCheckHeapSleep <t>
Sets the number of seconds to sleep (waiting for a debugger to attach) when MallocCheckHeapStart is set and a heap corruption is detected. The default is 100 seconds. Setting this to zero means not to sleep at all. Setting this to a negative number means to sleep (for the positive number of seconds) only the very first time a heap corruption is detected.
MallocCheckHeapAbort <b>
When MallocCheckHeapStart is set and this is set to a non-zero value, causes abort(3) to be called if a heap corruption is detected, instead of any sleeping.
MallocErrorAbort
If set, causes abort(3) to be called if an error was encountered in malloc(3) or free(3), such as a calling free(3) on a pointer previously freed.
MallocCorruptionAbort
Similar to MallocErrorAbort but will not abort in out of memory conditions, making it more useful to catch only those errors which will cause memory corruption. MallocCorruptionAbort is always set on 64-bit processes.
MallocHelp
If set, print a list of environment variables that are paid heed to by the allocation-related functions, along with short descriptions. The list should correspond to this documentation.
Note the comments under MallocCorruptionAbort about the behaviour of MallocErrorAbort.
For most of my own code, I use a series of wrapper functions — emalloc(), erealloc(), ecalloc(), efree(), estrdup(), etc — that check for failed allocations (efree() is a straight pass-through function for consistency) and do not return when an allocation fails. They either exit or abort. This is basically what Jesus Ramos suggests in his answer; I agree with what he suggests.
However, not all programs can afford to have that happen. I'm just in the process of fixing up some code I wrote which does use these functions so that it can be reused in a context where it is not OK to fail to on allocation error. For its original purpose (security checks during the very early stages of process startup), it was fine to exit on error, but now it needs to be usable after the system is running, when a premature exit is not allowed. So, the code has to deal with those paths where the code used to be able to assume 'no return on allocation failure'. That's a tad painful. It can still take a conservative view; an allocation failure means the request is not safe and process it appropriately. But not all code can afford to fail with abort on memory allocation failure.