How do I make dynamic allocations for a program run fail - c

Question

I have a C program which uses malloc (it could also have been C++ with new). I would like to test my program and simulate an "out of memory" scenario.
I would strongly prefer running my program from within a bash or sh shell environment without modifying the core code.
How do I make dynamic memory allocations fail for a program run?
Seems like it could be possible using ulimit but I can't seem to find the right parameters:
$ ulimit -d 50
$ ./program_which_heap_allocates
./program_which_heap_allocates: error while loading shared libraries: libc.so.6: cannot map zero-fill pages
$ ulimit -d 51
bash: ulimit: data seg size: cannot modify limit: Operation not permitted
I'm having trouble running the program in such a way that dynamic linking can occur (such as stdlib) but not the allocations from my program.

Answer 1

If you are under Linux and using glibc then there are Hooks for Malloc. The hooks allow you to catch calls to malloc and make them randomly fail.
Your test suite could use an environment variable to tell the code to insert the malloc hook and which call of malloc to fail. E.g. if you set FOOBAR_FAIL_MALLOC=10 then your malloc hook would count down and let the 10th use of malloc return 0.
FOOBAR_FAIL_MALLOC=0 could simply report the numbers of mallocs in a testcase. You would then run the test once with FOOBAR_FAIL_MALLOC=0 and capture the number of mallocs involved. Then repeat for FOOBAR_FAIL_MALLOC=1 to N to test every single malloc.
Unless after a failure of malloc you have more mallocs. Then you have to think of something more complex to specify which mallocs should fail.
You could also just make the hook fail randomly. Given enough runs every malloc call would fail at some point.
Note: a C++ new should also hot the malloc hook

Answer 2

You can have your test program include the .c under test and use a #define to override calls to malloc.
For example:
prog.c:
#include <stdio.h>
#include <stdlib.h>
void *foo(int x)
{
return malloc(x);
}
test.c:
#include <stdio.h>
#include <stdlib.h>
static char buf[100];
static int malloc_fail;
void *test_malloc(size_t n)
{
if (malloc_fail) {
return NULL;
} else {
return buf;
}
}
#define malloc(x) test_malloc(x)
#include "prog.c"
#undef malloc
int main()
{
void *p;
malloc_fail=0;
p = foo(5);
printf("buf=%p, p=%p\n", (void *)buf, p); // prints same value both times
malloc_fail=1;
p = foo(4);
if (p) {
printf("buf=%p, p=%p\n", (void *)buf, p);
} else {
printf("p is NULL\n"); // this prints
}
return 0;
}