Check amount of heap space by running a program; not matching with actual boundaries - c

Question

I need to know the amount of heap space allocated for a process. In theory call malloc(1) (in C program) only once will get a start address of heap in my code. And call sbrk(0) a system call which return end of heap space allocated to that process. Below is my test code.
#include <stdlib.h>
#include <stdio.h>
#include <unistd.h>
#ifndef ALLOCATE_BYTES
#define ALLOCATE_BYTES 0x1000000
#endif /* #ifndef ALLOCATE_BYTES */
void subroutine (void);
int main()
{
void *init_ptr = malloc((size_t) 1);
void *end_ptr = sbrk((intptr_t) 0);
printf(" Total Heap space = %lld B = %lld kB \n",
(long long) (init_ptr - end_ptr),
(long long) (init_ptr - end_ptr)/1024);
subroutine();
end_ptr = sbrk((intptr_t) 0); /* New edit 1 */
printf(" Total Heap space = %lld B = %lld kB \n",
(long long) (init_ptr - end_ptr),
(long long) (init_ptr - end_ptr)/1024);
return 0;
}
void subroutine (void)
{
void *ptr;
long long count = 0;
size_t size = ALLOCATE_BYTES;
size_t size_p = ALLOCATE_BYTES + 1;
long long mem_alocated = 0;
printf(" start rate to %u \n", size);
while (1)
{
ptr = malloc((size_t)size);
if (ptr != NULL)
{
mem_alocated += size;
count++;
}
if ((ptr == NULL) && (size == size_p))
{
size/=2;
printf("overflow --> reduced Bytes SIZE %*u & "
"current count = %lld ---> total bytes %lld \n"
, 7, size, count, mem_alocated);
}
if ((ptr == NULL) && (size == 1))
{
printf("overflow....!! at %lld for %lld bytes\n",
count, count * ALLOCATE_BYTES);
break;
}
size_p = size;
}
}
The following is the results:
$ gcc -o exmpl_heap_consume exmpl_heap_consume.c -DALLOCATE_BYTES=0x10000
$ ./exmpl_heap_consume
Total Heap space = -135160 B = -131 kB
start rate to 65536
overflow --> reduced Bytes SIZE 32768 & current count = 48792 ---> total bytes 3197632512
overflow --> reduced Bytes SIZE 16384 & current count = 49084 ---> total bytes 3207200768
overflow --> reduced Bytes SIZE 8192 & current count = 49371 ---> total bytes 3211902976
overflow --> reduced Bytes SIZE 4096 & current count = 49658 ---> total bytes 3214254080
overflow --> reduced Bytes SIZE 2048 & current count = 49945 ---> total bytes 3215429632
overflow --> reduced Bytes SIZE 1024 & current count = 50233 ---> total bytes 3216019456
overflow --> reduced Bytes SIZE 512 & current count = 50521 ---> total bytes 3216314368
overflow --> reduced Bytes SIZE 256 & current count = 50809 ---> total bytes 3216461824
overflow --> reduced Bytes SIZE 128 & current count = 51098 ---> total bytes 3216535808
overflow --> reduced Bytes SIZE 64 & current count = 51100 ---> total bytes 3216536064
overflow --> reduced Bytes SIZE 32 & current count = 51100 ---> total bytes 3216536064
overflow --> reduced Bytes SIZE 16 & current count = 51387 ---> total bytes 3216545248
overflow --> reduced Bytes SIZE 8 & current count = 51389 ---> total bytes 3216545280
overflow --> reduced Bytes SIZE 4 & current count = 51676 ---> total bytes 3216547576
overflow --> reduced Bytes SIZE 2 & current count = 51676 ---> total bytes 3216547576
overflow --> reduced Bytes SIZE 1 & current count = 51676 ---> total bytes 3216547576
overflow....!! at 51676 for 3386638336 bytes
Total Heap space = -135160 B = -131 kB
This result says theoretical I have "135160 bytes" of memory locations for heap. But if I start consuming all of it till the repeated malloc() function call returns NULL. While doing so I keep track of how many bytes of memory my program has consumed now.
But here the question is my theoretical heap space (135160 bytes) is not matching with my practical counts (3386638336 bytes). Am I missing anything?
Edit 2:
I added some checks for the pointer return by the malloc() call and aggregated it to see the total. I observed the total bytes allocated is just less than the my theoretical heap space. This would suggest two things malloc() internally not calling sbrk() and secondly it is allocating memory elsewhere. Am I right till this point or anything missing here?

Answer 1

You've completely misunderstood brk/sbrk. sbrk(0) will tell the location of current program break, aka the end of data segment. When malloc runs out of space, it will call sbrk with a positive offset that will resize the data segment, thus moving the program break further. The Linux man page for sbrk clarifies this:
DESCRIPTION
brk() and sbrk() change the location of the program break, which
defines the end of the process's data segment (i.e., the program break
is the first location after the end of the uninitialized data seg‐
ment). Increasing the program break has the effect of allocating mem‐
ory to the process; decreasing the break deallocates memory.
brk() sets the end of the data segment to the value specified by addr,
when that value is reasonable, the system has enough memory, and the
process does not exceed its maximum data size (see setrlimit(2)).
sbrk() increments the program's data space by increment bytes. Call‐
ing sbrk() with an increment of 0 can be used to find the current
location of the program break.
(emphasis mine)
Furthermore usually on modern malloc implementations allocations of sufficient size (say >128kB) are allocated directly with mmap outside of the data segment. I'd test if the pointer returned from malloc(65536) even were between an address that is in the initialized data segment (do a global int foo = 42;, then &foo should be before the .bss/uninitialized data) and the current program break.

Answer 2

I think what you are not remembering is that your `practical count' is the available RAM on your system, which is used for multiple things. For example, I modified your program to:
#include <stdint.h>
#include <stdlib.h>
#include <stdio.h>
#include <unistd.h>
#ifndef ALLOCATE_BYTES
#define ALLOCATE_BYTES 0x1000000
#endif /* #ifndef ALLOCATE_BYTES */
int main()
{
void *init_ptr = malloc((size_t) 1);
void *end_ptr = sbrk((intptr_t) 0);
printf("\nmalloc returned: 0x%08x\n", (uint64_t)init_ptr);
printf("sbrk returned: 0x%08x\n", (uint64_t)end_ptr);
printf(" Total Heap space = %lld B = %lld kB \n",
(long long) (end_ptr - init_ptr),
(long long) (end_ptr - init_ptr)/1024);
sleep(300);
return 0;
}
I compiled the above using gcc (version 4.8.4) on Ubuntu 14.04 as: gcc -std=c11 -pedantic -Wall temp.c -o temp; then I ran the following commands at the command prompt:
******#crossbow:~/junk$ ./temp &
[2] 2069
******#crossbow:~/junk$
malloc returned: 0x01cac010
sbrk returned: 0x01ccd000
Total Heap space = 135152 B = 131 kB
******#crossbow:ps
PID TTY TIME CMD
1681 pts/0 00:00:00 bash
1775 pts/0 00:00:15 emacs
2069 pts/0 00:00:00 temp
2070 pts/0 00:00:00 ps
******#crossbow:~/junk$ cat /proc/2069/maps
00400000- 00401000 r-xp 00000000 fc:02 10616872 /home/******/junk/temp
00600000- 00601000 r--p 00000000 fc:02 10616872 /home/******/junk/temp
00601000- 00602000 rw-p 00001000 fc:02 10616872 /home/******/junk/temp
01cac000- 01ccd000 rw-p 00000000 00:00 0 [heap]
7f16a2e44000- 7f16a2fff000 r-xp 00000000 fc:02 70779581 /lib/x86_64-linux-gnu/libc-2.19.so
7f16a2fff000- 7f16a31fe000 ---p 001bb000 fc:02 70779581 /lib/x86_64-linux-gnu/libc-2.19.so
7f16a31fe000- 7f16a3202000 r--p 001ba000 fc:02 70779581 /lib/x86_64-linux-gnu/libc-2.19.so
7f16a3202000- 7f16a3204000 rw-p 001be000 fc:02 70779581 /lib/x86_64-linux-gnu/libc-2.19.so
7f16a3204000- 7f16a3209000 rw-p 00000000 00:00 0
7f16a3209000- 7f16a322c000 r-xp 00000000 fc:02 70779574 /lib/x86_64-linux-gnu/ld-2.19.so
7f16a3407000- 7f16a340a000 rw-p 00000000 00:00 0
7f16a3428000- 7f16a342b000 rw-p 00000000 00:00 0
7f16a342b000- 7f16a342c000 r--p 00022000 fc:02 70779574 /lib/x86_64-linux-gnu/ld-2.19.so
7f16a342c000- 7f16a342d000 rw-p 00023000 fc:02 70779574 /lib/x86_64-linux-gnu/ld-2.19.so
7f16a342d000- 7f16a342e000 rw-p 00000000 00:00 0
7ffcee9ab000- 7ffcee9cc000 rw-p 00000000 00:00 0 [stack]
7ffcee9ef000- 7ffcee9f1000 r-xp 00000000 00:00 0 [vdso]
7ffcee9f1000- 7ffcee9f3000 r--p 00000000 00:00 0 [vvar]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
Now, from this you can see that currently the heap starts at 0x01cac000 and runs to 0x01ccd000. Notice, that malloc does not allocate memory starting from the beginning of the heap.
Also, notice, that the heap can grow upwards till it bumps into the area that dynamic libraries are loaded (in this case, this is at 0x7f16a2e44000).
From the memory map listing above, you can also see where the stack is located, if you have any interest in that, as well as where various parts of your program are loaded. You can determine what those blocks are by using the readelf utility, and you can see:
Program Headers:
Type Offset VirtAddr PhysAddr FileSiz MemSiz Flg Align
PHDR 0x000040 0x0000000000400040 0x0000000000400040 0x0001f8 0x0001f8 R E 0x8
INTERP 0x000238 0x0000000000400238 0x0000000000400238 0x00001c 0x00001c R 0x1
[Requesting program interpreter: /lib64/ld-linux-x86-64.so.2]
LOAD 0x000000 0x0000000000400000 0x0000000000400000 0x0008d4 0x0008d4 R E 0x200000
LOAD 0x000e10 0x0000000000600e10 0x0000000000600e10 0x000248 0x000250 RW 0x200000
DYNAMIC 0x000e28 0x0000000000600e28 0x0000000000600e28 0x0001d0 0x0001d0 RW 0x8
NOTE 0x000254 0x0000000000400254 0x0000000000400254 0x000044 0x000044 R 0x4
GNU_EH_FRAME 0x0007a8 0x00000000004007a8 0x00000000004007a8 0x000034 0x000034 R 0x4
GNU_STACK 0x000000 0x0000000000000000 0x0000000000000000 0x000000 0x000000 RW 0x10
GNU_RELRO 0x000e10 0x0000000000600e10 0x0000000000600e10 0x0001f0 0x0001f0 R 0x1
From looking at this, we can see that the area in memory from 0x00400000 to 0x00401000 is occupied by the contents of the file described by the second program header. This segment contains the following program sections: .interp .note.ABI-tag .note.gnu.build-id .gnu.hash .dynsym .dynstr .gnu.version .gnu.version_r .rela.dyn .rela.plt .init .plt .text .fini .rodata .eh_frame_hdr .eh_frame. Probably the most important for you would be the .text section, which contains your code. In fact, the programs entry point is located at 0x00400520 (as determined by readelf -h).
Looking at the next area in memory from 0x00601000 to 0x00602000 and referring to the above, we can see that this corresponds to the section of the file described by the third program header. This segment contains the following program sections: .init_array .fini_array .jcr .dynamic .got .got.plt .data .bss.