I've written a program that takes 3 number in input:
The size of memory to allocate in the heap with malloc()
Two int value
If q is an unsigned char pointer it gives q[i]=b from q[min] to q[max].
I thought that the heap was divided in pages and that the first call to malloc() would have given a pointer to the first byte of the page of my process. So why if try to get q[-1] my process is not killed?
Then I've tried with another pointer p and I noticed that between the two pointers there is a distance of 32byte, why they are not adjacent?
The last thing I notice is that both in p[-8]=q[-40(-32-8)] and q[-8] there is the number 33 00100001 (all the other bytes are setted to 0), it means anything?
Thank you!
I thought that the heap was divided in pages and that the first call to malloc would have given a pointer to the first byte of the page of my process. So why if try to get q[-1] my process is not killed?
Most likely because your malloc implementation stores something there. Possibly the size of the block.
Then I've tried with another pointer p and I noticed that between the two pointers there is a distance of 32byte, why they are not adjacents?
Same reason. Your implementation probably stores the size of the block in the block just before the address it returns.
The last thing I notice is that both in p[-8]=q[-40(-32-8)] and q[-8] there is the number 33 (00100001), it means anything?
It probably means something to your malloc implementation. But you can't really tell what without looking at the implementation.
The standard library uses the heap before calling main so anything you do won't be on a clean heap.
The heap implementation usually uses about 2 pointer at the starting of an allocation, and the total size is usually aligned to 2 pointers.
The heap implementation usually uses a lot of bytes at the start of each system allocation, it can sometimes be close to page size.
The heap is allocated in chunks much bigger than a page, on Windows it is at least 16 pages.
The heap can be adjacent to other allocations, on Linux it appeares right after the main executable so underflowing it won't crash.
Related
I learned that whenever malloc is called, the actual memory given to the program by the OS is not exactly the requested size, but is rounded up to page size. From what I know, page size is 1024 or 4096.
Based on this logic (and correct me if it's wrong), writing beyond my allocated block won't always cause segmentation fault, as this fault is given by the kernel (which has given me a full page and doesn't care what I do so long as I stay inside of it).
The weird thing is that in the program bellow I requested 8 bytes from malloc, and then wrote 80000 (sizeof(size_t) * 10000) bytes further, without getting segmentation fault. I did get invalid read and write valgrind errors though.
Can someone shed light on the topic?
#include <stdio.h>
#include <stdlib.h>
int main()
{
size_t *ptr = (size_t *)malloc(sizeof(size_t));
size_t forward = 10000;
ptr += forward;
*ptr = 8;
printf("%lu\n", *ptr);
ptr -= forward;
free(ptr);
return (0);
}
You are confusing two very different things. If you ask malloc for 1 byte, it may allocate a 4,096 byte page from the operating system but it may only reserve 16 bytes of that block for this call to malloc. The next call to malloc may get another 16 bytes from that same 4,096 byte page.
You can't assume that just because you didn't get a segmentation fault that means you only accessed space that the implementation has reserved for your block. You can be writing on top of other objects being used for other purposes.
You're assuming that malloc requested just one page from the OS for your allocation, and that the subsequent pages remained unmapped. That's not necessarily true. malloc will typically request many pages at a time and then use them for later allocations if possible, thus making fewer system calls and reducing that overhead.
So it's likely that the several pages following your allocation are also mapped for your process, which is why you don't get a segfault. Instead, you overwrote memory that could be in use for something important.
Indeed, it's possible that malloc requested a larger chunk before main started, for internal C library structures, and that your allocation is just being placed within that chunk. In my test, using strace to see the brk() system call, this chunk was 33 pages in size (132 kB), and your block is in the first page of that chunk. So your 80 kB overrun is still within that mapped region.
So the answer to the title question is "it depends on where your block is located within its page, and which surrounding pages happen to be mapped". These in turn depend on the precise algorithm used by malloc and the pattern of other allocations and frees done by your code, or library code, up to that point, none of which you can really predict. It is possible in principle to find out which pages are mapped (e.g. on Linux by parsing /proc/self/maps) but this can change unpredictably as memory is allocated and freed by your code or library functions. So the practical answer is "you don't know".
Basically, you shouldn't make any assumptions about what will or won't happen when you write outside the boundaries of malloced memory. The only time when you can really be assured of getting a segfault is when you mmaped and mprotected the memory yourself.
I'm trying to figure out how many bytes in a block are taken up by the boundary tags. I have been told that when trying to malloc an adjacent block of memory, a "jump" will appear in assembly code, and I can use that to determine the size of the boundary tag. I've tried this:
int* arr = malloc(8);
arr++;
arr = malloc(8);
But there isn't any jump in assembly code. Am I "trying to malloc an adjacent block of memory"?
EDIT: I think he means a jump will appear between address value. I use the beginning of the second block of memory subtract the payload size of the first block. But I'm still confused, how could I malloc an adjacent block of memory?
Unless you're writing an actual memory allocator, you can't actually allocate two consecutive chunks of memory. If you want to see some pretty gnarly code which does this, have a look at the Illumos malloc https://github.com/illumos/illumos-gate/blob/master/usr/src/lib/libc/port/gen/malloc.c.
If you want to see how Illumos (and Solaris) handle the redzone between allocated blocks, you should trawl through https://github.com/illumos/illumos-gate/tree/master/usr/src/lib/libumem/common.
The memory consumed by malloc(3) requires, for proper management of the actually used memory, of some structures that must be dynamically allocated also. For this reason, many allocators just do allocate the space required for the management data adjacent to the block space dedicated to the user. This makes that normally two consecutive junks of memory allocated by malloc(2) show some gap in their addresses.
There are other reasons to see gaps, one fundamental is that malloc normally gives you aligned memory addresses, so it is warranted that the data you store on that memory will be properly aligned.
And of course, there can be implementations (normally when heap allocation should be more robust in respect to buffer overruns) that the memory dedicated to storage of management data is completely unrelated and apart off the final given memory. In this case you could observe no gaps between memory allocations on some cases.
Anyway, your code has serious bugs, let's see:
int* arr = malloc(8);
You had better here to acquire just the memory you need, using the sizeof operator, as in int *arr = malloc(sizeof *arr); instead.
arr++;
this statement is useless, as you are going to overwrite the value of arr (the pointer) with new assignment statement after it from malloc(), so it is of no use to increment the pointer value. You also are somewhat losing the returned value of the previous malloc() (which is essential in order to return the allocated memory, but read below).
arr = malloc(8);
Until here, you had the chance to --arr decrementing the value of arr in order to be capable of free(3) that block. But this statement overwrites the value stored in arr so the previous pointer value is overwritten by the new pointer. Memory you acquired on the first malloc has no way to be accessed again. This is what is commonly known as a memory leak, and is normally a serious error (very difficult to catch) on long run programs (like servers or system daemons). The program allocates a bunch of memory in the inner part of a loop, that is not returned back with a call to free(3), so the program begins growing and growing until it exhausts all the available memory.
A final note, I don't understand what did you mean with malloc adjacent block of memory. Did you believe that incrementing the pointer would make malloc() to give you a special block of memory?
First, malloc has no idea of what are you going to do with the pointer it gives to you.
But also, it doesn't know anything about the variable contents of the pointer you are assigning to (you can even not store it in a variable, and pass it as another parameter to another functions) So there's no possibility for malloc to know that you have incremented the pointer value, or even the pointer location, from its body.
I cannot guess your interpretation. It would be nice to know what has made you to think that you can control how malloc(3) selects the block of memory to give to you. You have no control on the internals of malloc() You just specify the amount of continous memory you want, and mallocs provides it, giving you a pointer pointing to the start of that block. You cannot assume that the next time you call malloc (with the same or different amount of memory) it will give you an adjacent block. It just can be completely unrelated (above or below in memory) to the previous given block. And you cannot modify that pointer, because you need it to call free(3) once you don't need the block anymore, with exactly the same pointer value that malloc(3) gave to you. If, for some reason you modify the pointer, you need to restore it to the original value to be capable of calling free(3). Lack to do so, you'll probably crash your program at the next call to free(3).
I just see a memory leak. Malloc 2 times into different vars 8 bytes of space and see if the difference is more than 8 bytes or 2 int.
From what I understand, malloc(x) returns a block of memory x bytes long.
So to store a 4 byte integer, I would do:
int *p = (int *)malloc(4);
*p = 100;
Because sizeof(int) returns 4 for me.
However, if I do:
int *p = (int *)malloc(1);
*p = 100;
It seems to work exactly the same, with no issues storing the value.
Why does the amount of memory requested with malloc() not seem to matter? Shouldn't a 4 byte integer require malloc(4)?
If this works in your case it just works by chance and is not guaranteed to work. It is undefined behavior (compare this SO question) and everything can happen.
What did you expect to happen? Your program crash?
That might still happen if you call malloc and free a bit more often. malloc often takes some bytes more than requested and uses the extra space for Managing (linked list of all memory blocks, sizes of memory blocks). If you write some bytes before or after your allocated block then chances are high that you mess with the internal management structures and that subsequent malloc of free will crash.
If malloc internally always allocates a minimum of n bytes then your program might only crash if you access byte n+1. Also the operating system normally only be protects memory based on pages. If a page has a size of 512 bytes and your malloc-ed byte is in the middle of a page then your process might be able to read-write the rest of the page and will only crash when accessing the next memory page. But remember: even if this works it is undefined behavior.
malloc as all memory block allocation functions from C runtime or OS Kernel are optimized for memory access and object alignment.
Moreover, malloc specifically, allocate an hidden control block in front of the allocated space to keep track of the allocation (space required, space allocated, etc).
malloc must also to guarantee that the allocated memory address is suitably aligned for any storage object, this means that the block will start on an 8, 16, 32 or even 64 or 128 bytes boundary depending on the processor and generically from hardware (i.e. some special MMU). The boundary is also dependent on access speed, some processor have different behavior with different memory accesses (1, 2, 4, 8, ... bytes) and address boundaries. This constraints drive malloc code spec and allocator logical memory blocks partitions.
On practical side lets consider an allocator for X86 processor, it generally give back a block aligned on an 8 bytes boundary (32 bits code), that is useful for int, float and even doubles. To do this malloc divides the available memory arena in 'blocks' that are the minimal allocation space. When you allocate even 1 byte the function allocates at least one block. Eventually this block can host an integer, or even a double, but it is implementation dependent, and you can't consider it deterministic, because in future versions of the same function the behavior can change.
Now that, I hope, is clear because your code seems to work, keep well in mind that this is Undefined-Behavior and you must keep it for that. IT can work now, not with the next revision, it can crash on some hardware and not on another processor or machine.
For this we should know how malloc function works internally. To allocate memory dynamically, each operating system make use of system calls. We can dynamically allocate memory using these system calls. These system calls are different from one OS to the other.
So the system calls of one OS might not work for the other OS. And moreover if we are using system calls to allocate memory dynamically then our program will be platform dependent. So to avoid this dependency we use malloc function. Now it is the responsibility of malloc function to make the appropriate system calls based on the OS to allocate memory dynamically.
So malloc itself invokes system calls and it will be a very slow process because each time we are asking for dynamic memory it has to make use of system calls. To avoid this whenever we request dynamic memory it usually allocate extra memory so that next time the system call can be avoided and the remaining chunk of previously allocated memory can be used. And that's why your program is working as malloc is allocating extra memory.
The C programming language gives you the ability to shoot yourself in the foot.
It intentionally burdens the programmer with the charge that they ought to know what they are doing. Broadly speaking the reasons are to achieve performance, readability, and portability.
The behaviour of your code is undefined. If you ask for 1 byte then expect to get only one usable byte back. The fact that the operating system and C runtime library seems to be giving you back a little more than that is no more than a curious peculiarity.
On other occasions the compiler might simply eat your cat.
Finally, use sizeof in the call to malloc rather than hardcoding the size of an int type: on many systems sizeof(int) is 2, 4 is commonplace, and all values greater than 1 are allowed by the standard. In your case, using either sizeof(int) or sizeof(*p) is possible. Some folk prefer the latter since then you're not hardcoding the type of the variable in the sizeof call, so guarding you against possible variable type changes. (Note that sizeof(*p) is compile-time evaluable and uses static type information; hence it can be used before p itself "exists", if you get my meaning.)
It seems to work exactly the same, with no issues storing the value.
You invoke undefined behavior with your code, so you cannot tell that it works. In order to allocate memory for an integer, you should do:
int *p;
p = malloc(sizeof (*p) ); //you can use sizeof(*p) as p is already declared and here you use the size of its content, which is actually the size of an int
if (p != NULL)
*p = 100;
Simply, when you are allocating 1 byte for the int, there are 3 bytes following it that are not actually allocated for it, but you can still use them. You are getting lucky that these aren't being changed by something else during your tests, and that it isn't overwriting anything important (or maybe it is). So essentially this will cause an error once those 3 bytes are needed by something else -- always malloc the right size.
Usually malloc is implemented such a way that it allocates memory with the size not less than the size of the paragraph that is equal to 16 bytes.
So when you require for example 4 bytes of memory malloc actually allocates 16 bytes. However this behavior does not described in the C Standard and you may not rely on it. As result it means that the program you showed has undefined behavior.
I think this is because of padding, and even though you are calling malloc(1) padding bytes are coming with the memory.
please check this link http://www.delorie.com/gnu/docs/glibc/libc_31.html
So I have allocated 256 blocks in heap:
char* ptr1 = malloc(128);
char* ptr2 = malloc(128);
Now after I free ptr2 which I assume currently lies on top of the heap, the program break(the current location of the heap) does not decrease. However if I do another malloc the address returned by malloc is the same as the one that is freed.
So I have the following questions:
When I free a block why does not the program break decrease?
When I call free what exactly happens?How does it keep track of the freed memory so that next time I declare malloc the address is the same?
It's unspecified behavior. You can not rely on any single answer, unless you only care about one particular platform/os/compiler/libc combination. You did not specify an OS, and the C standard does not describe, or require any particular implementation. From C99 (I don't have the final published version of C11 yet):
7.20.3
The order and contiguity of storage allocated by successive calls to
the calloc, malloc, and realloc functions is unspecified. The pointer
returned if the allocation succeeds is suitably aligned so that it may
be assigned to a pointer to any type of object and then used to access
such an object or an array of such objects in the space allocated
(until the space is explicitly deallocated). The lifetime of an
allocated object extends from the allocation until the deallocation.
Each such allocation shall yield a pointer to an object disjoint from
any other object. The pointer returned points to the start (lowest
byte address) of the allocated space. If the space cannot be
allocated, a null pointer is returned. If the size of the space
requested is zero, the behavior is implementation- defined: either a
null pointer is returned, or the behavior is as if the size were some
nonzero value, except that the returned pointer shall not be used to
access an object.
This manual of GNU libc , might be of help.
Here's the gist
Occasionally, free can actually return memory to the operating system
and make the process smaller. Usually, all it can do is allow a later
call to malloc to reuse the space. In the meantime, the space remains
in your program as part of a free-list used internally by malloc.
When I free a block why does not the program break decrease?
I believe it doesn't decrease because that memory has already been given to the program.
When I call free() what exactly happens?
That section of memory is marked as allocatable, and its previous contents can be overwritten.
Consider this example...
[allocatedStatus][sideOfAllocation][allocatedMemory]
^-- Returned pointer
Considering this, the free() can then mark the [allocatedStatus] to false, so future allocations on the heap can use that memory.
How does it keep track of the free()d memory so that next time I declare
malloc() the address is the same?
I don't think it does. It just scanned for some free memory and found that previous block that had been marked as free.
Here is a rough idea how memory allocators work:
You have an allocator that has a bunch of "bins" ("free lists") which are just linked lists of free memory blocks. Each bin has a different block size associated with it (I.e.: you can have a list for 8 byte blocks, 16 byte blocks, 32 byte blocks, etc... Even arbitrary sizes like 7 or 10 byte blocks). When your program requests memory (usually through malloc()) the allocator goes to the smallest bin that would fit your data and checks to see if there are any free memory blocks in it. If not then it will request some memory from the OS (usually called a page) and cuts the block it gets back into a bunch of smaller blocks to fill the bin with. Then it returns one of these free blocks to your program.
When you call free, the allocator takes that memory address and puts it back into the bin (aka free list) it came from and everybody is happy. :)
The memory is still there to use so you don't have to keep paging memory, but with respect to your program it is free.
I believe it's entirely up to the operating system once you call free(), it may choose to immediately reclaim that memory or not care and just mark that memory segment as a possible acquisition for a later time (likely the same thing). To my knowledge that memory (if significant) shows up as available in the task manager right after free() on windows.
Keep in mind that the memory we are talking about here is virtual. So that means the operating system can tell you anything it wants and is likely not an accurate representation of the physical state of the machine.
Think about how you would manage memory allocation if you were writing an OS, you likely wouldn't want to do anything hasty that may waste resources. We are talking about 128 bytes here, would you want to waste valuable processing time handling it alone? It may be the reason for that behavior or not, at least plausible.
Do it in a loop and then free() in another loop or just allocate big chunks of memory, see what happens, experiment.
Trying to understand answers to my question
what happens when tried to free memory allocated by heap manager, which allocates more than asked for?
I wrote this function and puzzled by its output
int main(int argc,char **argv){
char *p,*q;
p=malloc(1);
strcpy(p,"01234556789abcdefghijklmnopqrstuvwxyz"); //since malloc allocates atleast 1 byte
q=malloc(2);
// free(q);
printf("q=%s\n",q);
printf("p=%s\n",p);
return 0;
}
Output
q=vwxyz
p=01234556789abcdefghijklm!
Can any one explain this behavior? or is this implementation specific?
also if free(q) is uncommented, I am getting SIGABRT.
You are copying more bytes to *p than you have allocated, overwriting whatever might have been at the memory locations after the allocated space.
When you then call malloc again, it takes a part of memory it knows to be unused at the moment (which happens to be a few bytes after *p this time), writes some bookkeeping information there and returns a new pointer to that location.
The bookkeeping information malloc writes happens to start with a '!' in this run, followed by a zero byte, so your first string is truncated. The new pointer happens point to the end of the memory you overwrote before.
All this is implementation specific and might lead to different results each run or depending on the phase of the moon. The second call to malloc() would also absolutely be in its right to just crash the program in horrible ways (especially since you might be overwriting memory that malloc uses internally).
You are just being lucky this time: this is an undefined behavior and don't count on it.
Ususally, but depending on the OS, memory is allocated in "pages" (i.e. multiple bytes). Malloc() on the other hand allocates memory from those "pages" in a more "granular" way: there is "overhead" associated with each allocation being managed through malloc.
The signal you are getting from free is most probably related to the fact that you mess up the memory management by writing past what you were allocated with p i.e. writing on the overhead information used by the memory manager to keep track of memory blocks etc.
This is a classical heap overflow. p has only 1 byte, but the heap manager pads the allocation (32 bytes in your case). q is allocated right after p, so it naturally gets the next available spot. For example if the address of p is 0x1000, the adress that gets assigned to q is 0x1020. This explains why q points to part of the string.
The more interesting question is why p is only "01234556789abcdefghijklm" and not "01234556789abcdefghijklmnopqrstuvwxyz". The reason is that memory manager uses the gaps between allocation for its internal bookkeeping. From a memory manager perspective the memory layout is as following:
p D q
where D is internal data structure of memory manager (0x1010 to 0x1020 in our example). While allocating memory for q, the heap manager writes its stuff to the bookkeeping area (0x1010 to 0x1020). A byte is changed to 0 truncates the string since it is treated as NULL terminator.
THE VALUE OF "p":
you allocated enough space to fit this: ""
[[ strings are null terminated, remember? you don't see it, but it's there -- so that's one byte used up. ]]
but you are trying to store this: "01234556789abcdefghijklmnopqrstuvwxyz"
the result, therefore, is that the "stuff" starting with "123.." is being stored beyond the memory you allocated -- possibly writing over other "stuff" elsewhere. as such your results will be messy, and as "jidupont" said you're lucky that it doesn't just crash.
OUTPUT OF PRINTING [BROKEN] "p"
as said, you've written way past the end of "p"; but malloc doesn't know this. so when you asked for another block of memory for "q", maybe it gave you the memory following what it gave you for "p"; and maybe it aligned the memory (typical) so it's pointer is rounded up to some nice number; and then maybe it uses some of this memory to store bookkeeping information you're not supposed to be concerned with. but you don't know, do you? you're not supposed to know either -- you're just not supposed to write to memory that you haven't allocated yourself!
and the result? you see some of what you expected -- but it's truncated! because ... another block was perhaps allocated IN the memory you used (and used without permission, i might add), or something else owned that block and changed it, and in any case some values were changed -- resulting in: "01234556789abcdefghijklm!". again, lucky that things didn't just explode.
FREEING "q"
if you free "q", then try to access it -- as you are doing by trying to print it -- you will (usually) get a nasty error. this is well deserved. you shouldn't uncomment that "free(q)". but you also shouldn't try to print "q", because you haven't put anything there yet! for all you know, it might contain gibberish, and so print will continue until it encounters a NULL -- which may not happen until the end of the world -- or, more likely, until your program accesses yet more memory that it shouldn't, and crashes because the OS is not happy with you. :)
It shouldn't be that puzzling that intentionally misusing these functions will give nonsensical results.
Two consecutive mallocs are not guaranteed to give you two consecutive areas of memory. malloc may choose to allocate more than the amount of memory you requested, but not less if the allocation succeeds. The behavior of your program when you choose to overwrite unallocated memory is not guaranteed to be predictable.
This is just the way C is. You can easily misuse the returned memory areas from malloc and the language doesn't care. It just assumes that in a correct program you will never do so, and everything else is up for grabs.
Malloc is a function just like yours :)
There is a lot of malloc implementations so i won't go into useless details.
At the first call malloc it asks memory to the system. For the example let's say 4096 which is the standard memory page size which is good. So you call malloc asking for 1 byte. The function malloc will asks 4096 bytes to the system. Next, it will use a small part of this memory to store internal data such the positions of the available blocks. Then it will cut one part of this block and send it back to you.
An internal algorithm will trys to reuse the blocks after a call to free to avoid re-asking memory to the system.
So with this little explanation you can now understand why you code is working.
You are writing in the memory asked my malloc to the system. This comportment doesn't bother the system because you stay in the memory allocated for your processes. The problem is you can't know for sure that you are not writing on critical parts of your software memory. This kind off error are called buffer overflow and are causing most of the "mystical bugs".
The best way to avoid them is to use valgrind on linux. This soft will tell you if you are writing or reading where you are not supposed to.
It that clear enough ?
I suggest reading this introduction.
Pointers And Memory
It helped me understand the difference between stack and heap allocation, very good introduction.