Related
Background:
I'm working through a project whose goal is to build a memory management system that replicates malloc() and free(). The program begins by requesting a specified amount of memory from the operating system with mmap and then uses a linked list to keep track of which sections of memory are in use or free. See below for a general idea of what I'm doing.
Output after allocating one chunk of memory
Question
Why do I hit a segmentation fault when trying to create a new list node to represent free memory? This segmentation fault occurs only when trying to request more than two chunks of memory - the first works as expected.
Details
The segmentation fault occurs when trying to assign values to the attributes of the node just created (new in the code below) to represent the remaining free memory space. This assignment works the first time I split memory into used v. free segments, but the second time I try to split the free memory into used v. free segments, my program encounters a segmentation fault - I am assuming this occurs because the attributes of the new node_t struct are undefined (compiler doesn't know where they live in memory). What I don't know is why this is occurring, and why it doesn't occur the first time I use this code.
NOTE: The answer shouldn't be to use malloc when initializing the *new node because the point of the project is to do this without using malloc. Malloc wouldn't allocate from the memory I'm managing - instead, it'd allocate from the heap my program is actually maintaining (but not using) on the side.
Struct definitions:
typedef enum {false, true} bool;
typedef struct _node_t {
int size;
struct _node_t *next;
struct _node_t *prev;
bool used;
} node_t;
Problem code. ret is the memory allocation I'm going to return to the caller and new is the remainder of the free memory block I just allocated part of.
// Code above this steps through the list elements to find the first free
// element that is large enough. I've debugged enough to know that's not the
// source of the problem.
node_t *new = ret + sizeof(node_t) + size; // Pointer to new free slot
assert(new != NULL);
new->prev = ret;
new->next = ret->next;
new->size = ret->size - sizeof(node_t) - size;
new->used = false;
ret->used = true;
ret->next = new;
ret->size = size;
return (ret + sizeof(node_t));
Calling code:
int main(int argc, char* argv[]) {
int success = Mem_Init(5000, P_FIRSTFIT);
if (success == -1)
return -1;
int *slot1 = Mem_Alloc(sizeof(int));
// This is where segmentation fault occurs
int *slot2 = Mem_Alloc(sizeof(int));
Link to code you can compile and run.
Although you don't include a definition of ret and Dropbox is totally useless for viewing code on a smartphone, I think we can deduce the declaration
node_t* ret;
Otherwise ret->next won't compile.
In that case, ret + sizeof(node_t) does not make sense because of the nature of pointer arithmetic in C. Remember that p + k is exactly the same as &p[k]. In other words, p + 1 is the address of the next element of size sizeof*p. So ret + sizeof(node_t) is the address of the 16th or 32nd element after ret, depending on the sizeof (node_t).
What you probably wanted was ret + 1, whose simplicity illustrates the rationale behind C pointer arithmetic.
The answer: It's all in pointer math. The issue was this statement:
node_t *new = ret + sizeof(node_t) + size; // Pointer to new free slot
Any integer in the right hand side of this equation doesn't increment the memory address by one byte... it increments it to the next spot where a node_t struct would begin. So in the code above, the address of new is set (32 + size) * 32 bytes further along in memory (assuming sizeof(node_t) is 32, which is true on a 64 bit system). Do that enough and you run well outside of the 500 bytes of memory you actually requested from the OS.
The code that works is:
node_t *new = ret + 1 + (size / sizeof(node_t)) + 1;
This code ends up over-allocating memory a bit if size isn't exactly equal to sizeof(node_t), but that's a problem with a more complicated solution. For illustration, consider this example:
ret = address 1000
sizeof(node_t) = 32 (in a 64 bit system)
size = 50
The end of the header for ret is address 1032 in this example. The program then allocates 2x32 bytes to handle the 50 byte request (math is (50 / 32) + 1 = 2, or two node_t structs). So the memory address of new is actually 1096.
To make this work overall, I'll also have to update the size calculation to reflect the actual allocation instead of the requested allocation. This will be necessary to make a safeguard against running outside of the bounds of memory requested from the OS work properly.
As far of my concern if realloc fails we loose the information and realloc set the Buffer(pointer) to NULL
Consider de following program:
#include<stdio.h>
#include<stdlib.h>
#include<string.h>
int main(void){
char *ptr = malloc(256);
if (!ptr){
printf("Error, malloc\n");
exit(1);
}
strcpy(ptr, "Michi");
ptr = realloc (ptr, 1024 * 102400000uL); /* I ask for a big chunk here to make realloc to fail */
if (!ptr){
printf("Houston we have a Problem\n");
}
printf("PTR = %s\n", ptr);
if (ptr){
free(ptr);
ptr = NULL;
}
}
And the output of course is:
Houston we have a Problem
PTR = (null)
I just lost the information inside ptr.
Now to fix this we should use a temporary buffer(pointer) before to see if we get that chunk of memory and if we get it we can use it, if not we still have the main buffer(pointer) safe.
Now please consider the following program, where instead of calling realloc I call malloc on a temporary buffer(pointer):
#include<stdio.h>
#include<stdlib.h>
#include<string.h>
int main(void){
char *ptr = malloc(256);
char *tmpPTR = NULL;
if (!ptr){
printf("Error, malloc\n");
exit(1);
}
strcpy(ptr, "Michi");
tmpPTR = malloc (1024 * 102400000uL);
if (tmpPTR){
strcpy(tmpPTR, ptr);
strcat(tmpPTR, " - Aloha");
if (ptr){
free(ptr);
ptr = NULL;
}
}else{
printf("Malloc failed on tmpPTR\n\n");
}
if (ptr){
printf("PTR = %s\n", ptr);
free(ptr);
ptr = NULL;
}else if (tmpPTR){
printf("tmpPTR = %s\n", tmpPTR);
free(tmpPTR);
ptr = NULL;
}
}
And the output is:
Malloc failed on tmpPTR
PTR = Michi
Now why should I ever use realloc?
Is there any benefit of using realloc instead of malloc based on this context?
Your problem is with how you use realloc. You don't have to assign the result of realloc to the same pointer that you re-allocate. And as you point out it even poses a problem if the realloc fails. If you immediately assign the result to ptr then indeed you lose the previous buffer when something goes wrong. However, if you assign the result of realloc to tmpPTR, then ptr remains fine, even if the realloc fails. Use realloc as follows:
char * ptr = malloc(256);
if(!ptr){
return 1;
}
char * tmpPTR = realloc(ptr, 512);
if(!tmpPTR){
printf("Houston, we have a problem");
// ptr is fine
}else{
ptr = tmpPTR;
}
// ptr is realloc()ed
In the above code, tmpPTR is not a new (temporary) buffer, but just a (temporary) pointer. If the realloc is succesful it points to the same buffer (but possibly in a different location), and if it fails it is NULL. realloc doesn't always need to allocate a new buffer, but may be able to change the existing one to fit the new size. But if it fails, the original buffer will not be changed.
If you use malloc with a temporary buffer, then (for this example) you need at least 256 + 512 = 768 bytes and you always need to copy the old data. realloc may be able to re-use the old buffer so copying is not necessary and you don't use more memory than requested.
You can use your malloc approach, but realloc is almost always more efficient.
The realloc scheme is simple. You do not need a separate call to malloc. For example if you initially have 256 bytes allocated for ptr, simply use a counter (or index, i below) to keep track of how much of the memory within the block allocated to ptr has been used, and when the counter reaches the limit (1 less than the max for 0-based indexes, or 2 less than the max if you are using ptr as a string), realloc.
Below shows a scheme where you are simply adding 256 additional bytes to ptr each time the allocation limit is reached:
int i = 0, max = 256;
char *ptr = malloc(max);
/* do whatever until i reaches 255 */
if (i + 1 >= max) {
void *tmp = realloc (ptr, max + 256);
if (!tmp) {
fprintf (stderr, "error: realloc - memory exhausted.\n")
/* handle error */
}
ptr = tmp;
max += 256;
}
note: your handle error can exit whatever loop you are in to preserve the existing data in ptr. You do not need to exit at that point.
The advantage of realloc over malloc is that it may be able to extend the original dynamic memory area so without the need to copy all the previous elements; you can't do that with malloc1. And whether this optimization is available costs no work to you.
Let's assume you have a previously allocated pointer:
char *some_string = malloc(size); // assume non-NULL
Then
if (realloc_needed) {
char *tmp = realloc(some_string, new_size);
if ( tmp == NULL )
// handle error
else
some_string = tmp; // (1)
At (1), you update the old pointer with the new one. Two things can happen: the address has effectively changed (and the elements been automatically copied) or it hasn't - you don't really care. Either way, your data is now at some_string.
Only the actual implementation (OS / libc) knows whether it's possible to enlarge the block: you don't get to see it, it's an implementation detail. You can however check your implementation's code and see how it's implemented.
Now to fix this we should use a temporary buffer(pointer) before to see if we get that chunk of memory and if we get it we can use it, if not we still have the main buffer(pointer) safe.
That not only doesn't help, it makes things worse because now you no longer have the pointer to the block you tried to reallocate. So how can you free it?
So it:
Wastes memory.
Require an extra allocate, copy, and free.
Makes the realloc more likely to fail because of 1.
Leaks memory since the pointer to the block you tried to reallocate is lost.
So no, that's not a good way to handle realloc returning NULL. Save the original pointer when you call realloc so you can handle failure sanely. The point of realloc to save you from having to manage two copies of the data and to avoid even making them when that's possible. So let realloc do this work for you whenever you can.
It is technically malloc(size) that is unneeded, because realloc(NULL, size) performs the exact same task.
I often read inputs of indeterminate length. As in the following function example, I rarely use malloc(), and instead use realloc() extensively:
#include <stdlib.h>
#include <errno.h>
struct record {
/* fields in each record */
};
struct table {
size_t size; /* Number of records allocated */
size_t used; /* Number of records in table */
struct record item[]; /* C99 flexible array member */
};
#define MAX_ITEMS_PER_READ 1
struct table *read_table(FILE *source)
{
struct table *result = NULL, *temp;
size_t size = 0;
size_t used = 0, n;
int err = 0;
/* Read loop */
while (1) {
if (used + MAX_ITEMS_PER_READ > size) {
/* Array size growth policy.
* Some suggest doubling the size,
* or using a constant factor.
* Here, the minimum is
* size = used + MAX_ITEMS_PER_READ;
*/
const size_t newsize = 2*MAX_ITEMS_PER_READ + used + used / 2;
temp = realloc(result, sizeof (struct table) +
newsize * sizeof (result->item[0]));
if (!temp) {
err = ENOMEM;
break;
}
result = temp;
size = newsize;
}
/* Read a record to result->item[used],
* or up to (size-used) records starting at result->item + used.
* If there are no more records, break.
* If an error occurs, set err = errno, and break.
*
* Increment used by the number of records read: */
used++;
}
if (err) {
free(result); /* NOTE: free(NULL) is safe. */
errno = err;
return NULL;
}
if (!used) {
free(result);
errno = ENODATA; /* POSIX.1 error code, not C89/C99/C11 */
return NULL;
}
/* Optional: optimize table size. */
if (used < size) {
/* We don't mind even if realloc were to fail here. */
temp = realloc(result, sizeof (struct table) +
used * sizeof table->item[0]);
if (temp) {
result = temp;
size = used;
}
}
result->size = size;
result->used = used;
errno = 0; /* Not normally zeroed; just my style. */
return result;
}
My own practical reallocation policies tend to be very conservative, limiting the size increase to a megabyte or so. There is a very practical reason for this.
On most 32-bit systems, userspace applications are limited to 2 to 4 gigabyte virtual address space. I wrote and ran simulation systems on a lot of different x86 systems (32-bit), all with 2 to 4 GB of memory. Usually, most of that memory is needed for a single dataset, which is read from disk, and manipulated in place. When the data is not in final form, it cannot be directly memory-mapped from disk, as a translation -- usually from text to binary -- is needed.
When you use realloc() to grow the dynamically allocated array to store such huge (on 32-bit) datasets, you are only limited by the available virtual address space (assuming there is enough memory available). (This especially applies to 32-bit applications on 64-bit systems.)
If, instead, you use malloc() -- i.e., when you notice your dynamically allocated array is not large enough, you malloc() a new one, copy the data over, and discard the old one --, your final data set size is limited to a lesser size, the difference depending on your exact array size growth policy. If you use the typical double when resizing policy, your final dataset is limited to about half (the available virtual address space, or available memory, whichever is smaller).
On 64-bit systems with lots and lots of memory, realloc() still matters, but is much more of a performance issue, rather than on 32-bit, where malloc() is a limiting factor. You see, when you use malloc() to allocate a completely new array, and copy the old data to the new array, the resident set size -- the actual amount of physical RAM needed by your application -- is larger; you use 50% more physical RAM to read the data than you would when using realloc(). You also do a lot of large memory-to-memory copies (when reading a huge dataset), which are limited to physical RAM bandwidth, and indeed slow down your application (although, if you are reading from a spinning disk, that is the actual bottleneck anyway, so it won't matter much).
The nastiest effect, and the most difficult to benchmark, are the indirect effects. Most operating systems use "free" RAM to cache recently accessed files not modified yet, and this really does decrease the wall clock time used by most workloads. (In particular, caching typical libraries and executables may shave off seconds from the startup time of large application suites, if the storage media is slow (ie. a spinning disk, and not a SSD).) Your memory-wasting malloc()-only approach gobbles up much more actual physical RAM than needed, which evicts cached, often useful, files from memory!
You might benchmark your program, and note that there is no real difference in run times between using your malloc()-only approach and realloc() approach I've shown above. But, if it works with large datasets, the users will notice that using the malloc()-only program slows down other programs much more than the realloc()-using program, with the same data!
So, although on 64-bit systems with lots of RAM using malloc() only is basically an inefficient way to approach things, on 32-bit systems it limits the size of dynamically allocated arrays when the final size is unknown beforehand. Only using realloc() can you there achieve the maximum possible dataset size.
Your assumption is wrong. Please do note that a pointer is not a buffer. When the function realloc() succeeds, it deallocates the old pointer(frees the original buffer) and return a new pointer to the new allocation(buffer), but when it fails, it leaves the old buffer intact and returns NULL.
So, you do not need a temporary buffer. You need a temporary pointer. I am going to borrow the example from kninnug, this is what you need to do:
char * ptr = malloc(256);
if (!ptr) {
return 1;
}
char * tmpPTR = realloc(ptr, 512);
if (!tmpPTR) {
printf("Houston, we have a problem");
// ptr is fine
}
else {
ptr = tmpPTR;
}
// ptr is realloc()ed
I want to create a program that will simulate an out-of-memory (OOM) situation on a Unix server. I created this super-simple memory eater:
#include <stdio.h>
#include <stdlib.h>
unsigned long long memory_to_eat = 1024 * 50000;
size_t eaten_memory = 0;
void *memory = NULL;
int eat_kilobyte()
{
memory = realloc(memory, (eaten_memory * 1024) + 1024);
if (memory == NULL)
{
// realloc failed here - we probably can't allocate more memory for whatever reason
return 1;
}
else
{
eaten_memory++;
return 0;
}
}
int main(int argc, char **argv)
{
printf("I will try to eat %i kb of ram\n", memory_to_eat);
int megabyte = 0;
while (memory_to_eat > 0)
{
memory_to_eat--;
if (eat_kilobyte())
{
printf("Failed to allocate more memory! Stucked at %i kb :(\n", eaten_memory);
return 200;
}
if (megabyte++ >= 1024)
{
printf("Eaten 1 MB of ram\n");
megabyte = 0;
}
}
printf("Successfully eaten requested memory!\n");
free(memory);
return 0;
}
It eats as much memory as defined in memory_to_eat which now is exactly 50 GB of RAM. It allocates memory by 1 MB and prints exactly the point where it fails to allocate more, so that I know which maximum value it managed to eat.
The problem is that it works. Even on a system with 1 GB of physical memory.
When I check top I see that the process eats 50 GB of virtual memory and only less than 1 MB of resident memory. Is there a way to create a memory eater that really does consume it?
System specifications: Linux kernel 3.16 (Debian) most likely with overcommit enabled (not sure how to check it out) with no swap and virtualized.
When your malloc() implementation requests memory from the system kernel (via an sbrk() or mmap() system call), the kernel only makes a note that you have requested the memory and where it is to be placed within your address space. It does not actually map those pages yet.
When the process subsequently accesses memory within the new region, the hardware recognizes a segmentation fault and alerts the kernel to the condition. The kernel then looks up the page in its own data structures, and finds that you should have a zero page there, so it maps in a zero page (possibly first evicting a page from page-cache) and returns from the interrupt. Your process does not realize that any of this happened, the kernels operation is perfectly transparent (except for the short delay while the kernel does its work).
This optimization allows the system call to return very quickly, and, most importantly, it avoids any resources to be committed to your process when the mapping is made. This allows processes to reserve rather large buffers that they never need under normal circumstances, without fear of gobbling up too much memory.
So, if you want to program a memory eater, you absolutely have to actually do something with the memory you allocate. For this, you only need to add a single line to your code:
int eat_kilobyte()
{
if (memory == NULL)
memory = malloc(1024);
else
memory = realloc(memory, (eaten_memory * 1024) + 1024);
if (memory == NULL)
{
return 1;
}
else
{
//Force the kernel to map the containing memory page.
((char*)memory)[1024*eaten_memory] = 42;
eaten_memory++;
return 0;
}
}
Note that it is perfectly sufficient to write to a single byte within each page (which contains 4096 bytes on X86). That's because all memory allocation from the kernel to a process is done at memory page granularity, which is, in turn, because of the hardware that does not allow paging at smaller granularities.
All the virtual pages start out copy-on-write mapped to the same zeroed physical page. To use up physical pages, you can dirty them by writing something to each virtual page.
If running as root, you can use mlock(2) or mlockall(2) to have the kernel wire up the pages when they're allocated, without having to dirty them. (normal non-root users have a ulimit -l of only 64kiB.)
As many others suggested, it seems that the Linux kernel doesn't really allocate the memory unless you write to it
An improved version of the code, which does what the OP was wanting:
This also fixes the printf format string mismatches with the types of memory_to_eat and eaten_memory, using %zi to print size_t integers. The memory size to eat, in kiB, can optionally be specified as a command line arg.
The messy design using global variables, and growing by 1k instead of 4k pages, is unchanged.
#include <stdio.h>
#include <stdlib.h>
size_t memory_to_eat = 1024 * 50000;
size_t eaten_memory = 0;
char *memory = NULL;
void write_kilobyte(char *pointer, size_t offset)
{
int size = 0;
while (size < 1024)
{ // writing one byte per page is enough, this is overkill
pointer[offset + (size_t) size++] = 1;
}
}
int eat_kilobyte()
{
if (memory == NULL)
{
memory = malloc(1024);
} else
{
memory = realloc(memory, (eaten_memory * 1024) + 1024);
}
if (memory == NULL)
{
return 1;
}
else
{
write_kilobyte(memory, eaten_memory * 1024);
eaten_memory++;
return 0;
}
}
int main(int argc, char **argv)
{
if (argc >= 2)
memory_to_eat = atoll(argv[1]);
printf("I will try to eat %zi kb of ram\n", memory_to_eat);
int megabyte = 0;
int megabytes = 0;
while (memory_to_eat-- > 0)
{
if (eat_kilobyte())
{
printf("Failed to allocate more memory at %zi kb :(\n", eaten_memory);
return 200;
}
if (megabyte++ >= 1024)
{
megabytes++;
printf("Eaten %i MB of ram\n", megabytes);
megabyte = 0;
}
}
printf("Successfully eaten requested memory!\n");
free(memory);
return 0;
}
A sensible optimisation is being made here. The runtime does not actually acquire the memory until you use it.
A simple memcpy will be sufficient to circumvent this optimisation. (You might find that calloc still optimises out the memory allocation until the point of use.)
Not sure about this one but the only explanation that I can things of is that linux is a copy-on-write operating system. When one calls fork the both processes point to the same physically memory. The memory is only copied once one process actually WRITES to the memory.
I think here, the actual physical memory is only allocated when one tries to write something to it. Calling sbrk or mmap may well only update the kernel's memory book-keep. The actual RAM may only be allocated when we actually try to access the memory.
Basic Answer
As mentioned by others, the allocation of memory, until used, does not always commit the necessary RAM. This happens if you allocate a buffer larger than one page (usually 4Kb on Linux).
One simple answer would be for your "eat memory" function to always allocate 1Kb instead of increasingly larger blocks. This is because each allocated blocks start with a header (a size for allocated blocks). So allocating a buffer of a size equal to or less than one page will always commit all of those pages.
Following Your Idea
To optimize your code as much as possible, you want to allocate blocks of memory aligned to 1 page size.
From what I can see in your code, you use 1024. I would suggest that you use:
int size;
size = getpagesize();
block_size = size - sizeof(void *) * 2;
What voodoo magic is this sizeof(void *) * 2?! When using the default memory allocation library (i.e. not SAN, fence, valgrin, ...), there is a small header just before the pointer returned by malloc() which includes a pointer to the next block and a size.
struct mem_header { void * next_block; intptr_t size; };
Now, using block_size, all your malloc() should be aligned to the page size we found earlier.
If you want to properly align everything, the first allocation needs to use an aligned allocation:
char *p = NULL;
int posix_memalign(&p, size, block_size);
Further allocations (assuming your tool only does that) can use malloc(). They will be aligned.
p = malloc(block_size);
Note: please verify that it is indeed aligned on your system... it works on mine.
As a result you can simplify your loop with:
for(;;)
{
p = malloc(block_size);
*p = 1;
}
Until you create a thread, the malloc() does not use mutexes. But it still has to look for a free memory block. In your case, though, it will be one after the other and there will be no holes in the allocated memory so it will be pretty fast.
Can it be faster?
Further note about how memory is generally allocated in a Unix system:
the malloc() function and related functions will allocate a block in your heap; which at the start is pretty small (maybe 2Mb)
when the existing heap is full it gets grown using the sbrk() function; as far as your process is concerned, the memory address always increases, that's what sbrk() does (contrary to MS-Windows which allocates blocks all over the place)
using sbrk() once and then hitting the memory every "page size" bytes would be faster than using malloc()
char * p = malloc(size); // get current "highest address"
p += size;
p = (char*)((intptr_t)p & -size); // clear bits (alignment)
int total_mem(50 * 1024 * 1024 * 1024); // 50Gb
void * start(sbrk(total_mem));
char * end((char *)start + total_mem);
for(; p < end; p += size)
{
*p = 1;
}
note that the malloc() above may give you the "wrong" start address. But your process really doesn't do much, so I think you'll always be safe. That for() loop, however, is going to be as fast as possible. As mentioned by others, you'll get the total_mem of virtual memory allocated "instantly" and then the RSS memory allocated each time you write to *p.
WARNING: Code not tested, use at your own risk.
I'm trying to create my own malloc() for practice. I got the code below from this thread.
typedef struct free_block {
size_t size;
struct free_block* next;
} free_block;
static free_block free_block_list_head = { 0, 0 };
// static const size_t overhead = sizeof(size_t);
static const size_t align_to = 16;
void* malloc(size_t size) {
size = (size + sizeof(free_block) + (align_to - 1)) & ~ (align_to - 1);
free_block* block = free_block_list_head.next;
free_block** head = &(free_block_list_head.next);
while (block != 0) {
if (block->size >= size) {
*head = block->next;
return ((char*)block) + sizeof(free_block);
}
head = &(block->next);
block = block->next;
}
block = (free_block*)sbrk(size);
block->size = size;
return ((char*)block) + sizeof(free_block);
}
void free(void* ptr) {
free_block* block = (free_block*)(((char*)ptr) - sizeof(free_block ));
block->next = free_block_list_head.next;
free_block_list_head.next = block;
}
I'm confused about treating memory chunks as a linked list. It seems to me that we basicaly call sbrk() everytime we need memory and we check if some of the memory that we requested before wasn't freed in the meantime.
But we have no way of checking other memory chunks that belong to other processes, we only check the memory that we requested before and added to our linked list.
If this is the case, is this optimal? Is this how the standard malloc() works?
Is there a way for us to work with all the memory on the heap ?
Please explain like I'm 5, I'm having a hard time understanding this concept.
Extending process data segment doesn't affect other processes. On most (recent) architectures process memory model is flat, i.e. each process has a virtual address space (2^32 or 2^64 bytes). When process requests extra memory (page), a virtual memory is added to process. In fact, that doesn't mean any physical memory allocation occurs, as virtual memory can be mapped to swap file, or unmapped before use altogether (address is given to process, but no actual resources assigned to it). Kernel takes care of mapping physical address to virtual one as per need/per resource availability.
What the algorithm does?
When user calls malloc the algorithm tries to find available empty block. In the beginning, there is none, so the algorithm tries to extend the process data segment.
However, you can see, that free doesn't release virtual memory (as it is not as trivial as allocating it), instead it adds this released block to a list of unused blocks.
So, when there are prereleased blocks, malloc attempts to reuse them instead of extendind the data segment.
Do standard mallocs work as above: no. The example you've provided is simple, yet really inefficient. There are many different algorithms available for memory management: small block heaps (when allocating data up to certain amount has a O(1) performance), thread-specific allocators (reducing congestion of accessing heap from multiple threads), allocators, that pre-allocate large chunks and then use them (similar to above, yet more efficient) and other.
You can try to goodle for "memory heap implementation" for more info
I have a code block that seems to be the code behind malloc. But as I go through the code, I get the feeling that parts of the code are missing. Does anyone know if there is a part of the function that's missing? Does malloc always combine adjacent chunks together?
int heap[10000];
void* malloc(int size) {
int sz = (size + 3) / 4;
int chunk = 0;
if(heap[chunk] > sz) {
int my_size = heap[chunk];
if (my_size < 0) {
my_size = -my_size
}
chunk = chunk + my_size + 2;
if (chunk == heap_size) {
return 0;
}
}
The code behind malloc is certainly much more complex than that. There are several strategies. One popular code is the dlmalloc library. A simpler one is described in K&R.
The code is obviously incomplete (not all paths return a value). But in any case this is not a "real" malloc. This is probably an attempt to implement a highly simplified "model" of 'malloc'. The approach chosen by the author of the code can't really lead to a useful practical implementation.
(And BTW, standard 'malloc's parameter has type 'size_t', not 'int').
Well, one error in that code is that it doesn't return a pointer to the data.
I suspect the best approach to that code is [delete].
When possible, I expect that malloc will try to put different requests close to each other, as it will have a block of code that is available for malloc, until it has to get a new block.
But, that also depends on the requirements imposed by the OS and hardware architecture. If you are only allowed to request a certain minimum size of code then it may be that each allocation won't be near each other.
As others mentioned, there are problems with the code snippet.
You can find various open-source projects that have their own malloc function, and it may be best to look at one of those, in order to get an idea what is missing.
malloc is for dynamically allocated memory. And this involves sbrk, mmap, or maybe some other system functions for Windows and/or other architectures. I am not sure what your int heap[10000] is for, as the code is too incomplete.
Effo's version make a little bit more sense, but then it introduce another black box function get_block, so it doesn't help much.
The code seems to be run on a metal machine, normally no virtual address mapping on such a system which only use physical address space directly.
See my understanding, on a 32 bits system, sizeof(ptr) = 4 bytes:
extern block_t *block_head; // the real heap, and its address
// is >= 0x80000000, see below "my_size < 0"
extern void *get_block(int index); // get a block from the heap
// (lead by block_head)
int heap[10000]; // just the indicators, not the real heap
void* malloc(int size)
{
int sz = (size + 3) / 4; // make the size aligns with 4 bytes,
// you know, allocated size would be aligned.
int chunk = 0; // the first check point
if(heap[chunk] > sz) { // the value is either a valid free-block size
// which meets my requirement, or an
// address of an allocated block
int my_size = heap[chunk]; // verify size or address
if (my_size < 0) { // it is an address, say a 32-bit value which
// is >0x8000...., not a size.
my_size = -my_size // the algo, convert it
}
chunk = chunk + my_size + 2; // the algo too, get available
// block index
if (chunk == heap_size) { // no free chunks left
return NULL; // Out of Memory
}
void *block = get_block(chunk);
heap[chunk] = (int)block;
return block;
}
// my blocks is too small initially, none of the blocks
// will meet the requirement
return NULL;
}
EDIT: Could somebody help to explain the algo, that is, converting address -> my_size -> chunk? you know, when call reclaim, say free(void *addr), it'll use this address -> my_size -> chunk algo too, to update the heap[chunk] accordingly after return the block to the heap.
To small to be a whole malloc implementation
Take a llok in the sources of the C library of Visual Studio 6.0, there you will find the implementation of malloc if I remeber it correctly