Background:
I'm working through a project whose goal is to build a memory management system that replicates malloc() and free(). The program begins by requesting a specified amount of memory from the operating system with mmap and then uses a linked list to keep track of which sections of memory are in use or free. See below for a general idea of what I'm doing.
Output after allocating one chunk of memory
Question
Why do I hit a segmentation fault when trying to create a new list node to represent free memory? This segmentation fault occurs only when trying to request more than two chunks of memory - the first works as expected.
Details
The segmentation fault occurs when trying to assign values to the attributes of the node just created (new in the code below) to represent the remaining free memory space. This assignment works the first time I split memory into used v. free segments, but the second time I try to split the free memory into used v. free segments, my program encounters a segmentation fault - I am assuming this occurs because the attributes of the new node_t struct are undefined (compiler doesn't know where they live in memory). What I don't know is why this is occurring, and why it doesn't occur the first time I use this code.
NOTE: The answer shouldn't be to use malloc when initializing the *new node because the point of the project is to do this without using malloc. Malloc wouldn't allocate from the memory I'm managing - instead, it'd allocate from the heap my program is actually maintaining (but not using) on the side.
Struct definitions:
typedef enum {false, true} bool;
typedef struct _node_t {
int size;
struct _node_t *next;
struct _node_t *prev;
bool used;
} node_t;
Problem code. ret is the memory allocation I'm going to return to the caller and new is the remainder of the free memory block I just allocated part of.
// Code above this steps through the list elements to find the first free
// element that is large enough. I've debugged enough to know that's not the
// source of the problem.
node_t *new = ret + sizeof(node_t) + size; // Pointer to new free slot
assert(new != NULL);
new->prev = ret;
new->next = ret->next;
new->size = ret->size - sizeof(node_t) - size;
new->used = false;
ret->used = true;
ret->next = new;
ret->size = size;
return (ret + sizeof(node_t));
Calling code:
int main(int argc, char* argv[]) {
int success = Mem_Init(5000, P_FIRSTFIT);
if (success == -1)
return -1;
int *slot1 = Mem_Alloc(sizeof(int));
// This is where segmentation fault occurs
int *slot2 = Mem_Alloc(sizeof(int));
Link to code you can compile and run.
Although you don't include a definition of ret and Dropbox is totally useless for viewing code on a smartphone, I think we can deduce the declaration
node_t* ret;
Otherwise ret->next won't compile.
In that case, ret + sizeof(node_t) does not make sense because of the nature of pointer arithmetic in C. Remember that p + k is exactly the same as &p[k]. In other words, p + 1 is the address of the next element of size sizeof*p. So ret + sizeof(node_t) is the address of the 16th or 32nd element after ret, depending on the sizeof (node_t).
What you probably wanted was ret + 1, whose simplicity illustrates the rationale behind C pointer arithmetic.
The answer: It's all in pointer math. The issue was this statement:
node_t *new = ret + sizeof(node_t) + size; // Pointer to new free slot
Any integer in the right hand side of this equation doesn't increment the memory address by one byte... it increments it to the next spot where a node_t struct would begin. So in the code above, the address of new is set (32 + size) * 32 bytes further along in memory (assuming sizeof(node_t) is 32, which is true on a 64 bit system). Do that enough and you run well outside of the 500 bytes of memory you actually requested from the OS.
The code that works is:
node_t *new = ret + 1 + (size / sizeof(node_t)) + 1;
This code ends up over-allocating memory a bit if size isn't exactly equal to sizeof(node_t), but that's a problem with a more complicated solution. For illustration, consider this example:
ret = address 1000
sizeof(node_t) = 32 (in a 64 bit system)
size = 50
The end of the header for ret is address 1032 in this example. The program then allocates 2x32 bytes to handle the 50 byte request (math is (50 / 32) + 1 = 2, or two node_t structs). So the memory address of new is actually 1096.
To make this work overall, I'll also have to update the size calculation to reflect the actual allocation instead of the requested allocation. This will be necessary to make a safeguard against running outside of the bounds of memory requested from the OS work properly.
Related
I was trying to see the working of oom_kill by invoking manually.
I allocated memory dynamically and tried to use them infinitely with while loop at first and then with the for loop to test out of memory.
But in the first case where I used the while loop it threw segmentation fault without swap and became unresponsive with swap whereas with the for loop out of memory (oom_kill) was invoked.
Sample codes of both:
First case: while:
int main (void) {
char* p;
while (1) {
p=malloc(1<<20);
memset (p, 0, (1<<20));
}
}
Second case : for :
int main (void) {
int i, n = 0;
char *pp[N];
for (n = 0; n < N; n++) {
pp[n] = malloc(1<<20);
if (pp[n] == NULL)
break;
}
printf("malloc failure after %d MiB\n", n);
for (i = 0; i < n; i++) {
memset (pp[i], 0, (1<<20));
printf("%d\n", i+1);
}
where N is some very large number to invoke oom. Referred this https://www.win.tue.nl/~aeb/linux/lk/lk-9.html for 2nd case.
Why does it happen so? What is the mistake I'm making with the while loop?
Kernel version : 4.15
Why does it happen so?
To invoke the OOM killer, you need to have a situation where an access to memory cannot be fulfilled because there is not enough RAM available to fulfill the access. To do that, you want to first have large allocations (virtual memory mappings), then write to them.
The procedure to trigger the OOM killer is very simple:
Allocate lots of memory
Write to the allocated memory
You must have enough preallocated memory to cause everything evictable from RAM to be evicted (things like memory-mapped files), and all of swap to be used, before the kernel will evoke the OOM killer to provide more RAM/swap space to fullfill the backing to the virtual memory being written to.
What is the mistake I'm making with the while loop?
One bug, and one logical error.
The bug is, you do not check if malloc() returns NULL. malloc() returns NULL, when there is no more virtual memory available (or kernel refuses to provide more, for any reason) for the process. (In normal operation, the virtual memory available to each process is limited for non-privileged users; run e.g. ulimit -a to see the current limits.)
Because you access the memory immediately when allocated, the kernel simply refuses to allow your process more when it runs out of RAM and SWAP, and malloc() returns NULL. You then dereference the NULL pointer (by using memset(NULL, 0, 1<<20)), which causes the Segmentation fault.
The logical problem is that that scheme will not trigger the kernel OOM killer.
Remember, in order to trigger the kernel OOM killer, your process must have allocated memory that it has not accessed yet. The kernel evokes the OOM killer only when it has already provided the virtual memory, but cannot back it with actual RAM, because there is nothing evictable in RAM, and swap is already full.
In your case, the OOM killer will not get evoked, because when the kernel runs out of RAM and swap, it can simply refuse to provide more (virtual memory), leading to malloc() returning NULL.
(The Linux kernel memory subsystem is one that is actively developed, so the exact behaviour you see depends on both the kernel version, the amount of RAM and swap, and the memory manager tunables (e.g., those under /proc/sys/vm/). The above describes the most common, or typical cases and configurations.)
You don't need an external array, either. You can for example chain the allocations to a linked list:
#include <stdlib.h>
#include <stdio.h>
#ifndef SIZE
#define SIZE (2*1024*1024) /* 2 MiB */
#endif
struct list {
struct list *next;
size_t size;
char data[];
}
struct list *allocate_node(const size_t size)
{
struct list *new_node;
new_node = malloc(sizeof (struct list) + size);
if (!new_node)
return NULL;
new_node->next = NULL;
new_node->size = size;
}
int main(void)
{
size_t used = 0;
struct list *root = NULL, *curr;
/* Allocate as much memory as possible. */
while (1) {
curr = allocate_node(SIZE - sizeof (struct list));
if (!curr)
break;
/* Account for allocated total size */
used += SIZE;
/* Prepend to root list */
curr->next = root;
root = curr;
}
printf("Allocated %zu bytes.\n", used);
fflush(stdout);
/* Use all of the allocated memory. */
for (curr = root; curr != NULL; curr = curr->next)
if (curr->size > 0)
memset(curr->data, ~(unsigned char)0, curr->size);
printf("Wrote to %zu bytes of allocated memory. Done.\n", used);
fflush(stdout);
return EXIT_SUCCESS;
}
Note, the above code is untested, and even uncompiled, but the logic is sound. If you find a bug in it, or have some other issue with it, let me know in a comment so I can verify and fix.
The document you're reading is from 2003. The impossibly large number it chose to allocate was 10,000 MiB.
Today, in 2018, when new computers are likely to come with 16GiB of RAM, this kind of allocation could definitely succeed without issues.
What is the mistake I'm making with the while loop?
The segmentation fault is likely the result of passing a null pointer to memset(), since malloc() will return NULL on error.
Your second example avoids this error by always checking the return value from malloc().
I used the while loop it ... became unresponsive with swap ...
From the very document that you mentioned that you are reading:
Sometimes processes get a segfault when accessing memory that the kernel is unable to provide, sometimes they are killed, sometimes other processes are killed, sometimes the kernel hangs.
Other than mentioning the kernel version, you are very vague with the OS and system description. Presumably this is a 32-bit version?
There are actually two ways of running out of memory. Your program could exceed the amount of (virtual) memory that is allocated, or the system could actually run out of memory pages.
Note that availability of memory (pages) is a complex combination of physical memory size, swap space size, memory usage and process load.
Reference: When Linux Runs Out of Memory
by Mulyadi Santosa or here.
I have created a linked list with the following code. As you can see, I have used malloc to created a list of size 3. But I ran the for loop over size 10 to initialize and print.
#include <stdio.h>
#include <malloc.h>
#include <stdlib.h>
struct node {
int value;
struct node *next;
};
int main() {
//code
struct node **head;
struct node *curr;
curr = (struct node *) malloc(sizeof(struct node)*3);
head = &curr;
printf("done 1\n");
(*head)->value = 0;
(*head)->next = NULL;
for(int i = 1; i < 10; i++) {
(*head+i-1)->next = (*head+i);
(*head+i)->value = i;
(*head+i)->next = NULL;
}
curr = *head;
printf("done 2\n");
for(int i = 0; i < 10; i++) {
printf("%d\t", (*head + i)->value);
//curr = curr->next;
}
printf("\ndone 3\n");
//free(curr);
return 0;
}
when I compile and run the code, the result obtained is,
done 1
done 2
0 1 2 3 154208560 842282289 876087600 154744882 808859448 875837236
done 3
Why am I able to assign values to the 4th node and access it when I actually created a list of size 3?
I see that garbage values were being printed from 5th to 10th node access. But how is the 4th node being created?
P.S:
I know that 10!=3. The code ran properly when I put the loop in its limits. I wanted to see what will happen when we go out of bounds. I see that the 4th node was also created as I was able to assign value when I actually created a list of size 3.
This is purely to see if I will get seg fault or not.
You're invoking undefined behavior. When you do this, the program may crash, it may appear to work properly, or it might behave in a seemingly random manner.
C doesn't perform any kind of bounds checks on arrays or allocated memory. It's one of the things that makes it fast. What that also means is that it will allow you to do things you're not supposed to do. It trusts the programmer to "do the right thing".
On your particular machine, you see random data after the third element. When I run the same code on my machine, I happen to get the expected output as though enough memory was allocated. Also, if I uncomment the call to free, the program crashes. That's undefined behavior.
Trying to understand undefined behavior is usually a futile effort. It all depends on the implementation details of your compiler and the machine it runs on. In this case, the memory being written to was probably unallocated memory in the heap after the memory that was properly allocated. Depending on how malloc is implemented, that part of memory may contain data needed for malloc and other functions to run properly. The point is that writing past the end of an array does not guarantee a crash, so you need to be careful.
This code will behave differently on the different operating systems.
You may get away even with assigning all 10 integers and print then correctly.
Your code puts the beginning of list on the heap with allocations for only 3 elements.
You may get lucky and use memory for all 10 element or your program will crash on the 4-th!
I want to create a program that creates link list which contains arrays.
However, I want each link to have a smaller array than previous one. at this moment the program works, but I want to know whether logically the program is actually creating smaller arrays, so no free space is wasted.
int main()
{
int c=3;
int d=0;
typedef struct mylist {
struct mylist *link;
int info[c-d];
}Node;
Node *a = (Node*) malloc (sizeof (Node));
a -> link = NULL;
Node *b = a;
int i=0,j=0;
while (i!=4){
while ((j)<=(2-d)){
printf("link%d array%d ",i,j);
scanf("%d",&a->info[j]);
j++;
}
j=0;
if (i !=3 ){
a -> link = (Node*) malloc (sizeof (Node));
a = a -> link;
}
d++;
i++;
}
d=0;
a = b;
while (a != NULL){
while ((j)<=(2-d)){
printf("%d ->", a->info[j]);
j++;
}
j=0;
d++;
a = a -> link;
}
a=b;
printf(" the end");
return 0;
}
First of all, you're not free()ing memory which you allocated, make sure you take care of that.
Now, you're asking whether are you not using any additional memory - reading your code is pretty hard for us, as you can see in the comments. It is always a good idea to try it out yourself - find out the size of the memory occupied during the initialization of the structure, either with the sizeof() operator or with checking your values.
However, I don't think your code will work properly, due to the fact that you are statically initializing the arrays size. As the values are determined compile-time, this is completely fine. However, the problem is that you have already decided the size of the int array at the start.
If you want to use dynamic allocation of memory, you need to use the malloc() function. When creating a new node, you will have to allocate memory for an array in it as well (and calculate how much size you want to give it). This will also solve your problem of not knowing the size of the array or not being sure about it very well - as you are explicitly declaring how much memory you're allocating for the array.
Just a small addition though. When allocating for example 20 bytes of memory, the OS won't give you specifically 20 bytes - you'll get an entire page, which will often be more memory than necessary. So unless you're working with a device where controlling memory properly is really important or you're working with huge sizes of memory, you should not need to overthink the difference of the size of an array of 4 ints and 3 ints.
This question is a continuation of Malloc call crashing, but works elsewhere
I tried the following program and I found it working (i.e. not crashing - and this was mentioned in the above mentioned link too). I May be lucky to have it working but I'm looking for a reasonable explanation from the SO experts on why this is working?!
Here are some basic understanding on allocation of memory using malloc() w.r.t structures and pointers
malloc(sizeof(struct a) * n) allocates n number of type struct a elements. And, this memory location can be stored and accessed using a pointer-to-type-"struct a". Basically a struct a *.
malloc(sizeof(struct a *) * n) allocates n number of type struct a * elements. Each element can then point to elements of type struct a. Basically malloc(sizeof(struct a *) * n) allocates an array(n-elements)-of-pointers-to-type-"struct a". And, the allocated memory location can be stored and accessed using a pointer-to-(pointer-to-"struct a"). Basically a struct a **.
So when we create an array(n-elements)-of-pointers-to-type-"struct a", is it
valid to assign that to struct a * instead of struct a ** ?
valid to access/de-reference the allocated array(n-elements)-of-pointers-to-type-"struct a" using pointer-to-"struct a" ?
data * array = NULL;
if ((array = (data *)malloc(sizeof(data *) * n)) == NULL) {
printf("unable to allocate memory \n");
return -1;
}
The code snippet is as follows:
#include <stdio.h>
#include <stdlib.h>
int main(void)
{
typedef struct {
int value1;
int value2;
}data;
int n = 1000;
int i;
int val=0;
data * array = NULL;
if ((array = (data *)malloc(sizeof(data *) * n)) == NULL) {
printf("unable to allocate memory \n");
return -1;
}
printf("allocation successful\n");
for (i=0 ; i<n ; i++) {
array[i].value1 = val++;
array[i].value2 = val++;
}
for (i=0 ; i<n ; i++) {
printf("%3d %3d %3d\n", i, array[i].value1, array[i].value2);
}
free(array);
printf("freeing successful\n");
return 0;
}
EDIT:
OK say if I do the following by mistake
data * array = NULL;
if ((array = (data *)malloc(sizeof(data *) * n)) == NULL) {
Is there a way to capture (during compile-time using any GCC flags) these kind of unintended programming typo's which could work at times and might blow out anytime! I compiled this using -Wall and found no warnings!
There seems to be a fundamental misunderstanding.
malloc(sizeof(struct a) * n) allocates n number of type struct a elements.
No, that's just what one usually does use it as after such a call. malloc(size) allocates a memory region of size bytes. What you do with that region is entirely up to you. The only thing that matters is that you don't overstep the limits of the allocated memory. Assuming 4 byte float and int and 8 byte double, after a successful malloc(100*sizeof(float));, you can use the first 120 of the 400 bytes as an array of 15 doubles, the next 120 as an array of 30 floats, then place an array of 20 chars right behind that and fill up the remaining 140 bytes with 35 ints if you wish. That's perfectly harmless defined behaviour.
malloc returns a void*, which can be implicitly cast to a pointer of any type, so
some_type **array = malloc(100 * sizeof(data *)); // intentionally unrelated types
is perfectly fine, it might just not be the amount of memory you wanted. In this case it very likely is, because pointers tend to have the same size regardless of what they're pointing to.
More likely to give you the wrong amount of memory is
data *array = malloc(n * sizeof(data*));
as you had it. If you use the allocated piece of memory as an array of n elements of type data, there are three possibilities
sizeof(data) < sizeof(data*). Then your only problem is that you're wasting some space.
sizeof(data) == sizeof(data*). Everything's fine, no space wasted, as if you had no typo at all.
sizeof(data) > sizeof(data*). Then you'll access memory you shouldn't have accessed when touching later array elements, which is undefined behaviour. Depending on various things, that could consistently work as if your code was correct, immediately crash with a segfault or anything in between (technically it could behave in a manner that cannot meaningfully be placed between those two, but that would be unusual).
If you intentionally do that, knowing point 1. or 2. applies, it's bad practice, but not an error. If you do it unintentionally, it is an error regardless of which point applies, harmless but hard to find while 1. or 2. applies, harmful but normally easier to detect in case of 3.
In your examples. data was 4 resp. 8 bytes (probably), which on a 64-bit system puts them into 1. resp. 2. with high probability, on a 32-bit system into 2 resp. 3.
The recommended way to avoid such errors is to
type *pointer = malloc(num_elems * sizeof(*pointer));
No.
sizeof(struct a*) is the size of a pointer.
sizeof(struct a) is the size of the entire struct.
This array = (data *)malloc(sizeof(data *) * n) allocates a sizeof(data*) (pointer) to struct data, if you want to do that, you need a your array to be a data** array.
In your case you want your pointer to point to sizeof(data), a structure in memory, not to another pointer. That would require a data** (pointer to pointer).
is it valid to assign that to struct a * instead of struct a ** ?
Well, technically speaking, it is valid to assign like that, but it is wrong (UB) to dereference such pointer. You don't want to do this.
valid to access/de-reference the allocated array(n-elements)-of-pointers-to-type-"struct a" using pointer-to-"struct a" ?
No, undefined behavior.
So I am trying to create a Memory Management System. In order to do this I have a set amount of space (allocated by malloc) and then I have a function myMalloc which will essentially return a pointer to the space allocated. Since we will then try and free it, we are trying to set a header of the allocated space to be the size of the allocated space, using memset.
memset(memPtr,sizeBytes,sizeof(int));
We then need to be able to read this so we can see the size of it. We are attempting to do this by using memcpy and getting the first sizeof(int) bytes into a variable. For testing purposes we are just trying to do memset and then immediately get the size back. I've included the entire method below so that you can see all declarations. Any help would be greatly appreciated! Thanks!
void* FirstFit::memMalloc(int sizeBytes){
node* listPtr = freelist;
void* memPtr;
// Cycle through each node in freelist
while(listPtr != NULL)
{
if(listPtr->size >= sizeBytes)
{
// We found our space
// This is where the new memory allocation begins
memPtr = listPtr->head;
memset(memPtr,sizeBytes,sizeof(int));
void *size;
memcpy(size, memPtr, sizeof(int));
// Now let's shrink freelist
listPtr->size = listPtr->size - sizeBytes;
int *temp = (int*)listPtr->head + (sizeBytes*sizeof(int));
listPtr->head = (int*) temp;
return memPtr;
}
listPtr = listPtr->next;
}
::Edit::
Sorry! When running this, we keep getting a seg fault when attempting to run the memcpy line. We have been playing with different ideas for the past hour or so and honestly just have no idea where the error is occurring.
::Edit2::
I also posted this as a comment, but figured I'd put it here as well, so it was easier to find...
Our problem is that we have an allocated space that we are allowed to work with, specified by one malloc call for 128MB. We can only use this, so we can't initialize size to anything using malloc. I guess, is there a way to do this without initializing size. If not, is there anyway to get the int that the header is set to without using memcpy.
The prototype for memcpy is
void * memcpy ( void * destination, const void * source, size_t num );
The problem lies here:
void *size; /* you haven't initialized this variable, and then you're writing to what it points to*/
memcpy(size, memPtr, sizeof(memPtr)); /* because size points to uninitialized memory it seg faults*/
EDIT1:
Please review this tutorial on pointers in C and C++ Unless you understand pointers you will not understand why those two lines of code, back to back, are a bad pair.
You code has numerous bugs in it - rather than go through them one-by-one, I'll give you a commented version of what it should look like:
void* FirstFit::memMalloc(size_t sizeBytes) // size_t is the appropriate type for memory sizes
{
node* listPtr = freelist;
void* memPtr;
// The actual allocation needs to be bigger, to have space to hold the size itself.
size_t allocSize = sizeBytes + sizeof allocSize;
// Test to make sure that allocSize didn't wrap around to zero
if (allocSize < sizeBytes)
{
return NULL;
}
// Cycle through each node in freelist
while(listPtr != NULL)
{
if(listPtr->size >= allocSize)
{
// We found our space
// This is where the new memory allocation begins
memPtr = listPtr->head;
// Copy the size to the start of the memory region
memcpy(memPtr, &allocSize, sizeof allocSize);
// Increment the pointer to be returned past the size
char *tempPtr = (char *)memPtr;
memPtr = (void *)(tempPtr + sizeof allocSize);
// Shrink the block
listPtr->size -= allocSize;
tempPtr = (char *)listPtr->head;
listPtr->head = (void *)(tempPtr + allocSize);
// TODO: If the block is now zero-sized, remove it from the linked list
return memPtr;
}
listPtr = listPtr->next;
}
/* No space */
return NULL;
}
void *size; is an uninitialized pointer, when you try to memcpy into it, your process will try to write this invalid location resulting seg fault.
Your use of memset is very odd:
memset(memPtr,sizeBytes,sizeof(int));
is equivalent to (assuming a 32 bit integer):
*((char *)memPtr + 0) = (sizeByte & 0xFF);
*((char *)memPtr + 1) = (sizeByte & 0xFF);
*((char *)memPtr + 2) = (sizeByte & 0xFF);
*((char *)memPtr + 3) = (sizeByte & 0xFF);
As you can see, it's setting each byte to the same value which is the lower byte of sizeBytes.
I'm not sure what you are intending to do so I can't offer a fix.
if you are writing it in windows... you can use
IsBadWritePtr
To Verify that the calling process has write access to the specified range of memory.
There may be three reason
1>pointers is either garbage or NULL
2>the amount you're trying to copy is too much.
i.e. copying past the end of the block of memory. Potentially "backwards" copy of string-literal would also cause this behaviour
char *s = "Hello";
char t[10];
memcpy(s, t, 6);
In creating your own memory management system, now that you have learned what memset() does and doesn't, hopefully already knowing enough of the low level stuff that you know the difference between memcpy() and memmove(), your next step is to learn about "alignment" and about the guarantees that malloc() fulfills, but your code doesn't.