I am going through this link and learning C. Interesting part on the page:
The real purpose of unions is to prevent memory fragmentation by arranging for a standard size for data in the memory. By having a standard data size we can guarantee that any hole left when dynamically allocated memory is freed will always be reusable by another instance of the same type of union.
I understand this part by the following code:
typedef struct{
char name[100];
int age;
int rollno;
}student;
typedef union{
student *studentPtr;
char *text;
}typeUnion;
int
main(int argc, char **argv)
{
typeUnion union1;
//use union1.studentPtr
union1.text="Welcome to StackOverflow";
//use union1.text
return 0;
}
Well, in the above code union1.text is reusing the space previously used by union1.studentPtr, not completely but still using.
Now, the part I don't understand is, when is the freed up space of malloc can't be used which leads to memory fragmentation?
edit: Going through the comments and answers, it is imperative to use the classic text, adding this edit to the post presuming it will help beginners like me
the comments have more expertise regarding unions in general.
Regarding your question specifically, this is my understanding:
union sets aside memory for the largest datatype in the union variable. So for example having a short int and a long int in the union will set aside enough memory for a long int
Imagine instead of union you declare a short int variable.
But then need to use a long int. So you use free on the short int
Then you use malloc to allocate memory for a long int. This has to be continguous memory. So now your memory looks like this.
With a free byte in the middle of an otherwise used block of memory. Sitting there waiting for you to request specifically 1 byte of memory.
Aside: If you're learning c I recommend the classic text. It's dated but I love the simplicity, clarity and text-book style approach.
That page is wrong, most programs simply assume memory usage will be fine and don't pay any attention to it.
In the following diagrams, assume each character represents, say 8 bytes. Letters represent different allocations, and underscores represent free memory. The scales involved are ludicrously small, and I'm skipping over details (like allocation metadata used by most malloc implementations), but the principles should be there.
Start with empty memory
_____________________________________________________
Then a bunch of 32-byte allocations occur as a program runs.
AAAABBBBCCCCDDDDEEEEFFFFGGGGHHHHIIIIJJJJKKKKLLLL________
The memory allocated by the program extends all the way past the last byte of the 'L' allocation, it's using 12*8*4 = 384 bytes.
Now the program frees every other allocation.
AAAA____CCCC____EEEE____GGGG____IIII____KKKK__________
Now the program is really only using 6*4*8 = 192 bytes, but the operating system has to keep all 352 bytes from the first 'A' to the last 'K' allocated for the program. Those freed gaps in between all the allocations are an example of memory fragmentation.
Now the program wants to allocate another 32-byte block. It could happen like this:
AAAAMMMMCCCC____EEEE____GGGG____IIII____KKKK_________
The new allocation fits in one of the gaps created by the frees, and this is fine since it recycles one of the gaps so we're wasting less space.
Now say the program needs to allocate a 40 byte block. None of the gaps is big enough, so the allocation has to go at the end and the operating system has to allocate more memory for the program, 352+40=392 bytes. All of the memory in those gaps is being wasted. This is the kind of waste due to memory fragmentation the webpage is talking about.
If all of the allocations had been 40 bytes to start with, then gap recycling could be maximized.
Related
I've the following struct:
#define M 3
#pragma pack(push)
#pragma pack(1)
struct my_btree_node {
struct my_btree_node *pointers[M];
unsigned char *keys[M - 1];
int data[M - 1];
unsigned char number_of_keys;
};
#pragma pack(pop)
The sizeof(struct my_btree_node) function returns a value of 49 byte for this struct. Does allocating memory for this struct using malloc return a 64 byte block because on 64 bit systems pointers are 16-byte-aligned or will it indeed be 49 bytes?
Is there a way to align memory with a smaller power of two than 16 and is it possible to get the true size of the allocated memory inside the application?
I want to reduce the number of padding bytes in order to save memory. My applications allocates millions of those structs and I do not want to waste memory.
malloc(3) is defined to
The malloc() and calloc() functions return a pointer to the allocated
memory, which is suitably aligned for any built-in type. On error,
these functions return NULL. NULL may also be returned by a
successful call to malloc() with a size of zero, or by a successful
call to calloc() with nmemb or size equal to zero.
So a conforming implementation has to return a pointer aligned to the largest possible machine alignment (with GCC, it is the macro __BIGGEST_ALIGNMENT__)
If you want less, implement your own allocation routine. You could for example allocate a large array of char and do your allocation inside it. That would be painful, perhaps slower (processors dislike unaligned data, e.g. because of CPU cache constraints), and probably not worthwhile (current computers have several gigabytes of RAM, so a few millions of hundred-byte sized data chunks is not a big deal).
BTW, malloc is practically implemented in the C standard library (but -on Linux at least- the compiler knows about it, thanks to __attribute__-s in GNU glibc headers; so some internal optimizations inside GCC know and take care of calls to malloc).
malloc uses internal heap-structure. It is implementation-dependent yet one may expect that the memory is allocated by a whole number of (internal) blocks. So usually it's not possible to allocate exactly 49 bytes by a single malloc call. You can build some subsystem of your own on top of malloc to do this, yet I see no reason why you may want it.
P.S. To reduce memory wasting, you can pre-allocate an array consisting of, say, 100 structs, when you need just one more, and return &a[i] until all free indexes are wasted. As arrays are never padded, the memory wasting would be reduced in about 100 times.
#include <stdio.h>
#include <string.h>
#include <stdlib.h>
struct Person
{
unsigned long age;
char name[20];
};
struct Array
{
struct Person someone;
unsigned long used;
unsigned long size;
};
int main()
{
//pointer to array of structs
struct Array** city;
//creating heap for one struct Array
struct Array* people=malloc(sizeof(struct Array));
city=&people;
//initalizing a person
struct Person Rob;
Rob.age=5;
strcpy(Rob.name,"Robert");
//putting the Rob into the array
people[0].someone=Rob;
//prints Robert
printf("%s\n",people[0].someone.name);
//another struct
struct Person Dave;
Dave.age=19;
strcpy(Dave.name,"Dave");
//creating more space on the heap for people.
people=realloc(people,sizeof(struct Array)*2);
//How do I know that this data is safe in memory from being overwritten?
people[1].someone=Dave;
//prints Dave
printf("%s\n",people[1].someone.name);
//accessing memory on the heap I do not owe?
people[5].someone=Rob;
//prints "Robert" why is this okay? Am I potentially overwriting memory?
printf("%s\n",people[5].someone.name);
return 0;
}
In the above code I attempt to make a pointer to a dynamic array of structs, unsure if I succeeded in that part, but my main concern is I use malloc to create space on the heap for the array 'people.' Later in the code I create another struct Person and use realloc to create more space on the heap for 'people.' I then write to memory outside of what I thought I gave space for by doing 'people[5].someone=Rob;.' This still works as I have access to the value at that memory location. My question is why does this work? Am I potentially overwriting memory by writing to memory I did not specifically define for people? Am I actually using malloc and realloc correctly? As I did hear that there were ways of testing if they were successful in another post. I am new to C so if my assumptions or terminology is off please correct me.
I'm not an expert in C, not even a middle, most of the time I program in C#, so some mistakes might be there.
Modern operating systems have a special mechanism called the memory manager. Using that mechanism we can ask OS to give us some amount of memory. In Windows there's a special function for that - VirtualAlloc. It's a really powerful function, you can read more about it on MSDN.
It works really great and gives us all the memory we require but there's a little problem - it gives us the whole physical pages (4KB). Well, actually that is not a big problem, you can use this memory in the same way as if it was allocated using malloc. There'll be no error.
But it is a problem because if we, for example, allocate a 10 byte chunk using VirtualAlloc, it will actually give us 4096 byte chunk as the memory size is rounded up to the page size boundary. So VirtualAlloc allocate a 4KB memory chunk, but we actually use only 10 bytes of it. The rest 4086 are "gone". If we create the second 10 byte array, VirtualAlloc will give us another 4096 byte chunk, so two 10 byte arrays will actually take 8KB of RAM.
To solve this problem, every C program uses malloc function, which is a part of the C runtime library. It allocates some space using VirtualAlloc and returns pointers to the parts of it. For example let's return to our previous arrays. If we allocate 10 byte array using malloc, the runtime library will call VirtualAlloc to allocate some space, and malloc will return pointer to the beginning of it. But if we allocate 10 byte array for the second time, malloc won't use VirtualAlloc. Instead, it will use the already allocated page, I mean the free space of it. After allocation of the first array, we got 4086 bytes of unused space in our memory chunk. So malloc will use this space wisely. In this case (for the second array) it will return pointer to "address of chunk" + 10 (that's a memory address).
Now we can allocate about 400 "ten byte arrays" and they will take only 4096 bytes if we use malloc. Naive way using VirtualAlloc would take 400 * 4096 bytes = 1600KB, that's a rather big figure in comparison to 4096 bytes using malloc.
There's another reason - performance, as VirtualAlloc is a really expensive operation. However, malloc will do some pointer math if you have free space in the allocated chunks, but if you don't have any free allocated space, it will call VirtualAlloc. Actually it's much more complicated than I say, but I think that would be enough to explain the cause.
Okay, let's return to the question. You allocate the memory for the Array array. Let's calculate it's size: sizeof(Person) = sizeof(long) + sizeof(char[20]) = 4 + 20 = 24 bytes; sizeof(Array) = sizeof(Person) + 2 * sizeof(long) = 24 + 8 = 32 bytes. The array of 2 elements will take 32 * 2 = 64 bytes. So, as I said before, malloc will call VirtualAlloc to allocate some memory, and it will return a 4096 bytes page. So, for example let's assume that the address of the chunk's beginning is 0. Application can modify any byte from 0 to 4096 as we allocated the page and we won't get any pagefault. What is an array indexation array[n]? It's just summation of the array's base and the offset calculated as array + n * sizeof(*array). In case of person[5] it will be 0 + sizeof(Array) * 5 = 0 + 5 * 64 = 320 bytes. Gotcha! We're still in the chunk's boundary, I mean we access the existing physical page. A pagefault would happen if we tried to access an unexisting virtual page, but in our case it exists at address 320 (from 0 to 4096 as we assumed). It's dangerous to access unallocated space as it can lead to lots of unknown consequences, but we can actually do it!
That's why you don't get any Access Violation at ****. But it's actually MUCH WORSE. Because if you for example try accessing the zero pointer you will get a pagefault and your app will just crash, therefore you WILL know the cause of the problem with a help of the debugger or something else. But if you overrun the buffer and you don't get any error, you will go crazy while looking for the problem's cause. Because it's REALLY HARD to find this kind of errors. And you can even be NOT AWARE of it. So NEVER OVERRUN THE BUFFER ALLOCATED IN THE HEAP. Actually Microsoft's C Runtime has a special "debug" version of malloc that can find these errors at runtime, but you need to compile application with "DEBUG" configuration. Also, there are some special thing like Valgrind, but I have a little experience in these stuff.
Well, I have written alot, sorry for my english, I'm still learning it. Hope it will help you.
First, never forget to free your memory.
// NEVER FORGET TO FREE YOUR MEMORY
free(people);
As for this part
//accessing memory on the heap I do not owe?
people[5].someone=Rob;
//prints "Robert" why is this okay? Am I potentially overwriting memory?
printf("%s\n",people[5].someone.name);
you are just being lucky (or unlucky in my opinion, since you don't see the logical mistake you are doing).
This is undefined behaviour, since you have two cells, but you access a 6th one, you go out of bounds.
I have I am doing this problem on SPOJ. http://www.spoj.com/problems/NHAY/. It requires taking input dynamically. In the code below even though I am not allocating memory to char *needle using malloc() - I am taking l = 1 - yet I am able to take input of any length and also it is printing out the entire string. Sometimes it gives runtime error. Why is this when I have not allocated enough memory for the string?
#include<stdio.h>
#include<malloc.h>
#include<ctype.h>
#include<stdlib.h>
int main()
{
long long l;
int i;
char *needle;
while(1){
scanf("%lld",&l);
needle =(char *)malloc(sizeof(char)*l);
scanf("%s",needle);
i=0;
while(needle[i]!='\0'){
printf("%c",needle[i]);
i++;
}
free(needle);
}
}
I also read on stackoverflow that a string is a char * so I should declare char *needle. How can I use this fact in the code? If I take l = 1 then no matter, what the length of the input string it should contain characters only up to the memory allocated for the char * pointer, i.e 1 byte. How can I do that?
Your code is producing an intentional buffer overflow by having sscanf copying a string bigger than the allocated space into the memory allocated by malloc. This "works" because in most cases, the buffer that is allocated is somewhere in the middle of a page so copying more data into the buffer "only" overwrites adjacent data. C (and C++) don't do any array bounds checking on plain C array and thus the error is uncaught.
In the cases where you end up with a runtime error, you most likely copied part of the string into unmapped and unallocated memory, which trigger an access violation.
Memory is usually allocated from the underlying OS in pages of a fixed size. For example, on x86 systems, pages are usually 4k in size. If the mapped address you are writing to is far enough away from the beginning and end of the page, the whole string will fit within the boundaries of the page. If you get close enough to the upper boundary, the code may attempt to write past the boundary, triggering the access violation.
[With some assumptions about the underlying system]
The reason it works for now is that the C library manages pools of memory allocated in pages from the operating system. The operating system only returns pages. The C library returns arbitrary amounts of data.
For your first allocation, you are getting read/write pages allocated by the operating system and managed by the pool. You are going off the edge of the data allocated by the library but are within the page returned by the operating system.
DOing what you are doing will corrupt the structure of the pool and a more extensive program using dynamic memory will eventually crash.
C language do not have default bound check. At the best it will crash while debugging, sometimes it will work as expected. Otherwise you will end up overwriting other memory blocks.
It will not always work. It is Undefined Behaviour.
I basically have this piece of code.
char (* text)[1][80];
text = calloc(2821522,80);
The way I calculated it, that calloc should have allocated 215.265045 megabytes of RAM, however, the program in the end exceeded that number and allocated nearly 700mb of ram.
So it appears I cannot properly know how much memory that function will allocate.
How does one calculate that propery?
calloc (and malloc for that matter) is free to allocate as much space as it needs to satisfy the request.
So, no, you cannot tell in advance how much it will actually give you, you can only assume that it's given you the amount you asked for.
Having said that, 700M seems a little excessive so I'd be investigating whether the calloc was solely responsible for that by, for example, a program that only does the calloc and nothing more.
You might also want to investigate how you're measuring that memory usage.
For example, the following program:
#include <stdio.h>
#include <stdlib.h>
#include <malloc.h>
int main (void) {
char (* text)[1][80];
struct mallinfo mi;
mi = mallinfo(); printf ("%d\n", mi.uordblks);
text = calloc(2821522,80);
mi = mallinfo(); printf ("%d\n", mi.uordblks);
return 0;
}
outputs, on my system:
66144
225903256
meaning that the calloc has allocated 225,837,112 bytes which is only a smidgeon (115,352 bytes or 0.05%) above the requested 225,721,760.
Well it depends on the underlying implementation of malloc/calloc.
It generally works like this - there's this thing called the heap pointer which points to the top of the heap - the area from where dynamic memory gets allocated. When memory is first allocated, malloc internally requests x amount of memory from the kernel - i.e. the heap pointer increments by a certain amount to make that space available. That x may or may not be equal to the size of the memory block you requested (it might be larger to account for future mallocs). If it isn't, then you're given at least the amount of memory you requested(sometimes you're given more memory because of alignment issues). The rest is made part of an internal free list maintained by malloc. To sum it up malloc has some underlying data structures and a lot depends on how they are implemented.
My guess is that the x amount of memory was larger (for whatever reason) than you requested and hence malloc/calloc was holding on to the rest in its free list. Try allocating some more memory and see if the footprint increases.
Well, I can't understand when and why it is needed to allocate memory using malloc.
Here is my code:
#include <stdlib.h>
int main(int argc, const char *argv[]) {
typedef struct {
char *name;
char *sex;
int age;
} student;
// Now I can do two things
student p;
// Or
student *ptr = (student *)malloc(sizeof(student));
return 0;
}
Why is it needed to allocate memory when I can just use student p;?
malloc is used for dynamic memory allocation. As said, it is dynamic allocation which means you allocate the memory at run time. For example, when you don't know the amount of memory during compile time.
One example should clear this. Say you know there will be maximum 20 students. So you can create an array with static 20 elements. Your array will be able to hold maximum 20 students. But what if you don't know the number of students? Say the first input is the number of students. It could be 10, 20, 50 or whatever else. Now you will take input n = the number of students at run time and allocate that much memory dynamically using malloc.
This is just one example. There are many situations like this where dynamic allocation is needed.
Have a look at the man page malloc(3).
You use malloc when you need to allocate objects that must exist beyond the lifetime of execution of the current block (where a copy-on-return would be expensive as well), or if you need to allocate memory greater than the size of that stack (i.e., a 3 MB local stack array is a bad idea).
Before C99 introduced VLAs, you also needed it to perform allocation of a dynamically-sized array. However, it is needed for creation of dynamic data structures like trees, lists, and queues, which are used by many systems. There are probably many more reasons; these are just a few.
Expanding the structure of the example a little, consider this:
#include <stdio.h>
int main(int argc, const char *argv[]) {
typedef struct {
char *name;
char *sex;
char *insurance;
int age;
int yearInSchool;
float tuitionDue;
} student;
// Now I can do two things
student p;
// Or
student *p = malloc(sizeof *p);
}
C is a language that implicitly passes by value, rather than by reference. In this example, if we passed 'p' to a function to do some work on it, we would be creating a copy of the entire structure. This uses additional memory (the total of how much space that particular structure would require), is slower, and potentially does not scale well (more on this in a minute). However, by passing *p, we don't pass the entire structure. We only are passing an address in memory that refers to this structure. The amount of data passed is smaller (size of a pointer), and therefore the operation is faster.
Now, knowing this, imagine a program (like a student information system) which will have to create and manage a set of records in the thousands, or even tens of thousands. If you pass the whole structure by value, it will take longer to operate on a set of data, than it would just passing a pointer to each record.
Let's try and tackle this question considering different aspects.
Size
malloc allows you to allocate much larger memory spaces than the one allocated simply using student p; or int x[n];. The reason being malloc allocates the space on heap while the other allocates it on the stack.
The C programming language manages memory statically, automatically, or dynamically. Static-duration variables are allocated in main memory, usually along with the executable code of the program, and persist for the lifetime of the program; automatic-duration variables are allocated on the stack and come and go as functions are called and return. For static-duration and automatic-duration variables, the size of the allocation must be compile-time constant (except for the case of variable-length automatic arrays[5]). If the required size is not known until run-time (for example, if data of arbitrary size is being read from the user or from a disk file), then using fixed-size data objects is inadequate. (from Wikipedia)
Scope
Normally, the declared variables would get deleted/freed-up after the block in which it is declared (they are declared on the stack). On the other hand, variables with memory allocated using malloc remain till the time they are manually freed up.
This also means that it is not possible for you to create a variable/array/structure in a function and return its address (as the memory that it is pointing to, might get freed up). The compiler also tries to warn you about this by giving the warning:
Warning - address of stack memory associated with local variable 'matches' returned
For more details, read this.
Changing the Size (realloc)
As you may have guessed, it is not possible by the normal way.
Error detection
In case memory cannot be allocated: the normal way might cause your program to terminate while malloc will return a NULL which can easily be caught and handled within your program.
Making a change to string content in future
If you create store a string like char *some_memory = "Hello World"; you cannot do some_memory[0] = 'h'; as it is stored as string constant and the memory it is stored in, is read-only. If you use malloc instead, you can change the contents later on.
For more information, check this answer.
For more details related to variable-sized arrays, have a look at this.
malloc = Memory ALLOCation.
If you been through other programming languages, you might have used the new keyword.
Malloc does exactly the same thing in C. It takes a parameter, what size of memory needs to be allocated and it returns a pointer variable that points to the first memory block of the entire memory block, that you have created in the memory. Example -
int *p = malloc(sizeof(*p)*10);
Now, *p will point to the first block of the consecutive 10 integer blocks reserved in memory.
You can traverse through each block using the ++ and -- operator.
In this example, it seems quite useless indeed.
But now imagine that you are using sockets or file I/O and must read packets from variable length which you can only determine while running. Or when using sockets and each client connection need some storage on the server. You could make a static array, but this gives you a client limit which will be determined while compiling.