Here's the code snippet:
void main() {
int i,*s;
for(i=1;i<=4;i++) {
s=malloc(sizeof(int));
printf("%lu \n",(unsigned long)s);
}
}
The size of int on my comp is 2 bytes, so shouldn't the printf command print address incremented by 16 bits, instead it prints the address as:
2215224120
2215224128
2215224136...
Why is this so?
How memory managed is entirely up to your operating system. It could allocate memory from all over the place, you can absolutely make no assumptions as to where the memory will be.
Most memory allocators also have some overhead, so even a simple 2-byte allocation might take up 8 bytes or more. Besides, addresses might need to be aligned for several reasons (like performance, and because some CPUs even crash when reading from unaligned addresses).
Bottom line - take the return value from malloc as it is, don't make any guesses or assumptions.
Its called alignment. Most CPUs have to align memory on some boundary, and its commonly 4 or 8. If you mis-align an address you will get a segfault or bus error.
malloc() does not provide any such guarantees. It just allocates some memory according to its own memory management decisions and returns you a pointer to that. In fact, many implementations use extra memory right before the pointer returned for memory management metadata.
malloc() gives you an abstraction on the underlying hardware, OS, drivers, etc. The memory allocation pattern may differ from machine to machine due to various parameters.
But the following are few things that always stays right about malloc()
The malloc() function allocates size bytes and returns a pointer to the allocated memory.
The memory is not initialized.
If size is 0,then malloc() returns either NULL, or a unique pointer value that can later be successfully passed to free().
The malloc() returns a pointer to the allocated memory that is suitably aligned for any kind of variable. On error, it returns NULL.
NULL may also be returned by a successful call to malloc() with a size of zero
On a side note, you can use %p format specifier for printing the pointers
I modified the program as follows
#include <stdlib.h>
int main(void) {
int i,*s;
printf("sizeof(int) = %zu \n", sizeof(int));
for(i=1;i<=4;i++) {
if ((s=malloc(sizeof(int))) == NULL) {
printf("unable to allocate memory \n");
return -1;
}
printf("%p \n",s);
}
return 0;
}
The output is as follows:
$ ./a.out
sizeof(int) = 4
0x9d5a008
0x9d5a018
0x9d5a028
0x9d5a038
$
You have no guarantees whatsoever about the pattern of addresses malloc returns to you.
Related
I was reading in a book:
The virtual address space of a process on a 32 bit machine is 2^32 i.e. 4Gb of space. And every address seen in the program is a virtual address. The 4GB of space is further goes through user/kernel split 3-1GB.
To better understand this, I did malloc() of 5Gb space and tried to print the all addresses. If I print the addresses, How is the application going to print whole 5Gb address when It has only 3GB of virtual address space? Am I missing something here?
malloc() takes size_t as an argument. On 32 bit system it's an alias to some unsigned 32 bit integer type. This means that you just cannot pass any value bigger than 2^32-1 as an argument for malloc() making it impossible request allocation of more than 4GB of memory using this function.
The same is true for all other functions that can be used to allocate memory. Ultimately they all end up as either brk() or mmap syscall. The length argument of mmap() is also of type ssize_t an in case of brk() you have to provide a pointer for the new end of your allocated space. The pointer is again 32 bit.
So there is absolutely no way to tell kernel you would like to get more than 4GB of memory allocated with one call) And it's not an accident - this just wouldn't make any sense anyway.
Now it's true that you could do several calls to malloc or other function that allocates memory, requesting more than 4GB in total. If you try this, the subsequent call (that would cause extending allocated memory to more than 3GB) will fail as there is just no address space available.
So I guess that you either didn't check the malloc return value or you did try to run code like this (or something similar):
int main() {
assert(malloc(5*1<<30));
}
and assumed that you succeeded in allocating 5GB without verifying that your argument overflowed and instead of requesting 5368709120 bytes, you requested 1073741824. One example to verify this on Linux is to use:
$ ltrace ./a.out
__libc_start_main(0x804844c, 1, 0xbfbcea74, 0x80484a0, 0x8048490 <unfinished ...>
malloc(1073741824) = 0x77746008
$
There's already a good answer. Just in case, the size of your virtual address space is easily verifiable like this:
#include <stdlib.h>
#include <stdio.h>
int main()
{
size_t size = (size_t)-1L;
void *foo;
printf("trying to allocate %zu bytes\n", size);
if (!(foo = malloc(size)))
{
perror("malloc()");
}
else
{
free(foo);
}
}
> gcc -m32 -omalloc malloc.c && ./malloc
trying to allocate 4294967295 bytes
malloc(): Cannot allocate memory
This must fail because parts of your address space are already occupied: by the mapped part of the kernel, by mapped shared libraries and by your program, of course.
You cannot do this because there is no function for you to alloc 5GB memory.
I wish to free blocks of memory which I don't have pointers to. In my program, I call malloc sequentially, hoping that the memory created by malloc(1), malloc(4), malloc(5) is continuous. Then I free these memory when I only have the pointer to malloc(5). But I can't think of how this can be done; I cannot simply create a pointer that reference to the address of ptr[-5] and then free 5 bytes of memory? How can this be done?
#include <string.h>
#include <stdio.h>
#include <stdlib.h>
int main(){
malloc(1);
malloc(4);
char* ptr = malloc(5);
free(ptr);
}
You cannot do what you want to do. You should not even try to do what you want to do.
Even if you work out exactly what malloc() is doing, your program would then be relying on undefined behavior. The behavior could change when a new version of the C library arrives, and your program would almost certainly fail if you compiled it using a different toolchain (switch from GNU C to Microsoft C or whatever).
Any time you allocate memory, you need to keep track of the pointer. If your program doesn't even know about the memory, there is no way to free it.
Keep track of your memory allocations. If you are designing data structures to be dynamically allocated, your design should include features to track them, such as keeping a list of addresses in a linked list or something.
If this seems like a lot of work, maybe consider using a managed language like C# or Java or Python or whatever.
free(void*)
[deallocate] A block of memory previously allocated by a call to malloc, calloc or realloc is deallocated, making it available again for further allocations.
If ptr does not point to a block of memory allocated with the above functions, it causes undefined behavior.
- http://www.cplusplus.com/reference/cstdlib/free/
There is no way.
But I can't think of how this can be done
That's because it is not possible. The blocks that you get back from malloc can come in truly arbitrary order. The only way to free a dynamically allocated block of memory is to keep a pointer to it accessible to your program. Anything else is undefined behavior.
Note: Implementations of malloc perform "bookkeeping" to figure out what kind of block you are releasing. While it is not impossible to hack into their implementation, there is no way of doing it in a standard-compliant, portable way.
You cannot create a [-5]...thing for a variety of reasons but the from a practical standpoint you have to remember that memory allocated with malloc() is coming off of the heap and not the stack so to "count" to it from somewhere else is difficult (since multiple calls to malloc are not guaranteed to be sequential).
What happens when a pointer loses its association to memory (or goes out of scope) without being freed is called a memory leak and without exhaustive techniques not readily available in C (Java's mark/sweep garbage collection for example, or mallocing the entire memory and scanning it or something) it is not possible to reclaim this memory.
So you cannot free memory in C when a pointer is not known.
First of all - as it seems you do not understand how malloc works - passing continuous numbers to malloc, won't make it allocate an array. malloc is defined as follows:
void* malloc (size_t size);
While an integer can be converted to size_t, it's still the number of bytes allocated, not the element number. If you want to allocate an array, do it as follows:
int* myDynamicArray = malloc(sizeof(int)*numberOfElements);
Then, you can access the elements by doing:
int i;
for(i=0;i<numberOfElements;i++)
printf("%d",myDynamicArray[i]);
Then, like others pointed out - you can deallocate the memory by calling the free function. free is defined as follows:
void free (void* ptr);
And you simply call it by doing:
free(myDynamicArray);
This is by no means an endorsement of what you have done, but it is possible assuming you know that the blocks were allocated continuously.
For example:
int main(){
char* ptr1=malloc(1);
char* ptr2=malloc(4);
char* ptr3=malloc(5);
// Verify that the memory is in fact continuous.
assert(ptr3==(ptr2+4));
assert(ptr3==(ptr1+5));
free(ptr3); // Frees 5 bytes at ptr3
free(ptr3-4); // Frees 4 bytes at ptr2
free(ptr3-5); // Frees 1 byte at ptr1
}
So, you if you have a pointer and know for a fact that you allocated a set of continuous bytes before it, you can simply offset the pointer with pointer arithmetic. It is highly dangerous and not recommended, but it is possible.
Edit:
I ran a test program and on my architecture, it allocated in 32 byte chunks, so ptr1+32==ptr2, and ptr2+32=ptr3. It did this for any chunks less than or equal to 24 bytes. So if I allocated 24 or less, then each ptr would be 32 bytes greater than the previous. If I allocated 25 or more, then it allocated an additional 16 bytes, making the total 48.
So, in my architecture, you'd need to be much more creative in how you generate your pointers using pointer arithmetic since it will not work as expected.
Here is an example program that works for all sizes of ptr1, ptr2, and ptr3 on my architecture.
#define ROUNDUP(number, multiple) (((number + multiple -1)/multiple)*multiple)
#define OFFSET(size) ((size < 24) ? 32 : ROUNDUP(size+8,16))
int main(int argc, char* argv[]){
char* ptr1, *ptr2, *ptr3;
int s1=atoi(argv[1]);
int s2=atoi(argv[2]);
int s3=atoi(argv[3]);
ptr1=(char*)malloc(s1);
ptr2=(char*)malloc(s2);
ptr3=(char*)malloc(s3);
fprintf(stdout, "%p %p %p\n", ptr1, ptr2, ptr3);
assert(ptr3==(ptr2+OFFSET(s2)));
assert(ptr2==(ptr1+OFFSET(s1)));
// Try to construct ptr2 from ptr3.
free(ptr3);
free(ptr3-OFFSET(s2));
free(ptr3-OFFSET(s2)-OFFSET(s1));
}
I've got the following code:
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
int main(int argc, char ** argv)
{
//just checking to see where the stack
printf("The stack is around %p\n", &argc); is, making sure arr isn't in it
char ** arr = malloc(8*sizeof(char*));
printf("arr is size %li, at %p\n", sizeof(arr), arr);
arr = realloc(arr, 100); //I picked a weird number to show it isn't doing anything. I've picked different numbers (like 200, 2*sizeof(char*)*sizeof(arr), and 16)
printf("arr is size %li, at %p\n", sizeof(arr), arr);
}
That's the entirety of the file (it's a unit test; I was noticing it elsewhere)
The output of the above is as follows:
The stack is around 0x7fff5b94d12c
arr is size 8, at 0x120f010
arr is size 8, at 0x120f010
Perhaps I'm misunderstanding what realloc should do. I'm expecting the following output.
The stack is around 0x7fff5b94d12c
arr is size 8, at 0x120f010
arr is size <size>, at <somewhere>
where <size> is... something odd like 12... at least not 8 and <somewhere> is most likely 0x120f010 but possibly anywhere reasonable.
Are my expectations wrong or am I using realloc incorrectly?
The output of your program is correct, because
Neither malloc nor realloc have anything to do with the automatic storage (i.e. "the stack"). They allocate memory from the dynamic storage area (i.e. "the heap"). One should not expect the position of the top of the stack to change in response to calls to malloc, realloc, or for that matter, any other function.
The value of sizeof(arr) does not depend on what you have allocated to it. It is computed at compile time, and it is always equal to the size of a pointer. On your system, pointers use 8 bytes.
malloc often gives you more memory that you ask, and stores the actual value in a special location that realloc can access at a later time. If you realloc down, or realloc within the bounds, the value returned by realloc does not change. That's the reason why it may perform better than simply calling malloc and memcpy.
sizeof arr
That's the same as
sizeof char**
The size of a pointer isn't going to change, and taking the size of a pointer is not going to tell you how much memory it refers to. Pointers are not arrays, and sizeof is evaluated at compile time.
As for the address bit, realloc doesn't guarantee that the memory block was moved. It could simply expand it successfully and return the same address.
Also, I realize this is just example code, but be aware that, if realloc failed, you leaked what arr originally pointed to.
It's not uncommon, and in fact a bit expected, that a malloc call followed directly by realloc would not change the address of the pointer. In many cases the allocator can just extend the amount of memory reserved at the address and not have to move the pointer. This is what is happening here.
This isn't something you should ever depend on though. It's just a quirk of the implementation
If your assumption is that realloc() has to return a different pointer, then your assumption is wrong.
Typically if you're reducing the size of the allocated memory or leaving it "as is", then realloc() can return the same pointer and avoid copying data, etc.
Sometimes if you're increasing the size of the allocated memory realloc() can check if there's free space above the existing space and still return the same pointer (and avoid copying data).
Mostly, it's only when there is no free space above the allocated memory that realloc() must copy the data somewhere else and return a different pointer.
I have a few related questions about managing aligned memory blocks. Cross-platform answers would be ideal. However, as I'm pretty sure a cross-platform solution does not exist, I'm mainly interested in Windows and Linux and to a (much) lesser extent Mac OS and FreeBSD.
What's the best way of getting a chunk of memory aligned on 16-byte boundaries? (I'm aware of the trivial method of using malloc(), allocating a little extra space and then bumping the pointer up to a properly aligned value. I'm hoping for something a little less kludge-y, though. Also, see below for additional issues.)
If I use plain old malloc(), allocate extra space, and then move the pointer up to where it would be correctly aligned, is it necessary to keep the pointer to the beginning of the block around for freeing? (Calling free() on pointers to the middle of the block seems to work in practice on Windows, but I'm wondering what the standard says and, even if the standard says you can't, whether it works in practice on all major OS's. I don't care about obscure DS9K-like OS's.)
This is the hard/interesting part. What's the best way to reallocate a memory block while preserving alignment? Ideally this would be something more intelligent than calling malloc(), copying, and then calling free() on the old block. I'd like to do it in place where possible.
If your implementation has a standard data type that needs 16-byte alignment (long long for example), malloc already guarantees that your returned blocks will be aligned correctly. Section 7.20.3 of C99 states The pointer returned if the allocation succeeds is suitably aligned so that it may be assigned to a pointer to any type of object.
You have to pass back the exact same address into free as you were given by malloc. No exceptions. So yes, you need to keep the original copy.
See (1) above if you already have a 16-byte-alignment-required type.
Beyond that, you may well find that your malloc implementation gives you 16-byte-aligned addresses anyway for efficiency although it's not guaranteed by the standard. If you require it, you can always implement your own allocator.
Myself, I'd implement a malloc16 layer on top of malloc that would use the following structure:
some padding for alignment (0-15 bytes)
size of padding (1 byte)
16-byte-aligned area
Then have your malloc16() function call malloc to get a block 16 bytes larger than requested, figure out where the aligned area should be, put the padding length just before that and return the address of the aligned area.
For free16, you would simply look at the byte before the address given to get the padding length, work out the actual address of the malloc'ed block from that, and pass that to free.
This is untested but should be a good start:
void *malloc16 (size_t s) {
unsigned char *p;
unsigned char *porig = malloc (s + 0x10); // allocate extra
if (porig == NULL) return NULL; // catch out of memory
p = (porig + 16) & (~0xf); // insert padding
*(p-1) = p - porig; // store padding size
return p;
}
void free16(void *p) {
unsigned char *porig = p; // work out original
porig = porig - *(porig-1); // by subtracting padding
free (porig); // then free that
}
The magic line in the malloc16 is p = (porig + 16) & (~0xf); which adds 16 to the address then sets the lower 4 bits to 0, in effect bringing it back to the next lowest alignment point (the +16 guarantees it is past the actual start of the maloc'ed block).
Now, I don't claim that the code above is anything but kludgey. You would have to test it in the platforms of interest to see if it's workable. Its main advantage is that it abstracts away the ugly bit so that you never have to worry about it.
I'm not aware of any way of requesting malloc return memory with stricter alignment than usual. As for "usual" on Linux, from man posix_memalign (which you can use instead of malloc() to get more strictly aligned memory if you like):
GNU libc malloc() always returns 8-byte aligned memory addresses, so
these routines are only needed if you require larger alignment values.
You must free() memory using the same pointer returned by malloc(), posix_memalign() or realloc().
Use realloc() as usual, including sufficient extra space so if a new address is returned that isn't already aligned you can memmove() it slightly to align it. Nasty, but best I can think of.
You could write your own slab allocator to handle your objects, it could allocate pages at a time using mmap, maintain a cache of recently-freed addresses for fast allocations, handle all your alignment for you, and give you the flexibility to move/grow objects exactly as you need. malloc is quite good for general-purpose allocations, but if you know your data layout and allocation needs, you can design a system to hit those requirements exactly.
The trickiest requirement is obviously the third one, since any malloc() / realloc() based solution is hostage to realloc() moving the block to a different alignment.
On Linux, you could use anonymous mappings created with mmap() instead of malloc(). Addresses returned by mmap() are by necessity page-aligned, and the mapping can be extended with mremap().
Starting a C11, you have void *aligned_alloc( size_t alignment, size_t size ); primitives, where the parameters are:
alignment - specifies the alignment. Must be a valid alignment supported by the implementation.
size - number of bytes to allocate. An integral multiple of alignment
Return value
On success, returns the pointer to the beginning of newly allocated memory. The returned pointer must be deallocated with free() or realloc().
On failure, returns a null pointer.
Example:
#include <stdio.h>
#include <stdlib.h>
int main(void)
{
int *p1 = malloc(10*sizeof *p1);
printf("default-aligned addr: %p\n", (void*)p1);
free(p1);
int *p2 = aligned_alloc(1024, 1024*sizeof *p2);
printf("1024-byte aligned addr: %p\n", (void*)p2);
free(p2);
}
Possible output:
default-aligned addr: 0x1e40c20
1024-byte aligned addr: 0x1e41000
Experiment on your system. On many systems (especially 64-bit ones), you get 16-byte aligned memory out of malloc() anyway. If not, you will have to allocate the extra space and move the pointer (by at most 8 bytes on almost every machine).
For example, 64-bit Linux on x86/64 has a 16-byte long double, which is 16-byte aligned - so all memory allocations are 16-byte aligned anyway. However, with a 32-bit program, sizeof(long double) is 8 and memory allocations are only 8-byte aligned.
Yes - you can only free() the pointer returned by malloc(). Anything else is a recipe for disaster.
If your system does 16-byte aligned allocations, there isn't a problem. If it doesn't, then you'll need your own reallocator, which does a 16-byte aligned allocation and then copies the data - or that uses the system realloc() and adjusts the realigned data when necessary.
Double check the manual page for your malloc(); there may be options and mechanisms to tweak it so it behaves as you want.
On MacOS X, there is posix_memalign() and valloc() (which gives a page-aligned allocation), and there is a whole series of 'zoned malloc' functions identified by man malloc_zoned_malloc and the header is <malloc/malloc.h>.
You might be able to jimmy (in Microsoft VC++ and maybe other compilers):
#pragma pack(16)
such that malloc( ) is forced to return a 16-byte-aligned pointer. Something along the lines of:
ptr_16byte = malloc( 10 * sizeof( my_16byte_aligned_struct ));
If it worked at all for malloc( ), I'd think it would work for realloc( ) just as well.
Just a thought.
-- pete
I have a question regarding memory allocation order.
In the following code I allocate in a loop 4 strings.
But when I print the addresses they don't seem to be allocated one after the other... Am I doing something wrong or is it some sort of defense mechanism implemented by the OS to prevent possible buffer overflows? (I use Windows Vista).
Thank you.
char **stringArr;
int size=4, i;
stringArr=(char**)malloc(size*sizeof(char*));
for (i=0; i<size; i++)
stringArr[i]=(char*)malloc(10*sizeof(char));
strcpy(stringArr[0], "abcdefgh");
strcpy(stringArr[1], "good-luck");
strcpy(stringArr[2], "mully");
strcpy(stringArr[3], "stam");
for (i=0; i<size; i++) {
printf("%s\n", stringArr[i]);
printf("%d %u\n\n", &(stringArr[i]), stringArr[i]);
}
Output:
abcdefgh
9650064 9650128
good-luck
9650068 9638624
mully
9650072 9638680
stam
9650076 9638736
Typically when you request memory through malloc(), the C runtime library will round the size of your request up to some minimum allocation size. This makes sure that:
the runtime library has room for its bookkeeping information
it's more efficient for the runtime library to manage allocated blocks that are all multiples of some size (such as 16 bytes)
However, these are implementation details and you can't really rely on any particular behaviour of malloc().
But when I print the addresses they don't seem to be allocated one after the other...
So?
Am I doing something wrong or is it some sort of defense mechanism implemented by the OS to prevent possible buffer overflows?
Probably "neither".
Just out of interest, what addresses do you get?
You shouldn't depend on any particular ordering or spacing of values returned by malloc. It behaves in mysterious and unpredictable ways.
Typically it is reasonable to expect that a series of chronological allocations will result in a memory addresses that are somehow related, but as others have pointed out, it is certainly not a requirement of the heap manager. In this particular case, though, it is possible that you are seeing results of the low fragmentation heap. Windows keeps lists of small chunks of memory that can quickly satisfy a request. These could be in any order.
You can't depend on malloc to give you contiguous addresses. It's entirely up to the implementation and presumably the current state of the heap; some implementations may, many won't.
If you need the addresses to be contiguous, allocate one large block of memory and set up your pointers to point to different areas within it.
As others have mentioned, there is no standard to specify in which order the memory blocks allocated by malloc() should be located in the memory. For example, freed blocks can be scattered all around the heap and they may be re-used in any order.
But even if the blocks happen to be one after each other, they most likely do not form a contiguous block. In order to reduce fragmentation, the heap manager only allocates blocks of specific size, for example power of two (64, 128, 256, 512 etc. bytes). So, if you reserve 10 bytes for a string, there may be perhaps 22 or 54 un-used bytes after that.
The memory overhead is another reason why it is not good idea to use dynamic memory allocation unless really necessary. It is much easier and safer just to use a static array.
Since you are interesting in knowing the addresses returned by malloc(), you should make sure you are printing them properly. By "properly", I mean that you should use the right format specifier for printf() to print addresses. Why are you using "%u" for one and "%d" for another?
You should use "%p" for printing pointers. This is also one of the rare cases where you need a cast in C: because printf() is a variadic function, the compiler can't tell that the pointers you're passing as an argument to it need to be of the type void * or not.
Also, you shouldn't cast the return value of malloc().
Having fixed the above, the program is:
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
int main(void)
{
char **stringArr;
int size=4, i;
stringArr = malloc(size * sizeof *stringArr);
for (i=0; i < size; i++)
stringArr[i] = malloc(10 * sizeof *stringArr[i]);
strcpy(stringArr[0], "abcdefgh");
strcpy(stringArr[1], "good-luck");
strcpy(stringArr[2], "mully");
strcpy(stringArr[3], "stam");
for (i=0; i<size; i++) {
printf("%s\n", stringArr[i]);
printf("%p %p\n", (void *)(&stringArr[i]), (void *)(stringArr[i]));
}
return 0;
}
and I get the following output when I run it:
abcdefgh
0x100100080 0x1001000a0
good-luck
0x100100088 0x1001000b0
mully
0x100100090 0x1001000c0
stam
0x100100098 0x1001000d0
On my computer, char ** pointers are 8 bytes long, so &stringArr[i+1] is at 8 bytes greater than &stringArr[i]. This is guaranteed by the standard: If you malloc() some space, that space is contiguous. You allocated space for 4 pointers, and the addresses of those four pointers are next to each other. You can see that more clearly by doing:
printf("%d\n", (int)(&stringArr[1] - &stringArr[0]));
This should print 1.
About the subsequent malloc()s, since each stringArr[i] is obtained from a separate malloc(), the implementation is free to assign any suitable addresses to them. On my implementation, with that particular run, the addresses are all 0x10 bytes apart.
For your implementation, it seems like char ** pointers are 4 bytes long.
About your individual string's addresses, it does look like malloc() is doing some sort of randomization (which it is allowed to do).