I'm learning Linux C programming recently,and there is a question have puzzled me long time.The question is that when we use malloc to allocate some memory,we can use the addresses that over the size we required,but when we access a large address than we required,the system may kill our process.just like the following codes:
int *p = malloc(10*sizeof(int));
*(p + 10) = 1;
when we use this clause,the system may not kill our process,but when we use:
*(p +10000) = 1;
the system may kill our process.
So why does the system do it in this way?
Actually, you can access memory you didn't allocate.
It's not C that (sometimes) prevents you from doing so (it's designed after the philosophy that the user is always right), it's the operating system using special MMU (memory management unit) hardware.
This hardware cannot (for performance and cost reasons) secure any arbitrary address, but only ranges of memory (pages). Thus illegal (from a programmer's standpoint) accesses are sometimes possible (if they are on the same memory page where you have legally allocated memory), other illegal accesses (outside pages with legal addresses) are prevented by the MMU that issues a page fault (segmentation violation).
This obviously doesn't mean you are allowed to access unallocated memory, it just explains why you sometimes get by with it.
Of course all this is only true for platforms that actually have a MMU. There are still a lot around that haven't, so better learn your lessons ;).
Obviously, you can only access memory you allocated upfront.
So what you're doing basically is:
int *p = malloc(10 * sizeof(int));
Here, you allocate memory for 10 ints.
*(p + 10) = 1;
This can be rewritten as (by removing pointer arithmetic):
p[10] = 1;
Now, you can clearly see that there's no memory for the 10th item. So in theory, your code should crash here already.
However, often, the OS decides to allocate a little more memory or you've allocated a block of memory next to p (in which case the OS won't intervene, even though your code doesn't work the way you want it to).
Let me give you an example:
char foo[4] = "foo";
char bar[4] = "bar";
Now (in theory, obviously it depends on what your compiler does) if you access the 5th char of foo, it actually maps to the first char of bar:
foo[5] = "c";
printf("%s %s\n", foo, bar); /* == foo car */
As I said, this depends on your compiler and I only give you this example because it might help you understand how the OS allocates memory.
One more thing:
The NULL pointer ((void *)0) is "protected" and the OS guarantees that it fails when you try to access the contents of that address.
if you dynamically allocate memory, you are allocating it on the heap. You do this by assigning a pointer the beginning of your memory block. So you can add an offset to this pointer and access a specific memory location. If this offset is large enough your pointer could end up pointing outside of the heap, or outside of your data section, or to a protected memory location. This will cause a segmentation fault - you may be accessing a memory location which is used for something else.
Your question is unclear, but I think you're asking why you "can" access an element outside of what you allocated. The answer is that you really can't. There's no telling what you'll be stepping on when you do that -- perhaps another variable, maybe even your code. For very large offsets, you'll definitely end up outside of the system memory that you "own", and will have your process killed.
It appears that you have no idea what you're doing with malloc(). Go back and read about it again.
Related
Take the following code :
int *p = malloc(2 * sizeof *p);
p[0] = 10; //Using the two spaces I
p[1] = 20; //allocated with malloc before.
p[2] = 30; //Using another space that I didn't allocate for.
printf("%d", *(p+1)); //Correctly prints 20
printf("%d", *(p+2)); //Also, correctly prints 30
//although I didn't allocate space for it
With the line malloc(2 * sizeof *p) I am allocating space for two integers, right ? But if I add an int to the third position, I still gets allocated correctly and retrievable.
So my question is, why do you specify a size when you use malloc ?
Simple logic: If you do not park in a legal parking space, nothing might happen but occasionally your car might get towed and you might get stuck with a huge fine. And, sometimes, as you try to find your way to the pound where your car was towed, you might get run over by a truck.
malloc gives you as many legal parking spots as you asked. You can try to park elsewhere, it might seem to work, but sometimes it won't.
For questions such as this, the Memory Allocation section of the C FAQ is a useful reference to consult. See 7.3b.
On a related (humorous) note, see also a list of bloopers by ART.
C kindly let you shoot yourself in the head. You have just used random memory on the heap. With unforeseeable consequences.
Disclaimer: My last real C programing was done some 15 years ago.
Let me give you an analogy to why this "works".
Let's assume you need to draw a drawing, so you retrieve a piece of paper, lay it flat on your table, and start drawing.
Unfortunately, the paper isn't big enough, but you, not caring, or not noticing, just continue to draw your drawing.
When done, you take a step back, and look at your drawing, and it looks good, exactly as you meant it to be, and exactly the way you drew it.
Until someone comes along and picks up their piece of paper that they left on the table before you got to it.
Now there's a piece of the drawing missing. The piece you drew on that other person's paper.
Additionally, that person now has pieces of your drawing on his paper, probably messing with whatever he wanted to have on the paper instead.
So while your memory usage might appear to work, it only does so because your program finishes. Leave such a bug in a program that runs for a while and I can guarantee you that you get odd results, crashes and whatnot.
C is built like a chainsaw on steroids. There's almost nothing you cannot do. This also means that you need to know what you're doing, otherwise you'll saw right through the tree and into your foot before you know it.
You got (un)lucky. Accessing p[3] is undefined, since you haven't allocated that memory for yourself. Reading/writing off the end of an array is one of the ways that C programs can crash in mysterious ways.
For example, this might change some value in some other variable that was allocated via malloc. That means it might crash later, and it'll be very hard to find the piece of (unrelated) code that overwrote your data.
Worse yet, you might overwrite some other data and might not notice. Imagine this accidentally overwrites the amount of money you owe someone ;-)
In fact, malloc is not allocating enough space for your third integer, but you got "lucky" and your program didn't crash. You can only be sure that malloc has allocated exactly what you asked for, no more. In other words, your program wrote to a piece of memory that was not allocated to it.
So malloc needs to know the size of the memory that you need because it doesn't know what you will end up doing with the memory, how many objects you plan on writing to the memory, etc...
This all goes back to C letting you shoot yourself in the foot. Just because you can do this, doesn't mean you should. The value at p+3 is definitely not guaranteed to be what you put there unless you specifically allocated it using malloc.
Try this:
int main ( int argc, char *argv[] ) {
int *p = malloc(2 * sizeof *p);
int *q = malloc(sizeof *q);
*q = 100;
p[0] = 10; p[1] = 20; p[2] = 30; p[3] = 40;
p[4] = 50; p[5] = 60; p[6] = 70;
printf("%d\n", *q);
return 0;
}
On my machine, it prints:
50
This is because you overwrote the memory allocated for p, and stomped on q.
Note that malloc may not put p and q in contiguous memory because of alignment restrictions.
Memory is represented as an enumerable contiguous line of slots that numbers can be stored in. The malloc function uses some of these slots for its own tracking info, as well as sometimes returning slots larger than what you need, so that when you return them later it isn't stuck with an unusably small chunk of memory. Your third int is either landing on mallocs own data, on empty space leftover in the returned chunk, or in the area of pending memory that malloc has requested from the OS but not otherwise parcelled out to you yet.
Depending on the platform, p[500] would probably "work" too.
When using malloc(), you are accepting a contract with the runtime library in which you agree to ask for as much memory as you are planning to use, and it agrees to give it to you. It is the kind of all-verbal, handshake agreement between friends, that so often gets people in trouble. When you access an address outside the range of your allocation, you are violating your promise.
At that point, you have requested what the standard calls "Undefined Behavior" and the compiler and library are allowed to do anything at all in response. Even appearing to work "correctly" is allowed.
It is very unfortunate that it does so often work correctly, because this mistake can be difficult to write test cases to catch. The best approaches to testing for it involve either replacing malloc() with an implementation that keeps track of block size limits and aggressively tests the heap for its health at every opportunity, or to use a tool like valgrind to watch the behavior of the program from "outside" and discover the misuse of buffer memory. Ideally, such misuse would fail early and fail loudly.
One reason why using elements close to the original allocation often succeeds is that the allocator often gives out blocks that are related to convenient multiples of the alignment guarantee, and that often results in some "spare" bytes at the end of one allocation before the start of the next. However the allocator often store critical information that it needs to manage the heap itself near those bytes, so overstepping the allocation can result in destruction of the data that malloc() itself needs to successfully make a second allocation.
Edit: The OP fixed the side issue with *(p+2) confounded against p[1] so I've edited my answer to drop that point.
You are asking for space for two integers. p[3] assumes that you have space for 4 integers!
===================
You need to tell malloc how much you need because it can't guess how much memory you need.
malloc can do whatever it wants as long as it returns at least the amount of memory you ask for.
It's like asking for a seat in a restaurant. You might be given a bigger table than you need. Or you might be given a seat at a table with other people. Or you might be given a table with one seat. Malloc is free to do anything it wants as long as you get your single seat.
As part of the "contract" for the use of malloc, you are required to never reference memory beyond what you have asked for because you are only guaranteed to get the amount you asked for.
Because malloc() allocates in BYTES. So, if you want to allocate (for example) 2 integers you must specify the size in bytes of 2 integers. The size of an integer can be found by using sizeof(int) and so the size in bytes of 2 integers is 2 * sizeof(int). Put this all together and you get:
int * p = malloc(2 * sizeof(int));
Note: given that the above only allocates space for TWO integers you are being very naughty in assigning a 3rd. You're lucky it doesn't crash. :)
Because malloc is allocating space on the heap which is part of the memory used by your program which is dynamically allocated. The underlying OS then gives your program the requested amount (or not if you end up with some error which implies you always should check return of malloc for error condition ) of virtual memory which it maps to physical memory (ie. the chips) using some clever magic involving complex things like paging we don't want to delve into unless we are writing an OS.
When you use * (p+3), you're addressing out of bounds even with using 2*sizeof(* p), hence you're accessing an invalid memory block, perfect for seg faults.
You specify the size b/c otherwise, the function doesn't know how big of a block out of the heap memory to allocate to your program for that pointer.
The reason for the size given to malloc() is for the memory manager to keep track of how much space has been given out to each process on your system. These tables help the system to know who allocated how much space, and what addresses are free()able.
Second, c allows you to write to any part of ram at any time. Kernel's may prevent you from writing to certain sections, causing protection faults, but there is nothing preventing the programmer from attempting.
Third, in all likelyhood, malloc()ing the first time probably doesn't simply allocate 8 bytes to your process. This is implementation dependent, but it is more likely for the memory manager to allocate a full page for your use just because it is easier to allocate page size chunks....then subsequent malloc()'s would further divide the previously malloc()ed page.
As everyone has said, you're writing to memory that isn't actually allocated, meaning that something could happen to overwrite your data. To demonstrate the problem, you could try something like this:
int *p = malloc(2 * sizeof(int));
p[0] = 10; p[1] = 20; p[2] = 30;
int *q = malloc(2 * sizeof(int));
q[0] = 0; // This may or may not get written to p[2], overwriting your 30.
printf("%d", p[0]); // Correctly prints 10
printf("%d", p[1]); // Correctly prints 20
printf("%d", p[2]); // May print 30, or 0, or possibly something else entirely.
There's no way to guarantee your program will allocate space for q at p[2]. It may in fact choose a completely different location. But for a simple program like this, it seems likely, and if it does allocate q at the location where p[2] would be, it will clearly demonstrate the out-of-range error.
Do :
int *p = malloc(2 * sizeof(*p)); // wrong (if type is something greater than a machine word)
[type] *p = malloc(2 * sizeof([type])); // right.
The following code shows an example that repeatedly allocates memory without first calling free. Instead, it frees **sign after the loop.
#include <stdio.h>
#include <stdlib.h>
float ** fun(int nloc)
{
float **sign;
int i,nt=100,it,k;
sign=(float **)calloc(nloc,sizeof(float *));
for (i=0;i<nloc;i++)
sign[i] = (float *)calloc(nt,sizeof(float *));
for (it=0;it<nt;it++){
for (k=0;k<nloc;k++){
sign[k][it]=it*0.2;
}
}
return sign;
}
int main(int argc, char *argv[])
{
int i,isrc,n=3,nloc=1;
float **sign=NULL;
for (isrc=0;isrc<n;isrc++){
sign = fun(nloc);
}
for (i=0;i<nloc;i++){
free(sign[i]);
}
free(sign);
exit(0);
}
This is a correct code snippet. My question is: why is it legal that we can allocate memory for a pointer in each iteration without having to free it first?
[Supplementary message]:
Hi all, I think there's one case we cannot free memory in the loop. If buffer=p and p is defined outside the loop, like:
float *buffer, *p;
/* Elements of p calculated */
for (...){
/* allocate memory for **buffer */
buffer = p;
free(buffer)
/* if free here will cause p lost */
}
If free buffer at the end of each loop, it may cause p lost because buffer and p share the same memory address.
why is it legal that we can allocate memory for a pointer in each iteration without having to free it first?
The responsibility of freeing dynamically allocated memory is left on the programmer. It is legal because the compiler does not enforce it, although there are code checking tools that can flag this problem.
Freeing dynamically allocated memory should be done in the reverse order of allocation. For ex:
for (i=0;i<nloc;i++)
free(sign[i]);
free(sign);
It is legal because in C, you as the programmer are responsible for memory management. There is a very good reason for this. To quote another thread
Garbage collection requires data structures for tracking allocations
and/or reference counting. These create overhead in memory,
performance, and the complexity of the language. C++ is designed to be
"close to the metal", in other words, it takes the higher performance
side of the tradeoff vs convenience features. Other languages make
that tradeoff differently. This is one of the considerations in
choosing a language, which emphasis you prefer.
Not only is performance and code size a consideration, but different systems have difference addressing schemes for memory. It is also for this reason that C is easy to port and maintain across platforms, given that there is no garbage collection to alter.
EDIT: Some answers mention freeing memory space as opposed to the pointer itself, it is worth further specifying what that means: free() simply marks the allocated space as available, it is not 'freed' or erased, nor does any other operation occur on that memory space. It is then still incumbent on the programmer to delete the address that was assigned to the pointer variable.
My question is: why is it legal that we can allocate memory for a pointer in each iteration without having to free it first?
Short answer
C trades away safety to gain simplicity and performance.
Longer answer
You don't free the pointer. You free the memory block the pointer is pointing at.
Think about what malloc (and calloc) does. They both allocate a piece of memory, and if the allocation is successful they return a pointer to that memory block. The allocation functions (like all functions) has no insight, nor control whatsoever of what you do with the return value. The functions does not even see the pointer that you are assigning the return value to.
It would be fairly complex (relatively speaking, C has a pretty simple structure) to implement a protection against it. First, consider this code:
int * a = malloc(1);
int * b = a;
a = malloc(1);
free(b);
free(a);
This code has no memory leaks, even though we did the precise thing you asked about. We reassigned a before calling free upon the memory from the first malloc. It works fine, because we have saved the address in b.
So in order to disallow reassigning pointers that points to a memory block that no other pointer points at, the runtime environment would need to keep track of this, and it is not entirely trivial. And it would also need to create extra code to handle all this. Since this check needs to be done at runtime, it may affect performance.
Also, remember that even though C may seem very low level today, it was considered a high level language when it came.
I'm currently learning C programming and since I'm a python programmer, I'm not entirely sure about the inner workings of C. I just stumbled upon a really weird thing.
void test_realloc(){
// So this is the original place allocated for my string
char * curr_token = malloc(2*sizeof(char));
// This is really weird because I only allocated 2x char size in bytes
strcpy(curr_token, "Davi");
curr_token[4] = 'd';
// I guess is somehow overwrote data outside the allocated memory?
// I was hoping this would result in an exception ( I guess not? )
printf("Current token > %s\n", curr_token);
// Looks like it's still printable, wtf???
char *new_token = realloc(curr_token, 6);
curr_token = new_token;
printf("Current token > %s\n", curr_token);
}
int main(){
test_realloc();
return 0;
}
So the question is: how come I'm able to write more chars into a string than is its allocated size? I know I'm supposed to handle mallocated memory myself but does it mean there is no indication that something is wrong when I write outside the designated memory?
What I was trying to accomplish
Allocate a 4 char ( + null char ) string where I would write 4 chars of my name
Reallocate memory to acomodate the last character of my name
know I'm supposed to handle mallocated memory myself but does it mean there is no indication that something is wrong when I write outside the designated memory?
Welcome to C programming :). In general, this is correct: you can do something wrong and receive no immediate feedback that was the case. In some cases, indeed, you can do something wrong and never see a problem at runtime. In other cases, however, you'll see crashes or other behaviour that doesn't make sense to you.
The key term is undefined behavior. This is a concept that you should become familiar with if you continue programming in C. It means just like it sounds: if your program violates certain rules, the behaviour is undefined - it might do what you want, it might crash, it might do something different. Even worse, it might do what you want most of the time, but just occasionally do something different.
It is this mechanism which allows C programs to be fast - since they don't at runtime do a lot of the checks that you may be used to from Python - but it also makes C dangerous. It's easy to write incorrect code and be unaware of it; then later make a subtle change elsewhere, or use a different compiler or operating system, and the code will no longer function as you wanted. In some cases this can lead to security vulnerabilities, since unwanted behavior may be exploitable.
Suppose that you have an array as shown below.
int arr[5] = {6,7,8,9,10};
From the basics of arrays, name of the array is a pointer pointing to the base element of the array. Here, arr is the name of the array, which is a pointer, pointing to the base element, which is 6. Hence,*arr, literally, *(arr+0) gives you 6 as the output and *(arr+1) gives you 7 and so on.
Here, size of the array is 5 integer elements. Now, try accessing the 10th element, though the size of the array is 5 integers. arr[10]. This is not going to give you an error, rather gives you some garbage value. As arr is just a pointer, the dereference is done as arr+0,arr+1,arr+2and so on. In the same manner, you can access arr+10 also using the base array pointer.
Now, try understanding your context with this example. Though you have allocated memory only for 2 bytes for character, you can access memory beyond the two bytes allocated using the pointer. Hence, it is not throwing you an error. On the other hand, you are able to predict the output on your machine. But it is not guaranteed that you can predict the output on another machine (May be the memory you are allocating on your machine is filled with zeros and may be those particular memory locations are being used for the first time ever!). In the statement,
char *new_token = realloc(curr_token, 6); note that you are reallocating the memory for 6 bytes of data pointed by curr_token pointer to the new_tokenpointer. Now, the initial size of new_token will be 6 bytes.
Usually malloc is implemented such a way that it allocates chunks of memory aligned to paragraph (fundamental alignment) that is equal to 16 bytes.
So when you request to allocate for example 2 bytes malloc actually allocates 16 bytes. This allows to use the same chunk of memory when realloc is called.
According to the C Standard (7.22.3 Memory management functions)
...The pointer returned if the allocation succeeds is suitably aligned so
that it may be assigned to a pointer to any type of object
with a fundamental alignment requirement and then used to access such an
object or an array of such objects in the space allocated
(until the space is explicitly deallocated).
Nevertheless you should not rely on such behavior because it is not normative and as result is considered as undefined behavior.
No automatic bounds checking is performed in C.
The program behaviour is unpredictable.
If you go writing in the memory reserved for another process, you will end with a Segmentation fault, otherwise you will only corrupt data, ecc...
Suppose I do the following
int *p = 1;
printf("%d", *p);
I get a segfault. Now, as I understand it, this memory location 1 is in the address space of my program. Why should there be a problem in reading this memory location ?
Your pointer doesn't point to anything valid. All you do is assign the value 1 to a pointer-to-int, but 1 isn't a valid memory location.
The only valid way to obtain an pointer value is either take the address-of a variable or call an allocation function:
int a;
int * p1 = &a; // OK
int * p2 = malloc(sizeof(int)); // also OK
*p1 = 2;
*p2 = 3;
As for "why there should be a problem": As far as the language is concerned, if you dereference an invalid pointer, you have undefined behaviour, so anything can happen -- this is really the only sensible way to specify the language if you don't want to introduce any arbitrary restrictions, and C is all about being easy-to-implement.
Practically, modern operating systems will usually have a clever virtual memory manager that needs to request memory when and as needed, and if the memory at address 1 isn't on a committed page yet, you'll actually get an error from the OS. If you try a pointer value near some actual address, you might not get an error (until you overstep the page boundary, perhaps).
To answer this properly will really depend upon a number of factors. However, it is quite likely that the address 0x00000001 (page 0) is not mapped and/or read protected.
Typically, addresses in the page 0 range are disallowed from a user application for one reason or another. Even in kernel space (depending upon the processor), addresses in page 0 are often protected and require that the page be both mapped and access enabled.
EDIT:
Another possible reason is that it could be segfaulting is that integer access is not aligned.
Your operating system won't let you access memory locations that don't belong to your program. It would work on kernel mode though...
No, address 1 is not in the address space of your program. You are trying to access a segment you do not own and receive segmentation fault.
You're trying to print 1, not the memory location. 1 isn't a valid memory location. To print the actual memory location do:
printf("%p", p);
Note the %p as icktoofay pointed out.
int main()
{
char *s1, *sTemp;
s1 = (char*)malloc(sizeof(char)*7);
*(s1 + 0) = 'a';
*(s1 + 1) = 'b';
*(s1 + 2) = 'c';
*(s1 + 3) = 'd';
*(s1 + 4) = 'e';
*(s1 + 5) = 'f';
*(s1 + 6) = '\0';
sTemp = (s1 + 3);
free(sTemp); // shud delete d onwards. But it doesn't !!
return 0;
}
Hello,
In the C above code sTemp should point to the 3rd cell beyond s1 ( occupied by 'd')
So on calling free(sTemp) i expect to have something deleted from this location onwards.
( I purposely mention 'something' as the motive of my experiment initially was to find out till which location the free() ing works )
However i recieve a SIGABRT at the free().
How does free() know that this is not the start of the chunk. and correspondingly can we free up memory only from start of chunks? [ are they only the free-able pointers that free() can accept?? ]
Looking forward to replies :)
From the man pages
free() frees the memory space pointed
to by ptr, which must have been
returned by a previous call to
malloc(), calloc() or realloc().
Otherwise, or if free(ptr) has already
been called before, undefined
behaviour occurs.
Source: http://linux.die.net/man/3/free
About the actual question "how does free know...":
free knows that it is not at the start of the chunk because it maintains metadata that you don't know about. If you think about it, that's necessarily so, because otherwise it could never know how much memory to free, given only an address.
It is not specified how exactly the runtime keeps book of allocation sizes, only that you must never pass any pointer to free that did not come from a function of the malloc family.
Usually this works by malloc allocating a little more memory than needed[1] and writing some metadata in memory preceding the address that is returned. free then just looks at e.g. ptr[-8] or whatever (implementation detail!) to read this metadata.
It can then optionally do some consistency checks, and can determine the size of the memory block (one trivial consistency check that is probably always done is checking proper alignment of the pointer).
Having mentioned that, please please please, don't even think about playing with this metadata.
[1] It often does that anyway to satisfy alignment requirements and because of allocator internals (most allocators manage memory in different "buckets" of fixed size), so if you allocate, say, 4 bytes, you nominally get 4 bytes, but the runtime really allocated 8 or 16 bytes most of the time (without you knowing).
You can only free() pointers that were actually malloced (or calloced, realloced). If you try to free only a portion of memory by passing in a different pointer (even one that is "part" of another pointer), the C runtime will not be pleased (as you can see.)
I think you cannot free the memory because free doesn't know how much memory to free.Your program has information about the address malloc() returned and not for every address in this space.So you can free(s1) but not free(s1+3).Also you can handle your pointers as an array in this example:
s1[0]='a';
You can only free() memory that was previously malloced, calloced, or realloced, as dlev and Daniel have said.
The reason for this is implementation-dependent, but involves the method of keeping track of allocated memory.
Efficient memory allocation is a difficult problem because different memory allocation algorithms work well depending upon how memory is allocated: a few small chunks, half of which are freed, then slightly larger blocks being grabbed, etc.
The objective is to keep the size of the memory block used by the program at a minimum (usually this chunk will be a continuous block of virtual memory), while keeping the usage of the space within that block extremely high (few gaps between used segments of memory).
Remember that these blocks can't be moved except when being realloced, so there's always going to be some wasted space.
To minimize the waste, metadata about (at least) the size of the block is stored just before it. The memory allocator can look through the used blocks when determining how to handle a new request. If you pick a random memory location, whether part of a previously-allocated region or no, this metadata will not be present and free will be unable to determine what should be freed.
You cannot free that pointer the way you are doing, check this question and its answer: Memory Clobbering Error