I spent an embarrassing amount of time last night tracking down a segfault in my application. Ultimately, it turned out I'd written:
ANNE_SPRITE_FRAME *desiredFrame;
*desiredFrame = anne_sprite_copy_frame(&sprite->current);
instead of:
ANNE_SPRITE_FRAME desiredFrame;
desiredFrame = anne_sprite_copy_frame(&sprite->current);
In line 1 I created a typed pointer, and in line 2 I set the value of the dereferenced pointer to the struct returned by anne_sprite_copy_frame().
Why was this a problem? And why did the compiler accept this at all? All I can figure is that the problem in example 1 is either:
I'm reserving space for the pointer but not the contents that it points to, or
(unlikely) it's trying to store the return value in the memory of the pointer itself
In line 1 I've created a typed pointer, and in line 2 I set the value of the dereferenced pointer to the struct returned by anne_sprite_copy_frame().
Both of these are allowed in C, which is why this is perfectly acceptable by the compiler.
The compiler doesn't check to make sure your pointer actually points to anything meaningful - it just dereferences and assigns.
One of the best and worst features of C is that the compiler does very little sanity checking for you - it follows your instructions, and does exactly what you tell it to do. You told it to do two legal operations - even though the variables were not initialized properly. As such, you get runtime issues, not compile time problems.
I'm reserving space for the pointer but not the contents that it points to
Yeah, exactly. But the compiler (unless it does some static analysis) can't infer that. It only sees that the syntax is valid and the types match, so it compiles your program. Dereferencing an uninitialized pointer is undefined behavior, though, so your program will most likely work erroneously.
The pointer is uninitialized, but it still has a value so it points somewhere. Writing the return value to that memory address overwrites whatever happens to be there, invoking undefined behavior.
Technically the compiler is not in the business of telling you that a syntactically valid construct will result in undefined (or even likely unexpected) behavior, but I would be surprised if there was no warning issued about this particular usage.
C is weakly typed. You can assign anything to anything with the obvious consequences. You have to be very careful and disciplined if you do not want to spend nights uncovering bugs that turn out "stupid". I mean no offense. I went through the same issues due to an array bound overflow that overwrote other variables and only showed up in some other part of the code trying to use these variables. Nightmare! That's why Java is so much easier to deal with. With C you are an acrobat without a net, with Java, you can afford to fall. That said, I do not mean to say Java is better. C has its raison d'etre.
Related
If I define a structure...
struct LinkNode
{
int node_val;
struct LinkNode *next_node;
};
and then create a pointer to it...
struct LinkNode *mynode = malloc(sizeof(struct LinkNode));
...and then finally free() it...
free(mynode);
...I can still access the 'next_node' member of the structure.
mynode->next_node
My question is this: which piece of the underlying mechanics keeps track of the fact that this block of memory is supposed to represent a struct LinkNode? I'm a newbie to C, and I expected that after I used free() on the pointer to my LinkNode, that I would no longer be able to access the members of that struct. I expected some sort of 'no longer available' warning.
I would love to know more about how the underlying process works.
The compiled program no longer has any knowledge about struct LinkedNode or field named next_node, or anything like that. Any names are completely gone from the compiled program. The compiled program operates in terms of numerical values, which can play roles of memory addresses, offsets, indices and so on.
In your example, when you read mynode->next_node in the source code of your program, it is compiled into machine code that simply reads the 4-byte numerical value from some reserved memory location (known as variable mynode in your source code), adds 4 to it (which is offset of the next_node field) and reads the 4-byte value at the resultant address (which is mynode->next_node). This code, as you can see, operates in terms of integer values - addresses, sizes and offsets. It does not care about any names, like LinkedNode or next_node. It does not care whether the memory is allocated and/or freed. It does not care whether any of these accesses are legal or not.
(The constant 4 I repeatedly use in the above example is specific for 32-bit platforms. On 64-bit platforms it would be replaced by 8 in most (or all) instances.)
If an attempt is made to read memory that has been freed, these accesses might crash your program. Or they might not. It is a matter of pure luck. As far as the language is concerned, the behavior is undefined.
There isn't and you can't. This is a classic case of undefined behavior.
When you have undefined behavior, anything can happen. It may even appear to work, only to randomly crash a year later.
It works by pure luck, because the freed memory has not yet been overwritten by something else. Once you free the memory, it is your responsibility to avoid using it again.
No part of the underlying Memory keeps track of it. It's just the semantics the programming language gives to the chunk of memory. You could e.g. cast it to something completely different and can still access the same memory region. However the catch here is, that this is more likely to lead to errors. Especially type-safty will be gone. In your case just because you called free doesn't mean that the underlying memory canges at all. There is just a flag in your operating system that marks this region as free again.
Think about it this way: the free-function is something like a "minimal" memory management system. If your call would require more than setting a flag it would introduce unneccessary overhead. Also when you access the member you (i.e. your operating system) could check if the flag for this memory region is set to "free" or "in use". But that's overhead again.
Of course that doesn't mean it wouldn't make sense to do those kind of things. It would avoid a lot of security holes and is done for example in .Net and Java. But those runtimes are much younger than C and we have much more ressources these days.
When your compiler translates your C code into executable machine code, a lot of information is thrown away, including type information. Where you write:
int x = 42;
the generated code just copies a certain bit pattern into a certain chunk of memory (a chunk that might typically be 4 bytes). You can't tell by examining the machine code that the chunk of memory is an object of type int.
Similarly, when you write:
if (mynode->next_node == NULL) { /* ... */ }
the generated code will fetch a pointer sized chunk of memory by dereferencing another pointer-sized chunk of memory, and compare the result to the system's representation of a null pointer (typically all-bits-zero). The generated code doesn't directly reflect the fact that next_node is a member of a struct, or anything about how the struct was allocated or whether it still exists.
The compiler can check a lot of things at compile time, but it doesn't necessarily generate code to perform checks at execution time. It's up to you as a programmer to avoid making errors in the first place.
In this specific case, after the call to free, mynode has an indeterminate value. It doesn't point to any valid object, but there's no requirement for the implementation to do anything with that knowledge. Calling free doesn't destroy the allocated memory, it merely makes it available for allocation by future calls to malloc.
There are a number of ways that an implementation could perform checks like this, and trigger a run-time error if you dereference a pointer after freeing it. But such checks are not required by the C language, and they're generally not implemented because (a) they would be quite expensive, making your program run more slowly, and (b) checks can't catch all errors anyway.
C is defined so that memory allocation and pointer manipulation will work correctly if your program does everything right. If you make certain errors that can be detected at compile time, the compiler can diagnose them. For example, assigning a pointer value to an integer object requires at least a compile-time warning. But other errors, such as dereferencing a freed pointer, cause your program to have undefined behavior. It's up to you, as a programmer, to avoid making those errors in the first place. If you fail, you're on your own.
Of course there are tools that can help. Valgrind is one; clever optimizing compilers are another. (Enabling optimization causes the compiler to perform more analysis of your code, and that can often enable it to diagnose more errors.) But ultimately C is not a language that holds your hand. It's a sharp tool -- and one that can be used to build safer tools, such as interpreted languages that do more run-time checking.
You need to assign NULL to mynode->next_node:
mynode->next_node = NULL;
after freeing the memory so it will indicate that you are not using anymore the memory allocated.
Without assigning the NULL value, it is still pointing to the previously freed memory location.
I have this piece of code, and it runs perfectly fine, and I don't why:
int main(){
int len = 10;
char arr[len];
arr[150] = 'x';
}
Seriously, try it! It works (at least on my machine)!
It doesn't, however, work if I try to change elements at indices that are too large, for instance index 20,000. So the compiler apparently isn't smart enough to just ignore that one line.
So how is this possible? I'm really confused here...
Okay, thanks for all the answers!
So I can use this to write into memory consumed by other variables on the stack, like so:
#include <stdio.h>
main(){
char b[4] = "man";
char a[10];
a[10] = 'c';
puts(b);
}
Outputs "can". That's a really bad thing to do.
Okay, thanks.
C compilers generally do not generate code to check array bounds, for the sake of efficiency. Out-of-bounds array accesses result in "undefined behavior", and one
possible outcome is that "it works". It's not guaranteed to cause a crash or other
diagnostic, but if you're on an operating system with virtual memory support, and your array index points to a virtual memory location that hasn't yet been mapped to physical memory, your program is more likely to crash.
So how is this possible?
Because the stack was, on your machine, large enough that there happened to be a memory location on the stack at the location to which &arr[150] happened to correspond, and because your small example program exited before anything else referred to that location and perhaps crashed because you'd overwritten it.
The compiler you're using doesn't check for attempts to go past the end of the array (the C99 spec says that the result of arr[150], in your sample program, would be "undefined", so it could fail to compile it, but most C compilers don't).
Most implementations don't check for these kinds of errors. Memory access granularity is often very large (4 KiB boundaries), and the cost of finer-grained access control means that it is not enabled by default. There are two common ways for errors to cause crashes on modern OSs: either you read or write data from an unmapped page (instant segfault), or you overwrite data that leads to a crash somewhere else. If you're unlucky, then a buffer overrun won't crash (that's right, unlucky) and you won't be able to diagnose it easily.
You can turn instrumentation on, however. When using GCC, compile with Mudflap enabled.
$ gcc -fmudflap -Wall -Wextra test999.c -lmudflap
test999.c: In function ‘main’:
test999.c:3:9: warning: variable ‘arr’ set but not used [-Wunused-but-set-variable]
test999.c:5:1: warning: control reaches end of non-void function [-Wreturn-type]
Here's what happens when you run it:
$ ./a.out
*******
mudflap violation 1 (check/write): time=1362621592.763935 ptr=0x91f910 size=151
pc=0x7f43f08ae6a1 location=`test999.c:4:13 (main)'
/usr/lib/x86_64-linux-gnu/libmudflap.so.0(__mf_check+0x41) [0x7f43f08ae6a1]
./a.out(main+0xa6) [0x400a82]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xfd) [0x7f43f0538ead]
Nearby object 1: checked region begins 0B into and ends 141B after
mudflap object 0x91f960: name=`alloca region'
bounds=[0x91f910,0x91f919] size=10 area=heap check=0r/3w liveness=3
alloc time=1362621592.763807 pc=0x7f43f08adda1
/usr/lib/x86_64-linux-gnu/libmudflap.so.0(__mf_register+0x41) [0x7f43f08adda1]
/usr/lib/x86_64-linux-gnu/libmudflap.so.0(__mf_wrap_alloca_indirect+0x1a4) [0x7f43f08afa54]
./a.out(main+0x45) [0x400a21]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xfd) [0x7f43f0538ead]
number of nearby objects: 1
Oh look, it crashed.
Note that Mudflap is not perfect, it won't catch all of your errors.
Native C arrays do not get bounds checking. That would require additional instructions and data structures. C is designed for efficiency and leanness, so it doesn't specify features that trade performance for safety.
You can use a tool like valgrind, which runs your program in a kind of emulator and attempts to detect such things as buffer overflows by tracking which bytes are initialized and which aren't. But it's not infallible, for example if the overflowing access happens to perform an otherwise-legal access to another variable.
Under the hood, array indexing is just pointer arithmetic. When you say arr[ 150 ], you are just adding 150 times the sizeof one element and adding that to the address of arr to obtain the address of a particular object. That address is just a number, and it might be nonsense, invalid, or itself an arithmetic overflow. Some of these conditions result in the hardware generating a crash, when it can't find memory to access or detects virus-like activity, but none result in software-generated exceptions because there is no room for a software hook. If you want a safe array, you'll need to build functions around the principle of addition.
By the way, the array in your example isn't even technically of fixed size.
int len = 10; /* variable of type int */
char arr[len]; /* variable-length array */
Using a non-const object to set the array size is a new feature since C99. You could just as well have len be a function parameter, user input, etc. This would be better for compile-time analysis:
const int len = 10; /* constant of type int */
char arr[len]; /* constant-length array */
For the sake of completeness: The C standard doesn't specify bounds checking but neither is it prohibited. It falls under the category of undefined behavior, or errors that need not generate error messages, and can have any effect. It is possible to implement safe arrays, various approximations of the feature exist. C does nod in this direction by making it illegal, for example, to take the difference between two arrays in order to find the correct out-of-bounds index to access an arbitrary object A from array B. But the language is very free-form, and if A and B are part of the same memory block from malloc it is legal. In other words, the more C-specific memory tricks you use, the harder automatic verification becomes even with C-oriented tools.
Under the C spec, accessing an element past the end of an array is undefined behaviour. Undefined behaviour means that the specification does not say what would happen -- therefore, anything could happen, in theory. The program might crash, or it might not, or it might crash hours later in a completely unrelated function, or it might wipe your harddrive (if you got unlucky and poked just the right bits into the right place).
Undefined behaviour is not easily predictable, and it should absolutely never be relied upon. Just because something appears to work does not make it right, if it invokes undefined behaviour.
Because you were lucky. Or rather unlucky, because it means it's harder to find the bug.
The runtime will only crash if you start using the memory of another process (or in some cases unallocated memory). Your application is given a certain amount of memory when it opens, which in this case is enough, and you can mess about in your own memory as much as you like, but you'll give yourself a nightmare of a debugging job.
Clang's scan-build reports quite a few dereference of null pointers in my project, however, I don't really see any unusual behavior (in 6 years of using it), ie:
Dereference of null pointer (loaded from variable chan)
char *tmp;
CList *chan = NULL;
/* This is weird because chan is set via do_lookup so why could it be NULL? */
chan = do_lookup(who, me, UNLINK);
if (chan)
tmp = do_lookup2(you,me,0);
prot(get_sec_var(chan->zsets));
^^^^
I know null derefs can cause crashes but is this really a big security concern as some people make it out to be? What should I do in this case?
It is Undefined Behavior to dereference a NULL pointer. It can show any behavior, it might crash or not but you MUST fix those!
The truth about Undefined Behavior is that it obeys Murphy's Law
"Anything that can go wrong will go wrong"
It makes no sense checking chan for NULL at one point:
if (chan)
tmp = do_lookup2(you,me,0); /* not evaluated if `chan` is NULL */
prot(get_sec_var(chan->zsets)); /* will be evaluated in any case */
... yet NOT checking it right at the next line.
Don't you have to execute both these statements within if branch?
Clang is warning you because you check for chan being NULL, and then you unconditionally dereference it in the next line anyway. This cannot possibly be correct. Either do_lookup cannot return NULL, then the check is useless and should be removed. Or it can, then the last line can cause undefined behaviour and MUST be fixed. Als is 100% correct: NULL pointer dereferences are undefined behaviour and are always a potential risk.
Probably you want to enclose your code in a block, so that all of it is governed by the check for NULL, and not just the next line.
You have to fix these as soon as possible. Or probably sooner. The Standard says that the NULL pointer is a pointer that points to "no valid memory location", so dereferencing it is undefined behaviour. It means that it may work, it may crash, and it may do strange things at other parts of your program, or maybe cause daemons to fly out of your nose.
Fix them. Now.
Here's how: put the dereference statement inside the if - doing otherwise (as you do: checking for NULL then dereferencing anyways) makes no sense.
if (pointer != NULL) {
something = pointer->field;
}
^^ this is good practice.
If you have never experienced problems with this code, it's probably because:
do_lookup(who, me, UNLINK);
always returns a valid pointer.
But what will it happen if this function changes? Or its parameters vary?
You definitely have to check for NULL pointers before dereferencing them.
if (chan)
prot(get_sec_var(chan->zsets));
If you are absolutely sure that neither do_lookup or its parameters will change in the future (and you can bet the safe execution of your program on it), and the cost of changing all the occurrences of similar functions is excessively high compared to the benefits that you would have in doing so, then:
you may decide to leave your code broken.
Many programmers did that in the past, and many more will do that in the future. Otherwise what would explain the existence of Windows ME?
If your program crashes because of a NULL pointer dereference, this can be classified as a Denial of Service (DoS).
If this program is used together with other programs (e.g. they invoke it), the security aspects now start to depend on what those other programs do when this one crashes. The overall effect can be the same DoS or something worse (exploitation, sensitive info leakage, and so on).
If your program does not crash because of a NULL pointer dereference and instead continues running while corrupting itself and possibly the OS and/or other programs within the same address space, you can have a whole spectrum of security issues.
Don't put on the line (or online) broken code, unless you can afford dealing with consequences of potential hacking.
We know that automatic variables are destroyed upon the return of the function.
Then, why is this C program returning correct value?
#include <stdio.h>
#include <process.h>
int * ReturningPointer()
{
int myInteger = 99;
int * ptrToMyInteger = &myInteger;
return ptrToMyInteger;
}
main()
{
int * pointerToInteger = ReturningPointer();
printf("*pointerToInteger = %d\n", *pointerToInteger);
system("PAUSE");
}
Output
*pointerToInteger = 99
Edit
Then why is this giving garbage values?
#include <stdio.h>
#include <process.h>
char * ReturningPointer()
{
char array[13] = "Hello World!";
return array;
}
main()
{
printf("%s\n", ReturningPointer());
system("PAUSE");
}
Output
xŤ
There is no answer to that question: your code exhibits undefined behavior. It could print "the right value" as you are seeing, it could print anything else, it could segfault, it could order pizza online with your credit card.
Dereferencing that pointer in main is illegal, it doesn't point to valid memory at that point. Don't do it.
There's a big difference between you two examples: in the first case, *pointer is evaluated before calling printf. So, given that there are no function calls between the line where you get the pointer value, and the printf, chances are high that the stack location pointer points to will not have been overwritten. So the value that was stored there prior to calling printf is likely to be output (that value will be passed on to printf's stack, not the pointer).
In the second case, you're passing a pointer to the stack to printf. The call to printf overwrites (a part of) that same stack region the pointer is pointing to, and printf ends up trying to print its own stack (more or less) which doesn't have a high chance of containing something readable.
Note that you can't rely on getting gibberish either. Your implementation is free to use a different stack for the printf call if it feels like it, as long as it follows the requirements laid out by the standard.
This is undefined behavior, and it could have launched a missile instead. But it just happened to give you the correct answer.
Think about it, it kind of make sense -- what else did you expect? Should it have given you zero? If so, then the compiler must insert special instructions at the scope end to erase the variable's content -- waste of resources. The most natural thing for the compiler to do is to leave the contents unchanged -- so you just got the correct output from undefined behavior by chance.
You could say this behavior is implementation defined. For example. Another compiler (or the same compiler in "Release" mode) may decide to allocate myInteger purely in register (not sure if it actually can do this when you take an address of it, but for the sake of argument...), in that case no memory would be allocated for 99 and you would get garbage output.
As a more illustrative (but totally untested) example -- if you insert some malloc and exercise some memory usage before printf you may find the garbage value you were looking for :P
Answer to "Edited" part
The "real" answer that you want needs to be answered in disassembly. A good place to start is gcc -S and gcc -O3 -S. I will leave the in-depth analysis for wizards that will come around. But I did a cursory peek using GCC and it turns out that printf("%s\n") gets translated to puts, so the calling convention is different. Since local variables are allocated on the stack, calling a function could "destroy" previously allocated local variables.
Destroying is the wrong word imho. Locals reside on the stack, if the function returns the stack space may be reused again. Until then it is not overwritten and still accessible by pointers which you might not really want (because this might never point to something valid)
Pointers are used to address space in memory, for local pointers the same as I described in 1 is valid. However the pointer seems to be passed to the main program.
If it really is the address storing the former integer it will result in "99" up until that point in the execution of your program when the program overwrite this memory location. It may also be another 99 by coincidence. Any way: do not do this.
These kind of errors will lead to trouble some day, may be on other machines, other OS, other compiler or compiler options - imagine you upgrade your compiler which may change the behaviour the memory usage or even a build with optimization flags, e.g. release builds vs default debug builds, you name it.
In most C/C++ programs their local variables live on the stack, and destroyed means overwritten with something else. In this case that particular location had not been overwritten yet when it was passed as a parameter to printf().
Of course, having such code is asking for trouble because per the C and C++ standards it exhibits undefined behavior.
That is undefined behavior. That means that anything can happen, even what you would expect.
The tricky part of UB is when it gives you the result you expect, and so you think that you are doing it right. Then, any change in an unrelated part of the program changes that...
Answering your question more specifically, you are returning a pointer to an automatic variable, that no longer exists when the function returns, but since you call no other functions in the middle, it happens to keep the old value.
If you call, for example printf twice, the second time it will most likely print a different value.
The key idea is that a variable represents a name and type for value stored somewhere in memory. When it is "destroyed", it means that a) that value can no longer be accessed using that name, and b) the memory location is free to be overwritten.
The behavior is undefined because the implementation of the compiler is free to choose what time after "destruction" the location is actually overwritten.
Assume we have the following:
int main(void) {
char* ptr;
printf("%c\n",ptr[24]); // junk value
ptr[24] = 'H';
printf("%c\n", ptr[24]); // prints H
return 0;
}
When I change the junk value to something else, does that mean I am corrupting memory or is this value literally junk so it doesn't matter what new value I assign to it?
Your program exhibits undefined behaviour which means: Literally anything may happen and it's still be coverd by the standard as being undefiend. And when I say anything, I mean it in the full extent. It would be even valid for your computer becoming sentient and chase you down the street.
Well, what's usually happens, but that's not warranted, is that you're writing into unmapped address space (on a modern OS with paged memory) causing a segmentation fault or a bus error (depending on architecture, OS and runtime implementation).
ptr is an unitialized pointer, which means the pointer's value is yet to be defined. A undefined pointer, by definition, points to nothing and everything, i.e. no valid object at all. The only way to make that pointer valid is assigning it the address of a proper C object of the type the pointer dereferences to.
BTW: Plain C has very, very strict typing rules. I sometimes say it's even stricter than C++, because its lack of the implicit conversion operator and function overloading. But its sloppy type casting and bad compilers ruined its reputation with respect to type safety.
You are accessing invalid memory locations which invokes undefined behavior. Anything might happen, it can't be predicted.
Since most C implementations allow you to access invalid memory locations, you are actually assigning the 'H' value to that position.
But you cannot trust what's gonna happen next. Maybe your program fails, maybe you damage memory in use by other program, or, in a multithreaded environment, another program may overwrite that value.