I have
x=(int *)malloc(sizeof(int)*(1));
but still I am able to read x[20] or x[4].
How am I able to access those values? Shouldn't I be getting segmentation error while accessing those memory?
The basic premise is that of Sourav Ghosh's answer: accessing memory returned from malloc beyond the size you asked for is undefined behavior, so a conforming implementation is allowed to do pretty much anything, including happily returning bizarre values.
But given a "normal" implementation on mainstream operating systems on "normal" machines (gcc/MSVC/clang, Linux/Windows/macOS, x86/ARM) why do you sometimes get segmentation faults (or access violations), and sometimes not?
Pretty much every "regular" C implementation doesn't perform any kind of memory check when reading/writing through pointers1; these loads/stores get generally translated straight to the corresponding machine code, which accesses the memory at a given location without much regard for the size of the "abstract C machine" objects.
However, on these machines the CPU doesn't straight access the physical memory (RAM) of the PC, but a translation layer (MMU) is introduced2; whenever your program tries to access an address, the MMU checks to see whether anything has been mapped there, and if your process has permissions to write over there. In case any of those checks fail3, you get a segmentation fault and your process gets killed. This is why uninitialized and NULL pointer values generally give nice segfaults: some memory at the beginning of the virtual address space is reserved unmapped just to spot NULL dereferences, and in general if you throw a dart at random into a 32 bit address space (or even better, a 64 bit one) you are most likely to find zones of memory that have never been mapped to anything.
As good as it is, the MMU cannot catch all your memory errors for several reasons.
First of all, the granularity of memory mappings is quite coarse compared to most "run of the mill" allocations; on PCs memory pages (the smallest unit of memory that can be mapped and have protection attributes) are generally 4 KB in size. There is of course a tradeoff here: very small pages would require a lot of memory themselves (as there's a target physical address plus protection attributes associated to each page, and those have to be stored somewhere) and slow down the MMU operation3. So, if you access memory out of "logical" boundaries but still within the same memory page, the MMU cannot help you: as far as the hardware is concerned, you are still accessing valid memory.
Besides, even if you go outside of the last page of your allocation, it may be that the page that follows is "valid" as far as the hardware is concerned; indeed, this is pretty common for memory you get from the so-called heap (malloc & friends).
This comes from the fact that malloc, for smaller allocations, doesn't ask the OS for "new" blocks of memory (which in theory may be allocated keeping a guard page at both ends); instead, the allocator in the C runtime asks the OS for memory in big sequential chunks, and logically partitions them in smaller zones (usually kept in linked lists of some kind), which are handed out on malloc and returned back by free.
Now, when in your program you step outside the boundaries of the requested memory, you probably don't get any error as:
the memory chunk you are using isn't near a page boundary, so your out-of-bounds read doesn't trigger an access violation;
even if it was at the end of a page, the page that follows is still mapped, as it still belongs to the heap; it may either be memory that has been given to some other code of your process (so you are reading data of some unrelated part of your code), or a free memory zone (so you are reading whatever garbage happened to be left by the previous owner of the block when it freed it), or a zone used by the allocator to keep its bookkeping data (so you are reading parts of such data).
In all these cases except for the "free block" one, even if you were to write there you wouldn't get a segmentation fault, but you could corrupt unrelated data or the data structures of the heap (which generally results in crashes later, as the allocator finds inconsistencies in its data).
Notes
Although modern compilers provide special instrumented builds to trap some of these errors; gcc and clang, in particular, provide the so-called "address sanitizer".
This allows to introduce transparent paging (swapping out to disk memory zones that aren't actively used in case of low physical memory availability) and, most importantly, memory protection and address space separation (when a user-mode process is running, it "sees" a full virtual address space containing only his stuff, and nothing from the other processes or the kernel).
And it's not a failure put there on purpose by the operating system to be notified that the processes is trying to access memory that has been swapped out.
Given that each access to memory needs to go through the MMU, the mapping must be very fast, so the most used page mappings are kept in a cache; if you make the pages very small and the cache can hold just as many entries, you effectively have a smaller memory range covered by the cache.
No, accessing invalid memory is undefined behavior, and segmantation fault is one of the many side effects of UB. It is not guaranteed.
That said,
Always check for the success of the malloc() by checking the returned pointer against NULL before using the returned pointer.
Please see this: Do I cast the result of malloc?
Related
I am trying to understand the scenarios in which call to memcpy can fail silently because invalid pointers will result in access violation/segfaults. Also, there will be issues in case of overlapping pointers. Apart from these, are there any other ways the memcpy call can fail? Or we can consider it'll pass all the time without any error. How to verify it?
The memcpy has the precondition that the memory areas will not overlap. You invoke undefined behavior if you violate that precondition.
Similarly, you also invoke undefined behavior if you read past the bounds of the source buffer or write past the bounds of the destination buffer. This is dictated in the standard.
When you invoke undefined behavior, you can't predict how the program will behave in the future (or even in the past). It could crash, it could output strange results, or it could appear to work normally.
Using a tool such as valgrind is very helpful in identifying when your program violates various memory constraints, such as reading or writing past the end of a buffer, using an uninitialized value, dereferencing a null pointer, or performing a double free.
If you give it valid pointers that do not overlap and you do not overrun the buffers with your reads/writes, it will not fail. If you do some of those things, it may still not fail, but could do unexpected things. It will always return dest.
I am trying to understand the scenarios in which call to memcpy can fail silently because invalid pointers will result in access violation/segfaults.
Typically when there's an access violation/segfault, software fails loudly. This can happen if memcpy() is given dodgy pointers or a bad size, which includes "correct pointers but heap was corrupted" (e.g. the metadata that malloc()/free() uses to keep track of allocated areas was overwritten by a bug causing free() to give the underlying virtual RAM back to the kernel for an area that should've been kept, and causing memcpy() to fail with an access violation because an area it should've been able to access can't be accessed).
The other cases are external failure conditions. If the OS decided to send some of the data to swap space but gets a read error from the device when trying to fetch the data back from swap space when you try to access it, there's very little to OS can do about it (your process and any other process using that data can't continue correctly). If you're using ECC RAM and the memory controller says there's an uncorrectable error with the RAM you're using it's similar. It's also possible for the OS to use "lazy page allocation" (e.g. pages of memory are allocated when you write to them and not when you thought you allocated them) and "over commit" (pretend that more pages were allocated than can be provided), so that when memcpy() writes to an area that was allocated the OS can't handle it (e.g. it triggers an "OOM/out of memory killer" that terminates a process to free up some RAM). Finally, it's possible for the code to be corrupted (e.g. faulty RAM without ECC, malicious attack like "Rowhammer", corrupted shared library, ...) so that (e.g.) using memcpy() causes a SIGILL. Of course all of these things aren't related to memcpy() itself and can just as easily happen anywhere else.
Also, there will be issues in case of overlapping pointers.
Yes. Some (most) implementations of memcpy() are optimised to copy larger blocks (e.g. optimised to use SSE on 80x86 and moving 16 bytes at a time) where if the areas overlap the data gets mangled. Some (most) implementations of memcpy() assume that it can copy data in one specific direction which will cause data to be corrupted if areas overlap in the wrong way (e.g. if the implementation uses the "lowest address first" direction and the destination area overlaps and is at a higher address than the source, then writes to the destination will overwrite source data that hasn't been copied yet).
Apart from these, are there any other ways the memcpy call can fail?
No, I think I covered all the possible failure cases above.
Or we can consider it'll pass all the time without any error. How to verify it?
For the "overlapping areas" problem it shouldn't be hard to write a wrapper around memcpy() that detects overlap and generates an error (so that it doesn't silently corrupt data). Unfortunately this only finds problems at run-time (after it's too late - e.g. possibly after it's been released and running on the end user's computer). For some of the cases might be easy enough to detect "overlapping areas" using a static source code analyser, but these cases are likely to be the "easily detected by testing at run-time before software is released" cases.
For some things (dodgy pointers, corrupted heap) there are tools (valgrind) to detect problems. Unfortunately these only detect problems when they actually happen (and don't detect problems that don't happen during testing but do happen when software is running on the end-user's computer).
For the remainder (OS failures and hardware failures), if you can't trust the OS or hardware then you can't assume any code that verifies anything will work properly either.
I just went through a bunch of stack vs heap threads here on Stack Overflow (and other random sites found through Google), and yet I can't find an answer that provides much (if any) depth to the issue of read/write speed (most answers and articles focus on allocation speed).
If I'm writing a C program (in my case, a game) where I'm storing object data in a buffer that's a few kilobytes in size, will it make a difference (in terms of access speed) if that buffer is allocated on the stack or on the heap? Or will the compiler treat a buffer of that size the same either way when it comes to deciding where on the hardware (i.e. cache or RAM) to store it?
On a typical modern CPU and OS, no, I wouldn't expect any difference in average access time, likelihood of cache miss, or (much worse) likelihood of page fault based on whether the buffer is in heap or stack. The only thing that should matter is your pattern of access to it, compared to other use of memory. However, there may be embedded or special-purpose platforms where the stack is kept in a different kind of memory than the heap.
You're ignoring a third possibility, non-heap memory (allocated by some OS-specific call, such as mmap, rather than malloc).
Finally, if you want to completely remove the possibility of page faults, the OS may provide a way to do that; for example, mlock on POSIX.
There is no difference in the access speed, its the same memory after all. Compiler never decide where buffer should be stored physically. It is hardware and operating system business.
No at all. heap ad stack are areas in the virtual address space that the operating system gives t your app. Their difference is what your app stores in the stack and what in the heap. All the reference types are stored in the heap and all the value types are stored in stack. Furthermore the lifetime of the things that are stored in the stack has the same duration as the lifetime of your app, while types stored in the heap may be garbage collected and the memory that they were using be reclaimed.
I have been playing around with the idea that memcpy() could be used for malevolent purposes. I made several test applications to see if I could "steal" data in memory from different regions. I have tested three regions thus far, heap, stack, and constant( read only ) memory. The constant memory was the only one to crash in my tests, provoking an error from MinGW.
Here is an example to illustrate my latest test :
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
void removeTerminatingCharacters( char ** string, const int length )
{
int i = 0;
for ( ; i < length; ++i )
if ( !( *string )[i] )
( *string )[i] = '0';
return;
}
int main()
{
int * naive = malloc( sizeof( int ) );
*naive = 0;
char * stolenData = malloc( 2000 );
memset( stolenData, 0, 2000 );
memcpy( stolenData, naive, 1999 );
removeTerminatingCharacters( &stolenData, 2000 );
printf( "%s\n", stolenData );
free( stolenData );
return 0;
}
Output :
0000-0:0Væ1lDk¦#:00000[æ0`Dk00p,:0-0:0MAIN=Computer0USERNAME=JohnDoe0USERPRO
FILE=C:\Users\JohnDoe0WATCOM=C:\watcom0windir=C:\Windows00?æ1+Ik?000S?000000
00000000000000000000000000000000000000000000000000000000000000000000000000000000
00??????????????????????????000000 00000000 000000?0?0?
00000000000 0 0 ?0000000000 0000000000 0000 00000???????????????????????0???????
0 00000000000000000000000000000000000000000000000
000000000000000000abcdefghijklmnopqrstuvwxyz000000ABCDEFGHIJKLMNOPQRSTUVWXYZ0000
0000â000000Ü0£0P00000000000è0î0Ä 0000000000¬0000000000¦0000¦00000aßGpSsµtFTOd8fe
ä?:0ú?:0-?:0T?:0)?:0`?:0£?:0-?:0p?:0?:0??:0+?:08?:0T?:0¦?:0µ?:0²?:0¶?:03?:0D?:0R
?:0ä :0- :0+ :0a :0v :0?!:0^!:0q!:0ë!:0ñ!:0+!:0±!:0¤":0P":0g":0ó":0¦":0+":0¦":0?
#:0.#:0B#:0W#:0x#:0ë#:00000ALLUSERSPROFILE=C:\ProgramData0APPDATA=C:\Users\Chris
topher\AppData\Roaming0asl.log=Destination=file0CLASSPATH=.;C:\Program Files (x8
6)\Java\jre6\lib\ext\QTJava.zip0CommonProgramFiles=C:\Program Files (x86)\Common
Files0CommonProgramFiles(x86)=C:\Program Files (x86)\Common Files0CommonProgram
W6432=C:\Program Files\Common Files0COMPUTERNAME=COMPUTER0ComSpec=C:\Windows\sys
tem32\cmd.exe0FPPUILang=en-US0FP_NO_HOST_CHECK=NO0HOMEDRIVE=C:0HOMEPATH=\Users\C
hristopher0HuluDesktopPath=C:\Users\JohnDoe\AppData\Local\HuluDesktop\instan
ces\0.9.13.1\HuluDesktop.exe0LOCALAPPDATA=C:\Users\JohnDoe\AppData\Local0LOG
ONSERVER=\\COMPUTER0NUMBER_OF_PROCESSORS=20OnlineServices=Online Services0OOBEUI
Lang=en-US0OS=Windows_NT0Path=.;F:\CodeBlocks\MinGW\bin;F:\CodeBlocks\MinGW;C:\M
inGW\bin;C:\MinGW;C:\Windows\System32;C:\Windows;C:\Windows\System32\wbem;C:\Pro
gram Files\Common Files\Microsoft Shared\Windows Live;C:\Program Files (x86)\Com
mon Files\Microsoft Shared\Windows Live;C:\Windows\System32\WindowsPowerShell\v1
.0;c:\Program Files (x86)\ATI Technologies\ATI.ACE\Core-Static;c:\Program Files
(x86)\Common F0Uô1m+k
My code isn't pretty, but it demonstrates my point. As you can see, the data is mostly garbage values, but there are a few interesting strings thrown in from the heap.
My primary question is why doesn't this action cause a memory access violation error?
Memory access violation errors are caught by virtual memory hardware, when you access an unmapped page. Not every address which is out of bounds is in an unmapped page. Pages are generally of equal sizes. A typical page size if 4096 bytes. (Page sizes are hardware-specific: some chips have memory management that allows for programmable page sizes, and even mixtures of different page sizes for different areas of memory.) Sometimes only part of a page contains valid data. It's not possible for just a fragment of a page to be unmapped, so the part which contains garbage is also mapped. Also, memory managers like malloc do not always give memory back to the operating system; they keep free areas for re-use. Those areas are valid memory (mapped pages). Also, making up pointer values, you can, by fluke, simply end up going beyond out-of-bounds, and can "land" in memory that corresponds to a valid object.
This is just how it works on your PC, with hits virtual memory OS. Virtual memory is not ubiquitous. On computers without virtual memory (nowadays, small embedded systems), you can access any location in memory. However, accessing certain areas may have side effects that change the state of hardware (namely I/O registers). Some address ranges may trigger a "bus error" type CPU exception because no hardware exists for that range, and so the access request times out. Other than that, as far as valid program memory goes, it is not protected from out-of-bounds accesses.
Operating systems without memory protection were used for early interactive systems in the 1960's. That history then repeated itself when personal microcomputers appeared: again, they had operating systems without memory protection, due to small memories and unsophisticated CPU's. On these operating systems, applications often stomped over each other's memory spaces, leading to frequent crashes. (Imagine that with your memcpy, you not only copy your own out-of-bounds area, such as some malloc block which was previously freed, but also an area from a completely different running program.) Users sometimes spotted patterns, like when certain programs were loaded into memory in certain orders, there were fewer problems, or so-called "conflicts" between applications.
Yes, it does can be used to malevolent purposes, but not in the way you may be thinking.
The memory your application read and write is your process' virtual memory, it is not the physical memory. You can't even know in what address of the physical memory yours or any application really is, only the system kernel knows that.
You cannot interact with active memory from other processes without proper permission and use of syscalls aswell.
You can howover, read the garbage left in unmapped memory areas that were once used by other processes, and you may even stumble in sensitive information left there, like passwords, certificate keys or personal stuff the user typed at some point, but you are on highly volatile grounds, the information there is most likely corrupted and there is no easy way to seek specific pieces of information.
Here is an article about that: https://security.stackexchange.com/questions/29019/are-passwords-stored-in-memory-safe
You must not try to access out of bounds memory.
Accessing memory you must not access results in Undefined Behavior.
Anything may happen.
You may be able to "steal" the memory. But that's not something you should rely on and write your code based on that.
memcpy() doesn't do any bounds check, the programmer has to take care of it. In your case, it causes a heap overflow.
This pretty much sums up the consequences of a heap overflow:
https://www.owasp.org/index.php/Heap_overflow
Description
A heap overflow condition is a buffer overflow, where the buffer that
can be overwritten is allocated in the heap portion of memory,
generally meaning that the buffer was allocated using a routine such
as the POSIX malloc() call.
Consequences
Availability: Buffer overflows generally lead to crashes. Other attacks leading to lack of availability are possible, including
putting the program into an infinite loop.
Access control (memory and instruction processing): Buffer overflows often can be used to execute arbitrary code, which is
usually outside the scope of a program's implicit security policy.
Other: When the consequence is arbitrary code execution, this can often be used to subvert any other security service.
Avoidance and mitigation
Pre-design: Use a language or compiler that performs automatic bounds checking.
Design: Use an abstraction library to abstract away risky APIs. Not a complete solution.
Pre-design through Build: Canary style bounds checking, library changes which ensure the validity of chunk data, and other such fixes
are possible, but should not be relied upon.
Operational: Use OS-level preventative functionality. Not a complete solution.
Discussion
Heap overflows are usually just as dangerous as stack overflows.
Besides important user data, heap overflows can be used to overwrite
function pointers that may be living in memory, pointing it to the
attacker's code.
Even in applications that do not explicitly use
function pointers, the run-time will usually leave many in memory. For
example, object methods in C++ are generally implemented using
function pointers. Even in C programs, there is often a global offset
table used by the underlying runtime.
To answer your question of "why doesn't this action cause a memory access violation error?", the answer is that it's because the memory in the area for the naive allocation has been allocated to the process. Probably it's memory that the C runtime has prepared for further possible malloc() requests.
On most desktop systems, the smallest unit of memory protection is a page, which can vary in size but will often be in the range of 4KB or so. You might well have run into a memory access violation if your read request were larger (but still maybe not - the heap might be much larger than your small allocation). Then again, your allocation could come near then end of a page and hit an access error right away (debug allocators will sometimes do this purposefully to help catch errors).
memcpy can't realistically be used for "malevolent" purposes. The only memory that memcpy can access is the memory belonging to your own process (which you own, anyway). If you screw up your own memory, your program takes the fall, but the OS will protect all other processes (and you cannot touch their memory spaces without mmap or similar).
Most of those "interesting strings" are environment variables set by the OS and passed to your program.
The question is self-explanatory (I think). If a program is accessing memory, say 4 bytes at a time, when does the memory happen to be non-readable, as opposed to say, hold garbage? Thanks.
Whenever the program is not allowed to read it.
This is the job of the MMU to allow or forbid access on memory. This is the job of the OS to specify which program is allowed to access which memory zone.
If you allocate/deallocate/access memory correctly, then you will never see this. You will only encounter this when you have done something wrong.
Typically, malloc is implemented with a sub-allocating memory manager. If you ask malloc for 4 bytes of heap memory, say, then the memory manager in the C runtime will allocate a larger block and then sub-allocate 4 bytes within that block to you. Subsequent requests for small amounts of memory will then be sub-allocated from one of these larger blocks.
You can read and write into areas of these large blocks of memory that have not yet been sub-allocated by malloc. Doing so is of course undefined behaviour. Please don't do this! You can also read and write into sub-blocks that have been freed, so long as the larger block has not been returned to the system. Again, please don't do this.
Most commonly a program will fault with a non-readable memory error (a.k.a. segmentation fault or access violation) when it tries to access an address that has been freed and the block of memory containing that address has been returned to the system. This is known as a stale pointer.
In practice, if you are engaging in correctly aligned read operations only, you have nothing to worry about. On real-world hardware, access granularity is always at the level of pages at least 4k in size. Mathematically, if a pointer p lies in a valid page, and p is a multiple of some alignment value m which divides the page size n, then
(p+0)/n = (p+1)/n = ... = (p+m-1)/n
i.e. p, p+1, ..., p+m-1 all point within the same page.
If p is misaligned to begin with, you have much bigger portability problems than the possibility of reading from an unmapped or unreadable page.
Can accessing (for read only) memory freed cause an access violation, and, if so, under what circumstances?
Yes, it can. "Access violation" ("segmentation fault", etc) is the response that is normally generated by OS/hardware when the process attempts to access (even just for reading) memory that is known to OS as "empty", "freed" or inaccessible for some other reason. The key moment here is that the OS/hardware must know that the memory is free. Memory management functions of C Standard Library don't necessarily return the 'free'd memory back to OS. They might (and will) keep it for future allocations. So in some cases accessing 'free'd memory will not result in "Access Violation" since from the OS's/hardware's point of view this memory has not been really freed. However, at some point the Standard Library might decide to return the collected free memory back to OS, after which an attempt to access that memory will normally result in "Access Violation".
You're asking "can" and not "will", so your answer is yes. It is undefined behavior to point to memory not owned by your program, therefore anything could happen.
Will it? Depends. That is very OS specific. You might be able to get away with it, but obviously you cannot depend on this. Trying to dereference it could cause an exception, because the OS has reclaimed the memory for it's own uses. (again, OS specific).
On Windows: Managing Virtual Memory in Win32
Free, Reserved, and Committed Virtual Memory
Every address in a process can be
thought of as either free, reserved,
or committed at any given time. A
process begins with all addresses
free, meaning they are free to be
committed to memory or reserved for
future use. Before any free address
may be used, it must first be
allocated as reserved or committed.
Attempting to access an address that is either reserved or free generates
an access violation exception.
Unlikely
Memory managers can theoretically return space to the OS but rarely if ever do so. And without returning the space all the way to the kernel, the MMU will never be involved and so a fault is not possible.
The problem is fragmentation. Allocation of variably-sized blocks is inefficient, and general purpose allocators cannot move allocated blocks around because they don't know what points to what, so they can't coalesce blocks unless they happen to be adjacent.
Measurements indicate that fragmentation overhead tends to be about 50% in steady-state processes, so with every-other-block untouchable it's impossible to return pages unless they are much smaller than blocks, and they generally are not.
Also, the book-keeping challenge of returning pages embedded within the heap is daunting, so most memory managers don't even have the capability, even in the unlikely case that they would have the opportunity.
Finally, the traditional process model was not a sparse object. This kind of low-level software is conservatively developed and lives for a long time. An allocator developed today just might attempt sparse allocation but most subsystems just use whatever is already in the C library, and that is not something that's wise to rewrite casually.
It's certainly allowed to; the C standard says straightforwardly that behavior is undefined if "The value of a pointer that refers to space deallocated by a call to the free or realloc function is used". In practice, owing to the way OSs work, you're more likely to get garbage than a crash, but quite simply you're not allowed to assume anything about what happens when you invoke undefined behavior.
freed memory doesn't belong to you anymore, exactly, corresponding physical memory page is out of your process address space which might have been remapped to other process address space already after your freeing, and that address you accessing have not been allocated physical page and do mapping yet; so "Access violation" or "segfault" will happen if access it even for reading only. it is triggered by the processor hardware in general, e.g. GP#, not by OS.
though if the specific physical page which owns your freed memory is still under controlling of your task context, say partial of the page is still used by your process, then "Access violation" or "segfault" may not occur.
If you're asking this because you've profiled your code and found that accessing freed memory would provide a significant performance boost, then the answer is very rarely, if the freed block is small. If you want to be sure, provide your own alternative implementation of malloc() and free().
We can access it but not encouraged. for example
void main()
{
char *str, *ptr;
str = (char *)malloc(10);
ptr = str;
strcpy(str, "Hello");
printf("%s", str);
free(str);
printf("%s", ptr);
}