I am writing a 64-bit application in C (with GCC) and NASM under Linux.
Is there a way to specify, where I want my heap and stack to be located. Specifically, I want all my malloc'ed data to be anywhere in range 0x00000000-0x7FFFFFFF. This can be done at either compile time, linking or runtime, via C code or otherwise. It doesn't matter.
If this is not possible, please explain, why.
P.S. For those interested, what the heck I am doing:
The program I am working on is written in C. During runtime it generates NASM code, compiles it and dynamically links to the already running program. This is needed for extreme optimization, because that code will be run thousands-if-not-billions of times, and is not known at compile time. So the reason I need 0x00000000-0x7FFFFFFF addresses is because they fit in immediates in assembler code. If I don't need to load the addresses separately, I can just about half the number of memory accesses needed and increase locality.
For Linux, the standard way of acquiring any Virtual Address range is using the mmap(2) function.
You can specify the starting virtual address and the size. If the address is not already in use and it not reserved by prior calls (or by the kernel) you will get access to the virtual address.
The success of this call can be checked by comparing the return value to the start address you passed. If the call fails, the function returns NULL.
In general mmap is used to map virtual addresses to file descriptors. But this mapping has to happen through physical pages on the RAM. Since the applications cannot directly access the disk.
Since you do not want any file backing, you can use the MAP_ANONYMOUS flag in the mmap call (also pass -1 as the fd).
This is the excerpt for the related part of the man-page -
MAP_ANONYMOUS
The mapping is not backed by any file; its contents are
initialized to zero. The fd argument is ignored; however,
some implementations require fd to be -1 if MAP_ANONYMOUS (or
MAP_ANON) is specified, and portable applications should
ensure this. The offset argument should be zero. The use of
MAP_ANONYMOUS in conjunction with MAP_SHARED is supported on
Linux only since kernel 2.4.
Related
I want to make sure the return address of sbrk is within a certain specific range. I read somewhere that sbrk allocates from an area allocated at program initialization. So I'm wondering if there's anyway I can enforce the program initialization to allocate from a specific address? For example, with mmap, I'll be able to do so with MAP_FIXED_NOREPLACE . Is it possible to have something similar?
No, this is not possible. brk and sbrk refer to the data segment of the program, and that can be loaded at any valid address that meets the needs of the dynamic linker. Different architectures can and do use different addresses, and even machines of the same architecture can use different ranges depending on the configuration of the kernel. Using a fixed address or address range is extremely nonportable and will make your program very brittle to future changes. I fully expect that doing this will cause your program to break in the future simply by upgrading libc.
In addition, modern programs are typically compiled as position-independent executables so that ASLR can be used to improve security. Therefore, even if you knew the address range that was used for one invocation of your program, the very next invocation of your program might use a totally different address range.
In addition, you almost never want to invoke brk or sbrk by hand. In almost all cases, you will want to use the system memory allocator (or a replacement like jemalloc), which will handle this case for you. For example, glibc's malloc implementation, like most others, will allocate large chunks of memory using mmap, which can significantly reduce memory usage in long-running programs, since these large chunks can be freed independently. The memory allocator also may not appreciate you changing the size of the data segment without consulting it.
Finally, in case you care about portability to other Unix systems, not all systems even have brk and sbrk. OpenBSD allocates all memory using mmap which improves security by expanding the use of ASLR (at the cost of performance).
If you absolutely must use a fixed address or address range and there is no alternative, you'll need to use mmap to allocate that range of memory.
According to mmap's manual:
MAP_NORESERVE
Do not reserve swap space for this mapping. When swap space
is reserved, one has the guarantee that it is possible to modify
the mapping. When swap space is not reserved one might get
SIGSEGV upon a write if no physical memory is available.
To my understanding, if a regular file is mapped into the virtual address range, there is no need for any swap space. Only MAP_ANONYMOUS may need some swap space.
So, is it correct to always add MAP_NORESERVE flag in mmap for a regular file?
Update:
To be more specific, is it correct to always add MAP_NORESERVE flag in mmap for a regular file, when MAP_SHARED is used?
To my understanding, if a regular file is mapped into the virtual address range, there is no need for any swap space. Only MAP_ANONYMOUS may need some swap space.
That depends on the mmap flags. If a regular file is mapped with MAP_PRIVATE then the memory region is initialized from the file but not backed by the file. The system will need swap space for such a mapping if it decides to swap out any of its pages.
So, is it correct to always add MAP_NORESERVE flag in mmap for a regular file?
It is not incorrect to specify MAP_NORESERVE for any mapping. It's simply a question of what guarantees you want to have about program behavior.
Moreover, you seem looking at this from the wrong direction. If a particular mapping can never require swap space then the system will not reserve swap space for it, regardless of the flags. It doesn't hurt to use MAP_NORESERVE in such a case, but it doesn't help, either, so what would be the point?
On the other hand, if you want to be sure that mapping cannot fail on account of using MAP_NORESERVE then the most appropriate course of action is to avoid using that flag. You can completely ignore its existence if you wish, and in fact you should do so if you want maximum portability, because MAP_NORESERVE is a Linux extension not specified by POSIX.
Update:
As I understand it, you are asserting that you can reproducibly observe successful mapping of existing ranges of existing files with MAP_SHARED to require MAP_NORESERVE. That is, two such mapping attempts that differ only in whether MAP_NORESERVE is specified will produce different results for you, in a manner that you can predict and reliably reproduce.
I find that surprising, even dubious. I do not expect pages of a process's virtual address space that the page table maps to existing regions of a regular file to have any association with swap space, and therefore I do not expect the system to try to reserve any swap space for such pages when it establishes a mapping, flags notwithstanding. If you genuinely observe different behavior then I would attribute it to a library or kernel bug, about which you should file an issue.
Consider, for example, the GNU libc manual, which explicitly says that a memory mapping can be larger than physical memory and swap space, but does not document Linux-specific MAP_NORESERVE at all.
With that said, again, it is not incorrect (on Linux) to specify MAP_NORESERVE for any given memory mapping. I expect it to be meaningless for a MAP_SHARED mapping of an existing region of a regular file, however, so I would not consider it good -- much less best -- practice to routinely use that flag for such mappings. On the other hand, if specifying that flag works around a library or kernel bug that otherwise interferes with establishing certain mappings then I see no particular reason to avoid doing that, but I would expect each such use to be accompanied by a documentary comment explaining why that flag is used.
Is there a way to malloc memory space and then copy function code inside the space in C?
This question might not make sense in practice. I ask this question out of curiosity so that I can get a better understanding about how c and its underlying implementation work.
Here's the follow-up questions if it is possible to copy the code into heap:
How to determine the size for the function binary code when copy?
Can we use function pointer to execute the code? (the code is placed inside malloc'd memory, and that part of memory might be marked as non-executable for safety reason, but I'm not sure about this)
This (or something like it) is possible on most machines, but the techniques you'd use are system-specific -- there's no standard C or C++ way to do it.
Even figuring out the length of a function so you can copy it is difficult. I don't think you can do it reliably if the function is in the same translation unit, because the compiler may have done optimization magic that you can't see. However, if the function is in a different file, then the interface to it will probably be more reliable (although there could be linker magic going on that you would have to understand and emulate to accomplish your goal.)
Other problems (on some systems) are that malloc'd memory may not be executable. (This is often the case to improve security by preventing execution of code placed in an overrun buffer area.) However, systems with executable protection often have an alternate memory allocation function that can give you a chunk of memory where executable code can be placed, and to which execution can transfer. Some variation of this feature is necessary to implement shared libraries.
Finally, although self modifying code is probably the first thing people probably think of when considering your question, a reasonable, legitimate use of the relevant techniques might be in a native-code, just-in-time compilation system.
You may get better answers by specifying a particular OS and CPU where you want to do this.
The C standard (e.g. C11, read n1570) or the C++ one (e.g. C++11, C++14 and notice that they have lambda expressions and std::function; read more about closures ...) does not define what is a function address or pointer (it only defines what calling such an address does, then function pointers should point to existing functions and there is no standard way to build new ones dynamically at runtime). In some systems (pure Harvard architectures) a function sits in a different address space than the C heap (and on these systems executing anything in malloc-ed heap makes no sense and is undefined behavior). so the C11 standard forbids casting function pointers to data pointers and vice-versa.
So, to your question
Is there a way to malloc memory space and then put function code inside the space in C?
the answer is NO in general (but on some systems you could generate code at runtime, see below).
However, on desktop or laptop PCs or server PCs or tablets (running common OSes like Linux, Windows, MacOSX, Android), you usually have a Von Neumann architecture and there is (for a given process) a single virtual address space sharing both code and data (notably heap data obtained with malloc). That virtual address space organised in pages, and each page has its own memory protection. Read more about computer architecture, instruction sets, MMUs. Quite often heap allocated data is non-executable thru the NX bit.
The operating system plays an essential role. You need to read an entire book about OS, such as Operating Systems : Three Easy Pieces.
(I am guessing that you want to "create" some new functions in your program at runtime and call them thru C function pointers; you should explain why; I suppose you are coding some application for a PC or a tablet with a Unix-like OS, practically a Linux-x86_64 distribution, but you could adapt my answer to Windows)
You could use some libraries for JIT compilation such as asmjit, libgccjit, LLVM (or libjit or GNU lightning) and they generate code which is executable.
You could also use dynamic loading techniques on some plugin; on POSIX systems look into dlopen & dlsym (which can be used to "create" function addresses from a loaded plugin, beyond what the C11 standard allows). A possible way would be to generate some C code in a temporary file, compile it into a plugin, and dlopen that generated plugin. See this answer for more details.
On Linux, you can use the mmap(2) and related system calls (used to implement malloc in your C standard library, and also by dlopen(3)) to change your virtual address space, and the mprotect(2) system call to change protection (on a page by page basis). So if you want to explicitly copy or generate some function code it has to go into an executable page (PROT_EXEC).
Notice that because of relocation issues (and offsets or absolute addresses in machine code), it is not easy to copy machine code. Copying with memcpy the bytes of a given function code into some executable page usually won't work without pain: often CALL or JUMP machine instructions are using PC-relative addressing, so copying them without changing their offset won't work.
if it is possible to copy the code into heap
No, it is not possible in general; and in practice it is much more difficult than what you believe (even on Linux-x86_64, where other approaches that I mentioned are preferable); if you want to go that route you need to care about low level implementation details (instruction set, processor, compiler, calling conventions, ABIs, relocation) and your code would be non-portable and brittle.
How to determine the size for the function binary code when copy?
That question (and the notion of function size) has no sense in general. Some optimizing compilers are able to emit some machine code which is shared between several C functions, or to emit several non-contiguous machine code chunks for a given function (and gcc -O2 is likely to do these optimizations, read about function cloning). On Linux you could use dladdr(3) (or the nm or readelf programs) to get a "symbol size" in the ELF sense, but that size might not mean much. And as I explained, you can't just byte-copy binary machine code, you need to relocate (some parts of) it.
I need to be able to unmap files that were opened through some libraries I'm linking with. The reason for needing to do this is that the mappings made by these libraries hold references to modules that may need to be reloaded while the program is executing (potentially long running executions). The problem is the modules cannot be unloaded while my process is holding a reference.
I've written C code to parse the information in proc/self/maps in an effort to read the address range of the mappings and calculate its length. I've calculated the length by subtracting the starting address from the ending address then I pass the starting address and the calculated length as the respective parameters to munmap. The problem is that munmap fails with EINVAL (Invalid Argument).
I've checked the size of the page my machine uses with sysconf(_SC_PAGESIZE) and it returned 4096, which is the value of my calculated length. The GNU manual says that munmap can fail with EINVAL if:
The memory range given was outside the user mmap range or wasn’t page
aligned.
Am I missing anything or is this at all not possible? My last resort would be to carefully comb through the system calls made and examine each mmap via strace, but I'd like for this to be a last resort, thanks.
The method you have chosen will not work, but, not necessarily for the reason(s) you think. However, there is a way, so read on ...
I need to be able to unmap files that were opened through some libraries I'm linking with.
Once you've let the ELF loader (e.g. ld.linux.so) load the libraries for you, you've lost control.
You can not just unmap the area [regardless of method]. The loader has already done relocations and symbol linkages for these libraries. The unmap removes the area, but now everything breaks because the various pointers set up by the linker now point to empty space. The loader will have no knowledge of what you've done.
Then, how would you remap the new version of the library [and where in memory]? Even remapping it to the same address is no guarantee because you can't adjust the things that the loader has already done.
The reason for needing to do this is that the mappings made by these libraries hold references to modules that may need to be reloaded while the program is executing (potentially long running executions).
Most programs that need to update to new versions of libraries simply reexec themselves. If you need to preserve the data, you can work out a dump/restore mechanism.
However, if you truly want to unload/load a newer library version, you can do this using dynamic linking.
Instead of using ld to link with (e.g.) libA, leave if off the ld command line and have the program do its own loading of libA.
You use dlopen/dlsym/dlclose to open/load the library under your control.
You'll have to keep track of the symbol tables, but changing to the new version is easy. When you want the new version, simply do dlclose and then a dlopen. You'll have to redo the dlsym calls to get the updated addresses, but all this is fairly easy and standard
The problem is the modules cannot be unloaded while my process is holding a reference.
The reason is that ELF loader did this. With dlopen et. al., you don't have the same problem.
Yes it is possible. I had a closer look at my code; there was something I was misunderstanding when it came to passing the start address of the mapping to munmap. I had initially read the start address into an unsigned long long and for some reason, I was converting this value into a hex string instead of just casting to a void pointer when calling munmap. In essence:
/* Values assigned here are really read from /proc/self/maps */
unsigned long long vm_start = 140013986873344;
unsigned long long vm_end = 140013986877440;
unsigned long long length = vm_end - vm_start;
munmap((void *)vm_start, length);
I've been looking at the different flags for the mmap function, namely MAP_FIXED, MAP_SHARED, MAP_PRIVATE. Can someone explain to me the purpose of MAP_FIXED? There's no guarantee that the address space will be used in the first place.
MAP_FIXED is dup2 for memory mappings, and it's useful in exactly the same situations where dup2 is useful for file descriptors: when you want to perform a replace operation that atomically reassigns a resource identifier (memory range in the case of MAP_FIXED, or fd in the case of dup2) to refer to a new resource without the possibility of races where it might get reassigned to something else if you first released the old resource then attempted to regain it for the new resource.
As an example, take loading a shared library (by the dynamic loader). It consists of at least three types of mappings: read+exec-only mapping of the program code and read-only data from the executable file, read-write mapping of the initialized data (also from the executable file, but typically with a different relative offset), and read-write zero-initialized anonymous memory (for .bss). Creating these as separate mappings would not work because they must be at fixed relative addresses relative to one another. So instead you first make a dummy mapping of the total length needed (the type of this mapping doesn't matter) without MAP_FIXED just to reserve a sufficient range of contiguous addresses at a kernel-assigned location, then you use MAP_FIXED to map over top of parts of this range as needed with the three or more mappings you need to create.
Further, note that use of MAP_FIXED with a hard-coded address or a random address is always a bug. The only correct way to use MAP_FIXED is to replace an existing mapping whose address was assigned by a previous successful call to mmap without MAP_FIXED, or in some other way where you feel it's safe to replace whole pages. This aspect too is completely analogous to dup2; it's always a bug to use dup2 when the caller doesn't already have an open file on the target fd with the intent to replace it.
If the file you are loading contains pointers, you will need to load it at a fixed location in order to ensure that the pointers are correct. In some cases, this can merely be an optimization.
Executables which are not position-independent must be loaded at fixed addresses.
Shared memory may contain pointers.
Executables which use prebinding will attempt to load dynamic libraries at predetermined memory locations as an optimization, but will fall back to normal loading techniques if a different location is used (or if the library has changed).
So MAP_FIXED is not typical usage.