Protect memory on x86-64 - c

Question

I'm implementing a compiler which compiles a simulated processor's instruction set to code in x86 instructions. No physical processor exists which runs on the to be simulated instruction set; there is just the simulation on x86. When executing the compiled machine code (the simulation), I want to make sure that only memory designated for the simulation is read or written. This serves two purposes: 1) An access outside of the designated memory area could mean that I have a bug in my compiler. I want to find such bugs. 2) An access outside of the designated memory area can also mean that the source instructions that I compiled have a logical error and therefor try to access a memory address which does not exist in my simulation, thus an error should be raised.
In a simpler form, you can imagine my code to look like this:
void simulate(char* designated_memory, size_t len) {
// code intended to access *designated_memory till *(designated_memory + len - 1) only
}
Is there a way on x86-64 and/or in linux to enforce that simulate() can only access its own stack and designated_memory, and any other access would generate an error. E.g. the code could look like this:
restrict_access_to(designated_memory, designated_memory + len - 1);
simulate(designated_memory, len);
remove_access_restriction();
A solution in C would be nice; asm is fine, too.
UPDATE:
Following Jester's comments, I came to try out this:
#include <stdio.h>
#include <unistd.h>
#include <malloc.h>
#include <sys/mman.h>
int main() {
size_t pagesize = sysconf(_SC_PAGESIZE);
printf("pagesize...........: %lu\n", pagesize);
char* m;
size_t len = 12345;
len = (len + pagesize - 1) / pagesize * pagesize;
posix_memalign(&m, pagesize, len);
printf("page aligned memory: %lx - %lx\n", (unsigned long) m, (unsigned long) m + len);
printf("protecting 0 till m..."); fflush(stdout);
mprotect(0, (size_t) m, PROT_NONE);
printf("done\n");
printf("protecting (m + len) till ?..."); fflush(stdout);
mprotect(m + len, 0x7fffffff, PROT_NONE);
printf("done\n");
printf("trying to modify memory..."); fflush(stdout);
*(m - 1000) = 5;
printf("done: %i\n", *(m - 1000));
free(m);
}
Which outputs:
pagesize...........: 4096
page aligned memory: 9ac000 - 9b0000
protecting 0 till m...done
protecting (m + len) till ?...done
trying to modify memory...done: 5
Segmentation fault (core dumped)
I think that this shows that modifying data outside of the allowed area still works, but that should not happen.

Answer 1

i do not use an interpreter. i use a compiler to compile the simulated >instructions to x86 instructions. i do not want to include all the range checking >in the compilation to x86 instructions for performance reasons.
So your task is very similar to what does qemu, when it does emulation for example ARM on x86, it convert arm instructions to x86 instructions,
so I suggest you look at source code of qemu: http://wiki.qemu.org/Main_Page
The next very similar to your project is valrgrind,
the way how valgrind works is very similar to what you do:
it execute program on some kind of virtual cpu to check memory accesses,
and to speedup things it uses jit (http://valgrind.org/).
And the last opensource project, that solve similar problems to your is
https://github.com/google/sanitizers/tree/master/address-sanitizer
Yes, it is instrumented code, but the result works much faster then valgrind,
the idea how to instrument generated code, but leave performance on suitable level you can find in this video about asan internals:
https://www.youtube.com/watch?v=Q2C2lP8_tNE

Answer 2

If you are executing arbitrary assembly instructions in user-space code, there isn't really a way to enforce protection of the memory -- memory protection generally requires having the ability to activate kernel-mode flags on the CPU.
However, since you are writing a simulator, and compiling another language to a set of assembly instructions of your choosing, you have a different option: control the instructions being emmitted. Instead of emitting raw memory access instructions for simulated memory accesses, have the simulation compiler replace instructions which would access memory with instructions which call a memory-access function of your own design, and implement memory access protection in that function. This could also be done as Lorehead states by having the memory region be something like a std::vector.
Note that just replacing normal move instructions with your own move function isn't really sufficient to protect your own wrapper code from being hacked by "simulated" code. If your simulation emits raw jump commands, then it can be broken by simulated code which jumps outside of the simulation. Or by carefully planned push/pop instructions which cause the wrapper code to return to other simulator code later, after you should have left the simulation.
To actually be (somewhat) safe (this is by no means exhaustive, or guaranteed to be sufficient to make the simulated code safe), you would need to make sure that your simulation compiler generates safe code; things which affect what memory may be accessed, including mov, jump, push, pop, call, and ret would need to be replaced with function calls that perform equivalent actions safely, rather than just executing assembly code that you can't trust.
You would also need to make sure to wrap this in code which saves your registers for your outer program, since the inner code may arbitrarily change their contents.

Answer 3

You write that you don’t want to add runtime bounds checking for performance reasons. I suggest you look closely at the actual runtime cost and reconsider: it’s literally a single instruction if you use indirect addressing (ja) which even the simplest branch predictors can get right if you put the exception handler last. If you can recognize and optimize (some) loops, you can do one bounds check before the loop, instead of one before every access inside the loop.
If you really can’t afford this overhead, one option you can try is to compile the executable and fork off a process that has no other memory in its address space. If you’re going to run in a process that uses the stack, though, it’s theoretically possible that native code could clobber the stack. You can also make it impossible to access memory outside the address space if you make the size of the address space exactly 2^16 or 2^32 bytes and have the compiled code access it with that size register. Otherwise, you’re stuck trying to poison every single other page of the address space.