mmap() for Remote File - c

Question

Currently I am implementing a version of mmap() which its objective is to map a remote file on a client machine. For the implementation, I cannot use any in-built or third party libraries. Having said that, I am in doubt whether the implementation will be based on either of the following two options:
Load the file on the client machine after reading the file contents from the client side and use the mmap() syscall by using the file descriptor obtained from the client machine or
Allocating memory for each chunk of file data received by the client side by using sbrk()
Any suggestions will be greatly appreciated!

Answer 1

This is quite possible to do in Linux, and even in a thread-safe fashion for a multithreaded process, but there is one very difficult function you'd need to implement either yourself, or by using some library.
You would need to decode and emulate any memory-accessing instruction, using an interface similar to
static void emulate(mcontext_t *const context,
void (*fetch)(void *const data,
const unsigned long addr,
size_t bytes),
void (*store)(const unsigned long addr,
const void *const data,
size_t bytes));
The instruction to decode is at (void *)context->gregs[REG_IP] on x86, and at (void *)context->gregs[REG_RIP] on x86-64. The function must skip the instruction by incrementing context->gregs[REG_IP]/context->gregs[REG_RIP]/etc. by the number of bytes in the machine instruction. If you don't, SIGSEGV will just be raised again and again, with the program code stuck in that instruction!
The function must use only the fetch and store callbacks to access the memory that caused the SEGV. In your case, they would be implemented as functions that contact the remote machine, asking it to perform the desired action on the specified bytes.
Assuming you have the above three functions implemented, the rest is just about trivial. For simplicity, lets assume you have
static void *map_base;
static size_t map_size;
static void *map_ends; /* (char *)map_base + map_size */
static void sigsegv_handler(int signum, siginfo_t *info, void *context)
{
if (info->si_addr >= map_base && info->si_addr < map_ends) {
const int saved_errno = errno;
emulate(&((ucontext_t *)context)->uc_mcontext,
your_load_function, your_store_function);
errno = saved_errno;
} else {
struct sigaction act;
sigemptyset(&act.sa_mask);
act.sa_handler = SIG_DFL;
act.sa_flags = 0;
if (sigaction(SIGSEGV, &act, NULL) == 0)
raise(SIGSEGV);
else
raise(SIGKILL);
}
}
static int install_sigsegv_handler(void)
{
struct sigaction act;
sigemptyset(&act.sa_mask);
act.sa_sigaction = handle_sigsegv;
act.sa_mask = SA_SIGINFO;
if (sigaction(SIGSEGV, &act, NULL) == -1)
return errno;
return 0;
}
If map_size was already obtained from the remote machine (and rounded up to sysconf(_SC_PAGESIZE)), then you just need to do
if (install_sigsegv_handler()) {
/* Failed; see errno. Abort. */
}
map_base = mmap(NULL, map_size, PROT_NONE,
MAP_PRIVATE | MAP_ANONYMOUS, -1, (off_t)0);
if ((void *)map_base != MAP_FAILED)
map_ends = (void *)(map_size + (char *)map_base);
else {
/* Failed; see errno. Abort. */
}
Now that I've scared everyone reading this out of their brains, I'm happy to also mention that there is a much easier, portable way to do this. It also tends to be more efficient.
This is not "memory mapping a remote file", but a co-operative scheme where multiple machines can share a mapping. From the user's perspective it's pretty much the same thing, but all parties using the mapping must participate in the work.
Instead of trying to catch every access to the mapped region, use page granularity and introduce the concept of page owner: each page of the mapping is accessible on at most one machine at a time, that machine owning said page.
Memory maps act on page-sized units (see sysconf(_SC_PAGESIZE)). You cannot set a specific byte or arbitrary byte range to be inaccessible or read-only -- unless it is aligned to page boundary. You can change any page to be readable and writable, readable only, or inaccessible (PROT_READ|PROT_WRITE, PROT_READ, and PROT_NONE, respectively; see mmap() and mprotect()).
The owner concept is quite simple. When a machine owns a page, it can freely read and write to the page, otherwise not. Note: If there is a file backing, updating the mapped file contents atomically is very difficult. I really recommend an approach where there is no backing file -- or that the backing file is updated in page-sized chunks using fcntl()-based leases or locking.)
Simply put, each page in the mapping is PROT_READ|PROT_WRITE on exactly one machine, and PROT_NONE in all others.
When somebody tries to write to a read-only page, the SIGSEGV handler on that machine is triggered. It contacts the other machines, and requests the ownership of that particular page. The then-owner, receiving such a message, changes its mapping to PROT_NONE, and sends the page to the new owner. The new owner updates the mapping, changing the protection to PROT_READ|PROT_WRITE, and returns from the SIGSEGV handler.
A couple of notes:
If the SIGSEGV handler returns before a change occurs in the mapping, nothing bad happens. The SIGSEGV signal simply gets immediately re-raised by the same instruction.
I recommend using a separate thread for receiving pages, and updating the local contents of the mapping. Then, the SIGSEGV handler only needs to make sure it has sent a request for ownership of that page, and sched_yield(), to not spin or "twiddle its thumbs" unnecessarily.
Program execution continues when the mapping is updated for that page. send() etc. are async-signal-safe, so you can send the request from the signal handler directly -- but not that you don't want to send the request every time slice (100-1000 times a second!), just once every while.
Remember: If the SIGSEGV signal handler does not resolve the problem, there is no harm done. The SIGSEGV just gets raised immediately again by the same instruction. However, I do warmly recommend using sched_yield(), so that other threads and processes on the machine get to use the CPU, instead of wasting CPU time raising a signal millions of times a second for nothing.
If writes are rare, but reads common, you can extend the ownership concept, to read-owner(s) and write-owner. Each page can be owned by any number of read-owners, as long as there is no write-owner. To modify the page, one needs to be write-owner, and that revokes any read-owners.
The logic is such that any thread can ask for read-ownership. If there is no write-owner, it is automatically granted; either the last write owner or any existing read-owners will send the read-only page contents. If there is a write-owner, it must downgrade its ownership to read-owner, and send the now read-only contents to the requester. To modify a page, one must already be a read-owner, and simply tells all other read-owners that they are now the write-owner.
In this case, the SIGSEGV handler is not much more complicated. If the page protections are PROT_NONE, it will ask for read ownership. If the page protections are PROT_READ, it already has read ownership, and therefore must ask to upgrade it to write-ownership. Note: using this scheme, we do not need to check the instruction whether it tried to access the memory for fetch or store -- indeed, it does not even matter. In the worst case -- write to a page not owned in any way by this thread -- SIGSEGV just gets raised twice: first to get read ownership, and second time to upgrade it to write-ownership.
Note that you cannot upgrade read-ownership to write-ownership in the SIGSEGV handler. If you did that, two threads on separate machines could upgrade their read-ownership at the same time, before the messages reach the other parties. All state changes can only occur after all necessary confirmation TCP messages have arrived.
(Since many-to-many message arbitration is quite complicated, it is almost always better to have a designated arbitrator (or "server"), which handles all the requests from each child. Page transfers can still be direct between members, although you do need to send a notification of each page transfer to the arbitrator/server, too.)
If there is no backing file -- i.e. it is MAP_ANONYMOUS -- you can replace the contents of any page atomically.
When receiving a page, you first get a new anonymous page using mmap(NULL, page, PROT_READ[|PROT_WRITE], MAP_PRIVATE|MAP_ANONYMOUS, -1, (off_t)0), and copy the new data into it. Then, you use mremap() to replace the old page with the new one. (The old page is effectively released as if munmap() was called, but this all happens atomically, so that no thread sees any intermediate state.)
This way you'll be sending just page-sized chunks around. For portability, you should actually use the smallest common multiple of all the page sizes involved, so that every machine can participate regardless of their possible page size differences. (Fortunately, they're always powers of two, and very often 4096, although I do seem to recall architectures that used 512, 2048, 8192, 16384, 32768, 65536, and 2097152 -byte pages, so please do not just hard-code your page size.)
Overall, both approaches have their benefits. The first (requiring the instruction emulator) allows any number of clients to access a memory mapping on one server with no co-operation needed from any of the other mappings to the same file on the server. The second needs co-operation from all parties using the mapping, but reduces the access latencies for multiple consecutive accesses; using the read-owner/write-owner logic, you should get a very performant shared memory management.
If you have difficulty deciding between brk()/sbrk() on one hand, and mmap() at other, I do fear both of these approaches are just too complex for you at this point. You should understand the inherent limitations of memory mapping first -- page granularity et cetera --, and perhaps even some of the cache theory (since this is essentially caching data), so that you can relatively easily manage the concepts involved.
Believe me, trying to program something you cannot really grasp at the conceptual level, leads to frustration. That said, grasping for the concepts, taking the time to learn them as you encounter them while programming, is fine; you just need to spend the time and effort.
Questions?

Answer 2

Here's an idea:
When the caller requests to "remote mmap" a region or an entire file, you allocate memory for that entire size right away and return that pointer. Also store a record of the allocation internally.
Use SFTP or similar to open the remote file. Don't do anything with it yet, just make sure it exists and has the right size.
You install a signal handler for SIGSEGV.
You use mprotect(2) to set the entire allocated space to be inaccessible (PROT_NONE).
When your signal handler is called, use the siginfo_t argument's si_addr parameter to know if the segmentation fault is in the region you allocated in step 1. If not, pass the segmentation fault along, it's probably going to be fatal as they usually are in most programs.
Now you know you have a region of memory which has been requested but is not yet accessible. Populate the memory by reading from the remote file opened in step 2 and return from your signal handler.
What we achieve then is something like "page faults" where we load on demand the required parts of the remote file. Of course, if you know something about the access pattern (e.g. that the entire file will always be needed in some particular order, or will be needed by multiple processes over time) you can do better, perhaps simpler things.