C/Linux: dual-map memory with different permissions - c

My program passes data pointers to third-party plugins with the intention that the data should be read-only, so it would be nice to prevent the plugins from writing to the data objects. Ideally there would be a segfault if a plugin attempts a write. I've heard there is some way to double-map a memory region, such that a second virtual address range points to the same physical memory pages. The second mapping would not have write permission, and the exported pointers would use this address range instead of the original (writable) one. I would prefer not to change the original memory allocations, whether they happen to use malloc or mmap or whatever. Can someone explain how to do this?

It is possible to get a dual mapping, but it requires some work.
The only way I know how to create such a dual mapping is to use the mmap function call. For mmap you need some kind of file-descriptor. Fortunately Linux allows you to get a shared memory object, so no real file on a storage medium is required.
Here is a complete example that shows how to create the shared memory object, creates a read/write and read-only pointer from it and then does some basic tests:
#include <stdio.h>
#include <fcntl.h>
#include <unistd.h>
#include <string.h>
#include <sys/mman.h>
#include <sys/stat.h>
#include <sys/types.h>
int main()
{
// Lets do this demonstration with one megabyte of memory:
const int len = 1024*1024;
// create shared memory object:
int fd = shm_open("/myregion", O_CREAT | O_RDWR, S_IRUSR | S_IWUSR);
printf ("file descriptor is %d\n", fd);
// set the size of the shared memory object:
if (ftruncate(fd, len) == -1)
{
printf ("setting size failed\n");
return 0;
}
// Now get two pointers. One with read-write and one with read-only.
// These two pointers point to the same physical memory but will
// have different virtual addresses:
char * rw_data = mmap(0, len, PROT_READ|PROT_WRITE, MAP_SHARED, fd,0);
char * ro_data = mmap(0, len, PROT_READ , MAP_SHARED, fd,0);
printf ("rw_data is mapped to address %p\n", rw_data);
printf ("ro_data is mapped to address %p\n", ro_data);
// ===================
// Simple test-bench:
// ===================
// try writing:
strcpy (rw_data, "hello world!");
if (strcmp (rw_data, "hello world!") == 0)
{
printf ("writing to rw_data test passed\n");
} else {
printf ("writing to rw_data test failed\n");
}
// try reading from ro_data
if (strcmp (ro_data, "hello world!") == 0)
{
printf ("reading from ro_data test passed\n");
} else {
printf ("reading from ro_data test failed\n");
}
printf ("now trying to write to ro_data. This should cause a segmentation fault\n");
// trigger the segfault
ro_data[0] = 1;
// if the process is still alive something didn't worked.
printf ("writing to ro_data test failed\n");
return 0;
}
Compile with: gcc test.c -std=c99 -lrt
For some reason I get a warning that ftruncate is not declared. No idea why. The code works well though. Example output:
file descriptor is 3
rw_data is mapped to address 0x7f1778d60000
ro_data is mapped to address 0x7f1778385000
writing to rw_data test passed
reading from ro_data test passed
now trying to write to ro_data. This should cause a segmentation fault
Segmentation fault
I've left the memory deallocation as an exercise for the reader :-)

Related

Confused about mmap() return - two different pointers?

Currently trying to understand how memory mapping works in Linux (or in general, really), and I'm following with this one example of Shared Memory in POSIX systems from Operating System Concepts. The two files are as follows:
Producer file
// This is the producer file for the Shared memory object
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <fcntl.h>
#include <sys/shm.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <unistd.h>
#include <errno.h>
#include <sys/mman.h>
int main()
{
// ---------------- PRODUCER ESTABLISHES SHARED MEMORY OBJECT AND WRITES TO IT ----------------
// Specifying the size in bytes of the shared memory object
const int SIZE = 4096;
// Name of the shared memory space
const char *name = "OS";
// The actual strings to write to shared memory
const char *message_0 = "Hello";
const char *message_1 = "World!";
// Shared memory file descriptor will be stored in this
int fd;
// Pointer to the shared memory object will be stored in this
char *ptr;
// Checking error
int errnum;
// Create a shared memory object. This opens (establishes a connection to) a shared memory object.
fd = shm_open(name, O_CREAT | O_RDWR,0666);
if (fd == -1)
{
errnum = errno;
fprintf(stdout, "Value of errno: %d\n", errno);
perror("Error printed by perror");
fprintf(stdout, "Error opening file: %s\n", strerror(errnum));
return 1;
}
// Configure the size of the shared-memory object to be 4096 bytes
ftruncate(fd, SIZE);
// Memory map the shared memory object
ptr = (char *) mmap(0, SIZE, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
printf("Ptr is: %p\n", ptr);
if (ptr == MAP_FAILED)
{
errnum = errno;
fprintf(stdout, "Value of errno in ptr: %d\n", errno);
perror("Error printed by perror");
fprintf(stdout, "Error opening file: %s\n", strerror(errnum));
return 1;
}
// Write to the shared memory object
sprintf(ptr, "%s", message_0);
ptr += strlen(message_0);
sprintf(ptr, "%s", message_1);
ptr += strlen(message_1);
return 0;
}
Consumer file
// This is the consumer file for the Shared memory object, in which it reads what is in the memory object OS
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <fcntl.h>
#include <sys/shm.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <unistd.h>
#include <errno.h>
#include <sys/mman.h>
int main()
{
// Size in bytes of shared memory object
const int SIZE = 4096;
// Name of the shared memory space
const char *name = "OS";
// Shared memory file descriptor will be stored in this
int fd;
// Pointer to the shared memory object will be stored in this
char *ptr;
// Checking error
int errnum;
// Open the shared memory object
fd = shm_open(name, O_RDWR, 0666);
// If error in shm_open()
if (fd == -1)
{
errnum = errno;
fprintf(stdout, "Value of errno: %d\n", errno);
perror("Error printed by perror");
fprintf(stdout, "Error opening file: %s\n", strerror(errnum));
return 1;
}
// Memory-map the shared memory object
ptr = (char *) mmap(0, SIZE, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
printf("Ptr is: %p\n", ptr);
if (ptr == MAP_FAILED)
{
errnum = errno;
fprintf(stdout, "Value of errno in ptr: %d\n", errno);
perror("Error printed by perror");
fprintf(stdout, "Error opening file: %s\n", strerror(errnum));
return 1;
}
// Read from the shared memory object
printf("%s\n", (char *) ptr);
// Remove the shared memory object (delete it)
shm_unlink(name);
return 0;
}
When I print the pointer to the shared memory object (printf("Ptr value: %p\n, ptr)), I get two different values for the consumer and producer files. Why does this happen?
As I understand, the pointer ptr points at the shared memory object, which is in the physical memory. This physical memory is just shared amongst two processes, by mapping it onto their address space. However, this would require that the pointer point to the same address in physical memory, no? Or is it pointing to the virtual memory (i.e. the address space of the processes)? If so, does that itself point to the physical memory?
Thanks!
Each process has its own virtual memory address space. The mappings from virtual memory to physical memory are, in general, different for each process. While the shared memory may be in physical memory at address 0x1000, the producer process may have it mapped into virtual memory at address 0x7000, and the consumer process may have it mapped into virtual memory at address 0x4000.
Further, if the shared memory is swapped out of memory for some reason, the system could later reload it to a different physical address, say 0x13000, and update the mappings in the processes so that it appears at the same addresses as before in each of the producer and consumer processes.
Producer and consumer are two different processes, so they have their own virtual memory space each. When you attach the shared segment, you specify the kernel that you have no preference on where (in your unallocated virtual address space) to put it, so you normally get different pointers because most probably both processes have different memory maps. Just think on an scenario where one process has allocated the memory map at, say, address A, and the other process has that address A occupied by some other thing it has allocated (e.g. dynamic memory for the heap, or a different shared library) It is clear that the kernel cannot allocate two different things in the same (virtual) address range (or even overlap some other mapping), so it uses a differen place, and returns a different pointer.
Just don't worry about it, as both virtual addresses map to the same phisical address (this is sure, as it is a shared memory segment), and the pointers finally end pointing to the same thing. The kernel is normally not aware of where it has placed that same segment in a different process, it doesn't need to do. This means that it is highly improbable that you get both pointers equal, contrary to what you though.
As none of these processes knows about the virtual address space of the other, there's no conflict, but never pass the address to the other process to use, because an address in a virtual address space has absolutely no meaning in the virtual address space of another, different process.
A process has normally no means (and no possibility) to know how the kernel assings the memory and builds the map of virtual to physicall addresses. Even for itself (the kernel also runs in it's virtual memory space, and only deals with the tables that map ---these tables are in the virtual address space of the kernel, but not in the virtual address space of any user process, so this is the reason that makes the kernel capable of changing the mappings but it is not possible for other processes)
It is impossible for a process to know where in actual memory a virtual address points to. That is possible for some administrator processes through a device (and a special device is needed for this) /dev/mem. But if you try to look in there without the actual mapping you'll get a mess of pages some belonging to a process, some to another, with no apparent structure, or you cannot also know what does it mean the contents of those pages.
There's another device /dev/kmem that maps to the kernel virtual address space, and allows a process (e.g. a debugger) to get access to the mapping table there.... but this is a very dangerous bend, as you can easily crash your system by tweaking there. Think that you cannot normally stop the kernel (this stops the system) or if you can do, you will stop essential parts of the system that should be running.

Mapping existing memory (data segment) to another memory segment

As the title suggests, I would like to ask if there is any way for me to map the data segment of my executable to another memory so that any changes to the second are updated instantly on the first. One initial thought I had was to use mmap, but unfortunately mmap requires a file descriptor and I do not know of a way to somehow open a file descriptor on my running processes memory. I tried to use shmget/shmat in order to create a shared memory object on the process data segment (&__data_start) but again I failed ( even though that might have been a mistake on my end as I am unfamiliar with the shm API). A similar question I found is this: Linux mapping virtual memory range to existing virtual memory range? , but the replies are not helpful.. Any thoughts are welcome.
Thank you in advance.
Some pseudocode would look like this:
extern char __data_start, _end;
char test = 'A';
int main(int argc, char *argv[]){
size_t size = &_end - &__data_start;
char *mirror = malloc(size);
magic_map(&__data_start, mirror, size); //this is the part I need.
printf("%c\n", test) // prints A
int offset = &test - &__data_start;
*(mirror + offset) = 'B';
printf("%c\n", test) // prints B
free(mirror);
return 0;
}
it appears I managed to solve this. To be honest I don't know if it will cause problems in the future and what side effects this might have, but this is it (If any issues arise I will try to log them here for future references).
Solution:
Basically what I did was use the mmap flags MAP_ANONYMOUS and MAP_FIXED.
MAP_ANONYMOUS: With this flag a file descriptor is no longer required (hence the -1 in the call)
MAP_FIXED: With this flag the addr argument is no longer a hint, but it will put the mapping on the address you specify.
MAP_SHARED: With this you have the shared mapping so that any changes are visible to the original mapping.
I have left in a comment the munmap function. This is because if unmap executes we free the data_segment (pointed to by &__data_start) and as a result the global and static variables are corrupted. When at_exit function is called after main returns the program will crash with a segmentation fault. (Because it tries to double free the data segment)
Code:
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <stdint.h>
#define _GNU_SOURCE 1
#include <unistd.h>
#include <sys/mman.h>
extern char __data_start;
extern char _end;
int test = 10;
int main(int argc, char *argv[])
{
size_t size = 4096;
char *shared = mmap(&__data_start, 4096, PROT_READ | PROT_WRITE, MAP_FIXED | MAP_ANONYMOUS | MAP_SHARED, -1, 0);
if(shared == (void *)-1){
printf("Cant mmap\n");
exit(-1);
}
printf("original: %p, shared: %p\n",&__data_start, shared);
size_t offset = (void *)&test - (void *)&__data_start;
*(shared+offset) = 50;
msync(shared, 4096, MS_SYNC);
printf("test: %d :: %d\n", test, *(shared+offset));
test = 25;
printf("test: %d :: %d\n", test, *(shared+offset));
//munmap(shared, 4096);
}
Output:
original: 0x55c4066eb000, shared: 0x55c4066eb000
test: 50 :: 50
test: 25 :: 25

How to write unnamed Posix Semaphore to Shared Memory?

I want to write a semaphore to shared memory. My first idea was to pass the pointer returned by mmap to sem_init():
#include <stdio.h>
#include <semaphore.h>
#include <string.h>
#include <errno.h>
#include <sys/mman.h>
#include <sys/stat.h>
#include <fcntl.h>
int main(void)
{
sem_t *sem_ptr;
int shm_fd = shm_open("Shm", O_CREAT | O_RDWR, DEFFILEMODE);
fprintf(stderr, "%s\n", strerror(errno));
sem_ptr = mmap(NULL, sizeof(sem_t), PROT_WRITE, MAP_SHARED, shm_fd, 0);
fprintf(stderr, "%p\n", strerror(errno));
sem_init(sem_ptr, 1, 1);
fprintf(stderr, "%s\n", strerror(errno));
sem_destroy(sem_ptr);
return 0;
}
But it leads to this error(when sem_init() is called): Process finished with exit code 135 (interrupted by signal 7: SIGEMT)
Then I tried to initialize the semaphore with a sem_t variable and write it to the shared memory:
int main(void)
{
sem_t *sem_ptr;
sem_t s;
int shm_fd = shm_open("Shm", O_CREAT | O_RDWR, DEFFILEMODE);
fprintf(stderr, "%s\n", strerror(errno));
sem_ptr = mmap(NULL, sizeof(sem_t), PROT_WRITE, MAP_SHARED, shm_fd, 0);
fprintf(stderr, "%p\n", strerror(errno));
sem_init(&s, 1, 1);
fprintf(stderr, "%s\n", strerror(errno));
*sem_ptr = s;
sem_destroy(&s);
return 0;
}
Now the line *sem_ptr = s; leads to the same error as in the first programm
Can anyone help me please?
Your first strategy for creating the semaphore is correct. You can't necessarily copy a sem_t object to a different memory address and have it still work.
I'm not sure why you're getting SIGEMT, which I thought was never generated by modern Unixes. But when I run either of your programs on my computer, they crash with SIGBUS instead, and that pointed me at a bug that I know how to fix. When you mmap a file (a shared memory object is considered to be a file), and the size you ask for in the mmap call is bigger than the file, and then you access the memory area beyond the end of the file (by far enough that the CPU can trap this), you get a SIGBUS. And let me quote you a key piece of the shm_open manpage:
O_CREAT: Create the shared memory object if it does not exist. [...]
A new shared memory object initially has zero length—the size of
the object can be set using ftruncate(2).
What you need to do is call ftruncate on shm_fd to make the shared memory object big enough to hold the semaphore.
Some less-important bugs you should fix at the same time:
All of the system calls that work with memory maps may malfunction if you give them offsets or sizes that aren't a multiple of the system page size. (They're supposed to round up for you, but historically there have been a lot of bugs in this area.) You get the system page size by calling sysconf(_SC_PAGESIZE), and you round up with a little helper function shown below.
Most C library functions are allowed to set errno to a nonzero value even if they succeed. You should check whether each function actually failed before printing strerror(errno). (In the code below I used perror instead for brevity.)
The name of a shared memory object is required to start with a slash, followed by up to NAME_MAX characters that are not slashes.
sem_init may read from as well as writing to the memory pointed to by sem_ptr, and subsequent use of sem_wait and sem_post definitely will, so you should use PROT_READ|PROT_WRITE in the mmap call.
Putting it all together, this is a revised version of your first program which works on my computer. Because of the SIGEMT thing I can't promise it will work for you.
#include <fcntl.h>
#include <semaphore.h>
#include <stdio.h>
#include <sys/mman.h>
#include <unistd.h>
#ifndef DEFFILEMODE
# define DEFFILEMODE 0666
#endif
static long round_up(long n, long mult)
{
return ((n + mult - 1) / mult) * mult;
}
int main(void)
{
long pagesize;
long semsize;
sem_t *sem_ptr;
int shm_fd;
pagesize = sysconf(_SC_PAGESIZE);
if (pagesize == -1) {
perror("sysconf(_SC_PAGESIZE)");
return 1;
}
shm_fd = shm_open("/Shm", O_CREAT|O_RDWR, DEFFILEMODE);
if (shm_fd == -1) {
perror("shm_open");
return 1;
}
semsize = round_up(sizeof(sem_t), pagesize);
if (ftruncate(shm_fd, semsize) == -1) {
perror("ftruncate");
return 1;
}
sem_ptr = mmap(0, semsize, PROT_READ|PROT_WRITE, MAP_SHARED, shm_fd, 0);
if (sem_ptr == MAP_FAILED) {
perror("mmap");
return 1;
}
if (sem_init(sem_ptr, 1, 1)) {
perror("sem_init");
return 1;
}
sem_destroy(sem_ptr);
shm_unlink("/Shm");
return 0;
}
An additional complication you should be aware of is that calling sem_init on a semaphore that has already been initialized causes undefined behavior. This means you have to use some other kind of locking around the creation of the shared memory segment and the semaphore within. Off the top of my head I don't know how to do this in a bulletproof way.

How to change characters in a text file using C's mmap()?

Let's say I have the standard "Hello, World! \n" saved to a text file called hello.txt. If I want to change the 'H' to a 'R' or something, can I achieve this with mmap()?
mmap does not exist in the standard C99 (or C11) specification. It is defined in POSIX.
So assuming you have a POSIX system (e.g. Linux), you could first open(2) the file for read & write:
int myfd = open("hello.txt", O_RDWR);
if (myfd<0) { perror("hello.txt open"); exit(EXIT_FAILURE); };
Then you get the size (and other meta-data) of the file with fstat(2):
struct stat mystat = {};
if (fstat(myfd,&mystat)) { perror("fstat"); exit(EXIT_FAILURE); };
Now the size of the file is in mystat.st_size.
off_t myfsz = mystat.st_size;
Now we can call mmap(2) and we need to share the mapping (to be able to write inside the file thru the virtual address space)
void*ad = mmap(NULL, myfsz, PROT_READ|PROT_WRITE, MAP_SHARED,
myfd, 0);
if (ad == MMAP_FAILED) { perror("mmap"); exit(EXIT_FAILURE); };
Then we can overwrite the first byte (and we check that indeed the first byte in that file is H since you promised so):
assert (*(char*ad) == 'H');
((char*)ad) = 'R';
We might call msync(2) to ensure the file is updated right now on the disk. If we don't, it could be updated later.
Notably for very large mappings (notably those much larger than available RAM), we can assist the kernel (and its page cache) with hints given thru madvise(2) or posix_madvise(3)...
Notice that a mapping remains in effect even after a close(2). Use munmap & mprotect or mmap with MAP_FIXED on the same address range to change them.
On Linux, you could use proc(5) to query the address space. So your program could read (e.g. after fopen, using fgets in a loop) the pseudo /proc/self/maps file (or /proc/1234/maps for process of pid 1234).
BTW, mmap is used by dlopen(3); it can be called a lot of times, my manydl.c program demonstrates that on Linux you could have many hundreds of thousands of dlopen-ed shared files (so many hundreds of thousands of memory mappings).
Here's a working example.
#include <stdio.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <sys/mman.h>
int main(){
int myFile = open("hello.txt", O_RDWR);
if(myFile < 0){
printf("open error\n");
}
struct stat myStat = {};
if (fstat(myFile, &myStat)){
printf("fstat error\n");
}
off_t size = myStat.st_size;
char *addr;
addr = mmap(NULL, size, PROT_READ|PROT_WRITE, MAP_SHARED, myFile, 0);
if (addr == MAP_FAILED){
printf("mmap error\n");
}
if (addr[0] != 'H'){
printf("Error: first char in file not H");
}
addr[0] = 'J';
return 0;
}

mprotect on a mmap-ed shared memory segment

When two processes share a segment of memory opened with shm_open and then it gets mmap-ed, does doing an mprotect on a portion of the shared memory in one process affects the permissions seen by the other process on this same portion? In other words, if one process makes part of the shared memory segment read-only, does it become read-only for the other process too?
I always like to address those questions in two parts.
Part 1 - Let's test it
Let's consider an example that is relatively similar to the one at shm_open(3).
Shared header - shared.h
#include <sys/mman.h>
#include <fcntl.h>
#include <semaphore.h>
#include <sys/stat.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#define errExit(msg) do { perror(msg); exit(EXIT_FAILURE); } while (0)
struct shmbuf {
char buf[4096];
sem_t sem;
};
Creator process - creator.c (Compile using gcc creator.c -o creator -lrt -lpthread)
#include <ctype.h>
#include <string.h>
#include "shared.h"
int
main(int argc, char *argv[])
{
if (argc != 2) {
fprintf(stderr, "Usage: %s /shm-path\n", argv[0]);
exit(EXIT_FAILURE);
}
char *shmpath = argv[1];
int fd = shm_open(shmpath, O_CREAT | O_EXCL | O_RDWR,
S_IRUSR | S_IWUSR);
if (fd == -1)
errExit("shm_open");
struct shmbuf *shm;
if (ftruncate(fd, sizeof(*shm)) == -1)
errExit("ftruncate");
shm = mmap(NULL, sizeof(*shm), PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
if (shm == MAP_FAILED)
errExit("mmap");
if (sem_init(&shm->sem, 1, 0) == -1)
errExit("sem_init");
if (mprotect(&shm->buf, sizeof(shm->buf), PROT_READ) == -1)
errExit("mprotect");
if (sem_wait(&shm->sem) == -1)
errExit("sem_wait");
printf("got: %s\n", shm->buf);
shm_unlink(shmpath);
exit(EXIT_SUCCESS);
}
Writer process - writer.c (Compile using gcc writer.c -o writer -lrt -lpthread)
#include <string.h>
#include "shared.h"
int
main(int argc, char *argv[])
{
if (argc != 3) {
fprintf(stderr, "Usage: %s /shm-path string\n", argv[0]);
exit(EXIT_FAILURE);
}
char *shmpath = argv[1];
char *string = argv[2];
int fd = shm_open(shmpath, O_RDWR, 0);
if (fd == -1)
errExit("shm_open");
struct shmbuf *shm = mmap(NULL, sizeof(*shm), PROT_READ | PROT_WRITE,
MAP_SHARED, fd, 0);
if (shm == MAP_FAILED)
errExit("mmap");
strcpy(&shm->buf[0], string);
if (sem_post(&shm->sem) == -1)
errExit("sem_post");
exit(EXIT_SUCCESS);
}
What's supposed to happen?
The creator process, creates a new shared memory object, "sets" its size, maps it into memory (as shm), uses mprotect to allow only writes to the buffer (shm->buf) and waits for a semaphore to know when the writer (which we will discuss in a moment finishes its thing).
The writer process starts, opens the same shared memory object, writes into it whatever we tell it to and signals the semaphore.
Question is, will the writer be able to write to the shared memory object even though the creator changed the protection to READ-ONLY?
Let's find out. We can run it using:
# ./creator.c /shmulik &
# ./writer.c /shmulik hi!
got: hi!
#
[1]+ Done ./creator /shmulik
As you can see, the writer was able to write to the shared memory, even though the creator set it's protection to READ-ONLY.
Maybe the creator does something wrong? Let's try to add the following line to creator.c:
if (mprotect(&shm->buf, sizeof(shm->buf), PROT_READ) == -1)
errExit("mprotect");
memset(&shm->buf, 0, sizeof(shm->buf)); // <-- This is the new line
if (sem_wait(&shm->sem) == -1)
errExit("sem_wait");
Let's recompile & run the creator again:
# gcc creator.c -o creator -lrt -lpthread
# ./creator /shmulik
Segmentation fault
As you can see, the mprotect worked as expected.
How about we let the writer map the shared memory, then we change the protection? Well, it ain't going to change anything. mprotect ONLY affects the memory protection of the process calling it (and it's descendants).
Part 2 - Let's understand it
First, you have to understand that shm_open is a glibc method, it's not a systemcall.
You can get the glibc source code from their website and just look for shm_open to see that yourself.
The underlying implementation of shm_open is a regular call for open, just like the man page suggests.
As we already saw, most of the magic happens in mmap. When calling mmap, we have to use MAP_SHARED (rather than MAP_PRIVATE), otherwise every process is going to get a private memory segment to begin with, and obviously one ain't gonna affect the other.
When we call mmap, the hops are roughly:
ksys_mmap_pgoff.
vm_mmap_pgoff.
do_mmap
mmap_region
At that last point, you could see that we take the process' memory management context mm and allocate a new virtual memory area vma:
struct mm_struct *mm = current->mm;
...
vma = vm_area_alloc(mm);
...
vma->vm_page_prot = vm_get_page_prot(vm_flags);
This memory area is not shared with other processes.
Since mprotect changes only the vm_page_prot on the per-process vma, it doesn't affect other processes that map the same memory space.
The POSIX specification for mprotect() suggests that changes in the protection of shared memory should affect all processes using that shared memory.
Two of the error conditions detailed are:
[EAGAIN]
The prot argument specifies PROT_WRITE over a MAP_PRIVATE mapping and there are insufficient memory resources to reserve for locking the private page.
[ENOMEM]
The prot argument specifies PROT_WRITE on a MAP_PRIVATE mapping, and it would require more space than the system is able to supply for locking the private pages, if required.
These strongly suggest that memory that is mapped with MAP_SHARED should not fail because of a lack of memory for making copies.
See also the POSIX specification for mmap().

Resources