Despite continued interest there is still no way to create a “cow-copy” of a memory region on Linux. With the inception of the memfd_create(2) syscall, the situation has slightely improved, as one does not have to create an explicit file any more for shared memory.
I am wondering, why isn't there a thing like the following?
void *ptr = mmap((void *)0, size, PROT_READ | PROT_WRITE, MAP_ANONYMOUS | MAP_PRIVATE, -1, (size_t)0);
void *ptr2= mremap(ptr, 0, size, MREMAP_MAYMOVE | MREMAP_COW);
The intended semantics are that ptr2 and ptr share the underlying memory, but a write from either will trigger a copy-on-write with page-granularity.
Is this just a case of “nobody bothered to implement this yet” or am I missing something technical?
Related
I would like to allocate a buffer within 32-bit address space on 64-bit ARM. In other words, I would like to ensure that my buffer is bound to the lower 32-bit address space. Do you know a nice C function which does that?
There is no C standard function to do so. However, since you tagged the question as Linux, take a look at mmap(2) and the MAP_ANONYMOUS and MAP_32BIT flags, e.g.:
mmap(
0, 1,
PROT_READ | PROT_WRITE,
MAP_PRIVATE | MAP_ANONYMOUS | MAP_32BIT,
-1, 0
);
Another option is passing an explicit address in the lower 32-bit address space using the MAP_FIXED flag:
mmap(
(void *)0x10000, 1,
PROT_READ | PROT_WRITE,
MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED,
-1, 0
);
I am trying to complete an exercise that is done with system calls and need to allocate memory for a struct *. My code is:
myStruct * entry = (myStruct *)mmap(0, SIZEOF(myStruct), PROT_READ|PROT_WRITE,
MAP_ANONYMOUS, -1, 0);
To clarify, I cannot use malloc() but can use mmap(). I was having no issues with this on Windows in Netbeans, now however I'm compiling and running from command line on Ubuntu I am getting "Segmentation Fault" each time I try to access it.
Is there a reason why it will work on one and not the other, and is mmap() a valid way of allocating memory in this fashion? My worry was I was going to be allocating big chunks of memory for each mmap() call initially, now I just cannot get it to run.
Additionally, the error returned my mmap is 22 - Invalid Argument (I did some troubleshooting while writing the question so the error check isn't in the above code). Address is 0, the custom SIZEOF() function works in other mmap arguments, I am using MAP_ANONYMOUS so the fd and offsetparameters must -1 and 0 respectively.
Is there something wrong with the PROT_READ|PROT_WRITE sections?
You need to specify MAP_PRIVATE in your flags.
myStruct * entry = (myStruct *)mmap(0, SIZEOF(myStruct),
PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0);
From the manual page:
The flags argument determines whether updates to the mapping are
visible to other processes mapping the same region, and whether
updates are carried through to the underlying file. This behavior is
determined by including exactly one of the following values in flags:
You need exactly one of the flags MAP_PRIVATE or MAP_SHARED - but you didn't give either of them.
A complete example:
#include <sys/mman.h>
#include <stdio.h>
typedef struct
{
int a;
int b;
} myStruct;
int main()
{
myStruct * entry = (myStruct *)mmap(0, sizeof(myStruct),
PROT_READ|PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
if (entry == MAP_FAILED) {
printf("Map failed.\n");
}
else {
entry->a = 4;
printf("Success: entry=%p, entry->a = %d\n", entry, entry->a);
}
return 0;
}
(The above, without MAP_PRIVATE of course, is a good example of what you might have provided as a an MCVE. This makes it much easier for others to help you, since they can see exactly what you've done, and test their proposed solutions. You should always provide an MCVE).
The man page for mmap() says that you must specify exactly one of MAP_SHARED and MAP_PRIVATE in the flags argument. In your case, to act like malloc(), you'll want MAP_PRIVATE:
myStruct *entry = mmap(0, sizeof *entry,
PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_ANONYMOUS, -1, 0);
(I've also made this more idiomatic C by omitting the harmful cast and matching the sizeof to the actual variable rather than its type).
I wanted to write some function void* share(void*, int) that should set up shared memory to share the data at the pointer.
My first attempt looked like (without checks etc.):
void* share(void *toBeShared, int size) {
int fd = shm_open(SHM_NAME, O_CREAT | O_RDWR | O_EXCL, S_IRUSR | S_IWUSR);
ftruncate(fd, size);
return mmap(toBeShared, size, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
}
but this does not seem to work as I would like it. The second attempt was something like:
void* share(void *toBeShared, int size) {
void *mem = NULL;
int fd = shm_open(SHM_NAME, O_CREAT | O_RDWR | O_EXCL, S_IRUSR | S_IWUSR);
ftruncate(fd, size);
mem = mmap(NULL, size, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0)
memcpy(mem, toBeShared, size);
return mem;
}
and this does work, but I need to copy the entire data, which I would like to avoid.
Therefore my question: is there a way to share memory that has already been allocated (if possible without having to copy too much around) and if yes, how could it be done?
Thanks in advance.
PS: I've seen more of these questions (e.g. here and here), but there are no answers given in there.
edit:
how I would like to use it:
typedef struct {
char *name;
int status;
} MyTask;
int main(int argc, char** argv) {
MyTask* taskList = NULL, sharedTaskList = NULL;
int length = 0;
...
readFile(&taskList, &length, ...);
sharedTaskList = share(taskList, length * sizeof(MyTask));
// or maybe even better: without needing to assign it to new variable
for(i = 0; i < NR_WORKERS; i++) {
switch(pid = fork()) {
//etc...
}
}
...
return 0;
}
How to share existing memory?
Don't share existing memory. Get some (small amount of) "fresh" shared memory and use (i.e. fill or read) it later.
Assuming you are on Linux, read shm_overview(7).
I guess that some of your functions might fail. You should test against failure each call, e.g.
int fd = shm_open(SHM_NAME, O_CREAT | O_RDWR | O_EXCL, S_IRUSR | S_IWUSR);
if (fd<0) {perror("shm_open"); exit(EXIT_FAILURE);};
and so on. Perhaps use also strace(1)
is there a way to share memory that has already been allocated
Short answer, no! (or not easily, and not in a portable way). You conventionally do the opposite: obtain some shared segment of known size, and use some pointers into it. (the same shared segment might have different virtual addresses in different processes, e.g. because of ASLR).
You could use mmap(2) with MAP_FIXED on some already used virtual address space subsegment (that would overwrite & replace the mapping with a new one, not share an existing mapping!), but I would suggest to avoid that. Notice that the virtual address space is managed in multiples of pages, so there is no way to share some data which is not page aligned. So your share function is impossible unless both toBeShared and size are page-aligned. You could consider the Linux specific mremap(2)
In other words, your applications should first allocate some shared memory and then put/use some data inside the obtained shared segment, not try to share some existing unshared virtual memory range. So you probably want to code some void* get_my_shared_memory(); (assuming the size is a compile time constant, and you call that function once per process, and its resulting virtual address would often vary from one process to another)
In practice, memory is a finite resource, and shared memory is a scarce and very limited resource. On most systems, you'll be able to share a few dozens of megabytes only... So sharing an arbitrary large amount of memory is unreasonable.
Perhaps your entire application might just use some server, e.g. some database server à la PostGreSQL, to share information, by making requests to that server (and using ACID properties of DBMS). Or you could organize it as a monitoring process exchanging messages (e.g. URL to be processed) -on pipes or sockets or fifos- with slave processes. But we don't know what kind of application are you coding.
BTW, sharing memory is not enough. You need to synchronize your processes.
I've got this struct:
typedef struct ip_row {
sem_t row_lock;
char row_name[NAME_SIZE];
char row_address4[NAME_SIZE]; // IPv4 address name
char row_address6[NAME_SIZE]; // IPv6 address name
} ip_row_t;
I would like to use the struct multiple times in a shared memory file.I have verified that for exactly one use, it works.
int shmfd; //Shared memory file descriptor
struct ip_row *data;
/*...creating shared memory and map...*/
shmfd = shm_open(shared_memory, O_CREAT | O_RDWR, S_IRUSR | S_IWUSR | S_IRGRP );
ftruncate(shmfd,FILESIZE);
data = (ip_row_t*)mmap(0, FILESIZE, PROT_READ | PROT_WRITE, MAP_SHARED, shmfd,0);
/*...getting to user inputs, this stuff is in a loop...*/
strcpy(data->row_name,ipstr1);
strcpy(data->row_address6,ipstr2);
strcpy(data->row_address4,ipstr3);
When I run the loop again, the writing starts at the beginning of the shared memory file, overwriting what was there before. How can I move the offset so I can support more entries into the shared memory? I've tried these two:
lseek(shmfd,sizeof(struct ip_row_t),SEEK_CUR); //segfault when we write agian
data = (ip_row_t*)mmap(0, FILESIZE, PROT_READ | PROT_WRITE, MAP_SHARED, shmfd,sizeof(struct ip_row_t)); //also segfaults when we try to read
Any advice would be greatly appreciated.
`
You should use pointer arithmetic on struct ip_row *data; to reach "further" in your mmaped shared memory since mmap returns nothing but a pointer to a memory region of FILESIZE, which happens to be mirrored into shared memory.
For example use data[0] to access first copy, data[1] to access second etc.
lseek on shared memory object is unspecified and your second mmap causes segfault because you are trying to mmap FILESIZE bytes to a FILESIZE memory region that has been allocated to you but at an sizeof(struct ip_row_t) offset, thus effectively going outside the memory you are allowed to access, besides, offset has to be multiple of getpagesize(), which in this case it probably is not.
I created my shared memory and mapped my object with following code:
shmfd = shm_open(SHMOBJ_PATH, O_CREAT | O_EXCL | O_RDWR, S_IRWXU | S_IRWXG);
ftruncate(shmfd, shared_seg_size);
bbuffer = (boundedBuffer *)mmap(NULL, shared_seg_size, PROT_READ | PROT_WRITE, MAP_SHARED, shmfd, 0);
Now I need to initialize and add/remove items to/from bbuffer. When I try to add/remove, I get Segmentation Fault: 11, which indicates the program accessed a memory location that was not assigned. What can I do to solve this problem?
A wild guess:
perhaps you don't have the header file for mmap included
and you are on an architecture with 64 bit void* and 32 bit int
What could happen there is that the compiler takes mmap as returning int by default, casts this into a pointer and by that clashes the higher order bits.
Never cast return values from functions such as malloc or mmap, and always take the warnings of your compiler seriously.