I am currently working on something using POSIX named semaphores and shared memory and I've read from the man pages that all open named semaphores are automatically closed on process termination. Is this also the case for shared memory objects, are they also closed and unmapped or simply just closed? I cannot find any information about this on the man pages.
The question seems to be about how and when to clean up POSIX shared memory used by one or more processes, or possibly about how to avoid shared memory being cleaned up prematurely.
POSIX shared memory is designed on a model intentionally similar to regular file access. In particular,
shm_open() will create and open a new, persistent shared-memory object or simply open an existing one, depending on whether there already is one with the specified name.
that region can be opened by other processes (and therefore must persist) until it is unlinked via shm_unlink().
a shared memory region lives after its unlinking as long as any process has it open, but it can no longer be opened via shm_open().
mapping a shared memory region via mmap() has the effect of holding it open while that mapping is in place, independent of the file descriptor used to map it
Moreover, memory mappings for a shared-memory region are in most respects the same as mappings for regular files. Mappings are per-process properties; they do not survive termination of the process to which they pertain. Mappings are preserved (duplicated) across fork()s.
On some systems, shared memory regions may even be accessible via the file system. Aside from the different functions for managing them, their most significant difference from regular files is probably that they do not persist across reboots.
Thus, you do not need to worry about termination of a process unwantedly tearing down a shared memory region that is in use by other processes. On the other hand, you can arrange for shared-memory regions to be cleaned up automatically by unlinking them after every process that needs to obtain access by name has done so. If you intend to grant access only to child processes (and maybe their children, etc.) then you can unlink immediately after creation. Children will inherit the mapping when you fork them.
So, in response to the actual question:
Is this also the case for shared memory objects, are they also closed and unmapped or simply just closed?
Shared memory objects open and / or mapped by a process are both closed and unmapped when that process terminates, but they are not automatically unlinked. They will persist at least until manually unlinked or the system is rebooted.
Is this also the case for shared memory objects, are they also closed
and unmapped or simply just closed?
They are unmapped but they may continue to occupy space in the backing filesystem (usually tmpfs/shmfs for /dev/shm on memory) if they are not explicitly unlinked.
On FreeBSD it is possible to get this automatic unlinking with the non-portable SHM_ANON flag. If you want this kind of behaviour you can either:
Use mmap(2) with the MAP_ANONYMOUS flag and share file descriptors via fork(2) or send them to other processes with sendmsg(2) using Unix domain sockets.
Use System V shared memory with the IPC_RMID flag, which automatically destroys the memory segment after the last process detaches it. The dettachment happens when the process dies or calls shmdt(2).
Use the newer Linux-only memfd_create(2) system call.
Related
Is it safe to use shm_unlink even there are other process currently open the shared memory?
For exmaple:
Process B shm_open and then Process A shm_unlink. Is it ok for Process B?
Yes, it is safe to call shm_unlink if another process has it opened. Per the man page for shm_unlink:
The operation of shm_unlink() is analogous to unlink(2): it removes a
shared memory object name, and, once all processes have unmapped the
object, de-allocates and destroys the contents of the associated
memory region. After a successful shm_unlink(), attempts to shm_open()
an object with the same name will fail (unless O_CREAT was specified,
in which case a new, distinct object is created).
When a process forks, would the child process have the customized shared library (.so file) in its address space?
If so, is the address of the shared library be same or different from its parent process (due to ASLR) ?
Would the function running before the main function __attribute__ ((constructor)) constructor be executed again in all the child process? What about thread?
Yes, the child will retain the parent's mappings. Ordinarily, Linux's virtual memory system will actually share the page between the two processes, up until either one tries to write new data. At that point, a copy will be made and each process will have its own unique version - at a different physical address but retaining the same virtual address. This is referred to as "copy on write" and is a substantial efficiency and resources advantage over systems which cannot support this, particularly running code which forks frequently.
Address Space Layout Randomization (ASLR) can't apply for libraries or objects which are already allocated virtual addresses, as to do so would break any pointers held anywhere in the code - something that a system running non-managed code can't know enough about to account for.
Since all previously constructed objects already exist in memory, constructors are not called again just because of the fork. Any objects which need to be duplicated because they are being uniquely modified have this done invisibly by the VM system behind the scenes - they don't really know that they are being cloned, and you could very well end up having a pair of objects where part of the implementation continues to share a physical page with identical contents while another part has been invisibly bifurcated into distinct physical pages with differing contents for each process.
You also asked about threads, and that is an area where things get complicated. Normally, only the thread which called fork() will exist in live form in the child (though data belonging to the others will exist in shared mappings, since it can't be known what might be shared with the forked thread). If you need to try to fork a multithreaded program, you will need to see the documentation of your threading implementation. For the common pthreads implementation on Linux, particularly pay attention to pthread_atfork()
My question is regarding initializing memory obtained from using shm_open() and mmap(). One common advice I have seen in several places is to call shm_open() with flags O_CREAT|O_EXCL: if that succeeds then we are the first user of the shared memory and can initialize it, otherwise we are not the first and the shared memory has already been initialized by another process.
However, from what I understand about shm_open and from the testing that I did on Linux, this wouldn't work: the shared memory objects get left over in the system, even after the last user of the shared memory object has unmapped and closed. A simple test program which calls shm_open with O_CREAT|O_EXCL, then closes the descriptor and exit, will succeed on the first run, but will still fail on the second run, even though nobody else is using the shared memory at that time.
It actually seems to me that (at least on the system that I tested) the behavior of shm_open is pretty much identical to open(): if I modify my simple test program to write something to the shared memory (through the pointer obtained by mmap) and exit, then the shared memory object will keep its contents persistently (I can run another simple program to read back the data I wrote previously).
So is the advice about using shm_open with O_CREAT|O_EXCL just wrong, or am I missing something?
I do know that the shared memory object can be removed with shm_unlink(), but it seems that will only cause more problems:
If a process dies before calling shm_unlink() then we are back to the problem described above.
If one process calls shm_unlink() while some other processes are still mapped into the same shared memory, these other processes will still continue using it as usual. Now, if another process comes and calls shm_open() with the same name and O_CREAT specified, it will actually succeed in creating new shared memory object with the same name, which is totally unrelated to the old shared memory object the other processes are still using. Now we have a process trying to communicate with other processes via the shared memory and totally unaware that it is using a wrong channel.
I m used to Windows semantics where shared memory object exists only as long as at least one handle is open to it, so this Posix stuff is very confusing.
Since you use the O_EXCL flag I will assume that you have a set of processes gathered around one master (the creator of the segment).
Then, your master process will create the shared memory segment using a call to shm_open :
shmid = shm_open("/insert/name/here", O_CREAT|O_EXCL, 0644);
if (-1 == shmid) {
printf("Oops ..\n");
}
Here, the slaves are ready to use the segment. Since the master HAS to create the segment, there is no need to use the O_CREAT flag in the slaves calls. You'll just have to handle possible errors if the slave call is performed when the segment is not created yet or already destroyed.
When any of your processes is done with the segment, it should call shm_unlink(). In this kind of architecture, the master is usually feeding the slaves. When it has nothing more to say, it just shuts up. The slaves have then the responsibility to handle corresponding errors gracefully.
As you stated, if a process dies before calling the shm_unlink procedure, then the segment will continue to live thereafter. To avoid this in some cases, you could define your own signal handlers in order to perform the operation when signals such as SIGINT are received. Anyway, you won't be able to cover the mess in case SIGKILL is sent to your process.
EDIT :
To be more specific, the use of O_CREAT | O_EXCL is wrong when unnecessary. With the little example above, you can see that it is required for the master to create the segment, thus those flags are needed. On the other hand, none of the slave processes would have to ever create it. Thus, you will absolutely forbid the use of O_CREAT in the related calls.
Now, if another process calls shm_open(..., O_CREAT, ...) when the segment is already created, it will just retrieve a file descriptor related to this very segment. It will thus be on the right channel (if it has the rights to do so, see the mode argument)
You can do the following :
int test = shmget(key_t key,size,0); Put this at the star of each process. Zero flag here tries to open an existing shared memory if its not created yet test will equal -1 so you can make a check after this statement if test -1 go and creat a shared memory else you just got an id to an existing shared memory ..... I hope this help
It's difficult to tell what is being asked here. This question is ambiguous, vague, incomplete, overly broad, or rhetorical and cannot be reasonably answered in its current form. For help clarifying this question so that it can be reopened, visit the help center.
Closed 10 years ago.
I want to share a variable between more than one independent C executables in Linux. That is, a program will write on an array and set a flag so that no other program can use it, and after this operation it'll unset the flag and then another program will read the array. I tried using the same custom header file (containing the variable) in every program, but it seems different instances of the variables are created when the programs are invoked.
Variables you declare in your headers will generate a copy where ever you include them (unless you declare them extern). Of course, when dealing with separate processes every process will have its own memory space. You need to use more sophisticated techniques to circumvent this, i.e. Inter Process Communication (IPC). For example:
(named) Pipes
Sockets
Shared Memory
Your question reads like shared memory is what you want, since hereby multiple processes can access the same memory regions to share some variables. Maybe take a look at this question and its answers for an example.
Your program would be required to create some shared memory, e.g. using shmget and to attach the shared memory object using shmat.
When multiple processes access same memory regions, it's always a healthy approach to add process synchronization during read/write on the variable, e.g. using a shared semaphore (semget, semop).
When you are done with your shared memory you need to detach (shmdt) from it. Thereby you tell the kernel that your process no longer needs access to it.
The process that created the shared memory/semaphore object also needs to destroy them at the end of your program(s). Otherwise it will reside in memory, probably until you reboot your machine (see shmctl, semctl, especially IPC_RMID).
Note that for shared memory objects "The segment will only actually be destroyed after the last process detaches it". So you want to make sure, that this actually happens for all of your processes (shmdt).
In response to the comments, here is the POSIX approach:
System V shared memory (shmget(2), shmop(2), etc.) is an older shared memory API. POSIX shared memory provides a simpler, and better designed interface; on the other hand POSIX shared memory is somewhat less widely available (especially on older systems) than System V shared memory.
shm_open - to get shared memory (via file descriptor)
ftruncate - to set the size of the shared memory
mmap - to get a pointer to the memory
sem_open - to get a semaphore
sem_wait, sem_post - for your read/write synchronisation
shm_unlink, sem_close - to clean up after all
See also this overview and here for examples.
Finally, note that
POSIX shared memory objects have kernel persistence: a shared memory object will exist until the system is shut down, or until all processes have unmapped the object and it has been deleted with shm_unlink(3)
In order to take into account the persistence of the shared memory objects, don't forget to add signal handlers to your application that will perform the clean up operations in case of an exceptional termination (SIGINT, SIGTERM etc.).
Look into using POSIX shared memory via shm_open and shm_unlink... I personally feel they are easier to use and more straight-forward than the older System-V IPC calls such as shmget, etc. since the handle returned works exactly like a file-descriptor that you can use with calls like read, write, etc. Otherwise, if you want access the shared memory object represented by the file-descriptor via normal pointers, you can use mmap on the file-descriptor returned by shm_open.
CreateFileMapping is used to create shared memory in win32. Do need to synchronized shared memory read/write or it is done automatically ?
Your question was not clear about whether you use this for shared memory between threads or processes, so just to be sure: for threads you dont need a file-mapping, all memory in a single process is visible to all threads (and needs sync).
To use it for shared memory between processes: yes you have to sync accesses to it explicitly because the system can not know whether consecutive writes to it are meant to be grouped as a single transaction or not. Tip to do this: To sync them you can not use criticalsections (they only work for threads within a process), you could use:
http://msdn.microsoft.com/en-us/library/aa904937%28v=VS.85%29.aspx