I'm taking over some C code running in Linux (Centos) with extensive use of semaphores.
The way the code is written :
./Program1
This program launches a bunch of processes which makes use of mutexes and semaphores.
./Program2
This program also launches a bunch of processes which makes use of mutexes and semaphores.
I've realised that Program1 and Program2, they make use of semaphores with the same names.
In Linux C programming, can different programs use the same semaphores?
My guess is no, but the same naming is confusing the hell out of me. They are using the same source code to launch and handle the semaphores.
The semaphores are invoked using the following commands:
semget
semctl
semop
I've read that these are called processes semaphores.. if Program1 creates SEMAPHORE1, can Program2 access SEMAPHORE1?
Appreciate any help here, thanks!
Assuming you mean named semaphores (or even unnamed semaphores stored in shared memory, both which can be created with sem_open), they generally are shared amongst processes.
Semaphores using semget and related calls use an ID key rather than a name but their usage patterns are similar.
Semaphores are one of the IPC (inter-process communication) methods.
You can create a one-process-only semaphore by using an unnamed variant in non-shared memory and this will only be accessible to threads of the given process but, in my experience, that's not a common use case. The semget family of calls can also give you process-private semaphores.
Mutexes, on the other hand, tend to be more used within a single process for inter-thread communication but there is even a variant of them that can work inter-process.
You create a pthread_mutexattr (attribute) which allows sharing of mutexes and then use that attribute when initialising the mutex you want to share. Obviously, the mutex needs to be in shared memory so that multiple processes can get at it.
Related
I'm currently working on a school projet (C language) which aims to make processes who can write into shared memory, one by one.
I can't use multi-threading neither fork.
I can have only one executable, which must be run as many times as we want.
For example if i do : ./main john then ./main toto, John would be able to write in shared memory while Toto has to wait that John unlock the semaphore.
But I just can't find any documentation who explain clearly how it works without doing threads, forks etc.
May someone can help me ? Thanks
Assuming that Linux is used:
Because multi-threading is not allowed, you may ignore pthread-semaphores.
Maybe system-V IPC semaphores are what you are looking for:
https://man7.org/linux/man-pages/man7/sysvipc.7.html
Those can be used between processes.
The system-V IPC also provides functions for shared memory handling.
Posix-semaphores and shm API are also another alternative:
https://linux.die.net/man/7/sem_overview
https://man7.org/linux/man-pages/man7/shm_overview.7.html
https://opensource.com/article/19/4/interprocess-communication-linux-storage has examples for using POSIX semaphores to synchronize shared memory access (in the second part of that article, past the file lock example).
It is not unlike synchronizing threads, except that it uses operating system facilities for inter process communication.
POSIX semaphores are available on POSIX systems, notably many *nixes.
I want to use anonymous semaphores in shared memory to synchronise multiple processes.
While multiple POSIX operating systems provide anonymous semaphores (through sem_init and associated functions), macOS doesn't support them.
I discovered that the Mach semaphores are available on macOS, but in semaphore_create, I see no equivalent to to the pshared parameter of sem_init and I have a hard time finding documentation indicating that Mach/XNU semaphores can actually be used for synchronising processes and not just threads. Is it possible?
Here is how I am using the semaphores currently (note that sema is a structure that contains different kinds of semaphores depending on the target OS, for instance it contains a sem_t semaphore on Linux):
For initialising the semaphore:
task_t task = current_task();
semaphore_create(task, &sema->semaphore, SYNC_POLICY_FIFO, value)
For waiting on it:
semaphore_wait(sema->semaphore);
For waiting on it with a timeout (equivalent functionality to sem_timedwait on Linux):
mach_timespec_t time;
time.tv_sec = seconds;
time.tv_nsec = useconds;
semaphore_timedwait(sema->semaphore, time);
For signalling it:
semaphore_signal(sema->semaphore);
All of the above code examples are actually contained in corresponding functions that allow to provide an OS agnostic API, the test calls I'm doing on these functions on Linux work as expected, but on macOS the process waiting on the semaphore is never signaled from the other process.
What am I doing wrong?
Edit
I found this in the docs:
Semaphores can be used any place where mutexes can occur. This
precludes their use in interrupt handlers or within the context of the
scheduler, and makes it strongly discouraged in the VM system.
What does the "context of the scheduler" refer to exactly? Would it be if you was writing your own scheduler or does this refer to any interaction with the scheduler (eg: processes).
In all cases, am I right that VM means that it shouldn't be used with processes that make use of the virtual memory, in other terms... all user processes?
Mach semaphores can be used across processes, but it requires moving their port handles with mach_msg to another process, and it's a bit of a pain.
But - why go there, when you have not POSIX semaphores, but Sys V ones? sem_init may not be available, but Sys V semaphores are fully supported.
Specifically you're looking at the following sys calls:
semsys
semctl
semget
semop : on top of which you have sem_post(2) and a few other sem_* operations
sem_unlink
sem_close
sem_open takes a semaphore path on the filesystem, creating a named semaphore which will have visibility in your other process as well. sem_wait, etc..
I have two processes that will be running, one that will be reading from shared memory (mmap) and one that will be writing to that shared memory (mmap). These processes are started separately in two different terminals, but they need to be synchronized so that while one process is writing, it writes the full amount before the other process reads from the memory. All of the posts I have seen relating to shared memory mutex locks have been spawning threads/processes from a single main program. Is there any way to create a shared mutex lock that can be used by two separate programs?
Sorry, but you are out of luck. Pthreads library does not have a concept of 'named' mutex, so two independent processes can't reliably share one.
Yes, you can create a mutex in shared memory and than use this mutex from the other process, but there is no way you can ensure the mutex is fully initialized by the first process when you are checking it in the second. For trully independent programms I strongly recommend using semaphores.
You can create a shared mutex into an mmapped file. If you're using Linux and have a sufficiently new kernel, you can even create an unlinked temp file, mmap it; initialize the mutex and only then link it to the final location. Or you can use file locking to deny access to it until the initialization has been completed.
The semaphore example from pthread_mutexattr_init POSIX manuals at linux.die.net did work on my Linux 4.2.0-27 Ubuntu.
I'm working with semaphores in C , especifically to control the access to a shared memory zone in linux. but there is one thing that I can't understand.
I am using a mutex to control the access to a specific zone because i have 2 processes that must read/write from that zone. the thing is, when we use the fork() to create a new child process, the whole program is "copied" to another program as if they were two seperate programs right ? so, when i do V(mutex) in one process, how does the other one know he can't access ?
I know its a noob question but nobody could explain this to me until now.
After the fork neither process is going to know about the memory actions of the other because they are separate copies. You have to put your shared variables in shared memory, including mutexes and semaphores. Then all the processes are operating on the same resource.
For unrelated (i.e. non-forked) process there are usually system facilities (e.g. named semaphores) that each process can open based on a path name or similar method that each can use to find and use the resource.
You synchronisation objects must be placed in process shared memory, for example created with mmap (... MAP_ANONYMOUS ...). In addition, they must have the PTHREAD_PROCESS_SHARED attribute set, for example, by using pthread_mutexattr_setpshared.
See here:
Semaphores and Mutex for Thread and Process Synchronization
So mutex in practice is often used in threads, which makes sharing trivial. For processes however, mutex could be stored as a part of the shared mem.
For semaphores however, linux has built in library, which identifies global semaphores by keys. See below.
http://beej.us/guide/bgipc/output/html/multipage/semaphores.html
Or you can use other IPC to sync. Signals, for example.
Hope this helps.
I am trying to create a shared memory which will be used by multiple processes. these processes communicate with each other using MPI calls (MPI_Send, MPI_Recv).
I need a mechanism to control the access of this shared memory I added a question yesterday to see if MPI provides any facility to do that. Shared memory access control mechanism for processes created by MPI , but it seems that there is no such provision by MPI.
So I have to choose between named semaphore or flock.
For named semaphore if any of the process dies abruptly without calling sem_cloe(), than that semaphore always remains and can be seen by ll /dev/shm/. This results in deadlock sometimes(if I run the same code again!), for this reason I am currently thinking of using flock.
Just wanted to confirm if flock is best suited for this type of operation ?
Are there any disadvantages of using flock?
Is there anything else apart from named semaphore and flock that can be used here ?
I am working on C under linux.
You can also use a POSIX mutex in shared memory; you just have to set the "pshared" attribute on it first. See pthread_mutexattr_setpshared. This is arguably the most direct way to do what you want.
That said, you can also call sem_unlink on your named semaphore while you are still using it. This will remove it from the file system, but the underlying semaphore object will continue to exist until the last process calls sem_close on it (which happens automatically if the process exits or crashes).
I can think of two minor disadvantages to using flock. First, it is not POSIX, so it makes your code somewhat less portable, although I believe most Unixes implement it in practice. Second, it is implemented as a system call, so it will be slower. Both pthread_mutex_lock and sem_wait use the "futex" mechanism on Linux, which only does a system call when you actually have to wait. This is only a concern if you are grabbing and releasing the lock a lot.