I understand that POSIX Message Queues do not require synchronization between different processes. However, how safe are they when being accessed by multiple threads in a single process?
I read this question: Does message queue support Multi-thread? which pertains to SysV Message Queues, and I would assume that POSIX has at least the same support. It seems to imply yes they are thread safe, however:
"Any resource which is shared globally among threads or processes is subject to race conditions"
This would lead me to believe that intra-process synchronization is still required.
In my case in particular both processes that communicate using the message queue implement the 'boss-worker' pattern, and therefore workers can collide into race conditions when attempting to perform operations on the message queues. Would my assumption be correct in stating that access to these queues still requires synchronization within each process?
If you're passing memory addresses of shared memory (either between threads in a process, or memory that's shared between processes) as the contents of your messages, then you still need memory synchronization to access that memory once you read its address out of the message. But you do not need any additional synchronization to perform operations on the message queue itself (aside from destroying it, which you should not do until the last user has finished with it, of course). That's the whole point of having message queues.
Related
I have 2 threads, the producer thread receives some data over an internet socket, constructs an object and adds it to a queue and the consumer thread pops items from that same queue and does some processing.
Since both threads modify the queue (by enqueuing and dequeuing respectively) I think using a mutex in both threads should be sufficient. Is this a good approach or should I use a semaphore? If so, why?
Either can be made to work. It depends on details of the actual design. How many producer/consumer requests will you allow simultaneously? Is it truly limited to two threads, or will the code spawn others as more requests occur?
I found this short blog interesting. It discusses and compares both the Mutex and Semaphore and may give you some ideas.
"A mutex object allows multiple process threads to access a single shared resource but only one at a time. On the other hand, semaphore
allows multiple process threads to access the finite instance of the
resource until available. In mutex, the lock can be acquired and
released by the same process at a time."
Examples in C
One queue accessed by multiple threads using mutex
Blocking Queue in C++ using semaphores (Not C specific, but concept will be same.)
The documentation of ØMQ mentions:
Individual ØMQ sockets are not thread safe except in the case where full memory barriers are issued when migrating a socket from one thread to another.
What exactly is meant by "full memory barriers?" Can I have multiple threads send over the same ØMQ socket if I synchronize this with mutexes?
As Ulrich has said, yes you can synchronise access to a single thread using mutexes, but really, why would you want to do that?
It's normally considered good practice to only access a socket from a single thread, and synchronise between threads using messages. Something like this:
Worker thread 1
\
Worker thread 2 - > Control thread -> msg out
/
Worker thread 3
where only the control thread can send messages directly over the socket. Messages from the worker threads would be sent to the control thread over an inproc zmq socket that you would create. The control thread would process just one message at a time which avoids the need for the mutexes, provided the workers have no shared state.
Message based designs are easier to implement and debug, and much easier to maintain than designs using mutexes. If you can change the design to do that, I'd advise doing so.
Acquiring a mutex implies a memory barrier. This basically means that write operations must not be reordered in a way that they cross this operation. Summary: Yes, use a mutex to protect access to the ZMQ socket and you're fine.
This is a design issue. Often Posix queues are preferred over anything custom because they are thoroughly tested, and offer advance features such as priority queuing that can be key to development. However, if we analyze it from an INTRA process communication point of view, how good they are if they are to be used only between multiple threads of same process sharing data. Does POSIX Queue optimize its messaging by removing unnecessary copy_to_user and copy_from_user once it recognize that the sender and receiver share the same address-space?
Your queues have two roles: exchanging data between threads and synchronizing them.
Apparently, your alternatives are posix queues, or your own queue (with a pthread mutex and condition variable for locking & synchronization).
For intra process communication, the volume of the data exchanged is not really a concern, since you can always just transmit the pointer of the data (and have the convention that the emitting thread is malloc-ing it while the receiving thread will free it after consumption).
I would guess (but you have to measure) that Posix queues might be slightly slower, because they probably involve a syscall for every operation. In contrast, pthread operations involve a syscall (futex(2)) only for contention (and in the common case of non-blocking operation, a mutex don't do syscall).
I am not sure the kernel could optimize the message passing to avoid the copy_to_user because it may not know when is a queue only for one process (it cannot predict that no other process would mq_open the same queue later).
And you could also use a pipe internal to the process (with poll on the recieving side).
But you really have to benchmark. I am not sure it is such a big issue.
I have 3 questions about thread and process communication.
Can the Linux function msgget(), msgsnd(), and msgrcv() be invoked by multiple threads in one process? These functions in different threads are to attempt to access(r/w) one process' message queue. Are all race conditions supposed to be taken care by the system? If not, is there any good method to support threads and send a message to its main thread(process)?
Can semop() function be used to synchronize threads in one process?
There is a shared memory which have the following entities to access.
process
several threads in one process.
Do I have to use semaphore of inter-process level and a semaphore of threads level at the same time? Any simple way to handle this?
A lot of question. :) thanks.
Can the Linux function msgget(), msgsnd(), and msgrcv() be invoked by multiple threads in one process?
You do not need to worry about race conditions, the system will take care of that, there is no race condition with these calls.
can semop() function be used to synchronize threads in one process?
Yes, read more in the documentation
Do I have to use semaphore of inter-process level and a semaphore of threads level?
Any resource which is shared globally among threads or processes is subject to race conditions due to one or more threads or processes trying to access it at the very same time, So you need to synchronize the access to such a shared global resource.
A thread is "lightweight" because most of the overhead has already been accomplished through the creation of its process.
I found this in one of the tutorials.
Can somebody elaborate what it exactly means?
The claim that threads are "lightweight" is - depending on the platform - not necessarily reliable.
An operating system thread has to support the execution of native code, e.g. written in C. So it has to provide a decent-sized stack, usually measured in megabytes. So if you started 1000 threads (perhaps in an attempt to support 1000 simultaneous connections to your server) you would have a memory requirement of 1 GB in your process before you even start to do any real work.
This is a real problem in highly scalable servers, so they don't use threads as if they were lightweight at all. They treat them as heavyweight resources. They might instead create a limited number of threads in a pool, and let them take work items from a queue.
As this means that the threads are long-lived and small in number, it might be better to use processes instead. That way you get address space isolation and there isn't really an issue with running out of resources.
In summary: be wary of "marketing" claims made on behalf of threads. Parallel processing is great (increasingly it's going to be essential), but threads are only one way of achieving it.
Process creation is "expensive", because it has to set up a complete new virtual memory space for the process with it's own address space. "expensive" means takes a lot of CPU time.
Threads don't need to do this, just change a few pointers around, so it's much "cheaper" than creating a process. The reason threads don't need this is because they run in the address space, and virtual memory of the parent process.
Every process must have at least one thread. So if you think about it, creating a process means creating the process AND creating a thread. Obviously, creating only a thread will take less time and work by the computer.
In addition, threads are "lightweight" because threads can interact without the need of inter-process communication. Switching between threads is "cheaper" than switching between processes (again, just moving some pointers around). And inter-process communication requires more expensive communication than threads.
Threads within a process share the same virtual memory space but each has a separate stack, and possibly "thread-local storage" if implemented. They are lightweight because a context switch is simply a case of switching the stack pointer and program counter and restoring other registers, wheras a process context switch involves switching the MMU context as well.
Moreover, communication between threads within a process is lightweight because they share an address space.
process:
process id
environment
folder
registers
stack
heap
file descriptor
shared libraries
instruments of interprocess communications (pipes, semaphores, queues, shared memory, etc.)
specific OS sources
thread:
stack
registers
attributes (for sheduler, like priority, policy, etc.)
specific thread data
specific OS sources
A process contains one or more threads in it and a thread can do anything a process can do. Also threads within a process share the same address space because of which cost of communication between threads is low as it is using the same code section, data section and OS resources, so these all features of thread makes it a "lightweight process".
Just because the threads share the common memory space. The memory allocated to the main thread will be shared by all other child threads.
Whereas in case of Process, the child process are in need to allocate the separate memory space.