In a spare-time project of mine, I'm implementing a duplex transcoding framework. The most essential functions I'm writing are Read and Write functions, which are to be called on different threads, to exchange data like Unix Pipe/FIFO.
Because they're on different threads, I need to be sure that they're properly synchronized, and that my use of synchronization APIs are correct.
When encountering EOF, I call pthread_{condvar,mutex}_destroy functions to destroy 2 condition variables and 1 mutex. The 2 condvars are used to blocks the read and write call respectively, until input/output space are available; the mutex is the big mutex that protects the entire duplex object.
The questions are:
Is it safe to signal a condition variable after it had been destroyed?
Is it safe to unlock a mutex after it had been destroyed?
Are there similar guarantees on other threading APIs (such as C11 Threads and C++ Threads)?
Is it safe to signal a condition variable after it had been destroyed?
No. Its docs say:
A destroyed condition variable object can be reinitialized using
pthread_cond_init(); the results of otherwise referencing the object
after it has been destroyed are undefined.
(emphasis added).
Is it safe to unlock a mutex after it had been destroyed?
No. It's docs say:
A destroyed mutex object can be reinitialized using
pthread_mutex_init(); the results of otherwise referencing the
object after it has been destroyed are undefined.
(emphasis added).
Are there similar guarantees on other threading APIs (such as C11
Threads and C++ Threads)?
About using synchronization objects after they have been destroyed? No. Why would there be? These sorts of APIs are for releasing resources. If the synchronization objects rely on resources that require cleanup in the first place, then how does it make sense that they could still function once those resources are released?
Generally speaking, then, before you can tear down synchronization objects such as mutexes and condition variables, you need to ensure that there are no circumstances under which any current thread could attempt to access them again. At least, not until after they have been re-initialized (in a framework where that is even possible). Under some circumstances it is reasonable simply not to tear them down at all. Otherwise, you need to be more creative.
This is not safe. Destroy will destroy any shared memory used by these objects. The solution to your problem is to join all the threads that are using your your mutexes and cond-vars (except one) and only then destroy them.
Related
I use pthread_mutex_t in my program for thread synchronization control.
Do I need to do some finishing works when the pthread_mutex_t is no longer in use?
Or, can I do nothing?
Thank you for your help
You mention that "the pthread_mutex_t is no longer in use".
I assume you mean that you no longer need to use it, ever, in any of your threads.
In this case:
The pthread_mutex_t must be in the unlocked state.
You should call pthread_mutex_destroy.
The requirement for the mutex to be unlocked appears in the documentation for pthread_mutex_destroy:
It shall be safe to destroy an initialized mutex that is unlocked.
Attempting to destroy a locked mutex results in undefined behavior.
(emphasis is mine)
This post contains some more info about the proper usage of pthread_mutex_destroy:
How to safely and correctly destroy a mutex in Linux using pthread_mutex_destroy?.
I use pthread_mutex_t in my program for thread synchronization
control. Do I need to do some finishing works when the
pthread_mutex_t is no longer in use? Or, can I do nothing?
TL;DR: You do not need to do any cleanup. In some cases you should, and in other cases it's a question of style. And in some cases, the question is moot because it's not possible to recognize that a mutex is no longer in use.
The relevant sense of "no longer in use" here would be that the mutex is not currently locked by any thread (including the one that might perform cleanup), and there is no possibility that any thread will attempt to lock it in the future. For this case, the pthread_mutex_destroy() function is available to release any resources that the mutex may be holding (not including the storage occupied by the mutex object itself). In any other case, destroying the mutex puts your program at risk of exercising undefined behavior.
If a given mutex object has ever been initialized, including via the static initializer, and its lifetime ends at a point when it has not been destroyed since its last initialization, then the end of its lifetime must be assumed to leak resources. But this is consequential only when the mutex's lifetime ends before the end of the program, because all resources belonging to a process are cleaned up by the OS when the process terminates. In particular, it is not consequential in the common case of mutex objects declared at file scope in any translation unit.
Guidance, then:
As a correctness matter, you must ensure that
The lifetime of a mutex object does not end while it is still in use.
No mutex is destroyed while it is still in use or after the end of its lifetime.
As a practical matter, you should avoid consequential resource leaks, as they may ultimately lead to program failure and / or overall system stress from resource exhaustion. In this context, that means using pthread_mutex_destroy() to clean up mutex objects having automatic, allocated, or thread storage duration before those objects' lifetimes end, when that occurs significantly before the end of the program overall.
As a style matter, you might choose to apply a similar discipline to mutex objects having static storage duration -- perhaps only those initialized via pthread_mutex_init(), or perhaps including also those initialized via the static initializer macro. I tend not to worry about these, myself, as there are rarely very many, and they rarely go out of use very much before the program is going to terminate anyway.
As a style matter, you should not make heroic efforts or overly complicate your code to ensure that mutexes are explicitly destroyed when the program is terminating. The OS is going to perform all necessary cleanup anyway, and any cleanup (or other) code that takes a lot of effort to write correctly in the first place is fertile ground for bugs and has high maintenance cost.
Finally, note well that there are cases when you can't even recognize before program termination that a given mutex is no longer in use. For example, consider a program that declares a file-scope mutex used to synchronize the operations of several daemon threads. It may well be that no thread in the system can determine whether all the (other) daemon threads have terminated, so as to know that the mutex is no longer in use, so there is no safe course but to avoid ever destroying it explicitly.
Details about the issue leading to the question
We're facing a SIGSEGV error under extremely rare circumstances when using the lmdb database library that are not easily reproducible. The best we got out of it is a stack-trace that looks like this through the core dump:
#0 in mdb_env_reader_dest (ptr=...) at .../mdb.c: 4935
#1 in __nptl_deallocate_tsd () at pthread_create.c:301
...
The line the stack-trace is pointing to is this (it's not the same because we attempted some changes to fix this).
Having tracked the issue for a while, the only explanation I have is that, when closing the lmdb environment from a thread other than the main thread, there's some kind of race between this line and this line, where the latter deletes the key, calls the custom destructor mdb_env_reader_dest, which causes SIGSEGV according to the documentation of lmdb, when attempting to use resources after freeing them.
The question
The documentation of pthread_key_create and pthread_key_delete are ambiguous to me, personally, in the sense whether they're talking about the calls to pthread_key_create and pthread_key_delete or the data underneath the pointers. This is what the docs say:
The associated destructor functions used to free thread-specific data at thread exit time are only guaranteed to work correctly when called in the thread that allocated the thread-specific data.
So the question is, can we call mdb_env_open and mdb_env_close from two different threads, leading to pthread_key_create and pthread_key_delete from different threads, and expect correct behavior?
I couldn't find such a requirement in the lmdb documentation. The closest I could find is this, which doesn't seem to reference the mdb_env_open function.
Are we allowed to call pthread_key_create and pthread_key_delete from different threads?
Yes.
However, a key must be created via pthread_key_create before it can be used by any thread, and that is not, itself, inherently thread safe. The key creation is often synchronized by performing it before starting any (other) threads that will use the key, but there are other alternatives.
Similarly, a key must not be deleted before all threads are finished with it, and the deletion is not, itself, inherently thread safe. TSD keys often are not deleted at all, and when they are deleted, that is often synchronized by first joining all (other) threads that may use the key. But again, there are other alternatives.
The documentation of pthread_key_create and pthread_key_delete are
ambiguous to me, personally, in the sense whether they're talking
about the calls to pthread_key_create and pthread_key_delete or the
data underneath the pointers. This is what the docs say:
The associated destructor functions used to free thread-specific data
at thread exit time are only guaranteed to work correctly when called
in the thread that allocated the thread-specific data.
The destructor functions those docs are talking about are the ones passed as the second argument to pthread_key_create().
And note well that that text is drawn from the Rationale section of the docs, not the Description section. It is talking about why the TSD destructor functions are not called by pthread_key_delete(), not trying to explain what the function does. That particular point is that TSD destructor functions must run in each thread carrying non-NULL TSD, as opposed to in the thread that calls pthread_key_delete().
So the question is, can we call mdb_env_open and mdb_env_close from
two different threads, leading to pthread_key_create and
pthread_key_delete from different threads, and expect correct
behavior?
The library's use of thread-specific data does not imply otherwise. However, you seem to be suggesting that there is a race between two different lines in the same function, mdb_env_close0, which can be the case only if that function is called in parallel by two different threads. The MDB docs say of mdb_env_close() that "Only a single thread may call this function." I would guess that they mean that to be scoped to a particular MDB environment. In any case, if you really have the race you think you have, then it seems to me that your program must be calling mdb_env_close() from multiple threads, contrary to specification.
So, as far as I know or can tell, the thread that calls mdb_env_close() does not need to be the same one that called mdb_env_open() (or mdb_env_create()), but it does need to be the only one that calls it.
How can I implement a binary semaphore using the POSIX counting semaphore API? I am using an unnamed semaphore and need to limit its count to 1. I believe I can't use a mutex because I need to be able to unlock from another thread.
If you actually want a semaphore that "absorbs" multiple posts without allowing multiple waits to succeed, and especially if you want to be strict about that, POSIX semaphores are not a good underlying promitive to use to implement it. The right set of primitives to implement it on top of is a mutex, a condition variable, and a bool protected by the mutex. When changing the bool from 0 to 1, you signal the condition variable.
With that said, what you're asking for is something of a smell; it inherently has ambiguous orderings. For example if threads A and B both post the semaphore one after another, and threads X and Y are both just starting to wait, it's possible with your non-counting semaphore that either both waits succeed or that only one does, depending on the order of execution: ABXY or AXBY (or other comparable permutation). Thus, the pattern is likely erroneous unless either there's only one thread that could possibly psot at any given time (in which case, why would it post more than once? maybe this is a non-issue) or ability to post is controlled by holding some sort of lock (again, in which case why would it post more than once?). So if you don't have a design flaw here, it's likely that just using a counting semaphore but not posting it more than once gives the behavior you want.
If that's not the case, then there's probably some other data associated with the semaphore that's not properly synchronized, and you're trying to use the semaphore like a condition variable for it. If that's the case, just put a proper mutex and condition variable around it and use them, and forget the semaphore.
One comment for addressing your specific situation:
I believe I can't use a mutex because I need to be able to unlock from another thread.
This becomes a non-issue if you use a combination of mutex and condition variable, because you don't keep the mutex locked while working. Instead, the fact that the combined system is in-use is part of the state protected by the mutex (e.g. the above-mentioned bool) and any thread that can obtain the mutex can change it (to return it to a released state).
Are there any guarantees on when a memory write in one thread becomes visible in other threads using pthreads?
Comparing to Java, the Java language spec has a section that specifies the interaction of locks and memory that makes it possible to write portable multi-threaded Java code.
Is there a corresponding pthreads spec?
Sure, you can always go and make shared data volatile, but that is not what I'm after.
If this is platform dependent, is there a de facto standard? Or should another threading library be used?
POSIX specifies the memory model in 4.11 Memory Synchronization:
Applications shall ensure that access to any memory location by more than one thread of control (threads or processes) is restricted such that no thread of control can read or modify a memory location while another thread of control may be modifying it. Such access is restricted using functions that synchronize thread execution and also synchronize memory with respect to other threads. The following functions synchronize memory with respect to other threads:
fork()
pthread_barrier_wait()
pthread_cond_broadcast()
pthread_cond_signal()
pthread_cond_timedwait()
pthread_cond_wait()
pthread_create()
pthread_join()
pthread_mutex_lock()
pthread_mutex_timedlock()
pthread_mutex_trylock()
pthread_mutex_unlock()
pthread_spin_lock()
pthread_spin_trylock()
pthread_spin_unlock()
pthread_rwlock_rdlock()
pthread_rwlock_timedrdlock()
pthread_rwlock_timedwrlock()
pthread_rwlock_tryrdlock()
pthread_rwlock_trywrlock()
pthread_rwlock_unlock()
pthread_rwlock_wrlock()
sem_post()
sem_timedwait()
sem_trywait()
sem_wait()
semctl()
semop()
wait()
waitpid()
The pthread_once() function shall synchronize memory for the first call in each thread for a given pthread_once_t object.
The pthread_mutex_lock() function need not synchronize memory if the mutex type if PTHREAD_MUTEX_RECURSIVE and the calling thread already owns the mutex. The pthread_mutex_unlock() function need not synchronize memory if the mutex type is PTHREAD_MUTEX_RECURSIVE and the mutex has a lock count greater than one.
Unless explicitly stated otherwise, if one of the above functions returns an error, it is unspecified whether the invocation causes memory to be synchronized.
Applications may allow more than one thread of control to read a memory location simultaneously.
I am not aware that POSIX threads give such guarantees. They don't have a model for atomic access to thread-shared objects. If it'd be for POSIX threads, the only guarantees that you can have for visibility of modifications is using some kind of lock.
Modern C, C11, (and probably also C++11) has a model for this kind of questions. It has threads and atomics (fences and all that stuff) that give you exact rules when you may assume that a modification done by one thread is visible by another.
The thread interface of C11 is a cooked-down version of POSIX threads, with less functionality. Unfortunately, the specification for the semantics of that thread interface is yet much to loose, basically the semantics are missing in many places. But a combination of C11 interfaces and POSIX thread semantics can give you a good view of how things work in modern systems.
Edit: So if you want to have guarantees for memory synchronization use either the lock interfaces that POSIX provides or go for atomic operations. All modern compilers have extensions that provide these, gcc and family (icc, opencc, clang) have e.g the series of __sync... builtins. Clang it its newest version also already has support of the new C11 _Atomic feature. There are also wrappers available that give you interfaces for the other compilers that come close to _Atomic.
If I have two threads and one global variable (one thread constantly loops to read the variable; the other constantly loops to write to it) would anything happen that shouldn't? (ex: exceptions, errors). If it, does what is a way to prevent this. I was reading about mutex locks and that they allow exclusive access to a variable to one thread. Does this mean that only that thread can read and write to it and no other?
Would anything happen that shouldn't?
It depends in part on the type of the variables. If the variable is, say, a string (long array of characters), then if the writer and the reader access it at the same time, it is completely undefined what the reader will see.
This is why mutexes and other coordinating mechanisms are provided by pthreads.
Does this mean that only that thread can read and write to it and no other?
Mutexes ensure that at most one thread that is using the mutex can have permission to proceed. All other threads using the same mutex will be held up until the first thread releases the mutex. Therefore, if the code is written properly, at any time, only one thread will be able to access the variable. If the code is not written properly, then:
one thread might access the variable without checking that it has permission to do so
one thread might acquire the mutex and never release it
one thread might destroy the mutex without notifying the other
None of these is desirable behaviour, but the mere existence of a mutex does not prevent any of these happening.
Nevertheless, your code could reasonably use a mutex carefully and then the access to the global variable would be properly controlled. While it has permission via the mutex, either thread could modify the variable, or just read the variable. Either will be safe from interference by the other thread.
Does this mean that only that thread can read and write to it and no other?
It means that only one thread can read or write to the global variable at a time.
The two threads will not race amongst themselves to access the global variable neither will they access it at the same time at any given point of time.
In short the access to the global variable is Synchronized.
First; In C/C++ unsynchronized read/write of variable does not generate any exceptions or system error, BUT it can generate application level errors -- mostly because you are unlikely to fully understand how the memory is accessed, and whether it is atomic unless you look at the generated assembler. A multi core CPU may likely create hard-to-debug race conditions when you access shared memory without synchronization.
Hence
Second; You should always use synchronization -- such as mutex locks -- when dealing with shared memory. A mutex lock is cheap; so it will not really impact performance if done right. Rule of thumb; keep the lcok for as short as possible, such as just for the duration of reading/incrementing/writing the shared memory.
However, from your description, it sounds like that one of your threads is doing nothing BUT waiting for the shared meory to change state before doing something -- that is a bad multi-threaded design which cost unnecessary CPU burn, so
Third; Look at using semaphores (sem_create/wait/post) for synchronization between your threads if you are trying to send a "message" from one thread to the other
As others already said, when communicating between threads through "normal" objects you have to take care of race conditions. Besides mutexes and other lock structures that are relatively heavy weight, the new C standard (C11) provides atomic types and operations that are guaranteed to be race-free. Most modern processors provide instructions for such types and many modern compilers (in particular gcc on linux) already provide their proper interfaces for such operations.
If the threads truly are only one producer and only one consumer, then (barring compiler bugs) then
1) marking the variable as volatile, and
2) making sure that it is correctly aligned, so as to avoid interleaved fetches and stores
will allow you to do this without locking.