I know that in Linux mutexes are implemented as futexes down below and futex uses compare-and-swap mechanism. And usually for acquiring locks, a user-space thread does not need to make a system call as the lock is resolved in user-space.
Now my question is what happens when there is high contention and many threads are trying to lock a mutex at the same time. Does a system call occurs then for the kernel to decide which thread to grant the mutex? Especially when thread priorities are different? I myself think so.
As long as there is no contention, there are no system calls made. If there is contention, then a system call is made to place the thread in a sleep queue that will then be used to find the first thread to wake up when the mutex becomes free. Additionally, an adjustment is made in the syscall to the value of the futex so that the currently owning thread will not go through the user-land "fast-path" unlock routine (which simply resets the futex back to a zero or "unlocked" value), but will instead make another system call to check the sleep queue for waiting threads to pass the lock ownership to. With more threads contending for a lock, there is of course going to be a higher chance of a contention being found, but again, if there is no contention, then there is no sys-call made.
Futexes only do a small number of loops before falling back to a syscall, so in case of high lock contension there is a high chance that threads will fallback to a syscall.
Related
According to this question here by using pthread_spin_lock is dangerous to lock a critical section as the thread might be interrupted by the scheduler out of the bloom and other threads contenting on that resource might be left spinning.
Suppose that I decide to switch from pthread_spin_lock to locks implemented via atomic built-in + compare_and_swap idion: will this thing improve or still I will suffer from this issue?
Since with pthread it seems to be nothing to disable preemption, is there something I can do in case I use locks implemented via atomics or anything I can have a look at?
I am interested in locking a small critical region.
pthread_mutex_lock typically has a fast path which uses an atomic operation to try to acquire the lock. In the event that the lock is not owned, this can be very fast. Only if the lock is already held, does the thread enter the kernel via a system call. The kernel acquires a spin-lock, and then reattempts to acquire the mutex in case it was released since the first attempt. If this attempt fails, the calling thread is added to a wait queue associated with the mutex, and a context switch is performed. The kernel also sets a bit in the mutex to indicate that there is a waiting thread.
pthread_mutex_unlock also has a fast path. If the waiting thread flag is clear, it can simply release the lock. If the flag is set, the thread must enter the kernel via a system call so the waiting thread can be woken. Again, the kernel must acquire a spin lock so that it can manipulate its thread control data structures. In the event that there is no thread waiting after all, the lock can be released by the kernel. If there is a thread waiting, it is made runnable, and ownership of the mutex is transferred without it being released.
There are many subtle race conditions in this little dance, and hopefully it all works properly.
Since a thread that attempts to acquire a locked mutex is context switched out, it does not prevent the thread the owns the mutex from running, which gives the owner an opportunity to exit its critical section and release the mutex.
In contrast, a thread that attempts to acquire a locked spin-lock simply spins, consuming CPU cycles. This has the potential of preventing the thread that owns the spin-lock from exiting its critical section and releasing the lock. The spinning thread can be preempted when its timeslice has been consumed, allowing the thread that owns the lock to eventually regain control. Of course, this is not great for performance.
In practice, spin-locks are used where there is no chance that the thread can be preempted while it owns the lock. A kernel may set a per-cpu flag to prevent it from performing a context switch from an interrupt service routine (or it may raise the interrupt priority level to prevent interrupts that can cause context switches, or it may disable interrupts altogether). A user thread can prevent itself from being preempted (by other threads in the same process) by raising its priority. Note that, in a uniprocessor system, preventing the current thread from being preempted eliminates the need for the spin lock. Alternatively, in a multiprocessor system, you can bind threads to cpus (cpu affinity) so that they cannot preempt one another.
All locks ultimately require an atomic primitive (well, efficient locks; see here for a counter example). Mutexes can be inefficient if they are highly contended, causing threads to constantly enter the kernel and be context switched; especially if the critical section is smaller than the kernel overhead. Spin locks can be more efficient, but only if the owner cannot be preempted and the critical section is short. Note that the kernel must still acquire a spin lock when a thread attempts to acquire a locked mutex.
Personally, I would use atomic operations for things like shared counter updates, and mutexes for more complex operations. Only after profiling would I consider replacing mutexes with spin locks (and figure out how to deal with preemption). Note that if you intend to use condvars, you have no choice but to use mutexes.
I want to implement a mutex lock.
From my understanding, mutex.lock() should work like
1) check lock owner
2) if lock is owned, put thread in waiting queue
3) suspend this thread until another thread send a wait up signal
However, there is nothing like pthread_suspend(), then how do I do suspend?
I found someone saying use pthread_con_wait(), but seems if I want to use that function, I have to set up a pthread_mutex lock first, which it doesn't make sense to use pthread_mutex inside my mutex.
Well, if my understanding of mutex is wrong, please correct me.
Thanks.
Mutexes, locks, and wait conditions are all different, distinct things. You need a mutex variable in order to implement both a lock and a wait condition.
A lock is a simple mechanism that prevents more than one thread from executing the same code at once by making all by one thread wait for the lock to become unlocked.
A wait condition is a slightly more complex structure that allows a thread to monitor a condition (usually a boolean flag) and only wake up when the flag has changed favourably.
In both cases, when a thread blocks (i.e. sleeps), the operating system's scheduling primitives automatically take care of descheduling the thread and using the available computing time elsewhere. Thread and task scheduling is not something you would normally have to worry about manually.
You can only make things that are at least as complex as the simplest pieces you have. If the simplest pieces you have are mutexes, then you can't make mutexes from the pieces you have. You can only make things at least as complex as a mutex or more so. If you have any pieces simpler than a mutex, tell us what they are, and we can tell you how to make a mutex out of them.
I suppose, if you want, you can make your own mutex out of pthread mutexes and condition variables. I'm not sure what the point is, but it's trivial to do. As you noted, you can use pthread_cond_wait to wait on your own kind of mutex.
The reason the pthreads standard gives you a mutex is because it's about the most flexible of the possible synchronization primitives.
mutex.lock() should work like:
1) check lock owner
2) if lock is owned, put thread in waiting queue
3) suspend this thread until THE THREAD THAT OWNS THE LOCK sends a wake up signal. No other thread can release the lock.
These steps should be performed as an atomic operation so that the correct behaviour is followed for all threads acquiring/releasing the mutex, no matter how such calls may be interrupted and reentered from other threads.
'However, there is nothing like pthread_suspend(), then how do I do suspend?' - usually, you don't. The OS kernel provides synchronization primitives that can block threads that should not run on. To implement a 'suspend' in user-space, you can only spin-wait - something that is a good strategy in a few cases, (underloaded multi-core box where the lock is only held for a very short time), but certainly not all, (and can lead to spectacularly disastrous livelocks across whole clusters of machines).
If you want a mutex, use an OS mutex - that's what any cross-platform lib. will do.
This question already has answers here:
Closed 11 years ago.
Possible Duplicate:
Why can't you sleep while holding spinlock?
As far as I know, spinlocks should be used in short duration, and are only choices in code such as interrupt handler where sleeping (preemption) is not allowed.
However, I do not know why there is such a "rule" that there SHOULD BE no sleeping at all while holding a spinlock. I know that it is not a recommended practice (since it is detrimental in performance), but I see no reason why sleeps SHOULD NOT be allowed in spinlocks.
You cannot hold a spin lock while you acquire a semaphore, because you might have to sleep while waiting for the semaphore, and you cannot sleep while holding a spin lock (from "Linux Kernel Development" by Robert Love).
The only reason I can see is for portability reasons, because in uniprocessors, spinlocks are implemented as disabling interrupts, and by disabling interrupts, sleeping is of course not allowed (but sleeping will not break code in SMP systems).
But I am wondering if my reasoning is correct or if there are any other reasons.
There are several reasons why, at least in Linux, sleeping in spinlocks is not allowed:
If thread A sleeps in a spinlock, and thread B then tries to acquire the same spinlock, a uniprocessor system will deadlock. Thread B will never go to sleep (because spinlocks don't have the waitlist necessary to awaken B when A is done), and thread A will never get a chance to wake up.
Spinlocks are used over semaphores precisely because they're more efficient - provided you do not contend for long. Allowing sleeping means that you will have long contention periods, erasing all the benefit of using a spinlock. Your system would be faster just using a semaphore in this case.
Spinlocks are often used to synchronize with interrupt handlers, by additionally disabling interrupts. This use case is not possible if you sleep (once you enter the interrupt handler, you cannot switch back to the thread to let it wake up and finish its spinlock critical section).
Use the right tool for the right job - if you need to sleep, semaphores and mutexes are your friends.
Actually, you can sleep with interrupts disabled or some other sort of exclusion active. If you don't, the condition for which you are sleeping could change state due to an interrupt and then you would never wake up. The sleep code would normally never be entered without an elevated priority or some other critical section that encloses the execution path between the decision to sleep and the context switch.
But for spinlocks, sleep is a disaster, as the lock stays set. Other threads will spin when they hit it, and they won't stop spinning until you wake up from the sleep. That could be an eternity compared to the handful of spins expected in the worst case at a spinlock, because spinlocks exist just to synchronize access to memory locations, they aren't supposed to interact with the context-switching mechanism. (For that matter, every other thread might eventually hit the spinlock and then you would have wedged every thread of every core of the entire system.)
You cannot when you use a spin lock as it is meant to be used. Spin locks are used where really necessary to protect critical regions and shared data structures. If you acquire one while also holding a semaphore, you lock access to whichever critical region (say) your lock is attached to (it is typically a member of a specific larger data structure), while allowing this process to possibly be put to sleep. If, say, an IRQ is raised while this process sleeps, and the IRQ handler needs access to the critical region still locked away, it's blocked, which can never happen with IRQs. Obviously, you could make up examples where your spin lock isn't used the way it should be (a hypothetical spin lock attached to a nop loop, say); but that's simply not a real spin lock found in Linux kernels.
Given that pthread_spin_lock is available, when would I use it, and when should one not use them ?
i.e. how would I decide to protect some shared data structure with either a pthread mutex or a pthread spinlock ?
The short answer is that a spinlock can be better when you plan to hold the lock for an extremely short interval (for example to do nothing but increment a counter), and contention is expected to be rare, but the operation is occurring often enough to be a potential performance bottleneck. The advantages of a spinlock over a mutex are:
On unlock, there is no need to check if other threads may be waiting for the lock and waking them up. Unlocking is simply a single atomic write instruction.
Failure to immediately obtain the lock does not put your thread to sleep, so it may be able to obtain the lock with much lower latency as soon a it does become available.
There is no risk of cache pollution from entering kernelspace to sleep or wake other threads.
Point 1 will always stand, but point 2 and 3 are of somewhat diminished usefulness if you consider that good mutex implementations will probably spin a decent number of times before asking the kernel for help waiting.
Now, the long answer:
What you need to ask yourself before using spinlocks is whether these potential advantages outweigh one rare but very real disadvantage: what happens when the thread that holds the lock gets interrupted by the scheduler before it can release the lock. This is of course rare, but it can happen even if the lock is just held for a single variable-increment operation or something else equally trivial. In this case, any other threads attempting to obtain the lock will keep spinning until the thread the holds the lock gets scheduled and has a chance to release the lock. This may never happen if the threads trying to obtain the lock have higher priorities than the thread that holds the lock. That may be an extreme case, but even without different priorities in play, there can be very long delays before the lock owner gets scheduled again, and worst of all, once this situation begins, it can quickly escalate as many threads, all hoping to get the lock, begin spinning on it, tying up more processor time, and further delaying the scheduling of the thread that could release the lock.
As such, I would be careful with spinlocks... :-)
The spinlock is a "busy waiting" lock. It's main advantage is that it keeps the thread active and won't cause a context switch, so if you know that you will only be waiting for a very short time (because your critical operation is very quick), then this may give better performance than a mutex. Conversely a mutex will cause less demand on the system if the critical section takes a long time and a context switch is desirable.
TL;DR: It depends.
The safest method with a performance boost is a hybrid of the two: an adaptive mutex.
When your system has multiple cores you spin for a few thousand cycles to capture the best case of low or no contention, then defer to a full mutex to yield to other threads for long contended locks.
Both POSIX (PTHREAD_MUTEX_ADAPTIVE_NP) and Win32 (SetCriticalSectionSpinCount) have adaptive mutexes, many platforms don't have a POSIX spinlock API.
Spinlock has only interest in MP context. It is used to execute pseudo-atomical tasks. In monoprocessor system the principle is the following :
Lock the scheduler (if the task deals with interrupts, lock interrupts instead)
Do my atomic tack
Unlock the scheduler
But in MP systems we have no guaranties that an other core will not execute an other thread that could enter our code section. To prevent this the spin lock has been created, its purpose is to postpone the other cores execution preventing concurrency issue. The critical section becomes :
Lock the scheduler
SpinLock (prevent entering of other cores)
My task
SpinUnlock
Task Unlock
If the task lock is omitted, during a scheduling, an other thread could try to enter the section an will loop at 100% CPU waiting next scheduling. If this task is an high-priority one, it will produce a deadlock.
I would like to know why spin locks are used instead of semaphores inside an interrupt handler.
Semaphores cause tasks to sleep on contention, which is unacceptable for interrupt handlers. Basically, for such a short and fast task (interrupt handling) the work carried out by the semaphore is overkill. Also, spinlocks can't be held by more than one task.
Whats the problem with semaphore & mutex. And why spinlock needed ?
Can we use the semaphore or mutex in interrupt handlers. The answer is yes and no. you can use the up and unlock, but you can’t use down and lock, as these are blocking calls which put the process to sleep and we are not supposed to sleep in interrupt handlers.
Note that semaphore is not a systemV IPC techniques, its just a synchronization techniques. And there are three functions to acquires the semaphore.
down() : acquire the semaphore and put into un-interruptible state.
down_trylock() : try if lock is available, if lock is not available , don't sleep.
up() :- its useful for releasing the semaphore
So, what if we want to achieve the synchronization in interrupt handlers ? Use spinlocks.
What spinlocks will do ?
Spinlock is a lock which never yields.
Similar to mutex, it has two operations – lock and unlock.
If the lock is available, process will acquire it and will continue in the critical section and unlock it, once its done. This is similar to mutex. But, what if lock is not available ? Here, comes the interesting difference. With mutex, the process will sleep, until the lock is available. But,
in case of spinlock, it goes into the tight loop, where it
continuously checks for a lock, until it becomes available
.
This is the spinning part of the spin lock. This was designed for multiprocessor systems. But, with the preemptible kernel, even a uniprocessor system behaves like an SMP.
The problem is that interrupt handlers (IH) are triggered asynchronously and in unpredictable way, out of the scope of any other activities running in the system. In fact, IHs run out of the scope of concept of the threads and scheduling at all. Due to this all mutual exclusion primitives which rely to the scheduler are unacceptable. Because they usage in the IH can dramatically increases the interrupt handling latencies (in case of IH running in the context of low priority thread) and is able to produce deadlocks (in case of IH running in the context of thread which hold the lock).
You can look at nice and detailed description of spinlocks at http://www.makelinux.net/ldd3/chp-5-sect-5.