Futexes and blocking in the kernel - c

Question

I was reading some documentation and trying out a few samples of code that issue the futex system call in Linux. I read that if a mutex is acquired by thread_a using FUTEX_LOCK_PI, and say if another thread thread_b tries to acquire the same, the latter(thread_b) is blocked in the kernel.
What exactly is meant by "blocked in the kernel"? Is it that the execution of thread_b in userspace does not resume? In the following example, the second thread seems to issue the printf after the syscall as well. Would that not suggest that it is not blocked in the kernel? What am I missing?
/*
* This example demonstrates that when futex_lock_pi is called twice, the
* second call is blocked inside the kernel.
*/
#include <stdio.h>
#include <unistd.h>
#include <pthread.h>
int mutex = 0;
#define __NR_futex 240
#define FUTEX_WAIT 0
#define FUTEX_WAKE 1
#define FUTEX_FD 2
#define FUTEX_REQUEUE 3
#define FUTEX_CMP_REQUEUE 4
#define FUTEX_WAKE_OP 5
#define FUTEX_LOCK_PI 6
#define FUTEX_UNLOCK_PI 7
#define FUTEX_TRYLOCK_PI 8
#define FUTEX_WAIT_REQUEUE_PI 11
#define FUTEX_CMP_REQUEUE_PI 12
void *thread(void *arg) {
int ret = 0;
pid_t tid = gettid();
printf("Entering thread[%d]\n", tid);
ret = syscall(__NR_futex, &mutex, FUTEX_LOCK_PI, 1, NULL, NULL, 0);
printf("Thread %d returned from kernel\n", tid);
printf("Value of mutex=%d\n", mutex);
printf("Return value from syscall=%d\n", ret);
}
int main(int argc, const char *argv[]) {
pthread_t t1, t2;
if (pthread_create(&t1, NULL, thread, NULL)) {
printf("Could not create thread 1\n");
return -1;
}
if (pthread_create(&t2, NULL, thread, NULL)) {
printf("Could not create thread 2\n");
return -1;
}
// Loop infinitely
while ( 1 ) { }
return 0;
}
The output can be found below :-
Entering thread[952]
Thread 952 returned from kernel
Entering thread[951]
Value of mutex=-2147482696
Return value from syscall=0
Thread 951 returned from kernel
Value of mutex=-1073740873
Return value from syscall=0

Answer 1

blocked in the kernel means that the thread is put to sleep state until an events wakes it up, in your case the event is the mutex to get available.
I changed a little bit your code to illustrate:
#include <stdio.h>
#include <unistd.h>
#include <pthread.h>
#include <sys/syscall.h>
static pthread_mutex_t mutex = PTHREAD_MUTEX_INITIALIZER;
void *thread(void *unused) {
printf("Entering thread %d\n", syscall(SYS_gettid));
pthread_mutex_lock(&mutex);
sleep(2);
pthread_mutex_unlock(&mutex);
return NULL;
}
int main(int argc, const char *argv[]) {
pthread_t t1, t2;
printf("using mutex #%p\n", &mutex);
if (pthread_create(&t1, NULL, thread, &t1)) {
printf("Could not create thread 1\n");
return -1;
}
if (pthread_create(&t2, NULL, thread, &t2)) {
printf("Could not create thread 2\n");
return -1;
}
pthread_join(t1, NULL);
pthread_join(t2, NULL);
return 0;
}
The code is basically the same but I use explicitly a mutex of the pthread lib.
If I run this code with strace, I get:
using mutex #0x6010a0
Process 19688 attached
Entering thread 19688
Process 19689 attached
Entering thread 19689
[pid 19689] futex(0x6010a0, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid 19688] futex(0x6010a0, FUTEX_WAKE_PRIVATE, 1) = 1
[pid 19689] <... futex resumed> ) = 0
[pid 19688] +++ exited with 0 +++
[pid 19689] futex(0x6010a0, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 19689] +++ exited with 0 +++
+++ exited with 0 +++
You may notice that we do not see the first thread taking the mutex, BUT we can see the next one caling futex for waiting (FUTEX_WAIT_PRIVATE). This is due to the fact that futex doesn't get called when the mutex is being taken.
But then you can see that the first thread (id 19688 here) then eventually calls futex(FUTEX_WAKE_PRIVATE) which tells the kernel to wake the other thread now the utex is free.
You may have noticed that the first call
[pid 19689] futex(0x6010a0, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
is unfinished, which means that the process was hanged waiting for the kernel to finish the job and give back the hand.
Then [pid 19689] <... futex resumed> ) = 0 sketches that the call is eventually finished (obviously because the mutex was released)