Process threads stuck with __pthread_enable_asynccancel () - c

Question

I am debugging an issue with a multi-threaded TCP server application on CentOS platform. The application suddenly stopped processing connections. Even the event logging with syslog was not seen in the log files. It was as if the application become a black hole.
I killed the process with signal 11 to get the core dump. In the core dump I observed that all threads are stuck with similar patterns.
Ex:
Thread 19 (Thread 0xb2de8b70 (LWP 3722)):
#0 0x00c9e424 in __kernel_vsyscall ()
#1 0x00c17189 in __pthread_enable_asynccancel () from /lib/libpthread.so.0
#2 0x080b367d in server_debug_stats_timer_handler (tmp=0x0) at server_debug_utils.c:75 ==> this line as a event print with syslog(...)
All most all threads are attempting a syslog(...) print but gets stuck with __pthread_enable_asynccancel ().
What is being done with __pthread_enable_asynccancel () and why it isn't returning?
Here is info reg from the thread mentioned:
(gdb) info reg
eax 0xfffffe00 -512
ecx 0x80 128
edx 0x2 2
ebx 0x154ea3c4 357475268
esp 0xb2de8174 0xb2de8174
ebp 0xb2de81a8 0xb2de81a8
esi 0x0 0
edi 0x0 0
eip 0xc9e424 0xc9e424 <__kernel_vsyscall+16>
eflags 0x200246 [ PF ZF IF ID ]
cs 0x73 115
ss 0x7b 123
ds 0x7b 123
es 0x7b 123
fs 0x0 0
gs 0x33 51
(gdb)
(gdb) print $orig_eax
$1 = 240
($orig_eax = 240 is SYS_futex)
One of the thread state is as shown below:
Thread 27 (Thread 0xa97d9b70 (LWP 3737)):
#0 0x00c9e424 in __kernel_vsyscall ()
#1 0x00faabb3 in inet_ntop () from /lib/libc.so.6
#2 0x00f20e76 in freopen64 () from /lib/libc.so.6
#3 0x00f96a55 in fcvt_r () from /lib/libc.so.6
#4 0x00f96fd7 in qfcvt_r () from /lib/libc.so.6
#5 0x00a19932 in app_signal_handler (signum=11) at appBaseClass.cpp:920
#6 <signal handler called>
#7 0x00f2aec5 in sYSMALLOc () from /lib/libc.so.6
#8 0x0043431a in CRYPTO_free () from /usr/local/lr/packages/stg_app/5.3.8/lib/ssl/libcrypto.so.10
#9 0x00000000 in ?? ()
(gdb) print $orig_eax $5 = 240`

Answer 1

Your stuck threads are in the futex() syscall, probably on an internal glibc lock taken within malloc() (the syslog() call allocates memory internally).
The reason for this deadlock isn't apparent, but I have two suggestions:
If you call syslog() (or any other non-AS-safe function, like printf()) from a signal handler anywhere in your program, that could cause a deadlock like this;
It's possible you're being bitten by a bug in futex() that was introduced in Linux 3.14 and fixed in 3.18. The bug was also backported RHEL 6.6 so would have been present for a while in CentOS too. The effect of this bug was to cause processes to fail to wake up from a FUTEX_WAIT when they should have.