I have been working with a simple bi-threaded application with two threads running simultaneously. Each thread plots a figure(a triangle/ a rectangle) to a predefined and mutually exclusive section of the screen. That is, both threads, never write in each other's screen space.
After running the program for a long time, say 5hours or so, the main thread keeps running , but the other thread freezes(it doesn't die though).
I get this in backtrace o/p, which shows a deadlock I guess.
Thread 2 (Thread 0xb542f440 (LWP 2142)):
#0 0xb6e83258 in __lll_lock_wait ()
from /lib/arm-linux-gnueabihf/libpthread.so.0
#1 0xb6e7de38 in pthread_mutex_lock ()
from /lib/arm-linux-gnueabihf/libpthread.so.0
#2 0xb6ef8de4 in SDL_mutexP ()
from /usr/lib/arm-linux-gnueabihf/libSDL-1.2.so.0
#3 0xb6ef4058 in ?? () from /usr/lib/arm-linux-gnueabihf/libSDL-1.2.so.0
#4 0xb6ef4058 in ?? () from /usr/lib/arm-linux-gnueabihf/libSDL-1.2.so.0
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
Is it really a libSDL1.2 issue? Do I have to write explicit user defined mutex for every SDL_Blit/or other operations with SDL, that I am doing? Or is it something else that I am missing? How would I solve this issue?
I am using:
libSDL1.2 available in Raspberry Pi
libPthreads
libSDL_ttf
EDIT: gdb trace of the thread 2 mutex:
(gdb) p *(pthread_mutex_t*) 0xb542ed38
$3 = {_data = {_lock = 39257160, __count = 1048617, __owner = 0,
__kind = 7078021, _nusers = 0, {_spins = 0, _list = {_next = 0x0}}},
__size = "H\004W\002)\000\020\000\000\000\000\000\205\000l\000\000\000\000\000\000\000\000", __align = 39257160}
Let me know if you need more information.
This approach will only work with software surfaces (w/o hardware acceleration). With hardware surfaces it, at best, would lock and wait - which have no performance benefits anyway; and in bad case (judging by quick look at SDL source) it could end badly, by the simple logic how SDL_LockSurface works.
I'm not even sure your external mutexes would help, since hardware accelerated graphics access should almost always be locked to one thread (OpenGL is good example - each thread have it's own separate drawing context).
In short, i see no reason why you should use two threads for drawing, but in any case it isn't guaranteed to be thread-safe.
Related
I've combined my Eagle GUI library with Allegro 5 for graphics and input. When I use Allegro 5's al_register_trace_handler function to pipe the output from allegro's debugging log to my own, I get a deadlock in a thread spawned by allegro to create a win32 window and display. It specifically hangs on a call to ALLEGRO_INFO which is a logging macro used by allegro. The CRITICAL_SECTION used to prevent race conditions in the log shows up as held by my main thread. When I output the CRITICAL_SECTION in gdb, I get the following report :
(gdb) p *(trace_info.trace_mutex.cs)
$1 = {DebugInfo = 0xffffffff, LockCount = -2, RecursionCount = 176, OwningThread = 0x4750, LockSemaphore = 0x0, SpinCount = 33556432}
Thread 4750 is Main as identified by gdb and info threads.
If I don't register a trace handler with allegro, everything works fine, but if I do, and I use a debugging level of 'Debug' or 'Info' it deadlocks in the mentioned log output call. I found a case where the allegro trace function wasn't releasing the CRITICAL_SECTION in the case of a registered trace handler and I thought that would fix it by releasing the lock, but it did nothing, and the output remains the same.
Does the value of the ReferenceCount field in the critical section indicate a failure to properly unlock the log's mutex (CS) and why is the lock still held by the main thread?
I'm reaching the end of my debugging skills. I log the state of all my own threads and none of them are in contention. But the fact that main holds the CRITICAL_SECTION being used by allegro in a different thread seems to indicate I've done something wrong.
So, any help getting relevant info out of allegro and gdb would be appreciated. Like I said, it works fine if I don't register a trace handler, but if I do, it hangs on allegro code.
Advice and debugging tips welcome. Please and thank you for helping me out.
Marc
The offending missing LeaveCriticalSection call was left out in the path in allegro code where a user trace handler was used. The following patch fixed the problem.
--- C:/Users/Marc/AppData/Local/Temp/TortoiseGit/debug-619c69e3.002.c Thu May 13 11:18:03 2021
+++ E:/usr/libs/Allegro52X/src/debug.c Wed May 12 11:20:57 2021
## -300,6 +300,7 ##
if (_al_user_trace_handler) {
_al_user_trace_handler(static_trace_buffer);
static_trace_buffer[0] = '\0';
+ _al_mutex_unlock(&trace_info.trace_mutex);
return;
}
I am working on a POC app on datagram sockets, I'm on the iOS part. It's a straightforward one screen app with a couple of buttons. Anyway, my issue is with the EDT thread, the GC thread and one of my IO threads. My IO thread has a bound datagram socket waiting for messages (recvfrom). Sometimes, I see that the EDT is stuck and when I look at the iOS threads stacks, I see that:
1 - The EDT thread is sleeping waiting for a boolean to turn false
while(threadStateData->threadBlockedByGC) {
usleep(1000);
}
#3 0x0000000100e6ed02 in java_lang_Thread_sleep___long at /dist/MyApplication-src/nativeMethods.m:1231
#4 0x0000000101194c44 in java_lang_System_gc__ at /dist/MyApplication-src/java_lang_System.m:257
#5 0x0000000100c431c1 in codenameOneGcMalloc at /dist/MyApplication-src/cn1_globals.m:791
#6 0x00000001011bac4a in __NEW_com_codename1_ui_Label_1 at /dist/MyApplication-src/com_codename1_ui_Label_1.m:31
#7 0x0000000101491019 in com_codename1_ui_Label___INIT_____java_lang_String_java_lang_String at /dist/MyApplication-src/com_codename1_ui_Label.m:1402
...
2 - The GC thread is also sleeping waiting for another boolean to turn true
while(t->threadActive) {
usleep(500);
#3 0x0000000100c428d6 in codenameOneGCMark at /dist/MyApplication-src/cn1_globals.m:426
#4 0x0000000100e6e950 in java_lang_System_gcMarkSweep__ at /dist/MyApplication-src/nativeMethods.m:1078
#5 0x000000010119521d in java_lang_System_access$200__ at /dist/MyApplication-src/java_lang_System.m:331
...
A quick watch on t shows the threadId=8
t ThreadLocalData * 0x600001616eb0 0x0000600001616eb0
threadId JAVA_LONG 8
3 - My IO thread seems to be the one with id 8 (the address in memory is the same as well)
A quick watch on threadStateData shows the threadId=8
threadStateData ThreadLocalData * 0x600001616eb0 0x0000600001616eb0
threadId JAVA_LONG 8
ssize_t result = recvfrom(socketDescriptor, buffer, sob, 0, (struct sockaddr *)&receiveSockaddr, &receiveSockaddrLen);
#1 0x0000000101100a00 in -[net_etc_net_impl_NativeDatagramSocketImpl receive:param1:param2:param3:] at /dist/MyApplication-src/net_et_net_impl_NativeDatagramSocketImpl.m:131
#2 0x0000000101615f6b in net_etc_net_impl_NativeDatagramSocketImplCodenameOne_receive___int_int_java_lang_String_int_R_int at /dist/MyApplication-src/native_net_et_net_impl_NativeDatagramSocketImplCodenameOne.m:51
#3 0x0000000100f7fc9e in net_etc_net_impl_NativeDatagramSocketStub_receive___int_int_java_lang_String_int_R_int at /dist/MyApplication-src/net_etc_net_impl_NativeDatagramSocketStub.m:87
#4 0x0000000100d59939 in virtual_net_etc_net_impl_NativeDatagramSocket_receive___int_int_java_lang_String_int_R_int at /dist/MyApplication-src/net_etc_net_impl_NativeDatagramSocket.m:91
#5 0x000000010156690f in net_etc_net_DatagramSocket_receive___byte_1ARRAY_int_R_int at /dist/MyApplication-src/net_etceterum_net_DatagramSocket.m:215
So my question is: what can I do to prevent this?
Thanks for your help.
Emmanuel
See this code in our socket implementation. I suggest adding yield/resume calls in your code to let the GC work. Just make sure you don't do any Java based allocations during that time.
What happens is this:
The GC needs to run so it loops over all the active threads and tries to collect
Your thread started on the Java side so it's marked as a GC thread
It's marked as alive
The GC wants it to suspend allocations so it can GC it
The thread is unaware of this because it's in C code for a long time... Deadlock
I am out of ideas of how to figure out where my problem is coming from.
I am trying to incorporate a Async UDP handler into an existing FreeRTOS emulator, both being pthreads based. The FreeRTOS implementation is essentially a wrapper around pthreads and the UDP handler spawns a FreeRTOS task which then spawns a pthread thread for each socket, such that the spawned threads can have their own sigaction to handle that specific UDP port with a specified callback.
As a sanity check I moved the UDP handler code into a stand alone build yesterday to test it and it works without fault, found here. All valgrind checks also showing no errors. The FreeRTOS Emulator is also stable when the UDP handler is not added, found here. The unstable integration can be found here.
Now when integrating the two I get behavior I have not been able to debug successfully yet. The bug presents itself as a heisenbug in that during debugging I am not able to recreate it always. All valgrind (memcheck, helgrind and drd) are not able to recreate the bug, only reporting errors in linked libraries such as SDL2, X11, mensa graphics etc. Post morten GDB is able to capture the fault as well as when using (gdb) set disable-randomization off.
The backtrace from gdb shows me the following
(gdb) bt
#0 0x00007faa2f45a41b in pthread_kill () from /usr/lib/libpthread.so.0
#1 0x0000564392f5c93b in prvResumeThread (xThreadId=0) at /home/alxhoff/git/GitHub/FreeRTOS-Emulator/lib/FreeRTOS_Kernel/portable/GCC/Posix/port.c:561
#2 0x0000564392f5c38b in vPortYield () at /home/alxhoff/git/GitHub/FreeRTOS-Emulator/lib/FreeRTOS_Kernel/portable/GCC/Posix/port.c:329
#3 0x0000564392f5d986 in xQueueGenericReceive (xQueue=0x564396692bd0, pvBuffer=0x0, xTicksToWait=4294967295, xJustPeeking=0) at /home/alxhoff/git/GitHub/FreeRTOS-Emulator/lib/FreeRTOS_Kernel/queue.c:1376
#4 0x0000564392f5b0d3 in vDemoTask1 (pvParameters=0x0) at /home/alxhoff/git/GitHub/FreeRTOS-Emulator/src/main.c:338
#5 0x0000564392f5c754 in prvWaitForStart (pvParams=0x5643966b2780) at /home/alxhoff/git/GitHub/FreeRTOS-Emulator/lib/FreeRTOS_Kernel/portable/GCC/Posix/port.c:496
#6 0x00007faa2f4524cf in start_thread () from /usr/lib/libpthread.so.0
#7 0x00007faa2efcd2d3 in clone () from /usr/lib/libc.so.6
The problem appears to be that prvResumeThread is not being passed a valid thread id as seen in #1. Going into the FreeRTOS sources I believe that this should not be the case as the same threads are created when the UDP handler and it's respective task are added, their addition somehow leads to FreeRTOS's pxCurrentTCB becoming invalid when executing xTaskGetCurrentTaskHandle which retrieves the thread handle for the faulting prvResumeThread call in #1 of the backtrace. Moving the task creation order around leads to the same error which makes me think I am dealing with some sort of memory leak but given that I cannot reproduce the error with valgrind I am unsure of how to diagnose the error.
I am worried this seems like a "debug my program" post but I am unsure of what methods or tools I can utilize to further my diagnosis, given my limited experience with multi-threaded debugging, and am in need of a push in the right direction.
Cheers
I have observed a blocking behavior of C NIFs when they were being called concurrently by many Erlang processes. Can it be made non-blocking? Is there a mutex at work here which I'm not able to comprehend?
P.S. A basic "Hello world" NIF can be tested by making it sleep for a hundred microseconds in case of a particular PID calling it. It can be observed that the other PIDs calling the NIF wait for that sleep to execute before their execution.
Non blocking behavior would be beneficial in cases where concurrency might not pose an issue(e.g. array push, counter increment).
I am sharing the links to 4 gists which comprise of a spawner, conc_nif_caller and niftest module respectively. I have tried to tinker with the value of Val and I have indeed observed a non-blocking behavior. This is confirmed by assigning a large integer parameter to the spawn_multiple_nif_callers function.
Links
spawner.erl,conc_nif_caller.erl,niftest.erl and finally niftest.c.
The line below is printed by the Erlang REPL on my Mac.
Erlang/OTP 17 [erts-6.0] [source] [64-bit] [smp:4:4] [async-threads:10] [hipe] [kernel-poll:false] [dtrace]
NIF's themselves don't have any mutex. You could implement one in C, and there is one when you load NIF's object, but this should be done only once with loading module.
One thing that's might be happening (and I would bet that's what is going on), is you C code messes up Erlang scheduler(s).
A native function that do lengthy work before returning will degrade responsiveness of the VM, and may cause miscellaneous strange behaviors. Such strange behaviors include, but are not limited to, extreme memory usage, and bad load balancing between schedulers. Strange behaviors that might occur due to lengthy work may also vary between OTP releases.
and description of what lengty work means and how you could solve it.
In very few words (with quite few simplifications):
For core one scheduler is created. Each has a list of processes which he can run. If ones scheduler list is empty, he will try to still work from another one. This can fail, if there is nothing (or not enough) to still.
Erlang schedulers spends some amount of work in one process, than moves to another, spend there some amount of work, and move to another. And so on, and so one. This is very similar to scheduling in system processes.
One thing that very important here is calculating amount of work. As default each function call has assigned some number of reductions. Addition could have two, calling function in your module will have one, sending a message also a one, some build-in could have more (like list_to_binary). If we collect 2 000 reductions we move to another process.
So what is the cost of your C function? It's only one reduction.
Code like
loop() ->
call_nif_function(),
loop().
could be taking all whole hour, but scheduler will be stuck in this one process, because he still haven't count to 2 000 reductions. Or to put it in other words, he could be stuck inside NIF without possibility to move forward (at least any time soon).
There are few ways around this but general rule is stat NIF's should not take long time. So if you have long running C code, maybe you should use drivers instead. They should be much easier to implement and manage, that tinkering with NIF's.
I think the responses about long-running NIFs are off the mark, since your question says you're running some simple "hello world" code and are sleeping for just 100 us. It's true that ideally a NIF call shouldn't take more than a millisecond, but your NIFs likely won't cause scheduler issues unless they run consistently for tens of milliseconds at a time or more.
I have a simple NIF called rev/1 that takes a string argument, reverses it, and returns the reversed string. I stuck a usleep call in the middle of it, then spawned 100 concurrent Erlang processes to invoke it. The two thread stacktraces shown below, based on Erlang/OTP 17.3.2, show two Erlang scheduler threads both inside the rev/1 NIF simultaneously, one at a breakpoint I set on the NIF C function itself, the other blocked on the usleep inside the NIF:
Thread 18 (process 26016):
#0 rev (env=0x1050d0a50, argc=1, argv=0x102ecc340) at nt2.c:9
#1 0x000000010020f13d in process_main () at beam/beam_emu.c:3525
#2 0x00000001000d5b2f in sched_thread_func (vesdp=0x102829040) at beam/erl_process.c:7719
#3 a0x0000000100301e94 in thr_wrapper (vtwd=0x7fff5fbff068) at pthread/ethread.c:106
#4 0x00007fff8a106899 in _pthread_body ()
#5 0x00007fff8a10672a in _pthread_start ()
#6 0x00007fff8a10afc9 in thread_start ()
Thread 17 (process 26016):
#0 0x00007fff8a0fda3a in __semwait_signal ()
#1 0x00007fff8d205dc0 in nanosleep ()
#2 0x00007fff8d205cb2 in usleep ()
#3 0x000000010062ee65 in rev (env=0x104fcba50, argc=1, argv=0x102ec8280) at nt2.c:21
#4 0x000000010020f13d in process_main () at beam/beam_emu.c:3525
#5 0x00000001000d5b2f in sched_thread_func (vesdp=0x10281ed80) at beam/erl_process.c:7719
#6 0x0000000100301e94 in thr_wrapper (vtwd=0x7fff5fbff068) at pthread/ethread.c:106
#7 0x00007fff8a106899 in _pthread_body ()
#8 0x00007fff8a10672a in _pthread_start ()
#9 0x00007fff8a10afc9 in thread_start ()
If there were any mutexes within the Erlang emulator preventing concurrent NIF access, the stacktraces would not show both threads inside the C NIF.
It would be nice if you were to post your code so those willing to help resolve this issue could see what you're doing and perhaps help you find any bottlenecks. It would also be helpful if you were to tell us what version(s) of Erlang/OTP you're using.
NIF calls block the scheduler to which the process that called them is bound. So, for your example, if those other processes are on the same scheduler, they cannot call into the NIF until the first process finishes.
You cannot make an NIF call non-blocking in this regard. You can, however, spawn your own threads and offload the brunt of your work to them.
Such threads can send messages to local Erlang processes (processes on the same machine), and as such you can still get the response you desire by waiting for your spawned thread to send back a message.
A bad example:
static ERL_NIF_TERM my_function(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[]) {
MyStruct* args = new MyStruct(); // I like C++; so sue me
args->caller = enif_self();
ErlNifTid thread_id;
// Please remember, you must at some point rejoin the thread,
// so keep track of the thread_id
enif_thread_create("my_function_thread", &thread_id, my_worker_function, (void*)args, NULL);
return enif_make_atom(env, "ok");
}
void* my_worker_function(void* args) {
sleep(100);
ErlNifEnv* msg_env = enif_alloc_env();
ERL_NIF_TERM msg = enif_make_atom(msg_env, "ok");
enif_send(NULL, args->caller, msg_env, msg);
delete args;
return NULL;
}
And in your erlang source:
test_nif() ->
my_nif:my_function(),
receive
ok -> ok
end.
Something to that effect, anyway.
I have created a TCP client that connects to a listening server.
We implemeted TCP keep alive also.
Some times the client crashes and core dumped.
Below are the core dump traces.
Problem is in linux kernel version Update 4, kernel 2.6.9-42.0.10.
we had two core dumps.
(gdb) where
#0 0x005e77a2 in _dl_sysinfo_int80 () from /ddisk/d303/dumps/mhx239131/ld-
linux.so.2
#1 0x006c8bd1 in connect () from /ddisk/d303/dumps/mhx239131/libc.so.6
#2 0x08057863 in connect_to_host ()
#3 0x08052f38 in open_ldap_connection ()
#4 0x0805690a in new_connection ()
#5 0x08052cc9 in ldap_open ()
#6 0x080522cf in checkHosts ()
#7 0x08049b36 in pollLDEs ()
#8 0x0804d1cd in doOnChange ()
#9 0x0804a642 in main ()
(gdb) where
#0 0x005e77a2 in _dl_sysinfo_int80 () from /ddisk/d303/dumps/mhx239131/ld-
linux.so.2
#1 0x0068ab60 in __nanosleep_nocancel (
from /ddisk/d303/dumps/mhx239131/libc.so.6
#2 0x080520a2 in Sleep ()
#3 0x08049ac1 in pollLDEs ()
#4 0x0804d1cd in doOnChange ()
#5 0x0804a642 in main ()
We have tried to reproduce the problem in our environment, but we could not.
What would cause the core file?
Please help me to avoid such situation.
Thanks,
Naga
_dl_sysinfo_int80 is just a function which does a system call into the kernel. So the core dump is happening on a system call (probably the one used by connect in the first example and nanosleep in the second example), probably because you are passing invalid pointers.
The invalid pointers could be because the code which calls these functions being broken or because somewhere else in the program is broken and corrupting the program's memory.
Take a look at two frames above (frame #2) in the core dump for both examples and check the parameters being passed. Unfortunately, it seems you did not compile with debug information, making it harder to see them.
Additionally, I would suggest trying valgrind and seeing if it finds something.
Your program almost cetainly did not coredump in either of the above places.
Most likely, you either have multiple threads in your process (and some other thread caused the core dump), or something external caused your process to die (such as 'kill -SIGABRT <pid>').
If you do have multiple threads, GDB 'info threads' and 'thread apply all where' are likely to provide further clues.