I am out of ideas of how to figure out where my problem is coming from.
I am trying to incorporate a Async UDP handler into an existing FreeRTOS emulator, both being pthreads based. The FreeRTOS implementation is essentially a wrapper around pthreads and the UDP handler spawns a FreeRTOS task which then spawns a pthread thread for each socket, such that the spawned threads can have their own sigaction to handle that specific UDP port with a specified callback.
As a sanity check I moved the UDP handler code into a stand alone build yesterday to test it and it works without fault, found here. All valgrind checks also showing no errors. The FreeRTOS Emulator is also stable when the UDP handler is not added, found here. The unstable integration can be found here.
Now when integrating the two I get behavior I have not been able to debug successfully yet. The bug presents itself as a heisenbug in that during debugging I am not able to recreate it always. All valgrind (memcheck, helgrind and drd) are not able to recreate the bug, only reporting errors in linked libraries such as SDL2, X11, mensa graphics etc. Post morten GDB is able to capture the fault as well as when using (gdb) set disable-randomization off.
The backtrace from gdb shows me the following
(gdb) bt
#0 0x00007faa2f45a41b in pthread_kill () from /usr/lib/libpthread.so.0
#1 0x0000564392f5c93b in prvResumeThread (xThreadId=0) at /home/alxhoff/git/GitHub/FreeRTOS-Emulator/lib/FreeRTOS_Kernel/portable/GCC/Posix/port.c:561
#2 0x0000564392f5c38b in vPortYield () at /home/alxhoff/git/GitHub/FreeRTOS-Emulator/lib/FreeRTOS_Kernel/portable/GCC/Posix/port.c:329
#3 0x0000564392f5d986 in xQueueGenericReceive (xQueue=0x564396692bd0, pvBuffer=0x0, xTicksToWait=4294967295, xJustPeeking=0) at /home/alxhoff/git/GitHub/FreeRTOS-Emulator/lib/FreeRTOS_Kernel/queue.c:1376
#4 0x0000564392f5b0d3 in vDemoTask1 (pvParameters=0x0) at /home/alxhoff/git/GitHub/FreeRTOS-Emulator/src/main.c:338
#5 0x0000564392f5c754 in prvWaitForStart (pvParams=0x5643966b2780) at /home/alxhoff/git/GitHub/FreeRTOS-Emulator/lib/FreeRTOS_Kernel/portable/GCC/Posix/port.c:496
#6 0x00007faa2f4524cf in start_thread () from /usr/lib/libpthread.so.0
#7 0x00007faa2efcd2d3 in clone () from /usr/lib/libc.so.6
The problem appears to be that prvResumeThread is not being passed a valid thread id as seen in #1. Going into the FreeRTOS sources I believe that this should not be the case as the same threads are created when the UDP handler and it's respective task are added, their addition somehow leads to FreeRTOS's pxCurrentTCB becoming invalid when executing xTaskGetCurrentTaskHandle which retrieves the thread handle for the faulting prvResumeThread call in #1 of the backtrace. Moving the task creation order around leads to the same error which makes me think I am dealing with some sort of memory leak but given that I cannot reproduce the error with valgrind I am unsure of how to diagnose the error.
I am worried this seems like a "debug my program" post but I am unsure of what methods or tools I can utilize to further my diagnosis, given my limited experience with multi-threaded debugging, and am in need of a push in the right direction.
Cheers
Related
I'm trying to debug my application that is based on an STM32F3 uC running FreeRTOS. I have manually set the PSP to an invalid value (e.g. 0) at random places in thread context in the application expecting my memManageFault/busFault/usageFault/hardFault handlers to fire. Unfortunately none of the fault handlers are executed, but the core locks up on the first push to the invalid stack. What am I missing?
Some more details from the lockup state:
SCB->SHCSR: 0x74001 (all three faultHandlers are enabled, busFault pending, memFault active)
SCB->HFSR:0x40000000 (fault escalated to hardFault even though all handlers are defined and enabled)
SCB->CFSR: 0x28601 (BFAR valid, precise error)
SCB->BFAR/SCB->MMFAR: 0xfffffff7 (erroneous SP after sub, I assume)
PRIMASK/FAULTMASK/BASEPRI: 0
MSP: 0x2000ffe0 (still valid, the handler should run just fine)
Any ideas are welcome.
It seems like once again the core is right and I am wrong. The mistake I made was that although I have implemented the HardFault_Handler as a naked function, all the other fault handlers were simple application failure hooks implemented in C, trying to access the stack in whatever context they interrupted. Needless to say, things went dirty quickly.
Implementing all handlers in asm solved the issue of the core locking up on corrupted a SP.
busFault pending, memFault active - memFault has caused busError - and it kills the micro
Exception stacking uses the same stack as the current context. By providing an invalid stack pointer, you've prevented any of the exception handlers being able to complete. Lockup specifically addresses this scenario.
I have an application that I am porting from the Keil IDE to build with the GNU toolchain due to license issues. I have successfully be able to set up, build, flash and run the application on the device.
The application on the GNU side is for some reason is getting stuck in the weak linked IRQ handler for the WWDG which is an infinite loop. The application does not enable the WWDG, and it is disabled at reset by default. I have also verified that the configuration registers are at their default startup values.
The only difference, other than compilers, are the linker and startup files. However, both the startup files, and linker files used by both toolchains are defaults generated by STM.
Any idea what may be causing this? I'm about at my wits end here.
Using the stm32f103XX, let me know if any other information would be helpful.
EDIT:
Using the comments below I was able to ascertain that it is, in fact, the HardFault_Handler that is being triggered.
I have included the backtrace output below if that may be of help
GDB BT:
0 HardFault_Handler ()
1 (signal handler called)
2 0x720a3de in ?? ()
3 0x80005534 in foo ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
2 things stand out to me, though im no gdb expert. 1) foo is not a function, it is a const array of chars and 2) 0x0720a3de is not a valid memory address the flash address range starts at 0x08000000
So thanks to the kick in the pants by D Krueger. I was able to figure out that the HardFault_Handler was what was actually being called. So, anyone that stumbles on this post, verify which IRQ is truly being called by writing temporary functions to cover the likely culprits i.e. HardFault. The true issue for the IRQ call is a bad memory access by memcpy which I am on my way to solving next.
I had exactly the same error as OP (apparent WWDG interrupt, but actually the HardFault_Handler firing) when porting an example for the STM32F3 Discovery board to compile in CooCox CoIDE 1.7.7 with STM32Cube F3 libraries (v1.1.0). The code ran fine as long as I didn't try using any interrupts, but as soon as I turned on the SysTick timer interrupt, the HardFault exception tripped.
The problem was that I had neglected to include the stm32f3xx_it.h and stm32f3xx_it.c files in the project. Their absence wasn't causing any compiler warnings/errors. Once they were compiled & linked in, the code with interrupts ran fine.
I've had this problem due to the same root cause as awilhite. I'm using Atollic TrueStudio 8.0.0. I used it to start a project for STM32F030 and (probably manually) added libraries folder with stm32f0xx.h, which defines ADC1_IRQn (IRQ channel number used in NVIC setup).
And I implemented ADC1_IRQHandler(void) in my main.c (as I'm used to and it always worked so far -- x_IRQn -> x_IRQHandler)
But after 2 days frustration, I found out, that startup_stm32f0xx.s in my project defines ADC1_COMP_IRQHandler.
So, ultimately, my ADC interrupt handler was undefined and when the ADC generated the interrupt, the program crashed (WWDG interrupt).
I hope this helps to people like me, who think they did implement their handler but in fact, they did not.
I had a very similar problem when merging two projects generated separately by STM32CubeMX for an STM32F2XX processor. One project was using the Ethernet peripheral, while the other was not. Besides that one difference, the two projects used the same set of peripherals.
After integrating the two projects together by manually copying files, the application would end up in the WWDG_IRQHandler after starting the first task (when interrupts are enabled for the first time). I first confirmed that the WDGA bit of the WWDG register was indeed not set and, therefore, that the WWDG peripheral was disabled. Next, I verified that the interrupt vector table was initialized correctly. Finally, after several hours of digging, I realized that I had not defined the ETH_IRQHandler function in stm32f2xx_it.c, which provoked the Ethernet interrupt to be handled by the default handler, masking itself as the WWDG_IRQHandler -- likely due to optimization.
The core problem is that the Default Handler is called instead of another irq handler. I doubt that our situations are the same but here is my solution:
I was working on a c++ project, the same happened to me. This was the first time I made a project from scratch & with CMSIS. After some unsuccessful attempts I went through a generated project when I noticed that in the stm32xxxx_it.h the IRQ handler function prototypes are guarded by these:
extern "C"
{
void TIM7_IRQHandler(void);
}
With these guards the linker could find my own interrupt handler functions.
I'll expand a bit on what led me here, and how I use the insight from #Mike to correct it.
I had a project running fine on a demo project in Eclipse SW4STM32, but with sources and headers scattered all over the place so I wanted to have a more "compact" project easier to customize and use as a base for minor modifications (and easier to follow in Git).
I created an empty AC6 project targetting the same board. It generated the HAL drivers, the startup_stm32.s and LinkerScript.ld. I then copied all of the .c and corresponding .h from the original project to my new project (which was a pain in itself because they were scattered in BSP, CMSIS, Components, Middlewares, etc. directories). Everything compiled and seemed to work, until I started modifying a bit.
In the debugger, it seemed all function calls were working until the while(1) main loop where I ended up in the Default_Handler defined in the startup_stm32.s, seemingly from WWDG_IRQHandler. That was, in fact, the default IRQ handler for not-user-defined handlers (WWDG_IRQHandler being the first one declared, it was reported as such by gdb, as indicated by #D Krüger).
I started looking at compiler and linker options or linker script, without much luck, until I realized the only file I didn't check was the startup_stm32.s, which was indeed different.
I blindly copy-pasted it and voilà!
The explanation I could give is that the STM32 is calling IRQ handlers defined in the startup_stm32.s when interrupt occur, all of them initially pointing to Default_Handler() (later overriden by the linker). So if a .c file you copied defines a handler with a slightly different name (but consistent with its own startup_xxx.s), you'll end up with the Default_Handler() being called (which is an infinite loop) instead of the one you defined. And things go wrong.
See https://www.freertos.org/Debugging-Hard-Faults-On-Cortex-M-Microcontrollers.html for more information.
N.B. I'm not happy to blindly copy-paste without fully understanding, but time constraints and milestones usually push you to territories you're not happy to explore...
Will add my 5 cents. I had this issue on stm32h7, but for me the cause was that the cube "forgot" to add TIM16_IRQHandler when TIM16 is used as the timebase source. It was not happening at the beginning but after several code regenerations. Looks like a bug in the cube, as the TIM16 was still set, but the interrupt handler got removed. So toggking to TIM17 and back resolved the issue.
In my case, I had a function written in the GCC assembly that was migrated from the ARM assembly. The problem went away after I had added the .thumb_func line to the assembly file.
I was getting this error:
(gdb) c
+c
Continuing.
Program received signal SIGINT, Interrupt.
WWDG_IRQHandler () at ...startup_stm32f40_41xxx.s:121
(gdb) bt
#0 WWDG_IRQHandler () at ...startup_stm32f40_41xxx.s:12
#1 <signal handler called>
#2 RTOS_SysTick_Handler () at ...osKernel.s:18
#3 <signal handler called>
#4 0x0800021a in task0 () at ...main.cpp:10
#5 0x08000214 in frame_dummy ()
#6 0x00000000 in ?? ()
RTOS_SysTick_Handler is a function written in assembly and the WWDG_IRQHandler was always triggered before any first assembly instructions in that function (tried different instructions and it didn't change anything).
I was doing some tweaks around the C code and at some point, I hit another handler: UsageFault which led me to the .thumb_func hint: ARM Cortex M4 SVC_Handler "UsageFault".
I have observed a blocking behavior of C NIFs when they were being called concurrently by many Erlang processes. Can it be made non-blocking? Is there a mutex at work here which I'm not able to comprehend?
P.S. A basic "Hello world" NIF can be tested by making it sleep for a hundred microseconds in case of a particular PID calling it. It can be observed that the other PIDs calling the NIF wait for that sleep to execute before their execution.
Non blocking behavior would be beneficial in cases where concurrency might not pose an issue(e.g. array push, counter increment).
I am sharing the links to 4 gists which comprise of a spawner, conc_nif_caller and niftest module respectively. I have tried to tinker with the value of Val and I have indeed observed a non-blocking behavior. This is confirmed by assigning a large integer parameter to the spawn_multiple_nif_callers function.
Links
spawner.erl,conc_nif_caller.erl,niftest.erl and finally niftest.c.
The line below is printed by the Erlang REPL on my Mac.
Erlang/OTP 17 [erts-6.0] [source] [64-bit] [smp:4:4] [async-threads:10] [hipe] [kernel-poll:false] [dtrace]
NIF's themselves don't have any mutex. You could implement one in C, and there is one when you load NIF's object, but this should be done only once with loading module.
One thing that's might be happening (and I would bet that's what is going on), is you C code messes up Erlang scheduler(s).
A native function that do lengthy work before returning will degrade responsiveness of the VM, and may cause miscellaneous strange behaviors. Such strange behaviors include, but are not limited to, extreme memory usage, and bad load balancing between schedulers. Strange behaviors that might occur due to lengthy work may also vary between OTP releases.
and description of what lengty work means and how you could solve it.
In very few words (with quite few simplifications):
For core one scheduler is created. Each has a list of processes which he can run. If ones scheduler list is empty, he will try to still work from another one. This can fail, if there is nothing (or not enough) to still.
Erlang schedulers spends some amount of work in one process, than moves to another, spend there some amount of work, and move to another. And so on, and so one. This is very similar to scheduling in system processes.
One thing that very important here is calculating amount of work. As default each function call has assigned some number of reductions. Addition could have two, calling function in your module will have one, sending a message also a one, some build-in could have more (like list_to_binary). If we collect 2 000 reductions we move to another process.
So what is the cost of your C function? It's only one reduction.
Code like
loop() ->
call_nif_function(),
loop().
could be taking all whole hour, but scheduler will be stuck in this one process, because he still haven't count to 2 000 reductions. Or to put it in other words, he could be stuck inside NIF without possibility to move forward (at least any time soon).
There are few ways around this but general rule is stat NIF's should not take long time. So if you have long running C code, maybe you should use drivers instead. They should be much easier to implement and manage, that tinkering with NIF's.
I think the responses about long-running NIFs are off the mark, since your question says you're running some simple "hello world" code and are sleeping for just 100 us. It's true that ideally a NIF call shouldn't take more than a millisecond, but your NIFs likely won't cause scheduler issues unless they run consistently for tens of milliseconds at a time or more.
I have a simple NIF called rev/1 that takes a string argument, reverses it, and returns the reversed string. I stuck a usleep call in the middle of it, then spawned 100 concurrent Erlang processes to invoke it. The two thread stacktraces shown below, based on Erlang/OTP 17.3.2, show two Erlang scheduler threads both inside the rev/1 NIF simultaneously, one at a breakpoint I set on the NIF C function itself, the other blocked on the usleep inside the NIF:
Thread 18 (process 26016):
#0 rev (env=0x1050d0a50, argc=1, argv=0x102ecc340) at nt2.c:9
#1 0x000000010020f13d in process_main () at beam/beam_emu.c:3525
#2 0x00000001000d5b2f in sched_thread_func (vesdp=0x102829040) at beam/erl_process.c:7719
#3 a0x0000000100301e94 in thr_wrapper (vtwd=0x7fff5fbff068) at pthread/ethread.c:106
#4 0x00007fff8a106899 in _pthread_body ()
#5 0x00007fff8a10672a in _pthread_start ()
#6 0x00007fff8a10afc9 in thread_start ()
Thread 17 (process 26016):
#0 0x00007fff8a0fda3a in __semwait_signal ()
#1 0x00007fff8d205dc0 in nanosleep ()
#2 0x00007fff8d205cb2 in usleep ()
#3 0x000000010062ee65 in rev (env=0x104fcba50, argc=1, argv=0x102ec8280) at nt2.c:21
#4 0x000000010020f13d in process_main () at beam/beam_emu.c:3525
#5 0x00000001000d5b2f in sched_thread_func (vesdp=0x10281ed80) at beam/erl_process.c:7719
#6 0x0000000100301e94 in thr_wrapper (vtwd=0x7fff5fbff068) at pthread/ethread.c:106
#7 0x00007fff8a106899 in _pthread_body ()
#8 0x00007fff8a10672a in _pthread_start ()
#9 0x00007fff8a10afc9 in thread_start ()
If there were any mutexes within the Erlang emulator preventing concurrent NIF access, the stacktraces would not show both threads inside the C NIF.
It would be nice if you were to post your code so those willing to help resolve this issue could see what you're doing and perhaps help you find any bottlenecks. It would also be helpful if you were to tell us what version(s) of Erlang/OTP you're using.
NIF calls block the scheduler to which the process that called them is bound. So, for your example, if those other processes are on the same scheduler, they cannot call into the NIF until the first process finishes.
You cannot make an NIF call non-blocking in this regard. You can, however, spawn your own threads and offload the brunt of your work to them.
Such threads can send messages to local Erlang processes (processes on the same machine), and as such you can still get the response you desire by waiting for your spawned thread to send back a message.
A bad example:
static ERL_NIF_TERM my_function(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[]) {
MyStruct* args = new MyStruct(); // I like C++; so sue me
args->caller = enif_self();
ErlNifTid thread_id;
// Please remember, you must at some point rejoin the thread,
// so keep track of the thread_id
enif_thread_create("my_function_thread", &thread_id, my_worker_function, (void*)args, NULL);
return enif_make_atom(env, "ok");
}
void* my_worker_function(void* args) {
sleep(100);
ErlNifEnv* msg_env = enif_alloc_env();
ERL_NIF_TERM msg = enif_make_atom(msg_env, "ok");
enif_send(NULL, args->caller, msg_env, msg);
delete args;
return NULL;
}
And in your erlang source:
test_nif() ->
my_nif:my_function(),
receive
ok -> ok
end.
Something to that effect, anyway.
I have been working with a simple bi-threaded application with two threads running simultaneously. Each thread plots a figure(a triangle/ a rectangle) to a predefined and mutually exclusive section of the screen. That is, both threads, never write in each other's screen space.
After running the program for a long time, say 5hours or so, the main thread keeps running , but the other thread freezes(it doesn't die though).
I get this in backtrace o/p, which shows a deadlock I guess.
Thread 2 (Thread 0xb542f440 (LWP 2142)):
#0 0xb6e83258 in __lll_lock_wait ()
from /lib/arm-linux-gnueabihf/libpthread.so.0
#1 0xb6e7de38 in pthread_mutex_lock ()
from /lib/arm-linux-gnueabihf/libpthread.so.0
#2 0xb6ef8de4 in SDL_mutexP ()
from /usr/lib/arm-linux-gnueabihf/libSDL-1.2.so.0
#3 0xb6ef4058 in ?? () from /usr/lib/arm-linux-gnueabihf/libSDL-1.2.so.0
#4 0xb6ef4058 in ?? () from /usr/lib/arm-linux-gnueabihf/libSDL-1.2.so.0
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
Is it really a libSDL1.2 issue? Do I have to write explicit user defined mutex for every SDL_Blit/or other operations with SDL, that I am doing? Or is it something else that I am missing? How would I solve this issue?
I am using:
libSDL1.2 available in Raspberry Pi
libPthreads
libSDL_ttf
EDIT: gdb trace of the thread 2 mutex:
(gdb) p *(pthread_mutex_t*) 0xb542ed38
$3 = {_data = {_lock = 39257160, __count = 1048617, __owner = 0,
__kind = 7078021, _nusers = 0, {_spins = 0, _list = {_next = 0x0}}},
__size = "H\004W\002)\000\020\000\000\000\000\000\205\000l\000\000\000\000\000\000\000\000", __align = 39257160}
Let me know if you need more information.
This approach will only work with software surfaces (w/o hardware acceleration). With hardware surfaces it, at best, would lock and wait - which have no performance benefits anyway; and in bad case (judging by quick look at SDL source) it could end badly, by the simple logic how SDL_LockSurface works.
I'm not even sure your external mutexes would help, since hardware accelerated graphics access should almost always be locked to one thread (OpenGL is good example - each thread have it's own separate drawing context).
In short, i see no reason why you should use two threads for drawing, but in any case it isn't guaranteed to be thread-safe.
I have created a TCP client that connects to a listening server.
We implemeted TCP keep alive also.
Some times the client crashes and core dumped.
Below are the core dump traces.
Problem is in linux kernel version Update 4, kernel 2.6.9-42.0.10.
we had two core dumps.
(gdb) where
#0 0x005e77a2 in _dl_sysinfo_int80 () from /ddisk/d303/dumps/mhx239131/ld-
linux.so.2
#1 0x006c8bd1 in connect () from /ddisk/d303/dumps/mhx239131/libc.so.6
#2 0x08057863 in connect_to_host ()
#3 0x08052f38 in open_ldap_connection ()
#4 0x0805690a in new_connection ()
#5 0x08052cc9 in ldap_open ()
#6 0x080522cf in checkHosts ()
#7 0x08049b36 in pollLDEs ()
#8 0x0804d1cd in doOnChange ()
#9 0x0804a642 in main ()
(gdb) where
#0 0x005e77a2 in _dl_sysinfo_int80 () from /ddisk/d303/dumps/mhx239131/ld-
linux.so.2
#1 0x0068ab60 in __nanosleep_nocancel (
from /ddisk/d303/dumps/mhx239131/libc.so.6
#2 0x080520a2 in Sleep ()
#3 0x08049ac1 in pollLDEs ()
#4 0x0804d1cd in doOnChange ()
#5 0x0804a642 in main ()
We have tried to reproduce the problem in our environment, but we could not.
What would cause the core file?
Please help me to avoid such situation.
Thanks,
Naga
_dl_sysinfo_int80 is just a function which does a system call into the kernel. So the core dump is happening on a system call (probably the one used by connect in the first example and nanosleep in the second example), probably because you are passing invalid pointers.
The invalid pointers could be because the code which calls these functions being broken or because somewhere else in the program is broken and corrupting the program's memory.
Take a look at two frames above (frame #2) in the core dump for both examples and check the parameters being passed. Unfortunately, it seems you did not compile with debug information, making it harder to see them.
Additionally, I would suggest trying valgrind and seeing if it finds something.
Your program almost cetainly did not coredump in either of the above places.
Most likely, you either have multiple threads in your process (and some other thread caused the core dump), or something external caused your process to die (such as 'kill -SIGABRT <pid>').
If you do have multiple threads, GDB 'info threads' and 'thread apply all where' are likely to provide further clues.