Why does the Linux kernel enable interrupts in preempt_schedule_irq? - c

I'm reading Linux source code to learn how scheduling works. I learn that in a preemptible kernel (CONFIG_PREEMPT is set), there is a chance for preemption after returning to kernel-space from interrupt handler by calling preempt_schedule_irq.
However, I also find the following code snippet in preempt_schedule_irq
do {
preempt_disable();
local_irq_enable(); //why enable interrupt here?
__schedule(true); //interrupt would be disabled inside it
local_irq_disable();
sched_preempt_enable_no_resched();
} while (need_resched());
There is a local_irq_enable() call inside it and this kind of confuses me. Why do we need to enable interrupt here since at the start of __schedule it would disabled again?.
My humble guess is that this gives a chance to processes with higher priority to be scheduled first. However, it doesn't make sense because the preemption is already disabled in preempt_schedule_irq, even if there is an interrupt, there would not be a preemption reschedule.
So what on earth is the point in preempting the scheduling procedure here? I think I must have missed something but I don't figure out.

Short answer: Because interrupts should be enabled as much as possible, only disabled to protect minimal critical sections. Arbitrarily extending the disabling beyond your critical section into non-critical sections of functions you're calling because you're assuming that at some point that function will disable them, is bad design.
Why doesn't __schedule() disable interrupts as it's very first instruction? Because it doesn't need to, the code at the start of __schedule() isn't a critical section so explicitly disabling interrupts before it would be a waste. The writer of __schedule() went out of their way to maximize the time when interrupts can be handled, why ignore that opportunity by not enabling interrupts?
Also, you have no guarantees about what __schedule() might do in the future. Since the start of __schedule() isn't a critical section, you have no guarantees that more stuff won't be added before the interrupt disabling. Remember, the person who's going to be making changes to __schedule() shouldn't have to consider that one of the callers decided to leave their interrupts disabled relying on the fact that __schedule() turns them off pretty soon anyway. __schedule() has no regard at all about interrupt status when it's called.
You should disable/enable interrupts around the critical sections of your code, not rely on the inner mechanics of some other function you're calling and hoping it doesn't change.
If you look through the history of the scheduler, you'll see that the code preceding the interrupt disabling has changed over time. Digging through the commits you see that the "sti" to enable interrupts was present since the very first commit to implement preemption in the kernel going back all the way to 2.5: https://github.com/schwabe/tglx-history/blob/ec332cd30cf1ccde914a87330ff66744414c8d24/arch/i386/kernel/entry.S#L235

Related

pvPortMalloc and vPortFree Usage (freeRTOS Heap4.c) in Baremetal STM32F4x

I have had a good experience in programming bare metal stm32f4x; however, i tried to shift my code to freeRTOS and for that i first wanted to see if i can use Heap4.c for memory allocation instead of standard C malloc and free calls to better manage the memory etc.
However, what I observed is that using these calls disables my interrupts and never turns them back on. Hence, anything which makes use of interrupts is not working; everything else, which has nothing to do with interrupts is working ok. Not even Systick timer interrupt handler is being triggered.
So, the question is that, how can we make use of pvPortMalloc and vPortFree with bare metal code considering that all other peripherals do make use of their interrupts and SysTick is basically used for simple time delays etc. When using these calls, I could not see any prints happening inside systick as there was no systick handler being called.
Here I would like to point out that I am not calling pvPortMalloc or vPortFree in any interrupt context at all. So, that is totally safe and nothing to worry about that.
I have read through few discussions and if i understand correctly, then any call to FreeRTOS scheduler to suspend tasks etc does not impact as there will be no tasks at all. So, I expect this heap4.c port to work just fine with bare metal as well as long as we stay away from using them within ISR context; but apparently it just disables interrupts and seem to never turn them back on.
I hope to have the opinion of experts here on using pvPortMalloc and vPortFree in bare metal instead of using freeRTOS.
Best regards,
Junaid
I think if you replace the xTaskSuspendAll() and xTaskResumeAll() to simply disable / enable interrupts it should work fine. In fact if your interrupts are not using the allocated memory you might not even need to do this, you could simply comment them out. Suspend and Resume are quite complex functions that can attempt to yield control to other tasks if required.
I suspect the reason interrupts are not getting re-enabled is that either taskEXIT_CRITICAL() is not defined correctly (portENABLE_INTERRUPTS) or the uxCriticalNesting is greater than one when truing to re-enable interrupts (enter critical called more times than exit critical).
However you will probably find the standard malloc and free are better if you are not using FreeRTOS.

The right way to clear an interrupt flag on STM32

I'm developping a bare-metal project on a STM32L4 and I'm starting from an existing code base.
The ISRs have been implemented the following way:
read interrupt status in the peripheral to know what event(s) provoked the interrupt
do something
clear the flags that have read at the beginning.
Is it the right way to clear the flag ? Shouldn't the flags be cleared at the very beginning of the ISR ? My understanding is that, if the same peripheral event is happening a second time during step 2, it will not provoke a second IRQ so it would be lost. On the other hand if you clear the flag as soon as you can, this second event would pulse the interrupt whose state in the CPU would change to "pending and active": a second IRQ would happen.
PS: From STM32 Processor Programming Manual I read: "STM32 interrupts are both level-sensitive and pulse-sensitive".
Definitely at the beginning (unless you have special reasons in the program logic) as some time is needed the for actual write to the flag clear register to propagate through the buses.
If you decide for some reason to put it at the end of the interrupt you should leave some instructions, place the barrier instruction or read back the register before the interrupt routine return to make sure that the clear operation has propagated across the buses. Otherwise you may have a "phantom" duplicate routine calls.

Why not put task context in interrupt

Here is the story.
Its a safety critical project and needs to run a time critical functional routine in 20KHz. Now the design is to put functional routine in a 20KHz FIQ interrupt, meanwhile safety interrupt also in FIQ. Thats the only two FIQ in system. (Surely there are couples of IRQ enabled in the MCU)
I know that its not good to put task context in interrupt ISR, the proper way of doing this to set mark and run in OS task. But seems current design harm nobody.
The routine takes about 10us (main clock 300MHz), so basically it will not blocks IRQ/FIQ for unacceptable time. It even save time for extra context switch compare with using OS task to run the functional routine. To me, currently it feels like the design is against every principle written on text book in university but can not find a reason to say no to it.
How could I convince myself to move functional routine from ISR to OS? Should I?
Let's recollect your situation:
you are coding a safety critical system
the software architecture isn't specified otherwise you wouldn't ask the question at hand
the system requirements weren't processed correctly otherwise 2) wouldn't be in question
someone told you to "use minimum interrupt if possible in safety critical system"
you want to use the highest priority & non-interruptible code for "just some math work"
Sorry for being a bit harsh but I wouldn't want to use/be in your safety critical system.
For your actual problem:
you have to make sure two things
the code in the FIQ must be deterministic and WCET tested
the registers of the timer must be protected and supervised. Why? An unwanted/erroneous manipulation of the timers registers by a lower safety level code can congest the CPU so much that effectively nothing else but the interrupt is processed.
All this under the assumption that your safe state depends entirely on an external hardware watchdog.
PS: Which are the hazards for users of your system? Annoyance? Injury? Lethal? Are you in a SIL or ASIL context?
The reason to move complex code away from ISR is precisely to avoid lengthy processing in the ISR and thus timing jitter and delayed interrupt servicing resulting from it.
You are stating the your processing is not lengthy so do it in the ISR! Otherwise you are just adding bloat.
20Khz = 50us between interrupts, with 10us of processing time it gives you roughly 20% of CPU time just for this "task", and a jitter of 10us in any other routine that runs in your CPU, it will also sum 10us of processing time for each 40us that any other task will consum, if it is ok for your project, and you keep your total CPU processing time below 70% (which is the common maximum acceptable for critical systems), IMHO it should work without any issue.

what's wrong in using interrupt handlers as event listeners

My system is simple enough that it runs without an OS, I simply use interrupt handlers like I would use event listener in a desktop program. In everything I read online, people try to spend as little time as they can in interrupt handlers, and give the control back to the tasks. But I don't have an OS or real task system, and I can't really find design information on OS-less targets.
I have basically one interrupt handler that reads a chunk of data from the USB and write the data to memory, and one interrupt handler that reads the data, sends the data on GPIO and schedule itself on an hardware timer again.
What's wrong with using the interrupts the way I do, and using the NVIC (I use a cortex-M3) to manage the work hierarchy ?
First of all, in the context of this question, let's refer to the OS as a scheduler.
Now, unlike threads, interrupt service routines are "above" the scheduling scheme.
In other words, the scheduler has no "control" over them.
An ISR enters execution as a result of a HW interrupt, which sets the PC to a different address in the code-section (more precisely, to the interrupt-vector, where you "do a few things" before calling the ISR).
Hence, essentially, the priority of any ISR is higher than the priority of the thread with the highest priority.
So one obvious reason to spend as little time as possible in an ISR, is the "side effect" that ISRs have on the scheduling scheme that you design for your system.
Since your system is purely interrupt-driven (i.e., no scheduler and no threads), this is not an issue.
However, if nested ISRs are not allowed, then interrupts must be disabled from the moment an interrupt occurs and until the corresponding ISR has completed. In that case, if any interrupt occurs while an ISR is in execution, then your program will effectively ignore it.
So the longer you spend inside an ISR, the higher the chances are that you'll "miss out" on an interrupt.
In many desktop programs, events are send to queue and there is some "event loop" that handle this queue. This event loop handles event by event so it is not possible to interrupt one event by other one. It also is good practise in event driven programming to have all event handlers as short as possible because they are not interruptable.
In bare metal programming, interrupts are similar to events but they are not send to queue.
execution of interrupt handlers is not sequential, they can be interrupted by interrupt with higher priority (numerically lower number in Cortex-M3)
there is no queue of same interrupts - e.g. you can't detect multiple GPIO interrupts while you are in that interrupt - this is the reason you should have all routines as short as possible.
It is possible to implement queues by yourself, feed these queues by interrupts and consume these queues in your super loop (consume while disabling all interrupts). By this approach, you can get sequential processing of interrupts. If you keep your handlers short, this is mostly not needed and you can do the work in handlers directly.
It is also good practise in OS based systems that they are using queues, semaphores and "interrupt handler tasks" to handle interrupts.
With bare metal it is perfectly fine to design for application bound or interrupt/event bound so long as you do your analysis. So if you know what events/interrupts are coming at what rate and you can insure that you will handle all of them in the desired/designed amount of time, you can certainly take your time in the event/interrupt handler rather than be quick and send a flag to the foreground task.
The common approach of course is to get in and out fast, saving just enough info to handle the thing in the foreground task. The foreground task has to spin its wheels of course looking for event flags, prioritizing, etc.
You could of course make it more complicated and when the interrupt/event comes, save state, and return to the forground handler in the forground mode rather than interrupt mode.
Now that is all general but specific to the cortex-m3 I dont think there are really modes like big brother ARMs. So long as you take a real-time approach and make sure your handlers are deterministic, and you do your system engineering and insure that no situation happens where the events/interrupts stack up such that the response is not deterministic, not too late or too long or loses stuff it is okay
What you have to ask yourself is whether all events can be services in time in all circumstances:
For example;
If your interrupt system were run-to-completion, will the servicing of one interrupt cause unacceptable delay in the servicing of another?
On the other hand, if the interrupt system is priority-based and preemptive, will the servicing of a high priority interrupt unacceptably delay a lower one?
In the latter case, you could use Rate Monotonic Analysis to assign priorities to assure the greatest responsiveness (the shortest execution-time handlers get the highest priority). In the first case your system may lack a degree of determinism, and performance will be variable under both event load, and code changes.
One approach is to divide the handler into real-time critical and non-critical sections, the time-critical code can be done in the handler, then a flag set to prompt the non-critical action to be performed in the "background" non-interrupt context in a "big-loop" system that simply polls event flags or shared data for work to complete. Often all that might be necessary in the interrupt handler is to copy some data to timestamp some event - making data available for background processing without holding up processing of new events.
For more sophisticated scheduling, there are a number of simple, low-cost or free RTOS schedulers that provide multi-tasking, synchronisation, IPC and timing services with very small footprints and can run on very low-end hardware. If you have a hardware timer and 10K of code space (sometimes less), you can deploy an RTOS.
I am taking your described problem first
As I interpret it your goal is to create a device which by receiving commands from the USB, outputs some GPIO, such as LEDs, relays etc. For this simple task, your approach seems to be fine (if the USB layer can work with it adequately).
A prioritizing problem exists though, in this case it may be that if you overload the USB side (with data from the other end of the cable), and the interrupt handling it is higher priority than that triggered by the timer, handling the GPIO, the GPIO side may miss ticks (like others explained, interrupts can't queue).
In your case this is about what could be considered.
Some general guidance
For the "spend as little time in the interrupt handler as possible" the rationale is just what others told: an OS may realize a queue, etc., however hardware interrupts offer no such concepts. If the event causing the interrupt happens, the CPU enters your handler. Then until you handle it's source (such as reading a receive holding register in the case of a UART), you lose any further occurrences of that event. After this point, until exiting the handler, you may receive whether the event happened, but not how many times (if the event happened again while the CPU was still processing the handler, the associated interrupt line goes active again, so after you return from the handler, the CPU immediately re-enters it provided nothing higher priority is waiting).
Above I described the general concept observable on 8 bit processors and the AVR 32bit (I have experience with these).
When designing such low-level systems (no OS, one "background" task, and some interrupts) it is fundamental to understand what goes on on each priority level (if you utilize such). In general, you would make the most real-time critical tasks the highest priority, taking the most care of serving those fast, while being more relaxed with the lower priority levels.
From an other aspect usually at design phase it can be planned how the system should react to missed interrupts, since where there are interrupts, missing one will eventually happen anyway. Critical data going across communication lines should have adequate checksums, an especially critical timer should be sourced from a count register, not from event counting, and the likes.
An other nasty part of interrupts is their asynchronous nature. If you fail to design the related locks properly, they will eventually corrupt something giving nightmares to that poor soul who will have to debug it. The "spend as little time in the interrupt handler as possible" statement also encourages you to keep the interrupt code reasonably short which means less code to consider for this problem as well. If you also worked with multitasking assisted by an RTOS you should know this part (there are some differences though: a higher priority interrupt handler's code does not need protection against a lower priority handler's).
If you can properly design your architecture regarding the necessary asynchronous tasks, getting around without an OS (from the no multitasking aspect) may even prove to be a nicer solution. It needs way more thinking to design it properly, however later there are much less locking related problems. I got through some mid-sized safety critical projects designed over a single background "task" with very few and little interrupts, and the experience and maintenance demands regarding those (especially the tracing of bugs) were quite satisfactory compared to some others in the company built over multitasking concepts.

How do you test your interrupt handling module?

I've got an interrupt handling module which controls the interrupt controller hardware on an embedded processor. Now I want to add more tests to it. Currently, the tests only tests if nesting of interrupts works by making two software interrupts from within an ISR, one with low priority and one with high priority. How can I test this module further?
I suggest that you try to create other stimuli as well.
Often, also hardware interrupts can be triggered by software (automatic testing) or the debugger by setting a flag. Or as an interrupt via I/O. Or a timer interrupt. Or you can just set the interrupt bit in an interrupt controller via the debugger while you are single stepping.
You can add some runtime checks on things which are not supposed to happen. Sometimes I elect to set output pins to monitor externally (nice if you have an oscilloscope or logic analyser...)
low_prio_isr(void)
{
LOW_PRIO_ISR=1;
if (1 == HIGH_PRIO_ISR)
{ this may never happen. dummy statement to allow breakpoint in debugger }
}
high_prio_isr(void)
{
HIGH_PRIO_ISR=1
}
The disadvantage of the software interrupt is that the moment is fixed; always the same instruction. I believe you would like to see evidence that it always works; deadlock free.
For interrupt service routines I find code reviews very valuable. In the end you can only test the situations you've imagined and at some point the effort of testing will be very high. ISRs are notoriously difficult to debug.
I think it is useful to provide tests for the following:
- isr is not interrupted for lower priority interrupt
- isr is not interrupted for same priority interrupt
- isr is interrupted for higher priority interrupt
- maximum nesting count within stack limitations.
Some of your tests may stay in the code as instrumentation (so you can monitor for instance maximum nesting level.
Oh, and one more thing: I've generally managed to keep ISRs so short that I can refrain from nesting.... if you can this will gain you additional simplicity and more performance.
[EDIT]
Of course, ISRs need to be tested on hardware in system too. Apart from the bit-by-bit, step-by-step approach you may want to prove:
- stability of system at maximum interrupt load (preferably several times the predicted maximum load; if your 115kbps serial driver can also handle 2MBps you'll be ok!)
- correct moment of enabling / disabling isr, especially if system also enters a sleep mode
- # of interrupts. Can be surprising if you add mechanical switches, mechanical rotary (hundreds of break/contact moments before reaching steady situation)
I recommend real-hardware testing. Interrupt handling is inherently random and unpredictable.
Use a signal generator and feed a square wave into the appropriate interrupt pin. Use multiple generators (or one with multiple outputs) to test multiple IRQ lines and verify priority handling.
Experiment with dialing the frequency up & down on the signal generators (vary the rates between them), and see what happens. Have lots of diagnostic code to verify the state of the interrupt controller in the various states.
Alternative: If your platform has timers that can trigger interrupts, you can use them instead of external hardware.
I'm not an embedded developer, so I don't know if this is possible, but how about decoupling the code that handles the interrupts from the callback-registration mechanism? This would allow you to write simulator code fireing interrupt-events as you like it...
For stuff like this I highly recommend something like the SPIN model checker. You wind up testing the algorithm, not the code, but the testing is exhaustive. Back in the day, I found a bug in gdb using this technique.

Resources