So I have just discovered that libuv is a fairly small library as far as C libraries go (compare to FFmpeg). I have spent the past 6 hours reading through the source code to get a feel for the event loop at a deeper level. But still not seeing where the "nonblockingness" is implemented. Where some event interrupt signal or whatnot is being invoked in the codebase.
I have been using Node.js for over 8 years so I am familar with how to use an async non-blocking event loop, but I never actually looked into the implementation.
My question is twofold:
Where exactly is the "looping" occuring within libuv?
What are the key steps in each iteration of the loop that make it non-blocking and async.
So we start with a hello world example. All that is required is this:
#include <stdio.h>
#include <stdlib.h>
#include <uv.h>
int main() {
uv_loop_t *loop = malloc(sizeof(uv_loop_t));
uv_loop_init(loop); // initialize datastructures.
uv_run(loop, UV_RUN_DEFAULT); // infinite loop as long as queue is full?
uv_loop_close(loop);
free(loop);
return 0;
}
The key function which I have been exploring is uv_run. The uv_loop_init function essentially initializes data structures, so not too much fancness there I don't think. But the real magic seems to happen with uv_run, somewhere. A high level set of code snippets from the libuv repo is in this gist, showing what the uv_run function calls.
Essentially it seems to boil down to this:
while (NOT_STOPPED) {
uv__update_time(loop)
uv__run_timers(loop)
uv__run_pending(loop)
uv__run_idle(loop)
uv__run_prepare(loop)
uv__io_poll(loop, timeout)
uv__run_check(loop)
uv__run_closing_handles(loop)
// ... cleanup
}
Those functions are in the gist.
uv__run_timers: runs timer callbacks? loops with for (;;) {.
uv__run_pending: runs regular callbacks? loops through queue with while (!QUEUE_EMPTY(&pq)) {.
uv__run_idle: no source code
uv__run_prepare: no source code
uv__io_poll: does io polling? (can't quite tell what this means tho). Has 2 loops: while (!QUEUE_EMPTY(&loop->watcher_queue)) {, and for (;;) {,
And then we're done. And the program exists, because there is no "work" to be done.
So I think I have answered the first part of my question after all this digging, and the looping is specifically in these 3 functions:
uv__run_timers
uv__run_pending
uv__io_poll
But not having implemented anything with kqueue or multithreading and having dealt relatively little with file descriptors, I am not quite following the code. This will probably help out others along the path to learning this too.
So the second part of the question is what are the key steps in these 3 functions that implement the nonblockingness? Assuming this is where all the looping exists.
Not being a C expert, does for (;;) { "block" the event loop? Or can that run indefinitely and somehow other parts of the code are jumped to from OS system events or something like that?
So uv__io_poll calls poll(...) in that endless loop. I don't think is non-blocking, is that correct? That seems to be all it mainly does.
Looking into kqueue.c there is also a uv__io_poll, so I assume the poll implementation is a fallback and kqueue on Mac is used, which is non-blocking?
So is that it? Is it just looping in uv__io_poll and each iteration you can add to the queue, and as long as there's stuff in the queue it will run? I still don't see how it's non-blocking and async.
Can one outline similar to this how it is async and non-blocking, and which parts of the code to take a look at? Basically, I would like to see where the "free processor idleness" exists in libuv. Where is the processor ever free in the call to our initial uv_run? If it is free, how does it get reinvoked, like an event handler? (Like a browser event handler from the mouse, an interrupt). I feel like I'm looking for an interrupt but not seeing one.
I ask this because I want to implement an MVP event loop in C, but just don't understand how nonblockingness actually is implemented. Where the rubber meets the road.
I think that trying to understand libuv is getting in your way of understanding how reactors (event loops) are implemented in C, and it is this that you need to understand, as opposed to the exact implementation details behind libuv.
(Note that when I say "in C", what I really means is "at or near the system call interface, where userland meets the kernel".)
All of the different backends (select, poll, epoll, etc) are, more-or-less, variations on the same theme. They block the current process or thread until there is work to be done, like servicing a timer, reading from a socket, writing to a socket, or handling a socket error.
When the current process is blocked, it literally is not getting any CPU cycles assigned to it by the OS scheduler.
Part of the issue behind understanding this stuff IMO is the poor terminology: async, sync in JS-land, which don't really describe what these things are. Really, in C, we're talking about non-blocking vs blocking I/O.
When we read from a blocking file descriptor, the process (or thread) is blocked -- prevented from running -- until the kernel has something for it to read; when we write to a blocking file descriptor, the process is blocked until the kernel accepts the entire buffer.
In non-blocking I/O, it's exactly the same, except the kernel won't stop the process from running when there is nothing to do: instead, when you read or write, it tells you how much you read or wrote (or if there was an error).
The select system call (and friends) prevent the C developer from having to try and read from a non-blocking file descriptor over and over again -- select() is, in effect, a blocking system call that unblocks when any of the descriptors or timers you are watching are ready. This lets the developer build a loop around select, servicing any events it reports, like an expired timeout or a file descriptor that can be read. This is the event loop.
So, at its very core, what happens at the C-end of a JS event loop is roughly this algorithm:
while(true) {
select(open fds, timeout);
did_the_timeout_expire(run_js_timers());
for (each error fd)
run_js_error_handler(fdJSObjects[fd]);
for (each read-ready fd)
emit_data_events(fdJSObjects[fd], read_as_much_as_I_can(fd));
for (each write-ready fd) {
if (!pendingData(fd))
break;
write_as_much_as_I_can(fd);
pendingData = whatever_was_leftover_that_couldnt_write;
}
}
FWIW - I have actually written an event loop for v8 based around select(): it really is this simple.
It's important also to remember that JS always runs to completion. So, when you call a JS function (via the v8 api) from C, your C program doesn't do anything until the JS code returns.
NodeJS uses some optimizations like handling pending writes in a separate pthreads, but these all happen in "C space" and you shouldn't think/worry about them when trying to understand this pattern, because they're not relevant.
You might also be fooled into the thinking that JS isn't run to completion when dealing with things like async functions -- but it absolutely is, 100% of the time -- if you're not up to speed on this, do some reading with respect to the event loop and the micro task queue. Async functions are basically a syntax trick, and their "completion" involves returning a Promise.
I just took a dive into libuv's source code, and found at first that it seems like it does a lot of setup, and not much actual event handling.
Nonetheless, a look into src/unix/kqueue.c reveals some of the inner mechanics of event handling:
int uv__io_check_fd(uv_loop_t* loop, int fd) {
struct kevent ev;
int rc;
rc = 0;
EV_SET(&ev, fd, EVFILT_READ, EV_ADD, 0, 0, 0);
if (kevent(loop->backend_fd, &ev, 1, NULL, 0, NULL))
rc = UV__ERR(errno);
EV_SET(&ev, fd, EVFILT_READ, EV_DELETE, 0, 0, 0);
if (rc == 0)
if (kevent(loop->backend_fd, &ev, 1, NULL, 0, NULL))
abort();
return rc;
}
The file descriptor polling is done here, "setting" the event with EV_SET (similar to how you use FD_SET before checking with select()), and the handling is done via the kevent handler.
This is specific to the kqueue style events (mainly used on BSD-likes a la MacOS), and there are many other implementations for different Unices, but they all use the same function name to do nonblocking IO checks. See here for another implementation using epoll.
To answer your questions:
1) Where exactly is the "looping" occuring within libuv?
The QUEUE data structure is used for storing and processing events. This queue is filled by the platform- and IO- specific event types you register to listen for. Internally, it uses a clever linked-list using only an array of two void * pointers (see here):
typedef void *QUEUE[2];
I'm not going to get into the details of this list, all you need to know is it implements a queue-like structure for adding and popping elements.
Once you have file descriptors in the queue that are generating data, the asynchronous I/O code mentioned earlier will pick it up. The backend_fd within the uv_loop_t structure is the generator of data for each type of I/O.
2) What are the key steps in each iteration of the loop that make it non-blocking and async?
libuv is essentially a wrapper (with a nice API) around the real workhorses here, namely kqueue, epoll, select, etc. To answer this question completely, you'd need a fair bit of background in kernel-level file descriptor implementation, and I'm not sure if that's what you want based on the question.
The short answer is that the underlying operating systems all have built-in facilities for non-blocking (and therefore async) I/O. How each system works is a little outside the scope of this answer, I think, but I'll leave some reading for the curious:
https://www.quora.com/Network-Programming-How-is-select-implemented?share=1
The first thing to keep in mind is that work must be added to libuv's queues using its API; one cannot just load up libuv, start its main loop, and then code up some I/O and get async I/O.
The queues maintained by libuv are managed by looping. The infinite loop in uv__run_timers isn't actually infinite; notice that the first check verifies that a soonest-expiring timer exists (presumably, if the list is empty, this is NULL), and if not, breaks the loop and the function returns. The next check breaks the loop if the current (soonest-expiring) timer hasn't expired. If neither of those conditions breaks the loop, the code continues: it restarts the timer, calls its timeout handler, and then loops again to check more timers. Most times when this code runs, it's going to break the loop and exit, allowing the other loops to run.
What makes all this non-blocking is the caller/user following the guidelines and API of libuv: adding your work to queues, and allowing libuv to perform its work on those queues. Processing-intensive work may block these loops and other work from running, so it's important to break your work into chunks.
btw, uv__run_idle, uv__run_check, uv__run_prepare 's source code is defined on src/unix/loop-watcher.c
In external interrupt function, I want to reset by calling main function. But afterwards, if I have a new interrupt trigger, MCU thinks that It's handling in interrupt function and It doesn't call interrupt function again. What is my solution? (in my project, I'm not allowed to call soft Reset function)
Calling main() in any event is a bad idea, calling it from an interrupt handler is a really bad idea as you have discovered.
What you really need is to modify the stack and link-register so that when the interrupt context exits,, it "returns" to main(), rather than from whence it came. That is a non-trivial task, probably requiring some assembler code or compiler intrinsics.
You have to realise that the hardware will not have been restored to its reset state; you will probably need at least to disable all interrupts to prevent them occurring while the system is re-initialising.
Moreover the standard library will not be reinitialised if you jump to main(); rather than the reset vector. In particular, any currently allocated dynamic memory will instantly leak away and become unusable. In fact all of the C run-time environment initialisation will be skipped - leaving amongst for example static and global data in its last state rather than applying correct initialisation.
In short it is dangerous, error-prone, target specific, and fundamentally poor practice. Most of what you would have to do to make it work is already done in the start-up code that is executed before main() is called, so it would be far simpler to invoke that. The difference between that and forcing a true reset (via the watchdog or AICR) is that the on-chip peripheral state remains untouched (apart from any initialisation explicitly done in the start-up). In my experience, if you are using a more complex peripheral such as USB, safely restarting the system without a true reset is difficult to achieve safely (or at least it is difficult to determine how to do it safely) and hardly worth the effort.
Reset by calling main() is wrong. There is code in front of main inserted by the linker and C-runtime that you will skip by such soft-reset.
Instead, call NVIC_SystemReset() or enable the IWDG and while(1){} to reset.
The HAL should have example files for the watchdog timer.
SRAM is maintained. Any value not initialized by the linker script will still be there.
Calling Main() from any point of your code is a wrong idea if you are not resetting the stack and setting the initial values.
There is always a initialization function ( that actually calls Main()) which is inside an interrupt vector, usually this function can be triggered by calling the function NVIC_SystemReset(void) , be sure than you enable this interrupt so it can be software triggered.
As far as I know, when get inside and interrupt code, other interruptions are inhibit, I am thinking on two different options:
Enable the interruptions inside the interruption and call the function NVIC_SystemReset(void)
Modify the stack and push the direction of the function NVIC_SystemReset(void) so when you go out of the interruption it could be executed.
I'm looking to create a state of uninterruptible sleep for a program I'm writing. Any tips or ideas about how to create this state would be helpful.
So far I've looked into the wait_event() function defined in wait.h, but was having little luck implementing it. When trying to initialize my wait queue the compiler complained
warning: parameter names (without types) in function declaration
static DECLARE_WAIT_QUEUE_HEAD(wq);
Has anyone had any experience with the wait_event() function or creating an uninterruptible sleep?
The functions that you're looking at in include/linux/wait.h are internal to the Linux kernel. They are not available to userspace.
Generally speaking, uninterruptible sleep states are considered undesirable. Under normal circumstances, they cannot be triggered by user applications except by accident (e.g, by attempting to read from a storage device that is not responding correctly, or by causing the system to swap).
You can make sleep 'signal-aware`.
sleep can be interrupted by signal. In which case the pause would be stopped and sleep would return with amount of time still left. The application can choose to handle the signal notified and if needed resume sleep for the time left.
Actually, you should use synchronization objects provided by the operating system you're working on or simply check the return value of sleep function. If it returns to a value bigger than zero, it means your procedure was interrupted. According to this return value, call sleep function again by passing the delta (T-returnVal) as argument (probably in a loop, in case of possible interrupts that might occur again in that time interval)
On the other hand, if you really want a real-uninterruptible custom sleep function, I may suggest something like the following:
void uninterruptible_sleep(long time, long factor)
{
long i, j;
__asm__("cli"); // close interrupts
for(i=0; i<time; ++i)
for(j=0; j<factor; ++j)
; // custom timer loop
__asm__("sti"); // open interrupts
}
cli and sti are x86 assembly instructions which allow us to set IF (interrupt flag) of the cpu. In this way, it is possible to clear (cli) or set (sti) all the interrupts. However, if you're working on a multi-processor system, there needs to be taken another synchronization precautions too, due to the fact that these instructions will only be valid for single microprocessor. Moreover, this type of function as I suggested above, will be very system (cpu) dependant. Because, the inner loop requires a clock-cycle count to measure an exact time interval (execution number of instructions per second) depending on the cpu frequency. Thus, if you really want to get rid of every possible interrupt, you may use a function as I suggested above. But be careful, if your program gets a deadlock situation while it's in cli state, you will need to restart your system.
(The inline assembly syntax I have written is for gcc compiler)
While referring to memory module of Linux kernel some functions are not clear to me. One of the functions is shown below:
static inline int __kprobes notify_page_fault(struct pt_regs *regs)
{
int ret = 0;
/* kprobe_running() needs smp_processor_id() */
if (kprobes_built_in() && !user_mode_vm(regs)) {
preempt_disable();
if (kprobe_running() && kprobe_fault_handler(regs, 14))
ret = 1;
preempt_enable();
}
return ret;
}
I am confused with the "__kprobes" between return type and function name. When I looked at the initialization of "__kprobes" in compiler.h, I found below:
/*Ignore/forbid kprobes attach on very low level functions marked by
this attribute:*/
#ifdef CONFIG_KPROBES
# define __kprobes __attribute__((__section__(".kprobes.text")))
#else
# define __kprobes
#endif
Well, I know that at compile time __kprobe is going to be replaced with its defined part.
Questions:
1.) What is the significance of __attribute__((__section__(".kprobes.text")))?
and
2.) What does it do at compile time and at run time when it is used before "function_name"?
I read about kprobe and found that it has to do something about breakpoints and back trace. What I understand about kprobe is it will help debugger in creating back traces and breakpoints. Could someone please explain me in simple words how does it really works and please correct me if I am wrong.
TL;DR
__attribute__((__section__(".kprobes.text"))) will place that function in separate section which is not findable by kprobes thus preventing infinite breakpoints.
You must use it before "function_name" to place whole "function_name" symbol in separate section.
Real answer
kprobes (kernel probes) is Linux kernel mechanism for dynamic tracing. It allows you to insert breakpoint at almost any kernel function, invoke your handler and then continue executing. It works by runtime patching kernel image with so-called kernel probe/kprobe - see struct kprobe. This probe will allow you to pass control to your handler, and that handler is usually do some tracing.
So, what's going under the hood:
You create your struct kprobe by defining address at which to break and handler to pass reference.
You register your probe with register_kprobe
Kernel kprobe subsystem finds address from your probe
Then kprobe:
inserts breakpoint CPU instruction (int 3 for x86) at given address
adds some wrapper code to save context(registers, etc.)
adds even more code to help you get access to function arguments or return values.
Now when kernel execution hits that probed address:
it will fall into CPU trap
it will save context
it will pass control to your handler via notifier_call_chain
...
after all it will restore context
That's how it works. As you can see it's a really neat and dirty hack, but some kernel function is so terribly low-level that it's just pointless to probe them. notify_page_fault is one of those functions - as a part of notifier_call_chain it's used in passing control to your handler.
So if you probe at notify_page_fault you'll get infinite loop of breakpoints, which is not what you want. What you really want is to protect that kind of functions and kprobes do this by placing it in separate section .kprobes.text. This will prevent to probe at that functions because kprobe will not lookup for address in that section. And that's a job for __attribute__((__section__(".kprobes.text"))).
I have an interrupt function called, interrupt_Foo() {...} which turns on a flag when 1 second has elapsed, and a user-defined function foo_calling() {...} which calls another function foo_called() {...}. I want to stop the process in foo_called() when 1 second has elapsed.
The code snippet below may elaborate further my need:
void interrupt interrupt_foo() {
...
if(1 second has elapsed) {
flag1s = 1;
} else {
flag1s = 0;
}
}
void foo_calling() {
// need something here to stop the process of foo_called()
...
(*fptr_called)(); // ptr to function which points to foo_called
...
}
void foo_called() {
// or something here to stop the process of this function
...
// long code
...
}
This is real time operating system so polling the 1 second flag inside foo_called() at some portion in the code is undesirable. Please help.
If you are willing to write non-portable code, and test the heck out of it before deploying it, and if the processor supports it, there may be a solution.
When the interrupt handler is called, the return address must be stored somewhere. If that is a location your code can query - like a fixed offset down the stack - then you can compare that address to the range occupied by your function to determine if 'foo_called is executing. You can get the address of the function by storing a dummy address, compiling, parsing the map file, then updating the address and recompiling.
Then, if your processor supports it, you can replace the return address with the address of the last instruction(s) of foo_called. (make sure you include the stack cleanup and register restoration code.). Then exit the interrupt as normal, and the interrupt handling logic will return code to the end of your interrupted function.
If the return address is not stored in the stack, but in an unwritable register, you still may be able to force quit your function - if the executable code is in writrable memory. Just store the instruction at the interruupt's return address, then overwrite it with a jump instruction which jumps to the function end. In the caller code, add a detector which restored the overwritten instruction.
I would expect that your RTOS has some kind of timer signal/interrupt that you can use to notify you when one second has passed. For instance if it is a realtime UNIX/Linux then you would set a signal handler for SIGALRM for one second. On a RT variant of Linux this signal will have more granularity and better guarantees than on a non-RT variant. But it is still a good idea to set the signal for slightly less than a second and busy-wait (loop) until you reach one second.