I'm implementing a custom serial bus driver for a certain ARM-based Linux board (a custom UART driver, actually). This driver shall enable communication with a certain MCU on the other end of the bus via a custom protocol. The driver will not (and actually must not) expose any of its functions to the userspace, nor it is possible to implement it in userspace at all (hence, the need for the custom driver instead of using the stock TTY subsystem).
The driver will implement the communication protocol and UART reads/writes, and it has to export a set of higher-level functions to its users to allow them to communicate with the MCU (e.g. read_register(), drive_gpios(), all this stuff). There will be only one user of this module.
The calling module will have to wait for the completion of the operations (the aforementioned read_register() and others). I'm currently considering using semaphores: the user module will call my driver's function, which will initiate the transfers and wait on a semaphore; the IRQ handler of my driver will send requests to the MCU and read the answers, and, when done, post to the semaphore, thus waking up the calling module. But I'm not really familiar with kernel programming, and I'm baffled by the multitude of possible alternative implementations (tasklets? wait queues?).
The question is: is my semaphore-based approach OK, or too naïve? What are the possible alternatives? Are there any pitfalls I may be missing?
Traditionally IRQ handling in Linux is done in two parts:
So called "upper-half" is actual working in IRQ context (IRQ handler itself). This part must exit as fast as possible. So it basically checks interrupt source and then starts bottom-half.
"Bottom-half". It may be implemented as work queue. It is where actual job is done. It runs in normal context, so it can use blocking functions, etc.
If you only want to wait for IRQ in your worker thread, better to use special object called completion. It is exactly created for this task.
Related
I am writing a device driver that receives interrupts from hardware. (32 IRQs using MSI)
From the driver, I'd like to signal/interrupt the application that opened the device file that an event occured.
I might be able to use signal but I think it's not really reliable and too slow. Moreover, only 2 SIGUSR are available.
I'd like to avoid adding overhead.
I'd like to avoid them because:
signal: not enough reliable and high latency
netlink: high latency, asynchronous and may loose packets
polling/read/ioctl: need to use a pthread and an infinity loop
Currently, I exchange data using ioctl/read/write syscalls.read/write syscalls.
What is the best practice to interrupt/signal an event to an user-space application from a kernel driver?
The method should support many interrupts/signals without loosing any of them, it has to be reliable and fast.
Basically, I'd like to use my user-space app as bottom half of the interrupts I receive in the driver.
The device file is opened by a unique app.
Sorry If this is a noob question, but I'm developing a software "add on" for a game. I'm doing this through a driver simply because the anti-cheat doesn't support ring 0 detection. I haven't seen much info on how IOCTL can be used and i was wondering if you can send custom inputs like process ids and other information that may change or is it all set in stone like a switch function or something. Once again sorry for noob question.
You can communicate with a kernel-mode device driver via IOCTL using the DeviceIoControl Win32 API routine. This routine internally calls NtDeviceIoControlFile (NTDLL) which performs a system call to get NtDeviceIoControlFile (NTOSKRNL) executed.
The DeviceIoControl routine is documented at MSDN: https://msdn.microsoft.com/en-us/library/windows/desktop/aa363216(v=vs.85).aspx
The kernel-mode device driver will have a prerequisite to fulfill: https://learn.microsoft.com/en-us/windows-hardware/drivers/kernel/named-device-objects
I haven't seen much info on how IOCTL can be used and i was wondering if you can send custom inputs like process ids and other information
The answer is yes, you can send custom buffers via IOCTL. You can also receive an output buffer back from your kernel-mode device driver to the user-mode application which initiated the IOCTL operation - this is optional of course.
If you need to send multiple pieces of information at the same time, consider using a structure.
I also recommend you read the following:
https://learn.microsoft.com/en-us/windows-hardware/drivers/kernel/methods-for-accessing-data-buffers
First of all, I am a device driver guy. This is my first time to handle an user mode program.
I used to have an interrupt service routine to response a hardware interrupt.
In other word, the hardware uses interrupt service routine to notify the driver to service.
I use ioctl to be a channel to communicate between the application and device driver now and poll it to wait the response.
Are there other ways that a device driver can notify an application when it finishes some task?
Any comments are welcome.
Thanks,
There are several mechanisms for this. First approach: user-space application makes poll() or select() system call, waiting for some event from kernel. Second approach is to use Netlink sockets. There are also others like mmap() or signals. Google by kernel user-space IPC and you will see the whole list.
As for your case (drivers development), I'd say go with next approach. Create sysfs file in your driver, and do sysfs_notify() (and maybe wait_for_completion_interruptible_timeout() or something like that). In user-space do select() system call for your driver sysfs file. See how line discipline installed from user-space for example.
Typically, the kernel never notifies an application unless the application requests the notification and is waiting for the notification. On unix systems, this will typically be done using the select or similar routines. One supplies select with a set of file descriptors and select will then wait until there is activity on one of the file descriptors at which time it returns.
Given that on unix all devices are files, you should be able to make use of this mechanism to wake an application when an interrupt comes in on some hardware device.
There are plenty of kernel-userspace communication interfaces in addition to ioctl (signals, sockets, etc). Please, refer to Kernel Space - User Space Interfaces tutorial for detailed explanation.
Problem definition:
We are designing an application for an industrial embedded system running Linux.
The system is driven by events from the outside world. The inputs to the system could be any of the following:
Few inputs to the system in the form of Digital IO lines(connected
to the GPIOs of the processor like e-stop).
The system runs a web-server which allows for the system to be
controlled via the web browser.
The system runs a TCP server. Any PC or HMI device could send commands over TCP/IP.
The system needs to drive or control RS485 slave devices over UART using Modbus. The system also need to control few IO lines like Cooler ON/OFF etc.We believe that a state machine is essential to define this application. The core application shall be a multi threaded application which shall have the following threads...
Main thread
Thread to control the RS485 slaves.
Thread to handle events from the Web interface.
Thread to handle digital I/O events.
Thread to handle commands over TCP/IP(Sockets)
For inter-thread communication, we are using Pthread condition signal & wait. As per our initial design approach(one state machine in main thread), any input event to the system(web or tcp/ip or digital I/O) shall be relayed to the main thread and it shall communicate to the appropriate thread for which the event is destined. A typical scenario would be to get the status of the RS485 slave through the web interface. In this case, the web interface thread shall relay the event to the main thread which shall change the state and then communicate the event to the thread that control's the RS485 slaves & respond back. The main thread shall send the response back to the web interface thread.
Questions:
Should each thread have its own state machine thereby reducing the
complexity of the main thread ? In such a case, should we still need
to have a state machine in main thread ?
Any thread processing input event can communicate directly to the
thread that handles the event bypassing the main thread ? For e.g
web interface thread could communicate directly with the thread
controlling the RS485 slaves ?
Is it fine to use pthread condition signals & wait for inter thread
communication or is there a better approach ?
How can we have one thread wait for event from outside & response
from other threads ? For e.g. the web interface thread usually waits
for events on a POSIX message queue for Inter process communication
from web server CGI bins. The CGI bin's send events to the web
interface thread through this message queue. When processing this
event, the web interface thread would wait for response from other
threads. In such a situation, it couldn't process any new event from
the web interface until it has completed processing the previous
event and gets back to the wait on the POSIX message queues.
sorry for the too big explanation...I hope I have put forward my explanation in the best possible way for others to understand and help me.
I could give more inputs if needed.
What I always try to do with such requirements is to use one state machine, run by one 'SM' thread, which could be the main thread. This thread waits on an 'EventQueue' input producer-cosumer queue with a timeout. The timeout is used to run an internal delta-queue that can provide timeout events into the state-machine when they are required.
All other threads communicate their events to the state engine by pushing messages onto the EventQueue, and the SM thread processes them serial manner.
If an action routine in the SM decides that it must do something, it must not synchronously wait for anything and so it must request the action by pushing a request message to an input queue of whatever thread/susbsystem can perform it.
My message class, (OK, *struct in your C case), typically contains a 'command' enum, 'result' enum, a data buffer pointer, (in case it needs to transport bulk data), an error-message pointer, (null if no error), and as much other state as is necessary to allow the asynchronous queueing up of any kind of request and returning the complete result, (whether success or fail).
This message-passing, one SM design is the only one I have found that is capable of doing such tasks in a flexible, expandable manner without entering into a nightmare world of deadlocks, uncontrolled communications and unrepeatable, undebuggable interactions.
The first question that should be asked about any design is 'OK, how can the system be debugged if there is some strange problem?'. In my design above, I can answer straightaway: 'we log all events dequeued in the SM thread - they all come in serially so we always know exactly what actions are taken based on them'. If any other design is suggested, ask the above question and, if a good answer is not immediately forthcoming, it will never be got working.
So:
If a thread, or threaded subsystem, can use a separate state-machine to do its own INTERNAL functionality, OK, fine. These SM's should be invisible from the rest of the system.
NO!
Use the pthread condition signals & wait to implement producer-consumer blocking queues.
One input queue per thread/subsystem. All inputs go to this queue in the form of messages. Commands/state in each message identify the message and what should be done with it.
BTW, I would 100% do this in C++ unless shotgun-at-head :)
I have implemented a legacy embedded library that was originally written for a clone (EC115/EC270) of Siemens ES122C terminal controller. This library and OS included more or less what you describe. The original hardware was based on 80186 cpu. The OS, RMOS for Siemens, FXMOS for us (don't google it was never published) had all the stuff needed for basic controller work.
It had preemptive multi-tasking, task-to-task communication, semaphores, timers and I/O events, but no memory protection.
I ported that stuff to RaspberryPi (i.e. Linux).
I used the pthreads to simulate our legacy "tasks" because we hadn't memory protection, so threads are semantically the closest.
The rest of the implementation then turned around the epoll API. This means that everything generates an event. An event is when something happens, a timer expires, another thread sends data, a TCP socket is connected, an IO pin changes state, etc.
This requires that all the event sources be transformed in file descriptors. Linux provides several syscalls that do exactly that:
for task to task communication I used classic Unix pipes.
for timer events I used timerfd API.
for TCP communication I used normal sockets.
for serial I/O I simply opened the right device /dev/???.
signals are not necessary in my case but Linux provides 'signalfd' if necessary.
I have then epoll_wait wrapped around to simulate the original semantic.
I works like a charm.
TL;DR
take a deep look at the epoll API it does what you probably need.
EDIT: Yes and the advices of Martin James are very good especially 4. Each thread should only ever be in a loop waiting on an event via epoll_wait.
Im writing a custom device driver in linux that has to be able to respond very rapidly on interrupts. Code to handle this already exists in a user-space implementation but that is too slow as it relies on software constantly checking the state of the interrupt line. After doing some research, I found that you can register these interrupt lines from a kernel module, and execute a function given by a function pointer. However the code we want to execute is in the user-space, is there a way to call a function in the user-space from a kernel space module?
You are out of luck with invoking user-space functions from the kernel since the kernel doesn't and isn't supposed to know about individual user-space application functions and logic, not to mention that each user-space application has its own memory layout, that no other process nor the kernel is allowed to invade in that way (shared objects are the exception here, but still you can't tap into that from the kernel space). What about the security model, you aren't supposed to run user-space code (which is automatically considered unsafe code in the kernel context) in the kernel context in the first place since that will break the security model of a kernel right there in that instant. Now considering all of the above mentioned, plus many other motives you might want to reconsider your approach and focus on Kernel <-> User-space IPC and Interfaces, the file system or the user-mode helper API(read bellow).
You can invoke user space apps from the kernel though, that using the usermode-helper API. The following IBM DeveloperWorks article should get you started on using the usermode-helper Linux kernel API:
Kernel APIs, Part 1: Invoking user-space applications from the kernel
I think the easiest way is to register a character device which becomes ready when the device has some data.
Any process which tries to read from this device, then gets put to sleep until the device is ready, then woken up, at which point it can do the appropriate thing.
If you just want to signal readyness, a reader could just read a single null byte.
The userspace program would then just need to execute a blocking read() call, and would be blocked appropriately, until you wake it up.
You will need to understand the kernel scheduler's wait queue mechanism to use this.
Sounds like your interrupt line is already available to userspace via gpiolib? (/sys/class/gpio/...)
Have you benchmarked if gpio edge triggering and poll() is fast enough for you? That way you don't have to poll the status from the userspace application but edge triggering will report it via poll(). See Documentation/gpio.txt in kernel source.
If the edge triggering via sysfs is not good enough, then the proper way is to develop a kernel driver that takes care of the time critical part and exports the results to userspace via a API (sysfs, device node, etc).
I am also facing the same problem, I read this document http://people.ee.ethz.ch/~arkeller/linux/multi/kernel_user_space_howto-6.html, so planning to use signals. In my case there is no chance of losing signals, because
1. the system is closed loop, after signals executed then only I will get another signal.
2. And I am using POSIX real-time signals.