Let’s say I have created a simple program in C, using GTK, that brings up a label. When this program is run using ./a.out from the command line, I am aware that a new process is forked, execve is called, etc.
But when exactly in the process of running a program is my GUI, with my label, drawn to the screen? At what point is X11 interfaced with? I am struggling to understand exactly when such GUIs are drawn in terms of the steps with the Linux process lifecycle.
To help illustrate my understanding, this is the link I am using to try and understand the general process life cycle - it contains no information on when GUIs are drawn however. http://glennastory.net/?p=870
The system for screen rendering has changed often on Linux. An extensive but probably not this much reliable summary can be found at: https://en.wikipedia.org/wiki/Direct_Rendering_Infrastructure.
The information is all there but it assumes a certain level of knowledge from the reader. I can summarize what I understand from the link and give some further information.
In the classic X Window System architecture the X Server is the only process with exclusive access to the graphics hardware, and therefore the one which does the actual rendering on the framebuffer. All that X clients do is communicate with the X Server to dispatch rendering commands. Those commands are hardware independent, meaning that the X11 protocol provides an API that abstracts the graphics device so the X clients don't need to know or worry about the specifics of the underlying hardware. Any hardware specific code lives inside the Device Dependent X, the part of the X Server that manages each type of video card or graphics adapter and which is also often called the video or graphics driver.
What this says is that the X server is started as a privileged process (root). A non-root process communicates with the X server using xlib. Xlib itself communicates with the X server using socket system calls. This allows to securely communicate with the X server (a form of secure IPC). The socket interface to the X server is device independent. Your non-root process will call the same function regardless of the underlying graphics card.
The device independent portion of the X server will make calls in the device dependent portion. The device dependent portion is basically a user mode driver implementation. It is not the same as a kernel driver because it is actually called by a user mode root process. The kernel driver is a separate entity that is added to the kernel at boot. The kernel driver is either programmed by some third party which reads the graphics card documentation or, more likely, by the graphics card vendor themselves. The kernel driver is nothing more than a fancy character device responding to ioctl system calls and doing PCI read/writes on registers that make the graphics card do DMA operations and interact with the screen.
The user mode portion of the driver (the device dependent portion of the X server) is also implemented by the graphics card vendor. It needs to be like this because the X server doesn't know (and it shouldn't) anything about how the driver works. The X server thus presents generic functions that are called by the device independent portion of it. It is these generic functions that are implemented by the graphics card vendor.
The rise of 3D rendering has shown the limits of this architecture. 3D graphics applications tend to produce large amounts of commands and data, all of which must be dispatched to the X Server for rendering. As the amount of inter-process communication (IPC) between the X client and X Server increased, the 3D rendering performance suffered to the point that X driver developers concluded that in order to take advantage of 3D hardware capabilities of the latest graphics cards a new IPC-less architecture was required. X clients should have direct access to graphics hardware rather than relying on a third party process to do so, saving all the IPC overhead. This approach is called "direct rendering" as opposed to the "indirect rendering" provided by the classical X architecture. The Direct Rendering Infrastructure was initially developed to allow any X client to perform 3D rendering using this "direct rendering" approach.
What this says is that the X server indirect (through sockets) rendering was not good enough for 3D. They thus implemented a direct rendering. After all, if you install a driver on your system, you are completely vulnerable to what it is going to do. You can count on a driver written by hardware vendors to collaborate with the X server to render where the driver should render. This is exactly what happens. There is still kernel and user mode portions of the driver. The kernel portion stays the same but the user mode portion becomes an implementation of a 3D rendering convention such as OpenGL. It is exactly what is stated a bit further:
the DRI client —an X client performing "direct rendering"— needs a hardware specific "driver" able to manage the current video card or graphics adapter in order to render on it. These DRI drivers are typically provided as shared libraries to which the client is dynamically linked. Since DRI was conceived to take advantage of 3D graphics hardware, the libraries are normally presented to clients as hardware accelerated implementations of a 3D API such as OpenGL, provided by either the 3D hardware vendor itself or a third party such as the Mesa 3D free software project.
Afterwards, it says:
the X Server provides an X11 protocol extension —the DRI extension— that the DRI clients use to coordinate with both the windowing system and the DDX driver.[9] As part of the DDX (device dependent X) driver, it's quite common that the X Server process also dynamically links to the same DRI driver that the DRI clients, but to provide hardware accelerated 3D rendering to the X clients using the GLX extension for indirect rendering (for example remote X clients that can't use direct rendering). For 2D rendering, the DDX driver must also take into account the DRI clients using the same graphics device.
The X server still does indirect rendering with the GLX extension.
the access to the video card or graphics adapter is regulated by a kernel component called the Direct Rendering Manager (DRM).[10] Both the X Server's DDX driver and each X client's DRI driver must use DRM to access to the graphics hardware. DRM provides synchronization to the shared resources of the graphics hardware —resources such as the command queue, the card registers, the video memory, the DMA engines, ...— ensuring that the concurrent access of all those multiple competing user space processes don't interfere with each other. DRM also serves as a basic security enforcer that doesn't allow any X client to access the hardware beyond what it needs to perform the 3D rendering.
Like I said earlier, there is a kernel and user mode portion of the graphics card driver. The DRM is partly generic (the same for every graphics card) and partly specific to one graphics card. This is made possible by how PCI devices work (all graphics card today are PCI if not all peripherals). PCI devices have some common registers that must be found on all devices. This represents the generic portion of the DRM. Among the common registers there is BAR registers that point to a device specific portion of the configuration space. This represents the specific portion of the DRM. The graphics card vendor thus provides a kernel module in the form of a character device. The kernel then detects using the PCI IDs of the graphics card that it must use this kernel module to drive it. Then, it will present a virtual character device file to user mode which can be opened to do further ioctl operations on it.
The DRI2 extension provides other core operations for the DRI clients, such as finding out which DRM device and driver should they use (DRI2Connect) or getting authenticated by the X Server in order to be able to use the rendering and buffer facilities of the DRM device (DRI2Authenticate).
What this says is that the DRI clients still need to get authenticated with the X server before they can actually render (make DRM system calls).
At least, this is what I understand from the wikipedia page. I cannot be completely sure but it must be something along those lines.
Related
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 2 years ago.
Improve this question
I studied the content from the osdev.org website and built a compiler, launched a test kernel, but I thought, how can I create a primitive text shell for the kernel with commands? Maybe someone can explain to me with an example how to implement this. There is nothing interesting on the site itself, of course there is an article but it is useless for me. I'm a beginner if that.
how can I create a primitive text shell for the kernel with commands?
The correct way is:
Write enough kernel code to manage various resources (memory, IRQs, IO ports, DMA channels, ...). This should include managing time (a scheduler), and should also include some kind of inter-process communication (so that the scheduler can be told "Don't give this task any more CPU time until/unless it receives data from inter-process communication").
Enumerate devices, determine each device's resources, and start drivers for whichever devices you find. Note that this is hierarchical. For example, if you enumerate PCI buses and find 2 USB controllers attached to PCI buses and start their device drivers, then you'll need to enumerate each of the USB controllers to find any USB devices attached to USB buses and might find 3 USB hubs, and then you'll need to enumerate all 3 USB hubs to see what is plugged into them. All of this should be coordinated by some kind of "device manager" which keeps track of a hierarchical tree of devices, so that (e.g.) if a device is unplugged or sent to a power saving state (or if its device driver crashes) you can inform drivers that depended on that device (e.g. if a USB hub is unplugged you can inform all drivers for devices that were attached to that hub).
write keyboard device driver/s. These should decode the data from the device (likely using tables and other information describing a "keyboard layout" that's loaded from file system) and send packets of data using the kernel's inter-process communication (so that any task can say "don't give me any CPU time until I receive data from the keyboard driver"). This will involve designing a standard way that all keyboard drivers (and all software emulating a keyboard - e.g. "on screen keyboard" for people using touchscreens, etc) will behave (e.g. the format of that data packet they send, etc); and probably should involve creating a formal "keyboard device driver interface" specification for your OS to describe whatever you designed (in addition to designing a file format for "keyboard layout files").
write video device driver/s. This will also include designing a suitable video driver interface for your OS (and should including writing a formal specification describing it). However; video is complex and you can cheat by only designing part of the video driver interface and leaving the rest of it (video mode setting, 3D, GPGPU, ...) until later. The same applies to the video driver itself - you will want to start with "generic raw frame buffer driver" (that just uses a frame buffer configured by the boot loader) and probably won't write actual drivers for specific video cards.
(optional) write some kind of upper layer to control which task is the main task for each set of user input/output devices. This allows the user to have multiple virtual consoles and switch between them (e.g. maybe with "control+alt+F1" to "control+alt+F12"), possibly allowing some virtual consoles to be associated with terminals and others to be associated with different GUIs. It can also make it easy to support "multiple seat" (e.g. if there's 2 keyboards and 2 monitors, then you can have 2 completely separate users with each user having one keyboard and one monitor).
create a task with a simple main loop that uses inter-process communication ("don't give me any more CPU time until/unless I receive data from the keyboard") and processes the data it received to build up a current command string, then (if/when the user presses the enter key) parses the command string and does whatever the command says. Note that if you get this far it's tempting to do a tiny little bit of extra work to support user-space, and make it a normal process instead of a kernel task.
The incorrect way is:
don't have a kernel that supports some/most of the things that device drivers and other code must rely on
don't do any kind of device enumeration. Instead, make wild assumptions about which devices are present and which resources they use.
don't put any thought into device driver interfaces. Just slap together whatever seemed convenient (and continually break everything whenever you change any device driver).
don't use tasks (or inter-process communication). Instead, build the "shell" into the keyboard driver's IRQ handler to make sure the entire OS "pauses" when someone enters any time consuming command.
don't continue working on the OS after you get the shell to "work". This will be necessary because the code will be too inflexible and too fragile (any attempt to do anything else will cause you to have to rewrite everything).
Note: In my experience people that ask questions like "how do I write a kernel shell" are likely to have skipped everything that matters (because they'd know how to write a shell if they've done everything that a shell depends on); and are so focused on having a shell that they're very tempted to do it all the incorrect way (and then get stuck later and regret it).
This can be a silly question as I do not know much about this topic at all... It seems that user applications can talk directly to GPU to render an image, for example using OpenGL, through mesa and libdrm, where the libdrm is a wrapper around various ioctl() calls, as illustrated in this graph. Does it mean that for every new frame of a 3D game, the game application needs to call the ioctl() once (or maybe even twice if KMS needs to be reached)? That sounds like a lot of user-kernel space barrier crossing (thinking about a 120 fps game).
libdrm is an user space wrapper to perform fine grained access of the underlying KMS driver features like modesetting, checking if plane being used is an overlay plane or primary plane etc. libdrm implementations are generally different for various CPU/GPU/OS combinations, as the h/w driver running in kernel tend to support different set of functionalities apart from the standard ones. The standard way of working with libdrm is to open a drm device available in /dev/ node and perform libdrm function calls using the fd returned from open().
More often than not, the display compositor software for a particular OS like X11, wayland, hardware-composer will need to be in control of the drm device, which means non privileged applications have no way of being DRM master. Most of the libdrm mode setting functionalities do not work if the application trying to use them are not the DRM master. Recommended practice instead of using libdrm directly, is to use a standard graphics library like openGL or VULKAN to prepare and render frames in your application.
The number of ioctls required to interact with the kernel DRM module is most likely not the biggest bottleneck you will face when trying to render high FPS applications. The preferred way to run high fps applications while cooperating with the display compositor of the target system is to have
a double or triple buffered setup for rendering, where the next buffer to be rendered is ready to be rendered before the current frame has finished rendered.
Take advantage of h/w acceleration wherever possible, e.g for performing scaling/resizing/image format conversions/color space conversions.
Pre compute and reuse shader elements
Try to reuse texture elements as much as possible instead of computing a lot of textures for every frame being rendered.
Use vector/SIMD/SSEv2,3,4/AVX/neon instructions wherever possible to take advantage of modern CPU pipelines
I'm doing an embedded-system project now. From my view, AT commands can be sent to a device to retrieve 4G information, dial and so on. On the other hand, we can also do that by calling APIs provided by the 4G vendor.
My question is what's the relationship between them? Is the API a wrapper for the AT commands?
TL;DR
Vendor's API (not only C, but also C++, Java or Python depending to the vendor and the modem model) can both be wrappers for AT commands and a wider, more powerful set of API were the user can port complex applications.
It depends on the vendor and on the model.
A jungle of modems produced by different vendors
It is impossible to define a general "rule" about API provided by a Cellular Module (not necessarily a 4G module).
First of all every vendor usually implements standard AT Commands (both Hayes commands and extended standard commands for cellular devices). In the same way every vendor has it's own implementation of the user application area where the customers can store their own application to control the modem's functionalities and to use them according to their own application requirements.
AT commands remain the interface to be used when the modem needs to be connected (and driven) by an external host. When the user application area is used, instead, a wider set of API is usually provided. They may include:
A library exporting a subset of the OS capabilities (threads management, events, semaphores, mutexes, SW timers, FS access and so on)
A library offering the capability to manage the specific HW of the device (GPIOs, SPI, I2C, ADC, DAC and so on)
A library offering a programmatical way to perform action, related to connectivity, that would normally be executed through AT commands (registration status check, PIN code insertion, PDP context activation, SMS management, TCP/UDP/TLS sockets)
The latter usually access a base layer involving all the functionalities provided by the modem. Usually this is the same layer invoked by the AT commands sent through modem's serial interface.
Sending AT commands... from the vendor's API?
Of course it often happens that the library mentioned above provides just a subset of the functionalities usually exported with the AT commands set so, in order to "fill the gap", a further set of APIs is usually exported as well:
A set of functions that allow the simulation of AT commands sent to the modem's serial port. Sending them and parsering the responses they send in the vitual internal serial/USB interface allow the user to port in the internal user application area the the application they previously run on an external host processor (with obvious BOM benefits).
As an example, please check Telit Appzone here and here. It was the inspiration of my answer because I know it very well.
I don't know why you name the title that there's a relationship between AT command and Linux-C API.
Regarding AT command, you can take a look at this wiki article for general information.
Each module has a specified AT command sets. Normally, the module manufacture just offers AT command set and what return values are.
Is API a wrapper of AT command?
If you can use the API provided by the manufacturer, then yes, it's a wrapper of the AT command handler.
My question is what's the relationship between them? Is the API a wrapper for the AT commands?
It is impossible to be sure without having any details of the device, but probably any C API for it wraps the AT command set, either by communicating with the device directly over an internal serial interface or by going through a device driver that uses AT commands to communicate with it.
However, it is at least conceivable, albeit unlikely, that the 4G device offers an alternative control path that the C API uses (definitely via a driver in this case).
I'm not quite sure what the point of the question is, though. If you are programming the device and its 4G component in C, and the manufacturer has provided a C API, then use it! If you are programming in some other language then at least consider using the C API, which you should be able to access from most other languages in some language-specific way. You should not expend effort on rolling your own without a compelling reason to reject the API already provided to you.
Let's say I have a webcam, and I installed the device driver for this webcam in my Linux OS, now a device file will be created for the device driver (for example: /dev/video0).
Now say I want to create a program in C that wants to access this webcam. How can my program access the device driver for the webcam, should my program use the device file (/dev/video0) to access the device driver, or is there another way?
You asked a general question, and then gave a specific example. I'll try to address both.
When you load a driver, the way to communicate with it from user space is by whatever means this driver defined. Typically, this is through a /dev device created for the driver. If that's the case, yes, that's the only way to communicate with it.
This is not universally true. Many drivers also have entries under the /sys sysfs pseudo file system, and some aspects can be modified through there. In fact, there are whole classes of drivers that are only accessible through the /sys fs. Prominent examples are GPIO and Led devices, that can be turned on and off via access to /sys/class/gpio and similar paths.
Another option, considered deprecated but still sometimes used, is to use the /proc pseudo file system. Again, this is up to the driver to define its communication method. As the user, you will have to follow whatever protocol the driver defined.
Also, some drivers don't have any file system presence at all. The most obvious standard example are network interfaces. The only way to communicate with them is via the networking system calls.
In the particular example you provided, you talked about a video camera that appears as /dev/video0. Such a camera is, usually, a Video4Linux (or v4l) camera, and those are accessed via their character devices.
With that said, the protocol for communicating with the camera might have wrappers that makes life easier. If you open the actual device, you might have to implement a rather complicated handshake with it. Instead, you can use the v4l library to wrap the details of the access.
Make no mistake. You're still talking to the character device in /dev. It's just that it's not your code that does it, but the library's.
On my ARM system (Tegra based), I'm running the mainline linux kernel. It uses the device tree system.
I have enabled a hardware driver for the General-Memory-Bus (part of the SoC) in the .dts file by setting its status="okay". Recompiled the dtb and booted the kernel. But no device (/dev/xx) appears.
The driver is compiled into the kernel and can be seen by
cat /lib/modules/$(uname -r)/modules.builtin
The command
cat /sys/firmware/devicetree/base/<path to device>/status
returns "okay".
Do I need to make some kind of "mknod"?
What else is nessesary?
The traditional UNIX "stream of bytes" device model is a pretty high-level abstraction of most modern hardware, and as such there are plenty of drivers which do not create /dev entries for the devices they control largely because they don't fit that model. Bus drivers in particular are very much a case of that - they exist, but only for the sake of discovering and allowing access to the devices behind them; there is no /dev/sata that lets you interact with the actual host controller, sending out raw commands on any old port regardless of what's connected or not; there is no /dev/usb that lets you attempt arbitrary transfers to arbitrary endpoints which may or may not exist.
Furthermore, your typical 'external interface' controller as in this case is orders of magnitude less complex than an interface like SATA or USB - the 'device' itself is often little more than a register block controlling some clocks and a chip-select multiplexer. Even if the driver did create something you could interact with directly, there's not exactly much you could do with it.
The correct way to proceed in this situation is to describe your FPGA device in the DT as a child of the GMI bus, accurately reflecting the hardware, no less, then develop your own driver for that. The bus driver itself just sits transparently in the middle. And if you do want a quick and dirty way to get started by just reading and writing bus addresses directly, well, it's behind a memory-mapped I/O region; that's exactly what /dev/mem exists for.