I'm studying databases now and I'd like to know if ODBC drivers really run in kernel space. Wouldn't it be easier just to use sockets for client connections? The only reasonable explanation I can think is that it would be performance issue with sockets. Or there are other reasons?
ODBC isn't a protocol! It's an API. ODBC isn't about how to talk with DBMS. It's about how to talk with a thing that know how to talk with DBMS. That thing is a C library. It's called "DBMS-specific library" or "ODBC-driver". ODBC specifies what C functions (and names of this functions) a library must have so that can be used as an ODBC-driver. ODBC-driver implements DBMS-specific protocol. It can use whatever it wants to talk with DBMS: sockets, pipes, shared-memory, pigeons, whatever
Also there is an interesting thing called "ODBC-ODBC bridge". It allows you to connect to DBMS remotely without having locally installed DBMS-specific library. Similar to a protocol, isn't it?
User-space driver is pretty common thing. So it's not weird that ODBC-libraries are called "drivers". It's very convenient to implement all logic in user-space and use kernel-space only to send data to device.
Related
I'm trying to implement an echo TCP server as a loadable kernel module.
Should I use sock_create, or sock_create_kern?
Should I use accept, or kernel_accept?
I mean it does make sense that I should use kernel_accept for example; but I don't know why. Can't I use normal sockets in the kernel?
The problem is, you are trying to shoehorn an user space application into the kernel.
Sockets (and files and so on) are things the kernel provides to userspace applications via the kernel-userspace API/ABI. Some, but not all, also have an in-kernel callable, for cases when another kernel thingy wishes to use something provided to userspace.
Let's look at the Linux kernel implementation of the socket() or accept() syscalls, in net/socket.c in the kernel sources; look for SYSCALL_DEFINE3(socket, and SYSCALL_DEFINE3(accept,, SYSCALL_DEFINE4(recv,, and so on.
(I recommend you use e.g. Elixir Cross Referencer to find specific identifiers in the Linux kernel sources, then look up the actual code in one of the official kernel Git trees online; that's what I do, anyway.)
Note how pointer arguments have a __user qualifier: this means the data pointed to must reside in user space, and that the functions will eventually use copy_from_user()/copy_to_user() to retrieve or set the data. Furthermore, the operations access the file descriptor table, which is part of the process context: something that normally only exist for userspace processes.
Essentially, this means your kernel module must create an userspace "process" (enough of one to satisfy the requirements of crossing the userspace-kernel boundary when using kernel interfaces) to "hold" the memory and file descriptors, at minimum. It is a lot of work, and in the end, it won't be any more performant than an userspace application would be. (Linux kernel developers have worked on this for literally decades. There are some proprietary operating systems where doing stuff in "kernel space" may be faster, but that is not so in Linux. The cost to do things in userspace is some context switches, and possibly some memory copies (for the transferred data).)
In particular, the TCP/IP and UDP/IP interfaces (see e.g. net/ipv4/udp.c for UDP/IPv4) do not seem to have any interface for kernel-side buffers (other than directly accessing the rx/tx socket buffers, which are in kernel memory).
You have probably heard of TUX web server, a subsystem patch to the Linux kernel by Ingo Molnár. Even that is not a "kernel module server", but more like a subsystem that an userspace process can use to implement a server that runs mostly in kernel space.
The idea of a kernel module that provides a TCP/IP and/or UDP/IP server, is simply like trying to use a hammer to drive in screws. It will work, after a fashion, but the results won't be pretty.
However, for the particular case of an echo server, it just might be possible to bolt it on top of IPv4 (see net/ipv4/) and/or IPv6 (see net/ipv6/) similar to ICMP packets (net/ipv4/icmp.c, net/ipv6/icmp.c). I would consider this route if and only if you intend to specialize in kernel-side networking stuff, as otherwise everything you'd learn doing this is very specialized and not that useful in practice.
If you need to implement something kernel-side for an exercise or something, I'd recommend steering away from "application"-type ideas (services or similar).
Instead, I would warmly recommend developing a character device driver, possibly implementing some kind of inter-process communications layer, preferably bus-style (i.e., one sender, any number of recipients). Something like that has a number of actual real-world use cases (both hardware drivers, as well as stranger things like kdbus-type stuff), so anything you'd learn doing that would be real-world applicable.
(In fact, an echo character device -- which simply outputs whatever is written to it -- is an excellent first target. Although LDD3 is for Linux kernel 2.6.10, it should be an excellent read for anyone diving into Linux kernel development. If you use a more recent kernel, just remember that the example code might not compile as-is, and you might have to do some research wrt. Linux kernel Git repos and/or a kernel source cross referencer like Elixir above.)
In short sockets are just a mechanism that enable two processes to talk, localy or remotely.
If you want to send some data from kernel to userspace you have to use kernel sockets sock_create_kern() with it's family of functions.
What would be the benefit of TCP echo server as kernel module?
It makes sense only if your TCP server provides data which is otherwise not accessible from userspace, e.g. read some post-mortem NVRAM which you can't read normally and to send it to rsyslog via socket.
I've found that Elixir programs can run C code either via NIFs (native implemented functions) or via OS-level ports. Having read those and similar links, I'm not a hundred percent clear on when to use one or the other method (or something else entirely?), and feel it would be good to have a direct comparison available, for myself and other novices. Can anyone provide?
What are ports?
Ports are basically separate programs which are run separately from the Erlang VM. The Erlang VM communicates with the running port over standard input/output, and the resulting port lives behind an Erlang process that owns it and can facilitate communication between the port and the rest of your Erlang or Elixir application. Ports are "safe" in the sense that if the port crashes, it doesn't bring down the whole Erlang VM.
Porcelain might be of interest as a possible improvement and expansion over what's already provided in the Port module. System.cmd/3 also uses ports in its underlying implementation.
What are NIFs?
Native inline functions or "NIFs" are functions defined in what are essentially shared libraries / DLLs loaded by the Erlang VM and written using some language which exposes a C-compatible ABI. NIFs are more efficient than ports (since they don't have to communicate over STDIN/STDOUT) and are simpler in many respects (since you don't have to deal with encoding and decoding data between your Elixir and non-Elixir codebases), but they're also much less safe; a NIF can crash the Erlang VM, and a long-running NIF can potentially lock up the Erlang VM (since the scheduler can't reason about native code).
What are port drivers?
Port drivers are kind of an in-between approach to integrating external code with an Erlang or Elixir codebase. Like NIFs, they're loaded into the Erlang VM, and a port driver can therefore crash or hang the whole VM. Like ports, they behave similarly to Erlang processes.
When should I use a port?
You want your external code to behave like an ordinary Erlang process (at least enough for such a process to wrap it and send/receive messages on behalf of your external code)
You want the Erlang VM to be able to survive your external code crashing
You want to implement a long-running task in your external code
You want to write your external code in a language that does not support C-compatible FFI (or otherwise don't want to deal with your language's FFI facilities)
When should I use a NIF?
You want your external code to behave like a collection of ordinary Erlang functions (particularly if you want to define an Erlang/Elixir module that exports functions implemented in native-compiled code)
You want to avoid any potential performance hits / overhead from communicating via standard input/output and/or you want to avoid having to translate between Erlang terms and something your external code understands
You are reasonably confident that the things your external code is doing are neither long-running nor likely to crash (including, in the latter case, if you're writing your NIFs in something like Rust; see also: Rustler), or...
You are reasonably confident that crashing or hanging the Erlang VM is acceptable for your use case (e.g. your code is both distributed and able to survive the sudden loss of an Erlang node, or you're writing a desktop application and an application-wide crash is not a big deal aside from being an inconvenience to users)
When should I use port drivers?
You want your external code to behave like an Erlang process
You want to avoid the overhead and/or complexity of communicating over standard input/output
You are reasonably confident that your port driver won't crash or hang the Erlang VM, or...
You are reasonably confident that a crash or hang of the Erlang VM is not a critical issue
What do you recommend?
There are two aspects to weigh here:
Process-like v. module-like
Safe v. efficient
If you want maximum safety behind a process-like interface, go with a port.
If you want maximum safety behind a module-like interface, go with a module with functions that either wrap System.cmd/3 or directly use a port to communicate with your external code
If you want better efficiency behind a process-like interface, go with a port driver.
If you want better efficiency behind a module-like interface, go with NIFs.
Is it possible to migrate a single and particular TCP connection inside a running process in one machine to another machine using CRIU tools in Linux?
What I want is to dump a particular TCP Connection information in a memory and transfer this information to a peer machine. Inside this machine, I will use the dumped information to recreate the the migrated TCP connection. Does anyone have an example or tutorial in c language?
I am aware about different solutions like SockMi which provides Kernel Module + User Space APIs to migrate a certain TCP Socket. However, I want to use CRIU tools since it is part of Linux Mainline.
Right now we only have the TCP migration functionality integrated into CRIU tool. It sits in the sk-tcp.c file, the whole TCP-repair code is there, though it's bound to the rest of CRIU.
On the other hand, we've been asked for TCP-only migration for quite a while, it's possible to pull this code into smth like libcriutcp.so, but it will require patching. You're welcome to participate to the https://github.com/xemul/criu/issues/72
How to run c code in Erlang? Please explain with simple example.
Basically I am trying to post data on a server with this.I will trigger the C code through Erlang and then this C code will post data to server.
This question comes back very often: one of the best answers is here
I would also add ports to the mix. Do not confuse them with port drivers: ports vs port drivers
For your use case ports should be the best option. Nifs can crash Erlang VM, so they are only for optimizing critical code. Port drivers have similar problems. C nodes mimic Erlang VM and I haven't seen good use case for them yet.
Ports are just normal OS processes with a protocol used to communicate with Erlang VM. This is usually the default way to go, when you want to call some C code from Erlang.
I need to make labview communicate with a C/C++ application. Both the applications run on the same machine. What is the IPC mechanism with lower overhead and highest speed available in LabView?
TCP, UDP, ActiveX, DDE, file transactions, or perhaps just directly calling a dll are the solutions that come to mind.
First I'd just call a dll if you can manage with that. Assuming you're tied in to using two separate applications then:
I'd use TCP or UDP. File transactions are clunky but easy to implement, DDE is older but might be viable (I'd recommend against it).
Basic TCP/IP in Labview
TCP/IP and UDP in Labview
Calling a dll from Labview
Have you investigated straight up TCP or UDP?
It'll make it easy if you ever need to separate the applications onto different machines later on down the road. Implementation is pretty straight forward too, although it may not be the fastest throughput.
What speeds are we talking about here?
NI has provided a thorough document explaining that: Using External Code in LabVIEW [pdf]. In brief, you can use:
Shared Libraries (on windows they are called DLLs). According to the above document, any
language can be used to write DLLs as long as the DLLs can be called
using one of the calling conventions LabVIEW supports, either
stdcall or C."
Code Interface Node (CIN), which is a block diagram node that links C/C++ source
code to LabVIEW.
.NET technology.
Note that "Shared Libraries" and "Code Interface Node" are supported on Windows, Max OS X, Linux and Solaris.