Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
Questions asking for code must demonstrate a minimal understanding of the problem being solved. Include attempted solutions, why they didn't work, and the expected results. See also: Stack Overflow question checklist
Closed 9 years ago.
Improve this question
My situation:
I would like the data received on a network card to reach my application as fast as possible. I have concluded that the best (as in lowest latency) solution is to implement a network stack in my user space.
The network traffic can be a proprietary protocol (if it makes writing the network stack easier) because it is simply between two local computers.
1) What is the bare minimum list of functions my network stack will need to implement?
2) Would I need to remove/disable whatever network stack is currently in my Linux/how would I do this?
3) How exactly would I write the driver? I presume I would need to find exactly where the driver code gets called and then instead of the driver/network stack being called, I would instead send the data to a piece of memory which I can access from my application?
I think the already built-in PF_PACKET socket type does exactly what you want to implement.
Drawback: The application must be started with root rights.
There are some enhancements to the PF_PACKET system that are described on this page:
Linux packet mmap
The Kernel is in control of the NIC card. Whenever you pass data between kernel and user-space, there is a context-switch between the kernel rings, which is costly. My understanding is that you would use the standard API's while setting the buffers to a larger size allowing larger chunks of data to be copied between user and kernel-space at a time, reducing the number of context switches for a given size of data.
As far as implementing your own stack, it is unlikely a single person can created a faster network stack than the one built into the kernel.
If the linux kernel is not capable of processing packets at a speed you require, you might want to investigate NIC cards with more onboard hardware processing power. These sorts of things are used for network throughput testing etc.
Related
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 2 years ago.
Improve this question
I studied the content from the osdev.org website and built a compiler, launched a test kernel, but I thought, how can I create a primitive text shell for the kernel with commands? Maybe someone can explain to me with an example how to implement this. There is nothing interesting on the site itself, of course there is an article but it is useless for me. I'm a beginner if that.
how can I create a primitive text shell for the kernel with commands?
The correct way is:
Write enough kernel code to manage various resources (memory, IRQs, IO ports, DMA channels, ...). This should include managing time (a scheduler), and should also include some kind of inter-process communication (so that the scheduler can be told "Don't give this task any more CPU time until/unless it receives data from inter-process communication").
Enumerate devices, determine each device's resources, and start drivers for whichever devices you find. Note that this is hierarchical. For example, if you enumerate PCI buses and find 2 USB controllers attached to PCI buses and start their device drivers, then you'll need to enumerate each of the USB controllers to find any USB devices attached to USB buses and might find 3 USB hubs, and then you'll need to enumerate all 3 USB hubs to see what is plugged into them. All of this should be coordinated by some kind of "device manager" which keeps track of a hierarchical tree of devices, so that (e.g.) if a device is unplugged or sent to a power saving state (or if its device driver crashes) you can inform drivers that depended on that device (e.g. if a USB hub is unplugged you can inform all drivers for devices that were attached to that hub).
write keyboard device driver/s. These should decode the data from the device (likely using tables and other information describing a "keyboard layout" that's loaded from file system) and send packets of data using the kernel's inter-process communication (so that any task can say "don't give me any CPU time until I receive data from the keyboard driver"). This will involve designing a standard way that all keyboard drivers (and all software emulating a keyboard - e.g. "on screen keyboard" for people using touchscreens, etc) will behave (e.g. the format of that data packet they send, etc); and probably should involve creating a formal "keyboard device driver interface" specification for your OS to describe whatever you designed (in addition to designing a file format for "keyboard layout files").
write video device driver/s. This will also include designing a suitable video driver interface for your OS (and should including writing a formal specification describing it). However; video is complex and you can cheat by only designing part of the video driver interface and leaving the rest of it (video mode setting, 3D, GPGPU, ...) until later. The same applies to the video driver itself - you will want to start with "generic raw frame buffer driver" (that just uses a frame buffer configured by the boot loader) and probably won't write actual drivers for specific video cards.
(optional) write some kind of upper layer to control which task is the main task for each set of user input/output devices. This allows the user to have multiple virtual consoles and switch between them (e.g. maybe with "control+alt+F1" to "control+alt+F12"), possibly allowing some virtual consoles to be associated with terminals and others to be associated with different GUIs. It can also make it easy to support "multiple seat" (e.g. if there's 2 keyboards and 2 monitors, then you can have 2 completely separate users with each user having one keyboard and one monitor).
create a task with a simple main loop that uses inter-process communication ("don't give me any more CPU time until/unless I receive data from the keyboard") and processes the data it received to build up a current command string, then (if/when the user presses the enter key) parses the command string and does whatever the command says. Note that if you get this far it's tempting to do a tiny little bit of extra work to support user-space, and make it a normal process instead of a kernel task.
The incorrect way is:
don't have a kernel that supports some/most of the things that device drivers and other code must rely on
don't do any kind of device enumeration. Instead, make wild assumptions about which devices are present and which resources they use.
don't put any thought into device driver interfaces. Just slap together whatever seemed convenient (and continually break everything whenever you change any device driver).
don't use tasks (or inter-process communication). Instead, build the "shell" into the keyboard driver's IRQ handler to make sure the entire OS "pauses" when someone enters any time consuming command.
don't continue working on the OS after you get the shell to "work". This will be necessary because the code will be too inflexible and too fragile (any attempt to do anything else will cause you to have to rewrite everything).
Note: In my experience people that ask questions like "how do I write a kernel shell" are likely to have skipped everything that matters (because they'd know how to write a shell if they've done everything that a shell depends on); and are so focused on having a shell that they're very tempted to do it all the incorrect way (and then get stuck later and regret it).
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 4 years ago.
Improve this question
I was trying to make REST calls from C; came across libcurl which was successful in doing that dynamically. But the code needs to be ported on to a Cortex M0 board, which need a lower footprint. Is there any workaround? All I need is to make a REST API call from C without any external library or overhead.
Well, how low do you want to go?
C doesn't know anything about REST, it doesn't know HTTP, not even TCP or something like a network interface. On bare metal, you'd start by reading the hardware specs of your network interface card and programming it (through ports, memory mapped registers, etc....) -- You'd have to understand ARP, IP, ICMP etc (and, of course, implement it), just to get a TCP connection on top of that.
Assuming there's an operating system in place, you'll be given some API, then the answer would depend on what this API allows. A typical level would be a "socket abstraction", like BSD sockets, which gives you functionality to establish a TCP connection. So, "all" you'd have to do is implement a HTTP client on top of that.
Unfortunately, HTTP itself is a complex protocol. You'd have to implement all the requests you need, with Content-Types, transfer encodings, etc and also handle all possible server responses appropriately. These are a lot. Bring content negotiation to the table, partial responses, etc ... it's "endless" work. That's exactly the reason there are libraries like curl that already implement all of this for you.
So, sorry to say that, but there's no simple answer possible giving you what you want here. If you want to get the job done, use a library. Maybe you can find something smaller than libcurl.
What you can do is to compile the library yourself, linking it statically and using compiler options like gcc's -ffunction-sections -fdata-sections and the linker option --gc-sections in an attempt to drop code from the library you don't use, this might help to reduce size.
Per http://www.solacesystems.com/blog/kernel-bypass-revving-up-linux-networking:
[...]a network driver called OpenOnload that use “kernel bypass” techniques to run the application and network driver together in user space and, well, bypass the kernel. This allows the application side of the connection to process many more messages per second with lower and more consistent latency.
[...]
If you’re a developer or architect who has fought with context switching for years kernel bypass may feel like cheating, but fortunately it’s completely within the rules.
What are the functions needed to do such kernel bypassing?
A TCP offload engine will "just work", no special application programming needed. It doesn't bypass the whole kernel, it just moves some of the TCP/IP stack from the kernel to the network card, so the driver is slightly higher level. The kernel API is the same.
TCP offload engine is supported by most modern gigabit interfaces.
Alternatively, if you mean "running code on a SolarFlare network adapter's embedded processor/FPGA 'Application Onload Engine'", then... that's card-specific. You're basically writing code for an embedded system, so you need to say which kind of card you're using.
Okay, so the question is not straight forward to answer without knowing how the kernel handles the network stack.
In generel the network stack is made up of a lot of layers, with the lowest one being the actual hardware, typically this hardware is supported by means of drivers (one for each network interface), the nic's typically provide very simple interfaces, think recieve and send raw data.
On top of this physical connection, with the ability to recieve and send data is a lot of protocols, which are layered as well, near the bottem is the ip protocol, which basically allows you to specify the reciever of your information, while at the top you'll find TCP which supports stable connections.
So in order to answer your question, you most first figure out which part of the network stack you'll need to replace, and what you'll need to do. From my understanding of your question it seems like you'll want to keep the original network stack, and then just sometimes use your own, and in that case you should really just implement the strategy pattern, and make it possible to state which packets should be handled by which toplevel of the network stack.
Depending on how the network stack is implemented in linux, you may or may not be able to achieve this, without kernel changes. In a microkernel architecture, where each part of the network stack is implemented in its own service, this would be trivial, as you would simply pipe your lower parts of the network stack to your strategy pattern, and have this pipe the input to the required network toplevel layers.
Do you perhaps want to send and recieve raw IP packets?
Basically you will need to fill in headers and data in a ip-packet.
There are some examples here on how to send raw ethernet packets:
:http://austinmarton.wordpress.com/2011/09/14/sending-raw-ethernet-packets-from-a-specific-interface-in-c-on-linux/
To handle TCP/IP on your own, i think that you might need to disable the TCP driver in a custom kernel, and then write your own user space server that reads raw ip.
It's probably not that efficient though...
Closed. This question is off-topic. It is not currently accepting answers.
Want to improve this question? Update the question so it's on-topic for Stack Overflow.
Closed 10 years ago.
Improve this question
I have done buffer overflow exploits for user level processes in the past. However this does not seem to work good if I try to overflow the buffer of a vulnerable kernel module. Here's what I do:
There is a vulnerable kernel module which I can open as a file and read/write to it. The write operation is done without bounds checking. So I do a write operation and overflow the buffer and overwrite the return address as the address of an environment variable that has my shellcode. But something is going wrong. The kernel crashes and after rebooting I opened /var/log/messages and find that the eip is correctly pointing to the address I overwrote. But still it crashes saying "Unable to handle kernel null pointer dereference at virtual address"
Any reason why this would happen? Why wouldn't the control be redirected to a overwritten return address?
Note: I ran this on redhat enterprise linux with exec-shield and ASLR turned off.
The kernel can't jump to user addresses without performing a kernel exit, since it is running in a privileged mode with a different configuration than userspace (e.g. different paging tables, CPU permission bits, etc.).
So, in order to get shellcode into the kernel, you'd have to pack the shellcode into the buffer written to the driver (and copied to the kernel), and somehow get the address of it. This is not so hard if you have access to the kernel map, but recently Linux distributions have begun locking access to such sensitive information to make it harder to exploit kernel bugs.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 4 years ago.
Improve this question
After writing several different custom serial protocols for various projects, I've started to become frustrated with re-inventing the wheel every time. In lieu of continuing to develop custom solutions for every project, I've been searching for a more general solution. I was wondering if anyone knows of a serial protocol (or better yet, implementation) that meets the following requirements:
Support multiple devices. We'd like to be able to support an RS485 bus.
Guaranteed delivery. Some sort of acknowledgement mechanism, and some simple error detection (CRC16 is probably fine).
Not master/slave. Ideally the slave(s) would be able to send data asynchronously. This is mostly just for aesthetic reasons, the concept of polling each slave doesn't feel right to me.
OS independence. Ideally it wouldn't rely on a preemptive multitasking environment at all. I'm willing to concede this if I can get the other stuff.
ANSI C. We need to be able to compile it for several different architectures.
Speed isn't too much of an issue, we're willing to give up some speed in order to meet some of those other needs. We would, however, like to minimize the amount of required resources.
I'm about to start implementing a sliding window protocol with piggybacked ACKs and without selective repeat, but thought that perhaps someone could save me the trouble. Does anyone know of an existing project that I could leverage? Or perhaps a better strategy?
UPDATE
I have seriously considered a TCP/IP implementation, but was really hoping for something more lightweight. Many of the features of TCP/IP are overkill for what I'm trying to do. I'm willing to accept (begrudgingly) that perhaps the features I want just aren't included in lighter protocols.
UPDATE 2
Thanks for the tips on CAN. I have looked at it in the past and will probably use it in the future. I'd really like the library to handle the acknowledgements, buffering, retries etc, though. I guess I'm more looking for a network/transport layer instead of a datalink/physical layer.
UPDATE 3
So it sounds like the state of the art in this area is:
A trimmed down TCP/IP stack. Probably starting with something like lwIP or uIP.
A CAN based implementation, it would probably rely heavily on the CAN bus, so it wouldn't be useful on other physical layers. Something like CAN Festival could help along the way.
An HDLC or SDLC implementation (like this one). This is probably the route we'll take.
Please feel free to post more answers if you come across this question.
Have you considered HDLC or SDLC?
There's also LAP/D (Link Access Protocol, D-Channel).
Uyless Black's "Data Link Protocols" is always nearby on my bookshelf - you might find some useful material in there too (even peruse the TOC & research the different protocols)
CAN meets a number of your criteria:
Support multiple devices: It supports a large number of devices on one bus. It's not, however, compatible with RS485.
Guaranteed delivery: The physical layer uses bit-stuffing and a CRC, all of which are implemented in hardware on an increasing number of modern embedded processors. If you need acknlowedgement, you need to add that on top yourself.
Not master/slave: There are no masters or slaves; all devices can transmit whenever they want. The processor hardware deals with arbitration and contention.
OS independence: Not applicable; it's a low-level bus. What you put on top of that is up to you.
ANSI C: Again, not applicable.
Speed: Typically, up to 1 Mbps up to 40 m; you can choose your own speed for your application.
As mentioned, its definition is fairly low-level, so there's still work to be done to turn it into a full protocol to meet your needs. However, the fact that a lot of the work is done in hardware for you does it make very useful for a variety of applications.
I'd guess a reasonable starting point could be uIP.
(Adding Wikipedia article on µIP since original link is dead.)
Would you consider the MODBUS protocol? It is master/slave oriented, so the slave could not initiate the transfer, but otherwise is lightweight for implementation, free, and well supported with high level tools. You should just get a grasp on their terminology /like holding register, input register, output coil etc).
Phy level could be RS232, RS485, Ethernet...
Have a look at Microcontroller Internet Network (MIN):
https://github.com/min-protocol/min
Inspired by CAN but using standard UART hardware, with Fletcher's checksum and frame format checking for error detection and byte-stuffing to mark a frame header.
Take a look at Profibus.
If you don't want master/slave, I think you ought to do the arbitration with hardware (Canbus, FlexRay).