I am looking to implement some kind of transmission protocol in C, to use on a custom hardware. I have the ability to send and receive through RF, but I need to rely in some protocol that validates the package integrity sent/received, so I though it would be a good idea to implement some kind of UDP library.
Of course, if there is any way that I can modify the existing implementations for UDP or TCP so it works over my RF device it would be of great help. The only thing that I think it needs to be changed is the way that a single bit is sent, if I could change that on the UDP library (sys/socket.h) it would save me a lot of time.
UDP does not exist in standard C99 or C11.
It is generally part of some Internet Protocol layer. These are very complex software (as soon as you want some performance).
I would suggest to use some existing operating system kernel (e.g. Linux) and to write a network driver (e.g. for the Linux kernel) for your device. Life is too short to write a competitive UDP like layer (that could take you dozens of years).
addenda
Apparently, the mention of UDP in the question is confusing. Per your comments (which should go inside the question) you just want some serial protocol on a small 8 bits PIC 18F4550 microcontroller (32Kbytes ROM + 2Kbytes RAM). Without knowing additional constraints, I would suggest a tiny "textual" like protocol (e.g. in ASCII lines, no more than 128 bytes per line, \n terminated ....) and I would put some simple hex checksum inside it. In the 1980s Hayes modems had such things.
What you should then do is define and document the protocol first (e.g. as BNF syntax of the message lines), then implement it (probably with buffering and finite state automaton techniques). You might invent some message format like e.g. DOFOO?123,456%BE53 followed by a newline, meaning do the command DOFOO with arguments 123 then 456 and hex checksum BE53
Related
If I want to sniffing packet in linux without set any filters, I saw 2 options.
Use libpcap
Use raw socket myself like https://www.binarytides.com/packet-sniffer-code-in-c-using-linux-sockets-bsd-part-2/
Why libpcap is better than use raw sockets myself?
Three reasons:
1) It's way easier to correctly set up.
2) It's portable, even to Windows, which uses a quite similar yet different API for sockets.
3) It's MUCH faster.
1 and 2, IMO, don't need much explanation. I'll dive into 3.
To understand why libpcap is (generally) faster, we need to understand the bottlenecks in the socket API.
The two biggest bottlenecks that libpcap tends to avoid are syscalls and copies.
How it does so is platform-specific.
I'll tell the story for Linux.
Linux, since 2.0 IIRC, implements what it calls AF_PACKET socket family, and later with it PACKET_MMAP. I don't exactly recall the benefits of the former, but the latter is critical to avoid both copying from kernel to userspace (there are still a few copies kernel-side) and syscalls.
In PACKET_MMAP you allocate a big ring buffer in userspace, and then associate it to an AF_PACKET socket. This ring buffer will contain a bit of metadata (most importantly, a marker that says if a region is ready for user processing) and the packet contents.
When a packet arrives to a relevant interface (generally one you bind your socket to), the kernel makes a copy in the ring buffer and marks the location as ready for userspace*.
If the application was waiting on the socket, it gets notified*.
So, why is this better than raw sockets? Because you can do with few or none syscalls after setting up the socket, depending on whether you want to busy-poll the buffer itself or wait with poll until a few packets are ready, and because you don't need the copy from the socket's internal RX buffer to your user buffers, since it's shared with you.
libpcap does all of that for you. And does it on Mac, *BSD, and pretty much any platform that provides you with faster capture methods.
*It's a bit more complex on version 3, where the granularity is in "blocks" instead of packets.
I'm trying to do the same protocol that is defined and described here in that wiki
https://wiki.trezor.io/Developers_guide-Message_Workflows
My toolset is Protobuf for embedded systems: Nanopb.
STM32F7 using Serial port.
I'm trying to communicate right now between PC and STM32F7, and usually the communication will be done between two STM32F7 boards.
My questions:
What kind of protocol that is sufficient for a request and answer like the one in trezor ?
I googled and I found I have to use something like HLDC, is it necessary for that purpose or it's an overhead ?
Coding and design issue: I will fire a serial interrupt that always polling for the data that is communicated between two boards, now there will a very big state machine to decode each message type and do the event based on the message type. Is there an alternative design ?
Firstly, AFAIK, nanopb doesn't support the full range of possibilities in the Protobuf schema language. So you'll need a schema that works for nanopb, and hopefully that'll be good enough for the needs. However it can be very annoying as (so far as I know) the very useful oneof doesn't work.
Secondly, protobuf wireformat is not self delimiting. So you'll be squirting data down the serial cable, but it's not possible to reliably (if at all) tell where one message ends and another starts. So you'll need to transmit some sort of inter-message sync pattern of bytes, chosen to be unlikely to be encountered in a message. You'll have to read in the bytes inbetween sync patterns, place them in a buffer, and parse from that.
Thirdly, if you're sending a variety of different messages and you can't use oneof, then you'll need some other way of identifying what type of message has arrived so that you can parse it to the right type of object. That "way" could simply be a fixed sequence of message types, or a byte whose value identifies the message type, or a field that does the same thing in all the messages. oneof is attractive (though not available to you) because it can be used as a carrier for a variety of different message types; you simply parse the received data using the oneof's parser.
I'm currently attempting to get the best sending performance for an 802.11 frame, I am using libpcap but I wondered if I could speed it up using raw sockets (or any other possible method).
Consider this simple example code for libpcap with a device handle already created previously:
char ourPacket[60][50] = { {0x01, 0x02, ... , 0x50}, ... , {0x01, 0x02, ... , 0x50} };
for( ; ; )
{
for(int i; i = 0; i < 60; ++i)
{
pcap_sendpacket(deviceHandle, ourPacket[i], 50);
}
}
This code segment is done on a thread for each separate CPU core. Is there any faster way to do this for raw 802.11 frame/packets containing Radiotap headers that are stored in an array?
Looking at pcap's source code for pcap_inject (the same function but different return value), it doesn't seem to be using raw sockets to send packets? No clue.
I don't care about capturing performance, as a lot of the other questions have answered that. Is raw sockets even for sending layer 2 packets/frames?
As Gill Hamilton mentioned, the answer will depend on a lot of things. If you see super gains on one system, you may not see them on another, even if both are "running Linux". You're better off testing the code yourself. That being said, here is some info from what my team has found:
Note 1: all the gains were for code that did not just write frames/packets to sockets, but analyzed them and processed them, so it is likely that much or most of our gains were there rather than the read/write.
We are writing a direct raw socket implementation to send/receive Ethernet frames and IP packets. We're seeing about a 250%-450% performance gain on our measliest R&D system which is a MIPS 24K 5V system on a chip with a MT7530 integrated Ethernet NIC/Switch which can barely handle sustained 80 Mbit. On a very modest but much beefier test system with an Intel Celeron J1900 and I211 Gigabit controllers it drops to about 50%-100% vs c libpcap. In fact, we only saw about 80%-150% vs. Python dpkt/scapy implementation. We only saw maybe about a 20% gain on a generic i5 Linux dual-gigabit system vs. a c libpcap implementation. So based on our non-rigorous testing, we saw up to a 20x difference in performance gains of the code depending on the system.
Note 2: All of these gains were when using maximum optimizations and strictest compile parameters during compiling of the custom c code, but not necessarily for the c libpcap code (making all warnings errors on some of the above systems make the libpcap code not compile, and who wants to debug that?), so the differences may be less dramatic. We need to squeeze out every last ounce of performance to enable some sophisticated packet processing using no more than 5.0V and 1.5A, so we'll ultimately be going with a custom ASIC which may be FPGA. That being said, it's A LOT of work to get it working without bugs and we're likely going to be implementing significant portions of the Ethernet/IP/TCP/UPD stack, so I don't recommend it.
Last note: The CPU usage on the MIPS 24K system was about 1/10 for the custom code, but again, I would say that that vast majority of that gain was from the processing.
I have developed an embedded solution which communicates over a Multi Drop Bus and now I would like to develop a PC based application which monitors traffic on the bus.
MDB supports true 9 data bits (plus start/stop/parity - and *no fudging* by using the parity bit as a 9th data bit) whereas standard Windows and Linux libraries offer a maximum of 8 data bits.
I have a StarTech PCI2S950 PC serial port card which supports 9-data bits, but am not sure how to code my monitoring app & have googled a lot to no great avail.
I would prefer to code in C (or Delphi, or C++). I have a slight preference for Cygwn, but am willing to use straightforward Windows or Linux.
Just anything to read/write 9 data bits over that PC serial port card.
Can anyone help?
The document at http://www.semiconductorstore.com/pdf/newsite/oxford/ox16c950b.pdf describes the differences between various UARTs.
While your StarTech board includes the 16C950, which is RS-485 (and 9-bit) capable, it uses it in RS-232 compatible (550) mode, similar to 16550/8250 from IBM-PC days, and supports max 8 bit data.
You need a board with the same chip (16C950) but that exposes the RS-485 compatible 950 mode that supports 9 bit data as per the spec. And any board claiming such support would have to come with custom drivers for Windows, since Microsoft's is 8 bit only.
There are several other chips that can do 9-bit RS-485 mentioned here but again finding Windows driver support will be tricky. And, of course, many boards use 16C950 but only in 8-bit and/or RS-232 only mode, and without appropriate drivers.
In answer to your related question on Superuser, sawdust suggested the Sealevel 7205e, which looks like a good choice, with Windows driver support. It is pricey but they specifically mention 9-bit, RS-485 support, and Windows drivers. It may well be your best option.
The card you selected is not suitable for this application. It has just plain RS-232 ports, it is not suitable for a multi-drop bus. You'll need to shop elsewhere for an EIA-485 style bus interface, you could only find those at industrial electronics suppliers. By far the best way is to go through the National Automatic Merchandising Association, the industry group that owns the MDB specification.
The 9-bit data format is just a trick and is used in the MDB protocol to mode-switch between address bytes and data bytes. All ports on the bus listen to address bytes, only the addressed port listens to data bytes.
The 9th bit is simply the parity bit that any UART can generate. The fundamental data size is still 8 bits. An UART auto-generates the parity bit from the way it was initialized, you can choose between mark, space, odd and even parity.
Now this is easy to do in a micro-controller that has an UART, the kind of processor used on a bus like this. You simply re-program the UART on-the-fly, telling it to generate mark parity when you send the address bytes. And re-program it again to space parity when you send the data bytes. Waiting for the fifo to empty will typically be necessary although it depends on the actual UART chip.
That is a lot harder to do on a regular Windows or Linux machine, there's a driver between the user mode program and the UART. The driver generates a "transmit buffer empty" status bit, like WaitCommmEvent() for EV_TXEMPTY on Windows, but this doesn't include the fifo empty status, it only indicates that the buffer is empty. A workaround would be to wait for the buffer empty status and then sleep() long enough to ensure that the fifo is emptied. A fifo is typically 16 bytes deep so sleep for 16 times the bit time. You'll need the datasheet for the UART on the card you selected to know these details for sure.
Under Win32 serial ports are just files, so you create a handle for it with CreateFile and then use a DCB structure to set up the configuration options (the members are documented here and include number of data bits as ByteSize).
There's a good walk through here:
http://www.codeproject.com/Articles/3061/Creating-a-Serial-communication-on-Win32
The link provided shows the card supports 9 data bits and Windows 8, so I would presume all the cards features are available to the application through the standard Windows API.
Apart from setting the correct data format in a DCB and opening the port, I would have thought the standard ReadFile would work. I wonder if the data read in would actually be 2*8 bit bytes which represent the 9 data bits, rather than a continuous 9 bits streamed in (which you will need to decode at a later date).
Is the 9th bit used for some purpose other than data?
I have been playing with creating sounds using mathematical wave functions in C. The next step in my project is getting user input from a MIDI keyboard controller in order to modulate the waves to different pitches.
My first notion was that this would be relatively simple and that Linux, being Linux, would allow me to read the raw data stream from my device like I would any other file.
However, research overwhelmingly advises that I write a device driver for the MIDI controller. The general idea is that even though the device file may be present, the kernel will not know what system calls to execute when my application calls functions like read() and write().
Despite these warnings, I did an experiment. I plugged in the MIDI controller and cat'ed the "/dev/midi1" device file. A steady stream of null characters appeared, and when I pressed a key on the MIDI controller several bytes appeared corresponding to the expected Message Chunks that a MIDI device should output. MIDI Protocol Info
So my questions are:
Why does the cat'ed stream behave this way?
Does this mean that there is a plug and play device driver already installed on my system?
Should I still go ahead and write a device driver, or can I get away with reading it like a file?
Thank you in advanced for sharing your wisdom in these areas.
Why does the cat'ed stream behave this way?
Because that is presumably the raw MIDI data that is being received by the controller. The null bytes are probably some sort of sync tick.
Does this mean that there is a plug and play device driver already installed on my system?
Yes.
However, research overwhelmingly advises that I write a device driver for the MIDI controller. The general idea is that even though the device file may be present, the kernel will not know what system calls to execute when my application calls functions like read() and write().
<...>
Should I still go ahead and write a device driver, or can I get away with reading it like a file?
I'm not sure what you're reading or how you're coming to this conclusion, but it's wrong. :) You've already got a perfectly good driver installed for your MIDI controller -- go ahead and use it!
Are you sure you are reading NUL bytes? And not 0xf8 bytes? Because 0xf8 is the MIDI time tick status and is usually sent periodically to keep the instruments in sync. Try reading the device using od:
od -vtx1 /dev/midi1
If you're seeing a bunch of 0xf8, it's okay. If you don't need the tempo information sent by your MIDI controller, either disable it on your controller or ignore those 0xf8 status bytes.
Also, for MIDI, keep in mind that the current MIDI status is usually sent once (to save on bytes) and then the payload bytes follow for as long as needed. For example, the pitch bend status is byte 0xeK (where K is the channel number, i.e. 0 to 15) and its payload is 7 bits of the least significant byte followed by 7 bits of the most significant bytes. Thus, maybe you got a weird controller and you're seeing only repeated payloads of some status, but any controller that's not stupid won't repeat what it doesn't need to.
Now for the driver: have a look at dmesg when you plug in your MIDI controller. Now if your OSS /dev/midi1 appears when you plug in your device (udev is doing this job), and dmesg doesn't shoot any error, you don't need anything else. The MIDI protocol is yet-another-serial-protocol that has a fixed baudrate and transmits/receives bytes. There's nothing complicated about that... just read from or write to the device and you're done.
The only issue is that queuing at some place could result in bad audio latency (if you're using the MIDI commands to control live audio, which I believe is what you're doing). It seems like those devices are mostly made for system exclusive messages, that is, for example, downloading some patch/preset for a synthesizer online and uploading it to the device using MIDI. Latency doesn't really matter in this situation.
Also have a look at the ALSA way of playing with MIDI on Linux.
If you are not developing a new MIDI controller hardware, you shouldn't worry about writing a driver for it. It's the user's concern installing their hardware, and the vendor's obligation to supply the drivers.
Under Linux, you just read the file. Now to interpret and make useful things with the data.