First of all: I'm not absolutely certain that this is the right place to ask, but I think the question fits here better then on superuser or serverfault, since it is a question from a programmer's perspective: I figured more programmers might have had the same question (although I couldn't find this specific question!).
I would like to have a feature in my program which allows users to send files to a 'friend'. You can find friends via an username: this all goes via a server which can provide the IP-adress of a friend.
I wanted to use a tcp connection to send the file. This becomes difficult, however, when one (or both) of the parties is behind a NAT. What is the best way to solve this? I heard that it's possible to send stuff via a server, but I'd rather send everything directly, to prevent server overhead.
I heard about a technique called hole punching, but also that it's pretty complex to implement and not 100% reliable. I could use UDP and implement some scheme to improve the reliability, but this seems a bit complex to me. I know skype, bittorrent and a whole lot of other programs do similiar things (but I don't know about the specifics, which protocol they use, if they use hole punching etc.).
I looked into FTP a bit, until I realised that this is just a protocol using TCP, so I should use TCP hole punching in order to let this work... Anyway, I hope someone can give me some advice on this :)
If you don't want to make data pass through a server, I'm not aware of other methods other than TCP Hole Punching or simple Port forwarding of a previously choosen port.
Related
This may not be in the right location, so tell me and I'll move it.
I am a recent EE grad and I was hired to build a system that exists on a SoC with a simple 32-bit processor. The system basically monitors several external devices and performs some DSP on it, and then is supposed to send the results using a WiFi device (in my case I have the ESP8266 using UDP) to an email server for logging/notification.
I have been trying to find a library that I can use, but my uC can only program in C and I have it set up for UDP, and everything is in C++ using some other protocol, or something else completely.
I am great at DSP, decent at SoC's and uC's, but when it come to this email server communication thing I am at a loss.
I have successfully configured everything for the sensors, the datapath, the DSP, and connected the system to my WiFi via UDP, but I have yet to figure out how to send data to any servers.
Could someone help me understand how I should go about this?
I have looked into some simple SMTP commands such as HELO, MAIL, RCPT, DATA, etc. but I cannot understand how I actually should implement them in my code.
When I send out the WiFi data via UDP what type of data do I send and how do I format it? Do I need to send any other kind of flags? How should I expect the response? I also know the data has to be transformed into base 64 which is confusing me further.
I am also not super familiar with UDP to begin with, I have been using libraries that are part of the SoC's default library to connect to my WiFi.
I know these may either seem like obvious or stupid questions but it is were I no longer have any knowledge, and everything I find online doesn't make sense, or doesn't attempt to explain it, just gives a pre-made solution
I have found the RFC2821 but it doesn't get any clearer.
I know that's a lot but any help at all would be a lifesaver!
Since you are asking this question, I'm assuming that you are not booting and running an OS suitable for micro-controllers such as an embedded variant of Linux or such. If you were, you would simply be able to take advantage of possibly built in applications or other existing code.
But you don't mention having written an Ethernet stack, so are you using some other library or operating environment which might have some of the functionality needed for an implementation of SMTP?
If you don't and really do need to write your own SMTP client to run directly on the processor you are using, then you should be able to find plenty of examples of source code for this. A quick google search of How To Write an SMTP client showed a few articles with some example code. One article seems to be an exact hit, but you need to look at it further.
However, I would highly suggest just sitting down with a telnet client and connect to an SMTP server you are allowed to use and try the commands you need to just send a message. If you only need to send text, you don't need to get involved in MIME encoding or anything like that.
I am working on a project involving a microcontroller communicating to a PC via Modbus over TCP. My platform is an STM32F4 chip, programming in C with no RTOS. I looked around and found LwIP and Freemodbus and have had pretty good success getting them both to work. Unfortunately, I'm now running into some issues which I'm not sure how to handle.
I've noticed that if I establish connection, then lose connection (by unplugging the Ethernet cable) I will not be able to reconnect (once I've plugged back in, of course). Freemodbus only allows one client and still has the first client registered. Any new clients trying to connect are ignored. It won't drop the first client until after a specific timeout period which, as far as I can tell, is a TCP/IP standard.
My thoughts are...
I need a Modbus module that will handle multiple clients. The new client request after communication loss will be accepted and the first client will eventually be dropped due to the timeout.
How do I modify Freemodbus to handle this? Are there examples out there? I've looked into doing it myself and it appears to be a decently sized project.
Are there any good Modbus packages out there that handle multiple clients, are not too expensive, and easy to use? I've seen several threads about various options, but I'm not sure any of them meet exactly what I need. I've had a hard time finding any on my own. Most don't support TCP and the ones that do only support one client. Is it generally a bad idea to support multiple clients?
Is something wrong with how I connect to the microcontroller from my PC?
Why is the PC changing ports every time it tries to reconnect? If it kept the same port it used before, this wouldn't be a problem
Should I drop the client from Freemodbus as soon as I stop communicating?
This seems to go against standards but might work.
I'm leaning towards 1. Especially since I'm going to need to support multiple connections eventually anyways. Any help would be appreciated.
Thanks.
If you have a limit on the number of modbus clients then dropping old connections when a new one arrives is actually suggested in the modbus implementation guide (https://www.modbus.org/docs/Modbus_Messaging_Implementation_Guide_V1_0b.pdf)
Nevertheless a mechanism must be implemented in case of exceeding the number of
authorized connection. In such a case we recommend to close the oldest unused
connection.
It has its own problems but everything is a compromise.
Regarding supporting multiple clients...if you think about modbus/rs server - it could only ever have one master at a time. Then replace the serial cable with TCP and you see why it's not uncommon to only support one client (and of course it's easier to program). It is annoying though.
Depending on what you are doing you wont need the whole modbus protocol and implementing the parts you do need is pretty easy. Of course if you have to support absolutely everything its a different prospect. I haven't used freemodbus, or any other library appropriate to your setup, so I can't help with suggestions there.
Regarding the PC using different TCP source port each time - that is how TCP is supposed to work and no fault on your side. If it did reuse the same source port then it wouldn't help you because e.g. sequence numbers would be wrong.
Regarding dropping clients. You are allowed to drop clients though its better not to. Some clients will send a modbus command, notice the connection has failed, reconnect, but not reissue the command. That may be their problem but still nicer to not see it that often where possible. Of course things like battery life might make the calculation different.
I want to write an Application layer protocol that uses TCP to return certain ASCII Text when a GET request is sent. I read the first HTTP specification and the SMTP specification but am still unclear on how to write the protocol connecting the two computers itself. How could I write this in C? Are there any tutorials or examples I could look at?
The heart of any communication protocol is the interface control document(ICD), which will describe the message structures that are allowed, like what is the size of your header, data, crc field etc. It is from this document you create the C structures. Usually people use bit fields to encapsulate the message fields appropriately. When you use existing communication methods, for example Ethernet you have the TCP or UDP sockets to send and receive the data. You can encode your messages in them. If you want to develop a new communication protocol then you have to make a logic of your own and embed it over existing media and proceed.
You're asking two questions. Your first question is "How can I create a new communications protocol", and your second question is "How can I implement this in C".
These are both far too generic to be good questions per the charter of this forum.
The answer to "How can I create a new communications protocol" is, as millimoose already pointed out, simple: A protocol is a document specifying the set of rules for how entities can communicate. Decide what a conversation should look like, starting from the "hello" or equivalent, specifying every possible request and every possible response, and every possible error response, through to how to say goodbye (and how to deal with a connection that gets dropped with saying goodbye), and write that all down. The SMTP protocol is actually a fairly good example of exactly that, in fact. (A TCP-based internet protocol will also typically specify a default TCP port to operate the protocol over.)
The answer to "How can I implement this in C", now that you have a fully specified protocol, is the same as the answer to "How can I implement this in Java", "How can I implement this in REBOL", or "How can I implement this in TCL": Write a basic server app that speaks the server half of the protocol and a basic client app that speaks the client half of the protocol.
(Of course, you might actually have been intending to ask "Regardless of the specific protocol, how can I write in C a server and client that communicate with each other?". This is also an excessively generic question, which can be answered through judicious searching on google.)
In practice, it is much better to use an existing TCP stack (e.g. tcp(7) socket implementation, at least on Linux), then use some HTTP server library above it, like e.g. libonion. Reinventing your TCP stack and your HTTP server layer would take you more than a year of work.
TL;DR available at the bottom
I've been trying to figure out a way to get two laptops (both running Ubuntu) to be able to pass basic messages back and forth without the need for them to be connected via a wireless network,either by an AP or ad-hoc. I want to reiterate here that ad-hoc networking is not what I'm looking for, I've seen many similar questions here with that as the answer.
I guess what I'm asking is: how do I achieve this? All I really need is for one computer to be able to send a packet, and then for another to pick it up via a packet sniffer of some kind.
Currently: I have both laptops in monitor mode (via a mon0 interface created from aircrack-ng's airmon-ng)so that they can sniff nearby traffic (with Wireshark, tcpdump,tcpcump.org's sample libpcap code, and opening a raw socket and just printing out all the packets. I tried each just because I thought one could be doing something differently/leaving something out). I also have a very basic program that consists of opening a raw socket to send crafted ethernet frames out to the air, but I can't get my two machines to see the other's packets. The sniffer running on each machine can only see the packets going out of that machine (in addition to nearby beacons/control traffic from wifi in the area).
Some things to note that might be important are:
-the packets I'm sending out appear in Wireshark (only on the sending machine) as malformed 802.11 packets (probably because I'm just filling them with junk data for now). I was under the impression that my other laptop would also see them as malformed packets, but it gets nothing
-the sockets I'm using are from a call to socket(PF_PACKET,SOCK_RAW,ETH_P_ALL). Raw sockets are something I just recently was aware of, so I could be misunderstanding how they work, but my impression is that I can craft a layer 2 packet by hand and ship out straight out to the wire/air.
If you're curious as to why I want to do something like this, it's part curiosity, part research for a project I'm working on. I want to streamline / automate the process of setting up an ad-hoc network, and what I'm trying to do here is for the laptops to do a small exchange to figure out the specifics of the adhoc network they are about to create and then make/join that network automatically, instead of either one person explicitly setting up the network OR having both people pre-decide the name, etc of the network and have both computers constantly trying to connect to that specific one.
I'm more interested if I'm going about this process in the right way rather than if my code works or not, if someone thinks me posting my (very basic, taken from another post on Stack Overflow) raw socket code will help, I can.
Edit: I am more than happy to post a complete set of code with instructions if I can get this working. I couldn't find much helpful info on this topic on the internet, and I'd love to put it up for future people trying to do the same thing.
TL;DR I want to send out a packet from one laptop and pick it up on another via a packent sniffer of some sort. No wifi network or ad-hoc network involved. Something akin to spoofing an AP's beacon frame (or similar) for the purpose of sending small amounts of data.
Edit 2:After some thought, perhaps what I'm looking for is some kind of raw 802.11 use? Having direct control of the wifi radio? Is such a thing possible?
I found out I was able to send packets out through my monitor mode interface as long as I had correct 802.11 with radiotap headers. I think the problem I was originally experiencing (not being able to sniff the packets) was because they were malformed and thus not actually getting sent out.
I was able to accomplish this by adapting the example code found here, courtesy of someone named Evan Jones, except I did not need to use an Atheros based card or Madwifi drivers, everything worked fine with the mon0 interface created with aircrack-ng.
I am certain that Apple Mac do this. Apple call it 'bonjour'. There may well be a proper IETF spec for it. This is an Article on Bonjour this is Wikipedia on an open component of bonjour which might help get you moving.
I was designing a file server using socket programming in C. I send the calls like open(), write() etc as plain strings using stream sockets and decipher it at the server end.i.e if it is a open call then we extract path, mode, flags. Is it ok or should I be using some kind of struct to store the file system calls and send to server where the server simply accesses the fields.
Is there some standard way i don't know?
Thanks
You're basically starting to define your own protocol. It would be a lot easier if you sent numbers describing operations instead of strings.
If you're serious about this you might want to look into RPC - RFC707 (you did ask for a standard way, right?).
Yes, there is a standard way. Look into NFS, AFP, CIFS, and WebDAV, then pick one.
You already have answers for the standard way, so I'll give you a few caveats you should look out for.
If you intend to deploy your file server in an un-trusted environment (e.g. on the Internet) think about securing it right away. Securing it is not just a question of slapping encryption on - you need to know how you intend to authenticate your users, how you want to authorize different types of access to different parts of the server, how you will insure the authenticity and the integrity of the data and how you intend to keep the data confidential.
You'll also need to think about your server's availability. That means that you should be fault-tolerant - i.e. connections can (and will) break (regardless of whether they're being broken on purpose or not) so you need to detect that, either will some kind of keep-alive (which will fail if the client left) or with some kind of activity time-out (which will expire if the client left). You also need to think about how many clients you are willing to support simultaneously - which can radically change the architecture of your server.
As for the open, close, read, write etc. commands, most file transfer protocols don't go into so much detail, but it may be interesting to be able to do so depending on your situation. If your files are huge and you only need some chunks of it, or if you want to be able to lock files to work on them exclusively, etc. you may want to go into such detail. If you don't have those requirements, simpler, transactional commands such as get & put (rather than open, read, read, read, close and open, write, write some more, close) may both be easier to implement and easier to work with.
If you want a human being to interact with your server and give it commands, text is a good approach: it's easy to debug when sniffing and humans understand text and can type is easily. If there are no humans involved, using integers as commands is probably a better approach: you can structure your command to start with an integer followed by a number of parameters and always simply expect the same thing on the server's end (and do a switch on the command you receive). Even in that case, though, it may be a good idea to have human-readable values in your integers. For example, putting 'READ' in an integer as the read command uses as many bytes as 0x00000001, but is easier to read when sniffed with WireShark.
Finally, you really take a look at the standard approaches and try to understand the trade-offs made in each case. Ask yourself, for example, why HTTP has such verbose headers and why WebDAV uses it. Why does FTP use two connections (one for commands and one for data) while many other protocols use only one? How did NFS evolve to where it is now, and why? Understanding the answers to these questions will help you develop your own protocol - if after you understand those answers, you still feel you need your own protocol.