Once in a while my server accept functions just stop working properly anymore.
There is a much deeper story behind this, I'm being flooded with SYN and SYN/ACK packets, my network router goes disco and accept keeps returning ECONNABORTED.... I already tried to debug and fix this specific attack, but without success. By now I gave up and rather look for a more generic server recover solution.
Anyway I figured out that simpy "restarting" the server socket by closing and calling socket again is helping. Theoretically very simple, but practically I'm facing here a huge challenge because (a) the server is quite complex by now and (b) when should I exactly restart the server socket.
My setup is one accept-thread that calls accept and feeds epoll, one listener-thread that listens for epoll read/write etc. events and feeds a queue of a thread pool.
I have not found any literature that guides one through restarting the server socket.
Particularly:
When do I actually restart the server socket? I mean I do not really know if a ECONNABORTED return value from accept is just a aborted connection or the accept/filedescriptor is going banana.
How does closing the server socket affect epoll and connected clients? Should I close the server socket immediately or rather have a buffer time such that all clients have finished first?
Or is it even best to have two alternating server sockets such that if one goes banana I just try the other one.
I am making some assumptions about the things you say in your question all being true and accurate even though some of them seems like they may be misdiagnosed. Unfortunately, you didn't really explain how you reached the conclusions presented, so I really can't do much other than assume they're true.
For example, you don't explain how or why you figured that closing and calling socket again will help. From just the information you gave, I would strongly suspect the opposite is true. But again, without knowing the evidence and rationale that lead you to figure that, all I can do is assume it's true despite my instinct and experience saying it's wrong.
When do I actually restart the server socket? I mean I do not really know if a ECONNABORTED return value from accept is just a aborted connection or the accept/filedescriptor is going banana.
If it really is the case that accepting connections will recover faster from a restart than without one and you really can't get any connections through, keep track of the last successful connection and the number of failures since the last successful connection. If, for example, you've gone 120 seconds or more without a successful connection and had at least four failed connections since the last successful one, then close and re-open. You may need to tune those parameters.
How does closing the server socket affect epoll and connected clients?
It has no effect on them unless you're using epoll on the server socket itself. In that case, make sure to remove it from the set before closing it.
Should I close the server socket immediately or rather have a buffer time such that all clients have finished first?
I would suggest "draining" the socket by calling accept without blocking until it returns EWOULDBLOCK. Then you can close it. If you get any legitimate connections in that process, don't close it since it's obviously still working.
A client that tries to get in between your close and getting around to calling listen on a new socket might get an error. But if they're getting errors anyway, that should be acceptable.
Or is it even best to have two alternating server sockets such that if one goes banana I just try the other one.
A long time ago, port DoS attacks were common because built-in defenses to things like SYN-bombs weren't as good as they are now. In those days, it was common for a server to support several different ports and for clients to try the ports in rotation. This is why IRC servers often accepted connections on ranges of ports such as 6660-6669. That meant an attacker had to do ten times as much work to make all the ports unusable. These days, it's pretty rare for an attack to take out a specific inbound port so the practice has largely gone away. But if you are facing an attack that can take out specific listening ports, it might make sense to open more listening ports.
Or you could work harder to understand the attack and figure out why you are having a problem that virtually nobody else is having.
Related
I have a program that needs to:
Handle 20 connections. My program will act as client in every connection, each client connecting to a different server.
Once connected my client should send a request to the server every second and wait for a response. If no request is sent within 9 seconds, the server will time out the client.
It is unacceptable for one connection to cause problems for the rest of the connections.
I do not have access to threads and I do not have access to non-blocking sockets. I have a single-threaded program with blocking sockets.
Edit: The reason I cannot use threads and non blocking sockets is that I am on a non-standard system. I have a single RTOS(Real-Time Operating System) task available.
To solve this, use of select is necessary but I am not sure if it is sufficient.
Initially I connect to all clients. But select can only be used to see if a read or write will block or not, not if a connect will.
So when I have connected to say 2 clients and they are all waiting to be served, what if the 3rd does not work, the connection will block causing the first 2 connections to time out as well.
Can this be solved?
I think the connection-issue can be solved by setting a timeout for the connect-operation, so that it will fail fast enough. Of course that will limit you if the network really is working, but you have a very long (slow) path to some of the server(s). That's bad design, but your requirements are pretty harsh.
See this answer for details on connection-timeouts.
It seems you need to isolate the connections. Well, if you cannot use threads you can always resort to good-old-processes.
Spawn each client by forking your server process and use traditional IPC mechanisms if communication between them is required.
If you can neither use a multiprocess approach I'm afraid you'll have a hard time doing that.
I am working on a project involving a microcontroller communicating to a PC via Modbus over TCP. My platform is an STM32F4 chip, programming in C with no RTOS. I looked around and found LwIP and Freemodbus and have had pretty good success getting them both to work. Unfortunately, I'm now running into some issues which I'm not sure how to handle.
I've noticed that if I establish connection, then lose connection (by unplugging the Ethernet cable) I will not be able to reconnect (once I've plugged back in, of course). Freemodbus only allows one client and still has the first client registered. Any new clients trying to connect are ignored. It won't drop the first client until after a specific timeout period which, as far as I can tell, is a TCP/IP standard.
My thoughts are...
I need a Modbus module that will handle multiple clients. The new client request after communication loss will be accepted and the first client will eventually be dropped due to the timeout.
How do I modify Freemodbus to handle this? Are there examples out there? I've looked into doing it myself and it appears to be a decently sized project.
Are there any good Modbus packages out there that handle multiple clients, are not too expensive, and easy to use? I've seen several threads about various options, but I'm not sure any of them meet exactly what I need. I've had a hard time finding any on my own. Most don't support TCP and the ones that do only support one client. Is it generally a bad idea to support multiple clients?
Is something wrong with how I connect to the microcontroller from my PC?
Why is the PC changing ports every time it tries to reconnect? If it kept the same port it used before, this wouldn't be a problem
Should I drop the client from Freemodbus as soon as I stop communicating?
This seems to go against standards but might work.
I'm leaning towards 1. Especially since I'm going to need to support multiple connections eventually anyways. Any help would be appreciated.
Thanks.
If you have a limit on the number of modbus clients then dropping old connections when a new one arrives is actually suggested in the modbus implementation guide (https://www.modbus.org/docs/Modbus_Messaging_Implementation_Guide_V1_0b.pdf)
Nevertheless a mechanism must be implemented in case of exceeding the number of
authorized connection. In such a case we recommend to close the oldest unused
connection.
It has its own problems but everything is a compromise.
Regarding supporting multiple clients...if you think about modbus/rs server - it could only ever have one master at a time. Then replace the serial cable with TCP and you see why it's not uncommon to only support one client (and of course it's easier to program). It is annoying though.
Depending on what you are doing you wont need the whole modbus protocol and implementing the parts you do need is pretty easy. Of course if you have to support absolutely everything its a different prospect. I haven't used freemodbus, or any other library appropriate to your setup, so I can't help with suggestions there.
Regarding the PC using different TCP source port each time - that is how TCP is supposed to work and no fault on your side. If it did reuse the same source port then it wouldn't help you because e.g. sequence numbers would be wrong.
Regarding dropping clients. You are allowed to drop clients though its better not to. Some clients will send a modbus command, notice the connection has failed, reconnect, but not reissue the command. That may be their problem but still nicer to not see it that often where possible. Of course things like battery life might make the calculation different.
C Language TCP server/client.. I want to assign a new socket for a particular client which requested my server from 8080 lets say the new socket is 8081 to get further request, and want to free the previous socket(8080) so that the other clients will request my server from 8080. is there any way of doing it in C language. (OS Ubuntu) Thanks
Your problem statement is incorrect. You can't do this even if you wanted to. The way that TCP sockets work is that accept() gives you a new socket for the incoming client connection, on the same port you are listening to. That's all you need and it's all you can get. You can't 'allocate a new socket' to the client on a new port without engaging in another TCP handshake with him, which would be nothing but a complete waste of time when you already have a connection to him. This does not preclude another connection bring accepted while this one is open. You need to read a TCP Sockets networking tutorial.
Mat and EJP have said the pertinent things above, but I thought it might help others to describe the situation more verbosely.
A TCP/IP connection is identified by a four-tuple: target IP address, target TCP port number, source IP address, and source TCP port number. The kernel will keep track of established connections based on these four things. A single server port (and IP address) can be connected to thousands of clients at the same time, limited in practice only by the resources available.
When you have a listening TCP socket, it is bound to some IP address (or wildcard address) and TCP port. Such a socket does not receive data, only new connections. When accept() is called, the server notes the new four-tuple of the connection, and hands off the file descriptor that represents that connection (as the accept() return value). The original socket is free to accept new connections. Heck, you can even have more than one thread accepting new connections if you want to, although establishing new connections in Linux is so fast you shouldn't bother; it's just too insignificant to worry about.
If establishing the connection at application level is resource-intensive -- this is true for for example encrypted connections, where agreeing to an encryption scheme and preparing the data structures needed takes typically several orders of magnitude more CPU resources than a simple TCP connection --, then it is natural to wish to avoid that overhead. Let's assume this is the point in OP's question: to avoid unnecessary application-level connection establishment when a recent client needs another connection.
The preferred solution is connection multiplexing. Simply put, the application-level protocol is designed to allow multiple data streams via a single TCP connection.
The OP noted that it would be necessary/preferable to keep the existing application protocol intact, i.e. that the optimization should be completely on the server side, transparent to the clients.
This turns the recommended solution to a completely new direction. We should not talk about application protocols, but how to efficiently implement the existing one.
Before we get to that, let's take a small detour.
Technically, it is possible to use the kernel packet filtering facilities to modify incoming packets to use a different port based on the source IP address, redirecting requests from specific IP addresses to separate ports, and making those separate ports otherwise inaccessible. Technically possible, but quite complex to implement, and with very questionable benefits.
So, let's ignore the direction OP assumed would bring the desired benefits, and look at the alternatives. Or, actually, the common approach used.
Structurally, your application has
- A piece of code accepting new connections
- A piece of code establishing the application-level resources needed for that connection
- A piece of code doing the communication with the client (serving the response to the client, per the client's request)
There is no reason for these three pieces to be consecutive, or even part of the same code flow. Use data structures to your advantage.
Instead of treating new incoming connections (accept()ed) as equal, they can be simply thrown into separate pools based on their source IP addresses. (Or, if you are up to it, have a data structure which clusters source IP addresses together, but otherwise keeps them in the order they were received.)
Whenever a worker completes a request by a client, it checks if that same client has new incoming connections. If yes, it can avoid most if not all of the application-level connection establishment by checking that the new connection matches the application-level parameters of the old one. (You see, it is possible that even if the source IP address is the same, it could be a completely different client, for example if the clients are under the same VPN or NATted subnet.)
There are quite a few warts to take care of, for example how to keep the priorities, and avoid starving new IP addresses if known clients try to hog the service.
For protocols like HTTP, where the client sends the request information as soon as the server accepts the connection, there is an even better pattern to apply: instead of connection pools, have request pools. A single thread or a thread pool can receive the requests (they may span multiple packets in most protocols), without acting on them; only detecting when the request itself is complete. (A careful server will limit the number of pending requests, and the number of incomplete request, to avoid vulnerability to DOS.)
When the requests are complete, they are grouped, so that the same "worker" who serves one request, can serve another similar request with minimal overhead. Again, some careful thought is needed to avoid the situation where a prolific client hogs the server resources by sending a lot of requests, but it's nothing some careful thought and testing won't resolve.
One question remains:
Do you need to do this?
I'd wager you do not. Apache, which is one of the best HTTP servers, does not do any of the above. The performance benefits are not considered worth the extra code complexity. Could you write a new HTTP server (or a server for whatever protocol you're working with), and use a scheme similar to above, to make sure you can use your hardware as efficiently as possible? Sure. You don't even need to be a wizard, just do some research and careful planning, and avoid getting caught in minute details, keeping the big picture in mind at all times.
I firmly believe that code maintainability and security is more important than efficiency, especially when writing an initial implementation. The information gained from the first implementation has thus far always changed how I perceive the actual "problem"; similar to opening new eyes. It has always been worth it to create a robust, easy to develop and maintain, but not necessarily terribly efficient implementation, for the first generation. If there is someone willing to support the development of the next generation, you not only have the first generation implementation to compare (and verify and debug) against, but also all the practical knowledge gained.
That is also the reason old hands warn so often against premature optimization. In short, you end up optimizing resource waste and personal pain, not the implementation you're developing.
If I may, I'd recommend the OP back up a few steps, and actually describe what they intend to implement, what the observed problem with the implementation is, and suggestions on how to fix and avoid the problem. The current question is like asking how to better freeze a banana, as it keeps shattering when you hammer nails with it.
Context: I'm developing a client-server application that is fairly solid most of the time, despite frequent network problems, outages, broken pipes, and so on. I use non-blocking sockets, select(), and OpenSSL to deliver messages between one or more nodes in a cluster, contingent on application-level heartbeats. Messages are queued and not removed from the queue until the entire message has been transferred and all the SSL_write()s return successfully. I maintain two sockets for each relationship, one incoming and one outgoing. I do this for a reason, and that's because it's much easier to detect a failed connection (very frequent) on a write than it is on a read. If a client is connecting, and I already have a connection, I replace it. Basically, the client performing the write is responsible for detecting errors and initiating a new connection (which will then replace the existing (dead) read connection on the server). This has worked well for me with one exception.
Alas, I'm losing messages. 99.9% of the time, the messages go through fine. But every now and then, I'll send, and I have no errors detected on either side for a few minutes... and then I'll get an error on the socket. The problem is that SSL_write has already returned successfully.
Let me guess: if I was blocking this would be fine, but since I'm non-blocking, I don't wait for the read on my remote end. As long as my TCP buffer can fit more, I keep stuffing things in the pipe. And when my socket goes poof, I lose anything in that buffer yet to be delivered?
How can I deal with this? Are application-level acks really necessary? (I'd rather not travel down the long road of complicated lost-acks and duplicate message complexity) Is there an elegant way to know what message I've lost? Or is there a way I can delay removal from my queue until I know it has been delivered? (Without an ack, how?)
Thanks for any help in advance.
I have some code that establishes a connection to servers. For a while, my code runs normally and everything works fine.
At some point though, when trying to connect out, my application sends a SYN flag , gets a SYN/ACK, and then will start sending a FIN flag which terminates the connection!
This is using FreeBSD 9. I have checked all of the limits, and as far as I can tell, I am not exceeding any open socket limits or anything, and I would have not exected the socket to open to even send the SYN flag if it was something like that going on.
What else can I do to debug this? After it happens for one outbound connection, it starts happening for all of them, so I think it must be some kind of systemic problem.
It's more likely to be a coding whoops than some complicated, sinister networking issue. I agree with netcoder, check your calls, check your return values. Check that you're not doing something daft, like eating file descriptors! Check your firewalls at both ends, I've seen that effect with a firewall getting overly protective before. Or post some code for us to look at...