Im working with TCP Sockets in C but yet dont really understand "how far" the delivery of data is ensured.
My main problem is that in my case the server sometimes sends a message to the client and expects an answer shortly after. If the client doesnt answer in time, the server closes the connection.
When reading through the manpages of the recv() function in C, I found the MSG_PEEK Flag which lets me look/peek into the Stream without actually reading the data.
But does the server even care if I read from the stream at all?
Lets say the server "pushes" a series of messages into the stream and a Client should receive them.
As long as the Client doesnt call recv() those messages will stay in the Stream right?
I know about ACK messages being send when receiving data, but is ACK sent when i call the recv() function or is the ACK already sent when the messsage successfully reached its destination and could (emphasising could) be received by the client if it choses to call recv()?
My hope is to trick the server into thinking the message wasnt completely send yet, because the client has not called recv() yet. Therefore the Client could already evaluate the message by using the MSG_PEEK flag and ensure it always answers in time.
Of course I know the timout thing with my server depends on the implementation. My question basically is, if PEEKING lets the server think the message hasnt reached it destination yet or if the server wont even care and when ACK is sent when using recv().
I read the manpages on recv() and wiki on TCP but couldnt really figure out how recv() takes part in the process. I found some similar questions on SO but no answer to my question.
TL;DR
Does the recv() function trigger sending the ACK?
No, not on any regular OS. Possibly on an embedded platform with an inefficient network stack. But it's almost certainly the wrong problem anyway.
Your question about finessing the details of ACK delivery is a whole can of worms. It's an implemention detail, which means it is highly platform-specific. For example, you may be able to modify the delayed ACK timer on some TCP stacks, but that might be a global kernel parameter if it even exists.
However, it's all irrelevant to your actual question. There's almost no chance the server is looking at when the packet was received, because it would need it's own TCP stack to even guess that, and it still wouldn't be reliable (TCP retrans can keep backing off and retrying for minutes). The server is looking at when it sent the data, and you can't affect that.
The closest you could get is if the server uses blocking writes and is single-threaded and you fill the receive window with un-acked data. But that will probably delay the server noticing you're late rather than actually deceiving it.
Just make your processing fast enough to avoid a timeout instead of trying to lie with TCP.
Related
There are many threads regarding how to detect if a socket is connected or not using various methods like getpeername / getsockopt w/ SO_ERROR. https://man7.org/linux/man-pages/man2/getpeername.2.html would be a good way for me to detect if a socket is connected or not. The problem is, it does not say anything about if the connection is in progress... So if i call connect, it is in progress, then i call getpeername, will it say it is an error (-1) even though the connection is still in progress?
If it does, I can implement a counter-like system that will eventually kill the socket if it is still in progress after x seconds.
Short Answer
I think that, if getpeername() returns ENOTCONN, that simply means that the tcp connection request has not yet succeeded. For it to not return ENOTCONN, I think the client end needs to have received the syn+ack from the server and sent its own ack, and the server end needs to have received the client's ack.
Thereafter all bets are off. The connection might subsequently be interrupted, but getpeername() has no way of knowing this has happened.
Long Answer
A lot of it depends on how fussy and short-term one wants to be about knowing if the connection is up.
Strictly Speaking...
Strictly speaking with maximum fussiness, one cannot know. In a packet switched network there is nothing in the network that knows (at any single point in time) for sure that there is a possible connection between peers. It's a "try it and see" thing.
This contrasts to a circuit switched network (e.g. a plain old telephone call), where there is a live circuit for exclusive use between peers (telephones); provided current is flowing, you know the circuit is complete even if the person at the other end of the phone call is silent.
Note that if the two computers were connected by a single Ethernet cable (no router, no switches, just a cable between NICs), that is effectively a fixed circuit (not even a circuit-switched network).
Relaxing a Little...
Focusing on what one can know about a connection in a packet switched network. As others have already said, the answer is that, really, one has to send and receive packets constantly to know if the network can still connect the two peers.
Such an exchange of packets occurs with a tcp socket connect() - the connecting peer sends a special packet to say "please can I connect to you", and the serving peer replies "yes", the client then says "thank you!" (syn->, <-syn+ack, ack->). But thereafter the packets flow between peers only if the applications send and receive data, or elects to close the connection (fin).
Calling something like getpeername() I think is somewhat misleading, depending on your requirements. It's fine, if you trust the network infrastructure and remote computer and its application to not break, and not crash.
It's possible for the connect() to succeed, then something breaks somewhere in the network (e.g. the peer's network connection is unplugged, or the peer crashes), and there is no knowledge at your end of the network that that has happened.
The first thing you can know about it is if you send some traffic and fail to get a response. The response is, initially, the tcp acks (which allows your network stack to clear out some of its buffers), and then possibly an actual message back from the peer application. If you keep sending data out into the void, the network will quite happily route packets as far as it can, but your tcp stack's buffers will fill up due to the lack of acks coming back from the peer. Eventually, your network socket blocks on a call to write(), because the local buffers are full.
Various Options...
If you're writing both applications (server and client), you can write the application to "ping pong" the connection periodically; just send a message that means nothing other than "tell me you heard this". Successful ping-ponging means that, at least within the last few seconds, the connection was OK.
Use a library like ZeroMQ. This library solves many issues with using network connections, and also includes (in modern version) socket heartbeats (i.e. a ping pong). It's neat, because ZeroMQ looks after the messy business of making, restoring and monitoring connections with a heartbeat, and can notify the application whenever the connection state changes. Again, you need to be writing both client and server applications, because ZeroMQ has it's own protocol on top of tcp that is not compatible with just a plain old socket. If you're interested in this approach, the words to look for in the API documentation is socket monitor and ZMQ_HEARTBEAT_IVL;
If, really, only one end needs to know the connection is still available, that can be accomplished by having the other end just sending out "pings". That might fit a situation where you're not writing the software at both ends. For example, a server application might be configured (rather than re-written) to stream out data regardless of whether the client wants it or not, and the client ignores most of it. However, the client knows that if it is receiving data it then also knows there is a connection. The server does not know (it's just blindly sending out data, up until its writes() eventually block), but may not need to know.
Ping ponging is also good in that it gives some indication of the performance of the network. If one end is expecting a pong within 5 seconds of sending a ping but doesn't get it, that indicates that all is not as expected (even if packets are eventually turning up).
This allows discrimination between networks that are usefully working, and networks that are delivering packets but too slowly to be useful. The latter is still technically "connected" and is probably represented as connected by other tests (e.g. calling getpeername()), but it may as well not be.
Limited Local Knowledge...
There is limited things one can do locally to a peer. A peer can know whether its connection to the network exists (e.g. the NIC reports a live connection), but that's about it.
My Opinion
Personally speaking, I default to ZeroMQ these days if at all possible. Even if it means a software re-write, that's not so bad as it seems. This is because one is generally replacing code such as connect() with zmq_connect(), and recv() with zmq_revc(), etc. There's often a lot of code removal too. ZeroMQ is message orientated, a tcp socket is stream orientated. Quite a lot of applications have to adapt tcp into a message orientation, and ZeroMQ replaces all the code that does that.
ZeroMQ is also well supported across numerous languages, either in bindings and / or re-implementations.
man connect
If the initiating socket is connection-mode, .... If the connection cannot be established immediately and O_NONBLOCK is not set for the file descriptor for the socket, connect() shall block for up to an unspecified timeout interval until the connection is established. If the timeout interval expires before the connection is established, connect() shall fail and the connection attempt shall be aborted.
If connect() is interrupted by a signal that is caught while blocked waiting to establish a connection, connect() shall fail and set errno to [EINTR], but the connection request shall not be aborted, and the connection shall be established asynchronously.
If the connection cannot be established immediately and O_NONBLOCK is set for the file descriptor for the socket, connect() shall fail and set errno to [EINPROGRESS], but the connection request shall not be aborted, and the connection shall be established asynchronously.
When the connection has been established asynchronously, select() and poll() shall indicate that the file descriptor for the socket is ready for writing.
If the socket is in blocking mode, connect will block while the connection is in progress. After connect returns, you'll know if a connection has been established (or not).
A signal could interrupt the (blocking/waiting) process, the connection routine will then switch to asynchronous mode.
If the socket is in non blocking mode (O_NONBLOCK) and the connection cannot be established immediately, connect will fail with the error EINPROGRESS and like above switching to asynchronous mode, that means, you'll have to use select or poll to figure out if the socket is ready for writing (indicates established connection).
I'm writing a C/S program and both the client and server may send data to peer (without explicit ack) at arbitrary time. I'm wondering if it could possibly deadlock if the client and server coincidentally write to the peer at the same time.
So does returning from write() mean that the peer application has already read() the data? Or it only means the peer's kernel has got the data and would deliver to the app on next read()?
(EJP's answer fixed my totally wrong understanding about write()/send()/.... To add some authoritative info I found this in the POSIX standard about send:
Successful completion of a call to send() does not guarantee delivery of the message. A return value of -1 indicates only locally-detected errors.
Linux's man page about send() is not very clear:
No indication of failure to deliver is implicit in a send(). Locally detected errors are indicated by a return value of -1.
Or it's because I cannot fully understand the first sentence as a non native English speaker. )
I'm wondering if it could possibly deadlock if the client and server coincidentally write to the peer at the same time.
It can't, unless one or both of the peers is very slow reading and has closed its receive window. TCP is full-duplex.
So does returning from write() mean that the peer application has already read() the data?
No.
Or it only means the peer's kernel has got the data and would deliver to the app on next read()?
No.
It means the data has reached your kernel, and is queued for transmission.
the returning from write() means the TCP ACK has already been received.
No it doesn't.
u mean returning from write() only means the data has reached the sender's kernel?
That is not only what I meant, it is what I said.
I'd think the sender's already received the TCP ACK so reached the peer's kernel.
No.
No. If you think of it as a data pipe, returning from write means that your data has entered the pipe, not that it has exited the pipe at the other end.
In fact, since the pipe is one where the data may take any of hundreds of different pathways, you're not even guaranteed that it will reach the other end :-) If that happens, you'll be notified about it at some later date, probably by a subsequent write failing.
It may be blocked:
trying to exit your machine due to a broken cable,
at a bottleneck in the path somewhere,
by the networking stack at the destination,
by a networking stack at some device in the networking path, more intelligent than a simple hub,
because the application at the other end is otherwise tied up,
and so on.
A successful return from write means that your local network stack has accepted your data and will process it in due course.
In an application I'm currently working on, I need to stop downloading some file if I realize it's not what I'm looking for. The protocol doesn't provide any way to know it before I start receiving the file (like headers or so).
As an example, in some cases I might be looking for a file of exactly X bytes in size, but after I have downloaded X bytes and I keep getting more bytes, this is not the file I'm looking for as its size is greater than X. In this case, I want to stop downloading to free network bandwidth resources. The protocol doesn't provide any way to notify the server about this.
I read somewhere that close(fd) or shutdown(fd, SHUT_RD) won't actually stop downloading as the server will continue to send() the file and this will continue to consume network bandwidth. I am also not sure about if I just stop calling recv() and packets still arrive, will they fill the buffer and then start to be discarded? If it matters, the protocol used is based on TCP (but I would like some solution that can also be used for UDP based protocols).
I became even more doubtful that stopping calls to recv() would solve it, after I searched for programmatic bandwidth control (sleep(), token bucket..) as an alternative solution (reduce download speed to about zero after I realize it's not the file I'm looking for). How can I control network bandwidth usage by reducing recv() calls if the server will still be send()ing? I didn't catch it.
Main idea is to entirely stop the download.
What would you suggest?
I read somewhere that close(fd) or shutdown(fd, SHUT_RD) won't actually stop download as server will continue to send() file and this will continue to consume network bandwidth.
If you shutdown(fd, SHUT_RD) your own recv() will unblock with a return code of zero, which will cause the code to close the socket, which will cause the localhost to issue an RST if any more data comes from the peer, which will cause an ECONNRESET at the sender (after a few more send() calls, not necessarily immediately).
Where did you read this nonsense?
I became even more doubtful about if stop calling recv() would solve it
It won't solve it, but it will eventually stop the sender from sending, because of TCP flow control. It isn't a solution to this problem.
Context: I'm developing a client-server application that is fairly solid most of the time, despite frequent network problems, outages, broken pipes, and so on. I use non-blocking sockets, select(), and OpenSSL to deliver messages between one or more nodes in a cluster, contingent on application-level heartbeats. Messages are queued and not removed from the queue until the entire message has been transferred and all the SSL_write()s return successfully. I maintain two sockets for each relationship, one incoming and one outgoing. I do this for a reason, and that's because it's much easier to detect a failed connection (very frequent) on a write than it is on a read. If a client is connecting, and I already have a connection, I replace it. Basically, the client performing the write is responsible for detecting errors and initiating a new connection (which will then replace the existing (dead) read connection on the server). This has worked well for me with one exception.
Alas, I'm losing messages. 99.9% of the time, the messages go through fine. But every now and then, I'll send, and I have no errors detected on either side for a few minutes... and then I'll get an error on the socket. The problem is that SSL_write has already returned successfully.
Let me guess: if I was blocking this would be fine, but since I'm non-blocking, I don't wait for the read on my remote end. As long as my TCP buffer can fit more, I keep stuffing things in the pipe. And when my socket goes poof, I lose anything in that buffer yet to be delivered?
How can I deal with this? Are application-level acks really necessary? (I'd rather not travel down the long road of complicated lost-acks and duplicate message complexity) Is there an elegant way to know what message I've lost? Or is there a way I can delay removal from my queue until I know it has been delivered? (Without an ack, how?)
Thanks for any help in advance.
Is there a way to get send() to wait until all the data that has been sent has been ACK-ed (or return -1 if the timeout for an ACK has been reached), or is there some other mechanism to wait for the ACK after the send() but before doing something else?
I am using the standard Unix Berkeley sockets API.
I know I could implement an application-layer ACK, but I'd rather not do that when TCP's ACK serves the purpose perfectly well.
AFAIK there is no way.
Also, it wouldn't be reliable, the ACK means only that the kernel received the data, in the meantime the client or its machine could have crashed. You would think the client received the data, but actually it never processed it.
Unfortunately standard API doesn't reserve any appropriate way to do this. There could be a way to query the current TCP send window size/usage, but unfortunately it may not be queried by the standard means.
Of course there are tricky ways to achieve what you want. For instance on Windows one may create a network filter driver to monitor packet-level trafic.
I seem to have found a solution. At least on Linux, if you set SO_SNDBUF to a value of 0, it seems to wait for every transaction before allowing the next transfer through. While it will immediately return, it will not proceed to allowing another send to succeed until the previous send has sent. I haven't tried using select(...) to determine if the data has been sent.
This works on my Linux 3.8 kernel, and am confident it works elsewhere.
You can write a wrapper for send(). I have answered a question similar to this in another thread.