I am debugging a c based linux socket program. As all the examples available in websites,
I applied the following structure:
sockfd= socket(AF_INET, SOCK_STREAM, 0);
connect(sockfd, (struct sockaddr *) &serv_addr, sizeof(serv_addr));
send_bytes = send(sockfd, sock_buff, (size_t)buff_bytes, MSG_DONTWAIT);
I can detect the disconnection when the remove server closes its server program. But if I unplug the ethernet cable, the send function still return positive values rather than -1.
How can I check the network connection in a client program assuming that I can not change server side?
But if I unplug the ethernet cable, the send function still return
positive values rather than -1.
First of all you should know send doesn't actually send anything, it's just a memory-copying function/system call. It copies data from your process to the kernel - sometime later the kernel will fetch that data and send it to the other side after packaging it in segments and packets. Therefore send can only return an error if:
The socket is invalid (for example bogus file descriptor)
The connection is clearly invalid, for example it hasn't been established or has already been terminated in some way (FIN, RST, timeout - see below)
There's no more room to copy the data
The main point is that send doesn't send anything and therefore its return code doesn't tell you anything about data actually reaching the other side.
Back to your question, when TCP sends data it expects a valid acknowledgement in a reasonable amount of time. If it doesn't get one, it resends. How often does it resend ? Each TCP stack does things differently, but the norm is to use exponential backoffs. That is, first wait 1 second, then 2, then 4 and so on. On some stacks this process can take minutes.
The main point is that in the case of an interruption TCP will declare a connection dead only after a seriously large period of silence (on Linux it does something like 15 retries - more than 5 minutes).
One way to solve this is to implement some acknowledgement mechanism in your application. You could for example send a request to the server "reply within 5 seconds or I'll declare this connection dead" and then recv with a timeout.
To detect a remote-disconnect, do a read()
Check this thread for more info:
Can read() function on a connected socket return zero bytes?
You can't detect the unplugged ethernet cable only with calling write() funcation.
That's because of tcp retransmission acted by tcp stack without your consciousness.
Here are solutions.
Even though you already set keepalive option to your application socket, you can't detect in time the dead connection state of the socket, in case of your app keeps writing on the socket.
That's because of tcp retransmission by the kernel tcp stack.
tcp_retries1 and tcp_retries2 are kernel parameters for configuring tcp retransmission timeout.
It's hard to predict precise time of retransmission timeout because it's calculated by RTT mechanism.
You can see this computation in rfc793. (3.7. Data Communication)
https://www.rfc-editor.org/rfc/rfc793.txt
Each platforms have kernel configurations for tcp retransmission.
Linux : tcp_retries1, tcp_retries2 : (exist in /proc/sys/net/ipv4)
http://linux.die.net/man/7/tcp
HPUX : tcp_ip_notify_interval, tcp_ip_abort_interval
http://www.hpuxtips.es/?q=node/53
AIX : rto_low, rto_high, rto_length, rto_limit
http://www-903.ibm.com/kr/event/download/200804_324_swma/socket.pdf
You should set lower value for tcp_retries2 (default 15) if you want to early detect dead connection, but it's not precise time as I already said.
In addition, currently you can't set those values only for single socket. Those are global kernel parameters.
There was some trial to apply tcp retransmission socket option for single socket(http://patchwork.ozlabs.org/patch/55236/), but I don't think it was applied into kernel mainline. I can't find those options definition in system header files.
For reference, you can monitor your keepalive socket option through 'netstat --timers' like below.
https://stackoverflow.com/questions/34914278
netstat -c --timer | grep "192.0.0.1:43245 192.0.68.1:49742"
tcp 0 0 192.0.0.1:43245 192.0.68.1:49742 ESTABLISHED keepalive (1.92/0/0)
tcp 0 0 192.0.0.1:43245 192.0.68.1:49742 ESTABLISHED keepalive (0.71/0/0)
tcp 0 0 192.0.0.1:43245 192.0.68.1:49742 ESTABLISHED keepalive (9.46/0/1)
tcp 0 0 192.0.0.1:43245 192.0.68.1:49742 ESTABLISHED keepalive (8.30/0/1)
tcp 0 0 192.0.0.1:43245 192.0.68.1:49742 ESTABLISHED keepalive (7.14/0/1)
tcp 0 0 192.0.0.1:43245 192.0.68.1:49742 ESTABLISHED keepalive (5.98/0/1)
tcp 0 0 192.0.0.1:43245 192.0.68.1:49742 ESTABLISHED keepalive (4.82/0/1)
In addition, when keepalive timeout ocurrs, you can meet different return events depending on platforms you use, so you must not decide dead connection status only by return events.
For example, HP returns POLLERR event and AIX returns just POLLIN event when keepalive timeout occurs.
You will meet ETIMEDOUT error in recv() call at that time.
In recent kernel version(since 2.6.37), you can use TCP_USER_TIMEOUT option will work well. This option can be used for single socket.
Finally, you can use read function with MSG_PEEK flag, which can let you check that the socket is okay. (MSG_PEEK just peeks if data arrived at kernel stack buffer and never copies the data into user buffer.)
So you can use this flag just for checking socket is okay without any side effect.
Check the return value, and see if it's equal to this value:
EPIPE
This socket was connected but the connection is now broken. In this case, send generates a SIGPIPE signal first; if that signal is ignored or blocked, or if its handler returns, then send fails with EPIPE.
Also add a check for the SIGPIPE signal in your handler, to make it be more controllable.
Related
In recv()'s man page I found that return value can be zero in such case:
Datagram sockets in various domains (e.g., the UNIX and Internet
domains) permit zero-length datagrams. When such a datagram is
received, the return value is 0.
When or why should one use zero-length datagrams in UNIX socket intercommunication? What's its purpose?
One such use is to unblock a recvfrom() call when you wish to close a UDP service thread - set a 'terminate' flag and send the zero-length datagram on the localhost stack.
One example I stumbled upon yesterday while researching the answer to another question.
In the old RFC 868 protocol for getting the current time of a remote server, the workflow for using UDP looks like:
When used via UDP the time service works as follows:
S: Listen on port 37 (45 octal).
U: Send an empty datagram to port 37.
S: Receive the empty datagram.
S: Send a datagram containing the time as a 32 bit binary number.
U: Receive the time datagram.
The server listens for a datagram on port 37. When a datagram
arrives, the server returns a datagram containing the 32-bit time
value. If the server is unable to determine the time at its site, it
should discard the arriving datagram and make no reply.
In this case, the server has to receive a datagram to be alerted that a user is requesting the time (And to know what address to send a reply to), due to UDP's connectionless nature (The TCP version just needs to connect to the server). The contents of that datagram are ignored, so might as well specify that it should be empty.
I am doing some test with TCP client application in a Raspberry Pi (server in the PC), with PPP (Point to Point Protocol) using a LTE Modem. I have used C program with sockets, checking system call's response. I wanted to test how socket works in a bad coverage area so I did some test removing the antenna.
I have followed the next steps:
Connect to server --> OK
Start sending data (write system call) --> OK (I also check in the server)
I removed the LTE modem's antenna (There is no network, it can't do ping)
Continue sending data (write system call) --> OK (server does not receive anything!!!)
It finished sending data and closed socket --> OK (connection is still opened and there is no data since the antenna was removed)
Program was finished
I put the antenna again
Some time later, the data has been uploaded and the connection closed. But I did another test following this steps but with more data, and it did not upload this data...
I do not know if there any way to ensure that the data written to TCP server is received by the server (I thought that TCP layer ensured this..). I could do it manually using an ACK but I guess that it has to be a better way to do.
Sending part code:
while(i<100)
{
sprintf(buf, "Message %d\n", i);
Return = write(Sock_Fd, buf, strlen(buf));
if(Return!=strlen(buf))
{
printf("Error sending data to TCP server. \n");
printf("Error str: %s \n", strerror(errno));
}
else
{
printf("write successful %d\n", i);
i++;
}
sleep(2);
}
Many thanks for your help.
The write()-syscall returns true, since the kernel buffers the data and puts it in the out-queue of the socket. It is removed from this queue when the data was sent and acked from the peer. When the OutQueue is full, the write-syscall will block.
To determine, if data has not been acked by the peer, you have to look at the size of the outqueue. With linux, you can use an ioctl() for this:
ioctl(fd, SIOCOUTQ, &outqlen);
However, it would be more clean and portable to use an inband method for determining if the data has been received.
TCP/IP is rather primitive technology. Internet may sound newish, but this is really antique stuff. TCP is needed because IP gives almost no guarantees, but TCP doesn't actually add that many guarantees. Its chief function is to turn a packet protocol into a stream protocol. That means TCP guarantees a byte order; no bytes will arrive out of order. Don't count on more than that.
You see that protocols on top of TCP add extra checks. E.g. HTTP has the famous HTTP error codes, precisely because it can't rely on the error state from TCP. You probably have to do the same - or you can consider implementing your service as a HTTP service. "RESTful" refers to an API design methodology which closely follows the HTTP philosophy; this might be relevant to you.
The short answer to your 4th and 5th topics was taken as a shortcut from this answer (read the whole answer to get more info)
A socket has a send buffer and if a call to the send() function succeeds, it does not mean that the requested data has actually really been sent out, it only means the data has been added to the send buffer. For UDP sockets, the data is usually sent pretty soon, if not immediately, but for TCP sockets, there can be a relatively long delay between adding data to the send buffer and having the TCP implementation really send that data. As a result, when you close a TCP socket, there may still be pending data in the send buffer, which has not been sent yet but your code considers it as sent, since the send() call succeeded. If the TCP implementation was closing the socket immediately on your request, all of this data would be lost and your code wouldn't even know about that. TCP is said to be a reliable protocol and losing data just like that is not very reliable. That's why a socket that still has data to send will go into a state called TIME_WAIT when you close it. In that state it will wait until all pending data has been successfully sent or until a timeout is hit, in which case the socket is closed forcefully.
The amount of time the kernel will wait before it closes the socket,
regardless if it still has pending send data or not, is called the
Linger Time.
BTW: that answer also refers to the docs where you can see more detailed info
I'm writing a non forking server, using poll() for multiple simultaneous connections. It works properly, except I have a problem with how to detect timeout the right way.
Let's say I have the following code:
#define POLL_SIZE 512
struct pollfd poll_set[POLL_SIZE];
timeout = 60000; // 60 secs
// setup server_sockfd with socket(), bind(), listen(), ...
poll_set[0].fd = server_sockfd;
poll_set[0].events = POLLIN;
numfds = 1;
while(1) {
rc = poll(poll_set, numfds, timeout);
if(rc == 0){
// handle timeout
}
for(fd_index = 0; fd_index < numfds; fd_index++) {
if(poll_set[fd_index].revents & POLLIN) {
// accept new connection or handle established connections
}
}
}
Let's assume, I have 15 clients connected, 14 clients are sending and receiving data, however one client is silent, no data to or from, ie. just occupying a socket on the server.
Now, the problem is that poll() can't spot this one specific client, because all the other 14 clients are providing data, so poll() says, it's ok.
How would you solve this problem by detecting this silent client, and close its connection?
Currently, I have nothing better, then create a time_t lastseen[POLL_SIZE] array, and keep track of the timestamp of the given connection when either data is read from the client or sent to client.
Then I use an alarm signal in every 60 seconds, and run through the lastseen array, compare their timestamp with the current timestamp, and tear down every connection being idle > 60 seconds.
Or perhaps a thread could do the same to avoid signaling. What do you suggest to solve the problem?
(Note that I experimented with libevent, and it's very nice. However, I had to abandon it, because I couldn't find support to add SSL/TLS to an already connected socket. Think of STARTTLS)
Detecting of errors related to socket is not poll's job. All it does it indicates whether one or more sockets are ready for read write operations. If error occurs with any awaited socket then poll marks that socket as ready (really it marked by OS) and POLLERR flag is indicated in revents field.
What about timeout. In general timeout is not transport layer error (and therefore is not tracked by sockets). You need to track it by yourself. For example you can remember timestamp of last read from socket (See clock_gettime(CLOCK_MONOTONIC, ...)) and set timeout in poll to minimum of all timeouts related to that sockets. After timeout expired you need to check whether it expired for the each socket or no.
Also consider use epoll - it is much faster for large number of sockets in one poll. And also for selection of nearest timeout you can use Heap data structure. So you can manage all sockets with O(log n) execution time.
I wrote a program on C , where a client sent one time some information to a server. I used TCP sockets.And some time the server calculated and should to sent result to the client. How can I detect if connection on The server or the client was broken?
You may want to try TCP keepalives.
# cat /proc/sys/net/ipv4/tcp_keepalive_time
7200
# cat /proc/sys/net/ipv4/tcp_keepalive_intvl
75
# cat /proc/sys/net/ipv4/tcp_keepalive_probes
9`
In the above example, TCP keep-alive timer kicks in after the idle time of 7200 seconds. If the keep-alive messages are unsuccessful then they are retried at the interval of 75 seconds. After 9 successive retry failure, the connection will be brought down.
The keepalive time can be modified at boot time by placing startup script at /etc/init.d.
There is a way to detect, on Linux, dead sockets without reading or writing to them:
Get the numeric (uint) file descriptor from the socket handler.
readlink the file /proc/[pid]/fd/[#hander]. If it's a socket it will return a string like socket:[#inode].
Read /proc/net/tcp, search for the line with that inode (11th column).
Read the status (st) column on that line (4th column). If it's 0x07 (Close) or 0x08 (TIME_WAIT), the socket is dead.
The only way I know for a program to know for certain that a TCP connection is dead is to attempt to send something on it. The attempt will either time-out or return with an error condition. Thus, the program doesn't need to do anything special -- just send the stuff it was designed to send. It does need to handle, however, all possible error-conditions. On time-out, it could retry for a limited time or decide that the connection is dead. The latter case is appropriate if sending the same data multiple times would be harmful. After this or an error condition, the program should close the current connection and, if appropriate, re-establish it.
TCP Keep-Alive is a reliable way to determine if the peer is dead.That is if the peer application exited without doing a proper closure of the open TCP connection.
http://tldp.org/HOWTO/TCP-Keepalive-HOWTO/overview.html
Lookout for how to enable tcp keep-alives(SO_KEEPALIVE) per socket using setsockopt call.
Another method is that the client and the server application agree on heartbeats at regular intervals. Non arrival of heartbeats should indicate that the peer is dead.
I have worked in non blocking TCP, in that both read and write can fail in non blocking case. TCP non blocking read can fail if there is no data available and TCP write can fail if peer side`s TCP buffer is full (I hope TCP buffer size is 64K).
Similarly UDP read (recvfrom) can fail if no data available. But what is the failure case for UDP write (sendto). I think in UDP write there will not be any non block error. Because TCP write sends data and wait for the ACK from other side. But this is not the case for UDP write it will just send and comes out and it doesnt wait for any ACK from peer side. If its not send to other side means its packet loss.
Whether my understanding of UDP non blocking write is correct ? Please explain ?
The most likely reason why a UDP non-blocking send would fail is that the UDP socket's in-kernel outgoing-data buffer is full. In this case, send()/sendto() would return -1 and errno would be set to EWOULDBLOCK.
Note that a non-blocking send()/sendto() doesn't actually send the data out the network device before it returns; rather it copies the data into an in-kernel buffer and returns immediately, and thereafter it is the kernel's responsibility to move that data out to the network as quickly as it can. The outgoing-data buffer can become full if your program tries to send a lot of data at once, because the CPU can add your new data to the buffer much faster than the network hardware can forward the buffer's data out to the network.
If you get a -1/EWOULDBLOCK error, usually the most graceful way to handle it is to stop trying to send on that socket until the socket select()'s (or poll()'s, or etc) as ready-for-write. When that happens, you know that the in-kernel buffer has been at least partially drained, and you can try the send()/sendto() call again.
Another (less likely) cause of an error from send() would be if the IP address you are trying to send to is invalid. In any case, you should check errno and find out what the errno value is, as that will give you better insight into what is going wrong.
Btw the behavior described above is not unique to UDP, either; you can and will have the same problem with a non-blocking TCP socket (even if the remote peer's receive window is not full) if you try to send() data on the socket faster than the local network card can drain the socket's in-kernel buffer.
Because TCP write sends data and wait for the ACK from other side.
No it doesn't. It copies your data into the socket send buffer, and if that is full it either blocks or returns -1/EWOULDBLOCK/EAGAIN.
But this is not the case for UDP write it will just send and comes out and it doesnt wait for any ACK from peer side.
No it doesn't. It copies your data into the socket send buffer, and if that is full it either blocks or returns -1/EWOULDBLOCK/EAGAIN.
In both cases the actual putting of bytes onto the wire is asynchronous to your program.