I'm working on a system running embedded linux. I'm trying to obtain the packet timestamp from a stream I'm receiving on a socket.
After creating the socket, I do the following:
if (fd != -1) {
int enabled = 1;
setsockopt(fd, SOL_SOCKET, SO_TIMESTAMP, &enabled, sizeof(enabled);
}
After that I bind the socket, the socket is of type SOCK_STREAM. I successfully receive data on the socket, by calling the function recv(fd, buf, size, 0). Now in order to obtain the timestamp of the received data, I'm currently trying the following:
ret = recv(fd, buf, size, 0);
if (ret > 0) {
struct timeval tv_ioctl;
tv_ioctl.tv_sec = 0;
tv_ioctl.tv_usec = 0;
int error = ioctl(fd, SO_TIMESTAMP, &tv_ioctl);
printf("%ld.%ld - error = %d", (long int)tv_ioctl.tv_sec,
(long int)tv_ioctl.tv_usec, error);
}
The output of the printf statement is always the following:
0.0 error = -1
The error = -1 indicates that the ioctl call has failed. I've performed a test with getsockopt to check if the SO_TIMESTAMP option is set, getsockopt returns 0 for the option SO_TIMESTAMP so it seems correctly set. I'm a bit lost here, how can I further investigate why the ioctl call seems to be failing?
The ioctl to retrieve the most recent timestamp on the socket is SIOCGSTAMP; SO_TIMESTAMP is a socket option, not an ioctl. Your code should read:
int error = ioctl(fd, SIOCGSTAMP, &tv_ioctl);
^^^^^^^^^^
The alternate method to retrieve timestamps is to change recv to recvmmsg and extract the timestamp from the ancillary data. This is more efficient as it involves fewer system calls (Socket reading and timestamps); the ioctl is simpler, though.
Note that SIOCGSTAMP and SO_TIMESTAMP are mutually exclusive - if you're going to use SIOCGSTAMP you should disable SO_TIMESTAMP (with enabled = 0). This is because SO_TIMESTAMP directs the kernel to make the timestamp available via recvmmsg ancillary data instead of via SIOCGSTAMP.
Related
I'm writing my own web server in C. And I'm kind of stuck with an annoying problem.
I'm waiting for incoming connections like this:
struct sockaddr_in caddr;
uint32_t caddr_len = sizeof(caddr);
int fd = accept(sfd, (struct sockaddr *)&caddr, &caddr_len);
if(fd < 0) {
err(EXIT_FAILURE, "accept()");
}
And when accept() succeeded, I'm starting to receive the data with:
errno = 0;
ssize_t r = recv(fd, buf, sizeof(buf), 0);
Sometimes it happens that I don't receive any data, when accessing with firefox.
When I set the timeout to 1s, errno is set to EAGAIN.
And when I set the timeout to 5s, errno will not be set but I still not receiving any data r == 0.
Is it possible to configure the socket so that accept() only returns when there is actual data available?
Note: I do not experience this behavior when accessing with Chrome.
EDIT: Some suggested that I should use poll()
When I use poll(), I have the same problem:
struct pollfd p[] = {{sfd, POLLIN}};
int r = poll(sfd, 1, 1000);
if(r <= 0) err("poll() -> %d", r);
r == 1, but I have still the same problem, because this poll() applies only to the listening socket. It doesn't tell me if there is actual payload when accepting.
Accept returns when the connection is accepted. If you want to wait until data is available to read then you need to use poll or select (or blocking read).
You can use select, poll or similar. See:
https://stackoverflow.com/a/12862015/4885321
How can I retry sending (let's say using a while loop or something similar) in the following code that I have, whenever I have a timeout? I abridged some parts of my code.
I am not familiar with C error codes and error handling so I don't know where to catch/handle the error and what error code to look for.
sock = socket(create socket....)
if (sock < 0 ) {
exit(EXIT_FAILURE);
}
servaddr initializations.....
sendto(sock, etc etc........);
struct timeval timeout;
timeout.tv_sec = 5;
timeout.tv_usec = 0;
if (setsockopt(sock, SOL_SOCKET, SO_RCVTIMEO,&timeout,sizeof(timeout)) < 0) {
perror("Error");
}
addrlen = sizeof(servaddr);
if(recvfrom (sock, etc, etc......) < 0)
{
printf("revfrom failed.\n");
}
From man 7 socket:
SO_RCVTIMEO and SO_SNDTIMEO:
Specify the receiving or sending timeouts until reporting an
error. The argument is a struct timeval. If an input or output function blocks for this period of time, and data has been sent or received, the return value of that function will be the amount of data transferred; if no data has been transferred and the timeout has been reached, then -1 is returned with errno set to EAGAIN or EWOULDBLOCK, or EINPROGRESS (for connect(2)) just as if the socket was specified to be nonblocking. If the timeout is set to zero (the default), then the operation will never timeout. Timeouts only have effect for system calls that perform socket I/O (e.g., read(2), recvmsg(2), send(2), sendmsg(2)); timeouts have no effect for select(2), poll(2), epoll_wait(2), and so on.
So, in your case, the code to keep trying if the timeout is reached would look something like this:
struct timeval timeout = {.tv_sec = 5, .tv_usec = 0};
if (setsockopt(sock, SOL_SOCKET, SO_SNDTIMEO, &timeout, sizeof(timeout)) < 0) {
perror("setsockopt failed");
// Handle error
}
// ...
while (sendto(sock, /*...*/) == -1) {
if (errno != EAGAIN && errno != EWOULDBLOCK) {
// Some unexpected error happened.
perror("sendto failed");
}
// Otherwise it was a timeout, just continue trying.
}
Note that SO_SNDTIMEO is for sending, and SO_RCVTIMEO is for receiving. If you want to set both then do two setsockopt calls.
In any case, it seems to me like you are wasting your time with this. If you want to keep trying until you get data, then just don't bother doing any setsockopt, as the default behavior is to wait indefinitely until data is received:
If the timeout is set to zero (the default), then the operation will never timeout.
I am trying to send and receive raw ethernet frames to include a network device as a media access controller in a simulation environment.
Therefore it is important that the receiving of the packets works through nonblocking statements.
Now the sending of the raw ethernet frames works fine but there's one thing about the receive path that is confusing me:
How do I know where the one frame ends and the other frame begins.
What I fundamentally do is to open a raw socket:
device.socket = socket(AF_PACKET, SOCK_RAW, IPPROTO_RAW);
setting it up as non blocking:
flags = fcntl(s,F_GETFL,0);
assert(flags != -1);
fcntl(s, F_SETFL, flags | O_NONBLOCK);
and then call the recv() function cyclical to get the data from the socket:
length = recv(s, buffer, ETH_FRAME_LEN_MY, 0);
But as far as I know the recv() function only returns the amount of bytes, that is currently availible in the receive buffer and therefore I do not know if another frame starts or if I am still reading the "old" packet.
And because of the fact, that the length of the ethernet frame is not included in the header I can not do this on my own.
Thank you in advance!
If anyone runs into the same problem here's a possible solution:
You can use the libpcap library(in windows winpcap) to open the device as a capture device:
char errbuf[PCAP_ERRBUF_SIZE]; /* error buffer */
Pcap_t *handle; /* packet capture handle */
/* open capture device*/
/* max possible length, not in promiscous mode, no timeout!*/
handle = pcap_open_live(dev, 65536, 0, 0, errbuf);
if (handle == NULL) {
fprintf(stderr, "Couldn't open device %s: %s\n", dev, errbuf);
}
/* set capture device to non blocking*/
if(pcap_setnonblock(pcap_t *p, int nonblock, char *errbuf)){
fprintf("Could not set pcap interface in non blocking mode: %s \n", errbuf);
}
Now you can cyclic call the pcap_dispatch function to receive packet(s):
int pcap_dispatch(pcap_t *p, int cnt, pcap_handler callback, u_char *user);
You have to provide a callback function in which the data is handled.
See https://www.freebsd.org/cgi/man.cgi?query=pcap_dispatch&apropos=0&sektion=3&manpath=FreeBSD+11-current&format=html for further information.
You can send raw ethernet frames by using the inject function:
pcap_inject(pcap,frame,sizeof(frame));
I'm trying to set socket options for a client socket returned from the accept() function. But they are not getting set correctly.
My aim is to time out the client after a particular time of inactivity. But the server should still be able to accept other client connections.
Below is my code where I set the socket option. Can you please suggest what is wrong?
while ((new_sock_fd = accept(socket_fd, (struct sockaddr *) &cli_addr, &clilen)) > 0)
{
if (new_sock_fd < 0)
printf("Accept Error");
else
{
struct timeval timeout;
timeout.tv_sec = 10;
timeout.tv_usec = 0;
if (setsockopt(new_sock_fd, SOL_SOCKET, SO_RCVTIMEO, (char *)&timeout, sizeof(timeout)) < 0)
error("setsockopt failed\n");
if (setsockopt(new_sock_fd, SOL_SOCKET, SO_SNDTIMEO, (char *)&timeout, sizeof(timeout)) < 0)
error("setsockopt failed\n");
pthread_create(&thread, NULL, client_handler, (void *) (intptr_t)new_sock_fd); //intptr_t is big enough to hold the integer prt
}
}
My aim is to time out the client after a particular time of
inactivity. [...] can you please suggest what when wrong..?
I suspect what's wrong is that you're misunderstanding what SO_RCVTIMEO and SO_SNDTIMEO are intended to do. From the man page:
SO_RCVTIMEO and SO_SNDTIMEO Specify the receiving or sending timeouts
until reporting an error. The argument is a struct timeval. If an
input or output function blocks for this period of time, and data has
been sent or received, the return value of that function will be the
amount of data transferred; if no data has been transferred and the
timeout has been reached then -1 is returned with errno set to EAGAIN
or EWOULDBLOCK, or EINPROGRESS (for connect(2)) just as if the socket
was specified to be nonblocking. If the timeout is set to zero (the
default) then the operation will never timeout.
... but it sounds like what you want is for the TCP connection be automatically closed after a certain amount of time with no traffic on the TCP connection, which is not the same thing as forcing a call to send() or recv() to return after a specified amount of time.
If what you are looking for is a mechanism to close an idle TCP connection, you can implement that yourself by recording the current time whenever data is sent or received on the socket. At a later time (e.g. after send() or recv() times out), you can subtract your recorded last-traffic-seen-at-time from the current time; if the difference is greater than your idle-timeout value, call close() on the socket yourself.
I have a multi-threaded server (thread pool) that is handling a large number of requests (up to 500/sec for one node), using 20 threads. There's a listener thread that accepts incoming connections and queues them for the handler threads to process. Once the response is ready, the threads then write out to the client and close the socket. All seemed to be fine until recently, a test client program started hanging randomly after reading the response. After a lot of digging, it seems that the close() from the server is not actually disconnecting the socket. I've added some debugging prints to the code with the file descriptor number and I get this type of output.
Processing request for 21
Writing to 21
Closing 21
The return value of close() is 0, or there would be another debug statement printed. After this output with a client that hangs, lsof is showing an established connection.
SERVER 8160 root 21u IPv4 32754237 TCP localhost:9980->localhost:47530 (ESTABLISHED)
CLIENT 17747 root 12u IPv4 32754228 TCP localhost:47530->localhost:9980 (ESTABLISHED)
It's as if the server never sends the shutdown sequence to the client, and this state hangs until the client is killed, leaving the server side in a close wait state
SERVER 8160 root 21u IPv4 32754237 TCP localhost:9980->localhost:47530 (CLOSE_WAIT)
Also if the client has a timeout specified, it will timeout instead of hanging. I can also manually run
call close(21)
in the server from gdb, and the client will then disconnect. This happens maybe once in 50,000 requests, but might not happen for extended periods.
Linux version: 2.6.21.7-2.fc8xen
Centos version: 5.4 (Final)
socket actions are as follows
SERVER:
int client_socket;
struct sockaddr_in client_addr;
socklen_t client_len = sizeof(client_addr);
while(true) {
client_socket = accept(incoming_socket, (struct sockaddr *)&client_addr, &client_len);
if (client_socket == -1)
continue;
/* insert into queue here for threads to process */
}
Then the thread picks up the socket and builds the response.
/* get client_socket from queue */
/* processing request here */
/* now set to blocking for write; was previously set to non-blocking for reading */
int flags = fcntl(client_socket, F_GETFL);
if (flags < 0)
abort();
if (fcntl(client_socket, F_SETFL, flags|O_NONBLOCK) < 0)
abort();
server_write(client_socket, response_buf, response_length);
server_close(client_socket);
server_write and server_close.
void server_write( int fd, char const *buf, ssize_t len ) {
printf("Writing to %d\n", fd);
while(len > 0) {
ssize_t n = write(fd, buf, len);
if(n <= 0)
return;// I don't really care what error happened, we'll just drop the connection
len -= n;
buf += n;
}
}
void server_close( int fd ) {
for(uint32_t i=0; i<10; i++) {
int n = close(fd);
if(!n) {//closed successfully
return;
}
usleep(100);
}
printf("Close failed for %d\n", fd);
}
CLIENT:
Client side is using libcurl v 7.27.0
CURL *curl = curl_easy_init();
CURLcode res;
curl_easy_setopt( curl, CURLOPT_URL, url);
curl_easy_setopt( curl, CURLOPT_WRITEFUNCTION, write_callback );
curl_easy_setopt( curl, CURLOPT_WRITEDATA, write_tag );
res = curl_easy_perform(curl);
Nothing fancy, just a basic curl connection. Client hangs in tranfer.c (in libcurl) because the socket is not perceived as being closed. It's waiting for more data from the server.
Things I've tried so far:
Shutdown before close
shutdown(fd, SHUT_WR);
char buf[64];
while(read(fd, buf, 64) > 0);
/* then close */
Setting SO_LINGER to close forcibly in 1 second
struct linger l;
l.l_onoff = 1;
l.l_linger = 1;
if (setsockopt(client_socket, SOL_SOCKET, SO_LINGER, &l, sizeof(l)) == -1)
abort();
These have made no difference. Any ideas would be greatly appreciated.
EDIT -- This ended up being a thread-safety issue inside a queue library causing the socket to be handled inappropriately by multiple threads.
Here is some code I've used on many Unix-like systems (e.g SunOS 4, SGI IRIX, HPUX 10.20, CentOS 5, Cygwin) to close a socket:
int getSO_ERROR(int fd) {
int err = 1;
socklen_t len = sizeof err;
if (-1 == getsockopt(fd, SOL_SOCKET, SO_ERROR, (char *)&err, &len))
FatalError("getSO_ERROR");
if (err)
errno = err; // set errno to the socket SO_ERROR
return err;
}
void closeSocket(int fd) { // *not* the Windows closesocket()
if (fd >= 0) {
getSO_ERROR(fd); // first clear any errors, which can cause close to fail
if (shutdown(fd, SHUT_RDWR) < 0) // secondly, terminate the 'reliable' delivery
if (errno != ENOTCONN && errno != EINVAL) // SGI causes EINVAL
Perror("shutdown");
if (close(fd) < 0) // finally call close()
Perror("close");
}
}
But the above does not guarantee that any buffered writes are sent.
Graceful close: It took me about 10 years to figure out how to close a socket. But for another 10 years I just lazily called usleep(20000) for a slight delay to 'ensure' that the write buffer was flushed before the close. This obviously is not very clever, because:
The delay was too long most of the time.
The delay was too short some of the time--maybe!
A signal such SIGCHLD could occur to end usleep() (but I usually called usleep() twice to handle this case--a hack).
There was no indication whether this works. But this is perhaps not important if a) hard resets are perfectly ok, and/or b) you have control over both sides of the link.
But doing a proper flush is surprisingly hard. Using SO_LINGER is apparently not the way to go; see for example:
http://msdn.microsoft.com/en-us/library/ms740481%28v=vs.85%29.aspx
https://www.google.ca/#q=the-ultimate-so_linger-page
And SIOCOUTQ appears to be Linux-specific.
Note shutdown(fd, SHUT_WR) doesn't stop writing, contrary to its name, and maybe contrary to man 2 shutdown.
This code flushSocketBeforeClose() waits until a read of zero bytes, or until the timer expires. The function haveInput() is a simple wrapper for select(2), and is set to block for up to 1/100th of a second.
bool haveInput(int fd, double timeout) {
int status;
fd_set fds;
struct timeval tv;
FD_ZERO(&fds);
FD_SET(fd, &fds);
tv.tv_sec = (long)timeout; // cast needed for C++
tv.tv_usec = (long)((timeout - tv.tv_sec) * 1000000); // 'suseconds_t'
while (1) {
if (!(status = select(fd + 1, &fds, 0, 0, &tv)))
return FALSE;
else if (status > 0 && FD_ISSET(fd, &fds))
return TRUE;
else if (status > 0)
FatalError("I am confused");
else if (errno != EINTR)
FatalError("select"); // tbd EBADF: man page "an error has occurred"
}
}
bool flushSocketBeforeClose(int fd, double timeout) {
const double start = getWallTimeEpoch();
char discard[99];
ASSERT(SHUT_WR == 1);
if (shutdown(fd, 1) != -1)
while (getWallTimeEpoch() < start + timeout)
while (haveInput(fd, 0.01)) // can block for 0.01 secs
if (!read(fd, discard, sizeof discard))
return TRUE; // success!
return FALSE;
}
Example of use:
if (!flushSocketBeforeClose(fd, 2.0)) // can block for 2s
printf("Warning: Cannot gracefully close socket\n");
closeSocket(fd);
In the above, my getWallTimeEpoch() is similar to time(), and Perror() is a wrapper for perror().
Edit: Some comments:
My first admission is a bit embarrassing. The OP and Nemo challenged the need to clear the internal so_error before close, but I cannot now find any reference for this. The system in question was HPUX 10.20. After a failed connect(), just calling close() did not release the file descriptor, because the system wished to deliver an outstanding error to me. But I, like most people, never bothered to check the return value of close. So I eventually ran out of file descriptors (ulimit -n), which finally got my attention.
(very minor point) One commentator objected to the hard-coded numerical arguments to shutdown(), rather than e.g. SHUT_WR for 1. The simplest answer is that Windows uses different #defines/enums e.g. SD_SEND. And many other writers (e.g. Beej) use constants, as do many legacy systems.
Also, I always, always, set FD_CLOEXEC on all my sockets, since in my applications I never want them passed to a child and, more importantly, I don't want a hung child to impact me.
Sample code to set CLOEXEC:
static void setFD_CLOEXEC(int fd) {
int status = fcntl(fd, F_GETFD, 0);
if (status >= 0)
status = fcntl(fd, F_SETFD, status | FD_CLOEXEC);
if (status < 0)
Perror("Error getting/setting socket FD_CLOEXEC flags");
}
Great answer from Joseph Quinsey. I have comments on the haveInput function. Wondering how likely it is that select returns an fd you did not include in your set. This would be a major OS bug IMHO. That's the kind of thing I would check if I wrote unit tests for the select function, not in an ordinary app.
if (!(status = select(fd + 1, &fds, 0, 0, &tv)))
return FALSE;
else if (status > 0 && FD_ISSET(fd, &fds))
return TRUE;
else if (status > 0)
FatalError("I am confused"); // <--- fd unknown to function
My other comment pertains to the handling of EINTR. In theory, you could get stuck in an infinite loop if select kept returning EINTR, as this error lets the loop start over. Given the very short timeout (0.01), it appears highly unlikely to happen. However, I think the appropriate way of dealing with this would be to return errors to the caller (flushSocketBeforeClose). The caller can keep calling haveInput has long as its timeout hasn't expired, and declare failure for other errors.
ADDITION #1
flushSocketBeforeClose will not exit quickly in case of read returning an error. It will keep looping until the timeout expires. You can't rely on the select inside haveInput to anticipate all errors. read has errors of its own (ex: EIO).
while (haveInput(fd, 0.01))
if (!read(fd, discard, sizeof discard)) <-- -1 does not end loop
return TRUE;
This sounds to me like a bug in your Linux distribution.
The GNU C library documentation says:
When you have finished using a socket, you can simply close its file
descriptor with close
Nothing about clearing any error flags or waiting for the data to be flushed or any such thing.
Your code is fine; your O/S has a bug.
include:
#include <unistd.h>
this should help solve the close(); problem