I wonder about the behaviour of socket on blocking and nonblocking actions. What happens to threads blocking on socket when the socket blocking mode changes ? Here is the scenario;
thread1(T1) creates a UDP socket and
fd = socket(AF_INET , SOCK_DGRAM, 0);
T1 waiting(sleeping) for receive
recv(fd, buf , sizeof(buf) , 0);
and thread2(T2) changes socket mode to non-blocking before socket receive any data
fcntl(fd, F_SETFL, O_NONBLOCK);
what happens to T1 ? Is it signalled/waked because the socket is no more blocking ?
The behavior is literally unspecified: fcntl is not required to unblock any threads.
Linux just sets the flag in the file description struct file and returns without unblocking any blocked threads.
A thread already blocked in recv can be scheduled to run only if:
data to read becomes available;
or an error condition on the file descriptor is detected (FIN/RST, socket read timeout, TCP keep-alive failure, the file descriptor is closed by another thread);
or a signal is received and the signal disposition does not include SA_RESTART;
or it is pthread_cancelled.
The fact that you are trying to change flags of a file descriptor of another thread suggests that your design requires a review. Ideally, threads must not share any data and not poke at each other's state, rather they should use message passing to communicate with each other.
You made me curious. First, it's quite obvious that since there is no standard that specifies that the socket should wake up, it will not be awoken because it would be quite a pain to implement this (since the non-blocking flag is in a different layer than where the socket is blocking). So we can say quite confidently that the socket will not wake up until we receive a packet. Or will it?
#include <pthread.h>
#include <signal.h>
#include <stdio.h>
#include <err.h>
#include <sys/types.h>
#include <netinet/in.h>
#include <string.h>
#include <time.h>
#include <fcntl.h>
int sock;
static void
sighand(int s)
{
write(1, "sig\n", 4);
}
static void *
rcv(void *v)
{
struct sigaction sa;
memset(&sa, 0, sizeof(sa));
sa.sa_handler = sighand;
sa.sa_flags = SA_RESTART;
if (sigaction(SIGUSR1, &sa, NULL) == -1)
err(1, "sigaction");
char buf[64];
ssize_t sz = recv(sock, buf, sizeof(buf), 0);
printf("recv %d\n", (int)sz);
return NULL;
}
pthread_t t1, t2;
static void *
beeper(void *v)
{
for (;;) {
nanosleep(&((struct timespec){.tv_sec = 1}), NULL);
if (pthread_kill(t1, SIGUSR1))
errx(1, "pthread_kill");
printf("beep\n");
}
}
int
main(int argc, char **argv)
{
if ((sock = socket(PF_INET, SOCK_DGRAM, 0)) == -1)
err(1, "socket");
struct sockaddr_in sin;
memset(&sin, 0, sizeof(sin));
sin.sin_family = AF_INET;
sin.sin_port = htons(4711);
sin.sin_addr.s_addr = htonl(INADDR_ANY);
if (bind(sock, (struct sockaddr *)&sin, sizeof(sin)) == -1)
err(1, "bind");
if (pthread_create(&t1, NULL, rcv, NULL))
errx(1, "pthread_create");
if (pthread_create(&t2, NULL, beeper, NULL))
errx(1, "pthread_create");
/* pretend that this is correct synchronization. */
nanosleep(&((struct timespec){.tv_sec = 3}), NULL);
printf("setting non-block\n");
if (fcntl(sock, F_SETFL, O_NONBLOCK) == -1)
err(1, "fcntl");
printf("set\n");
nanosleep(&((struct timespec){.tv_sec = 3}), NULL);
return 0;
}
The code above (sorry, couldn't make it shorter). Blocks the thread in recv, waits a bit and then sets non-blocking on the file descriptor. As expected nothing happens. But then I added a twist. The receiving thread has a signal handler that wakes up once a second with SA_RESTART. Of course, since we have SA_RESTART recv should not wake up.
On OSX this will "misbehave" if we consider any behavior correct. I'm pretty sure this will behave the same way on all BSDs. Same on Linux. In fact, the way SA_RESTART is usually implemented I'm pretty sure this will "misbehave" pretty much everywhere.
Funny. This of course doesn't mean that the code above is useful for anything, but it's an interesting curiosity. To answer your question, this is unspecified, but in most cases it won't wake up, unless it will. Please don't do weird things like this.
I think POSIX specification is pretty clear about this:
If no messages are available at the socket and O_NONBLOCK is not set on the socket's file descriptor, recv() shall block until a message arrives. If no messages are available at the socket and O_NONBLOCK is set on the socket's file descriptor, recv() shall fail and set errno to [EAGAIN] or [EWOULDBLOCK].
You call recv when O_NONBLOCK is not yet set, to it should block until message arrives (not until mode changes).
Unblocked thread (didn't block yet due to reading when it was in blocking mode) will behave as if it was always a non-blocking mode.
Reading recv(2) manual page
If no messages are available at the socket, and if the socket is nonblocking, the value -1 is returned and the external
variable errno is set to EAGAIN or EWOULDBLOCK.
Blocked threads (blocked while reading when in blocking mode) before changing to non-blocking.
As pointed by #Maxim sharing the code of the function which doesn't awaken threads, The blocked threads will only be awaken after a write is done (data is available).
Related
I'm building an application that sends requests to a server and I need to implement a way to control how long I try to connect() to the server.
I'd like to end the connect() function after ten seconds. I've read about select() and non-blocking sockets, but I don't completely understand how it works.
Could someone give me a a very simple example of how this could be done? Using C. Thanks.
Also, I'd like to add the same timeouts to the send and recv functions as well.
#include <stdio.h>
#include <string.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <stdlib.h>
#include <arpa/inet.h>
#include <fcntl.h>
#include <errno.h>
int main(void) {
struct sockaddr_in server;
int sockfd;
sockfd = socket(AF_INET, SOCK_STREAM, 0);
server.sin_addr.s_addr = inet_addr("10.0.0.1");
server.sin_family = AF_INET;
server.sin_port = htons(atoi("80"));
fcntl(sockfd, F_SETFL, O_NONBLOCK);
struct timeval tv;
fd_set writefds;
tv.tv_sec = 5;
tv.tv_usec = 500000;
FD_ZERO(&writefds);
FD_SET(sockfd, &writefds);
connect(sockfd, (struct sockaddr *)&server, sizeof(server));
if (errno == EINPROGRESS)
{
printf("In progress");
}
select(sockfd+1, NULL, &writefds, NULL, &tv);
Can you explain why the if condition for errno does not show until after the timeout? That doesn't make sense to me. And is there any additional code I should look into here to handle this correctly. Thanks.
Rather than write the actual code, I'll give you an outline. (I'm assuming Linux or something similar.)
Create the socket.
Call fcntl on the socket fd and set the
O_NONBLOCK option to make the socket nonblocking. (You may also be
able to do this by giving the SOCK_NONBLOCK option on the socket
call.)
Call connect. It will return immediately. It will most
likely "fail" with the error EINPROGRESS, which means that the
connect operation is in progress.
Call select or poll to wait
for the socket fd to become writable. Both of those calls have a
timeout option.
If the socket becomes writable (i.e., your select or poll
doesn't time out), check the connection status by calling
getsockopt to get the value of SO_ERROR, which will indicate
whether the connect succeeded or failed by returning an error code
or 0 on success.
You can use the same technique to time out sends and recvs as well (although for receive you need to wait for the socket to become readable).
See the following man pages for more information: socket(2), connect(2), select(2), poll(2), socket(7), getsockopt(2).
I have no problems with running the TCP Server and I like the fact that it's in blocking to avoid useless loops and sleeping code and useless cpu cycles.
The problem happens when shutting it down in Linux environment, it stays on, until the connected user sends something, then it turns off.
I figured it's because it's blocking even though the endless while loop is set to exit. But when it's blocking changing the socket id's to NON_BLOCKING doesn't help at all, most likely has to be set to NON_BLOCKING before the block occurs.
#include <pthread.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <stdio.h>
#include <errno.h>
#include <fcntl.h> /* Added for the nonblocking socket */
#define LISTEN_MAX 10 /* Maximum clients that can queue up */
#define LISTEN_PORT 32000
#define MAX_COMMANDS_AT_ONCE 4000
#define NANO_SECOND_MULTIPLIER 1000000 // 1 millisecond = 1,000,000 Nanoseconds
//Global so I can access these where I shut the threads off.
int listenfd, connfd; //sockets that must be set to non-blocking before exiting
int needQuit(pthread_mutex_t *mtx)
{
switch(pthread_mutex_trylock(mtx)) {
case 0: /* if we got the lock, unlock and return 1 (true) */
pthread_mutex_unlock(mtx);
return 1;
case EBUSY: /* return 0 (false) if the mutex was locked */
return 0;
}
return 1;
}
/* this is run on it's own thread */
void *tcplistener(void *arg)
{
pthread_mutex_t *mx = arg;
//keyboard event.
SDLKey key_used;
struct timespec ts;
//int listenfd,connfd,
int n,i, ans;
struct sockaddr_in servaddr,cliaddr;
socklen_t clilen;
pid_t childpid;
char mesg[MAX_COMMANDS_AT_ONCE];
listenfd=socket(AF_INET,SOCK_STREAM,0);
bzero(&servaddr,sizeof(servaddr));
servaddr.sin_family = AF_INET;
servaddr.sin_addr.s_addr=htonl(INADDR_ANY);
servaddr.sin_port=htons(LISTEN_PORT);
int option = 1;
if(setsockopt(listenfd, SOL_SOCKET,SO_REUSEADDR,(char*)&option,sizeof(option)) < 0)
{
printf("setsockopt failed\n");
close(listenfd);
}
bind(listenfd,(struct sockaddr *)&servaddr,sizeof(servaddr));
listen(listenfd,LISTEN_MAX);
while( !needQuit(mx) )
{
clilen=sizeof(cliaddr);
connfd = accept(listenfd,(struct sockaddr *)&cliaddr,&clilen);
while( !needQuit(mx) )
{
n = recv(connfd,mesg,MAX_COMMANDS_AT_ONCE,0);
if(n == 0 || n == -1) break;
//...Do Stuff here with mesg...
}
}
close(connfd);
}
close(connfd);
close(listenfd);
return NULL;
}
int main(int argc, char *argv[])
{
/* this variable is our reference to the thread */
pthread_t tcp_listener_thread;
pthread_mutex_t mxq; /* mutex used as quit flag */
/* init and lock the mutex before creating the thread. As long as the
mutex stays locked, the thread should keep running. A pointer to the
mutex is passed as the argument to the thread function. */
pthread_mutex_init(&mxq,NULL);
pthread_mutex_lock(&mxq);
/* create a hread which executes tcplistener(&x) */
if(pthread_create(&tcp_listener_thread, NULL, tcplistener, &mxq)) {
fprintf(stderr, "Error creating TCP Listener thread\n");
//clear thread for tcp listener on exit.
/* unlock mxq to tell the thread to terminate, then join the thread */
fcntl(listenfd, F_SETFL, O_NONBLOCK); /* Change the socket into non-blocking state */
fcntl(connfd, F_SETFL, O_NONBLOCK); /* Change the socket into non-blocking state */
pthread_mutex_unlock(&mxq);
pthread_join(tcp_listener_thread,NULL);
pthread_cancel(tcp_listener_thread);
pthread_exit(NULL);
return 0;
}
//End of the TCP Listener thread.
// Waits 500 milliseconds before shutting down
struct timespec ts;
ts.tv_sec = 0;
ts.tv_nsec = 500 * NANO_SECOND_MULTIPLIER;
nanosleep((&ts, NULL);
//Forces a shutdown of the program and thread.
//clear thread for tcp listener on exit.
/* unlock mxq to tell the thread to terminate, then join the thread */
fcntl(listenfd, F_SETFL, O_NONBLOCK); /* Change the socket into non-blocking state */
fcntl(connfd, F_SETFL, O_NONBLOCK); /* Change the socket into non-blocking state */
pthread_mutex_unlock(&mxq);
pthread_join(tcp_listener_thread,NULL);
pthread_cancel(tcp_listener_thread);
pthread_exit(NULL);
return 0;
}
I tried the fix EJP suggested like so, still hanging..
I've made connfd and listernfd both global scope
pthread_mutex_unlock(&mxq);
close(connfd); //<- this
close(listenfd); //<-- this
pthread_join(tcp_listener_thread,NULL);
pthread_cancel(tcp_listener_thread);
pthread_exit(NULL);
To unblock accept(), just close the listening socket. Make sure the code around accept() handles is correctly.
To unblock recv(), shutdown the receiving socket for input. That will cause recv() to return zero, which again must be handled correctly. Or else just close the socket as above, which might be better if you want the receive code to know that you're closing the application.
Your listener thread will indeed block in the accept().
The nasty way to fix this (almost) is to send a signal to the listener thread with pthread_kill(). This causes accept() to return with errno == EINTR, which you test for and then return.
However, that has a race condition: if the signal is received between testing the while (!needQuit(mx)) condition and entering the accept() then it'll be lost and the accept() will block again.
One correct way to solve this is to use something like select() and a pipe. You select for read over the pipe and the socket. When the main thread wants the listener thread to exit it writes a byte to the pipe. The listener thread's select() call returns either when a byte is readable from the pipe (in which case it exits) and/or when a client can be accepted.
Non-blocking sockets are primarily used to multiplex lots of sockets into one event loop (i.e. thread). That's a good idea for server scalability, but not necessary here.
I have C linux TCP Client/Server application.
I came up with a strange scenario, but I don't know if there are any consequences with this application.
I have a server side that can accept N connections, for example this server will accept 100 connections.
In this scenario I create the listen socket in the main thread, then I create 100 threads and each thread has an independent accept() and a select() iomux, also each thread can only accept one connection.
My concerns here, if two simultaneous accept() want to accept the same socket(connection) because of the select is a ready to read on the same socket, I don't know if the simultaneous accepts are thread safe in kernel and only one accept can handle that incoming connection and the other will wait for another connection?
I tried that on my RedHat machine that works fine, but I don't know If I am a lucky enough to avoid a deadlock!
Thanks
rc = bind(sd, (struct sockaddr_in *)& groupSock, sizeof(struct sockaddr_in));
CHECK_VALUE("Bind address error", rc, 0, goto cleanup);
rc = listen(sd, 10);
CHECK_VALUE("listen", rc, 0, goto cleanup);
for(; count< num_socks; count++){
par_data[count].sd = sd;
par_data[count].thread_num = count;
par_data[count].err_chk = -1;
rc = pthread_create(&thread_id[count], NULL, accept_sock_thread, (void *)& par_data[count]);
CHECK_VALUE("pthread_create", rc, 0, goto cleanup);
}
void * accept_sock_thread(void* atr){
int rc;
int sock = INVALID_SOCKET;
int datalen = config.traffic;
char *databuf = NULL;
struct thread_data *data = NULL;
struct sockaddr_in tcp_remote;
struct timeval t;
socklen_t size;
fd_set socks;
databuf = malloc(sizeof(char) * datalen);
memset(databuf, 0, datalen);
data = (struct thread_data*) atr;
DEBUG(my_debug_flags, ENTER_FUNCT, ("Enter Function accept_sock_thread thread_num %d \n", data->thread_num));
FD_ZERO(&socks);
FD_SET(data->sd, &socks);
t.tv_sec = 25;
t.tv_usec = 0;
rc = select(data->sd + 1 , &socks, NULL, NULL,&t);
if(rc < 0){
VL_MISC_ERR(("Error in select with Errno: %d", errno));
goto cleanup;
}
else if(rc == 0){
VL_MISC_ERR(("Accept Select returned a TIEMOUT."));
goto cleanup;
}
size = sizeof(struct sockaddr_in);
sock = accept(data->sd, (struct sockaddr *)& tcp_remote, &size);
CHECK_NOT_EQUAL("tcp accept error", sock, INVALID_SOCKET, goto cleanup);
cleanup:
// sleep(2); /* avoid EOF */
if(sock != INVALID_SOCKET){
rc = close(sock);
if(rc != 0){
data->err_chk = -1;
}
}
return NULL;
}
accept() is thread-safe and reentrant according to POSIX.
This means that two call of accept on the same descriptor should not give an undefined behaviour. One of the accept will open the socket and the other will return an error.
You can see a little more there :
Is accept() thread-safe?
Are BSD/Posix sockets reentrant?
Only one thread is going to accept the connection. The kernel insures this. It has been this way in the unix/posix world for a very long time now.
Only one thread would accept(), so far so good ... - but before this all threads will be triggered to have select() return, which might not be what you want.
So if you'd had N threads sleeping in select() and one connection comes in all threads would wake up, but only one would be needed, as only one would succeed in accept()ing.
To use accept() on same socket object from multiple threads/processes is not only standard, but also widly used strategy. From my knowledge apache does this. nginx does this too but in slightly different fashion.
Do note: select() can wake up multiple threads, but only one of them will accept connection, while others will either a) hang in accept() or b) return -1 and set errno to EAGAIN in case of non-blocking IO. Since you are using select() before accept() I suppose socket descriptor is in non blocking mode. So this can lead to some threads never serve a connection.
Also I would advise you to never close same socket in multiple threads. This can lead to very nasty and hard to debug consequences. Either use wrapper and shared_ptr or assign to one of threads a role of 'owner of socket'.
I have the following code in C.
void setNonBlocking(SOCKET fd){
int flags;
if (-1 == (flags = fcntl(fd, F_GETFL, 0)))
flags = 0;
fcntl(fd, F_SETFL, flags | O_NONBLOCK);
}
int main(){
int sock;
connect(sock, .....);
setNonBlocking(sock);
....
close(sock);
//we will do something here but the application exits in/after the close operation
}
I am using the socket in non-blocking mode with setNonBlocking function. When i close socket, the application immediately exits without segfault or anything else. I don't see this issue if i don't use setNonBlocking function.
How can i close the non-blocking socket without this issue?
Perhaps your application is getting SIGPIPE. You should normally handle or ignore the SIGPIPE signal when programming with sockets.
You are ignoring any error-result from fcntl. If fcntl returns -1, you should at least print out an error message (using perror, for example).
What is the difference between read() and recv(), and between send() and write() in socket programming in terms of performances, speed and other behaviors?
The difference is that recv()/send() work only on socket descriptors and let you specify certain options for the actual operation. Those functions are slightly more specialized (for instance, you can set a flag to ignore SIGPIPE, or to send out-of-band messages...).
Functions read()/write() are the universal file descriptor functions working on all descriptors.
Per the first hit on Google
read() is equivalent to recv() with a flags parameter of 0. Other values for the flags parameter change the behaviour of recv(). Similarly, write() is equivalent to send() with flags == 0.
read() and write() are more generic, they work with any file descriptor.
However, they won't work on Windows.
You can pass additional options to send() and recv(), so you may have to used them in some cases.
I just noticed recently that when I used write() on a socket in Windows, it almost works (the FD passed to write() isn't the same as the one passed to send(); I used _open_osfhandle() to get the FD to pass to write()). However, it didn't work when I tried to send binary data that included character 10. write() somewhere inserted character 13 before this. Changing it to send() with a flags parameter of 0 fixed that problem. read() could have the reverse problem if 13-10 are consecutive in the binary data, but I haven't tested it. But that appears to be another possible difference between send() and write().
Another thing on linux is:
send does not allow to operate on non-socket fd. Thus, for example to write on usb port, write is necessary.
On Linux I also notice that :
Interruption of system calls and library functions by signal handlers
If a signal handler is invoked while a system call or library function call is blocked, then either:
the call is automatically restarted after the signal handler returns; or
the call fails with the error EINTR.
... The details vary across UNIX systems; below, the details for Linux.
If a blocked call to one of the following interfaces is interrupted
by a signal handler, then the call is automatically restarted after
the signal handler returns if the SA_RESTART flag was used; otherwise the call fails with the error EINTR:
read(2), readv(2), write(2), writev(2), and ioctl(2) calls on "slow" devices.
.....
The following interfaces are never restarted after being interrupted by a signal handler, regardless of the use of SA_RESTART; they
always fail with the error EINTR when interrupted by a signal handler:
"Input" socket interfaces, when a timeout (SO_RCVTIMEO) has been set on the socket using setsockopt(2): accept(2), recv(2),
recvfrom(2), recvmmsg(2) (also with a non-NULL timeout argument), and recvmsg(2).
"Output" socket interfaces, when a timeout (SO_RCVTIMEO) has been set on the socket using setsockopt(2): connect(2), send(2),
sendto(2), and sendmsg(2).
Check man 7 signal for more details.
A simple usage would be use signal to avoid recvfrom blocking indefinitely.
An example from APUE:
#include "apue.h"
#include <netdb.h>
#include <errno.h>
#include <sys/socket.h>
#define BUFLEN 128
#define TIMEOUT 20
void
sigalrm(int signo)
{
}
void
print_uptime(int sockfd, struct addrinfo *aip)
{
int n;
char buf[BUFLEN];
buf[0] = 0;
if (sendto(sockfd, buf, 1, 0, aip->ai_addr, aip->ai_addrlen) < 0)
err_sys("sendto error");
alarm(TIMEOUT);
//here
if ((n = recvfrom(sockfd, buf, BUFLEN, 0, NULL, NULL)) < 0) {
if (errno != EINTR)
alarm(0);
err_sys("recv error");
}
alarm(0);
write(STDOUT_FILENO, buf, n);
}
int
main(int argc, char *argv[])
{
struct addrinfo *ailist, *aip;
struct addrinfo hint;
int sockfd, err;
struct sigaction sa;
if (argc != 2)
err_quit("usage: ruptime hostname");
sa.sa_handler = sigalrm;
sa.sa_flags = 0;
sigemptyset(&sa.sa_mask);
if (sigaction(SIGALRM, &sa, NULL) < 0)
err_sys("sigaction error");
memset(&hint, 0, sizeof(hint));
hint.ai_socktype = SOCK_DGRAM;
hint.ai_canonname = NULL;
hint.ai_addr = NULL;
hint.ai_next = NULL;
if ((err = getaddrinfo(argv[1], "ruptime", &hint, &ailist)) != 0)
err_quit("getaddrinfo error: %s", gai_strerror(err));
for (aip = ailist; aip != NULL; aip = aip->ai_next) {
if ((sockfd = socket(aip->ai_family, SOCK_DGRAM, 0)) < 0) {
err = errno;
} else {
print_uptime(sockfd, aip);
exit(0);
}
}
fprintf(stderr, "can't contact %s: %s\n", argv[1], strerror(err));
exit(1);
}
"Performance and speed"? Aren't those kind of ... synonyms, here?
Anyway, the recv() call takes flags that read() doesn't, which makes it more powerful, or at least more convenient. That is one difference. I don't think there is a significant performance difference, but haven't tested for it.
The only difference between recv() and read() is the presence of flags. With a zero flags argument, recv() is generally equivalent to read()
you can use write() and read() instead send() and recv() but send() and recv() offer much greater control over your data transmission