i came across this problem when writing my first async server with single select call:
if( (retv = select((hsock<highestsocket?highestsocket:hsock)+1, &rFdx, &wFdx, &eFdx, 0) ) > 0)
{
printf("select() ended...\n");
if(FD_ISSET(hsock, &rFdx))
{
// .... handle new connection
}
for(unsigned int i=0; i < ClientList.size(); i++)
{
ServerClient* client = ClientList[i];
if(FD_ISSET(client->socket, &rFdx))
{
// handle client read
}
if(FD_ISSET(client->socket, &wFdx))
{
// handle client write
}
}
}
I expect to select to stop waiting and then handle client write when calling FD_SET from other thread.
If i call FD_SET on client socket from same thread, everything works as expected.
However calling it from other thread does nothing, select keeps waiting until any data is received from client.
Of course you can't modify data in one thread and expect that modification to be visible in other threads without the proper synchronization. It's likely that select reads the fd_set immediately when it's called and then never looks at it again; but regardless, there's no way of obtaining the necessary synchronization if select has already been called. You really need to rethink your design. One solution would be the "self-pipe" trick: having a pipe open that select is always looking for input on, and having your other threads send a message over the pipe to cancel any pending select and get the select thread to rescan the table of file descriptors it should be looking at (with the proper synchronization!) and update its own fd_sets.
Related
Background: My code structure: I have a master socket on main thread, then each time a new client is coming, the threadpool will be notified and let one pre allocated thread take the task.
Inside this thread, I will pass a slave socket to it, and let it using accept call to listen to the client.
Scenario: In my thread pool, thread A is listening to a client right now, now I want to stop all the pre-allocated thread and close all the connection to the client, the main thread is trying to close the connection using close the connection to the client, and trying to terminate thread A using pthread_join.
main() {
// create threadpool
// logic to create mastersocket
startServer(masterSock)
IwantToCloseServer() // this function is not directly called in main, but simulated by a terminal signal , like kill -quit pid.
}
int startServer(int msock) {
int ssock; // slaveSocket
struct sockaddr_in client_addr; // the address of the client...
unsigned int client_addr_len = sizeof(client_addr); // ... and its length
while (!stopCondition) {
// Accept connection:
ssock = ::accept((int)msock, (struct sockaddr*)&client_addr, &client_addr_len); // the return value is a socket
// I was trying to replace this line of code to poll(), but it's not does the same thing as before
if (ssock < 0) {
if (errno == EINTR) continue;
perror("accept");
running =0;
return 0;
// exit(0);
} else {
// push task to thread pool to deal with logic
}
// main thread continues with the loop...
}
return 1;
}
IwantToCloseServer(slaveSocket) {
// when i want to close() or shutdown() function to close connections, these 2 function always return -1, because the thread is blocked on accept call
// logic try to terminate all the preallocated threads, the pthread_join function is stuck because the thread is blocked on accept
}
Problem: The thread A is keeping blocking on the ::accept function , the close and shutdown function return -1, they won’t close the connection , and the pthread_join is not keep going because thread A is blocked on accept.
Things I tried:
I have try to change my while loop related accept function, for example, set a flag stopCondition,
while(!stopConditon) {
ssock = ::accept((int)msock, (struct sockaddr*)&client_addr, &client_addr_len);
}
However, when the main thread change stopCondtion, the thread A is blocked inside the accept function.
It won’t go inside the while loop, so this solution won’t affect the accept function, it’s not working
I have also tried to send a signal to this blocked Thread A, using
pthread_cancel or pthread_kill(Thread A, 9)
However, if I do this, the whole process gets killed.
3.try to use poll() to replace the line, where the accept functions at, with a timeout
however, the program doesn't behave like before, the program can't listen to client anymore.
How do I terminate thread A (which is blocked on accept function call right now), so that I can clean this pre allocated thread and restart my server ?
btw i can not use library like boost in my current program. And this is under linux system not winsocket
to check periodically stopConditon in your while(!stopConditon) { first call accept/pool with a timeout to know if there is something new about msock, then depending on the result call accept etc else do nothing
I was trying to replace this line of code to poll()
try to use poll() to replace the line, where the accept functions at, with a timeout
you cannot replace accept by poll, you have to call accept / pool first and of course check the result then may be call accept
Out of that
while(!stopConditon) {
if(!stopCondtion) {
is redundant and can be replaced by
while(!stopConditon) {
I have a question regarding adding new socket file descriptors to an FDSET. Lets say we've already connected to a socket s1:
fd_set readfds;
//s1 = socket(...);
//connect(s1, ...)...
FD_ZERO(&readfds);
FD_SET(s1, &readfds);
and we are waiting for data to come down the socket, by calling select in a thread:
socket_reader_thread() {
for (;;)
{
int rv = select(n, &readfds, NULL, NULL, &tv);
if (rv == -1) {
perror("select"); // error occurred in select()
}
else if (rv == 0) {
printf("Timeout occurred! No data after 10.5 seconds.\n");
}
else {
// one the descriptors have data
.....
}
}
}
If I now wanted to add another socket (or may be two more socket etc) to the readfds set, given that select is blocking, how should I proceed? how can I interrupt select
Is the trick to add a zero timeout and use select like poll?
You need to use the "pipe-trick".
This is where an additional socket or pipe is created add it to the fd_set.
Then to interrupt a running or pending select, send a 1 byte message to it via another thread.
The select will then return and if the special pipe FD is one of the ones that are ready in the set, that means you need to say look at a list or something "do work" - like add any new FDs to the fd_set before returning to the select call.
You can interrupt select by sending (and catching) a signal to your process, for example using raise. select will return in this case with -1 and errno set to EINTR. You can then change the events you want to wait for and call select again.
Is the trick to add a zero timeout and use select like poll?
One can simply use a timeout of 0 in which case it will just do a non-blocking check if any of the events got triggered, i.e. polling. But this should only be done in a few cases since busy polling instead of a blocking wait uses lots of resources of machine. And I would even consider the interrupting of a blocking select a questionable design, although probably not as bad as busy polling.
I am trying to wait on waitpid() and read() in a while-true loop. Specifically, I am waiting for either one of these two events and then process it in each iteration of the loop. Currently, I have the following implementation (which is not I desired).
while (true) {
pid_t pid = waitpid(...);
process_waitpid_event(...);
ssize_t sz = read(socket, ....);
process_read_event(...);
}
The problem with this implementation is that the processing of the second event depends on the completion of the first event. Instead of processing these two events sequentially, I wish to process whichever event that comes first in each iteration of the loop. How should I do this?
If you don't want to touch threading, you can include this in the options of the call to waitpid:
pid_t pid = waitpid(pid, &status, WNOHANG);
As from the manpage for waitpid:
WNOHANG - return immediately if no child has exited.
As such, if waitpid isn't ready, it won't block and the program will just keep going to the next line.
As for the read, if it is blocking you might want to have a look at poll(2). You can essentially check to see if your socket is ready every set interval, e.g. 250ms, and then call read when it is. This will allow it to not block.
Your code might look a bit like this:
// Creating the struct for file descriptors to be polled.
struct pollfd poll_list[1];
poll_list[0].fd = socket_fd;
poll_list[0].events = POLLIN|POLLPRI;
// POLLIN There is data to be read
// POLLPRI There is urgent data to be read
/* poll_res > 0: Something ready to be read on the target fd/socket.
** poll_res == 0: Nothing ready to be read on the target fd/socket.
** poll_res < 0: An error occurred. */
poll_res = poll(poll_list, 1, POLL_INTERVAL);
This is just assuming that you're reading from a socket, judging from the variable names in your code. As others have said, your problem might require something a bit more heavy duty like threading.
The answer of #DanielPorteous should work too if you don't want to use thread in your program.
The idea is simple, not keeping the waitpid and the read function to wait unless they consumes some time to do their operation. The idea is keeping a timeout mechanism so that, if waitpid has nothing to create an impact to the whole operation, it will return immediately and the same thing goes for the read operation too.
If the read function takes very long time to read the whole buffer, you may restrict the reading manually from the read function so that it doesn't read the whole at once, rather it reads for 2 milliseconds and then pass the cycle to the waitpid function to execute.
But its safe to use threading for your purpose and its pretty easy to implement. Here's a nice guideline about how can you implement threading.
In your case you need to declare two threads.
pthread_t readThread;
pthread_t waitpidThread;
Now you need to create the thread and pass specific function as their parameter.
pthread_create(&(waitpidThread), NULL, &waitpidFunc, NULL);
pthread_create(&(readThread), NULL, &readFunc, NULL);
Now you may have to write your waitpidFunc and readFunc function. They might look like this.
void* waitpidFunc(void *arg)
{
while(true) {
pid_t pid = waitpid(...);
// This is to put an exit condition somewhere.
// So that you can finish the thread
int exit = process_waitpid_event(...);
if(exit == 0) break;
}
return NULL;
}
I think that the right tool in this situation is select or poll. Both are doing essentially the same job. They allow to select those descriptors where an input is available. Hence you can wait simultaneously on two sockets for example. However, it is not directly usable in your case as you want to wait for a process and socket. The solution will be to create a pipe which will receive something when the waitpid finishes.
You can launch a new thread and connect it with the original one with a pipe. The new thread will invoke waitpid and when it finished it will write its result to the pipe. The main thread will wait either for the socket or pipe using select.
I have a single threaded program. It sends message to four destinations every five seconds. I don't want connect() to be blocked. So I am writing my program like this:
int j, rc, non_blocking=1, sockets[4], max_fd=0;
struct sockaddr server=get_server_addr();
fd_set fdset;
const struct timeval conn_timeout = { 2, 0 }; /* 2 seconds */
for (j=0; j<4; ++j)
{
sockets[j]=socket( AF_INET, SOCK_STREAM, 0 );
ioctl(sockets[j], FIONBIO, (char *)&non_blocking);
connect(sockets[j], &server, sizeof (server));
}
/* prepare fd_set */
FD_ZERO ( &fdset );
for (j=0;j<4;++j)
{
if (sockets[j] != -1 )
{
FD_SET ( sockets[j], &fdset );
if ( sockets[j] > max_fd )
{
max_fd = sockets[j];
}
}
}
rc=select(max_fd + 1, NULL, &fdset, NULL, &conn_timeout );
if(rc > 0)
{
for (j=0;j<4;++j)
{
if(sockets[j]!=-1 && FD_ISSET(sockets[j],&fdset))
{
/* send() */
}
}
}
/* close all valid sockets */
However, it seems select() returns immediately after ONE file descriptor is ready instead of blocking for conn_timeout (2 seconds). So in this case how can I achieve my targets?
The program continues if all sockets are ready.
The program can block there for 2 seconds if any one of sockets are not ready.
Yeah, select was designed on the assumption that you would want to service each socket as soon as it became ready.
If I understand what you're trying to do, then the simplest way to accomplish it will be to remove each socket from the fdset as it becomes ready. If there are any sockets left in the set, use gettimeofday to adjust the timeout downward, and call select again. When the set is empty, all four sockets are usable and you can proceed.
There are three basic approaches:
If you want to stay strictly portable you need to iterate:
calculate end time from current time and timeout of your choice
Cycle:
-- Create fdset with those fds not yet ready
-- calculate max time to wait
-- select()
-- remeber those fds that are now ready
-- break if end time reached or all fds ready
End cycle
Now you have knowledge of the ready fds and the elapsed time
If you want to stay portable, but can use threads:
start n threads
select on one fd per thread
join all threads
If you do not need to be portable: Most OSes have a facility for such a situation, e.g. Windows/.NET has WaitAll (together with async send and an event)
I don't see the connection between your stated targets and your stated problem. You are correct in saying that select() blocks until at least one socket is ready, but according to target #2 above that is exactly what you want. There's nothing in your stated targets about blocking until all four sockets are ready at the same time.
You should also note that sockets are almost always ready for writing, unless the send buffer is full, which means the receiver's receive buffer is full, which means the receiver is slower than the sender. So using select() alone as the underlying write timer isn't a good idea.
I am trying to make a simple client-server chat program. On the client side I spin off another thread to read any incomming data from the server. The problem is, I want to gracefully terminate that second thread when a person logs out from the main thread. I was trying to use a shared variable 'running' to terminate, problem is, the socket read() command is a blocking command, so if I do while(running == 1), the server has to send something before the read returns and the while condition can be checked again. I am looking for a method (with common unix sockets only) to do a non-blocking read, basically some form of peek() would work, for I can continually check the loop to see if I'm done.
The reading thread loop is below, right now it does not have any mutex's for the shared variables, but I plan to add that later don't worry! ;)
void *serverlisten(void *vargp)
{
while(running == 1)
{
read(socket, readbuffer, sizeof(readbuffer));
printf("CLIENT RECIEVED: %s\n", readbuffer);
}
pthread_exit(NULL);
}
You can make socket not blockable, as suggested in another post plus use select to wait input with timeout, like this:
fd_set input;
FD_ZERO(&input);
FD_SET(sd, &input);
struct timeval timeout;
timeout.tv_sec = sec;
timeout.tv_usec = msec * 1000;
int n = select(sd + 1, &input, NULL, NULL, &timeout);
if (n == -1) {
//something wrong
} else if (n == 0)
continue;//timeout
if (!FD_ISSET(sd, &input))
;//again something wrong
//here we can call not blockable read
fcntl(socket, F_SETFL, O_NONBLOCK);
or, if you have other flags:
int x;
x=fcntl(socket ,F_GETFL, 0);
fcntl(socket, F_SETFL, x | O_NONBLOCK);
then check the return value of read to see whether there was data available.
note: a bit of googling will yield you lots of full examples.
You can also use blocking sockets, and "peek" with select with a timeout. It seems more appropriate here so you don't do busy wait.
The best thing is likely to get rid of the extra thread and use select() or poll() to handle everything in one thread.
If you want to keep the thread, one thing you can do is call shutdown() on the socket with SHUT_RDWR, which will shut down the connection, wake up all threads blocked on it but keep the file descriptor valid. After you have joined the reader thread, you can then close the socket. Note that this only works on sockets, not on other types of file descriptor.
Look for function setsockopt with option SO_RCVTIMEO.