Every once in a while we get
Mar 31, 2015 7:24:32 PM org.apache.tomcat.util.threads.ThreadPool logFull
SEVERE: All threads (200) are currently busy, waiting. Increase maxThreads (200) or check the servlet status
in the Catalina logs and server stops responding. The server is used very lightly and number of concurrent users not anywhere near 200 (2-3 on a busy day)
The thread dump shows 199 of these:
"TP-Processor200" daemon prio=10 tid=0x00002b513c31b000 nid=0x1c44 runnable [0x00002b514a9a7000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
- locked <0x00000007873208a0> (a java.io.BufferedInputStream)
at org.apache.jk.common.ChannelSocket.read(ChannelSocket.java:628)
at org.apache.jk.common.ChannelSocket.receive(ChannelSocket.java:566)
at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:693)
at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:898)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)
at java.lang.Thread.run(Thread.java:662)
and one of
"TP-Processor4" daemon prio=10 tid=0x00002b513c21a000 nid=0x7470 in Object.wait() [0x00002b5135520000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x0000000784a7a208> (a org.apache.tomcat.util.threads.ThreadPool)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.threads.ThreadPool.findControlRunnable(ThreadPool.java:339)
- locked <0x0000000784a7a208> (a org.apache.tomcat.util.threads.ThreadPool)
at org.apache.tomcat.util.threads.ThreadPool.runIt(ThreadPool.java:314)
at org.apache.jk.common.ChannelSocket.acceptConnections(ChannelSocket.java:676)
at org.apache.jk.common.ChannelSocket$SocketAcceptor.runIt(ChannelSocket.java:879)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)
at java.lang.Thread.run(Thread.java:662)
Can anybody figure out why it complaints about busy threads when they are clearly not?
First, the InputStream method when used with sockets (i.e. SocketInputStream) - will block by default "until data is received, the end of the stream is detected, or an exception is thrown." A timeout value can be set with setSoTimeout to limit blocking on read to x ms, which when reached will throw a SocketTimeoutException.
A socket that is blocked reading in another thread can be shut down for input which should then signal to the reading thread to close the socket and exit.
This can be done by using the shutdownInput method on a socket:
//Shutting down input to socket...
socket.shutdownInput();
Then, you could still send to an output socket with:
//Write to socket again
toSocket.print("is socket connection still available?\r\n");
//close socket
socket.close();
Also depending on whatever "device" is being used to deliver the stream...i.e. if it's something like a container or a COM port this behavior may differ based on implementation or by vendor specification...if this is the case you might have a look at some of the suggestions offered here: Java InputStream blocking read - and as suggested possibly wrapping the input with DataInputStream to have things behave with more "file-like" consistency, i.e. ensuring that an EOFException is thrown if nothing is received.
Still another option might be to switch to a non-blocking connector implementation like NIO - where threads are not blocked during the time the connections are idle, and instead are monitored by special "poller" threads which serve to limit over-consumption of the "normal" threads in the thread-pool.
Some useful information on NIO and thread contention can also be found here: http://tutorials.jenkov.com/java-nio/server-socket-channel.html
Related
I am coding a game-server that allows up to 1100 concurrent connections using thread-per-connection approach. Every time a login packet is read from the client socket I want to be able to give it 5 seconds to connect, otherwise gracefully the connection and release the thread to the pool.
I know about alarm() for sending the process a SIGALRM, but which thread receives the signal is undefined behavior. I also tried the setitimer function, but it also sends the signal to the process. Blocking the signal in all threads but ours is impossible because I need to get the signals in all 5 threads.
Is there any way of doing this without changing the entire server architecture?
Note: This is not a personal project, so changing the thread-per-connection model is not an option, please consider these answers out-of-topic.
Threads and signals don't mix well, for the reasons you found out -- it's indeterminate which thread will receive the signal.
A better way to get a timeout within a thread is to set the socket to non-blocking mode and then run a while-loop around select() and recv(). Use the timeout argument to select() to ensure that select() will wake up at the end of your 5-second deadline, pass your socket in as part of the read-fd_set argument, and keep in mind that if the connection is TCP, the data from your socket may arrive in multiple small chunks (hence the while-loop, to collect all of them into a buffer).
I'm programing a small c linux local server. I've read a lot of documentation about threads, select function, nonblocking options, etc.
But I can't find any documentation about how to deal with client-side failures.
More specifically, working with threads and blocking sockets (not the best idea, I know, but I'm just running some tests), what happens when the client connection goes too slow? or what happens when the client doesn't close the connection properly (or not closing it at all)? Will my socket remain blocked? or the thread will never finish?
How to deal with this situations?
When using blocking sockets you have a couple of options.
One is to have one thread per client such that when you wait for information it doesn't matter how long you block. Note that when a connection is closed, the blocked operation will terminate. You can read a much more detailed description of this here.
An alternative to multiple threads is to use select. This allows you to wait on multiple file descriptors until some subset of the file descriptors are ready in the sense that they will not block. So basically instead of blocking on a single file descriptor during a read or write, you instead block on select and then you know you won't later block on the read/write.
Finally, you can use the asynchronous I/O functions, aio_read and aio_write which will perform the read/write asynchronously from the calling thread of execution.
Typically, sockets have some timeout value which can be controlled by the client. If a connection runs too slowly, or the connection dies for some reason (e.g. poor internet connectivity), the socket operation might continue to block until the timeout expires. With Linux sockets, when the timeout expires, you'll get an ETIMEDOUT errno that you can then handle later.
Typical values for the timeout are on the order of 60-300 seconds, but you can set them lower if you want to know about timeouts sooner. You can also set it to infinite, but this isn't recommended if you're using direct blocking calls since you could hang your thread forever.
On Linux (and any other system using the BSD socket API), you may change the socket timeouts using
struct timeval timeout;
timeout.tv_sec = 60;
timeout.tv_usec = 0;
setsockopt(socket, SOL_SOCK, SO_RCVTIMEO, &timeout, sizeof(struct timeval));
reply = redisCommand(rcontext,"HGET %u %u",env->cr[3] ,KeyHandle);
if(reply == NULL)
{
printf("in preNtDeletKey rediscommand error ! and the err type is %d the string is %s \n" ,rcontext->err,rcontext->errstr)";
}
Here I got a error , the reply return NULL
the output is
in preNtDeletKey rediscommand error ! and the err type is 1 the string is Interrupted system call
I use this in my project . And i grep in the hiredis source don't find Interrupted system call
I want to know what the reason to cause a Interrupted system call
How hiredis write the string to the redisContext (because I don't find in the sourec)
How we avoid the Interrupted system call ?
The hiredis package marshals your command using the Redis protocol, and sends it to the Redis server. It then synchronously waits for the reply.
You will find the functions dealing with the sockets in the hiredis.c file:
int redisBufferRead(redisContext *c)
int redisBufferWrite(redisContext *c, int *done)
In these functions, the EAGAIN error is handled, but not the EINTR error which corresponds to the "Interrupted system call" message.
The consequence is any Unix signal, received by the process when hiredis is doing a write or (more likely) a read operation, can interrupt the operation and cause this error.
You need first to understand which kind of signal the application receives. Depending on the nature of the signal and the application, there are various ways to handle this situation:
masking or deferring signal handlers before doing Redis calls
binding the signal to an event loop handler (if any) to avoid the signal to be processed when it is not expected
dedicate a given thread to handle all signals (and avoid any Redis calls in this thread)
using the SA_RESTART option (in sigaction) to tell the system to replay interrupted system calls automatically
just try to do the operation again (it might not be possible though)
Personally, I would favor hiredis to handle the situation in a more graceful way (i.e. processing EINTR just like EAGAIN).
UPDATE:
The EAGAIN error is normally returned in two situations:
when the non blocking mode has been activated by calling redisConnectNonBlock or
redisConnectUnixNonBlock()
when the connection is in blocking mode (default) and the redisSetTimeout() method has been called to set a timeout
Please note calling the redisSetTimeout() functions on client side just set the SO_RCVTIMEO and SO_SNDTIMEO properties of the socket. It is completely unrelated to the timeout defined in the Redis configuration file which is a server side idle timeout (Redis server being able to close a connection if it has been inactive for more than N seconds).
Getting EAGAIN in the second situation means the Redis instance is not responsive enough for the provided timeout. You may want to simply increase the timeout or investigate further the latency issues on Redis server side.
No clue, but a quick search says that (on Linux kernel) a system call can be interrupted when nothing is wrong and when that happens, the typical thing is to just do it again. My guess, since there is nothing to go on here, is the Redis database or some part of your code isn't handling that situation. http://www.win.tue.nl/~aeb/linux/lk/lk-4.html
I have a thread that is essentially just for listening on a socket. I have the thread blocking on accept() currently.
How do I tell the thread to finish any current transaction and stop listening, rather than staying blocked on accept?
I don't really want to do non-blocking if I don't have to...
Use the select(2) call to check which fd are ready to read.
The file descriptors from call can be read with out it blocking. eg accept() on the returned fd will immediately create a new connection.
Basically you have two options, the first one is to use interrupts: i.e
http://www.cs.cf.ac.uk/Dave/C/node32.html (see the signal handler section, it also supply a th_kill example).
From accept man page:
accept() shall fail if:
EINTR
The system call was interrupted by a signal that was caught before a valid connection arrived.
Another option is to use Non blocking sockets and select(): i.e.:
http://publib.boulder.ibm.com/infocenter/iseries/v5r3/index.jsp?topic=%2Frzab6%2Frzab6xnonblock.htm
Anyhow, usually in multi-threaded servers there's one thread which accepts new connections and spawns other threads for each connections. Since accept()ing and than recv()ing, can delay new connections requests... (Unless you're working with one client, and then accept()ing and recieving might be OK)
Use pthread_cancel on the thread. You'll need to make sure you've installed appropriate cancellation handlers (pthread_cleanup_push) to avoid resource leaks, and you should disable cancellation except for the duration of the accept call to avoid race conditions where the cancellation request might get acted upon later by a different function than accept.
Note that, due to bugs in glibc's implementation of cancellation, this approach could lead to lost connections and file descriptor leaks. This is because glibc/NPTL provides no guarantee that accept did not already finish execution and allocate a new file descriptor for the new connection before the cancellation request is acted upon. It should be a fairly rare occurrence but it's still an issue to consider...
See: http://sourceware.org/bugzilla/show_bug.cgi?id=12683
and for a discussion of the issue: Implementing cancellable syscalls in userspace
From Wake up thread blocked on accept() call
I just used the shutdown() system call and it seems to work...
Can a socket be closed from another thread when a send / recv on the same socket is going on?
Suppose one thread is in blocking recv call and another thread closes the same socket, will the thread in the recv call know this and come out safely?
I would like to know if the behavior will differ between different OS / Platforms. If yes, how will it behave in Solaris?
In linux closing a socket won't wake up recv(). Also, as #jxh says:
If a thread is blocked on recv() or send() when the socket is closed
by a different thread, the blocked thread will receive an error.
However, it is difficult to detect the correct remedial action after
receiving the error. This is because the file descriptor number
associated with the socket may have been picked up by yet a different
thread, and the blocked thread has now been woken up on an error for a
"valid" socket. In such a case, the woken up thread should not call
close() itself.
The woken up thread will need some way to differentiate whether the
error was generated by the connection (e.g. a network error) that
requires it to call close(), or if the error was generated by a
different thread having called close() on it, in which case it should
just error out without doing anything further to the socket.
So the best way to avoid both problems is to call shutdown() instead of close(). shutdown() will make the file descriptor still available, so won't be allocated by another descriptor, also will wake up recv() with an error and the thread with the recv() call can close the socket the normal way, like a normal error happened.
I don't know Solaris network stack implementation but I'll throw out my theory/explanation of why it should be safe.
Thread A enters some blocking system call, say read(2), for this given socket. There's no data in socket receive buffer, so thread A is taken off the processor an put onto wait queue for this socket. No network stack events are initiated here, connection state (assuming TCP) has not changed.
Thread B issues close(2) on the socket. While kernel socket structure should be locked while thread B is accessing it, no other thread is holding that lock (thread A released the lock when it was put to sleep-wait). Assuming there's no outstanding data in the socket send buffer, a FIN packet is sent and the connection enters the FIN WAIT 1 state (again I assume TCP here, see connection state diagram)
I'm guessing that socket connection state change would generate a wakeup for all threads blocked on given socket. That is thread A would enter a runnable state and discover that connection is closing. The wait might be re-entered if the other side has not sent its own FIN, or the system call would return with eof otherwise.
In any case, internal kernel structures will be protected from inappropriate concurrent access. This does not mean it's a good idea to do socket I/O from multiple threads. I would advise to look into non-blocking sockets, state machines, and frameworks like libevent.
For me, shutdown() socket from another thread do the job in Linux
If a thread is blocked on recv() or send() when the socket is closed by a different thread, the blocked thread will receive an error. However, it is difficult to detect the correct remedial action after receiving the error. This is because the file descriptor number associated with the socket may have been picked up by yet a different thread, and the blocked thread has now been woken up on an error for a "valid" socket. In such a case, the woken up thread should not call close() itself.
The woken up thread will need some way to differentiate whether the error was generated by the connection (e.g. a network error) that requires it to call close(), or if the error was generated by a different thread having called close() on it, in which case it should just error out without doing anything further to the socket.
Yes, it is ok to close the socket from another thread. Any blocked/busy threads that are using that socket will report a suitable error.