I've got a Webservice deployed on Apache ServiceMix which uses Apache Camel to invoke an ActiveMQ driven route using code similar to the following:
context.createProducerTemplate().sendBody("activemq:startComplex", xml);
The invocation works fine but after some time the file descriptor limit on my Linux machine gets hit. The resources are eaten up by a whole bunch (a few thousand) of ActiveMQ threads. Under the jmx console I can see a lot of threads similar to the following:
Name: ActiveMQ Transport: tcp://localhost/127.0.0.1:61616
State: RUNNABLE
Total blocked: 0 Total waited: 0
Stack trace:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:5 0)
org.apache.activemq.transport.tcp.TcpTransport$2.fill(TcpTransport.java:589)
org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:5 8)
org.apache.activemq.transport.tcp.TcpTransport$2.read(TcpTransport.java:574)
java.io.DataInputStream.readInt(DataInputStream.java:370)
org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:275)
org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:222)
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:214)
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:197)
java.lang.Thread.run(Thread.java:662)
and
Name: ActiveMQ Transport: tcp:///127.0.0.1:46420
State: RUNNABLE
Total blocked: 0 Total waited: 2
Stack trace:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
org.apache.activemq.transport.tcp.TcpTransport$2.fill(TcpTransport.java:589)
org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
org.apache.activemq.transport.tcp.TcpTransport$2.read(TcpTransport.java:574)
java.io.DataInputStream.readInt(DataInputStream.java:370)
org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:275)
org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:222)
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:214)
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:197)
java.lang.Thread.run(Thread.java:662)
And ideas how to get rid of the hanging threads?
See this FAQ
http://camel.apache.org/why-does-camel-use-too-many-threads-with-producertemplate.html
You should not create a new producer template on every message send. And if you do, then remember to close it after usage.
I managed to get rid of the leaking threads issue by dropping all use of ProducerTemplate and ConsumerTemplate.
I am now using standard JMS APIs to send and receive messages from ActiveMQ.
Related
We are using Java 8, SOAP, Apache Camel and the camel-cxf component to route messageges. Everything works fine until we encounter a unmarshalling error in the response. What happens is that the CPU gets 100% on a I/O Dispatcher thread. The thread is RUNNING and not BLOCKED.
What we do is requesting a list of adresses that we send through a splitter that aggregates the responses.
.split(body()).executorServiceRef("splitterThreadPool")
.timeout(aggregationTimeout)
.parallelProcessing(true)
.aggregationStrategy(agpAggregationStrategy
...
.toD("direct:${exchangeProperty.AgpServiceComponentId}")
.end()
If one response causes an unmarshalling error we get this problem. The dispatcher thread will take all CPU until the service is restarted. Has anyone encountered a similar problem and how did you solve it?
There are other soap faults that do not create this problem.
I have a reliable acquisition flow where an inbound SFTP connector is polling and reading the file and publishing to JMS ActiveMq. I have an issue when ActiveMQ is down. JMS connector get in to reconnection mode lets say every minute but SFTP is actively reading the files and try to publish the file in ActiveMq. which causes the loss of message and the concept of deadletter queue also does not work as JMS is down.
Is there a way we can stop reading the file until the JMS successfully reconnect? also what if the message was in flight and JMS get down before message end up in the queue? will the message rollback?
I am using Mule 3.9
If you set the flow processing strategy to synchronous and use only 1 receiver thread in the connector configuration. Then it should not try to read a new file if the previous one wasn't processed. Without those changes it is not a correct implementation of the reliable pattern.
More information:
Documentation of reliable patterns: https://docs.mulesoft.com/mule-runtime/3.9/reliability-patterns.
KB Article: https://help.mulesoft.com/s/article/How-to-fetch-files-in-order-using-the-FTP-connector
Example:
<sftp:connector name="sftpConn">
<receiver-threading-profile maxThreadsActive="1" />
</sftp:connector>
<flow name="mainFlow" processingStrategy="synchronous">
...
We are facing issue with solr/zookeeper where zookeeper timeouts after 10000ms. Error below.
SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper <server1>:9181,<server2>:9182,<server2>:9183 within 10000 ms.
at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:184)
at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:121)
We are not getting any error in zookeeper logs.Except below logs
2018-12-19 04:35:22,305 [myid:2] - INFO [SessionTracker:ZooKeeperServer#354] - Expiring session 0x200830234de3127, timeout of 10000ms exceeded
2018-12-19 05:35:38,304 [myid:2] - INFO [SessionTracker:ZooKeeperServer#354] - Expiring session 0x200b4f912730086, timeout of 10000ms exceeded
During the issue threads go high and we could notice below in weblogic server.
Name: Connection evictor
State: TIMED_WAITING
Total blocked: 0 Total waited: 1
Stack trace:
java.lang.Thread.sleep(Native Method)
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
java.lang.Thread.run(Thread.java:748)
What could be going wrong here?
In my experience, ZK timeouts have almost always been due to something on the Solr node, rather than a problem in ZK.
You don't provide all the timestamps, but the theory is that:
Solr fails to send the heartbeat for some reason
ZK assumes the client has gone away and closes the connection
Solr tries to use the connection that ZK closed
So why might the Solr node fail to send the heartbeat? This could be because the Solr node was simply overloaded, (Is the thread spike a cause, or a symptom?) or just working through a very long GC pause could do it too.
I have:
Camel route (transacted=true), consuming from an MQ Queue
Using Spring's WebSphereUowTransactionManager
Transactionality works
Running on IBM Liberty
But, I get this message:
Setup of JMS message listener invoker failed for destination 'MY.QUEUE' - trying to recover.
Cause: Local JMS transaction failed to commit; nested exception is com.ibm.msg.client.jms.DetailedIllegalStateException:
MQJCA1020: The session is closed.
The application attempted to use a JMS session after it had closed the session.
Modify the application so that it closes the JMS session only after it has finished using the session.
This appears to be related to this other Stack Overflow question, but I've tried changing the configuration in server.xml, with no success.
You can try setting cacheLevelName=CACHE_CONSUMER which allows to re-use the JMS consumer and avoids endless of creation/destruction of JMS resources, as indicated by the error message may be the cause.
You can see more about the importanse of cache levels on the Camel JMS documentation: http://camel.apache.org/jms
Please help me in finalizing architecture for Asynchronous delivery support using Apache Camel and ActiveMQ and below I have explained point by point basis about my requirement.
I have Jetty Server receiving incoming messages and ActiveMQ to store it in disk using Kaha DB.
Active MQ sends ack once it stores in kaha DB back to client.
I have Spring AbstractPollingMessageListenerContainer JMS Message listener which picks up the message from activemq queue every 1 second and dispatch to Camel HTTP endpoints and then finally sent to actual remote receivers.
Once Dispatcher thread gets response from remote receivers it deletes message from ActiveMQ.
Assume that I have many slow remote receivers in that case my Dispatcher thread created by AbstractPollingMessageListenerContainer remains blocked until I get response from remote receivers. This results to creation of new Dispatcher threads since already created Dispatcher threads are not able to dispatch new messages from ActiveMQ queue.
Now creation of many Dispatcher threads result into more CPU usage which impacts overall performance.
Now my requirement is I want Dispatcher thread only to dispatch messages from ActiveMQ queue to HTTP endpoint and forget and also not do acknowledgement so that message is still in queue.
Also I will not let Dispatcher thread to wait till I get response so I have thought to handle response using separate thread and this same thread will only delete message from ActiveMQ queue.
So my current architecture is like below:
Camel Jetty Server ----> ActiveMQ queue ----> Dispatcher Thread ---> Camel Direct endpoint ----> Camel HTTP endpoint ---> remote receivers sending response back ---> response ---> Dispatcher Thread (sends ack to delete messages from ActiveMQ queue) ----> ActiveMQ Queue.
Here I feel since we are using Direct endpoint which is synchronous so Dispatcher thread remains active till it gets response and so same dispatcher thread is not able to process further new message from ActiveMQ queue.
Please suggest if some thing else I can use here to avoid Direct endpoint.
I used SEDA endpoint but drawback is it processes 1 message using 1 thread and also gets blocked till it gets response from receivers.
In this approach previously Dispatcher thread gets blocked but now Seda consumer threads gets blocked and could not dispatch new messages from in memory queue of SEDA towards remote receivers.
I feel some kind of design which helps me in keep on sending message to remote receivers and only when response comes back some daemeon thread gets notified and it will handle acknowledgement towards activeMQ. Also I thought to use NIO framework implementation like Camel netty/netty4-http component but could not find exact usage and how to fit it in current architecture.
Modified architecture should be like below:
Camel Jetty Server ----> ActiveMQ queue ----> Dispatcher Thread--->Unknown Stuff ----> Camel HTTP Endpoint ---> remote receivers sending response back--->Unknown Stuff (sends ack to delete messages from ActiveMQ queue) ----> ActiveMQ Queue
Please help me in finalising Unknown Stuff and I am posting my query after doing enough R & D.
Also new ideas are welcome and please give me idea with a restriction that I must persist the message and delete it only after getting success response from remote receiver. Also I have to design architecture only using Apache Camel routes.
Route Definitions:
1. Dispatcher Route:
from(fromUri)to(toUris);
fromUri:
[ActiveMQueue.http1270018081testEndpoint1:queue:ActiveMQueue?maxConcurrentConsumers=15&concurrentConsumers=3&maxMessagesPerTask=10&messageListenerContainerFactoryRef=AbstractPollingMessageListenerContainer ]
ToUris:
[ActiveMQ.DLQ:queue:ActiveMQ.DLQ, direct:http1270018081testEndpoint1]
2.Remote Receiver Proxy Route:
fromUri:direct:http1270018081testEndpoint1
from(fromUri).to(toUri).process(responseProcessor)
toUri:http://127.0.0.1:8081/testEndpoint1?bridgeEndpoint=true
responseProcessor: To process response received by remote receiver.
Overall Route looks like below:
Dispatcher Route---> Remote Receiver Route---> Remote Server
JMS message acknowledgement is done under the covers, so your only way to really "send the acknowledgement back to the queue" is to use a JMS transaction (doesn't need to be XA)
It sounds like a LLR-style transaction would be useful and drastically simplify things for you. If you consume the message from the queue using a JMS-local transaction, and only have one other endpoint, the message will only be acknowledged and removed from the queue when the http send is completed-- even though HTTP doesn't support transactions. You can then have a number of concurrent consumers to run in parallel and combine with throttling to help with rate limiting.
from: amq:queue:INPUT.REQUESTS?.. concurrentConsumers.. and transacted enabled
throttle
to: http://url