Our environment consists of 3 jboss servers(portal,jms,reconciliation).
Reconciliation server hosts camel routes which has a Route consuming from queue(SLAQueue).
JMS server has all our queues hosted.
Recently we identified a bug where some of messages in TaskQueue
hosted on JMS server are not being delivered to MDB's on portal server. For some reason they
are stuck and when we restart the JMS server the stuck messages are
delivered.
To investigate we enabled TRACE level logging on "org.apache.activemq.artemis". We are noticing lot of chatter between our camel jms component and JMS server. One instance of chatter is listed below, logs like these get written every 1 second.
Questions :
What is the mechanism camel jms component uses to get messages from queue? Does JMS component poll the queue every second(pull) ? or JMS component gets notified when the message arrives in the queue ?
Is the mechanism different from J2EE Message driven beans? The MDB' get notified when the message arrives in the queue.
Based on the chatter below i think it is polling. If it is polling can the polling window be configured ? I have tried receiveTimeout option with no luck.
**Example chatter in the logs between camel JMS component and JMS server : **
2021-11-17 16:39:24,612 TRACE [org.apache.activemq.artemis.ra] (Camel (camelContextReconciliation) thread #0 - JmsConsumer[SLAQueue]) createdConsumer ActiveMQSession->ClientSessionImpl [name=eebaf2ec-47b3-11ec-ad21-0242ac110003, username=null, closed=false, factory = org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl#413c962e, metaData=(jms-session=,)]#72d15942 consumer=org.apache.activemq.artemis.ra.ActiveMQRAMessageConsumer#1dd2e8b
2021-11-17 16:39:24,612 TRACE [org.apache.activemq.artemis.ra] (Camel (camelContextReconciliation) thread #0 - JmsConsumer[SLAQueue]) addConsumer(org.apache.activemq.artemis.ra.ActiveMQRAMessageConsumer#1dd2e8b)
2021-11-17 16:39:24,612 TRACE [org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl] (Thread-206 (ActiveMQ-remoting-threads-ActiveMQServerImpl::serverUUID=b5ba8675-1a62-11ec-b397-0242ac110002-1886035738)) RemotingConnectionID=eeb5e9d6-47b3-11ec-ad21-0242ac110003 handling packet PACKET(SessionConsumerFlowCreditMessage)[type=70, channelID=12, responseAsync=false, requiresResponse=false, correlationID=-1, packetObject=SessionConsumerFlowCreditMessage, consumerID=5086, credits=1048576]
2021-11-17 16:39:24,612 TRACE [org.apache.activemq.artemis.ra] (Camel (camelContextReconciliation) thread #0 - JmsConsumer[SLAQueue]) unlock()
2021-11-17 16:39:24,612 TRACE [org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnection] (Thread-206 (ActiveMQ-remoting-threads-ActiveMQServerImpl::serverUUID=b5ba8675-1a62-11ec-b397-0242ac110002-1886035738)) InVMConnection [serverID=0, id=eeb5e9d6-47b3-11ec-ad21-0242ac110003]::packet sent done
2021-11-17 16:39:24,612 TRACE [org.apache.activemq.artemis.ra] (Camel (camelContextReconciliation) thread #0 - JmsConsumer[SLAQueue]) unlock()
2021-11-17 16:39:24,612 TRACE [org.apache.activemq.artemis.ra] (Camel (camelContextReconciliation) thread #0 - JmsConsumer[SLAQueue]) lock()
2021-11-17 16:39:24,612 TRACE [org.apache.activemq.artemis.ra] (Camel (camelContextReconciliation) thread #0 - JmsConsumer[SLAQueue]) tryLock()
2021-11-17 16:39:24,612 TRACE [org.apache.activemq.artemis.ra] (Camel (camelContextReconciliation) thread #0 - JmsConsumer[SLAQueue]) getUseTryLock()
2021-11-17 16:39:24,612 TRACE [org.apache.activemq.artemis.ra] (Camel (camelContextReconciliation) thread #0 - JmsConsumer[SLAQueue]) getUseTryLock()
2021-11-17 16:39:24,612 TRACE [org.apache.activemq.artemis.ra] (Camel (camelContextReconciliation) thread #0 - JmsConsumer[SLAQueue]) lock()
2021-11-17 16:39:24,612 TRACE [org.apache.activemq.artemis.ra] (Camel (camelContextReconciliation) thread #0 - JmsConsumer[SLAQueue]) receive org.apache.activemq.artemis.ra.ActiveMQRAMessageConsumer#1dd2e8b timeout=120000
2021-11-17 16:39:24,612 TRACE [org.apache.activemq.artemis.ra] (Camel (camelContextReconciliation) thread #0 - JmsConsumer[SLAQueue]) checkState()
2021-11-17 16:39:24,612 TRACE [org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl] (Camel (camelContextReconciliation) thread #0 - JmsConsumer[SLAQueue]) org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl#5e7ed34b{consumerContext=ActiveMQConsumerContext{id=5086}, queueName=jms.queue.SLAQueue}:: receive(120000)
2021-11-17 16:39:24,612 TRACE [org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl] (Camel (camelContextReconciliation) thread #0 - JmsConsumer[SLAQueue]) org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl#5e7ed34b{consumerContext=ActiveMQConsumerContext{id=5086}, queueName=jms.queue.SLAQueue}::receive(120000, false)
2021-11-17 16:39:24,612 TRACE [org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler] (Thread-1288 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6#775fa129)) ServerSessionPacketHandler::handlePacket,PACKET(SessionConsumerFlowCreditMessage)[type=70, channelID=12, responseAsync=false, requiresResponse=false, correlationID=-1, packetObject=SessionConsumerFlowCreditMessage, consumerID=5086, credits=1048576]
2021-11-17 16:39:24,612 DEBUG [org.apache.activemq.artemis.core.server.impl.ServerConsumerImpl] (Thread-1288 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6#775fa129)) ServerConsumerImpl [id=5086, filter=null, binding=LocalQueueBinding [address=jms.queue.SLAQueue, queue=QueueImpl[name=jms.queue.SLAQueue, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::serverUUID=b5ba8675-1a62-11ec-b397-0242ac110002], temp=false]#3b0e1e9f, filter=null, name=jms.queue.SLAQueue, clusterName=jms.queue.SLAQueueb5ba8675-1a62-11ec-b397-0242ac110002]]::FlowControl::Received 1048576 credits, previous value = 0 currentValue = 1048576
2021-11-17 16:39:24,612 TRACE [org.apache.activemq.artemis.core.server.impl.ServerConsumerImpl] (Thread-1288 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6#775fa129)) ServerConsumerImpl [id=5086, filter=null, binding=LocalQueueBinding [address=jms.queue.SLAQueue, queue=QueueImpl[name=jms.queue.SLAQueue, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::serverUUID=b5ba8675-1a62-11ec-b397-0242ac110002], temp=false]#3b0e1e9f, filter=null, name=jms.queue.SLAQueue, clusterName=jms.queue.SLAQueueb5ba8675-1a62-11ec-b397-0242ac110002]]::calling promptDelivery from receiving credits
2021-11-17 16:39:24,612 TRACE [org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler] (Thread-1288 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6#775fa129)) ServerSessionPacketHandler::scheduling response::null
2021-11-17 16:39:24,612 TRACE [org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler] (Thread-1288 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6#775fa129)) ServerSessionPacketHandler::regular response sent::null
Yes, the JMS consumers are polling. By default Camel uses Spring's DefaultMessageListenerContainer under the hood.
The Spring-Listener itself uses the JMS API directly.
Spring uses a default timeout of 1 second to pull messages. I don't know if you really can change this with Camel's receiveTimeout option.
Related
I have a Apache Camel route that starts with a Mail consumer, followed by some custom processing steps as part of several sub-routes. All routs are connected directly. When some processing fails downstream, the failed exchange returns to the Mail consumer, which performs a rollback and puts the mail message currently being processed back into its original unread state in the inbox:
2022-11-18 16:23:13 WARN MailConsumer:572 - Exchange failed, so rolling back message status: Exchange[7A8EE72E3AFB7CC-00000000000000C0]
This again triggers the Camel mail consumer, resulting in an error loop. I do have a Camel error handler configured for some part of the processing, but not all of it.
How can I configure the mail consumer such that a downstream processing failure does not return the mail back to its unread state?
The mail consumer is configured as follows:
.from("imaps://{{imapPath}}?username={{imapUserName}}&password=RAW({{imapPassword}})&delay={{imapPollingPeriodMs}}&moveTo={{processedMailFolderName}}")
I need to send messages to a queue in Azure Service Bus. I was using HTTP Post to send that messages, but I need to improve my flow rate, then I decided to test AMQP protocol.
Below the code:
public void configure() throws Exception {
AMQPComponent amqp = AMQPComponent.amqpComponent("amqps://server.servicebus.windows.net",
"accessKey", "secretKey");
getContext().addComponent("amqp", amqp);
ActiveMQJMSConnectionFactory connection = new ActiveMQJMSConnectionFactory("tcp://localhost:61616",
"admin", "admin");
getContext().addComponent("amq", JmsComponent.jmsComponent(connection));
from("amq:TEST")
.routeId("fromQueueToAzure")
.autoStartup(true)
.removeHeaders("JMS*")
.to("amqp:amqp.queue")
.log("sent");
}
When I start this route, the communication works, but for every message that Camel are sending to Servicebus, I got this log:
2019-02-07 18:47:11 [main] INFO DefaultCamelContext:3202 - Apache Camel 2.22.0 (CamelContext: camel-1) started in 0.602 seconds
2019-02-07 18:47:12 [AmqpProvider :(1):[amqps://server.servicebus.windows.net:-1]] INFO SaslMechanismFinder:106 - Best match for SASL auth was: SASL-PLAIN
2019-02-07 18:47:12 [AmqpProvider :(1):[amqps://server.servicebus.windows.net:-1]] INFO JmsConnection:1329 - Connection ID:5f75145f-6f10-4867-a590-782e507d51a8:1 connected to remote Broker: amqps://server.servicebus.windows.net
2019-02-07 18:47:13 [Camel (camel-1) thread #1 - JmsConsumer[TEST]] INFO fromQueueToAzure:159 - sent
2019-02-07 18:47:14 [AmqpProvider :(2):[amqps://server.servicebus.windows.net:-1]] INFO SaslMechanismFinder:106 - Best match for SASL auth was: SASL-PLAIN
2019-02-07 18:47:14 [AmqpProvider :(2):[amqps://server.servicebus.windows.net:-1]] INFO JmsConnection:1329 - Connection ID:08ea246c-523e-4eb3-822e-c7d7b26aea85:2 connected to remote Broker: amqps://server.servicebus.windows.net
2019-02-07 18:47:15 [Camel (camel-1) thread #1 - JmsConsumer[TEST]] INFO fromQueueToAzure:159 - sent
2019-02-07 18:47:16 [AmqpProvider :(3):[amqps://server.servicebus.windows.net:-1]] INFO SaslMechanismFinder:106 - Best match for SASL auth was: SASL-PLAIN
2019-02-07 18:47:16 [AmqpProvider :(3):[amqps://server.servicebus.windows.net:-1]] INFO JmsConnection:1329 - Connection ID:c8c40237-a73c-43cf-970d-c5cbf726eb21:3 connected to remote Broker: amqps://server.servicebus.windows.net
2019-02-07 18:47:17 [Camel (camel-1) thread #1 - JmsConsumer[TEST]] INFO fromQueueToAzure:159 - sent
Camel is spending one second per message to send to Servicebus.
Is it a normal behavior? Is it possible to make Camel send faster?
It appears that on each send the Camel route is creating a new Connection so that would explain why your sends are slow. In order to improve performance you'd want to use something like PooledJMS to create a connection pool so that a new connection and accompanying resources aren't recreated on each send.
You could try something like the following which uses the camel-amqp component but uses Qpid JMS directly with PooledJMS to configure it.
JmsConnectionFactory cf = new JmsConnectionFactory("amqp://localhost:5672");
JmsPoolConnectionFactory pooledCF = new JmsPoolConnectionFactory();
pooledCF.setConnectionFactory(cf);
AMQPComponent component = new AMQPComponent();
component.setConnectionFactory(pooledCF);
CamelContext context = new DefaultCamelContext();
context.addComponent("amqp", component);
I am using camel jms in a synchronous way using request-reply pattern.The camel route exposes a cxf endpoint and it places the message in to the jms queue another component process it and sends the response in replyto queue.
A new thread is getting created while receiving the reply from replyto queue?Iam facing issues with using log4j MDC with in the camel route.The values store in MDC are no more available?
Is it the behaviour of camel jms request/reply pattern?
Below is the JMS endpoint:
<to id="QueueEndpoint" pattern="InOut" uri="hornetq:queue:{{esb.api.requestqueue}}?replyTo=queue:{{esb.api.responsequeue}}&useMessageIDAsCorrelationID=true&replyToType=Exclusive&requestTimeout={{esb.api.queue.requesttimeout}}"/>
I use camel with hazelcast component and faced the problem with exchange headers, when exchange goes through hazelcast seda.
I need to poll files from directory and send exchanges to seda hazelcast queue (to processing exchanges in cluster).
There is a simple example of this situation.
There are two routes, which runs in other JVMs:
First for polling files:
from("file:someFromFolder")
.to("hazelcast:seda:queue?hazelcastInstance=#hazelcast-instance");
And second to processed this:
from("hazelcast:seda:queue?hazelcastInstance=#hazelcast-instance")
.to("file:someToFolder");
When I tried to read headers like Exchange.FILE_NAME or others, i realized they are empty.
Is it possible to save headers while transferring exchanges through hazelcast seda component?
EDIT
When I set transferExchange=true as an option on the producer route endpoint, I catch exception:
Stacktrace
---------------------------------------------------------------------------------------------------------------------------------------
org.apache.camel.RuntimeExchangeException: Message body of type org.apache.camel.component.file.GenericFile is not supported by this marshaller. on the exchange: Exchange[ID-DF-240815-MS12-60568-1487853045890-0-12][file.txt]
at org.apache.camel.impl.DefaultExchangeHolder.marshal(DefaultExchangeHolder.java:99)
at org.apache.camel.impl.DefaultExchangeHolder.marshal(DefaultExchangeHolder.java:83)
at org.apache.camel.component.hazelcast.seda.HazelcastSedaProducer.checkAndStore(HazelcastSedaProducer.java:55)
at org.apache.camel.component.hazelcast.seda.HazelcastSedaProducer.process(HazelcastSedaProducer.java:42)
at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:141)
at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)
at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:460)
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:190)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:121)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:83)
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:190)
at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:442)
at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:214)
at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:178)
at org.apache.camel.impl.ScheduledPollConsumer.doRun(ScheduledPollConsumer.java:174)
at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:101)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
So could it be marshaller try to serialize all file data?
You need to set transferExchange=true as an option on the endpoint. By default its only the message body that is transferred. You can find this information in the documentation: http://camel.apache.org/hazelcast-component.html#HazelcastComponent-seda
File upload mechanism is added to my existing application which is deployed to WebSphere 7. I use commons-fileupload-1.2.2.jar for the file upload. During testing, I encountered below exception:
[5/2/13 18:00:00:773 CST] 0000001b SystemOut O 18:00:00,773 INFO AccountsController:156 - Session is valid.
[5/2/13 18:04:42:299 CST] 0000001e servlet E com.ibm.ws.webcontainer.servlet.ServletWrapper service SRVE0068E: Uncaught exception created in one of the service methods of the servlet dispatcher in application SunLinkOnline_ST. Exception created : org.springframework.web.util.NestedServletException: Request processing failed; nested exception is org.springframework.web.multipart.MultipartException: Could not parse multipart servlet request; nested exception is org.apache.commons.fileupload.FileUploadBase$IOFileUploadException: Processing of multipart/form-data request failed. Async operation timed out
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:583)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:511)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:738)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:831)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1657)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1597)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:131)
at ph.com.sunlife.sunlink.filter.MFSanitizerFilter.doFilter(MFSanitizerFilter.java:38)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:236)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)
at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:188)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:116)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain._doFilter(WebAppFilterChain.java:77)
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:908)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:934)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:502)
at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:179)
at com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:91)
at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:864)
at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1583)
at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:186)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:445)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:504)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:301)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.ready(HttpInboundLink.java:275)
at com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.sendToDiscriminators(NewConnectionInitialReadCallback.java:214)
at com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.complete(NewConnectionInitialReadCallback.java:113)
at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:165)
at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)
at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)
at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)
at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1563)
Caused by: org.springframework.web.multipart.MultipartException: Could not parse multipart servlet request; nested exception is org.apache.commons.fileupload.FileUploadBase$IOFileUploadException: Processing of multipart/form-data request failed. Async operation timed out
at org.springframework.web.multipart.commons.CommonsMultipartResolver.parseRequest(CommonsMultipartResolver.java:172)
at org.springframework.web.multipart.commons.CommonsMultipartResolver.resolveMultipart(CommonsMultipartResolver.java:149)
at org.springframework.web.servlet.DispatcherServlet.checkMultipart(DispatcherServlet.java:1006)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:851)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:809)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:571)
... 34 more
Caused by: org.apache.commons.fileupload.FileUploadBase$IOFileUploadException: Processing of multipart/form-data request failed. Async operation timed out
at org.apache.commons.fileupload.FileUploadBase.parseRequest(FileUploadBase.java:371)
at org.apache.commons.fileupload.servlet.ServletFileUpload.parseRequest(ServletFileUpload.java:126)
at org.springframework.web.multipart.commons.CommonsMultipartResolver.parseRequest(CommonsMultipartResolver.java:165)
... 39 more
Caused by: java.net.SocketTimeoutException: Async operation timed out
Could someone please share some insights what is causing above exception. Though the exception may point to server long response or due to handling of timeout in server but is there a way to resolve this? Like handling of exception in code side? Your help is much appreciated. Please let me know if I need to include other information to resolve this exceptions. Thanks!
It seems like file upload time out need to be increased. I came across same situation once but what i needed was to upload my application to websphere via admin console. Since my application is a large one i have to increase time out for admin console.
Since your scenario is for file upload in application hosted in websphere you have to change the time out of different queue.
Please find following URLs
IBM Websphere timeout custom properties
IBM changing time out in HTTP queues
what i did in my scenario