Write to s3 from blocking queue java using camel - apache-camel

I have a use case where I want to write in memory Java blocking queue contents using apache camel route to S3. Is this even possible??
The workaround I can think of is to pull records from blocking queue and flush to local file and then to S3 via file-s3 route.
Update:
Route :
fromF("seda:awsquue?concurrentConsumers=3&queue=#NonLimitQueue%s", getSourceId())
.convertBodyTo(byte[].class)
.setHeader(S3Constants.CONTENT_LENGTH, simple("${in.header.CamelFileLength}"))
.setHeader(S3Constants.KEY,simple("${in.header.CamelFileNameOnly}"))
.log(LoggingLevel.INFO, "route started")
.to("aws-s3://" +"bucket"
+ "?amazonS3Client=#s3ClientProfiler&serverSideEncryption=AES256&multiPartUpload=true");
Queue creation in different workflow :
context.registerBean("NonLimitQueue"+sourceId,ArrayBlockingQueue.class, () -> queue);
camelContext.addRoutes(new S3RouteBuilder(sourceId));
queue.add("qqqq");
When route starts it fails with exception :
java.lang.ClassCastException: class java.lang.String cannot be cast to class org.apache.camel.Exchange (java.lang.String is in module java.base of loader 'bootstrap'; org.apache.camel.Exchange is in unnamed module of loader 'app')
at org.apache.camel.component.seda.SedaConsumer.doRun(SedaConsumer.java:171)
at org.apache.camel.component.seda.SedaConsumer.run(SedaConsumer.java:125)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
which I believe is failing at queue.add("qqqq"). How to write to queue explicitly is not clear to me.

camel seda is a blocking queue for apache camel. default queue size is 1000 .you can increase or decrease this size.
from("aws-s3://helloBucket?accessKey=yourAccessKey&secretKey=yourSecretKey")
.to("seda:awsquue");
from("seda:awsquue?concurrentConsumers=3&queue=#NonLimitQueue").
// do something
// for quarkus it is
#ApplicationScoped
public class ConnectionConf {
#Named("NonLimitQueue")
#Produces
public BlockingQueue arrayDeque(){
return new ArrayBlockingQueue(30000);
}
}

What seems to be happening is Camel trying to read "qqqq" as an Exchange... Camel routes can only handle Exchange-objects and not arbitrary data-structures. What you should do is place the stream you wish to use in the body of an Exchange and send that exchange to the queue.
Something to the effect of this:
camelContext.addRoutes(new S3RouteBuilder(sourceId));
Exchange exchange = new DefaultExchange(camelContext);
exchange.getIn().setBody("qqqq");
queue.add(exchange);

Related

concurrentConsumers not created right away from beginning

I am using Camel in a Spring-Boot application to route from AMQ-Queue. Messages from this queue will be sent to a REST-Webservice. It is already working with this code line:
from("amq:queue:MyQueue").process("jmsToHttpProcessor").to(uri);
My uri looks like this:
http4://localhost:28010/application/createCustomer
Now I have the requirement that the routing to the Webservice should be done parallely:
In order to achive that, I configured concurrentConsumers in JmsConfiguration as follows:
#Bean
public JmsComponent amq(#Qualifier("amqConnectionFactory") ConnectionFactory amqConnectionFactory, AMQProperties amqProperties) {
JmsConfiguration jmsConfiguration = new JmsConfiguration(amqConnectionFactory);
jmsConfiguration.setConcurrentConsumers(50);
jmsConfiguration.setMaxConcurrentConsumers(50);
return new JmsComponent(jmsConfiguration);
}
#Bean
public ConnectionFactory amqConnectionFactory(AMQProperties amqProperties) throws Exception {
ConnectionFactoryParser parser = new ConnectionFactoryParser();
ConnectionFactory returnValue = parser.newObject(parser.expandURI(amqProperties.getUrl()), "amqConnectionFactory");
return returnValue;
}
It is working as expected, BUT not right away from the beginning. I have the phenomenon:
I have 100 messages in the ActiveMQ queue
I start my Spring application
Camel creates only 1 thread consuming 1 message after the previous one gets response
I observe that the amount of messages in queue only decreasing slowly(99.... 98... 97... 96...)
I am filling the queue with new 100 messages
NOW the concurrent consumers are being created as I can observe that the messages decreasing rapidly.
Does someone have any idea, why the concurrentConsumers is not working right away from the beginning?
I tried the advices. Unfortunately they dont change the behaviour. I found out, that the problem is that Camel already starts consuming the messages from the queue before the Spring boot application is startet. I can observe this from the log:
2021-04-01T20:26:33,901 INFO (Camel (CamelBridgeContext) thread #592 - JmsConsumer[MyQueue]) [message]; ...
2021-04-01T20:26:33,902 INFO (Camel (CamelBridgeContext) thread #592 - JmsConsumer[MyQueue]) [message]; ...
2021-04-01T20:26:33,915 INFO (main) [AbstractConnector]; _; Started ServerConnector#5833f5cd{HTTP/1.1,[http/1.1]}{0.0.0.0:23500}
2021-04-01T20:26:33,920 INFO (main) [BridgeWsApplication]; _; Started BridgeWsApplication in 12.53 seconds (JVM running for 13.429)
In this case, only one consumer with thread #592 is consuming all the messages.
In fact, if I start my Spring application first, and then fill the queue with messages, then concurrentConsumers will be used:
2021-04-01T20:30:20,159 INFO (Camel (CamelBridgeContext) thread #594 - JmsConsumer[MyQueue])
2021-04-01T20:30:20,159 INFO (Camel (CamelBridgeContext) thread #599 - JmsConsumer[MyQueue])
2021-04-01T20:30:20,178 INFO (Camel (CamelBridgeContext) thread #593 - JmsConsumer[MyQueue])
2021-04-01T20:30:20,204 INFO (Camel (CamelBridgeContext) thread #564 - JmsConsumer[MyQueue])
In this case, messages are being consumed from concurrentConsumers parallely.
In order to solve the problem, I tried setting autoStartUp to false in my RouteBuilder component:
#Override
public void configure() {
CamelContext context = getContext();
context.setAutoStartup(false);
// My Route
}
In my naive thinking, I let Camel starting after the Spring boot is started and running:
public static void main(String[] args) {
ConfigurableApplicationContext context = SpringApplication.run(BridgeWsApplication.class, args);
SpringCamelContext camel = (SpringCamelContext) context.getBean("camelContext");
camel.start();
try {
camel.startAllRoutes();
} catch (Exception e) {
e.printStackTrace();
}
}
Unfortunately, this does not change the behaviour. There must be a configuration to let Camel starts after Spring is started.

Polling byte / large messages from ActiveMQ Artemis using Apache Camel ConsumerTemplate

I'm struggling with a problem in an application based on Apache Camel when connecting to ActiveMQ Artemis via JMS. At the end of one of the Camel routes, messages are stored in an Artemis JMS queue. A legacy component running in the same application picks them up from there periodically using a ConsumerTemplate.
This works fine for Camel messages with plain text bodies, but causes errors when using byte array bodies: It seems Artemis treats any message with byte body as a "large message", which are streamed instead of kept in memory. Receiving via the ConsumerTemplate works, but as soon as the body or headers are accessed, an exception as follows is raised:
org.apache.camel.RuntimeCamelException: Failed to extract body due to: javax.jms.IllegalStateException: AMQ119023: The large message lost connection with its session, either because of a rollback or a closed session. Message: ActiveMQMessage[ID:90c4d1d5-3233-11ea-b0cc-44032c68a56f]:PERSISTENT/ClientLargeMessageImpl[messageID=2974, durable=true, address=mytest,userID=90c4d1d5-3233-11ea-b0cc-44032c68a56f,properties=TypedProperties[firedTime=Wed Jan 08 17:26:03 CET 2020,__AMQ_CID=90b4f34e-3233-11ea-b0cc-44032c68a56f,breadcrumbId=ID-NB045-evolit-co-at-1578500762151-0-1,_AMQ_ROUTING_TYPE=1,_AMQ_LARGE_SIZE=3]]
at org.apache.camel.component.jms.JmsBinding.extractBodyFromJms(JmsBinding.java:172) ~[camel-jms-2.22.1.jar:2.22.1]
at org.apache.camel.component.jms.JmsMessage.createBody(JmsMessage.java:221) ~[camel-jms-2.22.1.jar:2.22.1]
at org.apache.camel.impl.MessageSupport.getBody(MessageSupport.java:54) ~[camel-core-2.22.1.jar:2.22.1]
at org.apache.camel.example.cdi.JmsPoller.someMethod(JmsPoller.java:36) ~[classes/:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_171]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_171]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_171]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_171]
at org.apache.camel.component.bean.MethodInfo.invoke(MethodInfo.java:481) ~[camel-core-2.22.1.jar:2.22.1]
at org.apache.camel.component.bean.MethodInfo$1.doProceed(MethodInfo.java:300) ~[camel-core-2.22.1.jar:2.22.1]
at org.apache.camel.component.bean.MethodInfo$1.proceed(MethodInfo.java:273) ~[camel-core-2.22.1.jar:2.22.1]
at org.apache.camel.component.bean.AbstractBeanProcessor.process(AbstractBeanProcessor.java:188) ~[camel-core-2.22.1.jar:2.22.1]
at org.apache.camel.component.bean.BeanProcessor.process(BeanProcessor.java:53) ~[camel-core-2.22.1.jar:2.22.1]
at org.apache.camel.component.bean.BeanProducer.process(BeanProducer.java:41) ~[camel-core-2.22.1.jar:2.22.1]
at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:148) ~[camel-core-2.22.1.jar:2.22.1]
at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548) [camel-core-2.22.1.jar:2.22.1]
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201) [camel-core-2.22.1.jar:2.22.1]
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201) [camel-core-2.22.1.jar:2.22.1]
at org.apache.camel.component.timer.TimerConsumer.sendTimerExchange(TimerConsumer.java:197) [camel-core-2.22.1.jar:2.22.1]
at org.apache.camel.component.timer.TimerConsumer$1.run(TimerConsumer.java:79) [camel-core-2.22.1.jar:2.22.1]
at java.util.TimerThread.mainLoop(Timer.java:555) [?:1.8.0_171]
at java.util.TimerThread.run(Timer.java:505) [?:1.8.0_171]
Caused by: javax.jms.IllegalStateException: AMQ119023: The large message lost connection with its session, either because of a rollback or a closed session
at org.apache.activemq.artemis.core.client.impl.LargeMessageControllerImpl.saveBuffer(LargeMessageControllerImpl.java:273) ~[artemis-core-client-2.6.2.jar:2.6.2]
at org.apache.activemq.artemis.core.client.impl.ClientLargeMessageImpl.saveToOutputStream(ClientLargeMessageImpl.java:115) ~[artemis-core-client-2.6.2.jar:2.6.2]
at org.apache.activemq.artemis.jms.client.ActiveMQMessage.saveToOutputStream(ActiveMQMessage.java:853) ~[artemis-jms-client-2.6.2.jar:2.6.2]
at org.apache.activemq.artemis.jms.client.ActiveMQMessage.setObjectProperty(ActiveMQMessage.java:693) ~[artemis-jms-client-2.6.2.jar:2.6.2]
at org.apache.camel.component.jms.JmsBinding.createByteArrayFromBytesMessage(JmsBinding.java:251) ~[camel-jms-2.22.1.jar:2.22.1]
at org.apache.camel.component.jms.JmsBinding.extractBodyFromJms(JmsBinding.java:163) ~[camel-jms-2.22.1.jar:2.22.1]
... 21 more
Caused by: org.apache.activemq.artemis.api.core.ActiveMQIllegalStateException: AMQ119023: The large message lost connection with its session, either because of a rollback or a closed session
at org.apache.activemq.artemis.core.client.impl.LargeMessageControllerImpl.saveBuffer(LargeMessageControllerImpl.java:273) ~[artemis-core-client-2.6.2.jar:2.6.2]
at org.apache.activemq.artemis.core.client.impl.ClientLargeMessageImpl.saveToOutputStream(ClientLargeMessageImpl.java:115) ~[artemis-core-client-2.6.2.jar:2.6.2]
at org.apache.activemq.artemis.jms.client.ActiveMQMessage.saveToOutputStream(ActiveMQMessage.java:853) ~[artemis-jms-client-2.6.2.jar:2.6.2]
at org.apache.activemq.artemis.jms.client.ActiveMQMessage.setObjectProperty(ActiveMQMessage.java:693) ~[artemis-jms-client-2.6.2.jar:2.6.2]
at org.apache.camel.component.jms.JmsBinding.createByteArrayFromBytesMessage(JmsBinding.java:251) ~[camel-jms-2.22.1.jar:2.22.1]
at org.apache.camel.component.jms.JmsBinding.extractBodyFromJms(JmsBinding.java:163) ~[camel-jms-2.22.1.jar:2.22.1]
... 21 more
The problem also occurs for messages that do not exceed the minLargeMessageSize of Artemis, in a test program even for 3 bytes.
Coincidentally, the same problem occurred in a standalone application used for testing the application. There, I was able to solve the issue by keeping the JMS session and receiver open until the JMS message body and headers were completely read. With Camel, that's abstracted away in the Spring JmsTemplate that Camel is based on.
I consulted the user documentation of the Camel JMS component to find configuration options that might help me. I've tried the following:
eagerLoadingOfProperties=true on consumer side: no effect, only seems to affect MessageListenerContainer. The documentation says:
It uses [...] Spring’s JmsTemplate for sending and a MessageListenerContainer for consuming.
However, while debugging it seemed that MessageListenerContainer is only used when consuming messages from an JMS endpoint in a Camel route. Using a ConsumerTemplate like in my case uses a JmsTemplate for consuming.
messageConverter and mapJmsMessage on consumer side: no effect, they are executed when the session has already been closed
alwaysCopyMessage on producer side: I thought maybe copying prevents use of streamed large messages, no effect
streamMessageTypeEnabled=false on producer side: no effect
jmsMessageType=Bytes on both producer and consumer side: no effect
transferExchange=true on both producer and consumer side: this does seem to solve my specific case, but it feels like a workaround. Documentation advises to use the option with caution.
So right now, transferExchange seems to be my best bet, assuming it really solves my issue in all test cases. Nevertheless, I'd be glad to get better understanding on the issue or different solutions:
Why does Artemis treat small byte array messages as large messages anyway?
Does Camel ConsumerTemplate support streamed large messages at all?
My versions are Camel 2.22.1 and Artemis 2.10.1.
I've been able to reproduce my problem by modifying the Camel Example camel-example-cdi from the release package of Camel to have the minimal classes shown below.
In addition I've added camel-jms and Artemis dependencies and started Artemis locally, both like described in the camel-example-artemis-large-messages example.
public class MyRoutes extends RouteBuilder {
#Override
public void configure() {
setupJmsComponent();
from("timer:writeTimer?period=6000")
.log("writing to JMS")
.setBody(() -> new byte[]{0,1,2})
.to(JmsPoller.ENDPOINT);
from("timer:pollTimer?period=3000")
.to("bean:jmsPoller");
}
private void setupJmsComponent() {
ActiveMQJMSConnectionFactory connectionFactory = new ActiveMQJMSConnectionFactory("tcp://localhost:61616");
JmsComponent jmsComponent = new JmsComponent();
jmsComponent.setConnectionFactory(connectionFactory);
getContext().addComponent("jms", jmsComponent);
}
}
#Singleton
#Named("jmsPoller")
public class JmsPoller {
static final String ENDPOINT = "jms:queue:mytest";
#Inject
private ConsumerTemplate consumerTemplate;
public void someMethod(String body) {
Exchange exchange = consumerTemplate.receive(ENDPOINT, 1000L);
System.out.println("Received " + (exchange == null ? null : exchange.getIn().getBody()));
}
}
ActiveMQ Artemis doesn't treat just any message with a byte body as a "large" message. It's worth noting that the broker ultimately treats all message bodies as an array of bytes because that's exactly what they are. However, in order to be considered "large" the message has to exceed a certain size. The documentation states:
Any message larger than a certain size is considered a large message. Large messages will be split up and sent in fragments. This is determined by the URL parameter minLargeMessageSize.
Note:
Apache ActiveMQ Artemis messages are encoded using 2 bytes per character so if the message data is filled with ASCII characters (which are 1 byte) the size of the resulting Apache ActiveMQ Artemis message would roughly double. This is important when calculating the size of a "large" message as it may appear to be less than the minLargeMessageSize before it is sent, but it then turns into a "large" message once it is encoded.
The default value is 100KiB.
It looks like the application's use-case simply doesn't fit with the semantics of large message support in ActiveMQ Artemis since the session which the message came from is being closed before the message's body is fully received.
Therefore, I recommend that you either keep the session open until the body is read or increase the minLargeMessageSize on the URL of the application which is sending the message so that no messages are ever considered "large." The latter option may result in greater memory usage on the broker since the entire message body will be held in memory at once.

Rollback the message to Dead Letter Queue - Apache Camel

I have set up the Apache camel in which i consumes the message from one queue and do some kind of operation on it and then transfers it to other queue .
Now if the exception comes then i want that it should rollback and then after 6 attempts it to send to dead letter queue , Currently rollback happens 5-6 times but my message is not transferred to dead letter queue .
Here What happens -->
Queue1-->(Consumes)-->Operation(Exception thrown)--> rollback --> again Queue1-->(Consumes) --> Operation(Exception thrown)-->rollback -->... this happens 5-6 times and then my message is lost
I dont know where my message is going and why it is getting lost , and from my Active MQ GUI i can see it is dequeued.
#Bean
public RedeliveryPolicy redeliveryPolicy() {
RedeliveryPolicy redeliveryPolicy = new RedeliveryPolicy();
redeliveryPolicy.setMaximumRedeliveries(2);
redeliveryPolicy.setMaximumRedeliveryDelay(10000);
redeliveryPolicy.setRedeliveryDelay(10000);
return redeliveryPolicy;
}
---------------------Route extends SpringRouteBuilder-------------------
onException(MyException.class)
.markRollbackOnly()
.redeliveryPolicy(redeliveryPolicy)
.useExponentialBackOff()
.handled(true)
from("jms:queue:Queue1")
.process(new Processor(){
public void process(Exchange ex){
throw new RuntimeException();
}
}).to("jms:queue:myQueue)
I assume there are multiple problems.
markRollbackOnly stops the message. After this statement no further routing is done.
That is the reason why your RedeliveryPolicy and the rest of your onException route is completely ignored. You configure 2 redelivery attempts but you write it does 5 (the default redelivery of ActiveMQ).
To fix this, move markRollbackOnly to the end of your onException route
If you consume transacted from your JMS broker the message must not get lost.
Since you lose it in case of an error, there is a problem with your transaction config. Configure the ActiveMQ component of Camel to use local JMS transactions when consuming.
#Bean(name = "activemq")
#ConditionalOnClass(ActiveMQComponent.class)
public ActiveMQComponent activeMQComponent(ConnectionFactory connectionFactory) {
ActiveMQComponent activeMQComponent = new ActiveMQComponent();
activeMQComponent.setConnectionFactory(connectionFactory);
activeMQComponent.setTransacted(true);
activeMQComponent.setLazyCreateTransactionManager(false);
return activeMQComponent;
}
When this is in place, you can in fact remove the onException route because the redelivery is done by the JMS broker, so you have to configure the redelivery settings on your JMS connection. If the configured redelivery is exhausted and the message still produces a rollback, it is moved to the DLQ.
Be aware when using an additional onException route because this is pure Camel. The Camel error handler does NOT redeliver on route level, but on processor level. So if you configure both broker and Camel redelivery, it can multiply them.

StreamCache FileNotFound issues with bigger data in multicast routes

We are using camel 2.13.2 - I have a multicast route with an AggregationStrategy.
And in each multicast branch, we have a custom camel component that returns huge data (around 4 MB) and writes to Stream Cache (Cached Output Stream) and we need to aggregate the data in the multicast (Aggregation Strategy).
In the Aggregation strategy, I need to do XPath evaluation using camel XPathBuilder.
Hence, I try to read the body and convert from StreamCache to byte[] to avoid 'Error during type conversion from type: org.apache.camel.converter.stream.InputStreamCache.' in the XPathBuilder.
When I try to read the body in the beginning of the Aggregation Strategy, I get the following error.
*/tmp/camel/camel-tmp-4e00bf8a-4a42-463a-b046-5ea2d7fc8161/cos6047774870387520936.tmp (No such file or directory), cause: FileNotFoundException:/tmp/camel/camel-tmp-4e00bf8a-4a42-463a-b046-5ea2d7fc8161/cos6047774870387520936.tmp (No such file or directory).
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.(FileInputStream.java:138)
at org.apache.camel.converter.stream.FileInputStreamCache.createInputStream(FileInputStreamCache.java:123) at org.apache.camel.converter.stream.FileInputStreamCache.getInputStream(FileInputStreamCache.java:117)
at org.apache.camel.converter.stream.FileInputStreamCache.writeTo(FileInputStreamCache.java:93)
at org.apache.camel.converter.stream.StreamCacheConverter.convertToByteArray(StreamCacheConverter.java:102)
at com.sap.it.rt.camel.aggregate.strategies.MergeAtXPathAggregationStrategy.convertToByteArray(MergeAtXPathAggregationStrategy.java:169)
at com.sap.it.rt.camel.aggregate.strategies.MergeAtXPathAggregationStrategy.convertToXpathCompatibleType(MergeAtXPathAggregationStrategy.java:161)
*
Following is the line of code where it is throwing an error:
Object body = exchange.getIn().getBody();
if( body instanceof StreamCache){
StreamCache cache = (StreamCache)body;
xml = new String(convertToByteArray(cache,exchange));
exchange.getIn().setBody(xml);
}
By disabling stream cache to write to file by setting a threshold of 10MB in multicast related routes, we were able to work with the aggregation strategy. But we do not want to do that, as we may have incoming data that maybe bigger.
<camel:camelContext id="multicast_xml_1" streamCache="true">
<camel:properties>
<camel:property key="CamelCachedOutputStreamCipherTransformation" value="RC4"/>
<camel:property key="CamelCachedOutputStreamThreshold" value="100000000"/>
</camel:properties>
....
</camel:camelContext>
Note: The FileNotFound issue does not appear if we have the StreamCache based camel component in the route with other processors, but without Multicast + Aggregation.
After debugging, I could understand the issue with aggregating huge data from StreamCache with MulticastProcessor.
In MulticastProcessor.java: doProcessParallel() is called and on completion of the branch exchange of multicast, the CachedOutputStream deletes / cleans up the temporary file.
This happens even before the multicast branch exchange reaches the aggregation Strategy, which tries to read the data from the branch exchange. In case of huge data in StreamCache, the temporary file is already deleted, leading to FileNotFound issues.
public CachedOutputStream(Exchange exchange, boolean closedOnCompletion) {
this.strategy = exchange.getContext().getStreamCachingStrategy();
currentStream = new CachedByteArrayOutputStream(strategy.getBufferSize());
if (closedOnCompletion) {
// add on completion so we can cleanup after the exchange is done such as deleting temporary files
exchange.addOnCompletion(new SynchronizationAdapter() {
#Override
public void onDone(Exchange exchange) {
try {
if (fileInputStreamCache != null) {
fileInputStreamCache.close();
}
close();
} catch (Exception e) {
LOG.warn("Error deleting temporary cache file: " + tempFile, e);
}
}
#Override
public String toString() {
return "OnCompletion[CachedOutputStream]";
}
});
}
}
public void close() throws IOException {
currentStream.close();
cleanUpTempFile();
}
I was able to circumvent the issue, if I try to set closedOnCompletion= false, while writing to CachedOutputStream in any component in any Multicast branch.
But this is a leaky solution, because the streamcache temporary file(s) may then never get cleaned up... hence I try to close + clean up the cachestream, after reading the data in the AggregationStrategy.
Can the MulticastProcessor be adjusted so that the multicast branch exchanges reach 'completion' status only, after they have been aggregated at the end of multicast?
Please help / advise on the issue, as I am new to using camel Multicast.
Thanks,
Lakshmi
I have similar exception thrown when trying to send larger than 1MB JSON response to Restlet request (yes, I know 1MB JSON is too big):
java.io.FileNotFoundException: C:\Users\me\AppData\Local\Temp\camel\camel-tmp-7ad6e098-538d-4d4c-9357-2b7addb1f19d\cos6725022584818060586.tmp (The system cannot find the file specified)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(FileInputStream.java:146)
at org.apache.camel.converter.stream.FileInputStreamCache.createInputStream(FileInputStreamCache.java:123)
at org.apache.camel.converter.stream.FileInputStreamCache.getInputStream(FileInputStreamCache.java:117)
at org.apache.camel.converter.stream.FileInputStreamCache.read(FileInputStreamCache.java:112)
at java.io.InputStream.read(InputStream.java:170)
at java.io.InputStream.read(InputStream.java:101)
at org.restlet.engine.io.BioUtils.copy(BioUtils.java:81)
at org.restlet.representation.InputRepresentation.write(InputRepresentation.java:148)
at org.restlet.engine.adapter.ServerCall.writeResponseBody(ServerCall.java:510)
at org.restlet.engine.adapter.ServerCall.sendResponse(ServerCall.java:454)
at org.restlet.ext.servlet.internal.ServletCall.sendResponse(ServletCall.java:426)
at org.restlet.engine.adapter.ServerAdapter.commit(ServerAdapter.java:196)
at org.restlet.engine.adapter.HttpServerHelper.handle(HttpServerHelper.java:153)
at org.restlet.ext.servlet.ServerServlet.service(ServerServlet.java:1089)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1496)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:102)
Same workaround works for me:
getContext().getProperties().put(CachedOutputStream.THRESHOLD, "" + THREE_MEGABYTE_TRESHOLD_BEFORE_FILE_CACHE);
I don't use multicast in this route, just plain
restlet request -> Service -> Jackson marshall => error
I use Camel 2.14.0 & Restlet 2.2.2 with JDK 7 and Spring-boot 1.0.2 / Jetty
This Camel reverse proxy - no response stream caching might be related to my issue.

Camel Stopping With No Apparent Reason

I have a bean producer and a bean consumer, used in a single route. The producer is spawned via a thread and listens for data on an hazelcast queue (it could be anything else, even randomly generated data locally I believe).
The data are sent to a seda endpoint, to ensure concurrency.
The consumer gets data and forwards it to another hazelcast queue. But again it could be anything else.
It works well but after a while, Camel shuts down and I can't find why.
Here are some of the messages I see:
Processing a lot of data...
[ main] MainSupport INFO Apache Camel 2.10.3 stopping
[ main] DefaultCamelContext INFO Apache Camel 2.10.3 (CamelContext: camel-1) is shutting down
[ main] DefaultShutdownStrategy INFO Starting to graceful shutdown 1 routes (timeout 300 seconds)
[el-1) thread #2 - ShutdownTask] DefaultShutdownStrategy INFO Waiting as there are still 1 inflight and pending exchanges to complete, timeout in 300 seconds.
then processing still during 300 seconds and stopping.
Here some of the code:
Producer:
public void run()
{
try
{
IRequest service = ProxyHelper.createProxy(context.getEndpoint("seda:echo"), IRequest.class);
BlockingQueue<Request> q = client.getQueue(MainApp.sQueueReceive);
while(true)
{
Request request;
request = q.take();
// no response awaited
service.request(request);
}
}
Consumer:
public void onMessage(Request request)
{
nb_forwarded++;
BlockingQueue<Request> q = MainApp.client.getQueue(MainApp.sQueueForward);
try
{
q.put(request);
}
catch (InterruptedException e)
{
exit(2); --> it does not happen
}
And finally, the route:
from("seda:echo")
.setExchangePattern(ExchangePattern.InOnly)
.bean(new HazelcastForwarder(), "onMessage");
It's in InOnly as no response is awaited from the producer, it is just a forward.
So why Camel is stopping. There is no message appart from those saying that it is stopping. Is there such a default behaviour in Camel. In which cases?
Thanks!
Enable DEBUG or Trace logging to reveil the true reason why camel is stopping. It can be that the enclosing container is stopping (if you are running camel inside something) or similar.
I was facing the similar issue, where Camel Context is closing immediately after starting the process. I am posting here so that it would also help others with similar issue.
In my case, I am using Spring for loading the Camel context using 'FileSystemXmlApplicationContext' and instantiating it with in try block,
try(AbstractXmlApplicationContext appContext = new FileSystemXmlApplicationContext(camelContextPath)) {
}
as my Eclipse was complaining about Resource leak. So, as soon as the call was coming out of the try/catch it was closing the Spring context, which again closing the Camel Context.
To fix the issue need to initialize Spring context out side the try block.
AbstractXmlApplicationContext appContext = null;
try {
appContext = new FileSystemXmlApplicationContext(camelContextPath);
}

Resources