How to consume messages from a Topic ActiveMQ Artemis - apache-camel

I'm trying to work with topics on ActiveMQ Artemis.
I have created a Multicast Address and a Multicast Queue inside this Address.
Created 2 routes with Apache Camel to connect in this Topic, but when I post message only one Route consume the message and when I post another message, the secont Route that consume this message message.
Below the code and the output.
public class CamelRoutes {
public static void main(String[] args) throws Exception {
ActiveMQJMSConnectionFactory connection = new ActiveMQJMSConnectionFactory("tcp://localhost:61616", "admin", "admin");
CamelContext camel = new DefaultCamelContext();
camel.addComponent("amq", JmsComponent.jmsComponent(connection));
camel.addRoutes(new RouteBuilder(){
#Override
public void configure() throws Exception {
from("amq:TEST.TOPIC")
.routeId("Route1")
.log("ROUTE1: ${body}");
}
});
camel.addRoutes(new RouteBuilder(){
#Override
public void configure() throws Exception {
from("amq:TEST.TOPIC")
.routeId("Route2")
.log("ROUTE2: ${body}");
}
});
camel.start();
Thread.sleep(20000000);
}
}
2019-02-11 16:35:42 [Camel (camel-1) thread #1 - JmsConsumer[TEST.TOPIC]] INFO Route1:159 - ROUTE1: {"message":1}
2019-02-11 16:35:45 [Camel (camel-1) thread #2 - JmsConsumer[TEST.TOPIC]] INFO Route2:159 - ROUTE2: {"message":2}
2019-02-11 16:35:48 [Camel (camel-1) thread #1 - JmsConsumer[TEST.TOPIC]] INFO Route1:159 - ROUTE1: {"message":3}
2019-02-11 16:35:51 [Camel (camel-1) thread #2 - JmsConsumer[TEST.TOPIC]] INFO Route2:159 - ROUTE2: {"message":4}
2019-02-11 16:35:54 [Camel (camel-1) thread #1 - JmsConsumer[TEST.TOPIC]] INFO Route1:159 - ROUTE1: {"message":5}

You are consuming from the queue, not from the topic.
You need to correct your consumer's URI scheme.
Change your consumer to:
from("amq:topic:TEST.TOPIC");
This is how you can create queue consumer :
from("amq:queue:YOUR.QUEUE.NAME);
// or as queue is default value
from("amq:YOUR.QUEUE.NAME);
This is how you can create topic consumer :
from("amq:topic.YOUR.TOPIC.NAME);

Related

QuarkusIntegrationTest does not allow two tests running sequentially: application closed in 2nd test by Camel SimpleMainShutdownStrategy

I have a strange error: I can only run one Quarkus integration test in my mvn verify step. When I have two integration tests, the second one runs but immediately stops the application so nothing is happening and building hangs forever.
The logs are different in two tests. For the 1st one, the logs are:
2023-01-04 15:43:04,560 INFO [org.apa.cam.imp.eng.AbstractCamelContext] (main) Routes startup (total:8 started:8)
2023-01-04 15:43:04,560 INFO [org.apa.cam.imp.eng.AbstractCamelContext] (main) Started route1 (activemq://queue:myqueue)
2023-01-04 15:43:04,560 INFO [org.apa.cam.imp.eng.AbstractCamelContext] (main) Started route2 (activemq://queue:myqueue.online)
2023-01-04 15:43:04,560 INFO [org.apa.cam.imp.eng.AbstractCamelContext] (main) Started main (direct://main)
2023-01-04 15:43:04,560 INFO [org.apa.cam.imp.eng.AbstractCamelContext] (main) Started individual-endpoint (direct://individual-endpoint)
2023-01-04 15:43:04,560 INFO [org.apa.cam.imp.eng.AbstractCamelContext] (main) Started send-and-retry (direct://send-and-retry)
2023-01-04 15:43:04,560 INFO [org.apa.cam.imp.eng.AbstractCamelContext] (main) Started mapping-response (direct://mapping-response)
2023-01-04 15:43:04,560 INFO [org.apa.cam.imp.eng.AbstractCamelContext] (main) Started reply-queue (direct://reply-queue)
2023-01-04 15:43:04,560 INFO [org.apa.cam.imp.eng.AbstractCamelContext] (main) Started route3 (direct://receive.ack.response)
2023-01-04 15:43:04,561 INFO [org.apa.cam.imp.eng.AbstractCamelContext] (main) Apache Camel 3.14.1 (camel-1) started in 593ms (build:0ms init:206ms start:387ms)
2023-01-04 15:43:04,562 DEBUG [org.apa.cam.imp.DefaultCamelContext] (main) start() took 594 millis
2023-01-04 15:43:04,567 DEBUG [org.apa.cam.mai.SimpleMainShutdownStrategy] (camel-main) Await shutdown to complete
2023-01-04 15:43:04,583 INFO [com.example.MyLogger] (main) Application is up
2023-01-04 15:43:04,583 INFO [com.example.MyLogger] (main) Application is up
2023-01-04 15:43:04,767 INFO [com.example.ShutdownController] (main) Setting pre shutdown sleep time to 10 seconds.
2023-01-04 15:43:04,767 INFO [io.quarkus] (main) my-app 0.0.1-SNAPSHOT on JVM (powered by Quarkus 2.7.5.Final) started in 3.124s. Listening on: http://0.0.0.0:8081
2023-01-04 15:43:04,768 INFO [io.quarkus] (main) Profile integration activated.
...(application runs normally; sends a message to in memory AMQ broker and Camel starts consuming from the queue)
And the 2nd test runs with these logs:
2023-01-04 15:43:59,606 INFO [org.apa.cam.imp.eng.AbstractCamelContext] (main) Routes startup (total:8 started:8)
2023-01-04 15:43:59,606 INFO [org.apa.cam.imp.eng.AbstractCamelContext] (main) Started route1 (activemq://queue:myqueue)
2023-01-04 15:43:59,606 INFO [org.apa.cam.imp.eng.AbstractCamelContext] (main) Started route2 (activemq://queue:myqueue.online)
2023-01-04 15:43:59,606 INFO [org.apa.cam.imp.eng.AbstractCamelContext] (main) Started main (direct://main)
2023-01-04 15:43:59,606 INFO [org.apa.cam.imp.eng.AbstractCamelContext] (main) Started individual-endpoint (direct://individual-endpoint)
2023-01-04 15:43:59,606 INFO [org.apa.cam.imp.eng.AbstractCamelContext] (main) Started send-and-retry (direct://send-and-retry)
2023-01-04 15:43:59,606 INFO [org.apa.cam.imp.eng.AbstractCamelContext] (main) Started mapping-response (direct://mapping-response)
2023-01-04 15:43:59,606 INFO [org.apa.cam.imp.eng.AbstractCamelContext] (main) Started reply-queue (direct://reply-queue)
2023-01-04 15:43:59,606 INFO [org.apa.cam.imp.eng.AbstractCamelContext] (main) Started route3 (direct://receive.ack.response)
2023-01-04 15:43:59,606 INFO [org.apa.cam.imp.eng.AbstractCamelContext] (main) Apache Camel 3.14.1 (camel-1) started in 452ms (build:0ms init:202ms start:250ms)
2023-01-04 15:43:59,607 DEBUG [org.apa.cam.imp.DefaultCamelContext] (main) start() took 453 millis
2023-01-04 15:43:59,631 INFO [com.example.MyLogger] (main) Application is up
2023-01-04 15:43:59,631 INFO [com.example.MyLogger] (main) Application is up
2023-01-04 15:43:59,617 DEBUG [org.apa.cam.mai.SimpleMainShutdownStrategy] (camel-main) Await shutdown to complete
2023-01-04 15:43:59,756 INFO [com.example.MyLogger] (main) Application is down
2023-01-04 15:43:59,756 INFO [com.example.MyLogger] (main) Application is down
2023-01-04 15:43:59,757 DEBUG [org.apa.cam.mai.SimpleMainShutdownStrategy] (main) Shutdown called
2023-01-04 15:43:59,757 DEBUG [org.apa.cam.mai.MainLifecycleStrategy] (main) CamelContext: camel-1 is stopping, triggering shutdown of the JVM.
The class ShutdownController listens for Quarkus shutdown event.
public class ShutdownController implements ShutdownListener {
static final Logger LOGGER = LoggerFactory.getLogger(ShutdownController.class);
private final long preShutdownSleep;
public ShutdownController() {
this.preShutdownSleep = (Long)ConfigProvider.getConfig().getOptionalValue("com.example.shutdown-controller.pre-shutdown-sleep", Long.class).orElse(ProfileManager.getLaunchMode() == LaunchMode.TEST ? 0L : 10L);
LOGGER.info("Setting pre shutdown sleep time to {} seconds.", this.preShutdownSleep);
}
public ShutdownController(final int preShutdownSleep) {
this.preShutdownSleep = (long)preShutdownSleep;
}
public void preShutdown(final ShutdownNotification notification) {
LOGGER.info("Pre shutdown received. Waiting fully functionally for {} seconds.", this.preShutdownSleep);
this.sleep();
LOGGER.info("Continuing shutdown");
notification.done();
}
private void sleep() {
try {
TimeUnit.SECONDS.sleep(this.preShutdownSleep);
} catch (InterruptedException var2) {
Thread.currentThread().interrupt();
}
}
}
Notice that Camel SimpleMainShutdownStrategy is called but in the second test, it does not wait for 10s. Why?
I see this part in SimpleMainShutdownStrategy class:
#Override
public boolean isRunAllowed() {
return !completed.get();
}
#Override
public void addShutdownListener(ShutdownEventListener listener) {
listeners.add(listener);
}
#Override
public boolean shutdown() {
if (completed.compareAndSet(false, true)) {
LOG.debug("Shutdown called");
latch.countDown();
for (ShutdownEventListener l : listeners) {
try {
LOG.trace("ShutdownEventListener: {}", l);
l.onShutdown();
} catch (Throwable e) {
// ignore as we must continue
LOG.debug("Error during ShutdownEventListener: {}. This exception is ignored.", l, e);
}
}
return true;
}
return false;
}
#Override
public void await() throws InterruptedException {
LOG.debug("Await shutdown to complete");
latch.await();
}
So maybe it is because in the 1st case, ShutdownController has no instance so the constructor is called, and wait for 10s; but in the 2nd test, the instance is created so it does not call constructor. But why shutdown is called before the integration test runs?
In both cases, Quarkus app starts before Camel.
Both tests have similar structure:
#QuarkusIntegrationTest
#QuarkusTestResource(value = WiremockResource.class, restrictToAnnotatedClass = true)
#QuarkusTestResource(value = InMemoryAMQResource.class, restrictToAnnotatedClass = true)
#TestProfile(MyIntegrationTestProfile.class)
class MyOutboundMessageFormatIT extends MyIntegrationTestSupport { // the base class have a lot of fields like connetion, session, etc., which are used in #BeforeAll and #BeforeEach, etc.
private static final Logger LOG = Logger.getLogger(MyOutboundMessageFormatIT.class);
#InjectInMemoryAMQ
protected BrokerService brokerService;
#InjectWiremock
protected WireMockServer wireMockServer;
#Override
WireMockServer getWiremock() {
return wireMockServer;
}
#Override
BrokerService getBroker() {
return brokerService;
}
#BeforeAll
protected static void start() throws Exception {
LOG.debug("Connecting to AMQ");
connectToAMQ();
}
#AfterAll
protected static void stop() throws Exception {
LOG.debug("Stopping AMQ");
session.close();
connection.stop();
connection.close();
}
#BeforeEach
protected void stub() {
// global stubs
stubAllFieldsOK();
stubOauthOK();
}
#AfterEach
protected void clearWiremockRequestsJournal() throws JMSException {
// clear reply queue for next test
consumeOnlineReply();
assertEmptyNonOnlineReply();
waitTillTokenExpires();
wireMockServer.resetRequests();
}
...
At last, at the end of the log I see the cause:
2023-01-05 14:13:52,530 INFO [org.apa.cam.imp.eng.AbstractCamelContext] (main) Apache Camel 3.14.1 (camel-1) shutdown in 942ms (uptime:1s341ms)
2023-01-05 14:13:52,549 ERROR [io.qua.run.Application] (main) Port 8081 seems to be in use by another process. Quarkus may already be running or the port is used by another application.
2023-01-05 14:13:52,549 WARN [io.qua.run.Application] (main) Use 'netstat -anop | grep 8081' to identify the process occupying the port.
2023-01-05 14:13:52,549 WARN [io.qua.run.Application] (main) You can try to kill it with 'kill -9 <pid>'.
So actually there cannot be 2 integration tests running; seems Quarkus application can only start once and will occupy the port.
At last, I solved this by allowing duplicate message in my ActiveMQ in memory broker URL, and put all cases into one test.
The original issue is that I was reusing the JSON message body between tests, and somehow ActiveMQ is ignoring the message, so no message entered the queue, so 2nd test was not running; also, changing stubbing out of Wiremock resource class seems impossible(at first I let Wiremock return 200, but in the last test I want to simulate 404 case), that's why I wanted to separate the tests.
Now, I use an id to mark the stub I want to change, and in test I call wiremockServer.editStub() to change the stub, also I stop tests from running in parallel with #TestMethodOrder(value = MethodOrderer.MethodName.class), then things start to work. So no dividing into 2 test classes is needed now.

concurrentConsumers not created right away from beginning

I am using Camel in a Spring-Boot application to route from AMQ-Queue. Messages from this queue will be sent to a REST-Webservice. It is already working with this code line:
from("amq:queue:MyQueue").process("jmsToHttpProcessor").to(uri);
My uri looks like this:
http4://localhost:28010/application/createCustomer
Now I have the requirement that the routing to the Webservice should be done parallely:
In order to achive that, I configured concurrentConsumers in JmsConfiguration as follows:
#Bean
public JmsComponent amq(#Qualifier("amqConnectionFactory") ConnectionFactory amqConnectionFactory, AMQProperties amqProperties) {
JmsConfiguration jmsConfiguration = new JmsConfiguration(amqConnectionFactory);
jmsConfiguration.setConcurrentConsumers(50);
jmsConfiguration.setMaxConcurrentConsumers(50);
return new JmsComponent(jmsConfiguration);
}
#Bean
public ConnectionFactory amqConnectionFactory(AMQProperties amqProperties) throws Exception {
ConnectionFactoryParser parser = new ConnectionFactoryParser();
ConnectionFactory returnValue = parser.newObject(parser.expandURI(amqProperties.getUrl()), "amqConnectionFactory");
return returnValue;
}
It is working as expected, BUT not right away from the beginning. I have the phenomenon:
I have 100 messages in the ActiveMQ queue
I start my Spring application
Camel creates only 1 thread consuming 1 message after the previous one gets response
I observe that the amount of messages in queue only decreasing slowly(99.... 98... 97... 96...)
I am filling the queue with new 100 messages
NOW the concurrent consumers are being created as I can observe that the messages decreasing rapidly.
Does someone have any idea, why the concurrentConsumers is not working right away from the beginning?
I tried the advices. Unfortunately they dont change the behaviour. I found out, that the problem is that Camel already starts consuming the messages from the queue before the Spring boot application is startet. I can observe this from the log:
2021-04-01T20:26:33,901 INFO (Camel (CamelBridgeContext) thread #592 - JmsConsumer[MyQueue]) [message]; ...
2021-04-01T20:26:33,902 INFO (Camel (CamelBridgeContext) thread #592 - JmsConsumer[MyQueue]) [message]; ...
2021-04-01T20:26:33,915 INFO (main) [AbstractConnector]; _; Started ServerConnector#5833f5cd{HTTP/1.1,[http/1.1]}{0.0.0.0:23500}
2021-04-01T20:26:33,920 INFO (main) [BridgeWsApplication]; _; Started BridgeWsApplication in 12.53 seconds (JVM running for 13.429)
In this case, only one consumer with thread #592 is consuming all the messages.
In fact, if I start my Spring application first, and then fill the queue with messages, then concurrentConsumers will be used:
2021-04-01T20:30:20,159 INFO (Camel (CamelBridgeContext) thread #594 - JmsConsumer[MyQueue])
2021-04-01T20:30:20,159 INFO (Camel (CamelBridgeContext) thread #599 - JmsConsumer[MyQueue])
2021-04-01T20:30:20,178 INFO (Camel (CamelBridgeContext) thread #593 - JmsConsumer[MyQueue])
2021-04-01T20:30:20,204 INFO (Camel (CamelBridgeContext) thread #564 - JmsConsumer[MyQueue])
In this case, messages are being consumed from concurrentConsumers parallely.
In order to solve the problem, I tried setting autoStartUp to false in my RouteBuilder component:
#Override
public void configure() {
CamelContext context = getContext();
context.setAutoStartup(false);
// My Route
}
In my naive thinking, I let Camel starting after the Spring boot is started and running:
public static void main(String[] args) {
ConfigurableApplicationContext context = SpringApplication.run(BridgeWsApplication.class, args);
SpringCamelContext camel = (SpringCamelContext) context.getBean("camelContext");
camel.start();
try {
camel.startAllRoutes();
} catch (Exception e) {
e.printStackTrace();
}
}
Unfortunately, this does not change the behaviour. There must be a configuration to let Camel starts after Spring is started.

Getting Status Code: 500 errors when accessing localstack Kinesis stream with Camel

I'm receiving 500 status code errors when connecting with Camel to my localstack Kinesis stream. Accessing and putting msgs to the stream via awslocal works fine. So it's probably not a localstack issue.
Localstack
# Start localstack
DEBUG=1 SERVICES=kinesis AWS_CBOR_DISABLE=true CBOR_ENABLED=false localstack start
# Setup stream
awslocal kinesis create-stream --shard-count 4 --stream-name mystream
Java Config (I'm using Quarkus)
public void configure() {
BasicAWSCredentials awsCreds = new BasicAWSCredentials(mystream.getAccessKeyId(), mystream.getSecretKey());
AmazonKinesis client = AmazonKinesisClientBuilder.standard()
.withCredentials(new AWSStaticCredentialsProvider(awsCreds))
.withEndpointConfiguration(
new AwsClientBuilder.EndpointConfiguration(mystream.getServiceEndpoint(), mystream.getRegion()))
.build();
getContext().setTracing(true);
getContext().getRegistry().bind("mystreamClient", client);
from("aws-kinesis://mystream?amazonKinesisClient=mystreamClient&bridgeErrorHandler=true&shard-id=1").routeId("mystream1").log("Received mystream");
}
Error msg:
ERROR [org.apa.cam.pro.err.DefaultErrorHandler] (Camel (camel-1) thread #0 - aws-kinesis://mystream) Failed delivery for (MessageId: 76F94B329B1EE4F-0000000000000002 on ExchangeId: 76F94B329B1EE4F-0000000000000002). Exhausted after delivery attempt: 1 caught: com.amazonaws.services.kinesis.model.AmazonKinesisException: null (Service: AmazonKinesis; Status Code: 500; Error Code: null; Request ID: null)

Apache Camel upload email attachment to ftp

I'm using the following processor to fetch the first attachement of an email and upload it to a ftp-server.
Route configuration
<from uri="imaps://...
<to uri="ejb:java:global/Dms/MailProcessor"/>
<to uri="ftp://....
MailProcessor
#Named("MailProcessor")
#Stateless
public class MailProcessor implements Processor {
#Override
public void process(Exchange exchange) throws Exception {
exchange.getOut().setHeaders(exchange.getIn().getHeaders());
Map<String, DataHandler> attachments = exchange.getIn().getAttachments();
if (attachments.size() > 0) {
for (String name : attachments.keySet()) {
DataHandler dataHandler = attachments.get(name);
// SET ATTACHMENT FILENAME TO OUTPUT FILENAME HEADER
String filename = dataHandler.getName();
filename = MimeUtility.decodeText(filename);
exchange.getOut().setHeader("filename", filename);
// SET INPUT ATTACHMENT TO OUTPUT BODY
byte[] data = exchange.getContext().getTypeConverter().convertTo(byte[].class, dataHandler.getInputStream());
exchange.getOut().setBody(data);
// SET ONLY THE FIRST ATTACHMENT
break;
}
}else{
exchange.getOut().setBody(exchange.getIn().getBody());
}
}
This works in general, but it takes literally forever for "big" attachments. (half an hour for just a 5 mb attachment)
Log
TRACE [org.apache.camel.component.file.remote.FtpOperations] (Camel (example) thread #119 - imaps://mail.example.com) Changing directory: upload
TRACE [org.apache.camel.component.file.remote.FtpOperations] (Camel (example) thread #119 - imaps://mail.example.com) doStoreFile(ID-example-local-59752-1494841993139-21-11)
DEBUG [org.apache.camel.component.file.remote.FtpOperations] (Camel (example) thread #119 - imaps://mail.example.com) About to store file: ID-example-local-59752-1494841993139-21-11 using stream: com.sun.mail.util.BASE64DecoderStream#601a11dc
TRACE [org.apache.camel.component.file.remote.FtpOperations] (Camel (example) thread #119 - imaps://mail.example.com) Client storeFile: ID-example-local-59752-1494841993139-21-11
-- long pause --
I've also tried to convert the attachment like this:
byte[] data = exchange.getContext().getTypeConverter().convertTo( byte[].class, dataHandler.getInputStream() );
exchange.getOut().setBody(data);
but this gives me the following:
2017-05-15 11:23:20,968 TRACE [org.apache.camel.impl.converter.DefaultTypeConverter] (Camel (example) thread #133 - imaps://mail.example.com) Converting org.apache.camel.Processor$$$view63 -> org.apache.camel.Processor with value: Proxy for view class: org.apache.camel.Processor of EJB: MailProcessor
2017-05-15 11:23:20,968 TRACE [org.apache.camel.component.bean.BeanProcessor] (Camel (example) thread #133 - imaps://mail.example.com) Using a custom adapter as bean invocation: Proxy for view class: org.apache.camel.Processor of EJB: MailProcessor
2017-05-15 11:23:20,990 TRACE [org.apache.camel.impl.converter.DefaultTypeConverter] (Camel (example) thread #133 - imaps://mail.example.com) Converting com.sun.mail.util.BASE64DecoderStream -> byte[] with value: com.sun.mail.util.BASE64DecoderStream#28442828
2017-05-15 11:23:20,990 TRACE [org.apache.camel.impl.converter.DefaultTypeConverter] (Camel (example) thread #133 - imaps://mail.example.com) Using converter: StaticMethodTypeConverter: public static byte[] org.apache.camel.converter.IOConverter.toBytes(java.io.InputStream) throws java.io.IOException to convert [class com.sun.mail.util.BASE64DecoderStream=>class [B]
2017-05-15 11:23:20,990 TRACE [org.apache.camel.util.IOHelper] (Camel (example) thread #133 - imaps://mail.example.com) Copying InputStream: java.io.BufferedInputStream#24b2cffb -> OutputStream: with buffer: 4096 and flush on each write false
-- long pause --
Uploading to the ftp server using Filezilla works like a charm. Also when I use camel-ftp in another route ( triggered by a file upload ) the upload to the ftp server works pretty fast.
So I have a feeling that it's the conversion of the attachment which slows things down.
Questions:
Is my assumption correct and how can I speed things up?
Solution was to use <from uri="imaps://...&mail.imaps.partialfetch=false.
https://community.oracle.com/thread/2604843?start=0&tstart=0

Unable to process large files in camel

I am trying to do simple transformation on a Csv file.But my programm is getting stuck and not giving any output and on console its printing something like below.
22:38:02.001 [main] INFO o.a.camel.impl.DefaultCamelContext - Apache Camel 2.15.2 (CamelContext: camel-1) is shutting down
22:38:02.135 [main] INFO o.a.c.impl.DefaultShutdownStrategy - Starting to graceful shutdown 1 routes (timeout 300 seconds)
22:38:02.167 [main] DEBUG o.a.c.i.DefaultExecutorServiceManager - Created new ThreadPool for source: org.apache.camel.impl.DefaultShutdownStrategy#65ead16a with name: ShutdownTask. -> org.apache.camel.util.concurrent.RejectableThreadPoolExecutor#52c0a65f[Running, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0][ShutdownTask]
22:38:02.173 [Camel (camel-1) thread #1 - ShutdownTask] DEBUG o.a.c.impl.DefaultShutdownStrategy - There are 1 routes to shutdown
22:38:02.177 [Camel (camel-1) thread #1 - ShutdownTask] DEBUG o.a.c.impl.DefaultShutdownStrategy - Route: route1 suspended and shutdown deferred, was consuming from: Endpoint[file:///home/cloudera/Desktop/camelinput/?delay=15m&noop=true]
22:38:02.177 [Camel (camel-1) thread #1 - ShutdownTask] INFO o.a.c.impl.DefaultShutdownStrategy - Waiting as there are still 2 inflight and pending exchanges to complete, timeout in 300 seconds.
22:38:02.179 [Camel (camel-1) thread #1 - ShutdownTask] DEBUG o.a.c.impl.DefaultShutdownStrategy - There are 1 inflight exchanges:
InflightExchange: [exchangeId=ID-quickstart-cloudera-40574-1441345060577-0-2, fromRouteId=route1, routeId=route1, nodeId=unmarshal1, elapsed=10787, duration=10791]
22:38:05.436 [Camel (camel-1) thread #1 - ShutdownTask] INFO o.a.c.impl.DefaultShutdownStrategy - Waiting as there are still 2 inflight and pending exchanges to complete, timeout in 299 seconds.
22:38:05.437 [Camel (camel-1) thread #1 - ShutdownTask] DEBUG o.a.c.impl.DefaultShutdownStrategy - There are 1 inflight exchanges:
InflightExchange: [exchangeId=ID-quickstart-cloudera-40574-1441345060577-0-2, fromRouteId=route1, routeId=route1, nodeId=unmarshal1, elapsed=14045, duration=14049]
22:38:08.201 [Camel (camel-1) thread #1 - ShutdownTask] INFO o.a.c.impl.DefaultShutdownStrategy - Waiting as there are still 2 inflight and pending exchanges to complete, timeout in 298 seconds.
22:38:08.202 [Camel (camel-1) thread #1 - ShutdownTask] DEBUG o.a.c.impl.DefaultShutdownStrategy - There are 1 inflight exchanges:
InflightExchange: [exchangeId=ID-quickstart-cloudera-40574-1441345060577-0-2, fromRouteId=route1, routeId=route1, nodeId=unmarshal1, elapsed=16810, duration=16814]
Actually the same program worked for small file but when I try to do with large file I am getting this issue.I think it may problem with Threads .Please Help me out to figure out the issue.
Following is my Program
Main Class
TestRouter myRoute = new TestRouter();
HDFSTransfer hdfsTransfer = new HDFSTransfer();
String copy = hdfsTransfer.copyToLocal(
"hdfs://localhost:8020",
"/user/cloudera/input/CamelTestIn.csv",
"/home/cloudera/Desktop/camelinput/");
boolean flag = false;
if ("SUCCESS".equals(copy)) {
myContext.addRoutes(myRoute);
// Launching the context
myContext.start();
// Pausing to let the route do its work
Thread.sleep(10000);
myContext.stop();
flag = true;
}
if (flag) {
hdfsTransfer.moveFile(
"hdfs://localhost:8020",
"file:/home/cloudera/Desktop/camelout/out.csv",
"/user/cloudera/output/");
}
RouterBuilder Class
{
CsvDataFormat csv = new CsvDataFormat();
from("file:/home/cloudera/Desktop/camelinput/?noop=true&delay=15m")
.unmarshal(csv)
.convertBodyTo(List.class)
.process(new Processor() {
#Override
public void process(Exchange msg) throws Exception {
List<List<String>> data = (List<List<String>>) msg.getIn().getBody();
for (List<String> line : data) {
// Checks if column two contains text STANDARD
// and alters its value to DELUXE.
// System.out.println("line "+line);
/*
if("Aug-04".equalsIgnoreCase(line.get(6))){
line.set(6, "04-August");}
*/
}
}
}).marshal(csv)
.to("file:/home/cloudera/Desktop/camelout/?fileName=out.csv")
.log("done.").end();
}
If you have a bigger file then you need to sleep for longer than 10 seconds to let it have time to process the file.
Also mind that your current route reads the file into memory when means you can run out of memory if the file is very big.
See the lazyLoad option on: http://camel.apache.org/csv.html
Also if all your route is doing is to change some text in a big file, then there is better and faster ways doing that than maybe a Camel route.

Resources