I can't seem to get the camel-ftp component to die when no files are found.
I added a LimitedPollingConsumerPollStrategy with a limit of 1:
<bean id="noPoll" class="org.apache.camel.impl.LimitedPollingConsumerPollStrategy">
<property name="limit" value="1"/>
</bean>
and configured the URI to use it:
ftp://user#host.ftp/?password=pass&stepwise=false&binary=true&delete=false&noop=true&pollStrategy=#noPoll
It still just hangs, looking for files, when it doesn't find any.. so I added &sendEmptyMessageWhenIdle=true to the URI.
I added conditions to my route to output to log when a message comes through with a null body and I saw a flood of those messages so it seems the limit on the polling consumer isn't working. I tried changing it to &consumer.pollStrategy=#noPoll and it behaved the same.
The following PollStrategy will stop the consumer if no messages are consumed.
public class PollOncePollStrategy extends DefaultPollingConsumerPollStrategy {
#Override
public void commit(Consumer consumer, Endpoint endpoint, int polledMessages) {
try {
if (polledMessages == 0) {
log.info("No polled messages, stopping consumer");
endpoint.getCamelContext().createProducerTemplate().sendBody(String.format("controlbus:route?async=true&action=stop&routeId=%s", EndpointHelper.getRouteIdFromEndpoint(endpoint)), null);
}
} catch (Exception e) {
throw new RuntimeException(e);
}
}
}
Register it in the camel registry and use as follows: ftp://127.0.0.1/mydir?pollStrategy=#pollOnce
The LimitedPollingConsumerPollStrategy is for limiting when the consumer has failed for X number of times in a row. This is also what is explained in the documentation of it. Its not for stopping after 1 poll.
You can implement your own poll strategy that stops when the commit method is called with the parameter polledMessages = 0. Then you know there was no files polled.
Related
I am trying to accomplish a transactional JMS client in Java Spring Boot using Apache Camel, which connects to IBM MQ. Furthermore, the client needs to apply an exponential back-off redelivery behavior when processing of messages fails. Reason: Messages from MQ need to be processed and forwarded to external systems that may be down for maintenance for many hours. Using transactions to guarantee at-least once processing guarantees seems the appropriate solution to me.
I have researched this topic for many hours and have not been able to find a solution. I will start with what I currently have:
#Bean
UserCredentialsConnectionFactoryAdapter uccConnectionFactoryAdapter ()
throws IOException {
MQConnectionFactory factory = new MQConnectionFactory();
factory.setCCDTURL(tabFilePath);
UserCredentialsConnectionFactoryAdapter adapter =
new UserCredentialsConnectionFactoryAdapter();
adapter.setTargetConnectionFactory(factory);
adapter.setUsername(userName);
bentechConnectionFactoryAdapter.setPassword(password);
return adapter;
}
#Bean
PlatformTransactionManager jmsTransactionManager(#Autowired UserCredentialsConnectionFactoryAdapter uccConnectionFactoryAdapter) {
JmsTransactionManager txMgr = new JmsTransactionManager(uccConnectionFactoryAdapter);
return txMgr;
}
#Bean()
CamelContextConfiguration contextConfiguration(#Autowired UserCredentialsConnectionFactoryAdapter uccConnectionFactoryAdapter,
#Qualifier("jmsTransactionManager") #Autowired PlatformTransactionManager txMgr) {
return new CamelContextConfiguration() {
#Override
public void beforeApplicationStart(CamelContext context) {
JmsComponent jmsComponent = JmsComponent.jmsComponentTransacted(uccConnectionFactoryAdapter, txMgr);
// required for consumer-level redelivery after rollback
jmsComponent.setCacheLevelName("CACHE_CONSUMER");
jmsComponent.setTransacted(true);
jmsComponent.getConfiguration().setConcurrentConsumers(1);
context.addComponent("jms", jmsComponent);
}
#Override
public void afterApplicationStart(CamelContext camelContext) {
// Do nothing
}
};
}
// in a route builder
...
from("jms:topic:INPUT_TOPIC?clientId=" + CLIENT_ID + "&subscriptionDurable=true&durableSubscriptionName="+ SUBSCRIPTION_NAME)
.transacted()
.("direct:processMessage");
...
I was able to verify the transactional behavior through integration tests. If an unhandled exception occurs during message processing, the transaction gets rolled back and retried. The problem is, it gets immediately retried, several times per second, causing possibly significant load on the IBM MQ manager and external system.
For ActiveMQ, redelivery policies are easy to do, with plenty of examples on the net. The ActiveMQConnectionFactory has a setRedeliveryPolicy method, meaning, the ActiveMQ client library has redelivery logic built in. This from all I can tell in line with the documentation of Camel's Transactional Client EIP, which states:
The redelivery in transacted mode is not handled by Camel but by the backing system (the transaction manager). In such cases you should resort to the backing system how to configure the redelivery.
What I absolutely can't figure out is how to achieve the same thing for IBM MQ. IBM's MQConnectionFactory does not have any support for redelivery policies. In fact, searching for redeliverypolicy in the MQ Knowledge Center brings up exactly... drumroll... 0 hits. I even looked a bit through the implementation of the MQConnectionFactory and didn't discover anything either.
Another backing system I looked into was the JmsTransactionManager. Searches for "jmstransactionmanager redelivery policy" or "jmstransactionmanager exponential backoff" did not turn up anything useful either. There was some talk about TransactionTemplate and AbstractMessageListenerContainer but 1) I didn't see any connection to redelivery policies, and 2) I could not figure out how those interact with Camel and JMS.
Sooo, does anybody have any idea how to implement exponential backoff redelivery policies with Apache Camel and IBM MQ?
Closing note: Camel supports redelivery policies on errorHandler and onException are not the same as redelivery policies in the transaction/connection backing system. Those handlers retry at the point of failure using the 'Exchange' object in whichever state it is, without rolling back and reprocessing the message from the start of the route. The transaction remains active during entire rety period, and a rollback only occurs when the errorHandler or onException gives up. This is not what I want for retries that may go on for many hours.
Looks like #JoshMc pointed me in the right direction. I managed to implement a RoutePolicy that delays redeliveries with increasing delays. I have run a test session for a few hours and several thousand redeliveries of the same message to see if there are any problems like memory leak, MQ connection exhaustion or so. I did not observe any problems. There were two stable TCP connections to the MQ manager, and memory usage of the Java process moved within a close range.
import java.util.Timer;
import java.util.TimerTask;
import javax.jms.Session;
import lombok.extern.log4j.Log4j2;
import org.apache.camel.CamelContext;
import org.apache.camel.CamelContextAware;
import org.apache.camel.Exchange;
import org.apache.camel.Message;
import org.apache.camel.Route;
import org.apache.camel.component.jms.JmsMessage;
import org.apache.camel.support.RoutePolicySupport;
#Log4j2
public class ExponentialBackoffPolicy extends RoutePolicySupport implements CamelContextAware {
final static String JMSX_DELIVERY_COUNT = "JMSXDeliveryCount";
private CamelContext camelContext;
#Override
public void setCamelContext(CamelContext camelContext) {
this.camelContext = camelContext;
}
#Override
public CamelContext getCamelContext() {
return this.camelContext;
}
#Override
public void onExchangeDone(Route route, Exchange exchange) {
try {
// ideally we would check if the exchange is transacted but onExchangeDone is called after the
// transaction is already rolled back, and the transaction context has already been removed.
if (exchange.getException() == null)
{
log.debug("No exception occurred, skipping route suspension.");
return;
}
int deliveryCount = getRetryCount(exchange);
int redeliveryDelay = getRedeliveryDelay(deliveryCount);
log.info("Suspending route {} for {}ms after exception. Current delivery count {}.",
route.getId(), redeliveryDelay, deliveryCount);
super.suspendRoute(route);
scheduleWakeup(route, redeliveryDelay);
} catch (Exception ex) {
// only log exception and let Camel continue as of this policy didn't exist.
log.error("Exception while suspending route", ex);
}
}
void scheduleWakeup(Route route, int redeliveryDelay) {
Timer timer = new Timer();
timer.schedule(
new TimerTask() {
#Override
public void run() {
log.info("Resuming route {} after redelivery delay of {}ms.", route.getId(), redeliveryDelay);
try {
resumeRoute(route);
} catch (Exception ex) {
// only log exception and let Camel continue as of this policy didn't exist.
log.error("Exception while resuming route", ex);
}
timer.cancel();
}
},
redeliveryDelay);
}
int getRetryCount(Exchange exchange) {
Message msg = exchange.getIn();
return (int) msg.getHeader(JMSX_DELIVERY_COUNT, 1);
}
int getRedeliveryDelay(int deliveryCount) {
// very crude backoff strategy for now, will need to refine later
if (deliveryCount < 10) return 1000;
if (deliveryCount < 20) return 5000;
if (deliveryCount < 30) return 20000;
return 60000;
}
}
And this is how it being used in route definitions:
from(mqConnectionString)
.routePolicy(new ExponentialBackoffPolicy())
.transacted()
...
// and if you want to distinguish between retriable and non-retriable situations, apply the following two exception handlers
onException(NonRetriableProcessingException.class)
.handled(true)
.log(LoggingLevel.WARN, "Non-retriable exception occurred, discard message.");
onException(Exception.class)
.handled(false)
.log(LoggingLevel.WARN, "Retriable exception occurred, retry message.");
One thing to note is that the JMSXDeliveryCount header comes from the MQ manager, and the redelivery delay is calculated from that. When you restart an application using the ExponentialBackoff policy while a message permanently fails, upon restart it will immediately attempt to reprocess that message but in case of another failure apply a delay corresponding to the total number of redeliveries, and not start over with the initial short delay.
I am trying to create a camel component which consumes an API from an external service.
My Route is as follows
from("myComponent:entity?from=&to=")
.to("seda:one")
from("seda:one")
.aggregate(constant(true), new GroupedBodyAggregationStrategy())
.completionSize(5)
.completionTimeout(5000)
.process( new Processor1() )
to("seda:two")
.
.
.
from("seda:five")
.to("myComponent2:entity")
I implemented my component consumer as follows
public class MyComponentConsumer extends DefaultConsumer {
public MyComponentConsumer(MyComponentEndpoint endpoint, Processor processor) {
super(endpoint, processor);
}
#Override
protected void doStart() throws Exception {
super.doStart();
flag = true;
while ( flag ) {
//external API call
Resource resource = getNextResource();
if ( resource.next() == null ) {
flag = false;
}
Exchange ex = endpoint.createExchange(ExchangePattern.InOnly);
ex.getIn().setBody(resource.toString());
getAsyncProcessor().process(
ex
doneSync -> {
LOG.info("Message processed");
}
);
}
}
#Override
protected void doStop() throws Exception {
super.doStop();
System.out.println("stop ---- ");
}
}
Everything worked fine and the data was propogating through the route. My only problem was that data did not propogate to the next part until the whole of this process was completed. And the next parts were running asynchronously.
I looked at the example of StreamConsumer and tried to implement it to my code using a runnable and an executorService. But if I do that consumer stops as soon as it starts.
I changed the code to
public class MyComponentConsumer extends DefaultConsumer implements Runnable
and added
private ExecutorService executor;
getEndpoint().getCamelContext().getExecutorServiceManager().newSingleThreadExecutor(this, "myComponent");
executor.execute(this);
and moved my logic inside the run() method. But, the consumer thread ends as soon as it starts. and the async processor does not transfer the data properly.
Is there any other way to implement the functionality I need or am I mistaken somewhere here. Any help would be appreciated.
What version of camel are you using?
There was an issue with managing the state of consumer in camel 2.x which was fixed in camel 3.x CAMEL-12765 which can lead to the issue you are describing here.
If you are on camel 2.x try using newScheduledThreadPool instead of newSingleThreadExecutor.
Also executor.schedule(this, 5L, TimeUnit.SECONDS) instead of executor.execute(this).
Delayed start of executor might help avoid the problem you are facing.
I would like to run a route once and stop the context when the route is completed. Currently I do the usual Thread.sleep(3000) in the main Java class to leave some time for the route to finish but it's obviously not accurate, my route may take 1 second or 20 seconds I can not know in advance.
The Java class:
public static void main(String[] args) throws Exception {
try (ClassPathXmlApplicationContext context = new ClassPathXmlApplicationContext("camel-context.xml")) {
CamelContext camelContext = SpringCamelContext.springCamelContext(context);
// context.start(); // apparently not necessary
camelContext.startRoute("route1");
try { Thread.sleep(3000); } catch (InterruptedException e) { e.printStackTrace(); }
// context.stop(); // apparently not necessary
}
}
The Spring xml:
<route id="route1" autoStartup="false">
<from uri="timer://runOnce?repeatCount=1&delay=3000" />
...
</route>
After reading http://camel.465427.n5.nabble.com/Need-control-back-in-the-Main-routine-so-that-we-can-terminate-JVM-td4483312.html#a4484845 especially the 4th post, from Claus Ibsen, I was thinking of using camelContext.getRouteStatus() in a loop with a Thread.sleep() but wherever I try to get the route status in the code (even after the Thread.sleep(3000)), the status is always "started". I don't know any other way to detect when the route is done.
What is the recommended way to stop the Camel context when a/all route(s) is/are completed, using Spring?
The route will never stop because routes do not have complete state. They can just be started, stopped or paused. A route will always be running if it's in the started state unless you do something to change that.
To accomplish what you are looking for, you can do a couple of things:
You can use the controlbus component and stop the route in the last step of your route. That way you can check (for example the way you mentioned checking for camelContext.getRouteStatus()) when you should stop the context as well.
You can write a small Processor that whenever it receives an Exchange it will stop the camelContext. Once ready, you will add it to the last step of your route.
Camel supports onCompletion callbacks, which can be equivalent to the option above. See the camel page.
Probably, the first option is the easiest for your use case, however I would go for the second option. It seems cleaner to me.
A more elegant way would be to use a synchronisation mechanism provided by Java like CountDownLatch. Main thread will wait for the latch to be opened by the Route thread. Something like :
CountDownLatch latch = new CountDownLatch(1);
camelContext.addRoutes(createRoute(latch));
and somewhere in the createRoute Method add a processor at the end of the route to open the latch. This worked perfectly for me.
.process(new Processor() {
#Override
public void process(Exchange exchange) throws Exception {
latch.countDown();
}
});
I am using Camel for my messaging application. In my use case I have a producer (which is RabbitMQ here), and the Consumer is a bean.
from("rabbitmq://127.0.0.1:5672/exDemo?queue=testQueue&username=guest&password=guest&autoAck=false&durable=true&exchangeType=direct&autoDelete=false")
.throttle(100).timePeriodMillis(10000)
.process(new Processor() {
#Override
public void process(Exchange exchange) throws Exception {
MyCustomConsumer.consume(exchange.getIn().getBody())
}
});
Apparently, when autoAck is false, acknowledgement is sent when the process() execution is finished (please correct me if I am wrong here)
Now I don't want to acknowledge when the process() execution is finished, I want to do it at a later stage. I have a BlockingQueue in my MyCustomConsumer where consume() is putting messages, and MyCustomConsumer has different mechanism to process them. I want to acknowledge message only when MyCustomConsumer finishes processing messages from BlockingQueue. How can I achieve this?
You can consider to use the camel AsyncProcessor API to call the callback done once you processing the message from BlockingQueue.
I bumped into the same issue.
The Camel RabbitMQConsumer.RabbitConsumer implementation does
consumer.getProcessor().process(exchange);
long deliveryTag = envelope.getDeliveryTag();
if (!consumer.endpoint.isAutoAck()) {
log.trace("Acknowledging receipt [delivery_tag={}]", deliveryTag);
channel.basicAck(deliveryTag, false);
}
So it's just expecting a synchronous processor.
If you bind this to a seda route for instance, the process method returns immediately and you're pretty much back to the autoAck situation.
My understanding is that we need to make our own RabbitMQ component to do something like
consumer.getAsyncProcessor().process(exchange, new AsynCallback() {
public void done(doneSync) {
if (!consumer.endpoint.isAutoAck()) {
long deliveryTag = envelope.getDeliveryTag();
log.trace("Acknowledging receipt [delivery_tag={}]", deliveryTag);
channel.basicAck(deliveryTag, false);
}
}
});
Even then, the semantics of the "doneSync" parameter is not clear to me. I think it's merely a marker to identify whether we're dealing with a real async processor or a synchronous processor that was automatically wrapped into an async one.
Maybe someone can validate or invalidate this solution?
Is there a lighter/faster/stronger alternative?
Or could this be suggested as the default implementation for the RabbitMQConsumer?
Hope this doesn't sound ridiculous, but how can I discard a message in Camel on purpose?
Until now, I sent them to the Log-Component, but meanwhile I don't even want to log the withdrawal.
Is there a /dev/null Endpoint in Camel?
You can use the message filter eip to filter out unwanted messages.
http://camel.apache.org/message-filter
There is no dev/null, component.
Also there is a < stop /> you can use in the route, and when a message hit that, it will stop continue routing.
And the closest we got on a dev/null, is to route to a log, where you set logLeve=OFF as option.
With credit to my colleague (code name: cayha)...
You can use the Stub Component as a camel endpoint that is equivalent to /dev/null.
e.g.
activemq:route?abc=xyz
becomes
stub:activemq:route?abc=xyz
Although I am not aware of the inner workings of this component (and if there are dangers for memory leaks, etc), it works for me and I can see no drawbacks in doing it this way.
one can put uri/mock-uri to the config using property component
<camelContext ...>
<propertyPlaceholder id="properties" location="ref:myProperties"/>
</camelContext>
// properties
cool.end=mock:result
# cool.end=result
// route
from("direct:start").to("properties:{{cool.end}}");
I'm a little late to the party but you can set a flag on the exchange and use that flag to skip only that message (by calling stop) if it doesn't meet your conditions.
#Override
public void configure() throws Exception {
from()
.process(new Processor() {
#SuppressWarnings("unchecked")
#Override
public void process(Exchange exchange) throws Exception {
exchange.setProperty("skip", false);
byte[] messageBytes = exchange.getIn().getBody(byte[].class);
if (<shouldNotSkip>) {
} else { //skip
exchange.setProperty("skip", true);
}
}
}).choice()
.when(exchangeProperty("skip").isEqualTo(true))
.stop()
.otherwise()
.to();
}
I am using activemq route and needs to send reply in normal cases, so exchange pattern is InOut. When I configure a filter in the route I find that even it does not pass message to next step, the callback is executed(sending reply), just same as the behavior when calling stop(). And it will send the same message back to reply queue, which is not desirable.
What I do is to change the exchange pattern to InOnly conditionally and stop if I want to filter out the message, so reply is not sent. MAIN_ENDPOINT is a direct:main endpoint I defined to include normal business logic.
from("activemq:queue:myqueue" + "?replyToSameDestinationAllowed=true")
.log(LoggingLevel.INFO, "Correlation id is: ${header.JMSCorrelationID}; will ignore if not null")
.choice()
.when(simple("${header.JMSCorrelationID} == null"))
.to(MAIN_ENDPOINT)
.endChoice()
.otherwise()
.setExchangePattern(ExchangePattern.InOnly)
.stop()
.endChoice()
.end();
Note that this message is also consumed and not in the queue anymore. If you want to preserve the message in the queue(not consuming it), you may just stop() or just filter() so the callback(sending reply which is the original message) works, putting the message back to the queue.
Using only filter() would be much simpler:
from("activemq:queue:myqueue" + "?replyToSameDestinationAllowed=true")
.log(LoggingLevel.INFO, "Correlation id is: ${header.JMSCorrelationID}; will ignore if not null")
.filter(simple("${header.JMSCorrelationID} == null"))
.to(MAIN_ENDPOINT);