When Im sending multithreaded request to a route with throttling sometimes I get NPE exception. if route has no throttling everything is fine. Do I need to sycnhronize some where when using throttling in multithreaded environment ?
route ends with success:
from("direct:start").log("executed ${body}").to("mock:result");
route with throttling ends with NPE sometimes:
from("direct:start").throttle(method(ThrottlingTester.class, "getMessagesPerSecond")).timePeriodMillis(1000).rejectExecution(true).log("executed ${body}").to("mock:result");
Code sending requests:
MockEndpoint resultEndpoint = resolveMandatoryEndpoint("mock:result", MockEndpoint.class);
resultEndpoint.expectedMessageCount(6);
resultEndpoint.setResultWaitTime(3000);
for (int i = 0; i < 3; i++) {
new Thread(new Runnable() {
#Override
public void run() {
template.sendBody("direct:start", "<test1> </test1>");
}
}).start();
}
for (int i = 0; i < 5; i++) {
new Thread(new Runnable() {
#Override
public void run() {
template.sendBody("direct:start", "<test2> </test2>");
}
}).start();
}
resultEndpoint.assertIsSatisfied();
Code calculates throttling value:
public synchronized static long getMessagesPerSecond(Exchange exchange){
if(exchange.getIn().getBody(String.class).indexOf("test2")>0)
return 1000;
return 1;
}
Error :
Caused by: java.lang.NullPointerException
at org.apache.camel.processor.Throttler$TimeSlot.access$002(Throttler.java:158)
at org.apache.camel.processor.Throttler.calculateDelay(Throttler.java:123)
at org.apache.camel.processor.DelayProcessorSupport.process(DelayProcessorSupport.java:156)
Using Camel version 2.15.3
Related
I have client like this :
import org.basex.api.client.ClientSession;
#Slf4j
#Component(value = "baseXAircrewClient")
#DependsOn(value = "baseXAircrewServer")
public class BaseXAircrewClient {
#Value("${basex.server.host}")
private String basexServerHost;
#Value("${basex.server.port}")
private int basexServerPort;
#Value("${basex.admin.password}")
private String basexAdminPassword;
#Getter
private ClientSession session;
#PostConstruct
private void createClient() throws IOException {
log.info("##### Creating BaseX client session {}", basexServerPort);
this.session = new ClientSession(basexServerHost, basexServerPort, UserText.ADMIN, basexAdminPassword);
}
}
It is a singleton injected in a service which run mulitple queries like this :
Query query = client.getSession().query(finalQuery);
return query.execute();
All threads query and share the same session.
With a single thread all is fine but with multiple thread I get some random (and weird) error, like the result of a query to as a result of another.
I feel that I should put a synchronized(){} arround query.execute() or open and close session for each query, or create a pool of session.
But I don't find any documentation how the use the session in parrallel.
Is this implementation fine for multithreading (and my issue is comming from something else) or should I do it differently ?
I ended creating a simple pool by adding removing the client from a ArrayBlockingQueue and it is working nicely :
#PostConstruct
private void createClient() throws IOException {
log.info("##### Creating BaseX client session {}", basexServerPort);
final int poolSize = 5;
this.resources = new ArrayBlockingQueue < ClientSession > (poolSize) {
{
for (int i = 0; i < poolSize; i++) {
add(initClient());
}
}
};
}
private ClientSession initClient() throws IOException {
ClientSession clientSession = new ClientSession(basexServerHost, basexServerPort, UserText.ADMIN, basexAdminPassword);
return clientSession;
}
public Query query(String finalQuery) throws IOException {
ClientSession clientSession = null;
try {
clientSession = resources.take();
Query result = clientSession.query(finalQuery);
return result;
} catch (InterruptedException e) {
log.error("Error during query execution: " + e.getMessage(), e);
} finally {
if (clientSession != null) {
try {
resources.put(clientSession);
} catch (InterruptedException e) {
log.error("Error adding to pool : " + e.getMessage(), e);
}
}
}
return null;
}
I need to browse messages from an active mq using Camel route without consuming the messages.
The messages in the JMS queue are to be read(only browsed and not consumed) and moved to a database while ensuring that the original queue remains intact.
public class CamelStarter {
private static CamelContext camelContext;
public static void main(String[] args) throws Exception {
camelContext = new DefaultCamelContext();
ConnectionFactory connectionFactory = new ActiveMQConnectionFactory(ActiveMQConnectionFactory.DEFAULT_BROKER_URL);
camelContext.addComponent("jms", JmsComponent.jmsComponent(connectionFactory));
camelContext.addRoutes(new RouteBuilder() {
#Override
public void configure() throws Exception {
from("jms:queue:testQueue").to("browse:orderReceived") .to("jms:queue:testQueue1");
}
}
);
camelContext.start();
Thread.sleep(1000);
inspectReceivedOrders();
camelContext.stop();
}
public static void inspectReceivedOrders() {
BrowsableEndpoint browse = camelContext.getEndpoint("browse:orderReceived", BrowsableEndpoint.class);
List<Exchange> exchanges = browse.getExchanges();
System.out.println("Browsing queue: "+ browse.getEndpointUri() + " size: " + exchanges.size());
for (Exchange exchange : exchanges) {
String payload = exchange.getIn().getBody(String.class);
String msgId = exchange.getIn().getHeader("JMSMessageID", String.class);
System.out.println(msgId + "=" +payload);
}
As far as I know, not possible in Camel to read (without consuming !) JMS messages...
The only workaround I found (in a JEE app) was to define a startup EJB with a timer, holding a QueueBrowser, and delegating the msg processing to a Camel route:
#Singleton
#Startup
public class MyQueueBrowser {
private TimerService timerService;
#Resource(mappedName="java:/jms/queue/com.company.myqueue")
private Queue sourceQueue;
#Inject
#JMSConnectionFactory("java:/ConnectionFactory")
private JMSContext jmsContext;
#Inject
#Uri("direct:readMessage")
private ProducerTemplate camelEndpoint;
#PostConstruct
private void init() {
TimerConfig timerConfig = new TimerConfig(null, false);
ScheduleExpression se = new ScheduleExpression().hour("*").minute("*/"+frequencyInMin);
timerService.createCalendarTimer(se, timerConfig);
}
#Timeout
public void scheduledExecution(Timer timer) throws Exception {
QueueBrowser browser = null;
try {
browser = jmsContext.createBrowser(sourceQueue);
Enumeration<Message> msgs = browser.getEnumeration();
while ( msgs.hasMoreElements() ) {
Message jmsMsg = msgs.nextElement();
// + here: read body and/or properties of jmsMsg
camelEndpoint.sendBodyAndHeader(body, myHeaderName, myHeaderValue);
}
} catch (JMSRuntimeException jmsException) {
...
} finally {
browser.close();
}
}
}
Apache camel browse component is exactly designed for that. Check here for the documentation.
Can't say more since you have not provided any other information.
Let's asssume you have a route like this
from("activemq:somequeue).to("bean:someBean")
or
from("activemq:somequeue).process(exchange -> {})
All you got to do it put a browse endpoint in between like this
from("activemq:somequeue).to("browse:someHandler").to("bean:someBean")
Then write a class like this
#Component
public class BrowseQueue {
#Autowired
CamelContext camelContext;
public void inspect() {
BrowsableEndpoint browse = camelContext.getEndpoint("browse:someHandler", BrowsableEndpoint.class);
List<Exchange> exchanges = browse.getExchanges();
for (Exchange exchange : exchanges) {
......
}
}
}
I was trwing to construct a synchronous route with timeout using Apache Camel, and I couldn't find anything in the framework with resolve it.
So I decided to build a process with make it for me.
public class TimeOutProcessor implements Processor {
private String route;
private Integer timeout;
public TimeOutProcessor(String route, Integer timeout) {
this.route = route;
this.timeout = timeout;
}
#Override
public void process(Exchange exchange) throws Exception {
ExecutorService executor = Executors.newSingleThreadExecutor();
Future<Exchange> future = executor.submit(new Callable<Exchange>() {
public Exchange call() {
// Check for field rating
ProducerTemplate producerTemplate = exchange.getFromEndpoint().getCamelContext().createProducerTemplate();
return producerTemplate.send(route, exchange);
}
});
try {
exchange.getIn().setBody(future.get(
timeout,
TimeUnit.SECONDS));
} catch (TimeoutException e) {
throw new TimeoutException("a timeout problem occurred");
}
executor.shutdownNow();
}
And I call this process this way:
.process(new TimeOutProcessor("direct:myRoute",
Integer.valueOf(this.getContext().resolvePropertyPlaceholders("{{timeout}}")))
I wanna know if my way is the recomended way to do it, if it is not, what is the best way for build a synchronous route with timeout?
I want to thanks the people who answer me.
That is my final code:
public class TimeOutProcessor implements Processor {
private String route;
private Integer timeout;
public TimeOutProcessor(String route, Integer timeout) {
this.route = route;
this.timeout = timeout;
}
#Override
public void process(Exchange exchange) throws Exception {
Future<Exchange> future = null;
ProducerTemplate producerTemplate = exchange.getFromEndpoint().getCamelContext().createProducerTemplate();
try {
future = producerTemplate.asyncSend(route, exchange);
exchange.getIn().setBody(future.get(
timeout,
TimeUnit.SECONDS));
producerTemplate.stop();
future.cancel(true);
} catch (TimeoutException e) {
producerTemplate.stop();
future.cancel(true);
throw new TimeoutException("a timeout problem occurred");
}
}
}
Very old project I have 'inherited', Apache 2.2.0 and apparently has never 'worked right'.
The process seems to 'lock up' occasionally and never completes. If the process is killed, files moved back to inbound, and restarted it seems to work. So it's a weird intermittent problem.
I believe the issue may be due to a custom RoutePolicy implementation;
package com.company.integration.management;
import org.apache.camel.CamelContext;
import org.apache.camel.Endpoint;
import org.apache.camel.Exchange;
import org.apache.camel.Route;
import org.apache.camel.SuspendableService;
import org.apache.camel.component.seda.SedaEndpoint;
import org.apache.camel.impl.ThrottlingInflightRoutePolicy;
import org.apache.log4j.Logger;
public class ParserInflightRoutePolicy extends ThrottlingInflightRoutePolicy {
private static Route fileRoute;
private static final int MAX_FILES_INFLIGHT = 3;
private static final int MSG_HIGH_WATERMARK = 100;
private static final int MSG_LOW_WATERMARK = 25;
private Logger logger = Logger.getLogger(getClass());
#Override
public void onExchangeBegin(Route route, Exchange exchange) {
int sedaMessagesInflight = 0;
sedaMessagesInflight = sedaQueueSize(route);
if (route.getId().equalsIgnoreCase("initialRoute")) {
fileRoute = route;
try {
if ((sedaQueueSize(route, "seda://parser") >= MAX_FILES_INFLIGHT) || (sedaMessagesInflight >= MSG_HIGH_WATERMARK)) {
if ( ! ((SuspendableService)fileRoute.getConsumer()).isSuspended() ) {
logger.error("FLOW CONTROL: stop file component");
((SuspendableService)fileRoute.getConsumer()).suspend();
}
}
}
catch (Exception ex) {
logger.error("Error stopping route", ex);
}
}
if (route.getId().equalsIgnoreCase("pricingRoute") || route.getId().equalsIgnoreCase("persistenceRoute")) {
try {
if ((sedaQueueSize(route, "seda://parser") == 0) && (sedaMessagesInflight <= MSG_LOW_WATERMARK)) {
if ( ((SuspendableService)fileRoute.getConsumer()).isSuspended() ) {
logger.error("FLOW CONTROL: start file component");
((SuspendableService)fileRoute.getConsumer()).resume();
}
}
}
catch (Exception ex) {
logger.error("Error starting route", ex);
}
}
}
int sedaQueueSize(Route route) {
int sedaMessagesInflight = 0;
CamelContext camelContext = route.getRouteContext().getCamelContext();
for (Route rte : camelContext.getRoutes()) {
Endpoint endpoint = rte.getEndpoint();
if (endpoint instanceof SedaEndpoint) {
SedaEndpoint seda = (SedaEndpoint)endpoint;
if (seda.getQueue().size() > 0) {
sedaMessagesInflight += seda.getQueue().size();
}
}
}
logger.debug("SEDA messages inflight [" + sedaMessagesInflight + "]");
return sedaMessagesInflight;
}
int sedaQueueSize(Route route, String endPointUri) {
CamelContext camelContext = route.getRouteContext().getCamelContext();
Endpoint endpoint = camelContext.getEndpoint(endPointUri);
if (endpoint instanceof SedaEndpoint) {
return ((SedaEndpoint) endpoint).getQueue().size();
}
return 0;
}
#Override
public void onExchangeDone(Route route, Exchange exchange) {
;
}
}
Notice that this code does not appear to be threadsafe - no locking mechanism as implemented in ThottlingInflightRoutePolicy. Could this be causing the intermittent issue?
During the processing of an Exchange received from JMS I'm creating dynamically a route that fetches a file from FTP to the file system and when the batch is done I need to remove that same route. The following code fragment shows how I do this:
public void execute() {
try {
context.addRoutes(createFetchIndexRoute(routeId()));
} catch (Exception e) {
throw Throwables.propagate(e);
}
}
private RouteBuilder createFetchIndexRoute(final String routeId) {
return new RouteBuilder() {
#Override
public void configure() throws Exception {
from("ftp://" + getRemoteQuarterDirectory() +
"?fileName=" + location.getFileName() +
"&binary=true" +
"&localWorkDirectory=" + localWorkDirectory)
.to("file://" + getLocalQuarterDirectory())
.process(new Processor() {
RouteTerminator terminator;
#Override
public void process(Exchange exchange) throws Exception {
if (camelBatchComplete(exchange)) {
terminator = new RouteTerminator(routeId,
exchange.getContext());
terminator.start();
}
}
})
.routeId(routeId);
}
};
}
I'm Using a thread to stop a route from a route, which is an approach recommended in the Camel Documentation - How can I stop a route from a route
public class RouteTerminator extends Thread {
private String routeId;
private CamelContext camelContext;
public RouteTerminator(String routeId, CamelContext camelContext) {
this.routeId = routeId;
this.camelContext = camelContext;
}
#Override
public void run() {
try {
camelContext.stopRoute(routeId);
camelContext.removeRoute(routeId);
} catch (Exception e) {
throw Throwables.propagate(e);
}
}
}
In result the route does stop. But what I see in the jconsole is that the thread that corresponds to the route isn't removed. Thus in time these abandoned threads just keep accumulating.
Is there a way to properly stop/remove a route dynamically/programmatically and also to release the route's thread, so that they don't accumulate through time?
This is fixed in the next Camel release 2.9.2 and 2.10. Fixed by this ticket:
https://issues.apache.org/jira/browse/CAMEL-5072