To some extent, this is a bit of a shot in the dark, but we have a process that dramatically slows down over the course of a day. We've found everything running on Fuse begins to drag, but only when we've been running a specific process. Running JProfiler, I found there to be a memory usage increase over time marked on org.apache.camel.ProducreTemplate.send.
So my main question is, is there something I'm missing with the way we are using the ProducerTemplate here that is incorrect/could be causing this issue?
Exchange foo = new DefaultExchange(getCamelContext(), ExchangePattern.InOnly);
foo.getIn().setBody(obj);
Route r = exchange.getContext().getRoute("do_something_fun");
ProducerTemplate template = exchange.getContext().createProducerTemplate();
template.send(r.getEndpoint(), foo);
Normally you shouldn't create a ProducerTemplate on each request as is described here: http://camel.apache.org/why-does-camel-use-too-many-threads-with-producertemplate.html
However, because I don't have the complete picture of your application you could have a situation where you cannot reuse it but then you must remember to close it when you're done with it.
Related
I have a x86 application working on windows10 (64 bit environment).
One pf the app's features is to generate a lot of reports, so there is a lot of printing involved.
However, I noticed that every time I try to use call DefaultPrintTicket on the print queue the dllhost process (COM Surrogate) grows in memory.
I managed to isolate the code responsible and moved it to a test WPF app. When a button is clicked this code is being fired:
var localPrintServer = new LocalPrintServer();
var oneNotePrintQueue = localPrintServer.GetPrintQueues().FirstOrDefault(p => p.Description.Contains(OneNote));
var printTicket = oneNotePrintQueue?.DefaultPrintTicket;
The printing queue is irrelevant as I tried them all and the problem remains.
I am aware that this might be a duplicate to: PrintTicket DllHost.exe Memory Climbs
However, the solution provided there does not work as PrintTicked is not an IDisposable object.
I also tried some tweaks in the registry (i.e. finding AppId AA0B85DA-FDDF-4272-8D1D-FF9B966D75B0 and removing "AccessPermission", "LaunchPermission" and "RunAs") with no result.
I cannot rebuild the app as AnyCpu and I would like to avoid creating a separate 64bit process just for printing as it would be difficult to send a report generated in one app to another.
Any suggestions are greatly appreciated.
It seems that the topic is difficult.
Just want to share the solution I went with in case anyone else has the same problem.
In the end I created a separate x64 app which handles the printing.
Initially I wanted to go with a WCF service. However, I ran into problems with serializing of FixedDocuments and PrintQueue. Hence the separate app.
The solution if far from perfect and in my opinion is not nice at all. However, it solved the memory leak issue.
I have implemented a Source by extending RichSourceFunction for our Message Queue that Flink doesn't support.
When I implements the run method whose signature is:
override def run(sc: SourceFunction.SourceContext[String]): Unit = {
val msg = read_from_mq
sc.collect(msg)
}
When the run method is called, if there is no newer message in message queue,
Should I run without calling sc.collect or
I can wait until newer data comes(in this case, run method will be blocked).
I would prefer the 2nd one,not sure if this is the correct usage.
The run method of a Flink source should loop, endlessly producing output until its cancel method is called. When there's nothing to produce, then it's best if you can find a way to do a blocking wait.
The apache nifi source connector is another reasonable example to use as a model. You will note that it sleeps for a configurable interval when there's nothing for it to do.
As you probably know both options are functionally correct and will yield correct results.
This being said the second one is preferred because you're not holding the thread. In fact, if you take a look at the RabbitMQ connector implementation you'll notice that this exactly how it is implemented: inside its run it indirectly waits for messages to be placed on a BlockingQueue.
I am executing some external processes using camel-exec component.
These might be long running (or for whatever business reason), I would like to be able to kill some inflightExchanges based on some header values.
So far so good. I can filter the exchange(s) I want to kill, I am marking the exchange with e.setProperty(Exchange.ROUTE_STOP, Boolean.TRUE); and removing it from the InfightRepository.
But what I need is to forcibly stop the execution of the current task and stop the exchange from further routing.
What would you propose? Any ideas?
What I did to solve it is the following:
Created two caches anexecCommandCache<aKey, org.apache.camel.component.exec.ExecCommand> and a processCache<org.apache.camel.component.exec.ExecCommand, java.lang.Process>. Whenever a new executable is started, I am adding it in these caches.
When I need to cancel an executable I am browsing the InflightRepository and filter on the aKey Exchange header. Then I retrieve the ExecCommand from the execCommandCache and and with this I can find the actual Process from processCache.
Finally I am using Zeroturnaround zt-process-killer to kill the Process. and marking the exchange as done via exchange.setProperty(Exchange.ROUTE_STOP, Boolean.TRUE);
ProcessUtil.destroyGracefullyOrForcefullyAndWait(process, 5, TimeUnit.SECONDS, 5, TimeUnit.SECONDS);
It's not very clean but it does the trick. Hope it helps.
I'm using a Splitter to break up a large file, do some processing on the splits and then, with a custom AggregationStrategy, save the updated splits to a new file. The splitter is configured for streaming but not parallel processing. This works fine.
The problem is that the Splitter calls doAggregate (inherited from MulticastProcessor) which is synchronized. When there are concurrent requests on that route, the performance is significantly impacted by the synchronization. Obviously, the more concurrent requests, the worse it is.
Is there a way to create a pool of splitter instances that can be used for the route? Each incoming exchange could use a different splitter instance and thus avoid the synchronized doAggregate call. Is it possible to leverage a custom ProcessorFactory to do this?
Update:
I created a simple test to demonstrate what I'm talking about.
I have a route like this
from("direct:splitterTest").split(method(new MySplitter(), "rowIterator"), new MyAggregationStrategy()).to("log:someSplitProcessing?groupSize=500")
The MySplitter simply returns an iterator of 10000 String[] which emulates reading a file.
The MyAggregationStrategy pretends to perform some work and saves the records to a new file.
In my test, I added loop to emulate some processing like
Random random = new Random(System.currentTimeMillis());
for (int i = 0; i < 10000; i++) {
random.nextGaussian();
}
I submit requests to the route like this (not that I'm not passing in a file in this case because the splitter is just returning dummy data):
ProducerTemplate producerTemplate = camelContext.createProducerTemplate();
Future<File> future1 = producerTemplate.asyncRequestBody("direct:splitterTest", null, File.class);
Future<File> future2 = producerTemplate.asyncRequestBody("direct:splitterTest", null, File.class);
System.out.println(future1.get());
System.out.println(future2.get());
I wanted to post visualvm screenshots showing how 2 and 4 concurrent in-flight exchanges are impacted by the synchronization but this account is too new to be allowed to post images.
The point is this. When the route is created, there is a single Splitter instance for that route. Multiple in-flight exchanges on that route will synchronize in the doAggregate call which seriously impacts the processing time for each request. When there are 4 requests, you can see that 3 threads are blocked while one is in the doAggregate call.
Due to the nature of the processing that I'm doing, I'm not able to configure the splitter to process in parallel so what I'm looking for is a way to create multiple splitter instances. I could create, say, 4 copies of the route and then use a routingSlip or dynamicRouter to round robin requests to each but that seems kind of ugly and I am hoping there's a better way.
You can just use your own thread pool to save in parallel if you want from the doAggregate.
I am seeing a very odd memory leak in Camel (2.10.3) when consuming from a Tibco topic (using tibjms 4.4.3 library). From looking at heap dumps it appears that the memory consumption is a huge amount of ConcurrentHashMap stuff (Segment, HashEntry[], locks etc).
What I believe is happening is that the exchanges coming in from the topic are never getting marked as 'complete' by Camel, and it is holding onto references to them in memory. The problem goes away when I route them to '.stop()'.
I create a JMS Component with:
TibjmsConnectionFactory connectionFactory = new TibjmsConnectionFactory();
connectionFactory.setServerUrl(properties.getProperty(endpoints.getServerUrl()));
connectionFactory.setUserName(properties.getProperty(endpoints.getUsername()));
connectionFactory.setUserPassword(properties.getProperty(endpoints.getPassword()));
JmsComponent emsComponent = JmsComponent.jmsComponent(connectionFactory);
return emsComponent;
Register it on the context with:
camelContext.addComponent("positionems", emsComponent);
Then have created an incredibly simple test route just to reproduce the problem:
from("positionems:topic:UK.TOPIC4")
.to("mock:out");
What is interesting is that this will fill up the heap with ConcurrentHashMap stuff until the process falls over with Heap Space errors. BUT it runs fine forever if I change the route to:
from("positionems:topic:UK.TOPIC4")
.stop();
According to the javadoc for stop, it "Stops continue routing the current org.apache.camel.Exchange and marks it as completed." - presumably 'marks it as completed' is what I am missing when I send it to mock (or indeed when I run my full normal program, which behaves the same way memory-wise as sending it to mock).
I have tried lots of variations of the Jms route config, for example:
from("positionems:topic:UK.TOPIC4?disableReplyTo=true&deliveryPersistent=false")
And I have tried setting the route never to expect a response, but maybe I am doing this wrong:
from("positionems:topic:UK.TOPIC4")
.inOnly() // marked as deprecated?
.to("mock:out");
Is this a problem specifically with Tibco? Given the number of people that use ActiveMQ with no issues I find it hard to believe I have found an actual bug in Camel, hopefully I am doing something really simple wrong!
EDIT
I have tested with the latest Camel version (2.12.1) and this seems to be a bit better (number of ConcurrentHashMap Segments grows slower) but is still definitely a problem.
You send the message to "mock:out" endpoint which keeps a copy of the message in memory. So there is your leak :) - What you can do is to either configure the mock endpoint to not retain so many messages (see the documentation), or maybe better send the message to a log endpoint or something.
At the mock documentation there is this big red warning, which tells about the in-memory copy: http://camel.apache.org/mock