I'm getting a java.lang.OutOfMemoryError: unable to create new native thread error during transfer of around 20 files using SMB endpoint.
Camel version 2.13.
The route itself is quite simple:
<route id="Filetransfer">
<from uri="ftp://user#server//source/map?password=pwd&include=fileA.*.csv|fileB.*.csv|fileC.*.csv|fileD.*.csv|fileE.*.csv|fileF.*.csv&move=save&consumer.delay=30000" />
<log message="${routeId}: ${header.CamelFileName}" />
<to uri="smb://domain;user#server/target/map?password=pwd"/>
</route>
When I check the number of threads in Hawtio dashboard the thread count is hitting a peak of 1000. The route executes correctly when only some small files are transferred. When some biggger
files (>5Mb, >100.000 lines) are transferred, the route gives the error.
When I replace the SMB endpoint with a FILE endpoint like <to uri="file:///tmp/camel"/> the route also executes correctly, and all the files are transferred.
Splitting the file per line first, and then use the Append option in the SMB endpoint causes the same error.
What can I do to make the SMB endpoint work, regardless the size of the files?
Related
I want old messages to be automatically removed from topic so I'm using the option timeToLive which is described in the Camel ActiveMQ component documentation. However, it only works with queue and not with topic. Any idea why?
For testing purposes I made this simple route:
<route>
<from uri="timer://foo?repeatCount=1"/>
<to uri="activemq:topic:test?timeToLive=10000"/>
</route>
Route sends message to my test topic with TTL=10seconds. I would expect that after 10 seconds message would disappear.
When I use queue instead everything works as expected ( <to uri="activemq:queue:test?timeToLive=10000"/> )
This is my Camel Route:
<route>
<from uri="file:///c:/"/>
<to uri="file:///D:/"/>
</route>
In case of any failure in this route I want to persistently store files in Kaha DB so that files won't be lost. But am not aware of blueprint.xml configuration of Kaha DB persistence for storing files. And my Activemq.xml file is as follows
<broker brokerName="kahaDB_Persistence" persistent="true" useShutdownHook="false">
<persistenceAdapter>
<kahaDB directory="${data}/kahadb/"
journalMaxFileLength="100mb"
concurrentStoreAndDispatchQueues="false"
concurrentStoreAndDispatchTopics="false"/>
</persistenceAdapter>
</broker>
Please advise me how to connect to this KahaDB from blueprint.xml by considering above mentioned route.
Camel's file component has a built in archive feature that saves files that have been processed. It copies them into a folder called ".camel", but it can be changed with a configuration option.
I would not recommend using KahaDB, as it doesn't fit the "right tool for the job" mantra.
Camel File component docs
My use case is to poll a local directory for a list of new files periodically, and then upload them to a FTP server in 1 connection. The Camel route is defined in Spring XML as follows:
<route>
<from uri="file:data/inbox?noop=true&delay=1000&maxMessagesPerPoll=3" />
<to uri="ftp:uid:xxxxx#host:21/data?disconnect=false"/>
</route>
The route is functioning well, except that the FTP connection will retain connected until the FTP server timeout my connection. I hope to reuse the same connection to upload a batch of files and then close the connection immediately when the last file in the batch completed the upload. How can achieve this in Camel?
This is not possible currently. You will need to write some code to do the disconnect yourself.
You are welcome to log a JIRA to enhance this in camel-ftp: https://issues.apache.org/activemq/browse/CAMEL. For example a new option to disconnectOnBatchComplete.
There might be a way but it is not pretty.
You could try to wrap your route based on a cronSchedulePolicy. So say you kick start the route once every hour and poll the directory and send the files. Then you simply add a stop(). Not sure if the stop is exactly the same in the xml dsl. Alternatively, you could also write that onExchangeComplete(new Processor(StopProcessor(routeId)) and inside that processor you via exchange.getContext.stopRoute(routeid) stop the route. Again this depends on your requirements allow you to do this.
<route>
<from uri="file:data/inbox?noop=true&delay=1000&maxMessagesPerPoll=3" />
<to uri="ftp:uid:xxxxx#host:21/data?disconnect=false"/>
<stop/>
</route>
I currently have a polling html consumer that polls a http endpoint every 5 seconds. This works fine.
I need to now support a http endpoint that has SSE as the transport mode. The server will push every 3 seconds or so and I was hoping that camel could somehow handle this. In this case I am the consumer and I have the Server end point I wish to consumer from.
Any thoughts how I could do this?
thanks
You could certainly do this in Camel/Servlet as SSE is an HTTP protocol with a specific content-type, so you'd need something like this in your route.
<setHeader headerName="Content-Type">
<simple>text/event-stream</simple>
</setHeader>
You'd have to be careful about scale though as I imagine you'd get a new thread for each connection.
What's the best strategy to send SMS via SMPP with Camel ? Should I use the ProducerTemplate ?
I'm new to camel so I'm not confident if my strategy is the best.
In my application upon reception of an SMS, I have to send back an other SMS with some computed content.
I created a
route smsIn that looks like this
from "uri=smpp ..."
unmarshal ref="bindyDataFormat"
to "uri=bean:myBean
and a route smsOut with
from "uri=direct:smsOut"
to "uri=smpp ..."
The smsIn route, receives the sms, transforms its conent (csv data) in a pojo and send that pojo to myBean.
In myBean I do some processing and then call a ProducerTemplate which send my computed message to the endpoint "direct:smsOut".
The reason I use the producerTemplate is that I have to set some info from my pojo in the header (CamelSmppDestAddr) and the body of the Exchange.
I have tested with the logica SMSC simulator, this seems to work fine, but would like to have your opinion about this solution ?
What about reliability , transaction ?
Should I store my message before trying to send it to the SMSC ?
Should I store it in a database, post it to a queue ?
I'm not sure why you have a producer template, you could just build up the route instead (given that you return something from your bean or takes an Exchange as paramter).
<from uri="smpp: ..."/>
<bean ref="bean:myBean"/>
<to uri="jms:queue:myQueue"/>
then not use direct, but use a JMS queue that is transactional and persistent. Say your smpp call fails, the message would have been gone. Using a queue like this and make sure its transactional, you can make sure not to lose data in this stage of the route.
<from uri="jms:queue:myQueue"/>
<transactional/>
<to uri="smpp.."/>
I suggest using Apache ActiveMQ as JMS middleware. Actually, if you download ActiveMQ, you get camel bundled, so you could actually run your Camel routes from ActiveMQ.
You might want to tweak how retries and error handling occurs dependent on what you want to happen (retry every second forever?, retry five times, then put to error queue? etc).
Read this page: Transaction Error handling in Camel
For deeper info and more tweaks, you might also want to read this:
Transactional Client