I have a application which places message in JMS Queue.The message size can be upto 150 mb. JMS message contains byte array data with additional metadata as xml,once the message is placed in queue the camel route application picks the message and calls the external java api with the message.i would like to stream the message from the JMS Queue using camel jms component.
Can you please tell me if i can achieve it using camel.
Currently i am facing heap error issue.
Sample JMS message is as below.
<request>
<esb-metadata>
<source-transaction-id>1234567890</source-transaction-id>
<environment>Dev</environment>
<message-type>REQ</message-type>
<source-id>12345678</source-id>
<master-id>mst-id1234</master-id>
</esb-metadata>
<message-body>
<parameters>
<info>
<document-class>Document</document-class>
<document-guid>idd_C0A5DF59-0000-C51C-BCDA-E76E9476D5C2</document-guid>
</info>
</parameters>
<document>
<document-mime-type>std/text</document-mime-type>
<document-content>SGVhZGluZzEgQ2hhbmdlZCB0b2RheSBKYW4gMTEgc2FtcGxlIHRleHQgYmVsb3csDQpPbmNlIEkg
cmVjZWl2ZSB0aGlzIGluZm9ybWF0aW9uLCBJIHdpbGwgdGhlbiBjb250YWN0IHlvdSBpbW1lZGlh
dGVseSB0byBkaXNjdXNzIHRoZSByZXNvbHV0aW9uIG9mIHlvdXIgY2xhaW0uDQoNCg0KSGVhZGlu
ZzIgc2FtcGxlIHRleHQsDQpJZiBwb3NzaWJsZSwgSSBuZWVkIHRvIHJlY2VpdmUgdGhpcyBpbmZv
cm1hdGlvbiBmcm9tIHlvdSB3aXRoaW4gMzAgRGF5cy4NCg0KMQkxCTENCjIJMgkyDQozCTMJMw0K
NAk0CTQNCg0KSGVhZGluZzMgc2FtcGxlIHRleHQsDQpJIHdpbGwgYXNzdW1lIHlvdSBkbyBub3Qg
d2lzaCB0byBwdXJzdWUgdGhpcyBjbGFpbSBhbmQgSSB3aWxsIGNsb3NlIG15IGZpbGUuDQpBRERF
RCBUSElTIE5FVyBMSU5FIFRPIFNFRSBJRiBJVCBJUyBWSVNJQkxFDQoNCnRhYmxlIHN0YXJ0cyBo
ZXJlDQpwb2xpY3kgbnVtYmVyCWNsYWltIG51bWJlcglpbnN1cmVkDQpwMTIzCWMxMjMJYm9zY2gN
CnAzMjEJYzMyMQlibXcNCnRhYmxlIGVuZHMgaGVyZQ0KDQomIzk2Nzk7CUJ1bGxldCBwb2ludCAx
DQomIzk2Nzk7CUJ1bGxldCBwb2ludCAyDQomIzk2Nzk7CUJ1bGxldCBwb2ludCAzDQoNCjEuCU51
bWJlcmVkIHBvaW50IDENCjIuCU51bWJlcmVkIHBvaW50IDINCjMuCU51bWJlcmVkIHBvaW50IDMN
Cg== </document-content>
</document>
</message-body>
</request>
HornetQ has support for streaming large messages so that clients don't have to worry about hitting memory limits. This support works in the HornetQ JMS implementation, but the JMS API doesn't directly support streaming large messages so a standard JMS component (e.g. one that Camel provides) likely won't work for this use-case. You'd need to use a component written to support the HornetQ implementation specifically. You can read about how HornetQ JMS clients can stream large messages in the documentation.
Related
I have a reliable acquisition flow where an inbound SFTP connector is polling and reading the file and publishing to JMS ActiveMq. I have an issue when ActiveMQ is down. JMS connector get in to reconnection mode lets say every minute but SFTP is actively reading the files and try to publish the file in ActiveMq. which causes the loss of message and the concept of deadletter queue also does not work as JMS is down.
Is there a way we can stop reading the file until the JMS successfully reconnect? also what if the message was in flight and JMS get down before message end up in the queue? will the message rollback?
I am using Mule 3.9
If you set the flow processing strategy to synchronous and use only 1 receiver thread in the connector configuration. Then it should not try to read a new file if the previous one wasn't processed. Without those changes it is not a correct implementation of the reliable pattern.
More information:
Documentation of reliable patterns: https://docs.mulesoft.com/mule-runtime/3.9/reliability-patterns.
KB Article: https://help.mulesoft.com/s/article/How-to-fetch-files-in-order-using-the-FTP-connector
Example:
<sftp:connector name="sftpConn">
<receiver-threading-profile maxThreadsActive="1" />
</sftp:connector>
<flow name="mainFlow" processingStrategy="synchronous">
...
Does Apache Camel ActiveMQ component guarantees delivery message to broker?
If i understand correctly (reading this doc) Camel has persisentDelivery configuration enabled by default for JMS and that guarantees consuming messages from broker.
But i don't understand: how it works on producing from app to broker (if yes, what kind of storage does it use)? If this kind of guarantee not supported by default, does Camel provide a simple way to implement it?
Thanks in advance
No only when the message is acknowledgede by the broker its safely sent to the broker, where its guaranteed. The persistent option just tells the broker to store the message in the storage instead of keeping it in memory only.
So if Camel cannot send the message to the broker due to networking issues etc, then the operation will fail, and you would need to deal with this error in Camel.
What you can do is to have a local ActiveMQ broker alongside your Camel apps and then connect these brokers in a network of brokers where the brokers will route the messages safely among each others.
How can I determine what, if any, ActiveMQ queue a camel route is a consumer of? The route is running as a bundle within Karaf.
You have to define it yourself. Every Camel route starts with a from statement. For ActiveMQ this would look somehow like this
from("activemq:queue:myAwesomeQueue")...
This route would create an ActiveMQ consumer that consumes every message arriving on the myAwesomeQueue.
The connection to the broker is "hidden" behind the activemq:. This is a Camel component (the ActiveMQ component) that needs to be configured to connect to the broker.
EDIT: Add operational perspective
Hawtio is a webconsole that uses Jolokia to get data. Jolokia makes JMX information available through a REST API.
If JMX is enabled, you can get loads of information about the CamelContext and/or ActiveMQ. For example the endpoint of an ActiveMQ consumer as in your case.
Unfortunately I can't upload a screenshot because the image domain of SO is blocked, but Google gives you lots of them.
Some AMQP enabled brokers like Microsoft Service Bus or ActiveMQ allow only one active producer or consumer per session. The Apache JMS Camel component seems not be able to handle this correctly, which result that JMS throws exceptions when processing InOut messages sent from e.g. Service Bus.
A small JMS test application with separate sessions for producer and consumer works nice. With using the same session, it run into the same problem.
Issue is rooted in the Spring JMS template, which is used as base for the Camel JMS implementation.
Does anybody know how to overcome that behavior?
You can define a 2nd JMS component and use that for the "other".
Or you can turn off any kind of connection pooling maybe.
If you use XML then you can define a 2nd jms component
<bean id="jms2" class="org.apache.camel.component.jms.JmsComponent"/>
What's the best strategy to send SMS via SMPP with Camel ? Should I use the ProducerTemplate ?
I'm new to camel so I'm not confident if my strategy is the best.
In my application upon reception of an SMS, I have to send back an other SMS with some computed content.
I created a
route smsIn that looks like this
from "uri=smpp ..."
unmarshal ref="bindyDataFormat"
to "uri=bean:myBean
and a route smsOut with
from "uri=direct:smsOut"
to "uri=smpp ..."
The smsIn route, receives the sms, transforms its conent (csv data) in a pojo and send that pojo to myBean.
In myBean I do some processing and then call a ProducerTemplate which send my computed message to the endpoint "direct:smsOut".
The reason I use the producerTemplate is that I have to set some info from my pojo in the header (CamelSmppDestAddr) and the body of the Exchange.
I have tested with the logica SMSC simulator, this seems to work fine, but would like to have your opinion about this solution ?
What about reliability , transaction ?
Should I store my message before trying to send it to the SMSC ?
Should I store it in a database, post it to a queue ?
I'm not sure why you have a producer template, you could just build up the route instead (given that you return something from your bean or takes an Exchange as paramter).
<from uri="smpp: ..."/>
<bean ref="bean:myBean"/>
<to uri="jms:queue:myQueue"/>
then not use direct, but use a JMS queue that is transactional and persistent. Say your smpp call fails, the message would have been gone. Using a queue like this and make sure its transactional, you can make sure not to lose data in this stage of the route.
<from uri="jms:queue:myQueue"/>
<transactional/>
<to uri="smpp.."/>
I suggest using Apache ActiveMQ as JMS middleware. Actually, if you download ActiveMQ, you get camel bundled, so you could actually run your Camel routes from ActiveMQ.
You might want to tweak how retries and error handling occurs dependent on what you want to happen (retry every second forever?, retry five times, then put to error queue? etc).
Read this page: Transaction Error handling in Camel
For deeper info and more tweaks, you might also want to read this:
Transactional Client