I am using Camel 2.18 and Java 8. Is there any way to check size of the message in Camel so that I could validate it ?
So for example I have a processor that has access to the body:
Message msgIn = exchange.getIn();
msgIn.getBody()
Now I would like to check size that body takes in memory.
Reason for that is that I must raise an exception if a body is larger than 1MB.
The message that is POSTed to my Camel Endpoint is just simple json:
{
"field1": "b14",
"fieldWithContent": "there may be something big",
"field3": 0
}
Related
So I have a rest API that lives at https://foo.bar/api which returns either an empty json list [] or a list that contains 1 or more items:
[
{
"#class": "foo.bar.java.MyObject",
"name": "Joe Bloggs"
},
{
"#class": "foo.bar.java.MyObject",
"name": "Fred Flinstone"
}
]
Now I am trying to have camel take in this data from my endpoint and hand each object within the list to a processor. I've tried the following:
fromF("timer://foo-poll?fixedRate=true&delay=5s&period=%d&bridgeErrorHandler=true", pollRate)
.toF("https4://%s/%s", host, requestPath)
.log("Received: ${body}")
.split()
.jsonpath("$")
.log("Split: ${body}")
.process(barProccessor);
As well as various attempts to unmarshal the data using .unmarshal(new ListJacksonDataFormat(MyObject.class)) or .unmarshal().json(JsonLibrary.Jackson, List.class)
where nothing has worked.
Using the larger code block above, there are no errors nor is the "Split: ${body}" log message printers out.
Unmarshalling using either methods described above throws this regardless of how many items are returned from the API:
com.fasterxml.jackson.databind.exc.MismatchedInputException: No
content to map due to end-of-input
When there
Okay, managed to figure this out if anyone else is facing a similar issue. The working route builder:
fromF("timer://foo-poll?fixedRate=true&delay=5s&period=%d&bridgeErrorHandler=true", pollRate)
.toF("https4://%s/%s", host, requestPath)
.log("Received: ${body}")
.streamCaching("true")
.unmarshal(new ListJacksonDataFormat(MyObject.class))
.split()
.jsonpath("$")
.log("Split: ${body}")
.process(barProccessor);
I have enabled stream caching and unmarshalled the list using Jackson.
I'm struggling with a problem in an application based on Apache Camel when connecting to ActiveMQ Artemis via JMS. At the end of one of the Camel routes, messages are stored in an Artemis JMS queue. A legacy component running in the same application picks them up from there periodically using a ConsumerTemplate.
This works fine for Camel messages with plain text bodies, but causes errors when using byte array bodies: It seems Artemis treats any message with byte body as a "large message", which are streamed instead of kept in memory. Receiving via the ConsumerTemplate works, but as soon as the body or headers are accessed, an exception as follows is raised:
org.apache.camel.RuntimeCamelException: Failed to extract body due to: javax.jms.IllegalStateException: AMQ119023: The large message lost connection with its session, either because of a rollback or a closed session. Message: ActiveMQMessage[ID:90c4d1d5-3233-11ea-b0cc-44032c68a56f]:PERSISTENT/ClientLargeMessageImpl[messageID=2974, durable=true, address=mytest,userID=90c4d1d5-3233-11ea-b0cc-44032c68a56f,properties=TypedProperties[firedTime=Wed Jan 08 17:26:03 CET 2020,__AMQ_CID=90b4f34e-3233-11ea-b0cc-44032c68a56f,breadcrumbId=ID-NB045-evolit-co-at-1578500762151-0-1,_AMQ_ROUTING_TYPE=1,_AMQ_LARGE_SIZE=3]]
at org.apache.camel.component.jms.JmsBinding.extractBodyFromJms(JmsBinding.java:172) ~[camel-jms-2.22.1.jar:2.22.1]
at org.apache.camel.component.jms.JmsMessage.createBody(JmsMessage.java:221) ~[camel-jms-2.22.1.jar:2.22.1]
at org.apache.camel.impl.MessageSupport.getBody(MessageSupport.java:54) ~[camel-core-2.22.1.jar:2.22.1]
at org.apache.camel.example.cdi.JmsPoller.someMethod(JmsPoller.java:36) ~[classes/:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_171]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_171]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_171]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_171]
at org.apache.camel.component.bean.MethodInfo.invoke(MethodInfo.java:481) ~[camel-core-2.22.1.jar:2.22.1]
at org.apache.camel.component.bean.MethodInfo$1.doProceed(MethodInfo.java:300) ~[camel-core-2.22.1.jar:2.22.1]
at org.apache.camel.component.bean.MethodInfo$1.proceed(MethodInfo.java:273) ~[camel-core-2.22.1.jar:2.22.1]
at org.apache.camel.component.bean.AbstractBeanProcessor.process(AbstractBeanProcessor.java:188) ~[camel-core-2.22.1.jar:2.22.1]
at org.apache.camel.component.bean.BeanProcessor.process(BeanProcessor.java:53) ~[camel-core-2.22.1.jar:2.22.1]
at org.apache.camel.component.bean.BeanProducer.process(BeanProducer.java:41) ~[camel-core-2.22.1.jar:2.22.1]
at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:148) ~[camel-core-2.22.1.jar:2.22.1]
at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548) [camel-core-2.22.1.jar:2.22.1]
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201) [camel-core-2.22.1.jar:2.22.1]
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201) [camel-core-2.22.1.jar:2.22.1]
at org.apache.camel.component.timer.TimerConsumer.sendTimerExchange(TimerConsumer.java:197) [camel-core-2.22.1.jar:2.22.1]
at org.apache.camel.component.timer.TimerConsumer$1.run(TimerConsumer.java:79) [camel-core-2.22.1.jar:2.22.1]
at java.util.TimerThread.mainLoop(Timer.java:555) [?:1.8.0_171]
at java.util.TimerThread.run(Timer.java:505) [?:1.8.0_171]
Caused by: javax.jms.IllegalStateException: AMQ119023: The large message lost connection with its session, either because of a rollback or a closed session
at org.apache.activemq.artemis.core.client.impl.LargeMessageControllerImpl.saveBuffer(LargeMessageControllerImpl.java:273) ~[artemis-core-client-2.6.2.jar:2.6.2]
at org.apache.activemq.artemis.core.client.impl.ClientLargeMessageImpl.saveToOutputStream(ClientLargeMessageImpl.java:115) ~[artemis-core-client-2.6.2.jar:2.6.2]
at org.apache.activemq.artemis.jms.client.ActiveMQMessage.saveToOutputStream(ActiveMQMessage.java:853) ~[artemis-jms-client-2.6.2.jar:2.6.2]
at org.apache.activemq.artemis.jms.client.ActiveMQMessage.setObjectProperty(ActiveMQMessage.java:693) ~[artemis-jms-client-2.6.2.jar:2.6.2]
at org.apache.camel.component.jms.JmsBinding.createByteArrayFromBytesMessage(JmsBinding.java:251) ~[camel-jms-2.22.1.jar:2.22.1]
at org.apache.camel.component.jms.JmsBinding.extractBodyFromJms(JmsBinding.java:163) ~[camel-jms-2.22.1.jar:2.22.1]
... 21 more
Caused by: org.apache.activemq.artemis.api.core.ActiveMQIllegalStateException: AMQ119023: The large message lost connection with its session, either because of a rollback or a closed session
at org.apache.activemq.artemis.core.client.impl.LargeMessageControllerImpl.saveBuffer(LargeMessageControllerImpl.java:273) ~[artemis-core-client-2.6.2.jar:2.6.2]
at org.apache.activemq.artemis.core.client.impl.ClientLargeMessageImpl.saveToOutputStream(ClientLargeMessageImpl.java:115) ~[artemis-core-client-2.6.2.jar:2.6.2]
at org.apache.activemq.artemis.jms.client.ActiveMQMessage.saveToOutputStream(ActiveMQMessage.java:853) ~[artemis-jms-client-2.6.2.jar:2.6.2]
at org.apache.activemq.artemis.jms.client.ActiveMQMessage.setObjectProperty(ActiveMQMessage.java:693) ~[artemis-jms-client-2.6.2.jar:2.6.2]
at org.apache.camel.component.jms.JmsBinding.createByteArrayFromBytesMessage(JmsBinding.java:251) ~[camel-jms-2.22.1.jar:2.22.1]
at org.apache.camel.component.jms.JmsBinding.extractBodyFromJms(JmsBinding.java:163) ~[camel-jms-2.22.1.jar:2.22.1]
... 21 more
The problem also occurs for messages that do not exceed the minLargeMessageSize of Artemis, in a test program even for 3 bytes.
Coincidentally, the same problem occurred in a standalone application used for testing the application. There, I was able to solve the issue by keeping the JMS session and receiver open until the JMS message body and headers were completely read. With Camel, that's abstracted away in the Spring JmsTemplate that Camel is based on.
I consulted the user documentation of the Camel JMS component to find configuration options that might help me. I've tried the following:
eagerLoadingOfProperties=true on consumer side: no effect, only seems to affect MessageListenerContainer. The documentation says:
It uses [...] Spring’s JmsTemplate for sending and a MessageListenerContainer for consuming.
However, while debugging it seemed that MessageListenerContainer is only used when consuming messages from an JMS endpoint in a Camel route. Using a ConsumerTemplate like in my case uses a JmsTemplate for consuming.
messageConverter and mapJmsMessage on consumer side: no effect, they are executed when the session has already been closed
alwaysCopyMessage on producer side: I thought maybe copying prevents use of streamed large messages, no effect
streamMessageTypeEnabled=false on producer side: no effect
jmsMessageType=Bytes on both producer and consumer side: no effect
transferExchange=true on both producer and consumer side: this does seem to solve my specific case, but it feels like a workaround. Documentation advises to use the option with caution.
So right now, transferExchange seems to be my best bet, assuming it really solves my issue in all test cases. Nevertheless, I'd be glad to get better understanding on the issue or different solutions:
Why does Artemis treat small byte array messages as large messages anyway?
Does Camel ConsumerTemplate support streamed large messages at all?
My versions are Camel 2.22.1 and Artemis 2.10.1.
I've been able to reproduce my problem by modifying the Camel Example camel-example-cdi from the release package of Camel to have the minimal classes shown below.
In addition I've added camel-jms and Artemis dependencies and started Artemis locally, both like described in the camel-example-artemis-large-messages example.
public class MyRoutes extends RouteBuilder {
#Override
public void configure() {
setupJmsComponent();
from("timer:writeTimer?period=6000")
.log("writing to JMS")
.setBody(() -> new byte[]{0,1,2})
.to(JmsPoller.ENDPOINT);
from("timer:pollTimer?period=3000")
.to("bean:jmsPoller");
}
private void setupJmsComponent() {
ActiveMQJMSConnectionFactory connectionFactory = new ActiveMQJMSConnectionFactory("tcp://localhost:61616");
JmsComponent jmsComponent = new JmsComponent();
jmsComponent.setConnectionFactory(connectionFactory);
getContext().addComponent("jms", jmsComponent);
}
}
#Singleton
#Named("jmsPoller")
public class JmsPoller {
static final String ENDPOINT = "jms:queue:mytest";
#Inject
private ConsumerTemplate consumerTemplate;
public void someMethod(String body) {
Exchange exchange = consumerTemplate.receive(ENDPOINT, 1000L);
System.out.println("Received " + (exchange == null ? null : exchange.getIn().getBody()));
}
}
ActiveMQ Artemis doesn't treat just any message with a byte body as a "large" message. It's worth noting that the broker ultimately treats all message bodies as an array of bytes because that's exactly what they are. However, in order to be considered "large" the message has to exceed a certain size. The documentation states:
Any message larger than a certain size is considered a large message. Large messages will be split up and sent in fragments. This is determined by the URL parameter minLargeMessageSize.
Note:
Apache ActiveMQ Artemis messages are encoded using 2 bytes per character so if the message data is filled with ASCII characters (which are 1 byte) the size of the resulting Apache ActiveMQ Artemis message would roughly double. This is important when calculating the size of a "large" message as it may appear to be less than the minLargeMessageSize before it is sent, but it then turns into a "large" message once it is encoded.
The default value is 100KiB.
It looks like the application's use-case simply doesn't fit with the semantics of large message support in ActiveMQ Artemis since the session which the message came from is being closed before the message's body is fully received.
Therefore, I recommend that you either keep the session open until the body is read or increase the minLargeMessageSize on the URL of the application which is sending the message so that no messages are ever considered "large." The latter option may result in greater memory usage on the broker since the entire message body will be held in memory at once.
Apache Camel is throwing Invalid Correlation Key exception when trying to aggregate messages from my AWS SQS queue.
The messages were placed in the queue using ZipSplitter and they all appear in the queue with matching "parentId" values (which I added using a random uuid as part of the splitting -I've tried CamelSourceFile as well). I get the Exception repeatedly until the retries are exhausted.
My aggregate expression:
from(--queue--).aggregate(header("parentId"), customAggregationStrategy).completionTimeout(3000).processor(new Processor() {...}.to(--next queue--);
There is no logging emitted from my customAggregationStrategy nor from any of the subsequent processors. It fails to aggregate:
... DeadLetterChannel - Failed delivery for (MessageId: ...). On Delivery attempt: 0 caught ...CamelExchangeException: Invalid correlation key. Exchange[ID...]
The delivery attempt is 0 through 9 for my retry attempts.
The infuriating thing is that the code works everywhere but locally...which you think would narrow things down, but neither the exception nor anything else logged sheds any light onto what is going on here.
You could try to use Camel simple language when expressing the correlation key, ie:
.aggregate(simple("${headers.parentId}", customAggregationStrategy)
This way, the exceptions might be silently ignored ?
Did you activate Camel tracer (http://camel.apache.org/tracer.html) to analyse your exchanges and ease the debugging ?
I suspect you have an Exchange which does NOT have the "parentId" header. If you want to skip them, just activate the ignoreInvalidCorrelationKeys option (see http://camel.apache.org/aggregator2.html)
I am trying to check :
<simple>${body} contains 'verification'</simple>
Body is the json:
{"verification": {"email": "bb#wp.pl", "code": "1234"}}
But this condition doesn't work. I've tried as well:
<simple>${body} contains 'verification'</simple>
<simple>${bodyAs(String)} contains 'verification'</simple>
<simple>${body.verification} != null</simple>
Could you please suggest me something?
I guess the body is maybe not a String, then try with
<simple>${bodyAs(String)} contains 'verification'</simple>
And btw what version of Camel do you use?
Actually this case:
<simple>${bodyAs(String)} contains 'verification'</simple>
didn't work cause:
In Camel the message body can be of any types. Some types are safely readable multiple times, and therefore do not 'suffer' from becoming 'empty'.
It fixes by Stream caching
We are using camel 2.13.2 - I have a multicast route with an AggregationStrategy.
And in each multicast branch, we have a custom camel component that returns huge data (around 4 MB) and writes to Stream Cache (Cached Output Stream) and we need to aggregate the data in the multicast (Aggregation Strategy).
In the Aggregation strategy, I need to do XPath evaluation using camel XPathBuilder.
Hence, I try to read the body and convert from StreamCache to byte[] to avoid 'Error during type conversion from type: org.apache.camel.converter.stream.InputStreamCache.' in the XPathBuilder.
When I try to read the body in the beginning of the Aggregation Strategy, I get the following error.
*/tmp/camel/camel-tmp-4e00bf8a-4a42-463a-b046-5ea2d7fc8161/cos6047774870387520936.tmp (No such file or directory), cause: FileNotFoundException:/tmp/camel/camel-tmp-4e00bf8a-4a42-463a-b046-5ea2d7fc8161/cos6047774870387520936.tmp (No such file or directory).
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.(FileInputStream.java:138)
at org.apache.camel.converter.stream.FileInputStreamCache.createInputStream(FileInputStreamCache.java:123) at org.apache.camel.converter.stream.FileInputStreamCache.getInputStream(FileInputStreamCache.java:117)
at org.apache.camel.converter.stream.FileInputStreamCache.writeTo(FileInputStreamCache.java:93)
at org.apache.camel.converter.stream.StreamCacheConverter.convertToByteArray(StreamCacheConverter.java:102)
at com.sap.it.rt.camel.aggregate.strategies.MergeAtXPathAggregationStrategy.convertToByteArray(MergeAtXPathAggregationStrategy.java:169)
at com.sap.it.rt.camel.aggregate.strategies.MergeAtXPathAggregationStrategy.convertToXpathCompatibleType(MergeAtXPathAggregationStrategy.java:161)
*
Following is the line of code where it is throwing an error:
Object body = exchange.getIn().getBody();
if( body instanceof StreamCache){
StreamCache cache = (StreamCache)body;
xml = new String(convertToByteArray(cache,exchange));
exchange.getIn().setBody(xml);
}
By disabling stream cache to write to file by setting a threshold of 10MB in multicast related routes, we were able to work with the aggregation strategy. But we do not want to do that, as we may have incoming data that maybe bigger.
<camel:camelContext id="multicast_xml_1" streamCache="true">
<camel:properties>
<camel:property key="CamelCachedOutputStreamCipherTransformation" value="RC4"/>
<camel:property key="CamelCachedOutputStreamThreshold" value="100000000"/>
</camel:properties>
....
</camel:camelContext>
Note: The FileNotFound issue does not appear if we have the StreamCache based camel component in the route with other processors, but without Multicast + Aggregation.
After debugging, I could understand the issue with aggregating huge data from StreamCache with MulticastProcessor.
In MulticastProcessor.java: doProcessParallel() is called and on completion of the branch exchange of multicast, the CachedOutputStream deletes / cleans up the temporary file.
This happens even before the multicast branch exchange reaches the aggregation Strategy, which tries to read the data from the branch exchange. In case of huge data in StreamCache, the temporary file is already deleted, leading to FileNotFound issues.
public CachedOutputStream(Exchange exchange, boolean closedOnCompletion) {
this.strategy = exchange.getContext().getStreamCachingStrategy();
currentStream = new CachedByteArrayOutputStream(strategy.getBufferSize());
if (closedOnCompletion) {
// add on completion so we can cleanup after the exchange is done such as deleting temporary files
exchange.addOnCompletion(new SynchronizationAdapter() {
#Override
public void onDone(Exchange exchange) {
try {
if (fileInputStreamCache != null) {
fileInputStreamCache.close();
}
close();
} catch (Exception e) {
LOG.warn("Error deleting temporary cache file: " + tempFile, e);
}
}
#Override
public String toString() {
return "OnCompletion[CachedOutputStream]";
}
});
}
}
public void close() throws IOException {
currentStream.close();
cleanUpTempFile();
}
I was able to circumvent the issue, if I try to set closedOnCompletion= false, while writing to CachedOutputStream in any component in any Multicast branch.
But this is a leaky solution, because the streamcache temporary file(s) may then never get cleaned up... hence I try to close + clean up the cachestream, after reading the data in the AggregationStrategy.
Can the MulticastProcessor be adjusted so that the multicast branch exchanges reach 'completion' status only, after they have been aggregated at the end of multicast?
Please help / advise on the issue, as I am new to using camel Multicast.
Thanks,
Lakshmi
I have similar exception thrown when trying to send larger than 1MB JSON response to Restlet request (yes, I know 1MB JSON is too big):
java.io.FileNotFoundException: C:\Users\me\AppData\Local\Temp\camel\camel-tmp-7ad6e098-538d-4d4c-9357-2b7addb1f19d\cos6725022584818060586.tmp (The system cannot find the file specified)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(FileInputStream.java:146)
at org.apache.camel.converter.stream.FileInputStreamCache.createInputStream(FileInputStreamCache.java:123)
at org.apache.camel.converter.stream.FileInputStreamCache.getInputStream(FileInputStreamCache.java:117)
at org.apache.camel.converter.stream.FileInputStreamCache.read(FileInputStreamCache.java:112)
at java.io.InputStream.read(InputStream.java:170)
at java.io.InputStream.read(InputStream.java:101)
at org.restlet.engine.io.BioUtils.copy(BioUtils.java:81)
at org.restlet.representation.InputRepresentation.write(InputRepresentation.java:148)
at org.restlet.engine.adapter.ServerCall.writeResponseBody(ServerCall.java:510)
at org.restlet.engine.adapter.ServerCall.sendResponse(ServerCall.java:454)
at org.restlet.ext.servlet.internal.ServletCall.sendResponse(ServletCall.java:426)
at org.restlet.engine.adapter.ServerAdapter.commit(ServerAdapter.java:196)
at org.restlet.engine.adapter.HttpServerHelper.handle(HttpServerHelper.java:153)
at org.restlet.ext.servlet.ServerServlet.service(ServerServlet.java:1089)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1496)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:102)
Same workaround works for me:
getContext().getProperties().put(CachedOutputStream.THRESHOLD, "" + THREE_MEGABYTE_TRESHOLD_BEFORE_FILE_CACHE);
I don't use multicast in this route, just plain
restlet request -> Service -> Jackson marshall => error
I use Camel 2.14.0 & Restlet 2.2.2 with JDK 7 and Spring-boot 1.0.2 / Jetty
This Camel reverse proxy - no response stream caching might be related to my issue.