I'm using the following processor to fetch the first attachement of an email and upload it to a ftp-server.
Route configuration
<from uri="imaps://...
<to uri="ejb:java:global/Dms/MailProcessor"/>
<to uri="ftp://....
MailProcessor
#Named("MailProcessor")
#Stateless
public class MailProcessor implements Processor {
#Override
public void process(Exchange exchange) throws Exception {
exchange.getOut().setHeaders(exchange.getIn().getHeaders());
Map<String, DataHandler> attachments = exchange.getIn().getAttachments();
if (attachments.size() > 0) {
for (String name : attachments.keySet()) {
DataHandler dataHandler = attachments.get(name);
// SET ATTACHMENT FILENAME TO OUTPUT FILENAME HEADER
String filename = dataHandler.getName();
filename = MimeUtility.decodeText(filename);
exchange.getOut().setHeader("filename", filename);
// SET INPUT ATTACHMENT TO OUTPUT BODY
byte[] data = exchange.getContext().getTypeConverter().convertTo(byte[].class, dataHandler.getInputStream());
exchange.getOut().setBody(data);
// SET ONLY THE FIRST ATTACHMENT
break;
}
}else{
exchange.getOut().setBody(exchange.getIn().getBody());
}
}
This works in general, but it takes literally forever for "big" attachments. (half an hour for just a 5 mb attachment)
Log
TRACE [org.apache.camel.component.file.remote.FtpOperations] (Camel (example) thread #119 - imaps://mail.example.com) Changing directory: upload
TRACE [org.apache.camel.component.file.remote.FtpOperations] (Camel (example) thread #119 - imaps://mail.example.com) doStoreFile(ID-example-local-59752-1494841993139-21-11)
DEBUG [org.apache.camel.component.file.remote.FtpOperations] (Camel (example) thread #119 - imaps://mail.example.com) About to store file: ID-example-local-59752-1494841993139-21-11 using stream: com.sun.mail.util.BASE64DecoderStream#601a11dc
TRACE [org.apache.camel.component.file.remote.FtpOperations] (Camel (example) thread #119 - imaps://mail.example.com) Client storeFile: ID-example-local-59752-1494841993139-21-11
-- long pause --
I've also tried to convert the attachment like this:
byte[] data = exchange.getContext().getTypeConverter().convertTo( byte[].class, dataHandler.getInputStream() );
exchange.getOut().setBody(data);
but this gives me the following:
2017-05-15 11:23:20,968 TRACE [org.apache.camel.impl.converter.DefaultTypeConverter] (Camel (example) thread #133 - imaps://mail.example.com) Converting org.apache.camel.Processor$$$view63 -> org.apache.camel.Processor with value: Proxy for view class: org.apache.camel.Processor of EJB: MailProcessor
2017-05-15 11:23:20,968 TRACE [org.apache.camel.component.bean.BeanProcessor] (Camel (example) thread #133 - imaps://mail.example.com) Using a custom adapter as bean invocation: Proxy for view class: org.apache.camel.Processor of EJB: MailProcessor
2017-05-15 11:23:20,990 TRACE [org.apache.camel.impl.converter.DefaultTypeConverter] (Camel (example) thread #133 - imaps://mail.example.com) Converting com.sun.mail.util.BASE64DecoderStream -> byte[] with value: com.sun.mail.util.BASE64DecoderStream#28442828
2017-05-15 11:23:20,990 TRACE [org.apache.camel.impl.converter.DefaultTypeConverter] (Camel (example) thread #133 - imaps://mail.example.com) Using converter: StaticMethodTypeConverter: public static byte[] org.apache.camel.converter.IOConverter.toBytes(java.io.InputStream) throws java.io.IOException to convert [class com.sun.mail.util.BASE64DecoderStream=>class [B]
2017-05-15 11:23:20,990 TRACE [org.apache.camel.util.IOHelper] (Camel (example) thread #133 - imaps://mail.example.com) Copying InputStream: java.io.BufferedInputStream#24b2cffb -> OutputStream: with buffer: 4096 and flush on each write false
-- long pause --
Uploading to the ftp server using Filezilla works like a charm. Also when I use camel-ftp in another route ( triggered by a file upload ) the upload to the ftp server works pretty fast.
So I have a feeling that it's the conversion of the attachment which slows things down.
Questions:
Is my assumption correct and how can I speed things up?
Solution was to use <from uri="imaps://...&mail.imaps.partialfetch=false.
https://community.oracle.com/thread/2604843?start=0&tstart=0
Related
I am having a number of type conversion issues using the Java DSL with Camel 3.14.3. For a simple example I have a route that uses a direct endpoint to trigger a pollEnrich for a file endpoint.
public class BasicRoute extends RouteBuilder {
#Override
public void configure() {
from("direct:test")
.pollEnrich("file://watchDirectory", 10000)
.to("mock:result");
}
}
When the route starts I get the following exception...
Exception in thread "main" org.apache.camel.FailedToCreateRouteException: Failed to create route route1 at: >>> PollEnrich[constant{file://watchDirectory}] <<< in route: Route(route1)[From[direct:test] -> [PollEnrich[constant{file... because of Error parsing [10000] as a java.time.Duration.
...
Caused by: org.apache.camel.NoTypeConversionAvailableException: No type converter available to convert from type: java.lang.String to the required type: java.time.Duration with value 10000
I am running this within a simple OG java app, so I am sure I am missing something in the context initialization, but I cannot find it.
I am using Camel in a Spring-Boot application to route from AMQ-Queue. Messages from this queue will be sent to a REST-Webservice. It is already working with this code line:
from("amq:queue:MyQueue").process("jmsToHttpProcessor").to(uri);
My uri looks like this:
http4://localhost:28010/application/createCustomer
Now I have the requirement that the routing to the Webservice should be done parallely:
In order to achive that, I configured concurrentConsumers in JmsConfiguration as follows:
#Bean
public JmsComponent amq(#Qualifier("amqConnectionFactory") ConnectionFactory amqConnectionFactory, AMQProperties amqProperties) {
JmsConfiguration jmsConfiguration = new JmsConfiguration(amqConnectionFactory);
jmsConfiguration.setConcurrentConsumers(50);
jmsConfiguration.setMaxConcurrentConsumers(50);
return new JmsComponent(jmsConfiguration);
}
#Bean
public ConnectionFactory amqConnectionFactory(AMQProperties amqProperties) throws Exception {
ConnectionFactoryParser parser = new ConnectionFactoryParser();
ConnectionFactory returnValue = parser.newObject(parser.expandURI(amqProperties.getUrl()), "amqConnectionFactory");
return returnValue;
}
It is working as expected, BUT not right away from the beginning. I have the phenomenon:
I have 100 messages in the ActiveMQ queue
I start my Spring application
Camel creates only 1 thread consuming 1 message after the previous one gets response
I observe that the amount of messages in queue only decreasing slowly(99.... 98... 97... 96...)
I am filling the queue with new 100 messages
NOW the concurrent consumers are being created as I can observe that the messages decreasing rapidly.
Does someone have any idea, why the concurrentConsumers is not working right away from the beginning?
I tried the advices. Unfortunately they dont change the behaviour. I found out, that the problem is that Camel already starts consuming the messages from the queue before the Spring boot application is startet. I can observe this from the log:
2021-04-01T20:26:33,901 INFO (Camel (CamelBridgeContext) thread #592 - JmsConsumer[MyQueue]) [message]; ...
2021-04-01T20:26:33,902 INFO (Camel (CamelBridgeContext) thread #592 - JmsConsumer[MyQueue]) [message]; ...
2021-04-01T20:26:33,915 INFO (main) [AbstractConnector]; _; Started ServerConnector#5833f5cd{HTTP/1.1,[http/1.1]}{0.0.0.0:23500}
2021-04-01T20:26:33,920 INFO (main) [BridgeWsApplication]; _; Started BridgeWsApplication in 12.53 seconds (JVM running for 13.429)
In this case, only one consumer with thread #592 is consuming all the messages.
In fact, if I start my Spring application first, and then fill the queue with messages, then concurrentConsumers will be used:
2021-04-01T20:30:20,159 INFO (Camel (CamelBridgeContext) thread #594 - JmsConsumer[MyQueue])
2021-04-01T20:30:20,159 INFO (Camel (CamelBridgeContext) thread #599 - JmsConsumer[MyQueue])
2021-04-01T20:30:20,178 INFO (Camel (CamelBridgeContext) thread #593 - JmsConsumer[MyQueue])
2021-04-01T20:30:20,204 INFO (Camel (CamelBridgeContext) thread #564 - JmsConsumer[MyQueue])
In this case, messages are being consumed from concurrentConsumers parallely.
In order to solve the problem, I tried setting autoStartUp to false in my RouteBuilder component:
#Override
public void configure() {
CamelContext context = getContext();
context.setAutoStartup(false);
// My Route
}
In my naive thinking, I let Camel starting after the Spring boot is started and running:
public static void main(String[] args) {
ConfigurableApplicationContext context = SpringApplication.run(BridgeWsApplication.class, args);
SpringCamelContext camel = (SpringCamelContext) context.getBean("camelContext");
camel.start();
try {
camel.startAllRoutes();
} catch (Exception e) {
e.printStackTrace();
}
}
Unfortunately, this does not change the behaviour. There must be a configuration to let Camel starts after Spring is started.
Camel Version 2.22.0
Runtime: SpringBoot : 2.0.2.RELEASE
JDK version: 1.8.0_121
EIP: recipientList.
Problem: Exception raised from parallel process of recipientList is not caught at route level onException clause.
Below is the DSL
#Override
public void configure() throws Exception {
restConfiguration().clientRequestValidation(true)
//.contextPath("/pss/v1.0/")
.port("8080").host("0.0.0.0")
.enableCORS(true)
.apiContextPath("/api-doc")
.apiProperty("api.title", "Test REST API")
.apiProperty("api.version", "v1")
.apiContextRouteId("doc-api")
.component("servlet")
.bindingMode(RestBindingMode.json);
rest("/api/").clientRequestValidation(true)
.id("api-route")
.consumes("application/json")
.get("/bean/{name}")
.bindingMode(RestBindingMode.json)
.to("direct:remoteService");
from("direct:remoteService")
.onException(Exception.class).handled(true)
.log("Exception Caught : ${exception.message}")
.end()
.recipientList(constant("direct:route1, direct:route2"), ",").parallelProcessing().aggregationStrategy(new GroupedBodyAggregationStrategy())
.stopOnException()
.end()
.log("The final Exchange data : ${exception.message}");
from("direct:route1")
.setHeader( Exchange.CONTENT_ENCODING, simple("gzip"))
.setBody(simple("RESPONSE - [ { \"id\" : \"bf383eotal length is 16250]]"))
.log("${body}");
from("direct:route2")
.log("${body}")
.process(e-> {
List<String> myList = new ArrayList();
myList.add("A");
myList.add("b");
myList.add("C");
e.getIn().setBody(myList);
})
.split(body())
.parallelProcessing(true)
.aggregationStrategy(new GroupedBodyAggregationStrategy())
.stopOnException()
.log("${body}")
.choice()
.when(simple("${body} == 'b'"))
.throwException(new Exception("jsdhfjkASDf"));
}
Try make onException as global like this:
onException(Exception.class).handled(true)
.log("Exception Caught : ${exception.message}")
.end();
from("direct:remoteService")
.recipientList(constant("direct:route1, direct:route2"), ",").parallelProcessing().aggregationStrategy(new GroupedBodyAggregationStrategy())
.stopOnException()
.end()
.log("The final Exchange data : ${exception.message}")
;
UPD: So you need to disable error handlers in recipient routes. Try like this (can't insert normally code sample)
That's a classical mistake: (exactly like the split EIP) each recipient will process a copy of the original Exchange. Any failure on these copies will not affect (raise an exception on) the route processing the master Exchange, as every single exchange runs in a completely separate unit of work.
If you enable the "shareUnitOfWork" option (on the recipientList), exceptions should be propagated.
hi~ i am using camel http component. and i can't extract body message.
here is my code
.log(LoggingLevel.INFO, "ToUri ===> ${body}")
.toD("${body}")
.log(LoggingLevel.INFO, "Result ===> ${body}")
.process(new Processor() {
public void process(Exchange exchange) throws Exception {
long startTime = System.currentTimeMillis();
Message inboundMessage = exchange.getIn();
Object body = exchange.getIn().getBody();
String msg = inboundMessage.getBody(String.class);
System.out.println("body:"+body);
System.out.println("getInBody msg:"+msg);
System.out.println("getInBody body:"+body.toString());
=======================================================================
body : org.apache.camel.converter.stream.CachedOutputStream$WrappedInputStream#28936ba4
getInBody msg:
getInBody bodybodybody:org.apache.camel.converter.stream.CachedOutputStream$WrappedInputStream#28936ba4
the log is good works. like this
09:56:53.523 INFO route1 - ToUri ===> https://translation.googleapis.com/language/translate/v2?key=tesetKey&source=en&target=ja&q=hi
09:56:54.545 INFO route1 - Result ===> {
"data": {
"translations": [
{
"translatedText": "こんにちは"
}
]
}
}
i want to extract translatedText using camel.
how can i handle CachedOutputStream and what is this?
i search camel doc.and cant understand.please give me a hint to solve my problem.
thanks.
See stream-caching for information about CachedOutputStream: http://camel.apache.org/stream-caching.html
To get the message body as string from the processor, you just do
String body = exchange.getIn().getBody(String.class);
That will tell Camel that you want the message as a String and it will automatic covert the message body from CachedOutputStream to String. Then you can grab that text you want via regular Java code.
Also note there is jsonpath you can use to work with json data and get information, however its syntax can take a little bit to learn: http://camel.apache.org/jsonpath
You already took the stream data (with .log) before calling processor. Stream data can only fetched once apparently. Try remove the log step and you can get in the processor:
.log(LoggingLevel.INFO, "Result ===> ${body}")
.process(new Processor() {
after spend 2 days experimenting
you can use convertBodyTo(Class<?> type) method for that, like this
.log(LoggingLevel.INFO, "Result ===> ${body}")
.convertBodyTo(String.class)
.process(new Processor() { ... }
I know my client works as I post to another website and it all works. I changed the client to post to url http://localhost:9000/enrollment/enroll and ran the following server...
#Path("/enrollment")
public class MockHpsRestApi {
private static final Logger log = LoggerFactory.getLogger(MockHpsRestApi.class);
public static void main(final String[] args) throws InterruptedException,
IOException, MuleException {
log.info("starting the server");
JAXRSServerFactoryBean sf = new JAXRSServerFactoryBean();
sf.setResourceClasses(MockHpsRestApi.class);
sf.setResourceProvider(MockHpsRestApi.class, new SingletonResourceProvider(new MockHpsRestApi()));
sf.setAddress("http://0.0.0.0:9000/");
sf.create();
}
#POST
#Path("/enroll")
#Consumes("application/xml")
public String enrollment(MultipartBody body) throws IOException {
log.info("received request="+body);
Attachment att = body.getRootAttachment();
ByteArrayOutputStream str = new ByteArrayOutputStream(100);
InputStream in = att.getDataHandler().getInputStream();
try {
IOUtils.copy(in, str);
log.info("body payload received="+str);
String xml = "success";
return xml;
} finally {
IOUtils.closeQuietly(in);
IOUtils.closeQuietly(str);
}
}
}
My logs though say this when it receives the request...
2012-03-16 09:30:30,295 INFO 300 [main] c.i.e.s.i.MockHpsRestApi - starting the server
Mar 16, 2012 9:30:30 AM org.apache.cxf.endpoint.ServerImpl initDestination
INFO: Setting the server's publish address to be http://0.0.0.0:9000/
2012-03-16 09:30:30,769 INFO 774 [main] org.eclipse.jetty.server.Server - jetty-7.5.4.v20111024
2012-03-16 09:30:30,801 INFO 806 [main] o.e.jetty.server.AbstractConnector - Started SelectChannelConnector#0.0.0.0:9000 STARTING
2012-03-16 09:30:30,824 INFO 829 [main] o.e.j.server.handler.ContextHandler - started o.e.j.s.h.ContextHandler{,null}
Mar 16, 2012 9:30:37 AM org.apache.cxf.jaxrs.utils.JAXRSUtils readFromMessageBody
WARNING: No message body reader has been found for request class MultipartBody, ContentType : application/xml.
Mar 16, 2012 9:30:37 AM org.apache.cxf.jaxrs.impl.WebApplicationExceptionMapper toResponse
WARNING: WebApplicationException has been caught : no cause is available
Why isn't this working?
hmmm, turning up the log levels, is this some kind of bug in CXF, though that would mean no one is really doing basic posts, so that doesn't seem right....(again, my client posts fine to another service today).
2012-03-16 09:51:01,414 DEBUG 10916 [qtp669588045-18 - /enrollment/enroll] o.a.c.j.i.JAXRSInInterceptor - Request path is: /enrollment/enroll
2012-03-16 09:51:01,414 DEBUG 10916 [qtp669588045-18 - /enrollment/enroll] o.a.c.j.i.JAXRSInInterceptor - Request HTTP method is: POST
2012-03-16 09:51:01,414 DEBUG 10916 [qtp669588045-18 - /enrollment/enroll] o.a.c.j.i.JAXRSInInterceptor - Request contentType is: application/xml
2012-03-16 09:51:01,415 DEBUG 10917 [qtp669588045-18 - /enrollment/enroll] o.a.c.j.i.JAXRSInInterceptor - Accept contentType is: */*
2012-03-16 09:51:01,415 DEBUG 10917 [qtp669588045-18 - /enrollment/enroll] o.a.c.j.i.JAXRSInInterceptor - Found operation: enrollment
2012-03-16 09:51:01,420 WARN 10922 [qtp669588045-18 - /enrollment/enroll] o.apache.cxf.jaxrs.utils.JAXRSUtils - No message body reader has been found for request class MultipartBody, ContentType : application/xml.
2012-03-16 09:51:01,423 WARN 10925 [qtp669588045-18 - /enrollment/enroll] o.a.c.j.i.WebApplicationExceptionMapper - WebApplicationException has been caught : no cause is available
2012-03-16 09:51:01,428 DEBUG 10930 [qtp669588045-18 - /enrollment/enroll] o.a.c.j.i.WebApplicationExceptionMapper - no cause is available
javax.ws.rs.WebApplicationException: null
at org.apache.cxf.jaxrs.utils.JAXRSUtils.readFromMessageBody(JAXRSUtils.java:1052) ~[cxf-bundle-2.5.2.jar:2.5.2]
at org.apache.cxf.jaxrs.utils.JAXRSUtils.processParameter(JAXRSUtils.java:616) ~[cxf-bundle-2.5.2.jar:2.5.2]
at org.apache.cxf.jaxrs.utils.JAXRSUtils.processParameters(JAXRSUtils.java:580) ~[cxf-bundle-2.5.2.jar:2.5.2]
at org.apache.cxf.jaxrs.interceptor.JAXRSInInterceptor.processRequest(JAXRSInInterceptor.java:238) ~[cxf-bundle-2.5.2.jar:2.5.2]
at org.apache.cxf.jaxrs.interceptor.JAXRSInInterceptor.handleMessage(JAXRSInInterceptor.java:89) ~[cxf-bundle-2.5.2.jar:2.5.2]
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:263) [cxf-api-2.5.2.jar:2.5.2]
at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:123) [cxf-rt-core-2.5.2.jar:2.5.2]
at org.apache.cxf.transport.http_jetty.JettyHTTPDestination.serviceRequest(JettyHTTPDestination.java:323) [cxf-bundle-2.5.2.jar:2.5.2]
at org.apache.cxf.transport.http_jetty.JettyHTTPDestination.doService(JettyHTTPDestination.java:289) [cxf-bundle-2.5.2.jar:2.5.2]
at org.apache.cxf.transport.http_jetty.JettyHTTPHandler.handle(JettyHTTPHandler.java:72) [cxf-bundle-2.5.2.jar:2.5.2]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:943) [jetty-server-7.5.4.v20111024.jar:7.5.4.v20111024]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:879) [jetty-server-7.5.4.v20111024.jar:7.5.4.v20111024]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117) [jetty-server-7.5.4.v20111024.jar:7.5.4.v20111024]
When using #POST and #Consumes("application/xml"), it's normal for the (single un-annotated) argument type on the method to be a JAXB-annotated class that CXF will deserialize the XML into before handing it to you. Is there a particular reason for not going that well-travelled route?
After debugging the code, it turns out you can use the following types as parameters to the method above.
Source
XMLSource
Document
byte[]
InputStream
I ended up using InputStream.