Apache Camel 2.12.1
I have a route setup like this:
public void configure() throws Exception {
from("direct:start")
.process(new AuthorizationHeaderProcessor(configureCreds()))
.to(httpSourceEndpoint)
.process(new GenerateSQLFromMessageProcessor))
.enrich("jdbc:dataSource", new DBAggregator())
//...do things with result...
1) HttpSourceEndPoint is a get request from some url.
2) I then want to use the result of this to generate the SQL as per GenerateSQLFromMessageProcessor, which is provided as input to the JDBC route.
My problem is, within the DBAggregator, the parameters that come through are:
oldExchange = the raw SQL string that was sent to the JDBC call
newExchange = the result set from the DB query
ie there is no sign of the original message that came from the http source endpoint, which is how I would have expected aggregation to work. How am I supposed to combine the 2 streams? Was it the GenerateSQLFromMessageProcessor call that consumed the original message? If so, should you specify the SQL in a bean for an enrich?
EDIT
So setting in the header like this:
public void configure() throws Exception {
from("direct:start")
.process(new AuthorizationHeaderProcessor(configureCreds()))
.to(httpSourceEndpoint)
.setHeader(new BeanExpression(MySQLBean.class, "methodToGenerateSQL")
.enrich("jdbc:dataSource", new DBAggregator())
//...do things with result...
results in my aggregator looking like this:
public class DBAggregator implements AggregationStrategy {
#Override
public Exchange aggregate(Exchange oldExchange, Exchange newExchange) {
Here I have:
oldExchange = the resulting SQL string that methodToGenerateSQL generated
newExchange = the result set from the SQL query
The problem is I do not have access to the original message that came from httpSourceEndpoint.
As this is an aggregator I would have expected oldExchange to be the incoming message, not just an SQL string.
After all, it is an aggregator, and yet I have effectively lost the incoming message - this is not "enriching"!
Thanks,
Mr Tea
Related
Issue:
I have multiple route sending messages to an ActiveMQ queue which later gets processed by a processor to save status information
The same message sending from ProducerTemplate to ActiveMQ queue somehow breakes the same code by not triggering logs on console, and saving status information to a randomly generated file name.
Desired Behavior:
Both sending method gets the messages processed the same way
Code Explanation:
On below codes the Save processor is the one producing the weird behavior where logs dont show up on console and writes to some random file, file name is basically the clientID from ActiveMQ
StartRoute calling the activemq:save works correctly
Code:
public class Save implements Processor {
public void process(Exchange exchange) throws Exception {
try {
Map<String, Object> map = new HashMap<String, Object>() {};
map.put(".....", ".....");
map.put(".....", ".....");
map.put(".....", ".....");
map.put(".....", ".....");
ProducerTemplate template = exchange.getContext().createProducerTemplate();
String response = template.requestBodyAndHeaders("activemq:save", "Batch Job Started", map, String.class);
FluentProducerTemplate FluentTemplate = exchange.getContext().createFluentProducerTemplate();
String result = FluentTemplate
.withHeader(".....", ".....")
.withHeader(".....", ".....")
.withHeader(".....", ".....")
.withHeader(".....", ".....")
.withHeader(".....", ".....")
.to("activemq:save")
.request(String.class);
} catch (Exception e) {
.....
}
}
}
public class StartRoute extends RouteBuilder {
restConfiguration()
.component("servlet")
.enableCORS(true);
rest("/start").id("start")
.post("/{filename}")
.route()
.....
.process(new Save())
.wireTap(startBackgroundProcessRoute)
.to("activemq:save")
.endRest();
}
public class SaveRoute extends RouteBuilder {
from("activemq:save")
.log("started saving")
.process(new FormatText())
.to("file:///status");
}
This question came from my original problem described here:
Camel Multicast Route call order
Solution can be found also there:
Camel Multicast Route call order
However to satisfy this question:
It seems Camel have few bugs regarding producer templates and maybe ActiveMQ, this is my initial conclusion.
The Only way i was able to use ProducerTemplate without issue is to use the send function with Exchanges, send() sends the messages correctly to the ActiveMQ same way as the to() however for whatever reason the content was still not written to the file.
After I dropped the ActiveMQ between the routes, everything started to work consistently. So possible miss configuration on ActiveMQ component or possible another camel framework bug.
If anyone knows the exact answer i would be happy to hear / see the reason for this behavior.
Code example:
public class SaveProcessor implements Processor {
public void process(Exchange exchange) throws Exception {
ProducerTemplate template = exchange.getContext().createProducerTemplate();
template.send(Utilities.trackBatchJobStatus, exchange);
/** NOTE
* DO NOT USE any other functions which are not working with EXCHANGES.
* Functions that uses body, header values and such are bugged
*/
}
}
10:39
I want to create a route that returns a JSON response to a user via REST and then sends data to BigQuery. Something like:
rest()
.get("/getSchedule")
.route()
.process("business logic - creates object with schedule AND big query statement")
The problem is that these are 2 different things. For the REST response to be correct I have to put the object with the schedule into the exchange body. But in order to send the BigQuery statement to the BigQuery component I have to set the BQ statement into the exchange object. Which messes up the REST response.
How can I accomplish this?
OK, I figured this out. What I want is a solution that returns with the REST response immediately and sends some other data to BigQuery as well. Since the 2nd step may block I will put an asynchronous process in between (could be a queue but in my case was Google PubSub). My route looks like this (leaving out things like .consumes, .outTypes for clarity):
rest()
.get("/getSchedule")
.route()
.processor(<business_logic>)
.wiretap("direct:pubsub");
from("direct:pubsub")
.process("processorA")
.to("google-pubsub topic");
from("google-pubsub subscription")
.processor("processorB")
.to("google-bigquery");
The first processor with business logic computes desired schedule data, puts it into a POJO, and puts a bigquery statement into an exchange header named "event".
#Override
public void process(Exchange exchange)
{
<business logic>
ScheduleData sData = new ScheduleData();
<insert values>
exchange.getIn().setBody(sData)
String insertStatement = "<insert_stmt>"
exchange.getIn().setHeader("event",insertStatement )
}
The ScheduleData data becomes the response I want. Now ProcessorA does some needed work before we put the data into the topic:
#Override
public void process(Exchange exchange)
{
String stmt = (String)exchange.getIn().getHeader("event")
byte[] data = <convert "stmt" into UTF8 bytes>
exchange.getIn().setBody(data);
}
Now the BigQuery data is sent through PubSub, received by Camel, and sent to ProcessorB.
public void process(Exchange exchange)
{
byte[] data = exchange.getIn().getBody(byte[].class)
String stmt = <convert UTF8 bytes to string>
exchange.getIn().setBody(stmt);
Putting the statement back in the exchange body it can now be sent to the BiqQuery component. Which I have not solved yet. But at least the wiretap method was the answer to my use case.
I've faced with behavior that I can't understand. This issue happens when Split with AggregationStrategy is executed and during one of the iterations, an exception occurs. An exception occurs inside of Splitter in another route (direct endpoint which is called for each iteration). Seems like route execution stops just after Splitter.
Here is sample code.
This is a route that builds one report per each client and collects names of files for internal statistics.
#Component
#RequiredArgsConstructor
#FieldDefaults(level = PRIVATE, makeFinal = true)
public class ReportRouteBuilder extends RouteBuilder {
ClientRepository clientRepository;
#Override
public void configure() throws Exception {
errorHandler(deadLetterChannel("direct:handleError")); //handles an error, adds error message to internal error collector for statistic and writes log
from("direct:generateReports")
.setProperty("reportTask", body()) //at this point there is in the body an object of type ReportTask, containig all data required for building report
.bean(clientRepository, "getAllClients") // Body is a List<Client>
.split(body())
.aggregationStrategy(new FileNamesListAggregationStrategy())
.to("direct:generateReportForClient") // creates report which is saved in the file system. uses the same error handler
.end()
//when an exception occurs during split then code after splitter is not executed
.log("Finished generating reports. Files created ${body}"); // Body has to be List<String> with file names.
}
}
AggregationStrategy is pretty simple - it just extracts the name of the file. If the header is absent it returns NULL.
public class FileNamesListAggregationStrategy extends AbstractListAggregationStrategy<String> {
#Override
public String getValue(Exchange exchange) {
Message inMessage = exchange.getIn();
return inMessage.getHeader(Exchange.FILE_NAME, String.class);
}
}
When everything goes smoothly after splitting there is in the Body List with all file names. But when in the route "direct:generateReportForClient" some exception occurred (I've added error simulation for one client) than aggregated body just contains one less file name -it's OK (everything was aggregated correctly).
BUT just after Split after route execution stops and result that is in the body at this point (List with file names) is returned to the client (FluentProducer) which expects ReportTask as a response body.
and it tries to convert value - List (aggregated result) to ReportTask and it causes org.apache.camel.NoTypeConversionAvailableException: No type converter available to convert from type
Why route breaks after split? All errors were handled and aggregation finished correctly.
PS I've read Camel In Action book and Documentation about Splitter but I haven't found the answer.
PPS project runs on Spring Boot 2.3.1 and Camel 3.3.0
UPDATE
This route is started by FluentProducerTemplate
ReportTask processedReportTask = producer.to("direct:generateReports")
.withBody(reportTask)
.request(ReportTask.class);
The problem is error handler + custom aggregation strategy in the split.
From Camel in Action book (5.3.5):
WARNING When using a custom AggregationStrategy with the Splitter,
it’s important to know that you’re responsible for handling
exceptions. If you don’t propagate the exception back, the Splitter
will assume you’ve handled the exception and will ignore it.
In your code, you use the aggregation strategy extended from AbstractListAggregationStrategy. Let's look to aggregate method in AbstractListAggregationStrategy:
#Override
public Exchange aggregate(Exchange oldExchange, Exchange newExchange) {
List<V> list;
if (oldExchange == null) {
list = getList(newExchange);
} else {
list = getList(oldExchange);
}
if (newExchange != null) {
V value = getValue(newExchange);
if (value != null) {
list.add(value);
}
}
return oldExchange != null ? oldExchange : newExchange;
}
If a first exchange is handled by error handler we will have in result exchange (newExchange) number of properties set by Error Handler (Exchange.EXCEPTION_CAUGHT, Exchange.FAILURE_ENDPOINT, Exchange.ERRORHANDLER_HANDLED and Exchange.FAILURE_HANDLED) and exchange.errorHandlerHandled=true. Methods getErrorHandlerHandled()/setErrorHandlerHandled(Boolean errorHandlerHandled) are available in ExtendedExchange interface.
In this case, your split finishes with an exchange with errorHandlerHandled=true and it breaks the route.
The reason is described in camel exception clause manual
If handled is true, then the thrown exception will be handled and
Camel will not continue routing in the original route, but break out.
To prevent this behaviour you can cast your exchange to ExtendedExchange and set errorHandlerHandled=false in the aggregation strategy aggregate method. And your route won't be broken but will be continued.
#Override
public Exchange aggregate(Exchange oldExchange, Exchange newExchange) {
Exchange aggregatedExchange = super.aggregate(oldExchange, newExchange);
((ExtendedExchange) aggregatedExchange).setErrorHandlerHandled(false);
return aggregatedExchange;
}
The tricky situation is that if you have exchange handled by Error Handler as not a first one in your aggregation strategy you won't face any issue. Because camel will use the first exchange(without errorHandlerHandled=true) as a base for aggregation.
I am newcomer in Apache Camel. Please have a look to my code bellow:
I have a service which exposed as cxf webservice:
interface CxfService{
public OutputType hello(InputType input);
}
This is my route:
from("cxf:/test?serviceClass=" + CxfService.class.getName())
.to("log:cxfLog1")
.recipientList(simple("direct:${header.operationName}"));
from("direct:hello")
.process(new Processor(){
public void process(Exchange exchange) throws Exception {
InputType file = exchange.getIn().getBody(InputType.class);
exchange.getOut().setBody(new OutputType());
}
});
The code works as expected, it consume InputType and produce OutputType.
I want to borrow my body to do another stuffs, so i rewrite that like this:
from("cxf:/test?serviceClass=" + CxfService.class.getName())
.to("log:cxfLog1")
.recipientList(simple("direct:${header.operationName}"));
from("direct:hello")
.process(new Processor(){
public void process(Exchange exchange) throws Exception {
InputType file = exchange.getIn().getBody(InputType.class);
exchange.getOut().setHeader("header.temporary", new OutputType());
}
})
.to("some endpoint")
.setBody(simple("${header.temporary}"));
This webservice consume InputType and produce nothing. What wrong with that?
In your second piece of code, when setting the header.temporary, you should change two things:
setHeader("temporary", new OutputType()) - the 'header' prefix isn't
needed - you're addressing headers directly via the method call.
Use getIn() instead of getOut(). The input will get copied to the
output. You may want to do some research into the procedure for
Camel building the out message for details - I'm not 100% sure of
this one.
Change
exchange.getOut().setHeader("header.temporary", new OutputType());
To
exchange.getIn().setHeader("temporary"), new OutputType());
.setHeader() is when you use the simple language. In 99% of the cases getIn() is sufficient.
From a programming point of view, I have a very simple business case. However, I can't figure out how to implement it using Apache Camel... Well, I have 2 JMS queues: one to receive commands, another - to store large number of message which should be delivered to external system in a batches of 1000 or less.
Here is the concept message exchange algorithm:
upon receiving a command message in 1st JMS queue I prepare XML
message
Send the XML message to external SOAP Web Service to obtain a usertoken
Using the usertoken, prepare another XML message and send it to a REST service to obtain jobToken
loop:
4.1. aggregate messages from 2nd JMS queue in batches of 1000, stop aggregation at timeout
4.2. for every batch, convert it to CSV file
4.3. send csv via HTTP Post to a REST service
4.4. retain batchtoken assigned to each batch
using the jobtoken prepare XML message and send to REST service to commit the batches
using batchtoken check execution status of each batch via XML message to REST service
While looking at Camel I could create a sample project where I can model out the exchange 1-3, 5:
from("file:src/data?noop=true")
.setHeader("sfUsername", constant("a#fd.com"))
.setHeader("sfPwd", constant("12345"))
.to("velocity:com/eip/vm/bulkPreLogin.vm?contentCache=false")
.setHeader(Exchange.CONTENT_TYPE, constant("text/xml; charset=UTF-8"))
.setHeader("SOAPAction", constant("login"))
.setHeader("CamelHttpMethod", constant("POST"))
.to("http4://bulklogin") // send login
.to("xslt:com/eip/xslt/bulkLogin.xsl") //xslt transformation to retrieve userToken
.process(new Processor() {
#Override
public void process(Exchange exchange) throws Exception {
String body = (String) exchange.getIn().getBody();
String[] bodyParts = body.split(",");
exchange.getProperties().put("userToken", bodyParts[0]);
.....
}
})
.to("velocity:com/eip/vm/jobInsertTeamOppStart.vm")
.setHeader(Exchange.CONTENT_TYPE, constant("application/xml; charset=UTF-8"))
.setHeader("X-Session", property("userToken"))
.setHeader("CamelHttpMethod", constant("POST"))
.to("http4://scheduleJob") //schedule job
.to("xslt:com//eip/xslt/jobInfoTransform.xsl")
.process(new Processor() {
#Override
public void process(Exchange exchange) throws Exception {
String body = (String) exchange.getIn().getBody();
exchange.getProperties().put("jobToken",body.trim());
}
})
//add batches in a loop ???
.to("velocity:com/eip/vm/jobInsertTeamOppEnd.vm")
.setHeader(Exchange.HTTP_URI, simple("https://na15.com/services/async/job/${property.jobToken}"))
.setHeader(Exchange.CONTENT_TYPE, constant("application/xml; charset=UTF-8"))
.setHeader("X-ID-Session", property("userToken"))
.setHeader("CamelHttpMethod", constant("POST"))
.to("http4://closeJob") //schedule job
//check batch?
.bean(new SomeBean());
So, my question is:
How can I read messages from my 2nd JMS queue?
This doesn't strike me as a very good use-case for a single camel route. I think you should implement the main functionality in a POJO and use Camels Bean Integration for consuming and producing messages. This will result in much more easy to maintain code, and also for easier Exception handling.
See https://camel.apache.org/pojo-consuming.html