I am reading camel-example-sql, it has two routes defined as below:
<!-- route that generate new orders and insert them in the database -->
<route id="generateOrder-route">
<from uri="timer:foo?period=5s"/>
<transform>
<method ref="orderBean" method="generateOrder"/>
</transform>
<to uri="sql:{{sql.insertOrder}}"/>
<log message="Inserted new order ${body[id]}"/>
</route>
<!-- route that process the orders by picking up new rows from the database
and when done processing then update the row to mark it as processed -->
<route id="processOrder-route">
<from uri="sql:{{sql.selectOrder}}?consumer.onConsume={{sql.markOrder}}"/>
<to uri="bean:orderBean?method=processOrder"/>
<log message="${body}"/>
</route>
I can understand first route, which fires every 5 second. But when does second route be triggered?
The 2nd route uses the SQL select statement from {{sql.selectOrder}} to query the database, and if there is a resultset, thean each row becomes a message that is routed. If the resultset is empty then no message is routed.
The 2nd route use a scheduler that runs every every 500 millis (eg the consumer.delay) option.
http://camel.apache.org/sql-component
Related
I'm trying to implement a simple streaming pipeline:
Get list of users from a remote REST endpoint, spliting the list into individual messages
For each user I have to enrich it with information from a SQL parametric table (departments), e.g.:
Initial message (User 1)
id:1
departmentId: 1
parentDepartmentId: 23
Departments
Department 1
id: 1
name: Dep1
Department 2
id: 23
name: Dep23
Enriched User 1
id:1
departmentId: 1
departmentName: Dep1
parentDepartmentId: 23
parentDepartmentName: Dep23
Route context:
<routeContext id="route1" xmlns="http://camel.apache.org/schema/spring">
<route id="route1">
<from id="_from1" uri="timer:mytimer"/>
<setHeader headerName="CamelHttpMethod">
<constant>POST</constant>
</setHeader>
<setHeader headerName="Content-Type">
<constant>application/json</constant>
</setHeader>
<setBody>
<simple>{ "new": "true" }</simple>
</setBody>
<to id="_apiCall1" uri="https4://myapi/v1/users/search"/>
<split streaming="true">
<simple>${body}</simple>
<to uri="direct:processNewUser"/>
</split>
</route>
<route id="processNewUser">
<from uri="direct:processNewUser"/>
<enrich strategyRef="myAggregationStrategy">
<simple>sql:select * from departments"</simple>
</enrich>
</route>
</routeContext>
I need to get the whole departments table to be able to enrich the user information but I want to avoid doing so for every message.
Is there a way to store the content of the sql query and reuse it during the enrichment phase?
Can't you get the departments first? It seems you could do that. Then, you'd set the result to an exchange property for instance and use it to enrich each user.
There are actually many ways to achieve that.
I am trying to use selectors on an amqp with Azure Service Bus consumer. However, for some reason the route is also consuming messages that do not match the selector.
Here's an example:
This route generates messages and append a header:
<route id="MessageGenerator">
<from uri="timer:generator?delay=5000&period=5000"/>
<setHeader headerName="INSTANCE_ID">
<simple>{{env:INSTANCE_ID}}</simple>
</setHeader>
<to uri="amqp:queue:external_queue" />
</route>
While this route should consume only those that contain INSTANCE_ID matching 2 possible values: env:INSTANCE_ID or Any.
<route id="ExternalConsumer">
<from uri="amqp:queue:external_queue?selector=INSTANCE_ID IN ('{{env:INSTANCE_ID}}', 'Any')"/>
<log message="{{env:INSTANCE_ID}} consumed message with Instance ID: ${header.INSTANCE_ID}" logName="AMQP_TEST" loggingLevel="INFO"/>
</route>
But the the log shows that it is consuming any message, regardless of the selector specifying which ones.
Am I missing something?
Thanks!
Issue here was that Azure Service Bus does NOT support selectors on queues. I switched to topics, which already have filters per subscription.
I have created multiple routes(say department, Employee) which takes input from file system folders(say department, Empployee) and process those files.
Now, I want to make them dependent. So, if I upload both emp.csv and dept.csv in those folders then it will process department file first and once complete it will start processing file for employee.
is there any way in camel to achieve this.
I looked at Route startupOrdering and AutoStartup feature, but it will work only for the first time when starting routes. However, I need same behavior for entire route life.
Thanks.
<route id="b" xmlns="http://camel.apache.org/schema/spring">
<from uri="file:/home/dev/code/Integration/RunCamleExample/src/main/resources/csv/Department?repeatCount=1&noop=true&delay=10000"/>
<log message="Department data is : ${body}"/>
</route>
<route id="employee" xmlns="http://camel.apache.org/schema/spring">
<from uri="file:/home/dev/code/Integration/RunCamleExample/src/main/resources/csv/Employee?noop=true&delay=10000"/>
<log message="Employee data is : ${body}"/>
</route>
I suggest to use other logic to handle the task. Two simple ways to go:
Use pollEnrich
Use pollEnrich to collect extra resource (e.g. a file with known name in file system) once at the middle of a route
Flow: Collect department files (From Endpoint) --(for each department file from file system) -> collect single employee file (trigger pollEnrich once with known name) ----> do anything else (if any)
Use ControlBus
Use ControlBus component to control the status of routes (only one of the route in 'start' status)
Flow: Start route A --(when route A complete its goal)-> Suspend route A ---> Start route B --(when route B complete its goal)-> Suspend route B ---> Start route A [loop back to head]
Dependent route execution first can be achieved in Camel using "RouteContext".
Example: If route 'A' is executed before route 'B' then route 'A' should be defined as 'RouteContext' and route be is defined inside "camelContext" like below:
<routeContext id="A" xmlns="http://camel.apache.org/schema/spring">
<route id="A">
<from uri="file:/home/dev/code/Integration/RunCamleExample/src/main/resources/csv/Department?repeatCount=1&noop=true&delay=10000"/>
<log message="Department data is : ${body}"/>
</route>
</routeContext>
Then regular "camelContext" should be defined with reference to this routeContext first.
<camelContext id="test" xmlns="http://camel.apache.org/schema/spring">
<routeContextRef ref="A"/>
<route id="B">
<from uri="file:/home/dev/code/Integration/RunCamleExample/src/main/resources/csv/Employee?noop=true&delay=10000"/>
<log message="Employee data is : ${body}"/>
</route>
</camelContext>
I am processing file in cluster environment. The cluster works fine. It is being processed on Only one server.
But on the second server It identifies as duplicates but still execute the form route delete=true
ERROR:
org.apache.camel.component.file.GenericFileOperationFailedException: Cannot delete file:
I am setting header CamelRouteStop to true but the exchange still try's to delete a file, instead of stop executing the route.
All I need is to end the route if it is duplicate.
<camelContext xmlns="http://camel.apache.org/schema/spring">
<route>
<from uri="smb:url?delete=true"/>
<idempotentConsumer messageIdRepositoryRef="myRepo">
<header>messageId</header>
<setHeader headerName="fileExist">
<simple>true</simple>
</setHeader>
</idempotentConsumer>
<when>
<simple>${header.fileExist} == null</simple>
<log message="File ${header.CamelFileName} processing/processed by other Nodes - DUPLICATE" loggingLevel="INFO" />
<setHeader headerName="CamelRouteStop">
<simple <simple resultType="java.lang.Boolean">true</simple>>true</simple>
</setHeader>
</when>
</route>
</camelContext>
For CamelRouteStop you need to use setProperty, not setHeader.
I have a route that, when lauched, requests messages from two datasources (routeA and routeB) and aggregates them into a single message. Each aggregated message MUST contain exactly one routeA message and one routeB message, if not, then drop it.
This process must be launched at specific intervals (i.e: every 5 min).
My question is, how can I let the aggregator know that all messages from routeA and routeB where processed and the messages that didn't find their pair, must be droped?
I'm currently using completionTimeout feature, but I don't like this solution for obvious reasons.
I know camel has a completionFromBatchConsumer feature, but I don't know how to use it with with multiple datasets.
Am greatefull for any advice.
Here's what I have right now:
<!-- main route -->
<route id="main">
<camel:from uri="timer://timer1?period=20000"/>
<multicast>
<to uri="direct:startA"/>
<to uri="direct:startB"/>
</multicast>
</route>
<!-- messages from route A -->
<route id="routeA" />
<from uri="direct:startA" />
<to uri="sql:select * from sampleDB?dataSource=ds"/>
<split>
<simple>${body}<simple>
<marshal ref="ObjectAJsonConverter"/>
<unmarshal ref="ObjectAJsonConverter"/>
<to uri="bean:myProcessor?method=addObjectACorrelationKey"/>
<to uri="seda:myAggregator"/>
</split>
<!-- messages from route B -->
<route id="routeB" />
<from uri="direct:startB"/>
<to uri="ldap:ldapcontext?base=DC=company,DC=net"/>
<split>
<simple>${body}<simple>
<marshal ref="ObjectBJsonConverter"/>
<unmarshal ref="ObjectBJsonConverter"/>
<to uri="bean:myProcessor?method=addObjectBCorrelationKey"/>
<to uri="seda:myAggregator"/>
</split>
<!-- aggregate the messages, create new ObjectC that contains ObjectA and ObjectB -->
<!-- wait 200000 ms for all messages from routeA and routeB to enter the aggregator -->
<route id="aggretatorRoute">
<from uri="seda:myAggregator"/>
<aggregate ref="myEntityAggregator" completionSize="2" completionTimeout="200000" discartOnCompletionTimeout="true" ignoreInvalidCorrelationKeys="true">
<correlationExpression><simple>${in.header.objectid}</simple></correlationExpression>
<to uri="bean:myProcessor?method=doSomethingWithObjectC"/>
</aggregate>
You can just in your AggregationStrategy only aggregate one of ObjectA and one of ObjectB. So if you see a 2nd of either of them, then just not aggregate it. And if you then want to drop what you have done so far, then you can mark the exchange to stop, by setting
exchange.setProperty("CamelRouteStop", true);
And if you then want to drop this immediately, then add a completionPredicate, that checks if that stop has been set.
<completionPredicate><simple>${property.CamelRouteStop} == true</simple></completionPredicate>
And for the correlationExpression, you can likely just use <constant>true</constant> as it seems you only work on one group.
Thanks Claus. Actually I took a different approach. Instead using an aggregator to join three different object, I now query a list of ids and use those to progressively build my complex object.
<route id="composeObject">
<from uri="sql:select id from people?oneSource">
<split><simple>${body}</simple>
<to uri="direct:getobjectOne"/>;
<to uri="bean:addToComplexObject"/>
<to uri="direct:getObjectTwo/>
<to uri="bean:addToComplexObject"/>
<to uri="direct:getobjectThree/>
<to uri="bean:addToComplexObject"/>
<to uri="seda:outChannel"/>
</split>
</route>