I am new to Apache camel. I am working with Red Hat JBoss developer studio 11.0.0.GA. I am trying to read a bulk of records from a table and insert them into another one. since I couldn't insert all records at once (I think camel has limitation on inserting 7000 records). I used a camel loop. First I got all the records. Did some process on them and set the result list on a property. In addition, I set an Integer-value-property to imply that how many records we have to skip for the current iteration. Now my problem is that for the third round (iteration) the exchange object is changed. I debugged it and found out that the exchangeId is different. So the properties I had set are all gone.
<camelContext id="_context1" xmlns="http://camel.apache.org/schema/blueprint">
<route id="_route1">
<from id="_from1" uri="file:work/input"/>
<to id="_to2" uri="sqlAbniyeh: select id , CRATERSLIST from ABNIYEH_BRIDGE "/>
<setProperty propertyName="isThereAnyRecord">
<simple resultType="java.lang.Boolean">true</simple>
</setProperty>
<loop doWhile="true" id="_loop1" >
<simple>${exchangeProperty.isThereAnyRecord} != false</simple>
<process id="_process1" ref="craters"/>
<to id="_to1" uri="sqlAbniyeh: insert into Z_ABNIYEH_BRIDGE_CRATERS( ID , NUMBER1 , LENGTH , BRIDGE_ID) values( HIBERNATE_SEQUENCE.nextval , :#value1 , :#value2 , :#id)?batch=true"/>
</loop>
</route>
</camelContext>
This is my process method:
public void process(Exchange exchange) throws Exception {
try
{
System.out.println("Entered Process method!");
List<Map<String, Object>> currentList = new ArrayList<Map<String,Object>>();
List<Map<String,Object>> newList = new ArrayList<Map<String,Object>>();
int numberOfRecordsToSkip = 0;
int numberOfRecordsToSkipForTheNextTime ;
List<Map<String, Object>> currentListOnProperties = (List<Map<String,Object>>) exchange.getProperty("listOfRecords");
numberOfRecordsToSkip = exchange.getProperty("numberOfRecordsToSkip") != null ?
(Integer)exchange.getProperty("numberOfRecordsToSkip"): 0;
if(currentListOnProperties != null)
{
newList = currentListOnProperties;
}
else
{
// It occurs just the first time
currentList = (List<Map<String,Object>>)exchange.getIn().getBody();
newList = OrganizeListForInsert(currentList);
}
int temp = (numberOfRecordsToSkip + NUMBER_OF_RECORDS_FOR_EACH_ROUND);
if(temp < newList.size())
{
numberOfRecordsToSkipForTheNextTime = temp;
}
else
{
numberOfRecordsToSkipForTheNextTime = numberOfRecordsToSkip + ( currentList.size() - numberOfRecordsToSkip);
exchange.removeProperty("isThereAnyRecord");
exchange.setProperty("isThereAnyRecord", false);
}
exchange.removeProperty("numberOfRecordsToSkip");
exchange.setProperty("numberOfRecordsToSkip", new Integer(numberOfRecordsToSkipForTheNextTime));
exchange.setProperty("listOfRecords", newList);
List<Map<String, Object>> sublistOfNewList =
new ArrayList<Map<String,Object>>(newList.subList(numberOfRecordsToSkip, numberOfRecordsToSkipForTheNextTime));
exchange.getIn().setBody(sublistOfNewList);
}
catch(Exception e)
{
e.printStackTrace();
}
System.out.println("End of everything!!");
}
As Bedla said I should have used EIPs instead of writing everything in java. I used Splitter EIP and it works well for my case. Here is the new code:
<camelContext id="_context1" xmlns="http://camel.apache.org/schema/blueprint">
<route id="_route1">
<from id="_from1" uri="file:work/input"/>
<to id="_to2" uri="sqlAbniyeh: select id , CRATERSLIST from ABNIYEH_BRIDGE "/>
<process id="_process1" ref="craters"/>
<split>
<simple>${body}</simple>
<to id="_to1" uri="sqlAbniyeh: insert into ABNIYEH_BRIDGE_CRATERS( ID , NUMBER1 , LENGTH , BRIDGE_ID) values( HIBERNATE_SEQUENCE.nextval , :#value1 , :#value2 , :#id)"/>
</split>
</route>
</camelContext>
and the process method:
#Override
public void process(Exchange exchange) throws Exception {
try
{
List<Map<String, Object>> currentList = new ArrayList<Map<String,Object>>();
List<Map<String,Object>> newList = new ArrayList<Map<String,Object>>();
currentList = (List<Map<String,Object>>)exchange.getIn().getBody();
newList = OrganizeListForInsert(currentList);
exchange.getIn().setBody(newList);
}
catch(Exception e)
{
e.printStackTrace();
}
}
Related
Available examples of the usage of the Camel Test component show how to test the expectations of a route:
However what I need to do is mock the body (manually setting it) of an intermediate route, e.g.:
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd http://camel.apache.org/schema/spring http://camel.apache.org/schema/spring/camel-spring.xsd">
<bean id="exampleBean" class="xxx.ExampleBean"/>
<routeContext id="routesTest" xmlns="http://camel.apache.org/schema/spring">
<route>
<from uri="direct:route1" />
<to uri="direct:route2" />
<log message="${body}"/>
</route>
<route>
<from uri="direct:route2"/>
<to uri="bean:exampleBean"/>
<to uri="direct:route3" />
</route>
<route>
<from uri="direct:route3"/>
<log message="${body}"/>
</route>
</routeContext>
</beans>
In this scenario I want to completely avoid the actual execution of bean:exampleBean, mocking the result of its execution.
My test class:
public class MyTests extends CamelSpringTestSupport {
#Produce(uri = "direct:route1")
protected ProducerTemplate inputProducerTemplate;
#EndpointInject(uri = "mock:bean:exampleBean")
protected MockEndpoint mockBeanExampleBean;
#Test
public void testRoute() throws Exception {
CompletableFuture<Object> future = inputProducerTemplate.asyncSendBody("direct:route1", "Some message");
Object o = future.get();
}
#Override
public String isMockEndpoints() {
return "bean:exampleBean";
}
#Override
protected AbstractApplicationContext createApplicationContext() {
return new ClassPathXmlApplicationContext("spring/gesti-test-application-context.xml");
}
}
public class ExampleBean {
public String enhance(String message) {
System.out.println(message);
//Here I would call a REST API
return "MY API RESULT";
}
}
When using mockBeanExampleBean.whenAnyExchangeReceived(exchange -> exchange.getMessage().setBody("My message")); it allows to override the input to exampleBean, but doesn't avoid its execution.
In the context of your unit test, route2 might be a "mock" component instead. A clean way of achieving that is to declare the route(s) in the properties file. The legibility of the routes gets harder, though.
Then, you could:
#EndpointInject("mock://route2")
MockEndpoint mockSecondStep;
mockSecondStep.whenExchangeReceived(1, e -> {
List whatever = new ArrayList<>();
e.getMessage().setBody(whatever);
});
I solved it using an InterceptStrategy:
public class MyTests extends CamelSpringTestSupport {
#Test
public void testRoute() throws Exception {
CompletableFuture<Object> future = template.asyncSendBody("direct:route1", "Some message");
Object o = future.get();
assertEquals("INTERCEPTED!", o);
}
#Override
protected AbstractApplicationContext createApplicationContext() {
return new ClassPathXmlApplicationContext("spring/gesti-test-application-context.xml");
}
#Override
protected RouteBuilder createRouteBuilder() {
return new RouteBuilder() {
#Override
public void configure() {
context.getProcessorDefinition("bean:exampleBean").addInterceptStrategy(
(context, definition, target, nextTarget) -> exchange -> exchange.getOut().setBody("INTERCEPTED!"));
}
};
}
}
public class ExampleBean {
public String enhance(String message) {
System.out.println(message);
//Here I would call a REST API
return "MY API RESULT";
}
}
I serialize the files to a byte arrays and send it to the ActiveMQ queue (I using the code from this example: Apache ActiveMQ File Transfer Example):
private void sendFileAsBytesMessage(File file) throws JMSException, IOException {
BytesMessage bytesMessage = session.createBytesMessage();
// Set file name here
bytesMessage.setStringProperty("fileName", file.getName());
// Set file body here
bytesMessage.writeBytes(fileManager.readfileAsBytes(file));
MessageProduce msgProducer =
session.createProducer(session.createQueue(queueName));
MessageProducer msgProducer.send(bytesMessage);
}
Method readfileAsBytes is presented below:
public byte[] readfileAsBytes(File file) throws IOException {
try (RandomAccessFile accessFile = new RandomAccessFile(file, "r")) {
byte[] bytes = new byte[(int) accessFile.length()];
accessFile.readFully(bytes);
return bytes;
}
}
In my OSGi bundle I have a custom processor:
public class Deserializer implements Processor {
#Override
public void process(Exchange exchange) throws Exception {
// Get file body here
byte[] bytes = exchange.getIn().getBody(byte[].class);
}
}
I use it in my Spring DSL route as follows:
<?xml version="1.0"?>
<blueprint xmlns="http://www.osgi.org/xmlns/blueprint/v1.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="
http://www.osgi.org/xmlns/blueprint/v1.0.0 http://www.osgi.org/xmlns/blueprint/v1.0.0/blueprint.xsd
http://camel.apache.org/schema/blueprint http://camel.apache.org/schema/blueprint/camel-blueprint.xsd">
<bean id="activemq" class="org.apache.activemq.camel.component.ActiveMQComponent">
<property name="brokerURL" value="tcp://localhost:61616" />
<property name="userName" value="admin" />
<property name="password" value="admin" />
</bean>
<bean id="deserializer" class="org.fusesource.example.Deserializer"/>
<camelContext id="blueprintContext" trace="false" xmlns="http://camel.apache.org/schema/blueprint">
<route id="testRoute">
<from uri="activemq:source-queue"></from>
<process ref="deserializer"/>
<to uri="activemq:sink-queue"></to>
</route>
</camelContext>
</blueprint>
I need to get the file name in my processor. How can I do that?
The problem can be solved as follows. For example, I send the README.html file to the alfresco-queue. Then the body of the file and the file name, as well as some additional information can be obtained as follows:
public class MyRouteBuilder extends RouteBuilder {
#Override
public void configure() throws Exception {
from("activemq:alfresco-queue?username=admin&password=admin")
.process(new Processor() {
public void process(Exchange exchange) throws Exception {
// Get file body
byte[] bytes = exchange.getIn().getBody(byte[].class);
for(int i = 0; i < bytes.length; i++) {
System.out.print((char) bytes[i]);
}
// Get headers
Map<String, Object> headers = exchange.getIn().getHeaders();
Iterator iterator = headers.entrySet().iterator();
while (iterator.hasNext()) {
Map.Entry pair = (Map.Entry)iterator.next();
System.out.println(pair.getKey() + " == " + pair.getValue());
}
}
})
// SKIPPED
It gives the following output:
<head>
root<> title>README
</title>
</head>
<body>
Please refer to http://java.com/licensereadme
</body>
</html>
breadcrumbId == ID:63-DP-TAV-59000-1531813754416-1:1:1:1:1
fileName == README.html
JMSCorrelationID == null
JMSCorrelationIDAsBytes == null
JMSDeliveryMode == 2
JMSDestination == queue://alfresco-queue
JMSExpiration == 0
JMSMessageID == ID:63-DP-TAV-59000-1531813754416-1:1:1:1:1
JMSPriority == 4
JMSRedelivered == false
JMSReplyTo == null
JMSTimestamp == 1531813754610
JMSType == null
JMSXGroupID == null
JMSXUserID == null
I have a task with CSV files. I need to limit the size of the CSV because the backend engine has constraints on payload sizes.
The problem is extracting the header, the first record/row, saving it off and adding it back to the remaining splitted data, thus creating multiple files all with the same header. I was hoping to find an elagant way of handling this. what I have works but it is well, less than desirable coding.
Also, I need the group By parm to be programable, I am trying to find out now if this can be set by a property in the camelContext.
This is what I have, it works, but... and I can't get the groupBy to accept a parameter.
my route
<!-- route on Weekends -->
<route id="inRouteWkEndBfmt1" routePolicyRef="startPolicyWkEnd" autoStartup="false" >
<from id="mainProcessingRouteWkEnd" ref="AsciiGatewayBackfillmt1" />
<convertBodyTo type="java.lang.String" />
<log message="File ${file:name} was received."/>
<setHeader headerName="messageDateTime">
<simple>${date:now:MM-dd-yyyy-HH:mm:ss}</simple>
</setHeader>
<split streaming="true" >
<tokenize token="\n" group="50"/>
<log message="Split line Body: ${body}"/>
<process ref="asciiSplitterProcessor" />
<log loggingLevel="INFO" message="Successfully sent ${file:name} to MT1 Core for Analytics Observation." />
<to id="windowsShareTargetWkEnd" uri="file://{{target.folder}}" />
</split>
<process ref="asciiCleanUp" />
</route>
Code
public void process(Exchange exchange) throws Exception {
log.info("Ascii Splitter Processor :: start");
String inBody = exchange.getIn().getBody(String.class);
String fileName = (String) exchange.getIn().getHeader("CamelFileName");
String fileSuffix = fileName.substring(fileName.lastIndexOf("."), fileName.length());
String filePrefix = fileName.substring(0, fileName.lastIndexOf("."));
fileName = filePrefix + "_" + cntr + fileSuffix;
exchange.getIn().setHeader("CamelFileName",fileName);
cntr++;
fileName = (String) exchange.getIn().getHeader("CamelFileName");
log.info("File being processed: " + fileName );
log.debug("Message record: " + inBody);
StringBuilder sb = new StringBuilder();
Scanner sc = new Scanner(inBody);
if ( ! hdrFlag ) {
while ( sc.hasNextLine() ) {
record = sc.nextLine();
log.debug("record: " + record);
log.debug("HEADER FLAG: " + hdrFlag);
if ( !hdrFlag ){
HEADER = record + "\r\n";
hdrFlag = true;
log.debug("HEADER: " + HEADER);
}
sb.append(record).append("\r\n");
}
} else {
sb.append(HEADER).append(inBody);
}
sc.close();
exchange.getIn().setBody(sb.toString());
sb = new StringBuilder();
I think this is a bit more elagant than above. unfortunately I'm not on camel 2.9. but this works for sub unit of work that need to be joined at the server side for large payloads of CSV's, I'm converting to Json and sending to server.
thanks everyone. hope this helps someone else with the use case.
public void process(Exchange exchange) throws Exception {
log.info("Entering Extract Header Processor ...");
//if file is split in to multiple files modify the name with an index
String fileName = (String) exchange.getIn().getHeader("CamelFileName");
String fileSuffix = fileName.substring(fileName.lastIndexOf("."), fileName.length());
String filePrefix = fileName.substring(0, fileName.lastIndexOf("."));
fileName = filePrefix + "_" + fileCounter + fileSuffix;
fileCounter++;
//fileName = filePrefix + "_" + Integer.valueOf((int)exchange.getProperty("CamelSplitSize")) + fileSuffix; // need camel 2.9 for this to work, bummer
exchange.getIn().setHeader("CamelFileName",fileName);
log.info(" FILE NAME: " + exchange.getIn().getHeader("CamelFileName", fileName));
//log.info("File Counter: " + Integer.valueOf((int)exchange.getProperty("CamelSplitSize"))); // need camel 2.9 for this to work, bummer
log.info("File Counter: " + fileCounter );
//if this is the first split body, get the header to attach to the other split bodies
String body = exchange.getIn().getBody(String.class);
StringBuilder sb = new StringBuilder();
if ( (Integer.valueOf((int)exchange.getProperty("CamelSplitIndex")) == 0 ) ) {
List<String> serviceRecords = new ArrayList<String>(Arrays.asList(body.split(System.lineSeparator())));
StringBuilder header = getHeader( serviceRecords );
HEADER = header.toString();
exchange.getIn().setBody(body);
} else {
sb.append(HEADER).append(System.lineSeparator()).append(body);
exchange.getIn().setBody(sb.toString());
}
sb = new StringBuilder();
log.debug("HEADER: " + HEADER);
log.info("Exiting Extract Header Processor ... :: Finish");
}
public StringBuilder getHeader(List<String> serviceRecords) {
StringBuilder sb = new StringBuilder();
for ( int i = 0; i < 1; i++ ) {
log.debug("record: : " + serviceRecords.get(i).toString());
if ( i == 0 ) {
String[] sa = serviceRecords.get(i).toString().split(",");
for ( int j = 0; j < sa.length; ++j) {
if ( j != 0 ) {
sb.append(sa[j]).append(",");
}
}
sb.deleteCharAt(sb.lastIndexOf(",", sb.length()));
} else {
break;
}
}
return sb;
}
public void cleanHeader() {
HEADER = "";
fileCounter = 0;
}
}
Route
<route
id="core.accept.file.type.route"
autoStartup="true" >
<from uri="{{fileEntranceEndpoint}}" />
<choice>
<when>
<simple>${header.CamelFileName} regex '^.*\.(csv|CSV)$'</simple>
<log message="${file:name} accepted for processing..." />
<choice>
<when>
<simple>${header.CamelFileName} regex '^.*\.(CSV)$'</simple>
<setHeader headerName="CamelFileName">
<simple>${file:name.noext}.csv</simple>
</setHeader>
</when>
</choice>
<split streaming="true" >
<tokenize token="\n" group="600" />
<log message="Split Group Body: ${body}"/>
<to uri="bean:extractHeader" />
<to id="acceptedFileType" ref="pConsumer" />
</split>
<to uri="bean:extractHeader?method=cleanHeader"/>
<!-- <to id="acceptedFileType" ref="pConsumer" /> -->
</when>
<otherwise>
<log message="${file:name} is an unknown file type, sending to unhandled repo." loggingLevel="INFO" />
<to uri="{{unhandledArchive}}" />
</otherwise>
</choice>
</route>
My select query is
sql.selectOrder=select STID,CLLTR from GSI_DEVL.POLLUX_DATA WHERE PROCESSED='FALSE'
My Route is
<route id="markRowsAsProcessed-Route" >
<!-- <from uri="timer://markRowsAsProcessed?delay=5000"/>-->
<from uri="sqlComponent:{{sql.selectOrder}}?consumer.useIterator=false" />
<doTry>
<to uri="bean:rowProcessController"/>
<to uri="sqlComponent:{{sql.markRows}}?batch=true"/>
<doCatch>
<exception>java.sql.SQLException</exception>
<exception>java.lang.IllegalStateException</exception>
<exception>java.sql.SQLException</exception>
<exception>java.lang.ClassCastException</exception>
</doCatch>
</doTry>
</route>
My bean is
public class RowProcessController {
List<Map<String, Object>> stationUnMarkedList = new ArrayList<Map<String, Object>>();
List<Map<String, Object>> stationMarkedList = new ArrayList<Map<String, Object>>();
Map<String,Object> stationMap = null;
#SuppressWarnings("unchecked")
#Handler
public List<Map<String, Object>> markRowAsProcessed(Exchange exchange)
{
stationUnMarkedList = (List<Map<String, Object>>)exchange.getIn().getBody();
for(Map<String,Object> data: stationMarkedList) {
System.out.println(data.get("STID"));
stationMap=new HashMap<String,Object>();
stationMap.put("stationId", ((String)data.get("STID")));
stationMap.put("callLetter", ((String)data.get("CLLTR")));
stationMarkedList.add(stationMap);
}
return stationMarkedList;
}
}
I want to update the result set processed column to done or some value.
I tried
sql.markRows=UPDATE POLLUX_DATA SET PROCESSED='DONE' where stid=:stationId
But this does not update any values in the database. Why not?
Can you try and change the query's named parameter notation to :# instead of just :
sql.markRows=UPDATE POLLUX_DATA SET PROCESSED='DONE' where stid=:#stationId
I have a camel route that looks like this:
<camel:camelContext id="context" xmlns="http://camel.apache.org/schema/spring">
<propertyPlaceholder id="camelContextProperties" location="ref:lefrogen"/>
<contextScan/>
<template id="producerTemplate"/>
<template id="consumerTemplate"/>
<route id="chibzo">
<from uri="activemq:queueName"/>
<to uri="seda:internal?multipleConsumers=true"/>
</route>
<route id="chon">
<from uri="seda:internal?multipleConsumers=true"/>
<bean ref="beanName"/>
<to uri="seda:external?multipleConsumers=true"/>
</route>
</camel:camelContext>
The bean class is like this:
#Component
public class BeanName {
private final static Logger LOGGER = Logger.getLogger(BeanName.class.getName());
public static String message = null;
public static String map(String custom) {
message = custom;
return custom;
}
}
I have written a unit test which is supposed to get the data from the bean and then compare it with the data from seda:external. The problem is that I am getting null when I am fetching data from the seda queue. Here's the unit test:
public class Test{
#EndpointInject
ProducerTemplate producerTemplate;
#EndpointInject
ConsumerTemplate consumerTemplate;
#Autowired
BeanName beanName;
#Test
public void testName() throws Exception {
producerTemplate.sendBody("seda:internal","Good bye");
Thread.sleep(10000); //needs to sleep so that the bean gets executed before accessing
//beanName.message;
String message = beanName.message;
LOGGER.info("Printing" + message); //prints (Good bye)
String body = consumerTemplate.receiveBodyNoWait("seda:external", String.class);
LOGGER.info("Body is" + body); //prints null
}
Any reason why the queue is giving null?
Its due concurrent, and you are using recieveBodyNoWait which does a fast check. Instead use receiveBody and provide may a timeout value to avoid waiting if there is no messages on the queue.
Or add sleep in the code if you want to use receiveBodyNoWait still.