I am trying to setup a simple camel route which reads from a sqlite table and prints the record (later it would be written to a file).
The flow I have setup is below
bindToRegistry("sqlConsumer", new SqliteConsumer());
bindToRegistry("sqliteDatasource", dataSource());
from("sql:select * from recordsheet_record_1 where col_1 = 'A5'?dataSource=#sqliteDatasource")
.to("bean:sqlConsumer?method=consume")
.end();
And the SqliteConsmer as below
public class SqliteConsumer {
public void consume(Map<String, Object> data, Exchange exchange) {
System.out.println("Map: '" + data + "'");
//TODO: append to file
}
}
When I execute the route, it should only execute once (prints once), but, it keeps on printing... Am I doing anything wrong here?
I am new to camel framework so any help or guide would be much appreciated.
Thanks.
It is a polling consumer so it polls the source according to the configuration, you can find more info here: https://camel.apache.org/components/latest/eips/polling-consumer.html
Related
I want to run a processor upon file appearance in a directory. My file url is like this:
file:{{file.root}}in?include=.*\.csv&charset=windows-1251&move=../out/done
A procedure that associates an url with a processor is like this:
MessageProcessor getOrCreateConsumer(CamelContext context, String uri) {
Endpoint endpoint = context.getEndpoint(uri);
endpoint.setCamelContext(context); // added this out of desperation, doesn't help
processor = new MessageProcessor();
try {
Consumer consumer = endpoint.createConsumer(processor);
endpoint.start(); // do we need this at all? works the same without it
consumer.start();
} catch (Exception e) {
throw new RuntimeException(e);
}
return processor;
}
}
MessageProcessor is a processor that does some things to an exchange.
Everything seems to work except the file doesn't get moved to the ../out/done directory. While debugging I can't get when the endpoint is configured to provide the file message exchange with this post operation.
I think I am missing some magic call that is normally invoked by a RouteBuilder and that will fully configure the file endpoint. Can you please help me out?
I have written a camel route which polls a folder and sends it to Azure Blob Container
I followed the example mentioned in the Azure document page
https://github.com/apache/camel/blob/master/components/camel-azure/src/main/docs/azure-blob-component.adoc
I am reversing the route. Instead of a consumer, I am using the Azure Blob Producer.
This is my route. I have used Java DSL.
from("file://C:/camel/source1").to("azure-blob://datastorage/container1/BLOB1?credentials=#credentials&operation=updateBlockBlob")
When I placed a file, I got the following error.
**java.lang.IllegalArgumentException: Unsupported blob type:org.apache.camel.component.file.GenericFile
at org.apache.camel.component.azure.blob.BlobServiceProducer.getInputStreamFromExchange(BlobServiceProducer.java:474) ~[camel-azure-2.19.2.jar:2.19.2]
at org.apache.camel.component.azure.blob.BlobServiceProducer.updateBlockBlob(BlobServiceProducer.java:143) ~[camel-azure-2.19.2.jar:2.19.2]
at org.apache.camel.component.azure.blob.BlobServiceProducer.process(BlobServiceProducer.java:79) ~[camel-azure-2.19.2.jar:2.19.2]**
I was able to fix this. I rewrote my route as.
from("file://C:/camel/source1")
.process(new Processor() {
#Override
public void process(Exchange exchange) throws Exception {
Object file = exchange.getIn().getMandatoryBody();
exchange.getOut().setBody(
GenericFileConverter.genericFileToInputStream(
(GenericFile<?>) file, exchange));
}
})
.to("azure-blob://datastorage/container1/BLOB1?credentials=#credentials&operation=updateBlockBlob")
.to("mock:Result");
My Question is, do I need to really write the processor? Shouldn't the camel component be receiving a stream or a File Object?
Yeah this is a little bug. I have logged a ticket: https://issues.apache.org/jira/browse/CAMEL-11844
You can do the workaround you did, or you can add a .convertBodyTo and convert to a FileInputStream, String etc.
from("file://C:/camel/source1")
.convertBodyTo(String.class)
...
I have two routes which execute some command every 2 seconds on different servers and print the output to the same file:
camelCtx.addRoutes(new RouteBuilder() {
#Override
public void configure() {
from("ssh://username:password#host1:port?delay=2&pollCommand=whoami")
.to("file:///tmp/?fileName=test.txt");
}
});
camelCtx.addRoutes(new RouteBuilder() {
#Override
public void configure() {
from("ssh://username:password#host2:port?delay=2&pollCommand=whoami")
.to("file:///tmp/?fileName=test.txt");
}
});
I would like to be sure that the two routes were started at the same time and also prefix output of each command with timestamp when route was started. For the second problem I tried custom process:
.process(exchange -> {
String body = exchange.getIn().getBody(String.class);
exchange.getIn().setBody(System.currentTimeMillis() + " " + body);
})
but it obviously gives the time when the output was received.
I also can execute date +%s%N before executing the command, so that pollCommand parameter would look like this:
...&pollCommand=date +%s%N;whoami"...
but in this case it's the time when connection to the server is already established, which is a bit too late...
So how to get the 'start time' of the route?
And also how to synchronise several routes so that they execute simultaneously?
If you mean the time that the exchange (camel message) on the route was started/created then you can access that information from the exchange property.
For example from a Camel Processor you can do:
Date created = exchange.getProperty(Exchange.CREATED_TIMESTAMP, Date.class);
You can use that information to build a file name which you can set with the header Exchange.FILE_NAME when will then override the filename configured in the endpoint uri, so you can include the timestamp.
We are using apache Camel for routing and extracting from the files.
I have a situation where I need to get the data from a file on a shared folder and data from database. I need to combine the data only when data from both sides have arrived. If either side has not received then my data combine process should wait till both sides are present.
Is it possible to possible? How can I achieve that? Any sample code?
Something must trigger the process - either the file or the database so pick one.
Then you can use the enricher pattern to populate the other source (when data is ready). An aggregation strategy is used to combine the data. You typically write the aggregation strategy in java.
The link has examples how to enrich and merge data. You can find out how to handle database and files in the Camel documentation.
I use this for zip processed file with processing log together. I attached an example, hope it will help you.
//Archived
from("direct:" + EnvironmentSetup.ARCHIVED)
.routeId(ROUTES.ARCHIVED.name())
.setHeader(HEADER_ZIP_AGG_ID, header(Exchange.FILE_NAME))
.setHeader(HEADER_AFTER_ZIP_DEST).constant(getArchiveUri())
.setHeader(HEADER_STATUS).constant(STATUS.SUCCESS)
.pipeline()
.to("direct:" + EnvironmentSetup.ARCHIVED_ZIP)
.end()
.pipeline()
.setHeader(Exchange.FILE_NAME, header(Exchange.FILE_NAME).append(".report"))
.setBody(header(ProcessManager.PROCESS_LOG).convertToString())
.to("direct:" + EnvironmentSetup.ARCHIVED_ZIP)
.end()
.end();
from(
"direct:" + EnvironmentSetup.DECRYPT_FAILED_ZIP,
"direct:"+EnvironmentSetup.PROCESS_FAILED_ZIP,
"direct:"+EnvironmentSetup.ARCHIVED_ZIP
)
.routeId("ZIP")
.aggregate(header(HEADER_ZIP_AGG_ID), new CopiedGroupedExchangeAggregationStrategy())
.completionSize(2)
.marshal(zipFileDataFormat)
.multicast()
.pipeline()
.setHeader(Exchange.FILE_NAME, simple(String.format(
"${in.header.%s}/${in.header.%s}", HEADER_EMAIL, Exchange.FILE_NAME))) //header(HEADER_EMAIL). header(Exchange.FILE_NAME))
//.dynamicRouter(header(HEADER_AFTER_ZIP_DEST))
.to("direct:dynamic")
.end()
.pipeline()
.marshal(encryption)
.setHeader(Exchange.FILE_NAME, simple(String.format(
"${in.header.%s}/${in.header.%s}.gpg", HEADER_EMAIL, Exchange.FILE_NAME)))
//.setHeader(Exchange.FILE_NAME, header(Exchange.FILE_NAME).append(".gpg"))
.to("direct:"+EnvironmentSetup.SEND_BACK)
.end()
.end() //end aggregate
.end();
CopiedGroupedExchangeAggregationStrategy.java
public class CopiedGroupedExchangeAggregationStrategy extends
AbstractListAggregationStrategy<Exchange> {
#Override
public boolean isStoreAsBodyOnCompletion() {
// keep the list as a property to be compatible with old behavior
return true;
}
#Override
public Exchange getValue(Exchange exchange) {
return exchange.copy();
}
}
I'm working on a camel prototype which uses two start points in the same camel context.
The first route consumes messages which are used to "configure" the application. Messages are loaded in a configuration repository through a configService bean:
// read configuration files
from("file:data/config?noop=true&include=.*.xml")
.startupOrder(1)
.to("bean:configService?method=loadConfiguration")
.log("Configuration loaded");
The second route implements a recipient list eip pattern, delivering a different kind of input messages to a number of recipients, which are read dinamically from the same configuration repository:
// process some source files (using configuration)
from("file:data/source?noop=true")
.startupOrder(2)
.unmarshal()
.to("setupProcessor") // set "recipients" header
.recipientList(header("recipients"))
// ...
The question that arises now is how to synchronize them, so the second route "waits" if the first is processing new data.
I'm new to Apache Camel and pretty lost on how to approach such a problem, any suggestion would be appreciated.
Use aggregate in combination with the possibility to start and stop routes dynamically:
from("file:data/config?noop=true&include=.*.xml")
.id("route-config")
.aggregate(constant(true), new MyAggregationStrategy()).completionSize(2).completionTimeout(2000)
.process(new Processor() {
#Override
public void process(final Exchange exchange) throws Exception {
exchange.getContext().startRoute("route-source");
}
});
from("file:data/source?noop=true&idempotent=false")
.id("route-source") // the id is needed so that the route is found by the start and stop processors
.autoStartup(false) // this route is only started at runtime
.aggregate(constant(true), new MyAggregationStrategy()).completionSize(2).completionTimeout(2000)
.setHeader("recipients", constant("direct:end")) // this would be done in a separate processor
.recipientList(header("recipients"))
.to("seda:shutdown"); // shutdown asynchronously or the route would be waiting for pending exchanges
from("seda:shutdown")
.process(new Processor() {
#Override
public void process(final Exchange exchange) throws Exception {
exchange.getContext().stopRoute("route-source");
}
});
from("direct:end")
.log("End");
That way, route-source is only started when route-config is completed. route-config and consequently route-source are restarted if new files are found in the config directory.
You can also place an "on completion" http://camel.apache.org/oncompletion.html in the first route that activates the second one.
Apache camel File will create a lock for the file that being processed. Any other File process on this file will not pool on if there is a lock (except if you put consumer.exclusiveReadLock=false)
source :
http://camel.apache.org/file.html => URI Options => consumer.exclusiveReadLock