I am trying to download the same file multiple times using Camel ConsumerTemplate. It's not getting downloaded multiple times, but I am getting it done while using camel route.
I want to download the file using ConsumerTemplate multiple times. Here is the code that I am trying with ConsumerTemplate:
import org.apache.camel.CamelContext;
import org.apache.camel.ConsumerTemplate;
import org.apache.camel.Exchange;
import org.apache.camel.Processor;
public class DynamicConsumer implements Processor {
#Override
public void process(Exchange inExchange) throws Exception {
CamelContext camelContext = inExchange.getContext();
ConsumerTemplate consumerTemplate = camelContext.createConsumerTemplate();
String resource = "sftp://tester#localhost:22/myfiles?password=password&noop=true&idempotent=false&readLockMarkerFile=false&readLock=none&filter=#myFileFilter";
consumerTemplate.start();
Exchange resourceExchange = consumerTemplate.receive(resource,20000);
consumerTemplate.stop();
if(resourceExchange != null) {
inExchange.getOut().setBody(resourceExchange.getIn().getBody());
inExchange.getProperties().putAll(resourceExchange.getProperties());
inExchange.getOut().getHeaders().putAll(resourceExchange.getIn().getHeaders());
} else {
inExchange.getOut().setBody(null);
}
}
}
This dynamic consumer is getting called from the camel routes multiple times. So, each time, I expect to download the file present in given location. But it is not happening.
Here the code that I tried with camel route.
<from uri="sftp://tester#localhost:22/myfiles?password=password&noop=true&idempotent=false&readLockMarkerFile=false&readLock=none&filter=#myFileFilter"/>
As the documentation states you should call the ConsumerTemplate's –doneUoW(Exchange) function:
If you have used any of the receive methods which returns a Exchange
type then you need to invoke this method when you are done using the
returned Exchange.
-- ConsumerTemplate documentation
Try to add the following before consumerTemplate.stop():
consumerTemplate.doneUoW(resourceExchange);
Related
I have just started working around the Apache Camel. I have a requirement to implement an FTP/FTPS/SFTP client, which would be used to fetch the files from the respective servers. I was looking into the possibility of using Apache Camel to do this but I am still confused after going through the examples and the tutorials.
The requirement is to fetch the files from the FTP/SFTP servers when the request is received from the scheduler.
Following is the route created using EndPoint-DSL
#Component
public class FtpReceiveRoute extends EndpointRouteBuilder {
#Override
public void configure() throws Exception {
from(
ftp("localhost:2001/home/admin")
.account("admin")
.password("admin12345")
.recursive(true)
)
.routeId("ftpReceive")
.log("From done!")
.to("log:ftp-log")
.log("To done!!");
}
}
I am trying to use the above route by invoking it when the request is made to fetch the file like below.
#Override
protected FtpResponse doMessage(String param, FtpRequest req) {
FtpResponse response = new FtpResponse ();
CamelContext ctx = new DefaultCamelContext();
ctx.addRoutes(##route); //FtpReceiveRoute, add the Routebuilder instance as EndpointRouteBuilder is acceptable.
ctx.start();
//Might need to induce sleep so that all the files are downloaded
ctx.stop();
return response;
}
The confusion is around how to invoke the Camel process with the route. I have used EndpointRouteBuilder to create the route because of the type-safe creation of the endpoint URI. I am not getting an option to add this route to the CamelContext as it expects the RouteBuilder instance which is not type-safe.
Further, the CamelContext is the engine and to invoke the route I would need to start and stop this engine. This I am not able to digest if I need to start and stop the engine to execute a route then I would need to induce some sleep in between so that all files are downloaded. Just to add there are more routes that I need to add with the implementation. Once the engine is started it would load and execute all the added routes which is not the requirement.
Maybe I am not getting how to use this properly. Any resources aiding my situation are welcome. Thanks.
You should not create and start new camel context every time you want to fetch file from server. What you should do instead is start one when your application starts and use that for all your exchanges.
You can use Spring-boot to initialize CamelContext and add annotated RouteBuilders to it automatically. Check the maven archetype camel-archetype-spring-boot for example.
If you want to call camel routes from Java you can Inject CamelContext to your bean and use it to create ProducerTemplate. This can be used to invoke Routes defined in the RouteBuilder.
Using ProducerTemplate.send you can get the resulting exchange.
Using producer template
Using File-component which works very similary to ftp-component.
package com.example;
import org.apache.camel.builder.endpoint.EndpointRouteBuilder;
import org.springframework.stereotype.Component;
#Component
public class MySpringBootRouter extends EndpointRouteBuilder {
#Override
public void configure() {
from(direct("fileFromFTP"))
.routeId("fileFromFTP")
// reads files from <project>/input using file consumer endpoint
.pollEnrich(file("input"), 1000)
// If file is found, convert body to string.
// Which in this case will read contents of the file to string.
.filter(body().isNotNull())
.convertBodyTo(String.class)
.end()
;
}
}
package com.example;
import org.apache.camel.CamelContext;
import org.apache.camel.Exchange;
import org.apache.camel.ProducerTemplate;
import org.apache.camel.support.DefaultExchange;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.context.annotation.Configuration;
import org.springframework.scheduling.annotation.EnableScheduling;
import org.springframework.scheduling.annotation.Scheduled;
import static org.apache.camel.builder.endpoint.StaticEndpointBuilders.direct;
#Configuration
#EnableScheduling
public class MySpringBean {
#Autowired
CamelContext camelContext;
#Scheduled(fixedRate = 1000)
public void scheduledTask() {
System.out.println("Scheduled Task!");
if(camelContext.isStopped()) {
System.out.println("Camel context not ready yet!");
return;
}
useProducerTemplate();
}
public void useProducerTemplate(){
ProducerTemplate producerTemplate = camelContext.createProducerTemplate();
Exchange inExchange = new DefaultExchange(camelContext);
//synchronous call!
Exchange result = producerTemplate.send(direct("fileFromFTP").toString(), inExchange);
String resultBody = result.getMessage().getBody(String.class);
String fileName = result.getMessage().getHeader(Exchange.FILE_NAME, String.class);
if(resultBody != null){
System.out.println("Consumed file: "+ fileName + " contents: " + resultBody.toString());
}
else{
System.out.println("No file to consume!");
}
}
}
Depending on what you need to do with the files you could probably do that inside camel route. Then you would only need to call the producerTemplate.sendBody.
public void useProducerTemplate(){
ProducerTemplate producerTemplate = camelContext.createProducerTemplate();
Exchange inExchange = new DefaultExchange(camelContext);
producerTemplate.sendBody(direct("fileFromFTP").toString(), inExchange);
}
Starting stopping camel route
If you want to start polling file consumer only for a short while you can do start the route and use for example aggregation timeout to shutdown the route when no new files have been received in any given duration.
#Component
public class MySpringBootRouter extends EndpointRouteBuilder {
#Override
public void configure() {
AggregationStrategy aggregateFileNamesStrategy = AggregationStrategies
.flexible(String.class)
.accumulateInCollection(ArrayList.class)
.pick(header(Exchange.FILE_NAME))
;
from(file("input"))
.routeId("moveFilesRoute")
.autoStartup(false)
.to(file("output"))
.to(seda("moveFilesRouteTimeout"));
;
from(seda("moveFilesRouteTimeout"))
.routeId("moveFilesRouteTimeout")
.aggregate(constant(true), aggregateFileNamesStrategy)
.completionTimeout(3000)
.log("Consumed files: ${body.toString()}")
.process(exchange -> {
exchange.getContext().getRouteController().stopRoute("moveFilesRoute");
})
.end()
;
}
}
public void startMoveFilesRoute() {
try {
System.out.println("Starting moveFilesRoute!");
camelContext.getRouteController().startRoute("moveFilesRoute");
//Sending null body moveFilesRouteTimeout to trigger timeout if there are no files to transfer
camelContext.createProducerTemplate().sendBody(seda("moveFilesRouteTimeout").toString(), null);
} catch(Exception e) {
System.out.println("failed to stop route. " + e);
}
}
I am trying to create a camel component which consumes an API from an external service.
My Route is as follows
from("myComponent:entity?from=&to=")
.to("seda:one")
from("seda:one")
.aggregate(constant(true), new GroupedBodyAggregationStrategy())
.completionSize(5)
.completionTimeout(5000)
.process( new Processor1() )
to("seda:two")
.
.
.
from("seda:five")
.to("myComponent2:entity")
I implemented my component consumer as follows
public class MyComponentConsumer extends DefaultConsumer {
public MyComponentConsumer(MyComponentEndpoint endpoint, Processor processor) {
super(endpoint, processor);
}
#Override
protected void doStart() throws Exception {
super.doStart();
flag = true;
while ( flag ) {
//external API call
Resource resource = getNextResource();
if ( resource.next() == null ) {
flag = false;
}
Exchange ex = endpoint.createExchange(ExchangePattern.InOnly);
ex.getIn().setBody(resource.toString());
getAsyncProcessor().process(
ex
doneSync -> {
LOG.info("Message processed");
}
);
}
}
#Override
protected void doStop() throws Exception {
super.doStop();
System.out.println("stop ---- ");
}
}
Everything worked fine and the data was propogating through the route. My only problem was that data did not propogate to the next part until the whole of this process was completed. And the next parts were running asynchronously.
I looked at the example of StreamConsumer and tried to implement it to my code using a runnable and an executorService. But if I do that consumer stops as soon as it starts.
I changed the code to
public class MyComponentConsumer extends DefaultConsumer implements Runnable
and added
private ExecutorService executor;
getEndpoint().getCamelContext().getExecutorServiceManager().newSingleThreadExecutor(this, "myComponent");
executor.execute(this);
and moved my logic inside the run() method. But, the consumer thread ends as soon as it starts. and the async processor does not transfer the data properly.
Is there any other way to implement the functionality I need or am I mistaken somewhere here. Any help would be appreciated.
What version of camel are you using?
There was an issue with managing the state of consumer in camel 2.x which was fixed in camel 3.x CAMEL-12765 which can lead to the issue you are describing here.
If you are on camel 2.x try using newScheduledThreadPool instead of newSingleThreadExecutor.
Also executor.schedule(this, 5L, TimeUnit.SECONDS) instead of executor.execute(this).
Delayed start of executor might help avoid the problem you are facing.
I have written a camel route which polls a folder and sends it to Azure Blob Container
I followed the example mentioned in the Azure document page
https://github.com/apache/camel/blob/master/components/camel-azure/src/main/docs/azure-blob-component.adoc
I am reversing the route. Instead of a consumer, I am using the Azure Blob Producer.
This is my route. I have used Java DSL.
from("file://C:/camel/source1").to("azure-blob://datastorage/container1/BLOB1?credentials=#credentials&operation=updateBlockBlob")
When I placed a file, I got the following error.
**java.lang.IllegalArgumentException: Unsupported blob type:org.apache.camel.component.file.GenericFile
at org.apache.camel.component.azure.blob.BlobServiceProducer.getInputStreamFromExchange(BlobServiceProducer.java:474) ~[camel-azure-2.19.2.jar:2.19.2]
at org.apache.camel.component.azure.blob.BlobServiceProducer.updateBlockBlob(BlobServiceProducer.java:143) ~[camel-azure-2.19.2.jar:2.19.2]
at org.apache.camel.component.azure.blob.BlobServiceProducer.process(BlobServiceProducer.java:79) ~[camel-azure-2.19.2.jar:2.19.2]**
I was able to fix this. I rewrote my route as.
from("file://C:/camel/source1")
.process(new Processor() {
#Override
public void process(Exchange exchange) throws Exception {
Object file = exchange.getIn().getMandatoryBody();
exchange.getOut().setBody(
GenericFileConverter.genericFileToInputStream(
(GenericFile<?>) file, exchange));
}
})
.to("azure-blob://datastorage/container1/BLOB1?credentials=#credentials&operation=updateBlockBlob")
.to("mock:Result");
My Question is, do I need to really write the processor? Shouldn't the camel component be receiving a stream or a File Object?
Yeah this is a little bug. I have logged a ticket: https://issues.apache.org/jira/browse/CAMEL-11844
You can do the workaround you did, or you can add a .convertBodyTo and convert to a FileInputStream, String etc.
from("file://C:/camel/source1")
.convertBodyTo(String.class)
...
To retrieve some open data from a remote web server to process, I'm trying out Apache Camel.
The problem is that it seems that the data is never received. I have tried the jetty, ahc and cxf components but can't get it to work. For example like this:
import org.apache.camel.CamelContext;
import org.apache.camel.builder.RouteBuilder;
import org.apache.camel.impl.DefaultCamelContext;
public class CamelHttpDemo {
public static void main(final String... args) {
final CamelContext context = new DefaultCamelContext();
try {
context.addRoutes(new RouteBuilder() {
#Override
public void configure() throws Exception {
this.from("direct:start")
.to("ahc:http://camel.apache.org/")
.process(exchange -> {
System.out.println(exchange);
});
}
});
context.start();
Thread.sleep(10000);
context.stop();
} catch (final Exception e) {
e.printStackTrace();
}
}
}
No output is written so the line System.out.println(exchange); is never executed and I assume the data is not retrieved.
I'm using the most recent version of Apache Camel, 2.17.1.
You need some message producer in your route to emit Exchange that would trigger the http component. Your route starts with direct:start which cannot emit new Exchanges, it just sits and waits for someone to initiate the process.
The easiest way to make your route work is to replace direct:start with some producer. For instance, replacing it with this timer .from("timer://foo?fixedRate=true&period=10000") will trigger your http-request once every 10 seconds.
If you want to initiate the request manually, you need to create a ProducerTemplate and use it to send a message to direct:start. That would be:
ProducerTemplate template = context.createProducerTemplate();
template.sendMessage("direct:start", "Message body");
I'm trying to put together my first Akka/Camel application from "scratch" (read, "noob") using the following lib versions:
akka-camel: 2.2.0-RC1
According to all of the documentation I can find (Akka docs, user groups, etc.) all I have to do to consume from a file-based queue is set up my system this way:
Main class:
actorSystem = ActorSystem.create("my-system");
Props props = new Props(Supervisor.class);
ActorRef supervisor = actorSystem.actorOf(props, "supervisor");
Camel camel = CamelExtension.get(actorSystem);
CamelContext camelContext = camel.context();
camelContext.start();
Supervisor class:
import akka.actor.ActorRef;
import akka.actor.Props;
import akka.camel.javaapi.UntypedConsumerActor;
import org.apache.camel.Message;
/**
* Manages creation and supervision of UploadBatchWorkers.
*/
public class Supervisor extends UntypedConsumerActor {
#Override
public String getEndpointUri() {
return "file:///Users/myhome/queue";
}
#Override
public void preStart() {
String test = "test";
}
#Override
public void onReceive(Object message) {
if (message instanceof CamelMessage) {
// do something
}
}
My problem is that even though I know the supervisor object is being created and breaks during debugging on the preStart() method's "test" line (not to mention that if I explicitly "tell" it something it processes fine), it does not consume from the defined endpoint, even though I have another application producing messages to the same endpoint.
Any idea what I'm doing wrong?
Ok, the problem was my own fault and is clearly visible in the example code if you look at the Consumer trait from which the UntypedConsumerActor inherits.
This method:
#Override
public void preStart() {
String test = "test";
}
overrides its parent's preStart() method, right? Well, that parent method is actually the one that registers the consumer with the on-the-fly created endpoint, so while you can override it, you must call super() or it will not work.
Hope this is useful to someone down the road!
Try changing your instanceof inside of onReceive to this:
if (message instanceof CamelMessage){
//do processing here
}
Where CamelMessage is from package akka.camel. That's what the examples in the akka camel docs are doing.