Camel - Enrich CSV from FTP with CSV from local disk using Camel Bindy - apache-camel

The goal is to produce a report every hour by comparing two CSV files with
use of Camel 3.0.0. One is located on a FTP server, the other on disk. How to use poll enrich pattern in combination with unmarshalling the CSV on disk with Bindy Dataformat?
Example code (for simplicity the FTP endpoint is replaced by a file endpoint):
#Component
public class EnricherRoute extends RouteBuilder {
#Override
public void configure() {
from("file://data?fileName=part_1.csv&scheduler=quartz2&scheduler.cron=0+0+0/1+*+*+?")
.unmarshal().bindy(BindyType.Csv, Record.class)
.pollEnrich("file://data?fileName=part_2.csv", new ReportAggregationStrategy())
.marshal().bindy(BindyType.Csv, Record.class)
.to("file://reports?fileName=report_${date:now:yyyyMMdd}.csv");
}
}
The problem in this example is that in the ReportAggregationStrategy the resource (coming from data/part_2.csv, see below) is not unmarshalled. How to unmarshal data/part_2.csv as well?
public class ReportAggregationStrategy implements AggregationStrategy {
#Override
public Exchange aggregate(Exchange original, Exchange resource) {
final List<Record> originalRecords = original.getIn().getBody(List.class);
final List<Record> resourceRecords = resource.getIn().getBody(List.class); // Results in errors!
...
}
}

You can wrap enrichment with direct endpoint and do unmarshaling there.
from("file://data?fileName=part_1.csv&scheduler=quartz2&scheduler.cron=0+0+0/1+*+*+?")
.unmarshal().bindy(BindyType.Csv, Record.class)
.enrich("direct:enrich_record", new ReportAggregationStrategy())
.marshal().bindy(BindyType.Csv, Record.class)
.to("file://reports?fileName=report_${date:now:yyyyMMdd}.csv");
from("direct:enrich_record")
.pollEnrich("file://data?fileName=part_2.csv")
.unmarshal().bindy(BindyType.Csv, Record.class);

Related

Apache Camel FTP integration

I have just started working around the Apache Camel. I have a requirement to implement an FTP/FTPS/SFTP client, which would be used to fetch the files from the respective servers. I was looking into the possibility of using Apache Camel to do this but I am still confused after going through the examples and the tutorials.
The requirement is to fetch the files from the FTP/SFTP servers when the request is received from the scheduler.
Following is the route created using EndPoint-DSL
#Component
public class FtpReceiveRoute extends EndpointRouteBuilder {
#Override
public void configure() throws Exception {
from(
ftp("localhost:2001/home/admin")
.account("admin")
.password("admin12345")
.recursive(true)
)
.routeId("ftpReceive")
.log("From done!")
.to("log:ftp-log")
.log("To done!!");
}
}
I am trying to use the above route by invoking it when the request is made to fetch the file like below.
#Override
protected FtpResponse doMessage(String param, FtpRequest req) {
FtpResponse response = new FtpResponse ();
CamelContext ctx = new DefaultCamelContext();
ctx.addRoutes(##route); //FtpReceiveRoute, add the Routebuilder instance as EndpointRouteBuilder is acceptable.
ctx.start();
//Might need to induce sleep so that all the files are downloaded
ctx.stop();
return response;
}
The confusion is around how to invoke the Camel process with the route. I have used EndpointRouteBuilder to create the route because of the type-safe creation of the endpoint URI. I am not getting an option to add this route to the CamelContext as it expects the RouteBuilder instance which is not type-safe.
Further, the CamelContext is the engine and to invoke the route I would need to start and stop this engine. This I am not able to digest if I need to start and stop the engine to execute a route then I would need to induce some sleep in between so that all files are downloaded. Just to add there are more routes that I need to add with the implementation. Once the engine is started it would load and execute all the added routes which is not the requirement.
Maybe I am not getting how to use this properly. Any resources aiding my situation are welcome. Thanks.
You should not create and start new camel context every time you want to fetch file from server. What you should do instead is start one when your application starts and use that for all your exchanges.
You can use Spring-boot to initialize CamelContext and add annotated RouteBuilders to it automatically. Check the maven archetype camel-archetype-spring-boot for example.
If you want to call camel routes from Java you can Inject CamelContext to your bean and use it to create ProducerTemplate. This can be used to invoke Routes defined in the RouteBuilder.
Using ProducerTemplate.send you can get the resulting exchange.
Using producer template
Using File-component which works very similary to ftp-component.
package com.example;
import org.apache.camel.builder.endpoint.EndpointRouteBuilder;
import org.springframework.stereotype.Component;
#Component
public class MySpringBootRouter extends EndpointRouteBuilder {
#Override
public void configure() {
from(direct("fileFromFTP"))
.routeId("fileFromFTP")
// reads files from <project>/input using file consumer endpoint
.pollEnrich(file("input"), 1000)
// If file is found, convert body to string.
// Which in this case will read contents of the file to string.
.filter(body().isNotNull())
.convertBodyTo(String.class)
.end()
;
}
}
package com.example;
import org.apache.camel.CamelContext;
import org.apache.camel.Exchange;
import org.apache.camel.ProducerTemplate;
import org.apache.camel.support.DefaultExchange;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.context.annotation.Configuration;
import org.springframework.scheduling.annotation.EnableScheduling;
import org.springframework.scheduling.annotation.Scheduled;
import static org.apache.camel.builder.endpoint.StaticEndpointBuilders.direct;
#Configuration
#EnableScheduling
public class MySpringBean {
#Autowired
CamelContext camelContext;
#Scheduled(fixedRate = 1000)
public void scheduledTask() {
System.out.println("Scheduled Task!");
if(camelContext.isStopped()) {
System.out.println("Camel context not ready yet!");
return;
}
useProducerTemplate();
}
public void useProducerTemplate(){
ProducerTemplate producerTemplate = camelContext.createProducerTemplate();
Exchange inExchange = new DefaultExchange(camelContext);
//synchronous call!
Exchange result = producerTemplate.send(direct("fileFromFTP").toString(), inExchange);
String resultBody = result.getMessage().getBody(String.class);
String fileName = result.getMessage().getHeader(Exchange.FILE_NAME, String.class);
if(resultBody != null){
System.out.println("Consumed file: "+ fileName + " contents: " + resultBody.toString());
}
else{
System.out.println("No file to consume!");
}
}
}
Depending on what you need to do with the files you could probably do that inside camel route. Then you would only need to call the producerTemplate.sendBody.
public void useProducerTemplate(){
ProducerTemplate producerTemplate = camelContext.createProducerTemplate();
Exchange inExchange = new DefaultExchange(camelContext);
producerTemplate.sendBody(direct("fileFromFTP").toString(), inExchange);
}
Starting stopping camel route
If you want to start polling file consumer only for a short while you can do start the route and use for example aggregation timeout to shutdown the route when no new files have been received in any given duration.
#Component
public class MySpringBootRouter extends EndpointRouteBuilder {
#Override
public void configure() {
AggregationStrategy aggregateFileNamesStrategy = AggregationStrategies
.flexible(String.class)
.accumulateInCollection(ArrayList.class)
.pick(header(Exchange.FILE_NAME))
;
from(file("input"))
.routeId("moveFilesRoute")
.autoStartup(false)
.to(file("output"))
.to(seda("moveFilesRouteTimeout"));
;
from(seda("moveFilesRouteTimeout"))
.routeId("moveFilesRouteTimeout")
.aggregate(constant(true), aggregateFileNamesStrategy)
.completionTimeout(3000)
.log("Consumed files: ${body.toString()}")
.process(exchange -> {
exchange.getContext().getRouteController().stopRoute("moveFilesRoute");
})
.end()
;
}
}
public void startMoveFilesRoute() {
try {
System.out.println("Starting moveFilesRoute!");
camelContext.getRouteController().startRoute("moveFilesRoute");
//Sending null body moveFilesRouteTimeout to trigger timeout if there are no files to transfer
camelContext.createProducerTemplate().sendBody(seda("moveFilesRouteTimeout").toString(), null);
} catch(Exception e) {
System.out.println("failed to stop route. " + e);
}
}

Using Project Reactor with Apache Camel

I'd like to know if it is possible to use Project Reactor with Apache Camel, so applications be fully reactive and non-blocking IO. I'd like to know how does the Project Reactor support works when integrating other Apache Camel's components.
Can I read for example from S3 reactively (therefore I'll need to use the Async S3 client behind the scenes)? Or will I block when reading from S3 and then just create a Flux out of what has been returned?
Where reactiveness is needed, you should use the relevant spring and reactor libraries. there are pseudo camel code also u can db call in camel bean or processors etc
#RestController
#RequestMapping(value = "/api/books")
#RequiredArgsContructor
public class HomeController {
private final BookRepository bookRepository;
private final ProducerTemplate template
#GetMapping("")
public Flux<Book> getHome() {
List <Book> books=bookRepository.findAll();
X ret = template.requestBody("direct:something", books, X.class);
}
}
#Component
public class SomeRoute extends RouteBuilder {
#Override
public void configure() throws Exception {
from("direct:something")
.process(e-> {
List<Book> books = e.getIn.getBody(List.class)
// some logic
e.getIn.setBody(new X( ))
})
}

Camel ProducerTemplate and ActiveMQ message and persistence issues

Issue:
I have multiple route sending messages to an ActiveMQ queue which later gets processed by a processor to save status information
The same message sending from ProducerTemplate to ActiveMQ queue somehow breakes the same code by not triggering logs on console, and saving status information to a randomly generated file name.
Desired Behavior:
Both sending method gets the messages processed the same way
Code Explanation:
On below codes the Save processor is the one producing the weird behavior where logs dont show up on console and writes to some random file, file name is basically the clientID from ActiveMQ
StartRoute calling the activemq:save works correctly
Code:
public class Save implements Processor {
public void process(Exchange exchange) throws Exception {
try {
Map<String, Object> map = new HashMap<String, Object>() {};
map.put(".....", ".....");
map.put(".....", ".....");
map.put(".....", ".....");
map.put(".....", ".....");
ProducerTemplate template = exchange.getContext().createProducerTemplate();
String response = template.requestBodyAndHeaders("activemq:save", "Batch Job Started", map, String.class);
FluentProducerTemplate FluentTemplate = exchange.getContext().createFluentProducerTemplate();
String result = FluentTemplate
.withHeader(".....", ".....")
.withHeader(".....", ".....")
.withHeader(".....", ".....")
.withHeader(".....", ".....")
.withHeader(".....", ".....")
.to("activemq:save")
.request(String.class);
} catch (Exception e) {
.....
}
}
}
public class StartRoute extends RouteBuilder {
restConfiguration()
.component("servlet")
.enableCORS(true);
rest("/start").id("start")
.post("/{filename}")
.route()
.....
.process(new Save())
.wireTap(startBackgroundProcessRoute)
.to("activemq:save")
.endRest();
}
public class SaveRoute extends RouteBuilder {
from("activemq:save")
.log("started saving")
.process(new FormatText())
.to("file:///status");
}
This question came from my original problem described here:
Camel Multicast Route call order
Solution can be found also there:
Camel Multicast Route call order
However to satisfy this question:
It seems Camel have few bugs regarding producer templates and maybe ActiveMQ, this is my initial conclusion.
The Only way i was able to use ProducerTemplate without issue is to use the send function with Exchanges, send() sends the messages correctly to the ActiveMQ same way as the to() however for whatever reason the content was still not written to the file.
After I dropped the ActiveMQ between the routes, everything started to work consistently. So possible miss configuration on ActiveMQ component or possible another camel framework bug.
If anyone knows the exact answer i would be happy to hear / see the reason for this behavior.
Code example:
public class SaveProcessor implements Processor {
public void process(Exchange exchange) throws Exception {
ProducerTemplate template = exchange.getContext().createProducerTemplate();
template.send(Utilities.trackBatchJobStatus, exchange);
/** NOTE
* DO NOT USE any other functions which are not working with EXCHANGES.
* Functions that uses body, header values and such are bugged
*/
}
}

Apache Camel route to pojo

I have an Account Instance that is not in the format I need. I am trying Camel
but just not getting
the basic understanding. I have the un-formatted Instance successfully coming in and I display it in the log.
I have tried to extract the data from the body using a processor and directly to a POJO.
The POJO uses fixed length definitions. Which is needed do to passing data to a Mainframe DB. Please point me in a direction.
Got message:12 xZZ200ZZZZZZZ
public class AccountAddRoute extends RouteBuilder{
public void configure() throws Exception {
DataFormat bindy = new BindyFixedLengthDataFormat(AccountAddData.class);
from("direct:input")
.log(" In AccountAddRoute ")
.marshal(bindy)
.log("Got message:${body}");
.to(???????_)
My Apologies. Not a good explanation of what I am trying to do.
I have An Account POJO that is being built thru FORM. I need to save
it in 2 different formats. Incoming just the way it is and in a
fixed format to be sent to Mainframe. This is what I get from the form
Code1=12 x , Code2=200
I initiate rout by sending body of form in accountAddData Instance
producerTemplate.sendBody("direct:input", accountAddData);
public class AccountAddRoute extends RouteBuilder{
public void configure() throws Exception {
DataFormat bindy = new BindyFixedLengthDataFormat(AccountAddData.class);
from("direct:input")
.log(" In AccountAddRoute ")
.marshal(bindy)
.log("Got message:${body}");
.to(???????_)
Just for Testing purposes my Account Bindy Pojo is below
#DataField(pos =1, length=15, paddingChar='Z', trim=true, align="L")
private String Code1;
#DataField(pos =2, length=10, paddingChar='Z', align="L")
private String Code2;
When I log Body, I get >>
Got message:12 xZZ200ZZZZZZZ
However, my question is how do I create the new Instance in the
BINDY Format SO I can use it
==========
I did add processor and when I do display of body in processor >>
log.info(" In MyProcessor = " + exchange.getIn().getBody());
In MyProcessor = [B#3338b2b9
I was expecting to have the getter/setters available after getBody for the
accoountAddData instance. But obviously I don’t have a good understanding
of camel
exchange.getIn().getBody().????
Are you trying to unmarshal the incoming data? If thats the case, then you would need to use the following code.
from("direct:input")
.log(" In AccountAddRoute ")
.unmarshal(new BindyFixedLengthDataFormat(AccountAddData.class))
.log("Got message:${body}");
.to(???????_)

How to get the actual file object from Camel FTP route exchange

In my Camel router:
from(<SourceURI>)
.process(new Processor() {
#Override
public void process(Exchange exchange) throws Exception {
// I want to extract the file object from the exchange
}
.to(<targetURI>).
How can I achieve this?
I tried e.g. exchange.getIn().getHeader(Exchange.FILE_NAME, String.class) which gives me the file name.
I am searching for something Exchange.FILE which gives me the actual file object. My Ultimate goal is to extract the file in the processor as the routed exchange is an archive file.
Get the file from the body. Camel uses a 'org.apache.camel.component.file.GenericFile' to store as the file body. But you can use Camel's type converters to get the file in a type you want.
For example you can get the content in different types, such as:
String text = exchange.getIn().getBody(String.class);
byte[] bytes = exchange.getIn().getBody(byte[].class);
InputStream is = exchange.getIn().getBody(InputStream.class);
For those who have a from("file:...") the following works:
File in = exchange.getIn().getBody(File.class);

Resources