Pick up files from multiple location using camel FTP - apache-camel

i have a situation i need to pick up files from two different location using camel FTP.
currently my code is
try {
from(format("sftp://%s#%s:22/../../..%s?password=%s&delete=true", ftpUserName, ftpServer, responsePath, ftpPassword ))
.filter(header("CamelFileName").endsWith(".PDF"))
.to(format("sftp://%s#%s:22/../../..%s/processed?password=%s", ftpUserName, ftpServer, responsePath, ftpPassword))
.process(documentProcessor)
/*.log(LoggingLevel.INFO, (org.slf4j.Logger) logger, "Download file ${file:name} complete.")*/
/*.to(downloadLocation)*/;
/*.to(format("smtp://relay.us.signintra.com?to=%s&from=noreply#dbschenker.com&subject=GTM response from cisco", emailTo))*/
;
} catch (Exception e) {
e.printStackTrace();
}
This is picking up the file that is mentioned in the application.properties files. How can i do this to puck up files from multile locations.

You can configure multiple FTP consumer routes and forward the message to shared direct-endpoint.
Example:
from("ftp:{{target.host}}:{{ftp.port}}/{{target.path1}}...")
.routeId("PollForFilesInPath1")
.to("direct:handleFileFromFTP");
from("ftp:{{target.host}}:{{ftp.port}}/{{target.path2}}...")
.routeId("PollForFilesInPath2")
.to("direct:handleFileFromFTP");
from("direct:handleFileFromFTP")
.routeId("handleFileFromFTP")
.log("file received from ftp: ${headers.CamelFileName }")
// Do stuff with the file
If you need to call 2 FTP consumer endpoints from single route you can use poll-enrich.

Related

can't get Camel unmarshal zipfile working?

I have a compressed or ZIP file inbound on camel route. This zip file has multiple CSV files within. I also need to condition the file content before further processing. It seems I can successfully unzip the file but cannot work on it is unmarshalled. It seems that multiple splitters are a problem. ??? hoping someone can tell me what I'm doing wrong. If I take the files out of the zip file, I can process them all individually, but I can't process them from the zip file.
I've tried ZipFileDataFormat, also ZipSplitter with same results. tried .split(bodyAs(Iterator.class))
} else if (Boolean.parseBoolean(isCompressedOnly)) { //Only Zipped or Compressed
ZipFileDataFormat zipFile = new ZipFileDataFormat();
zipFile.setUsingIterator(true);
from(fromStr)
.routeId(routeId)
.log(LoggingLevel.INFO, "Message received ${file:name} for Only Zipped or Compressed files from host " + host)
.unmarshal(zipFile)
.split(body(Iterator.class))
.streaming()
.convertBodyTo(String.class)
.wireTap("file:" + fileArchive)
.split(body()).streaming()
.process(new EndpointParametersProcessor(decoderName))
.to(toStr)
.end();
adding tokenize("\n") in the splitter operation that works on the individual csv files appears to have corrected my problem. also, to rejoin the message records to a single packet, include an aggregation strategy.
} else if (Boolean.parseBoolean(isCompressedOnly)) { //Only Zipped or Compressed
ZipFileDataFormat zipFile = new ZipFileDataFormat();
zipFile.setUsingIterator(true);
from(fromStr)
.routeId("Zipped.Only")
.log(LoggingLevel.INFO, "Message received ${file:name} for Only Zipped or Compressed files from host " + host)
.unmarshal(zipFile)
.split(body(Iterator.class))
.streaming()
.convertBodyTo(String.class)
.wireTap("file:" + fileArchive)
.split(body().tokenize("\n"), new FleetAggregationStrategy()).streaming()
.process(new EndpointParametersProcessor(decoderName))
.end()
.to(toStr);

Camel , catch error and move file in different folder

I am trying to doing ETL job (read,convert, store an excel) .
This route work fine :
from("file:src/data?move=processed&moveFailed=error&idempotent=true")
.bean(ExcelTransformer.class,"process")
.to("jpa:cie.service.receiver.DatiTelefonici?entityType=java.util.ArrayList");
but I need a customization in case of Exception : I need to move file in other folder , then applied :
onException(Throwable.class).maximumRedeliveries(0).to("file:src/data?move=error");
But file component can't move the file because is locket by first file comp instance.
Then I am tryinf to use try/catch but it doesn't work (probably the move operation inside catch is unaware of correct file name ? )
from("file:src/data?noop=true")
.doTry()
.bean(ExcelTransformer.class,"process")
.to("jpa:cie.service.receiver.DatiTelefonici?entityType=java.util.ArrayList")
.to("file:src/data?move=processed")
.doCatch(Throwable.class)
.to("file:src/data?move=error")
.end();
tanks
After many comments my current code looks like :
from("file:src/data?noop=false&delete=true")
.doTry()
.bean(ExcelTransformer.class,"process") .to("jpa:cie.service.receiver.DatiTelefonici?entityType=java.util.ArrayList")
.to("file:src/data/processed")
.doCatch(Throwable.class)
.to("file:src/data/error")
/*
.doFinally()
.to("file:src/data:delete=true")
*/
.end();
It move correctly the file in processed and error folder but the file remain in main folder and is preocessed more ,recursively
If I understood your question well then you need remove the idempotent=true from the parameters, then it should work:
from("file:src/data?move=processed&moveFailed=error")
.bean(ExcelTransformer.class,"process")
.to("jpa:cie.service.receiver.DatiTelefonici?entityType=java.util.ArrayList");
The previous route moves the file to the processed folder if the routing was successful otherwise it moves the file to the error folder (if any exception happens). The filename won't be changed.
Other solution with try-catch
from("file://src/data?delete=true")
.doTry()
.bean(ExcelTransformer.class,"process")
.to("jpa:cie.service.receiver.DatiTelefonici?entityType=java.util.ArrayList")
.to("file://src/data/processed")
.doCatch(Throwable.class)
.to("file://src/data/error")
.end();

Mule - Delete files with Files Connector

This is my main mule flow:
HTTP > Payload > File (Create) > Property (File Name) JMS
And that JMS calls to other flow that starts with JMS
JMS > JSON TO XML > File (Delete by file name)
Firstly, I create a file with the JSON message.
And finally, when JSON message is transformed to XML, I try to delete the file by name calling a property that tell me the name.
But the files doesn't delete. Its output this action by log:
Writing file to: C:\errors\91fa03a0-dd33-11e5-a0d9-408d5c4c1bf3.dat
How can I do to delete it?
I tried with the option autoDelete in the Connector Configuration but it doesn't delete.
<file:connector name="File1" outputPattern="#[message.inboundProperties.fileName]" autoDelete="true" streaming="false" validateConnections="true" doc:name="File"/>
You can't use this file outbound endpoint to delete a file.
If you would use something like S3 for example you can, but those connectors are 'operation-based'.
It's meant to write an new file or append data to an existing one.
The autoDelete option is for deleting the file once it has been read by a file inbound endpoint.
More info:
https://docs.mulesoft.com/mule-user-guide/v/3.7/file-transport-reference
If you want to delete and for your usecase you should use a piece of JAVA code and use methods from the File Class:
https://docs.oracle.com/javase/7/docs/api/java/io/File.html

StreamCache FileNotFound issues with bigger data in multicast routes

We are using camel 2.13.2 - I have a multicast route with an AggregationStrategy.
And in each multicast branch, we have a custom camel component that returns huge data (around 4 MB) and writes to Stream Cache (Cached Output Stream) and we need to aggregate the data in the multicast (Aggregation Strategy).
In the Aggregation strategy, I need to do XPath evaluation using camel XPathBuilder.
Hence, I try to read the body and convert from StreamCache to byte[] to avoid 'Error during type conversion from type: org.apache.camel.converter.stream.InputStreamCache.' in the XPathBuilder.
When I try to read the body in the beginning of the Aggregation Strategy, I get the following error.
*/tmp/camel/camel-tmp-4e00bf8a-4a42-463a-b046-5ea2d7fc8161/cos6047774870387520936.tmp (No such file or directory), cause: FileNotFoundException:/tmp/camel/camel-tmp-4e00bf8a-4a42-463a-b046-5ea2d7fc8161/cos6047774870387520936.tmp (No such file or directory).
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.(FileInputStream.java:138)
at org.apache.camel.converter.stream.FileInputStreamCache.createInputStream(FileInputStreamCache.java:123) at org.apache.camel.converter.stream.FileInputStreamCache.getInputStream(FileInputStreamCache.java:117)
at org.apache.camel.converter.stream.FileInputStreamCache.writeTo(FileInputStreamCache.java:93)
at org.apache.camel.converter.stream.StreamCacheConverter.convertToByteArray(StreamCacheConverter.java:102)
at com.sap.it.rt.camel.aggregate.strategies.MergeAtXPathAggregationStrategy.convertToByteArray(MergeAtXPathAggregationStrategy.java:169)
at com.sap.it.rt.camel.aggregate.strategies.MergeAtXPathAggregationStrategy.convertToXpathCompatibleType(MergeAtXPathAggregationStrategy.java:161)
*
Following is the line of code where it is throwing an error:
Object body = exchange.getIn().getBody();
if( body instanceof StreamCache){
StreamCache cache = (StreamCache)body;
xml = new String(convertToByteArray(cache,exchange));
exchange.getIn().setBody(xml);
}
By disabling stream cache to write to file by setting a threshold of 10MB in multicast related routes, we were able to work with the aggregation strategy. But we do not want to do that, as we may have incoming data that maybe bigger.
<camel:camelContext id="multicast_xml_1" streamCache="true">
<camel:properties>
<camel:property key="CamelCachedOutputStreamCipherTransformation" value="RC4"/>
<camel:property key="CamelCachedOutputStreamThreshold" value="100000000"/>
</camel:properties>
....
</camel:camelContext>
Note: The FileNotFound issue does not appear if we have the StreamCache based camel component in the route with other processors, but without Multicast + Aggregation.
After debugging, I could understand the issue with aggregating huge data from StreamCache with MulticastProcessor.
In MulticastProcessor.java: doProcessParallel() is called and on completion of the branch exchange of multicast, the CachedOutputStream deletes / cleans up the temporary file.
This happens even before the multicast branch exchange reaches the aggregation Strategy, which tries to read the data from the branch exchange. In case of huge data in StreamCache, the temporary file is already deleted, leading to FileNotFound issues.
public CachedOutputStream(Exchange exchange, boolean closedOnCompletion) {
this.strategy = exchange.getContext().getStreamCachingStrategy();
currentStream = new CachedByteArrayOutputStream(strategy.getBufferSize());
if (closedOnCompletion) {
// add on completion so we can cleanup after the exchange is done such as deleting temporary files
exchange.addOnCompletion(new SynchronizationAdapter() {
#Override
public void onDone(Exchange exchange) {
try {
if (fileInputStreamCache != null) {
fileInputStreamCache.close();
}
close();
} catch (Exception e) {
LOG.warn("Error deleting temporary cache file: " + tempFile, e);
}
}
#Override
public String toString() {
return "OnCompletion[CachedOutputStream]";
}
});
}
}
public void close() throws IOException {
currentStream.close();
cleanUpTempFile();
}
I was able to circumvent the issue, if I try to set closedOnCompletion= false, while writing to CachedOutputStream in any component in any Multicast branch.
But this is a leaky solution, because the streamcache temporary file(s) may then never get cleaned up... hence I try to close + clean up the cachestream, after reading the data in the AggregationStrategy.
Can the MulticastProcessor be adjusted so that the multicast branch exchanges reach 'completion' status only, after they have been aggregated at the end of multicast?
Please help / advise on the issue, as I am new to using camel Multicast.
Thanks,
Lakshmi
I have similar exception thrown when trying to send larger than 1MB JSON response to Restlet request (yes, I know 1MB JSON is too big):
java.io.FileNotFoundException: C:\Users\me\AppData\Local\Temp\camel\camel-tmp-7ad6e098-538d-4d4c-9357-2b7addb1f19d\cos6725022584818060586.tmp (The system cannot find the file specified)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(FileInputStream.java:146)
at org.apache.camel.converter.stream.FileInputStreamCache.createInputStream(FileInputStreamCache.java:123)
at org.apache.camel.converter.stream.FileInputStreamCache.getInputStream(FileInputStreamCache.java:117)
at org.apache.camel.converter.stream.FileInputStreamCache.read(FileInputStreamCache.java:112)
at java.io.InputStream.read(InputStream.java:170)
at java.io.InputStream.read(InputStream.java:101)
at org.restlet.engine.io.BioUtils.copy(BioUtils.java:81)
at org.restlet.representation.InputRepresentation.write(InputRepresentation.java:148)
at org.restlet.engine.adapter.ServerCall.writeResponseBody(ServerCall.java:510)
at org.restlet.engine.adapter.ServerCall.sendResponse(ServerCall.java:454)
at org.restlet.ext.servlet.internal.ServletCall.sendResponse(ServletCall.java:426)
at org.restlet.engine.adapter.ServerAdapter.commit(ServerAdapter.java:196)
at org.restlet.engine.adapter.HttpServerHelper.handle(HttpServerHelper.java:153)
at org.restlet.ext.servlet.ServerServlet.service(ServerServlet.java:1089)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1496)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:102)
Same workaround works for me:
getContext().getProperties().put(CachedOutputStream.THRESHOLD, "" + THREE_MEGABYTE_TRESHOLD_BEFORE_FILE_CACHE);
I don't use multicast in this route, just plain
restlet request -> Service -> Jackson marshall => error
I use Camel 2.14.0 & Restlet 2.2.2 with JDK 7 and Spring-boot 1.0.2 / Jetty
This Camel reverse proxy - no response stream caching might be related to my issue.

Camel SFTP Route fails to continue onException

I've fairly simple looking route
sftp://hostname:22//incoming/folder/location/?username=username&password=xxxxx
&localWorkDirectory=/tmp&readLock=changed&readLockCheckInterval=2000
&move=processed/$simple{date:now:yyyy}/$simple{date:now:MM}/$simple{date:now:dd}${file:name}
&consumer.delay=450000&stepwise=false&streamDownload=true&disconnect=true
I also have an onException clause
onException(ValidationException.class)
.handled(true)
.logStackTrace(true)
.filter(header("VALIDATION_ERROR").isEqualTo(true))
.choice()
.when(header("CamelFileName").contains("Param1"))
.to(sftp://hostname:22//One/error/folder?password=xxxxxx&username=username)
.when(header("CamelFileName").contains("Param2"))
.to(sftp://hostname:22//Two/error/folder?password=xxxxxx&username=username)
.endChoice();
When I have single file, the route seems to work as expected. When more than one file and exception occurs, I get many different exceptions like
org.apache.camel.component.file.GenericFileOperationFailedException: Cannot list directory: incoming/folder/location
Caused by: java.lang.IndexOutOfBoundsException
I tried using all the attributes mentioned in the route viz. streamDownload, stepwise, readLock, localWorkDirectory et al. However, the error handling while multiple files is not working. I see the first file getting processed. However, it doesn't move to the processed folder once the exception occurs and then incoming/folder/location becomes non listable. I tried using continued(true) as well instead of handled(true)
The problem was with multiple files being handled in the same exchange. On exception, the route was trying to FTP back the error file on the same server. The solution was to split body into multiple exchanges so that each file has its own exchange and process them separately.
from(sftp://hostname:22//incoming/folder/location/?username=username&password=xxxxx
&localWorkDirectory=/tmp&readLock=changed&disconnect=true&stepwise=false
&move=processed/$simple{date:now:yyyy}/$simple{date:now:MM}/$simple{date:now:dd}${file:name}
&consumer.delay=450000).split(body()).processRef("incomingProcessor").end();

Resources