can't get Camel unmarshal zipfile working? - apache-camel

I have a compressed or ZIP file inbound on camel route. This zip file has multiple CSV files within. I also need to condition the file content before further processing. It seems I can successfully unzip the file but cannot work on it is unmarshalled. It seems that multiple splitters are a problem. ??? hoping someone can tell me what I'm doing wrong. If I take the files out of the zip file, I can process them all individually, but I can't process them from the zip file.
I've tried ZipFileDataFormat, also ZipSplitter with same results. tried .split(bodyAs(Iterator.class))
} else if (Boolean.parseBoolean(isCompressedOnly)) { //Only Zipped or Compressed
ZipFileDataFormat zipFile = new ZipFileDataFormat();
zipFile.setUsingIterator(true);
from(fromStr)
.routeId(routeId)
.log(LoggingLevel.INFO, "Message received ${file:name} for Only Zipped or Compressed files from host " + host)
.unmarshal(zipFile)
.split(body(Iterator.class))
.streaming()
.convertBodyTo(String.class)
.wireTap("file:" + fileArchive)
.split(body()).streaming()
.process(new EndpointParametersProcessor(decoderName))
.to(toStr)
.end();

adding tokenize("\n") in the splitter operation that works on the individual csv files appears to have corrected my problem. also, to rejoin the message records to a single packet, include an aggregation strategy.
} else if (Boolean.parseBoolean(isCompressedOnly)) { //Only Zipped or Compressed
ZipFileDataFormat zipFile = new ZipFileDataFormat();
zipFile.setUsingIterator(true);
from(fromStr)
.routeId("Zipped.Only")
.log(LoggingLevel.INFO, "Message received ${file:name} for Only Zipped or Compressed files from host " + host)
.unmarshal(zipFile)
.split(body(Iterator.class))
.streaming()
.convertBodyTo(String.class)
.wireTap("file:" + fileArchive)
.split(body().tokenize("\n"), new FleetAggregationStrategy()).streaming()
.process(new EndpointParametersProcessor(decoderName))
.end()
.to(toStr);

Related

Pick up files from multiple location using camel FTP

i have a situation i need to pick up files from two different location using camel FTP.
currently my code is
try {
from(format("sftp://%s#%s:22/../../..%s?password=%s&delete=true", ftpUserName, ftpServer, responsePath, ftpPassword ))
.filter(header("CamelFileName").endsWith(".PDF"))
.to(format("sftp://%s#%s:22/../../..%s/processed?password=%s", ftpUserName, ftpServer, responsePath, ftpPassword))
.process(documentProcessor)
/*.log(LoggingLevel.INFO, (org.slf4j.Logger) logger, "Download file ${file:name} complete.")*/
/*.to(downloadLocation)*/;
/*.to(format("smtp://relay.us.signintra.com?to=%s&from=noreply#dbschenker.com&subject=GTM response from cisco", emailTo))*/
;
} catch (Exception e) {
e.printStackTrace();
}
This is picking up the file that is mentioned in the application.properties files. How can i do this to puck up files from multile locations.
You can configure multiple FTP consumer routes and forward the message to shared direct-endpoint.
Example:
from("ftp:{{target.host}}:{{ftp.port}}/{{target.path1}}...")
.routeId("PollForFilesInPath1")
.to("direct:handleFileFromFTP");
from("ftp:{{target.host}}:{{ftp.port}}/{{target.path2}}...")
.routeId("PollForFilesInPath2")
.to("direct:handleFileFromFTP");
from("direct:handleFileFromFTP")
.routeId("handleFileFromFTP")
.log("file received from ftp: ${headers.CamelFileName }")
// Do stuff with the file
If you need to call 2 FTP consumer endpoints from single route you can use poll-enrich.

Mule File Inbound - empty files are not triggered

I have a scenario wherein I need to read files from a particular folder. So I had a File inbound as below, its reading all non-empty files. But empty files are not read and sits in the same location as is.
<file:inbound-endpoint path="${file.path}" responseTimeout="10000" doc:name="File" moveToDirectory="${audit.location}">
<file:filename-regex-filter pattern="file.employee(.*).xml,file.client(.*).xml"
caseSensitive="true"/>
</file:inbound-endpoint>
I removed File filter, but still it doesn't read empty files.
Is there a way to enable file inbound to read empty files too?
According the the Mule File Connector documentation:
The File connector as inbound endpoint does not process empty (0 bytes) files.
So this behavior is expected. There is no documented way to process non empty file with the File Inbound Endpoint.
However you can still write your own connector to do this, or use a workaround such as fill your "empty" file with a single character (such as a space) to make it non-empty
If you want to read a file with the size of 0 KB, then you can`t achieve this with File Connector, but we can read a file by using MuleRequester in the flow. I will share sample snippet soon. Please let me know,If you need any help.
Regards,
Sreenivas B
Mule File connector does not process empty (0 bytes) files as inbound endpoint
As per my knowledge File Inbound connector will not process (0 KB) size files.
On the File Connector, the class org.mule.transport.file.FileMessageReceiver.java in method poll has :
if (file.length() == 0)
{
if (logger.isDebugEnabled())
{
logger.debug("Found empty file '" + file.getName() + "'. Skipping file.");
}
continue;
}
that prevents it from proccessing empty files
But you can create your own CustomFileMessageReceiver.java, create your package:
package com.mycompany.mule.transport.file;
And the class that extends AbstractPollingMessageReceiver
public class CustomFileMessageReceiver extends AbstractPollingMessageReceiver
Copy the original FileMessageReceiver.java methods but comment the above lines and change FileMessageReceiver to CustomFileMessageReceiver where needed.
The call fileConnector.move(file, workFile) is a protected method from the original package, commented and beware you cannot use workdir.
In the same package create a copy of org.mule.transport.file.ReceiverFileInputStream.java
Configure your connector:
<file:connector name="FILE" readFromDirectory="${incoming.directory}" autoDelete="true" streaming="false" recursive="true" validateConnections="true" doc:name="File" writeToDirectory="${processed.directory}">
<service-overrides messageReceiver="com.mycompany.mule.transport.file.CustomFileMessageReceiver" />
</file:connector>
Or you may implement your own file connector, as stated in the above answers.

Camel , catch error and move file in different folder

I am trying to doing ETL job (read,convert, store an excel) .
This route work fine :
from("file:src/data?move=processed&moveFailed=error&idempotent=true")
.bean(ExcelTransformer.class,"process")
.to("jpa:cie.service.receiver.DatiTelefonici?entityType=java.util.ArrayList");
but I need a customization in case of Exception : I need to move file in other folder , then applied :
onException(Throwable.class).maximumRedeliveries(0).to("file:src/data?move=error");
But file component can't move the file because is locket by first file comp instance.
Then I am tryinf to use try/catch but it doesn't work (probably the move operation inside catch is unaware of correct file name ? )
from("file:src/data?noop=true")
.doTry()
.bean(ExcelTransformer.class,"process")
.to("jpa:cie.service.receiver.DatiTelefonici?entityType=java.util.ArrayList")
.to("file:src/data?move=processed")
.doCatch(Throwable.class)
.to("file:src/data?move=error")
.end();
tanks
After many comments my current code looks like :
from("file:src/data?noop=false&delete=true")
.doTry()
.bean(ExcelTransformer.class,"process") .to("jpa:cie.service.receiver.DatiTelefonici?entityType=java.util.ArrayList")
.to("file:src/data/processed")
.doCatch(Throwable.class)
.to("file:src/data/error")
/*
.doFinally()
.to("file:src/data:delete=true")
*/
.end();
It move correctly the file in processed and error folder but the file remain in main folder and is preocessed more ,recursively
If I understood your question well then you need remove the idempotent=true from the parameters, then it should work:
from("file:src/data?move=processed&moveFailed=error")
.bean(ExcelTransformer.class,"process")
.to("jpa:cie.service.receiver.DatiTelefonici?entityType=java.util.ArrayList");
The previous route moves the file to the processed folder if the routing was successful otherwise it moves the file to the error folder (if any exception happens). The filename won't be changed.
Other solution with try-catch
from("file://src/data?delete=true")
.doTry()
.bean(ExcelTransformer.class,"process")
.to("jpa:cie.service.receiver.DatiTelefonici?entityType=java.util.ArrayList")
.to("file://src/data/processed")
.doCatch(Throwable.class)
.to("file://src/data/error")
.end();

Cant download file from webservice using MTOM and Axis2 stub

I m trying to download a file from my Axis2 webservice server using MTOM and ADB.
I can download the file if I dont enable the MTOM both on server and the client sides. Any suggestions or code sample would be nice :)
Client side
ServerWSStub stub = new ServerWSStub();
stub._getServiceClient().getOptions().setProperty(Constants.Configuration.ENABLE_MTOM,Constants.VALUE_TRUE);
Server side axis2.xml
<parameter name="enableMTOM">optional</parameter>
This is my Server
public DataHandler download(String konum) throws Exception {
System.out.println("The filePath for download: " + konum);
FileDataSource dataSource = new FileDataSource(konum);
DataHandler datahandler = new DataHandler(dataSource);
OMFactory fac = OMAbstractFactory.getOMFactory();
OMNamespace ns = fac.createOMNamespace("http://benim.projem.org/dosya", "dosyam");
OMText textData = fac.createOMText(datahandler, true);
OMElement ele = fac.createOMElement("sonuc", ns);
ele.addChild(textData);
System.out.println(ele);
return datahandler;
This is my Client
ServerWSStub stub = new ServerWSStub();
//stub._getServiceClient().getOptions().setProperty(Constants.Configuration.ENABLE_MTOM,Constants.VALUE_TRUE);
//when uncommented i get java.lang.NoClassDefFoundError: org/apache/james/mime4j/MimeException
//while trying to invoke _operationClient.execute(true); in ServerWSStub
//I guess it is because of wrong unparsing
Download download = new Download();
download.setKonum(konum);
try {
DownloadResponse downloadResponse = stub.download(download);
DataHandler dh =(DataHandler) downloadResponse.get_return();
File file = new File("C:/dosya/"+fileNameType);
if (!file.getParentFile().exists())
file.getParentFile().mkdirs();
if(!file.exists()){
file.createNewFile();
}
FileOutputStream fileOutputStream = new FileOutputStream(file);
dh.writeTo(fileOutputStream);
fileOutputStream.flush();
fileOutputStream.close();
} catch (ServerWSExceptionException e) {
e.printStackTrace();
}
Any
I finally got the solution I guess. The Stream closes before the client gets the whole file thats why first I used the getOptions().setTimeOutInMilliSeconds(10000) method but it was also useless and then in the Stub file I commented
_messageContext.getTransportOut().getSender().cleanup(_messageContext);//look for the method's finally part
part so that during a large file transportation the stream had not been closed and i could download the whole file without any silly exceptions :)
--MIMEBoundary_e56e8a77b94fbdd7678582aa5ca53f50b1d56c0d828499ea
Content-Type: application/xop+xml; charset=UTF-8; type="text/xml"
Content-Transfer-Encoding: binary
Content-ID: <0.146e8a77b94fbdd7678582aa5ca53f50b1d56c0d828499ea#apache.org>
<?xml version='1.0' encoding='UTF-8'?><soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><ns:downloadResponse xmlns:ns="http://servis.ws.projem.tez.benim"><ns:return><xop:Include xmlns:xop="http://www.w3.org/2004/08/xop/include" href="cid:1.046e8a77b94fbdd7678582aa5ca53f50b1d56c0d828499ea#apache.org" /></ns:return></ns:downloadResponse></soapenv:Body></soapenv:Envelope>
--MIMEBoundary_e56e8a77b94fbdd7678582aa5ca53f50b1d56c0d828499ea
Content-Type: text/plain
Content-Transfer-Encoding: binary
Content-ID: <1.046e8a77b94fbdd7678582aa5ca53f50b1d56c0d828499ea#apache.org>
binary code here
Simply add apache-mime4j-core-0.7.2.jar to WEB-INF/lib on the service (server) side. The jar can be found here.

StreamCache FileNotFound issues with bigger data in multicast routes

We are using camel 2.13.2 - I have a multicast route with an AggregationStrategy.
And in each multicast branch, we have a custom camel component that returns huge data (around 4 MB) and writes to Stream Cache (Cached Output Stream) and we need to aggregate the data in the multicast (Aggregation Strategy).
In the Aggregation strategy, I need to do XPath evaluation using camel XPathBuilder.
Hence, I try to read the body and convert from StreamCache to byte[] to avoid 'Error during type conversion from type: org.apache.camel.converter.stream.InputStreamCache.' in the XPathBuilder.
When I try to read the body in the beginning of the Aggregation Strategy, I get the following error.
*/tmp/camel/camel-tmp-4e00bf8a-4a42-463a-b046-5ea2d7fc8161/cos6047774870387520936.tmp (No such file or directory), cause: FileNotFoundException:/tmp/camel/camel-tmp-4e00bf8a-4a42-463a-b046-5ea2d7fc8161/cos6047774870387520936.tmp (No such file or directory).
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.(FileInputStream.java:138)
at org.apache.camel.converter.stream.FileInputStreamCache.createInputStream(FileInputStreamCache.java:123) at org.apache.camel.converter.stream.FileInputStreamCache.getInputStream(FileInputStreamCache.java:117)
at org.apache.camel.converter.stream.FileInputStreamCache.writeTo(FileInputStreamCache.java:93)
at org.apache.camel.converter.stream.StreamCacheConverter.convertToByteArray(StreamCacheConverter.java:102)
at com.sap.it.rt.camel.aggregate.strategies.MergeAtXPathAggregationStrategy.convertToByteArray(MergeAtXPathAggregationStrategy.java:169)
at com.sap.it.rt.camel.aggregate.strategies.MergeAtXPathAggregationStrategy.convertToXpathCompatibleType(MergeAtXPathAggregationStrategy.java:161)
*
Following is the line of code where it is throwing an error:
Object body = exchange.getIn().getBody();
if( body instanceof StreamCache){
StreamCache cache = (StreamCache)body;
xml = new String(convertToByteArray(cache,exchange));
exchange.getIn().setBody(xml);
}
By disabling stream cache to write to file by setting a threshold of 10MB in multicast related routes, we were able to work with the aggregation strategy. But we do not want to do that, as we may have incoming data that maybe bigger.
<camel:camelContext id="multicast_xml_1" streamCache="true">
<camel:properties>
<camel:property key="CamelCachedOutputStreamCipherTransformation" value="RC4"/>
<camel:property key="CamelCachedOutputStreamThreshold" value="100000000"/>
</camel:properties>
....
</camel:camelContext>
Note: The FileNotFound issue does not appear if we have the StreamCache based camel component in the route with other processors, but without Multicast + Aggregation.
After debugging, I could understand the issue with aggregating huge data from StreamCache with MulticastProcessor.
In MulticastProcessor.java: doProcessParallel() is called and on completion of the branch exchange of multicast, the CachedOutputStream deletes / cleans up the temporary file.
This happens even before the multicast branch exchange reaches the aggregation Strategy, which tries to read the data from the branch exchange. In case of huge data in StreamCache, the temporary file is already deleted, leading to FileNotFound issues.
public CachedOutputStream(Exchange exchange, boolean closedOnCompletion) {
this.strategy = exchange.getContext().getStreamCachingStrategy();
currentStream = new CachedByteArrayOutputStream(strategy.getBufferSize());
if (closedOnCompletion) {
// add on completion so we can cleanup after the exchange is done such as deleting temporary files
exchange.addOnCompletion(new SynchronizationAdapter() {
#Override
public void onDone(Exchange exchange) {
try {
if (fileInputStreamCache != null) {
fileInputStreamCache.close();
}
close();
} catch (Exception e) {
LOG.warn("Error deleting temporary cache file: " + tempFile, e);
}
}
#Override
public String toString() {
return "OnCompletion[CachedOutputStream]";
}
});
}
}
public void close() throws IOException {
currentStream.close();
cleanUpTempFile();
}
I was able to circumvent the issue, if I try to set closedOnCompletion= false, while writing to CachedOutputStream in any component in any Multicast branch.
But this is a leaky solution, because the streamcache temporary file(s) may then never get cleaned up... hence I try to close + clean up the cachestream, after reading the data in the AggregationStrategy.
Can the MulticastProcessor be adjusted so that the multicast branch exchanges reach 'completion' status only, after they have been aggregated at the end of multicast?
Please help / advise on the issue, as I am new to using camel Multicast.
Thanks,
Lakshmi
I have similar exception thrown when trying to send larger than 1MB JSON response to Restlet request (yes, I know 1MB JSON is too big):
java.io.FileNotFoundException: C:\Users\me\AppData\Local\Temp\camel\camel-tmp-7ad6e098-538d-4d4c-9357-2b7addb1f19d\cos6725022584818060586.tmp (The system cannot find the file specified)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(FileInputStream.java:146)
at org.apache.camel.converter.stream.FileInputStreamCache.createInputStream(FileInputStreamCache.java:123)
at org.apache.camel.converter.stream.FileInputStreamCache.getInputStream(FileInputStreamCache.java:117)
at org.apache.camel.converter.stream.FileInputStreamCache.read(FileInputStreamCache.java:112)
at java.io.InputStream.read(InputStream.java:170)
at java.io.InputStream.read(InputStream.java:101)
at org.restlet.engine.io.BioUtils.copy(BioUtils.java:81)
at org.restlet.representation.InputRepresentation.write(InputRepresentation.java:148)
at org.restlet.engine.adapter.ServerCall.writeResponseBody(ServerCall.java:510)
at org.restlet.engine.adapter.ServerCall.sendResponse(ServerCall.java:454)
at org.restlet.ext.servlet.internal.ServletCall.sendResponse(ServletCall.java:426)
at org.restlet.engine.adapter.ServerAdapter.commit(ServerAdapter.java:196)
at org.restlet.engine.adapter.HttpServerHelper.handle(HttpServerHelper.java:153)
at org.restlet.ext.servlet.ServerServlet.service(ServerServlet.java:1089)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1496)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:102)
Same workaround works for me:
getContext().getProperties().put(CachedOutputStream.THRESHOLD, "" + THREE_MEGABYTE_TRESHOLD_BEFORE_FILE_CACHE);
I don't use multicast in this route, just plain
restlet request -> Service -> Jackson marshall => error
I use Camel 2.14.0 & Restlet 2.2.2 with JDK 7 and Spring-boot 1.0.2 / Jetty
This Camel reverse proxy - no response stream caching might be related to my issue.

Resources