I am implementing some application with camel.
My scenario is like blow.
1) Collect files in input directory.
2) Unmarshal the file with specific data format (fixedLengthFormat).
3) If the file is abnormal during unmarshal, some exception is generated.
4) If the exception occurs, camel moves original file to exception directory.
I found exception like below.
org.apache.camel.component.file.GenericFileOperationFailedException: Cannot store file: C:\home\WRK\PRC\WLNE\IPTVKR\KRLPPM10\FATAL\SLPNPM_FGIDRO01_ID0006_T20190312050500.DAT
at org.apache.camel.component.file.FileOperations.storeFile(FileOperations.java:354) ~[camel-core-2.23.1.jar:2.23.1]
at org.apache.camel.component.file.GenericFileProducer.writeFile(GenericFileProducer.java:305) ~[camel-core-2.23.1.jar:2.23.1]
at org.apache.camel.component.file.GenericFileProducer.processExchange(GenericFileProducer.java:169) ~[camel-core-2.23.1.jar:2.23.1]
at org.apache.camel.component.file.GenericFileProducer.process(GenericFileProducer.java:80) ~[camel-core-2.23.1.jar:2.23.1]
....
This is an error caused by bad parsing of body.
So, I added tag with String type before tag.
I did this and there was no error.
But, moved file has data format to unmarshal, not original data.
How should I handle some code in order to get original data?
Data Format for unmarshal (KRFixedFormat.java)
#FixedLengthRecord(length=3005, header=KRHeader.class, footer=KRTailer.class)
public class KRFixedFormat implements Serializable {
private static final long serialVersionUID = 1L;
#DataField(pos=1, length=1,trim=true)
private String sType;
#DataField(pos=2, length=7, trim=true, align="L")
private String sRecordSeq;
#DataField(pos=9, length=1, trim=true, defaultValue="2")
private char sServiceType;
...
Camel Context
<doTry id="_CheckException">
<camel:convertBodyTo charset="ISO-8859-1"
id="_ConvertEncoding" type="java.lang.String"/>
<camel:unmarshal id="_FileParsing">
<camel:bindy
classType="com.ktds.openmzn.dao.KRFixedFormat"
locale="korea" type="Fixed"/>
</camel:unmarshal>
<doCatch id="_ParsingException">
<exception>org.apache.camel.component.bean.MethodNotFoundException</exception>
<exception>org.apache.camel.NoTypeConversionAvailableException</exception>
<convertBodyTo charset="ISO-8859-1"
id="_ConvertEncoding" type="java.lang.String"/>
<toD id="_MoveErrorFile" uri="file:${header.CamelFileParent}/FATAL"/>
</doCatch>
</doTry>
...
Original File
H20190312050500GIDRO01LPNPM
R0000001269056802668 D201903 ....
T000000120190312050500000000003091
Output File
KRFixedFormat [sType=R, sRecordSeq=0000001, sServiceType=2, sChrgId=69056802668, ....
Related
I am having a number of type conversion issues using the Java DSL with Camel 3.14.3. For a simple example I have a route that uses a direct endpoint to trigger a pollEnrich for a file endpoint.
public class BasicRoute extends RouteBuilder {
#Override
public void configure() {
from("direct:test")
.pollEnrich("file://watchDirectory", 10000)
.to("mock:result");
}
}
When the route starts I get the following exception...
Exception in thread "main" org.apache.camel.FailedToCreateRouteException: Failed to create route route1 at: >>> PollEnrich[constant{file://watchDirectory}] <<< in route: Route(route1)[From[direct:test] -> [PollEnrich[constant{file... because of Error parsing [10000] as a java.time.Duration.
...
Caused by: org.apache.camel.NoTypeConversionAvailableException: No type converter available to convert from type: java.lang.String to the required type: java.time.Duration with value 10000
I am running this within a simple OG java app, so I am sure I am missing something in the context initialization, but I cannot find it.
I am trying to add a custom sftp component in Apache Camel to wrap the username, host, port and password in a configuration object to be passed to a sftpcomponent.
Below is the code that I have tried:
#Configuration
class SftpConfig {
#Bean("sourceSftp")
public SftpComponent getSourceSftpComponent(
#Qualifier("sftpConfig")
SftpConfiguration sftpConfig) throws Exception{
SftpComponent sftpComponent = new SftpComponent();
// not getting way to set the configuration
return sftpComponent;
}
#Bean("sftpConfig")
public SftpConfiguration getSftpConfig(
#Value("${host}") String host,
#Value("${port}") int port,
#Value("${applicationUserName}") String applicationUserName,
#Value("${password}") String password) {
SftpConfiguration sftpConfiguration = new SftpConfiguration();
sftpConfiguration.setHost(host);
sftpConfiguration.setPort(port);
sftpConfiguration.setUsername(applicationUserName);
sftpConfiguration.setPassword(password);
return sftpConfiguration;
}
}
//In other class
from("sourceSftp:<path of directory>") ---custom component
A similar approach in JMSComponent works fine where I have created a bean for sourcejms, but I am not able to do it for sftp as SftpComponent doesn't have set call for sftpconfiguration.
The Camel maintainers seem to be moving away from providing individual components with a "setXXXConfiguration" method to configure their properties. The "approved" method of providing properties -- which works with the SFTP -- is to specify them on the connection URL:
from ("sftp://host:port/foo?username=foo&password=bar")
.to (....)
An alternative approach is to instantiate an endpoint and set its properties, and then use a reference to the endpoint in the from() call. There's a gazillion ways of configuring Camel -- this works for me for XML-based configuration:
<endpoint id="fred" uri="sftp://acme.net/test/">
<property key="username" value="xxxxxxx"/>
<property key="password" value="yyyyyyy"/>
</endpoint>
<route>
<from uri="fred"/>
<to uri="log:foo"/>
</route>
You can customize it by extending the SftpComponent. This allows you to define multiple endpoints without providing the username/password for each endpoint definition.
Step 1: Extend SftpComponent and give your component a custom name, ie customSftp
#Component("customSftp")
public class CustomSftpComponent extends SftpComponent {
private static final Logger LOG = LoggerFactory.getLogger(CustomSftpComponent.class);
#Value("${sftp.username}")
private String username;
#Value("${sftp.password}")
private String password;
#SuppressWarnings("rawtypes")
protected void afterPropertiesSet(GenericFileEndpoint<SftpRemoteFile> endpoint) throws Exception {
SftpConfiguration config = (SftpConfiguration) endpoint.getConfiguration();
config.setUsername(username);
config.setPassword(password);
}
}
Step 2: Create a camel route to poll 2 different folders using your custom component name.
#Component
public class PollSftpRoute extends RouteBuilder {
#Override
public void configure() throws Exception {
from("{{sftp.endpoint1}}").routeId("pollSftpRoute1")
.log(LoggingLevel.INFO, "Downloaded file from input folder 1.")
.to("file:data/out1");
from("{{sftp.endpoint2}}").routeId("pollSftpRoute2")
.log(LoggingLevel.INFO, "Downloaded file from input folder 2.")
.to("file:data/out2");
}
}
Step 3: Place this in application.properties
camel.springboot.main-run-controller=true
sftp.endpoint1=customSftp://localhost.net/input/1?delay=30s
sftp.endpoint2=customSftp://localhost.net/input/2?delay=30s
sftp.username=sftp_user1_l
sftp.password=xxxxxxxxxxxx
With this you don't have to repeat the username/password for each endpoints.
Note: With this approach you wont be able to set the username/password in URI endpoint configuration. Anything you set in URI will be replaced in afterPropertiesSet.
I've an SFTP route (in Spring XML), and its from path ends in a daily changing directory (ie. /yyyyMMdd), and everything is working well when autoCreate=true or the directory exists when the route starts. But it is not permitted to me to create the directory if not exists!
When the dir exists, the route get files and terminates itself.
When the dir not exists, the route is polling permanently with a warning (ie. org.apache.camel.component.file.GenericFileOperationFailedException: Cannot change directory to: 20160917) and never stops.
How can I avoid this behaviour (eg. convert the warning to an empty message or exception or ...)? I've done experiments with startingDirectoryMustExist, consumer.bridgeErrorHandler and many others without any success.
The simplified route (before start, fill the elmu.sftp.importDir property with the actual date):
<from
uri="sftp://{{elmu.sftp.host}}:{{elmu.sftp.port}}{{elmu.sftp.importDir}}?username={{elmu.sftp.userName}}&password={{elmu.sftp.password}}&
autoCreate=false&preferredAuthentications=password&binary=true&include={{elmu.importMask}}&initialDelay=100&
noop=true&sortBy=file:name&sendEmptyMessageWhenIdle=true"/>
<choice>
<when>
<simple>${body} != null</simple>
... a lot of stuff ...
<to uri="bean:shutdownRoute" />
</when>
<otherwise>
<to uri="bean:shutdownRoute" />
</otherwise>
</choice>
With directoryMustExist=true and startingDirectoryMustExist=true the result is an endless loop (poll) with this warning:
08:30:14,658 WARN SftpConsumer - Consumer Consumer[sftp://xxx.xxx.xx:22/DBHtest/ELMUteszt/Kiadott_adatok/20160918?autoCreate=false&binary=true&directoryMustExist=true&include=%5E.*%24&initialDelay=100&noop=true&password=xxxxxx&preferredAuthentications=password&sendEmptyMessageWhenIdle=true&sortBy=file%3Aname&startingDirectoryMustExist=true&username=xxx] failed polling endpoint: Endpoint[sftp://xxx:22/DBHtest/ELMUteszt/Kiadott_adatok/20160918?autoCreate=false&binary=true&directoryMustExist=true&include=%5E.*%24&initialDelay=100&noop=true&password=xxxxxx&preferredAuthentications=password&sendEmptyMessageWhenIdle=true&sortBy=file%3Aname&startingDirectoryMustExist=true&username=xxx]. Will try again at next poll. Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - Cannot change directory to: 20160918]
org.apache.camel.component.file.GenericFileOperationFailedException: Cannot change directory to: 20160918
at org.apache.camel.component.file.remote.SftpOperations.doChangeDirectory(SftpOperations.java:576)
at org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:564)
at org.apache.camel.component.file.remote.SftpConsumer.doPollDirectory(SftpConsumer.java:107)
at org.apache.camel.component.file.remote.SftpConsumer.pollDirectory(SftpConsumer.java:79)
at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:131)
at org.apache.camel.impl.ScheduledPollConsumer.doRun(ScheduledPollConsumer.java:175)
at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:102)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: 2: No such file
at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2846)
at com.jcraft.jsch.ChannelSftp._realpath(ChannelSftp.java:2340)
at com.jcraft.jsch.ChannelSftp.cd(ChannelSftp.java:342)
at org.apache.camel.component.file.remote.SftpOperations.doChangeDirectory(SftpOperations.java:574)
... 13 more
It doesn't work with stepwise=false:
11:52:19,210 WARN SftpConsumer - Consumer Consumer[sftp://xxx:22/DBHtest/ELMUteszt/Kiadott_adatok/20160918?autoCreate=false&binary=true&directoryMustExist=true&include=%5E.*%24&initialDelay=100&noop=true&password=xxxxxx&preferredAuthentications=password&sendEmptyMessageWhenIdle=true&sortBy=file%3Aname&startingDirectoryMustExist=true&stepwise=false&username=xxx] failed polling endpoint: Endpoint[sftp://xxx:22/DBHtest/ELMUteszt/Kiadott_adatok/20160918?autoCreate=false&binary=true&directoryMustExist=true&include=%5E.*%24&initialDelay=100&noop=true&password=xxxxxx&preferredAuthentications=password&sendEmptyMessageWhenIdle=true&sortBy=file%3Aname&startingDirectoryMustExist=true&stepwise=false&username=xxx]. Will try again at next poll. Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - Cannot list directory: DBHtest/ELMUteszt/Kiadott_adatok/20160918]
org.apache.camel.component.file.GenericFileOperationFailedException: Cannot list directory: DBHtest/ELMUteszt/Kiadott_adatok/20160918
UPDATE (according to #ruffp's answer):
I've tried to set up a custom PollingConsumerPollStrategy, but I cannot stop the route from it. Only the third (commented row) stops the route, but I've several routes and I cannot know the name of the actual route. How can I get it?
#Override
public boolean rollback(Consumer consumer, Endpoint endpoint, int retryCounter, Exception cause) throws Exception {
consumer.getEndpoint().stop(); // 1
consumer.stop(); // 2
consumer.getEndpoint().getCamelContext().stopRoute(route???); // 3
return false;
}
Finally I've solved it with consumer.exceptionHandler. But it seems, that this option is not in list of the usable options (http://camel.apache.org/file2.html which I've been reading again and again) just mentioned once with an example at the bottom of the huge page. Unfortunately, it was so "hidden" that I missed it up to now.
I set up a new class and implemented the handleExceptions methods:
public class DirNotExistsExHandler implements ExceptionHandler
which get the exceptions and decide what to do. In the context, I did a bean definition:
<bean id="dirNotExistsExHandler" class="hu.dbit.eleo.DirNotExistsExHandler" />
And in the consumer, passed the bean to the handler:
consumer.exceptionHandler=#dirNotExistsExHandler
Many thanks for your help!
In my opinion the best way to do it would be:
Enable the option throwExceptionOnConnectFailed on the sftp endpoint like this:
sftp://{{elmu.sftp.host}}:{{elmu.sftp.port}}{{elmu.sftp.importDir}}?username={{elmu.sftp.userName}}&password={{elmu.sftp.password}}&
autoCreate=false&preferredAuthentications=password&binary=true&include={{elmu.importMask}}&initialDelay=100&
noop=true&sortBy=file:name&sendEmptyMessageWhenIdle=true&throwExceptionOnConnectFailed=true
This could help to manage with the exception and to be catch it by the camel routes onException. However I am not sure it is really necessary in your case.
Make a special processing for your connection exception
// 2a) Solution to redirect an empty body to a destination
onException(GenericFileOperationFailedException.class)
.handled(true)
.log(LoggingLevel.INFO, "Source directory not present: send empty body to shutdown route...")
.setBody(null)
.to("bean:shutdownRoute");
Or another way:
// 2b)Solution to just stop the processing without log in warn or error
onException(GenericFileOperationFailedException.class)
.handled(true)
.log(LoggingLevel.INFO, "Source directory not present: stop the process and wait until next time...")
.stop();
UPDATE:
I found this post and apparently there is not another way than implementing your own PollingConsumerPollStrategy because the GenericFileOperationFailedException is apparently handled inside the default implementation.
Handle the exception using doCatch block. If no exception is thrown then check file exists using your code and throw exception manually.
http://camel.apache.org/try-catch-finally.html
Otto Csatari: To get Route Id of failed route in rollback method use below code snipped
CamelContext context = endpoint.getCamelContext();
List<Route> routes = context.getRoutes();
SftpEndpoint sftpEndpoint = (SftpEndpoint)endpoint;
Route failedRoute = routes.stream().filter(route -> ((SftpEndpoint) route.getEndpoint())
.getConfiguration().getDirectoryName().equals(sftpEndpoint.getConfiguration().getDirectoryName()))
.findAny().orElse(null);
I am struggling with type conversion in my camel route for handling ftp files. My route looks like this (in Spring DSL):
<route id="processIncomingFtpFile" errorHandlerRef="parkingCitErrHandler">
<from uri="{{ftp.parkingcit.input.path}}"/>
<bean ref="ftpZipFileHandler"/>
<unmarshal ref="bindyCsvFormat"/>
<bean ref="parkingTicketsHandler"/>
<split>
<simple>${body}</simple>
<marshal ref="jaxbFormatter"/>
<convertBodyTo type="java.io.File"/>
<to uri="{{ftp.parkingcit.output.path}}"/>
</split>
</route>
And my handler signature looks like this:
public File handleIncomingFile(File incomingfile)...
However, this yields the following type conversion problem:
org.apache.camel.InvalidPayloadException: No body available of type: java.io.File but has value: RemoteFile[test.zip] of type: org.apache.camel.component.file.remote.RemoteFile on: test.zip. Caused by: No type converter available to convert from type: org.apache.camel.component.file.remote.RemoteFile to the required type: java.io.File with value RemoteFile[test.zip]. Exchange[test.zip]. Caused by: [org.apache.camel.NoTypeConversionAvailableException - No type converter available to convert from type: org.apache.camel.component.file.remote.RemoteFile to the required type: java.io.File with value RemoteFile[test.zip]]
My question is: should I be able to handle my ftp file in-memory, without explicitly telling camel to write it to disk, with type converters doing the work automagically behind the scenes for me? Or is what I am trying to do senseless, given that my handler wants a java.io.File as its input parameter, i.e. I must write the data to disk for this to work?
java.io.File is for file systems, and not for FTP files.
You would need to change the type in your bean method signature to either Camel's GenericFile or the actual type which is from the commons-net FTP client library that Camel uses.
hmm, I would recommend to work with a local file (I always do that with the ftp endpoint). There is an option in the ftp endpoint called localWorkDirectory that lets you define a directory to download the file ( typically, /tmp ) before further processing. The idea is to avoid any error due to network issue or being disconnected during process
Can you try it It's easy enough (just add &localWorkDirectory =mydir in the uri ) and it will dl the file for you
see https://camel.apache.org/ftp2.html.
Just make sure you have write right on the directory, of course
I've created a new converter with the following structure:
package com.mycompany;
#Converter
public final class MyCustomConverter {
#Converter
public static TypeB convert(TypeA typeA) {
// do the conversion
}
}
I've also created in my src/main/java/resources folder the following package, META-INF.services.org.apache.camel and within that a TypeConverter file which just says:
com.mycompany
Within my route from TypeA I have got:
<convertBodyTo type="com.mycompany.TypeB" />
Yet my tests constantly fail to pick up the file and thus cannot find the converter, with the exception being:
Caused by: org.apache.camel.NoTypeConversionAvailableException: No
type converter available to convert from type: com.mycompany.TypeA to
the required type: com.mycompany.TypeB with value TypeA[value1="blah"]
Am I meant to do something else to get my test to pick up the TypeConverter file? Surely putting it in that resources folder with the exact structure adds it to the classpath and so it would be accessible.
Its recommended in the TypeConverter file to list the FQN of all the type conveters, eg
com.mycompany
Should be
com.mycompany.MyCustomConverter
This is also what we say on this page: http://camel.apache.org/type-converter.html
And could you check inside the generated JAR file of yours, that the META-INF/services/org/apache/camel directory is there, and that the TypeConverter file is present (and it is not in some directory like META-INF/org.apache.camel).
Also what is the runtime environment you use? Do you run Camel standalone, Tomcat, OSGi or something else?