Apache Camel: Idempotent SFTP is throwing exception on other node unnecessarily? (Infinispan) - apache-camel

Question

Following is code:
return "sftp://"+getSftpHostName() +getSftpImportDirectory()
+ "?username="+getSftpUserName()
+ "&password="+getSftpPassword() // Stored on wildfly server
+ "&download=true" //Shall be read chunk by chunk to avoid heap space issues. Earlier download=true was used: Harpreet
+ "&useList=true"
+ "&stepwise=false"
+ "&disconnect=true"
+ "&passiveMode=true"
+ "&reconnectDelay=10000"
+ "&bridgeErrorHandler=true"
+ "&delay="+getSftpDelay()
+ "&include="+ getSftpFileName()
+ "&preMove=$simple{file:onlyname}.$simple{date:now:yyyy-MM-dd'T'hh-mm-ss}.processing"
+ "&move="+getSftpSuccessDirectory()+"$simple{file:onlyname.noext}.$simple{date:now:yyyy-MM-dd'T'hh-mm-ss}.success"
+ "&moveFailed="+getSftpFailedDirectory()+"$simple{file:onlyname.noext}.$simple{date:now:yyyy-MM-dd'T'hh-mm-ss}.failed"
+ "&readLock=idempotent-changed"
+ "&idempotentRepository=#infinispan"
+ "&readLockRemoveOnCommit=true";
We have three nodes. This locks file on Node 1 and route starts all logs printed but on Node 2 it shows following exception only (Node 3 is fine):
Cannot rename file from: Hrm/test/From_HRM/import/Integrator_2.xml to: Hrm/test/From_HRM/import/Integrator_2.xml.2020-05-07T01-27-19.processing, StackTrace: org.apache.camel.component.file.GenericFileOperationFailedException: Cannot rename file from: Hrm/test/From_HRM/import/Integrator_2.xml to: Hrm/test/From_HRM/import/Integrator_2.xml.2020-05-07T01-27-19.processing
at org.apache.camel.component.file.remote.SftpOperations.renameFile(SftpOperations.java:467)
at org.apache.camel.component.file.strategy.GenericFileProcessStrategySupport.renameFile(GenericFileProcessStrategySupport.java:113)
at org.apache.camel.component.file.strategy.GenericFileRenameProcessStrategy.begin(GenericFileRenameProcessStrategy.java:45)
at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:360)
at org.apache.camel.component.file.remote.RemoteFileConsumer.processExchange(RemoteFileConsumer.java:137)
at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:219)
at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:183)
at org.apache.camel.impl.ScheduledPollConsumer.doRun(ScheduledPollConsumer.java:174)
at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:101)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: 2: No such file
at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2873)
at com.jcraft.jsch.ChannelSftp.rename(ChannelSftp.java:1950)
at org.apache.camel.component.file.remote.SftpOperations.renameFile(SftpOperations.java:463)
Why node 2 even trying to try that file? (Not able to understand what changes shall i do?)
(A small issue we face once a month i.e file gets loss and above error appears on all three nodes. If anyone can comment on same that would be help too)
Updated INfinispan Settings:
public class InfinispanIdempodentRepositoryProducer {
#Resource(lookup = "java:jboss/infinispan/container/camel")
private CacheContainer container;
#Produces
#Named("infinispan")
public InfinispanIdempotentRepository createInfinispanInfinispanIdempotentRepository() {
return new InfinispanIdempotentRepository(container, "camel-default-cache");
}
}
***Configuration***
The configuration of an infinispan cache container
Aliases
Default Cache: camel-default-cache
Module: org.jboss.as.clustering.infinispan
Statistics Enabled: false
***Async Operations***
Defines a thread pool used for asynchronous operations.
Keepalive Time: 60000MILLISECONDS
Max Threads: 25
Min Threads: 25
Queue Length: 1000
***Expiration***
Defines a thread pool used for for evictions.
Keepalive Time: 60000MILLISECONDS
Max Threads: 1
***Listener***
Defines a thread pool used for asynchronous cache listener notifications.
Keepalive Time: 60000MILLISECONDS
Max Threads: 1
Min Threads: 1
Queue Length: 100000
***Persistence***
Defines a thread pool used for interacting with the persistent store.
Keepalive Time: 60000MILLISECONDS
Max Threads: 4
***Remote Command***
Defines a thread pool used to execute remote commands.
Keepalive Time 60000MILLISECONDS
Max Threads: 200
Min Threads: 1
Queue Length: 0
***State Transfer***
Defines a thread pool used for for state transfer.
Keepalive Time: 60000MILLISECONDS
Max Threads: 60
Min Threads: 1
Queue Length: 0
***Transport***
Defines a thread pool used for asynchronous transport communication.
Keepalive Time: 60000MILLISECONDS
Max Threads: 25
Min Threads: 25
Queue Length: 100000
***JGroups***
The description of the transport used by this cache container
Channel: ee
Lock Timeout: 240000MILLISECONDS
More Settings in replicated cache:
***Locking***
Acquire Timeout:15000MILLISECONDS
Concurrency Level:1000
Isolation:READ_COMMITTED
Striping:false
***Partition Handling***
Enabled:false
***State TRansfer***
Chunk Size:512
Timeout:240000MILLISECONDS
***TRansaction***
Locking:OPTIMISTIC
Mode:FULL_XA
Stop Timeout:10000MILLISECONDS
***Expiration***
Interval:60000MILLISECONDS
Lifespan:-1MILLISECONDS
Max Idle:-1MILLISECONDS
***Attributes***
Module:
Remote Timeout:17500MILLISECONDS
Statistics Enabled:false
***Memory***
Size:-1