Apache camel file component not moving files - apache-camel

I'm using apache camel 2.12.1 to create a route and then move some files in my local directory, the exmple runs fine but the files are never moved, this is the code for the class.
public class MoveFilesTest {
private static final Log LOG = LogFactory.getLog(MoveFilesTest.class);
public static void main(String args[]) throws Exception {
LOG.debug("create CamelContext");
CamelContext context = new DefaultCamelContext();
// add our route to the CamelContext
context.addRoutes(new RouteBuilder() {
File file = null;
public void configure() {
from("file:data/inbox?delay=100&noop=true")
.process( new Processor() {
public void process(Exchange msg) throws Exception {
File file = msg.getIn().getBody(File.class);
LOG.debug("Processing file: " + file.getName());
}
})
.to("file:data/outbox").end();
}
});
LOG.debug("start the route and let it do its work");
context.start();
context.stop();
}
}
as a note, this code just to work, now i'm working on mac os x 10.7, this is the debug log. i added the noop=false and the delete=true, but the result is the same. thank you
DEBUG [main] (MoveFilesTest.java:24) - create CamelContext
DEBUG [main] (MoveFilesTest.java:45) - start the route and let it do its work
INFO [main] (DefaultCamelContext.java:1498) - Apache Camel 2.12.1 (CamelContext: camel-1) is starting
INFO [main] (ManagedManagementStrategy.java:187) - JMX is enabled
INFO [main] (DefaultTypeConverter.java:50) - Loaded 176 type converters
INFO [main] (DefaultCamelContext.java:1689) - StreamCaching is not in use. If using streams then its recommended to enable stream caching. See more details at http://camel.apache.org/stream-caching.html
INFO [main] (FileEndpoint.java:83) - Endpoint is configured with noop=true so forcing endpoint to be idempotent as well
INFO [main] (FileEndpoint.java:89) - Using default memory based idempotent repository with cache max size: 1000
INFO [main] (DefaultCamelContext.java:2183) - Route: route1 started and consuming from: Endpoint[file://data/inbox?delay=100&noop=true]
INFO [main] (DefaultCamelContext.java:1533) - Total 1 routes, of which 1 is started.
INFO [main] (DefaultCamelContext.java:1534) - Apache Camel 2.12.1 (CamelContext: camel-1) started in 8.936 seconds
INFO [main] (DefaultCamelContext.java:1706) - Apache Camel 2.12.1 (CamelContext: camel-1) is shutting down
INFO [main] (DefaultShutdownStrategy.java:172) - Starting to graceful shutdown 1 routes (timeout 300 seconds)
INFO [Camel (camel-1) thread #2 - ShutdownTask] (DefaultShutdownStrategy.java:600) - Route: route1 shutdown complete, was consuming from: Endpoint[file://data/inbox?delay=100&noop=true]
INFO [main] (DefaultShutdownStrategy.java:217) - Graceful shutdown of 1 routes completed in 0 seconds
INFO [main] (DefaultCamelContext.java:1780) - Apache Camel 2.12.1 (CamelContext: camel-1) uptime 8.953 seconds
INFO [main] (DefaultCamelContext.java:1781) - Apache Camel 2.12.1 (CamelContext: camel-1) is shutdown in 0.013 seconds

Yes, you start Camel and stop it immediately. So, when you put a file to a folder. It wont process cuz camel is already stopped.
Camel contains Main implementation to keep Camel running in standalone application.
There is link: http://camel.apache.org/running-camel-standalone-and-have-it-keep-running.html

CamelContext.start does not block, so basically you are starting the context and then immediately stopping it. You need to wait or block on something until the context should stop. You can reference this thread for some ways of doing this.

I had a similar problem but that was to do with streamCaching() not getting set properly and so the code below was failing
context.getStreamCachingStrategy().setSpoolDirectory(spoolDirectory);
from(uri.toString()).streamCaching().to(destination);
I set the Streaming directly on CamelContext and that solved the problem
CamelContext context = getContext();
context.setStreamCaching(true);
context.getStreamCachingStrategy().setSpoolDirectory(localSpoolDirectory);
context.getStreamCachingStrategy().setSpoolThreshold(Long.parseLong(spoolThreshold.trim()));
context.getStreamCachingStrategy().setBufferSize(Integer.parseInt(bufferSize.trim()));

Related

How to configure REST properties in Java-only Camel Main?

I'm working on a simple "hello-world" Camel3 (Camel version 3.4.3) REST API example with the following setup:
REST API is described in OpenAPI specification (Swagger v2.0)
The API spec is converted to Java DSL with camel-restdsl-swagger-plugin
Using Camel Main
100% Java (i.e. no XML)
The example below works as expected but I fail to set jetty port. Therefore on every run the web server runs on a random port. How do I set the port (e.g. to 8081) in the Java code ? It also seems to be possible to set the port in application.properties file. Example of that option is also appreciated.
// https://camel.apache.org/components/latest/others/main.html
import org.apache.camel.builder.RouteBuilder;
import org.apache.camel.main.Main;
import org.apache.camel.main.RestConfigurationProperties;
import rest.dsl.generated.Api101;
public class App {
public static void main(String[] args) throws Exception {
Main main = new Main();
// How do I set the rest properties ?
// The code below doesn't seem to have effect.
RestConfigurationProperties p = main.configure().rest()
.withComponent("jetty")
.withHost("localhost")
.withPort(8081)
;
main.configure().addConfiguration(p);
// The route generated by camel-restdsl-swagger-plugin
main.configure().addRoutesBuilder(
new Api101()
);
// The actual route.
main.configure().addRoutesBuilder(
new RouteBuilder() {
public void configure() {
from("direct:rest1")
.routeId("Rest1Route")
.log("START:")
.setBody(constant("{hello: \"Hello, World!\"}"))
.log("END:")
;
}
}
);
main.run(args);
}
}
Generated Api101 code:
package rest.dsl.generated;
import javax.annotation.Generated;
import org.apache.camel.builder.RouteBuilder;
/**
* Generated from Swagger specification by Camel REST DSL generator.
*/
#Generated("org.apache.camel.generator.swagger.PathGenerator")
public final class Api101 extends RouteBuilder {
/**
* Defines Apache Camel routes using REST DSL fluent API.
*/
public void configure() {
rest("/api101")
.get("/hello")
.description("Basic Hello World")
.to("direct:rest1");
}
}
Compilation & run example:
/tmp/openapi$ mvn clean compile exec:java
[INFO] Scanning for projects...
[INFO]
[INFO] ----------------------< com.stackoverflow:openapi >----------------------
[INFO] Building openapi 1.0
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) # openapi ---
[INFO] Deleting /tmp/openapi/target
[INFO]
[INFO] --- camel-restdsl-swagger-plugin:3.4.3:generate-with-dto (default) # openapi ---
[INFO] reading from /tmp/openapi/swagger.yaml
[INFO] Generating Camel DSL source in directory: /tmp/openapi/target/generated-sources/restdsl-swagger
[INFO] Generating DTO classes using io.swagger:swagger-codegen-maven-plugin:2.4.12
[INFO] reading from /tmp/openapi/swagger.yaml
[WARNING] Output directory does not exist, or is inaccessible. No file (.swagger-codegen-ignore) will be evaluated.
[WARNING] 'host' not defined in the spec. Default to 'localhost'.
[INFO] writing file /tmp/openapi/target/generated-sources/swagger/src/main/java/io/swagger/client/model/InlineResponse200.java
[WARNING] 'host' not defined in the spec. Default to 'localhost'.
[INFO]
[INFO] --- build-helper-maven-plugin:3.2.0:add-source (default) # openapi ---
[INFO] Source directory: /tmp/openapi/target/generated-sources/restdsl-swagger added.
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) # openapi ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 1 resource
[INFO]
[INFO] --- maven-compiler-plugin:3.8.1:compile (default-compile) # openapi ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 3 source files to /tmp/openapi/target/classes
[INFO]
[INFO] --- exec-maven-plugin:3.0.0:java (default-cli) # openapi ---
[ App.main()] BaseMainSupport INFO Using properties from: classpath:application.properties;optional=true
[ App.main()] DefaultRoutesCollector INFO No additional Camel XML routes discovered from: classpath:camel/*.xml
[ App.main()] DefaultRoutesCollector INFO No additional Camel XML rests discovered from: classpath:camel-rest/*.xml
[ App.main()] AbstractCamelContext INFO Apache Camel 3.4.3 (camel-1) is starting
[ App.main()] AbstractCamelContext INFO StreamCaching is not in use. If using streams then its recommended to enable stream caching. See more details at http://camel.apache.org/stream-caching.html
[ App.main()] JettyHttpComponent WARN JMX disabled in CamelContext. Jetty JMX extensions will remain disabled.
[ App.main()] log INFO Logging initialized #8774ms to org.eclipse.jetty.util.log.Slf4jLog
[ App.main()] Server INFO jetty-9.4.29.v20200521; built: 2020-05-21T17:20:40.598Z; git: 77c232aed8a45c818fd27232278d9f95a021095e; jvm 11.0.8+10-post-Ubuntu-0ubuntu120.04
[ App.main()] ContextHandler INFO Started o.e.j.s.ServletContextHandler#4d9f9a55{/,null,AVAILABLE}
[ App.main()] AbstractConnector INFO Started ServerConnector#157aa401{HTTP/1.1, (http/1.1)}{0.0.0.0:44267}
[ App.main()] Server INFO Started #8938ms
[ App.main()] InternalRouteStartupManager INFO Route: route1 started and consuming from: jetty:http://0.0.0.0:0/api101/hello
[ App.main()] InternalRouteStartupManager INFO Route: Rest1Route started and consuming from: direct://rest1
[ App.main()] AbstractCamelContext INFO Total 2 routes, of which 2 are started
[ App.main()] AbstractCamelContext INFO Apache Camel 3.4.3 (camel-1) started in 0.225 seconds
[ qtp1580225589-17] Rest1Route INFO START:
[ qtp1580225589-17] Rest1Route INFO END:
^C[ad #0 - CamelHangupInterceptor] DefaultMainShutdownStrategy INFO Received hang up - stopping the main instance.
[ App.main()] AbstractCamelContext INFO Apache Camel 3.4.3 (camel-1) is shutting down
[ad #0 - CamelHangupInterceptor] DefaultMainShutdownStrategy INFO Waiting for CamelContext to graceful shutdown, elapsed: 0.000 seconds
[ App.main()] DefaultShutdownStrategy INFO Starting to graceful shutdown 2 routes (timeout 45 seconds)
[el-1) thread #1 - ShutdownTask] DefaultShutdownStrategy INFO Route: Rest1Route shutdown complete, was consuming from: direct://rest1
[el-1) thread #1 - ShutdownTask] AbstractConnector INFO Stopped ServerConnector#157aa401{HTTP/1.1, (http/1.1)}{0.0.0.0:0}
[el-1) thread #1 - ShutdownTask] ContextHandler INFO Stopped o.e.j.s.ServletContextHandler#4d9f9a55{/,null,UNAVAILABLE}
[el-1) thread #1 - ShutdownTask] DefaultShutdownStrategy INFO Route: route1 shutdown complete, was consuming from: rest://get:/api101:/hello
[ App.main()] DefaultShutdownStrategy INFO Graceful shutdown of 2 routes completed in 0 seconds
[ App.main()] MainLifecycleStrategy INFO CamelContext: camel-1 has been shutdown, triggering shutdown of the JVM.
[ App.main()] AbstractCamelContext INFO Apache Camel 3.4.3 (camel-1) uptime 44.386 seconds
[ App.main()] AbstractCamelContext INFO Apache Camel 3.4.3 (camel-1) is shutdown in 0.037 seconds
/tmp/openapi$
As you can see from the above the web server is available in port 44267:
Started ServerConnector#157aa401{HTTP/1.1, (http/1.1)}{0.0.0.0:44267}
And can be successfully accessed:
$ sudo netstat -tulpn | grep /java
tcp6 0 0 :::44267 :::* LISTEN 16926/java
$ curl http://0.0.0.0:44267/api101/hello
{hello: "Hello, World!"}
$
I found an answer to my secondary question: this is the way to define the port with the properties:
$ cat src/main/resources/application.properties
camel.rest.port=8081
$
This is also the method how to set all the other configuration options mention in the documentation. Might be obvious for Java veterans, but it was not obvious for me.

Apache Camel Route is working under Windows but not Linux

I am executing Camel code in Windows using Eclipse and it is working fine.
However, when I execute the same code in standalone from Linux, the route has print first log but when fetching file it stops without any error.
Here is my code:
from("timer://alertstrigtimer?period=90s&repeatCount=1")
.log(LoggingLevel.INFO, "*******************************Job-Alert-System: Started: alertstrigtimer******************************" + getFileURI(getWorkFilePath(), getWorkFileName()))
.pollEnrich(getFileURI(getWorkFilePath(), getWorkFileName()))
.log(LoggingLevel.INFO, "*******************************Job-Alert-System: Started: alertstrigtimer******************************" + getFileURI(getWorkFilePath(), getWorkFileName()))
.choice()
.when(header("CamelFileName").isNull())
.log(LoggingLevel.INFO, "No File")
.process(new Processor() {
public void process(Exchange exchange) throws Exception {
log.info("Job-Alert-System: No Date File Exist!!!! Calculate 15 Minutes Back and fetching data from Masterdata");
// Do something
}
})
.otherwise()
.log(LoggingLevel.INFO, "Job Alert System: Date File Loaded: ${header.CamelFileName} at ${header.CamelFileLastModified}")
.process(new Processor() {
// Do something by a processor
})
public static String getFileURI(String filePath, String fileName) {
return "file://" + filePath + "?fileName=" + fileName
+ "&preMove=$simple{file:onlyname.noext}.$simple{date:now:yyyy-MM-dd'T'hh-mm-ss}";
}
Here are my logs from the Linux environment:
[main] INFO org.apache.camel.impl.DefaultCamelContext - Apache Camel 2.21.1 (CamelContext: camel-1) is starting
[main] INFO org.apache.camel.management.ManagedManagementStrategy - JMX is enabled
[main] INFO org.apache.camel.impl.converter.DefaultTypeConverter - Type converters loaded (core: 194, classpath: 0)
[main] INFO org.apache.camel.impl.DefaultCamelContext - StreamCaching is not in use. If using streams then its recommended to enable stream caching. See more details at http://camel.apache.org/stream-caching.html
[main] INFO org.apache.camel.impl.DefaultCamelContext - Route: route1 started and consuming from: timer://alertstrigtimer?period=90s&repeatCount=1
[main] INFO org.apache.camel.impl.DefaultCamelContext - Route: loadDataAndAlerts started and consuming from: direct://loadDataAndAlerts
[main] INFO org.apache.camel.impl.DefaultCamelContext - Total 2 routes, of which 2 are started
[main] INFO org.apache.camel.impl.DefaultCamelContext - Apache Camel 2.21.1 (CamelContext: camel-1) started in 0.664 seconds
[Camel (camel-1) thread #1 - timer://alertstrigtimer] INFO route1 - *******************************Job-Alert-System: Started: alertstrigtimer******************************file:///shared/wildfly/work-files/alerts?fileName=LastExecutionTime_JobAlerts.txt&preMove=.2020-10-12T06-48-16
It stops here. It creates a directory structure, but does not move forward.
Logs from My Local Machine:
[main] INFO org.apache.camel.impl.DefaultCamelContext - Apache Camel 2.21.1 (CamelContext: camel-1) is starting
[main] INFO org.apache.camel.management.ManagedManagementStrategy - JMX is enabled
[main] INFO org.apache.camel.impl.converter.DefaultTypeConverter - Type converters loaded (core: 194, classpath: 5)
[main] INFO org.apache.camel.impl.DefaultCamelContext - StreamCaching is not in use. If using streams then its recommended to enable stream caching. See more details at http://camel.apache.org/stream-caching.html
[main] INFO org.apache.camel.impl.DefaultCamelContext - Route: route1 started and consuming from: timer://alertstrigtimer?period=90s&repeatCount=1
[main] INFO org.apache.camel.impl.DefaultCamelContext - Route: loadDataAndAlerts started and consuming from: direct://loadDataAndAlerts
[main] INFO org.apache.camel.impl.DefaultCamelContext - Total 2 routes, of which 2 are started
[main] INFO org.apache.camel.impl.DefaultCamelContext - Apache Camel 2.21.1 (CamelContext: camel-1) started in 0.845 seconds
[Camel (camel-1) thread #1 - timer://alertstrigtimer] INFO route1 - *******************************Job-Alert-System: Started: alertstrigtimer******************************file://null?fileName=null&preMove=null.2020-10-12T10-28-51
[Camel (camel-1) thread #1 - timer://alertstrigtimer] INFO route1 - Job Alert System: Date File Loaded: null.2020-10-12T10-28-51 at 0
It creates directory structure in addition to a file, but the file is not present and it moves forward.

Apache Camel shuts down right after start, no reason logged, no exception thrown

I observe, that errors in the route definition lead to a silent shutdown of the application, caused by Camel. How can I configure Camel to telling me what exactly it dislikes.
Simple example: Assigning duplicate route names
If I define two routes with different names [by using: .id ("route name")], then the application starts and reports readiness.
If I mistakenly use a name twice, the application fails to start completely and Camel announces:
Apache Camel 2.22.0 (CamelContext: camel-1) is shutting down
Apache Camel 2.22.0 (CamelContext: camel-1) uptime 0.332 seconds
Apache Camel 2.22.0 (CamelContext: camel-1) is shutdown in 0.017 seconds
Stopping service [Tomcat]
HikariPool-1 - Shutdown initiated ...
HikariPool-1 - Shutdown completed.
Process finished with exit code 0
The exit code 0 seems to suggest a non-exceptional shutdown.
The same early shutdown happened, when I added a CronScheduledRoutePolicy to a route. [e.g. with .routePolicy (policy)]
It shuts down if I add a default instance or if I make any settings to the policy.
I increased the log level and got significantly more background noise, but no new findings. [using application.yml: logging: level: ROOT: DEBUG]
A try catch around the route definition didn't help. No exception was caught at the time of the route definition.
I tried to set the CronScheduledRoutePolicy an exception handler. [E.g. with policy.setExceptionHandler (myCamelExceptionHandlerLoggingEverything)]
Nothing got logged. No breakpoint was hit.
I would be very grateful for your help or references to any solutions.
Ralf
There are multiple ways to print an exact error and it depends on how developer design app/class with the camel route. I have mentioned below example which will give exact error you are looking for.
public class CamelPrintingExample {
public static void main(String[] args) throws Exception {
CamelContext camelContext = new DefaultCamelContext();
try {
camelContext.addRoutes(new RouteBuilder() {
public void configure() {
from("file:C:\\input\\").routeId("demo")
.to("file:C:\\output\\").end();
from("file:C:\\input\\").routeId("demo")
.to("file:C:\\output\\").end();
}
});
camelContext.start();
Thread.sleep(300000);
camelContext.stop();
} catch (Exception camelException) {
camelException.printStackTrace();
}
}
}
By running this I got below error.
org.apache.camel.FailedToStartRouteException: Failed to start route demo because of duplicate id detected: demo. Please correct ids to be unique among all your routes.
You can try defining a custom shutdown strategy. Read here
Try to add this in your route and then start it:
getContext().setTracing(true);
P.S. a problem could also be in your error handler

Apache Camel: wireTap and graceful shutdown

While I was debugging my Camel application I realized, that the graceful shutdown of a route ignores the outstanding tasks that have been triggered by wireTap().
If I have a route definition like this:
from("direct:start")
.wireTap("bean:myWireTapBean")
.to("mock:result");
and I set a debugging breakpoint in myWireTapBean (i.e. suspend the asynchronous processing of wireTap) then a CamelContext.stopRoute(routeId) call produces the following log messages:
11:36:12.352 [Thread-1] INFO o.a.camel.spring.SpringCamelContext - Apache Camel 2.19.1 (CamelContext: camel-1) is shutting down
11:36:12.352 [Thread-1] INFO o.a.c.impl.DefaultShutdownStrategy - Starting to graceful shutdown 1 routes (timeout 300 seconds)
11:36:12.352 [Camel (camel-1) thread #6 - ShutdownTask] INFO o.a.c.impl.DefaultShutdownStrategy - Route: route1 shutdown complete, was consuming from: direct://myRoute
11:36:12.352 [Thread-1] INFO o.a.c.impl.DefaultShutdownStrategy - Graceful shutdown of 1 routes completed in 0 seconds
11:36:12.478 [Thread-1] WARN o.a.c.impl.DefaultInflightRepository - Shutting down while there are still 1 inflight exchanges.
11:36:12.478 [Thread-1] INFO o.a.camel.spring.SpringCamelContext - Apache Camel 2.19.1 (CamelContext: camel-1) uptime 1.411 seconds
11:36:12.478 [Thread-1] INFO o.a.camel.spring.SpringCamelContext - Apache Camel 2.19.1 (CamelContext: camel-1) is shutdown in 0.126 seconds
Is there any way to prevent Camel from shutting down while there are still inflight exchanges in the DefaultInflightRepository that have been created by wireTap?
I've already read the FAQ: How can I stop a route from a route but this doesn't seem to be an answer to this question.
Yeah the current implementation of WireTap does not take in account of active tasks if they are not being routed.
I logged a ticket to add support for deferring shutdown of the WireTap EIP if it has inflight tasks: https://issues.apache.org/jira/browse/CAMEL-11539
The workaround is to create a route such as direct and then call that route where you can then call the bean, then when you shutdown Camel the direct route will have inflight exchanges and therefore wait for it to complete.
The DefaultShutdownStrategy's default behavior is to complete route shutdowns, so it is one of those things where the default works for many scenarios, but not all and you are in the latter. If the exchange is run within a transaction, it would rollback, you might look at sourcing from a JMS queue or other so you can rollback and resume processing.
The Javadocs have good information on the behavior and various options you can use to configure the DefaultShutdownStrategy before considering writing a custom one:
DefaultShutdownStrategy

Camel not finding "sftp" component in registry and shutting down

I'm using the camel-sftp component to upload a file to an SFTP server.
The code is simple:
File source = new File(path);
final String sftpUri = "sftp://" + userId + "#" + serverAddress + "/" + remoteDirectory+"?password="+pwd;
CamelContext context = new DefaultCamelContext();
context.addRoutes(new RouteBuilder() {
#Override
public void configure() throws Exception {
from("file:/" + path).to(sftpUri);
}
});
context.start();
Thread.sleep(10000);
context.stop();
However, camel has problems finding the sftp component. Activating the debug logs in Camel it complains:
| 62 - org.apache.camel.camel-core - 2.13.2 | Using ComponentResolver: org.apache.camel.impl.DefaultComponentResolver#29668a43 to resolve component with name: sftp
| 62 - org.apache.camel.camel-core - 2.13.2 | Found component: sftp in registry: null
| 62 - org.apache.camel.camel-core - 2.13.2 | Apache Camel 2.13.2 (CamelContext: camel-16) is shutting down
Any ideas why Camel is behaving this way? In fact, running this code in a standalone application (a Java class with a main method) works correctly.
And I can see:
11:22:13.237 [main] DEBUG o.a.c.impl.DefaultComponentResolver - Found component: sftp in registry: null
11:22:13.239 [main] DEBUG o.a.c.impl.DefaultComponentResolver - Found component: sftp via type: org.apache.camel.component.file.remote.SftpComponent via: META-INF/services/org/apache/camel/component/sftp
Inside of Karaf, though, only the first line appears, for some reason or other it does not find META-INF/services/org/apache/camel/component/sftp and as a result the sftp component is not found.
If you run Camel inside OSGi, you should use the OSGi CamelContext from camel-core-osgi. And then there is a few more steps to setup this for OSGi.
Though its often easier to use a OSGi blueprint application and bootstrap Camel in the blueprint xml file, which does this correctly.
But for Java code its some manual process. In the upcoming Camel 2.15 release there is a new camel-scr component for working with OSGi and SCR (Declarative Services) which makes it easier to do java code with Camel in OSGi using camel-scr.
I would suggest to check its current source code for inspiration how you can setup Camel in OSGi from Java code
https://github.com/apache/camel/tree/master/components/camel-scr

Resources