I'm trying to put together my first Akka/Camel application from "scratch" (read, "noob") using the following lib versions:
akka-camel: 2.2.0-RC1
According to all of the documentation I can find (Akka docs, user groups, etc.) all I have to do to consume from a file-based queue is set up my system this way:
Main class:
actorSystem = ActorSystem.create("my-system");
Props props = new Props(Supervisor.class);
ActorRef supervisor = actorSystem.actorOf(props, "supervisor");
Camel camel = CamelExtension.get(actorSystem);
CamelContext camelContext = camel.context();
camelContext.start();
Supervisor class:
import akka.actor.ActorRef;
import akka.actor.Props;
import akka.camel.javaapi.UntypedConsumerActor;
import org.apache.camel.Message;
/**
* Manages creation and supervision of UploadBatchWorkers.
*/
public class Supervisor extends UntypedConsumerActor {
#Override
public String getEndpointUri() {
return "file:///Users/myhome/queue";
}
#Override
public void preStart() {
String test = "test";
}
#Override
public void onReceive(Object message) {
if (message instanceof CamelMessage) {
// do something
}
}
My problem is that even though I know the supervisor object is being created and breaks during debugging on the preStart() method's "test" line (not to mention that if I explicitly "tell" it something it processes fine), it does not consume from the defined endpoint, even though I have another application producing messages to the same endpoint.
Any idea what I'm doing wrong?
Ok, the problem was my own fault and is clearly visible in the example code if you look at the Consumer trait from which the UntypedConsumerActor inherits.
This method:
#Override
public void preStart() {
String test = "test";
}
overrides its parent's preStart() method, right? Well, that parent method is actually the one that registers the consumer with the on-the-fly created endpoint, so while you can override it, you must call super() or it will not work.
Hope this is useful to someone down the road!
Try changing your instanceof inside of onReceive to this:
if (message instanceof CamelMessage){
//do processing here
}
Where CamelMessage is from package akka.camel. That's what the examples in the akka camel docs are doing.
Related
I have just started working around the Apache Camel. I have a requirement to implement an FTP/FTPS/SFTP client, which would be used to fetch the files from the respective servers. I was looking into the possibility of using Apache Camel to do this but I am still confused after going through the examples and the tutorials.
The requirement is to fetch the files from the FTP/SFTP servers when the request is received from the scheduler.
Following is the route created using EndPoint-DSL
#Component
public class FtpReceiveRoute extends EndpointRouteBuilder {
#Override
public void configure() throws Exception {
from(
ftp("localhost:2001/home/admin")
.account("admin")
.password("admin12345")
.recursive(true)
)
.routeId("ftpReceive")
.log("From done!")
.to("log:ftp-log")
.log("To done!!");
}
}
I am trying to use the above route by invoking it when the request is made to fetch the file like below.
#Override
protected FtpResponse doMessage(String param, FtpRequest req) {
FtpResponse response = new FtpResponse ();
CamelContext ctx = new DefaultCamelContext();
ctx.addRoutes(##route); //FtpReceiveRoute, add the Routebuilder instance as EndpointRouteBuilder is acceptable.
ctx.start();
//Might need to induce sleep so that all the files are downloaded
ctx.stop();
return response;
}
The confusion is around how to invoke the Camel process with the route. I have used EndpointRouteBuilder to create the route because of the type-safe creation of the endpoint URI. I am not getting an option to add this route to the CamelContext as it expects the RouteBuilder instance which is not type-safe.
Further, the CamelContext is the engine and to invoke the route I would need to start and stop this engine. This I am not able to digest if I need to start and stop the engine to execute a route then I would need to induce some sleep in between so that all files are downloaded. Just to add there are more routes that I need to add with the implementation. Once the engine is started it would load and execute all the added routes which is not the requirement.
Maybe I am not getting how to use this properly. Any resources aiding my situation are welcome. Thanks.
You should not create and start new camel context every time you want to fetch file from server. What you should do instead is start one when your application starts and use that for all your exchanges.
You can use Spring-boot to initialize CamelContext and add annotated RouteBuilders to it automatically. Check the maven archetype camel-archetype-spring-boot for example.
If you want to call camel routes from Java you can Inject CamelContext to your bean and use it to create ProducerTemplate. This can be used to invoke Routes defined in the RouteBuilder.
Using ProducerTemplate.send you can get the resulting exchange.
Using producer template
Using File-component which works very similary to ftp-component.
package com.example;
import org.apache.camel.builder.endpoint.EndpointRouteBuilder;
import org.springframework.stereotype.Component;
#Component
public class MySpringBootRouter extends EndpointRouteBuilder {
#Override
public void configure() {
from(direct("fileFromFTP"))
.routeId("fileFromFTP")
// reads files from <project>/input using file consumer endpoint
.pollEnrich(file("input"), 1000)
// If file is found, convert body to string.
// Which in this case will read contents of the file to string.
.filter(body().isNotNull())
.convertBodyTo(String.class)
.end()
;
}
}
package com.example;
import org.apache.camel.CamelContext;
import org.apache.camel.Exchange;
import org.apache.camel.ProducerTemplate;
import org.apache.camel.support.DefaultExchange;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.context.annotation.Configuration;
import org.springframework.scheduling.annotation.EnableScheduling;
import org.springframework.scheduling.annotation.Scheduled;
import static org.apache.camel.builder.endpoint.StaticEndpointBuilders.direct;
#Configuration
#EnableScheduling
public class MySpringBean {
#Autowired
CamelContext camelContext;
#Scheduled(fixedRate = 1000)
public void scheduledTask() {
System.out.println("Scheduled Task!");
if(camelContext.isStopped()) {
System.out.println("Camel context not ready yet!");
return;
}
useProducerTemplate();
}
public void useProducerTemplate(){
ProducerTemplate producerTemplate = camelContext.createProducerTemplate();
Exchange inExchange = new DefaultExchange(camelContext);
//synchronous call!
Exchange result = producerTemplate.send(direct("fileFromFTP").toString(), inExchange);
String resultBody = result.getMessage().getBody(String.class);
String fileName = result.getMessage().getHeader(Exchange.FILE_NAME, String.class);
if(resultBody != null){
System.out.println("Consumed file: "+ fileName + " contents: " + resultBody.toString());
}
else{
System.out.println("No file to consume!");
}
}
}
Depending on what you need to do with the files you could probably do that inside camel route. Then you would only need to call the producerTemplate.sendBody.
public void useProducerTemplate(){
ProducerTemplate producerTemplate = camelContext.createProducerTemplate();
Exchange inExchange = new DefaultExchange(camelContext);
producerTemplate.sendBody(direct("fileFromFTP").toString(), inExchange);
}
Starting stopping camel route
If you want to start polling file consumer only for a short while you can do start the route and use for example aggregation timeout to shutdown the route when no new files have been received in any given duration.
#Component
public class MySpringBootRouter extends EndpointRouteBuilder {
#Override
public void configure() {
AggregationStrategy aggregateFileNamesStrategy = AggregationStrategies
.flexible(String.class)
.accumulateInCollection(ArrayList.class)
.pick(header(Exchange.FILE_NAME))
;
from(file("input"))
.routeId("moveFilesRoute")
.autoStartup(false)
.to(file("output"))
.to(seda("moveFilesRouteTimeout"));
;
from(seda("moveFilesRouteTimeout"))
.routeId("moveFilesRouteTimeout")
.aggregate(constant(true), aggregateFileNamesStrategy)
.completionTimeout(3000)
.log("Consumed files: ${body.toString()}")
.process(exchange -> {
exchange.getContext().getRouteController().stopRoute("moveFilesRoute");
})
.end()
;
}
}
public void startMoveFilesRoute() {
try {
System.out.println("Starting moveFilesRoute!");
camelContext.getRouteController().startRoute("moveFilesRoute");
//Sending null body moveFilesRouteTimeout to trigger timeout if there are no files to transfer
camelContext.createProducerTemplate().sendBody(seda("moveFilesRouteTimeout").toString(), null);
} catch(Exception e) {
System.out.println("failed to stop route. " + e);
}
}
I am trying to create a camel component which consumes an API from an external service.
My Route is as follows
from("myComponent:entity?from=&to=")
.to("seda:one")
from("seda:one")
.aggregate(constant(true), new GroupedBodyAggregationStrategy())
.completionSize(5)
.completionTimeout(5000)
.process( new Processor1() )
to("seda:two")
.
.
.
from("seda:five")
.to("myComponent2:entity")
I implemented my component consumer as follows
public class MyComponentConsumer extends DefaultConsumer {
public MyComponentConsumer(MyComponentEndpoint endpoint, Processor processor) {
super(endpoint, processor);
}
#Override
protected void doStart() throws Exception {
super.doStart();
flag = true;
while ( flag ) {
//external API call
Resource resource = getNextResource();
if ( resource.next() == null ) {
flag = false;
}
Exchange ex = endpoint.createExchange(ExchangePattern.InOnly);
ex.getIn().setBody(resource.toString());
getAsyncProcessor().process(
ex
doneSync -> {
LOG.info("Message processed");
}
);
}
}
#Override
protected void doStop() throws Exception {
super.doStop();
System.out.println("stop ---- ");
}
}
Everything worked fine and the data was propogating through the route. My only problem was that data did not propogate to the next part until the whole of this process was completed. And the next parts were running asynchronously.
I looked at the example of StreamConsumer and tried to implement it to my code using a runnable and an executorService. But if I do that consumer stops as soon as it starts.
I changed the code to
public class MyComponentConsumer extends DefaultConsumer implements Runnable
and added
private ExecutorService executor;
getEndpoint().getCamelContext().getExecutorServiceManager().newSingleThreadExecutor(this, "myComponent");
executor.execute(this);
and moved my logic inside the run() method. But, the consumer thread ends as soon as it starts. and the async processor does not transfer the data properly.
Is there any other way to implement the functionality I need or am I mistaken somewhere here. Any help would be appreciated.
What version of camel are you using?
There was an issue with managing the state of consumer in camel 2.x which was fixed in camel 3.x CAMEL-12765 which can lead to the issue you are describing here.
If you are on camel 2.x try using newScheduledThreadPool instead of newSingleThreadExecutor.
Also executor.schedule(this, 5L, TimeUnit.SECONDS) instead of executor.execute(this).
Delayed start of executor might help avoid the problem you are facing.
I want to use a non serializable object in stream.map() like this
stream.map { i =>
val obj = new SomeUnserializableClass()
obj.doSomething(i)
}
It is very inefficient, because I create many SomeUnserializableClass instance. Actually, it can be created only once in each worker.
In Spark, I can use mapPartition to do this. But in flink stream api, I don't known.
If you are dealing with a non serializable class what I recommend you is to create a RichFunction. In your case a RichMapFunction.
A Rich operator in Flink has a open method that is executed in the taskmanager just one time as initializer.
So the trick is to make your field transient and instantiate it in your open method.
Check below example:
public class NonSerializableFieldMapFunction extends RichMapFunction {
transient SomeUnserializableClass someUnserializableClass;
#Override
public void open(Configuration parameters) throws Exception {
super.open(parameters);
this.someUnserializableClass = new SomeUnserializableClass();
}
#Override
public Object map(Object o) throws Exception {
return someUnserializableClass.doSomething(o);
}
}
Then your code will looks like:
stream.map(new NonSerializableFieldMapFunction())
P.D: I'm using java syntax, please adapt it to scala.
I have a Camel route which converts JSON to an object and passes it on to a processor class. Code is below. This ActiveMQ consumer is not acknowledging some messages, causing the topic to get backed up. The code does not explicitly set acknowledgement mode but a breakpoint shows these values -
acknowledgementMode = -1
acknowledgementModeName = null
What should be changed to ensure acknowledgements are sent on both successful processing and when an exception occurs inside the processor class?
#Component
public class MyRoute extends RouteBuilder {
private String mySubscription;
private MyProcessor myProcessor;
public MyRoute(#Value("${my.topic}") String tripSubscription, MyProcessor myProcessor) {
this.mySubscription = mySubscription;
this.myProcessor = myProcessor;
}
#Override
public void configure() {
from(mySubscription)
.unmarshal().json(JsonLibrary.Jackson, MyDTO.class)
.bean(myProcessor, "process(${body})")
.end();
}
}
The processor class -
#Slf4j
#Component
#AllArgsConstructor
public class MyProcessor {
public void process(MyDTO dto) {
//code that throws exception
}
}
The Camel JMS component docs at Github says that the default acknowledge mode is AUTO_ ACKNOWLEDGE.
However, the older docs at camel.apache.org says the default is -1 what corresponds to the value you see. Either the default was changed in a recent version or the new docs at Github are wrong.
The value -1 is somehow invalid because it is none of the defined modes.
Therefore you could give it a try to explicitly set acknowledgementModeName=AUTO_ACKNOWLEDGE on your consumer.
Side note… looks like you’re not setting “tripSubscription” into and instance variable if that was your intent…
It is possible to pass parameters to a Camel route?, for instance, in the next code snippet:
public class MyRoute extends RouteBuilder {
public void configure() throws Exception {
from("direct:start")
.to("cxf:bean:inventoryEndpoint?dataFormat=PAYLOAD");
}
}
The value for dataFormat is in hard code, but, what if I want set it dynamically?, passing a value from the code where route is called. I know this is possible adding a constructor and passing parameters in it, like this:
public class MyRoute extends RouteBuilder {
private String type;
public MyRoute(String type){
this.type = type;
}
public void configure() throws Exception {
from("direct:start")
.to("cxf:bean:inventoryEndpoint?dataFormat=" + type);
}
}
There is another way?
Thanks so much!
As you mentioned, you can use a constructor (or setters or any other Java/Framework instruments) if the parameters are static from a Camel point of view.
The parameters are configurable in the application, but after the application is started they do no more change. So, every message processed by the Camel route uses the same value.
In contrast, when the parameters are dynamic - i.e. they can change for every processed message, you can use the dynamic endpoint toD() of Camel. These endpoint addresses can contain expressions that are computed on runtime. For example the route
from("direct:start")
.toD("${header.foo}");
sends messages to a dynamic endpoint and takes the value from the message header named foo.
Or to use your example
.toD("cxf:bean:inventoryEndpoint?dataFormat=${header.dataFormat}");
This way you can set the dataformat for every message individually through a header.
You can find more about dynamic endpoints on this Camel documentation page