Modify the message body, or other wise modify some data in a camel route - apache-camel

What is the best way to make a small modification to some data in a Camel route?
I'm pulling in a BSON document from Mongo. I need to use a timestamp from it in an http call, but I need to convert it from milliseconds to seconds.
I tried setting a header.
.setHeader("test").jsonpath("$.startTime")
Which lets me add the timestamp to the URL with a Simple expression.
.toD("https://test.com/api/markets?resolution=60&start_time=${headers.test}")
But I can't find a way to modify the value of the header.
I also tried using a process
.process(new Processor() {
public void process(Exchange exchange) throws Exception {
DocumentContext message = JsonPath.parse(exchange.getMessage().getBody());
String time = message.read("$.startTime").toString();
time = "111100000";
// do something with the payload and/or exchange here
//exchange.getIn().setBody("Changed body");
}
})
But here the exchange isn't passed back out. I based this on how I used an enrich EIP, with an aggregation strategy that returned an Exchange with the changes I made. This Process doesn't seem to work that way.

You can modify body, header or property using Lambda, processor or a bean. With processor you need to use Message.setHeader method to modify the value of the header at least for value types and strings. Bean methods receive body value by default so if you want to pass in header you'll need to specify it using simple language.
import org.apache.camel.Exchange;
import org.apache.camel.Processor;
import org.apache.camel.RoutesBuilder;
import org.apache.camel.builder.RouteBuilder;
import org.apache.camel.component.mock.MockEndpoint;
import org.apache.camel.test.junit4.CamelTestSupport;
import org.junit.Test;
public class SetHeaderTest extends CamelTestSupport {
#Test
public void testGreeting() throws Exception {
MockEndpoint resultMockEndpoint = getMockEndpoint("mock:result");
resultMockEndpoint.expectedMessageCount(3);
template.sendBodyAndHeader("direct:modifyGreetingLambda",
null, "greeting", "Hello");
template.sendBodyAndHeader("direct:modifyGreetingProcessor",
null, "greeting", "Hello");
template.sendBodyAndHeader("direct:modifyGreetingBean",
null, "greeting", "Hello");
resultMockEndpoint.assertIsSatisfied();
}
#Override
protected RoutesBuilder createRouteBuilder() throws Exception {
return new RouteBuilder(){
#Override
public void configure() throws Exception {
from("direct:modifyGreetingLambda")
.routeId("modifyGreetingLambda")
.setHeader("greeting").exchange(exchange -> {
String modifiedGreeting = (String)exchange.getMessage().getHeader("greeting");
modifiedGreeting += " world!";
return modifiedGreeting;
})
.log("${headers.greeting}")
.to("mock:result");
from("direct:modifyGreetingProcessor")
.routeId("modifyGreetingProcessor")
.process(new Processor(){
#Override
public void process(Exchange exchange) throws Exception {
String modifiedGreeting = (String)exchange.getMessage().getHeader("greeting");
modifiedGreeting += " world!";
exchange.getMessage().setHeader("greeting", modifiedGreeting);
}
})
.log("${headers.greeting}")
.to("mock:result");
from("direct:modifyGreetingBean")
.routeId("modifyGreetingBean")
.setHeader("greeting").method(new ModifyGreetingBean(),
"modifyGreeting('${headers.greeting}')")
.log("${headers.greeting}")
.to("mock:result");
}
};
}
public class ModifyGreetingBean {
public String modifyGreeting(String greeting) {
return greeting + " world!";
}
}
}
Aside from these you can also use expression languages like simple or groovy.

In the route you can set the header with the milliseconds value .setHeader("test").jsonpath("$.startTime").
Then in a processor you can retrieve this value:
String milliSecondsValue = (String) exchange.getIn().getHeader("test");
Then you transform the milliSecondsValue to the value you want and you set it back on the exchange:
exchange.getIn().setHeader("test", secondsValue);
After that call .toD("https://test.com/api/markets?resolution=60&start_time=${header.test}") and it will use the seconds value

Related

How can I add message key to KafkaSink in Apache Flink 1.14

As stated in the title I need to set a custom message key in KafkaSink. I cannot find any indication on how to achieve this in the Apache Flink 1.14 docs.
At the moment I'm correctly setting up the KafkaSink and the data payload is correctly written in the topic, but the key is null.
Any suggestions? Thanks in advance
You should implement a KafkaRecordSerializationSchema that sets the key on the ProducerRecord returned by its serialize method.
You'll create the sink more-or-less like this:
KafkaSink<UsageRecord> sink =
KafkaSink.<UsageRecord>builder()
.setBootstrapServers(brokers)
.setKafkaProducerConfig(kafkaProps)
.setRecordSerializer(new MyRecordSerializationSchema(topic))
.setDeliverGuarantee(DeliveryGuarantee.EXACTLY_ONCE)
.setTransactionalIdPrefix("my-record-producer")
.build();
and the serializer will be something like this:
public class MyRecordSerializationSchema implements
KafkaRecordSerializationSchema<T> {
private static final long serialVersionUID = 1L;
private String topic;
private static final ObjectMapper objectMapper =
JsonMapper.builder()
.build()
.registerModule(new JavaTimeModule())
.configure(SerializationFeature.WRITE_DATES_AS_TIMESTAMPS, false);
public MyRecordSerializationSchema() {}
public MyRecordSerializationSchema(String topic) {
this.topic = topic;
}
#Override
public ProducerRecord<byte[], byte[]> serialize(
T element, KafkaSinkContext context, Long timestamp) {
try {
return new ProducerRecord<>(
topic,
null, // choosing not to specify the partition
element.ts.toEpochMilli(),
element.getKey(),
objectMapper.writeValueAsBytes(element));
} catch (JsonProcessingException e) {
throw new IllegalArgumentException(
"Could not serialize record: " + element, e);
}
}
}
Note that this example is also setting the timestamp.
FWIW, this example comes from https://github.com/alpinegizmo/flink-mobile-data-usage/blob/main/src/main/java/com/ververica/flink/example/datausage/records/UsageRecordSerializationSchema.java.
This example is for scala programmers. Here, we are defining a key by generating UUID for each events.
import org.apache.flink.connector.kafka.sink.KafkaRecordSerializationSchema
import org.apache.kafka.clients.producer.ProducerRecord
import java.lang
class MyRecordSerializationSchema extends KafkaRecordSerializationSchema[String] {
override def serialize(element: String, context: KafkaRecordSerializationSchema.KafkaSinkContext, timestamp: lang.Long): ProducerRecord[Array[Byte], Array[Byte]] = {
return new ProducerRecord(
kafkaTopicName,
java.util.UUID.randomUUID.toString.getBytes,
element.getBytes
)
}
}
In the main class, will have to pass an instance of this class while defining the kafka sink like this:
val sinkKafka: KafkaSink[String] = KafkaSink.builder()
.setBootstrapServers(bootstrapServerUrl) //Bootstrap server url
.setRecordSerializer(new MyRecordSerializationSchema())
.build()

Apache Camel - call simple language without side effect

Is it possible to call an object with the Simple language directly within the route and without side effects? The 2 approaches i've tried are;
.toD("${header.exchangeHelper.inc1()}") //works but fails if there is a return type from the called method
.bean(new Simple("${header.exchangeHelper.inc1()}")) //works but sets the body to false
Neither of which give the ideal solution.
You can store the result to exchange property or header instead. This way you'll keep the original body and get the result from your method in case you need it later. Alternatively you can just call the method using a processor.
These are generally better approaches with Java-DSL for something like this than using simple-language since they benefit from IDE's refactoring tools, error highlighting and many forms of linting.
package com.example;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.times;
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.when;
import org.apache.camel.Exchange;
import org.apache.camel.RoutesBuilder;
import org.apache.camel.builder.RouteBuilder;
import org.apache.camel.component.mock.MockEndpoint;
import org.apache.camel.test.junit4.CamelTestSupport;
import org.junit.Test;
public class ExampleTest extends CamelTestSupport {
ExchangeHelper exchangeHelper = mock(ExchangeHelper.class);
#Test
public void useSetPropertyTest() throws Exception {
MockEndpoint resultMockEndpoint = getMockEndpoint("mock:result");
resultMockEndpoint.expectedMessageCount(1);
resultMockEndpoint.message(0).body().isEqualTo("Hello");
when(exchangeHelper.inc1()).thenReturn(true);
template.sendBodyAndHeader("direct:useSetProperty", "Hello",
"exchangeHelper", exchangeHelper);
verify(exchangeHelper, times(1)).inc1();
resultMockEndpoint.assertIsSatisfied();
}
#Test
public void justUseProcessorTest() throws Exception {
MockEndpoint resultMockEndpoint = getMockEndpoint("mock:result");
resultMockEndpoint.expectedMessageCount(1);
resultMockEndpoint.message(0).body().isEqualTo("Hello");
when(exchangeHelper.inc1()).thenReturn(true);
template.sendBody("direct:justUseProcessor", "Hello");
verify(exchangeHelper, times(1)).inc1();
resultMockEndpoint.assertIsSatisfied();
}
#Test
public void useHeaderFromProcessorTest() throws Exception {
MockEndpoint resultMockEndpoint = getMockEndpoint("mock:result");
resultMockEndpoint.expectedMessageCount(1);
resultMockEndpoint.message(0).body().isEqualTo("Hello");
when(exchangeHelper.inc1()).thenReturn(true);
template.sendBodyAndHeader("direct:useHeaderFromProcessor", "Hello",
"exchangeHelper", exchangeHelper);
verify(exchangeHelper, times(1)).inc1();
resultMockEndpoint.assertIsSatisfied();
}
#Override
protected RoutesBuilder createRouteBuilder() throws Exception {
return new RouteBuilder(){
#Override
public void configure() throws Exception {
from("direct:useSetProperty")
.setProperty("result")
.simple("${header.exchangeHelper.inc1()}")
.log("Body: ${body} Result header: ${exchangeProperty.result}")
.to("mock:result")
.removeProperty("result");
from("direct:justUseProcessor")
.process( ex -> { exchangeHelper.inc1(); })
.log("Body: ${body}")
.to("mock:result");
from("direct:useHeaderFromProcessor")
.process( ex -> {
ex.getMessage()
.getHeader("exchangeHelper", ExchangeHelper.class)
.inc1();
})
.log("Body: ${body}")
.to("mock:result");
}
};
}
interface ExchangeHelper {
public boolean inc1();
}
}
Not tried, but why not using the wiretap EIP to issue an extra (and separated!) request to your requestHelper method ?
Something like:
from("direct:demo")
.wireTap("bean:${header.exchangeHelper.inc1()}")
.to("direct:doSomething");
You can simply use Camel Script, something like that:
from("direct:exampleScript")
.script().simple("${header.exchangeHelper.inc1()}")
.log("End")
;

Camel-Kafka Integration Issue

I am trying Camel-Kafka integration.
I have two queues :
queue1 and queue2.
There are three routes :
Route1 puts a list of two messages in queue1 (It should do it only once).
Route2 reads the list from queue1, splits it, and puts the individual messages in queue2
Route3 reads the messages from queue2 and just prints it.
The code is as follows :
import java.util.ArrayList;
import java.util.List;
import org.apache.camel.CamelContext;
import org.apache.camel.Exchange;
import org.apache.camel.Processor;
import org.apache.camel.builder.RouteBuilder;
import org.apache.camel.impl.DefaultCamelContext;
public class CamelListTest {
public static void main(String[] args) throws Exception {
CamelContext context = new DefaultCamelContext();
context.addRoutes(new CamelListRoute());
context.start();
Thread.sleep(30000);
context.stop();
}
}
class CamelListRoute extends RouteBuilder {
#Override
public void configure() throws Exception {
//Route1, expected to run once
from("timer://timerName?repeatCount=1").process(new Processor() {
#Override
public void process(Exchange exchange) throws Exception {
List<String> inOrderList = new ArrayList<String>();
inOrderList.add("1");
inOrderList.add("2");
exchange.getIn().setBody(inOrderList, ArrayList.class);
}
})
.to("kafka:<ip>:9092?topic=queue1");
//Route2
from("kafka:<ip>:9092?topic=queue1&groupId=testing&autoOffsetReset=latest&consumersCount=1")
.split()
.body().process(new Processor() {
#Override
public void process(Exchange exchange) throws Exception {
System.out.println("2nd Route : " + (exchange.getIn().getBody().toString()));
}
})
.to("kafka:<ip>:9092?topic=queue2");
//Route3
from("kafka:<ip>:9092?topic=queue2&groupId=testing&autoOffsetReset=latest&consumersCount=1")
.process(new Processor() {
#Override
public void process(Exchange exchange) throws Exception {
System.out.println("3rd Route : " + (exchange.getIn().getBody().toString()));
}
});
}
}
It is not working as expected, and there are few issues observed :
The first route, which is expected to run only once (repeatCount=1), runs continuously, putting the same message in queue1 again and again.
The second route reads the messages from queue1, splits it, but does not put it in queue2
Since second route does not put anything in queue2, this route does not get any messages.
Can anyone help me figure out what is wrong here?
I see couple of things:
I hope you are giving Kafka Url like this: "kafka://localhost:9092?topic=queue1"
note: kafka://
Providing zookeeper urls for consumers eg: kafka://?topic=queue1&zookeeperConnect=&consumerStreams=1&groupId=testing&autoOffsetReset=largest
Note in previous point autoOffsetReset value will be largest or smallest instead of latest.
I think you shoud exchange the message.
in processor do something like:
exchng.getOut().setHeader("type", "queue");
exchng.getOut().setBody(exchng.getIn().getBody() );
then could add a choice in the second route, does not require the third route.
I believe the first issue of running continuously and putting the same message in queue1 again and again is happening because you are using the same consumer groupId, groupId=testing for both your kafka consumers in routes 2 and 3.
Amend the kafka consumers to consume from different groupIds like so and this will ensure that the message is not consumed again and again.
//Route2
from("kafka:<ip>:9092?topic=queue1&groupId=testing-queue1&autoOffsetReset=latest&consumersCount=1")
and
//Route3
from("kafka:<ip>:9092?topic=queue2&groupId=testing-queue2&autoOffsetReset=latest&consumersCount=1")
The other issues of producing to queue2 and consuming from it to print, I think might be due to version incompatibilities. I have used camel-kafka version 2.20.1 (that uses kafka-clients 0.11.0.1 under the hood) and 2.21.0 (that uses kafka-clients 1.0.0 under the hood) and changed the routes to reflect the changes like so and this seems to consume-produce-consume fine.
class CamelListRoute extends RouteBuilder {
#Override
public void configure() throws Exception {
//Route1, expected to run once
from("timer://timerName?repeatCount=1").process(new Processor() {
#Override
public void process(Exchange exchange) throws Exception {
List<String> inOrderList = new ArrayList<String>();
inOrderList.add("1");
inOrderList.add("2");
exchange.getIn().setBody(inOrderList, ArrayList.class);
}
})
.to("kafka:queue1?brokers=<ip>:9092");
//Route2
from("kafka:queue1?brokers=<ip>:9092&groupId=testing-queue1&autoOffsetReset=latest&consumersCount=1")
.split()
.body().process(new Processor() {
#Override
public void process(Exchange exchange) throws Exception {
System.out.println("2nd Route : " + (exchange.getIn().getBody().toString()));
}
})
.to("kafka:queue2?brokers=<ip>:9092");
//Route3
from("kafka:queue2?brokers=<ip>:9092&groupId=testing-queue2&autoOffsetReset=latest&consumersCount=1")
.process(new Processor() {
#Override
public void process(Exchange exchange) throws Exception {
System.out.println("3rd Route : " + (exchange.getIn().getBody().toString()));
}
});
}
}

Camel FTP with pollStrategy fails

I have a standard route with a ftp uri as a consumer endpoint with a pollStrategy defined and added to the registry. However, I am getting the following error:
Caused by: java.lang.IllegalArgumentException: Could not find a suitable setter for property: pollStrategy as there isn't a setter method with same type: java.lang.String nor type conversion possible: No type converter available to convert from type: java.lang.String to the required type: org.apache.camel.spi.PollingConsumerPollStrategy with value #pollingStrategy
at org.apache.camel.util.IntrospectionSupport.setProperty(IntrospectionSupport.java:588)
at org.apache.camel.util.IntrospectionSupport.setProperty(IntrospectionSupport.java:616)
at org.apache.camel.util.IntrospectionSupport.setProperties(IntrospectionSupport.java:473)
at org.apache.camel.util.IntrospectionSupport.setProperties(IntrospectionSupport.java:483)
at org.apache.camel.util.EndpointHelper.setProperties(EndpointHelper.java:255)
at org.apache.camel.impl.DefaultComponent.setProperties(DefaultComponent.java:257)
at org.apache.camel.component.file.GenericFileComponent.createEndpoint(GenericFileComponent.java:67)
at org.apache.camel.component.file.GenericFileComponent.createEndpoint(GenericFileComponent.java:37)
at org.apache.camel.impl.DefaultComponent.createEndpoint(DefaultComponent.java:114)
at org.apache.camel.impl.DefaultCamelContext.getEndpoint(DefaultCamelContext.java:568)
I have tried different combinations but always end up with this error. Can anyone spot what I am missing? My code seems fairly similar to the Camel unit tests I looked at. The route looks like this:
import org.apache.camel.*;
import org.apache.camel.builder.RouteBuilder;
import org.apache.camel.impl.DefaultPollingConsumerPollStrategy;
import org.apache.camel.spi.PollingConsumerPollStrategy;
import org.apache.camel.util.ServiceHelper;
import org.apache.commons.lang3.StringUtils;
import java.util.HashMap;
import java.util.Map;
import java.util.concurrent.CountDownLatch;
import static org.apache.camel.builder.ProcessorBuilder.setBody;
public class Test extends RouteBuilder {
final CamelContext camelContext = getContext();
final org.apache.camel.impl.SimpleRegistry registry = new org.apache.camel.impl.SimpleRegistry();
final org.apache.camel.impl.CompositeRegistry compositeRegistry = new org.apache.camel.impl.CompositeRegistry();
private final CountDownLatch latch = new CountDownLatch(1);
#Override
public void configure() throws Exception {
ExceptionBuilder.setup(this);
compositeRegistry.addRegistry(camelContext.getRegistry());
compositeRegistry.addRegistry(registry);
((org.apache.camel.impl.DefaultCamelContext) camelContext).setRegistry(compositeRegistry);
registry.put("pollingStrategy", new MyPollStrategy());
from("ftp://user#localhost/receive/in?password=1234&autoCreate=false&startingDirectoryMustExist=true&pollStrategy=#pollingStrategy&fileName=test.csv&consumer.delay=10m")
.convertBodyTo(String.class)
.log(LoggingLevel.INFO, "TEST", "${body} : ${headers}");
}
private class MyPollStrategy implements PollingConsumerPollStrategy {
int maxPolls=3;
public boolean begin(Consumer consumer, Endpoint endpoint) {
return true;
}
public void commit(Consumer consumer, Endpoint endpoint, int polledMessages) {
if (polledMessages > maxPolls) {
maxPolls = polledMessages;
}
latch.countDown();
}
public boolean rollback(Consumer consumer, Endpoint endpoint, int retryCounter, Exception cause) throws Exception {
return false;
}
}
}
Note, if I remove the pollStrategy reference in the uri then everything works.
Ok found the solution..must have had one too many beers when working on this..a bit too obvious.
final CamelContext camelContext = getContext();
final org.apache.camel.impl.SimpleRegistry registry = new org.apache.camel.impl.SimpleRegistry();
final org.apache.camel.impl.CompositeRegistry compositeRegistry = new org.apache.camel.impl.CompositeRegistry();
That part should be in the configure method and not in the class variable declaration part.

camel return value from external Web Service

I need to invoke an external Web service running on WildFly from camel.
I managed to invoke it using the following route:
public class CamelRoute extends RouteBuilder {
final String cxfUri =
"cxf:http://localhost:8080/DemoWS/HelloWorld?" +
"serviceClass=" + HelloWorld.class.getName();
#Override
public void configure() throws Exception {
from("direct:start")
.id("wsClient")
.log("${body}")
.to(cxfUri + "&defaultOperationName=greet");
}
}
My question is how to get the return value from the Web service invocation ? The method used returns a String :
#WebService
public class HelloWorld implements Hello{
#Override
public String greet(String s) {
// TODO Auto-generated method stub
return "Hello "+s;
}
}
If the service in the Wild Fly returns the value then to see the values you can do the below
public class CamelRoute extends RouteBuilder {
final String cxfUri =
"cxf:http://localhost:8080/DemoWS/HelloWorld?" +
"serviceClass=" + HelloWorld.class.getName();
#Override
public void configure() throws Exception {
from("direct:start")
.id("wsClient")
.log("${body}")
.to(cxfUri + "&defaultOperationName=greet").log("${body}");
//beyond this to endpoint you can as many number of componenets to manipulate the response data.
}
}
The second log will log the response from the web service that you are returning. If you need to manipulate or do some routing and transformation with the response then you should look at the type of the response and accordingly you should use appropriate transformer.
Hope this helps.

Resources