I am trying to hit a REST endpoint on Camel and convert that data into a class (and for simplicity and testing convert that class into a JSON string) and make a POST to a local server. I can get it to do all but make that final post and just seems to hang.
App:
#SpringBootApplication
public class App {
/**
* A main method to start this application.
*/
public static void main(String[] args) {
SpringApplication.run(App.class, args);
}
#Component
public class RestTest extends RouteBuilder {
#Override
public void configure() throws Exception {
restConfiguration().component("restlet").host("localhost").port(8000).bindingMode(RestBindingMode.json);
rest("/test").enableCORS(true)
.post("/post").type(User.class).to("direct:transform");
from("direct:transform")
.transform().method("Test", "alter")
.to("http4:/localhost:8088/ws/v1/camel");
}
}
}
Bean:
#Component("Test")
public class Test {
public void alter (Exchange exchange) {
ObjectMapper mapper = new ObjectMapper();
User body = exchange.getIn().getBody(User.class);
try {
String jsonInString = mapper.writeValueAsString(body);
exchange.getOut().setHeader(Exchange.HTTP_METHOD, constant(HttpMethods.POST));
exchange.getOut().setHeader(Exchange.CONTENT_TYPE, MediaType.APPLICATION_JSON);
exchange.getOut().setBody(jsonInString);
} catch (JsonGenerationException e) {
e.printStackTrace();
} catch (JsonMappingException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
}
}
User:
public class User {
#JsonProperty
private String firstName;
#JsonProperty
private String lastName;
public void setFirstName(String firstName) {
this.firstName = firstName;
}
public String getFirstName() {
return firstName;
}
public void setLastName(String lastName) {
this.lastName = lastName;
}
public String getLastName() {
return lastName;
}
}
UPDATE
Able to get it to work with process instead of transform but errors when a response is sent back to Camel from the POST:
from("direct:transform")
.process(new Processor() {
public void process(Exchange exchange) throws Exception {
ObjectMapper mapper = new ObjectMapper();
User body = exchange.getIn().getBody(User.class);
try {
String jsonInString = mapper.writeValueAsString(body);
exchange.getOut().setHeader(Exchange.HTTP_METHOD, constant(HttpMethods.POST));
exchange.getOut().setHeader(Exchange.CONTENT_TYPE, MediaType.APPLICATION_JSON);
exchange.getOut().setBody(jsonInString);
} catch (JsonGenerationException e) {
e.printStackTrace();
} catch (JsonMappingException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
}
})
.to("http4://0.0.0.0:8088/ws/v1/camel");
Error
com.fasterxml.jackson.databind.JsonMappingException: No serializer found for class org.apache.camel.converter.stream.CachedOutputStream$WrappedInputStream and no properties discovered to create BeanSerializer (to avoid exception, disable SerializationFeature.FAIL_ON_EMPTY_BEANS)
at com.fasterxml.jackson.databind.JsonMappingException.from(JsonMappingException.java:284)
at com.fasterxml.jackson.databind.SerializerProvider.mappingException(SerializerProvider.java:1110)
at com.fasterxml.jackson.databind.SerializerProvider.reportMappingProblem(SerializerProvider.java:1135)
at com.fasterxml.jackson.databind.ser.impl.UnknownSerializer.failForEmpty(UnknownSerializer.java:69)
at com.fasterxml.jackson.databind.ser.impl.UnknownSerializer.serialize(UnknownSerializer.java:32)
at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:292)
at com.fasterxml.jackson.databind.ObjectWriter$Prefetch.serialize(ObjectWriter.java:1429)
at com.fasterxml.jackson.databind.ObjectWriter._configAndWriteValue(ObjectWriter.java:1158)
at com.fasterxml.jackson.databind.ObjectWriter.writeValue(ObjectWriter.java:988)
at org.apache.camel.component.jackson.JacksonDataFormat.marshal(JacksonDataFormat.java:155)
at org.apache.camel.processor.MarshalProcessor.process(MarshalProcessor.java:69)
at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:109)
at org.apache.camel.processor.MarshalProcessor.process(MarshalProcessor.java:50)
at org.apache.camel.component.rest.RestConsumerBindingProcessor$RestConsumerBindingMarshalOnCompletion.onAfterRoute(RestConsumerBindingProcessor.java:363)
at org.apache.camel.util.UnitOfWorkHelper.afterRouteSynchronizations(UnitOfWorkHelper.java:154)
at org.apache.camel.impl.DefaultUnitOfWork.afterRoute(DefaultUnitOfWork.java:278)
at org.apache.camel.processor.CamelInternalProcessor$RouteLifecycleAdvice.after(CamelInternalProcessor.java:317)
at org.apache.camel.processor.CamelInternalProcessor$InternalCallback.done(CamelInternalProcessor.java:246)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:109)
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:197)
at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:97)
at org.apache.camel.component.restlet.RestletConsumer$1.handle(RestletConsumer.java:68)
at org.apache.camel.component.restlet.MethodBasedRouter.handle(MethodBasedRouter.java:54)
at org.restlet.routing.Filter.doHandle(Filter.java:150)
at org.restlet.routing.Filter.handle(Filter.java:197)
at org.restlet.routing.Router.doHandle(Router.java:422)
at org.restlet.routing.Router.handle(Router.java:639)
at org.restlet.routing.Filter.doHandle(Filter.java:150)
at org.restlet.routing.Filter.handle(Filter.java:197)
at org.restlet.routing.Router.doHandle(Router.java:422)
at org.restlet.routing.Router.handle(Router.java:639)
at org.restlet.routing.Filter.doHandle(Filter.java:150)
at org.restlet.engine.application.StatusFilter.doHandle(StatusFilter.java:140)
at org.restlet.routing.Filter.handle(Filter.java:197)
at org.restlet.routing.Filter.doHandle(Filter.java:150)
at org.restlet.routing.Filter.handle(Filter.java:197)
at org.restlet.engine.CompositeHelper.handle(CompositeHelper.java:202)
at org.restlet.Component.handle(Component.java:408)
at org.restlet.Server.handle(Server.java:507)
at org.restlet.engine.connector.ServerHelper.handle(ServerHelper.java:63)
at org.restlet.engine.adapter.HttpServerHelper.handle(HttpServerHelper.java:143)
at org.restlet.engine.connector.HttpServerHelper$1.handle(HttpServerHelper.java:64)
at com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:79)
at sun.net.httpserver.AuthFilter.doFilter(AuthFilter.java:83)
at com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:82)
at sun.net.httpserver.ServerImpl$Exchange$LinkHandler.handle(ServerImpl.java:675)
at com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:79)
at sun.net.httpserver.ServerImpl$Exchange.run(ServerImpl.java:647)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Leading to the question of what is the fundamental difference between process and transform?
A "Processor" in camel is the lowest level message processing primitive. Under the covers a transform definition is executed as a org.apache.camel.processor.TransformProcessor which is itself a processor. In fact mostly everything is a Processor under the hood so strictly anything you can accomplish with transform you can accomplish with a pure processor:
Your last error is because you need to unmarshal the output of your HTTP call to an object, before it can be marshalled back to JSON using Jackson. Something like that:
from("direct:transform")
.transform().method("Test", "alter")
.to("http4:/localhost:8088/ws/v1/camel")
.unmarshal().json(JsonLibrary.Jackson, User.class);
Related
When I need to work with I/O (Query DB, Call to the third API,...), I can use RichAsyncFunction. But I need to interact with Google Sheet via GG Sheet API: https://developers.google.com/sheets/api/quickstart/java. This API is sync. I wrote below code snippet:
public class SendGGSheetFunction extends RichAsyncFunction<Obj, String> {
#Override
public void asyncInvoke(Obj message, final ResultFuture<String> resultFuture) {
CompletableFuture.supplyAsync(() -> {
syncSendToGGSheet(message);
return "";
}).thenAccept((String result) -> {
resultFuture.complete(Collections.singleton(result));
});
}
}
But I found that message send to GGSheet very slow, It seems to send by synchronous.
Most of the code executed by users in AsyncIO is sync originally. You just need to ensure, it's actually executed in a separate thread. Most commonly a (statically shared) ExecutorService is used.
private class SendGGSheetFunction extends RichAsyncFunction<Obj, String> {
private transient ExecutorService executorService;
#Override
public void open(Configuration parameters) throws Exception {
super.open(parameters);
executorService = Executors.newFixedThreadPool(30);
}
#Override
public void close() throws Exception {
super.close();
executorService.shutdownNow();
}
#Override
public void asyncInvoke(final Obj message, final ResultFuture<String> resultFuture) {
executorService.submit(() -> {
try {
resultFuture.complete(syncSendToGGSheet(message));
} catch (SQLException e) {
resultFuture.completeExceptionally(e);
}
});
}
}
Here are some considerations on how to tune AsyncIO to increase throughput: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-Async-IO-operator-tuning-micro-benchmarks-td35858.html
I have following POJO class,
import com.datastax.driver.mapping.annotations.Column;
import com.datastax.driver.mapping.annotations.Table;
#Table(keyspace = "testKey", name = "contact")
public class Person implements Serializable {
private static final long serialVersionUID = 1L;
#Column(name = "name")
private String name;
#Column(name = "timeStamp")
private LocalDateTime timeStamp;
}
and Mapper code is,
DataStream<Reading> sideOutput = stream.flatMap(new FlatMapFunction<String, Person>() {
#Override
public void flatMap(String value, Collector<Person> out) throws Exception {
try {
out.collect(objectMapper.readValue(value, Person.class));
} catch (JsonProcessingException e) {
e.printStackTrace();
}
}
}).getSideOutput(new OutputTag<>("contact", TypeInformation.of(Person.class)));
env.execute();
CassandraSink.addSink(sideOutput)
.setHost("localhost")
.setMapperOptions(() -> new Mapper.Option[]{Mapper.Option.saveNullFields(true)})
.build();
It's not working without .getSideOutput(new OutputTag<>("contact", TypeInformation.of(Person.class))); also.
The sideOutput is not emitting value to store in Cassandra. any idea where I am doing wrong?
I would say, env.execute(); should be called after the pipeline is build, i.e. after the CassandraSink and would get rid of side output. Somethink like this should work:
DataStream<Reading> ds = stream.flatMap(new FlatMapFunction<String, Person>() {
#Override
public void flatMap(String value, Collector<Person> out) throws Exception {
try {
out.collect(objectMapper.readValue(value, Person.class));
} catch (JsonProcessingException e) {
e.printStackTrace();
}
}
});
CassandraSink.addSink(ds)
.setHost("localhost")
.setMapperOptions(() -> new Mapper.Option[]{Mapper.Option.saveNullFields(true)})
.build();
env.execute();
I have StreamExecutionEnvironment job that consumes from kafka simple cql select queries.
I try to handle this queries asynchronically using following code:
public class GenericCassandraReader extends RichAsyncFunction {
private static final Logger logger = LoggerFactory.getLogger(GenericCassandraReader.class);
private ExecutorService executorService;
private final Properties props;
private Session client;
public ExecutorService getExecutorService() {
return executorService;
}
public GenericCassandraReader(Properties props, ExecutorService executorService) {
super();
this.props = props;
this.executorService = executorService;
}
#Override
public void open(Configuration parameters) throws Exception {
client = Cluster.builder().addContactPoint(props.getProperty("cqlHost"))
.withPort(Integer.parseInt(props.getProperty("cqlPort"))).build()
.connect(props.getProperty("keyspace"));
}
#Override
public void close() throws Exception {
client.close();
synchronized (GenericCassandraReader.class) {
try {
if (!getExecutorService().awaitTermination(1000, TimeUnit.MILLISECONDS)) {
getExecutorService().shutdownNow();
}
} catch (InterruptedException e) {
getExecutorService().shutdownNow();
}
}
}
#Override
public void asyncInvoke(final UserDefinedType input, final AsyncCollector<ResultSet> asyncCollector) throws Exception {
getExecutorService().submit(new Runnable() {
#Override
public void run() {
ListenableFuture<ResultSet> resultSetFuture = client.executeAsync(input.query);
Futures.addCallback(resultSetFuture, new FutureCallback<ResultSet>() {
public void onSuccess(ResultSet resultSet) {
asyncCollector.collect(Collections.singleton(resultSet));
}
public void onFailure(Throwable t) {
asyncCollector.collect(t);
}
});
}
});
}
}
each response of this code provides Cassandra ResultSet with different amount of fields .
Any Ideas for handling Cassandra ResultSet in Flink or should I use another technics to reach my goal ?
Thanks for any help in advance!
Cassandra ResultSet is not thread-safe. Better try to use Flink Cassandra connector. Or at least write your implementation in a similar way
During the processing of an Exchange received from JMS I'm creating dynamically a route that fetches a file from FTP to the file system and when the batch is done I need to remove that same route. The following code fragment shows how I do this:
public void execute() {
try {
context.addRoutes(createFetchIndexRoute(routeId()));
} catch (Exception e) {
throw Throwables.propagate(e);
}
}
private RouteBuilder createFetchIndexRoute(final String routeId) {
return new RouteBuilder() {
#Override
public void configure() throws Exception {
from("ftp://" + getRemoteQuarterDirectory() +
"?fileName=" + location.getFileName() +
"&binary=true" +
"&localWorkDirectory=" + localWorkDirectory)
.to("file://" + getLocalQuarterDirectory())
.process(new Processor() {
RouteTerminator terminator;
#Override
public void process(Exchange exchange) throws Exception {
if (camelBatchComplete(exchange)) {
terminator = new RouteTerminator(routeId,
exchange.getContext());
terminator.start();
}
}
})
.routeId(routeId);
}
};
}
I'm Using a thread to stop a route from a route, which is an approach recommended in the Camel Documentation - How can I stop a route from a route
public class RouteTerminator extends Thread {
private String routeId;
private CamelContext camelContext;
public RouteTerminator(String routeId, CamelContext camelContext) {
this.routeId = routeId;
this.camelContext = camelContext;
}
#Override
public void run() {
try {
camelContext.stopRoute(routeId);
camelContext.removeRoute(routeId);
} catch (Exception e) {
throw Throwables.propagate(e);
}
}
}
In result the route does stop. But what I see in the jconsole is that the thread that corresponds to the route isn't removed. Thus in time these abandoned threads just keep accumulating.
Is there a way to properly stop/remove a route dynamically/programmatically and also to release the route's thread, so that they don't accumulate through time?
This is fixed in the next Camel release 2.9.2 and 2.10. Fixed by this ticket:
https://issues.apache.org/jira/browse/CAMEL-5072
i'm newbie with apache camel (I'm using 2.8.1 version). I'm working with this framework and i understand (i hope) concept like route. Now i have this route definition
try {
context.addRoutes(new RouteBuilder() {
#Override
public void configure() throws Exception {
// TODO Auto-generated method stub
from("vm:internal").
split().method(DefaultSplitLogic.class, "split").
dynamicRouter(bean(router, "route"));
}
});
}catch (DefaultSplitLogicException e) {
// TODO: handle exception
e.printStackTrace();
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
This is the DefaultSpliLogic.class
public class DefaultSplitLogic {
public Object[] split(Object o) throws DefaultSplitLogicException{
if(o instanceof Collection<?>){
Collection c = (Collection) o;
return c.toArray();
}
else {
throw new DefaultSplitLogicException("Default Splitting Logic not correct");
}
}
}
This is DefaultSplitLogicException.class
public class DefaultSplitLogicException extends Exception{
/**
*
*/
private static final long serialVersionUID = 1L;
public DefaultSplitLogicException(String msg) {
// TODO Auto-generated constructor stub
super(msg);
System.err.println(msg);
}
public DefaultSplitLogicException(Throwable cause) {
super(cause);
}
}
I leaved router definition.
Now i want to capture my exception (i'm sure that my exception is thrown).
I'm using the onException clause into the route definition
try {
context.addRoutes(new RouteBuilder() {
#Override
public void configure() throws Exception {
// TODO Auto-generated method stub
onException(DefaultSplitLogicException.class).handled(false);
from("vm:internal").
split().method(DefaultSplitLogic.class, "split").
dynamicRouter(bean(router, "route"));
}
});
}catch (DefaultSplitLogicException e) {
// TODO: handle exception
e.printStackTrace();
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
but i cannot manage my exception. I tried to use differently this clause without success. Why?
Thank you all
I think i found the answer. It should be a bug of 2.8.1. version, fixed with 2.8.2+
http://camel.465427.n5.nabble.com/Cannot-handle-Exception-thrown-from-Splitter-Expression-td3286043.html