Is there any embedded persistence solution shipped with Camel API? I want to save my dead letter messages. It can be done through DB or JMS queue but I want to avoid it while shipping to clients.
I read about HawtDB in 'Camel In Action'. But it seems to be tightly coupled with Aggregator EIP. Later i want to create a report on all the dead letter messages and show it to Client.
You can just use any persistence component to save your dead letter messages (Hibernate, JPA, JDBC, DB4O, etc.). In case of majority of persistence components you will not save the message itself, but selected (meta)data from it (the notable exception from that is the case of using DB4O component).
For example if you're sending Hibernates entities in messages, you can use Hibernate component:
from("seda:deadMessageStore").
setBody().groovy("new DeadMessageLog(request.headers, request.body) ").
to("hibernate:com.foo.DeadMessageLog");
errorHandler(deadLetterChannel("seda:deadMessageStore"));
If you cannot predict what kind of classes will you send to the dead letter queue, consider using DB4O component. DB4O is extremely flexible in that matter since it allows you to store any Java object in the your queue (without explicit mapping, like in the case of the Hibernate or JPA).
// store any Java object without transformations
errorHandler(deadLetterChannel("db4o:java.lang.Object"))
Also creating embedded DB40 container is very simple:
ObjectContainer db = Db4oEmbedded.openFile(Db4oEmbedded.newConfiguration(), DB4OFILENAME);
Related
So, we have a use case in our production systems where we could probably use Akka streams. To understand how Akka streams exactly provide back pressure, I would like to go a bit deeper into our requirements.
We have a Solr cluster that hosts some of our data. Next, we have a Play app that serves the front-end customer facing site. Every in-coming request ultimately boils down to fetching a good deal of data from Solr using the /sql handler that Solr provides. Once we fetch the entire dataset from Solr, we write it back after morphing it, to a Cassandra cluster. This can be converted into a problem which can be solved using Akka streams where the Solr stream from the /sql handler will be the akka Source and the Cassandra storage will be the Sink and everything in between will be custom Flows.
I was studying Akka streams and understand it's an implementation of the Reactive streams. Most notably, the way Akka streams provide back pressure to make sure the customer isn't overwhelmed by the producer. Now, with respect to my use case, I want to understand how Akka provides back pressure.
As I can see it, there's a reactive streams library for Cassandra. Since it's the consumer in our case, this driver will be capable of signalling to the producer about how much data it will be able to receive. That would mean, there has to be a corresponding driver on the producer side that can react to this signal and control the emitting of elements. Specifically, since the producer in our case is Solr, isn't it correct that I would also have to use a reactive-compliant Solr driver that I can use to fetch documents from Solr and stream it in my application? This driver would then be capable of controlling the rate at which it has to fetch the documents from the Solr cluster whenever the Cassandra reactive driver signals it to backpressure. Isn't this correct?
If that is indeed the case, will using Akka streams without a non-reactive driver on the producer side provide any benefits? Specifically, are there other ways that Akka publishers can provide back pressure capabilities in such cases when the driver isn't reactive-compliant?
For Solr, there's also a fully reactive Akka Streams implementation from the Alpakka project, so using that as the Source would handle backpressure, though it would mean not using the SQL interface for expressing the query.
On the other hand, since the Solr SQL interface is essentially a JDBC facade which uses Solr, it's possible to use the Alpakka Slick integration as long as you define an instance of slick.jdbc.JdbcProfile which uses the Solr JDBC driver.
I am considering which technology to use for the following use case:
the system is event driven
there is a flow (which is mostly without forks except for error handling)
the flow itself should be sync , but non-blocking
the possibilities I have so far are:
pure java -
this make the code not so clear as I have to nest the callbacks inside one another
and have to write everything myself
apache-camel - use the camel routes
i.e.:
from(URI)
.transform(creatUserExpression) //prepare msg to send to db
.inOut(DB.URI) //send to db
.transform(UserCreatedExpression) //prepare msg to send to next step
.inout(OtherService.URI)
.end();
this looks like a nice solution, but is camel suited to handling all my business logic - all the flows for the events?
camel is mostly used for integration between services, so we are not sure if it would be correct to use it for the business logic
java RX - looks like a possible option , still don't know it enough, and is it production ready?
current release is 0.20.7 - not yet a 1.X version
akka - tried to use it for the flow - but in order to make sure the flows goes only 1 way we needed to use FSM which caused the code to be too complicated and we decided against it
any other suggestions will be appreciated
I agree that most of the time you typically try and stay clear from as much business logic in Integration routes. Business logic on a ESB is typically a big no, some of the more rigid architects I have known will break into violent swearing when they see business logic in the integration layer. This point of view makes sense when you are using a ESB system to integrate services.
In a SOA/Services world you don't want the producers and consumers to be tightly coupled and adding business logic to the integration layer breaks that abstraction. A consumer should be able to consume data from a SAP system, C# web service, Java service or any other service without knowing how the producer work. It should just understand the data.
Apache Camel is not a ESB, it is a EIP toolkit/framework. You can use Apache camel in a client application as well. This is the one of the reasons I really have a soft spot for Camel. It is library I can use to create integration routes. It is flexible and can be used by itself without the need for a full scale server.
So in your case I don't see a problem using apache camel for this purpose. If you are going to install ServiceMix, FuseESB or another full ESB system you are just going to overly complicate the whole setup.
My suggestion(it is just a suggestion) is that in this case having business logic in your route is not going to be bad as this is not really(from your description) about integration but leveraging the power of Camel to create and maintain a event system. Remember Camel does not come with a runtime environment so you still need to host this route somewhere. A simple run time container would be Apache Karaf. You can use this OSGi kernel to install and run your routes on. Last time I check the Karaf project was like under 40MB unzipped so compared to some of the other run times it is really small.
I have used Camel in this fashion to create and host services for android client for example. I guess my main message is that Camel can be considered as a routing engine or routing engine builder which specialises in integration. Camel is not an ESB so the concerns about business logic in here is not always applicable.
I'm currently developing a Camel Integration app in which resumption from a previous state of processing is important. When there's a power outage, for instance, it's important that all previously processed messages are not re-processed. The processing should resume from where it left off before the outage.
I've gone through a number of possible solutions including Terracotta and Apache Shiro. I'm not sure how to use either as documentation on the integration with Apache Camel is scarce. I've not settled on the two, however.
I'm looking for suggestions on the potential alternatives I can use or a pointer to some tutorial to get me started.
The difficulty in surviving outages lies primarily in state, and what to do with in-flight messages.
Usually, when you're talking state within routes the solution is to flush it to disk, or other nodes in the cluster. Taking the aggregator pattern as an example, aggregated state is persisted in an aggregation repository. The default implementation is in memory, so if the power goes out, all the state is lost. However, there are other implementations, including one for JDBC, and another using Hazelcast (a lightweight in-memory data grid). I haven't used Hazelcast myself, but JDBC does a synchronous write to disk. The aggregator pattern allows you to resume from where you left off. A similar solution exists for idempotent consumption.
The second question, around in-flight messages is a little more complicated, and largely depends on where you are consuming from. If you're in the middle of handling a web service request, and the power goes out, does it matter if you have lost the message? The user can simply retry. Any effects on external systems can be wrapped in a transaction, or an idempotent consumer with JDBC idempotent repository.
If you are building out integrations based on messaging, you should consume within a transaction, so that if your server goes down, the messages go back into the broker and can be replayed to another consumer.
Be careful when using seda: or threads blocks, these use an in-memory queue to pass exchanges between threads, any messages flowing down these sorts of routes will be lost if someone trips over the power cable. If you can't afford message loss, and need this sort of processing model, consider using a JMS queue as the endpoints between the two routes (with transactions to ensure you pick up where you left off).
I am building a zeromq PUB-SUB pattern with lots of subscribers to one publisher.
I want to build durable Subscribers, so I found out about the ZMQ_IDENTITy socket-option.
What should I take into account when choosing the value for the Identity?
And can I take the same for all subscribers?
Since they are all the same type but on different machines, that should be a problem, right?
Furthermore, are UUIDs only generated vie inproc service as in the zguide example explained?
Don't use the IDENTITY option. Durable subscribers were removed from ZeroMQ/3.x so your application would be stuck on the 2.x version.
The reason that durable subscribers are so bad we removed them is that they consume arbitrary amounts of memory on the publisher (the messages have to stay somewhere), which leads too easily to out-of-memory, the fastest way to kill your pubsub system for all subscribers.
If you want a durable subscriber model, you can construct it quite easily on top of ZeroMQ, and there's an full worked example in the Guide (Clone pattern).
I'm not sure if anyone's built a durable pubsub broker over ZeroMQ, it would be a fun exercise and not so difficult.
Cound I create a trigger to send a record message to JMS? If yes, how can I do?
Thanks in advance!
I would summarize your options as follows:
Databases Supporting JMS
Oracle is the only database that I am aware of that supports JMS natively in the form of Oracle Advanced Queueing. If your message receiver is not too keen on that JMS implementation, it is usually possible to find some sort of messaging bridge that will transform and forward messages from one JMS implementation to another. For example:
Apache Active MQ
JBoss Messaging
Databases Supporting Java
Some databases such as Oracle and DB2 have a built in Java Virtual Machine and support the loading of third party libraries (Jars) and custom classes that can be invoked by trigger code. Depending on the requirements of your JMS client, this be an issue on account of the version of Java supported (if you need Java 5+ but the DB only supports Java 3). Also keep in mind that threading in some of these embedded JVMs is not what you might expect it to be, but, one might also expect that the sending of JMS messages might be more forgiving of this than the receiving of the same.
Databases Supporting External Invocations (but not in Java)
Several databases support different means of triggering asynchronous events out to connected clients which can in turn forward JMS messages built from the payload of the event:
Oracle: DBMS_ALERT (synchronous), DBMS_PIPE, DCN
Postgres: SQLNotify
Some databases (all of the above and including SQLServer) allow you to send SMTP messages from database procedural code (which can be invoked by triggers). While it is not JMS, a mail listener could potentially listen for mail messages (that might conveniently have a JSON or XML message body) and forward this content as a JMS message.
A basic alternate of this is databases packages that allow HTTP posts to call out to external sources where you might have a servlet listening and forwarding the submitted content as a JMS message.
Other databases such as Postgres support non-java languages such as Perl, Python and Tcl where you might employ some clever scripting to send a message to an external message transformer that will forward as JMS. Active MQ (and therefore its message bridge) supports a multi-language JMS client that includes Python and Perl (and many others).
Lowest Common Denominator
Short of all that, your trigger can write an event to a table and an external client can poll the contents of the table, looking for new data and forwarding JMS messages when it finds it. The JMS message can either include the content, or simply indicate that content exists and what the PK is, and the consumer can come and get it.
This is a technique widely supported in Apache Camel which has adapters (technically called Components) specifically for polling databases:
Hibernate
JDBC
SQL
iBatis
Events read from a database table by Camel can then be transformed and routed to a variety of destinations including a JMS server (in the form of a JMS message). Implementing this Camel is fairly straight forward and well documented, so this is not a bad way to go.
I hope this was helpful.
IBM z and i series servers have db2 with native SQL function such as MQSEND to send to IBM MQ and also MQREAD to read from IBM MQ. Also these server you can create triggers that call a program.