I have a ActiveMQ queue that is being filled by messages from two diffrent processes. The first one supplies the queue with actual messages, the other one adds the "keepAlive" type of message to the queue every few seconds. Messages from that queue are consumed by a single process with camel-netty endpoint.
The "keepAlive" message has to be sent only when there are no other messages being sent in a given time frame. That's why I want to modify the "keepAlive" process to fill the queue only when there are no other messages in that queue which meet certain criteria (e.g. one of the header values equals something)
Is that achievable using only XML route and process definition (without custom code implementation) ?
Related
I'm having some issues trying to figure out the solution for this problem:
I need to implement a DLC on Apache Camel, though when message are dequeued from the dead letter queue I have on ActiveMQ, every single one of them has to be compared with the latest massage present on another AMQ queue.
So to be clear: when Camel is consuming from queue1 (dead letter queue) the message M1, before trying to resend it to a certain route, it has to compare M1 (for example header comparison) with the latest message present on queue2, M2. M2 is not to be removed from queue2 (it shall serve also for the next comparison) while M1 has to be removed from queue1.
I want to understand if this is possible and which EIP I'm missing in order to implement this.
What you need is a QueueBrowser to browse the messages of queue2 without consuming them.
Alternatively you could also consume from queue2 in a transaction and then force a rollback so that the message is not consumed. But when "latest message present on queue2" does not mean the first message, this will not work because you can only process the first message like this.
In producer using librdkafka, is it possible to know the number of items produced but not sent yet. I have turned ack off, only interested in sending the message out from the producer. I wish to avoid producing more messages if there are certain number of messages on the send queue(ie the messages produced but not yet sent).
If you set statistics.interval.ms in the producer configuration file, it will start sending the internal metrics in the callback you register.
The callback you can set using the function rd_kafka_conf_set_stats_cb. The statistics are received in the callback in the form of json data.
One the fields from the json data, msg_cnt, tell you the queued messages that you are looking for.
More details are available at https://github.com/edenhill/librdkafka/blob/master/STATISTICS.md
I'm consuming messages from SQS FIFO queue with maxMessagesPerPoll=5 set.
Currently I'm processing each message individually which is a total waste of resources.
In my case, as we are using FIFO queue and all of those 5 messages are related to the same object, I could process them all toghether.
I though this might be done by using aggregate pattern but I wasn't able to get any results.
My consumer route looks like this:
from("aws-sqs://my-queue?maxMessagesPerPoll=5&messageGroupIdStrategy=usePropertyValue")
.process(exchange -> {
// process the message
})
I believe it should be possible to do something like this
from("aws-sqs://my-queue?maxMessagesPerPoll=5&messageGroupIdStrategy=usePropertyValue")
.aggregate(const(true), new GroupedExchangeAggregationStrategy())
.completionFromBatchConsumer()
.process(exchange -> {
// process ALL messages together as I now have a list of all exchanges
})
but the processor is never invoked.
Second thing:
If I'm able to make this work, when does ACK is sent to SQS? When each individual message is processed or when the aggregate process finishes? I hope the latter
When the processor is not called, the aggregator probably still waits for new messages to aggregate.
You could try to use completionSize(5) instead of completionFromBatchConsumer() for a test. If this works, the batch completion definition is the problem.
For the ACK against the broker: unfortunately no. I think the message is commited when it arrives at the aggregator.
The Camel aggregator component is a "stateful" component and therefore it must end the current transaction.
For this reason you can equip such components with persistent repositories to avoid data loss when the process is killed. In such a scenario the already aggregated messages would obviously be lost if you don't have a persistent repository attached.
The problem lies in GroupedExchangeAggregationStrategy
When I use this strategy, the output is an "array" of all exchanges. This means that the exchange that comes to the completion predicate no longer has the initial properties. Instead it has CamelGroupedExchange and CamelAggregatedSize which makes no use for the completionFromBatchConsumer()
As I don't actually need all exchanges being aggregated, it's enough to use GroupedBodyAggregationStrategy. Then exchange properties will remain as in the original exchange and just the body will contain an "array"
Another solution would be to use completionSize(Predicate predicate) and use a custom predicate that extracts necessary value from groupped exchanges.
I have a EIP design related query.I have a requirement to process csv file by chunks and call a Rest API.After completion of processing of whole file i need to call another Rest API telling processing is complete.I wanted the route to be transacted so i have queue in between in case of end system not available the retry will happen at broker level.
My flow is as below.
First flow:
csv File->Split by chunk of 100 records->Place message in queue
the second flow(Transacted route):
Picks message from queue ->call the rest API
the second flow is transacted.Since iam breaking the flow and it is asynchronous iam not sure how to call to the completion call.I do not have a persistent store to status of each chunk processing.
is there anyway i can achive it using JMS functionality or Camel?
What you can use for your first flow is the Camel Splitter EIP:
http://camel.apache.org/splitter.html
And closely looking at the doc, you will find that there are three exchange properties available for each split exchange:
CamelSplitIndex: A split counter that increases for each Exchange being split. The counter starts from 0.
CamelSplitSize: The total number of Exchanges that was splitted. This header is not applied for stream based splitting. From Camel 2.9 onwards this header is also set in stream based splitting, but only on the completed Exchange.
CamelSplitComplete: Whether or not this Exchange is the last.
As they are exchange properties, you should put them to JMS headers before sending the messages to a queue. But then you should be able to make use of the information at the second flow, so you can know which is the last message.
Keep in mind, though, that it's all asynchronous so the CamelSplitComplete flag doesn't necessarily mean the last message at the second flow. You may create a stateful counter or utilise the Resequencer EIP http://camel.apache.org/resequencer.html to deal with the asynchronicity.
Requirement : A UDP server that on receiving an UDP packet and stores the received packet to one of the two queues. A worker thread is associated with each queue, and the associated thread picks up packet from the front of the queue, processes it and writes it into a in-memory cache system.
Constraints : solution has to be based on event-loop(libuv) and written in C
My Solution
register a call-back for incoming UDP which adds the received packet to one of the two queues and raises a uv_async_send
two global uv_sync_t objects are created one for each queue and used as parameter for uv_async_send. For ex : if packet is added to queue-one then uv_sync_t object-1 is used as parameter for uv_async_send. Similarly, if packet is added to queue-two then uv_sync_t object-2 is used
two threads are started, each having their own loop and a handle bound with a callback
In thread-one uv_sync_t object-1 is bound against a function (say funcA).
In thread-two uv_sync_t object-2 is bound against another function (say funcB)
funcA and funcB reads "SINGLE" packet from corresponding queue and stores it in the in-memory cache
The problem
The Client sends large number of packets which registers large number of events in the server. Now the problem is that libuv combines multiple calls into one and invokes a single callback(which removes a SINGLE node from queue). This leads to situation where nodes are being added to the queue at a faster rate and removed at a very slow rate. Can these rates be balanced ?
Is there a better way to design the server using event-looping library libuv ?
Since you are queueing the packets in one thread but processing in another, it's possible that they work at slightly different rates. I'd use a thread-safe queue (have a look at concurrencykit.org) and process the entire queue on the async callback, instead of just processing a single packet.