#camel Hi Devs, I am currently working on a camel to transform messages from Source to Target systems, I am stuck with an issue i.e., I want redelivery my message when any exceptions occurred or due to failure caused by endpoints. I had checked the camel docs then I got a info related to Redelivery Polices It is working as per the given delay time. But my problem is used to replay messages whenever I want. For example last year there are some messages which got failure those payloads are stored in my system. So I want to replay those messages this year. like Replay. Can any devs help me on this cause? Thanks.
You can simply use the DeadLetterChannel EIP (https://camel.apache.org/components/3.16.x/eips/dead-letter-channel.html).
This will put your failed messages in a special channel (typically a persistant JMS queue). To replay a message, you only have to move it from the DLC ("myqueue.dead") to the original queue ("myqueue") .
Related
When I install a Mosquitto Broker and publish messages to a topic and subscribe the the messages using an Apache Beam MQTTIO pipeline and print the message in the console, I am able to get all the messages.
Even after a gap of 5 minutes if I publish a message, I am able to see the message in the console of the Beam application
Now I changed nothing but the ServerUri in the ConnectionConfiguration of MQTTIO.Read() and gave a corresponding topic that is available in the that broker. Refer Documentation
This change made the application work differently, only the topics that have higgh frequency of messages are being printed in the console. When the message frequency if 1 message per minute or somewhere near to that range of frequency, only the first message is getting printed in the console.
I even tried out withMaxNumRecords and withMaxReadTime but still it just lists the first message.
Is there Some timeout that is happening in the broker configuration because of its idlenesss between messages?
The same behaviour was observed in runners - DirectRunner and FlinkRunner
Changing the options to --streaming=true for the pipeline using the arguments did not work as well
As the title says, I would like to send a notification to an endpoint if messages to it start getting throttled, and another message when the throttling stops.
I currently have the following (very basic) route configuration:
from("test-jms:queue:test.queue")
.throttle(2)
.to("file://test");
This configuration throttles messages just fine, but I need a way to let the consumer know that the messages are being throttled.
When the Throttler starts throttling, I would like to send a notification to the 'to' endpoint so those reading the messages know that they are being throttled. I would also like to be able to send another message when the Throttler is no longer throttling, so the consumer knows the messages are up to date.
This doesn't appear to be something the Throttler does. The only way I see of getting a notification when it starts throttling is setting rejectExecution to true, at which point it will throw an exception. The problem is that execution stops at that point, and no more messages are passed through (since an exception was thrown).
My current thoughts are that I will need to create a custom bean/processor/something that performs essentially the same function as the Throttler, but also injects a message when the throttling starts or stops. I don't want to do that unless I really need to, though. Any help is appreciated. Thanks!
No the throttler eip does not support such information (however you may be able to grab some statistics via JMX). A different thought would be to reverse the direction so the consumers signals upstream when they want new messages (this is what reactive systems does).
I assume the above to write to a file is just some example, what consumers are you using in real life, and do they really need to know that some messages are backed up within a short-time period of 1 second because they are throttled? Also since your source is JMS, you can also look at the route throttling policy, where you can suspend/resume the JMS consumer instead of using the throttler EIP.
In Google PubSub, the publish call from the client can be called asynchronously. Because of this, I would think that it would be possible to have multiple publish requests triggered and sent to the server, all at the same time, especially if the batch thresholds are too low.
If this is true, how does the pubsub client control the number of simultaneous publish requests that can be created? Is there a hard limit, or an error that can occur if too many requests are created? Is this the intended use of having an asynchronous publisher, or is simply to allow for other non-publishing activity to occur?
Though this question applies to any of the clients, we are specifically having an issue with the C# client, and are intermittently receiving the following error:
Grpc.Core.RpcException: Status(StatusCode=DeadlineExceeded, Detail="Deadline Exceeded")
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Google.Api.Gax.Grpc.ApiCallRetryExtensions.<>c__DisplayClass0_0`2.<<WithRetry>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
My thought is that we are sending too many publish requests..., but I am not sure.
I would advice using the raw gRPC code, but use the client library that has a very thin wrapper.
Looking at the client source code always helps me, you can find for c# code here PublisherClient.cs (thin wrapper)
If you are using PublishAsync it queues/batch the messages anyway, the behaviour is controlled the settings you give to the client (see PublisherServiceApiClient for how to tune it). You can also control the number of client connections that are used to send the queues in the client. I suggest playing with the batch-size first, then the number of connections till you found your sweet spot for your throughput.
I am able to read the messages from activemq using camel context[xml], but i could like to read only no of the messages, for example if queue contains 10 000 messages, we want to read only first 1 000 messages, remaining shouldn't be touched.
I am new to the camel
It is not quite clear how you want your program to work. Do you want to stop the route after 1000 messages? Or your program? Or just finish them before processing the rest?
Anyway, the Jms component has a maxMessagesPerTask parameter that is the number of messages a task can receive after which it's terminated. That might do what you want.
"jms:queue:order?maxMessagesPerTask=1000"
What if there is only 500 messages in the queue, should you wait until you have received additional 500, so the total is 1000. And what if you restart your application etc.
It's a bit strange use-case. The Camel JMS component is designed to continuesly consume from the queue. If you want to stop then look at the Control Bus EIP where you can control Camel routes, and stop them. And also look at the RoutePolicy where you can control routes using that, for example look at the throttling route policy which can start/stop routes depending on load etc.
The CiA2 book also has coverage of managing and controlling Camel routes, you can look at in the management chapter.
http://camel.apache.org/controlbus.html
http://camel.apache.org/routepolicy.html
Iam trying to implement replay mechanisam with camel ie., i will have to retrieve all the messages already persisted and forward to appropriate camel route to reprocess.This will be triggred by quartz scheduler.
I achieved the same by using below.
1) once the quartz scheduler is triggered, fwd to processor which will query db and have the message as list and set the same in camel exchange properties as list.
2) Use the camel in which LoopProcessor will set appropriate xml in the iteration in the exchange.
3) fwd it to activemq which will be forwarded to appropriate camel route to reprocess.
Every thing is fine.
I see the following TWO issues
a) there might be 'n' number of msges(10,000+) which will be available in the camel exchange properties in the form of List - i can remove the message which is sent for processing but i think this will do more good on performance and memory usage.
b) I don want to forward all the 10,000+ messages to activemq which i guess will make it exhaustive. Is there a better mechanism to forward 10000+ messages to activemq.
-- I am thinking to use SEDA/VM(using different camel contexts).how good this can give me considering above questions.
Thanks.
Regards
Senthil Kumar Sekar
If the number of messages is a problem, then not all messages should be loaded at once.
Process as follows (see also my answer for your other SO):
Limit the number of results when querying the DB.
Set a marker (e.g. processedFlag) for the DB entries that are processed
Begin at 1. and query only the not already processed entries until all records are processed.
However, you should test the ActiveMQ approach as well, if 10,000+ messages are really a problem or not.