I’m stuck at one particular requirement in camel , I’m processing a set of files and when any exception occurs i send a message to an exception queue , and from the exception queue im sending an email about the failure , the functionality is working fine but I end up sending many mails, like if 10 files fail im sending 10 emails, is there a way to send only one mail , like I would want to wait for the entire route to finish , then go look the exception queue and send a single mail stating what has failed (by processing the exception queue )
I'm open for suggestions.
I had to do this scenario once (inversed though - mail on success). I had a handy MySQL database configured and ready, so I just added each event from the queue to the database. Then once every now and then, extracted all info (and deleted it)- simply select * from events; delete from events; from the database and created a mail.
You could process the error queue with the aggregator pattern, it is very nice for these tasks. http://camel.apache.org/aggregate-example.html . You still need to know WHEN the aggregator should fire off a message. If you can, trigger a "finish, send mail" event such as in the example in the link above.
The most simple way would be to time schedule these mail notifications. Take a look at: http://camel.apache.org/simplescheduledroutepolicy.html . You can set it to run your route for some good choice of time, then when it fires off, you set the aggreator to complete upon timeout, and make the timeout good enough to empty any reasonable queue size of errors, but not too large.
At least that's my suggestions to your issue
Related
I have about 10K messages in a CSV file. Each message has an associated timestamp for it. When the time is reached, I want the message delivered to an MQ. The timestamps are not uniformly spread. Is this possible with Apache Camel?
As far as I know Apache Camel by default has no consumer endpoint components that you could configure to trigger with specific messages at specified times.
There is however a timer component that you can setup to trigger for example once a second. Then in the route you could use a processor to check if a list contains any messages that should be send at the given time and send them to the MQ.
You could also trigger route from java code using a ProducerTemplate that you can create using the CamelContext.
The list could be populated using your csv file and ordered by timestamp so you could use it like a stack and only check the first few entries instead of going through all 10K every second.
The real problem here would be persistence i.e to figure out which of the messages listed on the csv have already been sent if the application closes before all 10K messages have been sent.
A message is being dropped to Service bus queue with ScheduledEnqueueTimeUTC and Service Bus Connector in Logic app has trigger set to pick messages from queue at 12:05AM EST EveryDay.
Problem: Logic app has picked the same message twice one with Service bus message properties State='Scheduled' and other with state='Active' with same sequenceNumber. May i know when this happens and how can this be solved.
Problem: Logic app has picked the same message twice one with Service
bus message properties State='Scheduled' and other with state='Active'
with same sequenceNumber. May i know when this happens and how can
this be solved.
Here we discovered one of the workarounds that will meet your needs. To pick and send a message only once, we must set our settings to split on as seen below.
NOTE: I tried using Logic app standard, as this option is not available in Consumption plan
Please refer this MS DOC & SO THREAD for more information .
A single process in my ASP.NET Website will be fired from different locations at ~ same time.
Trying to use SQL Server Broker to Queue the requests and execute one by one.
Enabled SQL Broker and created a Queue and Service in the database. On receiving a request; I begin a dialog conversation using only the 1 Service and log the conversation token in a table.
Have written an activated procedure to access the data passed into Queue and Initiate processing.
End conversation inside activated procedure after Processing is complete.
I have doubts regarding the pattern I am following. The conversation does not get closed correctly. In some examples; I have seen an Initiator and Target Queue pattern where conversation is closed at both endpoints. Please help me to figure out the pattern needed in this case.
UPDATE
Sorry for not updating; got busy with some other work. I changed to using 2 each of Queues(Initiator and Target), Services and the corresponding Activated procs.
The connections get closed properly now. When we insert into the Target queue while 1st request is being processed; do we need to specify any setting or command to ensure that 1st request gets completed before starting the 2nd one?
Today, I tried to simulate a scenario where in the camel "to" tag I supplied a mis-spelt queue name(which was not there) Camel or RabbitMq instead of throwing an exception back continued to finish the route flow.
Intrgigued I did write a sample program to send a message using "channel.basicPublish" with a wrong queue name. I never got any exception thrown back from rabbit mq client.
however if the exchange name was wrong, I did get an exception back. Is this expected behaviour?
I tried adding return listener, confirm listener,exception handler, etc., but none of them got invoked.
Any clues?
Messages are published to exchanges, so the exchange must be there when publishing messages. At publish time RabbitMQ doesn't care about queues, unless the mandatory flag is provided, or the channel is in confirm mode.
App A wants to send domain events to App B through a middleware like RabbitMQ.
Let's take the example of one domain event called UserHasBeenRegistered, involving by the creation of the User entity.
A would inform B that this latter should send a welcome email, by sending this event.
I have in mind two workflows:
First:
- App A registers the user and the event is generated.
- App A sends the event directly to B through a queue provided by RabbitMQ
Second:
- App A registers the user and the event is generated.
- App A saves the event in some kind of event store as a database table (if relational) in the same local transaction used for persisting in database this new user.
- An asynchronous scheduler queries the event store, find this new user registration and sends the message through the RabbitMQ's queue.
Do you see the difference?
Yes, one is longer than the other... but the second is far more safe! albeit less performant.
Indeed, what while in the first case, the registration is rollback due to an exception thrown just after the publishing was made? => the mail would be sent whereas the user was not persisted.
This could be fixed by implementing a global XA transaction (two-phases commit), but it is well known that some middleware don't support it.
Therefore, is the second workflow mostly used in critical application?
What are its drawbacks?
I plan to implement one of both solutions for my project.
I had the same task and it was done as a mix of your two workflows:
App A registers the user and the event is generated.
App A sends the event which has ttl set to non-zero value directly to B through a queue provided by RabbitMQ.
App B receive event and send welcome message to user and store flag that welcome message sent.
There are background script which check whether there are newly registered users from last ttl + 1 time interval who doesn't receive messages.
You can remove background script and flag storing and stick with first workflow from you q. The cases when messages lost or any other cases are damn rare (with welcome messages sending it might be 1 failure per 1billion users) and unnecessary application complication may give you more errors.
The second workflow looks also stable, but why you are using RabbitMQ then?