Google pubsub allows to add custom attributes to messages before publishing them, e.g.:
{
data: {
something: "value"
},
attributes: {
attribute1: "value"
}
};
When receiving a message in a subscription, you must ack() or nack() the message to allow the broker knowing if the message was acknowledged or if it must be removed from the inventory and scheduled to be redelivered.
I would like to update the custom attributes in the message before nack() it, adding a property called ErrorCode - so, after X retries, the message should be automatically published to the DLQ topic with the ErrorCode in its attributes.
However, when I update the message attributes and nack() it, the next time the subscriber receives the same message the attributes don't get updated:
message.attributes = {ErrorCode: "123"}
message.nack();
In this example, the value of attributes in the message keeps having only the property attribute1 in the next time the subscriber receives the message.
Does someone know how the custom attributes can be updated for the current message? If it is not possible (it seems that it is the case), what would be the thoughts about a solution for this?
Attributes in a message are immutable once the message has been published. If you are changing the attributes, you are only changing them in the local copy. If you want to change the attributes in a message when it gets sent to a dead letter topic, you'd instead have to do the process manually, publishing to this other topic yourself instead of depending on Pub/Sub automatic dead letter capabilities.
Related
In ActiveMQ (and Hawtio) there is Admin UI functionality to move a messages from one queue to another. This process keeps the body and all the headers all the same as the original message.
Is there a way to do this in Camel? A simple from(queue1).to(queue2) always seems to change headers.
thanks
edit: tried
from(
"cMQConnectionFactory1:queue:queue1"
+ "?forceSendOriginalMessage=" + true
+ "&mapJmsMessage=" + false)
.routeId("testMove_cJMS_1")
.to("cMQConnectionFactory1:queue:queue2"
+ "?forceSendOriginalMessage=" + true
+ "&mapJmsMessage=" + false).id("testMove_cJMS_2");
}`
The Camel documentation for the JMS component describes the forceSendOriginalMessage option saying:
When using mapJmsMessage=false Camel will create a new JMS message to send to a new JMS destination if you touch the headers (get or set) during the route. Set this option to true to force Camel to send the original JMS message that was received.
forceSendOriginalMessage defaults to false.
The docs say this about mapJmsMessage:
Specifies whether Camel should auto map the received JMS message to a suited payload type, such as javax.jms.TextMessage to a String etc.
mapJmsMessage defaults to true.
Therefore it seems like you need to set mapJmsMessage=false & forceSendOriginalMessage=true.
Keep in mind that Camel will be using the JMS API to consume the message and then resend it. Even though the new message will have the same body and headers as the old message it will be slightly different because the JMS specification dictates that when a message is sent the broker must assign it a message ID and timestamp. Therefore the JMSMessageID and JMSTimestamp on the new message will be different from the old message, and there's really no way around that. If you need to identify the message uniquely you should set the correlation ID on the original message and use that to identify the message rather than the JMSMessageID. Also, if you need to preserve the original time the message was sent then set that in a custom property.
The reason the JMSMessageID and JMSTimestamp are not changed when moving the messages via the management console is because the broker moves the messages internally with a completely different mechanism than JMS.
I am currently designing a message broker in CZMQ (High-level C binding for ZeroMQ). My broker design is somewhat like the typical star-topology but with additional REP-sockets to communicate with the publishers and the subscribers. I also have a PUB-socket to notify publishers about subscribers.
When a node connects to the broker it has to do a handshake through this REP-socket and that node then gets added to an internal list in the server. My node structure looks like this:
struct node {
int type;
char *uuid;
char *path;
size_t path_len;
int64_t timestamp;
};
The path variable is the subscription path that the publisher/subscriber is using, e.g. /path/to/subscription.
I keep two different internal lists, one for publishers and one for subscribers. I want to design it so that when a publisher has no subscriber the server will notify that publisher so it can stop sending out messages until another subscriber that subscribes to that publisher.
The problem I am having is that I don't know how to figure out if a publisher has no subscribers, for example lets say that a publisher publishes to /path/to/data, I then need to find if there are any subscriber that are subscribed to /path, /path/to or /path/to/data.
Each node in this network has a unique ID, the uuid. Each publisher has a SUB-socket that subscribes to it's own uuid so that it can receive updates from the server.
Several options:
Alt.1)
Create your own explicit subscription-management layer, independent of the standard PUB/SUB Scalable Formal Communication Behaviour Archetype Pattern. There you can achieve whatever functionality your design wants to have.
Alt.2)
Use the standard XPUB/XSUB Archetype, where both subscription and unsubscription messages are enclosed. Yet, these are weak-type signalling and one may simply miss or never 've been intended to receive a such message, so your handling strategy must become robust enough not to get into troubles, if weak-signalling messages are not presented.
ZMQ_XPUB
Same as ZMQ_PUB except that you can receive subscriptions from the peers in form of incoming messages. Subscription message is a byte 1 (for subscriptions) or byte 0 (for unsubscriptions) followed by the subscription body. Messages without a sub/unsub prefix are also received, but have no effect on subscription status.
ZMQ_XSUB
Same as ZMQ_SUB except that you subscribe by sending subscription messages to the socket. Subscription message is a byte 1 (for subscriptions) or byte 0 (for unsubscriptions) followed by the subscription body. Messages without a sub/unsub prefix may also be sent, but have no effect on subscription status.
Last but not least, one need not worry about no SUB-s being present, as the Context()-instances' internal data-pumps' management is well-aware about connections' state and not having troubles on the empty list of SUB-s. If using a .setsockopt( ZMQ_CONFLATE, 1 ) method, the locally allocated resources for the SUB-s' Queue storage-capacity management will keep just the one, most recent message in the local Queue-head-end storage for each almost"live"-SUB.
Thats cute, isn't it?
I'm consuming messages from SQS FIFO queue with maxMessagesPerPoll=5 set.
Currently I'm processing each message individually which is a total waste of resources.
In my case, as we are using FIFO queue and all of those 5 messages are related to the same object, I could process them all toghether.
I though this might be done by using aggregate pattern but I wasn't able to get any results.
My consumer route looks like this:
from("aws-sqs://my-queue?maxMessagesPerPoll=5&messageGroupIdStrategy=usePropertyValue")
.process(exchange -> {
// process the message
})
I believe it should be possible to do something like this
from("aws-sqs://my-queue?maxMessagesPerPoll=5&messageGroupIdStrategy=usePropertyValue")
.aggregate(const(true), new GroupedExchangeAggregationStrategy())
.completionFromBatchConsumer()
.process(exchange -> {
// process ALL messages together as I now have a list of all exchanges
})
but the processor is never invoked.
Second thing:
If I'm able to make this work, when does ACK is sent to SQS? When each individual message is processed or when the aggregate process finishes? I hope the latter
When the processor is not called, the aggregator probably still waits for new messages to aggregate.
You could try to use completionSize(5) instead of completionFromBatchConsumer() for a test. If this works, the batch completion definition is the problem.
For the ACK against the broker: unfortunately no. I think the message is commited when it arrives at the aggregator.
The Camel aggregator component is a "stateful" component and therefore it must end the current transaction.
For this reason you can equip such components with persistent repositories to avoid data loss when the process is killed. In such a scenario the already aggregated messages would obviously be lost if you don't have a persistent repository attached.
The problem lies in GroupedExchangeAggregationStrategy
When I use this strategy, the output is an "array" of all exchanges. This means that the exchange that comes to the completion predicate no longer has the initial properties. Instead it has CamelGroupedExchange and CamelAggregatedSize which makes no use for the completionFromBatchConsumer()
As I don't actually need all exchanges being aggregated, it's enough to use GroupedBodyAggregationStrategy. Then exchange properties will remain as in the original exchange and just the body will contain an "array"
Another solution would be to use completionSize(Predicate predicate) and use a custom predicate that extracts necessary value from groupped exchanges.
I have a Symfony2/AngularJS application and using FOSRestBundle with JMS Serializer Bundle.
For normal entities every thing works great but in one of my entities that contains a collection of messages (topic entity) I need to return subset of messages in different situations.
For example for the Topic Entity Owner I want to return all messages of the topic for message owner I want to return just the message that posts with the message owner and for other users I don't want to return any messages but they can post a message on topic.
I'm not sure where to implement this. In the topic entity or in the controller or ...
The short answer is: In the controller.
Long version:
That you have a Topic entity that holds a collection of messages does not mean that you are not allowed to use just the message entity. Thus in case you don't need a whole topic then you should not use it also. Instead use an (array of the) message entity directly in your controller. You can use the findBy() or findOneBy() function to retrieve the message(s) you need.
I've set up a topic exchange such that the consumer queue is bound with "#.topic". I'd like to use different acknowledgement strategies based on the prefix. Is the full routing key sent to the consumer? If so, how do I access it? An answer in terms of AMQP concepts would probably be sufficient, but an answer involving rabbitmq-c would be ideal.
Even when you do a binding like you have given in your example the message received contains the full routing key. This means you can extract that in order to help you process the message. Unfortunately I only know how to do this in Java so try to extrapolate from there.
QueueingConsumer.Delivery delivery = consumer.nextDelivery();
String routingKey = delivery.getEnvelope().getRoutingKey();
The delivery object contains a body which is the payload and can be retrieve with delivery.getBody() and an Envelope object which contains other information like the full routing key.