Is it possible to have a list of sources passed to an MPI_Recv (or equivalent) call? Currently, my code looks something like:
do i=nod1,nod2
call mpi_recv(tmp,n,MPI_REAL,MPI_ANY_SOURCE,tag,MPI_COMM_WORLD,status,ierr)
... do stuff with tmp here
call mpi_send(tmp,n,MPI_REAL,status(MPI_SOURCE),tag,MPI_COMM_WORLD,ierr)
enddo
Of course, this doesn't guarantee that it does what I want. (if nod1 sends two messages here before nod2 can send one message, then nod2's message is not recieved during this iteration, which would be bad.) In my application, that can't happen since nod1 and nod2 have other constraints which force them to be synchronized (enough) with each other ... but it got me wondering if there is way to specify a list of procs that are permitted to be recieved from.
Not as such. However, you can use MPI_Probe() with MPI_ANY_SOURCE, and then compare the MPI_SOURCE field of the status object against your list of processes you want to receive from. If there's a match, you can then proceed to receive from that source with a regular blocking receive.
Related
I am trying to understand the cardinality between Messages and Exchanges. In the following route, by the time one message makes it to the final log how many Exchanges are created?
from("timer:one-second-timer")
.bean(messageTransformer, "transformMessage")
.to("log:logging-end-point");
Since a Message can hold only one "in" message, I imagine there will be one Message for each end-point the message hops on. Is this true?
You can consider an Exchange as being the envelope containing the Message + some meta-data (the Properties), allowing this message to be transported between endpoints.
The javadoc says:
An Exchange is the message container holding the information during
the entire routing of a Message received by a Consumer.
So, in your example, if the timer is fired 10 times, this will result in 10 distinct exchanges of one message.
Can anyone to tell me one thing ?
It is about ServiceBroker.
https://gallery.technet.microsoft.com/scriptcenter/Using-Service-Broker-for-360c961a
Following this short tutorial we can come across two thnigs:
CREATE MESSAGE TYPE [//SBM/RequestMessage] VALIDATION=WELL_FORMED_XML;
CREATE MESSAGE TYPE [//SBM/ReplyMessage] VALIDATION=WELL_FORMED_XML;
What is aim of this stuff ? What benefit from separating messages to two types ? And second thing: Is it possible to check if queue was disabled by poison ? I know that then queue is blocked, but I would like to be able to check it.
One thing I use message types for is to know how I should process the message without actually having to look at the message itself. Let's say we're creating a system that will deal with conversations from disparate applications that service both the HR and Finance teams. You could argue that those belong in separate queues, but let's assume that all of those messages are going into one queue. In the procedure that I use to dequeue messages from the queue, I would have something like this (simplified):
declare #message_type_name sysname, #message_body xml);
RECEIVE TOP (1)
#message_type_name = [message_type_name],
#message_body = CAST([message_body] AS XML)
FROM [repl].[CoreQueue]
IF (#message_type_name = 'TimeOffRequest')
EXEC dbo.ProcessTimeOffRequest #message_body;
ELSE IF (#message_type_name = 'ReimbursementRequest')
EXEC dbo.ProcessReimbursementRequest #message_body;
Note I'm employing some anti-patterns here (RECEIVEing only one message at a time, no error handling, etc) in the interest of clarity around your actual question.
Because I know something about the content of the message based on the message type, I can quickly route the message to something that can handle it. Add to that the ability to bundle message types into contracts which can be prioritized with broker priorities and you have a fairly flexible system.
I notice in the documentation for the SEND statement that it allows for sending the same message on multiple conversation handles at once. Let's say that in my situation, the number of places I want to send a given message is small (fewer than 5), but every message I want to send should go to all of those places. Is there any practical difference between the following:
declare #ch1 uniqueidentifier,
#ch2 uniqueidentifier,
#ch3 uniqueidentifier,
#message xml;
-- approach #1
send on conversation (#ch1, #ch2, #ch3)
message type [foo]
(#message);
-- approach #2
send on conversation (#ch1)
message type [foo]
(#message);
send on conversation (#ch2)
message type [foo]
(#message);
send on conversation (#ch3)
message type [foo]
(#message);
SEND ON (#h1, #h2, #h3, ... , #hN) is going to store the message body only once in sys.transmission_queue. As opposed to SEND ON (#h1), SEND ON (#h2), ... , SEND ON (#hN) which will store the message body N times. This is, basically, the real difference. If the message body is of significant size it can have noticeable impact on perf.
For local delivery, when sys.transmission_queue is usually not involved, there will be no significant difference.
I depends on you needs, As you say you have only five conversion you can go both ways . No difference between them... But here is the catch
Do you relay want to check your message individually(whether it went or not) OR you want to rollback from certain case or person
Do you want to count or do something on the sending process
Your 1st approach is like a machine gun.In small case it wont create pressure or data lose in a server but in large case i can not give you the guaranty(i mean Like machine gun, Server do jam ).
Messages in the transmission queues for an instance are transmitted in sequence based on:
The priority level of their associated conversation endpoint.
Within priority level, their send sequence in the conversation.
What I want to do is exactly like this:
Solr: How to perform a batch request to an external system from a PostFilter?
and the approach I took is similar:
-don't call super.collect(docId) in the collect method of the PostFilter but store all docIds in an internal map
-call the external system in the finish() then call super.collect(docId) for all the docs that pass the external filtering
The problem I have: docId exceeds maxDoc "(docID must be >= 0 and < maxDoc=100000 (got docID=123456)"
I suspect I am storing local docIds and when Reader is changed, docBase is also changed so the global docId, which I believe is constructed in super.collect(docId) using the parameter docId and docBase, becomes incorrect. I've tried storing super.delegate.getLeafCollector(context) along with docId and call super.delegate.getLeafCollector(context).collect() instead of super.collect() but this doesn't work either (got a null pointer exception)
Look at the code for the CollapsingQParserPlugin in the Solr codebase, particularly CollapsingScoreCollector.finish.
The docId's you receive in the collect call are not globally unique. The Collapsing collector makes them unique by adding the docBase from the context to the local docId to create a globalDoc during the collect() phase.
Then in the finish() phase, you must find the context containing the doc in question and set the reader/leafDelegate depending on what version of Solr your running. Specifying the right docId with the wrong context will throw Exceptions. For the Collapsing collector, you iterate through the contexts until you find the first docBase smaller than the globalDoc.
Finally, if you added docBase in collect(), don't forget to subtract docBase in finish() when you call collect() on the appropriate DelegationCollector object, as the author may or may not have done the first time.
I effectively want a flush, or a completionSize but for all the aggregations in the aggregator. Like a global completionSize.
Basically I want to make sure that every message that comes in a batch is aggregated and then have all the aggregations in that aggregator complete at once when the last one has been read.
e.g. 1000 messages arrive (the length is not known beforehand)
aggregate on correlation id into bins
A 300
B 400
C 300 (size of the bins is not known before hand)
I want the aggregator not to complete until the 1000th exchange is aggregated
thereupon I want all of the aggregations in the aggregator to complete at once
The CompleteSize applies to each aggregation, and not the aggregator as a whole unfortunately. So if I set CompleteSize( 1000 ) it will just never finish, since each aggregation has to exceed 1000 before it is 'complete'
I could get around it by building up a single Map object, but this is kind of sidestepping the correlation in aggregator2, that I would prefer to use ideally
so yeah, either global-complete-size or flushing, is there a way to do this intelligently?
one option is to simply add some logic to keep a global counter and set the Exchange.AGGREGATION_COMPLETE_ALL_GROUPS header once its reached...
Available as of Camel 2.9...You can manually complete all current aggregated exchanges by sending in a message containing the header Exchange.AGGREGATION_COMPLETE_ALL_GROUPS set to true. The message is considered a signal message only, the message headers/contents will not be processed otherwise.
I suggest to take a look at the Camel aggregator eip doc http://camel.apache.org/aggregator2, and read about the different completion conditions. And as well that special message Ben refers to you can send to signal to complete all in-flight aggregates.
If you consume from a batch consumer http://camel.apache.org/batch-consumer.html then you can use a special completion that complets when the batch is done. For example if you pickup files or rows from a JPA database table etc. Then when all messages from the batch consumer has been processed then the aggregator can signal completion for all these aggregated messages, using the completionFromBatchConsumer option.
Also if you have a copy of the Camel in Action book, then read chapter 8, section 8.2, as its all about the aggregate EIP covered in much more details.
Using Exchange.AGGREGATION_COMPLETE_ALL_GROUPS_INCLUSIVE worked for me:
from(endpoint)
.unmarshal(csvFormat)
.split(body())
.bean(CsvProcessor())
.choice()
// If all messages are processed,
// flush the aggregation
.`when`(simple("\${property.CamelSplitComplete}"))
.setHeader(Exchange.AGGREGATION_COMPLETE_ALL_GROUPS_INCLUSIVE, constant(true))
.end()
.aggregate(simple("\${body.trackingKey}"),
AggregationStrategies.bean(OrderAggregationStrategy()))
.completionTimeout(10000)