Suppose we have two applications A and B with their own different databases A1 (MySQL) and B1 (Postgres) correspondingly. We create entities X and Y associated with each other in application A. Y belongs to X. On every insert to database A1 (after commit) we publish message to RabbitMQ to make application B aware of brand new entities. One event per entity – X1 and Y1. Everything is good, if RabbitMQ keeps the order of the messages, so workers in application B may process X1 first and Y1 second to establish right association between new A and B records in database B1. But as far as I understand RabbitMQ is not intended to keep messages order and does it in very specific circumstances, like publish within one channel, send to one exchange, push to one queue, consume within one channel.
So my question is about the correct direction and general approach:
Should I choose another message queue, that guarantees messages order?
Have I missed something specific in RabbitMQ messages order specifics?
Should I implement some kind of retry mechanism in application B side, that will re-enqueue messages back to RabbitMQ in case if message order was not as expected?
Maybe it will give more sensible context – B1 is a data warehouse, that aggregates data not only from A1, but also other databases.
Everything is good, if RabbitMQ keeps the order of the messages
Is that the only solution?
Can't X and Y instances get a sequence number assigned, which will then be used to rebuild the right sequence within B?
to establish right association between new A and B records
Can't X1, Y1 express an explicit relation allowing a creation of A, B without relying on sequence?
The point is:
Message ordering is always expensive. You either have hard constraints (e.g. one consumer) or you have less availability and speed. Your best bet is finding a way to not rely on order.
Related
Say we have web-service A, B, C and D and all of them are querying to access/modify a record in the same table.
What we want to achieve is, if service A being called first it should acquire a lock on the record; in order to prevent others (B, C and D in this case) to access/modify if they tried querying the previously locked record.
I came across this well articulated answer that gave me some pointers below;
Should we consider locking at database level we have specific processes (remember A, B, ..) in question that need to be identified by the application and database layer as the ones capable of performing lock/release.
Note: Its a deprive all, until the first got filled situation.
There is a question for doing this Pessimistically or Optimistically, but since in our case the logic in these processes open up to the place which we actually want to keep inaccessible: going for pessimistic locking seems a good choice. But how to achieve that I mean it should expand beyond certain transactions started from specified processes only.
We are open to suggestions to do this without the database lock as well. Thanks a bunch!
I'm confused about using Paxos algorithm.
Seems that Paxos can be used to such scenario: multiple server (a cluster, I assume each server has all 3 roles, proposer, acceptor, leaner) need to keep the same command sequences to achieve consistence and backup. I assume there are some clients sending commands to this server (clients may send in parallel). Each time the command is dispatched to multiple server by one Paxos instance.
Different clients can send different commands to different proposers, Right?
If so, one command from some client will raise a Paxos instance. So,
Multiple Paxos instance may run at the same time?
If so, client-A sends "A += 1" command to proposer-A, and client-B sends "B += 2" command to proposer-B at nearly the same time, I suppose to see each server has received 2 commands, "A += 1" and "B += 2".
However,
Given 5 servers, say S1-S5, S1 send command "A += 1" and S5 send command "B += 1", S2 promise S1 however S3, S4 promise S5, so finally S3,S4,S5 got "B += 1" but S1,S2 got nothing because the number of promise is not majority. Seems like the Paxos does not help at all. We don't get the expected "A += 1" and "B += 2" at all 5 servers?
So I guess in practical application of Paxos, no parallel Paxos instances are allowed? If so, how to avoid parallel Paxos instances, seems that we still need a centralized server to flag whether there is a Paxos running or not if we allowed multiple clients and multiple proposers.
Also, I have questions about the proposer number. I search the internet and some claims the following
is a solution:
5 servers, given corresponding index k(0-4), each server uses number 5*i + k for this server's "i"th proposal.
For me, this seems not meet the requirements at all, because server-1's first proposal number is always 1 and server-4's first proposal number is always 4, but server-4 may raise the proposal earlier than server-1, however it's proposal number is bigger.
So I guess in practical application of Paxos, no parallel Paxos
instances are allowed? If so, how to avoid parallel Paxos instances,
seems that we still need a centralized server to flag whether there is
a Paxos running or not if we allowed multiple clients and multiple
proposers.
You don't need a centralised service only need nodes to redirect clients to the current leader. If they don't know who the leader is they should throw an error and the client should select another node from DNS/config until they find or are told the current leader. Clients only need a reasonably up to date list of which nodes are in the cluster so that they can contact a majority of current nodes then they will find the leader when it becomes stable.
If nodes get separated during a network partition you may get lucky and its a clean and stable partition which only leads to one majority and it will get a new stable leader. If you get an unstable partition or some dropped directional packets such that two or nodes start to be "duelling leaders" then you can use randomised timeouts and adaptive back-off to minimise two nodes attempting to get elected at the same time leading to failed rounds and wasted messages. Clients will get wasted redirects, errors or timeouts and will be scanning for the leader during a duel until it is resolved to a stable leader.
Effectively paxos goes for CP out of CAP so it can loose the A (availability) due to duelling leaders or no majority being able to communicate. Perhaps if this really was high risk in practice people would be writing about having nodes blacklist any node which repeatedly tries to lead but which never gets around to committing due to persistent unstable network partition issues. Pragmatically one can imagine that folks monitoring the system will get alerts and kill such a node and fix the network before trying to add complex "works unattended no matter what" features into their protocol.
With respect to the number for proposals a simple scheme is a 64bit long with a 32bit counter packed into the highest bits and the 32bit IP address of the node packed into the lowest bits. That makes all numbers unique and interleaved and only assumes you don't run multiple nodes on the same machine.
If you take a look at Multi-Paxos on Wikipedia it's about optimising the flow for a stable leader. Failovers should be really rare so once you get a leader it can gallop with accept messages only and skip proposals. When you are galloping if you are bit packing the leader identity into the event numbers a node can see that subsequent accepts are from the same leader and are sequential for that leader. The galloping with a stable leader is a good optimisation and creating a new leader with proposals is an expensive thing requiring proposal messages and the risk of duelling leaders so should be avoid unless it is cluster startup or failover.
however it's proposal number is bigger.
That's exactly the point of partitioning the proposal space. The rule is, that only the most recent proposal, the one with the highest number seen, shall be accepted. Thus, if three proposals were sent out, only the one proposal with the largest number will ever get an accepted majority.
If you do not do that, chances are that multiple parties continue to spit out proposals with simply incremented numbers (nextNumber = ++highestNumberSeen) like crazy and never come to a consensus.
We are trying to build a system with multiple instances of a service on different machines that share the load of processing.
Each of these will check a table, if there are rows to be processed on that table, it will pick the first, mark it processing, then process it, then mark it done. Rinse repeat.
What is the best way to prevent a racing condition where 2 instances A and B do the following
A (1) read the table, finds row 1 to process,
B (1) reads the table, finds row 1 to process,
A (2) marks it row processing
B (2) Marks it row processing
In a single app we could use locks or mutexs.
I can just put A1 and A2 in a single transaction, is it that simple, or is there a better, faster way to do this?
Should I just turn it on it's head so that the steps are:
A (1) Mark the next row as mine to process
A (2) Return it to me for processing.
I figure this has to have been solved many times before, so I'm looking for the "standard" solutions, and if there are more than one, benefits and disadvantages.
Transactions are a nice simple answer, with two possible drawbacks:
1) You might want to check with the fine print of your database. Sometimes the default consistency settings don't guarantee absolute consistency in every possible circumstance.
2) Sometimes the pattern of accesses associated with using a database to queue and distribute work is hard on a database that isn't expecting it.
One possibility is to look at reliable message queuing systems, which are seem to pretty good match to what you are looking for - worker machines could just read work from a shared queue. Possible jumping-off points are http://en.wikipedia.org/wiki/Message_queue and http://docs.oracle.com/cd/B10500_01/appdev.920/a96587/qintro.htm
I want to make a simple GAE app in Go that will let users vote and store their answers in two ways. First way will be raw data (Database store of "voted for X"), the second will be a running count of those votes ("12 votes for X, 10 votes for Y"). What is an effective way to store both of those values with the app being accessed by multiple people at the same time? If I retrieve the data from the Datastore, change it, and save it back for one instance, another might be wanting to do the same in parallel, and I`m not sure if the final result will be correct.
It seems like a good way to do that is to simply store all vote events as separate entities (the "voted for X" way) and use the Task Queue for the recalculation (the "12 votes for X, 10 votes for Y" way), so the recalculation is done offline and sequentially (without any races and other concurrency issues). Then you'd have to put the recalc task every once in a while to the queue so the results are updated.
The Task Queue doesn't allow adding another task with the same name as an existing one, but doesn't allow checking whether a specific task is already enqueued, so maybe simply trying adding a task with a same name to the queue will be enough to be sure that multiple recalc tasks are not there.
Another way would be to use a goroutine waiting for a poke from an input channel in order to recalculate the results. I haven't run such goroutines on App Engine so I'm not sure of the general behavior of this approach.
I’m building a system that generates “work items” that are queued up for back-end processing. I recently completed a system that had the same requirements and came up with an architecture that I don’t feel is optimal and was hoping for some advice for this new system.
Work items are queued up centrally and need to be processed in an essentially FIFO order. If this were the only requirement, then I would probably favor an MSMQ or SQL Server Service Broker solution. However, in reality, I need to select work items in a modified FIFO order. A work item has several attributes, and they need to be assigned in FIFO order where certain combinations of attribute values exist.
As an example, a work item may have the following attributes: Office, Priority, Group Number and Sequence Number (within group). When multiple items are queued for the same Group Number, they are guaranteed to be queued in Sequence Number order and will have the same priority.
There are several back-end processes (currently implemented as Windows Services) that pull work times in modified FIFO order given certain configuration parameters for the given service. The service running Washington, DC is configured to process only work items for DC, while the service in NY may be configured to process both NY and DC items (mainly to increase overall throughput). In addition to this type of selectivity, higher priority items should be processed first, and items that contain the same “Group Number” must be processed in Sequence Number order. So if the NY service is working on a DC item in group 100 with sequence 1, I don’t want the DC service to pull off DC item in group 100 sequence 2 because sequence 1 is not yet complete. Items in other groups should remain eligible for processing.
In the last system, I implemented the queues with SQL tables. I created stored procedures to submit items and, more importantly, to “assign” items to the Windows Services that were responsible for processing them. The assignment stored procedures contain the selection logic I described above. Each Windows Service would call the assignment stored procedure, passing it the parameters that were unique to that instance of the service (e.g. the eligible offices). This assignment stored procedure stamps the work item as assigned (in process) and when the work is complete, a final stored procedure is called to remove the item from the “queue” (table).
This solution does have some advantages in that I can quickly examine the state of these “queues” by a simple SQL select statement. I’m also able to manipulate the queues easily (e.g. I can bump priorities with a simple SQL update statement). However, on the downside, I occasionally have to deal with deadlocks on these queue tables and have the burden of writing these stored procedures (which gets tedious after a while).
Somehow I think that either MSMQ (with or without WCS) or Service Broker should be able to provide a more elegant solution. Rolling my own queuing/work-item-processing system just feels wrong. But as far as I know, these technologies don’t offer the flexibility that I need in the assignment process. I am hoping that I am wrong. Any advice would be welcome.
It seems to me that your concept of an atomic unit of work is a Group. So I would suggest that you only queue up a message that identified a Group Id, and then your worker will have to go to a table that maps Group Id to 1 or more Work Items.
You can handle your other problems by using more than one queue - NY-High, NY-Low, DC-High, DC-Low, etc.
In all honesty, though, I think you are better served to fix your deadlock issues in your current architecture. You should be reading the TOP 1 message from your queue table with Update Lock and Read Past hints, ordered by your priority logic and whatever filter criteria you want (Office/Location). Then you process your 1 message, change it's status or move it to another table. You should be able to call that stored procedure in parallel without a deadlock issue.
Queues are for FIFO order, not random access order. Even though you are saying that you want FIFO order, you want FIFO order with respect to a random set of variables, which is essentially random order. If you want to use queues, you need to be able to determine order before the message goes in the queue, not after it goes in.