Can someone explain conversation groups in service broker?
Currently, I'm using service broker to send messages from one SQL server to another. On the sending server, I'm trying to correlate the messages so they are processed in serial on the receiving side. Based on the documentation, conversation groups seem to be a perfect fit for this, but on the receiving server, the messages get assigned to a different conversation group from the one I specified when sending the message.
I've search around the web and saw that this behavior seems to be intended (http://social.msdn.microsoft.com/forums/en-US/sqlservicebroker/thread/baf48074-6804-43ab-844a-cb28a6dce02b/), but then I'm confused about the usefulness of the syntax from (http://msdn.microsoft.com/en-us/library/ms178624.aspx)
WAITFOR(
GET CONVERSATION GROUP #conversation_group_id FROM [dbo].[ReceiveQueue]
)
If the conversation group doesn't come across with the message from the sender and messages sent with the same conversation group id don't have the same conversation group id on the receive side, what's the point of the code above?
Conversation groups are a local primitve used for locking. Messages within a conversation group have no order guarantees, and conversation groups do not flow across the wire.
The message order is guaranteed by Service Broker within a conversation. So to preserve the order of corrleated messages in processing, send them on the same conversation.
Conversation groups are needed for groupping a set of conversations that are related to each other. Both GET CONVERSATION GROUP and RECEIVE verbs guarantee that they will lock an entire converstaion group, thus preventing any other thread from processing related messages. For example consider a traveling site. It receives a message with a request to book a holiday package. As a result it initiates a conversation with a hotel booking service and sends a request to reserve a room, it initiates a conversation with an airline reservation service and asks for travel reservation, it initiates a conversation with a car rental agency service and asks for a car reservation. These three new conversation it created are all in the same group with the initial conversation that the request was received on (the application has used the WITH RELATED_CONVERSATION clause of BEGIN DIALOG on all 3 of them). It then commits and proceed to process the messages in the queue. Later responses from these 3 correlated requests start comming in, at pretty much random times. Say the hotel resposnse comes in first. The message gets picked up by the applicaiton and it goes ahead to update the status of the request with the response from the hotel. At the same time, the airline response comes in. If another thread would be allowed to pick it up, it would try to update the status of the same request, thus resulting in blocking or even deadlock against the thread that is processing the hotel response. When the hotel response is processed, the thread commits and thus unlocks the whole conversation group, allowing for any thread (including itself) to pick up the airline response and process it.
Related
As an example user performs some update action and three actions need to be performed together:
update database, send email and perform network request to other service. Sending email and performing network request are non-transactional and if one fails and other succeeds we end up in the incorrect state. Are there common patterns for dealing with this?
A common pattern that works for me is to create an Email table as a queue of emails, inserts to this table can be easily included in transactions.
There is another process reading this Email table and sending requests to external processes (smtp server, sendgrid,...) and marking the email as "sent" if we get a success response from external processes. In case there is a failure, the email is not marked as "sent" and could be retried. There could be a situation that an email was sent successfully but there was an error in the network causing a failure response, retrying could send duplicate emails, but that would be rare and should not cause issues.
You can also have a look at database email which is a similar pattern: https://learn.microsoft.com/en-us/sql/relational-databases/database-mail/database-mail?view=sql-server-ver15
We can extend this pattern further with sending network requests with the same idea. You can have a look at this https://microservices.io/patterns/data/transactional-outbox.html
I have a scenario and looking for a good database design. My application provide 3 different roles. Admin, Customer, and Narrator.
Customer choose a Narrator and request a service. His request process by Admin to check whether it is valid or not. If it is not valid, Admin will send it back to the Customer with a message. Otherwise, Admin will send the request for the chosen Narrator. The Narrator perform the request and send it back to the Admin. Admin will check it again against some circumstances. If validation fail, Admin send the request back to the Narrator with a message. Otherwise, Admin will mark that request as finished and send it to the Customer.
Here is my basic tables:
User
username
password
email
Narrator
image
rank
user_id (FK)
Customer
phone_number
address
user_id (FK)
Request
customer_id (FK)
narrator_id (FK)
...
Actually my problem is how I should design my tables in order to support such a transaction.
I appreciate your suggestions.
Your text explaining what kind of service you want to deliver is a good lead about what type of models you need. You write about 'admins' and 'messages' but those models are omitted. I also think that 'request' is a property of a Service.
Maybe go for more general terms? Client, Supplier, Message, Service, Manager, Product. The Service needs a state field: new, assigned, needs more X...
Then your story goes as follows:
A Client chooses a Supplier and sends a Message to request a Service. A Service request is created with the status 'new'. The request will be reviewed by a Manager: He sets the state of the Service to approved and with that one click also send notification Messages to both Client and Supplier. The Supplier will preform the Service and on completion change the state... et cetera.
I wonder where Invoice comes in and maybe you need to deliver a Product in the end.
What is Synchronizing client with gmail ? Can anybody give a detailed explanation, because i want to have a better understanding over this concept.
For example, if your client keeps any local cache of the Gmail mailbox data like the Message.Id and labels, or headers, or the entire email. Then in order to update your client you're synchronizing it with Gmail--pulling new updates down to your client. In cases of clients designed for offline use, then synchronizing may also mean pushing local updates back up to the server (e.g. label updates made by client while "offline" that get applied at some later point). That's the general definition of synchronizing.
For the Gmail API specific case, Gmail has a backend mailbox-wide history Id. Any change that affects that account in any way gets a history identifier and most (but not all) history changes affect the state of email messages. Like adding a new message, changing the labels on a message, or deleting a message. Clients of the Gmail API can poll the history Id and find out what's changed since the last time they synchronized and pull down updates to maintain their sync.
What is the best practice to handle transaction between application and paypal.
Consider:
I'm Alice and I want send money to Bob
In my DB I see that Bob has $200 and I want to send him $150.
Once transaction is sent I want to update the Bob's account such that it would contain $50.
Now according to PayPal API I can send Pay and receive success. However what happens
if I for example send Pay it succeeds but I fail to receive a response due to network problem. So I assume that error happened and try again and technically I'll send $300 to Bob instead of $150?
How can I handle such a transaction - between a local database that keeps an account and the remote PayPal API?
I had this exact concern recently with an ASP.NET MVC project I was completing for a client.
I learned two things:
Communication between Paypal and your database cannot be trusted (well, didn't really learn this, but it was entirely reinforced)
I now understand why so many websites that have Paypal as a transaction type mention there could be a processing period between the time that the transaction was completed and shipping/delivery of the product is completed.
The way you handle the situation is similar to the way a business could handle personal checks:
A personal check looks like currency (and typically is), but many businesses would like some sort of verification from the bank that funds are available before they accept payment - so they use a machine that asks the bank if funds are actually available.
If the machine says the funds are available, the business trusts it and you complete the transaction. However, the machine can give an error message that typically means "the funds are not available or something went wrong" and the business has a decision to make:
We can trust the customer and accept the check, deliver the product, and hope for the best when later depositing the check to the bank.
Or we can tell the customer that it will take time for the check to clear, deposit the check, wait for the funds to actually arrive in our account, and (if successful) deliver the product after the business receives funding.
This sounds inefficient with the way many businesses operate today, but it is something that does come up. In fact, this is why a lot of businesses stray away from accepting personal checks, they are unreliable when compared to other methods of payment.
Now how does this correlate to handling a Paypal payment?
A Paypal payment looks like currency (and it typically is), but many businesses would like some sort of verification from the Paypal that funds are available before they accept payment - so they use Paypal PDT, IPN, or other method for checking that the transaction was handled appropriately.
If Paypal properly responds to one of the verification requests, the business can trust it and complete the transaction. However, your website may throw an error of some sort (i.e. Paypal could reply with an IPN response of NOTVALID, or you could never get a reply from Paypal). The business has a decision to make:
The business can trust the customer and accept that they have made a Paypal payment and everything should be alright (very bad decision in the case of a Paypal transaction)
Or the business can tell the customer at check-out time that there may be a 72 hour processing period for Paypal payments.
This may not sound like the best way to operate your business, but it is the way we have to deal with an imperfect internet.
I would set up the Paypal payment flow similar to this:
UserA wants to send $100 to another UserB using Paypal
UserA enters the value in the 'checkout field' and is sent over to Paypal to verify the transaction.
UserA is sent back to your website from Paypal and your website performs the IPN check with the details that Paypal has POSTed to your site(I chose IPN in this case - as if we were using Express Checkout as opposed to some other payment gateway that Paypal offers).
If the IPN is VALID, process the transaction as expected.
If the IPN is not VALID, mention to the customer that there may be a delay in processing, have your application send you a notification that a possible Paypal transaction issue has occurred (you may want to include a reference id so that you can quickly find which transaction this notification is referencing), and mark the transaction as pending as opposed to complete or something similar.
An admin of the site who handles these notifications will manually investigate the transaction (or force the website to check with Paypal again - see the Paypal API documentation for details on this) and manually mark the transaction as complete or failed.
Notify those involved of the status of the transaction.
It is annoying that we have to have extra steps involved to make sure the money was transferred, but, as mentioned earlier, we are using an imperfect system and we want to be very certain of the success / failure of financial transactions.
An added bonus to this process is that there is likely to be notifications when someone is tampering with the Paypal payment system - leaving you better equipped to deal with evil-doers in the future.
Please refer this link ,Hope PayPal Authorization & Capture method will be suitable for you ,since you don't want lose the response as well as miscalculated amount transfers, PayPal provides correlation id that can be referred for PayPal to confirm your order status,it will be better to pass the order id to PayPal API.
https://cms.paypal.com/us/cgi-bin/?cmd=_render-content&content_ID=developer/howto_admin_authcapture
http://www.scribd.com/doc/6303345/40/CorrelationID-for-Reporting-Problems-to-PayPal
Good Luck!
This is known as 2-phase commit. As long as paypal does not participate in the same transaction, you will run into problems.
I would debit Alice the $150 and reflect the transaction is "Pending Confirmation", then periodically poll PayPal to synchronize your DB, since you have no control of when the network or PayPal may be available, post, reverse or adjust the transaction. Once PayPal processes the transaction, you can change the status in your DB from "Pending" to "Completed". BTW, this how bank accounts and credit cards are processed. You could apply a double-entry accounting method to your DB. (see this Q&A)
From what i see you need to make sure the transaction is complete otherwise nothing should be done .
If you deposit the money into PayPal API and you do not receive an response from PAYPAL API then you need to rollback the transaction in you DB.
Is there any way to get the Channel ID on the server or transmit it inside a RequestFactory call?
Situation:
User starts the application, a channel is being opened.
User persists an entity with RequestFactory (requests.persist().using(...).fire(...)).
The persist() method on the server pings all connected clients to tell them that the entity has been updated.
But the user that made the initial change doesn't have to be pinged. Is there a way to find out which client made the change? It's not enough to know the user, because one user may have opened several windows (channels).
Honestly I haven't used Channel API yet but according to documentation each client is treated as separate user. So the solution lies beyond GAE API and I think you have two options:
Create logical User ID on the client that will be mapped to possibly multiple channels. That way you'll know what channels to skip.
Ping all channels anyway but send numeric Version of newly persisted entity. Then client will compare received version with what it has and if it's higher it means it needs to call findModel(id) again.