I'm in need of an Enterprise Service Bus/Message Queueing solution for publisher/subscriber functionality. I know MANY exist... MSMQ, MS Series, RabbitMQ, NServiceBus, etc etc etc...
My one requirement is that in a shared hosting solution, the only dependency that I can guarantee will exist is SQL 2005 and later... this leads me directly to SQL Service Broker.
If it sounds like I'm trying to shoehorn ESB functionality into SSB... I suppose I am...
My question is: does anyone know of a .NET API or framework that sits on top of SQL Service Broker and already provides much of the plumbing?
If I were to use pure ADO.net, I could add items to the queues by calling a stored procedure, but then:
Do to the nature of conversations, would I make one conversation per message?
If so, do I lose sequential message processing?
How do I receive messages (I know the receive syntax in t-SQL), do I call a stored procedure repeatedly in a message loop to try to
get a message off the queue?
Or would I WAITFORever? Keeping the connection open and executing the stored procedure forever?
SQL Service Broker doesn't support monologue conversations, but I read they can be implemented...
It's these kind of questions that make me wish there existed a .net solution that already managed all of this.
There was an effort to package a WCF Transport Channel for SQL Server Service Broker but, afaik, is abandonware.
But NServiceBus supports Service Broker as a transport, see Using NServiceBus and ServiceBroker.net and there are github projects like A simple wrapper API for SQL Service Broker and an ITransport plugin for NServiceBus. While not exactly mainstream, some support and community effort does exists.
As an ESB I think you will have problems due to lack of true pub-sub and broadcast. SQL Server 2012 has the ability to SEND a message to multiple targets, see How to Multicast messages with SQL Server Service Broker, but you will still have to implement the pub-sub infrastructure (publishing topics, subscribers etc) from scratch. MySpace did that and was a major effort, see Scale out SQL Server by using Reliable Messaging. My observation reffers to the low level direct use of SSB, I have never used NServiceBus so I cannot tell how well does it abstracts/expose unicast/broadcast/multicast/pub-sub over SSB.
As for your specific questions, I recommend reading Writing Service Broker Procedures and Reusing Conversations.
Related
I have an application using Rebus with SQL Server as a transport layer (yes, I know this is not the ideal transport layer) and I'm, trying to find an out-of-the-box tool to do real-time monitoring of the queues. I tried Rebus Snoop but I found that it is not compatible with SQL Server.
Does anyone know a way to monitor Rebus queues?
Thank you very much.
"Rebus queues" is not really a thing, so you most likely want to turn to whatever broker you are using and use some kind of monitoring tool for that.
E.g. if you're using Azure Service Bus, you can use the Azure Portal's built-in metrics to figure out queue lengths, request rates, etc.
But since it looks you're using SQL Server as the transport, there doesn't really (to my knowledge) exist any kind of compatible "queue monitoring tool" - you'd probably find loads of tools (e.g. from RedGate) that can help with looking at your SQL Server's performance metrics, but it doesn't know that you happen to be using your SQL Server as a message broker.
You can of course also turn to Rebus Pro that comes with Fleet Manager, which can help with some of the things. It's a commercial product though, so it might not be the answer you were hoping for. 😉
In SQL Server, What is enable_broker?
What is the risk?
SQL Table Dependency wants that.
1) https://learn.microsoft.com/en-us/sql/database-engine/configure-windows/sql-server-service-broker?view=sql-server-2017
SQL Server Service Broker provides native support for messaging and
queuing applications in the SQL Server Database Engine. This makes it
easier for developers to create sophisticated applications that use
the Database Engine components to communicate between disparate
databases. Developers can use Service Broker to easily build
distributed and reliable applications.
2) https://www.sqlservercentral.com/Forums/Topic818423-146-1.aspx
One risk being if there are already service brokers setup to use that
DB they will probably break, and any live connections will be killed
and rollback.
3) Sql Dependency with Service Broker
It's not necessary but if you want to see changes you need it or you can use an other service like periodic polling
Be careful using the SqlDependency class to monitor changes in the
database tables - it has the problems with the memory leaks.
I hope it will help you !
My first ever question on stack overflow so please go easy. I have a long running windows application that continually processes sql server commands. I also have a web front end that users use occasionally use to update the same db. I've noticed that sometimes (depending on what the windows application is processing at the time) that if a user submits something to the db I receive out of memory exceptions on the server. I realise I need to dig around a bit more and optimise the code. However I cannot afford the server to go down and expect that in the future i'll be allowing more and more users on the frontend. What i really need is a system that will queue the users requests (they are not time critical) and process them when the db is ready.
I'm using SQL 2012 express.
Is SQL Service Broker the best solution, i've also looked into MSMQ.
If so can someone point me in the right direction for it would be appreciate. In my search i'm just finding a lot of things it does that i don't think i need.
Cheers
It depends on where you're doing the persistence work, and / or calculations. If you're doing the hard work in your Windows Application, then using a Service Broker queue won't be worthwhile, as all you will be doing is receiving a message from the Service Broker queue in your Windows Application, doing your calculations and / or queries from the Windows Application, and then persisting the results to the database: as your database is already under memory pressure, this seems like an unnecessary extra load as you could just as easily queue and retrieve the message from MSMQ (or any other queueing technology).
If however you are doing all the work in the database and your Windows Application just acts as a marshalling service - eg taking the request and palming it off to a stored procedure for actioning - then Service Broker Queues may be worth using: because they are already operating within the context of the database, they can be very efficient at persisting amd querying data.
You would also want to take into failure modes, depending on whether or not you can afford to lose any messages. To ensure message persistence in MSMQ you have to use Transactional Messaging: Service Broker is more efficient at transactional queue processing than MSMQ (because it has transaction support built in, unlike MSMQ which has to use DTC, which adds an overhead) - but if your volume of messages is low, this may not be an issue.
Service Broker
What can be the Benefits of Programming with Service Broker?
In which condition one should use Service Broker?
Do you have Sample link for Code/Video that easily/precisely explains it's usage?
Queuing and asynchronous messaging are needed for many database
applications today. Service Broker provides a new, queue-based durable
messaging framework to address these needs. Using the Transact-SQL API
provided by Service Broker, you can easily develop services to handle
application requirements for queuing or asynchronous communications.
Some of the benefits of programming with Service Broker are:
Flexible development: The programs used in a single distributed
application can be written in multiple languages. Each program
provides the functionality of each distributed application component.
Improved security: You can express security requirements via
certificates, so application components do not need to share the same
security context. Service Broker uses SQL Server security features to
help you secure your applications.
Transactional processing: Message processing occurs within SQL Server
transactions to ensure data integrity. Service Broker supports remote
transactional messaging over a standard connection to the database.
Guaranteed ordering: Service Broker provides strong guarantees
regarding the delivery and processing of a related set of messages
exactly once and in order, so no additional coding is required to
provide this functionality.
Reliable delivery: All of the data needed for a conversation--a set of
related communications between two or more services--is persisted in
SQL Server. Service Broker supports clustering and database mirroring.
A conversation may be maintained through system restarts, server
failover, network outages, and so on without failing or losing data.
Improved scalability: Service Broker routing delivers messages based
on the name of the service, rather than on the network address of the
computer where the service runs. This allows you to install an
application on multiple computers without changing application code.
Ability to leverage existing knowledge: Service Broker uses
Transact-SQL to create objects. Applications that use Service Broker
are most often implemented in Transact-SQL or Microsoft .NET
Framework-compatible languages. You do not have to learn a new
language to create Service Broker applications.
For further detail check this link
http://dotnet.dzone.com/articles/c-sqldependency-monitoring
Service brokers are useful when we want do asynchronous processing in the database. which will improve the performance.
When we need inter database communication.
you can find sample code for creating and sending the messages using service broker here.
I am looking for a best practice or example of how I might be able to generate events for all update events on a given SQL Server 2008 R2 db. To be more descriptive, I am working on a POC where I would essentially publish update events to a queue (RabbitMq in my case) that could then be consumed by various consumers. This would be the first part of implementing a CQRS query-only data model via event sourcing. By placing on the que anybody could then subscribe to these events for replication into any number of query-only data models. This part is clear and fairly well-defined. the problem I am having is determining the best approach for generating the events out from SQL server. I have been given a few ideas such as monitoring the transaction log and SSIS. However, I'm not entirely sure if these options are adviseable or even feasible.
Does anybody have any experience with this sort of thing or have any notions on how to go about such an adventure? any help or guidance would be greatly appreciated.
You cannot monitor the log because, even if you would be able to understand it, you have the problem of the log being recycled before you had a chance to read it. Unless the log is somehow marked not to be truncated it will reused. For instance when transactional replication is enabled the log be pinned until is read by the replication agent and only then truncated.
SSIS is a very broad concept and saying that 'using SSIS to detect changes' is akin to saying 'I'll use a programing language to solve my problem'. The details is how would you use SSIS? There is no way, with or without SSIS, to reliably detect data changes on an arbitrary schema. Even data models specifically designed to allow for detecting changes have issues, specially at detecting deletes.
However there are viable alternatives. You can deploy Change Data Capture and delegate to the engine itself to track the changes. Consuming these detected changes and publishing them to consumers (via RabbitMQ if that's your fancy) is a something SSIS would be good at. But you have to understand that SSIS does not fare well to continuos, real-time tasks. It is designed to run periodically on batches, so your change notification consumers will be notified in spikes, with long delays (minutes), when the SSIS jobs run.
For a real-time approach a better solution is Service Broker. One possibility is to SEND Service Broker messages from triggers, but I would not recommend it. A better design is to have the application itself publish the changes by SEND-ing the message explicitly, when it does the data modification. With SQL Server 2012 is possible to multicast Service Broker messages to other SQL Server consumers (including SQL Server Express). SSB message delivery is fully transactional (no message gets sent if transaction rolls back) and does not require two-phase-commit with a message store resource manager. But to broadcast via RabbitMQ you would need to bridge the communication, ie. RECEIVE the SSB messages and transform them into RabbitMQ notifications.