I wish to comply with Remus Rusanu's dialog recycling technique in my SSB implementation. I wrote some activation procedure for initiator queue, in order to hook EndDialog message back from target and clean the Dialog table from the closed conversation handle.
Nevertheless, though EndDialog ack properly reaches initiator side, no activation is triggered, so my message handler cannot operate and clean the place.
CREATE PROCEDURE fdwh.ProcessResponse
AS
BEGIN
DECLARE #dlgId UNIQUEIDENTIFIER;
DECLARE #msgTypeName SYSNAME;
DECLARE #msgBody VARBINARY(MAX);
DECLARE #payloadHistoryId INT;
BEGIN TRY
BEGIN TRANSACTION
WAITFOR(
RECEIVE TOP(1)
#dlgId = [conversation_handle],
#msgTypeName = message_type_name,
#msgBody = message_body
FROM [fdwh].[SenderQueue]), TIMEOUT 10;
-- Message is regular end of conversation, terminate it
IF (#msgTypeName = N'http://schemas.microsoft.com/SQL/ServiceBroker/EndDialog')
BEGIN
END CONVERSATION #dlgId;
DELETE FROM DWH_BOARD.dbo.Dialog
WHERE (DbId = DB_ID()) AND
(DialogId = #dlgId);
END
-- Message is error, extracts and logs number and description
IF (#msgTypeName = N'http://schemas.microsoft.com/SQL/ServiceBroker/Error')
BEGIN
[...]
I expect queue activation to be triggered and http://schemas.microsoft.com/SQL/ServiceBroker/EndDialog message to be processed as well, but it's not. Isn't EndDialog ACK a regular message?
Please find below a Profiler trace screenshot that is self explaining: .
Example is pure local (single instance/two DBs).
Thanks,
Update
A few more metrics for failing queue:
`SELECT que.[name], que.is_activation_enabled, que.is_receive_enabled, que.is_poison_message_handling_enabled, que.activation_procedure, que.max_readers, [execute_as] = (SELECT pri.[name] FROM sys.database_principals pri WHERE pri.principal_id = que.execute_as_principal_id) FROM sys.service_queues que WHERE que.[name] = 'SenderQueue';
GO
SELECT conversation_handle, to_service_name, message_type_name, is_conversation_error, is_end_of_dialog, enqueue_time, transmission_status FROM sys.transmission_queue;
GO
SELECT [name], is_broker_enabled, log_reuse_wait_desc FROM sys.databases WHERE database_id = 8;
GO
EXEC sp_spaceused 'fdwh.SenderQueue';
GO
SELECT * FROM sys.dm_broker_activated_tasks WHERE database_id=8;
GO
SELECT [state], last_activated_time, tasks_waiting FROM sys.dm_broker_queue_monitors WHERE database_id = 8;
GO
`
Activation occurs only when new messages arrive.
"When STATUS = ON, the queue starts the stored procedure specified
with PROCEDURE_NAME when the number of procedures currently running is
less than MAX_QUEUE_READERS and when messages arrive on the queue
faster than the stored procedures receive messages."
https://learn.microsoft.com/en-us/sql/t-sql/statements/alter-queue-transact-sql?view=sql-server-2017
The activation procedure is expected to continue consuming messages until the queue is empty, and remains empty for the duration of its WATFOR ... RECEIVE.
Your activation procedure is missing the loop. It's RECEIVING a single message and exiting. So every time a new message arrives a single old message is consumed. This may appear to work for a while, but if you ever have get a backlog of messages, you'll never catch up.
Related
I am reading an expert from a google preview book.
shorturl.at/htF38
I want to basically simulate a poision message in a Service Broker Queue.
The author presents this code fragment to simulate 5 rollbacks .
My question is how does this pieces of code simulate 5 rollbacks beats me.
Per me On line 31 if it does not see a message in the queue the rollback happens once
and the break statement breaks the loop no ?
DECLARE #ch UNIQUEIDENTIFIER
DECLARE #messagetypename NVARCHAR(256)
DECLARE #messagebody XML
WHILE (1=1)
BEGIN
BEGIN TRANSACTION
WAITFOR (
RECEIVE TOP (1)
#ch = conversation_handle,
#messagetypename = message_type_name,
#messagebody = CAST(message_body AS XML)
FROM TargetQueue
),
TIMEOUT 60000
IF (##ROWCOUNT = 0)
BEGIN
ROLLBACK TRANSACTION
BREAK **--Line No 31**
END
-- Rolling back the current transaction
PRINT 'Rollback the current transaction - simulating a poison message...'
ROLLBACK TRANSACTION
END
GO
Now, when you place a message into the TargetQueue and execute the service program
shown in Listing ,you'll see that Service Broker deactivates the queue automatically, and
you’ll get an error message
I am writing a stored procedure that would perform the following operations:
BEGIN a transaction
perform some tasks
use Service Broker to kick-off a background process
wait for a response message (with job status) from Service Broker
COMMIT or ROLLBACK transaction, depending on the response message
The issue is that Service Broker communication is not working inside a TRANSACTION:
the message queue has activation enabled, but the associated stored procedure is not executed (PRINT statements in the stored procedure are not written to ERRORLOG file)
RECEIVE command times out
Here's an extract of my code:
-- Comment out the following line to make everything work
begin tran t1
DECLARE #Update_Msg XML([sb].[Service_Broker_xxx_Schemas]) = '
<Request xmlns="xxx">
<Table xmlns="xxx">
<Fields>
xxx
</Fields>
</Table>
<Requested_By>xxx</Requested_By>
</Request>'
DECLARE #conversation_handle UNIQUEIDENTIFIER
,#message_body varbinary(max)
,#message_type_name nvarchar(256)
,#timestamp datetime2
BEGIN DIALOG CONVERSATION #conversation_handle
FROM SERVICE [xxx_Initiating_Service]
TO SERVICE 'xxx_Target_Service'
ON CONTRACT xxx_Contract
WITH ENCRYPTION = OFF;
SEND ON CONVERSATION #conversation_handle
MESSAGE TYPE [xxx_Command](#Update_Msg);
select * from sys.transmission_queue with(nolock)
--PRINT #conversation_handle
WAITFOR (
-- just handle one message at a time
RECEIVE TOP(1) #conversation_handle = conversation_handle -- the identifier of the dialog this message was received on
,#message_type_name = message_type_name
,#message_body=message_body -- the message contents
,#timestamp = GETDATE()
FROM [sb].[xxx_Initiator_Queue]
WHERE conversation_handle = #conversation_handle
), TIMEOUT 1000 -- if the queue is empty for one second, give UPDATE and go away
IF ##ROWCOUNT > 0
BEGIN
SELECT ##ROWCOUNT, #message_type_name, CONVERT(XML, #message_body)
END CONVERSATION #conversation_handle;
END
ELSE
BEGIN
PRINT 'Did not receive any response from Service Broker.'
END
-- Comment out the following line to make everything work
commit tran t1
What is the correct way to implement Service Broker messaging inside a transaction?
Sending messages via Service Broker is transactional. That is, if you do begin tran; send;, the message isn't actually sent until you commit.
I am experiencing poison messages and I am not sure why.
My broker setup looks like this:
CREATE MESSAGE TYPE
[//DB/Schema/RequestMessage]
VALIDATION = WELL_FORMED_XML;
CREATE MESSAGE TYPE
[//DB/Schema/ReplyMessage]
VALIDATION = WELL_FORMED_XML;
CREATE CONTRACT [//DB/Schema/Contract](
[//DB/Schema/RequestMessage] SENT BY INITIATOR,
[//DB/Schema/ReplyMessage] SENT BY TARGET
)
CREATE QUEUE Schema.TargetQueue
CREATE SERVICE [//DB/Schema/TargetService]
ON QUEUE Schema.TargetQueue (
[//DB/Schema/Method3Contract]
)
CREATE QUEUE Schema.InitiatorQueue
CREATE SERVICE [//DB/Schema/InitiatorService]
ON QUEUE Schema.InitiatorQueue
Then I have my internal activation procedure:
CREATE PROCEDURE Schema.Import
AS
DECLARE #RequestHandle UNIQUEIDENTIFIER;
DECLARE #RequestMessage VARCHAR(8);
DECLARE #RequestMessageName sysname;
WHILE (1=1)
BEGIN
BEGIN TRANSACTION;
WAITFOR (
RECEIVE TOP(1)
#RequestHandle = conversation_handle,
#RequestMessage = message_body,
#RequestMessageName = message_type_name
FROM
Schema.TargetQueue
), TIMEOUT 5000;
IF (##ROWCOUNT = 0)
BEGIN
COMMIT TRANSACTION;
BREAK;
END
EXEC Schema.ImportStep1 #ID = #RequestMessage;
--EXEC Schema.ImportStep2 #ID = #RequestMessage;
END CONVERSATION #RequestHandle;
COMMIT TRANSACTION;
END
My activation is enabled by:
ALTER QUEUE Schema.TargetQueue
WITH
STATUS = ON,
ACTIVATION
( STATUS = ON,
PROCEDURE_NAME = Schema.Import,
MAX_QUEUE_READERS = 10,
EXECUTE AS SELF
)
I initiate this process with this stored procedure
CREATE PROCEDURE Schema.ImportStart
AS
BEGIN
DECLARE #ID VARCHAR(8);
DECLARE Cursor CURSOR FOR
SELECT ID FROM OtherDatabase.OtherSchema.ImportTable
EXCEPT
SELECT ID FROM Table
OPEN Cursor;
FETCH NEXT FROM Cursor INTO #ID;
WHILE ##FETCH_STATUS = 0
BEGIN
DECLARE #InitiateHandle UNIQUEIDENTIFIER;
DECLARE #RequestMessage VARCHAR(8);
BEGIN TRANSACTION;
BEGIN DIALOG
#InitiateHandle
FROM SERVICE
[//DB/Schema/InitiatorService]
TO SERVICE
N'//DB/Schema/TargetService'
ON CONTRACT
[//DB/Schema/Contract]
WITH
ENCRYPTION = OFF;
SELECT #RequestMessage = #ID;
SEND ON CONVERSATION
#InitiateHandle
MESSAGE TYPE
[//DB/Schema/RequestMessage]
(#RequestMessage);
COMMIT TRANSACTION;
FETCH NEXT FROM Cursor INTO #ID;
END
CLOSE Cursor;
DEALLOCATE Cursor;
END
So how this should work is:
I execute ImportStart
A message for each ID gets generated
Internal activation makes Import steps execute
Instead, I get poison messaging and the queue becomes disabled.
If however,
I I set Schema.TargetQue Activation to OFF
EXEC schema.ImportStart
EXEC schema.Import manually
It works fine.
Any insights anyone?
Well:
Your message types are defined as well_formed_xml, yet you send varchar(8) as a message body. Does it really work?
You use [//DB/Schema/Method3Contract] for the target queue, but do not define it. A misspelling, most likely.
You specify EXECUTE AS SELF in the queue activation. BOL says some mystical thing about this case:
SELF
Specifies that the stored procedure executes as the current user. (The database principal executing this ALTER QUEUE statement.)
I'm not really sure I understand the quoted statement, because it apparently contradicts with your experience. If it would be your user account, everything should have been fine, because you seem to have all permissions necessary to do the job.
So, just in case - who is the owner of the Schema schema? What permissions does this principal possess? And, if it's not you, who executes the alter queue statement (and why)?
Without access to logs, it's significantly more difficult to diagnose the problem, but I would start with creating a new user account with permissions identical to yours, setting it the owner of the Schema schema and then slowly working it down, revoking unnecessary permissions until it breaks. Assuming, of course, it will work at all.
One of the basics of building a distributed application that uses the asynchronous communication can be expressed as Do not wait actively for any event! This way, the natural solution based on SQL Service Broker is to use the activation of a stored procedure by the message that arrived to the queue.
The Lesson 2: Creating an Internal Activation Procedure from the official Microsoft tutorial shows how to bind the stored procedure to the message queue. It also suggests the way how the sp should be implemented.
(I am new to SQL. But should not be there one more BEGIN after the CREATE PROCEDURE... AS, and one more END before the GO?)
Do I understand it corectly? See my questions below the code...
CREATE PROCEDURE TargetActivProc
AS
DECLARE #RecvReqDlgHandle UNIQUEIDENTIFIER;
DECLARE #RecvReqMsg NVARCHAR(100);
DECLARE #RecvReqMsgName sysname;
WHILE (1=1)
BEGIN
BEGIN TRANSACTION;
WAITFOR
( RECEIVE TOP(1)
#RecvReqDlgHandle = conversation_handle,
#RecvReqMsg = message_body,
#RecvReqMsgName = message_type_name
FROM TargetQueueIntAct
), TIMEOUT 5000;
IF (##ROWCOUNT = 0)
BEGIN
ROLLBACK TRANSACTION;
BREAK;
END
IF #RecvReqMsgName =
N'//AWDB/InternalAct/RequestMessage'
BEGIN
DECLARE #ReplyMsg NVARCHAR(100);
SELECT #ReplyMsg =
N'<ReplyMsg>Message for Initiator service.</ReplyMsg>';
SEND ON CONVERSATION #RecvReqDlgHandle
MESSAGE TYPE
[//AWDB/InternalAct/ReplyMessage]
(#ReplyMsg);
END
ELSE IF #RecvReqMsgName =
N'http://schemas.microsoft.com/SQL/ServiceBroker/EndDialog'
BEGIN
END CONVERSATION #RecvReqDlgHandle;
END
ELSE IF #RecvReqMsgName =
N'http://schemas.microsoft.com/SQL/ServiceBroker/Error'
BEGIN
END CONVERSATION #RecvReqDlgHandle;
END
COMMIT TRANSACTION;
END
GO
When the message arrives, the procedure is called, and it enters to the "infinite" loop. Actually, the loop is not infinite because of the BREAK after the ROLLBACK when no data arrived (after the TIMEOUT).
If the data arrived, the BREAK is skipped. If the expected message arrived, the reply is sent back. If the ...EndDialog or ...Error message is received, the END CONVERSATION is executed. Can also some other kind of message be observed here?
As some message arrived (and was processed) the transaction is commited.
But why the loop now? Is the intention to process the other messages that got stuck in the queue because of the broken communication line in the past? Or because of more messages came at once and could not be processed so quickly?
What happens when another message is queued up, and the stored procedure is still running. Is another working process assigned for its processing? Can another stored procedure be launched in parallel? If yes, then why the loop?
Thanks for your help, Petr
Internal activation is not like a trigger. Specifically, the activated procedure does not get launched for each message that arrived. Instead the procedure is launched when there is something to process and is supposed to dequeue messages contiguously (in a loop) while the SSB infrastructure is monitoring the progress and, if necessary, launches a second procedure to help, up to the max specified. See Understanding Queue Monitors.
Having a loop in the activated procedure is not not strictly required, things should work fine even w/o the loop. The loop should perform better in a very busy environment. See also this old MSDN discussion.
We recently identified a problem with one of our databases where as a result of a 'fire & forget' setup (i.e: conversations being closed immediately after sending), our sys.conversation_endpoints table was filling up with DI/DISCONNECTED_INBOUND messages. This eventually spilled over into the tempDB, causing it to grow enormously and eat up precious disk space. We eventually resolved this issue by commenting out the line
END CONVERSATION #handle WITH CLEANUP
in our sending SP and closing the conversations in our receiving SP using the same code,
END CONVERSATION #handle WITH CLEANUP
However, we now have a new issue. Since moving servers (and migrating from SQL Server 2005 to SQL Server 2008) we've recently discovered that sys.conversation_endpoints is now filling up with CO/CONVERSING messages, indicating that the conversations are not being closed. The receiving SP is closing them, or at least is running the command to do so, so I don't understand where these messages are coming from.
I've tried going back to ending the conversation at the point of send, but it has no effect. Is it wrong to end conversations on the receiving end using WITH CLEANUP? Or is there some other problem?
This post on techtarget seems to suggest its a bug, and that running a job to cleanup the leftovers is the only solution...
UPDATE:
Pawel pointed out below that I should be avoiding the Fire & Forget Pattern, and I've added an activated SP to the initiator queue to end any conversations. However, sys.conversation_endpoints is STILL filling up, this time with CD/CLOSED messages. Here's the structure of my queues
Send_SP:
DECLARE #h UNIQUEIDENTIFIER
BEGIN DIALOG CONVERSATION #h
FROM SERVICE 'InitiatorQueue' TO SERVICE 'TargetQueue'
ON CONTRACT 'MyContract' WITH ENCRYPTION = OFF;
SEND ON CONVERSATION #h MESSAGE TYPE 'MyMessage' (#msg)
Receive_SP (Activated SP on TargetQueue)
DECLARE #type SYSNAME, #h UNIQUEIDENTIFIER, #msg XML;
DECLARE #target TABLE (
[message_type_name] SYSNAME,
[message_body] VARBINARY(MAX),
[conversation_handle] UNIQUEIDENTIFIER
)
WHILE(1=1)
BEGIN TRANSACTION
WAITFOR(RECEIVE TOP (1000)
[message_type_name],[message_body],[conversation_handle]
FROM TargetQueue INTO #target), TIMEOUT 2000
IF(##rowcount!=0)
BEGIN
WHILE((SELECT count(*) FROM #target) > 0)
BEGIN
SELECT TOP (1) #type = [message_type_name],
#msg = [message_body],
#h = [conversation_handle] FROM #target;
// Handle Message Here
END CONVERSATION #h;
DELETE TOP (1) FROM #target;
END
END
COMMIT TRANSACTION;
End_SP (Activated SP on InitiatorQueue)
DECLARE #type SYSNAME, #h UNIQUEIDENTIFIER, #msg XML;
DECLARE #init TABLE (
[message_type_name] SYSNAME,
[message_body] VARBINARY(MAX),
[conversation_handle] UNIQUEIDENTIFIER
)
WHILE(1=1)
BEGIN TRANSACTION
WAITFOR(RECEIVE TOP (1000)
[message_type_name],[message_body],[conversation_handle]
FROM InitiatorQueue INTO #init), TIMEOUT 2000
IF(##rowcount!=0)
BEGIN
WHILE((SELECT count(*) FROM #init) > 0)
BEGIN
SELECT TOP (1) #type = [message_type_name],
#msg = [message_body],
#h = [conversation_handle] FROM #init;
END CONVERSATION #h;
DELETE TOP (1) FROM #init;
END
END
COMMIT TRANSACTION;
Using the fire-and-forget pattern will inevitably lead to this and other types of issues. Additionally it will make any hypothetical errors go unnoticed. Is there any reason why you can't change the message exchange pattern so that the target issues END CONVERSATION (without cleanup!) once it receives a message and then the initiator only calls END CONVERSATION (again, without cleanup) upon receiving end conversation message from the target?