I have a service broker queue with one conversation that is not being processed by anything. However it is not returning anything when I run:
RECEIVE TOP(1000) * FROM dbo.QueueName
However if I run this:
SELECT COUNT(*) FROM dbo.QueueName
I am getting a figure in the thousands. The figure is not changing either. I assume that some process has the Conversation Group locked but is not doing anything with it. How can I tell if this is the case and how can I tell which SPID has the lock?
If you have access to system views, sys.dm_os_waiting_tasks and sys.dm_tran_locks should get you where you need to go. The former is high level "what's my process waiting on" type of stuff. From that, you can get which SPID(s) are blocking your query. From that list of SPIDs, you can (if you're interested) query the locks view to see what locks they're holding. Incidentally, I found that one of either allow_snapshot_isolation or read_committed_snapshot on the database helped with queue locking in a recent engagement.
Related
We have a new sp getting released and during testing we found that when it runs its blocking other OLTP transactions. We found that initially it was because the new sp was causing lock escalation on a table and we reduced the number of batch size and was able to avoid that. even after avoiding lock escalation, it is still blocking oltp transactions that are coming in.
I think its locking the same row which the oltp transaction is updating.
I need to find a way to track all the locks held and release by the new sp. I tried trace/xevents(lock acquired/release) and it does not look like its capturing all the lock, may be cause it happens so fast.
Just to understand how lock acquired look like, i tested it out by doing a select * from atable. but it gives me different results. When we do select * doesnt it put a series of page locks, so i should be seeing shared page locks in the trace. but all i see is IS lock acquired and released.
what is the best way to track all the locks for a given transaction?
I ran below query in one session
begin tran
update orderstst
set unitprice=unitprice+1
waitfor delay '00:00:20'
and ran below dmv while the query is running on other session..
select resource_database_id,request_mode,request_type,request_status,txt.text
from sys.dm_tran_locks lck
join
sys.dm_exec_requests ec
on ec.session_id=lck.request_session_id
cross apply
sys.dm_exec_Sql_text(ec.sql_handle) txt
I got below data...
when the transaction is still not committed,but completed,i ran above dmv again.but didnt get any output.since this is not currently executing.
But running below dmv,will still give me lock info of all sessions holding locks..so you will be able to identify which session is holding more locks
select resource_database_id,request_mode,request_type,request_status
from sys.dm_tran_locks lck
join
sys.dm_exec_sessions ec
on ec.session_id=lck.request_session_id
Above query gives me below info..
So in summary,you have to run DMV1 or DMV2 for some period through sql agent job and insert into some table for later analaysis..
Further from SQL 2012,you can use extended events also..
Go to Management ->Extended Events ,Right Click and say ,start new session wizard.
Give it a name and check start at server startup
next screen gives you an option to select default template or not,i choose default template for locks as shown below and click next..
In the next screen,you can choose different events,in channel,select all channels and do the same in categories too and select the events of your interest,i choose below ..
In this screen,you can select actions ,i choose text ,sessionid
In next screen,filter like say for example ..gather events only for a databasename like 'somename' or query like some text..
Next screen is where you can save file to disk for later analysis..
Complete rest of screens and finally select start event session immediately option..
When you are done with gathering data,go to extended events and stop the session you created.Right click and say view target data..which shows you below screenn
EDIT: as of 12/3/2019 the start new session wizard is now located here:
I have a server broker application I inherited that abends with a false negative.
I think it is using sys.dm_broker_activated_tasks incorrectly, and I want to validate that my understanding of what that view shows is correct.
Can I assume that this view is showing tasks being activated, and no so much those that were activated, but are now in the process of completing?
The procedure I have monitors for completion of processing by looking for when there are no entries in sys.dm_broker_activated_tasks for that queue.
This appears to work (mostly), except occasionally at the end when processing in the queue is winding down.
The row in that table seems to disappears before the final message in the queue has completed.
And unfortunately, as this uses the fire and forget anti-pattern, I can't really at this time do more than make the polling monitor a bit smarter.
That view doesn't do much apart from:
Returns a row for each stored procedure activated by Service Broker.
https://msdn.microsoft.com/en-us/library/ms175029.aspx
Not sure if you have looked at the code, but I think a better usage of it is to combine it with sys.dm_exec_sessions
select
at.spid
,DB_NAME(at.database_id) AS [DatabaseName]
,at.queue_id
,at.[procedure_name]
,s.[status]
,s.login_time
from
sys.dm_broker_activated_tasks at
inner join
sys.dm_exec_sessions s
on
at.spid = s.session_id;
Another good place to troubleshoot Service Broker is sys.transmission_queue. You will see every message sent there until there is an acknowledgement that it was received.
Today I was troubleshooting a deadlock case and discovered a quite strange case (at least it seemed strange to me). I had two concurrent statements (UPDATE and SELECT) and they led to a deadlock scenario. No question there. Below is DeadLock graph which depicts my case:
What bothers me is the type of lock which SELECT holds. Why is it SIU (Share with Intern Update), but not S (Shared) or IS (Intent Shared) lock?
I found a post on MSDN forum, which explains quite similar case:
Profiler shows current cumulative lock from all sessions. After I exported deadlock event to xdl-file and opened it in text editor I found that process which selects data has S lock and process that updates data has IU lock (and wants IX lock). And resource is SIU-locked (S+IU).
For me it sounds like (and it doesn't make any sense):
SELECT has SIU lock, because some other session has an intent to do an UPDATE
Can anybody, please, explain me why SELECT holds a SIU lock?
UPDATE: The select statement is autogenerated by EF 6.1.2; update statement is a Stored Procedure.
I think it will turn out that there was DML executing under the same transaction. I know you stated this was not the case but I really don't see how selects can take write locks.
Attach profiler and capture all statements and transaction operations (the TM events). Maybe you'll find more going on in the same session and transaction than you expect.
Submitting this guess as an answers because it seems quite likely to be the solution.
I have a stored procedure, and I want to ensure it cannot be executed concurrently.
My (multi-threaded) application does all necessary work on the underlying table via this stored procedure.
IMO, locking the table itself is an unnecessarily drastic action to take, and so when I found out about sp_GetAppLock, which essentially enforces a critical section, this sounded ideal.
My plan was to encase the stored procedure in a transaction and to set up spGetAppLock with transaction scope. The code was written and tested successfully.
The code has now been put forward for review and I have been told that I should not call this function. However when asking the obvious question "why not?", the only reasons I am getting are highly subjective, to do with any form of locking being complicated.
I don't necessarily buy this, but I was wondering whether anyone had any objective reasons why I should avoid this construct. Like I say, given my circumstances a critical section sounds an ideal approach to me.
Further info: An application sits on top of this with 2 threads T1 and T2. Each thread is waiting for a different message M1 and M2. The business logic involved says that processing can only happen once both M1 and M2 have arrived. The stored procedure logs that Mx has arrived (insert) and then checks whether My is present (select). The built-in locking is fine to make sure the inserts happen serially. But the selects need to happen serially too and I think I need to do something over and above the built-in functionality here.
Just for clarity, I want the "processing" to happen exactly once. So I can't afford for the stored procedure to return either false positives or false negatives. I'm worried that if the stored proc runs twice in very quick succession, then both "selects" might return data which indicates that it is appropriate to perform processing.
What is the procedure doing that you cannot rely on SQL Servers built-in concurrency control mechanisms? Often queries can be rewritten to allow real concurrency.
But if this procedure indeed has to be executed "alone", locking the table itself on first access is most likely going to be a lot faster than using the call to sp_GetAppLock. It sounds like this procedure is going to be called often. If that is the case you should look for a way to achieve the goal with minimal impact.
If the table contains no other rows besides of M1 and M2 a table lock is still your best bet.
If you have multiple threads sending multiple messages you can get more fine-grained by using "serializable" as transaction level and check if the other message is there before you do the insert but within the same transaction. To prevent deadlocks in this case make sure you check for both messages for example like this:
SET TRANSACTION ISOLATION LEVEL SERIALIZABLE;
BEGIN TRAN;
SELECT
#hasM1 = MAX(CASE WHEN msg_type='M1' THEN 1 ELSE 0 END),
#hasM2 = MAX(CASE WHEN msg_type='M2' THEN 1 ELSE 0 END)
FROM messages WITH(UPDLOCK)
WHERE msg_type IN ('M1','M2')
INSERT ...
IF(??) EXEC do_other_stuff_and_delete_messages;
COMMIT
In the IF statement before(!) the COMMIT you can use the information collected before the insert together with the information that you inserted to decide if additional processing is necessary.
In that processing step make sure to either mark those messages as processed or to delete them all still within the same transaction. That will make sure that you will not process those messages twice.
SERIALIZABLE is the only transaction isolation level that allows to lock rows that do not exist yet, so the first select statement with the WITH(UPDLOCK) effectively prevents the other row being inserted while the first execution is still running.
Finally, these are a lot of things to be aware of that could go wrong. You might want to have a look at service broker instead. you could use three queues with that. one for type M1 and one for type M2. Every time a message arrives within those queues a procedure can automatically be called to insert a token into the third queue. The third queue then could activate a process to check if both messages exist and do work. That would make the entire process asynchronous but for that it would be easy to restrict the queue 3 response to always only do one check at a time.
Service broker on msdn, also look at "activation" for the automatic message processing.
sp_GetAppLock is just like many other tools and as such it can be misused, overused, or correctly used. It is an exact match for the type of problem described by the original poster.
This is a good MSSQL Tips post on the usage
Prevent multiple users from running the same SQL Server stored procedure at the same time
http://www.mssqltips.com/sqlservertip/3202/prevent-multiple-users-from-running-the-same-sql-server-stored-procedure-at-the-same-time/
We use sp_getapplock all the time, due to the fact that we support some legacy applications that have been re-worked to use a SQL back-end, and the SQL Server locking model is not an exact match for our application logic.
We tend to go for a 'pessimistic' locking model, where we lock an entity before allowing a user to edit it, and use the (NOLOCK) hint extensively when reading data to bypass any blocking from the native locks on the actual tables. sp_getapplock is a good match for this. We also use it to enforce critical paths in large multi-user systems. You have to be systematic about what you call the locks you place.
We've found no performance problems with large numbers of user/locks via this route, so I see no reason why it wouldn't work well for you. Just be aware that you can get blocking and deadlocks if you have processes that place the same named locks, but not necessarily in the same order.
You can create a table with a flag for each set of messages, so if one of the threads is first to start processing it will mark the flag as processing.
To make sure that record blocked properly once one of threads reaches it use:
SELECT ... FROM WITH(XLOCK,ROWLOCK,READCOMMITTED) ... WHERE ...
This peace of code will put Exclusive lock on the record meaning who first got to it owns the row.
Then you do your changes and update flag, other thread will get updated value because it will be blocked by Exclusive lock until first thread commmits or rollbacks transaction.
For this to work you always need to select records from table with XLOCK this way it will work as expected.
Hope this helps.
Exclusive lock prove:
USE master
GO
IF OBJECT_ID('dbo.tblTest') IS NOT NULL
DROP TABLE dbo.tblTest
CREATE TABLE tblTest ( id int PRIMARY KEY )
;WITH cteNumbers AS (
SELECT 1 N
UNION ALL
SELECT N + 1 FROM cteNumbers WHERE N<1000
)
INSERT INTO
tblTest
SELECT
N
FROM
cteNumbers
OPTION (MAXRECURSION 0)
BEGIN TRANSACTION
SELECT * FROM dbo.tblTest WITH(XLOCK,ROWLOCK,READCOMMITTED) WHERE id = 1
SELECT * FROM sys.dm_tran_locks WHERE resource_database_id = DB_ID('master')
ROLLBACK TRANSACTION
I need to develop a server application (in C#) that will read rows from a simple table (in SQL Server 2005 or 2008), do some work, such as calling a web service, and then update the rows with the resulting status (success, error).
Looks quite simple, but things get tougher when I add the following application requisites:
Multiple application instances must be running at the same time, for Load Balancing and Fault Tolerance purposes. Typically, the application will be deployed on two or more servers, and will concurrently access the same database table. Each table row must be processed only once, so a common synchronization/locking mechanism must be used between multiple application instances.
When an application instance is processing a set of rows, other application instances shouldn't have to wait for it to end in order to read a different set of rows waiting to be processed.
If an application instance crashes, no manual intervention should need to take place on the table rows that were being processed (such as removing temporary status used for application locking on rows that the crashing instance was processing).
The rows should be processed in a queue-like fashion, i.e., the oldest rows should be processed first.
Although these requisites don't look too complex, I'm having some trouble in coming up with a solution.
I've seen locking hint suggestions, such as XLOCK, UPDLOCK, ROWLOCK, READPAST, etc., but I see no combination of locking hints that will allow me to implement these requisites.
Thanks for any help.
Regards,
Nuno Guerreiro
This is a typical table as queue pattern, as described in Using tables as Queues. You would use a Pending Queue and the dequeue transaction should also schedule a retry in a reasonable timeout. Is not realistically possible to hold on to locks for the duration of the web calls. On success, you would remove the pending item.
You also need to be able to dequeue in batch, dequeuing one-by-one is too slow if you go into serious load (100 and thousands of operations per second). So taking the Pending Queue example from the article linked:
create table PendingQueue (
id int not null,
DueTime datetime not null,
Payload varbinary(max),
cnstraint pk_pending_id nonclustered primary key(id));
create clustered index cdxPendingQueue on PendingQueue (DueTime);
go
create procedure usp_enqueuePending
#dueTime datetime,
#payload varbinary(max)
as
set nocount on;
insert into PendingQueue (DueTime, Payload)
values (#dueTime, #payload);
go
create procedure usp_dequeuePending
#batchsize int = 100,
#retryseconds int = 600
as
set nocount on;
declare #now datetime;
set #now = getutcdate();
with cte as (
select top(#batchsize)
id,
DueTime,
Payload
from PendingQueue with (rowlock, readpast)
where DueTime < #now
order by DueTime)
update cte
set DueTime = dateadd(seconds, #retryseconds, DueTime)
output deleted.Payload, deleted.id;
go
On successful processing you would remove the item from the queue using the ID. On failure, or on crash, it would be retries automatically in 10 minutes. One think you must internalize is that as long as HTTP does not offer transactional semantics you will never be able to do this with 100% consistent semantics (eg. guarantee that no item is processed twice). You can achieve a very high margin for error, but there will always be a moment when the system can crash after the HTTP call succeeded before the database is updated, and will cause the same item to be retried since you cannot distinguish this case from a case when the system crashed before the HTTP call.
I initially suggested SQL Server Service Broker for this. However, after some research it turns out this is probably not the best way of handling the problem.
What you're left with is the table architecture you've asked for. However, as you've been finding, it is unlikely that you will be able to come up with a solution that meets all the given criteria, due to the great complexity of locking, transactions, and the pressures placed on such a scheme by high concurrency and high transactions per second.
Note: I am currently researching this issue and will get back to you with more later. The following script was my attempt to meet the given requirements. However, it suffers from frequent deadlocks and processes items out of order. Please stay tuned, and in the meantime consider a destructive reads method (DELETE with OUTPUT or OUTPUT INTO).
SET XACT_ABORT ON; -- blow up the whole tran on any errors
SET TRANSACTION ISOLATION LEVEL READ COMMITTED;
BEGIN TRAN
UPDATE X
SET X.StatusID = 2 -- in process
OUTPUT Inserted.*
FROM (
SELECT TOP 1 * FROM dbo.QueueTable WITH (READPAST, ROWLOCK)
WHERE StatusID = 1 -- ready
ORDER BY QueuedDate, QueueID -- in case of items with the same date
) X;
-- Do work in application, holding open the tran.
DELETE dbo.QueueTable WHERE QueueID = #QueueID; -- value taken from recordset that was output earlier
COMMIT TRAN;
In the case of several/many rows being locked at once by a single client, there is a possibility of the rowlock escalating to an extent, page, or table lock, so be aware of that. Also, normally holding long-running transactions that maintain locks is a big no-no. It may work in this special usage case, but I fear that high tps by multiple clients will make the system break down. Note that normally, the only processes querying your queue table should be those that are doing queue work. Any processes doing reporting should use READ UNCOMMITTED or WITH NOLOCK to avoid interfering with the queue in any way.
What is the implication of rows being processed out of order? If an application instance crashes while another instance is successfully completing rows, this delay will likely cause at least one row to be delayed in its completion, causing the processing order to be incorrect.
If the transaction/locking method above is not to your satisfaction, another way to handle your application crashing would be to give your instances names, then set up a monitor process that has the capacity to check periodically if those named instances are running. When a named instance starts up it would always reset any unprocessed rows that possess its instance identifier (something as simple as "instance A" and "instance B" would work). Additionally, the monitor process would check if the instances are running and if one of them is not, reset the rows for that missing instance, enabling any other instances to run. There would be a small lag between crash and recovery, but with proper architecture it could be quite reasonable.
Note: The following links should be edifying:
info about XLOCK
Tables as Queues
You can't do this with SQL transactions (or relying on transactions as your main component here). Actually, you can do this, but you shouldn't. Transactions are not meant to be used this way, for long locks, and you shouldn't abuse them like this.
Keeping a transaction open for that long (retrieve rows, call the web service, get back to make some updates) is simply not good. And there's no optimistic locking isolation level that will allow you to do what you want.
Using ROWLOCK is also not a good idea, because it's just that. A hint. It's subject to lock escalation, and it can be converted to a table lock.
May I suggest a single entry point to your database? I think it fits in the pub/sub design.
So there would be only one component that reads/updates these records:
Reads batches of messages (enough for all your other instances to consume) - 1000, 10000, whatever you see fit. It makes these batches available to the other (concurrent) components through some queued way. I'm not going to say MSMQ :) (it would be the second time today I recommend it, but it's really suitable in your case too).
It marks the messages as in progress or something similar.
Your consumers are all bound, transactionally, to the inbound queue and do their stuff.
When ready, after the web service call, they put the messages in an outbound queue.
The central component picks them up and, inside a distributed transaction, does an update on the database (if it fails the messages will stay in the queue). Since it is the only one that could do that operation you won't have any concurrency issues. At least not on the database.
In the mean time it can read the next pending batch and so on.