Problem: To solve data consistency on one of most important tables in our system, all writes will be covered with explicit transactions (appropriate to what given stored procedure / business application is doing). However, since there are many sources of writes, it would be best if we had numerical measure of progress.
Ratio of explicit to autocommited transactions on that particular table would be one such measure. If after all the rework, there are still autocommit transactions in stats, we have proof that we missed some procedure or application (or manual process).
How can SQL server be queried for such information?
As mentioned by #lptr you can use sys.dm_tran_session_transactions to check if a user transaction was opened.
CREATE OR ALTER TRIGGER dbo.check_tran
ON dbo.YourTable
AFTER INSERT, UPDATE, DELETE
AS
SET NOCOUNT ON;
IF (SELECT is_user_transaction FROM sys.dm_tran_session_transactions WHERE session_id = ##SPID) = 1
BEGIN
INSERT Audit_transactions (host_name, user_name, initial_batch)
SELECT HOST_NAME(), SUSER_SNAME(), CONCAT(b.parameters, ' ', b.event_info)
FROM sys.dm_exec_requests r
OUTER APPLY sys.dm_exec_input_buffer (##SPID, r.request_id) b
WHERE r.session_id = ##SPID
END;
sys.dm_tran_session_transactions requires VIEW SERVER STATE permissions, so if the user does not have that then you may need to sign the trigger with a certificate of a user that has that permission.
Related
I was working mostly on PostgreSQL, but recently I was assigned to project with SqlServer and I encountered very strange behavior of this engine. I am using transaction in my code and connect with server via System.Data.SqlClient library. The code in transaction is approximately 1000 lines long, so I would like to not copy it here, but transaction is handled via code below:
using (var transaction = connection.BeginTransaction(IsolationLevel.ReadCommited))
{
//here code goes
//1. inserting new table metadata via inserts and updates
//2. creating new tables according to users project
//3. post execute actions via inserts and updates
//here is intended transaction freeze
await Task.Delay(1000 * 60 * 2);
}
During this execution I cannot perform any operation on database (query execution in SSMS or some code execution in application - doesn't matter). Simple selects f.e. SELECT * FROM "TableA" hangs, retrieving database properties in SSMS hangs etc. Any independent query waits for this one transaction to be completed.
I found several articles and answers here on SO, and based on those I tried following solutions:
Use WITH (NOLOCK) or WITH (READPAST) in SELECT statement
Changing database property Is Read Commited Snapshot ON to true
Changing transaction isolation level in code (all possible levels were tested)
None of the above solutions works.
I tested on 3 different computers: desktop, two laptops - the same behavior (SqlServer and SSMS was installed with default options).
In this thread: Understanding locking behavior in SQL Server there are some explanation of transaction isolation levels and locks but the problem is that WITH (NOLOCK) doesn't work for me as mentioned in 1).
This is very big problem for me, because my asynchronous application works synchronously because of that weird locks.
Oracle and postgres databases works perfectly fine, the problem concerns SqlServer only.
I don't use EntityFramework - I handle connections myself.
Windows 10 Pro
Microsoft SQL Server Developer (64-bit) version 15.0.2000.5
System.Data.SqlClient version 4.8.3
.NET 6.0
Any clues?
Update 1:
As pointed out in comments I have indeed schema changes in my transaction - CREATE TABLE and ALTER TABLE statements mixed with standard UPDATES and SELECTS. My app allows user to create own tables (in limited functionality) and when this table is registered in table via INSERT there are some CREATES to adjust table structure.
Update 2:
I can perform SELECT * FROM sys.dm_tran_locks
I executed DBCC SQLPERF ('sys.dm_os_wait_stats', CLEAR);
The problem remains.
The cause of the locking issue is DDL (CREATE TABLE, etc.) within a transaction. This will acquire and hold restrictive locks on system table meta-data and block other database activity that need access to object meta-data until the transaction is committed.
This is an app design problem as one should not routinely execute DDL functions in application code. If that design cannot be easily remediated, perform the DDL operations separately in a short transaction (or with utocommit statements without an explict transaction) and handle DDL rollback in code.
You can use this useful store proc which I picked up somewhere along my travels. It recently helped me see the locking on a table and showed that after setting READ UNCOMMITED it was no longer doing row/page/table locks, but still had the schema lock. I believe you may have schema locks if you are modifying them! and also as commented don't keep a transaction open long, in and out is key.
What this does is runs the stored proc every seconds 20 times, so you will get a snapshot of locking, a really useful stored proc to remember.
EXEC [dbo].CheckLocks #Database = 'PriceBook'
WAITFOR DELAY '00:00:01'
GO 20
The stored proc is as follows
/*
This script can be run to find locks at the current time
We can run it as follows:
EXEC [dbo].CheckLocks #Database = 'PriceBook'
WAITFOR DELAY '00:00:01'
GO 10
*/
CREATE OR ALTER PROCEDURE [dbo].[CheckLocks]
#Database NVARCHAR(256)
AS
BEGIN
-- Get the sp_who details
IF object_id('tempdb..#WhoDetails') IS NOT NULL
BEGIN
DROP TABLE #WhoDetails
END
CREATE TABLE #WhoDetails (
[spid] INT,
[ecid] INT,
[status] VARCHAR(30),
[loginame] VARCHAR(128),
[hostname] VARCHAR(128),
[blk] VARCHAR(5),
[dbname] VARCHAR(128),
[cmd] VARCHAR(128),
[request_id] INT
)
INSERT INTO #WhoDetails EXEC sp_who
-- Get the sp_lock details
IF object_id('tempdb..#CheckLocks') IS NOT NULL
BEGIN
DROP TABLE #CheckLocks
END
CREATE TABLE #CheckLocks (
[spid] int,
[dbid] int,
[ObjId] int,
[IndId] int,
[Type] char(4),
[Resource] nchar(32),
[Mode] char(8),
[Status] char(6)
)
INSERT INTO #CheckLocks EXEC sp_lock
SELECT DISTINCT
W.[loginame],
L.[spid],
L.[dbid],
db_name(L.dbid) AS [Database],
L.[ObjId],
object_name(objID) AS [ObjectName],
L.[IndId],
L.[Type],
L.[Resource],
L.[Mode],
L.[Status]--,
--ST.text,
--IB.event_info
FROM #CheckLocks AS L
INNER JOIN #WhoDetails AS W ON W.spid = L.spid
INNER JOIN sys.dm_exec_connections AS EC ON EC.session_id = L.spid
--CROSS APPLY sys.dm_exec_sql_text(EC.most_recent_sql_handle) AS ST
--CROSS APPLY sys.dm_exec_input_buffer(EC.session_id, NULL) AS IB -- get the code that the session of interest last submitted
WHERE L.[dbid] != db_id('tempdb')
AND L.[Type] IN ('PAG', 'EXT', 'TAB')
AND L.[dbid] = db_id(#Database)
/*
https://learn.microsoft.com/en-us/sql/relational-databases/system-stored-procedures/sp-lock-transact-sql?view=sql-server-ver15
Lock modes are as follows
------------------------------
S = Shared
U = Update
X = Exclusive
IS = Indent Shared
IS = Intent Update
IX = Intent Exclusive
Sch-S = Schema Stability lock so no we cant remove tables or indexes in use
Lock Type are as follows:
------------------------------
RID = Single row lock
KEY = Lock within an index that protects a range of keys
PAG = Page level lock
EXT = Extend Lock
TAB = Table Lock
DB = Database lock
*/
END
This is what you might see if you can catch the locking, this was before an after example, left and right.
I'm am working in SharePoint and am trying to understand why there is a limit of 5000 records for views on lists that can have a far greater number.
My question is not about SharePoint, but SQL. Does SQL have the limit described below, or was this something imposed on SQL by the SharePoint team?
For performance reasons, whenever SQL Server executes a single query
that returns 5000 + items, locks escalation happens within the SQL
table. As a result, the entire table will be locked. Since the entire
Share Point data is stored as a single table, a single list view query
that exceeds 5000+ items will lock the entire Share Point data table
within that content. The database and all the users will face huge
performance degradation. The entire set of users using Share Point at
the time of lock escalation will have to wait for a longer time to
retrieve the data. Hence, you can see that the list threshold is a
limitation that is imposed upon Share Point by its backend SQL Server.
This issue is generated from SQL and the reason is row lock
escalation. In order to avoid this performance degradation, Share
Point has imposed the limitation of 5000 items to be queried at any
point of time. Any queries for 5000+ items will be dealt the threshold
error message. Ref link
Thanks
EDIT____________________________________
An article on this issue:
https://www.c-sharpcorner.com/article/sharepoint-list-threshold-issue-the-traditional-problem/
Does SQL Server lock tables when queries return results greater than 5000
records?
Not generally, no.
It is documented that 5,000 is a magic number for the database engine to first attempt lock escalation (followed by further attempts at 1,250 increments) but unless running at repeatable read or serializable isolation level this will not generally be hit just by returning 5,0000 items in a SELECT. The default read committed level will release locks as soon as the data is read so never hit the threshold.
You can see the effect of isolation level on this with the following example.
CREATE TABLE T(C INT PRIMARY KEY);
INSERT INTO T
SELECT TOP 10000 ROW_NUMBER() OVER (ORDER BY ##SPID)
FROM sys.all_objects o1, sys.all_objects o2
And (uses undocumented trace flags so should only be used in dev environment)
DBCC TRACEON(3604,611);
/*5,000 key locks are held*/
SET TRANSACTION ISOLATION LEVEL REPEATABLE READ;
BEGIN TRAN
SELECT COUNT(*) FROM (SELECT TOP 5000 C FROM T) T
SELECT resource_type, request_mode, count(*) FROM sys.dm_tran_locks where request_session_id = ##spid GROUP BY resource_type, request_mode;
COMMIT
/*No key locks are held. They have been escalated to an object level lock. The messages tab shows the lock escalation (in my case after 6248 locks not 5,000)*/
SET TRANSACTION ISOLATION LEVEL REPEATABLE READ;
BEGIN TRAN
SELECT COUNT(*) FROM (SELECT TOP 10000 C FROM T) T
SELECT resource_type, request_mode, count(*) FROM sys.dm_tran_locks where request_session_id = ##spid GROUP BY resource_type, request_mode;
COMMIT
/*No key locks are held. They are released straight away at this isolation level. The messages tab shows no lock escalation messages*/
SET TRANSACTION ISOLATION LEVEL READ COMMITTED;
BEGIN TRAN
SELECT COUNT(*) FROM (SELECT TOP 10000 C FROM T) T
SELECT * FROM sys.dm_tran_locks where request_session_id = ##spid
COMMIT
DBCC TRACEOFF(3604,611);
I have a part of application that constantly updates values in table rows (1-100 rows).
Since this data integrity is important i am using SERIALIZABLE lock on transaction in functions reading and updating those rows.
Now my question is if i execute a simple read only SELECT (without lock) on rows that is currently used by a transaction i could probably get a DEADLOCK exception right?
So would that mean that i would still need a retry mechanism in case of DEADLOCK even in case of simple SELECT?
Now that I know what your specific business scenario is (from the comments), here's how you'd do something like what you're proposing without having to implement serializable isolation
CREATE PROCEDURE [dbo].[updateUserState] (#UserID int)
AS
BEGIN
SET TRANSACTION ISOLATION LEVEL READ COMMITTED;
BEGIN TRANSACTION
SELECT [State]
FROM [dbo].[UserState] WITH (UPDLOCK)
WHERE [UserID] = #UserID;
IF ([State] = 'logged out')
BEGIN
UPDATE [us]
SET [State] = 'logged in'
FROM [dbo].[UserState] AS [us]
WHERE [UserID] = #UserID;
END
COMMIT TRANSACTION
END
Note that this is simplified, but presents the main idea. The UPDLOCK hint on the SELECT statement is the key. It says "try to select data as through I was going to do an update (which you are! just later) and keep it until the end of the transaction". In your example, if T2 comes in and T1 is still running, T2 will be unable to obtain the update lock and will thus wait until T1 is complete (either successfully or no). Also note that setting the transaction isolation level explicitly is just for completeness; READ COMMITTED is the default in SQL Server.
I have a table ImportSourceMetadata which I use to control an import batch process. It contains a PK column SourceId and a data column LastCheckpoint. The import batch process reads the LastCheckpoint for a given SourceId, performs some logic (on other tables), then updates the LastCheckpoint for that SourceId or inserts it if it doesn't exist yet.
Multiple instances of the process run at the same time, usually with disjunct SourceIds, and I need high parallelity for those cases. However, it can happen that two processes are started for the same SourceId; in that case, I need the instances to block each other.
Therefore, my code looks as follows:
BEGIN TRAN
SET TRANSACTION ISOLATION LEVEL READ COMMITTED
SELECT LastCheckpoint FROM ImportSourceMetadata WITH (UPDLOCK) WHERE SourceId = 'Source'
-- Perform some processing
-- UPSERT: if the SELECT above yielded no value, then
INSERT INTO ImportSourceMetadata(SourceId, LastCheckpoint) VALUES ('Source', '2013-12-21')
-- otherwise, we'd do this: UPDATE ImportSourceMetadata SET LastCheckpoint = '2013-12-21' WHERE SourceId = 'Source'
COMMIT TRAN
I'm using the transaction to achieve atomicity, but I can only use READ COMMITTED isolation level (because of the parallelity requirements in the "Perform some processing" block). Therefore (and to avoid deadlocks), I'm including an UPDLOCK hint with the SELECT statement to achieve a "critical section" parameterized on the SourceIdvalue.
Now, this works quite well most of the time, but I've managed to trigger primary key violation errors with the INSERT statement when starting a lot of parallel processes for the same SourceIdwith an empty database. I cannot reliably reproduce this, however, and I don't understand why it doesn't work.
I've found hints on the internet (e.g., here and here, in a comment) that I need to specify WITH (UPDLOCK,HOLDLOCK) (resp. WITH (UPDLOCK,SERIALIZABLE)) rather than just taking an UPDLOCK on the SELECT, but I don't really understand why that is. MSDN docs say,
UPDLOCK
Specifies that update locks are to be taken and held until the transaction completes.
An update lock that is taken and held until the transaction completes should be enough to block a subsequent INSERT, and in fact, when I try it out in SQL Server Management Studio, it does indeed block my insert. However, in some rare cases, it seems to suddenly not work any more.
So, why exactly is it that UPDLOCK is not enough, and why is it enough in 99% of my test runs (and when simulating it in SQL Server Management Studio)?
Update: I've now found I can reproduce the non-blocking behavior reliably by executing the code above in two different windows of SQL Server Management Studio simultaneously up to just before the INSERT, but only the first time after creating the database. After that (even though I deleted the contents of the ImportSourceMetadata table), the SELECT WITH (UPDLOCK) will indeed block and the code no longer fails. Indeed, in sys.dm_tran_locks, I can see a U-lock taken even though the row does not exist on subsequent test runs, but not on the first run after creating the table.
This is a complete sample to show the difference in locks between a "newly created table" and an "old table":
DROP TABLE ImportSourceMetadata
CREATE TABLE ImportSourceMetadata(SourceId nvarchar(50) PRIMARY KEY, LastCheckpoint datetime)
BEGIN TRAN
SET TRANSACTION ISOLATION LEVEL READ COMMITTED
SELECT LastCheckpoint FROM ImportSourceMetadata WITH (UPDLOCK) WHERE SourceId='Source'
SELECT *
FROM sys.dm_tran_locks l
JOIN sys.partitions p
ON l.resource_associated_entity_id = p.hobt_id JOIN sys.objects o
ON p.object_id = o.object_id
INSERT INTO ImportSourceMetadata VALUES('Source', '2013-12-21')
ROLLBACK TRAN
BEGIN TRAN
SET TRANSACTION ISOLATION LEVEL READ COMMITTED
SELECT LastCheckpoint FROM ImportSourceMetadata WITH (UPDLOCK) WHERE SourceId='Source'
SELECT *
FROM sys.dm_tran_locks l
JOIN sys.partitions p
ON l.resource_associated_entity_id = p.hobt_id JOIN sys.objects o
ON p.object_id = o.object_id
ROLLBACK TRAN
On my system (with SQL Server 2012), the first query shows no locks on ImportSourceMetadata, but the second query shows a KEY lock on ImportSourceMetadata.
In other words, HOLDLOCK is indeed required, but only if the table was freshly created. Why's that?
You also need HOLDLOCK.
If the row does exist then your SELECT statement will take out a U lock on at least that row and retain it until the end of the transaction.
If the row doesn't exist there is no row to take and hold a U lock in so you aren't locking anything. HOLDLOCK will lock at least the range where the row would fit in.
Without HOLDLOCK two concurrent transactions can both do the SELECT for a non existent row. Retain no conflicting locks and both move onto the INSERT.
Regarding the repro in your question it seems the "row doesn't exist" issue is a bit more complex than I first thought.
If the row previously did exist but has since been logically deleted but still physically exists on the page as a "ghost" record then the U lock can still be taken out on the ghost explaining the blocking that you are seeing.
You can use DBCC PAGE to see ghost records as in this slight amend to your code.
SET NOCOUNT ON;
DROP TABLE ImportSourceMetadata
CREATE TABLE ImportSourceMetadata
(
SourceId NVARCHAR(50),
LastCheckpoint DATETIME,
PRIMARY KEY(SourceId)
)
BEGIN TRAN
SET TRANSACTION ISOLATION LEVEL READ COMMITTED
SELECT LastCheckpoint
FROM ImportSourceMetadata WITH (UPDLOCK)
WHERE SourceId = 'Source'
INSERT INTO ImportSourceMetadata
VALUES ('Source', '2013-12-21')
DECLARE #DBCCPAGE NVARCHAR(100)
SELECT TOP 1 #DBCCPAGE = 'DBCC PAGE(0,' + CAST(file_id AS VARCHAR) + ',' + CAST(page_id AS VARCHAR) + ',3) WITH NO_INFOMSGS'
FROM ImportSourceMetadata
CROSS APPLY sys.fn_physloccracker(%%physloc%%)
ROLLBACK TRAN
DBCC TRACEON(3604)
EXEC (#DBCCPAGE)
DBCC TRACEOFF(3604)
The SSMS messages tab shows
Slot 0 Offset 0x60 Length 31
Record Type = GHOST_DATA_RECORD Record Attributes = NULL_BITMAP VARIABLE_COLUMNS
Record Size = 31
Memory Dump #0x000000001215A060
0000000000000000: 3c000c00 00000000 9ba20000 02000001 †<.......¢......
0000000000000010: 001f0053 006f0075 00720063 006500††††...S.o.u.r.c.e.
Slot 0 Column 1 Offset 0x13 Length 12 Length (physical) 12
I have upgraded from SQL Server 2005 to 2008. I remember that in 2005, ROWLOCK simply did not work and I had to use PAGELOCK or XLOCK to achieve any type of actual locking. I know a reader of this will ask "what did you do wrong?" Nothing. I conclusively proved that I could edit a "ROWLOCKED" row, but couldn't if I escalated the lock level. I haven't had a chance to see if this works in SQL 2008. My first question is has anyone come across this issue in 2008?
My second question is as follows. I want to test if a value exists and if so, perform an update on relevant columns, rather than an insert of the whole row. This means that if the row is found it needs to be locked as a maintenance procedure could delete this row mid-process, causing an error.
To illustrate the principle, will the following code work?
BEGIN TRAN
SELECT ProfileID
FROM dbo.UseSessions
WITH (ROWLOCK)
WHERE (ProfileID = #ProfileID)
OPTION (OPTIMIZE FOR (#ProfileID UNKNOWN))
if ##ROWCOUNT = 0 begin
INSERT INTO dbo.UserSessions (ProfileID, SessionID)
VALUES (#ProfileID, #SessionID)
end else begin
UPDATE dbo.UserSessions
SET SessionID = #SessionID, Created = GETDATE()
WHERE (ProfileID = #ProfileID)
end
COMMIT TRAN
An explanation...
ROWLOCK/PAGELOCK is granularity
XLOCK is mode
Granularity and isolation level and mode are orthogonal.
Granularity = what is locked = row, page, table (PAGLOCK, ROWLOCK, TABLOCK)
Isolation Level = lock duration, concurrency (HOLDLOCK, READCOMMITTED, REPEATABLEREAD, SERIALIZABLE)
Mode = sharing/exclusivity (UPDLOCK, XLOCK)
"combined" eg NOLOCK, TABLOCKX
XLOCK would have locked the row exclusively as you want. ROWLOCK/PAGELOCK wouldn't have.