We have a large multi-use application suffering from OBJECT and PAGE blocks on a number of tables. We are unable to reconsider the design, but need to reduce blocks that affect performance on client machines.
I have been considering disabling lock escalations on one table but need to know what pressure it will put on other resources. The disk i/o is already strained. Will additional, individual locks require more i/o than the automatic table locks? Will it affect our system databases more than our application db?
We don't do full table updates/reads. Each request will only deal with a very small portion of the table.
We have a large multi-use application suffering from OBJECT and PAGE
blocks on a number of tables.
...
The disk i/o is already strained.
Will additional, individual locks require more i/o than the automatic
table locks?
You misunderstand lock escalation, this is clear from the parts of your question that I made bold.
Lock escalation goes from rows to table or from pages to table (I excluded partition as it's not your case), so if now you have page locks it's NOT lock escalation.
Lock granularity is choosen by server unless you use hints (rowlock, paglock) and if it choses page locks there is no escalation. If it then removes all the page locks and substitutes them with table lock, it means lock escalation occured.
The second thing that is wrong is your thinking that locks have to do with IO. This is not true. Locks are held in memory and have nothing to do with reads. You can check this article to see how CPU usage and query duration increase when the locks are more granular: Why ROWLOCK Hints Can Make Queries Slower and Blocking Worse in SQL Server.
You should understand what causes your lock escalation.
Lock Escalation Thresholds
Lock escalation is triggered when lock escalation is not disabled on
the table by using the ALTER TABLE SET LOCK_ESCALATION option, and
when either of the following conditions exists:
A single Transact-SQL statement acquires at least 5,000 locks on a single nonpartitioned table or index.
A single Transact-SQL statement acquires at least 5,000 locks on a single partition of a partitioned table and the
ALTER TABLE SET LOCK_ESCALATION option is set to AUTO.
The number of locks in an instance of the Database Engine exceeds memory or configuration thresholds.
Lock Escalation (Database Engine)
So if you reach 5,000 locks per statement threshold you should split your operations to smaller batches.
And if you are under memory pressure, disabling lock escalation will make your situation even worse.
UPDATE
I've found this description of locks in the book Microsoft SQL Server 2012 Internals (Developer Reference)by Kalen Delaney (Author), Bob Beauchemin (Author), Conor Cunningham (Author), Jonathan Kehayias (Author), Paul S. Randal (Author), Benjamin Nevarez (Author
Locks aren’t on-disk structures. You won’t find a lock field directly
on a data page or a table header, and the metadata that keeps track of
locks is never written to disk. Locks are internal memory structures:
They consume part of the memory used for SQL Server. A lock is
identified by lock resource, which is a description of the resource
that’s locked (a row, index key, page, or table). To keep track of the
database, the type of lock, and the information describing the locked
resource, each lock requires 64 bytes of memory on a 32-bit system and
128 bytes of memory on a 64-bit system. This 64-byte or 128-byte
structure is called a lock block.
... The lock manager maintains a lock hash table. Lock resources,
contained within a lock block, are hashed to determine a target hash
slot in the hash table. All lock blocks that hash to the same slot are
chained together from one entry in the hash table. Each lock block
contains a 15-byte field that describes the locked resource. The lock
block also contains pointers to lists of lock owner blocks. Each of
the three states has a separate list for lock owners.
Hope it helps.
Related
The PostgreSQL documentation says:
PostgreSQL doesn't remember any information about modified rows in memory, so there is no limit on the number of rows locked at one time. However, locking a row might cause a disk write, e.g., SELECT FOR UPDATE modifies selected rows to mark them locked, and so will result in disk writes.
Does any other of the row-level locking modes can also write to the disk?
What are the conditions that make each type of row-level lock access the disk?
I was wondering why I would need to use lock mode page on a table.
Recently I came up to a pretty good case of why not. While I was trying to insert a row on a table I got a deadlock. After lots of investigation I figured out the the lock level of my table was Page and this was the actual reason that lead to the deadlock.
My guess is that this is a common scenario on large scale high performance environments with multiple applications hitting the same db
The only thing I found is that I should use page locking if I am processing rows in the same order as the paging occurs. This looks like a weak condition that can seldom be met (especially for scaling which could render this case obsolete).
I can see why one would lock a full table or use per row locking but the Page locking does not make much sense. Or does it?
You never need to use LOCK MODE PAGE on a table, but you may choose to do so.
It does no damage whatsoever if only a single row fits on a page (or a single row requires more than one page).
If you can fit multiple rows on a page, though, you have a meaningful choice between LOCK MODE PAGE and LOCK MODE ROW. Clearly, if you use LOCK MODE ROW, then the fact that one process has a lock on one row of a page won't prevent another process from gaining a lock on a different row on the same page, whereas LOCK MODE PAGE will prevent that.
The advantage of LOCK MODE PAGE is that it requires less locks when a single process updates multiple rows on a page in a single transaction.
So, you have to do a balancing act. You can take the view that there are so many rows in the database that the chances of two processes needing to lock different rows on the same page is negligible, and use LOCK MODE PAGE knowing that there's a small risk that you'll have processes blocking other processes that would not be blocked if you used LOCK MODE ROW. Alternatively, you can take the view that the risk of such blocking is unacceptable and the increased number of locks is not a problem, and decide to use LOCK MODE ROW anyway.
Historically, when the number of locks was a problem because memory was scarce (in the days when big machines had less than a 100 MiB of main memory!), saving locks by using LOCK MODE PAGE made more sense than it does now when systems have multiple gigabytes of main memory.
Note that it doesn't matter which lock mode you use if two processes want to update the same row; one will get a lock and block the other until the transaction commits (or until the statement completes if you aren't using explicit transactions).
Note that the default lock mode is still LOCK MODE PAGE, mainly in deference to history where that has always been the case. However, there is an ONCONFIG parameter, DEF_TABLE_LOCKMODE, that you can set to row (instead of page) that will set the default table lock mode to LOCK MODE ROW. You can still override that explicitly in a DDL statement, but if you don't specify an explicit lock mode, the default will be row or page depending on the setting of DEF_TABLE_LOCKMODE.
I'm getting very poor performance on a stored procedure that does well over a hundred (!) updates to a couple of very small tables and it appears that all the concurrent users are constantly blocking each other.
A full rewrite of the proc is scheduled for later this year, but in the meantime I wanted to see if we can alleviate the problem by forcing row-level locking on each of the affected tables.
On Sybase, you can (or at least could circa 2007) force row-level locking for a table using this statement:
alter table titles lock datarows
On SQL Server, it appears that the only way to get the same effect, is to use WITH (ROWLOCK) on every update or insert statement. And even then, it's only a hint that may be ignored.
Is there a way in SQL Server to force (or strongly favour) row-level locking for all updates to a given table?
First and foremost make sure the updates are not table scans. In other words you do have the proper indexes (UPDATE needs indexes too...). After you ensure that, upon careful consideration, disable page locks on the index used:
ALTER INDEX ... WITH (ALLOW_PAGE_LOCKS = OFF, ALLOW_ROW_LOCKS = ON);
Give this operation the same care you would give in joggling a barrel of TNT and a bottle of Napalm...
Additional information (from the comments below):
You can disable page locks on the clustered index, but you cannot disable page-locks on heaps (since they're physically structured and would not be possible to do proper lock hierarchy w/o locking the page).
Lock escalation is a (related, but) different subject. Locks escalation only comes into picture if the statement chooses row-level lock granularity and it decides to escalate during execution to rowset granularity level. Is true that the OP may be in fact victim of escalation, from how I read the OP I think the more likely cause is just lack of an index (ie. a high lock granularity is chosen upfront because of the scan, no escalation is triggered).
The initial granularity level is the result of the engine estimation of the task at hand. If the estimate indicates a large number of rows will have to be locked then it may opt for page granularity instead, since acquiring a large number of row locks is usually problematic. Lack of an index will trigger a scan, which will usually choose page granularity.
Also escalation is from row/page to rowset (object) granularity. Going through the intermediate page level first is riddled with concurrency problems so the 'big-hammer' is used.
You can use this to disable lock escalation:
ALTER TABLE titles SET (LOCK_ESCALATION=DISABLE)
See the docs about lock escalation.
trace flag 1211
turn off page level locking via alter index
combo of those 2 will give you row level locking. unless of course sql will decide to go directly to table locking :)
there is also 1224 flag, but it will ignore your request being under memory pressure, 1211 won't. memory consumption might increase big time, be warned and do some user activity simulation (RML or Benchmark Factory) and run perfmon to observe. I know people that have done it in the past.
This is really more of a discussion than a specific question about nolock.
I took over an app recently that almost every query (and there are lots of them) has the nolock option on them. Now I am pretty new to SQL server (used Oracle for 10 years) but yet I find this pretty disturbing. So this weekend I was talking with one of my friends who runs a rather large ecommerce site (name will be withheld to protect the guilty) and he says he has to do this with all of his SQL servers cause he will always end in deadlocks.
Is this just a huge short fall with SQL server? Is this just a failure in the DB design (mine is not 3rd level, but its close) Is anybody out there running an SQL server app without nolocks? These are issues that Oracle handles better with more grandulare recordlocks.
Is SQL server just not able to handle big loads? Is there some better workaround than reading uncommited data? I would love to hear what people think.
Thanks
SQL Server has added snapshot isolation in SQL Server 2005, this will enable you to still read the latest correct value without having to wait for locks. StackOverflow is also using Snapshot Isolation. The Snapshot Isolation level is more or less the same that Oracle uses, this is why deadlocks are not very common on an Oracle box. Just be aware to have plenty of tempdb space if you do enable it
from Books On Line
When the READ_COMMITTED_SNAPSHOT
database option is set ON, read
committed isolation uses row
versioning to provide statement-level
read consistency. Read operations
require only SCH-S table level locks
and no page or row locks. When the
READ_COMMITTED_SNAPSHOT database
option is set OFF, which is the
default setting, read committed
isolation behaves as it did in earlier
versions of SQL Server. Both
implementations meet the ANSI
definition of read committed
isolation.
If somebody says that without NOLOCK their application always gets deadlocked, then there is (more than likely) a problem with their queries. A deadlock means that two transactions cannot proceed because of resource contention and the problem cannot be resolved. An example:
Consider Transactions A and B. Both are in-flight. Transaction A has inserted a row into table X and Transaction B has inserted a row into table Y, so Transaction A has an exclusive lock on X and Transaction B has an exclusive lock on Y.
Now, Transaction A needs run a SELECT against table Y and Transaction B needs to run a SELECT against table X.
The two transactions are deadlocked: A needs resource Y and B needs resource X. Since neither transaction can proceed until the other completes, the situtation cannot be resolved: neither transactions demand for a resource may be satisified until the other transaction releases its lock on the resource in contention (either by ROLLBACK or COMMIT, doesn't matter.)
SQL Server identifies this situation and select one transaction or the other as the deadlock victim, aborts that transaction and rolls back, leaving the other transaction free to proceed to its presumable completion.
Deadlocks are rare in real life (IMHO). One rectifies them by
ensuring that transaction scope is as small as possible, something SQL server does automatically (SQL Server's default transaction scope is a single statement with an implicit COMMIT), and
ensuring that transactions access resources in the same sequence. In the example above, if transactions A and B both locked resources X and Y in the same sequence, there would not be a deadlock.
Timeouts
A timeout, on the other hand, occurs when a transaction exceeds its wait time and is rolled back due to resource contention. For instance, Transaction A needs resource X. Resource X is locked by Transaction B, so Transaction A waits for the lock to be released. If the lock isn't released within the queries timeout limimt, the waiting transaction is aborted and rolled back. Every query has a query timeout associated with it (the default value is 30s, I believe), after which time the transaction is aborted and rolled back. The query timeout can be set to 0s, in which case SQL Server will let the query wait forever.
This is probably what they are talking about. In my experience, timeouts like this usually occur in big databases when large batch jobs are updating thousands and thousands of records in a single transaction, although they can happen because a transaction goes to long (connect to your production database in Query Abalyzer, execute BEGIN TRANSACTION, update a single row in a frequently hit table in Query Analyzer and go to lunch without executing ROLLBACK or COMMIT TRANSACTION and see how long it takes for the production DBAs to go apes**t on you. Don't ask me how I know this)
This sort of timeout is usually what results in splattering perfectly innocent SQL with all sorts of NOLOCK hints
[TIP: if your going to do that, just execute SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED as the first statement in your stored procedure and have done with it.]
The problem with this approach (NOLOCK/READ UNCOMMITTED) is that you can read uncommitted data from other transaction: stuff that is incomplete or that may get rolled back later, so your data integrity is comprimised. You might be sending out a bill based on data with a high level of bogosity.
My general rule is that one should avoid the use of table hints insofar as possible. Let SQL Server and its query optimizer do their jobs.
The right way to avoid this sort of issue is to avoid the sort of transactions (insert a million rows all at one fell swoop, for instance) that cause the problems. The locking strategy implicit in relational database SQL is designed around small transactions of short scope. Lock should be small in scope and short in duration. Think "bank teller updating somebody's checking account with a deposit." as the underlying use case. Design your processes to work in that model and you'll be much happier all the way 'round.
Instead of inserting a million rows in one mondo insert statement, do the work in independent chunks and commit each chunk independently. If your million row insert dies after processing 999,000 rows, all the work done is lost (not to mention that the rollback can be a b*tch, and the table is still locked during rollback as well.) If you insert the million rows in block of 1000 rows each, committing after each block, you avoid the lock contention that causes deadlocks, as locks will be obtained and released and things will keep moving. If something goes south in the 999th block of 1000 rows, and the transaction get aborted and rolled back, you've still gotten 998,000 rows inserted; you've only lost 1000 rows of work. Restart/Retry is much easier.
Also, lock escalation occurs in large transactions. For effiency, locks escalate to larger and larger scope as the number of locks held by transaction increases. If a single transaction inserts/updates/deletes a single row in a table, I get a row lock. Keep doing that and once the number of row locks held by that transaction against that table hits a threshold value, SQL Server will escalate the locking strategy: the row locks will be consolidated and converted into a smaller number page locks, thus increasing the scope of the locks held. From that point forward, an insert/delete/update of a single row will lock that page in the table. Once the number of page locks held hits its threshold value, the page locks are again consolidated and the locking strategy escalates to table locks: the transaction now locks the entire table and nobody else may play until the transaction commits or rolls back.
Whether you can avoid functionally avoid the use of NOLOCK/READ UNCOMMITTED is entirely dependent on the nature of the processes hitting the underlying database (and the culture of the organization owning it).
Myself, I try to avoid its use as much as possible.
Hope this helps.
No, there is no need to use NOLOCK. Links: SO 1
As for load, we deal with 2000 rows per second which is small change compared to 35k TPS
Deadlocks are caused by lock contention and usually caused by inconsistent write order on tables in transactions. ORMs especially are rubbish at this. We get them very infrequently. A well written DAL should retry too as per MSDN.
In a traditional normalized OLTP environment, NOLOCK is a code smell and almost certainly unnecessary in a properly designed system.
In a dimensional model, I used NOLOCK extensively to avoid locking very large fact and dimension tables which were being populated with later fact data (and dimensions may have been expiring). In the dimensional model, the facts either never change or never change after a certain point. Similarly, any dimension which is referenced will also be static, so for example, the NOLOCK will stop your long analysis operation on yesterday's data from blocking a dimension expiration during a data load for today's data.
You should only use nolock on an unchanging table. Of course, this will be the same then as Read Committed Snapshot. Without the snapshot, you are only saving the time it takes to apply a shared lock, and then to remove it, which for most cases isn't necessary.
As for a changing table... No lock doesn't just mean getting a row before a transaction is done updating all of its rows. You can get ghost data as data pages split, or even index pages split. Or no data. That alone scared me away, but I think there may be even more scenarios where you simply get the wrong data.
Of course, nolock for getting rough estimates or to just check in on a process might be reasonable.
Basic rule of thumb -- if you care about the data at all, and the data is changing, then do not use NoLOCK.
I have two long running queries that are both on transactions and access the same table but completely separate rows in those tables. These queries also perform some update and inserts based on those queries.
It appears that when these run concurrently that they encounter a lock of some kind and it’s preventing the task from finishing and locks up when it goes to update one of the rows. I’m using an exclusive row lock on the rows being read and the lock that shows up on the process is a lck_m_ix lock.
Two questions:
When I update/insert a single row does it lock the entire table?
What can be done to work around this sort of issue?
Typically no, but it depends (most often used answer for SQL Server!)
SQL Server will have to lock the data involved in a transaction in some way. It has to lock the data in the table itself, and the data any affected indexes, while you perform a modification. In order to improve concurrency, there are several "granularities" of locking that the server might decide to use, in order to allow multiple processes to run: row locks, page locks, and table locks are common (there are more). Which scale of locking is in play depends on how the server decides to execute a given update. Complicating things, there are also classifications of locks like shared, exclusive, and intent exclusive, that control whether the locked object can be read and/or modified.
It's been my experience that SQL Server mainly uses page locks for changes to small portions of tables, and past some threshold will automatically escalate to a table lock, if a larger portion of a table seems (from stats) to be affected by an update or delete. The idea is that it is faster to lock a table (one lock) than obtaining and managing thousands of individual row or page locks for a big update.
To see what is happening in your specific case, you'd need to look at the query logic and, while your stuff is running, examine the locking/blocking conditions in sys.dm_tran_locks, sys.dm_os_waiting_tasks or other DMV's. You would want to discover what exactly is getting locked by what step in each of your processes, to discover why one is blocking the other.
The short version:
No
Fix your code.
The long version:
LCK_M_IX is an intent lock, meaning the operation will place an X lock on a subordinate element. Eg. When updating a row in a table, the operation table takes an IX lock on the table before locking X the row being updated/inserted/deleted. Intent locks are common strategy to deal with hierarchies, like table/page/row, because the lock manager cannot understand the physical structure of resources requested to be locked (ie. it cannot know that an X-lock on page P1 is incompatible with an S-lock on row R1 because R1 is contained in P1). For more details, see Lock Modes.
The fact that you are seeing contention on intent locks means you are trying to obtain high level object locks, like table locks. You will need to analyze your source code for the request being blocked (the one requesting the lock incompatible with LCK_M_IX) and remove the cause of the object level lock request. What that means will depend on your source code, I cannot know what you're doing there. My guess is that you use an erroneous lock hint.
A more general approach is to rely on SNAPSHOT ISOLATION. But this, most likely, will not solve the problem you're seeing, since snapshot isolation can only benefit row level contention issues, not applications that request table locks.
A frequent aim of using transactions: keep them as short and sweet as possible. I get the sense from your wording in the question that you are opening a transaction, then doing all kinds of things, some of which take a long time. Then expecting multiple users to be able to run this same code concurrently. Unfortunately, if you perform an insert at the beginning of that set of code, then do 40 other things before committing or rolling back, it is possible that that insert will block everyone else from running the same type of insert, essentially turning your operation from free-for-all to serial.
Find out what each query is doing, and if you are getting lock escalations that you wouldn't expect. Just because you say WITH (ROWLOCK) on a query doesn't mean SQL Server will be able to comply... if you are touched multiple indexes, indexed views, persisted computed columns etc. then there are all kinds of reasons why your rowlock may not hold any water. You also might have things later in the transaction that are taking longer than you think, and maybe you don't realize that the locks on all of the objects involved in the transaction (not just the statement that is currently running) can be held for the duration of the transaction.
Different databases have different locking mechanisms, but ones like SQL Server and Oracle have different types of locking.
The default on SQL Server appears to be pessimistic Page locking - so if you have a small number of records then all of them may get locked.
Most databases should not lock when running a script, so I'm wondering whether you're potentially running multiple queries concurrently without transactions.