We are trying to resolve a deadlock problem. The transaction that is getting rolled back is attempting to issue an Update (U) lock on a resource that another transaction has an Exclusive (X) lock on. According to Books Online (http://msdn.microsoft.com/en-us/library/ms175519.aspx), an Update lock is supposed to prevent deadlocks, not cause them.
So, my question is, why/when is an Update lock applied to a resource? We're a little confused about this because the resource that is attempting to have the Update lock applied to will not be updated by the process that is having the transaction rolled back.
Thanks for your help on this.
Randy
You are going to need to do a bit more research to find out what is actually locking, what isolation level each query is in, etc.
Some helpful resources.
SQL Server Transaction Isolation Levels and their Locks
SQL Server Lock Types and Lock Hints
There's a whole universe of "what if" behind what causes deadlocks (by which I mean, there's no way to tell from your initial post what's really going on). Could be table locks, could be locks on indexes; could be oustanding transactions you are not aware of; could be table header locks, could be tempdb issues (very unlikely), who knows?
The best method I've ever found to diagnose deadlocks works like so:
Fire up SQL Profiler, configure it with the "Deadlock Graph" and "Lock: Deadlock" events, and be sure to include the TextData column
While Profiler is running, cause your application to generate the deadlock
Select the "Deadlock Graph" profiler row, and you'll get a simple yet confusing graphic display of what's going on. This might help you figure out what's really going on.
If that doesn't help, the graphic is generated by Profiler based on a very detailed lump of XML. Extract this string (select the Deadlock Graph row, Ctrl+C to copy, paste in your text editor of choice, and delete all the columns but the XML), and then review the XML (in your preferred XML editor).
To-date, once I've gotten and worked through that XML, I've always been able to figure out what was causing the deadlock. It's a good way to learn how weird and convoluted some of SQL's internals can get.
Probably you do not handle your transaction isolation properly and are in serialized transaction mode.
You should
Use the proper transaction isolation on the connection.
Sometimes use hints on SQL, like the NoLock hint when you basically just read some data and don't do anything else with it anyway in a transaction.
Related
An app I am working on has to handle lots of ajax requests that needs to update some data on DB.
[Macromedia][SQLServer JDBC Driver][SQLServer]Transaction (Process ID
66) was deadlocked on lock | communication buffer resources with
another process and has been chosen as the deadlock victim. Rerun the
transaction.
For reads, I've already used the WITH (NOLOCK) hint and that prevented a lot of deadlocks on reads.
What can I do to better deal with writes?
CFLock the update code in CF?
Or is there a way to ask SQL Server to lock a Row instead of a Table?
Have anyone tried implementing CQRS? Seems to solve the problem but I am not clear on how to handle:
ID generation (right now it uses Auto Increment on DB)
How to deal with update request fail if server couldn't send the error back to the client right away.
Thanks
Here are my thoughts on this.
From the ColdFusion server side
I do believe that using named <cflock> tags around your ColdFusion code that updates the database could prevent the deadlock issue on the database server. Using a named lock would make each call single threaded. However, you could run into timeouts on the ColdFusion server side, waiting for the <cflock>, if transactions are taking a while. Handling it in this way on the ColdFusion server side may also slow down your application. You could do some load testing before and after to see how this method affects your app.
From the database server side
First of all let me say that I don't think deadlocks on the database server can be entirely prevented, just minimized and handled appropriately. I found this reference on TechNet for you - Minimizing Deadlocks. The first sentence from that page:
Although deadlocks cannot be completely avoided, following certain coding conventions can minimize the chance of generating a deadlock.
Here are the key points from that reference. They go into a bit more detail about each topic so please read the original source.
Minimizing deadlocks can increase transaction throughput and reduce system overhead because fewer transactions are:
Rolled back, undoing all the work performed by the transaction.
Resubmitted by applications because they were rolled back when deadlocked.
To help minimize deadlocks:
Access objects in the same order.
Avoid user interaction in transactions.
Keep transactions short and in one batch.
Use a lower isolation level.
Use a row versioning-based isolation level.
Set READ_COMMITTED_SNAPSHOT database option ON to enable read-committed transactions to use row versioning.
Use snapshot isolation.
Use bound connections.
The "row versioning-based isolation level" may answer your question Or is there a way to ask SQL Server to lock a Row instead of a Table?. There are some notes mentioned in the original source regarding this option.
Here are some other references that came up during my search:
Avoiding deadlock by using NOLOCK hint
How to avoid sql deadlock?
Tips to avoid deadlocks? - This one mentions being careful when using the NOLOCK hint.
The Difficulty with Deadlocks
Using Row Versioning-based Isolation Levels
I have a process with a Select which takes a long time to finish, on the order of 5 to 10 minutes. I am currently not using NOLOCK as a hint to the MS SQL database engine.At the same time we have another process doing updates and inserts into the same database and same tables. The first process has started, recently to end prematurely with a message
SQLEXCEPTION: Transaction was deadlocked on lock resources with another process and has been chosen as the deadlock victim.
This first process is running at other sites in identical conditions but with smaller databases and thus the select statement in question takes a much shorter period of time (on the order of 30 seconds or so). In these other sites, I don't get the deadlock message in these other sites. I also did not get this message at the site that is having the problem initially, but, I assume, as the database has grown, I believe I must have crossed some threshold. Here are my questions:
Could the time it takes for a transaction to execute make the associated process more likely to be flagged as a deadlock victim.
If I execute the select with a NOLOCK hint, will this remove the problem?
I suspect that a datetime field that is checked as part of the WHERE clause in the select statement is causing the slow lookup time. Can I create an index based on this field? Is it advisable?
Q1:Could the time it takes for a transaction to execute make the associated process more likely to be flagged as a deadlock victim.
No. The SELECT is the victim because it had only read data, therefore the transaction has a lower cost associated with it so is chosen as the victim:
By default, the Database Engine chooses as the deadlock victim the
session running the transaction that is least expensive to roll back.
Alternatively, a user can specify the priority of sessions in a
deadlock situation using the SET DEADLOCK_PRIORITY statement.
DEADLOCK_PRIORITY can be set to LOW, NORMAL, or HIGH, or alternatively
can be set to any integer value in the range (-10 to 10).
Q2. If I execute the select with a NOLOCK hint, will this remove the problem?
No. For several reasons:
you should first try to eliminate the deadlock properly, by investigating the root cause
dirty reads are inconsistent reads.
the proper way to specify dirty reads is to use transaction isolation levels
there is a much better solution: read committed snapshot.
Q3. I suspect that a datetime field that is checked as part of the WHERE clause in the select statement is causing the slow lookup time. Can I create an index based on this field? Is it advisable?
Probably. The cause of the deadlock is almost very likely to be a poorly indexed database.10 minutes queries are acceptable in such narrow conditions, that I'm 100% certain in your case is not acceptable.
With 99% confidence I declare that your deadlock is cased by a large table scan conflicting with updates. Start by capturing the deadlock graph to analyze the cause. You will very likely have to optimize the schema of your database. Before you do any modification, read this topic Designing Indexes and the sub-articles.
Here is how this particular deadlock problem actually occurred and how it was actually resolved. This is a fairly active database with 130K transactions occurring daily. The indexes in the tables in this database were originally clustered. The client requested us to make the indexes nonclustered. As soon as we did, the deadlocking began. When we reestablished the indexes as clustered, the deadlocking stopped.
The answers here are worth a try, but you should also review your code. Specifically have a read of Polyfun's answer here:
How to get rid of deadlock in a SQL Server 2005 and C# application?
It explains the concurrency issue, and how the usage of "with (updlock)" in your queries might correct your deadlock situation - depending really on exactly what your code is doing. If your code does follow this pattern, this is likely a better fix to make, before resorting to dirty reads, etc.
Although #Remus Rusanu's is already an excelent answer, in case one is looking forward a better insight on SQL Server's Deadlock causes and trace strategies, I would suggest you to read Brad McGehee's How to Track Down Deadlocks Using SQL Server 2005 Profiler
When you create a SQL Server transaction, what gets locked if you do not specify a table hint? In order to lock something, must you always use a table hint? Can you lock rows/tables outside of transactions (i.e. in ordinary queries)? I understand the concept of locking and why you'd want to use it, I'm just not sure about how to implement it in SQL Server, any advice appreciated.
You should use query hints only as a last resort, and even then only after expert analysis. In some cases, they will cause a query to perform badly. So, unless you really know what you are doing, avoid using query hints.
Locking (of various types) happens automatically everytime you perform a query (unless NOLOCK is specified). The default Transaction Isolation level is READ COMMITTED
What are you actually trying to do?
Understanding Locking in SQL Server
"Can you lock rows/tables outside of transactions (i.e. in ordinary queries)?"
You'd better understand that there are no ordinary queries or actions in SQL Server, they are ALL, without any exceptions, transactional. This is how ACID-ness is achieved, see, for ex., [1]. If client tools or developer interactively do not specify transaction explicitly with BEGIN TRANSACTION and COMMIT/ROLLBACK, then implicit transactions are used.
Also, transaction is not synonym of locking/locks engagement. There is a plethora of mechanisms to control concurrency without locking (for example, versioning. etc.) as well as READ UNCOMMITTED transaction "isolation" (in this case, absence of any isolation) level does not control it at all.
Update2:
In order to lock something, must you always use a table hint?
As far as, transaction isolation level is not READ UNCOMMITTED or one of row-versioning (snapshot) isolation levels, for ex., default READ COMMITTED or set by, for ex.,
SET TRANSACTION ISOLATION LEVEL SERIALIZABLE
the locks are issued (I do not know where to start and how to end this topic[2]). Table hints, which can be used in statements, override these settings.
[1]
Paul S. Randal. Understanding Logging and Recovery in SQL Server
What is Logging?
http://technet.microsoft.com/en-us/magazine/2009.02.logging.aspx#id0060003
[2]
Insert trailing ) upon clicking this link
http://en.wikipedia.org/wiki/Isolation_(database_systems)
Update:
5 min ago I had reputation 784 (the same as 24h ago) and, now, without any visible downvotes, it dropped to 779.
Where can I ask this question if I am banned from meta.stackoverflow.com?
I'm writing some logging/auditing code that will be running in production (not just when errors are thrown or while developing). After reading Coding Horror's experiences with dead-locking and logging, I decided I should seek advice. (Jeff's solution of "not logging" won't work for me, this is legally mandated security auditing)
Is there an suitable Isolation level for minimizing contention and dead-locking? Any query hints I can add to the insert statement or the stored procedure?
I care deeply about the transactional integrity for everything except the audit table. The idea is that so much will be logged that if a few entries fail, it's not a problem. If the logging stops a some other transaction-- that would be bad.
I can log to a database or a file, although logging to a file is less attractive because I need to be able to display the results somehow. Logging to a file would (almost) guarantee the logging wouldn't interfere with other code though.
A normal transaction (ie. READ COMMITTED) insert already does the 'minimal' locking. Insert intensive applications will not deadlock on the insert, no matter the order of how the insert is mixed with other operations. At worst an intensive insert system may cause page latch contention on the hot spot where insert occurs, but not deadlocks.
To cause deadlocks as described by Jeff there has to be more at play, like any one of the following:
The system is using a higher isolation level (they had it coming then and well deserve it)
They were reading from the log table during the transaction (so is no longer 'append-only')
The deadlock chain involved application layer locks (ie. .Net lock statements in the log4net framework) resulting in undetectable deadlocks (ie. application hangs). Given that solving the problem involved looking at process dumps, I guess this is the scenario they were having.
So as long as you do insert only logging in READ COMMITTED isolation level transactions you are safe. If you expect the same problem I suspect SO had (ie. deadlocks involving application layer locks) then no amount of database wizardry can save you, as the problem can still manifest even if you log on separate transaction or into separate connection.
If you don't care about consistency on your logging table, why not perform all the logging from a separate thread.
I probably would not wait for transactions to complete before logging, since the log can be pivotal in diagnosing long running transactions. Also, this enables you to see all the work a transaction that rolled back did.
Grab the stack trace and all of your logging data in the logging thread, chuck it on a queue when there are new logging messages, flush them to the db in a single transaction.
Steps to minimizing locking:
(KEY) perform all appends to the logging table outside of the main thread/connection/transaction.
Ensure your logging table has a monotonically increasing clustered index (Eg. int identity ) that is increasing each time you append a log message. This ensures the pages being inserted into are usually in memory and avoids the performance hits you get with heap tables.
Perform multiple appends to the log in a transaction (10 inserts in a transaction are faster than 10 inserts out of a transaction and usually acquire/release less locks)
Give it a break. Only perform logging to your db every N milliseconds. Batch up bits of works.
If you need to report on stuff historically, you can consider partitioning your logging table. Example: You could create a new logging table every month, and at the same time have a log VIEW that is a UNION ALL of all the older logging tables. Perform the reporting against the most appropriate source.
You will get better performance by flushing multiple logging messages in a single (smallish) transaction, and have the advantage that if 10 threads are doing work and logging stuff, only a single thread is flushing stuff to the logging table. This pipelining actually makes stuff scale better.
Since you don't care about the transactional integrity of the audit table, you can obviously perform logging outside of the transaction (i.e. after it completes). That will minimise impact on the transaction.
Also, if you want to minimize locking, you should try to ensure that as much of your query workload as possible has covering non-clustered indexes. (SQL Server 2005 and above, the use of the INCLUDE statement in NC indexes can make a big difference)
One easy way to prevent your logging from having locking issues with your 'regular' database is to not use the same database. Just create another database for your logging. As a bonus, the rapid growth of your logging database won't result in fragmentation in your main DB. Personall, I usually prefer to log to a file -- but then again, I'm used to doing heavy text manipulation in my editor - VIM. Logging to a separate DB should help avoid deadlocking issues.
Just make sure that if you try writing your own database appender for the logging framework you use, you be very careful about your locks (which I'm guessing is what tripped up Jeff in the blog post you reference). Properly written (see several of the comments in Jeff's post), you shouldn't have locking issues with your logging framework unless they do something odd.
Here's my situation (SQL Server):
I have a web application that utilizes nHibernate for data access, and another 3 desktop applications. All access the same database, and are likely to utilize the same tables at any one time.
Now, with the help of NH I'm batching selects in order to load an aggregate with all of its hierarchy - so I would see 4 to maybe 7 selects being issued at once (not sure if it matters).
Every few days one of the applications will get a : "Transaction has been chosen as the deadlock victim." (this usually appears on a select)
I tried changing to snapshot isolation on the database , but that didn't helped - I was ending up with :
Snapshot isolation transaction aborted
due to update conflict. You cannot use
snapshot isolation to access table
'...' directly or indirectly in
database '...' to update,
delete, or insert the row that has
been modified or deleted by another
transaction. Retry the transaction or
change the isolation level for the
update/delete statement.
What suggestions to you have for this situation ? What should I try, or what should I read in order to find a solution ?
EDIT:
Actually there's no raid in there :). The number of users per day is small (I'll say 100 per day - with hundreds of small orders on a busy day), the database is a bit bigger at about 2GB and growing faster every day.
It's a business app, that handles orders, emails, reports, invoices and stuff like that.
Lazy loading would not be an option in this case.
I guess taking a very close looks at those indexes is my best bet.
Deadlocks are complicated. A deadlock means that at least two sessions have locks and are waiting for one another to release a different lock; since both are waiting, the locks never get released, neither session can continue, and a deadlock occurs.
In other words, A has lock X, B has lock Y, now A wants Y and B wants X. Neither will give up the lock they have until they are finished with their transaction. Both will wait indefinitely until they get the other lock. SQL Server sees that this is happening and kills one of the transactions in order to prevent the deadlock. Snapshot isolation won't help you - the DB still needs to preserve atomicity of transactions.
There is no simple answer anyone can give as to why a deadlock would be occurring. You'll need to profile your application to find out.
Start here: How to debug SQL deadlocks. That's a good intro.
Next, look at Detecting and Ending Deadlocks on MSDN. That will give you a lot of good background information on why deadlocks occur, and help you understand what you're looking at/for.
There are also some previous SO questions that you might want to look at:
Diagnosing Deadlocks in SQL Server 2005
Zero SQL deadlock by design
Or, if the deadlocks are very infrequent, just write some exception-handling code into your application to retry the transaction if a deadlock occurs. Sometimes it can be extremely hard (if not nearly impossible) to prevent certain deadlocks. As long as you write transactionally-safe code, it's not the end of the world; it's completely safe to just try the transaction again.
Is your hardware properly configured (specifically RAID configuration)? Is it capable of matching your workload?
If hardware is all good and humming, you should ensure you have the 'right' indexes to match your query workload.
Many locking/deadlock problems can be eliminated with the correct indexes (covering indexes can take pressure off the clustered index during inserts).
BTW: turning on snapshot isolation will put increased pressure on your tempDB. How is tempDB configured? RAID 0 is preferred (and even better use an SSD if tempDB is a bottleneck).
While it's not uncommon to find this error in NHibernate sessions with large numbers of users, it seems to be happening too often in your case.
Perhaps your objects are very large resulting in long-running selects? And if your selects are taking too long, that might indicate problems with your indexes (as Mitch Wheat explains)
If everything is in order, you could also try Lazy Loading to postpone your selects until when you really need your data. This might not be appropriate for your exact situation so you do have to see if it works.