I am struggling to find a SQL Server replacement for select for update that works.
I have a master table that contains a column which is used for next order number. The application does a select from update on this row, reads the current value (while locked) adds one to this value and then updates the row, then uses the number it received. This process works perfectly on all databases I've tried but for SQL Server which does not seem to have any process for selecting data for exclusive use.
How do I do a locked read and update of something like a next order number from a sequence table is SQL Server?
BTW, I know I can use things like IDENTITY cols and stuff, to do this, but in this case I must read from this existing column. Get the value and inc it, and do it in a safe locked manner to avoid 2 users getting the same value.
UPDATE::
Thank you, that works for this case :)
DECLARE #Output char(30)
UPDATE scheme.sysdirm
SET #Output = key_value = cast(key_value as int)+1
WHERE system_key='OPLASTORD'
SELECT #Output
I have one other place I do something similar. I read and lock a stock record too.
SELECT STOCK
FROM PRODUCT
WHERE ID = ? FOR UPDATE.
I then do some validation and the do
UPDATE PRODUCT SET STOCK = ?
WHERE ID=?
I can't just use your above method here, as the value I update is based on things I do from the stock I read. But I need to ensure no one else can mess with the stock while I do this. Again, easy on other DB's with SELECT FOR UPDATE... is there a SQL Server workaround?? :)
You can simple do an UPDATE that also reads out the new value into a SQL Server variable:
DECLARE #Output INT
UPDATE dbo.YourTable
SET #Output = YourColumn = YourColumn + 1
WHERE ID = ????
SELECT #Output
Since it's an atomic UPDATE statement, it's safe against concurrency issues (since only one connection can get an update locks at any one given time). A potential second session that wants to get the incremented value at the same time will have to wait until the first one completes, thus getting the next value from the table.
As an alternative you can use the OUTPUT clause of the UPDATE statement, although this will insert into a table variable.
Create table YourTable
(
ID int,
YourColumn int
)
GO
INSERT INTO YourTable VALUES (1, 1)
GO
DECLARE #Output TABLE
(
YourColumn int
)
UPDATE YourTable
SET YourColumn = YourColumn + 1
OUTPUT inserted.YourColumn INTO #Output
WHERE ID = 1
SELECT TOP 1 YourColumn
FROM #Output
**** EDIT
If you want to ensure that no-one can change the data after you have read it, you can use a repeatable read. You should be aware that any reads of any tables you do will be locked for Update (pessimistic locking) and may cause Deadlocking. You can also sue the SELECT ... FROM TABLE (UPDLOCK) hint within a transaction.
SET TRANSACTION ISOLATION LEVEL REPEATABLE READ
BEGIN TRANSACTION
SELECT STOCK
FROM PRODUCT
WHERE ID = ?
.....
...
UPDATE Product
SET Stock = nnn
WHERE ID = ?
COMMIT TRANSACTION
Related
We are using the technique outlined here to generate random record IDs without collisions. In short, we create a randomly-ordered table of every possible ID, and mark each record as 'Taken' as it is used.
I use the following Stored Procedure to obtain an ID:
ALTER PROCEDURE spc_GetId #retVal BIGINT OUTPUT
AS
DECLARE #curUpdate TABLE (Id BIGINT);
SET NOCOUNT ON;
UPDATE IdMasterList SET Taken=1
OUTPUT DELETED.Id INTO #curUpdate
WHERE ID=(SELECT TOP 1 ID FROM IdMasterList WITH (INDEX(IX_Taken)) WHERE Taken IS NULL ORDER BY SeqNo);
SELECT TOP 1 #retVal=Id FROM #curUpdate;
RETURN;
The retrieval of the ID must be an atomic operation, as simultaneous inserts are possible.
For large inserts (10+ million), the process is quite slow, as I must pass through the table to be inserted via a cursor.
The IdMasterList has a schema:
SeqNo (BIGINT, NOT NULL) (PK) -- sequence of ordered numbers
Id (BIGINT) -- sequence of random numbers
Taken (BIT, NULL) -- 1 if taken, NULL if not
The IX_Taken index is:
CREATE NONCLUSTERED INDEX (IX_Taken) ON IdMasterList (Taken ASC)
I generally populate a table with Ids in this manner:
DECLARE #recNo BIGINT;
DECLARE #newId BIGINT;
DECLARE newAdds CURSOR FOR SELECT recNo FROM Adds
OPEN newAdds;
FETCH NEXT FROM newAdds INTO #recNo;
WHILE ##FETCH_STATUS=0 BEGIN
EXEC spc_GetId #newId OUTPUT;
UPDATE Adds SET id=#newId WHERE recNo=#recNo;
FETCH NEXT FROM newAdds INTO #id;
END;
CLOSE newAdds;
DEALLOCATE newAdds;
Questions:
Is there any way I can improve the SP to extract Ids faster?
Would a conditional index improve peformance (I've yet to test, as
IdMasterList is very big)?
Is there a better way to populate a table with these Ids?
As with most things in SQL Server, if you are using cursors, you are doing it wrong.
Since you are using SQL Server 2012, you can use a SEQUENCE to keep track of what random value you already used and effectively replace the Taken column.
CREATE SEQUENCE SeqNoSequence
AS bigint
START WITH 1 -- Start with the first SeqNo that is not taken yet
CACHE 1000; -- Increase the cache size if you regularly need large blocks
Usage:
CREATE TABLE #tmp
(
recNo bigint,
SeqNo bigint
)
INSERT INTO #tmp (recNo, SeqNo)
SELECT recNo,
NEXT VALUE FOR SeqNoSequence
FROM Adds
UPDATE Adds
SET id = m.id
FROM Adds a
INNER JOIN #tmp tmp ON a.recNo = tmp.recNo
INNER JOIN IdMasterList m ON tmp.SeqNo = m.SeqNo
SEQUENCE is atomic. Subsequent calls to NEXT VALUE FOR SeqNoSequence are guaranteed to return unique values, even for parallel processes. Note that there can be gaps in SeqNo, but it's a very small trade off for the huge speed increase.
Put a PK inden of BigInt on each table
insert into user (name)
values ().....
update user set = user.ID = id.ID
from id
left join usr
on usr.PK = id.PK
where user.ID = null;
one
insert into user (name) value ("justsaynotocursor");
set #PK = select select SCOPE_IDENTITY();
update user set ID = (select ID from id where PK = #PK);
Few ideas that came to my mind:
Try if removing the top, inner select etc. helps to improve the performance of the ID fetching (look at statistics io & query plan):
UPDATE top(1) IdMasterList
SET #retVal = Id, Taken=1
WHERE Taken IS NULL
Change the index to be a filtered index, since I assume you don't need to fetch numbers that are taken. If I remember correctly, you can't do this for NULL values, so you would need to change the Taken to be 0/1.
What actually is your problem? Fetching single IDs or 10+ million IDs? Is the problem CPU / I/O etc. caused by the cursor & ID fetching logic, or are the parallel processes being blocked by other processes?
Use sequence object to get the SeqNo. and then fetch the Id from idMasterList using the value returned by it. This could work if you don't have gaps in IdMasterList sequences.
Using READPAST hint could help in blocking, for CPU / I/O issues, you should try to optimize the SQL.
If the cause is purely the table being a hotspot, and no other easy solutions seem to help, split it into several tables and use some kind of simple logic (even ##spid, rand() or something similar) to decide from which table the ID should be fetched. You would need more checking if all tables have free numbers, but it shouldn't be that bad.
Create different procedures (or even tables) to handle fetching of single ID, hundreds of IDs and millions of IDs.
Postgresql allows me to lock the row while selecting it with the following syntax:
select id, amount from table where id = 1234 for update
This allows me to update the row while ensuring that concurrent reads who also want to update the same row will get updated and correct results.
Basically I can do something like this (pseudocode):
begin transaction
select id, amount from table where id = 1234 for update
if(amount == new_amount) then
delete from table where id = 1234;
else
update table set amount = amount - new_amount where id = 1234
end
commit transaction
The same syntax is allowed in Sql Server only for DECLARE CURSOR. Is there a way to read the row while locking it for update with just simple select?
select * from T WITH (UPDLOCK)
The WITH hints are very flexible and well designed. I guess they are more precise and general than for update.
I am using SQL Server 2008. I have a table where orders with SKU are recorded, a table for inventory that has counts and a table where the relationship between the SKU sold and inventory items is recorded.
In the end, I got the report like this
Inventory CurrentQuantity OpenedOrder
SKU1 300 50
SKU2 100 10
Each order will be processed individually. How can I have the database automatically update the inventory tablet after each order is processed?
i.e
If the order has 2 SKU1 in it got processed, the the inventory table will automatically show 298.
Thanks
I would use a Stored Procedure, and perform the order insert and quantity update in one hit:
CREATE PROC dbo.ProcessOrder
#Item int,
#Quantity int
AS
BEGIN
--Update order table here
INSERT INTO dbo.Orders(ItemID,Quantity)
VALUES (#ItemID, #Quantity)
--Update Inventory here
UPDATE dbo.Inventory
SET CurrentQuantity = CurrentQuantity - Quantity
WHERE ItemID = #ItemID
END
I think what you are looking for is a trigger
Basically, set up a trigger that will update the appropriate columns using the inserted/updated data given. Without a full schema set, that is the best answer I can give at this time
I wouldn't be looking at a trigger myself for this.
My check out process
Start a transaction
Check stock level.
If OK, (optional validation / authorisation)
Add a check out record
Reduce the stock
Possibly add some record to invoice teh recipent etc.
Commit the transaction
While you could do it with triggers, I simply fail to see the point, a nice simple clear and all in one place SP_CheckOut stored procedure is where I'd be going.
I would normally advise to use a trigger but stock manipulation is that kind of operation that's usually done a lot of times, sometimes on batches and this is not the best scenario for triggers to be honest.
I think PKG's idea is very good, but you should never forget to add transaction control to it, otherwise you can endup with non-matching stocks:
CREATE PROC dbo.ProcessOrder
#Item int,
#Quantity int
AS
BEGIN
begin transaction my_tran
begin try
--Update order table here
INSERT INTO dbo.Orders(ItemID,Quantity)
VALUES (#ItemID, #Quantity)
--Update Inventory here
UPDATE dbo.Inventory
SET CurrentQuantity = CurrentQuantity - Quantity
WHERE ItemID = #ItemID
commit transaction
end try
begin catch
rollback transaction
--raise error if necessary
end catch
END
you can use the trigger,also use the procedure,and the specific steps on the top,use the procedure need to open the atuo exec feature in mastaer DB.
I have a deadlock problem between 2 processes that insert data in the same table
These 2 processes run exactly the same SQL orders on a table with a primary key (identity) and a unique index.
the sequence of SQL order is the following, for each process in an explicit transaction :
begin trans
select CUSTID from CUSTOMERS where CUSTNUMBER='unique value'
------- the row is never found in this case so... insert the data
insert into CUST(CUSTNUMBER) values('unique value')
------- then we must read the value generated for the pk
select CUSTID from CUSTOMERS where CUSTNUMBER='unique value'
commit
each process work on a distinct data set and have no common values for "CUSTNUMBER"
the deadlock occurs in this case :
spid 1 : select custid... for unique value 1
spid 2 : select custid... for unique value 2
spid 1 : insert unique value 1
spid 2 : insert unique value 2
spid 2 : select custid again for value 2 <--- Deadlock Victim !
spid 1 : select custid again for value 1
The deadlock graph show that the problem occurs on the unique index on CUSTNUMBER
The killed process had a lock OwnerMode:X and was RequestMode:S on the unique index for the same HoBt ID.
The winner process was OnwerMode:X and RequestMode:S for the same HoBt ID
I have no idea to explain that, maybe someone can help me ?
try using OUTPUT to get rid of the final SELECT:
begin trans
select CUSTID from CUSTOMERS where CUSTNUMBER='unique value'
------- the row is never found in this case so... insert the data
insert into CUST(CUSTNUMBER) OUTPUT INSERTED.CUSTID values('unique value')
--^^^^^^^^^^^^^^^ will return a result set of CUSTIDs
commit
OR
DECLARE #x table (CUSTID int)
begin trans
select CUSTID from CUSTOMERS where CUSTNUMBER='unique value'
------- the row is never found in this case so... insert the data
insert into CUST(CUSTNUMBER) OUTPUT INSERTED.CUSTID INTO #x values('unique valu')
--^^^^^^^^^^^^^^^^^^^^^^ will store a set of CUSTIDs
-- into the #x table variable
commit
I have no explanation to the deadlock only another way of doing what you are doing using merge and output. It requires that you use SQL Server 2008 (or higher). Perhaps it will take care of your deadlock issue.
declare #dummy int;
merge CUSTOMERS as T
using (select 'unique value') as S(CUSTNUMBER)
on T.CUSTNUMBER = S.CUSTNUMBER
when not matched then
insert (CUSTNUMBER) values(S.CUSTNUMBER)
when matched then
update set #dummy = 1
output INSERTED.CUSTID;
This will return the newly created CUSTID if there was no match and the already existing CUSTID if there where a match for CUSTNUMBER.
It would be best if you post the actual deadlock graph (the .xml file, not the picture!). W/o that noone can be sure, but is likely that you see a case of the read-write deadlock that occurs due to the order of using vs. applying updates to the secondary indexes. I cannot reommend a solution w/o seeing the deadlock graph and the exact table schema (clustered index and all non-clustered indexes).
On a separate note the SELECT->if not exists->INSERT pattern is always wrong under concurrency, there isn't anything to prevent two threads from trying to insert the same row. A much better patter is to simply insert always and catch the duplicate key violation exception that occurs (is also more performant). As for your second SELECT, use OUTPUT clause as other have already suggested. so basically this whole ordeal can be reduced an insert int a try/catch block. MERGE will also work.
An alternative to using output is replacing the last select with a select scope_identity() if the CUSTID column is an identity column.
I have a business requirement that the InvoiceNumber field in my Invoices table be totally sequential - no gaps or the auditors might think our accountants are up to something fishy!
My first thought was to simply use the primary key (identity) but if a transaction is rolled back a gap appears in the sequence.
So my second thought is to use a trigger which, at the point of insert, looks for the highest InvoiceNumber value in the table, adds 1 to it, and uses it as the InvoiceNumber for the new row. Easy to implement.
Are there potential issues with near-simultaneous inserts? For example, might two near simultaneous inserts running the trigger at the same time get the same 'currently highest InvoiceNumber' value and therefore insert rows with the same InvoiceNumber?
Are there other issues I might be missing? Would another approach be better?
Create a table which keeps tracks of 'counters'.
For your invoices, you can add some record to that table which keeps track of the next integer that must be used.
When creating an invoice, you should use that value, and increase it. When your transaction is rolled back, the update to that counter will be rollbacked as well. (Make sure that you put a lock on that table, to be sure that no other process can use the same value).
This is much more reliable than looking at the highest current counter that is being used in your invoice table.
You may still get gaps if data gets deleted from the table. But if data only goes in and not out, then with proper use of transactions on an external sequence table, it should be possible to do this nicely. Don't use MAX()+1 because it can have timing issues, or you may have to lock more of the table (page/table) than required.
Have a sequential table that has only one single record and column. Retrieve numbers from the table atomically, wrapping the retrieval and usage in a single transaction.
begin tran
declare #next int
update seqn_for_invoice set #next=next=next+1
insert invoice (invoicenumber,...) value (#next, ....)
commit
The UPDATE statement is atomic and cannot be interrupted, and the double assignment make the value of #next atomic. It is equivalent to using an OUTPUT clause in SQL Server 2005+ to return the updated value. If you need a range of numbers in one go, it is easier to use the PRE-update value rather than the POST-update value, i.e.
begin tran
declare #next int
update seqn_for_invoice set #next=next, next=next+3 -- 3 in one go
insert invoice (invoicenumber,...) value (#next, ....)
insert invoice (invoicenumber,...) value (#next+1, ....)
insert invoice (invoicenumber,...) value (#next+2, ....)
commit
Reference for SQL Server UPDATE statement
SET #variable = column = expression sets the variable to the same value as the column. This differs from SET #variable = column, column = expression, which sets the variable to the pre-update value of the column.
CREATE TABLE dbo.Sequence(
val int
)
Insert a row with an initial seed. Then to allocate a range of sufficient size for your insert (call it in the same transaction obviously)
CREATE PROC dbo.GetSequence
#val AS int OUTPUT,
#n as int =1
AS
UPDATE dbo.Sequence
SET #val = val = val + #n;
SET #val = #val - #n + 1;
This will block other concurrent attempts to increment the sequence until the first transaction commits.