I am facing a problem when I query master table (having ~700 Million records and high transactional table) to look for newly inserted records. My aim is to get all the newly created IDs from the #IDs temp table (Min and Max records) and dump it in another child table. But random IDs are missing in the child table.
Setup:
We have a primary and secondary server (SQL Server 2016) and they are in sync mode.
Tables:
CREATE TABLE tblMaster
(
ID BIGINT IDENTITY(1,1) NOT NULL,
EmployeeID INT NOT NULL
)
CREATE TABLE tblChild
(
ChildID IDENTITY(1,1),
ID BIGINT NOT NULL,
TransactionDate Datetime NOT NULL
)
tblChild.ID references tblMaster.ID.
Stored procedure:
DECLARE #MaxID BIGINT
SELECT #MaxID = MAX(ID) FROM tblChild WITH(NOLOCK)
SET #MaxID = ISNULL(#MaxID, 0)
DROP TABLE IF EXISTS #IDS
SELECT ID
INTO #IDS
FROM tblMaster WITH(NOLOCK)
WHERE ID > #MaxID
--25k RECORDS BATCH INSERT INTO tblChild - MAINLY TAKE CARE NEWLY inserted records
STARTIDS:
IF EXISTS (SELECT TOP 1 * FROM #IDS)
BEGIN
DROP TABLE IF EXISTS #TOPIDS
SELECT TOP 25000 ID INTO #TOPIDS
FROM #IDS
ORDER BY ID ASC
INSERT INTO tblChild (ID, CreatedBy, CreatedDate)
SELECT ID, SYSTEM_USER, GETDATE()
FROM #TOPIDS
DELETE AA
FROM #IDS AA
INNER JOIN #TOPIDS BB ON AA.ID = BB.ID
GOTO STARTIDS
END
Please help where it's going wrong.
Related
I'm trying to implement the following logic in SQL Server:
every time data is inserted into MainTable, all this data should be also inserted into a backup table MainTable_BACKUP, and every row inserted into MainTable should have a foreign key BackupRecordId pointing to MainTable_BACKUP.
Can it be achieved using a trigger?
CREATE TRIGGER TRG_MainTable
ON MainTable
AFTER INSERT AS
BEGIN
INSERT INTO MainTable_BACKUP
SELECT *
FROM INSERTED
-- UPDATE INSERTED SET BackupRecordId = ??? somehow...
END
Yes you can.
Assuming that you have an identity column named BackupRecordId on you MainTable_BACKUP table, you can create an after insert trigger like this
Create Table MainTable
(
ID int IDENTITY(1, 1) PRIMARY KEY,
Description NVARCHAR(50),
BackupRecordId int
)
Create table MainTable_BACKUP
(
BackupRecordId int IDENTITY(1, 1) PRIMARY KEY,
[Id] int,
Description NVARCHAR(50)
)
CREATE TRIGGER TRG_MainTable
ON MainTable
AFTER INSERT AS
BEGIN
INSERT INTO MainTable_BACKUP([Id], Description)
SELECT [Id], Description FROM INSERTED
UPDATE MainTable
SET BackupRecordId = MP.BackupRecordId
FROM MainTable
INNER JOIN inserted i on i.Id = MainTable.Id
INNER JOIN MainTable_BACKUP MP ON MP.Id = MainTable.Id
END
You can try it with this:
insert into MainTable(Description)
values ('Testing')
select * from MainTable
select * from MainTable_BACKUP
I am using an INSTEAD OF insert trigger on a table to set an incrementing version number on the row and also copy the row to a 2nd history/audit table.
The rows are inserted to both tables without a problem.
However, I am having trouble returning the new identity from the 1st table back to the user.
Schema
CREATE TABLE Table1
(
id INT IDENTITY(1,1) PRIMARY KEY,
name VARCHAR(250) NOT NULL UNIQUE,
rowVersion INT NOT NULL
)
CREATE TABLE Table1History
(
id INT NOT NULL,
name VARCHAR(250) NOT NULL,
rowVersion INT NOT NULL
)
CREATE TRIGGER TRG_INS_Table1
ON Table1
INSTEAD OF INSERT
AS
DECLARE #OutputTbl TABLE (id INT, name VARCHAR(250))
BEGIN
--make the insert
INSERT INTO Table1 (name, rowVersion)
OUTPUT INSERTED.id, INSERTED.name INTO #OutputTbl(id, name)
SELECT i.name, 1
FROM INSERTED i
--copy into history table
INSERT INTO Table1History (id, name, rowVersion)
SELECT t.ID, i.name, 1
FROM INSERTED i
JOIN #OutputTbl t on i.name = t.name
END
CREATE TRIGGER TRG_UPD_Table1
ON Table1
INSTEAD OF UPDATE
AS
BEGIN
--make the update
UPDATE Table1
SET name = i.name,
rowVersion = (SELECT d.rowVersion + 1 FROM DELETED d WHERE d.id = i.id)
FROM INSERTED i
WHERE Table1.id = i.id
--copy into history table
INSERT INTO Table1History (id, name, rowVersion)
SELECT i.id ,i.name, (SELECT d.rowVersion + 1 FROM DELETED d WHERE d.id = i.id)
FROM INSERTED i
END
Joining on the name column in the insert trigger is not ideal, but it needs to handle multiple inserts at once.
eg INSERT INTO Table1 (name) VALUES('xxx'),('yyy')
Attempted Solutions
When doing an insert, SCOPE_IDENTITY is NULL.
INSERT INTO Table1(name)
VALUES('xxx')
SELECT SCOPE_IDENTITY()
or
INSERT INTO Table1(name)
VALUES('xxx')
RETURN SCOPE_IDENTITY()
I've also tried using OUTPUT - which returns 0:
DECLARE #IdentityOutput TABLE (id INT)
INSERT INTO Table1(name)
OUTPUT INSERTED.id INTO #IdentityOutput
VALUES('xxx')
SELECT id FROM #IdentityOutput
The rows are inserted fine and have IDs, but I cannot access them unless I use the below - which seems hacky:
INSERT INTO Table1(name)
VALUES('xxx')
SELECT id from Table1 WHERE name = 'xxx'
What is the proper way to get the new ID??
Solution
Impossible! You can't reliably return the identity when doing an INSERT on a table that has an INSTEAD OF trigger. Sidux's answer below is a good workaround for my situation (replace INSTEAD OF trigger with AFTER trigger and added DEFAULT columns).
CREATE TABLE Table1
(
id INT IDENTITY(1,1) PRIMARY KEY,
name VARCHAR(250) NOT NULL UNIQUE,
rowVersion INT NOT NULL
)
GO
CREATE TABLE Table1History
(
id INT NOT NULL,
name VARCHAR(250) NOT NULL,
rowVersion INT NOT NULL
)
GO
CREATE TRIGGER TRG_INS_Table1
ON Table1
INSTEAD OF INSERT
AS
DECLARE #OutputTbl TABLE (id INT, name VARCHAR(250))
BEGIN
--make the insert
INSERT INTO Table1 (name, rowVersion)
SELECT i.name, 1
FROM INSERTED i
END
GO
CREATE TRIGGER TRG_UPD_Table1
ON Table1
INSTEAD OF UPDATE
AS
BEGIN
--make the update
UPDATE Table1
SET name = i.name,
rowVersion = (SELECT d.rowVersion + 1 FROM DELETED d WHERE d.id = i.id)
FROM INSERTED i
WHERE Table1.id = i.id
END
GO
CREATE TRIGGER TRG_AFT_INS_Table1
ON Table1
AFTER INSERT, UPDATE
AS
BEGIN
INSERT INTO Table1History (id, name, rowVersion)
SELECT i.ID, i.name, i.rowversion
FROM INSERTED i
END
GO
INSERT INTO Table1 (name) VALUES('xxx'),('yyy')
SELECT * FROM Table1History
-----------------------------------------------
id name rowVersion
2 yyy 1
1 xxx 1
-----------------------------------------------
UPDATE Table1 SET name = 'xxx1' WHERE id = 1;
SELECT * FROM Table1History
-----------------------------------------------
id name rowVersion
2 yyy 1
1 xxx 1
1 xxx1 2
-----------------------------------------------
Basically you do not need TRG_INS_Table1 trigger, you can just use DEFAULT value = 1 for column and that's it. Also if you use DATETIME column instead of rowversion, you can just insert the state of INSERTED table to the history with the GETDATE() value. In that case you can order by Dtime column DESC and you have history.
There are two table exist in DB, Audit and AuditField, following is the Create table code:
-- Primary key: ID
CREATE TABLE [dbo].[Audit](
[ID] [int] IDENTITY(1,1) NOT NULL PRIMARY KEY,
[TypeName] [varchar](50) NOT NULL
)
GO
-- Primary key: ID
CREATE TABLE [dbo].[AuditField](
[ID] [int] IDENTITY(1,1) NOT NULL PRIMARY KEY,
[AuditID] [int] NOT NULL,
[Field1] [varchar](50) NOT NULL
)
GO
-- Set foreign key on AuditField table
ALTER TABLE [dbo].[AuditField]
ADD CONSTRAINT [FK_AuditFiled_Audit] FOREIGN KEY([AuditID])
REFERENCES [dbo].[Audit] ([ID])
GO
Then I prepared some test data:
DECLARE #audit TABLE
(
ID int not null,
TypeName varchar(50)
)
DECLARE #auditField TABLE
(
AuditID int not null,
Field1 varchar(50)
)
-- ADD TEST DATA
DECLARE #i int = 1
DECLARE #rowCount int = 500
WHILE #i<=#rowCount
BEGIN
INSERT INTO #audit
VALUES(#i, 'SomeTypeName')
INSERT INTO #auditField
(AuditID,Field1)
VALUES(#i,'SomeThing')
SET #i += 1
END
Finally, i run following transaction to insert test data to these two table:
begin transaction
INSERT INTO dbo.Audit
SELECT TypeName
FROM #audit
ORDER BY ID
declare #lastIdentity int = ##identity
declare #offSet int = #lastIdentity - #rowCount
INSERT INTO dbo.AuditField
SELECT AuditID+#offSet AS AuditID, Field1
FROM #auditField
ORDER BY AuditID
commit transaction
When this transaction run concurrent, dead lock occur, one process are failed, the other got an error：
Msg 547, Level 16, State 0, Line 40 The INSERT statement conflicted
with the FOREIGN KEY constraint "FK_AuditFiled_Audit". The conflict
occurred in database "MyDB", table "dbo.Audit", column 'ID'.
There is no trigger on Audit and AuditField table.
Sorry for the format of the code, I really need an answer why this dead lock occur, thanks.
One thing should be clear, the data of AuditField table comes from #auditField, As #Bogdan answer I rewrite like this:
begin transaction
INSERT INTO dbo.Audit
OUTPUT inserted.ID INTO #temp
SELECT TypeName
FROM #audit
INSERT INTO #idMapping
SELECT ROW_NUMBER() OVER(ORDER BY ID) AS RowNumber, ID
FROM #temp
INSERT INTO dbo.AuditField
SELECT m.ID AS AuditID, Field1
FROM #auditField af
INNER JOIN #idMapping m ON af.AuditID = m.RowNumber
commit transaction
This is and Read - Write deadlock:
As you can see, every transaction has successfully acquired an [e]X[clusive] lock and it requests a S[hared] lock. The question is why a transaction try to read rows X locked by another transaction.
And the answer is bellow:
1) Following piece of source code
declare #lastIdentity int = ##identity
declare #offSet int = #lastIdentity - #rowCount
assumes that IDENTITY values generated by every
INSERT INTO dbo.Audit
SELECT TypeName
FROM ...
statement are continues. This is completely wrong as you can see in the following picture:
This means that at some point in time, a transaction could successfully get X locks on inserted rows and then
1) Because inserted rows into Audit aren't continuous and
2) Because of
declare #lastIdentity int = ##identity
declare #offSet int = #lastIdentity - #rowCount
INSERT INTO dbo.AuditField
SELECT AuditID+#offSet AS AuditID, Field1 ...
this last INSERT tries to insert into dbo.AuditField, AuditID values that belong to another transaction and this requires FK validation and, also, means that SQL Server needs to read rows from dbo.Audit. For this S[hared] locks are needed.
To be clear: the root cause of this deadlock is not the FK constraint. The real problem is that source code.
Solution: I would rewrite thus:
begin transaction
INSERT INTO dbo.Audit
OUTPUT inserted.ID, inserted.TypeName INTO #audit (ID, TypeName)
SELECT TypeName
FROM #audit
-- ORDER BY ID -- Isn't necessary
... do something (ex. DELETE) with rows from #audit
INSERT INTO dbo.AuditField (AuditID, ...)
SELECT x.ID, ...
FROM #audit x
-- ORDER BY AuditID
/* or
INSERT INTO dbo.AuditField (AuditID, Field1, ....)
SELECT y.ID, y.ColumnName, ...
FROM (
SELECT x.ID, ...
FROM #audit x
UNPIVOT( ColumnValue FOR ColumnName IN ([TypeName], ...) )
) y
WHERE y.....
*/
commit transaction -- Isn't necessary
You are trying to insert an invalid foreign key value into dbo.AuditField:
SELECT AuditID+#offSet AS AuditID, Field1
Why the #offset? You won't necessarily have an AuditId with that value in the dbo.Audit table.
I have 3 tables similar to the sctructure below
CREATE TABLE [dbo].[EmpBasic](
[EmpID] [int] IDENTITY(1,1) NOT NULL Primary Key,
[Name] [varchar](50),
[Address] [varchar](50)
)
CREATE TABLE [dbo].[EmpProject](
[EmpID] [int] NOT NULL primary key, // referencing column with EmpBasic
[EmpProject] [varchar](50) )
CREATE TABLE [dbo].[EmpFull_Temp](
[ObjectID] [int] IDENTITY(1,1) NOT NULL Primary Key,
[T1Name] [varchar](50) ,
[T1Address] [varchar](50) ,
[T1EmpProject] [varchar](50)
)
The EmpFull_Temp table has the records with a dummy object ID column... I want to populate the first 2 tables with the records in this table... But with EmpID as a reference between the first 2 tables.
I tried this in a stored procedure...
Create Table #IDSS (EmpID bigint, objID bigint)
Insert into EmpBasic
output Inserted.EmpID, EmpFull_Temp.ObjectID
into #IDSS
Select T1Name, T1Address from EmpFull_Temp
Where ObjectID < 106
Insert into EmpProject
Select A.EmpID, B.T1EmpProject from #IDSS as A, EmpFull_Temp as B
Where A.ObjID = B.ObjectID
But it says.. The multi-part identifier "EmpFull_Temp.ObjectID" could not be bound.
Could you please help me in achieving this...
Edit : There is no guarantee that [Name]+[Address] would be unique across [EmpBasic] Table
With your EmpProject join table, you probably don't want the primary key constraint on only the EmpID column
DECLARE #Count int
DECLARE #NextEmpID int
DECLARE #StartObjectID int
DECLARE #EndObjectID int
-- range of IDs to transfer (inclusive)
SET #StartObjectID = 1
SET #EndObjectID = 105
BEGIN TRAN
-- lock tables so IDENT_CURRENT is valid
SELECT #Count = COUNT(*) FROM [EmpBasic] WITH (TABLOCKX, HOLDLOCK)
SELECT #Count = COUNT(*) FROM [EmpProject] WITH (TABLOCKX, HOLDLOCK)
SELECT #NextEmpID = IDENT_CURRENT('EmpBasic')
SET IDENTITY_INSERT [EmpBasic] ON
INSERT [EmpBasic] ([EmpID], [Name], [Address])
SELECT #NextEmpID + ROW_NUMBER() OVER(ORDER BY ObjectID), [T1Name], [T1Address]
FROM [EmpFull_Temp]
WHERE [ObjectID] BETWEEN #StartObjectID AND #EndObjectID
SET IDENTITY_INSERT [EmpBasic] OFF
INSERT [EmpProject]([EmpID], [EmpProject])
SELECT #NextEmpID + ROW_NUMBER() OVER(ORDER BY ObjectID), [T1EmpProject]
FROM [EmpFull_Temp]
WHERE [ObjectID] BETWEEN #StartObjectID AND #EndObjectID
COMMIT TRAN
The solution to this problem depends on whether the "parent" table (i.e. the one with the IDENTITY column) has a natural key (i.e. one or more fields which, when combined, are guaranteed to be unique, other than the surrogate primary key).
For example, in this case, is the combinaton of Name and Address aways going to be unique?
If the answer is yes then you can simply insert into EmpBasic without bothering to output and store the generated IDs. You can then insert into EmpProject joining back on to EmpBasic using the natural key (e.g. name and address) to fnd the correct EmpID.
Insert into EmpBasic
Select T1Name, T1Address from EmpFull_Temp
Where ObjectID < 106
Insert into EmpProject
Select A.EmpID, B.T1EmpProject from EmpBasic as A, EmpFull_Temp as B
Where A.Name = B.Name And A.Address = B.Address
If the answer is no then there is no easy solution I know of - in SQL Server 2005 (I've no idea if this is any different in 2008), it's not possible to OUTPUT values that are not inserted. I've got around this issue in the past by using one of the other fields (e.g. Name) to temporarily store the original ID (in this case, ObjectID), use that to join when inserting the child records as described above and then gone back to update the parent records o remove/replace the temporary values. It's not nice but I've not found a better way.
Insert into EmpBasic
Select cast(ObjectID as varchar(50)) as name, T1Address from EmpFull_Temp
Where ObjectID < 106
Insert into EmpProject
Select A.EmpID, B.T1EmpProject from EmpBasic as A, EmpFull_Temp as B
Where A.Name = cast(B.ObjectID as varchar(50))
Update EmpBasic
Set Name = B.T1Name
from EmpBasic as A, EmpFull_Temp as B
Where A.Name = cast(B.ObjectID as varchar(50))
Please note: I've not tested the sample SQL given above but I hope it gives you an idea of how you might approach this.
Add an ObjectID column to the EmpBasic table to facilitate the data transfer then drop it when you're done. I'm assuming this is a one-time operation, I don't recommend adding and dropping a column if this is on-going
I have used the Stack Exchange Data Explorer to investigate alternative solutions. The only one with promise at the moment is shown here. It is effectively #ScotHauder's answer, except using a temporary table that has the ObjectID column and using IDENTITY_INSERT to move the generated EmpId values into EmpBasic.
If you have to do this multiple times you need to get the EmpBasic_Temp EmpId IDENTITY starting value to be Max(EmpBasic.EmpID)+1.
The function Scope_Identity() will provide the last generated primary key value from a table insert. Is there any generally accepted way to get multiple keys from an insertion of a set (an insert resulting from a select query)?
In SQL Server 2005 onwards, you can use the OUTPUT clause to get a returned set of values. From the linked article:
The following example creates the
EmployeeSales table and then inserts
several rows into it using an INSERT
statement with a SELECT statement to
retrieve data from source tables. The
EmployeeSales table contains an
identity column (EmployeeID) and a
computed column (ProjectedSales).
Because these values are generated by
the SQL Server Database Engine during
the insert operation, neither of these
columns can be defined in #MyTableVar.
USE AdventureWorks ;
GO
IF OBJECT_ID ('dbo.EmployeeSales', 'U') IS NOT NULL
DROP TABLE dbo.EmployeeSales;
GO
CREATE TABLE dbo.EmployeeSales
( EmployeeID int IDENTITY (1,5)NOT NULL,
LastName nvarchar(20) NOT NULL,
FirstName nvarchar(20) NOT NULL,
CurrentSales money NOT NULL,
ProjectedSales AS CurrentSales * 1.10
);
GO
DECLARE #MyTableVar table(
LastName nvarchar(20) NOT NULL,
FirstName nvarchar(20) NOT NULL,
CurrentSales money NOT NULL
);
INSERT INTO dbo.EmployeeSales (LastName, FirstName, CurrentSales)
OUTPUT INSERTED.LastName,
INSERTED.FirstName,
INSERTED.CurrentSales
INTO #MyTableVar
SELECT c.LastName, c.FirstName, sp.SalesYTD
FROM HumanResources.Employee AS e
INNER JOIN Sales.SalesPerson AS sp
ON e.EmployeeID = sp.SalesPersonID
INNER JOIN Person.Contact AS c
ON e.ContactID = c.ContactID
WHERE e.EmployeeID LIKE '2%'
ORDER BY c.LastName, c.FirstName;
SELECT LastName, FirstName, CurrentSales
FROM #MyTableVar;
GO
SELECT EmployeeID, LastName, FirstName, CurrentSales, ProjectedSales
FROM dbo.EmployeeSales;
GO
Use the row count and last identity value....
DECLARE #LastID int
DECLARE #Rows int
--your insert from a select here
SELECT #LastID=##IDENTITY, #Rows=##ROWCOUNT
--set of rows you want...
SELECT * FROM YourTable Where TableID>#LastID-#Rows AND TableID<=#LastID