I have very big dataset about 40000000 items of data. I am trying to load this to a new table. Since the dataset is really big, I am trying to load them in batch so the log transaction would not be full. I am using the query below but still have issue with log transaction being full.
Is there a way to commit the transition after each batch so that it wont be saved in log and will go for next batch?
Please let me know if there is a way for me to solve this issue.
And SSIS is not an option for me currently.
DECLARE #I BIGINT;
DECLARE #icnt int;
DECLARE #n INT, #flag INT;
SELECT #icnt = (SELECT COUNT(*) FROM table1) A
PRINT #ICNT
BEGIN
SELECT #I = 0
WHILE #I <=#icnt
BEGIN
SELECT #n = 0
SELECT #flag = 1
BEGIN TRANSACTION
WHILE #flag != 0
BEGIN
;WITH A AS
(
SELECT
a.*,
ROW_NUMBER() OVER (ORDER BY (SELECT 1)) AS batchnum
FROM
table1
)
INSERT INTO table2 (*)
SELECT *
FROM A
WHERE batchnum >= #I AND batchnum <#I + 10000000
SELECT #I = #I + 10000000
SELECT #n = #n + 10000000
SELECT #flag = CASE WHEN #n >= 50000000 THEN 0
WHEN #I >= #icnt THEN 0 ELSE 1 END
PRINT #I
END
COMMIT TRANSACTION
END
END
Related
This question already has answers here:
Stored procedure hangs seemingly without explanation
(8 answers)
Closed 1 year ago.
I have this T-SQL that works just fine and it executes about 30 seconds.
DECLARE #BatchSize int = 1000,
#TransactionInterval tinyint = 5,
#MaxToDelete int = 10000,
#FactDeleted int = 0;
DECLARE #counter int = 1;
DECLARE #s datetime,
#e datetime,
#r int = 1;
SET #e = '20200606';
SELECT
#s = MIN(AtTime)
FROM
dbo.DevicePositions;
BEGIN TRANSACTION;
WHILE (#r > 0)
BEGIN
IF #r % #TransactionInterval = 1
BEGIN
COMMIT TRANSACTION;
BEGIN TRANSACTION;
END
DELETE TOP (#BatchSize)
FROM DevicePositions
WHERE AtTime >= #s
AND AtTime <= #e;
SET #FactDeleted = #FactDeleted + #BatchSize;
SET #r = ##ROWCOUNT;
SET #counter = #counter + 1;
IF #FactDeleted >= #MaxToDelete
BREAK;
END
IF ##TRANCOUNT > 0
BEGIN
COMMIT TRANSACTION;
IF #counter % 10 = 0 -- or maybe 100 or 1000
BEGIN CHECKPOINT;
END
END
GO
SELECT
A.Records
FROM
(SELECT
OBJECT_NAME(object_id) as ID,
SUM(row_count) AS Records
FROM
sys.dm_db_partition_stats
WHERE
object_id = OBJECT_ID('DevicePositions')
AND index_id < 2
GROUP BY
OBJECT_NAME(object_id)) A
I converted this code into a stored procedure and it won't complete so it runs forever.
The stored procedure:
ALTER PROCEDURE [dbo].[DeleteOldDevicePositions]
#MaxToDelete int , -- Max amount of records to delete
#BatchSize int , -- it is
#ToEndDate datetime -- Delete until this datetime
AS
BEGIN
SET NOCOUNT ON;
DECLARE #TransactionInterval tinyint = 5, #FactDeleted int = 0;
DECLARE #counter int = 1;
DECLARE #s datetime, #e datetime, #r int = 1;
SELECT #s = MIN(AtTime) FROM dbo.DevicePositions;
BEGIN TRANSACTION;
WHILE (#r > 0)
BEGIN
IF #r % #TransactionInterval = 1
BEGIN
COMMIT TRANSACTION;
BEGIN TRANSACTION;
END
DELETE TOP (#BatchSize) FROM DevicePositions WHERE AtTime >= #s AND AtTime <= #ToEndDate;
SET #FactDeleted = #FactDeleted +#BatchSize;
SET #r = ##ROWCOUNT;
SET #counter = #counter + 1;
IF #FactDeleted >= #MaxToDelete
BREAK;
END
IF ##TRANCOUNT > 0
BEGIN
COMMIT TRANSACTION;
IF #counter % 10 = 0 -- or maybe 100 or 1000
BEGIN
CHECKPOINT;
END
END
SELECT A.Records FROM (
SELECT OBJECT_NAME(object_id) as ID, SUM(row_count) AS Records FROM sys.dm_db_partition_stats WHERE
object_id = OBJECT_ID('DevicePositions') AND index_id < 2
GROUP BY OBJECT_NAME(object_id) ) A
END
And I start it like
EXEC [dbo].[DeleteOldDevicePositions] 10000, 1000, '20200606'
So it starts and has no end.
What did I miss?
Thank you!
Thanks to Stored procedure hangs seemingly without explanation
I have used http://whoisactive.com/ to see that there is a rollback on that table.
So I used
exec sp_who
KILL 53 WITH STATUSONLY;
And see that have to wait until it will be done.
May be Parameter Sniffing is one of the reason for this Issue. Please have a look on the below Link.
https://blog.sqlauthority.com/2019/12/19/sql-server-parameter-sniffing-simplest-example/
I have the a Requirement where I have to Delete in batches in sql server , and also track the number of count affected in the end. My Sample Code is as Follows:
Declare #count int
Declare #deletecount int
set #count=0
While(1=1)
BEGIN
BEGIN TRY
BEGIN TRAN
DELETE TOP 1000 FROM --CONDITION
SET #COUNT = #COUNT+##ROWCOUNT
IF (##ROWCOUNT)=0
Break;
COMMIT
END CATCH
BEGIN CATCH
ROLLBACK;
END CATCH
END
set #deletecount=#COUNT
Above Code Works fine, but how to keep track of #deletecount if Rollback happens in one of the batch.
Some potential issues with the code :
while loop never exits/breaks
declare #tbl table(foo bit); --deleting from an empty table
declare #COUNT int;
declare #iteration int = 0;
While(1=1)
BEGIN
--loop safety net
if #iteration >= 100
begin
break --exit after 100 iterations
end
select #iteration = #iteration+1
BEGIN TRY
DELETE TOP (1000) FROM #tbl; --#tbl is empty
SET #COUNT = #COUNT+##ROWCOUNT;-- <-- this always sets ##rowcount = 1
IF (##ROWCOUNT)=0 -- ... so this can never be 0 --> will never exit the loop
Break;
END TRY
BEGIN CATCH
END CATCH
END --end while
select #iteration as iterations;
Open transaction at the end of the batch
declare #tbl table(foo bit); --<-- empty table
declare #COUNT int;
declare #myrowcount int;
declare #iteration int = 0;
While(1=1)
BEGIN
--loop safety net
if #iteration >= 100
begin
break --exit after 100 iterations
end
select #iteration = #iteration+1
BEGIN TRY
BEGIN TRAN --<-- transaction begins on each iteration
DELETE TOP (1000) FROM #tbl; --#tbl is empty
set #myrowcount = ##rowcount;
SET #COUNT = #COUNT+#myrowcount;
IF (#myrowcount)=0
Break; --<-- if this breaks... transaction is neither committed nor rolledback
COMMIT --<-- commited on each iteration, when no error
END TRY
BEGIN CATCH
ROLLBACK --<-- or rolled back on each iteration error
END CATCH
END --end while
select #iteration as iterations, ##trancount as open_transactions;
rollback transaction;
loop keeps trying to delete the same erroneous batch (over and over again) and never exits
create table parent (id int primary key clustered);
create table child (parentid int references parent(id));
go
--some parents
insert into parent(id)
select top (200) row_number() over(order by ##spid)
from sys.all_objects;
--and a child
insert into child(parentid) values (75)
go
declare #COUNT int;
declare #myrowcount int;
declare #errormsg nvarchar(max);
declare #iteration int = 0;
While(1=1)
BEGIN
--loop safety net
if #iteration >= 100
begin
break --exit after 100 iterations
end
select #iteration = #iteration+1
BEGIN TRY
DELETE TOP (10) t --changed the batch size..for the example
from (select top (10000) * from parent order by id) as t;
set #myrowcount = ##rowcount;
SET #COUNT = #COUNT+#myrowcount;
IF (#myrowcount)=0
Break;
END TRY
BEGIN CATCH
select #errormsg = isnull(#errormsg, '') + error_message();
END CATCH
END --end while
select #iteration as iterations, #errormsg as errormsg;
--some parents have been deleted (8th batch kept failing, there is an fk to parent 75)
select *
from parent
go
drop table child
drop table parent
go
You could rectify all the above issues...(last one, to skip erroneous batches being a bit more "difficult"), by implementing logic in the where clause (eg. exclude rows which are referenced etc) according to your model. Trying to implement a straight deletion, without any rules, and skipping failures makes it a bit harder.
Just an example: when a batch fails, after 3 attempts, take another deletion route. Since each deletion is atomic, transactions are not really needed here (besides, if the batch is called from a parent module, a plain ROLLBACK in the batch would "invalidate" any transactions opened in the parent module, before the batch execution)
create table parent (id int primary key clustered);
create table child (parentid int references parent(id));
go
--some parents
insert into parent(id)
select top (200) row_number() over(order by ##spid)
from sys.all_objects;
--and two children
insert into child(parentid) values (75), (115) --, (9), (18) --: add 9 and 18 and nothing gets deleted, alternative route in the example does not work
go
declare #COUNT int;
declare #myrowcount int;
declare #iteration int = 0;
declare #errormsg nvarchar(max);
declare #ierrorcnt int=0;
declare #deletedids table(id int);
declare #lastsuccessfullydeletedid int = 0;
select #COUNT = 0;
While(1=1)
BEGIN
--loop safety net
if #iteration >= 100
begin
break --exit after 100 iterations
end
select #iteration = #iteration+1
BEGIN TRY
--when 10 consecutive errors in a single iteration..just quit the loop
if #ierrorcnt >= 10
begin
break;
end
BEGIN TRAN --<-- transaction begins on each iteration
if #ierrorcnt >= 3 --when 3 consecutive errors in the iteration..try to bypass the current batch
begin
delete top (10) t
output deleted.id into #deletedids(id)
from (select top (10) * from (select top (2*10) * from parent where id > #lastsuccessfullydeletedid order by id) as a order by id desc) as t
select #myrowcount = count(*), #lastsuccessfullydeletedid = max(id)
from #deletedids;
delete from #deletedids;
end
else
begin
DELETE TOP (10) FROM parent where id > #lastsuccessfullydeletedid;
set #myrowcount = ##rowcount;
end
SET #COUNT = #COUNT+#myrowcount;
COMMIT --<-- commited on each iteration, when no error
IF (#myrowcount)=0 and #ierrorcnt = 0
Break;
set #ierrorcnt = 0; --everything ok, set iteration error counter to 0
END TRY
BEGIN CATCH
ROLLBACK --<-- or rolled back on each iteration error
if #ierrorcnt = 0
begin
select #errormsg = isnull(#errormsg, '') +';'+ error_message();
end
set #ierrorcnt = #ierrorcnt + 1; --error, increase the iteration error counter
END CATCH
END --end while
select #iteration as iterations, ##trancount as open_transactions;
select #iteration as iterations, #errormsg as errormsg;
--some parents have been deleted
select * /*2 failed batches, 20 rows left*/
from parent
select #COUNT as [count/deleted] --<--this is also the deleted count
go
drop table child
drop table parent
go
..without any errors, in your original code, #count = #deletedcount
declare #tbl table(foo int); --<-- empty table
insert into #tbl(foo)
select top (100000) row_number() over(order by ##spid)
from sys.all_objects as a
cross join sys.all_objects as b;
--batch 1000, max iterations for the #tbl count
declare #maxiterations int;
select #maxiterations = 1+count(*) / 1000
from #tbl
declare #COUNT int;
declare #deletedcount int;
declare #myrowcount int;
declare #iteration int = 0;
select #COUNT = 0, #deletedcount = 0;
While(1=1)
BEGIN
--loop safety net
if #iteration >= #maxiterations
begin
break --exit after #maxiterations
end
select #iteration = #iteration+1;
BEGIN TRY
BEGIN TRAN
DELETE TOP (1000) FROM #tbl
where foo%5 = 0 ;
set #myrowcount = ##rowcount;
SET #COUNT = #COUNT+#myrowcount;
set #deletedcount = #deletedcount + #myrowcount;
COMMIT
IF #myrowcount=0
Break;
END TRY
BEGIN CATCH
ROLLBACK
END CATCH
END --end while
select #iteration as iterations, ##trancount as open_transactions;
select #count as _count, #deletedcount as deletedcount;
I'm trying to do an SQL script here but facing some dificulties as I don't have so much knowledges on that, here is my issue:
I need to create a temp table with pre-determined values (dbrsm01 to dbrsm30) and check iterating through these values if one of them is available on MSSQL server to be used as a new database, if one of these values is already in use it needs to be ignored as I need to create a new DB using this value.
Here is what I've done so far:
DECLARE #temp TABLE (id int, dbname varchar(10))
INSERT INTO #temp VALUES(1,'dbrsm01');
INSERT INTO #temp VALUES(2,'dbrsm02');
INSERT INTO #temp VALUES(3,'dbrsm03');
...
INSERT INTO #temp VALUES(27,'dbrsm27');
INSERT INTO #temp VALUES(28,'dbrsm28');
INSERT INTO #temp VALUES(29,'dbrsm29');
INSERT INTO #temp VALUES(30,'dbrsm30');
DECLARE #maxid INT, #counter INT, #tempname VARCHAR(10), #nameset VARCHAR(10)
SET #counter = 1
SELECT #maxid = COUNT(*) FROM #temp
WHILE (#counter <= #maxid OR #nameset = #tempname)
BEGIN
SET #tempname = (SELECT dbname FROM #temp WHERE id = #counter)
IF NOT EXISTS (SELECT name FROM sys.databases WHERE name = #tempname)
BEGIN
SET #nameset = #tempname
END
SET #counter = #counter + 1
END
SELECT #nameset as [#dbname]
Any help on that is appreciate! Thank you all!
This doesn't need to be iterative at all. You can do this in a single statement:
WITH Tally AS (
SELECT 1 AS i
UNION ALL
SELECT i + 1
FROM Tally
WHERE i + 1 <= 30)
SELECT 'dbrsm' + CONVERT(varchar(7),T.i)
FROM Tally T
LEFT JOIN sys.databases d ON 'dbrsm' + CONVERT(varchar(7),T.i) = d.[name]
WHERE d.database_id IS NULL;
Edit: A mindset you need to change when writing SQL is thinking programmatically. You don't want to think about what you're going to do to a row, you need to think about what you're going to do to a column. Using loops are not a way of thinking in a dataset approach (normally).
Edit: Nevermind, here's how to make a CREATE statement and make all the databases:
DECLARE #SQL nvarchar(MAX);
WITH Tally AS (
SELECT 1 AS i
UNION ALL
SELECT i + 1
FROM Tally
WHERE i + 1 <= 30)
SELECT #SQL = STUFF((SELECT NCHAR(10) + N'CREATE DATABASE ' + QUOTENAME(N'dbrsm' + CONVERT(varchar(7),T.i)) + N';'
FROM Tally T
LEFT JOIN sys.databases d ON 'dbrsm' + CONVERT(varchar(7),T.i) = d.[name]
WHERE d.database_id IS NULL
FOR XML PATH ('')),1,1,'');
PRINT #SQL; --This is your best friend for troubleshooting
--EXEC sp_executesql #SQL; --Uncomment to run your dynamic SQL
You could use dynamic SQL:
DECLARE #sql NVARCHAR(MAX)
,#maxid INT = (SELECT COUNT(*) FROM #temp)
,#counter INT = 1
,#tempname VARCHAR(10)
,#nameset VARCHAR(10);
WHILE (#counter <= #maxid)
BEGIN
SET #tempname = (SELECT dbname FROM #temp WHERE id = #counter);
IF NOT EXISTS (SELECT name FROM sys.databases WHERE name = #tempname)
BEGIN
SET #nameset = #tempname;
SET #sql = 'CREATE DATABASE ' + QUOTENAME(#nameset);
PRINT #sql;
EXEC (#sql);
END
SET #counter += 1;
END
You can use BREAK and do what you want, like this:
DECLARE #temp TABLE (id int, dbname varchar(10))
INSERT INTO #temp VALUES(1,'dbrsm01');
INSERT INTO #temp VALUES(2,'dbrsm02');
INSERT INTO #temp VALUES(3,'dbrsm03');
DECLARE #maxid INT, #counter INT, #tempname VARCHAR(10), #nameset VARCHAR(10)
SET #counter = 1
SELECT #maxid = COUNT(*) FROM #temp
WHILE (#counter <= #maxid /*OR #nameset = #tempname --YOU DON'T NEED THIS */)
BEGIN
SELECT #tempname = dbname FROM #temp WHERE id = #counter
PRINT #tempname
IF NOT EXISTS (SELECT name FROM sys.databases WHERE name = #tempname)
BEGIN
SET #nameset = #tempname
PRINT 'ITS FREE'
BREAK
END
ELSE
PRINT 'ITS IN USE!'
SET #counter = #counter + 1
END
SELECT #nameset as [#dbname]
If you need to create all DB's that dont exist, remove the BREAK and add your logic (CREATE DATABASE ...)
SELECT 'CREATE Database '+A.dbname+'
GO '
FROM
(
SELECT K.dbname
FROM #temp K
LEFT JOIN sys.databases Y
ON K.dbName = Y.Name
WHERE Y.name is NULL
)A
You can copy the result of this statement and run it.
I´m having some trouble about this query. The problem is that only one while is process, the firts one, don´t work for me.
This is my code:
The While #j, don´t work, but I don´t see the mistake ...
Thanks for your time.
CREATE PROCEDURE InsertVar
AS
DECLARE #i int
DECLARE #j int
SET NOCOUNT ON;
SET #i = 1
SET #j = 1
WHILE #j < 21
BEGIN
WHILE #i < 11
BEGIN
INSERT INTO Var(idline,idVar,CheckBox )
VALUES(#j,#i,0);
SET #i = #i + 1;
END;
SET #j = #j + 1;
END;
SELECT * FROM Var;
GO
There are other ways to do this logic (such as a single query). But your problem is that you do not re-initialize i inside the loop. Try this:
CREATE PROCEDURE InsertVar
AS
BEGIN
DECLARE #i int;
DECLARE #j int;
SET NOCOUNT ON;
SET #j = 1;
WHILE #j < 21
BEGIN
SET #i = 1;
WHILE #i < 11
BEGIN
INSERT INTO Var(idline, idVar, CheckBox)
VALUES(#j, #i, 0);
SET #i = #i + 1;
END;
SET #j = #j + 1;
END;
SELECT * FROM Var;
END;
GO
Use Recursive CTE to generate the rows. No need to use while loop or declaration of varaiables.
CREATE PROCEDURE Insertvar
AS
BEGIN
SET NOCOUNT ON;
;WITH cte
AS (SELECT 1 AS idline
UNION ALL
SELECT idline + 1 FROM cte
WHERE idline < 20),
cte1
AS (SELECT idline,1 AS idVar,0 AS CheckBox
FROM cte
UNION ALL
SELECT idline,idVar + 1,CheckBox FROM cte1
WHERE idVar < 10)
INSERT INTO [Var]
SELECT * FROM cte1
ORDER BY idline,idVar
SELECT * FROM [Var]
END
I need a help in generating SQL result based on a column value to calculate per day amount.
I have a SQL like below.
select guid, reccharge, dayCareDaysCharged, dayCareHoursPerDay, RATE from tableA.
Here if reccharge = 0 (weekly) then the result should have 7 rows with RATE as RATE/7 for each row (per day amount);
if reccharge = 1 (monthly) then the result should have 30 rows with RATE as RATE/30 for each row;
if reccharge = 2 (yearly) then the result should have 365 rows with RATE as RATE/365 for each row;
if reccharge = 3 (hourly) then there should be a row as calculate it for a day;
How can I achieve this. Please help.
The question seems a little strange to me, but if I've followed it correctly then maybe this;
CREATE TABLE [#temp1] (reccharge tinyint,rdays smallint)
GO
declare #rcount smallint
set #rcount=0
while #rcount<7
begin
set #rcount=#rcount+1
insert #temp1 values (0,7)
end
set #rcount=0
while #rcount<30
begin
set #rcount=#rcount+1
insert #temp1 values (1,30)
end
set #rcount=0
while #rcount<365
begin
set #rcount=#rcount+1
insert #temp1 values (2,365)
end
insert #temp1 values (3,1)
GO
select t1.[guid], t1.reccharge, t1.dayCareDaysCharged, t1.dayCareHoursPerDay, t1.RATE/t2.rdays as RATE from tableA t1
inner join #temp1 t2 on t1.reccharge=t2.reccharge
order by t1.[guid]
I have not idea to do it in single query, but you can do it by using CURSOR as like below :
declare #reccharge int
declare #count int
declare #i int
declare #strSQL nvarchar(max) = 'select 1, 1 where 1 <> 1'
declare CurTest cursor for select reccharge from test
open CurTest
fetch next from CurTest into #reccharge
while ##fetch_status = 0
begin
set #i = 0;
select #count = case when #reccharge = 0 then 7 when #reccharge = 1 then 30 when #reccharge = 2 then 365 else 1 end
while #i < #count
begin
set #strSQL = #strSQL + ' union all select reccharge, Rate/' + cast(#count as varchar(10)) + ' from test where reccharge = ' + cast(#reccharge as varchar(10))
set #i = #i + 1;
end
fetch next from CurTest into #reccharge
end
close CurTest
deallocate CurTest
exec(#strSQL)