I have the following stored procedure running against an ExtractedMessages table which might contain up to 100M records (100,000,000).
For the purpose of my application, this stored procedure should run in less than one second
CREATE PROCEDURE GetNextMessages
#taskId bigint
AS
BEGIN
SET NOCOUNT ON;
DECLARE #ci INT
DECLARE #cr INT
SELECT
#ci = CurrentIndex, #cr = CurrentResources
FROM
ExtractedTasks
WHERE
Id = #taskId
UPDATE ExtractedTasks
SET CurrentIndex = #ci + #cr
WHERE Id = #taskId
SELECT *
FROM ExtractedMessages
WHERE TaskId = #taskId
ORDER BY Id
OFFSET #ci ROWS
FETCH NEXT #cr ROWS ONLY
END
NB: cr can not be more than 1500
Instead of querying your table twice you could do the following:
DECLARE #ci INT
DECLARE #cr INT
DECLARE #T TABLE (
ci INT
, cr INT
);
UPDATE ExtractedTasks
SET CurrentIndex = CurrentIndex + CurrentResources
OUTPUT DELETED.CurrentIndex, DELETED.CurrentResources
INTO #T (ci, cr)
WHERE Id = #taskId;
SELECT
#ci = ci, #cr = cr
FROM
#T
SELECT *
FROM ExtractedMessages
WHERE TaskId = #taskId
ORDER BY Id
OFFSET #ci ROWS
FETCH NEXT #cr ROWS ONLY
The key here is OUTPUT clause. It's going to insert deleted records (pre-update values) into a table variable. Which means you replace multiple selects from a table with one. Other than correct indexes I don't see anything for improvement.
Also, make sure you select just the columns you need, not everything. It's generally a good practice to list exact columns you need.
Thanks Everyone for your kind help
I was able to identify the problem, after all it was a memory management issue on the server that is running SQL server. I have increased the dedicated memory of the server. And the issue has been resolved. The SP works smoothly now
Related
I am searching for a loop query over multiple databases and insert result into existing table in one database to collect al data.
There are 28 existing databases at the moment but when i start the query below it says table already exists at the second database.
when this works i want to loop a much larger query then this.
I also tried executing and union all but if a new database is added it must be collected autmatically.
See example i've tried below:
--drop table if exists [hulptabellen].dbo.HIdatabases
declare #dbList table (dbName varchar(128), indx int)
insert into #dbList
select dbName = dbname, row_number() over (order by dbname)
from [hulptabellen].dbo.HIdatabases
--declare variables for use in the while loop
declare #index int = 1
declare #totalDBs int = (select count(*) from #dbList)
declare #currentDB varchar(128)
declare #cmd varchar(300)
--define the command which will be used on each database.
declare #cmdTemplate varchar(300) =
'
use {dbName};
select * insert into [hulptabellen].dbo.cladrloc from {dbname}.dbo.cladrloc
'
--loop through each database and execute the command
while #index <= #totalDBs
begin
set #currentDB = (select dbName from #dbList where indx = #index)
set #cmd = replace(#cmdTemplate, '{dbName}', #currentDB)
execute(#cmd)
set #index += 1
end
Create the table outside your loop and insert into the table this way:
INSERT INTO [hulptabellen].dbo.cladrloc (col1,col2)
SELECT col1,col2
FROM {dbname}.dbo.cladrloc
FYI: When you use the following syntax, a new table is created, so it can be executed only once.
SELECT *
INTO [hulptabellen].dbo.cladrloc
FROM {dbname}.dbo.cladrloc
I am just new in SQL language and still studying it. I'm having hard time looking for answer on how can I use the stored procedure and insert value into a table.
I have this stored procedure:
CREATE PROCEDURE TestID
AS
SET NOCOUNT ON;
BEGIN
DECLARE #NewID VARCHAR(30),
#GenID INT,
#BrgyCode VARCHAR(5) = '23548'
SET #GenID = (SELECT TOP (1) NextID
FROM dbo.RandomIDs
WHERE IsUsed = 0
ORDER BY RowNumber)
SET #NewID = #BrgyCode + '-' + CAST(#GenID AS VARCHAR (30))
UPDATE dbo.RandomIDs
SET dbo.RandomIDs.IsUsed = 1
WHERE dbo.RandomIDs.NextID = #GenID
SELECT #NewID
END;
and what I'm trying to do is this:
INSERT INTO dbo.Residents([ResidentID], NewResidentID, [ResLogdate],
...
SELECT
[ResidentID],
EXEC TestID ,
[ResLogdate],
....
FROM
source.dbo.Resident;
There is a table dbo.RandomIDs containing random 6 digit non repeating numbers where I'm pulling out the value via the stored procedure and updating the IsUsed column of the table to 1. I'm transferring data from one database to another database and doing some processing on the data while transferring. Part of the processing is generating a new ID with the required format.
But I can't get it to work Sad I've been searching the net for hours now but I'm not getting the information that I need and that the reason for my writing. I hope someone could help me with this.
Thanks,
Darren
your question is little bit confusing, because you have not explained what you want to do. As i got your question, you want to fetch random id from randomids table and after performed some processing on nextid you want to insert it into resident table [newresidentid] and end of the procedure you fetch data from resident table. if i get anything wrong feel free to ask me.
your procedure solution is following.
CREATE PROCEDURE [TestId]
AS
SET NOCOUNT ON;
BEGIN
DECLARE #NEWID NVARCHAR(30)
DECLARE #GENID BIGINT
DECLARE #BRGYCODE VARCHAR(5) = '23548'
DECLARE #COUNT INTEGER
DECLARE #ERR NVARCHAR(20) = 'NO IDS IN RANDOM ID'
SET #COUNT = (SELECT COUNT(NEXTID) FROM RandomIds WHERE [IsUsed] = 0)
SET #GENID = (SELECT TOP(1) [NEXTID] FROM RandomIds WHERE [IsUsed] = 0 ORDER BY [ID] ASC)
--SELECT #GENID AS ID
IF #COUNT = 0
BEGIN
SELECT #ERR AS ERROR
END
ELSE
BEGIN
SET #NEWID = #BRGYCODE + '-' + CAST(#GENID AS varchar(30))
UPDATE RandomIds SET [IsUsed] = 1 WHERE [NextId] = #GENID
INSERT INTO Residents ([NewResidentId] , [ResLogDate] ) VALUES (#NEWID , GETDATE())
SELECT * FROM Residents
END
END
this procedure will fetch data from your randomids table and perform some processing on nextid than after it directs insert it into resident table and if you want to insert some data through user you can use parameter after declaring procedure name
E.G
CREATE PROCEDURE [TESTID]
#PARAM1 DATATYPE,
#PARAM2 DATATYPE
AS
BEGIN
END
I'm not convinced that your requirement is a good one but here is a way to do it.
Bear in mind that concurrent sessions will not be able to read your update until it is committed so you have to kind of "lock" the update so you will get a block until you're going to commit or rollback. This is rubbish for concurrency, but that's a side effect of this requirement.
declare #cap table ( capturedValue int);
declare #GENID int;
update top (1) RandomIds set IsUsed=1
output inserted.NextID into #cap
where IsUsed=0;
set #GENID =(select max( capturedValue) from #cap )
A better way would be to use an IDENTITY or SEQUENCE to solve your problem. This would leave gaps but help concurrency.
I have an application with a requirement that the user can "copy" a record. This will duplicate the record in the table, and the associated records in any child tables.
I will use a trigger to execute a stored procedure to do the copy. The issue I am facing is that I want to increment the ID field for the copied record, which is also the FK in the child tables. The ID field is not a standard format, so using an increment won't work. In order to make this future-proof, I was going to use dynamic SQL to pull the columns for each table so that I don't need to modify the code if I add a new field to one of the tables. The client's system admin can also add columns to the table via the GUI, but they have no access to the SQL backend so they would need to contact us to modify the code (not ideal).
Example:
Declare #ColumnNames varchar(2000)
Declare #BLDGCODE char(4)
set #BLDGCODE = '001'
select #ColumnNames = COALESCE(#ColumnNames + ', ', '') + COLUMN_NAME
from
INFORMATION_SCHEMA.COLUMNS
where
TABLE_NAME='FMB0'
Declare #DynSqlStatement varchar(max);
set #DynSqlStatement = 'Insert into dbo.FMB0('+ #ColumnNames + ')
select * from dbo.FMB0 where BLDGCODE= ' + cast(#BLDGCODE as char(4));
print(#DynSqlStatement);
This solves the issue for a new column being added to one of the tables. However, how can I increment the ID (BLDGCODE in this example). Is my only solution to script out the columns by name so I can increment the ID, or is there a function I am overlooking?
Hopefully this made sense. I am an intermediate SQL user at best, so forgive the naivete if there's an obvious solution.
UPDATE
So I've decided to use #temp tables to hold the record that was changed, modify the id there, and then insert back into the main table from the #temp table. This is working pretty well, with one exception. I get the following error:
The column "FLOORID_" cannot be modified because it is either a computed column or is the result of a UNION operator.
Below is my stored procedure. I've investigated a STUFF approach, but not sure where to insert that code. Using STUFF with calculated column. I am now back to thinking I need to call out the columns specifically for the one table with the computed column, and if we add a new field, I just need to modify this stored procedure. Anyone have any other ideas?
ALTER PROCEDURE [dbo].[LDAC_BLDGCOPY]
#BLDGCODE CHAR(4)
AS
BEGIN
SET NOCOUNT ON;
--COPY BUILDING RECORD
BEGIN
DECLARE #ColumnNamesB0 VARCHAR(2000);
SELECT *
INTO #TEMPB0
FROM FMB0
WHERE BLDGCODE = #BLDGCODE;
UPDATE #TEMPB0
SET BLDGCODE = CONVERT(CHAR(4), (CAST((#BLDGCODE) AS INT) +
100)),
auto_key = dbo.GetAutoKey(),
BLDGCOPY = 0;
SELECT #ColumnNamesB0 = COALESCE(#ColumnNamesB0+', ',
'')+COLUMN_NAME
FROM INFORMATION_SCHEMA.COLUMNS
WHERE TABLE_NAME = 'FMB0';
DECLARE #DynSqlStatementB0 VARCHAR(MAX);
SET #DynSqlStatementB0 = 'Insert into dbo.FMB0('+#ColumnNamesB0+')
select * from #TEMPB0';
EXEC (#DynSqlStatementB0);
END;
--COPY FLOOR RECORDS
BEGIN
DECLARE #ColumnNamesL0 VARCHAR(2000);
--DECLARE #Val INT =
RTRIM(CONVERT(CHAR(10),CAST(LEFT(#BL_KEY,LEN(RTRIM(#BL_KEY))-2)+1 as
INT)))+'01
SELECT *
INTO #TEMPL0
FROM FML0
WHERE BLDGCODE = #BLDGCODE;
UPDATE #TEMPL0
SET BLDGCODE = CONVERT(CHAR(4), (CAST((#BLDGCODE) AS INT) +
100)),
auto_key = dbo.GetAutoKey();
UPDATE #TEMPL0
SET FLOORID_ = auto_key;
SELECT #ColumnNamesL0 = COALESCE(#ColumnNamesL0+', ',
'')+COLUMN_NAME
FROM INFORMATION_SCHEMA.COLUMNS
WHERE TABLE_NAME = 'FML0';
DECLARE #DynSqlStatementL0 VARCHAR(MAX);
SET #DynSqlStatementL0 = 'Insert into dbo.FML0('+#ColumnNamesL0+')
select * from #TEMPL0';
EXEC (#DynSqlStatementL0);
END;
--COPY ROOM RECORDS
BEGIN
DECLARE #ColumnNamesA0 VARCHAR(2000);
--DECLARE #Val INT =
RTRIM(CONVERT(CHAR(10),CAST(LEFT(#BL_KEY,LEN(RTRIM(#BL_KEY))-2)+1 as
INT)))+'01
SELECT *
INTO #TEMPA0
FROM FMA0
WHERE BLDGCODE = #BLDGCODE;
UPDATE #TEMPA0
SET
BLDGCODE = CONVERT(CHAR(4), (CAST((#BLDGCODE) AS INT) +
100)),
auto_key = dbo.GetAutoKey(),
FLOORID = #TEMPL0.FLOORID_
FROM #TEMPA0
INNER JOIN #TEMPL0 ON CONVERT(CHAR(4), (CAST((#BLDGCODE) AS
INT) + 100)) = #TEMPL0.BLDGCODE;
SELECT #ColumnNamesA0 = COALESCE(#ColumnNamesA0+', ',
'')+COLUMN_NAME
FROM INFORMATION_SCHEMA.COLUMNS
WHERE TABLE_NAME = 'FMA0';
DECLARE #DynSqlStatementA0 VARCHAR(MAX);
SET #DynSqlStatementA0 = 'Insert into dbo.FMA0('+#ColumnNamesA0+')
select * from #TEMPA0';
EXEC (#DynSqlStatementA0);
DROP TABLE #TEMPB0;
DROP TABLE #TEMPL0;
DROP TABLE #TEMPA0;
END;
END;
ORIGINAL PROBLEM
I have created a custom script to retrieve data from a remote SQL server into our local copy in our office. I had some issues with the script where selected tables had some data inserted twice, thus creating duplicates. I know that for all the tables in all databases there should be no duplicates.
This issue has made me paranoid that other tables may have had this problem historically, and therefore I'd like to verify this.
SOLUTION
I have created a SQL script to insert the count and distinct count of all columns into a table for all the databases on our server (excluding the 4 system databases):
DECLARE #TableFullName AS NVARCHAR(MAX)
DECLARE #SQLQuery AS NVARCHAR(MAX)
DECLARE #TableHasDuplicates AS BIT
DECLARE #TempTableRowCount AS INT
DECLARE #ResultsTable TABLE ([CompleteTableName] NVARCHAR(200), [CountAll] INT, [CountDistinct] INT)
DECLARE #CountAll INT
DECLARE #CountDistinct INT
SET NOCOUNT ON
DECLARE #AllTables TABLE ([CompleteTableName] NVARCHAR(200))
INSERT INTO #AllTables ([CompleteTableName])
EXEC sp_msforeachdb 'SELECT ''['' + [TABLE_CATALOG] + ''].['' + [TABLE_SCHEMA] + ''].['' + [TABLE_NAME] + '']'' FROM [?].INFORMATION_SCHEMA.TABLES'
SET NOCOUNT OFF;
DECLARE [table_cursor] CURSOR FOR
(SELECT *
FROM #AllTables
WHERE [CompleteTableName] NOT LIKE '%master%' AND [CompleteTableName] NOT LIKE '%msdb%' AND [CompleteTableName] NOT LIKE '%tempdb%' AND [CompleteTableName] NOT LIKE '%model%');
OPEN [table_cursor]
PRINT N'There were ' + CAST(#CountAll AS NVARCHAR(10)) + ' tables with potential duplicate data'
FETCH NEXT FROM [table_cursor]
INTO #TableFullName
WHILE ##FETCH_STATUS = 0
BEGIN
SET #SQLQuery = 'SELECT #CntAll = COUNT(*) FROM ' + #TableFullName + ' SELECT #CntDistinct = COUNT(*) FROM (SELECT DISTINCT * FROM ' + #TableFullName + ') AS [sq] IF #CntAll > #CntDistinct SELECT #BitResult=1 ELSE SELECT #BitResult=0';
EXEC sp_executesql #SQLQuery, N'#BitResult BIT OUTPUT, #CntAll INT OUTPUT, #CntDistinct INT OUTPUT', #BitResult = #TableHasDuplicates OUTPUT, #CntAll = #CountAll OUTPUT, #CntDistinct = #CountDistinct OUTPUT;
IF #TableHasDuplicates = 1
BEGIN
INSERT INTO #ResultsTable ([CompleteTableName], [CountAll], [CountDistinct])
SELECT #TableFullName, #CountAll, #CountDistinct
END;
FETCH NEXT FROM [table_cursor]
INTO #TableFullName
END
CLOSE [table_cursor];
DEALLOCATE [table_cursor];
SELECT *
FROM #ResultsTable
An overview of how it works is the table variable #AllTables uses sp_msforeachdb with INFORMATION_SCHEMA.TABLES to list all the tables in all databases (there are 16537 tables). A table cursor is used to store all non-system entries and then I use dynamic SQL to undertake a count and distinct count which is stored in another table variable #ResultsTable.
THE PROBLEM WITH THIS SOLUTION
When I run this query, it will run for circa 3 minutes then throw an error saying that the tempdb PRIMARY filegroup is full:
I am my own DBA, and I used Brent Ozar's guide to setting up my SQL server instance, and my tempdb is set up with 8 x 3GB mdf/ndf files (the server has 8 cores):
These files show as having 23997MB available under 'General' properties.
MY QUESTIONS
If I have circa 24GB of tempdb free space, why is this relatively simple query running out of tempdb space?
Is there a better/more efficiency way of getting a count and distinct count of all tables in all databases?
You should always consider contention before adding TempDb file. Adding 7 additional TempDb file won't really help.
If I have circa 24GB of tempdb free space, why is this relatively
simple query running out of tempdb space?
No, it should not. But are you sure that you aren't dealing with large amount of data or you don't have other process running on SQL? Cursors, Temp tables and even table variables use TempDb extensively. Check which object is consuming more TempDb space:
SELECT
SUM (user_object_reserved_page_count)*8 as usr_obj_kb,
SUM (internal_object_reserved_page_count)*8 as internal_obj_kb,
SUM (version_store_reserved_page_count)*8 as version_store_kb,
SUM (unallocated_extent_page_count)*8 as freespace_kb,
SUM (mixed_extent_page_count)*8 as mixedextent_kb
FROM sys.dm_db_file_space_usage
So, if your user and internal objects are more then it clearly means that you have low TempDb space because of cursors and SQL Server internal usage (Ex: intermediate tables, Hash joins, Hash aggregation etc)
Is there a better/more efficiency way of getting a count and distinct
count of all tables in all databases?
You can use below code to get the count of all tables in all databases
DECLARE #Stats TABLE (DBNAME VARCHAR(40), NAME varchar(200), Rows INT)
INSERT INTO #Stats
EXECUTE sp_MSForEachDB
'USE ?; SELECT DB_NAME()AS DBName,
sysobjects.Name
, sysindexes.Rows
FROM
sysobjects
INNER JOIN sysindexes
ON sysobjects.id = sysindexes.id
WHERE
type = ''U''
AND sysindexes.IndId < 2'
SELECT * FROM #Stats
I have written an article on TempDb recommendation; I would suggest you to read that to understand objects which can affect TempDb and how to solve common problems of it. Ideally, your total TempDb size should be calculated based on observation which in your case > 24 GB.
** Edit 1**
If you are unsure about stats update then use below query to get count of all tables
Note: Replace databases for which you don't want stats
DECLARE #ServerStats TABLE (DatabaseName varchar(200), TableName varchar(200), RowsCount INT)
INSERT INTO #ServerStats
exec sp_msforeachdb #command1='
use #;
if ''#'' NOT IN (''master'', ''model'', ''msdb'', ''tempdb'',''ReportServer'')
begin
print ''#''
exec sp_MSforeachtable #command1=''
SELECT ''''#'''' AS DATABASENAME, ''''?'''' AS TABLENAME, COUNT(*) FROM ? ;
''
end
', #replacechar = '#'
SELECT * FROM #ServerStats
similarly you can take distinct in all tables for all databases with below query
DECLARE #ServerStatsDistinct TABLE (DatabaseName varchar(200), TableName varchar(200), RowsCount INT)
INSERT INTO #ServerStatsDistinct
exec sp_msforeachdb #command1='
use #;
if ''#'' NOT IN (''master'', ''model'', ''msdb'', ''tempdb'',''ReportServer'')
begin
print ''#''
exec sp_MSforeachtable #command1=''
SELECT ''''#'''' AS DATABASENAME, ''''?'''' AS TABLENAME, COUNT(*) FROM (
SELECT DISTINCT *
FROM ?
) a ;
''
end
', #replacechar = '#'
SELECT * FROM #ServerStatsDistinct
Be gentle, I'm a SQL newbie. I have a table named autonumber_settings like this:
Prefix | AutoNumber
SO | 112320
CA | 3542
A whenever a new sales line is created, a stored procedure is called that reads the current autonumber value from the 'SO' row, then increments the number, updates that same row, and return the number back from the stored procedure. The stored procedure is below:
ALTER PROCEDURE [dbo].[GetAutoNumber]
(
#type nvarchar(50) ,
#out nvarchar(50) = '' OUTPUT
)
as
set nocount on
declare #currentvalue nvarchar(50)
declare #prefix nvarchar(10)
if exists (select * from autonumber_settings where lower(autonumber_type) = lower(#type))
begin
select #prefix = isnull(autonumber_prefix,''),#currentvalue=autonumber_currentvalue
from autonumber_settings
where lower(autonumber_type) = lower(#type)
set #currentvalue = #currentvalue + 1
update dbo.autonumber_settings set autonumber_currentvalue = #currentvalue where lower(autonumber_type) = lower(#type)
set #out = cast(#prefix as nvarchar(10)) + cast(#currentvalue as nvarchar(50))
select #out as value
end
else
select '' as value
Now, there is another procedure that accesses the same table that duplicates orders, copying both the header and the lines. On occasion, the duplication results in duplicate line numbers. Here is a piece of that procedure:
BEGIN TRAN
IF exists
(
SELECT *
FROM autonumber_settings
WHERE autonumber_type = 'SalesOrderDetail'
)
BEGIN
SELECT
#prefix = ISNULL(autonumber_prefix,'')
,#current_value=CAST (autonumber_currentvalue AS INTEGER)
FROM autonumber_settings
WHERE autonumber_type = 'SalesOrderDetail'
SET #new_auto_number = #current_value + #number_of_lines
UPDATE dbo.autonumber_settings
SET autonumber_currentvalue = #new_auto_number
WHERE autonumber_type = 'SalesOrderDetail'
END
COMMIT TRAN
Any ideas on why the two procedures don't seem to play well together, occasionally giving the same line numbers created from scratch as lines created by duplication.
This is a race condition or your autonumber assignment. Two executions have the potential to read out the same value before a new one is written back to the database.
The best way to fix this is to use an identity column and let SQL server handle the autonumber assignments.
Barring that you could use sp_getapplock to serialize your access to autonumber_settings.
You could use repeatable read on the selects. That will lock the row and block the other procedure's select until you update the value and commit.
Insert WITH (REPEATABLEREAD,ROWLOCK) after the from clause for each select.