i have written this t-sql stored procedure.
This is working fine , but since this stored procedure will be used to delete a lot of data (for example 1M to 2M) , I think, this can cause some locks in the table, or cause some db performance etc. So I am thinking, if we delete in batch for example at a time delete 1000 records. I am not totally sure about how to do this without cause any issue in db.
ALTER PROCEDURE [schema].[purge_data] #count INT---(count input can be in millions)
AS
DECLARE #p_number VARCHAR(22)
,#p_r_number VARCHAR(5)
DECLARE data_cursor CURSOR
FOR
SELECT TOP (#count) JRC_policy_number
,jrc_part_range_nbr
FROM [staging].[test].[p_location]
WHERE JRC_POLICY_TERM_DT < CAST('19950101 00:00:00.000' AS DATETIME)
AND jrc_policy_status = 'T'
OPEN data_cursor
FETCH NEXT
FROM data_cursor
INTO #p_number
,#p_r_number
WHILE ##FETCH_STATUS = 0
BEGIN
DELETE
FROM [staging].[test].[p_location]
WHERE JRC_policy_number = #p_number
AND jrc_part_range_nbr = #p_r_number
FETCH NEXT
FROM data_cursor
INTO #p_number
,#p_r_number
END
CLOSE data_cursor
DEALLOCATE data_cursor
Edit :
I had already tried without cursor - direct delete query like below.
DELETE TOP (1000) FROM [MyTab] WHERE YourConditions
It was very fast , it took 34 seconds to delete 1M records, but , during the 34 seconds, the table was locked completely. In production p_locator table is being used 24/7 , and being used by a very critical application, which expects response time in milliseconds, our purge script should not impact the the main application in any way. that's why I have chosen this cursor approach. pls guide
With some of your references I've written the below stored proc. Ofcourse there will be ALOT of scope for improvements. Pls share.
ALTER PROCEDURE [dbo].[purge_data] #count INT
AS
DECLARE #iteration INT
,#remainder INT
,#current_count INT
BEGIN
SELECT #current_count = count(*)
FROM PROD_TBL
WHERE JRC_POLICY_TERM_DT < CAST('19950101 00:00:00.000' AS DATETIME)
AND JRC_POLICY_STATUS = 'T'
AND JRC_PLCY_ADMIN_SYS_CD = 'X'
IF (#current_count < #count)
BEGIN
SET #count = #current_count
END
SET #iteration = #count / 10000
SET #remainder = #count % 10000
WHILE (#iteration > 0)
BEGIN
DELETE
FROM PROD_TBL
FROM (
SELECT TOP 10000 JRC_POLICY_NUMBER
,JRC_PART_RANGE_NBR
,JRC_PLCY_ADMIN_SYS_CD
FROM PROD_TBL
WHERE JRC_POLICY_TERM_DT < CAST('19950101 00:00:00.000' AS DATETIME)
AND JRC_POLICY_STATUS = 'T'
AND JRC_PLCY_ADMIN_SYS_CD = 'X'
) pol_locator_tbl
WHERE PROD_TBL.JRC_POLICY_NUMBER = pol_locator_tbl.JRC_POLICY_NUMBER
AND PROD_TBL.JRC_PART_RANGE_NBR = pol_locator_tbl.JRC_PART_RANGE_NBR
AND PROD_TBL.JRC_PLCY_ADMIN_SYS_CD=pol_locator_tbl.JRC_PLCY_ADMIN_SYS_CD
SET #iteration = #iteration - 1
END
IF (#remainder > 0)
BEGIN
DELETE
FROM PROD_TBL
FROM (
SELECT TOP (#remainder) JRC_POLICY_NUMBER
,JRC_PART_RANGE_NBR
,JRC_PLCY_ADMIN_SYS_CD
FROM PROD_TBL
WHERE JRC_POLICY_TERM_DT < CAST('19950101 00:00:00.000' AS DATETIME)
AND JRC_POLICY_STATUS = 'T'
AND JRC_PLCY_ADMIN_SYS_CD = 'X'
) pol_locator_tbl
WHERE PROD_TBL.JRC_POLICY_NUMBER = pol_locator_tbl.JRC_POLICY_NUMBER
AND PROD_TBL.JRC_PART_RANGE_NBR = pol_locator_tbl.JRC_PART_RANGE_NBR
AND PROD_TBL.JRC_PLCY_ADMIN_SYS_CD=pol_locator_tbl.JRC_PLCY_ADMIN_SYS_CD
END
END
END
I have a scenario where I need to run a stored procedure individually as well as I need to call from some other stored procedure.
Let me present the scenario: I have 3 stored procedures in sequential call from another one. Like 1st stored procedure is getting called from the application when some raw financial data is being imported; 2nd stored procedure is getting called from 1st stored procedure and in 2nd stored procedure, there is a While loop in which my 3rd stored procedure is getting called.
I am posting here 2nd and 3rd stored procedure code here:
2ND stored procedure code:
If #loopCount > 0
Begin
While(#i <= #loopCount)
Begin
Select #RecoString = '',
#CompanyId = 0,
#UserId = 0
Select #RecoString = MainRecord,
#CompanyId = CompanyId,
#UserId = UsersId
From #RecoData With (Nolock)
Where Id = #i
Order By Id
/* 3rd stored procedure is getting called - IF NO INSERT Statement */
----Exec USP_Temp #IsReco = 1,#ReconcileBy = 'system',#UserCompanyId = #UserCompanyId,#UserId = #UserId,#finalCollection = #RecoString
/* 3rd stored procedure is NOT getting called - IF INSERT Statement */
Insert Into dbo.ReconcileInsertUpdateLog(TransferDetailId,Msg,ReconcilationId,IsFutureTransferReconciled)
Exec dbo.USP_Temp #IsReco = 1, #ReconcileBy = 'system', #CompanyId = #CompanyId, #UserId = #UserId, #finalCollection = #RecoString, #isAutoReconcile = 0
Set #i = #i + 1
End
End
3RD stored procedure code:
ALTER PROCEDURE dbo.USP_Temp
#IsReco Bit
,#ReconcileBy Nvarchar(250)
,#UserCompanyId int
,#UserId int
,#finalCollection Nvarchar(Max) = ''
,#isAutoReconcile Bit = 0
AS
BEGIN
Set Nocount On;
Declare #TransName Varchar(100)
Select #TransName = 'USP_Temp'
Begin Try
Begin Transaction #TransName
Declare #Msg Nvarchar(Max) = ''
,#ParentReconcilationId Int = 0 -- 07.25.2019
,#IsFutureTransferReconciled Int = 0 -- 07.25.2019
------------------------------------------------------------
-- Return result
------------------------------------------------------------
Insert Into dbo.TempReco(Comments)
Select 'Reached to USP_Temp 1 step ahead of Return final Result'
Select 1 As TransferDetailId
,#Msg As Msg
,#ParentReconcilationId As ReconcilationId -- 07.25.2019
,#IsFutureTransferReconciled As IsFutureTransferReconciled -- 07.25.2019
Commit Transaction #TransName
GoTo EndLevel
End Try
Begin Catch
Set #Msg = Error_Message()
GoTo Error
End Catch
Error:
BEGIN
Insert Into dbo.TempReco(Comments) Select 'Reached to USP_Temp - Error Block'
Rollback Transaction #TransName
Select 0 As TransferDetailId
,#Msg As Msg
,0 As ReconcilationId -- 07.25.2019
,0 As IsFutureTransferReconciled -- 07.25.2019
END
EndLevel:
END
GO
Look at the 2nd stored procedure code, I have commented the code which is working if no insert into statement prior to Exec SPName and when calling stored procedure along with insert into SomeTable prior statement then stored procedure is not getting called. Does anyone have some idea on this?
I'm writing tsqlt against a proc that can be run against various parameter values. I initially built a proc that populates fake tables - and then 1 tsqlt test per possible value (ended up with 35 tests and they each worked).
What I would like to do is reduce these into 1 test (since they are all really testing the same functionality - just for different values). I thought I could do this with a cursor like so:
---- Declare Sproc Variables
DECLARE #ReviewId INT;
DECLARE #SourceId INT = 1;
CREATE TABLE #Present
(
SubmissionReviewId INT ,
username VARCHAR(50)
);
CREATE TABLE #Expected
(
SubmissionReviewId INT ,
username VARCHAR(50)
);
--Create Cursor to loop through each active value
DECLARE review_id CURSOR
FOR
SELECT ReviewId
FROM reftype.Rev
WHERE IsActive = 1;
OPEN review_id;
FETCH NEXT FROM review_id
INTO #ReviewId;
WHILE ##FETCH_STATUS = 0
BEGIN
--Setup Fake Data according to the specified test condition
EXEC ut_DataSetupProc #ReviewId = #ReviewId;
-- Run set cutover Sproc
EXEC Procbeing Tested #ReviewId = #ReviewId,
#SourceId = 1, #Username = 'blah';
-- Confirm appropriate review is present in Submission Review Active
DELETE FROM #Present;
DELETE FROM #Expected;
INSERT INTO #Present
SELECT SubmissionReviewId ,
LastModifiedBy
FROM review.SubmissionReviewActive
ORDER BY SubmissionReviewId ,
LastModifiedBy;
/**********************Create table holding expected values***************************/
INSERT INTO #Expected
--This confirms active reviews that belong to other sections/sources remain unaffected
SELECT SubmissionReviewId ,
LastModifiedBy
FROM review.SubmissionReviewActive
WHERE ( ReviewId != #ReviewId )
OR ( SourceId != #SourceId )
UNION
SELECT sra.SubmissionReviewId ,
sra.LastModifiedBy
FROM review.SubmissionReviewActive sra
JOIN review.SubmissionReviewFutureActive srfa ON srfa.IssuerId = sra.IssuerId
AND srfa.ReviewId = sra.ReviewId
AND srfa.Version < sra.Version
WHERE sra.ReviewId = #ReviewId
AND sra.SourceId = #SourceId
UNION
SELECT srfa.SubmissionReviewId ,
'jmarina' AS LastModifiedBy
FROM review.SubmissionReviewFutureActive srfa
JOIN review.SubmissionReviewActive sra ON srfa.IssuerId = sra.IssuerId
AND srfa.ReviewId = sra.ReviewId
AND srfa.Version > sra.Version
WHERE sra.ReviewId = #ReviewId
AND srfa.SourceId = #SourceId
UNION
SELECT srfa.SubmissionReviewId ,
'blah' AS LastModifiedBy
FROM review.SubmissionReviewFutureActive srfa
WHERE srfa.ReviewId = #ReviewId
AND srfa.SourceId = #SourceId
AND srfa.IssuerId NOT IN (
SELECT IssuerId
FROM review.SubmissionReviewActive
WHERE ReviewId = #ReviewId
AND SourceId = #SourceId )
UNION
SELECT sra.SubmissionReviewId ,
sra.LastModifiedBy
FROM review.SubmissionReviewActive sra
WHERE sra.ReviewId = #ReviewId
AND sra.SourceId = #SourceId
AND IssuerId NOT IN (
SELECT IssuerId
FROM review.SubmissionReviewFutureActive
WHERE ReviewId = #ReviewId
AND SourceId = #SourceId )
ORDER BY SubmissionReviewId ,
LastModifiedBy;
/*************************************************************/
EXEC tSQLt.AssertEqualsTable #Expected = '#Expected',
#Actual = '#Present', #Message = N'', -- nvarchar(max)
#FailMsg = N'Active Status is not a match'; -- nvarchar(max)
FETCH NEXT FROM review_id
INTO #ReviewId;
END;
CLOSE review_id;
DEALLOCATE review_id;
DROP TABLE #Expected;
DROP TABLE #Present;
END;
However, running this using
EXEC proc name #ReviewId = #ReviewId;
yields a message saying no tests were run. How can I sue a cursor to reduce my number of tests? Or is there another approach I should consider?
I'd suggest you write something called a parameterized test.
tSQLt does not (yet) have native support for that, but there is an easy workaround:
You start by writing one of your tests normally. But instead of hardcoding the pertinent values, you make them parameters of the procedure. (For data sets, you can use table parameters.)
You also name that procedure something that doesn't start with "test" (but lives in the same schema).
Then you write one real test per actual case, each one consisting of one line: the execution of your parameterized procedure.
That will lead to tests that are a lot easier to understand than your current approach. And additionally, if one of them fails, you immediately know which.
As a side note: You always want to hardcode your expected results. Your current code is way complex. You want to minimize things that can go wrong in the test itself. Really, your goal should be tests that can be understood with one glance.
In the end I achieved the end goal in a couple of steps:
1. Move the assert statement outside of the cursor
2. Created 'cased' temp table with pass/fail records
INSERT INTO #ActualAssert
SELECT p.SubmissionReviewId,e.SubmissionReviewId,
CASE WHEN ( e.SubmissionReviewId IS NULL
OR p.SubmissionReviewId IS NULL
) THEN 'Fail'
ELSE 'Pass'
END
FROM #Present p
LEFT JOIN #Expected e ON e.SubmissionReviewId = p.SubmissionReviewId
UNION
SELECT p.SubmissionReviewId,e.SubmissionReviewId ,
CASE WHEN ( e.SubmissionReviewId IS NULL
OR p.SubmissionReviewId IS NULL
) THEN 'Fail'
ELSE 'Pass'
END
FROM #Present p
RIGHT JOIN #Expected e ON e.SubmissionReviewId = p.SubmissionReviewId;
3. Outside of the cursor I set up a new parameter that takes any fails if they exist or 'pass' if they don't
SET #Result = ( SELECT DISTINCT TOP 1
TestStatus
FROM #ActualAssert
ORDER BY TestStatus ASC
);
4. Then I modified the assert to fail if #result is anything other than 'Pass'
EXEC tSQLt.AssertEqualsString #Expected = N'Pass', -- nvarchar(max)
#Actual = #Result, #Message = N''; -- nvarchar(max)
** A note I change previous present and expected temp tables into variable tables
I writing a cursor to select data from a table into variables.
Using oracle I could simply use a single declare statement that in effect creates a variable for every column with a data type that matches the type of the underlying table (as it is currently defined at runtime)..
In SQL server it would seem you have to manually declare a variable for every column.
Am I right. Does anyone have any thoughts?
I think this is a massive reason to dislike SQL Server when compared to oracle.
I am so disappointed that SS doesn't have %TYPE and %ROWTYPE. These save hours of work when initially writing code and in the event that the type of a column needs to change, it saves having to go back and re-work all the code.
In Oracle I used to write something like:
DECLARE #MyRowVar AS ATable%ROWTYPE;
In SQL Server, I just had to write this this:
DECLARE #External_ID AS BIGINT = NULL;
DECLARE #PlatformID AS INT = NULL;
DECLARE #ActorIDOfReseller AS INT = NULL;
DECLARE #ActorIDOfClient AS INT = NULL;
DECLARE #ActorIDOfExtension AS INT = NULL;
DECLARE #CallType AS NCHAR (10) = NULL;
DECLARE #CallInitiatedDate AS DATE = NULL;
DECLARE #CallInitiatedTimeHH24MI AS TIME (0) = NULL;
DECLARE #TimePeriodID AS INT = NULL;
DECLARE #CallAnswered AS DATETIME = NULL;
DECLARE #CallAnsweredYN AS BIT = NULL;
DECLARE #CallDispositionID AS INT = NULL;
DECLARE #CountryID AS INT = NULL;
DECLARE #CallPrefixID AS INT = NULL;
DECLARE #FromNumber AS VARCHAR (32) = NULL;
DECLARE #ToNumber AS VARCHAR (80) = NULL;
DECLARE #CallDuration AS INT = NULL;
DECLARE #CallCostToExtension AS DECIMAL (10, 6) = NULL;
DECLARE #CallCostToClient AS DECIMAL (10, 6) = NULL;
DECLARE #CallCostToReseller AS DECIMAL (10, 6) = NULL;
DECLARE #CallCostToAdmin AS DECIMAL (10, 6) = NULL;
DECLARE #Flow AS VARCHAR (3) = NULL;
DECLARE #CallStart AS DATETIME = NULL;
DECLARE #MoneyUnit AS VARCHAR (32) = NULL;
DECLARE #Prefix AS VARCHAR (32) = NULL;
DECLARE #External_CallID AS VARCHAR (255) = NULL;
This is making me very sad and is a waste of my life.
I tagged this as Oracle, as I'm sure oracle developers will want to know how lucky they are to work with this professional tool.
Harvey
Continued.
From the comment below it appears there is an easier way! Thanks very much for the input. I was ranting as I got a bit knarked....sorry.
Here's some more context:
I have a csv file, with unreliable data in each column.
I BULK LOAD this csv file into tableA which has all columns declared as VARCHAR(100). (The file loads even if the data type of values aren't what was expected in the column)
I used a single statement to transfer the data from tableA into tableB.
TableB has the same columns as tableA, but tableB columns have a variety of datatypes - which are the expected datatypes that the csv data should have.
(TableB also has a few other columns not populated form the file, eg an IDENTITY column, etc..).
The statement is something like:
INSERT INTO tableB
(column list)
SELECT
( {column list for select} )
FROM tableB;
Each column in the {column list for select} above, has is wrapped with the TRY_CAST function to convert it to the same data type as the destination table or NULL.
The above step 3 worked OK, but I had no way of identifying the invalid rows.
So I decided to try to use a cursor and use the same select statement as in step 3 but without the TRY_CAST wrapping each column. I used a TRY/CATCH around the whole INSERT statement that operated on each row of the select statement to log the invalid rows when the INSERT failed.
I've now written Step 4 but boy I wish I'd posted a "how do I do this" question here first. My code looks really rubbish and I've posted it below.
As you can probably tell, I've not been using SQL Server long and I'm pretty rusty at using big databases..
Any more input you can give is very welcome. I'll re-write this tomorrow!
I like Jermey's suggestion of creating a temp table using teh SQL below, but I've not completely got my head around this yet!
Select *
INTO #tmp
From mytable
Where 1 = 0;
Any way, here it is.
(I've run this now and it took 30 seconds to process 254,000 rows, - only one of which was a failure - but this time is acceptable.)
CREATE PROCEDURE [dbo].[Call_Process_FromXoomTalk_UsingCursor]
#PlatformId int
, #StartOfMonthDate as Date = NULL
, #EndOfMonthDate as Date = NULL
, #LimitImportToRowsWithThisCallType nVarchar(max) = NULL
, #LimitImportToRowsWithThisFlow nVarchar(max) = NULL
, #Raise1Print2Both3 as tinyint = 3
AS
BEGIN
DECLARE #msg as nVarchar(max);
DECLARE #RowCount as int = 0;
DECLARE #FETCHSTATUS as int = 0;
DECLARE #CountOfRowFailures AS INT = 0;
DECLARE #CountOfRowSuccesses AS INT = 0;
--------------------------------------------------------------------------------------------------
-- The order any type of the variables declared below
-- MUST match the order of the columns of the CURSOR SELECT statement below and
-- the data type of the underlying columns of the table "Call".
--
-- CALL TABLE COLUMNS:
--
DECLARE #External_ID AS BIGINT = NULL;
DECLARE #PlatformID2 AS INT = NULL;
DECLARE #ActorIDOfReseller AS INT = NULL;
DECLARE #ActorIDOfClient AS INT = NULL;
DECLARE #ActorIDOfExtension AS INT = NULL;
DECLARE #CallType AS NCHAR (10) = NULL;
DECLARE #CallInitiatedDate AS DATE = NULL;
DECLARE #CallInitiatedTimeHH24MI AS TIME (0) = NULL;
DECLARE #TimePeriodID AS INT = NULL;
DECLARE #CallAnswered AS DATETIME = NULL;
DECLARE #CallAnsweredYN AS BIT = NULL;
DECLARE #CallDispositionID AS INT = NULL;
DECLARE #CountryID AS INT = NULL;
DECLARE #CallPrefixID AS INT = NULL;
DECLARE #FromNumber AS VARCHAR (32) = NULL;
DECLARE #ToNumber AS VARCHAR (80) = NULL;
DECLARE #CallDuration AS INT = NULL;
DECLARE #CallCostToExtension AS DECIMAL (10, 6) = NULL;
DECLARE #CallCostToClient AS DECIMAL (10, 6) = NULL;
DECLARE #CallCostToReseller AS DECIMAL (10, 6) = NULL;
DECLARE #CallCostToAdmin AS DECIMAL (10, 6) = NULL;
DECLARE #Flow AS VARCHAR (3) = NULL;
DECLARE #CallStart AS DATETIME = NULL;
DECLARE #MoneyUnit AS VARCHAR (32) = NULL;
DECLARE #Prefix AS VARCHAR (32) = NULL;
DECLARE #External_CallID AS VARCHAR (255) = NULL;
-- END OF CALL TABLE COLUMNS:
--------------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------------
-- SETTINGS
SET NOCOUNT ON;
BEGIN TRY
BEGIN TRANSACTION
PRINT ' ';
PRINT '===================================================================================';
EXEC [PrintSystemTime] 'Starting: [Call_Process_FromXoomTalk_UsingCursor]';
PRINT '===================================================================================';
--
-- It is assumed that the file loaded into the CallData table
-- only contains calls made in the year and month provided to this procedure
--
--
-- Move rows from the staging table [CallDATA] into the main table
-- [Call]. Some simple looks and modifications are made
--
-----------------------------------------------------------------
-- Define the cursor
-- WARNING - For simplicity keep column order in step with the
-- variable declarations, select and insert use!
--
DECLARE curNewData CURSOR FAST_FORWARD
FOR
SELECT Table1.*
FROM
( SELECT
-- ID is identity field
[id] AS [ExternalID]
, #PlatformId AS [PlatformID]
, ( SELECT SubQry.[ID]
FROM [dbo].[Actor] AS SubQry
WHERE SubQry.[ExternalID] = [NewData].client_reseller_id
) AS Actor_ResellerID
, ( SELECT SubQry.[ID]
FROM [dbo].[Actor] AS SubQry
WHERE SubQry.[ExternalID] = NewData.client_client_id
) AS Actor_ClientID
, ( SELECT SubQry.[ID]
FROM [dbo].[Actor] AS SubQry
WHERE SubQry.[ExternalID] = NewData.client_extension_id
) AS Actor_ExtensionID
-- This code prevent invalid values being loaded
-- that would be rejected byt eh foreigh key
, CASE [calltype]
WHEN 'out' THEN 'out'
WHEN 'in' THEN 'in'
WHEN 'local' THEN 'local'
WHEN 'elocal' THEN 'elocal'
ELSE NULL
END AS [calltype]
-----------------------------------------------------------------------------------------------
--
-- Initiated date/time is split to produce derrived columns
--
-- , [initiated] as [initiated] -- Useful for testing
-- NOTE that CAST will convert 0000-00-00 00:00:00 to be NULL
, TRY_CAST([initiated] AS DATE) AS [CallInitiatedDate]
, TRY_CAST([initiated] AS TIME(0)) AS [CallInitiatedTimeHH24MI]
, ( SELECT isnull(GroupWithTHISID,[ID]) as ID
FROM [dbo].[DimTimePeriod] AS SubQry
WHERE SubQry.[ActorID] = NewData.[client_client_id]
AND TRY_CAST([initiated] AS TIME(0)) >= SubQry.[StartTimeHH24MI]
AND TRY_CAST([initiated] AS TIME(0)) < SubQry.[EndTimeHH24MI]
) AS [TimePeriodID]
-- What format are their dates! Do they have nano secs
, TRY_CAST([answer] as datetime2) AS [CallAnswered]
, iif([answer] is null,0,1) AS [CallAnsweredYN]
-----------------------------------------------------------------------------------------------
, ( SELECT X.[ID]
FROM [dbo].[DimCallDisposition] AS X
WHERE X.[Disposition] = NewData.[disposion]
) AS [CallDispositionID]
-----------------------------------------------------------------------------------------------
--
-- Population of Country and Area require significant processing that is done later
--
, NULL AS [CountryID]
, NULL AS [AreaID]
-----------------------------------------------------------------------------------------------
-- Useful for testing
-- , [extension_number] AS [FromClientAndNumber]
, RIGHT([extension_number],LEN([extension_number])-CHARINDEX('*',[extension_number],1)) AS [FromNumber]
, CASE LEFT([partyid],2)
WHEN '00' THEN
-- Replace 00 with a + ie for all international numbers
'+' + SUBSTRING([partyid],3,9999)
ELSE [partyid]
END AS [ToNumber]
, [duration] AS [CallDuration_SECS]
-----------------------------------------------------------------------------------------------
-- COST columns
--
-- NOTE: Some data in the cost fields in the file from the xoomtalk platform
-- was in scientific notation format
-- eg 4.2e-05 4.2e-05 7.2e-05 6.2e-05 8.3e-05 8.3e-05
-- Note that these were all very small values!
, CASE
WHEN [costext] like '%E-%' THEN LTRIM(RTRIM(TRY_CAST(TRY_CAST([costext] AS FLOAT) AS DECIMAL(10,6))))
WHEN [costext] like '%E+%' THEN NULL
ELSE TRY_CAST([costext] AS DECIMAL(10,6))
END AS [CallCostToExtension]
, CASE
WHEN NewData.[costcl] like '%E-%' THEN LTRIM(RTRIM(TRY_CAST(TRY_CAST([costcl] AS FLOAT) AS DECIMAL(10,6))))
WHEN [costcl] like '%E+%' THEN NULL
ELSE TRY_CAST([costcl] AS DECIMAL(10,6))
END AS [CallCostToClient]
, CASE
WHEN NewData.[costres] like '%E-%' THEN LTRIM(RTRIM(TRY_CAST(TRY_CAST([costres] AS FLOAT) AS DECIMAL(10,6))))
WHEN [costres] like '%E+%' THEN NULL
ELSE TRY_CAST([costres] AS DECIMAL(10,6))
END AS [CallCostToReseller]
, CASE
WHEN NewData.[costadmin] like '%E-%' THEN LTRIM(RTRIM(TRY_CAST(TRY_CAST([costadmin] AS FLOAT) AS DECIMAL(10,6))))
WHEN [costadmin] like '%E+%' THEN NULL
ELSE TRY_CAST([costadmin] AS DECIMAL(10,6))
END AS [CallCostToAdmin]
-----------------------------------------------------------------------------------------------
, [flow] AS Flow
, [start] AS [CallStart]
, [moneyunit] AS [moneyunit]
, [prefix] AS [prefix]
, [callid] AS [External_CallID]
-----------------------------------------------------------------------------------------------
FROM [dbo].[CallDATA] AS NewData
WHERE ( (#LimitImportToRowsWithThisCallType IS NULL)
OR NewData.[calltype] = #LimitImportToRowsWithThisCallType
)
AND ( (#LimitImportToRowsWithThisFlow IS NULL)
OR NewData.[flow] = #LimitImportToRowsWithThisFlow
)
AND TRY_CAST([initiated] as datetime2) >= #StartOfMonthDate
AND TRY_CAST([initiated] as datetime2) <= #EndOfMonthDate
) as Table1
ORDER BY [calltype]
, [flow]
, [CallInitiatedDate]
, [CallInitiatedTimeHH24MI]
, [CallDuration_SECS]
;
-----------------------------------------------------------------
-- Open and use the cursor
--
PRINT '------------------------------------------------------------------';
PRINT 'ABOUT TO LIST the external_ID of any ROWS that failed to insert';
OPEN curNewData;
SET #FETCHSTATUS = 0;
SET #CountOfRowFailures = 0;
SET #CountOfRowSuccesses = 0;
WHILE #FETCHSTATUS = 0
BEGIN
FETCH NEXT FROM curNewData
INTO
#External_ID
, #PlatformID2
, #ActorIDOfReseller
, #ActorIDOfClient
, #ActorIDOfExtension
, #CallType
, #CallInitiatedDate
, #CallInitiatedTimeHH24MI
, #TimePeriodID
, #CallAnswered
, #CallAnsweredYN
, #CallDispositionID
, #CountryID
, #CallPrefixID
, #FromNumber
, #ToNumber
, #CallDuration
, #CallCostToExtension
, #CallCostToClient
, #CallCostToReseller
, #CallCostToAdmin
, #Flow
, #CallStart
, #MoneyUnit
, #Prefix
, #External_CallID
;
BEGIN TRY
INSERT INTO dbo.Call
(
[External_ID]
, [PlatformID]
, [ActorID-OfReseller]
, [ActorID-OfClient]
, [ActorID-OfExtension]
, [CallType]
, [CallInitiatedDate]
, [CallInitiatedTimeHH24MI]
, [TimePeriodID]
, [CallAnswered]
, [CallAnsweredYN]
, [CallDispositionID]
, [CountryID]
, CallPrefixID
, [FromNumber]
, [ToNumber]
, [CallDuration]
, [CallCostToExtension]
, [CallCostToClient]
, [CallCostToReseller]
, [CallCostToAdmin]
, [Flow]
, [CallStart]
, [MoneyUnit]
, [Prefix]
, [External_CallID]
)
VALUES
( #External_ID
, #PlatformID2
, #ActorIDOfReseller
, #ActorIDOfClient
, #ActorIDOfExtension
, #CallType
, #CallInitiatedDate
, #CallInitiatedTimeHH24MI
, #TimePeriodID
, #CallAnswered
, #CallAnsweredYN
, #CallDispositionID
, #CountryID
, #CallPrefixID
, #FromNumber
, #ToNumber
, #CallDuration
, #CallCostToExtension
, #CallCostToClient
, #CallCostToReseller
, #CallCostToAdmin
, #Flow
, #CallStart
, #MoneyUnit
, #Prefix
, #External_CallID
)
;
SET #CountOfRowSuccesses = #CountOfRowSuccesses + 1;
END TRY
BEGIN CATCH
-- Ignore error
PRINT CONCAT(#External_ID, '|', ERROR_NUMBER(), '|', ERROR_MESSAGE());
SET #CountOfRowFailures = #CountOfRowFailures + 1;
END CATCH;
SET #FETCHSTATUS = ##FETCH_STATUS;
END;
CLOSE curNewData;
DEALLOCATE curNewData;
PRINT '------------------------------------------------------------------';
PRINT CONCAT('Count Of Row Failures =', #CountOfRowFailures)
PRINT CONCAT('Count Of Row Successes=', #CountOfRowSuccesses)
PRINT '------------------------------------------------------------------';
COMMIT;
PRINT '################################ ';
PRINT 'COMMIT DONE';
PRINT '################################ ';
EXEC [PrintSystemTime] 'Ending: [Call_Process_FromXoomTalk_UsingCursor]';
PRINT '=========================================================================';
END TRY
BEGIN CATCH
IF #Raise1Print2Both3 = 2 OR #Raise1Print2Both3 = 3
EXEC [dbo].[ErrPrintErrorDetails]
#CalledFromProcedure= '[dbo].[Call_Process_FromXoomTalk_UsingCursor]'
, #ExplainitoryErrorTextForUser = NULL
, #AddDottedLine = 1;
EXEC [PrintSystemTime] 'Ending with Error: [Call_Process_FromXoomTalk_UsingCursor]' ;
PRINT '=========================================================================';
ROLLBACK;
PRINT '################################ ';
PRINT 'ROLLBACK DONE';
PRINT '################################ ';
IF #Raise1Print2Both3 = 1 OR #Raise1Print2Both3 = 3
THROW;
END CATCH;
END
Unfortunately sometimes you have to change your tactics when you move from Oracle to SQL Server.
Back to your case, if you are processing a large data set and you want to catch the bad rows, try give your raw imported table an unique id column, start from the smallest id, copy limited rows to the new table (use cast, not try_cast) each time. When you see an error, the insert will fail and you can reduce the number of insert rows until you reach the bad row. Move it to a "bad row" and continue try other data. After all rows copied, you have one table for the good rows and one for the bad rows. Then deal with the bad ones.
It may not be the fast solution but you don't have to spend too many times perfecting your script and basically it's a one-pass solution.
Here is the concept :
Alter table imported_table add lid int identity(1,1) not null primary key;
select top 1 into bad_rows from imported_table ;
truncate table bad_rows;
Then run this code:
declare #last_id int, #r int,#c int;
select #last_id= 0; #r = 100;
while #last_id >=0 begin
begin try
insert into new_table (.....)
select top(#r) .... from imported_table where lid>#last_id order by lid;
select #c=##rowcount;
if #c<1 select #last_id=-1 --- end of loop
else select #last_id=#last_id + #c;
select #r = 100;
end try
begin catch
if #r <2 begin
insert into bad_rows(...)
select top(#r)... from imported_table
where lid>#last_id order by lid;
select #lid=#lid+1;
end
else select #r=#r / 2;
end catch;
end;
Others have stated that using temp tables is a possibility.
If you want to easily create a temp table with the same data types as your source, you can use Select INTO syntax such as
Select *
INTO #tmp
From mytable
Where 1 = 0
You can use this with only select columns, joins, etc to create a temporary working space without having to declare variables and data types
I have a stored procedure in production that does 2 things. It updates one table and then inserts a record into another. The first step (the update) seems to occur but we've found instances by examining the data where the second step did not occur. I have looked at the data and confirmed that it is not a data issue. I've confirmed that the queries return the appropriate data in order to ensure that the queries complete and in normal circumstances both should execute. I don't know if perhaps there is some sort of performance issue ... or blocking issue that is occurring on the second step that prevents that step from occurring.
The error handling for the stored procedure is as follows.
BEGIN TRY
BEGIN TRANSACTION;
-- perform update to data
-- insert record into second table.
IF ( ##ERROR = 0 AND ##TRANCOUNT > 0 )
COMMIT TRANSACTION;
END TRY
BEGIN CATCH
IF ( ##TRANCOUNT > 0 )
BEGIN
ROLLBACK TRANSACTION;
END
DECLARE #WebSafeErrorId INT;
EXEC dbo.spErrorInsert #WebSafeErrorId OUTPUT, 'Proc';
-- Reraise the error back to the client.
IF ( #WebSafeErrorId != 0 )
BEGIN
DECLARE #Error VARCHAR(20);
SET #Error = CAST( #WebSafeErrorId AS VARCHAR(20) );
RAISERROR( #Error, 11, 1 );
END
ELSE
BEGIN
RAISERROR( 'An error has occurred but there is no error to log.', 11, 1 );
END
END CATCH;
Surely if an error occurred in this procedure that cause the insert to not occur it would be logged and then raised. The code for spErrorInsert is below ...
CREATE PROCEDURE [dbo].[spErrorInsert]
#ReturnErrorId INT OUTPUT
, #ErrorSourceType VARCHAR(4) = NULL
, #ParentErrorId INT = NULL
, #StackTrace VARCHAR(MAX) = NULL
AS
SET NOCOUNT ON;
--SET XACT_ABORT ON;
-- Will indicate an error was not logged.
SET #ReturnErrorID = 0;
DECLARE
#ErrorSource VARCHAR(200)
, #ErrorMessage VARCHAR(MAX)
, #ComposedErrorMessage VARCHAR(MAX)
, #ErrorLine INT
, #ErrorSeverity INT
, #ErrorState INT
, #ErrorNumber INT;
SET #ErrorSource = ERROR_PROCEDURE();
SET #ErrorMessage = ERROR_MESSAGE();
SET #ErrorLine = ERROR_LINE();
SET #ErrorSeverity = ERROR_SEVERITY();
SET #ErrorState = ERROR_STATE();
SET #ErrorNumber = ERROR_NUMBER();
SET #ComposedErrorMessage = 'Message: Error occurred in procedure ' + CAST( #ErrorSource AS VARCHAR(MAX) )
+ ' on line ' + CAST( #ErrorLine AS VARCHAR(MAX) )
+ '. Error: ' + #ErrorMessage;
BEGIN TRY
INSERT INTO Errors(
ParentId
, ErrorSourceType
, ErrorSource
, [Message]
, [LineNo]
, Severity
, Stacktrace
, ts)
VALUES (#ParentErrorId
, #ErrorSourceType --#ErrorSourceType --- NOTE: move this into a parameter ...
, #ErrorSource
, #ComposedErrorMessage
, #ErrorLine
, #ErrorState
, #Stacktrace
, GETDATE()
);
SET #ReturnErrorId = SCOPE_IDENTITY();
END TRY
BEGIN CATCH
RAISERROR( 'An error has occurred but there is no error to log.', 11, 1 );
END CATCH;
I don't know if maybe there is a way to get a snapshot of what's going on the database at a specific time when a certain procedure is called ... I'm not sure how to determine if something isn't happening when it should? Are there any tools that I can make use of or sql features that I don't know about???
If you want to monitor the database, SQL Profiler is a good place to start but it is going to be deprecated.
Extended events are much more capable and I would suggest reading about those if you are really interested in monitoring what's going on.
As a thought, if your procedure code is using the same data to update the row as it is to insert to the other table, consider using OUTPUT.
Update table set col1 = 'value'
OUTPUT inserted.col INTO othertable
Where col3 = stuff
OUTPUT and OUTPUT INTO
Or if this is for some sort of Audit or Log table, you can use DELETED.col1
That will be the original value prior to it being updated. Note that INSERTED will return the value that you are updating or inserting, it's just called INSERTED for both.
If you have a copy of Visual Studio, try it. It allows you to step through stored procedures.
The approach I would try is to firstly take a copy of this procedure and comment out the try/catch. I have found that tsql does not raise errors if the error generating code is within a try/catch block - I guess this is the sql equivalent of an exception being handled by the catch clause.
I use a table in my database to keep a permanent record of errors as they occur (a trick I learned doing embedded programming)
The errors table creation code is :
CREATE TABLE dbo.errors (
id smallint NOT NULL IDENTITY(1, 1) PRIMARY KEY,
errordesc nvarchar (max) NOT NULL,
dateandtime smalldatetime NOT NULL, -- Date and time of last occurance
errorcount int NOT NULL) ;
My stored procedure for adding a record into the error table is:
CREATE PROCEDURE jc_diagnostics.jpwsp0005_adderrordescription(
#Errordescription nvarchar( max ))
AS
BEGIN
DECLARE
#Id smallint = 0 ,
#Currentcount int = 0;
IF((#Errordescription IS NULL) OR ( #Errordescription = ''))
BEGIN
SET #Errordescription = 'Error description missing';
END;
SELECT #Id = ( SELECT TOP ( 1 ) id
FROM jc_diagnostics.errors
WHERE errordesc = #Errordescription );
IF(#Id IS NOT NULL)
BEGIN
SELECT #Currentcount = (SELECT errorcount
FROM jc_diagnostics.errors
WHERE id = #Id );
SET #Currentcount = #Currentcount + 1;
UPDATE jc_diagnostics.errors
SET errorcount = #Currentcount
WHERE id = #Id;
UPDATE jc_diagnostics.errors
SET dateandtime = CONVERT(smalldatetime , GETDATE())
WHERE id = #Id;
END;
ELSE
BEGIN
--new entry
INSERT INTO jc_diagnostics.errors( errordesc ,
dateandtime ,
errorcount )
VALUES( #Errordescription ,
CONVERT(smalldatetime , GETDATE()) ,
1 );
END;
IF(#Id IS NULL)
BEGIN
SET #Id = SCOPE_IDENTITY( );
END;
RETURN #Id;
END;
The calling code when an error occurs is:
Declare #Failuredesc nvarchar(max) = 'Description of error';
EXEC #Retval = jc_diagnostics.jpwsp0005_adderrordescription #Failuredesc;
The return value #Retval contains the id of the record in the error table so you can look it up
Finally I would create some code to continuously call your procedure until an error is declared. You can then inspect the error table and see if this throws light on your problem.
Hope this helps.
Jude
Logically thinking - because you declare transaction before these 2 steps, any error would result in rollback of both transactions. So most likely there is no error at all here. I would suggest inspect your queries again as it seems that the problem could be in them rather than anywhere else. Please post the entire code if you like more suggestions.
Regards
Roman