Multiple SQL Server inserts fail, but go unnoticed in Mule - sql-server

When I run multiple insert queries together into a SQL Server database with Mule, if the second insert fails, it doesn't insert a row, and won't show as a failure in the flow or logs.
We use variables to collect together different SQL statements to insert into a header and detail table. I noticed last week that in some cases, the header record was there but no detail. There was nothing in the logs for this.
After some investigation it appears that Mule will take the result of the first SQL insert as the return code, regardless of whether the resulting SQL inserts worked or not.
I've tried changing this to a BULK UPDATE but i still get the same result.
Edit - code included for a sample insert. 4 insert statements, 3 will be successful, 1 will fail, but will simply pass back as successful -
insert into highjump.t_import_order(status,idoc_number,datetime_created,datetime_processed,error_message,wh_id,order_number,order_type,order_subtype,is_vas,is_shrinkwrap,is_mhe_packhold,is_consolidation,is_nonmhe_packhold,is_full_case,ship_to_account,ship_to_name,ship_to_address1,ship_to_address2,ship_to_address3,ship_to_city,ship_to_state,ship_to_zip,ship_to_country,sold_to_account,sold_to_name,sold_to_address1,sold_to_address2,sold_to_address3,sold_to_city,sold_to_state,sold_to_zip,telephone_number,sold_to_country,stock_pool,discount,box_type,service_level,telephone_number_alt,dest_type,carrier_code,route_code,inv_cat,cust_order_date,expected_ship_date,expected_delivery_date,dsv_tracking_number,postage_cost,carton_contents_type,unit_total,total_before_discount,total_after_discount,carton_cubing_indicator,req_proof_of_delivery,payment_type,is_cms,carrier_override_type,sales_org,pack_note_preference,shipper_order_id,master_order_number,currency_code,store_code,order_method,dsv_reference,email_address,ship_complete_flag,replen_type,carton_content_flags,partner_profile) values
(N'Z',N'0000000629673252','2019-04-12 09:57:38','2019-04-12 09:57:38',null,N'WST',N'6412210697',N'MCR',N'STD EU',0,0,0,0,0,0,N'MCRSHPTODE',N'Dave Smith',
N'888415936',N'PACKSTATION 432',null,N'Koettgenstr. 8',null,N'13629',N'DE',N'MCRSLDTODE',N'MCR SOLD TO DE',N'High St.',null,null,N'Street',null,N'BA330YA',null,
N'GB',N'MC01',0,N'BAG',N'10',null,N'RE','',N'01',N'W','2019-03-29 11:38:13','2019-03-29 11:38:13','2019-03-29 11:38:13',null,0,N'001',2,null,null,'91',1,
N'MCR CON - UK Orders',1,'1',null,N'N',null,N'623611121','GBP',null,null,null,N'Smith#arcor.com',null,'R',N'F', N'WWMULESFTH');
insert into highjump.t_import_order_cms
(order_id,delivery_from_date,delivery_to_date,pin_number,cms_location,cms_delivery_endpoint,cms_comm_preference,cms_dont_despatch_before,cms_market,cms_brand,is_gift,gift_message,loyalty_number,cms_dest_type,cms_time_delivery,cms_day_delivery,cms_customer_type,carrier_service_name,special_instructions) values ((select top(1) order_id from highjump.t_import_order where order_number='6412210697'),'2019-04-03','2019-04-03',null,N'432',N'PACKSTATIONPACKSTATION',null,null,null,N'CLA',null,null,null,N'PUDO',null,null,null,null,null);
insert into highjump.t_import_order_detail
(order_id,line_number,item_number,order_quantity,customer_item_number,ratio_pack_group,is_ratio_pack,ratio_pack_qty,uom,retail_price,freight_class,sales_order_number,customer_order_number,dsv_price_discount,customer_item_colour,price_paid,currency_code,customer_item_size)
values ((select top(1) order_id from highjump.t_import_order where order_number='6412210697'),00010,'261392464080',1.000,null,null,null,null,'U','0.0',null,N'623611121000010',N'623611121',null,null,99.95,null,null);
insert into highjump.t_import_order_detail (order_id,line_number,item_number,order_quantity,customer_item_number,ratio_pack_group,is_ratio_pack,ratio_pack_qty,uom,retail_price,freight_class,sales_order_number,customer_order_number,dsv_price_discount,customer_item_colour,price_paid,currency_code,customer_item_size)
values ((select top(1) order_id from highjump.t_import_order where order_number='6412210697'),00020,'261394324080',1.000,null,null,null,null,'U','0.0',null,N'623611121000020',N'623611121',null,null,89.95,null,null);


Structurally, these SQL queries seem to be fine. It is unclear to me why and how any of these queries would fail (or would not insert any data). It should just work fine, as far as I can see.
In the end, when you execute these queries in SQL Server Management Studio, they should all return value 1:
select count(*) from highjump.t_import_order where order_number = '6412210697';
select count(*) from highjump.t_import_order_cms where order_id = (select top (1) order_id from highjump.t_import_order where order_number = '6412210697');
select count(*) from highjump.t_import_order_detail where line_number = 10 and order_id = (select top (1) order_id from highjump.t_import_order where order_number = '6412210697');
select count(*) from highjump.t_import_order_detail where line_number = 20 and order_id = (select top (1) order_id from highjump.t_import_order where order_number = '6412210697');


Use transaction for multiple insert queries execution. In multiple SQL queries suppose one query gives error then it will be roll back.
BEGIN TRY
BEGIN TRANSACTION
//Here you will write multiple insert/delete/update queries
COMMIT
END TRY
BEGIN CATCH
ROLLBACK
END CATCH

Related

SQL trigger for audit table getting out of sync

I recently created a SQL trigger to replace a very expensive query I used to run to reduce the amount of updates my database does each day.
Before I preform an update I check to see how many updates have already occurred for the day, this used to be done by querying:
SELECT COUNT(*) FROM Movies WHERE DateAdded = Date.Now
Well my database has over 1 million records and this query is run about 1-2k a minute so you can see why I wanted to take a new approach for this.
So I created an audit table and setup a SQL Trigger to update this table when any INSERT or UPDATE happens on the Movie table. However I'm noticing the audit table is getting out of sync by a few hundred everyday (the audit table count is higher than the actual updates in the movie table). As this does not pose a huge issue I'm just curious what could be causing this or how to go about debugging it?
SQL Trigger:
ALTER TRIGGER [dbo].[trg_Audit]
ON [dbo].[Movies]
AFTER UPDATE, INSERT
AS
BEGIN
UPDATE Audit SET [count] = [count] + 1 WHERE [date] = CONVERT (date, GETDATE())
IF ##ROWCOUNT=0
INSERT INTO audit ([date], [count]) VALUES (GETDATE(), 1)
END
The above trigger only happens after an UPDATE or INSERT on the Movie table and tries to update the count + 1 in the Audit table and if it doesn't exists (IF ##ROWCOUNT=0) it then creates it. Any help would be much appreciated! Thanks.

Something like this should work:
create table dbo.Movies (
A int not null,
B int not null,
DateAdded datetime not null
)
go
create view dbo.audit
with schemabinding
as
select CONVERT(date,DateAdded) as dt,COUNT_BIG(*) as cnt
from dbo.Movies
group by CONVERT(date,DateAdded)
go
create unique clustered index IX_MovieCounts on dbo.audit (dt)
This is called an indexed view. The advantage is that SQL Server takes responsibility for maintaining the data stored in this view, and it's always right.
Unless you're on Enterprise/Developer edition, you'd query the audit view using the NOEXPAND hint:
SELECT * from audit with (noexpand)
This has the advantages that
a) You don't have to write the triggers yourself now (SQL Server does actually have something quite similar to triggers behind the scenes),
b) It can now cope with multi-row inserts, updates and deletes, and
c) You don't have to write the logic to cope with an update that changes the DateAdded value.

Rather than incrementing the count by 1 you should probably be incrementing it by the number of records that have changed e.g.
UPDATE Audit
SET [count] = [count] + (SELECT COUNT(*) FROM INSERTED)
WHERE [date] = CONVERT (date, GETDATE())
IF ##ROWCOUNT=0
INSERT INTO audit ([date], [count])
VALUES (GETDATE(), (SELECT COUNT(*) FROM INSERTED))

Get the highest updated ID of a multi row update

Update: I am using Sql Server 2008 R2.
I am going to update a large number of rows and to avoid unnecessary locking I will do this in matches of around a thousand lines per update.
Using SET ROWCOUND I can limit the update to 1000 lines and using WHERE ID > x I can set which batch it should run.
But for this to work I need to know the highest ID from the just processed batch.
I could user OUTPUTto return all affected ID's and find the highest one on code but I would like to be able to return just the highest ID.
I tried this
SELECT MAX(id)
FROM (
UPDATE mytable
SET maxvalue = (SELECT MAX(salesvalue) FROM sales WHERE cid = t.id GROUP BY cid)
OUTPUT inserted.id
FROM mytable t
WHERE au.userid > 0
) updates(id)
But it gives me this error
A nested INSERT, UPDATE, DELETE, or MERGE statement is not allowed in a SELECT statement that is not the immediate source of rows for an INSERT statement.
BUT if I try to insert the result into a table directly it is valid
CREATE TABLE #temp(id int)
INSERT INTO #temp
SELECT MAX(id)
FROM (
UPDATE mytable
SET maxvalue = (SELECT MAX(salesvalue) FROM sales WHERE cid = t.id GROUP BY cid)
OUTPUT inserted.id
FROM mytable t
WHERE au.userid > 0
) updates(id)
drop table #temp
Is there any workaround to this and can anyone explain why I can insert the result into a table but not just return the result?

DO NOT USE SET ROWCOUNT for this (or, at all), as BOL says:
Using SET ROWCOUNT will not affect DELETE, INSERT, and
UPDATE statements in the next release of SQL Server!
Do not use SET
ROWCOUNT with DELETE, INSERT, and UPDATE statements in new development
work, and plan to modify applications that currently use it. Also, for
DELETE, INSERT, and UPDATE statements that currently use SET ROWCOUNT,
we recommend that you rewrite them to use the TOP syntax.
You could do it with a table variable, too:
DECLARE #Log TABLE (id INT NOT NULL);
UPDATE TOP 1000 mytable
SET maxvalue = (SELECT MAX(salesvalue) FROM sales WHERE cid = t.id GROUP BY cid)
OUTPUT inserted.id INTO #Log
FROM mytable t
WHERE au.userid > 0
SELECT maxid = MAX(id)
FROM #Log

Weird Behavior of Trigger

I've got a trigger (SQL 2008 R2) that does one simple operation but the results are not logical.
Here is the overview:
A text file is fed to an SSIS package with ONE line(one record) that loads it into "ORDERS_IN_PROCESS" table. The Data Access Mode is set to "Table or view" in the "OLE DB Destination" to allow triggers to fire.
Here is my ORDERS table:
OrderID ItemNo
--------- ---------
9813 1
9813 2
9813 3
9817 1
So, SSIS executes and
ORDERS_IN_PROCESS gets one record inserted which is OrderID 9813
Trigger is fired:
INSERT INTO ORDERS_ARCHIVE SELECT * FROM ORDERS WHERE OrderID=INSERTED.OrderID
Pretty simple so far...
The results I get in my ORDERS_ARCHIVE (identical layout to ORDERS) Table are
OrderId ItemNo
--------- ----------
9813 3
Where are the rest of the 2 line items?
Note, it only inserted the last row read from ORDERS table into ORDERS_ARCHIVE.
I need all 3 of them in ORDERS_ARCHIVE.
Why does this happen?
I believe it has something to do with the way SSIS processes it using "OLE DB Destination" because if I insert a record into RLFL manually, the trigger does exactly what it's supposed to do and inserts all 3 records from BACK.
You may argue that trigger fires once per batch and I agree but in this case I have a batch of just ONE record.
I'm thinking of an sp, but i'd rather not add another level of complexity for something so trivial, supposedly.
Any ideas?
Thanks.

I don't think that this has something to do with SSIS at all, but with your trigger. Instead of than IN that you have there, try using a JOIN in your query:
INSERT INTO ORDERS_ARCHIVE
SELECT O.*
FROM ORDERS O
INNER JOIN INSERTED I
ON O.ORderID = I.OrderID

I concur with Lamark, misspelling intentional, with the assessment of your trigger is incorrect.
The logic you provided for your trigger does not compile. The WHERE clause is not valid. I'm assuming, as Lamak did that your intention was to join based on OrderID.
create table dbo.ORDERS_ARCHIVE
(
OrderID int
, ItemNo int
)
GO
create table dbo.ORDERS
(
OrderID int
, ItemNo int
)
GO
create trigger
trUpdate
ON
dbo.ORDERS
AFTER INSERT
AS
BEGIN
SET NOCOUNT ON;
-- This doesn't work
-- Msg 4104, Level 16, State 1, Procedure trUpdate, Line 12
-- The multi-part identifier "INSERTED.OrderID" could not be bound.
--INSERT INTO dbo.ORDERS_ARCHIVE
--SELECT *
--FROM ORDERS
--WHERE OrderID=INSERTED.OrderID;
-- I think you meant
INSERT INTO dbo.ORDERS_ARCHIVE
SELECT *
FROM ORDERS
WHERE OrderID=(SELECT INSERTED.OrderID FROM INSERTED);
END
GO
I then ginned up a simple SSIS package, I have a data source that supplies the 4 rows you indicated and writes to dbo.ORDERS. I ran the package 2 times and each one netted 4 rows in the ORDERS_ARCHIVE table. 3 rows with 9813, 1 with 9817 per batch.
I am getting the right count of rows in there so I believe the trigger is firing correctly. Instead, what is happening is the logic is incorrect. Since the OrderID is not unique in the ORDERS table, the database engine is going to pick the first row that happens to satisfy the search criteria. It just so happens that it picks the same row (ItemNo = 1) each time but since there is no guarantee of order without an ORDER BY clause, this is just random or an artifact of how the Engine chooses but no behaviour I would bank on remaining consistent.
How do you fix this?
Fix the trigger. Joining to the inserted virtual table only on the OrderID is resulting in multiple rows satisfying the condition.
create trigger
trUpdate
ON
dbo.ORDERS
AFTER INSERT
AS
BEGIN
SET NOCOUNT ON;
-- This trigger will add all the rows from the ORDERS table
-- that match what was just inserted based on OrderID and ItemNo
INSERT INTO dbo.ORDERS_ARCHIVE
SELECT O.*
FROM dbo.ORDERS O
INNER JOIN INSERTED I
ON O.OrderID = I.OrderID
AND O.ItemNo = I.ItemNo;
END
Now when I run the ETL, I see 4 rows in ORDERS_ARCHIVE with the correct ItemNo values.

Select and Delete in the same transaction using TOP clause

I have table in which the data is been continuously added at a rapid pace.
And i need to fetch record from this table and immediately remove them so i cannot process the same record second time. And since the data is been added at a faster rate, i need to use the TOP clause so only small number of records go to business logic for processing at the time.
I am using the below query to
BEGIN TRAN readrowdata
SELECT
top 5 [RawDataId],
[RawData]
FROM
[TABLE] with(HOLDLOCK)
WITH q AS
(
SELECT
top 5 [RawDataId],
[RawData]
FROM
[TABLE] with(HOLDLOCK)
)
DELETE from q
COMMIT TRANSACTION readrowdata
I am using the HOLDLOCK here, so new data cannot insert into the table while i am performing the SELECT and DELETE operation. I used it because Suppose if there are only 3 records in the table now, so the SELECT statement will get 3 records and in the same time new record gets inserted and the DELETE statement will delete 4 records. So i will loose 1 data here.
Is the query is ok in performance term? If i can improve it then please provide me your suggestion.
Thank you

Personally, I'd use a different approach. One with less locking, but also extra information signifying that certain records are currently being processed...
DECLARE #rowsBeingProcessed TABLE (
id INT
);
WITH rows AS (
SELECT top 5 [RawDataId] FROM yourTable WHERE processing_start IS NULL
)
UPDATE rows SET processing_start = getDate() WHERE processing_start IS NULL
OUTPUT INSERTED.RowDataID INTO #rowsBeingProcessed;
-- Business Logic Here
DELETE yourTable WHERE RowDataID IN (SELECT id FROM #rowsBeingProcessed);
Then you can also add checks like "if a record has been 'beingProcessed' for more than 10 minutes, assume that the business logic failed", etc, etc.

By locking the table in this way, you force other processes to wait for your transaction to complete. This can have very rapid consequences on scalability and performance - and it tends to be hard to predict, because there's often a chain of components all relying on your database.
If you have multiple clients each running this query, and multiple clients adding new rows to the table, the overall system performance is likely to deteriorate at some times, as each "read" client is waiting for a lock, the number of "write" clients waiting to insert data grows, and they in turn may tie up other components (whatever is generating the data you want to insert).
Diego's answer is on the money - put the data into a variable, and delete matching rows. Don't use locks in SQL Server if you can possibly avoid it!

You can do it very easily with TRIGGERS. Below mentioned is a kind of situation which will help you need not to hold other users which are trying to insert data simultaneously. Like below...
Data Definition language
CREATE TABLE SampleTable
(
id int
)
Sample Record
insert into SampleTable(id)Values(1)
Sample Trigger
CREATE TRIGGER SampleTableTrigger
on SampleTable AFTER INSERT
AS
IF Exists(SELECT id FROM INSERTED)
BEGIN
Set NOCOUNT ON
SET XACT_ABORT ON
Begin Try
Begin Tran
Select ID From Inserted
DELETE From yourTable WHERE ID IN (SELECT id FROM Inserted);
Commit Tran
End Try
Begin Catch
Rollback Tran
End Catch
End
Hope this is very simple and helpful

If I understand you correctly, you are worried that between your select and your delete, more records would be inserted and the first TOP 5 would be different then the second TOP 5?
If that so, why don't you load your first select into a temp table or variable (or at least the PKs) do whatever you have to do with your data and then do your delete based on this table?

I know that it's old question, but I found some solution here https://www.simple-talk.com/sql/learn-sql-server/the-delete-statement-in-sql-server/:
DECLARE #Output table
(
StaffID INT,
FirstName NVARCHAR(50),
LastName NVARCHAR(50),
CountryRegion NVARCHAR(50)
);
DELETE SalesStaff
OUTPUT DELETED.* INTO #Output
FROM Sales.vSalesPerson sp
INNER JOIN dbo.SalesStaff ss
ON sp.BusinessEntityID = ss.StaffID
WHERE sp.SalesLastYear = 0;
SELECT * FROM #output;
Maybe it will be helpfull for you.

SQl Server Express 2005 - updating 2 tables and atomicity?

First off, I want to start by saying I am not an SQL programmer (I'm a C++/Delphi guy), so some of my questions might be really obvious. So pardon my ignorance :o)
I've been charged with writing a script that will update certain tables in a database based on the contents of a CSV file. I have it working it would seem, but I am worried about atomicity for one of the steps:
One of the tables contains only one field - an int which must be incremented each time, but from what I can see is not defined as an identity for some reason. I must create a new row in this table, and insert that row's value into another newly-created row in another table.
This is how I did it (as part of a larger script):
DECLARE #uniqueID INT,
#counter INT,
#maxCount INT
SELECT #maxCount = COUNT(*) FROM tempTable
SET #counter = 1
WHILE (#counter <= #maxCount)
BEGIN
SELECT #uniqueID = MAX(id) FROM uniqueIDTable <----Line 1
INSERT INTO uniqueIDTableVALUES (#uniqueID + 1) <----Line 2
SELECT #uniqueID = #uniqueID + 1
UPDATE TOP(1) tempTable
SET userID = #uniqueID
WHERE userID IS NULL
SET #counter = #counter + 1
END
GO
First of all, am I correct using a "WHILE" construct? I couldn't find a way to achieve this with a simple UPDATE statement.
Second of all, how can I be sure that no other operation will be carried out on the database between Lines 1 and 2 that would insert a value into the uniqueIDTable before I do? Is there a way to "synchronize" operations in SQL Server Express?
Also, keep in mind that I have no control over the database design.
Thanks a lot!

You can do the whole 9 yards in one single statement:
WITH cteUsers AS (
SELECT t.*
, ROW_NUMBER() OVER (ORDER BY userID) as rn
, COALESCE(m.id,0) as max_id
FROM tempTable t WITH(UPDLOCK)
JOIN (
SELECT MAX(id) as id
FROM uniqueIDTable WITH (UPDLOCK)
) as m ON 1=1
WHERE userID IS NULL)
UPDATE cteUsers
SET userID = rn + max_id
OUTPUT INSERTED.userID
INTO uniqueIDTable (id);
You get the MAX(id), lock the uniqueIDTable, compute sequential userIDs for users with NULL userID by using ROW_NUMBER(), update the tempTable and insert the new ids into uniqueIDTable. All in one operation.
For performance you need and index on uniqueIDTable(id) and index on tempTable(userID).
SQL is all about set oriented operations, WHILE loops are the code smell of SQL.

You need a transaction to ensure atomicity and you need to move the select and insert into one statement or do the select with an updlock to prevent two people from running the select at the same time, getting the same value and then trying to insert the same value into the table.
Basically
DECLARE #MaxValTable TABLE (MaxID int)
BEGIN TRANSACTION
BEGIN TRY
INSERT INTO uniqueIDTable VALUES (id)
OUTPUT inserted.id INTO #MaxValTable
SELECT MAX(id) + 1 FROM uniqueIDTable
UPDATE TOP(1) tempTable
SET userID = (SELECT MAXid FROM #MaxValTable)
WHERE userID IS NULL
COMMIT TRANSACTION
END TRY
BEGIN CATCH
ROLLBACK TRANSACTION
RAISERROR 'Error occurred updating tempTable' -- more detail here is good
END CATCH
That said, using an identity would make things far simpler. This is a potential concurrency problem. Is there any way you can change the column to be identity?
Edit: Ensuring that only one connection at a time will be able to insert into the uniqueIDtable. Not going to scale well though.
Edit: Table variable's better than exclusive table lock. If need be, this can be used when inserting users as well.

Resources