So I have been struggling with handling transactions in SSIS. My requirement is to achieve transaction without enabling MSDTC service and I have partially achieved that but I just got another error which I feel like is one of the many bugs in SSIS. I used execute SQL task and explicitly mentioned begin tran and commit/rollback tran in my package. My package is working fine. All the tables are enclosed in a sequence container. I have a condition where one output from one table goes in 2 different tables and that's where the problem is. The funny part is even the package fails, I will still see insert in only these two tables. SSIS is shown in the attached image. I have disabled two tables. These two tables take input from Frholdsum and even if the package fails and there is no data in FDR holdssum tables. Microsft never ceases to amaze me :(. enter image description here
Set RetainSameConnection on your ConnectionManager to true.
https://munishbansal.wordpress.com/2009/04/01/how-to-retain-same-data-connection-across-multiple-tasks-in-ssis/
It's working fine if I explicitly write delete statements after rollback ran like this:
rollback tran; delete from dbo.UCOP_ENDOW_INVEST; delete from dbo.ucop_fdr_attrib ;
I shouldn't have to do this though :(
Related
I was testing possible issues with a query when the connection is lost or timed out. To do the test, I run a query with a fresh connection, and just seconds after I kill the program or disconnect the network. Then I check the impact of the query.
I believe that if a query, not within the explicit transaction fails for any reason, will roll back the effect. Of course, this makes sense for operations like DELETE, INSERT, UPDATE or DDL statements too. Implicit Transaction is OFF in the db.
My theory held true except when I ran a SELECT-INTO statement. Sample query that I tried -
SELECT * INTO test_table FROM audit
It failed due to Socket read timeout but later I found that even though there are no records inserted, the new table test_table was created as empty.
After browsing the docs for a while, according to official documentation, it's expected behavior. That's understandable. But the problem for me is that I can't really retry this query execution as the table already exists.
I guess to fix this I need to use the explicit transaction around such statements.
To help me with the feature - Am I going the right way? And are there any other SQL statements that can cause similar behavior?
Thanks in advance.
Edit:
Since I got suggestions on how to fix this, I am wondering now if there are any other SQL statements that can cause similar behavior.
1st .)
I have a Sequence container.
It has 4 different execute sql tasks and 4 different DFT where data is inserting into different tables .
I want to implement transaction with or without MSDTC service on the package failure i.e., each and every data should be rollback on failure of any of the DFT or execute SQL task .
How to implement it? when I am trying to implement with MSDTC service I get the "OLEDB Connection" error and without MSDTC the data is getting inserted only the last execute Sql task is getting rolled back . How to implement this on ssis 2017?
2nd.)
when I tried without MSDTC by setting the property of ServerConnection RetainSameConnection as TRUE and took two more execute sql task for begin transaction and commit. I faced a issue with the EVENT HANDLER i.e., I was not able to log error into different table. Either the Rollback is working or the Event Handler when tried to manipulate.
Soon as the error occurred the control goes to the event handler and then Rollback every thing including task in event handler
3rd.)
The Sequence Container is used for parallel execution of tasks. So the particular task among the 4 getting failed only that particular task getting rolled back rest SQL task was inserting data into tables.
Thanks in Advance!! ;-)
One option I've used (without MSDTC) is to configure your OLEDB conection as RetainSameConnection=True
(Via Properties Window)
Then Begin a Transaction Before your Sequence Container & Commit afterwards (all sharing the same OLEDB Connection.
Works quite well & pretty easy to implement.
According to my Scenario :
I used a sequence container(which contains different DFT's and tasks),and have taken 3 more Execute sql task :
1st Begin Transaction T1(before the sequence container)
2nd Commit transaction T1(after the sequence container)
3rd Rollback transaction T1(after the sequence container) with precedence as failure i.e, only when the sequence container fails the Execute Sql task containing rollback operation executes.
Note : I tried to rollback this way but only the current execute sql task i.e., the nearest to it was getting rolled back rest of the data were inserted. So whats the solution? In that same execute sql task I have truncated the table where rows were getting inserted. So, when the sequence container fails the execute sql task will truncate all the data in the respective table. (Rollback transaction T1 go truncate table Table_name1 go truncate table table_name2 go truncate table table_name3)
*IMPORTANT :***To make the above operation work make sure in the connection manager properties **RetainSameConnection is set to True by default it's false.
Now, to log errors into user defined tables we make use of event handler.So the scenario was, when the sequence container gets failed everything gets rolled back including the table used in execute sql task in the event handler. So whats the solution?
When you are not using SSIS transaction properties, by default every task have the properties set to supported. The Execute sql task in the Event handler also has the same properties as Supported so it follows the same transaction.To make the event handler work properly change the connection of the Execute Sql Task i.e, take a different connection and set its TransactionProperty to NotSupported. Thus, it will not follow the same transaction and when any error occurs it will log the errors into the table.
Note : we are using sequence container for parallel execution of tasks.What if? the error occurs inside the sequence container in any of the task and task does not allow to sequence container to fail.In that case, connect all the task serially.Yes, that makes no sense of the sequence container.I found my solution to work that way.
Hope it helps all! ;-)
I have integrated with a SQL Server database using Talened. The jobs insert, update and delete records along with many reads.
Recently, the ERP using this database was upgraded and now we have to run a Set statement before executing any queries in order to perform the transactions.
The Set statement looks like this:
Set Contact_Info [binary value]
I cannot figure out a way to do this in Talend. Doesn't seem to be possible in a tMSSQLOutput block. Any suggestions?
The components tMSSqlOutput and tMSSqlInput are not designed to set properties or execute commands, You to use the tMSSqlRow component to execute such command.
I followed Balazs' approach and it worked perfectly. I added a connection block and the row block. I set the row block and my update/insert block to both use the connection block. The connection is also set to autocommit. Worked like a charm.
I have successfully deployed many SSIS packages (in SQL Server 2008 R2).
I have discovered an issue that is stumping me. Examine the basic following flow.
In 'Download Files from FTP' I pick up the new files I want to operate on and download them to my local drive, saving the picked up files so they can be iterated upon in the Process Each File foreach container.
In 'Create Temp Table' I am creating a temp table as such:
IF not exists (SELECT * FROM tempdb.dbo.sysobjects WHERE name='##tempProcessFiles' and xtype='U')
CREATE TABLE ##tempProcessFiles
(
--my columns
)
GO
In 'Truncate Temp Table' I am doing:
TRUNCATE TABLE ##tempProcessFiles
Basically, I pick up some files, create a temp table and then loop through each file, loading the necessary junk to my database. In order to make sure everything runs smoothly, I truncate the temp table on each iteration so I have a fresh table to work with. On the very last step of this package, I drop the temp table. I also drop the temp table OnError.
The problem is, when I schedule this guy, create temp table executes fine but once it reaches 'truncate temp table', it throws an exception saying the temp table does not exist. Specifically:
Executing the query "TRUNCATE TABLE ##tempProcessFiles" failed with
the following error: "Cannot find the object "##tempProcessFiles"
because it does not exist or you do not have permissions.". Possible
failure reasons: Problems with the query, "ResultSet" property not set
correctly, parameters not set correctly, or connection not established
correctly.
Running this package in debug mode does not recreate this scenario. Everything works fine.
I discovered (after many days of frustration, chasing ideas that sent me in circles) that by removing the GO statement in 'Create Temp Table' my package would execute normally.
It was determined that there were no errors or no accidental dropping of my temp table. Plus, I had persist security info set to false like always, delay validation set to true like always; but my drop temp table task wasn't firing - I even tried deploying it with it disabled. None of my other packages used this global temp table and the database had no scheduled jobs or triggers dropping it or anything ridiculous like that.
All I can determine is that the GO operator here was causing my session to the database to terminate and cause my temp table to be dropped immediately. Is this how GO operators works in SSIS? If I were to run a script on SQL Server using the exact same syntax as I have in my package, I definitely would not experience this kind of thing, so it's thrown me for a loop. But I'm not a DBA, so it's probably something fundamental or subtle to that area of work.
Can anyone explain what really happened here? Or would I need to provide more detail about my package to get a sufficient answer? The only change I did was removing GO from 'Create Temp Table', so that was the definite fix.
It seems that the connection is being dropped after all, despite RetainSameConnection.
To gather more information, try following:
Run a trace while executing the package to confirm the above (add Security Audit > Login/Logout and Sessions > Existing connections). Pay attention to StartTime and EndTime, maybe it'll be something interesting like 30s (default command timeout)?
Then try running this package with Crate and Truncate tasks only (without the loop) to eliminate a possibility that other components are interfering (Data Sources using same Connection Manager?). If it works, try it with a loop, but with Truncate task only placed inside, and keep adding other components.
I was reading an article the other day the showed how to run SQL Update, Insert, or Deletes as a whatif type scenario. I don't remember the parameter that they talked about and now I can't find the article. Not sure if I was dreaming.
Anyway, does anyone know if there is a parameter in SQL2008 that lets you try an insert, update, or delete without actually committing it? It will actually log or show you what it would have updated. You remove the parameter and run it if it behaves as you would expect.
I don't know of a SQL2008 specific feature with any SQL service that supports transactions you can do this:
Start a transaction ("BEGIN TRANSACTION" in TSQL)
The rest of your INSERT/UPDATE/DELETE/what-ever code
(optional) Some extra SELECT statements and such if needed to output the result of the above actions, if the default output from step 2 (things like "X rows affected") is not enough
Rollback the transaction ("ROLLBACK TRANSACTION" in TSQL)
(optional) Repeat the testing code to show how things are without the code in step 2 having run
For example:
BEGIN TRANSACTION
-- make changes
DELETE people WHERE name LIKE 'X%'
DELETE people WHERE name LIKE 'D%'
EXEC some_proc_that_does_more_work
-- check the DB state after the changes
SELECT COUNT(*) FROM people
-- undo
ROLLBACK TRANSACTION
-- confirm the DB state without the changes
SELECT COUNT(*) FROM people
(you might prefer to do the optional "confirm" step before starting the transaction rather than after rolling it back, but I've always done it this way around as it keeps the two likely-to-be-identical sections of code together for easier editing)
If you use something like this rather then something SQL2008 specific the technique should be transferable to other RDBS too (just update the syntax if needed).
OK, finally figured it out. I've confused this with another project I was working on with PowerShell. PowerShell has a "whatif" parameter that can be used to show you what files would be removed before they are removed.
My apologies to those who have spent time trying to find an answer to this port and my thanks to those of you who have responsed.
I believe you're talking about BEGIN TRANSACTION
BEGIN TRANSACTION starts a local transaction for the connection issuing the statement. Depending on the current transaction isolation level settings, many resources acquired to support the Transact-SQL statements issued by the connection are locked by the transaction until it is completed with either a COMMIT TRANSACTION or ROLLBACK TRANSACTION statement. Transactions left outstanding for long periods of time can prevent other users from accessing these locked resources, and also can prevent log truncation.
Do you perhaps mean SET NOEXEC ON ?
When SET NOEXEC is ON, SQL Server
compiles each batch of Transact-SQL
statements but does not execute them.
When SET NOEXEC is OFF, all batches
are executed after compilation.
Note that this won't warn/indicate things like key violations.
Toad for SQL Server has a "Validate SQL" feature that checks queries against wrong table/column names etc. . Maybe you are talking about some new feature in SSMS 2008 similar to that...
I'm more than seven years late to this particular party but I suspect the feature in question may also have been the OUTPUT clause. Certainly, it can be used to implement whatif functionality similar to Powershell's in a t-sql stored procedure.
https://learn.microsoft.com/en-us/sql/t-sql/queries/output-clause-transact-sql
Use this in each insert/update/delete/merge query to let the SP output a meaningful resultset of the changes it makes e.g. outputting the table name and action performed as the first two columns then all the altered columns.
Then simply rollback the changes if a #whatif parameter is set to 1 or commit them if #whatif is set to 0.