1st .)
I have a Sequence container.
It has 4 different execute sql tasks and 4 different DFT where data is inserting into different tables .
I want to implement transaction with or without MSDTC service on the package failure i.e., each and every data should be rollback on failure of any of the DFT or execute SQL task .
How to implement it? when I am trying to implement with MSDTC service I get the "OLEDB Connection" error and without MSDTC the data is getting inserted only the last execute Sql task is getting rolled back . How to implement this on ssis 2017?
2nd.)
when I tried without MSDTC by setting the property of ServerConnection RetainSameConnection as TRUE and took two more execute sql task for begin transaction and commit. I faced a issue with the EVENT HANDLER i.e., I was not able to log error into different table. Either the Rollback is working or the Event Handler when tried to manipulate.
Soon as the error occurred the control goes to the event handler and then Rollback every thing including task in event handler
3rd.)
The Sequence Container is used for parallel execution of tasks. So the particular task among the 4 getting failed only that particular task getting rolled back rest SQL task was inserting data into tables.
Thanks in Advance!! ;-)
One option I've used (without MSDTC) is to configure your OLEDB conection as RetainSameConnection=True
(Via Properties Window)
Then Begin a Transaction Before your Sequence Container & Commit afterwards (all sharing the same OLEDB Connection.
Works quite well & pretty easy to implement.
According to my Scenario :
I used a sequence container(which contains different DFT's and tasks),and have taken 3 more Execute sql task :
1st Begin Transaction T1(before the sequence container)
2nd Commit transaction T1(after the sequence container)
3rd Rollback transaction T1(after the sequence container) with precedence as failure i.e, only when the sequence container fails the Execute Sql task containing rollback operation executes.
Note : I tried to rollback this way but only the current execute sql task i.e., the nearest to it was getting rolled back rest of the data were inserted. So whats the solution? In that same execute sql task I have truncated the table where rows were getting inserted. So, when the sequence container fails the execute sql task will truncate all the data in the respective table. (Rollback transaction T1 go truncate table Table_name1 go truncate table table_name2 go truncate table table_name3)
*IMPORTANT :***To make the above operation work make sure in the connection manager properties **RetainSameConnection is set to True by default it's false.
Now, to log errors into user defined tables we make use of event handler.So the scenario was, when the sequence container gets failed everything gets rolled back including the table used in execute sql task in the event handler. So whats the solution?
When you are not using SSIS transaction properties, by default every task have the properties set to supported. The Execute sql task in the Event handler also has the same properties as Supported so it follows the same transaction.To make the event handler work properly change the connection of the Execute Sql Task i.e, take a different connection and set its TransactionProperty to NotSupported. Thus, it will not follow the same transaction and when any error occurs it will log the errors into the table.
Note : we are using sequence container for parallel execution of tasks.What if? the error occurs inside the sequence container in any of the task and task does not allow to sequence container to fail.In that case, connect all the task serially.Yes, that makes no sense of the sequence container.I found my solution to work that way.
Hope it helps all! ;-)
Related
I have an SSIS package that consist of 2 main blocks within Begin and Commit/Rollback transaction blocks
1. Truncate tables (with truncate table query)
2. Import Data (Import data from flat file and insert to these truncated tables)
When I run the package the job is getting hung. The activity monitor shows that there is a LCK_M_SCH_S lock created that blocks further execution.
Sometimes this work and sometimes not.
To add, if I truncate these table separately and run the package removing the truncate block it executes fine.
Also to add there are not just one Import Data component. We have around 6 import data component for 6 different table. For time being i kept only one in the screen shot
,
Looking at your screenshots, the first thing I'd verify is that the property RetainSameConnection is set to true on your OLE DB Connection manager (right click on CM, select Properties, find RetainSameConnection). The default for this is False.
If that resolves the issue, then the root cause was you had two requests in different transactions attempting to modify the same resource.
If you had already switched the Connection Manager's property to true, then my next guess would be that you want to set the DelayValidation property for the Data Flow "Import Data" to True.
If that resolves the issue, then the root cause was the component was attempting to validate the metadata for the target table and was getting blocked by the truncate statement (or vice versa). Setting DelayValidation will prevent the package from validating that specific task until the last possible second - giving other processes time to get out of the way. This seems less likely but it's the only other opportunity for the package to be blocking itself.
We have a number of configuration scripts which are a bit complicated. Those scripts are using a number of available stored procedure in our database to insert the configuration data. If any script attempts to insert invalid data, stored procedures will make a call to
RAISERROR #msg, 16, 1
And then the SP will return. The SPs start their own named transactions and they will rollback/commit the named transaction, however, LB doesn't detect the error raised and it takes the execution as a successful execution. But we don't want to continue if that happens.
failOnError in the changeSets are set to true but LB still proceeds. Even in DATABASECHANGELOG that failed changeset marked as executed as if it was successful.
We also tried removing nested transactions (named transactions), no luck.
We removed the name from the transaction and just using BEGIN TRAN, the LB execution stops at the incorrect script but the problem is, LB fails to commit its own transaction and can't release the lock so it remains LOCKED.
Is there anyway to tell LB that an error happened and make it stop?
We are using Liquibase 3.5.0. and Microsoft SQL Server
=== EDIT
So after debugging Liquibase, we found two things:
When connected to MS SQL Server, if a RAISERROR occurs and there are also resultsets in the script, it won't throw exception unless we make a call to statement.getMoreResults(). The same thing happens with Sybase (we tested it with Sybase too). So we thought maybe in LB, after executing the statement we need to make a call to getMoreResults() until it throws exception or it returns false which means no error happened.
A script makes a call to a stored procedure. The stored procedure, has 'BEGIN TRAN' and at the end it either COMMIT or ROLLBACK. If a rollback occurs, it also does RAISERROR. Note that our scripts don't do any update/insert, they are only providing the data in a temp table, so we don't do transaction handling in our scripts. In this scenario, consider we added code to make a call to getMoreResults(), the exception is throws correctly but then in LB, the executor tries to database.rollback() and then later again in StandardLockService, before releasing the lock, it tries to database.rollback() which ends in exception because our SP has rolled back the transaction already. This last rollback in LB, causes the error raised by JDBC to be swallowed and as the result not only do we see the error that caused it but also the lock remained unreleased and this is the most concern because even if we re-run the script and fix it, the lock hasn't been released and we need to do it manually.
One may argue that our transaction handling is not correct but all I am trying to say is that releasing lock should not be affecting if our script is incorrect. LB should be releasing the lock and throw exception or continue if a script/changeset is not run successfully.
If anybody is facing this too: In my case I had a very complex SQL script only for MS SQL Server. This also failed to stop the execution of the LB changes if an error occures in the SQL script, anyway if I use RAISERROR or THROW.
Things I need to do, to get it to work:
remove (or comment) all places where resultsets were created (SELECT)
start the SQL script with "SET NOCOUNT ON;" to avoid results from
insert or update (... lines affected)
end the SQL script with "SET NOCOUNT OFF;" to enable LB to work properly just after executing the SQL script (set EXECTYPE)
Use the precondition https://docs.liquibase.com/concepts/advanced/preconditions.html
create another changeset and check for the execution result before proceeding to next.
So I have been struggling with handling transactions in SSIS. My requirement is to achieve transaction without enabling MSDTC service and I have partially achieved that but I just got another error which I feel like is one of the many bugs in SSIS. I used execute SQL task and explicitly mentioned begin tran and commit/rollback tran in my package. My package is working fine. All the tables are enclosed in a sequence container. I have a condition where one output from one table goes in 2 different tables and that's where the problem is. The funny part is even the package fails, I will still see insert in only these two tables. SSIS is shown in the attached image. I have disabled two tables. These two tables take input from Frholdsum and even if the package fails and there is no data in FDR holdssum tables. Microsft never ceases to amaze me :(. enter image description here
Set RetainSameConnection on your ConnectionManager to true.
https://munishbansal.wordpress.com/2009/04/01/how-to-retain-same-data-connection-across-multiple-tasks-in-ssis/
It's working fine if I explicitly write delete statements after rollback ran like this:
rollback tran; delete from dbo.UCOP_ENDOW_INVEST; delete from dbo.ucop_fdr_attrib ;
I shouldn't have to do this though :(
Let me also back up a step - I'm trying to implement a sanity check inside an IS package. The idea is that the entire package runs in a read uncommitted transaction, with the final step being a check that determines that certain row counts are present, that kind of stuff. If they are NOT, I want to raise an exception and rollback the transaction.
If you can tell me how to do this, or, even better, suggest a better way to implement a sanity check, that would be great.
In order to fail the package if your observed rowcount differs from your expected rowcount:
Create a Package Global Variable to hold your expected rowcount. This could be derived from a RowCount step in your DFT, set manually, etc.
Edit your Execute SQL Task that provides the observed rowcount, and set the Result Set to Single Row.
In the Result Set tab of your Execute SQL Task, assign this Result Set to a variable.
Edit your constraint condition prior to your final step. Set the Evaluation Operation to Expression and Constraint. Set the Value to Failure. In your expression, evaluate ResultSetVariable <> ExpectedRowCountVariable.
If the observed rowcount does not equal the expected rowcount, the package will fail.
You can raise an error and roll back the transaction in an SSIS "Execute SQL" task with the following SQL:
Raiserror('Something went wrong',16,1)
This will cause the "Execute SQL" task to return the error to the SSIS package and the task will follow the "red" (failure) path. You can then use this path to rollback the transaction and do any tidying-up.
This approach has the advantage you can do all your processing in the Execute SQL task then call Raiserror if you need to.
I need a way to execute a stored procedure whenever my ssis package ends.
Regardless if it’s a failure or success.
Is there any way to do this without connecting failure events from each of my tasks? I’ve been looking for an OnPackageEnd event or something but I can’t see it.
Do any of you have any ideas?
In the package put all the tasks in a container. And below the container put the execute proceduer task and for precedence constraint choose values as "Completion"(The line will be in Blue color, green by default). So irrespective of the package status (success or fail) the stored proc will be executed.
Well I think the simplest thing is to add the execution of the proc as a second step in the job that executes the package, you can specify there that you can go to the next step on failure as well as on success.
Or you put the Exec SQL task to execute the package at the end of the process (that gets the success branch) and put it in the event handler for all failures (we do the event handler at the package level not for individual steps), we do that for one step where we run the same proc but with different input values in case of failure or success.
You can create an event handler onPostExecute of the pacakge and add the proc on an execute SQL Task in there
OR
add all your current components inside a sequence container and drag the green arrow to a new execute sql task component with your procedure.
Double click the green arrow and select "Completion" instead of "Success"
OnPostExecute gets executed for every tasks. So this is not good for a requirement that expects OnPackageEnd kind of an event.