Earlier we were in SSSIS 2008, we have 3 projects under a solution, Project A, Project B, Project C. I have Master Package in Project B from which I am calling packages in ProjectA and ProjectC in execute package task using external reference as File System, everything worked well till here. We recently moved from SSIS 2008 to SSIS 2012 and used Project deployment model to deploy packages to SSISDB instead of File System using Package Deployment Model, so I replaced Execute Package task with Execute SQL task in my master package to call packages from Project A and C.
I created Connection to SSISDB and used it in Execute SQL task and placed the below T-SQL code in it, it worked fine here also.
Declare #execution_id bigint
EXEC [SSISDB].[catalog].[create_execution]
#package_name=N'Child_Package.dtsx',
#execution_id=#execution_id OUTPUT,
#folder_name=N'Test',
#project_name=N'ProjectA',
#use32bitruntime=False, #reference_id=Null
Select #execution_id
DECLARE #var0 smallint = 1
EXEC [SSISDB].[catalog].[set_execution_parameter_value]
#execution_id,
#object_type=50,
#parameter_name=N'LOGGING_LEVEL',
#parameter_value=#var0
EXEC [SSISDB].[catalog].[start_execution] #execution_id
GO
I have 2 question here
1) If I deploy my packages from development to Production server, will this T-SQL code inside Execute SQL task works there without any issues (I read in one post that it will have some issues when deployed to diff environment) I am not hard-coding any environment parameter value inside my T-SQL code, will this work in another environment as well?
2) I am trying to implement rollback transaction support in my master package, so if any child package fails, everything needs to be rolled back, so I have Changed the package TransactionOption property to required and all other tasks(Execute SQL TASK) as supported(Default), I am ending up with an error like
"[Execute SQL Task] Error: Executing the query "Declare #execution_id bigint
EXEC [SSISDB].[catalo..." failed with the following error: "Cannot use SAVE TRANSACTION within a distributed transaction.". Possible failure reasons: Problems with the query, "ResultSet" property not set correctly, parameters not set correctly, or connection not established correctly"
The package is working fine if the TransactionOption property of the package is supported, the problem comes only when I use TransactionOption property of package as Required.
Any help is much appreciated.
Related
I need to run SSIS2016 package from older SSIS (another server) because of extending legacy system. The package runs fine from SQL Agent, but I am unable to run it using tsql or DTEXEC. The problem is that package is unable to access network even when run in elevated cmd (under sysadmin)
I am able to access network drive from account I am running package (admin)
I know that package runs fine when it is run from Sql agent under same account.
The package runs fine when file is saved to local temp folder (both DTEXEC and TSQL)
The package fails to save file to network when run using execute is SSMS, TSQL or DTEXEC.
DTEXEC /ISSERVER "\SSISDB\XXXX\Exports\package.dtsx" /SERVER "." /Par
"$Package::FilePath(String)";"C:\TEMP"
From all what I have read on internet, it seems that the only way to run SSIS2016 package is by using SQL Agent and jobs.
I am surprised by the fact that DTEXEC ignores the security context of account which execute the command line.
Is there a way to force it to use proxy account?
I find it hard to believe, that only way to run package with permission to network is to use Sql agent and jobs.
Thank you.
p.s. Here is good article about running packages, but it completely omits network access problem.
EDIT 2019-06-10
I applied MSSQL2016 CU7 and restarted the server.
After this,
Iam able to save file to network using DTEXEC command ran on local server (logged as sysadmin)
Iam able to save file to network using sp ran from local server (logged as sysadmin)
I cant save file to network drive when logged as sysadmin on different machine. The error is
Flat File Destination failed the pre-execute phase and returned error
code 0xC020200E.
Cannot open the datafile ..
there is also warning "Access is denied" in data flow which should save the file.
Source code of stored procedure
DECLARE #execution_id BIGINT;
EXEC [SSISDB].[catalog].[create_execution]
#package_name = N'package.dtsx',
#execution_id = #execution_id OUTPUT,
#folder_name = N'XXXX',
#project_name = N'Exports',
#use32bitruntime = False,
#reference_id = NULL;
SELECT #execution_id;
DECLARE #var0 SQL_VARIANT = N'\\NETWORK\PATH\Export\test_files';
EXEC [SSISDB].[catalog].[set_execution_parameter_value]
#execution_id,
#object_type = 30,
#parameter_name = N'FilePath',
#parameter_value = #var0;
DECLARE #var1 SMALLINT = 1;
EXEC [SSISDB].[catalog].[set_execution_parameter_value]
#execution_id,
#object_type = 50,
#parameter_name = N'LOGGING_LEVEL',
#parameter_value = #var1;
EXEC [SSISDB].[catalog].[start_execution]
#execution_id;
I also added System::UserName to log on start of package and I see correct account.
I assume, that problem might be in double hop as #Piotr mentions.
Iam digging the fileserver logs to see.
EDIT2 2019-06-10
Ok. After much dancing around I can narrow problem down to delegation executing user rights to CIFS.
What i did
Setup a share on dev computer
Create package with scriptask which tries to create test file and get security context of calling user WindowsIdentity.GetCurrent().Name
Then I ran stored procedure with code above in scenario that always ended by access denied.
Until I enabled anonymous access (using this article), the package was not able to save file.
AFAIK we have enabled only MSSQLSvc kerberos delegation.
Closure:
The problem is caused by behaviour of SSIS catalog, when called from TSQL, which gets users security context, but in order to work this properly, when accessing netwrok shares, you need to setup Kerberos delegation for CIFS also.
I was not able to do this in our environment, so I reverted back to xp_cmdshell.
Thanks to all for support.
I'm trying to pass a variable value from SQL Server Agent job to SSIS package but the variable contains an apostrophe in it causing the SQL Server Agent job to fail
e.g In SQL Server Agent at Job Step Properties I'm entering the following details:
Property Path: \Package.Variables[User::VariableName].Properties[Value] Property
Value: Michael O'Callaghan.
Any idea how to resolve this issue?
If the package is deployed to SSISDB and executed from there, use SSISDB stored procedures to set the value and escape the quote like how you would via T-SQL. The SQL Agent job can then use a T-SQL script for this step instead. The example below uses the set_execution_parameter_value stored procedure to set this value and will still result in "Michael O'Callaghan" being passed in.
DECLARE #execution_id BIGINT
EXEC [SSISDB].[catalog].[create_execution] #package_name=N'Package.dtsx', #execution_id=#execution_id OUTPUT,
#folder_name=N'Project Folder', #project_name=N'Project', #use32bitruntime=False, #reference_id=Null
DECLARE #var0 SQL_VARIANT = N'Michael O''Callaghan'
EXEC [SSISDB].[catalog].[set_execution_parameter_value] #execution_id, #object_type=30, #parameter_name=N'Name', #parameter_value=#var0
DECLARE #var1 SMALLINT = 1
EXEC [SSISDB].[catalog].[set_execution_parameter_value] #execution_id, #object_type=50, #parameter_name=N'LOGGING_LEVEL', #parameter_value=#var1
EXEC [SSISDB].[catalog].[start_execution] #execution_id
Escape it. Just use a double apostrophe. '' (Not a quotation ", but a apostrophe apostrophe).
Try the standard way of maintaining a configuration file(if you are using 2008 or less) and pass the variables values through the file.
An alternative way to handle this, and frankly I think the best way, is to use Environment Variables. To my knowledge, this was introduced when Microsoft rolled out the project deployment model with SQL Server 2012 as a replacement to the package deployment model. The package deployment model required that package parameters be specified in a separate XML file to be deployed to the server. With the project deployment model, Microsoft has created a user-friendly user interface in SQL Server to manage this - the XML file has been removed.
In short, Environment Variables allow developers to link package parameters, but not package variables as those are internal to the package itself, to SQL Server and thus expose them on the server. This makes management of identical package parameters that exist across packages (e.g., connection managers, network folder locations in FQDN format, etc.) incredibly easy to manage. The idea here is that if packages need to be pointed to a new server or new network folder, then developers can simply change a single value in SQL Server, which would then propogate out to all packages without the need to open, change, and re-deploy the package.
For detailed steps on how to do this, see the following references:
Microsoft: This is a bit dry, but is comprehensive and from the horse's mouth.
Deploy Packages with SSIS
Lesson 1: Preparing to Create the Deployment Bundle
SQL Chick: More intuitive and provides screenshots, which I found helpful.
Parameterizing Connections and Values at Runtime Using SSIS Environment Variables
Thanks for your all you suggestions but unfortunately they didn't work, however I built a clever workaround for this.
SQL server agent wraps a variable value in single quote e.g specifying Jon Doe in sql server agent, the agent wraps it like this 'Jon Doe' and passes it to the SSIS package, so if you were to enter a value with an apostrophe it would break the sql server agent job and won't execute the SSIS package it would look like this E.G passing this value: 'John O' Doe' this would cause the agent the job to break so you need to pass your variable value as : John O''Doe and the agent wraps it as follows: 'John O''''Doe' so you would need to include the following logic in your SSIS package:
Declare #TempVar nVarchar(50)
SET #TempVar = REPLACE(?, '''''', CHAR(39))
The above code creates a variable to store the parameter value. It replaces the 4 single quotes to one. CHAR(39) is the ASCII representation of a single quote.
This would then cause the variable value to look like John O'Doe.
Hope this helps.
The reason I wanted to pass a variable value from the agent as I had to change the variable value very often from the SSIS package it would need to be deployed every time. So this way is faster.
I have a simple SSIS package that uses C# to generate an Excel file and put it to assigned location.
Package works fine in design mode.
Package been deployed successfully:
However when I try to execute it from SSMS - then then its only displays #execution_id.
DECLARE #execution_id BIGINT
EXEC [SSISDB].[catalog].[create_execution]
#package_name=N'ExcessCopy1.dtsx',
#execution_id=#execution_id OUTPUT,
#folder_name=N'MonthlyReports',
#project_name=N'MonthlyReports',
#use32bitruntime=False,
#reference_id=Null
SELECT #execution_id
DECLARE #var0 smallint = 1
EXEC [SSISDB].[catalog].[set_execution_parameter_value]
#execution_id,
#object_type=50,
#parameter_name=N'LOGGING_LEVEL',
#parameter_value=#var0
EXEC [SSISDB].[catalog].[start_execution] #execution_id
GO
If I comment out SELECT #execution_id, then I get
Command completed successfully
However it does not deliver file to assigned location.
What can be the problem and how can I troubleshoot this issue?
I can see the package under Integrational Service Catalog.
But I was unable to find this package in SSISBD:
I was able to see what is the error:
What can be the problem
The most likely explanation is that the package puts the file in a place you don't expect because it is now executing on the server and not your local box. Almost equally likely is that the package is encountering an error that it is not passing back to the SSMS window.
and how can I troubleshoot this issue?
If the package is in SSISDB, you can right click on it and look at the "All Executions" report to verify that the package did execute, and then drill down into the "All Messages" for that execution to see what happened.
Your SQL Server is missing the ACE OLEDB 12.0 driver. You can get it here: https://www.microsoft.com/en-us/download/details.aspx?id=13255
If you need both 64-bit and 32-bit versions installed for some reason, you can install the 64-bit from the link above, but it won't let you also install the 32-bit version, so you can use an older version of the 32-bit driver, which can coexist with the 64-bit one. You can get that one here: https://www.microsoft.com/en-us/download/details.aspx?id=23734
I am receiving the following error when trying to run the package from the Integration Services catalog in SSMS. I changed the 64BitRuntime option to FALSE but it still does not work. The error below is followed by an error that a connection cannot be made to my Excel connection manager. Any suggestions?
Package Error: The requested OLE DB provider Microsoft.Jet.OLEDB 4.0
is not registered. If the 64-bit driver is not installed, run the
package in 32-bit mode. Error code: 0x00000000
if you are executing the SSIS package from job , there is an option in job configuration a checkbox "enable 32 bit".
OR
if you are executing the SSIS package from BIDS or SSDT , go to project properties=> Configuration => debugging => turn 64BitRuntime from "True" to "False" as it is set to True by default.
You have to install Microsoft Access Database Engine 2010 Redistributable and
set 64BitRuntime option to FALSE
you can get it from the following link:
https://www.microsoft.com/en-us/download/details.aspx?id=13255
More info and details can be found in the following links:
http://sqlblog.com/blogs/john_paul_cook/archive/2010/03/24/running-32-bit-ssis-in-a-64-bit-environment.aspx
http://toddmcdermid.blogspot.com/2009/10/quick-reference-ssis-in-32-and-64-bits.html?m=1
https://msdn.microsoft.com/en-us/library/ms162810.aspx
You are attempting to run an SSIS package from the SSISDB catalog and need it to be in 32 bit mode.
The TSQL for such would look like the following
DECLARE #execution_id bigint;
EXEC SSISDB.catalog.create_execution
#package_name = N'Legacy_DataExport.dtsx'
, #execution_id = #execution_id OUTPUT
, #folder_name = N'Legacy_DataExport'
, #project_name = N'Legacy_DataExport'
, #use32bitruntime = True
, #reference_id = NULL;
SELECT
#execution_id;
DECLARE #var0 smallint = 1;
EXEC SSISDB.catalog.set_execution_parameter_value
#execution_id
, #object_type = 50
, #parameter_name = N'LOGGING_LEVEL'
, #parameter_value = #var0;
EXEC SSISDB.catalog.start_execution #execution_id;
GO
Of note is the penultimate parameter of the first EXEC where we specify #use32bitruntime = True
That says, please run the package Legacy_DataExport.dtsx which can be found in the project Legacy_DataExport which can be found in the folder Legacy_DataExport using the 32bit runtime.
From the UI perspective, it looks like
The click path within SSMS for this would be
Expand the "Integration Services Catalogs" node under "Management"
Expand the only option there of "SSISDB"
Expand the Folder where your project exists - "Legacy_DataExport" in my case
Expand the "Projects" node
Expand your actual project node - my project is also called "Legacy_DataExport"
Expand "Packages"
Find your package, again my example is "Legacy_DataExport.dtsx", right click it and select Execute...
Using the built in excel connection manager in SSIS, the package needs to run in 32-bit mode. Switching this:
64BitRuntime option to FALSE
Only allows SSDT to run the package in 32bit mode, but it does not affect how it will run once you deploy it. To run it in 32bit mode from SSMS:
If you are right clicking on the package in the Integration Services Catalog and hitting execute, go to the advanced tab of the dialogue and check 32-bit runtime.
If you are executing it via a SQL Agent job. In the step, go to configuration > Advanced and check 32-bit runtime.
I really like tsqlt to test procs and functions, but really would like to be able to also execute SSIS packages and take advantage of FakeTable and AssertEquals to determine if it was the SSIS package did what it was supposed to.
Has anyone explored this path, is it possible to call dtexec from with the transaction that tsqlt wraps your test in?
I believe I can answer your question Andrey, although this is a little late in coming. But I believe that it will benefit others.
We are using RedGate SQLTest(tSQLt) to do data quality testing as a part of our integration testing.
For example to test the completeness of the data being loaded into Staging, on test would be to AssertEqualsTable after a package loads a staging table. Here is the basic order of things:
Assemble
Create and load the expected table with data.
Act
Execute the SSIS Package in the catalog via t-sql. You can generate t-sql code to call any package in the catalog as follows:
Locate the package you're testing in it's folder in the catalog
Right click and select 'Execute'
The Execute Package dialogue box will open.
Click the scripting dropdown and select 'Script to Clipboard'
All the t-SQL Code needed to execute the package from a stored procedure or script is generated:
DECLARE #execution_id BIGINT
EXEC [SSISDB].[catalog].[create_execution]
#package_name=N'HistoricalLoad_import_rti_stores_s1.dtsx'
, #execution_id=#execution_id OUTPUT
, #folder_name=N'Testing'
, #project_name=N'Staging1_HistoricalLoad_RTIStores'
, #use32bitruntime=FALSE
, #reference_id=NULL
SELECT #execution_id
DECLARE #var0 SMALLINT = 1
EXEC [SSISDB].[catalog].[set_execution_parameter_value]
#execution_id
, #object_type=50
, #parameter_name=N'LOGGING_LEVEL'
, #parameter_value=#var0
EXEC [SSISDB].[catalog].[start_execution] #execution_id
Go back to your test stored proc and paste the code into the Act section.
Assert
- Select into the actual table from the SSIS destination table of the package being tested.
then validate that the expected and actual are equal
EXEC tSQLt.AssertEqualsTable 'expected', 'actual';
And that's all there is too it.
Take a look at the foreign key tests in the examples database to guide you on foreign key and referential integrity tests.
I've found it to be invaluable as a means of regression testing our data warehouse load functionality and also validating our orchestration. Because if we can verify that the data is flowing into the right place, at the right time, then things are executing as expected.
tSQLt is a Unit Testing Framework and it is designed for testing code in isolation.
So for testing how your code/data will be integrated with the other code/data typically used different types of tests - Integration Tests.
LATER UPDATE
Not exactly about the topic but it may be useful information about unit/integration testing of SSIS packages
There is a sample project with unit tests for SSIS at http://ssistester.codeplex.com/. Few samples show the use of the FakeSource and the FakeDestination to assert if data flow streams correctly read/write data.