I need to run SSIS2016 package from older SSIS (another server) because of extending legacy system. The package runs fine from SQL Agent, but I am unable to run it using tsql or DTEXEC. The problem is that package is unable to access network even when run in elevated cmd (under sysadmin)
I am able to access network drive from account I am running package (admin)
I know that package runs fine when it is run from Sql agent under same account.
The package runs fine when file is saved to local temp folder (both DTEXEC and TSQL)
The package fails to save file to network when run using execute is SSMS, TSQL or DTEXEC.
DTEXEC /ISSERVER "\SSISDB\XXXX\Exports\package.dtsx" /SERVER "." /Par
"$Package::FilePath(String)";"C:\TEMP"
From all what I have read on internet, it seems that the only way to run SSIS2016 package is by using SQL Agent and jobs.
I am surprised by the fact that DTEXEC ignores the security context of account which execute the command line.
Is there a way to force it to use proxy account?
I find it hard to believe, that only way to run package with permission to network is to use Sql agent and jobs.
Thank you.
p.s. Here is good article about running packages, but it completely omits network access problem.
EDIT 2019-06-10
I applied MSSQL2016 CU7 and restarted the server.
After this,
Iam able to save file to network using DTEXEC command ran on local server (logged as sysadmin)
Iam able to save file to network using sp ran from local server (logged as sysadmin)
I cant save file to network drive when logged as sysadmin on different machine. The error is
Flat File Destination failed the pre-execute phase and returned error
code 0xC020200E.
Cannot open the datafile ..
there is also warning "Access is denied" in data flow which should save the file.
Source code of stored procedure
DECLARE #execution_id BIGINT;
EXEC [SSISDB].[catalog].[create_execution]
#package_name = N'package.dtsx',
#execution_id = #execution_id OUTPUT,
#folder_name = N'XXXX',
#project_name = N'Exports',
#use32bitruntime = False,
#reference_id = NULL;
SELECT #execution_id;
DECLARE #var0 SQL_VARIANT = N'\\NETWORK\PATH\Export\test_files';
EXEC [SSISDB].[catalog].[set_execution_parameter_value]
#execution_id,
#object_type = 30,
#parameter_name = N'FilePath',
#parameter_value = #var0;
DECLARE #var1 SMALLINT = 1;
EXEC [SSISDB].[catalog].[set_execution_parameter_value]
#execution_id,
#object_type = 50,
#parameter_name = N'LOGGING_LEVEL',
#parameter_value = #var1;
EXEC [SSISDB].[catalog].[start_execution]
#execution_id;
I also added System::UserName to log on start of package and I see correct account.
I assume, that problem might be in double hop as #Piotr mentions.
Iam digging the fileserver logs to see.
EDIT2 2019-06-10
Ok. After much dancing around I can narrow problem down to delegation executing user rights to CIFS.
What i did
Setup a share on dev computer
Create package with scriptask which tries to create test file and get security context of calling user WindowsIdentity.GetCurrent().Name
Then I ran stored procedure with code above in scenario that always ended by access denied.
Until I enabled anonymous access (using this article), the package was not able to save file.
AFAIK we have enabled only MSSQLSvc kerberos delegation.
Closure:
The problem is caused by behaviour of SSIS catalog, when called from TSQL, which gets users security context, but in order to work this properly, when accessing netwrok shares, you need to setup Kerberos delegation for CIFS also.
I was not able to do this in our environment, so I reverted back to xp_cmdshell.
Thanks to all for support.
Related
I have deployed a SSIS Package to the Integration Services Catalogs. The issue that I am having is that the flat file source does not have access to the network drive.
How do I configure the package to run under another account.
The reason I chose this route is I wanted to be able to run the SSIS package from a WebAPI and just running a stored procedure.
EXECUTE AS LOGIN = 'Domain\User'
Declare #execution_id bigint
EXEC [SSISDB].[catalog].[create_execution] #package_name=N'Package.dtsx', #execution_id=#execution_id OUTPUT, #folder_name=N'CDMS', #project_name=N'Package Import', #use32bitruntime=False, #reference_id=Null
Select #execution_id
DECLARE #var0 smallint = 1
EXEC [SSISDB].[catalog].[set_execution_parameter_value] #execution_id, #object_type=50, #parameter_name=N'LOGGING_LEVEL', #parameter_value=#var0
EXEC [SSISDB].[catalog].[start_execution] #execution_id <----Fails
Update:
I tried 2 scenarios in SQL Management studio:
When logged in as a SQL Account: The current security context cannot be reverted. Please switch to the original database where 'Execute As' was called and try it again. ( Btw I am not even calling REVERT)
Logged in as Windows Auth and removed the Login statement: Cannot open the datafile...
Question in scenario 2, It is clear SQL is not accessing the network drive under my account. But when using EXECUTE as LOGIN..Does that then access the network drive under the login specified?
UPDATE 2
I have changed the permissions of the share to Modified for Everyone and I am still getting this error
Earlier we were in SSSIS 2008, we have 3 projects under a solution, Project A, Project B, Project C. I have Master Package in Project B from which I am calling packages in ProjectA and ProjectC in execute package task using external reference as File System, everything worked well till here. We recently moved from SSIS 2008 to SSIS 2012 and used Project deployment model to deploy packages to SSISDB instead of File System using Package Deployment Model, so I replaced Execute Package task with Execute SQL task in my master package to call packages from Project A and C.
I created Connection to SSISDB and used it in Execute SQL task and placed the below T-SQL code in it, it worked fine here also.
Declare #execution_id bigint
EXEC [SSISDB].[catalog].[create_execution]
#package_name=N'Child_Package.dtsx',
#execution_id=#execution_id OUTPUT,
#folder_name=N'Test',
#project_name=N'ProjectA',
#use32bitruntime=False, #reference_id=Null
Select #execution_id
DECLARE #var0 smallint = 1
EXEC [SSISDB].[catalog].[set_execution_parameter_value]
#execution_id,
#object_type=50,
#parameter_name=N'LOGGING_LEVEL',
#parameter_value=#var0
EXEC [SSISDB].[catalog].[start_execution] #execution_id
GO
I have 2 question here
1) If I deploy my packages from development to Production server, will this T-SQL code inside Execute SQL task works there without any issues (I read in one post that it will have some issues when deployed to diff environment) I am not hard-coding any environment parameter value inside my T-SQL code, will this work in another environment as well?
2) I am trying to implement rollback transaction support in my master package, so if any child package fails, everything needs to be rolled back, so I have Changed the package TransactionOption property to required and all other tasks(Execute SQL TASK) as supported(Default), I am ending up with an error like
"[Execute SQL Task] Error: Executing the query "Declare #execution_id bigint
EXEC [SSISDB].[catalo..." failed with the following error: "Cannot use SAVE TRANSACTION within a distributed transaction.". Possible failure reasons: Problems with the query, "ResultSet" property not set correctly, parameters not set correctly, or connection not established correctly"
The package is working fine if the TransactionOption property of the package is supported, the problem comes only when I use TransactionOption property of package as Required.
Any help is much appreciated.
The requirement is to call a web service through SSIS and calling the SSIS from a SQL Server Service Broker activated stored procedure.
Here is what I have currently doing:
Queue
CREATE QUEUE [schema].[ProccessingQueue] WITH STATUS = ON , RETENTION = OFF , ACTIVATION ( STATUS = ON , PROCEDURE_NAME = [schema].[usp_ProccessingQueueActivation] , MAX_QUEUE_READERS = 10 , EXECUTE AS N'dbo' ), POISON_MESSAGE_HANDLING (STATUS = ON)
My stored procedure:
ALTER PROCEDURE [schema].[usp_ProccessingQueueActivation]
WITH EXECUTE AS CALLER
AS
BEGIN
SET NOCOUNT ON;
<snip declaration>
BEGIN
BEGIN TRANSACTION;
WAITFOR
(
RECEIVE TOP (1)
#ConversationHandle = conversation_handle,
#MessageBody = CAST(message_body AS XML),
#MessageTypeName = message_type_name
FROM [schema].[ProccessingQueue]
), TIMEOUT 5000;
<snip awasome stuff>
EXEC dbo.RunSSIS <param>
DECLARE #ReplyMessageBody XML = #MessageBody;
SEND ON CONVERSATION #ConversationHandle MESSAGE TYPE [type] (#ReplyMessageBody);
END
<handle error>
COMMIT TRANSACTION;
END
END
Now here is what RunSSIS stored procedure looks like
ALTER PROCEDURE [dbo].[RunSSIS]
<params>
AS
BEGIN
DECLARE #exec_id BIGINT
EXEC [SSISDB].[catalog].[create_execution]
#package_name=N'<SSIS_package>',
#folder_name=N'<folder>',
#project_name=N'<projectName>',
#use32bitruntime=FALSE,
#reference_id=NULL,
#execution_id=#exec_id OUTPUT
EXEC [SSISDB].[catalog].[set_execution_parameter_value]
#exec_id,
#object_type=30,
#parameter_name=N'<param_Name>',
#parameter_value=<param>
SELECT #exec_id
EXEC [SSISDB].[catalog].[start_execution] #exec_id
END
Now this will throws the below exception in event-viewer as the Sql service broker activation security context isn't recognized in SSISDB environment.
The activated proc
'[schema].[usp_ProccessingQueueActivation]' running on
queue '' output the
following: 'The current security context cannot be reverted. Please
switch to the original database where 'Execute As' was called and try
it again.'
To resolve the problem I have tried those following approach
So I follow this link
http://www.databasejournal.com/features/mssql/article.php/3800181/Security-Context-of-Service-Broker-Internal-Activation.htm
and created a User with a self signed certificate (thinking that it
is user that doesn't has permission). But it is returning same error,
digging deeper I found that [internal].[prepare_execution] in
SSISDB has "REVERT" statement in line no 36 that throws the error as
it doesn't like Impersonation at all.
I tried to move the RunSSIS stored procedure to SSISDB and try to call it from activation stored procedure, it was shoot down as SSISDB it doesn't allow any user with SQL Server auth, It needs to have a Windows auth and User created by Certificate obviously doesn't has windows credential.
My question is
Am I on the correct path? I certainly doesn't anticipate using 2 component of SQL server together would be that difficult.
If not in correct approach what would be best approach to call a service from Service broker? I have seen "External Activation" for SQL Server Service broker but haven't explored is yet. But I would try to stick to something that lives inside server environment and scale-able, and don't like the idea of installing different component in prod environment (it is always a overhead for support personal,as there is one more point which can fail)
I am using Windows auth and my credential has sys_Admin access.
I think you can take out the "WITH EXECUTE AS CALLER" and everything (the proc and then the package that ends up getting called) will be run under the security context of the Service Broker. As long as that context has the permissions to do what you want to do, you should be fine.
I have not used a Service Broker in this way, but I do the same thing with jobs fired off by the SQL Agent. As long as the Agent's security context has the permissions needed in the procs/packages everything runs fine. We use network accounts for our services so it all works between servers as well.
This has a code smell of tight coupling and my first instinct is to decouple the queue, the DB that houses the proc, and the SSIS execution into a PowerShell script. Have the script get the messages from service broker then call SSISDB on a different connection without wrapping [catalog].[create_execution] and [catalog].[set_execution_parameter_value] in a stored proc. You can still run this script directly from Agent.
This approach gives you the most flexibility with regard to security contexts, if one of the components moves to a different server, if something is named differently in dev/QA, or technologies change (Azure ServiceBus instead of Broker for instance). You can also get creative with logging/monitoring.
I am trying to get SQL Server Agent to run a program with arguments (as a Operating system CmdExec job step), but everytime it runs the job I receive the following error: The process could not be created for Step 1 of job, reason: Access is denied).
The research I did online indicated that this could be a permissions issue. I set up a new user account in Windows and gave it full permissions on the program I wanted to run, then mapped this user profile to the SQLSERVERAGENT profile within MS SQL but I still get this error.
Any help with this would be appreciated.
above steps worked for me
Enable XP_cmdshell
EXEC sp_configure 'show advanced options', 1
GO
RECONFIGURE
GO
EXEC sp_configure 'xp_cmdshell', 1
GO
RECONFIGURE
GO
Create credential
CREATE CREDENTIAL cmdshell_agent WITH IDENTITY = 'account_name', SECRET = 'password';
GO
Create proxy
USE [msdb]
GO
EXEC msdb.dbo.sp_add_proxy #proxy_name=N'tst_Proxy',#credential_name=N'cmdshell_agent', #enabled=1
GO
EXEC msdb.dbo.sp_grant_proxy_to_subsystem #proxy_name=N'tst_Proxy', #subsystem_id=3
GO
After this use tst_proxy in the sql agent job to run the job. Job ran successfull
I had the same problem, and the solution was embarrassingly simple: my path to the .exe was off. It's a little confusing that this reflects as an "Access Denied" error (because the Agent account doesn't really know that the file doesn't exist, just that it can't reach such a file).
One simple step you can take while troubleshooting is to run the exact command in a command window. Does it work as expected? How about if you log in using the Agent account?
First you'll need to make sure that XP_CMDSHELL is allowed.
exec sp_configure 'xp_cmdshell',1
go
reconfigure
You'll need to create a credential with the user you created.
Create a proxy referencing the credential you created. Give this proxy access to the "Operating System(CmdExec)" subsytem.
In the job step itself, make sure that it is executing as this proxy (Run as:).
Got the same error message. It turned out that the SQL Agent account did not have permissions on the Tools directory where sqlcmd.exe is located. Gave Read & Execute permissions to the SQL Agent Account, problem solved.
If you're running into this issue with SQL 2014 Maintenance Plans, even if the RunAs account and service accounts are fine (as confirmed in EXECUTE AS testing), then try the following:
Check the Local Connection authentication in the Maintenance Plan.
if it's set to an account & it's appropriate to do so, switch to Windows NT Authentication, save out the plan
if it's already on WinNT, switch to an account, save out the plan, go back in, switch to Windows NT Authentication, save out the plan again.
There was no logging anywhere that would catch this. I was just thinking where else authentication could be involved, since the failures were around that, but everything was set up properly everywhere else....
As Grandma would say: "It's always in the last place you look". 😏
We need to make sure that the account through which the job is being executed has full access to the folders/document involved in the job steps.
This should resolve the issue without adding the proxy.
I am able to run the xp_fileexists command successfully on the local drives of my SQL Server instance. If a file exists, I get an output of 1. If a file does not exists, the output is 0. When I run the same command on a network drive that is mapped on the SQL Server machine, the output is always 0.
For example, if I have a file with a URL of '\\10.188.20.5\myfolder\myfile.txt'. I would run
SET #MYFILE = '\\10.188.20.5\myfolder\myfile.txt'
EXEC MASTER.DBO.XP_FILEEXIST #MYFILE, #MYOUTPUT OUT
PRINT #MYOUTPUT
The result would return 0.
If I run a bulk insert command on the same URL, the file would be successfully imported
BULK INSERT #mytable
FROM '\\10.188.20.5\myfolder\myfile.txt'
What is causing xp_fileexists to malfunction on network drives?
The problem turned out to be with the Admin account password used to run SQL Server. The password was changed, but SQL Server continued to run. It just happened 'MASTER.DBO.XP_FILEEXIST' didn't work because of the password conflict, but other SQL Server functions did. Once I changed the password associated with the Admin account used to run SQL Server, the function worked as expected.
It depends on the credetials of the Service
usualy local system, which has not network access
Go to >> Sql Server Configuration Manager >> on left panel go to SQL Server Services >> select your own Instance (MyPc\Sqlexpress) and enter the account name, the username and the password.
now it should work