Problem
I have developed an SSIS package for an ETL process, which I have deployed successfully to a remote server. I have windows and SQL logins for this server, but not admin. The data that this ETL process loads is generated on a different machine on a different domain early in the morning each day (i.e. before I get to work) and needs to be automatically loaded after the data generation process as soon as possible.
Currently, I manually run the ETL package in SSMS, and authenticate this by spoofing the windows user credentials on the remote server by launching SSMS with runas.exe. This is currently okay since we are in early development, but this is obviously not an acceptable solution in production. Hence, I would like to automate this process, since a) the data is extremely time sensitive and b) manually running this process is a PITA.
How should I go about automating execution of the ETL package such that:
Execution happens immediately after the data is available, and
No user input is required from me e.g. to enter passwords?
Attempted solutions
Scheduling a SQL server agent job on a schedule is not an option, because of the time sensitivity. (Also my sysadmin seems to have not enabled/disabled the SQL server agent)
Continuing to use runas.exe is not an option either, as the /netonly flag, which is required, and the /savecred flag, which would prevent the need for manual password entry, are mutually exclusive.
sqlcmd.exe can be scripted to run with a SQL user and password (bad practice putting passwords in plain text, I know) but then running a stored procedure that calls the SSIS package fails, as windows authentication is required to run the package.
I would strongly advice you to ask for working SQL Agent and create an Agent Job to run the package. This way has a lot of advantages:
Native Windows user credentials management with proxy accounts
You can configure specific values for package/project parameters, connection managers via SSMS interface
You can use environment references of SSIS 2012 and above, and manage it via SSMS as well
Job can be started on schedule or manually from SQL command. Thus you can automate package start.
In a nutshell - it is much easier to manage and you do not disclose user credentials needed to run the package - you might even do not know it.
For the alternative - you can try PsExec from SysInternals tools. It allows to start program on another computer explicitly specifying username and password as psexec \\server -u user -p password .... However, I would opt against it.
Related
I've inherited an SSIS package which loads CSV files into a SQL database.
The first thing the package does is call a .BAT file which maps a network drive. The .BAT file contains the username and password used to map the drive in plain text, so urgently needs to be replaced.
I've written a solution which uses New-PSDrive in a Powershell script and creates a credentials XML file with the password encrypted. If I execute the ps1 script it works Ok.
If I call the ps1 script from within SSIS and run it through VS then it also works fine.
When I call the SSIS package that calls the script task through a SQL Agent job (executed as the SQL Server Agent user account) the drive mapping doesn't seem to work, and the package complains it cannot access the file from the file share.
I presume this is no working because the SQL Server Agent user account can't run Powershell queries? The package only errors because it cannot access the share, not during the execution of the Powershell script.
So, two questions:
Is there an obvious issue with the SQL Agent deployment idea
or
Is there a different way of mapping a network drive from within SSIS without storing the password in plain text anywhere?
This is the pattern I suggest using for accessing files on a remote server:
Access the files via unc path: \\MyServer\Files
This means that the package needs to run under a context that has access to the path. There are a couple of options:
Run the package under a proxy, i.e. the account credentials that were stored in the bat file. This means that you have to also grant access to the account on the sql instance as well
Use a group managed service account (gMSA). In this case, the sql agent service account is replaced with the gMSA and the job runs under sql agent. The gMSA needs to be granted access to the remote share as well as the sql instance. This is a much more secure way of addressing the whole process because there is no password to manage (AD takes care of that) and the account cannot be used to log in anywhere. But there is set up work to do to get it created so it's not the fastest option.
I normally run an SSIS package using a Sql Agent Job and a proxy user as described here: https://www.sqlservercentral.com/articles/run-an-ssis-package-under-a-different-account
I now need to run the same package using the same proxy user using T-SQL. I've been trying to use the [catalog].[create_execution] and [catalog].[start_execution] procedures to do this but there doesn't seem to be a way to specify a user.
How do I execute a package as a different user?
Is my best recourse the use of T-SQL to execute a SQL Agent Job that is configured to use the proxy user instead?
I don't know how agent actually works to make proxy users work - especially with regard to SSIS packages.
In a "normal" sql session say in SSMS, if I wanted to run a query as another user
EXECUTE AS USER = 'TurgidWizard';
SELECT USER_NAME() AS WhoAmI;
REVERT;
That code would allow me to impersonate you until I hit the REVERT call.
But, if you swap out calls to create_execution/start_execution you'll run into the same issue as trying to use a local sql server user runs into with using the methods in the SSISDB - it doesn't work. The methods in the SSISDB all run checks before they begin to ensure users have the correct access level and there isn't impersonation going on. Because once those methods start running, they themselves do impersonation so I guess that doesn't work well.
How can I run a package using tsql under a different account? I would start SSMS/sqlcmd under the credentials using RUNAS For example, the following will open a new command window as you.
runas /netonly /user:corpdomain.com\turgiwizard "cmd"
From there, things I do will be under the aegis of your user so I could run sqlcmd calls like
sqlcmd -S TheServer -d SSISDB -Q "EXECUTE catalog.create_execution ...;"
Mouse click will be Ctrl+Shift+right click executable.
Your SSMS install location is version dependent but try various ten digit increments of 140 in the following path
C:\Program Files (x86)\Microsoft SQL Server\140\Tools\Binn\ManagementStudio\Ssms.exe
The downside for me with regard to runas is that I could not automate getting my credentials passed into it. I've seen articles about use auto hot key and such but never had any luck with it.
Cleanest/easiest approach for something that needs to run regularly is to use sql agent with a proxy, or you could use Windows Task Scheduler and create it as the target user. One off executions, I'd likely use the runas approach.
I set up a job to run a SSIS package in the SSIS Catalog, but the SSIS package doesn't start. If I right-click the package and run it, it completes successfully.
When I look in the execution history of the package in the Catalog, there is no sign that it the package was even started by the job.
I created a test package to make sure it isn't the specific package failing, so the package I'm testing with has a SQL Task that does a 'SELECT 1'. Still fails.
The job just stays in the "Executing" state until I stop the job.
I am running SQL Server 2014 Standard Edition, and running the job with a sysadmin account.
Why is the SQL Agent Job not running my package?
Edit: I had a look through this article but it either doesn't apply or I gave it a shot and it didn't work either.
Edit 2: I found an error under the SQL Server Agent Job History: Unable to terminate process 1850 launched by step 1 of job 0x5DAD5416FA09C445B82ABDBB49F75E38 (reason: Access is denied)
Does that mean my service account doesn't have enough permissions on the local machine? However, I AM able to run the job with DTEXEC using the service account without issues...
I have also faced this scenario many times but when I checked running the package manually,its completing successfully because I was using for each loop container and sequence container as well.In both cases for each loop and sequence were completing without validating other ones.So I checked precedence constraint and change it,Now it working and all the component ran successfully. Sometimes we miss to choose appropriate precedence constraint, there are many option like on Success ,failure,completion and then for you can choose values from Constraint ,Expression,Expression AND Constraint and Expression OR Constraint. Initially i was using Expression OR Constraint for success and now changed it to Expression AND Constraint, its working fine for me. You also need to do this,it will definitely work please try and let me know.
When you look at the connections in the SSIS package, did you put in a user id and password? If so, did you add a package password? The reason I ask is that if you don't add a package password, it will strip out things it thinks should be secure (like passwords). You might want to try putting the connection password in a configuration file, and add that file to the job. Either that or password encrypt the SSIS package but you will have to take special care when you run the package for the first time (or when you create the job) because you will need to provide the password that you used to encrypt the package.
Reboot fixes this issue. Reason is because if you look at this message. Unable to terminate process 1850. That is a windows process. I would say reboot. If you really like to enquire before rebooting. You could use sysinternals process explorer. Download it and use that to find process id 1850 which may be hung process in windows.
Thanks
Ali
I am an Oracle guy who suddenly got SQL Server and SSIS dropped in his lap, so I am probably not using the terminology in the correct manner, but here goes:
There is a SSIS package that pulls data from the Oracle database into our SQL Server 2008 R2 warehouse. If I open this package in SSIS Visual Studio 2008, I get prompted for a password:
The sensitive data in the package 'MyRefresh.dtsx' is encrypted with a password.
for the package itself. I enter the password. I run the package. Works great.
The previous guy had loaded this package into SQL Server with a job to run at 1am every day and it worked great there too.
Recently, there were some database changes. The package, of course, stopped working. I was able to fix it, and, again, it runs great if run through SSIS Visual Studio 2008. However, when I loaded it into SQL Server, and the job runs, it fails with:
0xC001405F Failed to decrypt an encrypted XML node because the password was not specified or not correct.
Where/how to I specify the password so the job can run?
Late answer, but might be helpful to other users/thread visitors
In short, to load the package to SQL Server it must be exported with new credentials specified, and then imported back into specified folder.
Here is the article I found on setting the SSIS package encryption manually in SSMS, that provides a quick tutorial on how to Import/Export an encrypted package.
Please note that the Protection level option regards sensitive data, in one case, or all the data included in particular package in other. Data that is considered sensitive is set by default in Integration Services: variables previously marked as delicate, non-changeable XML tags, which are controlled by the SSIS service, and password, which can be considered sensitive if the ‘Encrypt all data with password’ is chosen.
Package protection levels:
Do not save sensitive data: if sensitive data exists, it will not be included after the exporting of the new package, remaining unavailable;
Encrypt sensitive data with user key: sensitive data will be encrypted with current user credentials, and package still can be used on local server. Which data will be considered as sensitive, depends on the creator/owner of the package;
Encrypt sensitive data with password: with this level, a password must be provided – this kind of encryption is desirable, if user want to keep only sensitive data private.
Encrypt all data with user key: same as the encryption of sensitive data, it can be used on local server, but it regards all the data within the package;
Encrypt all data with password: this level encrypts all data within the package, password is required, and it provides a 100% privacy.
Hope this info is helpful.
If you have the opportunity I suggest you no longer use the EncryptAllWithPassword protection level. Read here for more info about package encryption levels:
http://sqlblog.com/blogs/eric_johnson/archive/2010/01/12/understanding-the-ssis-package-protection-level.aspx
In short the idea of package encryption is to stop people opening up the package XML to extract plain text passwords. But generally this is implemented in a insecure manner which defeats the purpose.
I suggest you use windows authentication throughout instead:
Ensure your Oracle server supports external authentication
Create an externally identified login to Oracle using the SQL Agent windows service account
In your Oracle connection manager, use external authentication (login with user / and no password)
If you have any SQL Server connection managers you need to do the same (in SQL Server this is called windows authentication)
Lastly ensure that all developers are set up with windows authentication in SQL Server and Externally identified authentication in Oracle so they can run the package in BIDS
Now you don't need to encrypt your package anymore (you can use DontSaveSensitive). The authority for all operations are against the SQL Agent service account.
You don't need to remember a package password or an Orace login password any more.
Also for example if you need to rotate the password on your Oracle login, originally you would have to go and change this password in Oracle and in your package. But by using windows authentication this is no longer necessary.
I can give you more info if you are interested.
You can use the /de switch along with the dtexec utility for your password like so:
dtexec /f <filename> /de <password>
We are in the process of migrating from SQL 2000 to SQL 2005. We have hundreds of DTS pacakges, that the development team is reluctant to redevelop using SSIS.
When migrating these packages to SSIS, I am faced with a problem - many of these packages read from Excel files.
Given that my production Box is 64 bit, I am forced to use CmdExec sub-system to call the 32 bit runtime to execute these packages.
My question here is : What are the security risks involved with using CmdExec subsystem to schedule these SSIS packages as SQL agent jobs?
Thanks,
Raj
Whatever account running the job will potentially have access to run commands from the command line - so you need to think about how it will be running and what permissions the account will have.
For example, if a user could create a job that would run under the context of your sqlagent and your sql agent was overpriviledged (rights to change security), she could grant herself elevated privs or hurt your machine.
SQL 2008 introduced a switch for DTExec that allows you to run the packages in 32 bit mode using the native SQL Agent task for SSIS. On the execution tab of the job step properties there is a check box for 32 bit, which translates to the "/X86" switch when looking at the command line view.
If you are stuck using SQL 2005 then the CMDEXEC option is the only one I know of.
xp_cmdshell is the biggest security risk in SQL Server because it allows a compromised SQL Server box to elevate the attack to the host operating system itself, and from there to the entire network.
The typical vector of attack is web site HTTP form -> SQL injection -> xp_cmdshell -> take over SQL hosting machine -> take over domain. If xp_cmdshell is shut down then the attacker has to find other means to elevate its attack from SQL to the host.
Other scenarios exists, like insider users using it to elevate privileges, or using the cmdshell for other purposes, eg. steal a database. All are based on the fact that xp_cmdshell allows arbitrary commands to be executed and on the host, and in some cases the commands executed also inherit the SQL Server service account privileges.
There are other commands and extend procedures that can be used by an attacker if xp_cmdshell is blocked, but they far less known. Using the xp_cmdshell vector is in every SQL injection cheat sheet and forum discussion, so is known by everyone and their grand ma.