My SSIS Package works, but fails as a SSMS job (Error: 0xC0016016)
I am posting this question and sharing my solution to this issue as my own question because I didn't see the posed problem match the specific issue I encountered and the answers seemed to be scattered in different forum questions.
Background:
I have four SSIS packages on SQL Server 2012 that import a table from SQL Server 2008 R2, 2008, or 2005, depending on the specific package. I use a designated sql server login and password for the source database and integrated Windows security for the target database.
Within SSIS I am able to run each package without a problem.
To ensure this package ran on a schedule, I set up a SSMS job on the same server as the SSIS package. In Job step properties, I chose: SQL Server Integration Services Package > run as SQL Sever Agent Service Account > Package source: File system).
The Symptom:
When manually running the job to make sure it worked, I got an error and saw this in the Log File Viewer. This was the first of several errors, but as this was chronologically the first error, I looked into this one first.
Error: 2014-10-24 09:52:34.48
Code: 0xC0016016
Source: [Redacted -- the correct name of the table I was importing]
Description: Failed to decrypt protected XML node "DTS:Password" with
error 0x8009000B "Key not valid for use in specified state.". You may
not be authorized to access this information. This error occurs when
there is a cryptographic error. Verify that the correct key is
available.
End Error
I looked up the error code on Google and started looking at resolutions. Rather than retelling how I found the bits and pieces of the resolution, I am presenting an actionable result in sequence -- at least for me and the network infrastructure I'm working with.
In the "properties" panel of the SSIS solution (do this first) and each package in that solution, reset the "ProtectionLevel" attribute to EncryptSensitiveWithPassword and set a password. The package passwords must match the solution password.
Just a double-check, run your packages in debug mode to make sure that no other new issues arise. In my case, I needed to re-enter the sql server password for the source server database.
Rebuild your SSIS solution.
In SSMS, open your job and open the job step properties for the task in question.
Select the "Command Line" tab. A "Package Password" popup appears. Enter the password that you entered in step 1.
Select "Edit the command line manually" and place the same password from step 1 immediately after /DECRYPT.
Repeat steps 4-6 for each job step that runs this type of package.
Press the "OK" button and re-run your job.
I was able to run my job successfully after that.
Related
All of a sudden one day (on my DEV PC) my Microsoft SQL Server 2012 instance (installed as instance name "SQL2012") would not start (all my other installed instances did). Trying to start it manually under Services failed. I don't recall making any recent changes prior to this. The cause of the failure was a mystery.
On inspecting Event Viewer, under System it showed a rather amusing error message [emphasis mine]:
The SQL Server (SQL2012) service terminated with the following service-specific error:
WARNING: You have until SQL Server (SQL2012) to logoff. If you have not logged off at this time, your session will be disconnected, and any open files or devices you have open may lose data.
checking under Application Event Log, I found these 2 error messages (preceded by a number of MSSQL$SQL2012 informational messages):
Script level upgrade for database 'master' failed because upgrade step 'msdb110_upgrade.sql' encountered error 200, state 7, severity 25. This is a serious error condition which might interfere with regular operation and the database will be taken offline. If the error happened during upgrade of the 'master' database, it will prevent the entire SQL Server instance from starting. Examine the previous errorlog entries for errors, take the appropriate corrective actions and re-start the database so that the script upgrade steps run to completion.
followed by:
Cannot recover the master database. SQL Server is unable to run. Restore master from a full backup, repair it, or rebuild it. For more information about how to rebuild the master database, see SQL Server Books Online.
Fearing having lost my system databases (and not having a backup of them to restore - who makes backups of their system dbs anyway??) and needing to access the instance, and attached databases - I was willing to try anything. Even the possible restore of the system databases: Restoring the SQL Server Master Database Even Without a Backup - but that looked quite complex.
Fortunately, I was eventually able to start the instance (thank you to this answer: https://stackoverflow.com/a/59676743/4993856 which I trusted, because Pinal Dave also mentions that particular switch in: SQL SERVER – Script level upgrade for database ‘master’ failed because upgrade step msdb110_upgrade.sql encountered error 926, state 1, severity 25) if I ran:
net start mssqlserver$SQL2012 /T902
This pointed to some issue with the upgrade script... (Remember SQL is installed with instance name: SQL2012, hence the mssqlserver$SQL2012 used above for the named instance).
After some more searching I discovered this post: Installing service pack / cumulative update on SQL Server 2016 / 2017 breaks database engine (not exactly the same SQL version as mine) which pointed to the following possible Region Settings setting (Control Panel [when viewed by 'icons'] > All Control Panel Items > Region > Administrative > "Change system locale..."):
"Beta: Use Unicode UTF-8 for worldwide language support" in Region Settings
THAT WAS IT!!! After de-selecting that option (and possibly restarting my computer), the MSSQL Server 2012 Instance started up without any issue, and I was able to access all my previously attached databases.
I assume the pending upgrade scripts ran successfully. Thinking back about it now, it is possible that I agreed to installing a SQL Update, and never bothered to test access to the instance afterwards.
I also don't recall exactly why I chose to enable that specific setting under Region Settings, possibly due to some Linux compatibility, but it looks like it has become defaulted 'on' in recent Windows builds.
I got the same problem SQL2017 after update Windows Patch Hotfix3391(KB5001228)
after restart server MSSQL Fail to start and event viewer shown error below
Script level upgrade for database 'master' failed because upgrade step 'msdb110_upgrade.sql' encountered error 200, state 7, severity 25. This is a serious error condition which might interfere with regular operation and the database will be taken offline. If the error happened during upgrade of the 'master' database, it will prevent the entire SQL Server instance from starting. Examine the previous errorlog entries for errors, take the appropriate corrective actions and re-start the database so that the script upgrade steps run to completion.
Solution
Fix by remove Beta:Use Unicode UTF-8 for Worldwide lang.. in the Region Settings
Then it require restart server. After restart MSSQL can start as normal.
The problem is the msdb_110.sql update script, the script is a bit of a mess, with mixed tabs and spaces (wtf?).
It tries to run a couple of procedures that fail, on startup of sql-server. They fail when the code-page is 65001 (usually because the BETA utf-8 code page option has been selected) and so SQL server fails to start.
This appears to happen any time a SQL Server update is installed. I only experience this error with SQL Server 2017, not 2019
Why?
Don't know? The script is a mess.
Solution
Deselect the use utd-8 code page option
Restart the machine
Start sql server and let it run the script
(optional) reselect the use utd-8 code page option
Restart machine again and sql server
(optinal but recommended) uninstall windows, install a unix and run postgres
I am dealing with my problem on some Windows Server 2019 (Core) with one running SQL Server 2019 CU4 instance each.
What we try to do
We are currently building a data warehouse with distributed databases. The individual layers of the DWH are located on one database server each. The data exchange between the layers/servers takes place via SSIS ETLs, which use Linked Servers to reach the other layers and drag and drop data. Each layer also has its own SSIS service instance and executes the corresponding SSIS packets.
The SSIS packages are called by SQL Server Agent jobs. We have a job that executes the SSIS packets (#1), which in turn calls another job (#2) as the last step, which after a short wait time executes the calling job (#1). Thus, controlled by schedules, a loop is created and data is continuously transferred with ETLs.
I hope this was not too much unnecessary background
The error
Basically the job is running and there are numerous successful executions. However, we are observing interruptions at job #1 without helpful information regarding the error. This means that the job history log refers to the SSIS log, which again only contains an "unexpected termination". In the SSIS log, we only see behavior that indicates that the ETL packet active at that time stopped after validation. Depending on the log level, nothing is logged at all, not even the execution of single packages of the project. The package where this error occurs is different and not limited to a specific one.
What I have already tried
Re-create the jobs and SSIS Enviroments by hand (scripted before)
Using the 32Bit Runtime
Upgrade the SSIS project/package version to
2019
Increase the log level to "verbose"
Patching the SQL Server to CU4
Save ssis dump files (couldn't find them or they weren't created)
Search Windows and SQL Server Logfiles
Does anyone have some suggestions or some ideas how to become more error specific informations?
Thank you very much and take care :)
UPDATE We have an error message (OLE DB 0xC0202009 and 0X80004005)!
In order to exclude the use of environments as a cause, I manually set the parameters in the SSIS job step instead of overwriting them by selecting an environment.
Long story short: Today it turns out that the parameter for an OLE DB Connection String is not passed correctly.
The following is specified as a parameter in the job step:
However, the following connection string is specified in the context of the error message:
Please note that some arguments are added twice to the parameter (red).
What could have caused that?
I have created a SSIS package and I am trying to run it locally. We use package configurations that point to sql tables and a XML config file. The package ran successfully for about a week, even when deployed to a SQL Server Agent Job in our STAGE environment.
Now, the only way I can get the package to run is by not using the Package Configurations and choosing EncryptSensitivewithPassword. If I change the package to DontSaveSensitive, I continuously get the error below:
An OLE DB record is available. Source: "Microsoft SQL Server Native Client 11.0" Hresult: 0x80040E4D Description: "Login failed for user 'Test_User'.".
Error: 0xC020801C at AgentCompany, Lookup [37]: SSIS Error Code DTS_E_CANNOTACQUIRECONNECTIONFROMCONNECTIONMANAGER. The AcquireConnection method call to the connection manager "Test" failed with error code 0xC0202009. There may be error messages posted before this with more information on why the AcquireConnection method call failed.
It is so strange that about a week ago, this package ran fine with the configurations and the DontSaveSensitive Option.
I have updated the config file to ensure that it is establishing the connection string to the appropriate database. I also test the connectivity on the connection managers and they all test successfully.
I also double checked the SQL Database where the user is trying to connect to ensure that it has permissions there and it does.
I am very confused. Please Help!
Updating dtsconfig file
Re-creating the connection managers
Making some DFT task DelayValidation to true
Changing the RunTime to 32 bit
EncrpytPasswordSensitive with package configs removed---This works but this is not the standard at my company and this is not how I developed and tested the package before
When you open/run a package, an OnInformation event is fired that says something like
The package is attempting to configure from the XML file "c:\ssisdata\so_56776576.dtsconfig".
When Visual Studio/SSDT opens/runs a package which says it uses configuration but for reasons, cannot get them, you should then see messages like
Warning loading so_56776576.dtsx: Failure importing configuration file: "c:\ssisdata\so_56776576.dtsconfig"
and
Warning loading so_56776576.dtsx: The configuration file "c:\ssisdata\so_56776576.dtsconfig" cannot be found. Check the directory and file name.
and
Warning loading so_56776576.dtsx: Failed to load at least one of the configuration entries for the package. Check configuration entries for "Configuration 1" and previous warnings to see descriptions of which configuration failed.
If someone has manually edited the config file and broken the XML, you'd see a warning like
Cannot load the XML configuration file. The XML configuration file may be malformed or not valid
The important thing to note with regard to configuration - if a configuration cannot be found, SSIS will continue along with the design time values. That is why it is crucial to check the warnings emitted when your package runs. If you are running manually, ensure that you have /rep ew specified so you report Errors and Warnings.
Guesses as to root cause
The package has the protection level of EncryptSensitiveWithUserKey which means the AD credentials of the package creator are used to hash things that might have sensitive information in them. I could be using AD authentication in my connection string and specify that the connection should be trusted but that entire block is still going to get encrypted against my Active Directory account. When you come along and attempt to maintain the package, it's not going to be able to decrypt the sensitive data as you are not me.
The two ways around that are to use a shared key (EncryptSensitiveWithPassword/EncryptPackageWithPassword) which is cumbersome to deal with plus it goes against the whole spirit of secrecy since everyone knows the secret. The other approach as you've identified is DontSaveSensitive and that's my go to for all of this.
The problem to be overcome is that with DontSaveSensitive is that every time you save, SSIS is going to wipe out any knowledge of user name and password from places that might be holding on to it - like a connection manager. The 2005/2008 strategy to hedge against this was to use Configuration or explicit overrides at run time to supply user name and password. My typical approach was to use configuration based on a table instead of XML as I was better at securing sensitive data in a table than I was mucking with ACL on the file system. The other challenge we had with multiple developers and file based configuration was that either everyone had to set their file systems up the same (and we developers are unique rainbow snowflakes so that's unlikely) or we need to use a network shared file which is great until someone adds their own values to it and breaks it or removes your changes or any of a host of other challenges.
I'm facing a strange behavior of SQL Server Agent when executing SSIS packages.
I have a job that includes many steps (mainly SSIS packages). Some steps fail mostly every day even the configuration is the same for all the steps.
I tried to delete/create the job, delete/create the SQL Server Agent Proxy but with no sucess.
I can't find any difference between the steps that fail and the ones that succeed.
This is the error returned by SQL Server Agent :
The package failed to load due to error 0xC0011008 "Error loading from XML. No further detailed error information can be specified for this problem because no Events object was passed where detailed error information
SQL Server version : 2014
SSIS version : 2014
EDIT :
In the Event Log I found an Information Message from User Profile Service that says :
Windows detected your registry file is still in use by other applications or services. The file will be unloaded now. The applications or services that hold your registry file may not function properly afterwards
Process 5924 (\Device\HarddiskVolume2\Program Files\Microsoft SQL Server\120\DTS\Binn\DTExec.exe) has opened key \REGISTRY\USER\S-X-X-XX-XXXXXXXXXX-XXXXXXXXXX-XXXXXXXXXX-XXXX\Control Panel\International
Process 5924 (\Device\HarddiskVolume2\Program Files\Microsoft SQL Server\120\DTS\Binn\DTExec.exe) has opened key \REGISTRY\USER\S-X-X-XX-XXXXXXXXXX-XXXXXXXXXX-XXXXXXXXXX-XXXX\Software\Microsoft\Windows\CurrentVersion
The SID corresponds to the Proxy User used to execute the SQL Job steps. And the timestamp corresponds is the same when the error occures in SQL Agent.
I think this is what causes the steps to fail.
Could we prevent Windows unloading this registry ?
The error was indeed caused by the fact that the User Profile Service forces the unloading of the Registry.
The solution that worked for me was to change the policy setting Do not forcefully unload the user registry at user logoff from "Not Configured" to Enabled.
Start the Local Group Policy Editor (gpedit.msc)
Go to Computer Configuration > Administrative Templates > System > User Profiles
Set "Do not forcefully unload the user registry at user logoff" to Enabled
Run gpupdate command.
Details can be found here : https://support.microsoft.com/en-us/help/2287297/a-com-application-may-stop-working-on-windows-server-2008-when-a-user
I have an SSIS Package that I created through the Import Export (32-bit) Tool. When I executed the package manually through the Execute Package Utility the package run successfully with no issues. However when I try to run the package through a Job Agent in SSMS I keep getting errors. The primary error I get seems to be:
Failed to decrypt protected XML node "DTS:Password" with error
0x8009000B "Key not valid for use in specified state" You may not be
authorized to access this information. This error occurs when there is
a cryptographic error. Verify that the correct key is available.
I'm using SQL Server 2008 R2.
I have researched this error to some degree and I think it has something to do with the package protection level. I feel like I've tried the configurations that make the most sense but none seem to be working for me. The Options are:
Encrypt sensitive data with user key
Do not save sensitive data
Encrypt sensitive data with password
Encrypt all data with user key
Encrypt all data with password
Rely on server storage and roles for access control
I feel like Ishould be using the last option here (Rely on server storage...) because I prefer to use SQL Server Authentication. I use SQL Server Authentication on the 'Choose Destination' window of the SQL Server Import and Export Wizard, and similarly I use this with the same username and password when I create the Job Agent in SSMS on the General Tab of the Job Step Properties. Is it possible that there is something that I need to add to the User I'm using in SSMS - even though it works outside of SSMS?
Something else I wondered that might have an impact is having the option "Drop and Re-create destination table" checked in the Column Mappings window of the Import and Export Tool. I was using a stored procedure to remove the tables before executing the Job Agent and I feel like ti was working at one point - could that have something to do with it?
Again the thing that baffles me most is that it runs no problem when I execute it manually through the 'SQL Server Execute Utility Package' tool.
I've included images of some of the windows I mentioned above if that helps.