We've been recently trying to migrate out of SQL Server 2016 to SQL Server 2019 on our servers, that includes upgrading all the SSIS Packages we have on our catalog.
The migration wizard had no issues and migrated all packages with no errors, and on the surface everything seems OK. Even tried a test run on Visual Studio, and everything worked. But once we deployed all the packages on the catalog and tried a run via there, we started getting the following error:
none
Error: 0xC0014020 at Load ODI_PaymentDevice, ODBC Source [14]: SQLSTATE: HY010, Message: [Microsoft][ODBC Driver Manager] Function sequence error;
Error: 0xC0209029 at Load ODI_PaymentDevice, ODBC Source [14]: SSIS Error Code DTS_E_INDUCEDTRANSFORMFAILUREONERROR. The "ODBC Source.Outputs[ODBC Source Output]" failed because error code 0xC020F450 occurred, and the error row disposition on "ODBC Source" specifies failure on error. An error occurred on the specified object of the specified component. There may be error messages posted before this with more information about the failure.
Error: 0xC0047038 at Load ODI_PaymentDevice, SSIS.Pipeline: SSIS Error Code DTS_E_PRIMEOUTPUTFAILED. The PrimeOutput method on ODBC Source returned error code 0xC0209029. The component returned a failure code when the pipeline engine called PrimeOutput(). The meaning of the failure code is defined by the component, but the error is fatal and the pipeline stopped executing. There may be error messages posted before this with more information about the failure.
This is only on the packages that have an connection with Hive, using the ODBC Connector on SSIS, and then ODBC Data Source on a Data Flow
Based on the error code, it could be narrowed down to the ODBC Connection we have to our Hadoop-Hive cluster, the connection for sure works as we tested it on the Windows' ODBC Sources tool, and works as well on Visual Studio. We've researched a lot about this error and the different soulutions to this. So we tried a lot of different things.
Deleting the data source and then creating a new one (To update metadata)
Running in 32 Bit mode
Updating Microsoft's Hive ODBC driver
Switching to a different vendor's driver (CDATA)
Switching to an ADO.NET connection instead
Played around with the driver's configuration, almost all combinations possible
After trying all of this to no avail, we tried again on Visual Studio, and to our surprise, it also started to fail there too.
After trying a few different things, we could reproduce again the conditions in which the package worked, and it is the strangest thing, we could not find anyone with a similar issue on the internet so far.
So, as stated before, the connection works, and the package itself also does, BUT, we have a For Each Loop Container, that iterates through dates, to load data for the last X dates we have, so if there is any kind of loop container (For Each loop, for example) that contains a query against our ODBC source, it fails on the second loop around 100% of the time.
So that is the reason it worked on Visual Studio, because it only ran once (had only one date to process as test), but when deployed, it had to fetch real data, with a bunch of different dates.
To confirm that this is indeed the issue, deployed the package, and updated the table with dates to load, to have only available 1 day. And the package ran through. Also ruling out any parameter issue on the deployment/server/catalog.
After this discovery we tried a few different things:
Passing NULL on every column to see if there is some issues with metadata between loops
Also activated LOG_TRACE on the Hive ODBC driver, to have a very detailed log of what is happening, we see the query going out for the second loop on the log, and it also appears on TEZ (our Hive execution engine) but very briefly, only fractions of a second. And then it cancels itself, so the query is arriving the cluster, but somehow SSIS drops the connection by itself.
As mentioned before, we couldn't find anything like this before, and we cannot think of any other options to solve the issue without having to directly change the packages or not upgrading to 2019 at all, which is not ideal knowing that it is already outside of the mainstream support cycle.
Anyone has an idea how this might be solved or what may be causing this issue?
I have faced a very similar issue (if not the same) with the SSIS ODBC Source Component inside a For Loop for transferring records in batches from a remote PostgreSQL server to a database on MS SQL Server 2019. My Visual Studio is 2019 and the MS SQL Server is 2019 as well. The very weird thing was that the package was running as expected in VS (Debugging and Without Debugging), then it was working quite well through the SQL Job Agent of the SQL Server installed on my machine, but when deployed on the production SQL Server (the same version and psqlodbc driver installed there) the package was running successfully for the first iteration of the For Loop component and then unexpectedly was crashing, showing in the logs the same errors you have posted above: SQLSTATE: HY010, Message: [Microsoft][ODBC Driver Manager] Function sequence error;.....etc. After many hours spent on this without any success, I finally fixed it and now it is working like a charm; hence decided to share how I figured that out, so hopefully it may be of help to you or anyone facing that challenge.
What I found out is that for some reason the problem was happening inside the ODBC Source Component, but could not do much as it is like a black box. I fixed the problem by switching to a Script Source Component, so that I took control over the connection in the C# code. Here below, I also share the code:
#region Namespaces
using System;
using System.Data;
using System.Data.Odbc;
using Microsoft.SqlServer.Dts.Pipeline.Wrapper;
using Microsoft.SqlServer.Dts.Runtime.Wrapper;
#endregion
...........
[Microsoft.SqlServer.Dts.Pipeline.SSISScriptComponentEntryPointAttribute]
public class ScriptMain : UserComponent
{
...........
public override void CreateNewOutputRows()
{
string connectionString = this.Connections.PostgreSQLODBCConn.ConnectionString;
using (OdbcConnection conn = new OdbcConnection(connectionString))
{
using (OdbcCommand cmd = conn.CreateCommand())
{
cmd.CommandText = "SELECT * FROM fn_transfer_records(500000);";
cmd.CommandType = CommandType.Text;
DataTable dt = new DataTable();
conn.Open();
using (OdbcDataAdapter adapter = new OdbcDataAdapter(cmd))
{
adapter.Fill(dt);
foreach (DataRow row in dt.Rows)
{
Output0Buffer.AddRow();
Output0Buffer.col1 = (Int32)row["col1"];
Output0Buffer.col2= (double)row["col2"];
}
}
}
}
}
}
The PostgreSQLODBCConn used in the code above is the name of the Connection added to the connections collection of the Script Component added through the visual editor of the component when you double click on it.
Hope this would be of help...
Ensure your SQL Server Target version is set to SQL Server 2019. This can be found from the Project properties. This error is typical of a mismatched target server, as the issue is only present during deployment, and not during development.
I am dealing with my problem on some Windows Server 2019 (Core) with one running SQL Server 2019 CU4 instance each.
What we try to do
We are currently building a data warehouse with distributed databases. The individual layers of the DWH are located on one database server each. The data exchange between the layers/servers takes place via SSIS ETLs, which use Linked Servers to reach the other layers and drag and drop data. Each layer also has its own SSIS service instance and executes the corresponding SSIS packets.
The SSIS packages are called by SQL Server Agent jobs. We have a job that executes the SSIS packets (#1), which in turn calls another job (#2) as the last step, which after a short wait time executes the calling job (#1). Thus, controlled by schedules, a loop is created and data is continuously transferred with ETLs.
I hope this was not too much unnecessary background
The error
Basically the job is running and there are numerous successful executions. However, we are observing interruptions at job #1 without helpful information regarding the error. This means that the job history log refers to the SSIS log, which again only contains an "unexpected termination". In the SSIS log, we only see behavior that indicates that the ETL packet active at that time stopped after validation. Depending on the log level, nothing is logged at all, not even the execution of single packages of the project. The package where this error occurs is different and not limited to a specific one.
What I have already tried
Re-create the jobs and SSIS Enviroments by hand (scripted before)
Using the 32Bit Runtime
Upgrade the SSIS project/package version to
2019
Increase the log level to "verbose"
Patching the SQL Server to CU4
Save ssis dump files (couldn't find them or they weren't created)
Search Windows and SQL Server Logfiles
Does anyone have some suggestions or some ideas how to become more error specific informations?
Thank you very much and take care :)
UPDATE We have an error message (OLE DB 0xC0202009 and 0X80004005)!
In order to exclude the use of environments as a cause, I manually set the parameters in the SSIS job step instead of overwriting them by selecting an environment.
Long story short: Today it turns out that the parameter for an OLE DB Connection String is not passed correctly.
The following is specified as a parameter in the job step:
However, the following connection string is specified in the context of the error message:
Please note that some arguments are added twice to the parameter (red).
What could have caused that?
I use SQL Server 2014 and SSIS with Visual studio 2013.
I need to make some changes in one OLTP database (Here it is my destination database) in several tables with one of my SSIS package, and some times that takes a lot of time to execute and during this time it locks some tables in OLTP database and I can't anymore work with may application which works with OLTP database.
Before my package have been run one time in day and it was at midnight but now we decided to run this package every 30 min.
I need to use snapshot isolation level in OLTP databse to be sure that SSIS package does not lock the table for long time.
At first I want use IsolationLevel equal to Snapshot but it does not work.
Then I found that I should handel the Isolation leve manually inside the ControlFlow for using the Isolation Snapshot.
I have created a sample because my real SSIS package is really huge.
I have created a database and then, I have set Allow snapshot isolation with true.
After that I have created a SSIS package in which I have these variables:
Then I have created a Control flow like that :
In the first SQL Execution Task I have run this command:
SET TRANSACTION ISOLATION LEVEL SNAPSHOT
Then in second SQL Execution Task I have started a Transaction :
BEGIN TRANSACTION
Then I have added a Dataflow task. Inside of my DataFlow task there are two OLE DB. One as source and second one is destination.
The source and target are both in the same server and in the different database.
Then I change the property of my Connection Manager like below :
And then I have installed and configured my MSDTC in my computer. (Source and Destination are both on the same server and ssis package is running in the same server).
Then I configur my service like that :
Then in my firewall I have given the access to MSDTC like that :
Then I have configured my firewall's advance configuration like that :
All of the Task in my SSIS package have these configurations for IsolationLevel and TransactionOption :
My control flow has these configurations :
When I run the package I get this error in my OLE DB destination.
[OLE DB Destination [2]] Error: SSIS Error Code
DTS_E_CANNOTACQUIRECONNECTIONFROMCONNECTIONMANAGER.
The AcquireConnection method call to the connection manager
".<ServerName>.<DatabaseName>" failed with error code 0xC001A004.
There may be error messages posted before
this with more information on why the AcquireConnection method call failed.
[SSIS.Pipeline] Error:
OLE DB Destination failed validation and returned error code 0xC020801C.
What should I do to be able to handel manually the transaction and isolation level as Snapshot in my SSIS package?
My SSIS Package works, but fails as a SSMS job (Error: 0xC0016016)
I am posting this question and sharing my solution to this issue as my own question because I didn't see the posed problem match the specific issue I encountered and the answers seemed to be scattered in different forum questions.
Background:
I have four SSIS packages on SQL Server 2012 that import a table from SQL Server 2008 R2, 2008, or 2005, depending on the specific package. I use a designated sql server login and password for the source database and integrated Windows security for the target database.
Within SSIS I am able to run each package without a problem.
To ensure this package ran on a schedule, I set up a SSMS job on the same server as the SSIS package. In Job step properties, I chose: SQL Server Integration Services Package > run as SQL Sever Agent Service Account > Package source: File system).
The Symptom:
When manually running the job to make sure it worked, I got an error and saw this in the Log File Viewer. This was the first of several errors, but as this was chronologically the first error, I looked into this one first.
Error: 2014-10-24 09:52:34.48
Code: 0xC0016016
Source: [Redacted -- the correct name of the table I was importing]
Description: Failed to decrypt protected XML node "DTS:Password" with
error 0x8009000B "Key not valid for use in specified state.". You may
not be authorized to access this information. This error occurs when
there is a cryptographic error. Verify that the correct key is
available.
End Error
I looked up the error code on Google and started looking at resolutions. Rather than retelling how I found the bits and pieces of the resolution, I am presenting an actionable result in sequence -- at least for me and the network infrastructure I'm working with.
In the "properties" panel of the SSIS solution (do this first) and each package in that solution, reset the "ProtectionLevel" attribute to EncryptSensitiveWithPassword and set a password. The package passwords must match the solution password.
Just a double-check, run your packages in debug mode to make sure that no other new issues arise. In my case, I needed to re-enter the sql server password for the source server database.
Rebuild your SSIS solution.
In SSMS, open your job and open the job step properties for the task in question.
Select the "Command Line" tab. A "Package Password" popup appears. Enter the password that you entered in step 1.
Select "Edit the command line manually" and place the same password from step 1 immediately after /DECRYPT.
Repeat steps 4-6 for each job step that runs this type of package.
Press the "OK" button and re-run your job.
I was able to run my job successfully after that.
I implemented SSIS package which moves data from Sql Server database to another one. This package has set of Data Flow Tasks which copy data simultaneously in different tables. Each Data Flow Task contain OLE DB datasource and Sql Server destination.
Package worked fine until I decided to implement transaction. I found that it is not possible to just set TransactionOption to Supported on package level, because SSIS cannot handle transactions in multiple simultaneous processes. So, I decided to use this way:
http://consultingblogs.emc.com/jamiethomson/archive/2005/08/20/SSIS-Nugget_3A00_-RetainSameConnection-property-of-the-OLE-DB-Connection-Manager.aspx
But now I have another problem. I have "Unable to bulk copy data. You may need to run this package as an administrator" errors. These errors occur in random places. For example if I ran package in the first time Data Flow Task named "Task A" can be executed correctly, but when I run package in the second time it can throw the error.
How do I can implement transaction in my case? (Changing of package in order to perform execution of Data Flow tasks sequentially is not an option)
I got a recent error with our MS SQL Server 2008R2 and SSIS. Found the error:
[SQL Server Destination [16]] Error: Unable to bulk copy data. You may need to run this package as an administrator.
[SSIS.Pipeline] Error: component "SQL Server Destination" (16) failed the pre-execute phase and returned error code 0xC0202071.
but could not solve it with running as Admin. The error only came with one step and I finally found out that I get rid of the error when I increased the timeout of the SQL Server Destination. Funny is that with the read of external ADO NET Source I get a proper error that helped me to see the timeout is the problem.