ODBC Connection on SSIS package fails only when it goes through a loop - sql-server

Question

We've been recently trying to migrate out of SQL Server 2016 to SQL Server 2019 on our servers, that includes upgrading all the SSIS Packages we have on our catalog.
The migration wizard had no issues and migrated all packages with no errors, and on the surface everything seems OK. Even tried a test run on Visual Studio, and everything worked. But once we deployed all the packages on the catalog and tried a run via there, we started getting the following error:
none
Error: 0xC0014020 at Load ODI_PaymentDevice, ODBC Source [14]: SQLSTATE: HY010, Message: [Microsoft][ODBC Driver Manager] Function sequence error;
Error: 0xC0209029 at Load ODI_PaymentDevice, ODBC Source [14]: SSIS Error Code DTS_E_INDUCEDTRANSFORMFAILUREONERROR. The "ODBC Source.Outputs[ODBC Source Output]" failed because error code 0xC020F450 occurred, and the error row disposition on "ODBC Source" specifies failure on error. An error occurred on the specified object of the specified component. There may be error messages posted before this with more information about the failure.
Error: 0xC0047038 at Load ODI_PaymentDevice, SSIS.Pipeline: SSIS Error Code DTS_E_PRIMEOUTPUTFAILED. The PrimeOutput method on ODBC Source returned error code 0xC0209029. The component returned a failure code when the pipeline engine called PrimeOutput(). The meaning of the failure code is defined by the component, but the error is fatal and the pipeline stopped executing. There may be error messages posted before this with more information about the failure.
This is only on the packages that have an connection with Hive, using the ODBC Connector on SSIS, and then ODBC Data Source on a Data Flow
Based on the error code, it could be narrowed down to the ODBC Connection we have to our Hadoop-Hive cluster, the connection for sure works as we tested it on the Windows' ODBC Sources tool, and works as well on Visual Studio. We've researched a lot about this error and the different soulutions to this. So we tried a lot of different things.
Deleting the data source and then creating a new one (To update metadata)
Running in 32 Bit mode
Updating Microsoft's Hive ODBC driver
Switching to a different vendor's driver (CDATA)
Switching to an ADO.NET connection instead
Played around with the driver's configuration, almost all combinations possible
After trying all of this to no avail, we tried again on Visual Studio, and to our surprise, it also started to fail there too.
After trying a few different things, we could reproduce again the conditions in which the package worked, and it is the strangest thing, we could not find anyone with a similar issue on the internet so far.
So, as stated before, the connection works, and the package itself also does, BUT, we have a For Each Loop Container, that iterates through dates, to load data for the last X dates we have, so if there is any kind of loop container (For Each loop, for example) that contains a query against our ODBC source, it fails on the second loop around 100% of the time.
So that is the reason it worked on Visual Studio, because it only ran once (had only one date to process as test), but when deployed, it had to fetch real data, with a bunch of different dates.
To confirm that this is indeed the issue, deployed the package, and updated the table with dates to load, to have only available 1 day. And the package ran through. Also ruling out any parameter issue on the deployment/server/catalog.
After this discovery we tried a few different things:
Passing NULL on every column to see if there is some issues with metadata between loops
Also activated LOG_TRACE on the Hive ODBC driver, to have a very detailed log of what is happening, we see the query going out for the second loop on the log, and it also appears on TEZ (our Hive execution engine) but very briefly, only fractions of a second. And then it cancels itself, so the query is arriving the cluster, but somehow SSIS drops the connection by itself.
As mentioned before, we couldn't find anything like this before, and we cannot think of any other options to solve the issue without having to directly change the packages or not upgrading to 2019 at all, which is not ideal knowing that it is already outside of the mainstream support cycle.
Anyone has an idea how this might be solved or what may be causing this issue?

Answer 1

I have faced a very similar issue (if not the same) with the SSIS ODBC Source Component inside a For Loop for transferring records in batches from a remote PostgreSQL server to a database on MS SQL Server 2019. My Visual Studio is 2019 and the MS SQL Server is 2019 as well. The very weird thing was that the package was running as expected in VS (Debugging and Without Debugging), then it was working quite well through the SQL Job Agent of the SQL Server installed on my machine, but when deployed on the production SQL Server (the same version and psqlodbc driver installed there) the package was running successfully for the first iteration of the For Loop component and then unexpectedly was crashing, showing in the logs the same errors you have posted above: SQLSTATE: HY010, Message: [Microsoft][ODBC Driver Manager] Function sequence error;.....etc. After many hours spent on this without any success, I finally fixed it and now it is working like a charm; hence decided to share how I figured that out, so hopefully it may be of help to you or anyone facing that challenge.
What I found out is that for some reason the problem was happening inside the ODBC Source Component, but could not do much as it is like a black box. I fixed the problem by switching to a Script Source Component, so that I took control over the connection in the C# code. Here below, I also share the code:
#region Namespaces
using System;
using System.Data;
using System.Data.Odbc;
using Microsoft.SqlServer.Dts.Pipeline.Wrapper;
using Microsoft.SqlServer.Dts.Runtime.Wrapper;
#endregion
...........
[Microsoft.SqlServer.Dts.Pipeline.SSISScriptComponentEntryPointAttribute]
public class ScriptMain : UserComponent
{
...........
public override void CreateNewOutputRows()
{
string connectionString = this.Connections.PostgreSQLODBCConn.ConnectionString;
using (OdbcConnection conn = new OdbcConnection(connectionString))
{
using (OdbcCommand cmd = conn.CreateCommand())
{
cmd.CommandText = "SELECT * FROM fn_transfer_records(500000);";
cmd.CommandType = CommandType.Text;
DataTable dt = new DataTable();
conn.Open();
using (OdbcDataAdapter adapter = new OdbcDataAdapter(cmd))
{
adapter.Fill(dt);
foreach (DataRow row in dt.Rows)
{
Output0Buffer.AddRow();
Output0Buffer.col1 = (Int32)row["col1"];
Output0Buffer.col2= (double)row["col2"];
}
}
}
}
}
}
The PostgreSQLODBCConn used in the code above is the name of the Connection added to the connections collection of the Script Component added through the visual editor of the component when you double click on it.
Hope this would be of help...

Answer 2

Ensure your SQL Server Target version is set to SQL Server 2019. This can be found from the Project properties. This error is typical of a mismatched target server, as the issue is only present during deployment, and not during development.