SSIS Connection Manager - Azure SQL Database intermittent failure - sql-server

We have an SSIS job that connects to a hosted SQL Database in Azure.
Occasionally we get error messages when the connection manager is unable to acquire a connection.
Example error messages:
SSIS Error Code DTS_E_CANNOTACQUIRECONNECTIONFROMCONNECTIONMANAGER. The AcquireConnection method call to the connection manager "AzureCloud" failed with error code 0xC0202009. There may be error messages posted before this with more information on why the AcquireConnection method call failed. End Error
An OLE DB record is available. Source: "Microsoft SQL Server Native Client 10.0" Hresult: 0x80004005 Description: "Login timeout expired". An OLE DB record is available. Source: "Microsoft SQL Server Native Client 10.0" Hresult: 0x80004005 Description: "A network-related or instance-specific error has occurred while establishing a connection to SQL Server. Server is not found or not accessible. Check if instance name is correct and if SQL Server is configured to allow remote connections. For more information see SQL Server Books Online.". An OLE DB record is available. Source: "Microsoft SQL Server Native Client 10.0" Hresult: 0x80004005 Description: "Named Pipes Provider: Could not open a connection to SQL Server [53]. ". End Error Error: 2017-04-28 01:03:48.00 Code: 0xC020801C Source: Copy to stage Lookup [580] Description: SSIS Error Code DTS_E_CANNOTACQUIRECONNECTIONFROMCONNECTIONMANAGER. The AcquireConnection method call to the connection manager "Cloud" failed with error code 0xC0202009. There may be error messages posted before this with more information on why the AcquireConnection method call failed. End Error Error: 2017-04-28 01:03:48.83 Code: 0xC0047017 Source: Copy to stage SSIS.Pipeline Description: component "Lookup" (580) failed validation and returned error code 0xC020801C. End Error Error: 2017-04-28 01:03:49.65 Code: 0xC004700C Source: Copy to stage SSIS.Pipeline Description: One or more component failed validation. End Error Error: 2017-04-28 01:03:50.50 Code: 0xC0024107 Source: Copy to stage Description: There were errors during task validation. End Error DTExec: The package execution returned DTSER_FAILURE (1). Started: 1:00:12 AM Finished: 1:03:50 AM Elapsed: 218.339 seconds. The package execution failed. The step failed.,00:03:38,0,0,,,,0
Is there an Azure SQL Connection Manager that can use connection resiliency to retry acquiring a connection in the event that a network event (e.g. increased latency on the public internet) occurs?
Thanks!

There is no specified azure sql connector with more resiliency against transient errors. You need to implement this by yourself.
If you are using ssis packages in connection with sql server agent jobs, then you should consider using the retry feature of the agent's job steps.
if you are otherwise starting your packages from custom code, then utilize a retry library. For e.g. in c# there is a package called Polly, that handles retry policies very well.

Related

Sql jobs failed randomly to acquire connection after the activation of the Always ON feature

Since we activated the Always On mode on our server, we have some issues regarding our SQL jobs (running SSIS package) which failed randomly with this message :
Failed to acquire connection "DB". Connection may not be configured correctly or you may not have the right permissions on this connection.
More details :
This issue did not happen before the activation of the Always On
The connection is defined in an OLEDB component
We have another environment/server with this kind of architecture and there is no issue with it
Trace of the listener :
Message Executed as user: NT Service\SQLSERVERAGENT. Microsoft (R) SQL Server Execute Package Utility Version 15.0.2000.5 for 64-bit Copyright (C) 2019 Microsoft. All rights reserved.
Started: 2:18:01 PM Error: 2022-02-15 14:18:30.07
Code: 0xC0202009
Source: TestImport Connection manager "DB_PRIMARY"
Description: SSIS Error Code DTS_E_OLEDBERROR.
An OLE DB error has occurred. Error code: 0x80004005.
An OLE DB record is available. Source: "Microsoft SQL Server Native Client 11.0"
Hresult: 0x80004005 Description: "Login timeout expired".
An OLE DB record is available.
Source: "Microsoft SQL Server Native Client 11.0"
Hresult: 0x80004005
Description: "A network-related or instance-specific error has occurred while establishing a connection to SQL Server. Server is not found or not accessible. Check if instance name is correct and if SQL Server is configured to allow remote connections. For more information see SQL Server Books Online.".
An OLE DB record is available. Source: "Microsoft SQL Server Native Client 11.0" Hresult: 0x80004005 Description: "Named Pipes Provider: Could not open a connection to SQL Server [53]. ".
End Error Error: 2022-02-15 14:18:30.94 Code: 0xC00291EC
Source: Execute SQL Task Link coupons movement to collection Execute SQL Task D escription: Failed to acquire connection "DB_PRIMARY". Connection may not be configured correctly or you may not have the right permissions on this connection.
End Error DTExec: The package execution returned DTSER_FAILURE (1). Started: 2:18:01 PM Finished: 2:18:30 PM Elapsed: 29.984 seconds. The package execution failed. The step failed.
When enabling AlwaysOn, there is only one active database ("master") and synchronous or asynchornous copies that are call replicas ("slave") that you can't access to.
The SQL Agent operates on local database, that can be in the active or slave mode. When active, your job will operate. When slave your job will fail.
So is is necessary to ask is the database is active prior to all execution of SQL server Agent job.
I wrote some UDF to test if the database is reachable or not that you can have at :
SQL agent and AlwaysOn replicas (HADR – Availability Groups), is my database active?
And also to reproduce the SQL Server jobs on all replicas... !

SQL Agent job fails intermittently

SSIS packages deployed on SSIDB are scheduled through a SQL Agent job. The package ran with no issues when only one job is scheduled for it. Later when we scheduled another job to run the same package with different parameters, the job randomly fails with connectivity error.
As you can see from the below message, instead of passing the parameter value, the parameter name is referred to in Database.
SSIS Error Code DTS_E_CANNOTACQUIRECONNECTIONFROMCONNECTIONMANAGER.
The AcquireConnection method call to the connection manager
"Destination" failed with error code 0xC0202009. There may be error
messages posted before this with more information on why the
AcquireConnection method call failed.
SSIS Error Code DTS_E_OLEDBERROR. An OLE DB error has occurred. Error
code: 0x80004005.An OLE DB record is available. Source: "Microsoft SQL
Server Native Client 11.0" Hresult: 0x80004005 Description: "Login
failed for user 'NT SERVICE\SQLSERVERAGENT'.".An OLE DB record is
available. Source: "Microsoft SQL Server Native Client 11.0" Hresult:
0x80004005 Description: "Cannot open database "Server_N" requested by
the login. The login failed."
P.S. There is no overlap between these jobs.

Facing Login Timeout issues in SSIS Package

SSIS jobs are getting failed frequently with the below error message. Although it restarts and runs successfully in the next run, SSIS jobs are getting failed sporadically in the subsequent runs. These issues were happening after our SQL Server was patched with KB4338824 and KB4338815 updates. Any help?
Error: 2018-07-19 23:38:22.52 Code: 0xC0202009 Source:
ParallelExport Connection manager "XXXXConnectionString"
Description: SSIS Error Code DTS_E_OLEDBERROR. An OLE DB error has
occurred. Error code: 0x80004005. An OLE DB record is available.
Source: "Microsoft OLE DB Provider for SQL Server" Hresult:
0x80004005 Description: "Login timeout expired". End Error Error:
2018-07-19 23:38:22.52 Code: 0xC00291EC Source: Get Export List
to Process Execute SQL Task Description: Failed to acquire
connection "XXXXConnectionString". Connection may not be configured
correctly or you may not have the right permissions on this
connection. End Error

Package upgraded from ssis2005 to ssis2008r2 not executing from sql agent job

I upgraded a set of ssis packages from 2005 to 2008r2. the corresponding database has ben upgraded from sql2005 to sql2014. The packages run fine from BIDS. But when i execute them from new sql server agent job ( i have checked the use-32 bit runtime check box)., the package fails with error message as
Executed as user: ADMIN\SvcSQL13DVMDDEW03. Microsoft (R) SQL Server Execute Package Utility Version 10.50.6000.34 for 32-bit Copyright (C) Microsoft Corporation 2010. All rights reserved. Started: 11:57:06 Error: 2017-07-27 11:57:08.96 Code: 0xC0209302 Source: CBSBankRec50244961Transform Connection manager "TempRBGLINT" Description: SSIS Error Code DTS_E_OLEDB_NOPROVIDER_ERROR. The requested OLE DB provider SQLNCLI10.1 is not registered. Error code: 0x00000000. An OLE DB record is available. Source: "Microsoft OLE DB Service Components" Hresult: 0x80040154 Description: "Class not registered". End Error Error: 2017-07-27 11:57:08.97 Code: 0x00000008 Source: Check Database Connection Description: The script returned a failure result. End Error Error: 2017-07-27 11:57:24.55 Code: 0xC0202009 Source: CBSBankRec50244961Transform Connection manager "RBGLINT" Description: SSIS Error Code DTS_E_OLEDBERROR. An OLE DB error has occurred. Error code: 0x80004005. An OLE DB record is available. Source: "Microsoft SQL Server Native Client 10.0" Hresult: 0x80004005 Description: "Login timeout expired". An OLE DB record is available. Source: "Microsoft SQL Server Native Client 10.0" Hresult: 0x80004005 Description: "A network-related or instance-specific error has occurred while establishing a connection to SQL Server. Server is not found or not accessible. Check if instance name is correct and if SQL Server is configured to allow remote connections. For more information see SQL Server Books Online.". An OLE DB record is available. Source: "Microsoft SQL Server Native Client 10.0" Hresult: 0x80004005 Description: "Named Pipes Provider: Could not open a connection to SQL Server [2]. ". End Error Error: 2017-07-27 11:57:24.57 Code: 0xC00291EC Source: Log Package 'Started' Execute SQL Task Description: Failed to acquire connection "RBGLINT". Connection may not be configured correctly or you may not have the right permissions on this connection. End Error Error: 2017-07-27 11:57:40.21 Code: 0xC0202009 Source: CBSBankRec50244961Transform Connection manager "RBGLINT" Description: SSIS Error Code DTS_E_OLEDBERROR. An OLE DB error has occurred. Error code: 0x80004005. An OLE DB record is available. Source: "Microsoft SQL Server Native Client 10.0" Hresult: 0x80004005 Description: "Login timeout expired". An OLE DB record is available. Source: "Microsoft SQL Server Native Client 10.0" Hresult: 0x80004005 Description: "A network-related or instance-specific error has occurred while establishing a connection to SQL Server. Server is not found or not accessible. Check if instance name is correct and if SQL Server is configured to allow remote connections. For more information see SQL Server Books Online.". An OLE DB record is available. Source: "Microsoft SQL Server Native Client 10.0" Hresult: 0x80004005 Description: "Named Pipes Provider: Could not open a connection to SQL Server [2]. ". End Error Error: 2017-07-27 11:57:40.22 Code: 0xC00291EC Source: Log Package 'Failure' Execute SQL Task Description: Failed to acquire connection "RBGLINT". Connection may not be configured correctly or you may not have the right permissions on this connection. End Error DTExec: The package execution returned DTSER_FAILURE (1). Started: 11:57:06 Finished: 11:57:40 Elapsed: 34.102 seconds. The package execution failed. The step failed.
The provider used in the config files for the packages is SQLNCLI10.1 . in the ssis package there is a ole db connection manager - RBGLINT. Just for information - this was automatically upgraded from sql native client \oledb provider to sql native client \ ole db provider 10.0. in the connection manager. I do not see this causing the issue. But just for information.
Any inputs?
From the error message you provided the main error is:
The requested OLE DB provider SQLNCLI10.1 is not registered
From the link below you there is 2 methods for solving this issue:
The requested OLE DB provider SQLNCLI10.1 is not registered
you need to do is to open the connection manager/connection string and edit it to use Native Client 11.0 which is what comes as default with SSIS 2014
Another alternative is to install the Native Client 10.0 from SSIS 2008 installable:
https://blogs.iis.net/rickbarber/installing-sql-server-native-client-10-0
http://www.keepitsimpleandfast.com/2013/04/the-sqlncli10-provider-is-not.html

Package deployment error - change connection string to SQLNCLI10 or find and install support for SQLNCLI.1?

I tried to run a package under SQL server agent. Most of my error messages indicate that I have to - change connection string to SQLNCLI10 or find and install support for SQLNCLI.1.
I need to know the meaning of this error. Is it as simple as installing a driver and registering it or is it more than just drivers. Could there be non-driver related reasons for this problem ? I changed the provider as mentioned in the error and now i get new errors, which are given after 1st error.
Thanks.
Full error -
Code: 0xC0209302 Source: MyPackage Connection manager "MyOleDBConnection" Description: SSIS Error Code DTS_E_OLEDB_NOPROVIDER_ERROR. The requested OLE DB provider SQLNCLI.1 is not registered.
Error code: 0x00000000. An OLE DB record is available. Source: "Microsoft OLE DB Service Components" Hresult: 0x80040154 Description: "Class not registered".
End Error
Code: 0xC020F42A Source: MyPackage Connection manager "MyOleDBConnection" Description: Consider changing the PROVIDER in the connection string to SQLNCLI10 or
visit http://www.microsoft.com/downloads to find and install support for SQLNCLI.1.
End Error
Code: 0xC020801C Source: MyPackage Log provider "SSIS log provider for SQL Server" Description: SSIS Error Code DTS_E_CANNOTACQUIRECONNECTIONFROMCONNECTIONMANAGER.
The AcquireConnection method call to the connection manager "MyOleDBConnection" failed with error code 0xC0209302.
There may be error messages posted before this with more information on why the AcquireConnection method call failed.
COM error object information is available. Source: "MyPackage" error code: 0xC0209302
Description: "Consider changing the PROVIDER in the connection string to SQLNCLI10 or
visit http://www.microsoft.com/downloads to find and install support for SQLNCLI.1. ".
End Error
Code: 0xC0209302 Source: MyPackage Connection manager "MyOleDBConnection" Description: SSIS Error Code DTS_E_OLEDB_NOPROVIDER_ERROR.
The requested OLE DB provider SQLNCLI.1 is not registered. Error code: 0x00000000.
An OLE DB record is available. Source: "Microsoft OLE DB Service Components" Hresult: 0x80040154 Description: "Class not registered".
End Error
Code: 0xC020F42A Source: MyPackage Connection manager "MyOleDBConnection" Description: Consider changing the PROVIDER in the connection string to SQLNCLI10 or
visit http://www.microsoft.com/downloads to find and install support for SQLNCLI.1.
End Error
Code: 0xC020801C Source: DFT Insert Execution Log OLE_DST Insert SSIS Execution Log [25] Description: SSIS Error Code DTS_E_CANNOTACQUIRECONNECTIONFROMCONNECTIONMANAGER.
The AcquireConnection method call to the connection manager "MyOleDBConnection" failed with error code 0xC0209302.
There may be error messages posted before this with more information on why the AcquireConnection method call failed.
End Error
Code: 0xC0047017 Source: DFT Insert Execution Log SSIS.Pipeline Description: component "OLE_DST Insert SSIS Execution Log" (25) failed validation and returned error code 0xC020801C.
End Error
Code: 0xC004700C Source: DFT Insert Execution Log SSIS.Pipeline Description: One or more component failed validation.
End Error
Code: 0xC0024107 Source: DFT Insert Execution Log Description: There were errors during task validation.
End Error
DTExec: The package execution returned DTSER_FAILURE (1).
One more error -
Code: 0xC0202009 Source: MyPackage Connection manager "MyOLEDBConnection" Description: SSIS Error Code DTS_E_OLEDBERROR. An OLE DB error has occurred. Error code: 0x80040E4D.
An OLE DB record is available. Source: "Microsoft SQL Server Native Client 10.0" Hresult: 0x80040E4D Description: "Login failed for user 'MyUser'.".
End Error
Code: 0xC020801C Source: MyPackage Log provider "SSIS log provider for SQL Server" Description: SSIS Error Code DTS_E_CANNOTACQUIRECONNECTIONFROMCONNECTIONMANAGER. The AcquireConnection
method call to the connection manager "MyOLEDBConnection" failed with error code 0xC0202009. There may be error messages posted before this with more information on why the AcquireConnection method
call failed. COM error object information is available. Source: "MyPackage" error code: 0xC0202009 Description: "SSIS Error Code DTS_E_OLEDBERROR. An OLE DB error has occurred. Error code: 0x80040E4D.
An OLE DB record is available. Source: "Microsoft SQL Server Native Client 10.0" Hresult: 0x80040E4D Description: "Login failed for user 'MyUser'.". ".
End Error
Code: 0xC0202009 Source: MyPackage Connection manager "MyOLEDBConnection" Description: SSIS Error Code DTS_E_OLEDBERROR. An OLE DB error has occurred. Error code: 0x80040E4D.
An OLE DB record is available. Source: "Microsoft SQL Server Native Client 10.0" Hresult: 0x80040E4D Description: "Login failed for user 'MyUser'.".
End Error
Code: 0xC020801C Source: DFT Insert Execution Log OLE_DST Insert SSIS Execution Log [25] Description: SSIS Error Code DTS_E_CANNOTACQUIRECONNECTIONFROMCONNECTIONMANAGER.
The AcquireConnection method call to the connection manager "MyOLEDBConnection" failed with error code 0xC0202009. There may be error messages posted before this with more information
on why the AcquireConnection method call failed.
End Error
Code: 0xC0047017 Source: DFT Insert Execution Log SSIS.Pipeline Description: component "OLE_DST Insert SSIS Execution Log" (25) failed validation and returned error code 0xC020801C.
End Error
Code: 0xC004700C Source: DFT Insert Execution Log SSIS.Pipeline Description: One or more component failed validation.
End Error
Code: 0xC0024107 Source: DFT Insert Execution Log Description: There were errors during task validation.
End Error
Understanding the problem
The SQL Server Native Client provider is SQLNCLI* in an OLE DB Connection String.
2005 uses SQLNCLI.1
2008 uses SQLNCLI10
2012 uses SQLNCLI11
As of this posting, there is not a SQLNCLI12 included with the CTP2 release of SQL Server 2014.
These providers can generally talk to previous versions of SQL Server but version forwardness is less probable (with the exception of SQLNCLI11 communicating with 2014).
Root cause
You have an OLE DB Connection string using a provider that does not exist on that machine. Perhaps you built the package in a 2005 format but are running it on a 2008 instance. At any rate, your connection strings in the format of
Provider=SQLNCLI.1;Server=myServerAddress;Database=myDataBase;Uid=myUsername;Pwd=myPassword;
would need to be updated to match the provider on the machine
Provider=SQLNCLI10;Server=myServerAddress;Database=myDataBase;Uid=myUsername;Pwd=myPassword;
This can generally be addressed through judicious use of Configurations

Resources