Exporting To Azure SQL from Local SQL Database Failure - sql-server

I have a local SQL Server 2012 database that used to export tables and views to Azure SQL daily. It's been working flawlessly for 3 months.
The export tasks were created by the SQL Server Studio Import/Export Wizard. They were saved into a SQL Server SSIS package and ran daily by a SQL Server Job.
Issue I am having:
For the past few months, the export process have been failing randomly with multiple errors. It mentions that the connection has dropped. I don't think this is the problem.
I can run SELECT statements against the Azure database in SSMS with no problem. The Azure database is read by Microsoft Power BI with no problem too. So I think the problem is local to my network.
Executed as user: HIDDEN.
Microsoft (R) SQL Server Execute Package Utility Version 11.0.2100.60 for 64-bit
Copyright (C) Microsoft Corporation. All rights reserved.
Started: 10:43:37
Error: 2016-07-05 10:43:37.84
Code: 0xC0016016
Source: Azure - Table - INV1
Description: Failed to decrypt protected XML node
"DTS:Password" with error 0x8009000B "Key not valid for use in
specified state.". You may not be authorized to access this
information. This error occurs when there is a cryptographic error.
Verify that the correct key is available. End Error Error:
2016-07-05 10:45:05.82 Code: 0xC0202009 Source: Data Flow Task
1 Destination - INV1 [2] Description: SSIS Error Code
DTS_E_OLEDBERROR. An OLE DB error has occurred. Error code:
0x80004005. An OLE DB record is available. Source: "Microsoft OLE DB
Provider for SQL Server" Hresult: 0x80004005 Description:
"[DBNETLIB][ConnectionWrite (send()).]General network error. Check
your network documentation.". End Error Error: 2016-07-05
10:45:05.82 Code: 0xC0209029 Source: Data Flow Task 1
Destination - INV1 [2] Description: SSIS Error Code
DTS_E_INDUCEDTRANSFORMFAILUREONERROR. The "Destination -
INV1.Inputs[Destination Input]" failed because error code 0xC020907B
occurred, and the error row disposition on "Destination -
INV1.Inputs[Destination Input]" specifies failure on error. An error
occurred on the specified object of the specified component. There
may be error messages posted before this with more information about
the failure. End Error Error: 2016-07-05 10:45:05.82 Code:
0xC0047022 Source: Data Flow Task 1 SSIS.Pipeline Description:
SSIS Error Code DTS_E_PROCESSINPUTFAILED. The ProcessInput method on
component "Destination - INV1" (2) failed with error code 0xC0209029
while processing input "Destination Input" (15). The identified
component returned an error from the ProcessInput method. The error is
specific to the component, but the error is fatal and will cause the
Data Flow task to stop running. There may be error messages posted
before this with more information about the failure. End Error
Error: 2016-07-05 10:45:05.82 Code: 0xC02020C4 Source: Data
Flow Task 1 Source - INV1 [97] Description: The attempt to add a
row to the Data Flow task buffer failed with error code 0xC0047020.
End Error Error: 2016-07-05 10:45:05.82 Code: 0xC0047038
Source: Data Flow Task 1 SSIS.Pipeline Description: SSIS Error
Code DTS_E_PRIMEOUTPUTFAILED. The PrimeOutput method on Source - INV1
returned error code 0xC02020C4. The component returned a failure code
when the pipeline engine called PrimeOutput(). The meaning of the
failure code is defined by the component, but the error is fatal and
the pipeline stopped executing. There may be error messages posted
before this with more information about the failure. End Error
DTExec: The package execution returned DTSER_FAILURE (1). Started:
10:43:37 Finished: 10:45:05 Elapsed: 88.156 seconds. The package
execution failed.
Sometimes it works fine, other times it fails. I've noticed that smaller tables export fine, but larger ones like the error above have about 12,000 records. It fails randomly in different amount of records.
Has anyone any experience with what is going on?
Thanks!

Related

2 Hours for an SSIS Job With a Linked Server

I have an SSIS job which takes on average 2 hours to complete (I did not write it and have never written an SSIS package). It will occasionally fail with the error:
Executed as user: ICAT\SQL_AgentSvc. Microsoft [...] Started: 12:15:00 AM Error: 2018-01-24 02:00:47.97 Code: 0xC0202009 Source: Data Flow Task 5 Source 21 - sSlip [260] Description: SSIS Error Code DTS_E_OLEDBERROR. An OLE DB error has occurred. Error code: 0x80004005. An OLE DB record is available. Source: "Microsoft OLE DB Provider for SQL Server" Hresult: 0x80004005 Description: "[DBNETLIB][ConnectionRead (recv()).]General network error. Check your network documentation.". End Error Error: 2018-01-24 02:00:47.98 Code: 0xC0047038 Source: Data Flow Task 5 SSIS.Pipeline Description: SSIS Error Code DTS_E_PRIMEOUTPUTFAILED. The PrimeOutput method on Source 21 - sSlip returned error code 0xC0202009. The component returned a failure code when the pipeline engine called PrimeOutput(). The meaning of the failure code is defined by the component, but the error is fatal and the pipeline stopped executing. There may be error messages posted before this with more information about the failure. End Error DTExec: The package execution returned DTSER_FAILURE (1). Started: 12:15:00 AM Finished: 2:00:48 AM Elapsed: 6334.94 seconds. The package execution failed. The step failed.
Does this have anything to do with the time the job takes? I ask that because my usual resolution is to run the job manually in order to get the data the client is missing the day after it fails, but yesterday that failed and presented the error:
Login timeout expired
The job has several preparation steps and data flow steps. Each preparation step truncates tables in the destination database and then each data flow step grabs all data from identical tables in the linked source and moves the data to the destination. There are 23 tables it operates on. A couple of the tables have over one million records, one has 800k+ and several more have numbers in the tens of thousands.

SSIS Package Fails when run as Job

Within SSIS I am able to run each package without a problem. However when I run it via SQL job, I am encountering these errors:
Message:
Code: 0xC0047022 Description: SSIS Error Code DTS_E_PROCESSINPUTFAILED. The ProcessInput method on component "ALL KN and UNK BR" (120) failed with error code 0xC0047020 while processing input "Union All Input 2" (144). The identified component returned an error from the ProcessInput method. The error is specific to the component, but the error is fatal and will cause the Data Flow task to stop running. There may be error messages posted before this with more information about the failure.
Code: 0xC02020C4 Description: The attempt to add a row to the Data Flow task buffer failed with error code 0xC0047020.
Code: 0xC0047038 Description: SSIS Error Code DTS_E_PRIMEOUTPUTFAILED. The PrimeOutput method on SRC OLE DB returned error code 0xC02020C4. The component returned a failure code when the pipeline engine called PrimeOutput(). The meaning of the failure code is defined by the component, but the error is fatal and the pipeline stopped executing. There may be error messages posted before this with more information about the failure.
Code: 0xC0209029 Description: SSIS Error Code DTS_E_INDUCEDTRANSFORMFAILUREONERROR. The "OLE_DEST F_BR_Summary.Inputs[OLE DB Destination Input]" failed because error code 0xC020907B occurred, and the error row disposition on "OLE_DEST Summary.Inputs[OLE DB Destination Input]" specifies failure on error. An error occurred on the specified object of the specified component. There may be error messages posted before this with more information about the failure.
Code: 0xC0047022 Source: DFT Populate Summary SSIS.Pipeline Description: SSIS Error Code DTS_E_PROCESSINPUTFAILED. The ProcessInput method on component "Summary" (821) failed with error code 0xC0209029 while processing input "OLE DB Destination Input" (834). The identified component returned an error from the ProcessInput method. The error is specific to the component, but the error is fatal and will cause the Data Flow task to stop running. There may be error messages posted before this with more information about the failure. End Error Error: 2016-07-19 20:40:18.97
Code: 0x80004005 Source: DFT Populate F_BR_Detail SSIS.Pipeline Description: Unspecified error End Error Error: 2016-07-19 20:40:19.41
Code: 0xC02020C4 Source: DFT Populate Detail SRC OLE DB [608] Description: The attempt to add a row to the Data Flow task buffer failed with error code 0xC0047020. End Error Error: 2016-07-19 20:40:19.67 Code: 0xC0047038 Source: DFT Populate F_BR_Detail SSIS.Pipeline Description: SSIS Error Code DTS_E_PRIMEOUTPUTFAILED. The PrimeOutput method on SRC OLE DB returned error code 0xC02020C4. The component returned a failure code when ... The package execution fa... The step failed.
This error is caused by not having enough memory available. Solution is reduce the SQL Server engine max memory by running the sp_configure stored procedure.
The following example sets the max server memory option to 4 GB:
sp_configure 'show advanced options', 1;
GO
RECONFIGURE;
GO
sp_configure 'max server memory', 4096;
GO
RECONFIGURE;
GO
For more information: https://msdn.microsoft.com/en-us/library/ms178067.aspx

SSIS package runs successfully when i start my SQL server Agent job manually but it went to failure on scheduled time

That time it throws an error message as below:
Message
Executed as user: DOMAIN1\administrator. ...50.1600.1 for 32-bit Copyright (C) Microsoft Corporation 2010.
All rights reserved. Started: 1:00:01 AM Error: 2015-05-28 01:40:00.82 Code: 0xC0202009
Source: DFT RequestFactStg OLE DB Command [3499] Description: SSIS Error Code DTS_E_OLEDBERROR.
An OLE DB error has occurred. Error code: 0x80040E14. An OLE DB record is available. Source:
"Microsoft SQL Server Native Client 10.0" Hresult: 0x80040E14 Description: "Uncommittable transaction is detected
at the end of the batch. The transaction is rolled back.". An OLE DB record is available. Source:
"Microsoft SQL Server Native Client 10.0" Hresult: 0x80040E14 Description: "The current transaction cannot
be committed and cannot support operations that write to the log file. Roll back the transaction.". End Error
Error: 2015-05-28 01:40:00.82 Code: 0xC0202009 Source: DFT RequestFactStg OLE DB Destination [3509]
Description: SSIS Error Code DTS_E_OLEDBERROR. An OLE DB error has occurred. Error code: 0x80004005. An OLE
DB record is available. Source: "Microsoft SQL Server Native Client 10.0" Hresult: 0x80004005 Description:
"The INSERT statement conflicted with the FOREIGN KEY constraint "FK_DateDimStg_TO_RequestFactStg". The
conflict occurred in database "ENIStageUAT", table "dbo.DateDimStg", column 'DateKey'.". End Error Error:
2015-05-28 01:40:00.87 Code: 0xC0209029 Source: DFT RequestFactStg OLE DB Destination [3509]
Description: SSIS Error Code DTS_E_INDUCEDTRANSFORMFAILUREONERROR. The "input "OLE DB Destination Input"
(3522)" failed because error code 0xC020907B occurred, and the error row disposition on "input "OLE DB
Destination Input" (3522)" specifies failure on error. An error occurred on the specified object of the
specified component. There may be error messages posted before this with more information about the failure.
End Error Error: 2015-05-28 01:40:00.87 Code: 0xC0209029 Source: DFT RequestFactStg OLE DB Command
[3499] Description: SSIS Error Code DTS_E_INDUCEDTRANSFORMFAILUREONERROR. The "input "OLE DB Command
Input" (3504)" failed because error code 0xC020906E occurred, and the error row disposition on "input "OLE
DB Command Input" (3504)" specifies failure on error. An error occurred on the specified object of the
specified component. There may be error messages posted before this with more information about the
failure. End Error Error: 2015-05-28 01:40:00.91 Code: 0xC0047022 Source: DFT RequestFactStg SSIS.
Pipeline Description: SSIS Error Code DTS_E_PROCESSINPUTFAILED. The ProcessInput method on component
"OLE DB Destination" (3509) failed with error code 0xC0209029 while processing input "OLE DB Destination
Input" (3522). The identified component returned an error from the ProcessInput method. The error is specific
to the component, but the error is fatal and will cause the Data Flow task to stop running. There
may be error messages posted before this with more information about the failure. End Error Error:
2015-05-28 01:40:00.93 Code: 0xC0047022 Source: DFT RequestFactStg SSIS.Pipeline Description:
SSIS Error Code DTS_E_PROCESSINPUTFAILED. The ProcessInput method on component "OLE DB Command" (3499)
failed with error code 0xC0209029 while processing input "OLE DB Command Input" (3504). The identified
component returned an error from the ProcessInput method. The error is specific to the component, but the
error is fatal and will cause the Data Flow task to stop running. There may be error messages posted
before this with more information about the failure. End Error Error: 2015-05-28 01:40:01.86 Code:
0xC02020C4 Source: DFT RequestFactStg OLE DB Source [1617] Description: The attempt to add a row
to the Data Flow task buffer failed with error code 0xC0047020. End Error Error: 2015-05-28 01:40:01.90
Code: 0xC0047038 Source: DFT Reque... The package execution fa... The step failed.
So any pls help me on this...
Most often issue with users. In editing the step in job, you must select proxy user in Run as field.
Create proxy user with sufficient rights to run the SSIS package (for example with sysadmin rights)
EDIT: tutorial for creating agent proxy
EDIT 2: Well, my bad. Didn't see the conflict error in that chaos. Maybe you are just trying to insert bad data now.
"The INSERT statement conflicted with the FOREIGN KEY constraint "FK_DateDimStg_TO_RequestFactStg". The
conflict occurred in database "
This could potentially be an issue.
Make sure you'r using the same databases.

SSIS 2008 R2 Completes But Gives Error "Violation of PRIMARY KEY constraint"

I've found a number of links on this but so far not the one with an answer.
Issue: SSIS 2008 R2 package importing data from many tables (about 160) of SQL 2000 database (conversion project) into new structures. This has run cleanly a number of times. To be clear, the data appears to be actually importing correctly. However, the last 3 times, I've run into this error. The Task (54) Source (269) shown in the message below is the last step and item in the step.
I've confirmed that the table mention as a "duplicate key violation" actually has the same 49166 records as the source. It seems like it almost just has issue with threading or the time that has elapsed (long running query). I have TRUNCATED the 5 tables in the final step and re-run that single Task 54 and it completes just fine, all green.
It would be good to not have to go hunting through the process due to red herrings. Any idea on how to address this so it reports cleanly?
Thanks
Error message below:
Executed as user: [masked]. Microsoft (R) SQL Server Execute Package Utility Version 10.50.6000.34 for 64-bit Copyright (C) Microsoft Corporation 2010. All rights reserved. Started: 4:36:46 PM Error: 2015-05-20 16:36:48.49 Code: 0xC0016016 Source: Description: Failed to decrypt protected XML node "DTS:Password" with error 0x8009000B "Key not valid for use in specified state.". You may not be authorized to access this information. This error occurs when there is a cryptographic error. Verify that the correct key is available. End Error Error: 2015-05-20 16:36:48.50 Code: 0xC0016016 Source: Description: Failed to decrypt protected XML node "DTS:Password" with error 0x8009000B "Key not valid for use in specified state.". You may not be authorized to access this information. This error occurs when there is a cryptographic error. Verify that the correct key is available. End Error Error: 2015-05-20 17:19:33.78 Code: 0xC0202009 Source: Data Flow Task 54 Destination 269 - cadNotificationStatus [396] Description: SSIS Error Code DTS_E_OLEDBERROR. An OLE DB error has occurred. Error code: 0x80004005. An OLE DB record is available. Source: "Microsoft SQL Server Native Client 10.0" Hresult: 0x80004005 Description: "The statement has been terminated.". An OLE DB record is available. Source: "Microsoft SQL Server Native Client 10.0" Hresult: 0x80004005 Description: "Violation of PRIMARY KEY constraint 'PK__cadNotif__25843E4F1B0907CE'. Cannot insert duplicate key in object 'dbo.cadNotificationStatus'. The duplicate key value is (4, 2).". End Error Error: 2015-05-20 17:19:33.79 Code: 0xC0209029 Source: Data Flow Task 54 Destination 269 - cadNotificationStatus [396] Description: SSIS Error Code DTS_E_INDUCEDTRANSFORMFAILUREONERROR. The "input "OLE DB Destination Input" (409)" failed because error code 0xC020907B occurred, and the error row disposition on "input "OLE DB Destination Input" (409)" specifies failure on error. An error occurred on the specified object of the specified component. There may be error messages posted before this with more information about the failure. End Error Error: 2015-05-20 17:19:33.79 Code: 0xC0047022 Source: Data Flow Task 54 SSIS.Pipeline Description: SSIS Error Code DTS_E_PROCESSINPUTFAILED. The ProcessInput method on component "Destination 269 - cadNotificationStatus" (396) failed with error code 0xC0209029 while processing input "OLE DB Destination Input" (409). The identified component returned an error from the ProcessInput method. The error is specific to the component, but the error is fatal and will cause the Data Flow task to stop running. There may be error messages posted before this with more information about the failure. End Error Error: 2015-05-20 17:19:33.81 Code: 0xC02020C4 Source: Data Flow Task 54 Source 269 - cadNotificationStatus [218] Description: The attempt to add a row to the Data Flow task buffer failed with error code 0xC0047020. End Error Error: 2015-05-20 17:19:33.81 Code: 0xC0047038 Source: Data Flow Task 54 SSIS.Pipeline Description: SSIS Error Code DTS_E_PRIMEOUTPUTFAILED. The PrimeOutput method on component "Source 269 - cadNotificationStatus" (218) returned error code 0xC02020C4. The component returned a failure code when the pipeline engine called PrimeOutput(). The meaning of the failure code is defined by the component, but the error is fatal and the pipeline stopped executing. There may be error messages posted before this with more information about the failure. End Error DTExec: The package execution returned DTSER_FAILURE (1). Started: 4:36:46 PM Finished: 5:19:34 PM Elapsed: 2568.01 seconds. The package execution failed. The step failed.
As per the error, it is failing while to inserting duplicate records in 'dbo.cadNotificationStatus' table. Is there any logic to import only new records from source table which is in SQL 2000, which are not present in destination table? If no, data flow task importing data into this table is going to fail.
It looks like, it is running successfully because you are truncating the table and then importing the whole data again. If you can provide more info regarding source query, it will be more clear.
Just to be 100% sure I would redirect rows to a flat file that are causing you the issue. This will allow you to see what line item is causing the failure or triggering the error message. This link shows a good tutorial of how to do that:
http://www.techbrothersit.com/2013/07/ssis-how-to-redirect-invalid-rows-from.html
Also it looks like you have more going on in your error output from a security perspective based on the below:
Description: Failed to decrypt protected XML node "DTS:Password" with error 0x8009000B "Key not valid for use in specified state.". You may not be authorized to access this information.
My guess is that someone else created this package and used: "EncryptSensitiveWithUserKey"...
I would recommend:
While importing the package to SQL Server choose Protection Level: Either
1- Don't save sensitive data.
Or
2- Rely on Server Storage and roles for access control.

SQL job fails to execute package, but VS not

When I run package directly from Visual Studio then it doesn't throw error. When I execute the package via SQL Server Agent Job then it throws error:
Source: Fact_Invoice_Item PWBConsolidation [1] Description: SSIS
Error Code DTS_E_OLEDBERROR. An OLE DB error has occurred. Error
code: 0x80004005. An OLE DB record is available. Source: "Microsoft
SQL Server Native Client 10.0" Hresult: 0x00040EDA Description:
"Warning: Null value is eliminated by an aggregate or other SET
operation.". End Error Error: 2012-02-14 07:17:49.01 Code:
0xC0047038 Source: Fact_Invoice_Item SSIS.Pipeline
Description: SSIS Error Code DTS_E_PRIMEOUTPUTFAILED. The PrimeOutput
method on component "PWBConsolidation" (1) returned error code
0xC0202009. The component returned a failure code when the pipeline
engine called PrimeOutput(). The meaning of the failure code is
defined by the component, but the error is fatal and the pipeline
stopped executing. There may be error messages posted before this
with more information about the failure. End Error DTExec: The
package execution returned DTSER_FAILURE (1). Started: 7:14:35 AM
Finished: 7:17:50 AM Elapsed: 195.094 seconds. The package
execution failed. NOTE: The step was retried the requested number of
times (3) without succeeding. The step failed.
I have isolated a part that is causing the problem. In Ole Db Source I use query to get the data and when I remove the following line then the package is executed successfully via Sql Server Agent Job:
Checksum = Checksum(S.BrokerID, S.TeamID, II.DatabaseName, II.INVOICE, I.Invoice_Date, P.Expiration_Date, P.Effective_Date,
P.Binder_Effective, I.AGENCY, II.MARKET, P.POLICY, CT.LINE_OF_BUSINESS, II.Coverage_Type, SUB.INSURED,
LOB.Description, CT.Description, I.Date_Due, I.Installment, P.Installments, P.ENDORSES, I.Inv_Tdate, Inv_Acctcur,
INS.INDUSTRY, SUB.SUBMISSION
)
EDIT:
I have just checked and looks like part of data gets through data flow and at some point it just throws error. I have manually set Checksum=2 and still doesn't work
Any idea why is this so?
Thanks a lot,
Ilija
This looks like you are running into a constraint issue on your table that you are inserting into or values you are operating on...like aggregate function(max,sum,avg..) exists on a null value.
Try redirecting error rows to a flat file or something of that nature and see if you still get the error.
Also try set ANSI_WARNINGS OFF in your SQL statement
Enjoy and good luck

Resources