SSIS Scenario
I have a SQL variable of type object. It contains all the connection to different servers/databases. I want to connect to those databases one by one and run a query.
Expected Exception Handling
If the SSIS Connection manager(Dynamic connection manger) is unable to find the connection to the server (probably the server is down) I want to Skip that connection (Database/server) and log that into the table and move onto the next Connection (Database/Server). The SSIS package should not crash.
My Implementation
I have Successfully Configured the SSIS package to Use Connection Manager (Dynamic Connection Manager) and Foreach loop to loop through the SSIS variable of type object. but I am not able to skip the Connection if the server/database is not found. it generates error that the server/databsse not found / problem with the connection and the SSIS package fails.
my experience in SSIS is one week old
Any help will be appreciated.
How about setting --> Force execution Result property of the task to success
I was looking for the same fix too. Seems the normal OnError Eventhandling does not work for issues that arise when connecting to a source DB.
There is another workaround I wanted to mention. You can handle the error in the Data Flow Task (OnError Eventhandler, set the System-Variable "Propagate" in that Eventhandler to false). I think this is still required, but not sure. I use it also to log the exception.
Afterwards you can set MaximumErrorCount in ForEachLoop to "0" (which means unlimited). I'm not exactly sure why it works, but trying to find a way to handle the scenario you described, I found this.
==
Just as an interesting observation: For debugging purposes I added an OnError Eventhandler to the ForEachLoop and set a breakpoint there in a dummy script. It was never reached. Nonetheless the ForEachLoop failed all the time until I set MaximumErrorCount to 0.
I dont think its possible to continue the package execution once it encounters an error. You need to control this behavior through a SQL Server table (or any other table for that matter).
Once the package fails, you can set a flag in the table saying that the package failed. The next time the package runs, you can start from this point on and continue the execution. But automatically skipping the down server is kinda pulling a rabbit out of a hat.
Another way you can do this is to ping each server before hand in a separate package and store the ping results in a table. Only pick those records (servers) whose ping results were positive. Otherwise just skip the server.
Related
I am dealing with my problem on some Windows Server 2019 (Core) with one running SQL Server 2019 CU4 instance each.
What we try to do
We are currently building a data warehouse with distributed databases. The individual layers of the DWH are located on one database server each. The data exchange between the layers/servers takes place via SSIS ETLs, which use Linked Servers to reach the other layers and drag and drop data. Each layer also has its own SSIS service instance and executes the corresponding SSIS packets.
The SSIS packages are called by SQL Server Agent jobs. We have a job that executes the SSIS packets (#1), which in turn calls another job (#2) as the last step, which after a short wait time executes the calling job (#1). Thus, controlled by schedules, a loop is created and data is continuously transferred with ETLs.
I hope this was not too much unnecessary background
The error
Basically the job is running and there are numerous successful executions. However, we are observing interruptions at job #1 without helpful information regarding the error. This means that the job history log refers to the SSIS log, which again only contains an "unexpected termination". In the SSIS log, we only see behavior that indicates that the ETL packet active at that time stopped after validation. Depending on the log level, nothing is logged at all, not even the execution of single packages of the project. The package where this error occurs is different and not limited to a specific one.
What I have already tried
Re-create the jobs and SSIS Enviroments by hand (scripted before)
Using the 32Bit Runtime
Upgrade the SSIS project/package version to
2019
Increase the log level to "verbose"
Patching the SQL Server to CU4
Save ssis dump files (couldn't find them or they weren't created)
Search Windows and SQL Server Logfiles
Does anyone have some suggestions or some ideas how to become more error specific informations?
Thank you very much and take care :)
UPDATE We have an error message (OLE DB 0xC0202009 and 0X80004005)!
In order to exclude the use of environments as a cause, I manually set the parameters in the SSIS job step instead of overwriting them by selecting an environment.
Long story short: Today it turns out that the parameter for an OLE DB Connection String is not passed correctly.
The following is specified as a parameter in the job step:
However, the following connection string is specified in the context of the error message:
Please note that some arguments are added twice to the parameter (red).
What could have caused that?
I am using SSDT 2017. There are 3 Dataflow tasks (They are connected using precedence constraints) which loads data from Excel into Database tables. When I run each task individually it is showing as success but when I run the entire package it is showing as completed with out any green tick on the tasks which means the tasks are not getting executed. After I changed the RetainSameConnection Property of Excel to True, all the tasks are getting completed successfully.
I have not seen this behavior in the earlier versions. Is this a new change in 2017 version or am I missing anything ?
Trying to figure out the issue
Based on the question and comments you are using 3 Data flow Task that contains 3 Excel source components that use the same connection manager.
At the start of the package execution in the Validation phase, each one of this components try must acquire the connection from the connection manager to read the metadata and try to keep this connection open until it must be executed and since there are many files that try to open the same connection it will cause a problem.
When using RetainSameConnection the package run the acquireconnection method once and use it multiple time, then the issue is solved.
Something to try
Try changing the Delay Validation property to True, and ValidateExternalMetadata property to False for each Excel Source on all Data Flow Tasks, it may solve the issue if the connection is only acquired for validation purposes. If it doesn't works then it means that the acquire connection method is called to lock the file for read even if the data flow task execution is not started yet.
I am trouble shooting an error in a package.
Update MYTABLE for MYCOLUMN (REF to task name):Error: Executing the query "..." failed with the following error: "Invalid column name 'MYCOLUMN'.". Possible failure reasons: Problems with the query, "ResultSet" property not set correctly, parameters not set correctly, or connection not established correctly.
I have verified the table and column exists, the length of the field is way excessive than what it needs that is 14 where it is declared as varchar(250).
I have verified the script works on the server in SSMS outside of the context of the package.
I have verified the connection and database in the package is as I expect.
Is there away to verify on the server. I did try to look at the Connection Managers tab on the package configuration itself i.e. in the Integration Services Catalogs->SSISDB->solutionfolder->..->package.dtsx->Configure context menu but it is empty.
Any ideas on how to troubleshoot?
Just to add more context the package contains 27 other tasks, 9 tasks in a row linked to this task but all set to on completion, all seem to be doing stuff independent of the other. 1 task is a loop doing stuff and the rest are single independent tasks. So I don't know at this stage if it is a cascading connection issue perhaps however; I am just reading what the log says.
I kicked off the package at 9:54am, the timestamp on the error log says 11:45am so nearly 2 hours into running is this log reported.
I would suggest the below things to troubleshoot the issue.
I would suggest you to just have this task and disable all other
tasks to troubleshoot the issue. So that you can focus on this issue
specifically. That will tell you whether connection is working fine
without issues.
I would suggest you to edit the task and see whether parameters are
set properly. Different providers have different way of setting
parameters. Again check whether parameters are proper. Execute SQL
Task
one more thing, may be you are pointing the package to different
connection than the one you used for SSMS. So, it is working in SSMS
and in the connection being used in the package is not having schema
changes yet done.
I finally figure it out before I read the previous offered suggestion so will give some credit if I can! FYI: We have a lot of dev servers. I clicked on the overview hyperlink in the All Execution logs and it said another server. Also I found the connection on the job calling the package not the package itself so I have learnt something today. Anyhow the job said one server but the overview said another so I again I was back to square one scratching my head.
Then I decided to open the connection manager on the job and select the field and make no change rather then cancelling I clicked ok not thinking about it and noticed the field changed to bold face. So I am assuming if you make a manual change on the server in SSMS to anything it shows up in bold which is kind of useful. So I can only assume this is a MS SSMS or SSIS or VS deployment bug. That it does not overwrite, the previous connection although the SSMS interface says otherwise. Perhaps somebody can share some light. Having not checked the server before I made a change and deployed it I have no idea if the previous settings were changed manually by someone or the connection in the package was changed and deployed. Anyhow checking the job history shows it had been failing for awhile so it wasn't me so whoever and whenever a change was done by a previous developer didn't figure it our either or didn't bother or did not know how, or didn't observe it. Anyhow it is pointing to the correct server now!!!
I need to get data from several databases using SQL Server Integration Services (more than 20) and for each of them execute the package that transfers the data to a target database, there is a table with a list of connection strings of these databases in other database.
I try to execute the ExecutePackage task inside foreach task and send connection string from query as package's parameter, also I send connection string of target database and date. But package stops with error:
[Importing [11]] Error: SSIS error code: DTS_E_CANNOTACQUIRECONNECTIONFROMCONNECTIONMANAGER. The call to the AcquireConnection method of the connection manager "targetdb" failed with error code 0xC0202009. You may have previously published error messages that contain more detailed information about the reason for the failure of the AcquireConnection method.
I think the connection manager of targetdb does not pass validation, in this case I run package directly from Visual Studio. When I run package separately (not inside foreach and with default parameters), then it works right.
Is it right way for my task?
Decision was simple. I set DelayValidation property to true for package, which executed inside foreach loop, it decided problem.
When I run my SSIS package in Visual Studio it executes just fine. However when I try to execute this same package using the SQL server agent, by either CmdExec or SSIS package type, it gets canceled. The package is stored on the IS catalog. All on the same server.
The standard report tells me task x failed, but is not showing any error messages. I'm sure there is nothing wrong with the task itself as after an DB restore and rerun of the same package with the same data, a different task fails.
I have tried this several times and it always fails because of 1 (out of possible 3) task which are more or less close to each other in the control flow. Moreover the task that fails is only calling a stored procedure, which is completely encapsulated with try-catch.
The standard report is however throwing the following message at the time the package failed:
< ?xml version"1.0"?> < DTS:ProcessMemoryUsage xmlns: DTS=www.microsoft.com/sqlserver/DTS">
< DTS:Peakworkingsetsize>320692224< /DTS.................
which makes me think it has something to do with memory. However can't find anything usefull on google about this message.
I already trying things for a week now but not able to find the solution. Please help?
It might something going on in your tasks. I have noticed the same behavior while running SSAS cube. If it fails, gives not specific error. Try to disable all tasks and enable them one by one in order to find which contains the problem.