I have the following scenario in SSIS. There are two packages, Outer.dtsx and Inner.dtsx. the Inner package is called inside the Outer package in the workflow. To increase the performance, as the workload is heavy, I added a sequence container, and instead of having only one Inner package running, I managed several packages inside the container, so to have multiple instances (10 to be exact) of Inner package running in parallel. It is only one version of Inner package, however it is called several times.
How does this scenario differ from having multiple versions of Inner (Inner_1, Inner_2, ...... , Inner_10) and run them again inside the sequence container? Does having multiple packages with same definition improves the performance, compared to one version of the package, called several times ? Which scenario is more efficient and has best performance ?
From the definition of SSIS package - it is a definition of tasks and transformations written in XML format and being executed by SSIS engine. SSIS engine can execute several instances of the same package simultaneously.
On practice -- performed the following experiment. Created a dummy package loading from CSV file to MSSQL DB table with parameters of file name and table name - InnerPkg. Then created a copy of it - InnerPkg1. Also created two copies of the source file and the destination SQL Table.
Please note!!! I created different source and destinations to avoid resource locking.
OuterPkg_Parallel calls two instances of InnerPkg, passing different parameters of osurce filename and destination tablename at Execute Package Task.
OuterPkg_Copies calls InnerPkg and InnerPkg1 with appropriate parameters.
Results (average of 5 runs):
OuterPkg_Parallel - 12,72 seconds
OuterPkg_Copies - 12,77 seconds
So, the difference is negligible, to my understanding.
The tests were conducted on MS SQL - SSIS version 2016, OS - Windows Server 2016.
Bottom Line - use single package calling, as it has no visible performance penalty and greatly simplifies support.
We are moving data from Oracle to SQL Server via SSIS. The first steps are filling some (about 8 right now) Oracle tables with selections from the existing Oracle application tables.
The 8 Oracle tables should be filled sequentially, so one after the other, but the filling must be able to be initiated from SSIS, one by one (it's a lot of data and things go wrong!) or all in 1 go (again, one after the other). So we made 9 SSIS packages for this, 8 for filling oracle tables and 1 that calls the 8 packages sequentially. All 8 packages call Oracle procedures, that are contained in 1 Oracle package, to fill the tables.
For 1 of the 8 Oracle tables we made (oracle) parallel jobs to fill the table (for performance sake, via the Oracle job scheduler) . Drawback of Oracle job scheduler parallel execution is that the 'mother' job is finished as soon as all the child jobs are submitted (not completed). So the status wether all subjobs are finished, is put in an Oracle table/column/field (eg FINISHED = 1) through another Oracle procedure. For the next SSIS package to run, the table/column/field has to be a certain value (eg FINISHED = 1), before the next SSIS package can run (to fill the next Oracle table).
I have searched quite a bit, but cannot find a solution on how to do this in SSIS from A to Z. Some say via script task but do not provide screenshots or an example. I am absolutely no expert in SSIS, I just know some basic stuff.
Thank you in advance!
PS: I am looking for a loop-constuct because we don't know how long it wil take for the Oracle scheduler jobs to complete (we test with 1000 rows, up to 20 million)
We have an SSIS package that is apparently termed as 'slow' by the development team. Since they do not have a person with SSIS ETL, as a DBA I tried digging into it. Below is the information I found:
SQL Server was 2014 version upgraded -inplace to 2017 so it has SSIS of both versions.
They load a SQL Server table of size 200 GB into SSIS and then zip the data into flatfile using command line zip functionality.
The data flow task simple hits a select * from view - the view is nothing but containing the table with no other fancy joins.
While troubleshooting I found that on SQL Server, there is hardly any load coming, possibly because the select command is running in single thread and not utilizing SQL server cores.
When I run the same select * command (only for 5 seconds, since it is 200 GB table), even my command is single threaded.
The package has a configuration file that the SQL job shows (this is how the package runs) with some connection settings.
Opening the package in BIDS show defaultBufferMaxRows as 10000 only (possibly default value) (since configuration file or any variables does not has a customer value, I guess this is what the package is using too).
Both SQL and SSIS are on same server. SQL has been allocated max memory leaving around 100 GB for SSIS and OS.
Kindly share any ideas on how can I force the SQL Server to run this select command using multiple threads so that entire table gets inside SSIS buffer pool faster.
Edit: I am aware that bcp can read data faster than any process and save it to flatfile but at this point changes to the SSIS package has to be kept minimum and exploring options that can be incorporated within SSIS package.
Edit2: Parallelism works perfectly for my SQL Server as I verified for a lot of other queries.The table in question is 200 GB. It is something with SSIS only which is not hammering my DB as hard as it should.
Edit3: I have made some progress, adjusted the buffer value to 100 MB and max rows to 100000 and now the package seem to be doing better. when I run this package on the server directly using dtexec utility, it generates good load of 40- 50 MB per second but through SQL job it never generates lod more than 10 MB. so I am trying to figure out this behavior.
Edit4: I found that when I run the package directly from logging to the server and invoking dtexec utility, it runs good because it generates good load on the DB causing data I\O to remain steady between 30-50 MB\sec.
The same thing from SQL job never exceeds the I\O more than 10 MB\sec.
I even tried to run the package using agent and opting for cmdline operation but no changes. Agent literally sucks here, any pointers on what could be wrong here?
Final Try:
I am stumped at the observation I have finally:
1)Same package runs 3x faster when run from command prompt from windows node by invoking dtexc utility
2) Exact same package runs 3 times slower than above when involked by SQL agent which has sysadmin permissions on windows as well as SQL Server
In both cases, I tried to see the version of DTEXEC they invoke, and they both invoke the same version. So why one would be so slow is out of my understanding.
I don't think that there is a general solution to this issue since it is a particular case that you didn't provide much information. Since there are two components in your data flow task (OLE DB Source and Flat File Destination), I will try to give some suggestions related to each component.
Before giving suggestions for each component, it is good to mention the following:
If no transformations are applied within the data flow task, It is not recommended to use this task. It is preferable to use bcp utility
Check the TempDb and the database log size.
If a clustered index exists, try to rebuild it. If not, try to create a clustered index.
To check the component that is slowing the package execution, open the package in Visual Studio and try to remove the flat file destination and replace it with a dummy Script Component (write any useless code, for example: string s = "";). And then run the package; if it is fast enough, then the problem is caused by the Flat File Destination, else you need to troubleshoot the OLE DB Source.
Try executing the query in the SQL Server management studio and shows the execution plan.
Check the package TargetServerVersion property within the package configuration and make sure it is correct.
OLE DB Source
As you mentioned, you are using a Select * from view query where data is stored in a table that contains a considerable amount of data. The SQL Server query optimizer may find that reading data using Table Scan is more efficient than reading from indexes, especially if your table does not have a clustered index (row store or column store).
There are many things you may try to improve data load:
Try replacing the Select * from view with the original query used to create the view.
Try changing the data provider used in the OLE DB Connection Manager: SQL Server Native Client, Microsoft OLE DB provider for SQL Server (not the old one).
Try increasing the DefaultBufferMaxRows and DefaultBufferSize properties. more info
Try replacing using SQL Command with specific column names instead of selecting the view name (Table of View data access mode). more info
Try to load data in chunks
Flat File Destination
Check that the flat file directory is not located on the same drive where SQL Server instance is installed
Check that the flat file is not located on a busy drive
Try to export data into multiple flat files instead of one huge file (split data into smaller files) , since when the exported data size increase in a single file, writing to this file become slower, then the package will become slower. (Check the 5th suggestion above)
Any indexes on the table could slow loading. If there are any indexes, try dropping them before the load and then recreating them after. This would also update the index statistics, which would be skewed by the bulk insert.
Are you seeing SQL server utilizing other cores too for other queries? If not, maybe someone played with the following settings:
Check these under server configuration setting:
Maximum Degree of Parallelism
Cost Threshold for Parallelism (server configuration setting).
Does processors affinitized to a CPU.
Also, MaxDOP query hint can cause this too but you said there is no fancy stuff in the view.
Also, it seems you have enough memory on error, why not increase defaultBufferMaxRows to an extremely large number so that SQL server doesn't get slowed down waiting for the buffer to get empty. Remember, they are using the same disk and they will have to wait for each other to use the disk, which will cause extra wait times for the both. It's better SQL server uses it, put into the buffer, and then SSIS starts processing and writing it into disk.
DefaultBufferSize : default is 10MB, max possible 2^31-1 bytes
DefaultBufferMaxRows : default is 10000
you can set AutoAdjustBufferSize so that DefaultBufferSize is automatically calculated based on DefaultBufferMaxRows
See other performance troubleshooting ideas here
https://learn.microsoft.com/en-us/sql/integration-services/data-flow/data-flow-performance-features?view=sql-server-ver15
Edit 1: Some other properties you can check out. These are explained in the above link as well
MaxConcurrentExecutables (package property): This defines how many threads a package can use.
EngineThreads (Data Flow property): how many threads the data flow engine can use
Also try running dtsexec under the same proxy user used by SQL agent to see if you get different result with this account versus your account. You can use runas /user:... cmd to open a command window under that user and then execute dtexec.
Try changing the proxy user used in SQL Agent to a new one and see if it will help. Or try giving elevated permissions in the directories it needs access to.
Try keeping the package in file-system and execute through dtexec from the SQL Agent directly instead of using catalog.start_execution.
Not your case but for other readers: if you have "Execute Package Task", make sure the child packages to be executed are set to run in-process via ExecuteOutOfProcess property. This just reduces overhead of using more processes.
Not your case but for other readers: if you're testing in BIDS, it will run in debug mode by default and thus run slow. Use CTRL-F5 (start without debugging). The best is to use dtexec directly to test the performance
A data flow task may not be the best choice to move this data. SSIS Data Flow tasks are an ETL tool where you can do transformations, look ups, redirect invalid rows, add derived columns and a lot more. If the data flow task is simple and only moves data with no manipulation or redirection of rows then ditch the Data Flow task and use a simple Execute SQL Task and OPENROWSET to import the flat file that was generated from command line and zipped up. Assuming the flat file is a .csv file here are some working examples to query a .csv and insert the data to a table.
You need [Ad Hoc Distributed Queries] run_value set to 1
into dbo.Destination
SELECT *
from openrowset('MSDASQL', 'Driver={Microsoft Text Driver (*.txt; *.csv)};
DefaultDir=D:\YourCsv.csv;Extensions=csv;','select * from YourCsv.csv') File;
Here is some additional examples https://sqlpowershell.blog/2015/02/09/t-sql-read-csv-files-using-openrowset/
There are suggestions in this MSDN article: MSDN DataFlow performance features
Key ones appear to be:
Check the EngineThreads property of the DataFlow task, which tells SSIS how may source and worker threads it should use
If using OLE DB Source to select data from a view uses "SQL Command" and write a SELECT * From View rather than Table or View
Let us know how you get on
You may be facing I/O bottleneck while writing the 200GB to the flat file. I don't see any problem with SQL Query.
If possible create multiple files and split the data (either by modifying SSIS or changing the select query)
I have a table which is getting modified by SSIS package, this ssis package is running 20+ packages in parallel and all of the packages are inserting values in same table via stored procedure and this table should have distinct records. Since all packages are running in parallel few records are getting duplicate values.
My question if I set lock on this table, and 2 process/packages try to insert into table at the same time which will get 1st priority and if table 1 get the priority 1st then would I got a error message for table 2 or it will wait till table 1 release the lock.
Implementing lock on table would have performance impact (My initial thought process, I am ready to get challenged on it).
Can anyone suggest a solution for the problem of getting duplicate records from multiple processes.
Thanks
Suppose that your duplication issue is really caused by not locking the table.
Here are a few idea that might help:
Change TransactionOption for all the 20+ tasks ( assuming you are using 20+ 'Execute Package Task' in one package to run other 20+ packages) from 'Supported' to 'Required';
Instead of parallel running all tasks, put them in sequence. I would suggest to run them in sequence any way to derterminate the real cause of the duplicate.
Edit:
- If I were you, I will to try to load all 20+ package to 20+ seperated Staging tables, and when all packages are finished, then load all staging tables to your destination table in sequence and it should be very easy to verify duplication now since all staging tables are local.
I'm working with an SSIS package that itself calls multiple SSIS packages and hangs periodically during execution.
This is a once-a-day package that runs every evening and collects new and changed records from our census databases and migrates them into the staging tables of our data warehouse. Each dimension has its own package that we call through this package.
So, the package looks like
Get current change version
Load last change version
Identify changed values
a-z - Move changed records to staging tables (Separate packages)
Save change version for future use
All of those are execute SQL tasks except for the moving records tasks which are twenty some execute package tasks (data move tasks), which are executed somewhat in parallel. (Max four at a time.)
The strange part is that it almost always fails when executed by the SQL agent (using a proxy user) or dtexec, but never fails when I run the package through Visual Studio. I've added logging so that I can see where it stops, but it's inconsistent.
We didn't see any of this while working in our development / training environments, but the volume of data is considerably smaller. I wonder if we're just doing too much at once.
I may - to test - execute the tasks serially through the SQL Server agent to see if it's a problem with a package calling a package , but I'd rather not do this because we have a relatively short time in the evening to do this for seven database servers.
I'm slightly new to SSIS, so any advice would be appreciated.
Justin