I have about 7 projects deployed on a SQL Server. Each one contains a MasterPackage which run all the child packages of that project. The issue is that I want all 7 projects to run in parallel, starting at the same time, but as it is right now, they get queued up and start one after another. Can I make all the projects start at the same time?
You can always schedule packages' executions by the means of SQL Server Agent jobs. You will probably have to create a separate job for each project, but after that, whatever schedule you pick for them should be followed.
Just keep in mind that, if packages push a lot of data through, server might not cope with the total workload, so parallel execution might be slower than a serialised one.
Related
I have googled and read many questions/answers, but only one question has ever sounded exactly the same and it did not have an answer.
The situation:
My group has several SQL Servers that are running SQL Server 2017. They are configured virtually identically.
These servers are build boxes, meaning they pull data from a data ware house, or an extract file, run some ETL processing and then push to a prod box. SSIS packages are deployed on the box where the DB resides.
Just over a month ago (with no updates having occurred), one of these servers started having an issue where all the jobs that ran an SSIS package would "hang" on the step that ran the package. Any other step runs fine. But a job step that runs a package (all jobs do this), will not even start the package. The package shows no indication in the executions that anything has even tried to start it.
If the user executes the deployed package it will run successfully.
The only thing that will "fix" the issue is restarting the agent service.
I created a simple job to run a simple package every 5 mins. It had been running for about a week, the last time it ran was 4/11/2021 at 2:40am, the 2:45 run hung. I could find nothing in the event logs that occurred at that time. The server was rebooted as a normal scheduled process at 3:15 and was online by 3:25 because that is the next time it tried to run and it again just hung. So even a server reboot did not fix the issue.
I am at my wits end, since there is no error (the job hangs and the package does not even start) there is no logging that I can find that is showing any issues, I am at a loss as to what might cause this.
Thanks in advance.
Take a look at the SSISDB catalog database on each/all the servers involved. Has it grown exponentially and needs the history etc. cleared down or settings changed? How big are the transaction logs for those databases etc.?
So we are working on adding SQL DACPACs as part of our continues integration. We do this with powershell script running "sqlpackage.exe" with each DacPac file. we have about 12 DBs so it's about 12 DacPacs.
Everytime we run "sqlpackage.exe" to publish or script the DacPac we notice that it will take between 0.5-1+ min to complete initialization. Most of this time is being taken during initialization flow when "sqlpackage.exe" begins.
i'm trying to find a way to reduce this if possible since we have 12 DBs we are talking about at least 12 min for DB Deployment which is too much for us.
Do you know of any way to be able to reduce this?
The time up front is, I believe, used to get the current schema from the database, compare it to what's in the dacpac, and come up with the change scripts required.
There is no way to eliminate that using a dacpac.
However, you could do it ahead of time by having SqlPackage create the upgrade scripts and then at deploy-time just run the scripts. This could reduce downtime to just the time it takes to run the script, and, as suggested, if run in parallel the down time could be dramatically reduced.
If all your DBs are guaranteed to be in the same (schema) state you could just create a single upgrade script off the first DB and run it on all DBs.
This might be too late but potentially useful for others - we found that opening up all ports in the firewall on the SQL server solved our problem of deployment taking a long time. Went from about 4-5 minutes to about 1 minute.
I have an SSIS package that seems to be having concurrency issues.
I have set it up in a way that each of the 6 containers can be run all at the same time, or individually. They all run a SQL stored procedure and store data into a table object.
The first task is a SQL task that gets a list of clients from the database. The others are all foreach loops (for each client). When I run the package for ALL containers, it seems to fail after 1 loop of the 2nd and 3rd container. There si nothing in the output/debug, other than "Package has started". The first loop completes quite quickly for every client (< 10 seconds), whereas the others take about 2-3 minutes to run for each client (a lot more data).
If I run the package for a single for each loop, it completes without issue (it iterates 7 times). It only fails after 1 loop, if the other containers are also running. The first task that gets the client IDs stores them into a table, however there are 2 variables that the table data is stored in for each loop (Client ID and Client Name). My thinking is that once the first container is done (i.e. finished all 7 clients), the values in the variables have changed and the other loops fail.
http://i.imgur.com/AJwrLNF.png
I cannot read the SSIS tasks. The reason one does things in parallel in SSIS is to gain performance by using the built in parallelizing features of both SSIS and SQL Server. I once had to process millions of rows and by doing it in parralel I got it done in the window available to me. If you are processing 7 clients (and the tasks seem to be very different) you can be sure all kinds of locking (and probably deadlocks) are taking place. Just do it sequentially.
I work with an environment that uses Merge Replication to publish a dozen publications to 6 a dozen subscribers every 10 minutes. When certain jobs are running simultaneously, deadlocks and blocking is encountered and the replication process is not efficient.
I want to create a SQL Server Agent Job that runs a group of Merge Replication Jobs in a particular order waiting for one to finish before the next starts.
I created an SSIS package that started the jobs in sequence, but it uses sp_start_job and when run it immediately starts all the jobs so they are running together again.
A side purpose is to be able to disable replication to a particular server instead of individually disabling a dozen jobs or temporarily disabling replication completely to avoid 70+ individual disablings.
Right now, if I disable a Merge Replication job, the SSIS package will still start and run it anyway.
I have now tried creating an SSIS package for each Replication Job and then creating a SQL Server Agent job that calls these packages in sequence. That job takes 8 seconds to finish while the individual packages it is calling (starting a replication job) takes at least a minute to finish. In other words, that doesn't work either.
The SQL Server Agent knows when a Replication job finishes! Why doesn't an SSIS package or job step know? What is the point of having a control flow if it doesn't work?
Inserting waits is useless. the individual jobs can take anywhere from 1 second to an hour depending on what needs replicating.
May be I didn't see real problem but it is naturally that you need synchronization point and there are many ways to create it.
For example you could still run jobs simultaneously but let first job lock a resource that is needed for second, that will wait till resource will be unlocked. Or second job can listen log table in loop (with wait for a "minute" and self cancel after "an hour")...
I'm working with an SSIS package that itself calls multiple SSIS packages and hangs periodically during execution.
This is a once-a-day package that runs every evening and collects new and changed records from our census databases and migrates them into the staging tables of our data warehouse. Each dimension has its own package that we call through this package.
So, the package looks like
Get current change version
Load last change version
Identify changed values
a-z - Move changed records to staging tables (Separate packages)
Save change version for future use
All of those are execute SQL tasks except for the moving records tasks which are twenty some execute package tasks (data move tasks), which are executed somewhat in parallel. (Max four at a time.)
The strange part is that it almost always fails when executed by the SQL agent (using a proxy user) or dtexec, but never fails when I run the package through Visual Studio. I've added logging so that I can see where it stops, but it's inconsistent.
We didn't see any of this while working in our development / training environments, but the volume of data is considerably smaller. I wonder if we're just doing too much at once.
I may - to test - execute the tasks serially through the SQL Server agent to see if it's a problem with a package calling a package , but I'd rather not do this because we have a relatively short time in the evening to do this for seven database servers.
I'm slightly new to SSIS, so any advice would be appreciated.
Justin