There is an SQL Agent Job containing a complex Integration Services Package performing some ETL Jobs. It takes between 1 and 4 hours to run, depending on our data sources.
The Job currently runs daily, without problems. What I would like to do now is to let it run in an endless loop, which means: When it's done, start over again.
The scheduler doesn't seem to provide this option. I found that it would be possible to use the steps interface to go to step one after the last step is finished, but there's a problem using that method: If I need to stop the job, I would need to do that in a forceful way. However I would like to be able to let the job stop after the next iteration. How can I do that?
Thanks in advance for any help!
Since neither Martin nor Remus created an answer, here is one so the question can be accepted.
The best way is to simply set the run frequency to a very low value, like one minute. If it is already running, a second instance will not be created. If you want to stop the job after the current run, simply disable the schedule.
Thanks!
So you want that when you want to stop the job, after the running iteration, it should stop - if I am getting you correctly.
You can do one thing here.
Have one table for configuration which is having boolean value.
Add one step into the job. i.e. Before iteration, check the value from table. If it's true, then only run the ETL packages.
So, each time it finds its true, it'll follow endless loop.
When you want to stop the job, set that value in table to false.
When the current job iteration completes, it'll go to find the value from your table, will find it false, and the iteration will stop.
you can always set the "on success" action to go to step one, creating an endless loop, but as you said, if you want to stop the job you'll have to force it.
Other than that, an simple control table on the database with a status and a second job that queries this table and fires your main job depending on the status. Coupe of possible architectures here, just pick the one that suits you better
You could use service broker within the database. The job you need to run can be started by queuing a 'start' message and when it finishes it can send itself a message to start again.
To pause the process you can just deactivate the queue processor.
Related
My use case is that I need to track all the changes (insertions/updates/deletions) from a table.
My idea is to create a stream on that table, and consume that stream every second or so, exporting all the changes to another history table (mytable_history).
A task would be the perfect candidate for that. But unfortunately, a task can only be scheduled for 1 minute or more. I'll be getting same-row updates per second, so I'd really need the task to run every second at least.
My idea now is to run an infinite LOOP, using SYSTEM$WAIT to consume the stream every 1 second and inserting the data to the history table.
Is this a bad idea? What could go wrong?
Thanks
I can add two points to your idea:
Please note that "DML updates to the source object in parallel transactions are tracked by the change tracking system but do not update the stream until the explicit transaction statement is committed and the existing change data is consumed." (https://docs.snowflake.com/en/user-guide/streams-intro.html#table-versioning)
Your warehouse would run all day to process this, that's why your costs would increase noticeable.
Firstly, I am new to SSIS and not sure of what format to explain the problem in a repeatable way for testing
So I have a task that loops through a folder looking for flat files and if in the correct format loads them into a SQL server staging table. This seem to work correctly
The next bit is the strange bit. I have a split for success and a split for failure (for user notification etc) on both legs I run a SQL task to count the number of rows in the staging table to process. IF it greater than zero I want to run my SQL stored procedure that handles the load to into the main database table for the data, then clean up.
In my debug email I see the number of rows is greater than 0 but the task does not proceed. I can not understand why. I have tried a number of constrain combinations (On completion, On Success, On Success and RowsToProcess>0).
I have also tried removing and adding my SQL Task and remapping.
(BTW all my tasks function in SSMS etc)
Nothing seems to work The only thing is that 2 branches re-join at this point, but surely that would not affect it, would it? (see below original screen shots)
Here is my control flow, If I have missed anything please add a comment and I will supply the information if I know it
My results from executions
On further testing two executables does seem to be the problem! So, additional questions.
Is this expected behaviour that two constraints joining back to a SQL task stops the flow in SSIS (in BPA task centre it does not)?
If, this is the case does this mean you have to repeat code in parallel? (for example if I was to run a clean up script at the end of my flow I would have to write it tice once for success leg and once for failure leg). This seems inefficient, am I missing something?
I think I've found the answer you are looking for here (go give it an upvote)
I was not aware of it, but basically, it says that bot branches must complete successfully (logical AND) in order for the flow to proceed, which won' happen as the right branch in this case won't even run.
The default behavior can be changed in the "constraint properties" (right-click on the arrow that links the blocks) under the "Multiple Constraints" section.
That said
I've never used this method, but that's up to the use case. I would have used "sequence containers" to avoid replicating blocks or multiple constraints, which is not the same but works as long as you don't have to pass data from one block to the other.
In your case, I'd put all the blocks inside a sequence container except the last 2, and after the container completes execute the last two as they must be always run after the previous block of operations
note about containers:
Containers do not output data but are useful to group operations logically, and you can even run just a container instead of the whole package (from VS).
Seems today I've learned something new too...
Hope this helps you solve your issue
An Index Defrag was executed and was later on terminated. It impacted 2 other processes to long run. One process cancelled by itself and the other was terminated. While re-running I was checking the status from sys.dm_exec_requests and notice that on the last part of the query which will insert the data into a table, it is changing status from running to runnable: preemptive_os_reportevent, etc.. Later on, the job once again cancelled by itself.
I want to learn why is the script changing status like that? Is that expected? And if something else is causing it to long run, what else should I check?
Note: I was also checking other active scripts at the time it was running and none was using the same target table.
This has been resolved yesterday. Apparently, the syslogs was full and is prohibiting the script to write logs therefore it is stuck and can't complete. Feel free to add inputs if you have some.
I have created an SSIS package and it has a for Loop. I have a table in SQL Server that stores the time the package started. and a column for Remainging_Time(Minute), which is a countdown(in minutes) value. Now I want the SSIS For Loop to run until the Remainging_Time(Minute)value is 0.
Description in the pic:
I think your problem is two fold. First, the for loop is going to perform a task, then check the eval expression. If your counter is really just keeping track of elapsed time, it is possible that your time will expire while it is performing whatever tasks you assign.
If you don't care about that level of precision, I would store the Remainging_Time as a package variable, not in a database table. You can use a SQL task with expressions to update the value of your package variable as the last task inside your loop. It doesn't make sense to have to make DB call every time through the loop unless there is some external process that is updating that value in the DB... and if that's the case, that's not really a for loop, that's more of a while loop, in which case I'd do a script task instead of a for loop and handle whatever you're trying to do in there.
Let me know if that makes sense. I can add more detail if I know more about what you're trying to do.
I am running a bunch of database migration scripts. I find myself with a rather pressing problem, that business is waking up and expected to see their data, and their data has not finished migrating. I also took the applications offline and they really need to be started back up. In reality "the business" is a number of companies, and therefore I have a number of scripts running SPs in one query window like so:
EXEC [dbo].[MigrateCompanyById] 34
GO
EXEC [dbo].[MigrateCompanyById] 75
GO
EXEC [dbo].[MigrateCompanyById] 12
GO
EXEC [dbo].[MigrateCompanyById] 66
GO
Each SP calls a large number of other sub SPs to migrate all of the data required. I am considering cancelling the query, but I'm not sure at what point the execution will be cancelled. If it cancels nicely at the next GO then I'll be happy. If it cancels mid way through one of the company migrations, then I'm screwed.
If I cannot cancel, could I ALTER the MigrateCompanyById SP and comment all the sub SP calls out? Would that also prevent the next one from running, whilst completing the one that is currently running?
Any thoughts?
One way to acheive a controlled cancellation is to add a table containing a cancel flag. You can set this flag when you want to cancel exceution and your SP's can check this at regular intervals and stop executing if appropriate.
I was forced to cancel the script anyway.
When doing so, I noted that it cancels after the current executing statement, regardless of where it is in the SP execution chain.
Are you bracketing the code within each migration stored proc with transaction handling (BEGIN, COMMIT, etc.)? That would enable you to roll back the changes relatively easily depending on what you're doing within the procs.
One solution I've seen, you have a table with a single record having a bit value of 0 or 1, if that record is 0, your production application disallows access by the user population, enabling you to do whatever you need to and then set that flag to 1 after your task is complete to enable production to continue. This might not be practical given your environment, but can give you assurance that no users will be messing with your data through your app until you decide that it's ready to be messed with.
you can use this method to report execution progress of your script.
the way you have it now is every sproc is it's own transaction. so if you cancel the script you will get it update only partly up to the point of the last successfuly executed sproc.
you cna however put it all in a singel transaction if you need all or nothign update.