I have a question on how to set up precedence constraints in SSIS.
I have the following flow in the package:
The execute SQL task returns a string value in this format '1111' that is then stored in the variable called "data"
The point of it all is to control which script tasks gets to execute
For example, if the value is "1111" then all 4 scripts get to run.
If the value is "1011" then scripts 1,3,4 get to run... you get the picture
The scripts DEPEND on the previous one in all cases.
The constraints evaluate with an expression such as this: SUBSTRING(#[User::data], 2,1)=="1" for script 2, for example.
The problem:
If one of the scripts dont run, then the next one wont either (because the constraint never got to the evaluated). For example, for data = "1011", the scripts 3 and 4 never get to run because number 2 never ran...
Do you know a better way to make this work?
Using SQL Server 2008 with BIDS
I agree with #Iamdave for his direction that says modify your script in each script do the check there if it should or should not execute rather than using expression constraints.
However, because someone might want to do this with Data Flow Tasks or something here is a way to do it with SSIS components. Your problem is that you want to conditionally execute a task but whether or not that task gets executed you then want to conditionally execute another task. By placing each script task in a Sequence Container the precedence between sequence containers will always make the other container execute as long as that prior container does not fail or does not have a expression as a constraint. But once in the container you need to be able to set a Conditional Precedence. You can do that by adding a dummy task of some kind ahead of the script task and adding the constraint on that precedence.
Add some conditional code into your scripts.
For example, at the start of script task 1:
if firstcharacter(Variable) = 1
do things
else
do nothing
And then in script task 2:
if secondcharacter(Variable) = 1
do things
else
do nothing
and just make sure that all the data passes through, only being transformed if the right character in your variable is met.
Related
Firstly, I am new to SSIS and not sure of what format to explain the problem in a repeatable way for testing
So I have a task that loops through a folder looking for flat files and if in the correct format loads them into a SQL server staging table. This seem to work correctly
The next bit is the strange bit. I have a split for success and a split for failure (for user notification etc) on both legs I run a SQL task to count the number of rows in the staging table to process. IF it greater than zero I want to run my SQL stored procedure that handles the load to into the main database table for the data, then clean up.
In my debug email I see the number of rows is greater than 0 but the task does not proceed. I can not understand why. I have tried a number of constrain combinations (On completion, On Success, On Success and RowsToProcess>0).
I have also tried removing and adding my SQL Task and remapping.
(BTW all my tasks function in SSMS etc)
Nothing seems to work The only thing is that 2 branches re-join at this point, but surely that would not affect it, would it? (see below original screen shots)
Here is my control flow, If I have missed anything please add a comment and I will supply the information if I know it
My results from executions
On further testing two executables does seem to be the problem! So, additional questions.
Is this expected behaviour that two constraints joining back to a SQL task stops the flow in SSIS (in BPA task centre it does not)?
If, this is the case does this mean you have to repeat code in parallel? (for example if I was to run a clean up script at the end of my flow I would have to write it tice once for success leg and once for failure leg). This seems inefficient, am I missing something?
I think I've found the answer you are looking for here (go give it an upvote)
I was not aware of it, but basically, it says that bot branches must complete successfully (logical AND) in order for the flow to proceed, which won' happen as the right branch in this case won't even run.
The default behavior can be changed in the "constraint properties" (right-click on the arrow that links the blocks) under the "Multiple Constraints" section.
That said
I've never used this method, but that's up to the use case. I would have used "sequence containers" to avoid replicating blocks or multiple constraints, which is not the same but works as long as you don't have to pass data from one block to the other.
In your case, I'd put all the blocks inside a sequence container except the last 2, and after the container completes execute the last two as they must be always run after the previous block of operations
note about containers:
Containers do not output data but are useful to group operations logically, and you can even run just a container instead of the whole package (from VS).
Seems today I've learned something new too...
Hope this helps you solve your issue
I have an existing Snowflake task that is scheduled to run once an hour and is using the WHEN condition to check if a particular stream has data. I'd like to ALTER this task so that it no longer uses the when condition, however, unset does not seem to work and I cannot modify it to NULL. The one workaround we've found is you can modify it to TRUE which seems to work (the statement executes and the task does run), but I'd prefer to have it unset like the other tasks that do not have a when condition set. I also recognize I could drop the task and recreate it, however, then I'd lose the history. This is not a significant problem, but seems strange that it cannot be modified after the fact and I've not seen any documentation that indicates this is expected behavior.
Note: I've made sure the root task is suspended before trying to make any changes.
alter task example_task unset when;
SQL compilation error: invalid property 'when' for 'TASK'
alter task example_task modify when NULL;
Invalid expression for task condition expression. Expecting one of
the following: [SYSTEM$STREAM_HAS_DATA]
alter task example_task unset condition;
SQL compilation error: invalid property 'condition' for 'TASK'
(Note: tested ^ because the column is labeled 'condition' when you run show tasks)
alter task example_task remove when;
SQL compilation error: syntax error line 1 at position 31 unexpected
'when'.
(Note: tested ^ because you use 'remove' to change the 'after' parameter)
alter task example_task modify when TRUE;
Statement executed successfully.
As a side note, this came to our attention while trying to use the snowflake terraform provider which unsuccessfully tried to update the when condition. Now I think I know why.
As an alternative approach, why not replace the whole task using CREATE OR REPLACE TASK...?
I have created an SSIS package and it has a for Loop. I have a table in SQL Server that stores the time the package started. and a column for Remainging_Time(Minute), which is a countdown(in minutes) value. Now I want the SSIS For Loop to run until the Remainging_Time(Minute)value is 0.
Description in the pic:
I think your problem is two fold. First, the for loop is going to perform a task, then check the eval expression. If your counter is really just keeping track of elapsed time, it is possible that your time will expire while it is performing whatever tasks you assign.
If you don't care about that level of precision, I would store the Remainging_Time as a package variable, not in a database table. You can use a SQL task with expressions to update the value of your package variable as the last task inside your loop. It doesn't make sense to have to make DB call every time through the loop unless there is some external process that is updating that value in the DB... and if that's the case, that's not really a for loop, that's more of a while loop, in which case I'd do a script task instead of a for loop and handle whatever you're trying to do in there.
Let me know if that makes sense. I can add more detail if I know more about what you're trying to do.
we have a requirement where SSIS job should trigger based on the availability of value in the status table maintained,point to remember here that we are not sure about the exact time when the status is going to be available so my SSIS process must continuously look for the value in status table,if value(ex: success) is available in status table then job should trigger.here we have 20 different ssis batch processes which should invoke based on respective/related status value is available.
What you can do is:
Scheduled the SSIS package that run frequently.
For that scheduled package, assign the value from the table to a package variable
Use either expression for disabling the task or constraint expression to let the package proceeds.
Starting a SSIS package takes some time. So I would recommend to create a package with the following structure:
Package variable Check_run type int, initial value 1440 (to stop run after 24 hours if we run check every minute). This is to avoid infinite package run.
Set For Loop, check if Check_run is greater than zero and decrement it on each loop run.
In For loop check your flag variable in Exec SQL task, select single result value and assign its result to a variable, say, Flag.
Create conditional execution branches based on Flag variable value. If Flag variable is set to run - start other packages. Otherwise - wait for a minute with Exec SQL command waitfor delay '01:00'
You mentioned the word trigger. How about you create a trigger when that status column meets the criteria to run the packages:
Also this is how to run a package from T-SQL:
https://www.timmitchell.net/post/2016/11/28/a-better-way-to-execute-ssis-packages-with-t-sql/
You might want to consider creating a master package that runs all the packages associated with this trigger.
I would take #Long's approach, but enhance it by doing the following:
1.) use Execute SQL Task to query the status table for all records that pertain to the specific job function and load the results into a recordset. Note: the variable that you are loading the recordset into must be of type object.
2.) Create a Foreach Loop enumerator of type ADO to loop over the recordset.
3.) Do stuff.
4.) When the job is complete, go back to the status table and mark the record complete so that it is not processed again.
5.) Set the job to run periodically (e.g., minute, hourly, daily, etc.).
The enhancement hear is that no flags are needed to govern the job. If a record exists then the foreach loop does its job. If no records exist within the recordset then the job exits successfully. This simplifies the design.
I started a new job a few months ago and am noticing some of the older SSIS packages are built with dummy tasks. For example, if there is a Execute SQL Task and it produces a Boolean result, the True result may go to the task that needs something done. However, there is also a dummy task for False that does nothing and is labeled as such.
Here's an example of what I'm seeing.
Is this a best practice or proper way to code these types of Tasks? The way I code these types of tasks is to only code the needed task and ignore the other. In the case of my image example, I would code the True but not add a task for the False.
Just curious if I'm doing something wrong here.
Why adding a dummy task that does nothing?
(1) Informing Developers
If the goal is Informing developers that edits the package then you can write an annotation beside of the True precedence constraint, and write that if False result is not considered or does or does nothing
(2) For Debugging purposes
If the adding a dummy task is for debugging purposes, then it is not a bad idea. Because it will make the package flow more clear for the viewer. But you have to disable this component before deploying the package on the production server.
(3) To apply an expression
If the goal is to apply an expression in the precedence constraint like mentioned in the comments, then the proper way is to use an Expression Task (in case that it is connected to a dummy task)
Also based on this Microsoft article:
A precedence constraint can use an expression to define the constraint between two executables
So if expression are needed, then write it in the True precedence constraint, or if expression is needed to assign some values to a variable then like i mentioned above use an Expression Task
In one line standard practise is to create the log if something does not go according to desired logic. It should be logged for future debugging.
If you are expecting true as a result of some boolean logic and it turns out to be false then there should be some structure to log it.
Now coming to use of dummy task it is of no harm but instead of that, some task should be used which can create some logs for future reference.