I have a simple String variable with the following value: "C:\Test.txt".
Now I would like to edit the variable to point to a different file.
I cannot find a way to do that. I can change the Name, Data Type, but not the value itself!
Do I need to delete the variable and create the new one?
Update: The problem was caused by "ReadOnly" property set to "True". For typical scenarios, see the accepted answer below.
As #Yuck and #devarc have noted, there are two different and distinct values a Variable holds. The Design-time value is the value you assign when the variable is first created. In your case, the variable holds C:\Test.txt as the design-time value. Everytime you open the package, it would show C:\Test.txt until you change it in the
To make the value of a variable change while the package is running, your options are either to set the value or calculate it. Here I have created a package-level variable CurrentFile with the value of C:\Test.txt
One thing that often trips people up is that they have correctly changed the run-time value but when they run it in BIDS, they see the "old" value. The value displayed in the Variables window does not change during package execution.
During package execution, my Variables window still shows the design-time value (C:\Test.txt) but the true value is reflected in the Locals window (C:\Test2.txt)
Setting a value
The value of most anything in SSIS can be established at run-time through a set of verbose command-line options or through configuration sources. The biggest difference in my mind is that this approach is that the value will always be the value for the entire lifetime of package execution. Sequential or parallel invocations of a package can change that value but for that execution the value would remain constant (barring an explicit modification of the value.
/SET
Command-line execution (dtexec.exe), right clicking on a package and running from the filesystem (dtexecUI.exe) or creating a SQL Agent job step of SQL Server Integration Services all allow for providing a run-time value through the SET command. Using the above variable, the following command would set the run-time value to C:\Test2.txt
dtexec /file C:\Generated.dtsx /set \Package.Variables[User::CurrentFile].Properties[Value];"C:\Test2.txt"
Configuration
SSIS offers an option to create configuration sources to provide run-time values to packages. The article I linked to above does a much better job describing the pros and cons of the configuration options than I will do here. I will say that I typically use both - my SET command configures a connection manager which is then used by the package to find the "full" set of package configurations.
Calculating a value
There are a variety of tasks in SSIS that can change the value of a variable as well as the use of Expressions to change a value. I see these as things that operate on value whilst the package is in flight.
Tasks
A Script Task is one of the most commonly used mechanisms for those starting out but I find other tools in the SSIS toolkit usually better suited for changing variable values.
Foreach Loop Container and Execute SQL Task are two of the other big Tasks you should look at for assignment of a variable value.
Expressions
Expressions are the most glorious candy in the SSIS toolbox. Most every "thing" in SSIS exposes properties for configuration. That's helpful, but using assigning an expression to build those properties is outstanding.
For example, imagine 3 variables RootFolder, FileName and ComputedCurrentFile with values of C:\, File2.txt and empty string. On the Properties window for ComputedCurrentFile we'd change the value for EvaluateAsExpression from False to True and then use an expression like #[User::RootFolder]+ "\\" +#[User::FileName] That simply concatenates the value the first two variables together. This can be helpful if the file name for processing was standard but the source folder changed often. Or if we're talking about output, it's common to use expressions to build an output file name using the date and possibly time of when the package is running.
Finally, there is nothing that prevents a mixing and matching of these approaches. I typically use a configuration to point a file enumerator at the correct starting folder and then use calculated values to identify the current file for processing.
If you want to change it in designer just right click on free space and --> Variables.
But if you want to change it at runtime I suggest you to:
create script task
choose language
add your variable to ReadWriteVariables.
Edit script.
For example in VB:
Dts.Variables("myVariable").Value = #"C:\Test2.txt";
Dts.TaskResult = ScriptResults.Success
Found an easy way to handle this. Remove the Variable from Expression which will enable Value Box to edit. Once it is edited, add the Variable back in the Expression should get the updated value. Hope this helps.
I was also facing the same issue like you where once the variable is declared and define (for eg:var1=text1.csv)in SSIS Variable window I was not able to update the variable value(for eg: var1=text2.csv) in SSIS Variable Window by clicking on the variable value field.
Applied below fix:-
I noticed that I was using var1 variable as a Expression by using expression builder so to update the value(for eg:-var1=text2.csv) I used expression builder window.once you done using the expression builder,you can see the text2.csv is got mapped to var1.
Related
I have a SSIS package which reads from the MSSQL database and saves it into a xlsx file.
I dynamically change the extract with the following format: [filename - ddmmyyyy hhmm].xlsx
problems is this: my SSIS package creates a file like [filename - 18052021 1400].xlsx
however when it tries to email it for example, the time is now 1401 and it tries to find a file name like [filename - 18052021 1401].xlsx which does not exist and so it generates an error.
Is there any way to keep the variable static through the execution?
Many thanks,
The problem you're experiencing is that every time a variable with an Expression is read, it is evaluated. I vent about this from time to time in my answers because it can be an insidious little problem to track down.
Currently, you are building a file name something like
"FileName - " + (DT_WSTR, 2) day(getdate()) ...
The problem as I've already hinted and you're experiencing is that every time that expression is evaluated, SSIS checks the current time. If your package runs for more than a minute, you'll have crossed the boundary and now have a "new" name to deal with.
The way to resolve it, is to use a System scoped variable, #[System::StartTime], instead of the getdate
"FileName - " + (DT_WSTR, 2) day(#[System::StartTime])) ...
StartTime is the time the package itself starts. It could run for a minute or a day and the value will remain constant because it's what you expect - the time the package started.
If you need something that can change but remains constant for a specific scope, put all the pieces in a Sequence Container and then you can make use of a System scoped variable named (approximately) ContainerStartTime. The container (sequence, foreach, for) only has one start time but it can be 10 minutes later than the package itself started.
Presumably your variable's value is an expression then, not a Value? If so, then don't use an Expression for the variable, assign it a value, and then assign a new value of the variable at the start of your SSIS package using a expression Task. Without the variable name, nor your expression, I can't give the exact solution, but the expression task would have an expression like:
#[User::YourVariableName] = {Your original Expression}
I have package variable (TableName), its value is set by for each loop container.
For instance TableName = sales set by for each loop after looping through informationschema.tables then I want to use TableName's value in dataflow for this I am using script component.
When I use script component to read TableName's value in script-file I cannot see variable any idea why??
I am using script component as source and TableName is in readonlyvariables.
Please help as I am new to SSIS.
On the Variable Mappings page of the Foreach Loop Editor, assign variables to each item of data that is returned by a single enumerated item. For example, a Foreach File enumerator returns only a file name at Index 0 and therefore requires only one variable mapping, whereas an enumerator that returns several columns of data in each row requires you to map a different variable to each column that you want to use in the Script task. +
After you have mapped enumerated items to variables, then you must add the mapped variables to the ReadOnlyVariables property on the Script page of the Script Task Editor to make them available to your script
For more details pls chk the below link
https://learn.microsoft.com/en-us/sql/integration-services/extending-packages-scripting/task/using-variables-in-the-script-task
https://social.technet.microsoft.com/wiki/contents/articles/22194.use-ssis-variables-and-parameters-in-a-script-task.aspx
Please use the following Microsoft Documentation link - it describes differences between Script Task and Script Component, and gives small code samples, specifically on using Variables.
as per this I need to use this.variables.myvariable
Thanks everyone.
I'm doing an Excel loop through fifty or more Excel files. The loop goes through each Excel file, grabs all the data and inputs it into the database without error. This is the typical process of setting delay validation to true, and making sure that the expression for the Excel Connection is a string variable called EFile that is set to nothing (in the loop).
What is not working: trying to input the name of the Excel file into the database.
What's been tried (edit; SO changed my 2 to 1 - don't know why):
Add a derived column between the Excel file and database input, and add a column using the EFile expression (so under Expression in the Derived Column it would be #[User::EFile]). and add the empty. However, this inputs nothing a blank (nothing).
One suggestion was to add ANOTHER string variable and set its properties EvaluateAsExpression to True and set the Expression to the EFile variable (#[User::EFile]). The funny thing is that this does the same thing - inputs a blank into the database.
Numerous people on blogs claim they can do this, yet I haven't seen one actually address this (I have a blog and I will definitely be showing people how to do this when I get an answer because, so far, these others have fallen short). How do I grab an Excel file's name and input it in a database during a loop?
Added: Forgot to add, no scripts; the claim is that it can be done without them, so I want to see the solution without them.
Note: I already have the ability to import the data from the Excel files - that's easy (see my GitHub account, as I have two different projects for importing all sorts of txt, csv, xls, xlsx data). I am trying to also get the actual name of the file being imported also into the database. So, if there are fifty Excel files, along with the data in each file, the database will have the fifty file names alongside that data (so if each file has 1000 rows of data, each 1000 rows would also have the name of the file they came from next to them as an additional column). This point seems to cause a lot of confusion, as people assume I'm having trouble importing data in files - NOPE, see my GitHub; again that's easy. It's the FILENAME that needs to also be imported.
Test package: https://github.com/tmmtsmith/SSISLoopWithFileName
Solution: #jaimet pointed out that the Derived Column needed to be the #[User::CurrentFile] (see the test package). When I first ran the package, I still got a blank value in my database. But when we originally set up the connection, we do point it to an actual file (I call this "fooling the package"), then change the expression on the connecting later to the #[User::CurrentFile], which is blank. The Derived Column, using the variable #[User::CurrentFile], showed a string of 0. So, I removed the Derived Column, put the full file path and name in the variable, then added the variable to the Derived Column (which made it think the string was 91 characters long), then went back and set the variable to nothing (English teacher would hate the THENs about right now). When I ran the package, it inputted the full file path. Maybe, like the connection, it needs to initially think that a file exists in order for it to input the full amount of characters?
Appreciate all the help.
The issue is because of blank value in the variable #[User::FileNameInput] and this caused the SSIS package to assume that the value of this variable will always be of zero length in the Derived Column transformation.
Change the expression on the Derived column transformation from #[User::FileNameInput] to (DT_STR, 2000, 1252)#[User::FileNameInput].
Type casting the derived column to 2000 sets the column length to that maximum value. The value 1252 represents the code page. I assumed that you are using ANSI code page. I took the value 2000 from your table definition because the FilePath column had variable VARCHAR(2000). If the column data type had been NVARCHAR(2000), then the expression would be (DT_WSTR, 2000)#[User::FileNameInput]
Tim,
You're using the wrong variable in your Derived Column component. You are storing the filename in #[User::CurrentFile] but the variable that you're using in your Derived Column component is #[User::FileNameInput]
Change your Derived Column component to use #[User::CurrentFile] and you'll be good.
Hope that helps.
JT
If you are using a ForEach loop to process the files in a folder then I have have used the technique described in SSIS Junkie's blog to get the filename in to an SSIS variable: SSIS: Enumerating files in a Foreach loop
You can use the variable later in your flow to write it to the database.
TO all intents and purposes your method #1 should work. That's exactly how I would attempt to do it. I am baffled as to why it is not working. Could you perhaps share your package?
Tony, thanks very much for the link. Much appreciated.
Regards
Jamie
I am getting to know SSIS, I apologize if the question is too simple.
I got a set of tasks inside a foreach-loop-container.
The first task needs only to get executed on condition that a certain user variable is not null or empty.
Otherwise, the flow should skip the first task and continue to the second one.
How would I go about realizing this (in detail) ?
Issue 1: There are two ways to interpret your logic: "...a certain user variable is not null or empty":
The (Variable is Not Null) OR the (Variable is Empty).
The (Variable is Not Null) OR the (Variable is Not Empty).
It's all about the object(s?) of the word "not". The differences are subtle but will impact when the first task in the Foreach loop executes. For demonstration purposes, I am assuming you intend #1.
Issue 2: The first task can no longer be first. In order to accomplish what you desire using SSIS inside the BIDS environment, you need to place another task ahead of the task formerly known as "the first task". This is so you can set a Precedence Constraint on the former first task from the new first task.
It is possible to accomplish what you desire by designing your SSIS dynamically from managed code, but I don't think this issue warrants the overhead associated with that design choice.
I like using an empty Sequence Container as an "Anchor" task - a task that exists solely to serve as the starting endpoint of a Precedence Constraint. I heavily document them as such. I don't want anyone deleting the "unnecessary empty container" and roaming the halls for days shaking their heads and repeating "Andy, Andy, Andy..." but I digress.
In the example below, I have two precedence constraints leaving the empty Sequence Container. One goes to the task that may be skipped and the other to the task following the task that can sometimes be skipped. A third precedence constraint is required between the task that can sometimes be skipped and the task following. It is important to note this third precedence constraint must be edited and the Multiple Constraints option set to OR. This allows the task following to execute when either of the mutually exclusive previous paths are taken. By default, this is set to AND and will require both paths to execute. By definition, that will not - cannot - happen with mutually exclusive paths.
I test the value of an SSIS String variable named #MyVar to see if it's Null or Empty. I used the Expression Only Evaluation Option for the constraints leaving the empty Sequence Container. The expressions vary but establish the mutual exclusivity of the expression. My Foreach Loop Container looks like this:
I hope this helps.
:{>
The best thing can be to use the 'Disable Property' in expressions and giving the expression as per the condition. Just search how to use the disable property.
How about a simple solution instead of some of the more complex ones that have already been given. For the task you want to conditionally skip, add an expression to the disabled property. Any expression that produces a true or false result will work, so for the question example you could use:
ISNULL(#[User::MY_VAR]) || #[User::MY_VAR]==""
The only downside is that it may not as visible as some of the other solutions but it is far easier to implement.
I would create a For Loop Container around the task that needs the condition with the following conditions (#iis the loop counter, #foo is your user variable that you want to test):
InitExpression: #i=0
EvalExpression: #i<1 && !ISNULL(#Foo) && #Foo!=""
AssignExpression: #i=#i+1
there is no need to create a "script"
I think the best (and simpler) approach is to add a blank script task inside your loop container before your "first task", drag the green arrow from it to your "first task" (which obviously will become the second) and use the precedence constraint to do the check.
To do that, double click the arrow, select "expression" on the "evaluation operation" and write your expression. After hitting OK the arrow will become blue indicating that it isnt a simple precedence constraint, it has a expression assigned to it.
Hopefully I didn't misunderstand the question but a possible solution can be as written below.
I created a sample ForEach loop. The loop itself is an item enumerator. It enumerates the numbers 1, 2, 3. The acutal value is stored in a variable called LoopVariable.
There is another variable named FirstShouldRun which is a Boolean variable showing the first task in the foreach loop should be runned or not. I set this variable's EvaluateAsExpression property to true, and its expression is (#[User::LoopVariable] % 2) == 0. I would like to demonstrate with this that every second time the first task should be started.
The two tasks do nothing much but display a MessageBox showing the task has been started.
I started the package and first and the third time the first task didn't started. In the second loop the MessageBox (showing "First started") appeared.
After that you should set FirstShouldRun variable as you like.
As I mentioned in my first comment to the OP, this solution is based on the idea of Amos Wood written in another answer.
That's a bit tricky.
You have to create a Script Task and check if your variable is not null in there.
So first you have the script task in which you will have the following code in your Main() function:
public void Main()
{
if (Dts.Variables["User::yourVariable"].Value != null)
{
Dts.TaskResult = (int)ScriptResults.Failure;
}
else
{
Dts.TaskResult = (int)ScriptResults.Success;
}
}
Then you create two connections from your script task, one to the task that needs to be executed when your variable is not null, and one to the next task (or to another script, if you need to check again, if the variable is not null).
Then you right-click on the (green) arrow of your first connection and select "Failure". Right-click the connection to the next task / script and set it to "Completion".
It should then look something like this:
That's it.
I'm tring to create an SSIS package to import some dataset files, however given that I seem to be hitting a brick
wall everytime I achieve a small part of the task I need to take a step back and perform a sanity check on what I'm
trying to achieve, and if you good people can advise whether SSIS is the way to go about this then I would
appreciate it.
These are my questions from this morning :-
debugging SSIS packages - debug.writeline
Changing an SSIS dts variables
What I'm trying to do is have a For..Each container enumerate over the files in a share on the SQL Server. For each
file it finds a script task runs to check various attributes of the filename, such as looking for a three letter
code, a date in CCYYMM, the name of the data contained therein, and optionally some comments. For example:-
ABC_201007_SalesData_[optional comment goes here].csv
I'm looking to parse the name using a regular expression and put the values of 'ABC', '201007', and
'SalesData' in variables.
I then want to move the file to an error folder if it doesn't meet certain criteria :-
Three character code
Six character date
Dataset name (e.g. SalesData, in this example)
CSV extension
I then want to lookup the Character code, the date (or part thereof), and the Dataset name against a lookup table
to mark off a 'checklist' of received files from each client.
Then, based on the entry in the checklist, I want to kick off another SSIS package.
So, for example I may have a table called 'Checklist' with these columns :-
Client code Dataset SSIS_Package
ABC SalesData NorthSalesData.dtsx
DEF SalesData SouthSalesData.dtsx
If anyone has a better way of achieving this I am interested in hearing about it.
Thanks in advance
That's an interesting scenario, and should be relatively easy to handle.
First, your choice of the Foreach Loop is a good one. You'll be using the Foreach File Enumerator. You can restrict the files you iterate over to be just CSVs so that you don't have to "filter" for those later.
The Foreach File Enumerator puts the filename (full path or just file name) into a variable - let's call that "FileName". There's (at least) two ways you can parse that - expressions or a Script Task. Depends which one you're more comfortable with. Either way, you'll need to create three variables to hold the "parts" of the filename - I'll call them "FileCode", "FileDate", and "FileDataset".
To do this with expressions, you need to set the EvaluateAsExpression property on FileCode, FileDate, and FileDataset to true. Then in the expressions, you need to use FINDSTRING and SUBSTRING to carve up FileName as you see fit. Expressions don't have Regex capability.
To do this in a Script Task, pass the FileName variable in as a ReadOnly variable, and the other three as ReadWrite. You can use the Regex capabilities of .Net, or just manually use IndexOf and Substring to get what you need.
Unfortunately, you have just missed the SQLLunch livemeeting on the ForEach loop: http://www.bidn.com/blogs/BradSchacht/ssis/812/sql-lunch-tomorrow
They are recording the session, however.