I have a SSIS package which reads from the MSSQL database and saves it into a xlsx file.
I dynamically change the extract with the following format: [filename - ddmmyyyy hhmm].xlsx
problems is this: my SSIS package creates a file like [filename - 18052021 1400].xlsx
however when it tries to email it for example, the time is now 1401 and it tries to find a file name like [filename - 18052021 1401].xlsx which does not exist and so it generates an error.
Is there any way to keep the variable static through the execution?
Many thanks,
The problem you're experiencing is that every time a variable with an Expression is read, it is evaluated. I vent about this from time to time in my answers because it can be an insidious little problem to track down.
Currently, you are building a file name something like
"FileName - " + (DT_WSTR, 2) day(getdate()) ...
The problem as I've already hinted and you're experiencing is that every time that expression is evaluated, SSIS checks the current time. If your package runs for more than a minute, you'll have crossed the boundary and now have a "new" name to deal with.
The way to resolve it, is to use a System scoped variable, #[System::StartTime], instead of the getdate
"FileName - " + (DT_WSTR, 2) day(#[System::StartTime])) ...
StartTime is the time the package itself starts. It could run for a minute or a day and the value will remain constant because it's what you expect - the time the package started.
If you need something that can change but remains constant for a specific scope, put all the pieces in a Sequence Container and then you can make use of a System scoped variable named (approximately) ContainerStartTime. The container (sequence, foreach, for) only has one start time but it can be 10 minutes later than the package itself started.
Presumably your variable's value is an expression then, not a Value? If so, then don't use an Expression for the variable, assign it a value, and then assign a new value of the variable at the start of your SSIS package using a expression Task. Without the variable name, nor your expression, I can't give the exact solution, but the expression task would have an expression like:
#[User::YourVariableName] = {Your original Expression}
Related
I have an SSIS package that writes csv files from a database, copies them to a couple locations, and then emails a success message. The process is:
Retrieve public file location from database into a variable, #[User::varSQLCSVOutputFolder]
Loop through a list of database records:
Create a local CSV for each one.
Copy the local file to the location in #[User::varSQLCSVOutputFolder]
Send email with MessageSource defined in #[User:varEmailBody].
#[User::varEmailBody] = "Files successfully saved to " + #[User::varCNNTargetCSVFolder]
#[User::varCNNTargetCSVFolder] = #[User::varSQLCSVOutputFolder]
#[User::varSQLCSVOutputFolder] loads from the database, value = \\server.domain.com\TEST\Output Files AM
(to confirm, #[User::varCNNTargetCSVFolder] is just a pass-thru)
I can confirm the expressions flow through at design time. But when I execute it from SSISDB, I get the error
Error: An error occurred with the following error message:
Failed to lock variable "Files successfully saved to
\\server.domain.com\TEST\Output Files AM" for read access with error
0xC0010001 The variable cannot be found. This occurs when an attempt
is made to retrieve a variable from the Variables collection on a
container during execution of the package, and the variable is not
there. The variable name may have changed or the variable is not
being created.
I thought maybe it was a weird problem with escaping the backslashes, but I tried using a REPLACE() in the expression, no luck. I do use the underlying variable #[User::varSQLCSVOutputFolder] repeatedly, but I have precedent constraints set up, so there should be no overlap.... any other possibilities?
It seems to be reading the CONTENT of my variable as the NAME of the variable.
Okay, this was a fun one. So, I had an expression defined for MessageSource, BUT, I had chosen MessageSourceType as Variable, not Direct Input. See below for posterity.
I'd like to define start_date and end_date parameters in my SSIS package, and have a foreach container that runs for each date in between these 2 (inclusive), which executes a SQL query taking in the current date value (ie starting at start_date) and using it as a parameter for the query.
I'm quite new to SSIS programming and I cannot find information on how to do this.
You can simply add a for loop container and use these variables as mentioned in the image below:
Where #[User:Loop], #[User:MinDate], #[User::MaxDate] are of type System.DateTime
image reference
How do I loop through date values stored as numbers within For Loop container?
Passing parameters to Execute SQL Task
You can refer to the following posts for more details:
Passing Variables to and from an SSIS task
How to pass variable as a parameter in Execute SQL Task SSIS?
A For Loop would be the better option to do this. Assuming that the start and end dates as supplied as parameters to the package as indicated in your question, be aware that parameters cannot be updated in an SSIS package however variables can be. This, as well as an example of the process outlined in your question, is further detailed below.
Create an SSIS datetime variable. As mentioned earlier, this will be used to store in initial value of the start date parameter.
Next add a For Loop on the Control Flow. In the screenshot below, the variable #[User::vStartDate] is set to the same value as the package parameter #[$Package::pStartDate] in the InitExpression on the For Loop. Iterations of the loop continue while the start date variable is less than/equal to the end date parameter, which is specified in the EvalExpression field.
After the Execute SQL Task (or however the SQL query is executed) add a Script Task. This will increment the value of the start date variable, so make sure this is the last task in the loop. An example C# script is below, which simply sets the value of the start date SSIS variable to a C# variable, increments the C# variable by one day, then writes that value back to the SSIS variable. Make sure to add the SSIS start date variable in the ReadWriteVariables field on the Script Task. This will go in the Main method of the script as follows. Although there’s just an increment of the date and update of the variable done in the Script Task, having this in place will allow for easier sustainability in the long term in case more logic needs to be added to this as C# provides much more functionality.
Script Task:
public void Main()
{
//get value in current iteration of loop
DateTime currentIterationValue = Convert.ToDateTime(Dts.Variables["User::vStartDate"].Value);
//increment by one day
currentIterationValue = currentIterationValue.AddDays(1);
//update SSIS variable
Dts.Variables["User::vStartDate"].Value = currentIterationValue;
Dts.TaskResult = (int)ScriptResults.Success;
}
I used an Execute SQL Task to store the dates (results) as a Result Set in a user defined variable. Then, inside the foreach loop container, I used the foreach ADO Enumerator on the user defined variable which has the set of dates. Using the variable mapping in the foreach loop container, you can map the start_date and end_dates from the user defined variable and pass it to other variables.
For example:
I have a SELECT statement which selects 2 rows with columns start_date and end_date. This will be stored as a result set in a variable called "main_dates". The foreach ADO Enumerator will enumerate on this "main_dates" variable (for each row in main_dates run the for loop). Then in the Variable Mapping section, you can create 2 new variables called u_start_date and u_end_date and map the columns 0 and 1 to these variables.
Inside the foreach loop whenever you execute a stored procedure, you can pass the u_start_date and u_end_date variables as parameters.
In my SSIS package, I have a Script Component that creates a +1 new customer ID when the package runs.
I want to assign the numerical value of the ID to a variable. The variable will be used in a File System Task to create a directory with the Customer ID as the folder name.
Example - C:\Customer Orders\<CO_ID>
I have created a derived column (sequence) transformation in the data flow named (CO_ID). However, I am stuck on the proper syntax for the variable expression.
Any advice is appreciated.
Without the error message and more details, it is difficult to say what's wrong. But I think you're trying to convert a number to a string and then concatenate it to another string (the path). If that's the case, your derived column expression should be something like:
"c:\Customer Orders\" + (DT_STR, 10,1252) CO_ID
If the backslashes give you a problem (can't remember if they need to be escaped), then try doubling them up:
"c:\\Customer Orders\\" + (DT_STR, 10,1252) CO_ID
Derived column Transformation is used inside a Dataflow task, and Execute System Task is on the Control Flow level, so any variable update is not visible to the Control Flow level until the Dataflow Task execution is done. So it will only create a directory for the last CO_ID.
If you need to create a Directory for each customer, and assuming that you are creating ID inside the script Component. Just create the directory from it using the following code (Vb.Net)
If Not Io.Directory.Exists("c:\Customer Orders\" & COID.ToString()) Then
IO.Directory.CreateDirectory(c:\Customer Orders\" & COID.ToString())
End If
Else, if you only need to create columns and to loop over them later, i think that you only need to Cast the CO_ID column to DT_WSTR:
"C:\\Customer Orders\\" + (DT_WSTR, 50) [CO_ID]
I'm doing an Excel loop through fifty or more Excel files. The loop goes through each Excel file, grabs all the data and inputs it into the database without error. This is the typical process of setting delay validation to true, and making sure that the expression for the Excel Connection is a string variable called EFile that is set to nothing (in the loop).
What is not working: trying to input the name of the Excel file into the database.
What's been tried (edit; SO changed my 2 to 1 - don't know why):
Add a derived column between the Excel file and database input, and add a column using the EFile expression (so under Expression in the Derived Column it would be #[User::EFile]). and add the empty. However, this inputs nothing a blank (nothing).
One suggestion was to add ANOTHER string variable and set its properties EvaluateAsExpression to True and set the Expression to the EFile variable (#[User::EFile]). The funny thing is that this does the same thing - inputs a blank into the database.
Numerous people on blogs claim they can do this, yet I haven't seen one actually address this (I have a blog and I will definitely be showing people how to do this when I get an answer because, so far, these others have fallen short). How do I grab an Excel file's name and input it in a database during a loop?
Added: Forgot to add, no scripts; the claim is that it can be done without them, so I want to see the solution without them.
Note: I already have the ability to import the data from the Excel files - that's easy (see my GitHub account, as I have two different projects for importing all sorts of txt, csv, xls, xlsx data). I am trying to also get the actual name of the file being imported also into the database. So, if there are fifty Excel files, along with the data in each file, the database will have the fifty file names alongside that data (so if each file has 1000 rows of data, each 1000 rows would also have the name of the file they came from next to them as an additional column). This point seems to cause a lot of confusion, as people assume I'm having trouble importing data in files - NOPE, see my GitHub; again that's easy. It's the FILENAME that needs to also be imported.
Test package: https://github.com/tmmtsmith/SSISLoopWithFileName
Solution: #jaimet pointed out that the Derived Column needed to be the #[User::CurrentFile] (see the test package). When I first ran the package, I still got a blank value in my database. But when we originally set up the connection, we do point it to an actual file (I call this "fooling the package"), then change the expression on the connecting later to the #[User::CurrentFile], which is blank. The Derived Column, using the variable #[User::CurrentFile], showed a string of 0. So, I removed the Derived Column, put the full file path and name in the variable, then added the variable to the Derived Column (which made it think the string was 91 characters long), then went back and set the variable to nothing (English teacher would hate the THENs about right now). When I ran the package, it inputted the full file path. Maybe, like the connection, it needs to initially think that a file exists in order for it to input the full amount of characters?
Appreciate all the help.
The issue is because of blank value in the variable #[User::FileNameInput] and this caused the SSIS package to assume that the value of this variable will always be of zero length in the Derived Column transformation.
Change the expression on the Derived column transformation from #[User::FileNameInput] to (DT_STR, 2000, 1252)#[User::FileNameInput].
Type casting the derived column to 2000 sets the column length to that maximum value. The value 1252 represents the code page. I assumed that you are using ANSI code page. I took the value 2000 from your table definition because the FilePath column had variable VARCHAR(2000). If the column data type had been NVARCHAR(2000), then the expression would be (DT_WSTR, 2000)#[User::FileNameInput]
Tim,
You're using the wrong variable in your Derived Column component. You are storing the filename in #[User::CurrentFile] but the variable that you're using in your Derived Column component is #[User::FileNameInput]
Change your Derived Column component to use #[User::CurrentFile] and you'll be good.
Hope that helps.
JT
If you are using a ForEach loop to process the files in a folder then I have have used the technique described in SSIS Junkie's blog to get the filename in to an SSIS variable: SSIS: Enumerating files in a Foreach loop
You can use the variable later in your flow to write it to the database.
TO all intents and purposes your method #1 should work. That's exactly how I would attempt to do it. I am baffled as to why it is not working. Could you perhaps share your package?
Tony, thanks very much for the link. Much appreciated.
Regards
Jamie
I have a simple String variable with the following value: "C:\Test.txt".
Now I would like to edit the variable to point to a different file.
I cannot find a way to do that. I can change the Name, Data Type, but not the value itself!
Do I need to delete the variable and create the new one?
Update: The problem was caused by "ReadOnly" property set to "True". For typical scenarios, see the accepted answer below.
As #Yuck and #devarc have noted, there are two different and distinct values a Variable holds. The Design-time value is the value you assign when the variable is first created. In your case, the variable holds C:\Test.txt as the design-time value. Everytime you open the package, it would show C:\Test.txt until you change it in the
To make the value of a variable change while the package is running, your options are either to set the value or calculate it. Here I have created a package-level variable CurrentFile with the value of C:\Test.txt
One thing that often trips people up is that they have correctly changed the run-time value but when they run it in BIDS, they see the "old" value. The value displayed in the Variables window does not change during package execution.
During package execution, my Variables window still shows the design-time value (C:\Test.txt) but the true value is reflected in the Locals window (C:\Test2.txt)
Setting a value
The value of most anything in SSIS can be established at run-time through a set of verbose command-line options or through configuration sources. The biggest difference in my mind is that this approach is that the value will always be the value for the entire lifetime of package execution. Sequential or parallel invocations of a package can change that value but for that execution the value would remain constant (barring an explicit modification of the value.
/SET
Command-line execution (dtexec.exe), right clicking on a package and running from the filesystem (dtexecUI.exe) or creating a SQL Agent job step of SQL Server Integration Services all allow for providing a run-time value through the SET command. Using the above variable, the following command would set the run-time value to C:\Test2.txt
dtexec /file C:\Generated.dtsx /set \Package.Variables[User::CurrentFile].Properties[Value];"C:\Test2.txt"
Configuration
SSIS offers an option to create configuration sources to provide run-time values to packages. The article I linked to above does a much better job describing the pros and cons of the configuration options than I will do here. I will say that I typically use both - my SET command configures a connection manager which is then used by the package to find the "full" set of package configurations.
Calculating a value
There are a variety of tasks in SSIS that can change the value of a variable as well as the use of Expressions to change a value. I see these as things that operate on value whilst the package is in flight.
Tasks
A Script Task is one of the most commonly used mechanisms for those starting out but I find other tools in the SSIS toolkit usually better suited for changing variable values.
Foreach Loop Container and Execute SQL Task are two of the other big Tasks you should look at for assignment of a variable value.
Expressions
Expressions are the most glorious candy in the SSIS toolbox. Most every "thing" in SSIS exposes properties for configuration. That's helpful, but using assigning an expression to build those properties is outstanding.
For example, imagine 3 variables RootFolder, FileName and ComputedCurrentFile with values of C:\, File2.txt and empty string. On the Properties window for ComputedCurrentFile we'd change the value for EvaluateAsExpression from False to True and then use an expression like #[User::RootFolder]+ "\\" +#[User::FileName] That simply concatenates the value the first two variables together. This can be helpful if the file name for processing was standard but the source folder changed often. Or if we're talking about output, it's common to use expressions to build an output file name using the date and possibly time of when the package is running.
Finally, there is nothing that prevents a mixing and matching of these approaches. I typically use a configuration to point a file enumerator at the correct starting folder and then use calculated values to identify the current file for processing.
If you want to change it in designer just right click on free space and --> Variables.
But if you want to change it at runtime I suggest you to:
create script task
choose language
add your variable to ReadWriteVariables.
Edit script.
For example in VB:
Dts.Variables("myVariable").Value = #"C:\Test2.txt";
Dts.TaskResult = ScriptResults.Success
Found an easy way to handle this. Remove the Variable from Expression which will enable Value Box to edit. Once it is edited, add the Variable back in the Expression should get the updated value. Hope this helps.
I was also facing the same issue like you where once the variable is declared and define (for eg:var1=text1.csv)in SSIS Variable window I was not able to update the variable value(for eg: var1=text2.csv) in SSIS Variable Window by clicking on the variable value field.
Applied below fix:-
I noticed that I was using var1 variable as a Expression by using expression builder so to update the value(for eg:-var1=text2.csv) I used expression builder window.once you done using the expression builder,you can see the text2.csv is got mapped to var1.