I have package variable (TableName), its value is set by for each loop container.
For instance TableName = sales set by for each loop after looping through informationschema.tables then I want to use TableName's value in dataflow for this I am using script component.
When I use script component to read TableName's value in script-file I cannot see variable any idea why??
I am using script component as source and TableName is in readonlyvariables.
Please help as I am new to SSIS.
On the Variable Mappings page of the Foreach Loop Editor, assign variables to each item of data that is returned by a single enumerated item. For example, a Foreach File enumerator returns only a file name at Index 0 and therefore requires only one variable mapping, whereas an enumerator that returns several columns of data in each row requires you to map a different variable to each column that you want to use in the Script task. +
After you have mapped enumerated items to variables, then you must add the mapped variables to the ReadOnlyVariables property on the Script page of the Script Task Editor to make them available to your script
For more details pls chk the below link
https://learn.microsoft.com/en-us/sql/integration-services/extending-packages-scripting/task/using-variables-in-the-script-task
https://social.technet.microsoft.com/wiki/contents/articles/22194.use-ssis-variables-and-parameters-in-a-script-task.aspx
Please use the following Microsoft Documentation link - it describes differences between Script Task and Script Component, and gives small code samples, specifically on using Variables.
as per this I need to use this.variables.myvariable
Thanks everyone.
Related
I need to execute a package once for every item in flat file. The package's parameters will be a few of the current item's attributes. What is the best way to proceed? I'm assuming the last thing I do is place an Execute Package Task component inside of a Foreach Loop Container like below:
Simple Solution
First of all you have to create a Package variable of type Object
Add a script task that loop over flat file and convert it to a DataTable or List(of T) and store it into the Object variable.
Connect the Script Task to the foreach loop you are using
In the foreach loop container select the enumeration type to Ado enumerator and select the Object variable as Source and map the columns you want to use in the execute package task to variables
Reading Flat File into DataTable
you can refer to one of the following links to learn more:
How to Read text file to DataTable
DataTable from TextFile?
Efficient function for reading a delimited file into DataTable
Side Note: If you need more details, give me a reply
Each item had three attributes: username, password, and exactAccountName
These were then used to execute an SSIS package with each login credential.
In order to achieve this, do the following:
Create a package variable of type Object in the master package.
Add a script task that loop over flat file and convert it to a DataTable or List(of T) and store it into the Object variable. (This step is from Hadi's answer)
Create a foreach loop to execute next.
In the foreach loop container select the enumeration type to Ado enumerator and select the Object variable as Source. (Also from Hadi's answer)
Put a Script Task and Execute Package Task in the foreach loop. Like so:
Create three string variables in the master package for username, password, and exactAccountName.
Populate the main function of the Script Task with code like this:
Set the parameters of the child package to be the variables specified by the Script Task and you're done!
I have 1 .csv file that I import. Because of the layout of the file I need to import it again to get a certain value that needs to be used for the next Data Flow task that will use that value for every row.
Below is an example of the file:
I need STATEMENT DATE for all records.
This is how the package looks like:
Inside Get Statement Date task I have the following Data Flow:
I don't really want to export the column I just want it to go into a variable that I can use in the next task And I am not sure were this should be done.
I am still learning how SSIS works.
Any help would be much appreciated.
Use a script component as Destination and assign that column to your variable.
Keep in mind the variable Datatype should be compatible.
Declare a variable in class ScriptMain to store your value:
public DateTime t;
On Input0_ProcessInputRow(Input0Buffer Row):
t = Row.StatementDate;
On PostExecute:
Variables.teste = t;
It will assign it the number of rows you actually have.(it's not a problem since you are only retrieving one row I guess). Only the last row will actually be saved to your variable.
Use RecordSet Destination. This can be used to store temporary results which will be stored in a variable of type object. You can use this along with Foreach loop to process each record. you can refer this link for more info.
RecordSet Destination
I'm doing an Excel loop through fifty or more Excel files. The loop goes through each Excel file, grabs all the data and inputs it into the database without error. This is the typical process of setting delay validation to true, and making sure that the expression for the Excel Connection is a string variable called EFile that is set to nothing (in the loop).
What is not working: trying to input the name of the Excel file into the database.
What's been tried (edit; SO changed my 2 to 1 - don't know why):
Add a derived column between the Excel file and database input, and add a column using the EFile expression (so under Expression in the Derived Column it would be #[User::EFile]). and add the empty. However, this inputs nothing a blank (nothing).
One suggestion was to add ANOTHER string variable and set its properties EvaluateAsExpression to True and set the Expression to the EFile variable (#[User::EFile]). The funny thing is that this does the same thing - inputs a blank into the database.
Numerous people on blogs claim they can do this, yet I haven't seen one actually address this (I have a blog and I will definitely be showing people how to do this when I get an answer because, so far, these others have fallen short). How do I grab an Excel file's name and input it in a database during a loop?
Added: Forgot to add, no scripts; the claim is that it can be done without them, so I want to see the solution without them.
Note: I already have the ability to import the data from the Excel files - that's easy (see my GitHub account, as I have two different projects for importing all sorts of txt, csv, xls, xlsx data). I am trying to also get the actual name of the file being imported also into the database. So, if there are fifty Excel files, along with the data in each file, the database will have the fifty file names alongside that data (so if each file has 1000 rows of data, each 1000 rows would also have the name of the file they came from next to them as an additional column). This point seems to cause a lot of confusion, as people assume I'm having trouble importing data in files - NOPE, see my GitHub; again that's easy. It's the FILENAME that needs to also be imported.
Test package: https://github.com/tmmtsmith/SSISLoopWithFileName
Solution: #jaimet pointed out that the Derived Column needed to be the #[User::CurrentFile] (see the test package). When I first ran the package, I still got a blank value in my database. But when we originally set up the connection, we do point it to an actual file (I call this "fooling the package"), then change the expression on the connecting later to the #[User::CurrentFile], which is blank. The Derived Column, using the variable #[User::CurrentFile], showed a string of 0. So, I removed the Derived Column, put the full file path and name in the variable, then added the variable to the Derived Column (which made it think the string was 91 characters long), then went back and set the variable to nothing (English teacher would hate the THENs about right now). When I ran the package, it inputted the full file path. Maybe, like the connection, it needs to initially think that a file exists in order for it to input the full amount of characters?
Appreciate all the help.
The issue is because of blank value in the variable #[User::FileNameInput] and this caused the SSIS package to assume that the value of this variable will always be of zero length in the Derived Column transformation.
Change the expression on the Derived column transformation from #[User::FileNameInput] to (DT_STR, 2000, 1252)#[User::FileNameInput].
Type casting the derived column to 2000 sets the column length to that maximum value. The value 1252 represents the code page. I assumed that you are using ANSI code page. I took the value 2000 from your table definition because the FilePath column had variable VARCHAR(2000). If the column data type had been NVARCHAR(2000), then the expression would be (DT_WSTR, 2000)#[User::FileNameInput]
Tim,
You're using the wrong variable in your Derived Column component. You are storing the filename in #[User::CurrentFile] but the variable that you're using in your Derived Column component is #[User::FileNameInput]
Change your Derived Column component to use #[User::CurrentFile] and you'll be good.
Hope that helps.
JT
If you are using a ForEach loop to process the files in a folder then I have have used the technique described in SSIS Junkie's blog to get the filename in to an SSIS variable: SSIS: Enumerating files in a Foreach loop
You can use the variable later in your flow to write it to the database.
TO all intents and purposes your method #1 should work. That's exactly how I would attempt to do it. I am baffled as to why it is not working. Could you perhaps share your package?
Tony, thanks very much for the link. Much appreciated.
Regards
Jamie
I have a simple String variable with the following value: "C:\Test.txt".
Now I would like to edit the variable to point to a different file.
I cannot find a way to do that. I can change the Name, Data Type, but not the value itself!
Do I need to delete the variable and create the new one?
Update: The problem was caused by "ReadOnly" property set to "True". For typical scenarios, see the accepted answer below.
As #Yuck and #devarc have noted, there are two different and distinct values a Variable holds. The Design-time value is the value you assign when the variable is first created. In your case, the variable holds C:\Test.txt as the design-time value. Everytime you open the package, it would show C:\Test.txt until you change it in the
To make the value of a variable change while the package is running, your options are either to set the value or calculate it. Here I have created a package-level variable CurrentFile with the value of C:\Test.txt
One thing that often trips people up is that they have correctly changed the run-time value but when they run it in BIDS, they see the "old" value. The value displayed in the Variables window does not change during package execution.
During package execution, my Variables window still shows the design-time value (C:\Test.txt) but the true value is reflected in the Locals window (C:\Test2.txt)
Setting a value
The value of most anything in SSIS can be established at run-time through a set of verbose command-line options or through configuration sources. The biggest difference in my mind is that this approach is that the value will always be the value for the entire lifetime of package execution. Sequential or parallel invocations of a package can change that value but for that execution the value would remain constant (barring an explicit modification of the value.
/SET
Command-line execution (dtexec.exe), right clicking on a package and running from the filesystem (dtexecUI.exe) or creating a SQL Agent job step of SQL Server Integration Services all allow for providing a run-time value through the SET command. Using the above variable, the following command would set the run-time value to C:\Test2.txt
dtexec /file C:\Generated.dtsx /set \Package.Variables[User::CurrentFile].Properties[Value];"C:\Test2.txt"
Configuration
SSIS offers an option to create configuration sources to provide run-time values to packages. The article I linked to above does a much better job describing the pros and cons of the configuration options than I will do here. I will say that I typically use both - my SET command configures a connection manager which is then used by the package to find the "full" set of package configurations.
Calculating a value
There are a variety of tasks in SSIS that can change the value of a variable as well as the use of Expressions to change a value. I see these as things that operate on value whilst the package is in flight.
Tasks
A Script Task is one of the most commonly used mechanisms for those starting out but I find other tools in the SSIS toolkit usually better suited for changing variable values.
Foreach Loop Container and Execute SQL Task are two of the other big Tasks you should look at for assignment of a variable value.
Expressions
Expressions are the most glorious candy in the SSIS toolbox. Most every "thing" in SSIS exposes properties for configuration. That's helpful, but using assigning an expression to build those properties is outstanding.
For example, imagine 3 variables RootFolder, FileName and ComputedCurrentFile with values of C:\, File2.txt and empty string. On the Properties window for ComputedCurrentFile we'd change the value for EvaluateAsExpression from False to True and then use an expression like #[User::RootFolder]+ "\\" +#[User::FileName] That simply concatenates the value the first two variables together. This can be helpful if the file name for processing was standard but the source folder changed often. Or if we're talking about output, it's common to use expressions to build an output file name using the date and possibly time of when the package is running.
Finally, there is nothing that prevents a mixing and matching of these approaches. I typically use a configuration to point a file enumerator at the correct starting folder and then use calculated values to identify the current file for processing.
If you want to change it in designer just right click on free space and --> Variables.
But if you want to change it at runtime I suggest you to:
create script task
choose language
add your variable to ReadWriteVariables.
Edit script.
For example in VB:
Dts.Variables("myVariable").Value = #"C:\Test2.txt";
Dts.TaskResult = ScriptResults.Success
Found an easy way to handle this. Remove the Variable from Expression which will enable Value Box to edit. Once it is edited, add the Variable back in the Expression should get the updated value. Hope this helps.
I was also facing the same issue like you where once the variable is declared and define (for eg:var1=text1.csv)in SSIS Variable window I was not able to update the variable value(for eg: var1=text2.csv) in SSIS Variable Window by clicking on the variable value field.
Applied below fix:-
I noticed that I was using var1 variable as a Expression by using expression builder so to update the value(for eg:-var1=text2.csv) I used expression builder window.once you done using the expression builder,you can see the text2.csv is got mapped to var1.
I'm learning SSIS and this seems like an easy task but I'm stuck.
I have a CSV file Orders.csv with this data:
ProductId,Quantity,CustomerId
1,1,104
2,1,105
3,2,106
I also have a stored procedure ssis_createorder that takes as input parameters:
#productid int
#quantity int
#customerid int
What I want to do is create an SSIS package that takes the .csv file as input and calls ssis_createorder three times for each row in the .csv file (the first row contains column names).
Here is what I have done so far.
I have created an SSIS package (Visual Studio 2005 & SQL Server 2005).
In Control Flow I have a Data Flow Task.
The Data Flow has a Flat File source of my .csv file. All of of the columns are mapped.
I have created a variable named orders of type Object. I also have variables CustomerId, ProductId, & Quantity of type int32.
Next I have a Recordset Destination that is assigning the contents of the .csv file into the varialbe orders. I'm not sure about how to use this tool. I'm setting the VariableName (under Customer Properties) to User::orders. I think that now orders holds an ADO record set made up of the contents from the original .csv file.
Next I'm adding a ForEach Loop Container on the Control Flow tag and linking it to the Data Flow Task.
Inside of the ForEach Loop Container I'm setting the Enumerator to "ForEach ADO Enumerator". I'm setting "ADO object source variable" to User::orders". For Enumeration mode I'm selecting "Rows in the first table".
In the Variable Mapping tab I have User::ProductId index 0, User::Quantity index 1, User::CustomerId index 2. I'm not sure if this is correct.
Next I have a Script Task inside of the ForEach Loop Container.
I have ReadOnlyVariables set to ProductId.
In the Main method this is what I'm doing:
Dim sProductId As String = Dts.Variables("ProductId").Value.ToString
MsgBox("sProductId")
When I run the package my ForEach Loop Container turns Bright Red and I get the following error messages
Error: 0xC001F009 at MasterTest: The type of the value being assigned to variable "User::ProductId" differs from the current variable type. Variables may not change type during execution. Variable types are strict, except for variables of type Object.
Error: 0xC001C012 at Foreach Loop Container: ForEach Variable Mapping number 1 to variable "User::ProductId" cannot be applied.
Error: 0xC001F009 at MasterTest: The type of the value being assigned to variable "User::Quantity" differs from the current variable type. Variables may not change type during execution. Variable types are strict, except for variables of type Object.
Error: 0xC001C012 at Foreach Loop Container: ForEach Variable Mapping number 2 to variable "User::Quantity" cannot be applied.
Error: 0xC001F009 at MasterTest: The type of the value being assigned to variable "User::CustomerId" differs from the current variable type. Variables may not change type during execution. Variable types are strict, except for variables of type Object.
Error: 0xC001C012 at Foreach Loop Container: ForEach Variable Mapping number 3 to variable "User::CustomerId" cannot be applied.
Warning: 0x80019002 at MasterTest: SSIS Warning Code DTS_W_MAXIMUMERRORCOUNTREACHED. The Execution method succeeded, but the number of errors raised (12) reached the maximum allowed (1); resulting in failure. This occurs when the number of errors reaches the number specified in MaximumErrorCount. Change the MaximumErrorCount or fix the errors.
SSIS package "Package.dtsx" finished: Failure.
Dts.TaskResult = Dts.Results.Success
Any help would be appreciated
One of my coworkers just give me the answer.
You don't need the the ForEach Loop Container or the RecordSet Container.
All you need is the Flat File Source and an OLE DB Command. Connect to your database and inside the OLE DB Command select the appropriate connection.
In the Component Properties enter the following SQLCommand:
exec ssis_createorder ?, ?, ?
The "?" are place holders for the parameters.
Next under the Column Mappings tab map the .csv file columns to the stored procedure parameters.
You are finished go ahead and run the package.
Thanks Gary if you were on StackOverFlow I would give you an upvote and accept your answer.
If I understand correctly, what you want to do is execute a stored procedure 3 times for each row in the data source.
What if you just create a data flow with a flat file data source and pipe the data through 3 execute sql command tasks? Just map the columns in the data to the input params of your stored procedure.
Maybe I'm not seeing it correctly in your question and I'm thinking too simple, but in my experience you need to avoid using the foreach task in SSIS as much as possible.
I suspect that you need to look at your Data Flow task. It's likely that the values from the source CSV file are being interpreted as string values. You will probably need a Derived Column component or a Data Conversion component to convert your input values to the desired data type.
And, I think #StephaneT's solution would be good for executing the SP.
I'm not sure if this answers your question. But I was looking to do this and I achieved it using the BULK INSERT command. I created a staging table with all of the columns in the csv file, and instead of a stored procedure I used a INSTEAD OF INSERT trigger to handle the logic of inserting it into many tables.