I need to read a column on a db on ADF and use all it´s values as parameters in a foreach.
I tried reading the column using a dataflow and a cache sink to then in a pipeline use Set Variable and then the foreach...but instead of an array of values I get an array with one value that contains all the others I want (but i cant iterate over)
I am using:
#array(activity('myDataflow').output.runStatus.output.columName
Any help is appreciated, seems simple enough (column to array) but I am stuck
Use a lookup activity to get the data from SQL server, and run for each loop on the output of the lookup.
Example:
Create a new pipeline
Add a lookup activity
Choose your source dataset (in this example, an Azure SQL database)
Remove the checkbox from “First row only”
Choose a table, stored procedure or type in a query
SELECT 1 AS result UNION ALL
SELECT 2 AS result UNION ALL
SELECT 3 AS result UNION ALL
SELECT 4 AS result
Add a foreach activity
In the foreach activity, under settings tab: “Items” -
#activity('Lookup SQL query').output.value – where 'Lookup SQL
query' is the name of the lookup activity
Inside the foreach loop, add a wait activity
In the settings tab, “Wait time in seconds” : #item().result .
item() is the current loop, and result is the name of the SQL column
debug the pipeline. You can see that the foreach activity iterates 4 times, for every row returned from the sql query.
You can use append variable activity also, inside ForEach after lookup.
First create an array variable in the pipeline.
Then use append variable activity inside ForEach and give
#item.<your_column_name>
Result variable stored in a sample variable:
Result:
I tried reading the column using a dataflow and a cache sink to then in a pipeline use Set Variable and then the foreach
If you want to do it with dataflows instead of lookup, use the same above procedure and give the below dynamic content in the ForEach.
#activity('Data flow1').output.runStatus.output.sink1.value
Related
I'd like to define start_date and end_date parameters in my SSIS package, and have a foreach container that runs for each date in between these 2 (inclusive), which executes a SQL query taking in the current date value (ie starting at start_date) and using it as a parameter for the query.
I'm quite new to SSIS programming and I cannot find information on how to do this.
You can simply add a for loop container and use these variables as mentioned in the image below:
Where #[User:Loop], #[User:MinDate], #[User::MaxDate] are of type System.DateTime
image reference
How do I loop through date values stored as numbers within For Loop container?
Passing parameters to Execute SQL Task
You can refer to the following posts for more details:
Passing Variables to and from an SSIS task
How to pass variable as a parameter in Execute SQL Task SSIS?
A For Loop would be the better option to do this. Assuming that the start and end dates as supplied as parameters to the package as indicated in your question, be aware that parameters cannot be updated in an SSIS package however variables can be. This, as well as an example of the process outlined in your question, is further detailed below.
Create an SSIS datetime variable. As mentioned earlier, this will be used to store in initial value of the start date parameter.
Next add a For Loop on the Control Flow. In the screenshot below, the variable #[User::vStartDate] is set to the same value as the package parameter #[$Package::pStartDate] in the InitExpression on the For Loop. Iterations of the loop continue while the start date variable is less than/equal to the end date parameter, which is specified in the EvalExpression field.
After the Execute SQL Task (or however the SQL query is executed) add a Script Task. This will increment the value of the start date variable, so make sure this is the last task in the loop. An example C# script is below, which simply sets the value of the start date SSIS variable to a C# variable, increments the C# variable by one day, then writes that value back to the SSIS variable. Make sure to add the SSIS start date variable in the ReadWriteVariables field on the Script Task. This will go in the Main method of the script as follows. Although there’s just an increment of the date and update of the variable done in the Script Task, having this in place will allow for easier sustainability in the long term in case more logic needs to be added to this as C# provides much more functionality.
Script Task:
public void Main()
{
//get value in current iteration of loop
DateTime currentIterationValue = Convert.ToDateTime(Dts.Variables["User::vStartDate"].Value);
//increment by one day
currentIterationValue = currentIterationValue.AddDays(1);
//update SSIS variable
Dts.Variables["User::vStartDate"].Value = currentIterationValue;
Dts.TaskResult = (int)ScriptResults.Success;
}
I used an Execute SQL Task to store the dates (results) as a Result Set in a user defined variable. Then, inside the foreach loop container, I used the foreach ADO Enumerator on the user defined variable which has the set of dates. Using the variable mapping in the foreach loop container, you can map the start_date and end_dates from the user defined variable and pass it to other variables.
For example:
I have a SELECT statement which selects 2 rows with columns start_date and end_date. This will be stored as a result set in a variable called "main_dates". The foreach ADO Enumerator will enumerate on this "main_dates" variable (for each row in main_dates run the for loop). Then in the Variable Mapping section, you can create 2 new variables called u_start_date and u_end_date and map the columns 0 and 1 to these variables.
Inside the foreach loop whenever you execute a stored procedure, you can pass the u_start_date and u_end_date variables as parameters.
I got stuck with SSIS. When I try to use a foreach container to loop through one recordset I extract from an Excel file.
Here are two variables I used:
The first thing I do is to extract data from an Excel file, then store it into variable "CriteriaResult" as a recordset.
Then in the foreach loop, I use an ADO Enumerator to access this recordset:
Each record in this recordset will be converted to a string variable "CriteriaID":
Then I will insert this record to a table in SQL Server:
The parameter setting is the same as the column's data type in the SQL Server table:
However, I got an error at the foreach loop step:
Error: 0xC001F009 at Get excel data: The type of the value being assigned to variable "User::CriteriaID" differs from the current variable type. Variables may not change type during execution. Variable types are strict, except for variables of type Object.
The raw data in the Excel file is like this:
It starts with number 1 and 2, then change to 3A, 3B.... I guess that's why it claims the data type changes at a foreach loop step. Could I add any script task to format data within the recordset before looping through it?
The other potential issue could come from the merging cells. However, I tried a single cell before, and it works well.
How can I fix this problem?
Try using expressions instead of passing parameters:
Don't specify any parameter
Go to the expression tab → SqlStatement property
Use the following expression:
"INSERT INTO dbo.SSIS (CriteriaID) VALUES(" + #[User::CriteriaID] + ")"
I have a flat file and used conditional split to filter the record into a single row. For example, RecordType == "2" retrieves single row with record having multiple columns say A,B,C,D and E. I want to pass the result of Column C value to a variable. And then to use it to update the table like:
Update tablename
Set A = that variable
Where A is null
Could you please help me in find out the solution.
I would not use the variable but use a Ole DB Command object.
You set the connection.
Then add your SQL from above:
Update tablename Set A = ? Where A is null
The map to Col C.
However, what I might guess you are trying to do is add a column to your other record set that has the detail but no key.
I would use a script component to do this:
Similar to this example:
Importing Grouped Report Data to Database
I am getting a datetime column from SqlServer (SS). I want it to appear as a "String" column to SSIS so that I can directly fit it into an SSIS string. After that, I want to loop over this column with a Foreach loop and assign each row to a SSIS string called "str_SSIS". So, I tried CONVERT(VARCHAR(50), MyDatetimeColumn, 121) and dumped it into a result set.
But, SSIS complains that the type of the vaue being assigned to variable str_SSIS differs from the current variable type.
Now, I also get the error - Error: ForEach Variable Mapping number 1 to variable "strMyString" cannot be applied.
Why could this be happening and how do I fix it ?
Thanks.
You need to update the source in SSIS because SSIS persists data types, so if it's changed, the SSIS doesn't know and there's the error you have.
if i understood this correctly you have an execute sql task which outputs multiple records with string date values and you want to iterate through the values to use as filter for your dataflow source task,to do this you need 2 variables,an object type variable lets say obj_ssis and second string variable str_ssis.step1:in execute sql task change the result set to full result set and in resultset tab add the object variable.2)in foreach loop select Foreach ADO Enumerator and in ADO object source variable select "obj_SSIS" then in variable mapping select str_SSIS string variable and now you should have required values in the string value
I'm learning SSIS and this seems like an easy task but I'm stuck.
I have a CSV file Orders.csv with this data:
ProductId,Quantity,CustomerId
1,1,104
2,1,105
3,2,106
I also have a stored procedure ssis_createorder that takes as input parameters:
#productid int
#quantity int
#customerid int
What I want to do is create an SSIS package that takes the .csv file as input and calls ssis_createorder three times for each row in the .csv file (the first row contains column names).
Here is what I have done so far.
I have created an SSIS package (Visual Studio 2005 & SQL Server 2005).
In Control Flow I have a Data Flow Task.
The Data Flow has a Flat File source of my .csv file. All of of the columns are mapped.
I have created a variable named orders of type Object. I also have variables CustomerId, ProductId, & Quantity of type int32.
Next I have a Recordset Destination that is assigning the contents of the .csv file into the varialbe orders. I'm not sure about how to use this tool. I'm setting the VariableName (under Customer Properties) to User::orders. I think that now orders holds an ADO record set made up of the contents from the original .csv file.
Next I'm adding a ForEach Loop Container on the Control Flow tag and linking it to the Data Flow Task.
Inside of the ForEach Loop Container I'm setting the Enumerator to "ForEach ADO Enumerator". I'm setting "ADO object source variable" to User::orders". For Enumeration mode I'm selecting "Rows in the first table".
In the Variable Mapping tab I have User::ProductId index 0, User::Quantity index 1, User::CustomerId index 2. I'm not sure if this is correct.
Next I have a Script Task inside of the ForEach Loop Container.
I have ReadOnlyVariables set to ProductId.
In the Main method this is what I'm doing:
Dim sProductId As String = Dts.Variables("ProductId").Value.ToString
MsgBox("sProductId")
When I run the package my ForEach Loop Container turns Bright Red and I get the following error messages
Error: 0xC001F009 at MasterTest: The type of the value being assigned to variable "User::ProductId" differs from the current variable type. Variables may not change type during execution. Variable types are strict, except for variables of type Object.
Error: 0xC001C012 at Foreach Loop Container: ForEach Variable Mapping number 1 to variable "User::ProductId" cannot be applied.
Error: 0xC001F009 at MasterTest: The type of the value being assigned to variable "User::Quantity" differs from the current variable type. Variables may not change type during execution. Variable types are strict, except for variables of type Object.
Error: 0xC001C012 at Foreach Loop Container: ForEach Variable Mapping number 2 to variable "User::Quantity" cannot be applied.
Error: 0xC001F009 at MasterTest: The type of the value being assigned to variable "User::CustomerId" differs from the current variable type. Variables may not change type during execution. Variable types are strict, except for variables of type Object.
Error: 0xC001C012 at Foreach Loop Container: ForEach Variable Mapping number 3 to variable "User::CustomerId" cannot be applied.
Warning: 0x80019002 at MasterTest: SSIS Warning Code DTS_W_MAXIMUMERRORCOUNTREACHED. The Execution method succeeded, but the number of errors raised (12) reached the maximum allowed (1); resulting in failure. This occurs when the number of errors reaches the number specified in MaximumErrorCount. Change the MaximumErrorCount or fix the errors.
SSIS package "Package.dtsx" finished: Failure.
Dts.TaskResult = Dts.Results.Success
Any help would be appreciated
One of my coworkers just give me the answer.
You don't need the the ForEach Loop Container or the RecordSet Container.
All you need is the Flat File Source and an OLE DB Command. Connect to your database and inside the OLE DB Command select the appropriate connection.
In the Component Properties enter the following SQLCommand:
exec ssis_createorder ?, ?, ?
The "?" are place holders for the parameters.
Next under the Column Mappings tab map the .csv file columns to the stored procedure parameters.
You are finished go ahead and run the package.
Thanks Gary if you were on StackOverFlow I would give you an upvote and accept your answer.
If I understand correctly, what you want to do is execute a stored procedure 3 times for each row in the data source.
What if you just create a data flow with a flat file data source and pipe the data through 3 execute sql command tasks? Just map the columns in the data to the input params of your stored procedure.
Maybe I'm not seeing it correctly in your question and I'm thinking too simple, but in my experience you need to avoid using the foreach task in SSIS as much as possible.
I suspect that you need to look at your Data Flow task. It's likely that the values from the source CSV file are being interpreted as string values. You will probably need a Derived Column component or a Data Conversion component to convert your input values to the desired data type.
And, I think #StephaneT's solution would be good for executing the SP.
I'm not sure if this answers your question. But I was looking to do this and I achieved it using the BULK INSERT command. I created a staging table with all of the columns in the csv file, and instead of a stored procedure I used a INSTEAD OF INSERT trigger to handle the logic of inserting it into many tables.