I am getting a datetime column from SqlServer (SS). I want it to appear as a "String" column to SSIS so that I can directly fit it into an SSIS string. After that, I want to loop over this column with a Foreach loop and assign each row to a SSIS string called "str_SSIS". So, I tried CONVERT(VARCHAR(50), MyDatetimeColumn, 121) and dumped it into a result set.
But, SSIS complains that the type of the vaue being assigned to variable str_SSIS differs from the current variable type.
Now, I also get the error - Error: ForEach Variable Mapping number 1 to variable "strMyString" cannot be applied.
Why could this be happening and how do I fix it ?
Thanks.
You need to update the source in SSIS because SSIS persists data types, so if it's changed, the SSIS doesn't know and there's the error you have.
if i understood this correctly you have an execute sql task which outputs multiple records with string date values and you want to iterate through the values to use as filter for your dataflow source task,to do this you need 2 variables,an object type variable lets say obj_ssis and second string variable str_ssis.step1:in execute sql task change the result set to full result set and in resultset tab add the object variable.2)in foreach loop select Foreach ADO Enumerator and in ADO object source variable select "obj_SSIS" then in variable mapping select str_SSIS string variable and now you should have required values in the string value
Related
I have a Variable User:AllBatches (ObJect) populated from an ADO.NET SQL Query. The SQL query returns a single column of Integers.
That part of my package is working.
The next step is a ForEach Loop using the User:AllBatches as the variable enumerator.
In the variable mapping section choose a variable to map to the collection variable index 0. However if I choose any variable type other than Object, the package errors.
If I choose an Object variable then my SQL Task fails with Unsupported Data Type on Parameter binding and I can't then get the value of the Object to use in the SQL Task inside the loop.
If I set it to be an Int32 then I get The Tyoe of the Variable (DBNULL) differs from the current variable type (INT32).
SO if the variable collection mapping will only support an Object Type how do I get the INTEGER value from that output object variable.
You don't have to use a Variable enumerator, you have to use ADO enumerator instead, and select to loop over rows in the first table. Check the following example:
Looping Through a Result Set with the ForEach Loop
Note that the Execute SQL Task result is stored as an ADO Recordset within the Object variable. On the other hand, variable enumerator is used to loop over an object variable that contains a list or an array generated from a script.
You can read more about enumerator types in the following link:
Foreach Loop Container
I'd like to define start_date and end_date parameters in my SSIS package, and have a foreach container that runs for each date in between these 2 (inclusive), which executes a SQL query taking in the current date value (ie starting at start_date) and using it as a parameter for the query.
I'm quite new to SSIS programming and I cannot find information on how to do this.
You can simply add a for loop container and use these variables as mentioned in the image below:
Where #[User:Loop], #[User:MinDate], #[User::MaxDate] are of type System.DateTime
image reference
How do I loop through date values stored as numbers within For Loop container?
Passing parameters to Execute SQL Task
You can refer to the following posts for more details:
Passing Variables to and from an SSIS task
How to pass variable as a parameter in Execute SQL Task SSIS?
A For Loop would be the better option to do this. Assuming that the start and end dates as supplied as parameters to the package as indicated in your question, be aware that parameters cannot be updated in an SSIS package however variables can be. This, as well as an example of the process outlined in your question, is further detailed below.
Create an SSIS datetime variable. As mentioned earlier, this will be used to store in initial value of the start date parameter.
Next add a For Loop on the Control Flow. In the screenshot below, the variable #[User::vStartDate] is set to the same value as the package parameter #[$Package::pStartDate] in the InitExpression on the For Loop. Iterations of the loop continue while the start date variable is less than/equal to the end date parameter, which is specified in the EvalExpression field.
After the Execute SQL Task (or however the SQL query is executed) add a Script Task. This will increment the value of the start date variable, so make sure this is the last task in the loop. An example C# script is below, which simply sets the value of the start date SSIS variable to a C# variable, increments the C# variable by one day, then writes that value back to the SSIS variable. Make sure to add the SSIS start date variable in the ReadWriteVariables field on the Script Task. This will go in the Main method of the script as follows. Although there’s just an increment of the date and update of the variable done in the Script Task, having this in place will allow for easier sustainability in the long term in case more logic needs to be added to this as C# provides much more functionality.
Script Task:
public void Main()
{
//get value in current iteration of loop
DateTime currentIterationValue = Convert.ToDateTime(Dts.Variables["User::vStartDate"].Value);
//increment by one day
currentIterationValue = currentIterationValue.AddDays(1);
//update SSIS variable
Dts.Variables["User::vStartDate"].Value = currentIterationValue;
Dts.TaskResult = (int)ScriptResults.Success;
}
I used an Execute SQL Task to store the dates (results) as a Result Set in a user defined variable. Then, inside the foreach loop container, I used the foreach ADO Enumerator on the user defined variable which has the set of dates. Using the variable mapping in the foreach loop container, you can map the start_date and end_dates from the user defined variable and pass it to other variables.
For example:
I have a SELECT statement which selects 2 rows with columns start_date and end_date. This will be stored as a result set in a variable called "main_dates". The foreach ADO Enumerator will enumerate on this "main_dates" variable (for each row in main_dates run the for loop). Then in the Variable Mapping section, you can create 2 new variables called u_start_date and u_end_date and map the columns 0 and 1 to these variables.
Inside the foreach loop whenever you execute a stored procedure, you can pass the u_start_date and u_end_date variables as parameters.
I got stuck with SSIS. When I try to use a foreach container to loop through one recordset I extract from an Excel file.
Here are two variables I used:
The first thing I do is to extract data from an Excel file, then store it into variable "CriteriaResult" as a recordset.
Then in the foreach loop, I use an ADO Enumerator to access this recordset:
Each record in this recordset will be converted to a string variable "CriteriaID":
Then I will insert this record to a table in SQL Server:
The parameter setting is the same as the column's data type in the SQL Server table:
However, I got an error at the foreach loop step:
Error: 0xC001F009 at Get excel data: The type of the value being assigned to variable "User::CriteriaID" differs from the current variable type. Variables may not change type during execution. Variable types are strict, except for variables of type Object.
The raw data in the Excel file is like this:
It starts with number 1 and 2, then change to 3A, 3B.... I guess that's why it claims the data type changes at a foreach loop step. Could I add any script task to format data within the recordset before looping through it?
The other potential issue could come from the merging cells. However, I tried a single cell before, and it works well.
How can I fix this problem?
Try using expressions instead of passing parameters:
Don't specify any parameter
Go to the expression tab → SqlStatement property
Use the following expression:
"INSERT INTO dbo.SSIS (CriteriaID) VALUES(" + #[User::CriteriaID] + ")"
I am trying to get CSV IDs from a table from sql server and assign the result to a variable. below is the sql I have put inside the Execute SQL Task
set nocount on
declare #csv varchar(max) = ''
select #csv = #csv + cast(companyid as varchar(10)) + ',' from company where isprocessed = 0
select substring(#csv,1,len(#csv) - 1) as companyids
As you can see its a simple and standard way of getting csv of a field in t-sql. It works perfectly in query window but throwing below error when I run the Task in SSIS 2012
[Execute SQL Task] Error: The value type (__ComObject) can only be
converted to variables of type Object.
[Execute SQL Task] Error: An error occurred while assigning a value to
variable "sCSVCompanyIds": "The type of the value (DBNull) being
assigned to variable "User::sCSVCompanyIds" differs from the current
variable type (String). Variables may not change type during
execution. Variable types are strict, except for variables of type
Object. ".
Below are the settings of Execute SQL Task
In General tab, Resultset project is set to single row
In Result Set tab, Result Name is set to 0 (I also tried by setting it to csvids which is the alias column name in the select list) and Variable Name is set to User::sCSVCompanyIds
I have no clue why its not working. After wasting so much time I am worked out a hard way which by returning the result as Full Row set (same SQL which returns 1 row 1 column always) and add a for each loop container to loop throw the result set (which will always iterates only once for obvious reasons) and assign the result set's fields to the variable. It works for me but there should be easy way of doing it. What I am missing?
The problem is with the nvarchar(max) data type. I assume it will be same for varchar(max) also. Though relevant data types in SSIS are DT_NTEXT and DT_TEXT are present for some reason we are getting this error.
There are multiple options to handle this.
One Of course there might be more appropriate way to handle this by cast/converting the column within the query to fixed length instead of max something like cast(myvarcharmaxfield as varchar(8000)). In my case it doesn't work because I expect bigger length string. I am generating a csv string from a unique identifier column which itself is 36 chars long string and needs 3 extra chars for quoting them with signle quote and a comma as separator which will support only 205 values. So it doesn't work for me.
So I left with no option but to stick the way I implemented already which is in my question
After wasting so much time I am worked out a hard way which by
returning the result as Full Row set (same SQL which returns 1 row 1
column always) and add a for each loop container to loop throw the
result set (which will always iterates only once for obvious reasons)
and assign the result set's fields to the variable
The new way I learnt is from an unaccepted answer from this question
Change the Oledb connection to ADO.NET connection
I think this is fair enough but it requires me to create another connection manager (so that I don't have to change all existing tasks) and use it for this type of taks where I need csv but I did not buy it as I have very little time to doing research on creating connection string of it which appears to me with the type of erros I am getting is different from oledb.
I'm learning SSIS and this seems like an easy task but I'm stuck.
I have a CSV file Orders.csv with this data:
ProductId,Quantity,CustomerId
1,1,104
2,1,105
3,2,106
I also have a stored procedure ssis_createorder that takes as input parameters:
#productid int
#quantity int
#customerid int
What I want to do is create an SSIS package that takes the .csv file as input and calls ssis_createorder three times for each row in the .csv file (the first row contains column names).
Here is what I have done so far.
I have created an SSIS package (Visual Studio 2005 & SQL Server 2005).
In Control Flow I have a Data Flow Task.
The Data Flow has a Flat File source of my .csv file. All of of the columns are mapped.
I have created a variable named orders of type Object. I also have variables CustomerId, ProductId, & Quantity of type int32.
Next I have a Recordset Destination that is assigning the contents of the .csv file into the varialbe orders. I'm not sure about how to use this tool. I'm setting the VariableName (under Customer Properties) to User::orders. I think that now orders holds an ADO record set made up of the contents from the original .csv file.
Next I'm adding a ForEach Loop Container on the Control Flow tag and linking it to the Data Flow Task.
Inside of the ForEach Loop Container I'm setting the Enumerator to "ForEach ADO Enumerator". I'm setting "ADO object source variable" to User::orders". For Enumeration mode I'm selecting "Rows in the first table".
In the Variable Mapping tab I have User::ProductId index 0, User::Quantity index 1, User::CustomerId index 2. I'm not sure if this is correct.
Next I have a Script Task inside of the ForEach Loop Container.
I have ReadOnlyVariables set to ProductId.
In the Main method this is what I'm doing:
Dim sProductId As String = Dts.Variables("ProductId").Value.ToString
MsgBox("sProductId")
When I run the package my ForEach Loop Container turns Bright Red and I get the following error messages
Error: 0xC001F009 at MasterTest: The type of the value being assigned to variable "User::ProductId" differs from the current variable type. Variables may not change type during execution. Variable types are strict, except for variables of type Object.
Error: 0xC001C012 at Foreach Loop Container: ForEach Variable Mapping number 1 to variable "User::ProductId" cannot be applied.
Error: 0xC001F009 at MasterTest: The type of the value being assigned to variable "User::Quantity" differs from the current variable type. Variables may not change type during execution. Variable types are strict, except for variables of type Object.
Error: 0xC001C012 at Foreach Loop Container: ForEach Variable Mapping number 2 to variable "User::Quantity" cannot be applied.
Error: 0xC001F009 at MasterTest: The type of the value being assigned to variable "User::CustomerId" differs from the current variable type. Variables may not change type during execution. Variable types are strict, except for variables of type Object.
Error: 0xC001C012 at Foreach Loop Container: ForEach Variable Mapping number 3 to variable "User::CustomerId" cannot be applied.
Warning: 0x80019002 at MasterTest: SSIS Warning Code DTS_W_MAXIMUMERRORCOUNTREACHED. The Execution method succeeded, but the number of errors raised (12) reached the maximum allowed (1); resulting in failure. This occurs when the number of errors reaches the number specified in MaximumErrorCount. Change the MaximumErrorCount or fix the errors.
SSIS package "Package.dtsx" finished: Failure.
Dts.TaskResult = Dts.Results.Success
Any help would be appreciated
One of my coworkers just give me the answer.
You don't need the the ForEach Loop Container or the RecordSet Container.
All you need is the Flat File Source and an OLE DB Command. Connect to your database and inside the OLE DB Command select the appropriate connection.
In the Component Properties enter the following SQLCommand:
exec ssis_createorder ?, ?, ?
The "?" are place holders for the parameters.
Next under the Column Mappings tab map the .csv file columns to the stored procedure parameters.
You are finished go ahead and run the package.
Thanks Gary if you were on StackOverFlow I would give you an upvote and accept your answer.
If I understand correctly, what you want to do is execute a stored procedure 3 times for each row in the data source.
What if you just create a data flow with a flat file data source and pipe the data through 3 execute sql command tasks? Just map the columns in the data to the input params of your stored procedure.
Maybe I'm not seeing it correctly in your question and I'm thinking too simple, but in my experience you need to avoid using the foreach task in SSIS as much as possible.
I suspect that you need to look at your Data Flow task. It's likely that the values from the source CSV file are being interpreted as string values. You will probably need a Derived Column component or a Data Conversion component to convert your input values to the desired data type.
And, I think #StephaneT's solution would be good for executing the SP.
I'm not sure if this answers your question. But I was looking to do this and I achieved it using the BULK INSERT command. I created a staging table with all of the columns in the csv file, and instead of a stored procedure I used a INSTEAD OF INSERT trigger to handle the logic of inserting it into many tables.