I am querying a Cache database through an ADO net datasource in my dataflow in SSIS (SQL 2008 R2). I want to pass parameters to the query but can only do this through the expressions section of the dataflow item. The query itself is over 4000 characters hence I cant use the query in the expressions section (due to the 4000 character limitation) in its raw form.
I have tried using a script task to assign the query to a string variable [User::Query1] however when I click the Evaluate Expression button in the expression builder screen of the dataflow, it returns nothing. I have the following expression for [ADO NET Source].[SQLCommand]:
#[User::Query1]
When running the package, I get a an error saying that SQL command has not been set correctly. Check SQLCommand Property.
I've set ValidateExternalMetaData to false and I see the following error in the execution results;
Error: The variable User::Query1 contains a string that exceeds the maximum allowed length of 4000 characters.
Error: Reading the variable "User::Query1" failed with error code 0xC0047100.
Error: The expression "#[User::Query1]" on property "[ADO NET Source].[SqlCommand]" cannot be evaluated. Modify the expression to be valid.
In my script task I have the entire query assigned to a string variable and then I am assigning the string to the actual variable using the vb code below:
Dts.Variables("User::Query1").Value = sSql
MessageBox.Show(Dts.Variables("User::Query1").Value.ToString())
On the script task properties, I have [User::Query1] as a ReadWriteVariables. I have also made sure that EvaluateAsExpression for [User::Query1] is set to true.
In essence, I am trying to run the query using the expressions property of the dataflow as this will allow me to use dynamic parameter values.
Also variables are limited to 4000 chars in SSIS, you could split the query in multiple sub queries.
I have found a solution when you use an OLE DB connection. It is complex to prepare but finally it is running OK. Previously, you need to define the variable with a similar and same structure query and <4000 characters to validate the expression. When is all ok, you can redefine the variable with >4000 characters query in a script task.
Next link explain how to do it.
ssis more than 4000 chars
After version SQL 2012 (I think) the limitation of the variable size does not exist.
However for example in SQL 2017 you can put a large amount of data in a string variable (I tried with more than 20 MB) and it works fine. But when you try to do string operations on it, in an expression like FINDSTRING or such then you get DTS_E_EXPREVALSTRINGVARIABLETOOLONG error.
My solution to sort this out was to use a script task, because in .NET you do not have this limitation.
Related
I’m experiencing a frustrating issue when trying to call a proc in an OLE DB source task. I’m using the SQL command from variable data access mode but I can see that it isn’t evaluating my variable correctly.
My variable (with ValidateAsExpression set to True) uses an expression to create a sql command like “EXEC ProcName ‘Param'” where the value of Param comes from a variable who’s value I set using an EXEC SQL task. Below is the expression:
“EXEC ProcName ” + “‘” + #[User::vDateThreshold] + “‘”
If I use a variable in my source that references a static value it works fine, but the issue seems to be when I use a variable which reference another variable in its expression.
Has anyone else come across this issue?
I’m using this method because I’ve had a similar issue when trying to use a parameter with the sql command data access method.
Thanks in advance
I’m using this method because I’ve had a similar issue when trying to use a parameter with the sql command data access method.
The right way to do that is by using SQL Command with parameters:
EXEC ProcName ?
And select #[User::vDateThreshold] as parameter.
Parameterized OLEDB source query
If it is not working then check your procedure code and make sure it generate a specific result set. If the result set is dynamic and columns are not fixed then you have to define it in the query using WITH RESULTSETS keyword.
https://www.mssqltips.com/sqlservertip/2356/overview-of-with-result-sets-feature-of-sql-server-2012/
From the name of #[User::vDateThreshold it seems like an SSIS datetime variable. Try setting this to a variable with an explicit cast and then executing the stored procedure with the variable. Make sure there that are single quotes (') within the CAST function as you would use if this was done in SSMS. When concatenating a datetime variable within a string variable in SSIS, the datetime variable must be converted to text, which is done with (DT_STR, length, codepage) in the sample expression below. I'm not sure what version you're using, but testing this out on SSDT for Visual 2017 worked fine for me. This will cover if you still want to hold the SQL in a variable, however the solution that #Hadi posted is a good option if you'd prefer to go that route.
"DECLARE #pDate DATETIME
SET #pDate = CAST('" + (DT_STR, 50, 1252)#[User::vDateThreshold] + "' AS DATETIME)
EXEC ProcName #pDate"
Thank you for the responses to my question.
I actually found the issue was with the ordering of my tasks in the package. When I looked closer at the values assigned to the relevant variables by using a break point on my exec SQL task I could see the wrong date was being passed to my proc. When I set the value of vDateThreshold at an earlier point the correct date value was assigned.
I think this was a case of looking at something for long enough that I was missing the obvious.
I defined a variable with the text of a parameterized query like this
select * from t where col = ?
Now, I am using that variable #[User::sqltext] in an OLE DB source for a dataflow task.
Problem is I don't see a way to set the parameters as I would have if I had supplied the SQL text in the source directly.
What am I missing?
A little touch.
Set your variable #[User::sqltext] with property EvaluateAsExpression=true, and set the following value for the Expression
"select * from t where col = '"+#[User::SomeStringParameter]+"'"
Handling expressions for variables is much easier with BIDSHelper, an addon for Visual Studio. It provides a special editor for managing variable expressions, as shown below.
This shows sample for similar task - creation of SQL command. Evaluate button allows to check results before it is executed in the package.
I am trying to get CSV IDs from a table from sql server and assign the result to a variable. below is the sql I have put inside the Execute SQL Task
set nocount on
declare #csv varchar(max) = ''
select #csv = #csv + cast(companyid as varchar(10)) + ',' from company where isprocessed = 0
select substring(#csv,1,len(#csv) - 1) as companyids
As you can see its a simple and standard way of getting csv of a field in t-sql. It works perfectly in query window but throwing below error when I run the Task in SSIS 2012
[Execute SQL Task] Error: The value type (__ComObject) can only be
converted to variables of type Object.
[Execute SQL Task] Error: An error occurred while assigning a value to
variable "sCSVCompanyIds": "The type of the value (DBNull) being
assigned to variable "User::sCSVCompanyIds" differs from the current
variable type (String). Variables may not change type during
execution. Variable types are strict, except for variables of type
Object. ".
Below are the settings of Execute SQL Task
In General tab, Resultset project is set to single row
In Result Set tab, Result Name is set to 0 (I also tried by setting it to csvids which is the alias column name in the select list) and Variable Name is set to User::sCSVCompanyIds
I have no clue why its not working. After wasting so much time I am worked out a hard way which by returning the result as Full Row set (same SQL which returns 1 row 1 column always) and add a for each loop container to loop throw the result set (which will always iterates only once for obvious reasons) and assign the result set's fields to the variable. It works for me but there should be easy way of doing it. What I am missing?
The problem is with the nvarchar(max) data type. I assume it will be same for varchar(max) also. Though relevant data types in SSIS are DT_NTEXT and DT_TEXT are present for some reason we are getting this error.
There are multiple options to handle this.
One Of course there might be more appropriate way to handle this by cast/converting the column within the query to fixed length instead of max something like cast(myvarcharmaxfield as varchar(8000)). In my case it doesn't work because I expect bigger length string. I am generating a csv string from a unique identifier column which itself is 36 chars long string and needs 3 extra chars for quoting them with signle quote and a comma as separator which will support only 205 values. So it doesn't work for me.
So I left with no option but to stick the way I implemented already which is in my question
After wasting so much time I am worked out a hard way which by
returning the result as Full Row set (same SQL which returns 1 row 1
column always) and add a for each loop container to loop throw the
result set (which will always iterates only once for obvious reasons)
and assign the result set's fields to the variable
The new way I learnt is from an unaccepted answer from this question
Change the Oledb connection to ADO.NET connection
I think this is fair enough but it requires me to create another connection manager (so that I don't have to change all existing tasks) and use it for this type of taks where I need csv but I did not buy it as I have very little time to doing research on creating connection string of it which appears to me with the type of erros I am getting is different from oledb.
I've this query in my OLE DB Source, using a parameter inside it:
select *
from A
where A.A_DATE >= DATEADD(d,-1*(CAST(? as int)),GETDATE())
with parameter X used. For sample case, I use X = 1. If I run the above query in the SQL Server Management Studio, it returns a row (the correct one). But when I run it inside the SSIS package it doesn't return any row (which is incorrect). When remove the -1*xxx and go straight with the CAST(? as int) (the query looks like this:)
select *
from A
where A.A_DATE >= DATEADD(d,CAST(? as int),GETDATE())
and set the X value to -1, it shows the correct result (a row returned). Is there something wrong with the parameter multiplication with a negative value inside an OLE DB Source query?
update 1:
It seems there is another problem with the 2nd query, as if I change the value to 1, I still get row results (when it shouldn't). Any solutions??
update 2:
it seems the cast solution is used is also flawed, since it doesn't return the correct value either. I kept getting 2 rows returned when I should have only 1 row returned
Seems the query above is also flawed in SSIS, so I decided to use this query, so I don't have to use CAST:
"select *
from A
where A.A_DATE >= DATEADD(d," + #[User::Y] + ",GETDATE())"
and put it inside a variable (let's called it Src_Query) and then evaluate the string above as an Expression. I also used variable Y with String data type instead of X with data type Int32 I used previously, and to turn it to negative value I just used Script Task to deal with it. Then in the OLE DB Source I used the "SQL Command from variable" option. I run the package, and the query returned the correct result. It's also useful for variable that I used inside sub-queries.
The problem with the above solution: In my project I have some source query that has more than 4000 chars, and SSIS doesn't allow more than 4000 chars processed in the Expression Builder. Still searching a way to go around this problem.
Update 1: I've workaround my problem for more than 4000 chars-long query, by putting the where clause in a separate Conditional Split after the data retrieval, and I still can use my Int32-typed variable.
This is the expression that the conditional split used, with the query in OLE DB Source no longer has the where clause related to the date:
A_DATE >= DATEADD("d",#[User::X],(DT_DBDATE)GETDATE())
Wonder how it will affect the performance though, is it significant?
I'm learning SSIS and this seems like an easy task but I'm stuck.
I have a CSV file Orders.csv with this data:
ProductId,Quantity,CustomerId
1,1,104
2,1,105
3,2,106
I also have a stored procedure ssis_createorder that takes as input parameters:
#productid int
#quantity int
#customerid int
What I want to do is create an SSIS package that takes the .csv file as input and calls ssis_createorder three times for each row in the .csv file (the first row contains column names).
Here is what I have done so far.
I have created an SSIS package (Visual Studio 2005 & SQL Server 2005).
In Control Flow I have a Data Flow Task.
The Data Flow has a Flat File source of my .csv file. All of of the columns are mapped.
I have created a variable named orders of type Object. I also have variables CustomerId, ProductId, & Quantity of type int32.
Next I have a Recordset Destination that is assigning the contents of the .csv file into the varialbe orders. I'm not sure about how to use this tool. I'm setting the VariableName (under Customer Properties) to User::orders. I think that now orders holds an ADO record set made up of the contents from the original .csv file.
Next I'm adding a ForEach Loop Container on the Control Flow tag and linking it to the Data Flow Task.
Inside of the ForEach Loop Container I'm setting the Enumerator to "ForEach ADO Enumerator". I'm setting "ADO object source variable" to User::orders". For Enumeration mode I'm selecting "Rows in the first table".
In the Variable Mapping tab I have User::ProductId index 0, User::Quantity index 1, User::CustomerId index 2. I'm not sure if this is correct.
Next I have a Script Task inside of the ForEach Loop Container.
I have ReadOnlyVariables set to ProductId.
In the Main method this is what I'm doing:
Dim sProductId As String = Dts.Variables("ProductId").Value.ToString
MsgBox("sProductId")
When I run the package my ForEach Loop Container turns Bright Red and I get the following error messages
Error: 0xC001F009 at MasterTest: The type of the value being assigned to variable "User::ProductId" differs from the current variable type. Variables may not change type during execution. Variable types are strict, except for variables of type Object.
Error: 0xC001C012 at Foreach Loop Container: ForEach Variable Mapping number 1 to variable "User::ProductId" cannot be applied.
Error: 0xC001F009 at MasterTest: The type of the value being assigned to variable "User::Quantity" differs from the current variable type. Variables may not change type during execution. Variable types are strict, except for variables of type Object.
Error: 0xC001C012 at Foreach Loop Container: ForEach Variable Mapping number 2 to variable "User::Quantity" cannot be applied.
Error: 0xC001F009 at MasterTest: The type of the value being assigned to variable "User::CustomerId" differs from the current variable type. Variables may not change type during execution. Variable types are strict, except for variables of type Object.
Error: 0xC001C012 at Foreach Loop Container: ForEach Variable Mapping number 3 to variable "User::CustomerId" cannot be applied.
Warning: 0x80019002 at MasterTest: SSIS Warning Code DTS_W_MAXIMUMERRORCOUNTREACHED. The Execution method succeeded, but the number of errors raised (12) reached the maximum allowed (1); resulting in failure. This occurs when the number of errors reaches the number specified in MaximumErrorCount. Change the MaximumErrorCount or fix the errors.
SSIS package "Package.dtsx" finished: Failure.
Dts.TaskResult = Dts.Results.Success
Any help would be appreciated
One of my coworkers just give me the answer.
You don't need the the ForEach Loop Container or the RecordSet Container.
All you need is the Flat File Source and an OLE DB Command. Connect to your database and inside the OLE DB Command select the appropriate connection.
In the Component Properties enter the following SQLCommand:
exec ssis_createorder ?, ?, ?
The "?" are place holders for the parameters.
Next under the Column Mappings tab map the .csv file columns to the stored procedure parameters.
You are finished go ahead and run the package.
Thanks Gary if you were on StackOverFlow I would give you an upvote and accept your answer.
If I understand correctly, what you want to do is execute a stored procedure 3 times for each row in the data source.
What if you just create a data flow with a flat file data source and pipe the data through 3 execute sql command tasks? Just map the columns in the data to the input params of your stored procedure.
Maybe I'm not seeing it correctly in your question and I'm thinking too simple, but in my experience you need to avoid using the foreach task in SSIS as much as possible.
I suspect that you need to look at your Data Flow task. It's likely that the values from the source CSV file are being interpreted as string values. You will probably need a Derived Column component or a Data Conversion component to convert your input values to the desired data type.
And, I think #StephaneT's solution would be good for executing the SP.
I'm not sure if this answers your question. But I was looking to do this and I achieved it using the BULK INSERT command. I created a staging table with all of the columns in the csv file, and instead of a stored procedure I used a INSTEAD OF INSERT trigger to handle the logic of inserting it into many tables.