I defined a variable with the text of a parameterized query like this
select * from t where col = ?
Now, I am using that variable #[User::sqltext] in an OLE DB source for a dataflow task.
Problem is I don't see a way to set the parameters as I would have if I had supplied the SQL text in the source directly.
What am I missing?
A little touch.
Set your variable #[User::sqltext] with property EvaluateAsExpression=true, and set the following value for the Expression
"select * from t where col = '"+#[User::SomeStringParameter]+"'"
Handling expressions for variables is much easier with BIDSHelper, an addon for Visual Studio. It provides a special editor for managing variable expressions, as shown below.
This shows sample for similar task - creation of SQL command. Evaluate button allows to check results before it is executed in the package.
Related
I’m experiencing a frustrating issue when trying to call a proc in an OLE DB source task. I’m using the SQL command from variable data access mode but I can see that it isn’t evaluating my variable correctly.
My variable (with ValidateAsExpression set to True) uses an expression to create a sql command like “EXEC ProcName ‘Param'” where the value of Param comes from a variable who’s value I set using an EXEC SQL task. Below is the expression:
“EXEC ProcName ” + “‘” + #[User::vDateThreshold] + “‘”
If I use a variable in my source that references a static value it works fine, but the issue seems to be when I use a variable which reference another variable in its expression.
Has anyone else come across this issue?
I’m using this method because I’ve had a similar issue when trying to use a parameter with the sql command data access method.
Thanks in advance
I’m using this method because I’ve had a similar issue when trying to use a parameter with the sql command data access method.
The right way to do that is by using SQL Command with parameters:
EXEC ProcName ?
And select #[User::vDateThreshold] as parameter.
Parameterized OLEDB source query
If it is not working then check your procedure code and make sure it generate a specific result set. If the result set is dynamic and columns are not fixed then you have to define it in the query using WITH RESULTSETS keyword.
https://www.mssqltips.com/sqlservertip/2356/overview-of-with-result-sets-feature-of-sql-server-2012/
From the name of #[User::vDateThreshold it seems like an SSIS datetime variable. Try setting this to a variable with an explicit cast and then executing the stored procedure with the variable. Make sure there that are single quotes (') within the CAST function as you would use if this was done in SSMS. When concatenating a datetime variable within a string variable in SSIS, the datetime variable must be converted to text, which is done with (DT_STR, length, codepage) in the sample expression below. I'm not sure what version you're using, but testing this out on SSDT for Visual 2017 worked fine for me. This will cover if you still want to hold the SQL in a variable, however the solution that #Hadi posted is a good option if you'd prefer to go that route.
"DECLARE #pDate DATETIME
SET #pDate = CAST('" + (DT_STR, 50, 1252)#[User::vDateThreshold] + "' AS DATETIME)
EXEC ProcName #pDate"
Thank you for the responses to my question.
I actually found the issue was with the ordering of my tasks in the package. When I looked closer at the values assigned to the relevant variables by using a break point on my exec SQL task I could see the wrong date was being passed to my proc. When I set the value of vDateThreshold at an earlier point the correct date value was assigned.
I think this was a case of looking at something for long enough that I was missing the obvious.
I am querying a Cache database through an ADO net datasource in my dataflow in SSIS (SQL 2008 R2). I want to pass parameters to the query but can only do this through the expressions section of the dataflow item. The query itself is over 4000 characters hence I cant use the query in the expressions section (due to the 4000 character limitation) in its raw form.
I have tried using a script task to assign the query to a string variable [User::Query1] however when I click the Evaluate Expression button in the expression builder screen of the dataflow, it returns nothing. I have the following expression for [ADO NET Source].[SQLCommand]:
#[User::Query1]
When running the package, I get a an error saying that SQL command has not been set correctly. Check SQLCommand Property.
I've set ValidateExternalMetaData to false and I see the following error in the execution results;
Error: The variable User::Query1 contains a string that exceeds the maximum allowed length of 4000 characters.
Error: Reading the variable "User::Query1" failed with error code 0xC0047100.
Error: The expression "#[User::Query1]" on property "[ADO NET Source].[SqlCommand]" cannot be evaluated. Modify the expression to be valid.
In my script task I have the entire query assigned to a string variable and then I am assigning the string to the actual variable using the vb code below:
Dts.Variables("User::Query1").Value = sSql
MessageBox.Show(Dts.Variables("User::Query1").Value.ToString())
On the script task properties, I have [User::Query1] as a ReadWriteVariables. I have also made sure that EvaluateAsExpression for [User::Query1] is set to true.
In essence, I am trying to run the query using the expressions property of the dataflow as this will allow me to use dynamic parameter values.
Also variables are limited to 4000 chars in SSIS, you could split the query in multiple sub queries.
I have found a solution when you use an OLE DB connection. It is complex to prepare but finally it is running OK. Previously, you need to define the variable with a similar and same structure query and <4000 characters to validate the expression. When is all ok, you can redefine the variable with >4000 characters query in a script task.
Next link explain how to do it.
ssis more than 4000 chars
After version SQL 2012 (I think) the limitation of the variable size does not exist.
However for example in SQL 2017 you can put a large amount of data in a string variable (I tried with more than 20 MB) and it works fine. But when you try to do string operations on it, in an expression like FINDSTRING or such then you get DTS_E_EXPREVALSTRINGVARIABLETOOLONG error.
My solution to sort this out was to use a script task, because in .NET you do not have this limitation.
I am using SSDT and working on a simple SSIS package.
The Control flow:
1. A Foreach Loop Container and seek a folder exist a "importdata{}.csv" file or not.
2. If found, a script task will set variables:
- User::FullPath = (e.g C:\importdata{}.csv)
- User::varFileNameNoExt = (importdata{}) without extension.
The {} is possible in "toy","game","food".
3. Go to dataflow
The Data Flow:
1. Flat File Source with a flat file connection, the connection string is varible and mapped connection string expression.
2.ADO.NET Destination , insert data.
My question is how can i set the ADO.NET Destination [TableOrViewName] Property in variable?
Assume the table : importdatatoy,importdatagame and importdatafood is created on SQL Server.
I try to set as "dbo"."[User::varFileNameNoExt]" ,but it cannot resolve the table name on runtime.
ADO.NET Destination [TableOrViewName] parametrization can be done at Data flow level. In data flow properties, you can specify "ADO.NET Destination [TableOrViewName]".
Also specify the quotes while assigning value to variable
Eg: varFileNameNoExt = "dbo"."tableName"
But first you will need to create mapping with an existent table.
Can you post your error message? I'm thinking you won't be able to combine static text and a variable like that inside of the TableOrViewName field. Instead do the combination in a new [User::varTableName] SSIS variable and use the Advanced Properties Expression editor to set the TableOrViewName to this new SSIS variable. Have a look here.
I have a simple String variable with the following value: "C:\Test.txt".
Now I would like to edit the variable to point to a different file.
I cannot find a way to do that. I can change the Name, Data Type, but not the value itself!
Do I need to delete the variable and create the new one?
Update: The problem was caused by "ReadOnly" property set to "True". For typical scenarios, see the accepted answer below.
As #Yuck and #devarc have noted, there are two different and distinct values a Variable holds. The Design-time value is the value you assign when the variable is first created. In your case, the variable holds C:\Test.txt as the design-time value. Everytime you open the package, it would show C:\Test.txt until you change it in the
To make the value of a variable change while the package is running, your options are either to set the value or calculate it. Here I have created a package-level variable CurrentFile with the value of C:\Test.txt
One thing that often trips people up is that they have correctly changed the run-time value but when they run it in BIDS, they see the "old" value. The value displayed in the Variables window does not change during package execution.
During package execution, my Variables window still shows the design-time value (C:\Test.txt) but the true value is reflected in the Locals window (C:\Test2.txt)
Setting a value
The value of most anything in SSIS can be established at run-time through a set of verbose command-line options or through configuration sources. The biggest difference in my mind is that this approach is that the value will always be the value for the entire lifetime of package execution. Sequential or parallel invocations of a package can change that value but for that execution the value would remain constant (barring an explicit modification of the value.
/SET
Command-line execution (dtexec.exe), right clicking on a package and running from the filesystem (dtexecUI.exe) or creating a SQL Agent job step of SQL Server Integration Services all allow for providing a run-time value through the SET command. Using the above variable, the following command would set the run-time value to C:\Test2.txt
dtexec /file C:\Generated.dtsx /set \Package.Variables[User::CurrentFile].Properties[Value];"C:\Test2.txt"
Configuration
SSIS offers an option to create configuration sources to provide run-time values to packages. The article I linked to above does a much better job describing the pros and cons of the configuration options than I will do here. I will say that I typically use both - my SET command configures a connection manager which is then used by the package to find the "full" set of package configurations.
Calculating a value
There are a variety of tasks in SSIS that can change the value of a variable as well as the use of Expressions to change a value. I see these as things that operate on value whilst the package is in flight.
Tasks
A Script Task is one of the most commonly used mechanisms for those starting out but I find other tools in the SSIS toolkit usually better suited for changing variable values.
Foreach Loop Container and Execute SQL Task are two of the other big Tasks you should look at for assignment of a variable value.
Expressions
Expressions are the most glorious candy in the SSIS toolbox. Most every "thing" in SSIS exposes properties for configuration. That's helpful, but using assigning an expression to build those properties is outstanding.
For example, imagine 3 variables RootFolder, FileName and ComputedCurrentFile with values of C:\, File2.txt and empty string. On the Properties window for ComputedCurrentFile we'd change the value for EvaluateAsExpression from False to True and then use an expression like #[User::RootFolder]+ "\\" +#[User::FileName] That simply concatenates the value the first two variables together. This can be helpful if the file name for processing was standard but the source folder changed often. Or if we're talking about output, it's common to use expressions to build an output file name using the date and possibly time of when the package is running.
Finally, there is nothing that prevents a mixing and matching of these approaches. I typically use a configuration to point a file enumerator at the correct starting folder and then use calculated values to identify the current file for processing.
If you want to change it in designer just right click on free space and --> Variables.
But if you want to change it at runtime I suggest you to:
create script task
choose language
add your variable to ReadWriteVariables.
Edit script.
For example in VB:
Dts.Variables("myVariable").Value = #"C:\Test2.txt";
Dts.TaskResult = ScriptResults.Success
Found an easy way to handle this. Remove the Variable from Expression which will enable Value Box to edit. Once it is edited, add the Variable back in the Expression should get the updated value. Hope this helps.
I was also facing the same issue like you where once the variable is declared and define (for eg:var1=text1.csv)in SSIS Variable window I was not able to update the variable value(for eg: var1=text2.csv) in SSIS Variable Window by clicking on the variable value field.
Applied below fix:-
I noticed that I was using var1 variable as a Expression by using expression builder so to update the value(for eg:-var1=text2.csv) I used expression builder window.once you done using the expression builder,you can see the text2.csv is got mapped to var1.
I've this query in my OLE DB Source, using a parameter inside it:
select *
from A
where A.A_DATE >= DATEADD(d,-1*(CAST(? as int)),GETDATE())
with parameter X used. For sample case, I use X = 1. If I run the above query in the SQL Server Management Studio, it returns a row (the correct one). But when I run it inside the SSIS package it doesn't return any row (which is incorrect). When remove the -1*xxx and go straight with the CAST(? as int) (the query looks like this:)
select *
from A
where A.A_DATE >= DATEADD(d,CAST(? as int),GETDATE())
and set the X value to -1, it shows the correct result (a row returned). Is there something wrong with the parameter multiplication with a negative value inside an OLE DB Source query?
update 1:
It seems there is another problem with the 2nd query, as if I change the value to 1, I still get row results (when it shouldn't). Any solutions??
update 2:
it seems the cast solution is used is also flawed, since it doesn't return the correct value either. I kept getting 2 rows returned when I should have only 1 row returned
Seems the query above is also flawed in SSIS, so I decided to use this query, so I don't have to use CAST:
"select *
from A
where A.A_DATE >= DATEADD(d," + #[User::Y] + ",GETDATE())"
and put it inside a variable (let's called it Src_Query) and then evaluate the string above as an Expression. I also used variable Y with String data type instead of X with data type Int32 I used previously, and to turn it to negative value I just used Script Task to deal with it. Then in the OLE DB Source I used the "SQL Command from variable" option. I run the package, and the query returned the correct result. It's also useful for variable that I used inside sub-queries.
The problem with the above solution: In my project I have some source query that has more than 4000 chars, and SSIS doesn't allow more than 4000 chars processed in the Expression Builder. Still searching a way to go around this problem.
Update 1: I've workaround my problem for more than 4000 chars-long query, by putting the where clause in a separate Conditional Split after the data retrieval, and I still can use my Int32-typed variable.
This is the expression that the conditional split used, with the query in OLE DB Source no longer has the where clause related to the date:
A_DATE >= DATEADD("d",#[User::X],(DT_DBDATE)GETDATE())
Wonder how it will affect the performance though, is it significant?