I have created a variable in ssis called Vpop_summary_table.
I have used this variable in a sql statement.
my variable
Vpop_summary_table
is expected to give value
[dbo].[2019-02-02_pop_table]
It is fine in ssis.
DECLARE #Vpop_summary_table VARCHAR(100)
SET #Vpop_summary_table= ?
SELECT
,[Age Range]
,[Gender]
,[2023 Population]
INTO [#Vpop_summary_table]
FROM [dbo].[2018_Population_Table_CLARITAS]
My table is created as [#Vpop_summary_table] instead of [dbo].[2019-02-02_pop_table]
Table names and column names need to be static.
If you want to do this create the full SQL and then use sp_executesql to execute it although in terms of good practices having a table in a variable doesn't look so great...
You cannot pass table name as parameter, you have to use expressions to achieve that:
Open Execute SQL Task editor, Go To Expressions Tab, add an expression for the SQLStatementSource property as following:
"SELECT
[Age Range]
,[Gender]
,[2023 Population]
INTO " + #[User::Vpop_summary_table] + "
FROM [dbo].[2018_Population_Table_CLARITAS]"
More info at:
Common Examples of SSIS Expressions and Variables (check Working with SSIS expressions section)
Related
I have an SSIS data flow task where I am trying to load data from a source table using ADO.NET Source Editor into a SQL Server table.
The problem:
The source table name is dynamic and depends on the current date.
Example: If I want to load today's data then the table name would be Sample_03292017_data and if loading tomorrow's data then it would be Sample_03302017_Data.
I did some research and found how to pass parameters to an ADO.NET Source Editor to use in where conditions but I couldn't find anything on how to use this parameter in a table name.
Does anyone know how I can achieve this? My query is really simple:
select * from Sample_[DateParameter]_Data.
I am using Visual Studio Data Tools 2010.
Expression must be set on the Data Flow Task not in the Ado.net Source
Follow this steps to set an expression for an Ado.net Source:
in the control flow tab click on the Data Flow Task and press F4 to show the properties tab
Click on the expression button, it show up a form like shown below
choose [Ado.net Source].sqlcommand property and click on the expression builder button
write the following expression "select * from Sample_ " + #[User::DateParameter] + "_Data" (assuming that your data parameter is stored in a variable named DateParameter)
You should use dynamic SQL to achieve this. If you are using ADO.net as Connection Type, copy and paste the following to your SQL statement:
Declare #SQL VARCHAR(MAX)
SET #SQL = 'Select * from ' + #TABLE
EXEC(#SQL)
In the parameter page, you need to Add new parameter which has #TABLE as the parameter name and leave the size as -1. The #Table variable should be decided by the expression from variable setting page.
I am using Power query & I would like to assign a comma delimited string from an excel cell to a sql server variable. The power query I have so far is below. My Parameter is in a Excel "Table3" in "Column2" :
let
ProdParameterSource =Excel.CurrentWorkbook(){[Name="Table3"]}[Content]{0}[Column2],
Source = Sql.Database("server", "database", [Query="Declare #Product varchar(max) = "&(ProdParameterSource)&" select #Product"])
in
Source
I see the below error :
The parameters I am using as seen in the excel sheet are below :
How can I fix this error and see my input parameters in the sql server variable ABC as '44,216' (with the inverted comma).
I do not do a lot of power query but I do work in Power BI Desktop version.
First, you need to look at the reference to the SQL.Database() on MSDN.
It is looking for a whole query, either a dynamic string you make up or a call to a stored procedure.
My simple example below, pulls data from the Adventure Works DW 2012 Customer table.
What you are missing is a TSQL statement or multiple TSQL statements. Use the semicolon to combine statements into one batch to be called.
The example below create a variable #x as an integer and returns the value. This is almost like you example above.
You will be prompted for security (trusted credentials) and it will tell you that it is an unencrypted connection.
I did some looking around the NET. This article from Reeves Smith is like my last example but slanted towards power query and excel. Just remember, both products use the same engine.
https://reevessmith.wordpress.com/2014/08/19/power-query-and-stored-procedures-with-parameters/
Again
This will fix it: [Query="Declare #Product varchar(max) = '" & ProdParameterSource & "' select #Product"]).
This will fix it in a safer way, since you will escape any extra single-quotes which could break out of the string value: [Query="Declare #Product varchar(max) = '" & Text.Replace(ProdParameterSource, "'", "''") &"' select #Product"]).
What happened is that Power Query treats the text passed to Query as the entire script, and when you built the script you didn't put the value of ProdParameterSource in quotes, so the script appears to set a varchar value to 44,216 (without quotes). This is an invalid statement.
I am wanting to have a user enter the database path they will be using.
For Example:
#SourceDB = DatabaseA.dbo.
I would then like to use the variables in a Data Flow or Execute SQL task.
My question: How do I reference the variables in these two tasks? Neither of these ways seem to be working for me:
SELECT Field1, Field2 FROM #SourceDB + tablename
or
"SELECT Field1, Field2 FROM " + #[User::SourceDB] + "tablename"
or
SELECT Field1, Field2 FROM ? tablename
You won't be be able to use it as a parameter ?. Others have tried to do that and run into what you're experiencing.
What I would do is define a variable, call it SourceQuery, data type of String, scoped at the package level. In the properties window for that variable (F4), change the EvaluateAsExpression property to True. Then use a formula in the expression like you have already specified.
"SELECT Field1, Field2 FROM " + #[User::SourceDB] + "tablename"
Finally in your OLE DB Source, change the access mode to table or view to "SQL Command from variable" and then things should just work.
The one caveat to be aware of is that if the data types for these columns change, your SSIS packages will fail the Validation step.
I have a stored procedure that gets executed through SQL SSIS using a Execute SQL Task.
The task has the following:
USE [OPPY_DWUSD]
GO
DECLARE #return_value int
EXEC #return_value = [dbo].[generate_merge_scdbk]
#Schema = N'dim',
#Dimension = N'VARIETY',
#ETLSchema = N'stg',
#ETLTable = N'vw_VARIETY',
#Execute = 1
SELECT 'Return Value' = #return_value
GO
Right now the way I have this setup, I have multiple Execute SQL Tasks with the same code but different values, about 20 Execute SQL Tasks.
Is there a more cleaner way to pull this off?
Here is one way of doing this. The example uses SSIS 2008 R2 with SQL Server 2012 backend.
Create a table to store your parameter values. Let's say the table name is dbo.SProcValues. Based on your stored procedure definition, the table schema would look like this.
CREATE TABLE dbo.SProcValues(
Id int IDENTITY(1,1) NOT NULL,
SProcName nvarchar(40) NOT NULL,
SchemaName nvarchar(20) NOT NULL,
Dimension nvarchar(40) NOT NULL,
ETLSchema nvarchar(20) NOT NULL,
ETLTable nvarchar(40) NOT NULL,
IsExecute bit NOT NULL
)
GO
Let's insert some sample data using the following script.
INSERT INTO dbo.SProcValues
(SProcName, SchemaName, Dimension, ETLSchema, ETLTable, IsExecute) VALUES
('dbo.sp_generate_merge', 'dim1', 'dimension1', 'stg1', 'table1', 1),
('dbo.sp_generate_merge_scdbk', 'dim2', 'dimension2', 'stg2', 'table2', 1),
('dbo.sp_generate_merge_scdbk', 'dim3', 'dimension3', 'stg3', 'table3', 0),
('dbo.sp_generate_merge', 'dim4', 'dimension4', 'stg4', 'table4', 0);
GO
On the SSIS package, assuming that you have the data source and connection manager already established. Create the following variables. Variable SProcValues will hold the parameter set that we stored in the above-mentioned table. Variable SQLInnerQuery will hold the query that will be used later in the inner Execute SQL Task. Other variables relate to each column available in the table so we can loop through each row and hold it in a variable.
Paste the following query in the value of the variable SQLGetParameters
SELECT SProcName, SchemaName, Dimension, ETLSchema, ETLTable, IsExecute FROM dbo.SProcValues
Select the variable SQLInnerQuery and press F4 to view the properties. Set the property EvaluateAsExpression to True and then click the Ellipsis button against the Expression property.
We need to set an expression that will evaluate to the EXEC stored procedure statement that can be later supplied to the inner Execute SQL Task. Set the following expression.
"EXEC " + #[User::SProcName] + " #Schema = ?, #Dimension = ?, #ETLSchema = ?, #ETLTable = ?, #IsExecute = ?"
If you click Evaluate Expression button on the editor, you can see what the expression will evaluate to. You will also notice that there is no stored procedure name in the below screenshot that is because the package variable SProcName currently does not have any value. During runtime, the SProcName will be assigned with the value from the table and this expression will automatically resolve itself.
On the SSIS package, drag and drop an Execute SQL Task. This task will run the following query to fetch the list of parameter values that are stored in the table dbo.SProcValues. Configure the General page on the Execute SQL Task as shown below. The example uses OLEDB connection and the connection manager/data source is named as Practice.
Configure the Result Set page of Execute SQL Task to store the result set from the query to an object variable.
Now that the first Execute SQL Task is configured to get the list of parameter values that should be passed to the stored procedure, you need to loop through the records.
Drag and drop a Foreach Loop container. Connect the Execute SQL Task's precedence container to the Foreach Loop container. Configure the Collection page of the Foreach Loop container as shown below. We are looping through the result set using the ADO enumerator.
Configure the Variable Mappings page on Foreach Loop container as shown below. As we loop through each row, we are storing the column values in respective variables so we can pass it to the next Execute SQL Task to run the stored procedure.
Drag and drop an Execute SQL Task inside the Foreach Loop container so that this task is executed each time we loop through a row in the result set. Configure the Execute SQL Task as shown below.
NOTE
You might want to configure the ResultSet property on this second Execute SQL Task according to your requirements. If you choose ResultSet, then you need to configure an appropriate object variable to accept the result set. I left it as None for this example.
Configure the values to be passed as parameters to the stored procedure.
Finally, the control flow would look something like this.
When the package runs, the loop will execute the stored procedure for as many records are returned by the SELECT query mentioned above, provided that you have all the stored procedures defined in the table rows are available in the database. I had created the stored procedures dbo.sp_generate_merge_scdbk and dbo.sp_generate_merge with the same parameters definition. That's the reason the package executed successfully.
You have the right concept, just need to use some concepts like variables, a foreach loop and parameters on the Execute SQL Task.
Control Flow
Your Control Flow would look something like this
Variables
I have 6 variables defined in SSIS
Dimension | String | VARIETY
ETLSchema | String | stg
ETLTable | String | vw_VARIETY
Execute | Int32 | 1
RecordSet | Object | System.Object
Schema | String | dim
The first Execute SQL Task will be a query or something enumerable like it. Currently, have a hard coded query to produce the supplied query values. Your solution could just be a chain of SELECT's UNIONed together. The goal of this step is to populate the RecordSet variable.
My Execute SQL Task returns a full result set
and I push that into my object thusly
ForEach Loop Container (ADO Recordset)
The ForEach Loop Container is going to consume that enumerable thing we established beforehand. It will go through each row and we will pop the values out of the object and assign them into local variables.
Change the Enumerator to Foreach ADO Enumerator. Select the object we populated with results User::RecordSet and then use an enumeration mode of Rows in first table
In the Variable Mappings tab, we will identify the ordinal based location for the values (column 0 maps to variable X). The only trick here is to ensure your SSIS Variable data types match the data type in the result set from your source query. Do note it's a zero based ordinal system.
At this point, if you click run you see it enumerate through all the rows you have sent into the RecordSet variable. I find it helpful to run it at this point to make sure I have all of my data types aligned.
Inner Execute SQL Task
I have taken your query and replaced the hard coded values with place holder. An OLEDB connection will use ? while an ADO.NET will use named #varname.
In the Parameter Mapping tab, simply map those local variables to the place holders.
Now you have a nice template for running the same proc with varying values.
I have placed a execute sql task inside the foreach loop container , where i want to execute the stored procedure name obtaining from the ssis user defined variable
But i am not finding a right syntax to execute stored procedure name from variable. and i want to pass input parameter to execute the stored procedure which is also a another dts variable
Please can any one help to figure out the right syntax
Lets say you have two variables StoredProcedureName and ParameterValue
Create a new variable called QueryString. Open up the properties window by pressing F4 and then click on the variable name in the window that has list of variables. Set the EvaluateAsExpression property of your QueryString variable to True.
Then click the elliptical in Expression to open up the Expression Builder. Type the following expression
"Execute " + #[User::StoredProcedureName] + " #ParameterName = '" + #[User::ParameterValue]+ "'"
Click on Evaluate Expression and you should see
Execute MyProcedure #ParameterName = 'SomeValue'
Assign this variable to your SourceVariable for the Execute SQL Task.
You can build your string out as provided above but I would not start with it. For one, it assumes everything is string and second it could conceivably open you up to sql injection (highly improbable, I agree)
To actually use the native parameter mapping for ADO.NET you need to use the #PlaceHolder for your variable substitution. Based on your screenshot, the value of #[User::StoredProcedureName] would need to have the format of EXECUTE schema.ProcName #ParameterName Otherwise, you're looking at cobbling the expression from #Raj More's example without the =... piece
Mapping is as simple as the following. Ensure you select the correct Data Types and Parameter Names to correlate with your query.