I have a simple task but somehow getting bugged on it.
Through SSIS I want to empty 3 tables in my database. I'm using SQL Server 2008 R2.
I want to create a FEL and somwehere in the variable put the names of my two tables.
Then the loop will go and delete records from those two tables.
Tables are: Table1 Table2 and Table3.
Can you please give me a detailed step by step isntruction. Please don't paste links to other articles, as it might be even more confusing.
Thank you very much!
Create a variable named TableNames of type Object
Create a variable named TableName of type String
Create a variable named Query of type String
Set the EvaluateAsExpression property to True
Set the Expression property to "DELETE FROM " + #[User::TableName]
Add an Execute Sql Task and double click to open the Execute Sql Task Editor
Make the following changes under the General settings
Set the ResultSet property to Full Result Set
Set the SQLStatement property to
select *
from
(
select 'Table1' as t union
select 'Table2' as t union
select 'Table3' as t
) tables
Make the following changes under the Result Set settings
Click Add button, and set the Result Name to 0 and the Variable Name to User::TableNames
Add a Foreach Loop Container and double click to open the Foreach Loop Editor
Make the following changes under the Collection settings
Set the Enumerator property to Foreach ADO Enumerator
Set the ADO object source variable to User::TableNames
Make the following changes under the Variable Mappings settings
Set the Variable to User::TableName and the Index to 0
Add an Execute Sql Task inside the Foreach Loop and double click to open the Execute Sql Task Editor
Make the following changes under the General settings
Set the SQLSourceType property to Variable
Set the SourceVariable to User::Query
Related
I have an SSIS data flow task where I am trying to load data from a source table using ADO.NET Source Editor into a SQL Server table.
The problem:
The source table name is dynamic and depends on the current date.
Example: If I want to load today's data then the table name would be Sample_03292017_data and if loading tomorrow's data then it would be Sample_03302017_Data.
I did some research and found how to pass parameters to an ADO.NET Source Editor to use in where conditions but I couldn't find anything on how to use this parameter in a table name.
Does anyone know how I can achieve this? My query is really simple:
select * from Sample_[DateParameter]_Data.
I am using Visual Studio Data Tools 2010.
Expression must be set on the Data Flow Task not in the Ado.net Source
Follow this steps to set an expression for an Ado.net Source:
in the control flow tab click on the Data Flow Task and press F4 to show the properties tab
Click on the expression button, it show up a form like shown below
choose [Ado.net Source].sqlcommand property and click on the expression builder button
write the following expression "select * from Sample_ " + #[User::DateParameter] + "_Data" (assuming that your data parameter is stored in a variable named DateParameter)
You should use dynamic SQL to achieve this. If you are using ADO.net as Connection Type, copy and paste the following to your SQL statement:
Declare #SQL VARCHAR(MAX)
SET #SQL = 'Select * from ' + #TABLE
EXEC(#SQL)
In the parameter page, you need to Add new parameter which has #TABLE as the parameter name and leave the size as -1. The #Table variable should be decided by the expression from variable setting page.
I have tables from navision which names are
dbo.[CompanyName$Cust_ Ledger Entry].[Customer No_]
I have 30 companies . I am looking to loop over companies
I follow this solution https://social.msdn.microsoft.com/forums/sqlserver/en-US/a13d39ab-968b-41f2-bf85-cb46db763d4e/variable-to-dynamically-change-tablename-for-data-flow-task
But how can I use it in the data flow task ? how can I change table names dynamically ?
Ok I can see from your other post that you have attempted things rather than just asking how to do something, so I will give you a hand.
Here is an example of how you can do this:
Here is an example of the SSIS package.
For starters you will need 3 variables:
Two string variables i've called them CompanyName and SQLStatement and an Object Variable which i've called CompanyList.
We first of all need to populate the CompanyList, we do so in the Execute SQL Task.
We need to set the Result Set to be "Full Result Set", then in the SQL Statement you need to add your way of getting the full company names, for instance "Select distinct Name from dbo.company", here we then store these results in the Object variable:
This will now store all the results into the Object Variable.
We now need to setup our SQLStatement Variable. The variable is just a string variable however what we need to do it set it to "EvaluateAsExpression" - "True":
In the expression we need to enter the following:
"Select * from dbo.[" + #[User::CompanyName] + "$Cust_ Ledger Entry].[Customer No_]"
Now this is setup we can get on with the ForeachLoop.
Set the ForEachLoop as above, where we want to evaluate "Rows in first table" against the "CompanyList" object variable.
The next step is to map that to our final variable "CompanyName" which will be used in out SQL Expression:
In the ForEachLoop go to the VariableMapping tab and enter Select the "CompanyName" variable with Index 0, this will change the "CompanyName" variable each time the loop runs, effectively changing the SQL Statement Variable allowing us to loop through each of the companies.
The final step is to go into the DFT and setup the OLEDB Source:
For the OLEDB Source we need to select "SQL command from variable", here we then select the "SQLStatement" variable.
You can then carry on your DFT as you wish.
I'm trying to first check if there are any new records to process before I execute my package. I have a bit field called "processed" in a SQL Server 2008 R2 table that has a value of 1 if processed and 0 if not.
I want to query it thus:
select count(processed) from dbo.AR_Sale where processed = 0
If the result is 0 I want to send an e-mail saying the records are not there. If greater than zero, I want to proceed with package execution. I am new to SSIS and can't seem to figure out what tool to use for this.
My package has a data flow item with an OLE DB connection inside it to the database. The connection uses a query to return the records. Unfortunately, the query completes successfully (as it should) even if there are no records to process. Here is the query:
Select * from dbo.AR_Sale where processed = 0
I copy these records to a data warehouse and then run another query to update the source table by changing the processed field from 0 to 1.
Any help would be greatly appreciated.
One option would be to make use of precedence constraint in conjunction with Execute SQL task to achieve this functionality. Here is an example of how to achieve this in SSIS 2008 R2.
I created a simple table based on the information provided in the question.
Create table script:
CREATE TABLE dbo.AR_Sale(
Id int NOT NULL IDENTITY PRIMARY KEY,
Item varchar(30) NOT NULL,
Price numeric(10, 2) NOT NULL,
Processed bit NOT NULL
)
GO
Then populated the new table with some sample data. You can see that one of the row has Processed flag set to zero.
Populate table script:
INSERT INTO dbo.AR_Sale (Item, Price, Processed) VALUES
('Item 1', 23.84, 1),
('Item 2', 72.19, 0),
('Item 3', 45.73, 1);
On the SSIS package, create the following two variables.
Processed of data type Int32
SQLFetchCount of data type String with value set to SELECT COUNT(Id) ProcessedCount FROM dbo.AR_Sale WHERE Processed = 0
On the SSIS project, create a OLE DB data source that points to the database of your choice. Add the data source to the package's connection manager. In this example, I have used named the data source as Practice.
On the package's Control Flow tab, drag and drop Execute SQL Task from the toolbox.
Configure the General page of the Execute SQL Task as shown below:
Give a proper Name, say Check pre-execution
Change ResultSet to Single row because the query returns a scalar value
Set the Connection to the OLE DB datasource, in this example Practice
Set the SQLSourceType to Variable because we will use the query stored in the variable
Set the SourceVariable to User::SQLFetchCount
Click Result Set page on the left section
Configure the Result Set page of the Execute SQL Task as shown below:
Click Add button to add a new variable which will store the count value returned by the query
Change the Result Name to 0 to indicate the first column value returned by query
Set the Variable Name to User::Processed
Click OK
On the package's Control Flow tab, drag and drop Send Mail Task and Data Flow Task from the toolbox. The Control Flow tab should look something like this:
Right-click on the green arrow that joins the Execute SQL task and Send Mail Task. Click Edit... the Green Arrow is called as Precedence Constraint.
On the Precedence Constraint Editor, perform the following steps:
Set Evaluation operation to Expression
Set the Expression to #[User::Processed] == 0. It means that take this path only when the variable Processed is set to zero.
Click OK
Right-click on the green arrow that joins the Execute SQL task and Data Flow Task. Click Edit... On the Precedence Constraint Editor, perform the following steps:
Set Evaluation operation to Expression
Set the Expression to #[User::Processed] != 0. It means that take this path only when the variable Processed is not set to zero.
Click OK
Control flow tab would look like this. You can configure the Send Mail Task to send email and the Data Flow Task to update the data according to your requirements.
When I execute the package with the data set to based on the populate table script, the package will execute the Data Flow Task because there is one row that is not processed.
When I execute the package after setting Processed flag to 1 on all the rows in the table using the script UPDATE dbo.AR_Sale SET Processed = 1, the package will execute the Send Mail Task.
Your SSIS design should be
Src:
Select count(processed) Cnt from dbo.AR_Sale where processed = 0
Conditional Split stage [under data flow transformations]:
output1: Order 1, Name - EmailCnt, Condition - Cnt = 0
output2: Order 2, Name - ProcessRows, Condition - Cnt > 0
Output Links:
EmailCnt Link: Send email
ProcessRowsLink: DataFlowTask
I have a stored procedure that gets executed through SQL SSIS using a Execute SQL Task.
The task has the following:
USE [OPPY_DWUSD]
GO
DECLARE #return_value int
EXEC #return_value = [dbo].[generate_merge_scdbk]
#Schema = N'dim',
#Dimension = N'VARIETY',
#ETLSchema = N'stg',
#ETLTable = N'vw_VARIETY',
#Execute = 1
SELECT 'Return Value' = #return_value
GO
Right now the way I have this setup, I have multiple Execute SQL Tasks with the same code but different values, about 20 Execute SQL Tasks.
Is there a more cleaner way to pull this off?
Here is one way of doing this. The example uses SSIS 2008 R2 with SQL Server 2012 backend.
Create a table to store your parameter values. Let's say the table name is dbo.SProcValues. Based on your stored procedure definition, the table schema would look like this.
CREATE TABLE dbo.SProcValues(
Id int IDENTITY(1,1) NOT NULL,
SProcName nvarchar(40) NOT NULL,
SchemaName nvarchar(20) NOT NULL,
Dimension nvarchar(40) NOT NULL,
ETLSchema nvarchar(20) NOT NULL,
ETLTable nvarchar(40) NOT NULL,
IsExecute bit NOT NULL
)
GO
Let's insert some sample data using the following script.
INSERT INTO dbo.SProcValues
(SProcName, SchemaName, Dimension, ETLSchema, ETLTable, IsExecute) VALUES
('dbo.sp_generate_merge', 'dim1', 'dimension1', 'stg1', 'table1', 1),
('dbo.sp_generate_merge_scdbk', 'dim2', 'dimension2', 'stg2', 'table2', 1),
('dbo.sp_generate_merge_scdbk', 'dim3', 'dimension3', 'stg3', 'table3', 0),
('dbo.sp_generate_merge', 'dim4', 'dimension4', 'stg4', 'table4', 0);
GO
On the SSIS package, assuming that you have the data source and connection manager already established. Create the following variables. Variable SProcValues will hold the parameter set that we stored in the above-mentioned table. Variable SQLInnerQuery will hold the query that will be used later in the inner Execute SQL Task. Other variables relate to each column available in the table so we can loop through each row and hold it in a variable.
Paste the following query in the value of the variable SQLGetParameters
SELECT SProcName, SchemaName, Dimension, ETLSchema, ETLTable, IsExecute FROM dbo.SProcValues
Select the variable SQLInnerQuery and press F4 to view the properties. Set the property EvaluateAsExpression to True and then click the Ellipsis button against the Expression property.
We need to set an expression that will evaluate to the EXEC stored procedure statement that can be later supplied to the inner Execute SQL Task. Set the following expression.
"EXEC " + #[User::SProcName] + " #Schema = ?, #Dimension = ?, #ETLSchema = ?, #ETLTable = ?, #IsExecute = ?"
If you click Evaluate Expression button on the editor, you can see what the expression will evaluate to. You will also notice that there is no stored procedure name in the below screenshot that is because the package variable SProcName currently does not have any value. During runtime, the SProcName will be assigned with the value from the table and this expression will automatically resolve itself.
On the SSIS package, drag and drop an Execute SQL Task. This task will run the following query to fetch the list of parameter values that are stored in the table dbo.SProcValues. Configure the General page on the Execute SQL Task as shown below. The example uses OLEDB connection and the connection manager/data source is named as Practice.
Configure the Result Set page of Execute SQL Task to store the result set from the query to an object variable.
Now that the first Execute SQL Task is configured to get the list of parameter values that should be passed to the stored procedure, you need to loop through the records.
Drag and drop a Foreach Loop container. Connect the Execute SQL Task's precedence container to the Foreach Loop container. Configure the Collection page of the Foreach Loop container as shown below. We are looping through the result set using the ADO enumerator.
Configure the Variable Mappings page on Foreach Loop container as shown below. As we loop through each row, we are storing the column values in respective variables so we can pass it to the next Execute SQL Task to run the stored procedure.
Drag and drop an Execute SQL Task inside the Foreach Loop container so that this task is executed each time we loop through a row in the result set. Configure the Execute SQL Task as shown below.
NOTE
You might want to configure the ResultSet property on this second Execute SQL Task according to your requirements. If you choose ResultSet, then you need to configure an appropriate object variable to accept the result set. I left it as None for this example.
Configure the values to be passed as parameters to the stored procedure.
Finally, the control flow would look something like this.
When the package runs, the loop will execute the stored procedure for as many records are returned by the SELECT query mentioned above, provided that you have all the stored procedures defined in the table rows are available in the database. I had created the stored procedures dbo.sp_generate_merge_scdbk and dbo.sp_generate_merge with the same parameters definition. That's the reason the package executed successfully.
You have the right concept, just need to use some concepts like variables, a foreach loop and parameters on the Execute SQL Task.
Control Flow
Your Control Flow would look something like this
Variables
I have 6 variables defined in SSIS
Dimension | String | VARIETY
ETLSchema | String | stg
ETLTable | String | vw_VARIETY
Execute | Int32 | 1
RecordSet | Object | System.Object
Schema | String | dim
The first Execute SQL Task will be a query or something enumerable like it. Currently, have a hard coded query to produce the supplied query values. Your solution could just be a chain of SELECT's UNIONed together. The goal of this step is to populate the RecordSet variable.
My Execute SQL Task returns a full result set
and I push that into my object thusly
ForEach Loop Container (ADO Recordset)
The ForEach Loop Container is going to consume that enumerable thing we established beforehand. It will go through each row and we will pop the values out of the object and assign them into local variables.
Change the Enumerator to Foreach ADO Enumerator. Select the object we populated with results User::RecordSet and then use an enumeration mode of Rows in first table
In the Variable Mappings tab, we will identify the ordinal based location for the values (column 0 maps to variable X). The only trick here is to ensure your SSIS Variable data types match the data type in the result set from your source query. Do note it's a zero based ordinal system.
At this point, if you click run you see it enumerate through all the rows you have sent into the RecordSet variable. I find it helpful to run it at this point to make sure I have all of my data types aligned.
Inner Execute SQL Task
I have taken your query and replaced the hard coded values with place holder. An OLEDB connection will use ? while an ADO.NET will use named #varname.
In the Parameter Mapping tab, simply map those local variables to the place holders.
Now you have a nice template for running the same proc with varying values.
I have a table and it has 500 rows. I want to retrieve only 10 rows and i want to insert into another table using control flow only. Through data flow task we can use OLEDB source and OLEDB destination. But i want result in such a way that by using execute sql task and for each loop. Is it possible to do in that way? My Idea is, get the set of ten records and and by using foreach loop iterate to every row and insert into the table by using execute sql task. The destination table need to create on the fly. I tried with some approach but not moving towards. Please find the image file.
Example taken from Northwind
Create variables (in variable collection) which represent the columns in the table which u ll create at runtime
Example :-
Customer_ID as string
Order_Id as int
Then u need to create Execute SQL Task and write the below query to select first 10 rows
Select top 10* from orders
Use FullResultSet and in Result Set configuration store the table rows in a variableName :- User::Result ResultName:0
Drop one Execute SQL Task and create a table on fly
IF OBJECT_ID('myOrders') IS not NULL
drop table myOrders
Create table myOrders
(OrderID int,
CustomerID varchar(50)
)
combine the 2 flows from Execute sql task and connect it to the Foreach loop
Drag a foreach loop .In collection use enumerator type as Foreach ADO Enumerator
In enumerator configuration select user::Result variable which stores the top 10 rows from the execute sql task and select the radio button " Rows in the first table"
In variable mapping ,map the column variables which u have created in the first step and the index will 0 for first column and 1 for 2nd column
Drag a execute sql task inside a foreach loop and write the below query :
Insert into myOrders( OrderID,CustomerID)
values
(?,?)
Map the parameters using parameter mapping configuration in execute sql task
VariableName : OrderID Direction : Input DataType=Long ParamterName=0
VariableName : CustomerID Direction : Input DataType=varchar ParamterName=1
I hope you are doing this on a "study-mode". There is no reason why to do this on the control flow over the data flow.
Anyway, your print screen is correct, I would just add another execute sql task in the beginning to create your destination table.
Then, your execute sql task should have the query to bring the 10 rows you want, its result set should be set to "Full result set" and on the resultset tab you should map the result set to a variable like this:
and configure your foreach loop container like this:
on each loop of the foreach you will have access to the values on the variables, then you can use another execute sql task to insert then on the new crated table