I want to set connection string using SQL Query result from my table. I need to import excel file from certain folder every month but i need to import only latest file name (FILE_YYYYMMDD)
Here is my table
I already follow some instruction using Execute SQL Task to get the result of the query and i want to assign the result as connection string to import excel file since the folder will get new file every month.
Here is my variable and my Result Set in Execute SQL Task
I want to make FilePath with max FileDate as Excel connection string in SSIS.
Is there anyway to do that?
Thanks
Your query can be like below :
CREATE TABLE MyTable(ID INT, FileName VARCHAR(255),FileDate DATE)
INSERT INTO MyTable(ID,FileName,FileDate) VALUES
(5,'C:\Users\Public\Documents\File_20200130.xls','2020-01-30'),
(5,'C:\Users\Public\Documents\File_20200131.xls','2020-01-31'),
(5,'C:\Users\Public\Documents\File_20220301.xls','2022-03-01')
SELECT FileName FROM MyTable
WHERE FileDate = (SELECT MAX(FileDate) FROM MyTable )
| FileName |
| :------------------------------------------ |
| C:\Users\Public\Documents\File_20220301.xls |
db<>fiddle here
Create a variable for the value we need to pass, the in the Execute SQL Task component, put the query mentioned above that returns the path with the max value of the date. Please make sure to set the result set to single row.
Then go to the variables tab and add the variable there.
To store the FileName as variable and use it as connection string for Excel source :
Select the Connection Manager, and in the Properties window, you will find the Expressions entry where you can use/concatenate the variable(s) in the ConnectionString entity.
Related
I need to extract data from a DB2 database to a SQL Server. I need to create my query based on a Excel file I have 176 records, which I need to create repeating queries & put in SQL server
So for example;
I have an Excel with a Number, From date, To date, and a Country
So the query should use these information from the records
SELECT *
FROM dbo.Test
WHERE Number = excel.Number1 AND Date BETWEEN excel.fromDate1 AND excel.toDate1 AND Country = excel.country1
And then another query with
SELECT *
FROM dbo.Test
WHERE Number = excel.Number2 AND Date BETWEEN excel.fromDate2 AND excel.toDate2 AND Country = excel.country2
Etc...
How should I do something like this in SSIS?
If needed I can put the DB2 and Excel data in MS SQL
You can proceed with the following approach:
Extract data rows from Excel and put it into SSIS Object Variable
Proceed with a Foreach loop to get each row from the Object Variable, parsing Object Variable to separate variables
Inject variable values into SQL Select command with Expressions
Perform Data Flow task based on SQL command, transform and put it into the target
Overall, your task seems to be feasible, but requires some knowledge on parsing Object Variable in Foreach Loop, and writing Variable Expressions.
Pretty new to BI and SQL in general, but a few months ago I didn't even know what a model is and now here I am...trying to build a package that runs daily.
Currently running this is Excel via PowerQuery but because the data is so much, I have to manually change the query every month. Decided to move it into SSIS.
Required outcome: Pull the last date in my Database and use it as a variable in the model (as I have millions of rows, I only want to load lines with dates greater than what I have in my table already).
Here is my Execute SQL Task:
I set up a variable for the SQL query
and trying to use it in my OLE DB query like this
Execute SQL Task: results, are fine - returns date as "dd/mm/yyyy hh24:mi:ss"
SELECT MAX (CONVACCT_CREATE_DATE) AS Expr1 FROM GOMSDailySales
Variable for OLE DB SQL Query:
"SELECT fin_booking_code, FIN_DEPT_CODE, FIN_ACCT_NO, FIN_PROD_CODE, FIN_PROG_CODE, FIN_OPEN_CODE, DEBIT_AMT, CREDIT_AMT, CURRENCY_CODE, PART_NO, FIN_DOC_NO, CREATE_DATE
FROM cuown.converted_accounts
WHERE (CREATE_DATE > TO_DATE(#[User::GetMaxDate],'yyyy/mm/dd hh24:mi:ss'))
AND (FIN_ACCT_NO LIKE '1%')"
Currently getting missing expression error, if I add " ' " to my #[User::GetMaxDate], I get a year must be between 0 and xxxx error.
What am I doing wrong / is there a cleaner way to get this done?
In the OLEDB source use the following, change the data access mode to SQL command, and use the following command:
SELECT fin_booking_code, FIN_DEPT_CODE, FIN_ACCT_NO, FIN_PROD_CODE, FIN_PROG_CODE, FIN_OPEN_CODE, DEBIT_AMT, CREDIT_AMT, CURRENCY_CODE, PART_NO, FIN_DOC_NO, CREATE_DATE
FROM cuown.converted_accounts
WHERE (CREATE_DATE > TO_DATE(?,'yyyy/mm/dd hh24:mi:ss'))
AND (FIN_ACCT_NO LIKE '1%')
And click on the parameters button and map #[User::GetMaxDate] to the first parameter.
For more information, check the following answer: Parameterized OLEDB source query
Alternative method
If parameters are not supported in the OLE DB provider you are using, create a variable of type string and evaluate this variable as the following expression:
"SELECT fin_booking_code, FIN_DEPT_CODE, FIN_ACCT_NO, FIN_PROD_CODE, FIN_PROG_CODE, FIN_OPEN_CODE, DEBIT_AMT, CREDIT_AMT, CURRENCY_CODE, PART_NO, FIN_DOC_NO, CREATE_DATE
FROM cuown.converted_accounts
WHERE CREATE_DATE > TO_DATE('" + (DT_WSTR, 50)#[User::GetMaxDate] +
"' ,'yyyy/mm/dd hh24:mi:ss') AND FIN_ACCT_NO LIKE '1%'"
Then from the OLE DB source, change the data access mode the SQL Command from variable and select the string variable you created.
Your trying to use the SSIS variable like a variable in the query. When constructing a SQL query in a string variable you simply need to concatenate the strings together. The expression for your query string variable should look like this.
"SELECT fin_booking_code, FIN_DEPT_CODE, FIN_ACCT_NO, FIN_PROD_CODE, FIN_PROG_CODE, FIN_OPEN_CODE, DEBIT_AMT, CREDIT_AMT, CURRENCY_CODE, PART_NO, FIN_DOC_NO, CREATE_DATE
FROM cuown.converted_accounts
WHERE CREATE_DATE > " + #[User::GetMaxDate] +
"AND (FIN_ACCT_NO LIKE '1%')"
I am having the following problem with populating a dynamic file path for an XML file from a SSIS variable.
In Visual Studio 2017 I have an Execute SQL Task with a MS SQL stored procedure that returns two columns. The first column is a date (stored as a string) and the second is a URL. These two columns (single row) populate SSIS variables and are mapped on the Result Set tab of the Script Task. The next step is script task that uses the URL from the variable to download an xml file from a web service. The xml file is stored using a file Connection Manager. The connection string for the file is an expression that should be using the 1st variable (User::rateDate) from the Execute SQL Task
Connection String expression:
#[User::xmlFileLocation] + "ExchangeRates-" + #[User::rateDate] + ".xml"
This evaluates to
\server\ExchangeRates\ExchangeRates-.xml
XML file should be saved as ExchangeRates-2017-12-19.xml with 2017-12-19 being the result of the stored procedure, but instead the XML file is saved as ExchangeRates-.xml
If I manually populate the User::rateDate variable it will use that in the Connection String, but I haven't been able to get it to populate from the stored procedure result.
The date generated is part of the URL generation too so I want this both created in the same place, i.e. I don't want to assign the file name via some GETDATE() logic in the expression.
I have confirmed the variable is being populated is a Script Task C# pop up.
I have confirmed that it is not a date/string issue by changing the stored procedure result to an explicit string, like "test". It still doesn't get added to the Connection String.
Thanks, Tim
I will provide 2 solutions based on the Stored Procedure type:
Stored Procedure with a Select Statement
I will assume that you are using a Stored procedure that contains a SELECT statement that return a Table of 2 columns: ServerURL and rateDate
In this case you have to insert the result of this select statement into a temp table, then read from these temp table, as example:
CREATE TABLE #TBL(ServerURL varchar(4000), rateDate varchar(50))
INSERT INTO #TBL EXEC pr_rateDate
SELECT TOP 1 * FROM #TBL
In this way your variables mapping should work
Stored Procedure with Output Parameters
I will assume that you are using a stored procedure which require 2 output parameters to be passed, example:
EXEC sp_rateDate #ServerURL OUTPUT, #rateDate OUTPUT
So you have to use the following SQL statemment:
EXEC sp_rateDate ? OUTPUT, ? OUTPUT
and you have to add 2 output parameters in the Parameter Mapping tab
Useful Links
Parameters and Return Codes in the Execute SQL Task Read Using Parameters with Stored Procedures part
SQL Assist - SSIS Execute SQL Task
Map Result Sets to Variables in an Execute SQL Task
How to Execute Stored Procedure in SSIS Execute SQL Task in SSIS
SSIS Basics: Using the Execute SQL Task to Generate Result Sets
Change the datatype of variable User::rateDate to datetime. Then, change your connection string expression to the following:
#[User::xmlFileLocation] +
"ExchangeRates-" +
(DT_WSTR,4)DATEPART("yyyy",#[User::rateDate]) +
RIGHT("0" + (DT_WSTR,2)DATEPART("mm",#[User::rateDate]) ,2) +
RIGHT("0" + (DT_WSTR,2)DATEPART("dd",#[User::rateDate]),2) +
".xml"
I have a stored procedure that gets executed through SQL SSIS using a Execute SQL Task.
The task has the following:
USE [OPPY_DWUSD]
GO
DECLARE #return_value int
EXEC #return_value = [dbo].[generate_merge_scdbk]
#Schema = N'dim',
#Dimension = N'VARIETY',
#ETLSchema = N'stg',
#ETLTable = N'vw_VARIETY',
#Execute = 1
SELECT 'Return Value' = #return_value
GO
Right now the way I have this setup, I have multiple Execute SQL Tasks with the same code but different values, about 20 Execute SQL Tasks.
Is there a more cleaner way to pull this off?
Here is one way of doing this. The example uses SSIS 2008 R2 with SQL Server 2012 backend.
Create a table to store your parameter values. Let's say the table name is dbo.SProcValues. Based on your stored procedure definition, the table schema would look like this.
CREATE TABLE dbo.SProcValues(
Id int IDENTITY(1,1) NOT NULL,
SProcName nvarchar(40) NOT NULL,
SchemaName nvarchar(20) NOT NULL,
Dimension nvarchar(40) NOT NULL,
ETLSchema nvarchar(20) NOT NULL,
ETLTable nvarchar(40) NOT NULL,
IsExecute bit NOT NULL
)
GO
Let's insert some sample data using the following script.
INSERT INTO dbo.SProcValues
(SProcName, SchemaName, Dimension, ETLSchema, ETLTable, IsExecute) VALUES
('dbo.sp_generate_merge', 'dim1', 'dimension1', 'stg1', 'table1', 1),
('dbo.sp_generate_merge_scdbk', 'dim2', 'dimension2', 'stg2', 'table2', 1),
('dbo.sp_generate_merge_scdbk', 'dim3', 'dimension3', 'stg3', 'table3', 0),
('dbo.sp_generate_merge', 'dim4', 'dimension4', 'stg4', 'table4', 0);
GO
On the SSIS package, assuming that you have the data source and connection manager already established. Create the following variables. Variable SProcValues will hold the parameter set that we stored in the above-mentioned table. Variable SQLInnerQuery will hold the query that will be used later in the inner Execute SQL Task. Other variables relate to each column available in the table so we can loop through each row and hold it in a variable.
Paste the following query in the value of the variable SQLGetParameters
SELECT SProcName, SchemaName, Dimension, ETLSchema, ETLTable, IsExecute FROM dbo.SProcValues
Select the variable SQLInnerQuery and press F4 to view the properties. Set the property EvaluateAsExpression to True and then click the Ellipsis button against the Expression property.
We need to set an expression that will evaluate to the EXEC stored procedure statement that can be later supplied to the inner Execute SQL Task. Set the following expression.
"EXEC " + #[User::SProcName] + " #Schema = ?, #Dimension = ?, #ETLSchema = ?, #ETLTable = ?, #IsExecute = ?"
If you click Evaluate Expression button on the editor, you can see what the expression will evaluate to. You will also notice that there is no stored procedure name in the below screenshot that is because the package variable SProcName currently does not have any value. During runtime, the SProcName will be assigned with the value from the table and this expression will automatically resolve itself.
On the SSIS package, drag and drop an Execute SQL Task. This task will run the following query to fetch the list of parameter values that are stored in the table dbo.SProcValues. Configure the General page on the Execute SQL Task as shown below. The example uses OLEDB connection and the connection manager/data source is named as Practice.
Configure the Result Set page of Execute SQL Task to store the result set from the query to an object variable.
Now that the first Execute SQL Task is configured to get the list of parameter values that should be passed to the stored procedure, you need to loop through the records.
Drag and drop a Foreach Loop container. Connect the Execute SQL Task's precedence container to the Foreach Loop container. Configure the Collection page of the Foreach Loop container as shown below. We are looping through the result set using the ADO enumerator.
Configure the Variable Mappings page on Foreach Loop container as shown below. As we loop through each row, we are storing the column values in respective variables so we can pass it to the next Execute SQL Task to run the stored procedure.
Drag and drop an Execute SQL Task inside the Foreach Loop container so that this task is executed each time we loop through a row in the result set. Configure the Execute SQL Task as shown below.
NOTE
You might want to configure the ResultSet property on this second Execute SQL Task according to your requirements. If you choose ResultSet, then you need to configure an appropriate object variable to accept the result set. I left it as None for this example.
Configure the values to be passed as parameters to the stored procedure.
Finally, the control flow would look something like this.
When the package runs, the loop will execute the stored procedure for as many records are returned by the SELECT query mentioned above, provided that you have all the stored procedures defined in the table rows are available in the database. I had created the stored procedures dbo.sp_generate_merge_scdbk and dbo.sp_generate_merge with the same parameters definition. That's the reason the package executed successfully.
You have the right concept, just need to use some concepts like variables, a foreach loop and parameters on the Execute SQL Task.
Control Flow
Your Control Flow would look something like this
Variables
I have 6 variables defined in SSIS
Dimension | String | VARIETY
ETLSchema | String | stg
ETLTable | String | vw_VARIETY
Execute | Int32 | 1
RecordSet | Object | System.Object
Schema | String | dim
The first Execute SQL Task will be a query or something enumerable like it. Currently, have a hard coded query to produce the supplied query values. Your solution could just be a chain of SELECT's UNIONed together. The goal of this step is to populate the RecordSet variable.
My Execute SQL Task returns a full result set
and I push that into my object thusly
ForEach Loop Container (ADO Recordset)
The ForEach Loop Container is going to consume that enumerable thing we established beforehand. It will go through each row and we will pop the values out of the object and assign them into local variables.
Change the Enumerator to Foreach ADO Enumerator. Select the object we populated with results User::RecordSet and then use an enumeration mode of Rows in first table
In the Variable Mappings tab, we will identify the ordinal based location for the values (column 0 maps to variable X). The only trick here is to ensure your SSIS Variable data types match the data type in the result set from your source query. Do note it's a zero based ordinal system.
At this point, if you click run you see it enumerate through all the rows you have sent into the RecordSet variable. I find it helpful to run it at this point to make sure I have all of my data types aligned.
Inner Execute SQL Task
I have taken your query and replaced the hard coded values with place holder. An OLEDB connection will use ? while an ADO.NET will use named #varname.
In the Parameter Mapping tab, simply map those local variables to the place holders.
Now you have a nice template for running the same proc with varying values.
I have a table and it has 500 rows. I want to retrieve only 10 rows and i want to insert into another table using control flow only. Through data flow task we can use OLEDB source and OLEDB destination. But i want result in such a way that by using execute sql task and for each loop. Is it possible to do in that way? My Idea is, get the set of ten records and and by using foreach loop iterate to every row and insert into the table by using execute sql task. The destination table need to create on the fly. I tried with some approach but not moving towards. Please find the image file.
Example taken from Northwind
Create variables (in variable collection) which represent the columns in the table which u ll create at runtime
Example :-
Customer_ID as string
Order_Id as int
Then u need to create Execute SQL Task and write the below query to select first 10 rows
Select top 10* from orders
Use FullResultSet and in Result Set configuration store the table rows in a variableName :- User::Result ResultName:0
Drop one Execute SQL Task and create a table on fly
IF OBJECT_ID('myOrders') IS not NULL
drop table myOrders
Create table myOrders
(OrderID int,
CustomerID varchar(50)
)
combine the 2 flows from Execute sql task and connect it to the Foreach loop
Drag a foreach loop .In collection use enumerator type as Foreach ADO Enumerator
In enumerator configuration select user::Result variable which stores the top 10 rows from the execute sql task and select the radio button " Rows in the first table"
In variable mapping ,map the column variables which u have created in the first step and the index will 0 for first column and 1 for 2nd column
Drag a execute sql task inside a foreach loop and write the below query :
Insert into myOrders( OrderID,CustomerID)
values
(?,?)
Map the parameters using parameter mapping configuration in execute sql task
VariableName : OrderID Direction : Input DataType=Long ParamterName=0
VariableName : CustomerID Direction : Input DataType=varchar ParamterName=1
I hope you are doing this on a "study-mode". There is no reason why to do this on the control flow over the data flow.
Anyway, your print screen is correct, I would just add another execute sql task in the beginning to create your destination table.
Then, your execute sql task should have the query to bring the 10 rows you want, its result set should be set to "Full result set" and on the resultset tab you should map the result set to a variable like this:
and configure your foreach loop container like this:
on each loop of the foreach you will have access to the values on the variables, then you can use another execute sql task to insert then on the new crated table