I am trying to create an SSIS package for Stored Procedure Deployment and Backup for our project.
I have some .sql file, each file contains one stored procedure definition and the name of the file is the stored procedure name itself.
I am trying to do the following by using SSIS
Read all files names one by one
Find the definition of each stored procedure if it exists in the database
If exists, then save the definition with the same name in a different folder (In my case it's a ROLLBACK folder)
For all new SP it save in a same file named DropNewSp.sql with DROP STORED PROCEDURE command.
After completing the backup process, execute all files in the destination database.
I am able to generate the desire .sql files, but I am faceing the following problem
1. The package also generated unwanted blank .sql file for all new Stored Procedure
2. The execution process failed if the stored procedure has some dependency on subsequent stored proc
In this answer, I will provide the main steps with some references to get more information on how to achieve each step. Even if I agree with the comments mentioned above that this is not the job of SSIS.
Add a foreach loop container that loop over .sql files and store the file name inside a variable:
Looping over files with the Foreach Loop
Lesson 2-2: Add and configure the Foreach Loop container
Load multiple source files
Add an Expression Task to retrieve the file name from the File Full Path (variable)
#{User::FileNameWithoutExtension] = SUBSTRING (#[User::FullFilePath], LEN( #[User::FullFilePath] ) - FINDSTRING( REVERSE( #[User::FullFilePath] ), "\\", 1) + 2, LEN (RIGHT( #[User::FullFilePath], FINDSTRING( REVERSE( #[User::FullFilePath] ), "\\", 1 ) - 1 ) ) - FINDSTRING( REVERSE( #[User::FullFilePath] ), ".", 1 ) )
Add an Execute SQL Task inside the foreach loop container to check if the stored procedure is found in the database:
SELECT COUNT(*) FROM sys.objects WHERE type = 'P' AND name = ?)
Pass the procedure name as parameter to the execute sql task:
Passing Variables to and from an SSIS task
Store the count result inside a variable of type integer using Resultsets:
SSIS Basics: Using the Execute SQL Task to Generate Result Sets
Using precedence constraints with expressions add 2 paths from the execute sql task
the first using expression #[User::Count] == 0
the second using expression #[User::Count] > 0
Other references:
Working with Precedence Constraints in SQL Server Integration Services
Defining Workflow in SSIS using Precedence Constraints
On the second path add an Execute SQL Task to get the procedure definition using the same approach above:
SELECT OBJECT_DEFINITION (OBJECT_ID(N'<databasename>.<schemaname>.' + CAST(? as VARCHAR(100))));
And store the result inside a variable using a result set.
Add a Script Task to write the procedure definition into the destination file
On the first path add a File system task to move the file into the directory specified
Add another foreach loop to read new files and execute the content.
Related
I have SSIS package with data flow task and execute SQL task components in For each loop container. Package flow is, Date flow task(flat file--> Conditional split to insert data into SQL server tables)-->Execute SQL task(perform some SQL operations on inserted data and insert the calculated values in one final analysis table) . File name is like name1_name2_yyyymmdd_1234.txt. I want to fetch the date from file name and insert that date value in table in SQL Server as FileDate. I am trying to do it using derived column but unable to figure our where will I save it so that it will be available in Insert statement in Execute SQL Task which is after Data flow task.
This should be done outside the dataflow but within the ForEach loop.
Pass in two parameters (package scope) to a script task. One with the #filename (read only) from the forloop and to store the #fileDate (read/write).
Split will create a 0-based array in which you only care about the third piece.
Dts.Variables["fileDate"].Value = DateTime.ParseExact(Dts.Variables["fileName"].Value.Split('_')[2]
,"yyyyMMdd", System.Globalization.CultureInfo.InvariantCulture);
Now you can use #fileDate anywhere you would like.
I have created a package in ssis in which i use some date-variables inside my SQL statements ( i.e declare #DateIn ="2018-02-22" and declare #DateTo = "2018-03-22"), in order to load the corresponding data inside the tables of the data warehouse.
What I need to do is to create a task or a different package, which will give me the possibility to define externally the values of these variables, every time i run it, in order to fill in the tables of the warehouse with the data that corresponds to the dates i set every time.
From what I've read, I should maybe use a script task or an execute sql task or parameters
Could you help me please? Or could you suggest me a good tutorial/link?
I have found plenty but can't decide if they meet the needs of what i am describing above.
Thank you
Create DTSX package with variables #DateStart and #DateEnd
Create table containing 3 columns DateStart, DateEnd, Active
Create stored procedure that reads DateStart, DateEnd where Active = 1 from your newly created table and does an alter on the SQL Server Job updating your variables value that are inside of your DTSX package with your desired value using sp_update_jobstep
See link
Ex of command:
dtexec /f YourPackage.dtsx
/set \package.variables[DateStart].Value;myvalue
/set \package.variables[DateStart].Value;myvalue
Add sp_start_job inside the stored procedure to start the job with the new variable values.
Create job with 1 step containing the execute of the stored procedure from Step 3
All you need to do is update the values from your table created in Step 2 and then execute job to run the stored procedure to update DTSX job exec command and start it. You can trigger this from a website and control the tables values from textboxes.
Also specific Permissions are required and the SP that updates the SQL Agent job needs to be run by Sysadmin
Good question by the way for the new learner!
There are many ways for this scenario,few of them I have mentioned below.
1-Create variable in variable pane #DateIn and #DateTo for storing the date and data type will be date.
Now put 2 entry in Excel ,text or xml for these two variables and call it by using foreachloop container and assign this to variables.
2-Create a SQl table in which you can store those values either by manually on daily basis or load the table with excel ,text ,xml or csv file and call the table in Execute SQL Task and select the result set and pass the result set values to the variables.
I hope it will solve your problem.
I am having the following problem with populating a dynamic file path for an XML file from a SSIS variable.
In Visual Studio 2017 I have an Execute SQL Task with a MS SQL stored procedure that returns two columns. The first column is a date (stored as a string) and the second is a URL. These two columns (single row) populate SSIS variables and are mapped on the Result Set tab of the Script Task. The next step is script task that uses the URL from the variable to download an xml file from a web service. The xml file is stored using a file Connection Manager. The connection string for the file is an expression that should be using the 1st variable (User::rateDate) from the Execute SQL Task
Connection String expression:
#[User::xmlFileLocation] + "ExchangeRates-" + #[User::rateDate] + ".xml"
This evaluates to
\server\ExchangeRates\ExchangeRates-.xml
XML file should be saved as ExchangeRates-2017-12-19.xml with 2017-12-19 being the result of the stored procedure, but instead the XML file is saved as ExchangeRates-.xml
If I manually populate the User::rateDate variable it will use that in the Connection String, but I haven't been able to get it to populate from the stored procedure result.
The date generated is part of the URL generation too so I want this both created in the same place, i.e. I don't want to assign the file name via some GETDATE() logic in the expression.
I have confirmed the variable is being populated is a Script Task C# pop up.
I have confirmed that it is not a date/string issue by changing the stored procedure result to an explicit string, like "test". It still doesn't get added to the Connection String.
Thanks, Tim
I will provide 2 solutions based on the Stored Procedure type:
Stored Procedure with a Select Statement
I will assume that you are using a Stored procedure that contains a SELECT statement that return a Table of 2 columns: ServerURL and rateDate
In this case you have to insert the result of this select statement into a temp table, then read from these temp table, as example:
CREATE TABLE #TBL(ServerURL varchar(4000), rateDate varchar(50))
INSERT INTO #TBL EXEC pr_rateDate
SELECT TOP 1 * FROM #TBL
In this way your variables mapping should work
Stored Procedure with Output Parameters
I will assume that you are using a stored procedure which require 2 output parameters to be passed, example:
EXEC sp_rateDate #ServerURL OUTPUT, #rateDate OUTPUT
So you have to use the following SQL statemment:
EXEC sp_rateDate ? OUTPUT, ? OUTPUT
and you have to add 2 output parameters in the Parameter Mapping tab
Useful Links
Parameters and Return Codes in the Execute SQL Task Read Using Parameters with Stored Procedures part
SQL Assist - SSIS Execute SQL Task
Map Result Sets to Variables in an Execute SQL Task
How to Execute Stored Procedure in SSIS Execute SQL Task in SSIS
SSIS Basics: Using the Execute SQL Task to Generate Result Sets
Change the datatype of variable User::rateDate to datetime. Then, change your connection string expression to the following:
#[User::xmlFileLocation] +
"ExchangeRates-" +
(DT_WSTR,4)DATEPART("yyyy",#[User::rateDate]) +
RIGHT("0" + (DT_WSTR,2)DATEPART("mm",#[User::rateDate]) ,2) +
RIGHT("0" + (DT_WSTR,2)DATEPART("dd",#[User::rateDate]),2) +
".xml"
I have a stored procedure that gets executed through SQL SSIS using a Execute SQL Task.
The task has the following:
USE [OPPY_DWUSD]
GO
DECLARE #return_value int
EXEC #return_value = [dbo].[generate_merge_scdbk]
#Schema = N'dim',
#Dimension = N'VARIETY',
#ETLSchema = N'stg',
#ETLTable = N'vw_VARIETY',
#Execute = 1
SELECT 'Return Value' = #return_value
GO
Right now the way I have this setup, I have multiple Execute SQL Tasks with the same code but different values, about 20 Execute SQL Tasks.
Is there a more cleaner way to pull this off?
Here is one way of doing this. The example uses SSIS 2008 R2 with SQL Server 2012 backend.
Create a table to store your parameter values. Let's say the table name is dbo.SProcValues. Based on your stored procedure definition, the table schema would look like this.
CREATE TABLE dbo.SProcValues(
Id int IDENTITY(1,1) NOT NULL,
SProcName nvarchar(40) NOT NULL,
SchemaName nvarchar(20) NOT NULL,
Dimension nvarchar(40) NOT NULL,
ETLSchema nvarchar(20) NOT NULL,
ETLTable nvarchar(40) NOT NULL,
IsExecute bit NOT NULL
)
GO
Let's insert some sample data using the following script.
INSERT INTO dbo.SProcValues
(SProcName, SchemaName, Dimension, ETLSchema, ETLTable, IsExecute) VALUES
('dbo.sp_generate_merge', 'dim1', 'dimension1', 'stg1', 'table1', 1),
('dbo.sp_generate_merge_scdbk', 'dim2', 'dimension2', 'stg2', 'table2', 1),
('dbo.sp_generate_merge_scdbk', 'dim3', 'dimension3', 'stg3', 'table3', 0),
('dbo.sp_generate_merge', 'dim4', 'dimension4', 'stg4', 'table4', 0);
GO
On the SSIS package, assuming that you have the data source and connection manager already established. Create the following variables. Variable SProcValues will hold the parameter set that we stored in the above-mentioned table. Variable SQLInnerQuery will hold the query that will be used later in the inner Execute SQL Task. Other variables relate to each column available in the table so we can loop through each row and hold it in a variable.
Paste the following query in the value of the variable SQLGetParameters
SELECT SProcName, SchemaName, Dimension, ETLSchema, ETLTable, IsExecute FROM dbo.SProcValues
Select the variable SQLInnerQuery and press F4 to view the properties. Set the property EvaluateAsExpression to True and then click the Ellipsis button against the Expression property.
We need to set an expression that will evaluate to the EXEC stored procedure statement that can be later supplied to the inner Execute SQL Task. Set the following expression.
"EXEC " + #[User::SProcName] + " #Schema = ?, #Dimension = ?, #ETLSchema = ?, #ETLTable = ?, #IsExecute = ?"
If you click Evaluate Expression button on the editor, you can see what the expression will evaluate to. You will also notice that there is no stored procedure name in the below screenshot that is because the package variable SProcName currently does not have any value. During runtime, the SProcName will be assigned with the value from the table and this expression will automatically resolve itself.
On the SSIS package, drag and drop an Execute SQL Task. This task will run the following query to fetch the list of parameter values that are stored in the table dbo.SProcValues. Configure the General page on the Execute SQL Task as shown below. The example uses OLEDB connection and the connection manager/data source is named as Practice.
Configure the Result Set page of Execute SQL Task to store the result set from the query to an object variable.
Now that the first Execute SQL Task is configured to get the list of parameter values that should be passed to the stored procedure, you need to loop through the records.
Drag and drop a Foreach Loop container. Connect the Execute SQL Task's precedence container to the Foreach Loop container. Configure the Collection page of the Foreach Loop container as shown below. We are looping through the result set using the ADO enumerator.
Configure the Variable Mappings page on Foreach Loop container as shown below. As we loop through each row, we are storing the column values in respective variables so we can pass it to the next Execute SQL Task to run the stored procedure.
Drag and drop an Execute SQL Task inside the Foreach Loop container so that this task is executed each time we loop through a row in the result set. Configure the Execute SQL Task as shown below.
NOTE
You might want to configure the ResultSet property on this second Execute SQL Task according to your requirements. If you choose ResultSet, then you need to configure an appropriate object variable to accept the result set. I left it as None for this example.
Configure the values to be passed as parameters to the stored procedure.
Finally, the control flow would look something like this.
When the package runs, the loop will execute the stored procedure for as many records are returned by the SELECT query mentioned above, provided that you have all the stored procedures defined in the table rows are available in the database. I had created the stored procedures dbo.sp_generate_merge_scdbk and dbo.sp_generate_merge with the same parameters definition. That's the reason the package executed successfully.
You have the right concept, just need to use some concepts like variables, a foreach loop and parameters on the Execute SQL Task.
Control Flow
Your Control Flow would look something like this
Variables
I have 6 variables defined in SSIS
Dimension | String | VARIETY
ETLSchema | String | stg
ETLTable | String | vw_VARIETY
Execute | Int32 | 1
RecordSet | Object | System.Object
Schema | String | dim
The first Execute SQL Task will be a query or something enumerable like it. Currently, have a hard coded query to produce the supplied query values. Your solution could just be a chain of SELECT's UNIONed together. The goal of this step is to populate the RecordSet variable.
My Execute SQL Task returns a full result set
and I push that into my object thusly
ForEach Loop Container (ADO Recordset)
The ForEach Loop Container is going to consume that enumerable thing we established beforehand. It will go through each row and we will pop the values out of the object and assign them into local variables.
Change the Enumerator to Foreach ADO Enumerator. Select the object we populated with results User::RecordSet and then use an enumeration mode of Rows in first table
In the Variable Mappings tab, we will identify the ordinal based location for the values (column 0 maps to variable X). The only trick here is to ensure your SSIS Variable data types match the data type in the result set from your source query. Do note it's a zero based ordinal system.
At this point, if you click run you see it enumerate through all the rows you have sent into the RecordSet variable. I find it helpful to run it at this point to make sure I have all of my data types aligned.
Inner Execute SQL Task
I have taken your query and replaced the hard coded values with place holder. An OLEDB connection will use ? while an ADO.NET will use named #varname.
In the Parameter Mapping tab, simply map those local variables to the place holders.
Now you have a nice template for running the same proc with varying values.
I'm busy setting up an SSIS package that takes a flat file and loads its content into a staging table. From there it needs to take the file name of that file and save it into a field on another table, along with one of the lines of data. The loading of the data I do by using an execute SQL task that executes a stored procedure. This procedure takes the file name as input parameter (just so I can write it to that table)... I added a Script task before the execute SQL Task that does the following:
public void Main()
{
string inputFolder;
string fileName;
inputFolder = Dts.Variables["SourceFolder"].Value.ToString();
// MessageBox.Show(inputFolder);
string[] foundFiles = null;
foundFiles = System.IO.Directory.GetFiles(inputFolder);
if (foundFiles.Length > 0)
{
fileName = foundFiles[0];
Dts.Variables["ProcessedFileName"].Value = System.IO.Path.GetFileName(fileName);
// MessageBox.Show(fileName);
// MessageBox.Show(System.IO.Path.GetFileName(fileName));
}
Dts.TaskResult = (int)ScriptResults.Success;
}
The "ProcessedFileName" variable is a global variable in the SSIS package, as well as the "SourceFolder" variable.
The execute SQL task I set up using Direct input executing:
DECLARE #StoredProcedure varchar(100) = ?;
DECLARE #ProcessedFileName varchar(100) = ?;
EXEC ? ?;
The parameter mapping is to variables StoredProcedure and ProcessedFileName.
The problem is when I run this it tells me "No value given for one or more required parameters." Why could this be?
You cannot dynamically change a stored procedure name in an Execute SQL Task using parameters. That is what EXEC ? ?; is doing.
If you need to change the procedure name at run-time then you'll need to use string concatenation via expressions to build our your query.
Bigger picture, it seems that you are using a script task where a For Each (file) Enumerator task would suffice. There are reasons to use a script task over foreach (like if you need to process in order, etc) but it doesn't read like you're doing that.
Assorted answers that may be of help on how to accomplish expressions and foreach enumerator
SSIS: Get any flat file source from folder and cache the name as a super global variable
Passing a variable through DTEXEC with xp_cmdshell (SQL Server 2008)
Import most recent csv file to sql server in ssis
fetching data from multiple file and loading it into raw file destination(raw file should be dynamic) in ssis