I really like tsqlt to test procs and functions, but really would like to be able to also execute SSIS packages and take advantage of FakeTable and AssertEquals to determine if it was the SSIS package did what it was supposed to.
Has anyone explored this path, is it possible to call dtexec from with the transaction that tsqlt wraps your test in?
I believe I can answer your question Andrey, although this is a little late in coming. But I believe that it will benefit others.
We are using RedGate SQLTest(tSQLt) to do data quality testing as a part of our integration testing.
For example to test the completeness of the data being loaded into Staging, on test would be to AssertEqualsTable after a package loads a staging table. Here is the basic order of things:
Assemble
Create and load the expected table with data.
Act
Execute the SSIS Package in the catalog via t-sql. You can generate t-sql code to call any package in the catalog as follows:
Locate the package you're testing in it's folder in the catalog
Right click and select 'Execute'
The Execute Package dialogue box will open.
Click the scripting dropdown and select 'Script to Clipboard'
All the t-SQL Code needed to execute the package from a stored procedure or script is generated:
DECLARE #execution_id BIGINT
EXEC [SSISDB].[catalog].[create_execution]
#package_name=N'HistoricalLoad_import_rti_stores_s1.dtsx'
, #execution_id=#execution_id OUTPUT
, #folder_name=N'Testing'
, #project_name=N'Staging1_HistoricalLoad_RTIStores'
, #use32bitruntime=FALSE
, #reference_id=NULL
SELECT #execution_id
DECLARE #var0 SMALLINT = 1
EXEC [SSISDB].[catalog].[set_execution_parameter_value]
#execution_id
, #object_type=50
, #parameter_name=N'LOGGING_LEVEL'
, #parameter_value=#var0
EXEC [SSISDB].[catalog].[start_execution] #execution_id
Go back to your test stored proc and paste the code into the Act section.
Assert
- Select into the actual table from the SSIS destination table of the package being tested.
then validate that the expected and actual are equal
EXEC tSQLt.AssertEqualsTable 'expected', 'actual';
And that's all there is too it.
Take a look at the foreign key tests in the examples database to guide you on foreign key and referential integrity tests.
I've found it to be invaluable as a means of regression testing our data warehouse load functionality and also validating our orchestration. Because if we can verify that the data is flowing into the right place, at the right time, then things are executing as expected.
tSQLt is a Unit Testing Framework and it is designed for testing code in isolation.
So for testing how your code/data will be integrated with the other code/data typically used different types of tests - Integration Tests.
LATER UPDATE
Not exactly about the topic but it may be useful information about unit/integration testing of SSIS packages
There is a sample project with unit tests for SSIS at http://ssistester.codeplex.com/. Few samples show the use of the FakeSource and the FakeDestination to assert if data flow streams correctly read/write data.
Related
I'm trying to pass a variable value from SQL Server Agent job to SSIS package but the variable contains an apostrophe in it causing the SQL Server Agent job to fail
e.g In SQL Server Agent at Job Step Properties I'm entering the following details:
Property Path: \Package.Variables[User::VariableName].Properties[Value] Property
Value: Michael O'Callaghan.
Any idea how to resolve this issue?
If the package is deployed to SSISDB and executed from there, use SSISDB stored procedures to set the value and escape the quote like how you would via T-SQL. The SQL Agent job can then use a T-SQL script for this step instead. The example below uses the set_execution_parameter_value stored procedure to set this value and will still result in "Michael O'Callaghan" being passed in.
DECLARE #execution_id BIGINT
EXEC [SSISDB].[catalog].[create_execution] #package_name=N'Package.dtsx', #execution_id=#execution_id OUTPUT,
#folder_name=N'Project Folder', #project_name=N'Project', #use32bitruntime=False, #reference_id=Null
DECLARE #var0 SQL_VARIANT = N'Michael O''Callaghan'
EXEC [SSISDB].[catalog].[set_execution_parameter_value] #execution_id, #object_type=30, #parameter_name=N'Name', #parameter_value=#var0
DECLARE #var1 SMALLINT = 1
EXEC [SSISDB].[catalog].[set_execution_parameter_value] #execution_id, #object_type=50, #parameter_name=N'LOGGING_LEVEL', #parameter_value=#var1
EXEC [SSISDB].[catalog].[start_execution] #execution_id
Escape it. Just use a double apostrophe. '' (Not a quotation ", but a apostrophe apostrophe).
Try the standard way of maintaining a configuration file(if you are using 2008 or less) and pass the variables values through the file.
An alternative way to handle this, and frankly I think the best way, is to use Environment Variables. To my knowledge, this was introduced when Microsoft rolled out the project deployment model with SQL Server 2012 as a replacement to the package deployment model. The package deployment model required that package parameters be specified in a separate XML file to be deployed to the server. With the project deployment model, Microsoft has created a user-friendly user interface in SQL Server to manage this - the XML file has been removed.
In short, Environment Variables allow developers to link package parameters, but not package variables as those are internal to the package itself, to SQL Server and thus expose them on the server. This makes management of identical package parameters that exist across packages (e.g., connection managers, network folder locations in FQDN format, etc.) incredibly easy to manage. The idea here is that if packages need to be pointed to a new server or new network folder, then developers can simply change a single value in SQL Server, which would then propogate out to all packages without the need to open, change, and re-deploy the package.
For detailed steps on how to do this, see the following references:
Microsoft: This is a bit dry, but is comprehensive and from the horse's mouth.
Deploy Packages with SSIS
Lesson 1: Preparing to Create the Deployment Bundle
SQL Chick: More intuitive and provides screenshots, which I found helpful.
Parameterizing Connections and Values at Runtime Using SSIS Environment Variables
Thanks for your all you suggestions but unfortunately they didn't work, however I built a clever workaround for this.
SQL server agent wraps a variable value in single quote e.g specifying Jon Doe in sql server agent, the agent wraps it like this 'Jon Doe' and passes it to the SSIS package, so if you were to enter a value with an apostrophe it would break the sql server agent job and won't execute the SSIS package it would look like this E.G passing this value: 'John O' Doe' this would cause the agent the job to break so you need to pass your variable value as : John O''Doe and the agent wraps it as follows: 'John O''''Doe' so you would need to include the following logic in your SSIS package:
Declare #TempVar nVarchar(50)
SET #TempVar = REPLACE(?, '''''', CHAR(39))
The above code creates a variable to store the parameter value. It replaces the 4 single quotes to one. CHAR(39) is the ASCII representation of a single quote.
This would then cause the variable value to look like John O'Doe.
Hope this helps.
The reason I wanted to pass a variable value from the agent as I had to change the variable value very often from the SSIS package it would need to be deployed every time. So this way is faster.
Earlier we were in SSSIS 2008, we have 3 projects under a solution, Project A, Project B, Project C. I have Master Package in Project B from which I am calling packages in ProjectA and ProjectC in execute package task using external reference as File System, everything worked well till here. We recently moved from SSIS 2008 to SSIS 2012 and used Project deployment model to deploy packages to SSISDB instead of File System using Package Deployment Model, so I replaced Execute Package task with Execute SQL task in my master package to call packages from Project A and C.
I created Connection to SSISDB and used it in Execute SQL task and placed the below T-SQL code in it, it worked fine here also.
Declare #execution_id bigint
EXEC [SSISDB].[catalog].[create_execution]
#package_name=N'Child_Package.dtsx',
#execution_id=#execution_id OUTPUT,
#folder_name=N'Test',
#project_name=N'ProjectA',
#use32bitruntime=False, #reference_id=Null
Select #execution_id
DECLARE #var0 smallint = 1
EXEC [SSISDB].[catalog].[set_execution_parameter_value]
#execution_id,
#object_type=50,
#parameter_name=N'LOGGING_LEVEL',
#parameter_value=#var0
EXEC [SSISDB].[catalog].[start_execution] #execution_id
GO
I have 2 question here
1) If I deploy my packages from development to Production server, will this T-SQL code inside Execute SQL task works there without any issues (I read in one post that it will have some issues when deployed to diff environment) I am not hard-coding any environment parameter value inside my T-SQL code, will this work in another environment as well?
2) I am trying to implement rollback transaction support in my master package, so if any child package fails, everything needs to be rolled back, so I have Changed the package TransactionOption property to required and all other tasks(Execute SQL TASK) as supported(Default), I am ending up with an error like
"[Execute SQL Task] Error: Executing the query "Declare #execution_id bigint
EXEC [SSISDB].[catalo..." failed with the following error: "Cannot use SAVE TRANSACTION within a distributed transaction.". Possible failure reasons: Problems with the query, "ResultSet" property not set correctly, parameters not set correctly, or connection not established correctly"
The package is working fine if the TransactionOption property of the package is supported, the problem comes only when I use TransactionOption property of package as Required.
Any help is much appreciated.
So i have this database called crm and i need to select a list of all the stored procedures that depend on a specific table/column.
I did some research and i learned about sp_depends, i tried to execute but it reports me and error.
Here's the code:
EXEC sp_depends #objname = 'TCdPost'
TCdPost stands for a column that contains the postal code.
As i execute the query i get an error that says:
The object 'TCdPost' does not exist in database 'crm' or is invalid for this operation.
Does anyone know what causes this error or some better ways to select all the stored procedures that depend on a table/column?
Im using SQL Server 2008 R2
Thank you.
This feature will be removed in a future version of Microsoft SQL Server. Avoid using this feature in new development work, and plan to modify applications that currently use this feature. Use sys.dm_sql_referencing_entities and sys.dm_sql_referenced_entities instead.
So would suggest not to use it.
Also, you are passing wrong object name to the procedure.
Try This one:
USE crm
Go
EXEC sp_depends 'dbo.YourTableName'
If you want the more specific details for columns, refer this.
try this
USE crm
EXEC sp_depends 'TCdPost'
or if you to want to view the dependencies of the table you can see it by right click the table and then click "View Dependencies" thanks
Where I work the production databases are backed up nightly and restored to dev, test, and QA environments. When running any of the programs in non-production environments, to avoid making changes we don't want to make in production, such as sending real users email, our programs test the environments using a combination of internally parsing the command line, and calling an SQL user function. The function selects ##SERVERNAME, then parses the result looking for specific strings ie. if the ServerName contains "-PROD-", it is a production server.
The problem is the hardware group is implementing a high availability project so if a server fails ##SERVERNAME will return the name of the backup server. I am looking at extending the current logic to account for whatever the fail-over server names will be, but I was hoping there was a better way to test the environment than parsing text for static strings.
Store a setting in a database that is separate from your application database(s) then read that setting as-needed using a function. When your application runs in production, you'll get the production values. When your application runs in Development, you'll get the development values.
The nice thing about this is you can store all kinds of values and easily get to them from your SPROCS, PowerShell or whatever front end you have.
CREATE DATABASE SETTINGSDB
GO
USE SETTINGSDB
GO
-- A table to hold key/value pairs
CREATE TABLE MYSETTINGS
(
SettingName VARCHAR(50) PRIMARY KEY CLUSTERED ,
SettingValue VARCHAR(500)
)
GO
-- On DEVELOPMENT SERVER, run this
INSERT INTO MYSETTINGS
VALUES ('ENVIRONMENT', 'DEV'),
('SOME_SETTING', 'True')
-- On PRODUCTION SERVER, run this
INSERT INTO MYSETTINGS
VALUES ('ENVIRONMENT', 'PROD'),
('SOME_SETTING', 'False')
GO
-- A function to pull key/value pairs.
CREATE FUNCTION dbo.GetEnvVar( #SettingName VARCHAR(50) )
RETURNS VARCHAR(500)
AS
BEGIN
RETURN (SELECT SettingValue FROM SETTINGSDB.dbo.MYSETTINGS WHERE SettingName = #SettingName)
END
GO
Once you are setup, you can then check the value, and it will be different between DEV/PROD. For example:
-- Then use these values:
USE YourApplicationDatabaseNameHere
GO
CREATE PROCEDURE SampleApplicationSprocThatSendsEmail
#EmailAddress VARCHAR(50),
#Subject VARCHAR(50)
AS
IF (dbo.GetEnvVar('ENVIRONMENT') = 'PROD' )
BEGIN
-- Only Executes in Production
-- TODO: SEND THE EMAIL
END ELSE
BEGIN
-- Only Executes in Development
PRINT 'Send email to ' + #EmailAddress
END
Instead of finding all possible objects that might cause harm or annoyance to some unknown number of targets (mailboxes, files, databases, etc...) why not just isolate the dev/test environment at the network layer? Put them in an isolated network/subnet where only inbound is permitted, all outbound gets blocked or rerouted. It's hard to know for sure you've gotten every single endpoint. For example, what happens if your admins add more secondaries for additional protection and read-only queries?
We used SQL Server's trace replay capabilities regularly for many years for service pack, upgrade (app and db), regression, cross-workload and other tests. Every so often we'll have some smart, highly motivated new team member (new hire or transfer) who would write scripts or "cleansing" apps to scrub the trace files and databases so the tests can be run on his/her workstation. Every single one of them will learn that's a bad idea, sometimes in a very, very hard way (e.g. re-index a clustered index on a highly volatile 2billion row table in mid-morning).
Blocking at the network layer has the added benefit of minimal prep/setup work for each test run plus you can have an identical setup as prod. When you encounter bugs or regressions, you have a few items less to check.
My company considers database scripts we write part of our intellectual property.
With new releases, we deliver a 2-part setup for our users:
a desktop application
an executable that wraps up the complexities of initializing/updating a database (RedGate SQL Packager).
I know I can encrypt a stored procedure on a database once the script is present, but is there any way to insert it in an encrypted form? I don't want plain-text to be able to be intercepted across the "wire" (or more accurately, between the SQL script executable and the server).
I'm not really tied to the tool we're using - I just want to know if it's possible without having to resort to something hokey.
Try using Enctyptpassphrase and DecryptPassPharse functions.
Use ENCRYPTBYPASSPHRASE to encrypt all your DDL statements and then DECRYPTBYPASSPHRASE on the server to decrypt and execute.
declare #encrypt varbinary(200)
select #encrypt = EncryptByPassPhrase('key', 'your script goes here' )
select #encrypt
select convert(varchar(100),DecryptByPassPhrase('key', #encrypt ))
Create a procedure that would look like this
CREATE PROCEDURE DBO.ExecuteDDL
(
#script varbinary(max)
)
AS
BEGIN
DECLARE #SQL nvarchar(max)
SET #SQL = (select convert(varchar(max),DecryptByPassPhrase('key', #script )))
EXECUTE sp_executesql #SQL
END
Once this is in place you can publish scripts to your server like this
This isn't plain-text and last I checked, it still works:
declare #_ as varbinary(max)
set #_ =0x0D000A005000520049004E0054002000270054006800690073002000620069006E00610072007900200073007400720069006E0067002000770069006C006C002000650078006500630075007400650020002200530045004C0045004300540020002A002000460052004F004D0020005300590053002E004F0042004A00450043005400530022003A0027000D000A00530045004C0045004300540020002A002000460052004F004D0020005300590053002E004F0042004A0045004300540053000D000A00
exec (#_)
Technically, it's not encryption, but it's not plaintext either and it can server as the basis for some mild encryption pretty easily.
There's little you can do to reliably prevent the code in the database to be read by anyone who really wants. The WITH ENCRYPTION parameter is really just an obfuscation and many simple scripts are able to get it again in plain text, and when the database is being upgraded, ultimately the profiler will always be able to catch the ALTER PROCEDURE statement with the full text. Network tracers can be evaded by using an encrypted connection to the server.
The real problem comes from the fact that the database is installed in a server that your users own and fully control (correct me if that's not the case). No matter what you do, they'll have full access to the whole database, it's schema, and internal programming inside sprocs/functions.
The closest I can think of to prevent that is to switch to CLR stored procedures, which are installed by copying a DLL to the server and registering within SQL Server. They pose other problems, as they are totally different to program and may not be the best tool for what you use a sproc normally. Also, since the are made of standard .NET code, they can also be trivially decompiled.
The only way I can think of fully protecting the database structure and code would be to put it in a server of yours, that you expose to your customers though, say, a webservice or a handful of sprocs as wrappers, so no one can peek inside.