Calling SSIS with SSISDB implementation from SQL Server Service broker - sql-server

The requirement is to call a web service through SSIS and calling the SSIS from a SQL Server Service Broker activated stored procedure.
Here is what I have currently doing:
Queue
CREATE QUEUE [schema].[ProccessingQueue] WITH STATUS = ON , RETENTION = OFF , ACTIVATION ( STATUS = ON , PROCEDURE_NAME = [schema].[usp_ProccessingQueueActivation] , MAX_QUEUE_READERS = 10 , EXECUTE AS N'dbo' ), POISON_MESSAGE_HANDLING (STATUS = ON)
My stored procedure:
ALTER PROCEDURE [schema].[usp_ProccessingQueueActivation]
WITH EXECUTE AS CALLER
AS
BEGIN
SET NOCOUNT ON;
<snip declaration>
BEGIN
BEGIN TRANSACTION;
WAITFOR
(
RECEIVE TOP (1)
#ConversationHandle = conversation_handle,
#MessageBody = CAST(message_body AS XML),
#MessageTypeName = message_type_name
FROM [schema].[ProccessingQueue]
), TIMEOUT 5000;
<snip awasome stuff>
EXEC dbo.RunSSIS <param>
DECLARE #ReplyMessageBody XML = #MessageBody;
SEND ON CONVERSATION #ConversationHandle MESSAGE TYPE [type] (#ReplyMessageBody);
END
<handle error>
COMMIT TRANSACTION;
END
END
Now here is what RunSSIS stored procedure looks like
ALTER PROCEDURE [dbo].[RunSSIS]
<params>
AS
BEGIN
DECLARE #exec_id BIGINT
EXEC [SSISDB].[catalog].[create_execution]
#package_name=N'<SSIS_package>',
#folder_name=N'<folder>',
#project_name=N'<projectName>',
#use32bitruntime=FALSE,
#reference_id=NULL,
#execution_id=#exec_id OUTPUT
EXEC [SSISDB].[catalog].[set_execution_parameter_value]
#exec_id,
#object_type=30,
#parameter_name=N'<param_Name>',
#parameter_value=<param>
SELECT #exec_id
EXEC [SSISDB].[catalog].[start_execution] #exec_id
END
Now this will throws the below exception in event-viewer as the Sql service broker activation security context isn't recognized in SSISDB environment.
The activated proc
'[schema].[usp_ProccessingQueueActivation]' running on
queue '' output the
following: 'The current security context cannot be reverted. Please
switch to the original database where 'Execute As' was called and try
it again.'
To resolve the problem I have tried those following approach
So I follow this link
http://www.databasejournal.com/features/mssql/article.php/3800181/Security-Context-of-Service-Broker-Internal-Activation.htm
and created a User with a self signed certificate (thinking that it
is user that doesn't has permission). But it is returning same error,
digging deeper I found that [internal].[prepare_execution] in
SSISDB has "REVERT" statement in line no 36 that throws the error as
it doesn't like Impersonation at all.
I tried to move the RunSSIS stored procedure to SSISDB and try to call it from activation stored procedure, it was shoot down as SSISDB it doesn't allow any user with SQL Server auth, It needs to have a Windows auth and User created by Certificate obviously doesn't has windows credential.
My question is
Am I on the correct path? I certainly doesn't anticipate using 2 component of SQL server together would be that difficult.
If not in correct approach what would be best approach to call a service from Service broker? I have seen "External Activation" for SQL Server Service broker but haven't explored is yet. But I would try to stick to something that lives inside server environment and scale-able, and don't like the idea of installing different component in prod environment (it is always a overhead for support personal,as there is one more point which can fail)
I am using Windows auth and my credential has sys_Admin access.

I think you can take out the "WITH EXECUTE AS CALLER" and everything (the proc and then the package that ends up getting called) will be run under the security context of the Service Broker. As long as that context has the permissions to do what you want to do, you should be fine.
I have not used a Service Broker in this way, but I do the same thing with jobs fired off by the SQL Agent. As long as the Agent's security context has the permissions needed in the procs/packages everything runs fine. We use network accounts for our services so it all works between servers as well.

This has a code smell of tight coupling and my first instinct is to decouple the queue, the DB that houses the proc, and the SSIS execution into a PowerShell script. Have the script get the messages from service broker then call SSISDB on a different connection without wrapping [catalog].[create_execution] and [catalog].[set_execution_parameter_value] in a stored proc. You can still run this script directly from Agent.
This approach gives you the most flexibility with regard to security contexts, if one of the components moves to a different server, if something is named differently in dev/QA, or technologies change (Azure ServiceBus instead of Broker for instance). You can also get creative with logging/monitoring.

Related

Service Broker queue message [duplicate]

I am learning how to use the Service Broker of SQL Server 2008 R2. When following the tutorial Completing a Conversation in a Single Database. Following the Lesson 1, I have successfully created the message types, contract, the queues and services. Following the Lesson 2, I have probably sent the message. However, when trying to receive the message, I get the NULL for the ReceivedRequestMsg instead of the sent content.
When looking at the sys.transmission_queue, the transmission_status for the message says:
An exception occurred while enqueueing a message in the target queue. Error: 15517, State: 1. Cannot execute as the database principal because the principal "dbo" does not exist, this type of principal cannot be impersonated, or you do not have permission.
I have installed SQL Server using the Windows login like Mycomp\Petr. I am using that login also for the lessons.
Can you guess what is the problem? What should I check and or set to make it working?
Edited 2012/07/16: For helping to reproduce the problem, here is what I did. Can you reproduce the error if you follow the next steps?
Firstly, I am using Windows 7 Enterprise SP1, and Microsoft SQL Server 2008 R2, Developer Edition, 64-bit (ver. 10.50.2500.0, Root Directory located at C:\Program Files\Microsoft SQL Server\MSSQL10_50.SQL_PRIKRYL05\MSSQL).
Following the tutorial advice, I have downloaded the AdventureWorks2008R2_Data.mdf sample database, and copied it into C:\Program Files\Microsoft SQL Server\MSSQL10_50.SQL_PRIKRYL05\MSSQL\DATA\AdventureWorks2008R2_Data.mdf
The SQL Server Management Studio had to be launched "As Administrator" to be able to attach the data later. Then I connected the SQL Server.
Right click on Databases, context menu Attach..., button Add..., pointed to AdventureWorks2008R2_Data.mdf + OK. Then selected the AdventureWorks2008R2_Log.ldf from the grid below (reported as Not found) and pressed the Remove... button. After pressing OK, the database was attached and the AdventureWorks2008R2_log.LDF was created automatically.
The following queries were used for looking at "Service Broker enabled/disabled", and for enabling (the Service Broker was enabled successfully for the database):
USE master;
GO
SELECT name, is_broker_enabled FROM sys.databases;
GO
ALTER DATABASE AdventureWorks2008R2
SET ENABLE_BROKER
WITH ROLLBACK IMMEDIATE;
GO
SELECT name, is_broker_enabled FROM sys.databases;
GO
Then, following the tutorial, the queries below were executed to create the message types, the contract, the queues, and the services:
USE AdventureWorks2008R2;
GO
CREATE MESSAGE TYPE
[//AWDB/1DBSample/RequestMessage]
VALIDATION = WELL_FORMED_XML;
CREATE MESSAGE TYPE
[//AWDB/1DBSample/ReplyMessage]
VALIDATION = WELL_FORMED_XML;
GO
CREATE CONTRACT [//AWDB/1DBSample/SampleContract]
([//AWDB/1DBSample/RequestMessage]
SENT BY INITIATOR,
[//AWDB/1DBSample/ReplyMessage]
SENT BY TARGET
);
GO
CREATE QUEUE TargetQueue1DB;
CREATE SERVICE
[//AWDB/1DBSample/TargetService]
ON QUEUE TargetQueue1DB
([//AWDB/1DBSample/SampleContract]);
GO
CREATE QUEUE InitiatorQueue1DB;
CREATE SERVICE
[//AWDB/1DBSample/InitiatorService]
ON QUEUE InitiatorQueue1DB;
GO
So far, so good.
Then the following queries are used to look at the queues (now empty when used):
USE AdventureWorks2008R2;
GO
SELECT * FROM InitiatorQueue1DB WITH (NOLOCK);
SELECT * FROM TargetQueue1DB WITH (NOLOCK);
SELECT * FROM sys.transmission_queue;
GO
The problem manifests when the message is sent:
BEGIN TRANSACTION;
BEGIN DIALOG #InitDlgHandle
FROM SERVICE
[//AWDB/1DBSample/InitiatorService]
TO SERVICE
N'//AWDB/1DBSample/TargetService'
ON CONTRACT
[//AWDB/1DBSample/SampleContract]
WITH
ENCRYPTION = OFF;
SELECT #RequestMsg =
N'<RequestMsg>Message for Target service.</RequestMsg>';
SEND ON CONVERSATION #InitDlgHandle
MESSAGE TYPE
[//AWDB/1DBSample/RequestMessage]
(#RequestMsg);
SELECT #RequestMsg AS SentRequestMsg;
COMMIT TRANSACTION;
GO
When looking at the queues, the Initiator... and the Target... queues are empty, and the sent message can be found in sys.transmission_queue with the above mentioned error reported via the transmission_status.
alter authorization on database::[<your_SSB_DB>] to [sa];
The EXECUTE AS infrastructure requires dbo to map to a valid login. Service Broker uses the EXECUTE AS infrastructure to deliver the messages. A typical scenario that runs into this problem is a corporate laptop when working from home. You log in to the laptop using cached credentials, and you log in into the SQL using the same Windows cached credentials. You issue a CREATE DATABASE and the dbo gets mapped to your corporate domain account. However, the EXECUTE AS infrastructre cannot use the Windows cached accounts, it requires direct connectivity to the Active Directory. The maddening part is that things work fine the next day at office (your laptop is again in the corp network and can access to AD...). You go home in the evening, continue with Lesson 3... and all of the sudden it doesn't work anymore. Make the whole thing seem flimsy and unreliable. Is just the fact that AD conectivity is needed...
Another scenatio that leads to the same problem is caused by the fact that databases reteint the SID of their creator (the Windows login that issues the CREATE DATABASE) when restored or attached. If you used a local account PC1\Fred when you create the DB and then copy/attach the database to PC2, the account is invalid on PC2 (it is scoped to PC1, of course). Again, not much is affected but EXECUTE AS is, and this causes Service Broker to give the error you see.
And last example is when the DB is created by a user that later leaves the company and the AD account gets deleted. Seems like revenge from his part, but he's innocent. The production DB just stops working, simply because it's his SID that the dbo maps too. Fun...
By simply changing the dbo to sa login you fix this whole EXECUTE AS thing and all the moving parts that depend on it (and SSB is probably the biggest dependency) start working.
You would need to grant receive on your target queue to your login. And it should work!
USE [YourDatabase]
GRANT RECEIVE ON [dbo].[YourTargetQueue]
TO [Mycomp\Petr];
GO
And you also need to grant send for your user, permission on Target Service should be sufficient, but let's enable on both services for the future.
USE AdventureWorks2008R2 ;
GO
GRANT SEND ON SERVICE::[//AWDB/1DBSample/InitiatorService]
TO [Mycomp\Petr] ;
GO
GRANT SEND ON SERVICE::[//AWDB/1DBSample/TargetService]
TO [Mycomp\Petr] ;
GO

SQL Server job scheduling and PHP

I've been developing an application that displays graphs on a website by reading from a SQL Server. Since some of the tables are very large, my SQL scripts take a couple minutes to execute, so I devised a plan to get around this by having a job execute on a monthly basis. I have my T-SQL scripts that process the data and other scripts to create and display the graphs, so the hard part is done. The problem I'm having is creating a scheduled job in SQL Server 2005 (cant upgrade since it's a work computer).
I've been having a great deal of trouble as it seems I don't have the stored procedures installed necessary to create a job or schedule. For example, when I ran:
USE TEST1 ;
GO
EXEC sp_add_job
#job_name = N'MY_FIRST_JOB2' ;
GO
EXEC sp_add_jobstep
#job_name = N'MY_FIRST_JOB2',
#step_name = N'Set database to read only',
#subsystem = N'TSQL',
#command = N'SELECT * FROM [TEST1].[dbo].[BACK1]',
#retry_attempts = 5,
#retry_interval = 5 ;
GO
I received errors saying the stored procedures sp_add_job and sp_add_jobstep could not be found. I tried putting [dbo] before the stored procedures and it didn't help. When I checked my Programmability -> Stored Procedures -> System Stored Procedure folder, there were a lot but none for jobs or scheduling! Interestingly, when I ran
SELECT * FROM [sys].[procedures];
I got nothing, not a single row other than the column headers.
When I tried using the interface in SQL Server Agent, I'm able to create a job and set the schedule but it's not executing. When I open the job, the last executed box doesn't always update even though it should have executed. Furthermore, in the view history/logs, only a few of my attempts appeared (all errored out) for different reasons, some were due to permission even when I set the user to NT AUTHORITY\SYSTEM.
This brings me to 2 questions:
1) Is there a way I can download these stored procedures?
2) Is there any other way I can do this? I have access to SQLExpress but since it doesn't have scheduling and the SQL Server Agent, I opted out and chose the easier route.
I've been at this for a while and am at wits end.
The problem is that the stored procedures you are referencing are not part of the database. All of the job related system stored procedures are in the msdb database.
An example of a proper call would be:
EXEC msdb.dbo.sp_add_job
#job_name = N'MY_FIRST_JOB2';

How to push the data from database to application?

I want to push the data from database to application instead of application pull the data. I have installed ms sql server and apache tomcat server. I have my application in apache tomcat, here I made connection to database. Now I want database send the data whenever there is update in data. But all I know is fetch the data from database is not good idea, because application needs to monitor the database for updated data will lead to fire the query for every 5 sec, this is not efficient as well.
I google it I got some answers they are Query Notification here, Sql server Agent Job to schedule the task automatically. If you have any other suggestion please post it.
There surely are several possibilities to do that:
Implement unsafe CLR trigger
Implement unsafe CLR procedure
Use xp_cmdshell
Call web service
Use Query Notification
You can read a little about them in this discussion:
Serial numbers, created and modified in SQL Server.
Personally I would prefer Query Notification over other methods, because it already has support fopr various cases (e.g. sync/async communication) and you don't have to reinvent the wheel. And is in your case recommended by Microsoft.
Polling is another method you've mentioned. It's is a more like traditional method and there can be some performance penalties related, but you shouldn't worry about them if you are careful enough. For example, if you already have an authentication built in your application, you can create another column in your Users table that is set if there are any changes related to that user. And then, there can be just a thread in your app that will perform a query every second against this table (even dirty reads with NOLOCK shouldn't be a problem here) and maintain some in-memory structure (e.g. thread-safe dictionary) that says which client should get pushed. Another thread polls your dictionary and when it finds there something for the client, performs a db query that extracts data and sends it to the client. This looks like a lot of unnccessary work, but at the end you get two independent workers which somewhat helps to separate concerns; first one is just an informer which performs 'lightweight' database polling; second one extract real data and performs server push. You can even optimize the push-worker in the way that when it runs, it checks if multiple clients need some data and then executes the select for all of those who need it. You would probably want the second worker to run less frequently than first one.
EDIT
If you wish to use non-.NET technology to achieve the same functionality, you will have to get more into SQL Server Service Broker. Query Notification is a simplified layer built in .NET on top of SQL Server Service Broker, and you would have to build at least part of that layer by yourself. This includes creating queue, message type, service and stored procedures with SEND and RECEIVE on the other side. You will have to take care of the conversation/dialog by yourself. SB is actually a async-messaging world adjusted to work in RDBMS environment, so you will see some new TSQL expressions. However, MSDN is here to help:
http://msdn.microsoft.com/en-us/library/ms166061(v=sql.105).aspx
http://msdn.microsoft.com/en-us/library/bb522893.aspx
This could help as well: Externally activate non-.NET application from Service Broker
Example on how to code the stuff:
-- First you have to enable SB for your database
USE master
ALTER DATABASE Playground
SET ENABLE_BROKER
GO
USE Playground
GO
-- Then create a message type; usually it will be XML
-- because it's very easy to serialize/deserialize it
CREATE MESSAGE TYPE [//Playground/YourMessageType]
VALIDATION = WELL_FORMED_XML
GO
-- Then create a contract to have a rule for communication
-- Specifies who sends which message type
CREATE CONTRACT [//Playground/YourContract] (
[//Playground/YourMessageType] SENT BY ANY)
GO
--Creates queues, one for initiator (1) and one for target (2)
CREATE QUEUE MyQueue1
GO
CREATE QUEUE MyQueue2
GO
-- Finally, configure services that 'consume' queues
CREATE SERVICE [//Playground/YourService1]
ON QUEUE MyQueue1 ([//Playground/YourContract])
GO
CREATE SERVICE [//Playground/YourService2]
ON QUEUE MyQueue2 ([//Playground/YourContract])
GO
-- Now you can send a message from service to service using contract
DECLARE
#dHandle uniqueidentifier,
#Msg nvarchar(max)
BEGIN DIALOG #dHandle
FROM SERVICE [//Playground/YourService1]
TO SERVICE '//Playground/YourService2'
ON CONTRACT [//Playground/YourContract]
WITH ENCRYPTION = OFF
SELECT #Msg = (
SELECT TOP 3 *
FROM Table1
FOR XML PATH('row'), ROOT('Table1'))
;SEND ON CONVERSATION #dHandle
MESSAGE TYPE [//Playground/YourMessageType] (#Msg)
PRINT #Msg
GO
-- To get the message on the other end, use RECEIVE
-- Execute this in another query window
DECLARE #dHandle uniqueidentifier
DECLARE #MsgType nvarchar(128)
DECLARE #Msg nvarchar(max)
;RECEIVE TOP(1)
#dHandle = conversation_handle,
#Msg = message_body,
#MsgType = message_type_name
FROM MyQueue2
SELECT #MsgType
SELECT #Msg
END CONVERSATION #dHandle
GO

Can Execute Procedure But Job That Runs Procedure Errors?

I have a procedure that pulls data from my database and uses MSDB.DBO.sp_send_dbmail to send out e-mails. The procedure itself runs without a hitch when I use 'EXEC myprocedure'. But when I set up a job for the procedure, the job fails with the following error
'Error formating query, probably invalid parameters [SQLState
42000](Error 22050). The step failed'
There is one step in the job with the TSQL statement 'EXEC myprocedure', using the database that my procedure is stored on. Does anyone know what could be causing this error?
Update
I've narrowed the problem down. It's something with the Exchange server I use. I was using a domain address (ie. mail.mycompany.com) as the 'Server name' under Database Mail's account configuration wizard. I was unable to send e-mails to listservs and external users using this domain address. I talked to our Exchange guy and he recommended using the actual IP address of one of the mail servers (ie. 10.123.53.53). That fixed the problem with listservs and external users, but now I am unable to send e-mails when I run my procedure using a job (the procedure itself executes properly when I manually run it). Does anyone know what criteria on our Exchange server I will need to change to fix this?
This is like to be a permissions issue. It is impossible from the information given to specify exactly what permission is the problem, but a TSQL job step will run under the context of the Server Agent Service Account, so ensure that that account has SELECT permissions to the tables that are being used to build the query, and then make sure that it has permissions to the profile that you are using within Database Mail.
The problem was with the #query parameter used by sp_send_dbmail. For some reason, my SQL Server Agent account (which was a domain administrator) was denied access to use sp_send_dbmail with the #query parameter. I ended up dropping the #query parameter altogether and putting everything in the #body parameter. The #body parameter became a concatenation of regular text and what returned from a SELECT query.

Scheduled run of stored procedure on SQL server

Is it possible to set up somehow Microsoft SQL Server to run a stored procedure on regular basis?
Yes, in MS SQL Server, you can create scheduled jobs. In SQL Management Studio, navigate to the server, then expand the SQL Server Agent item, and finally the Jobs folder to view, edit, add scheduled jobs.
If MS SQL Server Express Edition is being used then SQL Server Agent is not available. I found the following worked for all editions:
USE Master
GO
IF EXISTS( SELECT *
FROM sys.objects
WHERE object_id = OBJECT_ID(N'[dbo].[MyBackgroundTask]')
AND type in (N'P', N'PC'))
DROP PROCEDURE [dbo].[MyBackgroundTask]
GO
CREATE PROCEDURE MyBackgroundTask
AS
BEGIN
-- SET NOCOUNT ON added to prevent extra result sets from
-- interfering with SELECT statements.
SET NOCOUNT ON;
-- The interval between cleanup attempts
declare #timeToRun nvarchar(50)
set #timeToRun = '03:33:33'
while 1 = 1
begin
waitfor time #timeToRun
begin
execute [MyDatabaseName].[dbo].[MyDatabaseStoredProcedure];
end
end
END
GO
-- Run the procedure when the master database starts.
sp_procoption #ProcName = 'MyBackgroundTask',
#OptionName = 'startup',
#OptionValue = 'on'
GO
Some notes:
It is worth writing an audit entry somewhere so that you can see that the query actually ran.
The server needs rebooting once to ensure that the script runs the first time.
A related question is: How to run a stored procedure every day in SQL Server Express Edition?
Yes, if you use the SQL Server Agent.
Open your Enterprise Manager, and go to the Management folder under the SQL Server instance you are interested in. There you will see the SQL Server Agent, and underneath that you will see a Jobs section.
Here you can create a new job and you will see a list of steps you will need to create. When you create a new step, you can specify the step to actually run a stored procedure (type TSQL Script). Choose the database, and then for the command section put in something like:
exec MyStoredProcedure
That's the overview, post back here if you need any further advice.
[I actually thought I might get in first on this one, boy was I wrong :)]
Probably not the answer you are looking for, but I find it more useful to simply use Windows Server Task Scheduler
You can use directly the command sqlcmd.exe -S "." -d YourDataBase -Q "exec SP_YourJob"
Or even create a .bat file. So you can even 2x click on the task on demand.
This has also been approached in this HERE
I'll add one thing: where I'm at we used to have a bunch of batch jobs that ran every night. However, we're moving away from that to using a client application scheduled in windows scheduled tasks that kicks off each job. There are (at least) three reasons for this:
We have some console programs that need to run every night as well. This way all scheduled tasks can be in one place. Of course, this creates a single point of failure, but if the console jobs don't run we're gonna lose a day's work the next day anyway.
The program that kicks off the jobs captures print messages and errors from the server and writes them to a common application log for all our batch processes. It makes logging from withing the sql jobs much simpler.
If we ever need to upgrade the server (and we are hoping to do this soon) we don't need to worry about moving the jobs over. Just re-point the application once.
It's a real short VB.Net app: I can post code if any one is interested.
You could use SQL Server Service Broker to create custom made mechanism.
Idea (simplified):
Write a stored procedure/trigger that begins a conversation (BEGIN DIALOG) as loopback (FROM my_service TO my_service) - get conversation handler
DECLARE #dialog UNIQUEIDENTIFIER;
BEGIN DIALOG CONVERSATION #dialog
FROM SERVICE [name]
TO SERVICE 'name'
...;
Start the conversation timer
DECLARE #time INT;
BEGIN CONVERSATION TIMER (#dialog) TIMEOUT = #time;
After specified number of seconds a message will be sent to a service. It will be enqueued with associated queue.
CREATE QUEUE queue_name WITH STATUS = ON, RETENTION = OFF
, ACTIVATION (STATUS = ON, PROCEDURE_NAME = <procedure_name>
, MAX_QUEUE_READERS = 20, EXECUTE AS N'dbo')
, POISON_MESSAGE_HANDLING (STATUS = ON)
Procedure will execute specific code and reanable timer to fire again.
You can find fully-baked solution(T-SQL) written by Michał Gołoś called Task Scheduler
Key points from blog:
Pros:
Supported on each version (from Express to Enterprise). SQL Server Agent Job is not available for SQL Server Express
Scoped to database level. You could easiliy move database with associated tasks (especially when you have to move around 100 jobs from one enviromnent to another)
Lower privileges needed to see/manipulate tasks(database level)
Proposed distinction:
SQL Server Agent (maintenance):
backups
index/statistics rebuilds
replication
Task Scheduler (business processes):
removing old data
preaggregations/cyclic recalculations
denormalization
How to set it up:
get source code from section: "Do pobrania" - To download
(enabling broker/setting up schema tsks/configuration table + triggers + stored procedure)/setting up broker things)
set up configuration table [tsks].[tsksx_task_scheduler] to add new tasks (columns names are self-descriptive, sample task included)
Warning: Blog is written in Polish but associated source code is in English and it is easy to follow.
Warning 2: Before you use it, please make sure you have tested it on non-production environment.
Using Management Studio - you may create a Job (unter SQL Server Agent)
One Job may include several Steps
from T-SQL scripts up to SSIS Packages
Jeb was faster ;)
You should look at a job scheduled using the SQL Server Agent.

Resources