I'm currently working on a new project where I was hoping to automate the execution of all stored procedures within a specific schema in a database dynamically. I'd like to be able to simply add the stored procedures to the specific schema (e.g. Build), and have a process that runs on a set schedule which simply iterates through all the stored procedures in the schema, and runs them in parallel.
We have an existing custom ETL system that we've built, that will let us setup a bunch of jobs, this currently relies on using multiple agent jobs, that pick up the stored procedures, and executes them. I'm hoping for our new project to use something better, and was thinking the Service Broker would be the answer.
I've investigated this product: http://partitiondb.com/go-parallel-sql/ which seems to provide some pre-built code that will allow me to queue up the procedures, however I can't extract the database they provide, apparently it has a corrupted header :-(
I'm new to the world of service brokers, and am having a little difficulty in working out how I could automatically get something to queue up a bunch of stored procedures that get executed in parallel.
Any help would be much appreciated.
Cheers
To paraphrase one of my favorite poets - "mo' tech, mo' problems". Service Broker is a great solution to asynchronous processing, but it doesn't seem like a good fit here. Specifically, if all you're looking to do is run a (possibly unknown) set of stored procedures on a schedule, dynamic SQL seems like a better fit to me. Something like (untested)
create or alter procedure dbo.gottaRunEmAll
as
begin
declare p cursor fast_forward, local for
select name
from sys.objects
where schema_id = schema_id('Build');
open p;
declare #name sysname, #sql nvarchar(max);
while(1=1)
begin
fetch next from p into #name;
if (##rowcount <> 0)
break;
set #sql = concat('exec [Build].', quotename(name), ';');
exec(#sql)
end
close p
deallocate p
end
Better (imo) would be to have the above procedure maintained explicitly to call the procedures that you want and how you want them.
Related
I'm not sure if this is possible but if so, my scenario would be about as simple as they come. Assuming I have 6 stored procedures called:
dbo.SyncSources
dbo.SyncData
dbo.UpdateStatistics
dbo.TruncateSourceTable
dbo.ValidateData
dbo.SearchData
None of them require any variables to be input, they simply need to be executed in the order in which they appear above. Can I create one stored procedure that will run all of them, sequentially? Keep in mind that some of the middle ones in that list take several hours to run. Also, this is on an Azure Cloud database, so SQL Server Agent is unfortunately not an option.
You can try this
create procedure OneProcedureTorunThemAll
as
begin tran
exec dbo.SyncSources
exec dbo.SyncData
exec ...
commit
it could be nice to catch errors, verify return values, etc.
I have a stored procedure that has boolean result parameter as output. But my project needs to use 3 databases. Basically, there is a main database and 2 others. The other databases using same stored procedure but they just depend on params. If I explain the scenario, you will understand. Firstly, Sorry for bad explaining.
This application using main database for session management, user configurations and so on. This is okay. The problem is same user has to use 2 different databases for creating invoices. We can pretend that these users are IT support staff. They works for 2 different companies and supporting their products and they are managing their solutions in different databases.
User
Main Database
A Company Data
B Company Data
Users have to create their invoices for each company's customers. Basically they are using same database but the name of databases are totally different. Maybe my problem has easier solution so I want to ask that How can I use Dynamic Database Name in my Stored Procedure? and my current question is I'm initialising Stored Procedure as String and after that I execute the stored procedure as String.
Procedure has no error, also says me completed successfully but there is no insert (in stored procedure). When I use the sql command (which I set manually as String) in Management Studio, Query is running perfectly.
As a summary,
I have 3 databases, Main database need to execute generating Invoice stored procedure. But database names are must be dynamic because of different companies.
When I send the database name A_COMPANY_DB, stored procedure should execute in A_COMPANY_DB. When I send B_COMPANY_DB, stored procedure should execute B_COMPANY_DB.
Both of A_COMPANY and B_COMPANY databases are same. I have to manage sql query as String because of Dynamic Database Name. So I can't reach the output parameter.
My stored procedure has only one output parameter which is bit type. But I can't use it like:
Set #dynsql = 'USE ' + QUOTENAME(#DbName) + ' exec.[dbo].[spName] ' + other params
EXECUTE sp_executesql #dynsql
In this situation I couldn't reach Output parameter. How can I set or use my Output parameter in this stored procedure?
EXECUTE sp_executesql #dynsql #outputparam OUT
Is it possible or any solution?
Sorry for bad explaining.
I am going to use dynamic sql in my stored procedure to remove some of the code duplication. But I see one big drawback for me in this case: I have rather big DB with lots of objects. My stored procedure is using few tables and since it is compiled I can find dependencies easily from Sql Server management studio.But when I rewrite it to dynamically build some of the repeating queries I will loose dependency possibility and next time when I would need to find who is using this table I will need to do raw text search in my code repository rather than asking sql server for dependency. This is probably small concern, but I would still like to try to find solution.
So my question is: is there anything I can do to still have ability to see what dependencies my stored proc has? Like declare some dependencies upfront etc?
You can get dependencies to show up for the stored procedure for sections of code that never execute. For example, if you wanted to "declare" a dependency on a table named TestTable, you could use
CREATE PROC MyStoredProc
AS
DECLARE #SQL VarChar(4000)
SET #SQL = 'SELECT * FROM TestTable'
EXEC (#SQL)
RETURN
SELECT 0 FROM TestTable -- Never executes but shows as dependency
END
I have three websites which uses an abstract database structure with tables like: Items, Places, Categories, etc... and stored procedures like GetItemsByCategory, GetRelatedItems, etc... Actually im using exactly the same database structure for these 3 different websites.
From a code perspective im using the same code for all websites (except the HTML which is specific foreach one), and all the common code is in few projects used by all websites, so everytime that i detect a bug (which is in all websites) i just fix it on one place (the common part used by all) and automatically all websites get the fix.
Actually im using Asp.net MVC3 and Sql server.
Everytime i want to extend some funcionality, and i need a new table, stored procedure or something related with database, i have to do the modification in each database.
Do you know any approach that i could use to be able to have the same flexibility and do database modifications only one time for all websites?
Do you think I'm using a good approach or i should use something different in your opinion?
If the databases are on a single server, you could generate the script for the procedure from Management Studio, and make sure to use the option to "check for object existence" (Tools > Options > SQL Server Object Explorer > Scripting). This will yield something like this (most importantly it produces your stored procedure code as something you can execute using dynamic SQL):
USE DBName;
GO
SET ANSI_NULLS ON;
GO
SET QUOTED_IDENTIFIER ON;
GO
IF NOT EXISTS (...)
BEGIN
EXEC dbo.sp_executesql #statement = N'CREATE PROCEDURE dbo.whatever ...
'
END
GO
Now that you have this script, you can modify it to work across multiple databases - you just need to swipe the #statement = portion and re-use it. First you need to stuff the databases where you want this to work into a #table variable (or you can put this in a permanent table, if you want). Then you can build a command to execute in each database, e.g.
DECLARE #dbs TABLE (name SYSNAME);
INSERT #dbs(name) SELECT N'db1';
INSERT #dbs(name) SELECT N'db2';
INSERT #dbs(name) SELECT N'db3';
-- now here is where we re-use the create / alter procedure command from above:
DECLARE #statement NVARCHAR(MAX) = N'CREATE PROCEDURE dbo.whatever ...
';
-- now let's build some dynamic SQL and run it!
DECLARE #sql NVARCHAR(MAX);
SET #sql = N'';
SELECT #sql = #sql + '
EXEC ' + QUOTENAME(name) + '.dbo.sp_executesql N''' + #statement + ''';'
FROM #dbs;
EXEC sys.sp_executesql #sql;
Alternatively, you could create a custom version of my sp_msforeachdb or sp_ineachdb replacements:
Making a more reliable and flexible sp_MSforeachdb
Execute a Command in the Context of Each Database in SQL Server
I used to use a tool called SQLFarms Combine for this, but the tool doesn't seem to exist anymore, or perhaps it has been swallowed up / re-branded by another company. Red Gate has since produced SQL Multi Script that has similar functionality.
If you added a column to all your tables called websiteId you could just have one database. Store the unique websiteId in each site's web.config and just pass it with each request for data. Obviously each site's data is stored with their websiteId so data can be queried per website.
It means a bit of refactoring in your db and any calls to your your db, but once done, you only have one database to maintain.
Of course this is assuming your databases are on the same server...
Where I'm at we have a software package running on a mainframe system. The mainframe makes a nightly dump into sql server, such that each of our clients has it's own database in the server. There are a few other databases in the server instance as well, plus some older client dbs with no data.
We often need to run reports or check data across all clients. I would like to be able to run queries using sp_msforeachdb or something similar, but I'm not sure how I can go about filtering unwanted dbs from the list. Any thoughts on how this could work?
We're still on SQL Server 2000, but should be moving to 2005 in a few months.
Update:
I think I did a poor job asking this question, so I'm gonna clarify my goals and then post the solution I ended up using.
What I want to accomplish here is to make it easy for programmers working on queries for use in their programs to write the query using one client database, and then pretty much instantly run (test) code designed and built on one client's db on all 50 or so client dbs, with little to no modification.
With that in mind, here's my code as it currently sits in Management Studio (partially obfuscated):
use [master]
declare #sql varchar(3900)
set #sql = 'complicated sql command added here'
-----------------------------------
declare #cmd1 varchar(100)
declare #cmd2 varchar(4000)
declare #cmd3 varchar(100)
set #cmd1 = 'if ''?'' like ''commonprefix_%'' raiserror (''Starting ?'', 0, 1) with nowait'
set #cmd3 = 'if ''?'' like ''commonprefix_%'' print ''Finished ?'''
set #cmd2 =
replace('if ''?'' like ''commonprefix_%''
begin
use [?]
{0}
end', '{0}', #sql)
exec sp_msforeachdb #command1 = #cmd1, #command2 = #cmd2, #command3 = #cmd3
The nice thing about this is all you have to do is set the #sql variable to your query text. Very easy to turn into a stored procedure. It's dynamic sql, but again: it's only used for development (famous last words ;) ). The downside is that you still need to escape single quotes used in the query and much of the time you'll end up putting an extra ''?'' As ClientDB column in the select list, but otherwise it works well enough.
Unless I get another really good idea today I want to turn this into a stored procedure and also put together a version as a table-valued function using a temp table to put all the results in one resultset (for select queries only).
Just wrap the statement you want to execute in an IF NOT IN:
EXEC sp_msforeachdb "
IF '?' NOT IN ('DBs','to','exclude') BEGIN
EXEC sp_whatever_you_want_to
END
"
Each of our database servers contains a "DBA" database that contains tables full of meta-data like this.
A "databases" table would keep a list of all databases on the server, and you could put flag columns to indicate database status (live, archive, system, etc).
Then the first thing your SCRIPT does is to go to your DBA database to get the list of all databases it should be running against.
We even have a nightly maintenance script that makes sure all databases physically on the server are also entered into our "DBA.databases" table, and alerts us if they are not. (Because adding a row to this table should be a manual process)
How about taking the definition of sp_msforeachdb, and tweaking it to fit your purpose? To get the definition you can run this (hit ctrl-T first to put the results pane into Text mode):
sp_helptext sp_msforeachdb
Obviously you would want to create your own version of this sproc rather than overwriting the original ;o)
Doing this type of thing is pretty simple in 2005 SSIS packages. Perhaps you could get an instance set up on a server somewhere.
We have multiple servers set up, so we have a table that denotes what servers will be surveyed. We then pull back, among other things, a list of all databases. This is used for backup scripts.
You could maintain this list of databases and add a few fields for your own purposes. You could have another package or step, depending on how you decide which databases to report on and if it could be done programmatically.
You can get code here for free: http://www.sqlmag.com/Articles/ArticleID/97840/97840.html?Ad=1
We based our system on this code.