I need to move data between PROD and DEV SQL Server databases using stored procedure. Obviously I have to use linked servers to perform this task. But I need the same functionality in both directions.
For example something like:
CREATE PROCEDURE dbo.spMoveData(
)
AS
INSERT INTO Target.dbo.TestTable
SELECT * FROM Source.dbo.TestTable
And just pass proper connection strings to linked servers. I think it is better then creation of two identical stored procedures. Is it possible? If not, are there any best practices to do such tasks?
Maybe it makes sense to drop and create required linked servers before procedure execution?
Related
Scenario
Note: I am using SQL Server 2017 Enterprise
I am looping through a list of databases and copying data to them out of one database. This database will only be accessed by the script (no other transactions will be made against it from something else). The copy includes copying straight table to table, or will have more complex, longer-running queries or stored procedures. All of this is done with SQL Server jobs calling procedures; I'm not using anything like SSIS.
Question
Instead of looping through all the databases and running the statements one at a time, I want to be able to run them in parallel. Is there an easy way to do this?
Options I've thought of:
Run each data transfer as a job and then run all the jobs at once. From my understanding, they would be executed asynchronously, but I'm not 100% sure.
Generate the SQL statements and write a script outside of SQL Server (e.g. Powershell or Python) and run all the commands in parallel
Leverage SSIS
I prefer not to do this, since this would take too much work and I'm not very familiar with it. This may be used down the road though.
Use powershell...
Create a table on the central database to house instance / connection string details. (Remember to obfuscate for security)
Create another table to house the queries.
Create a third table to map Instance to Query.
In powershell create a collection / list based object. Deserialized from your data entries. The object will be made up of three properties {Source / Destination / Query}
Write a method / function to carry out the ETL based work. CONNECT TO DB, READ FROM SOURCE, WRITE TO DEST.
Iterate over the collection using Foreach-Parallel construct with your function nested within. This will initiate a new SPID based on the number of elements in the collection and pass those values into your function where the work will be carried out.
We have the same app source (with some custom "if x client then") and basically the same SQL Server database structure. But, some clients need slightly different stored procedures.
What would be best practice in this scenario for long term maintaining the databases and keeping the correct structure? As of now, for example when I change the procedure in one database and need to do the same in 9/10 others I just ALTER procedure and USE different database. But, I can't keep track of which procedures are different in that special snowflake client.
Any ideas? The plan is of course to get more clients so that's looking for trouble.
I try to push the "one fits all" concept but hey, what can you do...
Maybe have that "if x client then" as case statement in a SQL Server stored procedure and then you can just ALTER mindlessly?
I need to unit test results between several stored procedures on a single database (certain values between different results sets ought to be the same). Also, I need to be able to copy these unit test such that several identical databases will perform the unit tests identically when I choose to start the tests.
I want to use OpenRowSet to dump these results to temp tables and then compare these tables, possibly using a stored procedure that I can execute once a week.
Before I configure the servers to allow this are there any reasons not to use OpenRowSet? If so then what other options might I have?
The main reason to not use OpenRowSet is that you don't need to use it. Since you want to do testing, you should use a testing framework. I am a huge fan of DbFit ( http://dbfit.github.io/dbfit/ ). Your tests are completely isolated from your database. It is very easy to set up and modify. And you can even compare result sets between two Stored Procedures. It is very easy to automate. It is easy to create subsections and only run tests in a particular subsection, or an individual test. You can stage the test with DML statements and everything will get rolled back at the end of the test. You can use variables to grab data from a query or procedure and use that in calls / queries that follow.
Well... Perhaps using a unit testing framework is a step too far for you. If you don't want to go that far, try the below.
From MSDN OpenRowset "is an alternative to accessing tables in a linked server and is a one-time, ad hoc method of connecting and accessing remote data by using OLE DB". You have stated that there are several databases on the same server (i.e. no linked server). Therefore OpenRowSet seems to be overkill. You can still get the bulk performance gains by using "Select Into" statements to create your data tables in a new unit testing database (I wouldn't advokate creating test tables in your prod databases). This would have a stored proc that calls each individual database using 3 or 4 part naming. If you really wanted to you could have a table of database\stored procedures and use dynamic SQL to execute them all. once you have all the data your stored proc just needs to compare them.
Another way would be introduce a kind of ReturnType in your stored
procedures, or use an existing parameter to send that value.
When the ReturnType is set to 'INSERT_RESULTS_TO_TEST' or some such,
then have the stored procedures' final return statements, insert
records into test tables designed for testing, instead of their
default return.
If necessary, have additional columns in the test tables to indicate,
which server, which database, which stored procedure is producing the
result. Say call these ResultSetID.
Then, for your comparison, use self-joins on the test tables
comparing for the values between different ResultSetIDs.
I am trying to create a SQL Job that sits on what I like to call a Maintenance Server where details about all the other server databases sit. So I have a stored procedure to gather all the required data I have and this stored procedure is now in MaintenanceServer.MaintenanceDB.pr_MyProcedure.
Now what i'd like to do this is. I am creating Linked Servers on my MaintenanceServer which points to all the other Servers I'd like to maintain. Next on my list is to be able to run this procedure pr_MyProcedure against all those LinkedServers. I have over 25 servers to maintain and what i want to do is to run the procedure against them and insert the result to my local MaintenanceDB.
Can someone point me to a good way to do this? Also i'd like to hear the drawbacks of this method.
Thanks and Cheers!
EDIT:
I was also thinking about storing the SP on all the master databases of my other Servers and running them remotely on my Maintenance Server. Then i'll use OPENQUERY() to gather all the results that I need. However, I don't think I want to touch all the master databases on my server list. Please tell me if this would be better than the once I stated above.
The nice way to do it is run the SP on the target server, as that's nicely modular. If you want to query the remote server, it's going to be more painful. I'm not sure whether the undocumented sp_foreachdb deals with linked databases, if not you are down to dynamic sql.
As in build a string and call it with Exec(#somesql), that will need to deal with the results itself, as any variables will only be in scope inside the exec.
Personally I'd consider inverting all this. Have each server call it's own SP and the call an SP on the maintenance box to store the results.
EXEC [RemoteServer] .DatabaseName.DatabaseOwner.StoredProcedureName
‘Params’
Example:
EXEC [RemoteServer] .DatabaseName.DatabaseOwner.StoredProcedureName ’PramValue’
I have a database from a 3rd party. They supply a tool to update the database data weekly. The tool is pretty old and uses ODBC. Updates can either be incremental or can delete all database data then recreate the data. The update can take several hours. In order to have high availability, it was suggested to have 2 SQL databases, and store a "active database" setting in another database to determine which of the two databases applications should use (while the other could be being updated).
One issue we are running into is: How to do reference the active database in stored procedures in other databases?
Is this the right approach? Is there a simple, perhaps-infrastructure-based approach? (Should this be posted on ServerFault?)
Note: Databases are read-only besides the update tool.
If the databases are on different servers, you can create an alias for the server which will redirect to the other server in SQL Server Configuration Manager. Under SQLNative Client 10.0 Configuration (or 9.0 if you're in SQL Server 2005) you can add a new alias.
Otherwise, you can always rename the databases using sp_dbrename so thata your client applications are always using database1 while you are updating database2.
If you want to use different databases inside a stored procedure you either need to:
Duplicate all the calls. Ugly. You would end with a lot of:
if #firstDatabase=1
select * from database1..ExampleTable where ...
else
select * from database2..ExampleTable where ...
Use dynamic queries. Less ugly:
set #sqlQuery='select * from '+#currentDatabase+'..ExampleTable where...'
exec sp_executesql #sqlQuery
I admit that neither solution is perfect...
I'd take the approach of having the stored procedures in both databases with some sort of automatic trigger to update the stored procedures in the other database if a stored procedure is changed.