Where I work the production databases are backed up nightly and restored to dev, test, and QA environments. When running any of the programs in non-production environments, to avoid making changes we don't want to make in production, such as sending real users email, our programs test the environments using a combination of internally parsing the command line, and calling an SQL user function. The function selects ##SERVERNAME, then parses the result looking for specific strings ie. if the ServerName contains "-PROD-", it is a production server.
The problem is the hardware group is implementing a high availability project so if a server fails ##SERVERNAME will return the name of the backup server. I am looking at extending the current logic to account for whatever the fail-over server names will be, but I was hoping there was a better way to test the environment than parsing text for static strings.
Store a setting in a database that is separate from your application database(s) then read that setting as-needed using a function. When your application runs in production, you'll get the production values. When your application runs in Development, you'll get the development values.
The nice thing about this is you can store all kinds of values and easily get to them from your SPROCS, PowerShell or whatever front end you have.
CREATE DATABASE SETTINGSDB
GO
USE SETTINGSDB
GO
-- A table to hold key/value pairs
CREATE TABLE MYSETTINGS
(
SettingName VARCHAR(50) PRIMARY KEY CLUSTERED ,
SettingValue VARCHAR(500)
)
GO
-- On DEVELOPMENT SERVER, run this
INSERT INTO MYSETTINGS
VALUES ('ENVIRONMENT', 'DEV'),
('SOME_SETTING', 'True')
-- On PRODUCTION SERVER, run this
INSERT INTO MYSETTINGS
VALUES ('ENVIRONMENT', 'PROD'),
('SOME_SETTING', 'False')
GO
-- A function to pull key/value pairs.
CREATE FUNCTION dbo.GetEnvVar( #SettingName VARCHAR(50) )
RETURNS VARCHAR(500)
AS
BEGIN
RETURN (SELECT SettingValue FROM SETTINGSDB.dbo.MYSETTINGS WHERE SettingName = #SettingName)
END
GO
Once you are setup, you can then check the value, and it will be different between DEV/PROD. For example:
-- Then use these values:
USE YourApplicationDatabaseNameHere
GO
CREATE PROCEDURE SampleApplicationSprocThatSendsEmail
#EmailAddress VARCHAR(50),
#Subject VARCHAR(50)
AS
IF (dbo.GetEnvVar('ENVIRONMENT') = 'PROD' )
BEGIN
-- Only Executes in Production
-- TODO: SEND THE EMAIL
END ELSE
BEGIN
-- Only Executes in Development
PRINT 'Send email to ' + #EmailAddress
END
Instead of finding all possible objects that might cause harm or annoyance to some unknown number of targets (mailboxes, files, databases, etc...) why not just isolate the dev/test environment at the network layer? Put them in an isolated network/subnet where only inbound is permitted, all outbound gets blocked or rerouted. It's hard to know for sure you've gotten every single endpoint. For example, what happens if your admins add more secondaries for additional protection and read-only queries?
We used SQL Server's trace replay capabilities regularly for many years for service pack, upgrade (app and db), regression, cross-workload and other tests. Every so often we'll have some smart, highly motivated new team member (new hire or transfer) who would write scripts or "cleansing" apps to scrub the trace files and databases so the tests can be run on his/her workstation. Every single one of them will learn that's a bad idea, sometimes in a very, very hard way (e.g. re-index a clustered index on a highly volatile 2billion row table in mid-morning).
Blocking at the network layer has the added benefit of minimal prep/setup work for each test run plus you can have an identical setup as prod. When you encounter bugs or regressions, you have a few items less to check.
Related
My company considers database scripts we write part of our intellectual property.
With new releases, we deliver a 2-part setup for our users:
a desktop application
an executable that wraps up the complexities of initializing/updating a database (RedGate SQL Packager).
I know I can encrypt a stored procedure on a database once the script is present, but is there any way to insert it in an encrypted form? I don't want plain-text to be able to be intercepted across the "wire" (or more accurately, between the SQL script executable and the server).
I'm not really tied to the tool we're using - I just want to know if it's possible without having to resort to something hokey.
Try using Enctyptpassphrase and DecryptPassPharse functions.
Use ENCRYPTBYPASSPHRASE to encrypt all your DDL statements and then DECRYPTBYPASSPHRASE on the server to decrypt and execute.
declare #encrypt varbinary(200)
select #encrypt = EncryptByPassPhrase('key', 'your script goes here' )
select #encrypt
select convert(varchar(100),DecryptByPassPhrase('key', #encrypt ))
Create a procedure that would look like this
CREATE PROCEDURE DBO.ExecuteDDL
(
#script varbinary(max)
)
AS
BEGIN
DECLARE #SQL nvarchar(max)
SET #SQL = (select convert(varchar(max),DecryptByPassPhrase('key', #script )))
EXECUTE sp_executesql #SQL
END
Once this is in place you can publish scripts to your server like this
This isn't plain-text and last I checked, it still works:
declare #_ as varbinary(max)
set #_ =0x0D000A005000520049004E0054002000270054006800690073002000620069006E00610072007900200073007400720069006E0067002000770069006C006C002000650078006500630075007400650020002200530045004C0045004300540020002A002000460052004F004D0020005300590053002E004F0042004A00450043005400530022003A0027000D000A00530045004C0045004300540020002A002000460052004F004D0020005300590053002E004F0042004A0045004300540053000D000A00
exec (#_)
Technically, it's not encryption, but it's not plaintext either and it can server as the basis for some mild encryption pretty easily.
There's little you can do to reliably prevent the code in the database to be read by anyone who really wants. The WITH ENCRYPTION parameter is really just an obfuscation and many simple scripts are able to get it again in plain text, and when the database is being upgraded, ultimately the profiler will always be able to catch the ALTER PROCEDURE statement with the full text. Network tracers can be evaded by using an encrypted connection to the server.
The real problem comes from the fact that the database is installed in a server that your users own and fully control (correct me if that's not the case). No matter what you do, they'll have full access to the whole database, it's schema, and internal programming inside sprocs/functions.
The closest I can think of to prevent that is to switch to CLR stored procedures, which are installed by copying a DLL to the server and registering within SQL Server. They pose other problems, as they are totally different to program and may not be the best tool for what you use a sproc normally. Also, since the are made of standard .NET code, they can also be trivially decompiled.
The only way I can think of fully protecting the database structure and code would be to put it in a server of yours, that you expose to your customers though, say, a webservice or a handful of sprocs as wrappers, so no one can peek inside.
I want to push the data from database to application instead of application pull the data. I have installed ms sql server and apache tomcat server. I have my application in apache tomcat, here I made connection to database. Now I want database send the data whenever there is update in data. But all I know is fetch the data from database is not good idea, because application needs to monitor the database for updated data will lead to fire the query for every 5 sec, this is not efficient as well.
I google it I got some answers they are Query Notification here, Sql server Agent Job to schedule the task automatically. If you have any other suggestion please post it.
There surely are several possibilities to do that:
Implement unsafe CLR trigger
Implement unsafe CLR procedure
Use xp_cmdshell
Call web service
Use Query Notification
You can read a little about them in this discussion:
Serial numbers, created and modified in SQL Server.
Personally I would prefer Query Notification over other methods, because it already has support fopr various cases (e.g. sync/async communication) and you don't have to reinvent the wheel. And is in your case recommended by Microsoft.
Polling is another method you've mentioned. It's is a more like traditional method and there can be some performance penalties related, but you shouldn't worry about them if you are careful enough. For example, if you already have an authentication built in your application, you can create another column in your Users table that is set if there are any changes related to that user. And then, there can be just a thread in your app that will perform a query every second against this table (even dirty reads with NOLOCK shouldn't be a problem here) and maintain some in-memory structure (e.g. thread-safe dictionary) that says which client should get pushed. Another thread polls your dictionary and when it finds there something for the client, performs a db query that extracts data and sends it to the client. This looks like a lot of unnccessary work, but at the end you get two independent workers which somewhat helps to separate concerns; first one is just an informer which performs 'lightweight' database polling; second one extract real data and performs server push. You can even optimize the push-worker in the way that when it runs, it checks if multiple clients need some data and then executes the select for all of those who need it. You would probably want the second worker to run less frequently than first one.
EDIT
If you wish to use non-.NET technology to achieve the same functionality, you will have to get more into SQL Server Service Broker. Query Notification is a simplified layer built in .NET on top of SQL Server Service Broker, and you would have to build at least part of that layer by yourself. This includes creating queue, message type, service and stored procedures with SEND and RECEIVE on the other side. You will have to take care of the conversation/dialog by yourself. SB is actually a async-messaging world adjusted to work in RDBMS environment, so you will see some new TSQL expressions. However, MSDN is here to help:
http://msdn.microsoft.com/en-us/library/ms166061(v=sql.105).aspx
http://msdn.microsoft.com/en-us/library/bb522893.aspx
This could help as well: Externally activate non-.NET application from Service Broker
Example on how to code the stuff:
-- First you have to enable SB for your database
USE master
ALTER DATABASE Playground
SET ENABLE_BROKER
GO
USE Playground
GO
-- Then create a message type; usually it will be XML
-- because it's very easy to serialize/deserialize it
CREATE MESSAGE TYPE [//Playground/YourMessageType]
VALIDATION = WELL_FORMED_XML
GO
-- Then create a contract to have a rule for communication
-- Specifies who sends which message type
CREATE CONTRACT [//Playground/YourContract] (
[//Playground/YourMessageType] SENT BY ANY)
GO
--Creates queues, one for initiator (1) and one for target (2)
CREATE QUEUE MyQueue1
GO
CREATE QUEUE MyQueue2
GO
-- Finally, configure services that 'consume' queues
CREATE SERVICE [//Playground/YourService1]
ON QUEUE MyQueue1 ([//Playground/YourContract])
GO
CREATE SERVICE [//Playground/YourService2]
ON QUEUE MyQueue2 ([//Playground/YourContract])
GO
-- Now you can send a message from service to service using contract
DECLARE
#dHandle uniqueidentifier,
#Msg nvarchar(max)
BEGIN DIALOG #dHandle
FROM SERVICE [//Playground/YourService1]
TO SERVICE '//Playground/YourService2'
ON CONTRACT [//Playground/YourContract]
WITH ENCRYPTION = OFF
SELECT #Msg = (
SELECT TOP 3 *
FROM Table1
FOR XML PATH('row'), ROOT('Table1'))
;SEND ON CONVERSATION #dHandle
MESSAGE TYPE [//Playground/YourMessageType] (#Msg)
PRINT #Msg
GO
-- To get the message on the other end, use RECEIVE
-- Execute this in another query window
DECLARE #dHandle uniqueidentifier
DECLARE #MsgType nvarchar(128)
DECLARE #Msg nvarchar(max)
;RECEIVE TOP(1)
#dHandle = conversation_handle,
#Msg = message_body,
#MsgType = message_type_name
FROM MyQueue2
SELECT #MsgType
SELECT #Msg
END CONVERSATION #dHandle
GO
So we've got a bunch of apps running on our SQL servers, and today we realised that a number of them had a bunch of Partition Schemes/Functions that we didn't create.
The partition schemes and functions were called ifts_comp_fragment_data_space_{hash} and ifts_comp_fragment_partition_function_{hash} respectively.
Digging deeper, we realised that they are marked as system entries (is_system set to 1 in sys.partition_schemes) which means we can't even delete them.
After some research we found out that SQL server will create them to partition the fulltext catalogs if they become too large, or something like that see here. The problem is - we just deleted all the catalogs, and these were left abandoned, with NO way of clearing them out.
I wouldn't worry too much, except I NEED to delete them, since I'm trying to export our DB as a .bacpac file, and that crashes complaining that the DB contains partition schemes/functions and they're not supported.
Is there ANY way of forcing the SQL server to drop those objects, or any other alternative that I could do?
You can change that is_system flag from 1 to 0 and then drop the partition scheme like any other. To do this:
First allow updates on your server:
exec sp_configure 'allow updates', 1
go
reconfigure with override
go
Shutdown your SQL server
Start it back up in Single User mode by running "C:\Program Files\Microsoft SQL Server\MSSQL11.MSSQLSERVER\MSSQL\Binn\sqlservr.exe -m" from a console with elevated privs.
Login to the server using the SQL Server DAC http://technet.microsoft.com/en-us/library/ms178068(v=sql.105).aspx
If we do an SP_HELPTEXT on the sys.partition_schemes view, you'll see that the is_system column is based on a status flag in the sys.sysclsobjs table. "sysconv(bit, o.status & 0x4) AS is_system,"
So to change the flag, we have to look at what the current value of the status is and unmark the 4 bit. My value was 4 so I updated it to 0.
update sys.sysclsobjs set status = 0 where name =
'ifts_comp_fragment_data_space_033D368C'
Now you can shutdown the single user mode SQL Server process by just closing your console window and start your sql server windows service. Then just login as you normally would and drop the partition scheme.
Finally, set your 'allow updates' setting back to 0.
This might need to be planned downtime for a production server.
DISCLAIMER This probably isn't a Microsoft supported way of doing this, you may want to test on some non-prod servers before diving in.
I'm trying to understand how I can use an alias to reference another database in the same instance, without having to use a hardcoded name.
The scenario is as below:
I have a data db with stores data, an audit db which keeps all changes made. for various reason, i want to keep the audit data in a separate database, not least because it can get quite large and for reporting purposes.
In the data db, I don't want to reference this by a hardcoded name but an alias so that in different environments, I don't have to change the name and various sp's to reference the new name.
for example:
mydevdata
mydevaudit
If a sp exists in mydevdata such as which calls the mydevaudit, I don't want to change the sp when I go to test where the db's may be called mytestdata and mytestaudit. Again, for various reasons, the database names can change, more to do with spaces an instances etc.
So if I had procedure in mydevdata:
proc A
begin
insert into mydevaudit.table.abc(somecol)
select 1
end
when I go to test, I don't want to be change the procedure to reference another name, (assume for sake of argument that happened)
Instead I am looking to do something like:
proc A
begin
insert into AUDITEBALIAS.table.abc(somecol)
select 1
end
I am interested in finding out how I could do something like that, and the pro's and cons.
Also, dymnamic SQL is not an option.
thanks in advance for you help.
You may be able to use synonyms
CREATE SYNONYM WholeTableAliasWithDBetc FOR TheDB.dbo.TheTable
This means all object references in the local DB are local to that DB, except for synonyms that hide the other database from you.
You can also use stored procedures in the audit DB. There is a 3rd form of EXEC that is little used where you can parametrise the stored proc name
DECLARE #module_name_var varchar(100)
SET #module_name_var = 'mydevaudit.dbo.AuditProc'
-- SET #module_name_var = 'whatever.dbo.AuditProc'
EXEC #module_name_var #p1, #p2, ...
Obviously you can change module_name_var to use whatever DB you like
I've just posted this to How to create Sql Synonym or "Alias" for Database Name? which is a workaround for the same situation:
There is a way to simulate this using a linked server. This assumes you have two SQL servers with the same set of databases one for development/test and one live.
Open SQL Server Management Studio on your development/test server
Right click Server Objects > Linked Servers
Select New Linked Server...
Select the General page
Specify alias name in Linked server field - this would normally be the name of your live server
Select SQL Native Client as the provider
Enter sql_server for Product Name
In Data Source specify the name of the development server
Add Security and Server Options to taste
Click OK
The above is for SQL Server 2005 but should be similar for 2008
Once you've done that you can write SQL like this:
SELECT * FROM liveservername.databasename.dbo.tablename
Now when your scripts are run on the development server with the linked server back to itself they will work correctly pulling data from the development server and when the exact same scripts are run on the live server they will work normally.
Is it possible to set up somehow Microsoft SQL Server to run a stored procedure on regular basis?
Yes, in MS SQL Server, you can create scheduled jobs. In SQL Management Studio, navigate to the server, then expand the SQL Server Agent item, and finally the Jobs folder to view, edit, add scheduled jobs.
If MS SQL Server Express Edition is being used then SQL Server Agent is not available. I found the following worked for all editions:
USE Master
GO
IF EXISTS( SELECT *
FROM sys.objects
WHERE object_id = OBJECT_ID(N'[dbo].[MyBackgroundTask]')
AND type in (N'P', N'PC'))
DROP PROCEDURE [dbo].[MyBackgroundTask]
GO
CREATE PROCEDURE MyBackgroundTask
AS
BEGIN
-- SET NOCOUNT ON added to prevent extra result sets from
-- interfering with SELECT statements.
SET NOCOUNT ON;
-- The interval between cleanup attempts
declare #timeToRun nvarchar(50)
set #timeToRun = '03:33:33'
while 1 = 1
begin
waitfor time #timeToRun
begin
execute [MyDatabaseName].[dbo].[MyDatabaseStoredProcedure];
end
end
END
GO
-- Run the procedure when the master database starts.
sp_procoption #ProcName = 'MyBackgroundTask',
#OptionName = 'startup',
#OptionValue = 'on'
GO
Some notes:
It is worth writing an audit entry somewhere so that you can see that the query actually ran.
The server needs rebooting once to ensure that the script runs the first time.
A related question is: How to run a stored procedure every day in SQL Server Express Edition?
Yes, if you use the SQL Server Agent.
Open your Enterprise Manager, and go to the Management folder under the SQL Server instance you are interested in. There you will see the SQL Server Agent, and underneath that you will see a Jobs section.
Here you can create a new job and you will see a list of steps you will need to create. When you create a new step, you can specify the step to actually run a stored procedure (type TSQL Script). Choose the database, and then for the command section put in something like:
exec MyStoredProcedure
That's the overview, post back here if you need any further advice.
[I actually thought I might get in first on this one, boy was I wrong :)]
Probably not the answer you are looking for, but I find it more useful to simply use Windows Server Task Scheduler
You can use directly the command sqlcmd.exe -S "." -d YourDataBase -Q "exec SP_YourJob"
Or even create a .bat file. So you can even 2x click on the task on demand.
This has also been approached in this HERE
I'll add one thing: where I'm at we used to have a bunch of batch jobs that ran every night. However, we're moving away from that to using a client application scheduled in windows scheduled tasks that kicks off each job. There are (at least) three reasons for this:
We have some console programs that need to run every night as well. This way all scheduled tasks can be in one place. Of course, this creates a single point of failure, but if the console jobs don't run we're gonna lose a day's work the next day anyway.
The program that kicks off the jobs captures print messages and errors from the server and writes them to a common application log for all our batch processes. It makes logging from withing the sql jobs much simpler.
If we ever need to upgrade the server (and we are hoping to do this soon) we don't need to worry about moving the jobs over. Just re-point the application once.
It's a real short VB.Net app: I can post code if any one is interested.
You could use SQL Server Service Broker to create custom made mechanism.
Idea (simplified):
Write a stored procedure/trigger that begins a conversation (BEGIN DIALOG) as loopback (FROM my_service TO my_service) - get conversation handler
DECLARE #dialog UNIQUEIDENTIFIER;
BEGIN DIALOG CONVERSATION #dialog
FROM SERVICE [name]
TO SERVICE 'name'
...;
Start the conversation timer
DECLARE #time INT;
BEGIN CONVERSATION TIMER (#dialog) TIMEOUT = #time;
After specified number of seconds a message will be sent to a service. It will be enqueued with associated queue.
CREATE QUEUE queue_name WITH STATUS = ON, RETENTION = OFF
, ACTIVATION (STATUS = ON, PROCEDURE_NAME = <procedure_name>
, MAX_QUEUE_READERS = 20, EXECUTE AS N'dbo')
, POISON_MESSAGE_HANDLING (STATUS = ON)
Procedure will execute specific code and reanable timer to fire again.
You can find fully-baked solution(T-SQL) written by Michał Gołoś called Task Scheduler
Key points from blog:
Pros:
Supported on each version (from Express to Enterprise). SQL Server Agent Job is not available for SQL Server Express
Scoped to database level. You could easiliy move database with associated tasks (especially when you have to move around 100 jobs from one enviromnent to another)
Lower privileges needed to see/manipulate tasks(database level)
Proposed distinction:
SQL Server Agent (maintenance):
backups
index/statistics rebuilds
replication
Task Scheduler (business processes):
removing old data
preaggregations/cyclic recalculations
denormalization
How to set it up:
get source code from section: "Do pobrania" - To download
(enabling broker/setting up schema tsks/configuration table + triggers + stored procedure)/setting up broker things)
set up configuration table [tsks].[tsksx_task_scheduler] to add new tasks (columns names are self-descriptive, sample task included)
Warning: Blog is written in Polish but associated source code is in English and it is easy to follow.
Warning 2: Before you use it, please make sure you have tested it on non-production environment.
Using Management Studio - you may create a Job (unter SQL Server Agent)
One Job may include several Steps
from T-SQL scripts up to SSIS Packages
Jeb was faster ;)
You should look at a job scheduled using the SQL Server Agent.