Background
I have a production SQL Server 2005 server to which 4 different applications connect and make changes.
There are no foreign keys and in some cases no primary keys.
Unfortunately throwing the whole thing out and starting from scratch is not an option.
So my solution is to start migrating each of the applications to a service layer approach so that there is only one application directly connecting to the database.
However there are problems that need to be fixed before that service layer is written and all the applications are migrated over.
So rather than make changes and hope they don't break any one of the 4 badly written applications (with no way of quickly testing all functionality) my solution is to start auditing the database
Problem
How do I audit what stored procedures, tables, columns, views are being accessed/updated/called by each user on SQL Server 2005.
I can find out which tables are being updated but I have no idea which columns and by what users.
I also don't know if certain tables are being accessed only through stored procedures/views.
I know that SQL Server 2008 has better auditing features but if I could do this without spending money that would be great. That said if the best solution is to upgrade or buy software that's also an option.
Check out SQL Server 2008's CDC feature. You can't use this directly in 2005 but you can write a trigger for each table to log all data changes to a new audit table. i.e. you'd have an audit table for each table in your db, with all the same columns plus some additional columns saying what the operation was and when it occurred.
If the nature of your applications means you can get user information and/or application information from CURRENT_USER and APP_NAME() you could include that information in the audit table too.
And check out this answer for more goodness.
Related
My client has been using Microsoft Access 2010 for quite a while and they received some Security Audit Requirements. They are using a Linked Tables approach connecting to Microsoft SQL Server 2012 Express.
The requirements states that all actions against the data must be logged. (INSERT, UPDATE, DELETE AND SELECT statements)
For the INSERT, UPDATE, DELETE statements I could create a trigger which would log the changes.
The issue is around the audit of SELECT statements. If the data was read-only, I could have used a Stored Procedure which would have logged the query. But executing a Stored Proc makes a Recordset not updatable.
Does anyone have an idea how to approach this challenge?
I'm open to a lot of strategies... (Connecting Access to SQL through a web service, anything...)
It's important to note that my client does not have $30k to spend on an Enterprise edition of SQL Sever as they are a small-business with less than 10 employees.
SELECT statements are part of the database-level audit action groups in SQL Server. (Search that page for "database-level audit actions".) But that level of auditing requires SQL Server Enterprise edition.
Theoretically, you can limit all access to use only stored procedures regardless of whether the data is read-only. Write the stored procedure to write auditing information to the log first, then do whatever else needs to be done--SELECT, INSERT, etc.
Practically, you might not be able to do that. It depends on the applications that hit your database. Limiting all access to use only stored procedures can break applications that expect other things. (How would a Ruby on Rails application respond if you switched to just stored procedures?)
A bulletproof audit system that makes your database unusable isn't very good; it's simpler and cheaper to just shut down the database server altogether.
You could upgrade to a SQL Server edition that supports SQL Server profiler.
The other option is to get other tools to audit like sql audit for example.
You could turn on JET showplan. This would log all queries used by Access.
http://www.techrepublic.com/article/use-microsoft-jets-showplan-to-write-more-efficient-queries/?siu-container
As I pointed out in comments you really fooling the audit requirements UNLESS each form is opened using a where clause that limits the viewing of data in that form to the ONE record. If you don’t do this, then a form opened to a linked table could have 1000’s of records, and user(s) hitting ctrl-f to find and jump to one record means the SELECT statement tells you ZERO about what the user actually looked at. So while you can turn on show plan, the audit concept would not tell you anything about what the user actually looked at unless application design changes are made to restrict forms to one record. And to be fair, 99% of my applications in fact do open and restrict the main editing form to the one record via a where clause.
So while you can technology wise log all SELECT commands as per above, it not really in the sprit of such a log since such a log would not be of any use to determine what actual records the user looked at.
I have an access 2003 database that holds all of my business data. This access database gets updated every few hours during the day.
We're currently writing a website that will need to use the data from the access database. This website (for the time being) will have only read only capabilities. Meaning there will only need to be one way transfer of data (Access -> SQL).
I'm imaging there's a way to perform this data migration from access to SQL server programatically. Does anyone have any links to something I can read about?
If this practice sounds odd, and you'd like to suggest another way to do this (or a situation where data can go both ways (Access -> SQL, SQL -> Access), that's perfectly fine.
The company is going to continue using Access 2003 for their business functionality. There's no way around that. But I'd like to build the (readonly) website on top of SQL Server.
The strategy you outlined can be very challenging. You could use INSERT queries to copy new Access rows to SQL Server, as described in another answer.
However, if you have changes to existing Access rows, and you also want those changes propagated to SQL Server, it won't be so simple. And it will be more complicated still if you want deleted Access rows deleted from SQL Server, too.
It seems more reasonable to me to use a different approach. Migrate the data to SQL Server once. Then replace the tables in your Access database with ODBC links to the SQL Server tables. Thereafter, changes to the data from within your Access application will not require a separate synchronization step ... they will already be in SQL Server. And you won't need to write any code to synchronize them.
If your concern is that the connections between the web server and SQL Server be read-only, just set them up that way. You can still independently allow read-write permissions for your Access application.
To do the initial data migration and set the SQL Server automatically, I would use the SQL Server Migration Assistant. The only thing you should definitely change that I can think of would be to turn off the Identity property on any columns that have it - to be explained below (MS Access calls Identity autonumber). Once you have your tables loaded, you can set up a dsnless connection to the database (and tables) you just created.
I haven't used the method just linked, but I believe it allows you to use SQL Server authentication to connect to the db. The benefit of using this method is you can easily change which SQL Server instance and/or database your are connecting to for development and testing.
There might be a better, automated way, but you can create several insert queries doing left joins from the primary key of the Access table to the SQL Server table, and putting a WHERE clause that specifies the SQL Server PrimaryKey must be null. This is why you need to turn off the Identity property in the SQL Server tables, so that you can insert the new data.
Finally, put the name of each query in one function, then run the function periodically.
I have used Microsoft's free SQL Server Migration Assistant (SSMA) to migrate Access to SQL Server. The tool is very simple to use. The only problem I have encountered with the tool was overloaded data types when migrating. What I mean by this is a small string will get converted to a NVARCHAR(MAX) in some instances. Otherwise, the tool is very handy and can be reused after setting up a 'profile'.
I want to save any kind of log/tables with every query executed by my application.
I know I could do this by coding it (before I make any query, I insert a new row in a log table with the query and the user who is executing it.
I have read it can be done automatically but I'm not sure how can it work with WCF Services. I mean every query is going to be executed by the same SQL user and this wouldn't be very useful for audit operations (I need to know WHO made every query, and users will be validated against my own users tables).
Have you ever had a similar scenario? Thanks in advance!
As a starting point it may be worth looking into doing this via SQL Server Profiler. You can normally find this in the Tools Menu in Management Studio.
You can set up a trace to capture all SQL run on a server. More importantly you have a myriad of filter options which can be applied so that you only capture the data you are interested in (e.g. DatabaseName, UserName).
This information can be stored directly in a SQL Table, which should give you the abillity to join onto. Of course running anything like this will result in some overhead on the SQL box.
You can try the SQL Server Audit feature. It audits singe or groups of events both on server and database level. However, be advised that the database level auditing is available in SQL Server Enterprise and Developer editions only
I’m after a bit of advice on the best way to go about this is SQL server 2008R2 express. I have a number of applications that are in separate databases on the same server. They are all “plugins” that use a central staff/structure list that will be in a separate database. The application is in the process of being migrated from JET.
What I’m looking for is the best way of all the “plugin” databases being able to see the central database and use those tables in standard queries and views etc.
As I’m using express that rules out any replication solution and so far the only option I can think of is to use triggers or a stored procedure to “push” out all the changes to the plugins. The information needs to be populated on a near enough real time basis however the number of changes will be very small maybe up to 100 a day and the biggest table only has about 1000 rows at the moment (the staff names table).
Hopefully that will cover all everything but if anyone needs any more details then just ask
Thanks
Apologies if I've misunderstood, but from your description it sounds like all these databases are hosted on the same instance of SQL Server - it's your mention of replication that makes me uncertain.
Assuming that's the case, you should be able to replace any copies of tables from the central database which are held in the "plugin" databases with views or synonyms which reference the central tables directly, since SQL server allows you to make references between databases on the same server using three-part naming (database_name.schema_name.object_name)
For example, if each plugin db has a table StaffNames, you could replace this with a view by dropping the table, then creating a view:
drop table StaffNames
go
create view StaffNames
as
select * from <centraldbname>.<schema - probably dbo>.StaffNames
go
and your code should continue to work seamlessly, as long as permissions are set up.
Alternatively, you could replace all the references to the shared tables in the plugin databases with three-part name references to the central database, but the view method requires less work.
This involves data replication, kind of:
We have many sites with SQL Express installed, there is an 'audit' database on each site that has one table in 1st normal form (to make life simple :)
Now I need to get this table from each site, and copy the contents (say, with a Date Time Value > 1/1/200 00:00, but this will change obviously) and copy it to a big 'super table' in sql server proper, that also has the primary key as the Site Name (That needs injecting in) and the current primary key from the SQL Express table)
e.g. Many SQL Express DBs with the following table columns
ID, Definition Name, Definition Type, DateTime, Success, NvarChar1, NvarChar2 etc etc etc
And the big super table needs to have:
SiteName, ID, Definition Name, Definition Type, DateTime, Success, NvarChar1, NvarChar2 etc etc etc
Where items in bold are the primary key(s)
Is there a Microsoft (or non MS I suppose) app/tool/thing to manager copying all this data accross already, or do we need to write our own?
Many thanks.
You can use SSIS (which comes with SQL Server) to populate, it can be set up with variables to change the connection string to the various databases. I have one that loops through the whole list and does the same process using three differnt files from three differnt vendors. You could so something simliar to loop through the different site databases. Put the whole list of database you want to copy the audit data from in a table and loop through it changing the connection string each time.
However, why on earth would you want one mega audit table per site? If every table in the database populates the audit table as changes happen, then the audit table eventually becomes a huge problem for performance. Every insert, update and delete has to hit this table and then you are proposing to add an export on top of that. This seems to me to be a guaranteed structure for locking and deadlocks and all sorts of nastiness. Do yourself a favor and limit each audit table to the table it is auditing.
Things to consider:
Linked servers and sp_msforeachdb as part of a do-it-yourself solution.
SQL Server Replication (by Microsoft) (which I believe can pull data from SQL Server Express)
SQL Server Integration Services which can pull data from SQL Server Express instances.
Personally, I would investigate Integration Services first.
Good luck.
You could do this with SymmetricDS. SymmetricDS is open source, web-enabled, database independent, data synchronization/replication software. It uses web and database technologies to replicate tables between relational databases in near real time. The software was designed to scale for a large number of databases, work across low-bandwidth connections, and withstand periods of network outage.
As of right now, however, you would need to implement a custom IDataLoaderFilter extension point (in Java) to add the extra column. The metadata would be available though because your SiteName would be the external_id.