Are there any programs that will allow you to follow a sql transaction through to it's end? For instance, say I've inherited a rather complex sql database with a data dictionary. The data dictionary is pretty good, but not as good as say, SQL Doc. I've taken a look at Red Gate's Dependency Tracker and, while that does a very good job of putting things together (triggers, stored procedures, tables, views, etc.), it still does nothing for following a transaction through it's various tracks.
What I'd like is software that will allow me to enter a transaction and based on everything in my db, tree it out visually to let me see what's happening during the transaction. Does that make sense or do I need to elaborate?
Edited to elaborate: While the answers below were very good, it's not exactly what I'm looking for. The front end of the sql database was built with PowerBuilder. I can use SqlSpy during frontend data input and that helps to follow transactions through, but it's all the t-sql and as you can imagine, reams of output. I'm looking for something that would do the same thing, but lay it out visually with the ability to tree out the detail if I wanted to.
If you open a connection in Visual Studio, you can run sprocs and queries in debug mode. i.e. you can go step by step over a query with intellisense and see the values in the variables.
The SQL Profiler will allow you to trace a transaction through, and see exactly what is going on. It is very flexible and allows you to be shown only the events you are interested in, but it won't show it in the tree format I think you're asking about.
The profiler is installed as part of the SQL Server Client Tools (along with Enterprise Manager and Query Analyser).
I guess if this way is doable. You can set up a test environment and run the transaction. Once transaction is submitted, compare the database in test environment with the prodution database.
You can use some tool to do this comparation. Like OpenDBDiff or you can look for other tool by googling.
I haven't used this yet, it appears to do what you're asking: apply IDE-style debugging tools to T-SQL statements. I woul be interested to see how it works out.
Related
I want to do this, because I would like to know how many times a particular row has been changed.
Is this possible?
Thanks
Reading the log file either takes a commercial tool, or an incredible amount of SQL internals knowledge to achieve. You can see some of the raw output by using:
Select * from ::fn_DBlog(null,null)
Actually decoding to find the same record being altered and ensuring any alteration was committed etc would be a difficult task to put it lightly. So it is 'possible' but not very 'probable' that you will be able to do it.
If you need that functionality within a database then you should be looking at triggers / logic within the code.
Late answer but I hope it will be useful to new readers…
One more function you can try is DBCC LOG but unfortunately this is undocumented function same like fn_dblog.
Problem with transaction log in SQL Server is that it was never meant to be used for this but only to allow point in time recovery and transaction properties.
There is a commercial log reader from ApexSQL that you can try.
Here are also couple similar posts that might get you in the right direction.
Read the log file (*.LDF) in sql server 2008
SQL Server Transaction Log Explorer/Analyzer
you can use this program to do it
http://www.red-gate.com/products/SQL_Log_Rescue/index.htm
Consider using SQL Server 2008.
There is a feature new to SQL Server 2008 called Change Data Capture that does exactly what you require, that is to track data modifications over time.
Looking to inspect the log file in order to track changes is not a wise practice. Doing so will provide you with a limited history, the scope of which would also be dependent on the Recovery Model that you use for your database.
You could "roll your own" solution with a small amount of development, by using a log table and populating it using SQL Server Triggers. The suitability of such a solution is of course dependent on your business case.
Take a look at the following TechNet article for some interesting reading:
Tracking Changes in Your Enterprise Database
My DBA just lost some development work that he did on our development database. Poor fella. So naturally our manager asked him, at our status meeting, how this could happen and how we could avoid this happening in the future. "Source control could alleviate the problem" I suggested... The dba's response; "No, we just backup the server more often". Now I would like to help my DBA understand what source control is and how it fits together with a database schema and development on that schema.
Previously I've tried to explain him that there's nothing special about the source code behind tables and stored procedures and it should be in a source control system (TFS in this case). But he just didn't bite. Now, while this misap is in recent memory, I would like to take another stab at it.
So my question is, do you know of any good advice I could pass on to my DBA and maybe even a couple of resources explaining how you would go about migrating a DB schema to be under source control and find its proper place in the build and deployment processes?
A couple of facts about the environment:
Source Control on a TFS 2008 Server.
Database is a MS SQL server 2008 with >300 tables and >300 other objects (sprocs, triggers, functions etc.).
Clarification:
We have been using DB Ghost and other change management solutions on other projects with other DBAs, in the past. We even have the license for VS DB edition! The problem is getting the DBA to even think about this way of developing for the database. He's really old school (i.e. migrating changes manually from environment to environment), and unfortunately hes the only one who knows anything about this particular DB.
See how to version control sql server databases and Do you source control your databases, among many others. Or use the search page. Basically, your approach seems correct. Good luck persuading the DBA...
If you are using Visual Studio Team System, I recommend having a stab at their Database Edition (i think these days it comes with the Developer Edition if you are an MSDN Subscriber). What this will allow you to do is to script out all your schema, stored procs, views, triggers, etc and source control these. This should also make the dba more comfortable since he will be working with a "Database" version of the tool rather than the "Developer" version (naming can go a great lengths with people). As you make changes from Visual Studio, you can manage script changes as you work, and source control them.
If your company has an MSDN license, they can use the Visual Studio Database edition. There's a video tutorial of it here.
I have no power of purchase, so I don't know what the cost breakdowns are. But it has the capability of source controlling all the parts of a DB schema, and includes creating change-scripts as well as auto-deploying straight from VS if you want (I wouldn't recommend that).
In general though, it's pretty solid as a database source control option.
Source control for databases can be quite contentious. It's different to use source control for something that produces a binary because you can't lock the source: a stored proc is a row in a table and there is not single table to read to get a table definition.
Also, version to version is mostly a set of ALTER statements. You script out CREATEs and add them to source control. This makes it harder to use in cases like this.
To me, this is more a procedural error.
Why was the change not done from a script? Forget where the script lives, but why no reproducable and re-runnable script? Perhaps linked to the change tracking number? If the database is reset (loaded from prod) then how would the change have been re-applied to prepare for production. And other questions.
I believe in source control and we use it: but it has limits for database work.
First you are approaching this incorrectly. If the dba won't bite on Source Control and he is making errors that affect the system, the person you need to persuade is his boss.
If it helps, I'm from the old school too and I love having our database objects in source control. How nice to be able to revert one table without having to restore the whole database backup to a different location and then move the table. How much faster and simpler. How nice to be able to compare two different versions and see what changed. How nice to deploy a change and know exactly which database changes (say, for instance only twelve of the 23 possible ones) go with the part you are deploying and not some other unfinished project. How nice to know exactly which scripts were involved in a particular change you had to rollback. How nice that nobody is making on-the-fly changes on production since we now require all production changes to be from source control scripts. There are so many fewer errors and issues to worry about.
Yes it was a change in how we did business, but we did it through a policy change from on high so three was no argument and the dbas went through a couple of times and reverted any objects different from source control to the source control version, so now nobody will even think of doing a database change without it being in source control.
As the product manager for SQL Compare I've spoken to many 'traditional' DBAs who are uncomfortable with third party tools mainly because they have a system that works for them and sometimes changing can be difficult. There are many situations where I am convinced that they would benefit from our tools if only they gave them a chance. Frustrating.
One thing you might consider trying is Red Gate's upcoming tool, SQL Source Control. This is designed to build source control into SSMS, in other words it doesn't require DBAs to leave the comfort zone of their management environment. The bad news is that the tool hasn't been released yet. The good news is that we have an Early Access Program. Please visit the following link to find out more about the tool:
http://www.red-gate.com/Products/SQL_Source_Control/index.htm
you can't really put a large database under source control, so your DBA is right.
what you can do practically is to put your schema under source control, and maybe a few smallish 'configuration' tables.
One way to source control database is to store the data in and about the database separately
You can have the all the tables, procedures and function scripts as SQL files and add them to source control.
Export the database data as insert statements into SQL files, each with a fixed size. This is a cumbersome process as it would involve a lot of files that are to be tracked and controlled.
I am not sure if the VSS/SVN are able to read and keep history of changes to dump files created by the database backup options.
Its not clear from you question if you want to protect the data in the Db or the schemas in the Db. If the latter then you could identify all the important schemas and run an cron job that pulls the schema definitions from the Db and inserts them automatically into a source control system (perhaps even via triggers on the schemas??).
But this still just amounts to backing the system up more often. For what you envision you would need source control integrated with the Db tools and I don't know of any product that does that.
(and I shudder to think of VSS integrated into SQL management studio :-(( )
My answer to this same problem was to export all DB objects to text form (more than 136,000 of them) and then create the SourceSafe projects to hold them. Any New or changed objects in the DB now go to the SourceSafe structure, while unchanged are left alone.
In the place I work, very often it happens that a developer and QA session goes like this:
(This is in reference to SQL Server 2005)
QA: I get Invalid object name 'customers'
DEV: huh? can u send me the exact SQL statement you used?
QA: select * from customers
DEV: hmm. (after some thinks) Are you sure you're using CUSTDB?
QA: yes
DEV: (after figuring out that QA was using CUSTDB_PRODUCTION) Please add "USE CUSTDB" and then tell me what you get with that SQL.
QA: Oh, sorry, I was using wrong DB.
The tab-text for the SQL window shows the information of which database the query is running on, but how do you ensure that QA follows this?
I will admit that I have made this mistake of using the wrong DB many times. I don't tend to read the text in the tab.
What are your experiences with this type of scenario? Have you found a way to help mitigate such a problem?
if your QA is using SSMS for testing you should try the window coloring options in SSMS Tools Pack free add-in for SSMS. this way you could immediately differentiate between servers.
if that's not an option don't allow QA to access production server at all. they shouldn't be able to anyway.
I think you need to formalise how QA will report an error.
You need to specify a set of information that they'll supply with every error report, including:
what they were doing (exactly)
their configuration (including the database!)
time/date (so you can match stuff in logs)
how to repeat it (if repeatable)
etc. You can act on that immediately, or log it in an incident tracking system and come back to it later (in which case the above is invaluable, otherwise it's all lost).
The above can be as simple as an email draft/template. But you need to be rigorous about this, otherwise (as you've discovered) you're going to go round in circles, perhaps without all the salient information you require.
If QA are allowed access to both live and dev databases, using SSMS, then there must be some level of accepted responsibility on their part and/or some level of training of them on your part.
They have been given a tool that allows them to ask questions of the data, but are asking the wrong questions, then complaining to you - if I was the DBA, I'd simply remove their access until they could demonstrate they knew what they where doing! I sympathise that that might not go down to well, but at least threatening to do it might make them think a little for themselves.
Think of this question as 'someone is doing something wrong'
There are 2 simple answers:
remove their ability to 'do something wrong'
train them to do it right
On the same note as Mladen Prajdic, you can colour code query windows in SQL2008 SSMS too.
Personally I use the fully qualified name in all queries (server.datatabase.owner.table - well I only use server if I'm deliberately using a linked server) because I move from database to database so much. If you specify the database in the queries to be run, they still work if connected to a differnt database on the same server or if you have a linked server. Have your QA adopt doing this as their standard if they are writing their own queries; if you are writing the test queries then you should be specifying the database name in the query not through a use statment.
How do you track/manage your stored procedures, views, and functions in SQL Server?
I'd like to use Subversion, but it looks like I would have to just save & commit the CREATE/ALTER statements. That might work okay for me, but I suspect I'd end up doing a lot of nagging.
Is anyone using versioning with their databases? Is there a better way?
In the past, people have just commented out parts of the code and left it in. Or, they add little "added on 2/31/2010" comments all over. It drives me nuts, because I know there is a better way.
We do log changes in the object's header, but that's pretty limited. It would make my life easier to be able to diff versions.
Additional Info
We are using SQL Server 2005. I have Subversion (via VisualSVN Server) and TortoiseSVN installed, but I'm open to other suggestions.
By database objects, I specifically mean stored procedures, views, and functions.
There are only a few tables I would need to track. The database is the backend for a commercial application, and we mostly pull information out for reporting
I found a related question about stored procedure versioning
We script everything and put it into Subversion. Nothing can be loaded to Prod without a script (developers do not have rights to prod) and the people with rights on prod only accept scripts they loaded from Subversion.
We revision our database, schema creation, dw, etl, stored procedures just like any other piece of code, because it's code!
I have also seen people type dates in headers, etc. This is normally due to them completely missing the point of revision control.
Have a look at liquibase, here
It manages your sql changes/scripts for you, and can apply them in conjunction with svn via hooks or scripts. Makes doing all sorts of setup easy, and helps eliminate the case of the missing trigger/sproc/etc...
I'm not sure what you all mean with "database objects". Are these only the tables, views, procedures etc or also data? I mean daily created data?
Assumed you mean the database schema definition. By my experience there is only one way to handle database schema definitions (if you don't have NHibernate or some similar tool). You write sql scripts that create your database from scratch and check them in. You use the same scripts for installation of your software. You see the differences by just comparing the scripts files.
Whenever I've gone through this excercise, it's come down to 3 main things that need to be source-controlled:
Stored Procedures / Views / Triggers (more or less anything that can be fairly expressed as "code". These are fairly simple, include a conditional drop and create at the top of the file.
Table Schema - DROP / CREATE statements as above. You can try to get fancy with ALTER statements, but it tends to get really messy.
The biggest challenge we faced was this forces you into a system where your DB goes back to an initial state often - if there's a fair amount of work involved in bringing DBs to something usable / testable, it can be a pain. In that case we kept a library of scripts that brought a DB to various usable states, and source controlled those as well.
Data within tables. We looked at a couple of approaches here - either a series of INSERT statements stored in a file like "TableName_Data.sql" or a CSV file with custom build tooling that parsed and inserted when the DB was rebuilt.
Ultimately we went with the INSERT statements for simplicity's sake.
So recently on a project I'm working on, we've been struggling to keep a solution's code base and the associated database schema in synch (Database = SQL Server 2008).
Database changes occur fairly regularly (adding columns, constraints, relationships, etc) and as a result it's not uncommon for people to do a 'Get Latest' from source control and
find that they also need to rebuild the database as well (and sometimes they forget to do the latter).
We're not using VSTS: Database Edition (DataDude) but the standard Visual Studio database project with a script (batch file) which tears down and recreates the database from T-SQL scripts. The solution is a .Net & ASP.net solution with LINQ to SQL underlying as the ORM.
Anyone have ideas on an approach to take (automated or not) which would keep everyone up to date with the latest database schema?
Continuous integration with MSBuild is an option, but only helps pick up any breaking changes committed, it doesn't really help in the scenario I highlighted above.
We are using Team Foundation Server, if that helps..
We try to work forward from the creation scripts.
i.e a change to the database is not authorised unless the script has been tested and checked into source control.
But this assumes that the database team is integrated with your app team which is usually not the case in a large project...
(I was tempted to answer this "with great difficulty")
EDIT: Tools won't help you if your process isn't right.
Ok although its not the entire solution, you should include an assertion in the Application code that links up to the database to assert the correct schema is being used, that way at least it becomes obvious, and you avoid silent bugs and people complaining that stuff went crazy all of the sudden.
As for the schema version, you could use some database specific functionality if available, but i personally prefer to declare a schema version table and keep the version number in there, that way its portable and can be checked with a simple select statement
have a look at DB Ghost - you can create a dbp using the scripter in seconds and then manage all your database code with the change manager. www.dbghost.com
This is exactly what DB Ghost was designed to handle.
We basically do things the way you are, with the generation script checked into source control as well. I'm the designated database master so all changes to the script itself are done through me. People send me scripts of the changes they have made, I update my master copy of the schema, run a generate scripts (SSMS) to produce the new DB script, and then check it in. I keep my copy of the code current with any changes that are being made elsewhere. We're a small shop so this works pretty well for us. I realize that it probably doesn't scale.
If you are not using Visual Studio Database Professional Edition, then you will need another tool that can break the database down into its elemental pieces so that they are managable and changeable in an easier manner.
I'd recommend seriously considering Redgate's SQL tools if you want to maintain sanity over all your database changes and updates.
SQL Packager
SQL Multi Script
SQL Refactor
Use a tool like RedGate SQL Compare to generate the change schema between any given version of the database. You can then check that file into source code control
Have a look at this question: dynamic patching of databases. I think it's similar enough to your problem to be helpful.
My solution to this problem is simple. Define everything as XML, and make sure that both the database, the ORM and the UI are generated from this XML, no exceptions. That way, you can use code generation tools to quickly regenerate the database creation script, which will alter your schema while (hopefully) preserving some data. It takes some effort to do, but the net result is well worth it.