I'm a developer at a small company where we're struggling for consistent change control. I'm running into issues where non-dev staff are tweaking stored procedures and triggers in production installations. Their changes are being overwritten when we apply upgrades because they've gone outside of the process the dev team uses to verify db changes are incorporated into source control.
How would you recommend approaching this problem from a technical as well as personal perspective?
Edit 1: A little background on our current process might help this along. We're using a continuous integration server (TeamCity) to generate install artifacts and label svn upon check in. I'm using NMigrations to manage schema and sp/trigger changes when we apply fixes. Unfortunately it's beyond my ability to stop unauthorized schema changes so what I would love to find is a design pattern that allows for an overridable trigger/sp definition.
You need to clearly separate:
source control management
release management
Tweaking in prod shouldn't be possible if the release environment is protected through strict ACL preventing anyone duly appointed to deploy and change stuff.
If that deployment process is automated, then all changes will go through the proper channel because anyone will known a simple "push button" process will be enough to deploy the hotfix.
But if getting that fix in source control and deploy it is complicated, then a tweak directly in prod is usually the result...
Limit rights to change stored procedures and triggers, especially on production. Go ahead and let them know first so they aren't blindsided, but clearly protect production from all unauthorized changes.
Related
we have multiple features that needs to be developed but management decides which features go into Live.. this requires us to have a script for each User Story/Change . But how can I link a DB schema change to a User story in TFS?
What we have now:
TFS with User Story/Task
CC.net Buildserver
I've done research for SSDT, looks awesome! but how can i Link this with TFS?
thanks for reading,
Andy.
As long as your database scripts (or SSDT projects if you choose to move that direction) are checked into TFS version control, there is a deep connection between changesets and work items. When looking at your pending changes before checking in code, there is an option to associate a work item (i.e. a User Story or a Task). If you are using Visual Studio 2012, this is what associating a work item would look like:
And this is what it would look like if you are checking in via Windows Explorer:
There can be a lot of complexity associated with making changes to a system independent of each other. The best solution I've found involves creating "feature toggles" so that you enable or disable a feature after it has been deployed. Take a look at continuous delivery as a topic. Jez Humble wrote a great book on the subject.
Database schema changes can be more complex in some cases than just enabling or disabling a feature. I would suggest using an expand / contract model. You would add any new structure to the database in advance and get that deployed into production in a non-breaking way. Then when you enable the feature that has a dependency on that structure, it's already there. If you need to clean up the database schema after you've removed something then you could do the "contract" cycle out of band with other software changes to reduce the surface area of test.
I am in the process of writing an offline-capable smartclient that will have syncing capability back to the main backend when a connection can be made. As a side note, I considered the Microsoft Sync Framework but since I'm really only going one-way I didn't feel it would buy me enough to justify it.
The question I have is related to SQLite vs. SQLCE and ClickOnce deployments. I've dealt with SQLite before (impressive little tool) and I've dealt with ClickOnce, but never together. If I setup an installer for my app via ClickOnce, how do I ensure during upgrades the local database doesn't get wiped out? Is it possible to upgrade the database (table structure, etc. if necessary) as part of the installer? Or is it better to use SQLCE for something like this? I definitely don't want to go the route of installing SQL Express or anything as the overhead would be far too high for what I am doing.
I can't speak about SQLLite, having never deployed it, but I do have some info about SQLCE.
First, you don't have to deploy it as a prerequisite. You can just include the dll's in your project. You can check this article which explains how. This gives you finite control over what version is being used, and you don't have to deal with installing it per se.
Second, I don't recommend that you deploy the database as a data file and let ClickOnce manage it. When you change that file, ClickOnce will publish it again and put it in the data directory. Then it will take the previous one and put it in the \pre subfolder, and if you have no code to handle that, your user will lose his data. So if you open the database file to look at the table structure, you might be unpleasantly surprised to get a phone call from your user about the data being gone.
If you need to retain the data between updates, I recommend you move the database to the [LocalApplicationData] folder the first time the application runs, and reference it there. Then if you need to do any updates to the structure, you can do them programmatically and control when they happen. This article explains how to do this and why.
The other advantage to putting the data in LocalApplicationData is that if the user has a problem and has to uninstall and reinstall the application, his data is retained.
Regardless of the embedded database you choose your database file (.sqlite or .sdf) will be a part of your project so you will be able to use "Build Action" and "Copy to Output Directory" properties of that file to control what happens with the file during the install/update.
If you choose "Do not copy" it will not copy the database file and if you choose "Copy if newer" it will only copy if you have a new version of your database file.
You will need to experiment a little but by using these two properties you can have full control of how and when your database file is deployed/updated...
Developers have expressed a desire to deploy a single database object from a Sql Server 2008 project, such as a stored procedure, without having to use the build/deploy or schema comparison functions.
To enable this, developers have created their database object scripts including 'if exists.. drop' checks at the top of the script and have included the grant statements for the objects in their scripts.
This results in build errors that then prevent the build/deploy or schema compare functions from operating. So then, developers mark the object as "not in build" but then the object can't be deployed at all using build/deploy or schema compare.
Has anyone found a way of quickly deploying a single database object from visual studio that does not involve schema compare or build/deploy which does not remove the object from the normal build process? Manual copy/paste is not an option but scripting/macros which effectively do the same would be viable.
SQL Server Data Tools (SSDT) now provides this functionality by way of comparison. Individual differences identified in the comparison may be deployed. We have found that during development, publishing tends to result in overlaying the simultaneous changes that other developers are making on the shared development database server. The comparison tool has been working fairly well, except for a very annoying crash issue that occurs when the comparison window is closed. We are using 32bit Vista with 3GB of RAM and VS 2010. This issue may not occur in other configurations.
First I'd like to comment that you seem to be fighting the intended paradigm with regards database projects.
If your project is deployed somewhere [1],
then there should be a corresponding branch / label [2] in your source repository.
If you change a single object [delta] from the above baseline [2], and build the new version [3],
then when you deploy [3] to [1], the deployment script should find only one difference, and the net effect of the change [delta] is all that will be applied.
So, in theory there's no reason not to just simply build and deploy.
TIP: When taking on a new paradigm, it pays to embrace it fully; partial adoption tends to cause its own set of problems.
However, that said, I may be able to help.
We have a similar need because actual deployment is out of our control. We control only part of the database, and have to provide our changes to another team for review.
We have to provide individual 'self-contained' scripts for each object with if exists..drop at the drop and grant permission at the bottom.
However, we want the other benefits of working with the database project, and then simply copy out the individual script files when we "deploy".
The solution we came up with was to place the extra "bits" in a comment block as follows:
/*SINGLE_OBJECT_DEPLOYMENT
if exists (...)
DROP ...
--*/
--/*SINGLE_OBJECT_DEPLOYMENT
if exists (...)
DROP ...
--*/
Note that a simple search and replace of /*SINGLE_OBJECT_DEPLOYMENT with --/*SINGLE_OBJECT_DEPLOYMENT enables the commented out code, so it can be easily put into a macro, or other semi-automated process.
We are building a webapp which is shipped to several client as a debian package. Each client runs his own server. But the update and support is done by us.
We make regular releases of the product, with a clean version number. Most of the users get an automatic update (by Puppet), some others don't.
We want to keep a trace of the version of the application (in order to allow the user to check the version in an "about" section, and for our support to help the user more accurately).
We plan to store the version of the code and the version of the base in our database, and to keep the info up to date automatically.
Is that a good idea ?
The other alternative we see is a file.
EDIT : The code and database schema are updated together. ( if we update to version x.y.z , both code and database go to x.y.z )
Using a table to track every change to a schema as described in this post is a good practice that I'd definitely suggest to follow.
For the application, if it is shipped independently of the database (which is not clear to me), I'd embed a file in the package (and thus not use the database to store the version of the web application).
If not and thus if both the application and the database versions are maintained in sync, then I'd just use the information stored in the database.
As a general rule, I would have both, DB version and application version. The problem here is how "private" is the database. If the database is "private" to the application, and user never modifies the schema then your initial solution is fine. In my experience, databases which accumulate several years of data stop being private, it means that users add a table or two and access data using some reporting tool; from that point on the database is not exclusively used by the application any more.
UPDATE
One more thing to consider is users (application) not being able to connect to the DB and calling for support. For this case it would be better to have version, etc.. stored on file system.
Assuming there are no compelling reasons to go with one approach or the other, I think I'd go with keeping them in the database.
I'd put them in both places. Then when running your about function you quickly check that they are both the same, and if they aren't you can display extra information about the version mismatch. If they're the same then you will only need to display one of them.
I've generally found users can do "clever" things like revert databases back to old versions by manually copying directories around "because they can" so defensively dealing with it is always a good idea.
Assumption: live/production web app suppresses errors being shown to end-users.
Suppose your tech support team wants to see live data but through the eyes of the development-side of the application (maybe you want to see what errors are occurring, or want to see when you've got an issue fixed using an end-user's data).
Right now we've got one database serving both the dev and live boxes (not my idea - I know it's gross).
Ideas?
Edit: Best/handy tools for implementing your suggestion?
We replicate the data back to a different database. Yes, there is a delay, but it keeps people hands out of the production servers. This also allows us to "hide" information that tech support (and other people for that matter) aren't supposed to see.
In addition to replicating data down, on production, we see who's logged into the application, and if it's a member of the company, send them to the real error page versus the happy kitten playing with a ball of yarn apologizing.
Back up and restore from live to dev on a regular basis (once, twice a day). It doesn't need to be realtime (as you might be entering data from the dev side anyway, which could cause problems).
If you have PCI or HIPAA data, make sure you don't put that in your dev environment -- that might break laws.
I generally like to have a 3-tier system for web development:
Development
Testing
Live
Most of the time testing is an exact copy of the live system, except that errors are turned on, when a new version is about to be moved live it's replaced with the new version BEFORE live is, to detect upgrade issues.
Development is completely separate from live, to allow for major changes to things like the database, or changes to the production environment.
I would firstly make errors are either emailed to someone with details of how the user got there or at minimum logged so you can watch the error log while you perform similar actions to see if you get the same messages in the log.
And yes, copying the database on the dev server/site is probably your only option. You don't want any changes made by the development team to live data and you'll probably also have changes that won't work with the production database at some point.
I wouldn't recommend doing a nightly copy as a developer might be in the middle of some new feature where they have added data and then it's erased that night. I usually copy the production database(s) to dev each time a major version is released. This also allows me to do speed testing with a lot of live data. On some systems I also change everyones password to a default so I can login easily as any user.
If your configuration permits it:
a. Add a logging function (if there isn't one already) to write messages of interest to a log file.
b. Run the unix command
tail -f < logfile.txt
which will stream the growing log file to your console.
http://www.monkey.org/cgi-bin/man2html?tail
If you have Windows, you might try this:
http://tailforwin32.sourceforge.net/