We are thinking of integrating Flyway in our application but are concerned about the way it maintains its own versions and how that works with the Software development life cycle (SDLC).
In essence our problem with the approach is that you are maintaining a set of SQL scripts separated by version in the file name instead of maintaining a trunk in version control and releasing/tagging that trunk as a specific version. With Flyway a developer could go back and change an old migration script that relates to a released version of your application and break a version you've already integrated/tested/staged and shipped to a production environment.
What we are considering doing is maintaining the SQL migrations in a project under version control (i.e. my-app-db/trunk/migration.sql) and releasing/tagging from there when a SQL developer is stating it is ready as a release (V1.0.0__blah.sql). The trunk/migration.sql is then wiped out so that the next 1.0.1 or 1.1.0 script can be developed and tagged. A wrapper script will then export the SQL files from the tags, call Flyway with that directory to perform the migration, and clean up the export.
Does this seem like a valid point/approach? Will Flyway ever support something like version control?
Flyway 3.0 will open APIs that will make it possible for end users to extend it in this direction. Out of the box support for SCM integration is currently not on the agenda.
Related
I'm working with the Redgate SQL Comparison SDK at the moment and have got it set up to nicely diff 2 databases.
What I would like to do now is be able to diff an sqlproj from source control with a destination database.
I have tried pulling the sql files using the tfs/vsts SDKs but to no avail.
Is there any way to either build a sqlproj from source control into a dacpac and then pull this in as a source database, or to directly pull the sqlproj in as a source?
Edit:
My ultimate goal with this is to be able to basically compare the version of the database that is in source control with the the database running across many different environments and create delpoyment scripts for the diffs.
I have another couple of Redgate tools that accomplish this (SQL compare & SQL Source), but these can only be installed on 1 (maybe 2 max?) devices, the difficulty I have is in using Amazon RDS (where the endpoints are unreachable outside the VPC), I cannot connect one central install of these tools to all of my environments, and I can't buy an additional license for every environment. So I was trying to use the Comparison SDK to attempt to "roll my own" middle ground.
Many Thanks,
I also work at Redgate, please do email me via dlm#red-gate.com if you want to go into more details into your specific questions and I'll set up a call for us.
In general the process that Redgate recommends for what you're doing below would be to keep the canonised schema that you want all the database to have in version control. You could get that schema in either by each developer using the SQL Source Control product to bring their changes in from SSMS as they develop them, or by using the SQL Compare product to put a version in at the end of a sprint.
You can then use our DLM Automation tools in conjunction with a CI server to automate creating difference reports and sync scripts for your target servers. DLM Automation is a set of PowerShell commandlets and plugins for common CI servers like TeamCity, Jenkins, VSTS, TFS etc. You could also use the SQL Compare Pro command line.
If your whole team have our SQL Toolbelt product then you're licensed to install the DLM Automation tools as many times as you like on build/release agents, so you don't need additional licences per environment.
Are you doing this in the context of an automation build/ci system? You mention VSTS, so the way this normally works is that this would have already pulled the files from source control. Once the files are in the build agent's working folder, you should be able to point the SDK (or SQL Compare command line) at this. Bear in mind that a sql proj isn't an officially supported data source for Redgate tools, although it will work in many instances.
It would be good if you could edit your question and give some background on the higher level problem you're trying to solve just in case we (I work for Redgate) can recommend a more suited set of tools or techniques.
My team develop a web application using ASP.NET. The application is very much based on database (We use SQL Server). Most features require database development, in addition to server and client side code. We use GIT as our source code management system.
In order to check in (and later deploy) the database changes, we create SQL scripts and check them in. Our installer knows to run them and this is how we deploy these changes. Using scripts is very uncomfortable to merge changes (for example, if two developers added a column to the same table).
So my question is what other method or tool can you suggest? I know Visual Studio has a database project which may be useful, I still haven't learned about it, I wonder if there are other options out there before I start learning about it.
Thanks!
I think, you have to add in worlflow and use Liquibase from the first steps of database development (check Liquibase Quick-Start, where changelog started from creating initial structures).
From developers POV adding Liquibase means appearing of additional XML-file(s) in source-tree, when database-schema have to be changed in some changeset
First full disclosure that I work for Red Gate who make this product...
You might be interested in taking a look at SQL Source Control. It's a plugin for SSMS that connects your development database to your existing version control system, git in your case.
When a developer makes a change to a dev database (either in a dedicated local copy, or in a shared dev database) then this change is detected and can then be committed to the repository. Other developers can pick up this change, and you can propagate it up to other environments.
When it comes to deployment you can then use the SQL Compare tool to deploy from a specific revision in your repository that you check out.
It's really handy in cases like your example with two developers making a change to the same table. Either the 2nd developer can pick up the change from the version control system before they commit their change. Or you can use the branching/merging features of git to track these in separate branches and deploy them as separate changes. There's scope to couple this into CI systems too.
Some specifics on running SQL Source Control with git:
http://datachomp.com/archives/git-and-red-gate-sql-source-control/
And a link to a more general set-up guide
http://www.troyhunt.com/2010/07/rocking-your-sql-source-control-world.html
My main problem is where does database go?
The project will be on SVN and is developed using asp.net mvc repository pattern. Where do I put the sql server database (mdf file)? If I put it in app_data, then my other team mates can check out the source and database and run it with the database being deployed in the vs instance.
The problem with this method are:
I cannot use SQL Management Studio with this database.
Most web hosts require me to deploy the database using their UI or SQL Management studio. Putting it in App Data will make no sense.
Connection String has to be edited each time I'm moving from testing locally to testing on the web host.
If I create the database using SQL Management studio, my problems are:
How do I keep this consistent with the source control (team mates have to re-script the db if the schema changes).
Connection string again. (I'd like to automatically use the string when on production server).
Is there a solution to all my problems above? Maybe some form of patterns of tools that I am missing?
Basically your two points are correct - unless you're working off a central database everyone will have to update their database when changes are made by someone else. If you're working off a central database you can also get into the issues where a database change is made (ie: a column dropped), and the corresponding source code isn't checked in. Then you're all dead in the water until the source code is checked in, or the database is rolled back. Using a central database also means developers have no control over when databsae schema changes are pushed to them.
We have the database installed on each developer's machine (especially good since we target different DBs, each developer has one of the supported databases giving us really good cross platform testing as we go).
Then there is the central 'development' database which the 'development' environment points to. It is build by continuous integration each checkin, and upon successful build/test it publishes to development.
Changes that developers make to the database schema on their local machine need to be checked into source control. They are database upgrade scripts that make the required changes to the database from version X to version Y. The database is versioned. When a customer upgrades, these database scripts are run on their database to bring it up from their current version to the required version they're installing.
These dbpatch files are stored in the following structure:
./dbpatches
./23
./common
./CONV-2345.dbpatch
./pgsql
./CONV-2323.dbpatch
./oracle
./CONV-2323.dbpatch
./mssql
./CONV-2323.dbpatch
In the above tree, version 23 has one common dbpatch that is run on any database (is ANSI SQL), and a specific dbpatch for the three databases that require vendor specific SQL.
We have a database update script that developers can run which runs any dbpatch that hasn't been run on their development machine yet (irrespective of version - since multiple dbpatches may be committed to source control during a single version's development).
Connection strings are maintained in NHibernate.config, however if present, NHibernate.User.config is used instead, however NHibernate.User.config is ignored from source control. Each developer has their own NHibernate.User.config, which points to their local database and sets the appropriate dialects etc.
When being pushed to development we have a NAnt script which does variable substitution in the config templates for us. This same script is used when going to staging as well as when doing packages for release. The NAnt script populates a templates config file with variable values from the environment's settings file.
Use management studio or Visual Studios server explorer. App_Data isn't used much "in the real world".
This is always a problem. Use a tool like SqlCompare from Redgate or the built in Database Compare tools of Visual Studio 2010.
Use Web.Config transformations to automatically update the connection string.
I'm not an expert by any means but here's what my partner and I did for our most recent ASP.NET MVC project:
Connection strings were always the same since we were both running SQL Server Express on our development machines, as were our staging and production servers. You can just use a dot instead of the computer name (eg. ".\SQLEXPRESS" or ".\SQL_Named_Instance").
Alternatively you could also use web.config transformations for deploying to different machines.
As far as the database itself, we just created a "Database Updates" folder in the SVN repository and added new SQL scripts when updates needed to be made. I always thought it was a good idea to have an organized collection of database change scripts anyway.
A common solution to this type of problem is to have the database versioning handled in code rather than storing the database itself in version control. The code is typically executed on app_start but could be triggered in other ways (build/deploy process). Then developers can run their own local databases or use a shared development database. The common term for this is called database migrations (migrating from one version to the next). Here is a stackoverflow question for .net tools/libraries to make this easier: https://stackoverflow.com/questions/8033/database-migration-library-for-net
This is the only way I would handle this on projects with multiple developers. I've used this successfully with teams of over 50 developers and it's worked great.
The Red Gate solution would be to use SQL Source Control, which integrates into SSMS. Its maintains a sql scripts folder structure in source control, which you can keep in the same folder/ respository that you keep your app code in.
http://www.red-gate.com/products/SQL_Source_Control/
How do you manage your sql server database build/deploy/migrate for visual studio projects?
We have a product that includes a reasonable database part (~100 tables, ~500 procs/functions/views), so we need to be able to deploy new databases of the current version as well as upgrade older databases up to the current version. Currently we maintain separate scripts for creation of new databases and migration between versions. Clearly not ideal, but how is anyone else dealing with this?
This is complicated for us by having many customers who each have their own db instance, rather than say just having dev/test/live instances on our own web servers, but the processes around managing dev/test/live for others must be similar.
UPDATE: I'd prefer not to use any proprietary products like RedGate's (although I have always heard they're really good and will look into that as a solution).
We use Red-Gate SQLCompare and SQLDataCompare to handle this. The idea is simple. Both compare products let you maintain a complete image of the schema or data from selected tables (e.g. configuration tables) as scripts. You can then compare any database to the scripts and get a change script. We keep the scripts in our Mercurial source control and tag (label) each release. Support can then go get the script for any version and use the Redgate tools to either create from scratch or upgrade.
Redgate also has an API product that allows you to do the compare function from your code. For example, this would allow you to have an automatic upgrade function in your installer or in the product itself. We often use this for our hosted web apps as it allows us to more fully automate the rollout process. In our case, we have an MSBuild task that support can execute to do an automatic rollout and upgrade. If you distribute to third-parties, you have to pay a small additional license fee for each distribution that includes the API.
Redgate also has a tool that automatically packages a database install or upgrade. We don't use that one as we have found that the compare against scripts for a version gives us more flexibility.
The Redgate tools also help us in development because they make it trivial to source control the schema and configuration data in a very granular way (each database object can be placed in its own file)
The question was asked before SSDT projects appeared, but that's definitely the way I'd go nowadays, along with hand-crafting migration scripts for structural db changes where there is data that would be affected.
There's also the MS VSTS method (2008 description here), anyone got a good article on doing this with 2010 and the pros/cons of using these tools?
For the last few years I was the only developer that handled the databases we created for our web projects. That meant that I got full control of version management. I can't keep up with doing all the database work anymore and I want to bring some other developers into the cycle.
We use Tortoise SVN and store all repositories on a dedicated server in-house. Some clients require us not to have their real data on our office servers so we only keep scripts that can generate the structure of their database along with scripts to create useful fake data. Other times our clients want us to have their most up to date information on our development machines.
So what workflow do larger development teams use to handle version management and sharing of databases. Most developers prefer to deploy the database to an instance of Sql Server on their development machine. Should we
Keep the scripts for each database in SVN and make developers export new scripts if they make even minor changes
Detach databases after changes have been made and commit MDF file to SVN
Put all development copies on a server on the in-house network and force developers to connect via remote desktop to make modifications
Some other option I haven't thought of
Never have an MDF file in the development source tree. MDFs are a result of deploying an application, not part of the application sources. Thinking at the database in terms of development source is a short-cut to hell.
All the development deliverables should be scripts that deploy or upgrade the database. Any change, no matter how small, takes the form of a script. Some recommend using diff tools, but I think they are a rat hole. I champion version the database metadata and having scripts to upgrade from version N to version N+1. At deployment the application can check the current deployed version, and it then runs all the upgrade scripts that bring the version to current. There is no script to deploy straight the current version, a new deployment deploys first v0 of the database, it then goes through all version upgrades, including dropping object that are no longer used. While this may sound a bit extreme, this is exactly how SQL Server itself keeps track of the various changes occurring in the database between releases.
As simple text scripts, all the database upgrade scripts are stored in version control just like any other sources, with tracking of changes, diff-ing and check-in reviews.
For a more detailed discussion and some examples, see Version Control and your Database.
Option (1). Each developer can have their own up to date local copy of the DB. (Up to date meaning, recreated from latest version controlled scripts (base + incremental changes + base data + run data). In order to make this work you should have the ability to 'one-click' deploy any database locally.
You really cannot go wrong with a tool like Visual Studio Database Edition. This is a version of VS that manages database schemas and much more, including deployments (updates) to target server(s).
VSDE integrates with TFS so all your database schema is under TFS version control. This becomes the "source of truth" for your schema management.
Typically developers will work against a local development database, and keep its schema up to date by synchronizing it with the schema in the VSDE project. Then, when the developer is satisfied with his/her changes, they are checked into TFS, and a build and then deployment can be done.
VSDE also supports refactoring, schema compares, data compares, test data generation and more. It's a great tool, and we use it to manage our schemas.
In a previous company (which used Agile in monthly iterations), .sql files were checked into version control, and (an optional) part of the full build process was to rebuild the database from production then apply each .sql file in order.
At the end of the iteration, the .sql instructions were merged into the script that creates the production build of the database, and the script files moved out. So you're only applying updates from the current iteration, not going back til the beginning of the project.
Have you looked at a product called DB Ghost? I have not personally used it but it looks comprehensive and may offer an alternative as part point 4 in your question.