Can Flyway accomodate different SQL dialects in the same migration script? - database

in my Company we manage a SaaS infrastructure that uses PostgreSQL as DB solution. However, some clients wants us to perform an on-premise deployment, in which they provide MySQL instead.
This of course entails identifying and isolate PG-specific datatypes and functions, find a corresponding MySQL equivalent, and make them run properly.
If we were using Liquibase, that would be easy, since it would require me to write databaseChangeLog(s) with a precondition like this:
<preConditions onFail="HALT">
<dbms type="mysql"/>
</preConditions>
and include as many as I want within the same databaseChangeLog
With Flyway however, it seems like the only possible way is to define different folders, under which place the SQL files for a particular RDBMS.
This seems quite overkill and error-prone to me, so I would like to know: does Flyway have some specific syntax that I missed, which allows me to write different SQL-dialect-specific queries within the same .sql file? I.E. I would hypothetically expect something like this :
--; if(mysql)
CREATE TABLE test (start_date TIMESTAMP NOT NULL);
--; endif
--; if(postgresql)
CREATE TABLE test (start_date TIMESTAMPTZ NOT NULL);
--; endif
...
thank you in advance!

This is not a feature of flyway. SQL scripts in flyway are meant to be purely valid SQL (with some exceptions for variable injection).
However, you may be able to look at Java migrations to do conditional execution but you would not be able to do that with your existing SQL scripts.

Flyway doesn't support conditional blocks within migration files but it is possible for SQL dialects to use the same migration scripts.
I've used flyway placeholders to do this, rather than conditional blocks. I've got a demonstration of the technique here, using PowerShell as the glue. Yours sounds an interesting application, I have the same requirement for an application that can be installed in various environments that use a different brand of RDBMS, and I can see the value of block structures. The reason that I used placeholders was that the difference between SQL dialects us usually fairly trivial. Maybe a combination of techniques would be a winner.
I would guess that you can use SQL conditional blocks together with placeholders. You would just then need a placeholder with the name of your rdbms, taken from the flyway.conf jdbc connection-string.

Related

Can Flyway be used to migrate tables between different databases without migration file?

There are many questions like this on Stack Overflow, but it seems to me that they already have the migration scripts in place. For example, insert and update statements are available, as such, Flyway can just use those scripts to create the tables in the target database and its data.
However, my question is that, what if we don't have those scripts? For example, tables are being created manually or with some other tools and the data are being inserted over the years with the bound application, now we want to switch to a different SQL database. Can Flyway be used as a tool to transfer all the tables and databases only with providing connections?
If the answer is no, how this sort of migration can be done and what are the best practices.
I did a search and went through Flyway documentation but they are all vague and doesn't give you a clear example of that. Some of these tools I found are used for Salesforce but I need a tool/library that possibly can be used in Java using JDBC connection, or other languages such as Python etc, as our databases - for security reasons - cannot be accessed directly and are cloud based.
For your information, we are using a range of databases PostgreSQL, Aurora MySQL, SQL Server.
No, Flyway can't do this sort of thing.
Flyway is a deployment tool. While it certainly can include data movement, as with the deployment of database objects, the scripts supporting data movement have to be completely idempotent or completely isolated in their deployment. Neither of these is lending itself to what you're talking about.
What you're talking about is something like Redgate SQL Compare along with SQL Data Compare. These two would allow you to compare two databases, identify the differences, then generate the necessary scripts. I'm aware of no open source tools that do all that, especially that do all that across multiple data platforms. And that tool only supports SQL Server (there is a second one for Oracle, but no others).
The thing is, if you're allowing deployments to occur using manual processes or 3rd party mechanisms, without going through source control as centralized management of your code, you can't use Flyway anyway. Flyway requires a consistent and stable process wherein it is the thing running deployments. Allowing, or even encouraging, drift through out-of-band deployments will break your Flyway deployments.
DISCLOSURE: I work for Redgate, but we're not the solution you're looking for.

Can I apply changes across different databases on the same server using liquibase

We have a large enterprise system which has many databases on a single server (Sybase)
Developers will make a change in one db, script it, then maybe make a change in another db, add that to the list of scripts and so on.
Our release then runs through these scripts making changes to the objects in different databases in the same order.
Reading the Liquibase documentation, it seems like it would work if you applied all the changes to one db, then another, then another. Which wouldnt really work in our case as a change in one db may rely a change done earlier on another db and vice versa.
How could I use Liquibase to do the same?
You probably need to start looking at Datical DB (disclaimer: I work at Datical), which is a set of tools and extensions around Liquibase to handle these kinds of situations.
Alternatively, you could do something similar, writing your own tools to control Liquibase. Liquibase is controllable at several different levels - you could use a scripting tool to execute the Liquibase command line, or you could use Java or Groovy (or any other language that integrates with tools in the JVM) and use the Liquibase classes more directly.
Liquibase does not currently support connecting to multiple different databases. It is being considered for version 4.0.
If your databases are the same database instance but different schemas, you can use the schemaName attribute to target changeSets at different schemas from one changeSet to the next. You will need a single connection URL that has access to all of the schemas.
If your databases are different instances or not all accessible from a single connection URL, you can probably create custom change classes or extensions that allow you to run SQL against different connections, although that will not be as clean or easy as the schemaName option.

Generating database scripts for SQL Server database versioning

In the scope of responsible programming and versioning, I would like to start to version my database changes especially since I am developing on my database instance then moving it to production. I haven't found any thing that truly makes sense to me on how to do this. I am using Visual Studio 2010 Pro as my IDE. Is there a document that makes this process simple and able to detect changes to the database with relative ease? Or what should I change in my workflow to make this easier?
One way that I've successfully done this sort of thing in the past, is via Sql Source Control. Visual Studio does not offer this functionality for you.
Alternatively, you can use SSMS to generate the Database scripts for you and save it as a file; then you can check in the script. You would chose whether you generate the whole DB script in one file or whether you do it on an object by object basis. The syncing part will have to be done by you by executing your scripts in production. In conclusion a total nightmare.
Redgate also offers Sql Compare, which is great for syncing databases. Take a look at their products if you or your company can afford them.
We use our own DB solution in-house which brings all the tools required for proper DB versioning. While I realize that it may not be a perfect solution for everyone, I invite you to have a look at it (it is open-source): bsn ModuleStore
The versioning aspect is as follows: the tool can script out the SQL semi-automatically, and it does reformat the source code to be in an uniform format. The files will therefore always be identical for the same source, no matter of when and by whom something has been scripted; this therefore works nicely with non-locking source control systems (especially SVN, Git or Mercurial).
The reformat puts all statements in the same form (e.g. optional keywords such as AS, INNER, OUTER etc. are dealt with), scripts everything to the "dbo" schema (even if it was in a different one), puts all identifiers into the square braces ([something]), uppercases all reserved words, does the indentation etc.
Besides versioning, the runtime part of the tool can diff the running DB and the CREATE scripts (DB source code) and apply updates automatically for all non-destructive changes (e.g. updating indexes, constraints, views, stored procedures, triggers, custom types, new tables etc.). Destructuve changes have to be scriped manually (table changes which then usually require data transformations). The runtime will make sure that all updates are performed in a transaction and rollback if the resulting DB doesn't match the CREATE scripts, therefore you get the safety of knowing that the DB is exactly on the version required by the application, even if it has been tampered with manually.
Also, multiple "modules" can be used in a single database. Each module is stored as a schema and independent of other schemas, thereby making it possible to add or remove modules from one single DB, and avoiding the need to create multiple databases for different parts of the application. Also, the use of schemas to do this makes sure that there are no name collisions.
It may be worth noting that the toolset has no dependency to the SMO, it is autonomous.
Save Your Database scripts at SVN. Here is the Refernce How to use SVN Tortoise
OR
Save your database script at VSS. Here is the reference What is VSS ? How can we use that ?
In both cases you can keep track of the changes done so that in future you can check the history which in saved in the form of versions.
You can use Red Gate product also
EDIT
How do you pull out what what has changed?
Use comparison feature to check the changes made in the previous versions.
How do I apply the changes to the live database server?
Download the latest file from server.
I hope you are not using the Drop statements for the Table in your consolidated script. As it will delete all records from the table.
Drop statements will take place for Stored Pro, View, Function etc.
Please note that you have to run the complete latest database script file on the production server with below mentioned action plans
1. Remove Drop Statement for Schema DDL
2. Add Drop/Create Statements for Stored Proc/Views
3. Include Alter statements DML of schema.
Hope this will definitely help you.

How to separate programming logic and data in MS SQL Server 2005?

I am developing a data driven website and quite a lot of programming logic resides in database stored procedures and database functions. I found myself changing the stored proc/functions quite a lot in order to fix bugs or add new functionality. The data (tables) have remained mostly untouched.
The issue I am having is keeping track of versions of stored proc/functions. Currently I am incrementing version of whole database when I do a set of changes. As data is huge (10 Gb) I get issues having to run development version and release versions of databases in parallel.
I wish to put all the stored procs and functions in one database and keep data in one database, so that I can better manage the changes.
I am sure others would have encountered similar suggest and request suggestions on how to best handle this situation.
I would also recommend using source control keyword expansion in your stored procedures ($Version:$)
That way you can eyeball, grep, search syscomments, etc to see what version you have on your deployed database.
You can version just the schema dumps. In combination with source control keword expansion (as suggested by Rawheiser), you just take a look at what version you have in the database, generate a diff and apply it.
Also, there are several excellent tools to compare databases and their schemas, generate DDL scripts etc.: SQL Workbench, Power Architect, DDLUtils and Redgate SQL Compare, to name a few. SQL Compare is likely to work best with SQL Server, although all the others are FOSS and provide a higher ROI (in terms of time spent learning and what you can do with them) as they are platoform and RDBMS independent.
Finally, I have to say...I understand that the immediate results you get with logic in the DB are tempting, but if you've gone beyond more than a couple of procedures in the database, you're setting your self up for quite a lot of pain, sifting through what easily turns into spaghetti code and locking your application to a single database vendor. You might have your reasons, but I've been there and didn't like it very much. Logic can live very nicely in a different layer.
For source control you have several options:
Use a Visual Studio Database project.
Use SQL Server 2005's built-in support for source control
Use a third part tool such as SQL Compare
IMO Option 1. is preferable.

Need good scheme/workflow for managing database objects using Subversion

How do you track/manage your stored procedures, views, and functions in SQL Server?
I'd like to use Subversion, but it looks like I would have to just save & commit the CREATE/ALTER statements. That might work okay for me, but I suspect I'd end up doing a lot of nagging.
Is anyone using versioning with their databases? Is there a better way?
In the past, people have just commented out parts of the code and left it in. Or, they add little "added on 2/31/2010" comments all over. It drives me nuts, because I know there is a better way.
We do log changes in the object's header, but that's pretty limited. It would make my life easier to be able to diff versions.
Additional Info
We are using SQL Server 2005. I have Subversion (via VisualSVN Server) and TortoiseSVN installed, but I'm open to other suggestions.
By database objects, I specifically mean stored procedures, views, and functions.
There are only a few tables I would need to track. The database is the backend for a commercial application, and we mostly pull information out for reporting
I found a related question about stored procedure versioning
We script everything and put it into Subversion. Nothing can be loaded to Prod without a script (developers do not have rights to prod) and the people with rights on prod only accept scripts they loaded from Subversion.
We revision our database, schema creation, dw, etl, stored procedures just like any other piece of code, because it's code!
I have also seen people type dates in headers, etc. This is normally due to them completely missing the point of revision control.
Have a look at liquibase, here
It manages your sql changes/scripts for you, and can apply them in conjunction with svn via hooks or scripts. Makes doing all sorts of setup easy, and helps eliminate the case of the missing trigger/sproc/etc...
I'm not sure what you all mean with "database objects". Are these only the tables, views, procedures etc or also data? I mean daily created data?
Assumed you mean the database schema definition. By my experience there is only one way to handle database schema definitions (if you don't have NHibernate or some similar tool). You write sql scripts that create your database from scratch and check them in. You use the same scripts for installation of your software. You see the differences by just comparing the scripts files.
Whenever I've gone through this excercise, it's come down to 3 main things that need to be source-controlled:
Stored Procedures / Views / Triggers (more or less anything that can be fairly expressed as "code". These are fairly simple, include a conditional drop and create at the top of the file.
Table Schema - DROP / CREATE statements as above. You can try to get fancy with ALTER statements, but it tends to get really messy.
The biggest challenge we faced was this forces you into a system where your DB goes back to an initial state often - if there's a fair amount of work involved in bringing DBs to something usable / testable, it can be a pain. In that case we kept a library of scripts that brought a DB to various usable states, and source controlled those as well.
Data within tables. We looked at a couple of approaches here - either a series of INSERT statements stored in a file like "TableName_Data.sql" or a CSV file with custom build tooling that parsed and inserted when the DB was rebuilt.
Ultimately we went with the INSERT statements for simplicity's sake.

Resources