I believe database migrations are related to the objects of the database like tables, views, data etc.
Can we add / edit users and change their login passwords as part of the migration with flyway ? Is it a considered best practice?
Yes, anything that is valid SQL can be run in a migration.
However, the usual use case for Flyway is that migration scripts are stored somewhere permanent so that you have a trail of how the database got to its current state. You will need to take care that credentials are not accidentally exposed in source control (including history), collections of migration scripts on build servers, or anywhere else.
According to the Flyway docs:
SQL-based migrations are typically used for
DDL changes (CREATE/ALTER/DROP statements for TABLES,VIEWS,TRIGGERS,SEQUENCES,…)
Simple reference data changes (CRUD in reference data tables)
Simple bulk data changes (CRUD in regular data tables)
So yes the migrations can contain DML as well DDL.
Related
in a DevOps (CI/CD) scenario, when Liquibase is triggered by a specific step of a pipeline, is a good practice that Liquibase drops all application ( microservice ) DB tables and recreate all DDL/DML using changesets (only for test and pre-production environment) ?\
If it is, why?
thanks
Liquibase is designed to maintain database model consistency between all environments.
And when you drop database in one of them you break that consistency as you wouldn't want to drop production database as well. Check out Roll Back the Database or Fix Forward? article
If you need to drop some table, you should write additional <dropTable> changeSet.
If you need to test initial application deployment on an empty database, you can always use containers, as suggested by #bilak in the comments.
I have searched for this answer on many websites,and I am getting almost same answer everywhere.
Liquibase helps ensure software releases are always delivered with
the matching state of the database so your features are deployed
quickly and confidently.
We can specify changes once and test them on different databases.
We could pick and choose the changes to be applied in different environments(dev, staging, QA) if we have such a requirement.
This can be done by using Jpa too (using different properties file for staging, dev, production). Then when we should use liquibase, and when it is required?
Liquibase & JPA are very different & have completely separate use cases.
Liquibase is a library for tracking, managing, and applying database schema changes automatically. Liquibase helps track all of your database schema changes in your own files (sql, xml, yml, json). You can think of liquibase as a versioning-type tool for database scripts.
Say somebody inserts new data into a table but, only did it for one environment when it should have been done for all of them. Now an even worse case scenario is that person leaves their company & now no one knows they are missing data. This is a perfect example of how you lose consistency overtime due to careless errors when managing multiple databases or multiple schemas. With liquibase they wouldn't have to worry because that database change is now properly saved & will execute for each environment that that code is deployed to. Essentially, your liquibase scripts are now somewhat part of your project's code (depending on where they live of course).
On the other hand, JPA can validate your existing codebase against your database but, it won't make the necessary DB changes for you (unless you specify something like create-drop which I don't recommend for most use cases). Whenever you use any of their annotations such as #Entity, #Table, #Id, #Blob, etc., none of those changes are automatically applied to your database. JPA is great for modeling how your tables, relationships, & constraints should look in your database. However, it is usually up to you to execute the necessary SQL (or other query language) for such changes. That's where liquibase can conveniently handle it for you.
I currently have 2 databases used by 2 services (let's call them database/service A and database/service B), both of them with their own schemas.
I need to migrate some of the tables from DB A into DB B, once that's all completed re-point service A to service B. I know could easily do the schema migration by using pg_dump utility and that seems to be the "easy" bit.
The problem I have is that both services use Flyway for database version control, hence when I re-point service A to DB B there's a bunch of migrations that are clashing on the same version number because of checksum mismatch.
I've seen that there's a "baseline" functionality in Flyway (https://flywaydb.org/documentation/command/baseline), but at first look that doesn't seem to be what I need.
How could I resolve this problem?
On first considering this problem, the immediate answer is that your move from DbB to DbA is done through one migration on top of the existing migrations in DbA. You don't try to modify the database outside of the Flyway process. Instead, you incorporate the Flyway process into your database change. Flyway is very agnostic to the set of changes you introduce. So, you're just adding another change to the existing set. This shouldn't result in a repair or a baseline to get to the required point.
Let's say the last migration for DbA is V6.3__XXX, we just add V6.4__MigratingDbB to our chain of changes. What's in that script is the necessary set of changes. That should do it.
Grant's answer is definitely the best, but an alternative solution if the database objects for the two services are completely independent, is to have two Flyway configurations which refer to the script collections for each service, and which have distinct history tables. The problem is if there are dependencies between the two services; the migrations from one service would then need to know the current state of play in the other, which could get you in a tangle actually executing them.
I want to be able to create a database and tables the same way Amazon's DynamoDb client does it but with Sql Server. Is that possible?
I'm using .net Core and this is for integration tests. Figured I can throw in the code in the fixture.
Anyone have any ideas?
EF Core Migrations:
"The migrations feature in EF Core provides a way to incrementally
update the database schema to keep it in sync with the application's
data model while preserving existing data in the database."
Create and Drop APIs:
"The EnsureCreated and EnsureDeleted methods provide a lightweight
alternative to Migrations for managing the database schema. These
methods are useful in scenarios when the data is transient and can be
dropped when the schema changes. For example during prototyping, in
tests, or for local caches."
to create your tables at runtime.
And then use one of the Data Seeding techniques:
Data seeding is the process of populating a database with an initial
set of data. There are several ways this can be accomplished in EF
Core:
Model seed data
Manual migration customization
Custom initialization logic
to populate them with known data.
You could start the SQL Server (at least the logfiles) on a RAM disk. and/or use delayed durability ALTER DATABASE x SET DELAYED_DURABILITY = forced. You could also use memory optimized tables but I think you won’t get full compatibility.
BTW: it is dangerous to use such shortcuts if your development process relies entirely on it since developers very late get feedback on bad habits and performance problems.
For that kind of volatile databases (also applies to containers) you need to add some code to your test pipeline or product tomactually create and populate the DB. (If you use containers you can think about packaging a pre-populated DB snapshot)
Using
SQL Server 2008 (not R2)
Visual Studio 2012 Premium
SQL Server Database Project/SQL Server Data Tools (SSDT)
Are there any problems\potential issues with altering a table schema using a post-deployment script?
My company has a web application (App A) whose backend database has tables that are replicated to another company application's database (App B) using CDC Replication. Making schema changes to these tables causes SSDT to use DROP/CREATE when generating the deployment script. This is a problem for App B's database that uses CDC Replication on these tables, because when the table is dropped and recreated, App B's database's CT_[table_name] tables are dropped, bringing App B down. My solution is to use a post-deployment script to make ALTERations to these tables, instead of allowing SSDT to generate DROP/CREATE. Are there any potential problems or issues with this approach?
I could really use some help.
You could conceivably handle such table changes using a Post-Deployment script if you were to exclude those tables from the SSDT project model. This can be achieved in either of the following ways:
For each of the table files involved in CDC replication, set the Build Action property to None
Or simply remove the affected table files from the project altogether
This would prevent SSDT from attempting to perform any actions on that table at all, so you wouldn't have to worry about the comparison engine producing scripts that break your CDC instances.
Naturally, this would mean that any objects that depend on the excluded table objects (such as procs or views) would also need to be moved to Post-Deployment scripts. This would result in the traceability of the database being reduced, as all excluded those tables would no longer have per-file history stored in source control.
Even if a solution can be found that doesn't result in these drawbacks, for example using a pre-compare script per Ed's excellent blog post, there is the issue of deployment atomicity to consider. If part of the deployment occurs in the script that SSDT generates, and another part occurs in a Post-Deployment script, then it's possible for an error to occur that leaves the database in a half-deployed state. This is because SSDT only uses a transaction for the parts of the deployment that it is responsible for; anything included in a Post-Deployment script will be executed after the initial transaction is committed.
This means your Post-Deployment script needs to be written in an idempotent way, so that it can be re-executed if something goes wrong (sorry if that is a bit of an obvious statement... it just seems always like a good point to make whenever post-deploy scripts are mentioned!).
If a higher degree control over the way that your table changes are deployed is desired, without the potential loss of traceability or deployment atomicity, then may I suggest considering a migrations-driven deployment tool like ReadyRoll (disclaimer: I work for Redgate Software). ReadyRoll is a project sub-type of SSDT but uses quite a different deployment style to SSDT: instead of waiting until deployment to find out that the table will be dropped/recreated, the migration script is produced at development time, allowing changes to the sync operations to be made before committing it to source control.
For more information about how SSDT and ReadyRoll compare, have a look at the ReadyRoll FAQ:
http://www.red-gate.com/library/readyroll-frequently-asked-questions-faq
Are you using the in-built refactoring support to rename columns/tables? If you do then the deployment should generate a sp_rename.
If you are and it is a bug because of cdc:
raise a connect item
do your own alter but you will need to run a script before the deployment otherwise it will make the changes it want for you (I call it a pre-compare script)
See https://the.agilesql.club/Blog/Ed-Elliott/Pre-Compare-and-Pre-Deploy-Scripts-In-SSDT for more details.
Ed