I'm trying to implement Database versioning and Rollback features to my springboot project and I have explored some of the database versioning tools like Flyway, Liquibase, DB Geni and almost all the tools are creating the tracking tables in the database itself. For example If I'm connecting to Database1 which is integrated with flyway and tracking tables are getting created in the Database1 itself but I need a tool which Performs script execution on Database1 but tracking tables needs to be created in Database2.
Can you please help me with any of the tools which will do like this?
Thanks in advance.
Related
I maintain a Windows based application backed by SQL Server DB so there is a set of SQL entities, like tables, views. With time I add new features and fix bugs so schema of the tables and views changes. Once I need to deploy a new version of the application I deploy the DB part by relying on DacPac/DacFx which automatically generates a difference between already deployed DB and the supplied DacPac so the already deployed DB is altered to match the DacPac's content. This way I don't have to write a code which compares 2 schemas and then generates a difference - DacFx does that for me.
That works well but now I need to expand the application so it also supports SQLite DB, I will for sure have to create a new application layer working with SQLite which is doable but one place I need help with is being able to create and maintain SQLite DB schema in the same way I do for SQL Server with DacPac/DacFx so a difference in schemas is computed and applied. While doing that I ideally want do write the SQL schema once so it could be applied to SQL Server as well as SQLite. Ideally, I need to generate SQLite schema based on the schema specific to SQL Server.
I looked into sqldiff which is capable of generating difference between 2 SQLIte DBs and thought I could:
use a technique from here to migrate SQL Server schema to SQLite
generate a temporary SQLite DB based on the generated above schema
compare the above deployed temporary DB to an existing SQLite DB by using the sqldiff and finally apply the difference to the target SQLite DB
but the sqldiff, as stated in the Limitations section:
The sqldiff utility is not designed to support schema migrations
In addition it has limitations around views:
The sqldiff.exe utility does not (currently) display differences in
TRIGGERs or VIEWs.
So I interpret that like that tool could probably be used for some migration cases but it is not really recommended.
How do you suggest generating and applying the schema differences?
I'm also interested to know how others solve the task of incrementally updating schema of their SQLite DB even if I take SQL Server completely out of equation and would instead maintain SQLite schema, in the source code, only. Does everyone create their own schema comparing tools instead of using something similar like DacFx in SQL Server world?
I believe database migrations are related to the objects of the database like tables, views, data etc.
Can we add / edit users and change their login passwords as part of the migration with flyway ? Is it a considered best practice?
Yes, anything that is valid SQL can be run in a migration.
However, the usual use case for Flyway is that migration scripts are stored somewhere permanent so that you have a trail of how the database got to its current state. You will need to take care that credentials are not accidentally exposed in source control (including history), collections of migration scripts on build servers, or anywhere else.
According to the Flyway docs:
SQL-based migrations are typically used for
DDL changes (CREATE/ALTER/DROP statements for TABLES,VIEWS,TRIGGERS,SEQUENCES,…)
Simple reference data changes (CRUD in reference data tables)
Simple bulk data changes (CRUD in regular data tables)
So yes the migrations can contain DML as well DDL.
Im new to working with databases and Im trying to do the following
Copy all databaseA schemas (which has several tables each and permissions attached) without any data to my
existing databaseB table as record fields (which now contains only one schema and
also a few tables and permissions attached)
databaseA is an Amazon redshift database & databaseB is an Amazon RDS database. Im connecting to both using DBeaver, databaseA I'm using a redshift driver & databaseB I'm using a PostgreSQL driver
After the inital copy I want to run a daily cron job that checks for the following
a. Compare databaseA to databaseB table
b. If databaseA does not match databaseB (in terms of schema & table permissions)
c. Then switch all perms to match databaseB table
Any feedback on how to approach this would be appreciated!
You could create a python script that connects to both of the databases. You could set up a cron job to spot the differences daily and to update the database.
You can have a query like this for PG:
SELECT table_schema,table_name
FROM information_schema.tables
ORDER BY table_schema,table_name;
And something like this for Redshift:
SELECT schemaname, tablename
FROM PG_TABLE_DEF;
From there it`s just a matter of comparing the two and deciding if you want to update certain tables. Good luck.
I don't have experience with AWS. Im translating the little knowledge I have from OCS which is a younger solution than AWS.
First, Amazon Redshift is tailored for Data warehousing. RDS is a cloud relational database. Im not sure what your aim is to copy from Redshift to RDS. It would be more natural to have the DB or multiple DB copy/clone to the data warehouse unless this was some form of backup. You might need to look into the architecture of your solution.
Oracle Cloud which is fairly new provides a service for copying. Amazon should have a similar solution as they have been in the cloud business longer.
I have had a look at the Amazon documentation. Your challenge has a solution "backwards" here.
After copying the two dbs my assumption is they would be similar structurally. What is affecting the changes on dbA ? It feels like you don't want to use the permissions on dbA maybe its compromised.
My suggestion is to use permissions to prevent changes to dbA. Look at the IAM documentation and check the logs for dbA. If you really need to develop a solution use the API or CLI to interface with the db
I have a requirement of a syncing set of tables (with data) to a different database from our production database. This is an Azure SQL DB.
I created replicated DB (using Azure Geo Replication) and it is a read-only DB. My plan was to enable SQL Server Change Tracking (CT) in replicated DB and query those changes from Change Tables, so that Production DB will not have any impact because of change tracking. But then I found out it is not possible to enable Change Tracking or even access Change Tables in the DB replica I created.
Then I saw Azure 'Sync to other databases' feature and tried it out with the replicated DB. But it is also not possible since this feature does not support syncing data from a read-only db.
1) What is the solution for this? I cannot afford using 'Sync to other databases' feature on my Production db, because it uses DB Triggers to track these changes.
On the other hand, I cannot afford to enable CT in Production DB either.
2) Is there a way of enabling and tracking changes using CT from a replicated DB?
3) Or is there a way to use 'Sync to other databases' feature with a replicated DB?
The application trying to build is an analytics application. So I am trying to get data I want from a couple of other production DBs.
Thank you.
What is the effort to migrate from Blackfish to Firebird ?
I use DBX4
No stored procedure, or function or views
I access all my data trough SQL queries
If you don't rely on specific Blackfish features (SP, trigger, admin packages calls etc), the change should not be a pain.
I've did this kind of migration in a DEMO for a customer.
If you have used Blackfish as a "collection of tables accessible via SQL" then the change will not be difficult at all.
There is a driver for delphi and dbx4 firebird
http://sites.google.com/site/dbxfirebird/
my guess is that is easy for you to dump the blackfish tables into csv
and then to import them in firebird and later modify only the application to access
the firebird db using the dbx4 firebird settings
For database pumping you can use tools like :
IBExpert (full version)
Database Workbench (full version)
or database comparer