is it possible to migrate data from one database table into another database table using liquibase?
Now we are running liquibase changesets on two different databases as we have two executions in pom maven file. But is it possible to write one changeset which selects data from one database table and copies to another database table?
You could use your preferred scripting language to query the data from the table and generate insert statements with the result. Once you have the insert statements, put them in a liquibase formatted sql file. and run them on your target database.
The goal is to have already created the files when the data was originally inserted. If your database already existed before you started using liquibase, then it may be a good idea to restore from a backup taken from the day you started using liquibase and sync it from there.
Related
I have an application that used to run on Snowflake but was migrated to a SQL Managed Instance. i used SnowHub to export all the DDL and then zipped it up should I ever need to look up some source code for a deprecated report or other process.
But how do dump the entire database to a compressed file. You never know when an audit might come up and I might need to restore the data in the state it was a few months or years ago.
You have to use UNLOADING to load data from your table to an internal/external stage in your preferred format.
https://docs.snowflake.com/en/user-guide/data-unload-overview.html
Unfortunately, you cannot unload the whole database with one statement, but need to run one COPY INTO per table. As an alternative you could write a procedure looping over your metadata about tables and run COPY INTO per loop execution.
I made a custom application that is running from several years and is full of company data.
Now I need to replicate the application for another customer, so I set up a new server then i cloned the databases and empty all the tables.
Then I made a database and file shrink.
On the SQL Server side, the databases looks empty but if I run a grep search on the database files .mdf and .log I still can find recurrence of the previous company name also in system databases.
How do I really clean a SQL Server database?
Don't use backup/restore to clone a database for distribution to different clients. These commands copy data at the physical page/extent level, which may contain artifacts of deleted data, dropped objects, etc.
The best practice for this need is to create a new database with schema and system data from scratch using T-SQL scripts (ideally source controlled). If you don't already have these scripts, T-SQL scripts for schema/data can be generated from an existing database using the SMO API via .NET code or PowerShell. Here's the first answer I found with a search that uses the Microsoft.SqlServer.Management.SMO.Scripter class. Note you can include scripts data too (insert statements) by specifying the ScriptData scripting option for desired tables.
I used the following command for restoring a database:
RESTORE DATABASE TorqueNet
FROM DISK = 'C:\Backup.bak';
GO
This command deletes the existing data and performs a complete data replacement.
Instead of that, I want to append new data and replace existing records if there are any changes but keep existing data.
If I understand you, no you cannot. A database restore will overwrite the destination.
It sounds like you are wanting to merge one database into another, which can be done with 3rd party tools.
You would need to
Restore the backup to another database
Merge the newly restored database into the current database.
If this is a task that you need to repeat, then you could probably create an SSIS package to help you automate the process.
If you have a search for SQL Merge tools, or look at this question
A restore is replacing the database, its not running any kinda of insert or update transact sql it really is just overwriting the database.
Hi I am using SSIS (MSSQL) to copy data between multiple tables. This has been working fine up until recently when the S.A.P. team keeps updating the schema of the tables without telling me.
I have multiple tables that they continue to add columns to; this in turn makes my SSIS job of copying the data across fail.
Is there a way in SSIS that I can look at the source table and adjust my destination table to reflect the changes on the fly?
I'm quite new at SSIS and don't mind running a script out of the GUI but wondered if this was an option within the GUI I'm already familiar with.
So in short, can I in SSIS allow for new columns being added to source tables and update my destination tables automatically to stop my jobs failing
(Oh and map source to destination tables automatically)?
You'll have to include the new columns in the data flow, i.e. source and destination (include and map them). So basically you CANNOT automate what you're looking for in SSIS. Hope it helps.
Look into BiML Script, which lets you create and execute SSIS packages dynamically based on the meta data available at run time.
During our SQL Server database deployments, we create a temporary table which contains the new desired state of data for a particular table. We then merge the temp table into the target table (we actually use individual insert, update and delete statements, but that's probably not relevant). The inserts/updates/deletes performed are captured and written out to a log.
We would like to be able to report on what changes would be applied by a deployment, without actually applying them. This is currently done by rolling back the transaction at the end of the above process. This doesn't feel particularly great though.
Now what we are thinking of doing is, instead of performing the changes and rolling them back, we will generate a migration script for the table (generate some SQL code that performs the necessary inserts, updates and deletes). If we want to do the actual deployment, this code will be dynamically executed. If not, the code will just be printed to a log.
It shouldn't take long to put together some code which can generate migration scripts for two specified tables, but I first wanted to verify that there isn't already an existing tool which can do this?
Searching on Google, I can find lots of talk about migrating whole databases, but nothing about generating a data migration script to effectively merge one table into another.
So my question is, does anyone know of such a tool?
There are several data compare tools like:
SQL Data Compare from Red Gate
SQL Server Data Tools
dbForge Data Compare from Devart
Is that what you're looking for?