I am currently working on Postgressql database and have to create a database backup on a daily basis using Airflow DAG.
I have created a DAG and I am able to create the backup using BashOperator and pg_dump command. But this approach reveals the sensitive information in Airflow logs.
Is there an option we can mask the passwords and secret key?
Or Can we use PostgresOperator or Postgreshook to create the backup?
Related
We have an on-prem SQL Server DB (SQL Server 2017 Comp 140) that is about 1.2 TB. We need to do a repeatable migration of just the data to an on cloud SQL (Paas). The on-prem has procedures and functions that do cross DB queries which eliminates the Data Migration Assistant. Many of the tables that we need to migrate are system versioned tables (just to make this more fun). Ideally we would like to move the data into a different schema of a different DB so we can avoid the use of External tables (worried about performance).
Moving the data is just the first step as we also need to do an ETL job on the data to massage it into the new table structure.
We are looking at using ADF but it has trouble with versioned tables unless we turn them off first.
What are other options that we can look and try to be able to do this quickly and repeatedly? Do we need to change to IaaS or use a third party tool? Did we miss options in ADF to handle this?
If I summarize your requirements, you are not just migrating a database to cloud but a complete architecture of your SQL Server, which includes:
1.2 TB of data,
Continuous data migration afterwards,
Procedures and functions for cross DB queries,
Versioned tables
Point 1, 3, and 4 can be done easily by creating and exporting .bacpac file using SQL Server Management Studio (SSMS) from on premises to Azure Blob storage and then importing that file in Azure SQL Database. The .bacpac file that we create in SSMS allows us to include all version tables which we can import at destination database.
Follow this third-party tutorial by sqlshack to migrate data to Azure SQL Database.
The stored procedures can also be moved using SQL Scripts. Follow the below steps:
Go the server in Management Studio
Select the database, right click on it Go to Task.
Select Generate Scripts option under Task
Once its started select the desired stored procedures you want to copy
and create a file of them and then run script from that file to the Azure SQL DB which you can login in SSMS.
The repeatable migration of data is challenging part. You can try it with Change Data Capture (CDC) but I'm not sure that is what exactly your requirement. You can enable the CDC on database level using below command:
Use <databasename>;
EXEC sys.sp_cdc_enable_db;
Refer to know more - https://www.qlik.com/us/change-data-capture/cdc-change-data-capture#:~:text=Change%20data%20capture%20(CDC)%20refers,a%20downstream%20process%20or%20system.
I have to copy table data from one Azure SQL Database to another Azure SQL Database which are under same Azure server.
Is there any way to do this using Azure data factory? Also, this needs to be scheduled as a daily feed.
Edit : How can we add more tables to the existing dataset ? I have created this for 3 tables, now i want to add two more tables to this, how ?
Did you have a look at Copy data to and from SQL Server by using Azure Data Factory?.
In Azure Data Factory, you can use the Copy activity to copy data
among data stores located on-premises and in the cloud. After you copy
the data, you can use other activities to further transform and
analyze it
You can have a look at the steps from here on how to configured a triggered pipeline.
One important thing to remember is that you'll have to define the data set (with or without schema) for all tables that require copy for any source-destination combination.
you can think of elastic queries(preview)-for cross database queries and elastic jobs (preview) for job scheduling.
Utilize Elastic query for bringing result from another database on the same server. Read more on Elastic Query. The advantage is it is coming as free with Azure SQL.
Elastic database query (preview) for Azure SQL Database allows you to
run T-SQL queries that span multiple databases using a single
connection point.
Schedule Elastic job(currently in preview) which can be used to schedule job in a Azure SQL database. Read more on Elastic jobs
Elastic Database Jobs (preview) are Job Scheduling services that
execute custom jobs on one or many Azure SQL Databases.
I'm setting up an Azure SQL DB for our Web App. We have enabled Point In Time Retention (PITR) and Long Term Retention (LTR). Our process is to keep backups for 1 year.
Periodically, we need to upgrade the DB by applying SQL scripts. Sometimes there is a problem with the upgrade scripts and the upgrade fails. We need to rollback the database to the previous version.
To rollback the DB I tried the restore feature. However, the restore feature seems to only create new DBs; therein lies the problem. Restoring to a new DB and removing the old one works great, but we lose all our backup history. It appears backups are tied to the DB (probably to the ResourceId).
So, how can I use Azure SQL DB and periodically restore a DB and still maintain all the back up history?
Unfortunately, restoring from a backup in Azure SQL Database always creates a new database. The secret here may be to rename the newly restored database with the name of the original database. You will even see that the restored database once renamed it then shows all the security recommendations, automatic tuning recommendations of the original database.
So delete existing database, restored the database, and rename it as the original database.
You can reference this document Recover an Azure SQL database using automated database backups , it gives the answer that all the recover are creating new database.
By default, SQL Database backups are stored in geo-replicated blob storage (RA-GRS). The following options are available for database recovery using automated database backups:
Create a new database on the same SQL Database server recovered to a specified point in time within the retention period.
Create a database on the same SQL Database server recovered to the
deletion time for a deleted database.
Create a new database on any SQL Database server in the same region
recovered to the point of the most recent backups.
Create a new database on any SQL Database server in any other region
recovered to the point of the most recent replicated backups.
If you configured backup long-term retention, you can also create a new database from any LTR backup on any SQL Database server.
improtant:
You cannot overwrite an existing database during restore.
"So, how can I use Azure SQL DB and periodically restore a DB and still maintain all the back up history?"
You can use Database replacement:
If the restored database is intended as a replacement for the original database, you should specify the original database's compute size and service tier. You can then rename the original database and give the restored database the original name using the ALTER DATABASE command in T-SQL.
Hope this helps.
I have a TDE protected database which I want to extend to Azure using replication. My question is, do I have to restore certificate on the Azure database before I start setting up the database as the subscriber?
do I have to restore certificate on the Azure database before I start setting up the database as the subscriber?
No. Per docs:
"Replication does not automatically replicate data from a TDE-enabled database in an encrypted form. You must separately enable TDE if you want to protect the distribution and subscriber databases. Snapshot replication, as well as the initial distribution of data for transactional and merge replication, can store data in unencrypted intermediate files; for example, the bcp files. "
Transparent Data Encryption and Replication
Is there any way to transfer the large volume of data from Azure SQL to on-premises SQL Server 2016 Enterprise/Standard? The requirements prescribed as follows:
Weekly full database transfer
Daily delta transfer before midnight
I read about SSIS for Azure Blob Storage but am not sure whether it is applicable to this context.
Updated: I found an article on Azure Data Sync; according to that article, it seems doable. Please share your experiences. That would be extremely helpful.
https://www.mssqltips.com/sqlservertip/3062/understanding-sql-data-sync-for-sql-server/
Weekly full database transfer
SSIS Doesn't provide a way to do Full transfer of data(i mean backup),unless you want to truncate and insert from source..
For Weekly full database transfer,i would go with SQLAzure Export/Import functionality
Refer below links for more details..
1.https://github.com/richorama/SQLDatabaseBackup
2.I need to automate SQL Azure database backup in SQL Script files. How can i do so?
Daily delta transfer before midnight
You will need a way to identify delta..so create a table with all table names and last run time
create a console application which uses bulk insert functionality,which uses above table as base and insert in onpremises