I have a database deployed on Azure SQL. Due to various requirements, my company would like to make a continuous mirror of this database to an on-premise SQL Server database. We have explored several ways to implement this, so far to no avail.
The way recommended online, using the Azure Sync tool, didn't work for us -- and seeing as this tool is currently in CTP, it is not currently supported by Microsoft, so we want to implement this ourselves.
The requirements for this mirroring solution aren't extremely high - we only want to make backups every half an hour or so, so continuous synchronization isn't an issue. What would be the best way to approach this without resorting to commercial solutions that seem to be an overkill for us?
Hummm... it depends what you call an overkill. One way to do this is to wrap up your own BCP commands out of SQL Azure, and BCP into a database on premises. But depending on your database complexity, this may very quickly get out of hands, unless you can identify the top 2 or 3 tables that you really want to export/import to minimize your work. In fact you might event want to consider using SSIS; you will need to manage your own retry logic regardless of the approach you take.
We ended up choosing a complete backup-restore from the Azure account to the on-premise VM, since implementing custom retry & incremental query generation sounded way too complicated for this case.
Azure SQL Database support Import Export Database Service including Automated Exports to Azure Storage account.
Using Automated Exports you can define the export interval per day and bacpac files will be created in your Azure storage account which can be imported into your on-premise server.
You can find more detail about Configuring Automated Exports
Related
I use a SQL Server 2017 database to perform ad-hoc data analytics activities to support my team. In order to source data from various databases, I either mount a backup on my environment (if the target DB holding the data I’m after is also SQL Server) or use linked servers to establish a direct connection (where I need data from Oracle or iSeries).
More recently though I’m coming across SaaS based systems and was wondering if there’s any way I can establish a direct connection between my database and the SaaS database? I’m not sure whether SSIS packages will do the trick. Any pointers would be gratefully appreciated as I’m struggling to get the right, scalable solution for this problem!
Data integration with SaaS solutions is a mixed bag. You have to discover on a case-by-case basis what integration or data export functionality each SaaS application has. Few will allow any kind of direct database access, but you might find an ETL tool that has pretty broad connectivity.
In Azure you can look at Logic Apps Connectors, and Power Query Connectors. Or products like Boomi. Which have connectors for many popular SaaS Applications.
I need to regularly (but incrementally) sync (one way) the contents of a set of SQL Server Azure tables to a PostgreSQL Azure instance.
Here are some of the avenues I've considered:
Linked server from SQL Server. No go. Apparently Azure doesn't support linked servers.
Foreign Data Wrapper from PostgreSQL. No go. PostgreSQL on Azure only supports the postgres_fdw, not the needed tds_fdw.
Azure Data Factory. No go. The data copy process doesn't work incrementally, and the sink pipeline component doesn't support PosgreSQL.
Commercial replication solutions. Too expensive for a startup and most aren't hosted.
SymmetricDS or ReplicaDB. These might work, but aren't hosted so we may or may not save time over building a custom solution after all the time and effort of configuration and debugging.
Am I missing an obvious solution?
Congratulations, you solved your problem. It will be better that if you could share us more detail about your simple replication system.
I have a huge (500+ table) Azure SQL database (SQL Server). I need to create a clone of this database on Azure, and sync the two databases once daily. The clone is for reporting purposes.
What is the best way to implement the sync, outside of Azure Data Sync? We've experimented with Azure Data Sync, and it's proven unreliable due to the large size of the database.
I've looked into transactional replication, but I cannot find any documentation that states that it is supported from an Azure database to another Azure database. Geo-replication may be another option, though I'm not sure it is a good fit for this use case.
To my knowledge, your best option is Azure Data Factory. It has a very easy to use Wizard as explained here. You can create yourself your copy activities as explained here and here.
You can schedule ADF execution as explained here too.
SQL Data Sync is in Preview and for that reason not recommended for Production environment.
Geo-Replication cannot scheduled for synchronization.
Another option is to use Cross-Database queries as mentioned here, and schedule execution of synchronization procedure created by yourself using elastic jobs or Azure Automation.
Hope this helps.
I have a SQL 2012 database that I want to sync/replicate up into Azure.
Because we intend to use this in production we cannot use the Azure Sync tool that is currently in preview.
This needs to be near real time, and only one directional. (Only ever pushing data to Azure like a cache)
What are some recommended tools for doing this?
In my opinion:
SSDT - Sql Server Data Tools
http://thomaslarock.com/2013/01/sql-server-data-tools-why-must-life-be-so-hard/
http://msdn.microsoft.com/en-us/library/dn266028(v=vs.103).aspx
http://msdn.microsoft.com/en-us/data/tools.aspx
if you're using Azure SQL Database, you can have a look at SSIS, Sync Framework, etc...
the biggest challenge you'll have is not the sync, but the near real time requirement that you have.
The best approach to this problem is to use the Sync Framework, create an agent that sits on a Worker Role in your Azure environment, and a client agent that is on your network.
Here is an end-to-end sample: http://code.msdn.microsoft.com/Windows-Azure-Sync-Service-60293622
Here is another option, this one uses a two-tier approach that is simpler and may be more suited to your push-only scenario: http://blogs.msdn.com/b/zkap/archive/2012/04/12/synchronize-sql-server-and-sql-azure-using-sync-framework.aspx
I'm using SQL Azure on a project and it works great. The problem is that the usual backup features do not exist. I have exported the database a couple of times using SQLAzureMW ( http://sqlazuremw.codeplex.com/ ) but this tool is now choking trying to download the database data with bcp. In any case, it's not as nice a solution as SQL Server backups.
Is anyone aware of a commercial or open source tool, or other technique, for making reliable backups of SQL Azure databases? This is really a showstopper.
Starting with update 4, SQL Azure now supports database copies. You can make a copy of your database, kept in Azure, and use that to retrieve data in the event of an accidental deletion or schema bugaboo:
http://msdn.microsoft.com/en-us/library/ff951624.aspx
It's still up to you to get that database off Azure and onto your own local SQL Server, though, but at least you've got a mechanism for making a point-in-time copy.
Microsoft takes care of the backups for you. There is no reason to back up SQL Azure databases yourself.
Yes, we had the same problem and couldn't find any good/simple solutions, so we cobbled together a solution using Red Gate: http://mooneyblog.mmdbsolutions.com/index.php/2011/01/11/simple-database-backups-with-sql-azure
SQL Azure will support PIT (Point in time) backup/restore (mainly restore) later this year (2011), CTP in summer. There is some (little) preliminary info here info here.