I have a huge (500+ table) Azure SQL database (SQL Server). I need to create a clone of this database on Azure, and sync the two databases once daily. The clone is for reporting purposes.
What is the best way to implement the sync, outside of Azure Data Sync? We've experimented with Azure Data Sync, and it's proven unreliable due to the large size of the database.
I've looked into transactional replication, but I cannot find any documentation that states that it is supported from an Azure database to another Azure database. Geo-replication may be another option, though I'm not sure it is a good fit for this use case.
To my knowledge, your best option is Azure Data Factory. It has a very easy to use Wizard as explained here. You can create yourself your copy activities as explained here and here.
You can schedule ADF execution as explained here too.
SQL Data Sync is in Preview and for that reason not recommended for Production environment.
Geo-Replication cannot scheduled for synchronization.
Another option is to use Cross-Database queries as mentioned here, and schedule execution of synchronization procedure created by yourself using elastic jobs or Azure Automation.
Hope this helps.
Related
I want to move all data from one Azure SQL Server to different Azure SQL Server which more than 90 days old, and after moving need to delete moved data from first Azure SQL Server.
I want to run these steps on daily basis.
I am new to Azure and able to do same with Azure Data Factory. Can you please suggest any other best suited approach?
You are already using the best approach.
Azure Data Factory is an easy to use when it comes to extract and copy the data between the services. It also provide scheduling the triggers, i.e., triggering the copy pipeline after specific interval of time or any event. Refer Create a trigger that runs a pipeline on a schedule.
If the volume of data is large, you can re-configure the Integration Runtime (IR) resources (Compute type and Core count) to overcome the performance issue, if required. Refer below image.
I need to regularly (but incrementally) sync (one way) the contents of a set of SQL Server Azure tables to a PostgreSQL Azure instance.
Here are some of the avenues I've considered:
Linked server from SQL Server. No go. Apparently Azure doesn't support linked servers.
Foreign Data Wrapper from PostgreSQL. No go. PostgreSQL on Azure only supports the postgres_fdw, not the needed tds_fdw.
Azure Data Factory. No go. The data copy process doesn't work incrementally, and the sink pipeline component doesn't support PosgreSQL.
Commercial replication solutions. Too expensive for a startup and most aren't hosted.
SymmetricDS or ReplicaDB. These might work, but aren't hosted so we may or may not save time over building a custom solution after all the time and effort of configuration and debugging.
Am I missing an obvious solution?
Congratulations, you solved your problem. It will be better that if you could share us more detail about your simple replication system.
I'm relatively new to Azure and am having trouble finding what options are out there for connecting to an existing SQL database to push data into it.
The situation is that we have an external client who needs to connect to our Azure SQL database to push data into it, on an on-going basis. We can't give them permission to get into our database, so we're looking at what we can do allow data in. At this point the best option seems to be to create a web service deployed in Azure that will validate the data and then push it into our database.
The question I have is, are there other options to do this in an easier way? Are there Azure services or processes that can be set up to automatically process a file and pull the data into a database? Any other go-between options when each side has their own database and for security reasons can't just open up access to it?
Azure Data Factory works great for basic ETL. If neither party can grant direct access, you can use an intermediate repository like Blob Storage to drop csv/xml/json files for ingestion. If they'll grant you access to pull, you can setup a linked service that more or less functions the same as a linked server in MSSQL. As of the last release ADF now supports Azure hosted SSIS packages too.
I would do this via SSIS using SQL Studio Managemenet Studio (if it's a one time operation). If you plan to do this repeatedly, you could schedule the SSIS job to execute on schedule. SSIS will do bulk inserts using small batches so you shouldn't have transaction log issues and it should be efficient (because of bulk inserting). Before you do this insert though, you will probably want to consider your performance tier so you don't get major throttling by Azure and possible timeouts.
I have a database deployed on Azure SQL. Due to various requirements, my company would like to make a continuous mirror of this database to an on-premise SQL Server database. We have explored several ways to implement this, so far to no avail.
The way recommended online, using the Azure Sync tool, didn't work for us -- and seeing as this tool is currently in CTP, it is not currently supported by Microsoft, so we want to implement this ourselves.
The requirements for this mirroring solution aren't extremely high - we only want to make backups every half an hour or so, so continuous synchronization isn't an issue. What would be the best way to approach this without resorting to commercial solutions that seem to be an overkill for us?
Hummm... it depends what you call an overkill. One way to do this is to wrap up your own BCP commands out of SQL Azure, and BCP into a database on premises. But depending on your database complexity, this may very quickly get out of hands, unless you can identify the top 2 or 3 tables that you really want to export/import to minimize your work. In fact you might event want to consider using SSIS; you will need to manage your own retry logic regardless of the approach you take.
We ended up choosing a complete backup-restore from the Azure account to the on-premise VM, since implementing custom retry & incremental query generation sounded way too complicated for this case.
Azure SQL Database support Import Export Database Service including Automated Exports to Azure Storage account.
Using Automated Exports you can define the export interval per day and bacpac files will be created in your Azure storage account which can be imported into your on-premise server.
You can find more detail about Configuring Automated Exports
I have a SQL 2012 database that I want to sync/replicate up into Azure.
Because we intend to use this in production we cannot use the Azure Sync tool that is currently in preview.
This needs to be near real time, and only one directional. (Only ever pushing data to Azure like a cache)
What are some recommended tools for doing this?
In my opinion:
SSDT - Sql Server Data Tools
http://thomaslarock.com/2013/01/sql-server-data-tools-why-must-life-be-so-hard/
http://msdn.microsoft.com/en-us/library/dn266028(v=vs.103).aspx
http://msdn.microsoft.com/en-us/data/tools.aspx
if you're using Azure SQL Database, you can have a look at SSIS, Sync Framework, etc...
the biggest challenge you'll have is not the sync, but the near real time requirement that you have.
The best approach to this problem is to use the Sync Framework, create an agent that sits on a Worker Role in your Azure environment, and a client agent that is on your network.
Here is an end-to-end sample: http://code.msdn.microsoft.com/Windows-Azure-Sync-Service-60293622
Here is another option, this one uses a two-tier approach that is simpler and may be more suited to your push-only scenario: http://blogs.msdn.com/b/zkap/archive/2012/04/12/synchronize-sql-server-and-sql-azure-using-sync-framework.aspx