The client has on-premise SQL Server on his side. We want to replicate his database to our PostgreSQL Azure. The replication we want can have maximum 1min delay. ADF is not the option - copying is too frequent.
What tools can we use? We thought about Azure Sync or SymetricDS. Eventually Would SQL Server -> SQL Server be easier to execute? Any suggestions even beyond Azure are much appreciated.
As #Martin Cairney said on-prem SQL Server -> Azure PostgreSQL doesn't exist in Azure DataSync, so you would need to make use of offical or a 3rd party replication tool. Azure DataSync only support Azure SQL as the Hub Database.
Offical tool, such as Azure data factory. You can set a trigger condition to reduce the frequency of copy activities.
3rd party tool, sunch as skyvia. You said SymmetricDS maybe not support Azure PostgreSQL, according to its official doc.
Related
I am somewhat surprised (still after all these SQL Server Installed Instance (Windows VM Azure) that pushing data, on a nightly schedule, to an SQL Azure database is not straight forward. I see some articles and direction to 'migrate' schemas and data, but what about a nightly job to push from my SQL Server instance to individual client SQL Azure data stores?
Should I start with SSIS? Azure data factory? Python libraries? Why isn't a connection between the two 'native'?
Again, all links and references so far have been for one time migration. I want the two in a data Eco-system with reliable flow.
John
We do this using SSIS running from the on prem side, because we already have a bunch of SSIS projects hosted on prem, and have yet to migrate anything into azure data factory. We are using SQL authentication to make the connection to the SQL Azure database.
I'm trying to find out how I can use a Microsoft Azure Machine Learning notebook to connect to SQL Server and Azure SQL databases.
I am aware of how to connect to SQL Server databases with regular Jupyter files with the use of ODBC connections. But, it looks like I have to try something different when using Azure Machine Learning notebooks.
Could someone describe the best approach to accomplish this?
Note: I am referring to the new Microsoft Azure Machine Learning service, which is currently in preview mode as of May 2020.
great question -- the answer depends on whether or not your data sources are in Azure.
data in Azure
for Azure based storage (blob, data lake, Azure SQL, Azure Databricks) you're in luck with Azure ML Datasets, and abstraction on top of azureml-dataprep, a component package of azureml-sdk. IMHO, Azure ML Datasets are slick, TabularDatasets in particular with their to_pandas_dataframe() and .to_spark_dataframe() methods.
Check out the following articles for guidance on how to:
How to connect to data and register as a Dataset
How to access data during training
Follow the recommendations in this tutorial recommendation would be to make a TabularDataset FileDataset
data not in Azure
For on-premise or IaaS SQL servers, you've got two options that I'm aware of:
Put your SQL server inside the same network at the Azure ML service and ComputeTarget and access the server directly with pyodbc library.
Use ADF to move the SQL server data to Azure Storage, (you'll need an ADF integration runtime on the SQL server)
The question: Is it possible to point a view in database A running on Azure SQL Database service to tables/views on a SQL server running in a VM? I've tried external tables but come up short.
Scenario:
Two applications that is exchanging data, from two different vendors.
We've got three databases in total. One for each application and an integration database with views that both applications use either directly or through views.
The issue now is that we want to migrate to Azure and would prefer to use the Azure SQL Database service as much as possible, but one of the applications is not ready for it and therefor it's database has to be hosted on either a managed instance or in a VM.
The issue now is that there's view's in database A (running on Azure SQL Database service) that points to views and tables in the integration database, which again references tables in database B (running on SQL Server on the VM).
The short answer is "no". You can use external tables to query other SQL Azure databases, but there is no exact analogue for linked servers in Azure SQL Database. You can use SQL Azure Managed Instance (which supports SQL-SQL linked servers but not arbitrary linked servers).
There is a workaround, however. You can run SQL Server in an Azure VM and have it point to SQL Azure as a target as well as the other sources you want to connect. Then you can push data to/from Azure SQL DB using the SQL Server in a VM. You don't have the same management overhead in this approach since you don't really need to host data in the SQL Server if you don't want to do so. Note that this will be slower than doing direct connections to SQL Azure, but you can try to do this for a period of time if it would help you during a migration.
I'd like to make a direct comparison of a physical server to a serverless Azure SQL environment by using an hour snapshot of all SQL Server activity from some existing infrastructure.
Is it possible to use the SQL Server Profiler to record trace data and replay this on an Azure SQL instance? I am not attempting to tweak or performance tune the existing system, but want to compare how an Azure server at various tiers will perform that workload.
Another option might be to use the SQL Server Distributed Replay functionality if this is an option in Azure.
SQL Server profiler and Distributed Replay are not available.
https://feedback.azure.com/forums/217321-sql-database/suggestions/431943-profiler-for-sql-azure
You may want to try SQL Workload Profiler.
https://cbailiss.wordpress.com/sql-workload-profiler/
SQL Azure Managed Instances will soon have SQL Profiler available:
https://www.youtube.com/watch?v=0uT46lpjeQE&feature=youtu.be&t=1415
your entire question boils down to how can i choose the correct tier in sql azure for my on premises instances..
As alberto points out,there is no way to handle this now directly,for now you can use below workaround
go to http://dtucalculator.azurewebsites.net/ and download powershell script and schedule it for some duration and upload the generated log files to azure website.Once done, you will be provided with chart like below ..
References:
https://www.simple-talk.com/cloud/platform-as-a-service/azure-sql-database-how-to-choose-the-right-service-tier/
Reading some conflicting answers from Google searches, not sure if the answer is Yes, No, or maybe.
I thought it was pretty clear when reading this:
Amazon RDS currently does not support the following SQL Server features:
The ability to run Reporting, Analysis, Integration, or Master Data Services on the same server as the DB instance. If you need to do this, we recommend that you either install SQL Server on an EC2 instance or use an on-premise SQL Server instance to act as the Reporting, Analysis, Integration, or Master Data Services server.
Amazon now supports SSIS on RDS as of May 2020.
More info here