I am somewhat surprised (still after all these SQL Server Installed Instance (Windows VM Azure) that pushing data, on a nightly schedule, to an SQL Azure database is not straight forward. I see some articles and direction to 'migrate' schemas and data, but what about a nightly job to push from my SQL Server instance to individual client SQL Azure data stores?
Should I start with SSIS? Azure data factory? Python libraries? Why isn't a connection between the two 'native'?
Again, all links and references so far have been for one time migration. I want the two in a data Eco-system with reliable flow.
John
We do this using SSIS running from the on prem side, because we already have a bunch of SSIS projects hosted on prem, and have yet to migrate anything into azure data factory. We are using SQL authentication to make the connection to the SQL Azure database.
Related
I'm in the process of testing/playing about with Azure. We have moved an on-premise SQL Server database to an Azure Managed Instance, however, this contained a linked server to a Progress database via an ODBC DSN connection.
Of course this no longer works. Does anyone know if this is possible or will I have to import the various Progress tables into Azure via a different way?
Thanks.
You can't install drivers on a Managed Instance, so it only supports linked servers that use the SQL Server drivers.
As a workaround you can "double-hop" through a Linked Server on a small Azure VM running Windows and SQL Server Express Edition. See eg [Using Heterogeneous Linked Servers from Azure SQL MI][1]
Which is enough of a pain that you'd want to load the data into MI using some external tool, like SSIS or Azure Data Factory.
[1]: https://techcommunity.microsoft.com/t5/modernization-best-practices-and/using-heterogeneous-linked-servers-from-azure-sql-mi/ba-p/3074530
Is there a proper way to move a SQL Server 2016 sharded database from self-hosted to Azure Database?
I have a SQL Server 2016 sharded database that is part of the platform (Sitecore 10) I'm working with. It has a Shard Map Manager database and two Shard databases. I want to migrate the databases from self-hosted to Azure Database.
When the database is in a SQL Server Instance, it has one login mapped to the Shard Map Manger user and both Shard users. Everything works great. Since Azure Database does not use logins the same way and the databases are partially contained, I created separate users in each database in Azure with the same name and password.
Next, I migrated the schema with Azure Data Migration Services and then the data. There are some stored procedures that Sitecore provides to setup permissions and I ran those.
Finally, I updated the ServerName and DatabaseName fields in the [__ShardManagement].[ShardsGlobal] table for the ShardMapManager DB and the [__ShardManagement].[ShardsLocal] table in both Shard databases to match the new server and database names.
When I updated the app to use the new database, it spewed errors into the log, ultimately tracking to a failed login for the user. There are 13 other databases for the app which were also migrated and work fine, only the sharded database does not work in the app. I ran a PowerShell script that tests the connection string for all of the databases. All of the connection strings were successful. Backing out only the connection strings for the sharded database fixes the errors proving that the connection to the sharded databases is the problem.
What did I do wrong and what should I do to fix it?
If the requirement is only to migrate the databases from local to Azure, the best possible way is to use SQL Server Management Studio (SSMS).
The advantage is that you can mention the new database name on Azure while migrating itself and hence no need to change later. But just make sure same named database shouldn't available already in Azure SQL Server.
You can follow the migration steps from my this answer.
Also, my suggestion is to go through the official documents Assessment rules for SQL Server to Azure SQL Database migration, Troubleshoot connecting to the SQL Server Database Engine. I'm sure you will find the useful insights there which could help to make migrated database work properly with the apps.
I'm trying to find out how I can use a Microsoft Azure Machine Learning notebook to connect to SQL Server and Azure SQL databases.
I am aware of how to connect to SQL Server databases with regular Jupyter files with the use of ODBC connections. But, it looks like I have to try something different when using Azure Machine Learning notebooks.
Could someone describe the best approach to accomplish this?
Note: I am referring to the new Microsoft Azure Machine Learning service, which is currently in preview mode as of May 2020.
great question -- the answer depends on whether or not your data sources are in Azure.
data in Azure
for Azure based storage (blob, data lake, Azure SQL, Azure Databricks) you're in luck with Azure ML Datasets, and abstraction on top of azureml-dataprep, a component package of azureml-sdk. IMHO, Azure ML Datasets are slick, TabularDatasets in particular with their to_pandas_dataframe() and .to_spark_dataframe() methods.
Check out the following articles for guidance on how to:
How to connect to data and register as a Dataset
How to access data during training
Follow the recommendations in this tutorial recommendation would be to make a TabularDataset FileDataset
data not in Azure
For on-premise or IaaS SQL servers, you've got two options that I'm aware of:
Put your SQL server inside the same network at the Azure ML service and ComputeTarget and access the server directly with pyodbc library.
Use ADF to move the SQL server data to Azure Storage, (you'll need an ADF integration runtime on the SQL server)
The question: Is it possible to point a view in database A running on Azure SQL Database service to tables/views on a SQL server running in a VM? I've tried external tables but come up short.
Scenario:
Two applications that is exchanging data, from two different vendors.
We've got three databases in total. One for each application and an integration database with views that both applications use either directly or through views.
The issue now is that we want to migrate to Azure and would prefer to use the Azure SQL Database service as much as possible, but one of the applications is not ready for it and therefor it's database has to be hosted on either a managed instance or in a VM.
The issue now is that there's view's in database A (running on Azure SQL Database service) that points to views and tables in the integration database, which again references tables in database B (running on SQL Server on the VM).
The short answer is "no". You can use external tables to query other SQL Azure databases, but there is no exact analogue for linked servers in Azure SQL Database. You can use SQL Azure Managed Instance (which supports SQL-SQL linked servers but not arbitrary linked servers).
There is a workaround, however. You can run SQL Server in an Azure VM and have it point to SQL Azure as a target as well as the other sources you want to connect. Then you can push data to/from Azure SQL DB using the SQL Server in a VM. You don't have the same management overhead in this approach since you don't really need to host data in the SQL Server if you don't want to do so. Note that this will be slower than doing direct connections to SQL Azure, but you can try to do this for a period of time if it would help you during a migration.
This question already has answers here:
How do I copy SQL Azure database to my local development server?
(24 answers)
Closed 9 years ago.
Okay this is not about migrating existing sql server database to azure database. Its the other way round.
I want to stop using azure sql database, and moved to full version of SQL Server. How do I migrate existing database in AZURE Sql to a new SQL Server 2008 database?
If you don't want to lose any data, you have to stop using it for the moment of transfer. I can suggest you couple of ways:
First way is to use the SQL Azure Migration Wizard. Yes. Although it is originally designed to migrate your local SQL DB to SQL Azure, it can be used the other way around. I've done in numerous times. Just first point the migration wizard to your SQL Azure instance, then target the local server.
Second way is to use the Azure SQL Database backup. Create a bacpac archive either using the portal or via SSMS. Then import this BacPac package into your local SQL Server instance.
If you don't want downtime and want full data synchronization I will suggest doing the following:
Move schema and data initially using any of the above methods (for live systems I highly recommend to first a new database AS COPY and then create the BacPac from it, or use the Migration wizard to transfer it).
Enable the SQL Data Sync to begin synchronizing local data with on-going live data.
Wait while local is fully in sync.
Switch the connection strings in your app.
Cut down SQL Azure.
There are a lot of third party tools that will allow you to synchronize your SQL Azure database with full SQL Server database. Just use any of these to synchronize structures and then data.
You can try tools from Red Gate, ApexSQL, Idera or any other… Even though these tools are not really cheap you can still use them in trial mode (its fully functional in most cases) to get the job done.