I need to copy several hundred tables from an on-premise SQL Server to an Azure SQL Server using ADF. I don't have access to the DB or the network it's on, but I was able to get the on-prem data gateway installed, given an AD account with sufficient DB privileges, and then use the "Copy Data (Preview)" to copy all tables to blob storage.
My problem is that I don't have access to the DB's schema, so I can't easily provision the Azure SQL Server with the necessary tables/columns since there are several hundred tables & performing manually would be extremely time consuming. I found that copying to an Azure Data Warehouse has a "Auto table creation" feature & I am able to copy from on-prem SQL Server directly to Azure DW without defining a schema at the destination, but this isn't supported on Azure SQL Server.
Is there a way to obtain the same script/method that provisions the Azure DW schema & use it for Azure SQL Server? Is there any other way to obtain the source DB's schema via the on-prem data gateway?
given that you were able to run the Copy Data tool to extract data out of on-premises SQL Server, you must have credentials to access the database. Can you run SSMS (SQL Server Management Studio) on the on-prem data gateway and examine/extract the schema?
"Auto table creation" feature is currently only available for Azure Data Warehouse. Supporting this feature when loading into Azure SQL Server is on our backlog but we don't have a committed timeline for this yet.
Shirley Wang
Couldn't you use DMG to run a query against the database to generate the schema for your tables, assuming your AD account has read access to the metadata?
So with your pipeline, instead of it selecting * from each table, have it run a query to exract the schema, some examples here: How can I show the table structure in SQL Server query?
You would then output that to a blob.
Related
Is there a proper way to move a SQL Server 2016 sharded database from self-hosted to Azure Database?
I have a SQL Server 2016 sharded database that is part of the platform (Sitecore 10) I'm working with. It has a Shard Map Manager database and two Shard databases. I want to migrate the databases from self-hosted to Azure Database.
When the database is in a SQL Server Instance, it has one login mapped to the Shard Map Manger user and both Shard users. Everything works great. Since Azure Database does not use logins the same way and the databases are partially contained, I created separate users in each database in Azure with the same name and password.
Next, I migrated the schema with Azure Data Migration Services and then the data. There are some stored procedures that Sitecore provides to setup permissions and I ran those.
Finally, I updated the ServerName and DatabaseName fields in the [__ShardManagement].[ShardsGlobal] table for the ShardMapManager DB and the [__ShardManagement].[ShardsLocal] table in both Shard databases to match the new server and database names.
When I updated the app to use the new database, it spewed errors into the log, ultimately tracking to a failed login for the user. There are 13 other databases for the app which were also migrated and work fine, only the sharded database does not work in the app. I ran a PowerShell script that tests the connection string for all of the databases. All of the connection strings were successful. Backing out only the connection strings for the sharded database fixes the errors proving that the connection to the sharded databases is the problem.
What did I do wrong and what should I do to fix it?
If the requirement is only to migrate the databases from local to Azure, the best possible way is to use SQL Server Management Studio (SSMS).
The advantage is that you can mention the new database name on Azure while migrating itself and hence no need to change later. But just make sure same named database shouldn't available already in Azure SQL Server.
You can follow the migration steps from my this answer.
Also, my suggestion is to go through the official documents Assessment rules for SQL Server to Azure SQL Database migration, Troubleshoot connecting to the SQL Server Database Engine. I'm sure you will find the useful insights there which could help to make migrated database work properly with the apps.
When a database is fully contained, it is having all the objects within the database boundary. It manages connection also at database level. Contained Database
I have few questions:
If I host a contained database in Azure SQL Single DB then what is
the difference between a Azure SQL single db and a contained database
as Azure SQL single db ?
Why do we have a separate offering as Azure SQL Single Db, when
contained database also is something similar to it ?
Does making a database as contained database will help in easier
migration to Azure SQL single DB ?
In Azure speak, "single" is a deployment option that differentiates Azure SQL Database from Managed and Elastic Pools. There's really no difference from a containment perspective because Single and Azure SQL Database are one and the same.
Azure SQL Database inherently provides containment features like database-level user authentication. With on-prem SQL Server databases, one must opt-in for with CONTAINMENT=PARTIAL
to permit database-level authentication.
Does making a database as contained database will help in easier migration to Azure SQL single DB ?
CONTAINMENT=PARTIAL permits database-level authentication in on-prem versions, facilitating migration of database security principals. As long as user database entities stay within the database boundary and one doesn't need features unavailable in contained databases (e.g. CDC), migration is typically easy.
A consideration, though, is that partially contained databases implicitly use catalog collation Latin1_General_100_CI_AS_KS_WS_SC whereas Azure SQL Database catalog collation must be either the chosen DATABASE_DEFAULT collation or SQL_Latin1_General_CP1_CI_AS. This is generally an issue only when case-sensitivity of object/variables names is desired.
Existing uncontained references can be identified by querying sys.dm_db_uncontained_entities:
SELECT * FROM sys.dm_db_uncontained_entities;
The above query will also identify dynamic SQL references that will need to be examined manually.
The question: Is it possible to point a view in database A running on Azure SQL Database service to tables/views on a SQL server running in a VM? I've tried external tables but come up short.
Scenario:
Two applications that is exchanging data, from two different vendors.
We've got three databases in total. One for each application and an integration database with views that both applications use either directly or through views.
The issue now is that we want to migrate to Azure and would prefer to use the Azure SQL Database service as much as possible, but one of the applications is not ready for it and therefor it's database has to be hosted on either a managed instance or in a VM.
The issue now is that there's view's in database A (running on Azure SQL Database service) that points to views and tables in the integration database, which again references tables in database B (running on SQL Server on the VM).
The short answer is "no". You can use external tables to query other SQL Azure databases, but there is no exact analogue for linked servers in Azure SQL Database. You can use SQL Azure Managed Instance (which supports SQL-SQL linked servers but not arbitrary linked servers).
There is a workaround, however. You can run SQL Server in an Azure VM and have it point to SQL Azure as a target as well as the other sources you want to connect. Then you can push data to/from Azure SQL DB using the SQL Server in a VM. You don't have the same management overhead in this approach since you don't really need to host data in the SQL Server if you don't want to do so. Note that this will be slower than doing direct connections to SQL Azure, but you can try to do this for a period of time if it would help you during a migration.
I have DB in my SQL Server. I want to replicate the same DB with all the Views inside it into Azure and sync the same SQL Server DB with replicated DB on Azure on daily basis.
I have no idea about Azure.
You can migrate a SQL Server 2005 or later database to a single or pooled database in Azure SQL Database. For best performance during the import process, it’s better to drop indexed views and recreate them once finished. For more details, you may refer to this article: https://learn.microsoft.com/en-us/azure/sql-database/sql-database-cloud-migrate.
You can set up Azure SQL Data Sync by creating a hybrid sync group that contains both Azure SQL Database and SQL Server instances. For more details on how to achieve this, you may refer to this document: https://learn.microsoft.com/en-us/azure/sql-database/sql-database-get-started-sql-data-sync.
Note: Does SQL Data Sync fully create and provision tables?
Views and Stored Procedures are not created on the destination database.
You can use SQL Data Sync to sync the base tables of your views but SQL Data Sync cannot sync views. Sync data from views is quite complex and not always achievable and for that reason SQL Data Sync does not do it.
Use the step-by-step procedure described here to configure the synchronization. You need to make your SQL Server instance a "hub", your SQL Azure Database a "member" on the sync configuration. Configure the "Sync frequency" parameter/field to occur every X days or X hours.
Goal: I need to create a msdb on my Systems Database.
Why: I am trying to move an application database to Azure so multiple computers can connect to it. I created a bacpac file and was able to upload my database, but when I try to connect to the Azure database from the application it won't let me because "msdb..backupset is not supported on this version of SQL Server."
Problem: Azure's SQL Server only has master in its Systems Database. Why does it only have master? How can I get msdb and the others there as well? Do I need a VM?
Problem: Azure's SQL Server only has master in its Systems Database. Why does it only have master? How can I get msdb and the others there as well? Do I need a VM?
yes you need a VM or SQLAZURE managed instance*(which is same as on premises sql server with SQLAgent,clr,...,but with high available capacity of SQLAZure)
The reason why you don't have system databases is ..SQLAZURE is database as service and you just pay for the database you used
*Currently in preview,expected to be GA by the end if this year