Is there a proper way to move a SQL Server 2016 sharded database from self-hosted to Azure Database?
I have a SQL Server 2016 sharded database that is part of the platform (Sitecore 10) I'm working with. It has a Shard Map Manager database and two Shard databases. I want to migrate the databases from self-hosted to Azure Database.
When the database is in a SQL Server Instance, it has one login mapped to the Shard Map Manger user and both Shard users. Everything works great. Since Azure Database does not use logins the same way and the databases are partially contained, I created separate users in each database in Azure with the same name and password.
Next, I migrated the schema with Azure Data Migration Services and then the data. There are some stored procedures that Sitecore provides to setup permissions and I ran those.
Finally, I updated the ServerName and DatabaseName fields in the [__ShardManagement].[ShardsGlobal] table for the ShardMapManager DB and the [__ShardManagement].[ShardsLocal] table in both Shard databases to match the new server and database names.
When I updated the app to use the new database, it spewed errors into the log, ultimately tracking to a failed login for the user. There are 13 other databases for the app which were also migrated and work fine, only the sharded database does not work in the app. I ran a PowerShell script that tests the connection string for all of the databases. All of the connection strings were successful. Backing out only the connection strings for the sharded database fixes the errors proving that the connection to the sharded databases is the problem.
What did I do wrong and what should I do to fix it?
If the requirement is only to migrate the databases from local to Azure, the best possible way is to use SQL Server Management Studio (SSMS).
The advantage is that you can mention the new database name on Azure while migrating itself and hence no need to change later. But just make sure same named database shouldn't available already in Azure SQL Server.
You can follow the migration steps from my this answer.
Also, my suggestion is to go through the official documents Assessment rules for SQL Server to Azure SQL Database migration, Troubleshoot connecting to the SQL Server Database Engine. I'm sure you will find the useful insights there which could help to make migrated database work properly with the apps.
in my development environment we support the application both on MSSQL Server as well as Oracle. The database schema of both of these RDBMS are same.
while development we found that the developer made a mistake and forgot to change the oracle database for the last 1 yr. therfore the oracle script is quite behind in term of schema from SQL Server schema script.
now the question is how i can compare the two RDBMS systems to find the difference and make the oracle script updated
If there are no track log from which it's possible to find and reproduce all changes applied to SQL Server since first detected inconsistency with Oracle version, or that changes was applied, but only partially, you really need to compare objects presented in both databases.
In this case setup a link between databases on any side and use system dictionary views to compare table structures and other objects to find differences and, possible, to generate script for Oracle scheme rollup.
If you want to act from MS SQL Server side:
Install and configure Oracle Instant Client
Install Oracle ODAC
Follow Microsoft recomendations (64-bit version)
Connect as any user with dba role (or use same Oracle schema where object resides) to Oracle from MS SQL database
If you want to act from Oracle Server side:
Install and configure Oracle Database Gateway for SQL Server.
Create database link to MS SQL Server.
After successful configuration you may join Information schema views on SQL Server side with Data dictionary views on Oracle side to find differences.
Of course there are many troubles at this way like different data types, but it gives a chance to automate at least part of work.
Does anyone know how I can copy a SQL Azure database to my development machine? I'd like to stop paying to have a development database in the cloud, but it's the best way to get production data. I copy my production database to a new development database but I'd like to have that same database local.
Any suggestions?
There are multiple ways to do this:
Using SSIS (SQL Server Integration Services). It only imports data in your table. Column properties, constraints, keys, indices, stored procedures, triggers, security settings, users, logons, etc. are not transferred. However it is very simple process and can be done simply by going through wizard in SQL Server Management Studio.
Using combination of SSIS and DB creation scripts. This will get you data and all missing metadata that is not transferred by SSIS. This is also very simple. First transfer data using SSIS (see instructions below), then create DB Create script from SQL Azure database, and re-play it on your local database.
Finally, you can use Import/Export service in SQL Azure. This transfers data (with a schema objects) to Azure Blob Storage as a BACPAC. You will need an Azure Storage account and do this in Azure web portal. It is as simple as pressing an "Export" button in the Azure web portal when you select the database you want to export. The downside is that it is only manual procedure, I don't know a way to automate this through tools or scripts -- at least the first part that requires a click on the web page.
Manual procedure for method #1 (using SSIS) is the following:
In Sql Server Management Studio (SSMS) create new empty database on your local SQL instance.
Choose Import Data from context menu (right click the database -> Tasks -> Import data...)
Type in connection parameters for the source (SQL Azure). Select ".Net Framework Data Provider for SqlServer" as a provider.
Choose existing empty local database as destination.
Follow the wizard -- you will be able to select tables data you want to copy. You can choose to skip any of the tables you don't need. E.g. if you keep application logs in database, you probably don't need it in your backup.
You can automate it by creating SSIS package and re-executing it any time you like to re-import the data. Note that you can only import using SSIS to a clean DB, you cannot do incremental updates to your local database once you already done it once.
Method #2 (SSID data plus schema objects) is very simple. First go though a steps described above, then create DB Creation script (righ click on database in SSMS, choose Generate Scripts -> Database Create). Then re-play this script on your local database.
Method #3 is described in the Blog here: http://dacguy.wordpress.com/2012/01/24/sql-azure-importexport-service-has-hit-production/. There is a video clip with the process of transferring DB contents to Azure Blob storage as BACPAC. After that you can copy the file locally and import it to your SQL instance. Process of importing BACPAC to Data-Tier application is described here: http://msdn.microsoft.com/en-us/library/hh710052.aspx.
Copy Azure database data to local database:
Now you can use the SQL Server Management Studio to do this as below:
Connect to the SQL Azure database.
Right click the database in Object Explorer.
Choose the option "Tasks" / "Deploy Database to SQL Azure".
In the step named "Deployment Settings", connect local SQL Server and create New database.
"Next" / "Next" / "Finish"
In SQL Server 2016 Management Studio, the process for getting an azure database to your local machine has been streamlined.
Right click on the database you want to import, click Tasks > Export data-tier application, and export your database to a local .dacpac file.
In your local target SQL server instance, you can right click Databases > Import data-tier application, and once it's local, you can do things like backup and restore the database.
I just wanted to add a simplified version of dumbledad's answer, since it is the correct one.
Export the Azure SQL Database to a BACPAC file on blob storage.
From inside SQL Management studio, right-click your database, click "import data-tier application".
You'll be prompted to enter the information to get to the BACPAC file on your Azure blob storage.
Hit next a couple times and... Done!
I think it is a lot easier now.
Launch SQL Management Studio
Right Click on "Databases" and select "Import Data-tier application..."
The wizard will take you through the process of connecting to your Azure account, creating a BACPAC file and creating your database.
Additionally, I use Sql Backup and FTP (https://sqlbackupandftp.com/) to do daily backups to a secure FTP server. I simply pull a recent BACPAC file from there and it import it in the same dialog, which is faster and easier to create a local database.
You can also check out SQL Azure Data Sync in the Windows Azure Management Portal. It allows you to retrieve and restore an entire database, including schema and data between SQL Azure and SQL Server.
Using msdeploy.exe
Caveat: msdeploy.exe fails to create the destination database on its own, so you need to create it manually first.
Copy the connection string on the database properties page. Adjust it so that it contains a correct password.
Get the connection string for the destination DB.
Run msdeploy.exe like this:
"c:\Program Files\IIS\Microsoft Web Deploy V3\msdeploy.exe" -verb:sync -dest:dbDacFx="destination_DB_connection_string",dropDestinationDatabase=true -source:dbDacFx="azure_DB_connection_string",includeData=true -verbose
Using SqlPackage.exe
Export the azure DB to a bacpac package.
"c:\Program Files (x86)\Microsoft SQL Server\110\DAC\bin\SqlPackage.exe" /a:Export /ssn:"azure_db_server" /sdn:"azure_db_name" /su:"user_name" /sp:"password" /tf:"file.bacpac"
Import the package to a local DB.
"c:\Program Files (x86)\Microsoft SQL Server\110\DAC\bin\SqlPackage.exe" /a:Import /SourceFile:"file.bacpac" /TargetServerName:".\SQLEXPRESS" /TargetDatabaseName:CopyOfAzureDb
It's pretty easy. This worked for me...in terms of getting an Azure SQL database down onto your local machine...:
Open your SQL Management Studio and connect to your Azure SQL Server.
Select the database you would like to get down onto your local machine, and right-click...select "Generate Scripts". Follow the prompts...
BUT, be careful in that if you ALSO want the DATA, as well as the scripts, be sure to check the Advanced Options before beginning the generating...scroll down to "Types of data to script", and make sure you have "Schema and data"...or whatever is appropriate for you.
It will give you a nice SQL script file which can then be run on your local machine and it will create the database as well as populate it with all the data.
Bare in mind that in my case, I have no FK or other constraints...also, it wasn't a lot of data.
I don't recommend this as a backup mechanism in general...
In SQL Server Management Studio
Right click on the database you want to import, click Tasks > Export data-tier application, and export your database to a local .dacpac file.
In your local target SQL server instance, you can right click Databases > Import data-tier application, and once it's local, you can do things like backup and restore the database.
I couldn't get the SSIS import / export to work as I got the error 'Failure inserting into the read-only column "id"'. Nor could I get http://sqlazuremw.codeplex.com/ to work, and the links above to SQL Azure Data Sync didn't work for me.
But I found an excellent blog post about BACPAC files: http://dacguy.wordpress.com/2012/01/24/sql-azure-importexport-service-has-hit-production/
In the video in the post the blog post's author runs through six steps:
Make or go to a storage account in the Azure Management Portal.
You'll need the Blob URL and the Primary access key of the storage
account.
The blog post advises making a new container for the bacpac file and
suggests using the Azure Storage
Explorer for that. (N.B.
you'll need the Blob URL and the Primary access key of the storage
account to add it to the Azure Storage Explorer.)
In the Azure Management Portal select the database you want to
export and click 'Export' in the Import and Export section of the
ribbon.
The resulting dialogue requires your username and password for the
database, the blob URL, and the access key. Don't forget to include
the container in the blob URL and to include a filename (e.g.
https://testazurestorage.blob.core.windows.net/dbbackups/mytable.bacpac).
After you click Finish the database will be exported to the BACPAC
file. This can take a while. You may see a zero byte file show up
immediately if you check in the Azure Storage Explorer. This is the
Import / Export Service checking that it has write access to the
blob-store.
Once that is done you can use the Azure Storage Explorer to download
the BACPAC file and then in the SQL Server Management Studio
right-click your local server's database folder and choose Import
Data Tier Application that will start the wizard which reads in the
BACPAC file to produce the copy of your Azure database. The wizard
can also connect directly to the blob-store to obtain the BACPAC
file if you would rather not copy it locally first.
The last step may only be available in the SQL Server 2012 edition of the SQL Server Management Studio (that's the version I am running). I do not have earlier ones on this machine to check. In the blog post the author uses the command line tool DacImportExportCli.exe for the import which I believe is available at http://sqldacexamples.codeplex.com/releases
Regarding the " I couldn't get the SSIS import / export to work as I got the error 'Failure inserting into the read-only column "id"'.
This can be gotten around by specifying in the mapping screen that you do want to allow Identity elements to be inserted.
After that, everything worked fine using SQL Import/Export wizard to copy from Azure to local database.
I only had SQL Import/Export Wizard that comes with SQL Server 2008 R2 (worked fine), and Visual Studio 2012 Express to create local database.
The accepted answer is out of date. I found a better answer: Use Import Data-tier Application
More detailed information please see this article:
Restoring Azure SQL Database to a Local Server
You can try with the tool "SQL Database Migration Wizard".
This tool provide option to import and export data from azure sql.
Please check more details here.
https://sqlazuremw.codeplex.com/
I always use Import/Export Feature which seems to be the easiest one among all.
Step 1:
Get the backup from the azure instance as follows, Select the database → Right click → Tasks → Export Data Tier Application.
Step 2:
Give a specific name for the backup file and save it in your desired location
Step 3: That's it you have taken a backup of the database from sql instance to your local. Lets restore it to the local. Copy the backed up database to your C drive. Now open the PowerShell with administrator rights and navigate to C drive
Step 4: Lets download the powershell script to remove the master keyRemoveMasterKey.ps1 have the script on the same drive in this case its C.
Step 5 : Run the script as follows,
.\RemoveMasterKey.ps1 -bacpacPath "C:\identity.bacpac"
That's it, now you can restore it on MSSQL 2017 in your local environment.
Step 6: Connect to your local server, and click Databases → Import-Data-Tier-Application
Step 7 : Give a name for your database to restore.
Now you will see everything in green!
Read my blog with diagrams.
Using SSMS v18.9+, you can use the Deploy Database to Microsoft Azure SQL Database.
It's not really intuitive, but the wizard allow you to select a local db even if the name of the task is Deploy Database to Microsoft Azure SQL Database.
Connect to your Azure Database using SSMS
Right-click on the database, select Tasks > Deploy Database to Microsoft Azure SQL Database.
Select your local SQL server as the target connection.
Follow the additional steps.
You can use the new Azure Mobile Services to do a nightly backup export from SQL Azure to a .bacpac file hosted in Azure Storage. This solution is 100% cloud, doesn't require a 3rd party tool and doesn't require a local hosted SQL Server instance to download/copy/backup anything.
There's about 8 different steps, but they're all easy:
http://geekswithblogs.net/BenBarreth/archive/2013/04/15/how-to-create-a-nightly-backup-of-your-sql-azure.aspx
Looks like the functionality is missing from Management Studio in 2022. Here is how I do it using Azure Data Studio:
Install Azure Data Studio
Open Azure Data Studio
Install the extension called Admin Pack for SQL Server
Restart Azure Data Studio
Setup connections for both source database (Azure) and destination (local machine)
Execute statement in your local DB Engine on master database:
sp_configure 'contained database authentication', 1;
GO
RECONFIGURE;
GO
Right click on source database and select Data-tier Application Wizzard
Select Export to .bacpac file [Export bacpac] (4th, last option in my current version)
Finish the wizard
Right click on destination database and select Data-tier Application
Wizzard
Select Import from .bacpac [Import bacpac] (3rd option in my version)
Select the .bacpac file previously created and finish the wizard
After importing the logins and users are created, but their default schema isn’t set. If default shcema is required that needs to be handled manually. In that case:
Open an admin connection to the local copy and run:
USE [imported-db]; ALTER USER imported-user WITH DEFAULT_SCHEMA = whatever;
Download Optillect SQL Azure Backup - it has 15-day trial, so it will be enough to move your database :)
The trick for me was to start replicating PKs/FKs/constraints on empty DB, then temporarily disable constraints while importing data (see https://stackoverflow.com/a/161410).
More precisely:
Create empty target DB manually;
Right-click source DB > Tasks > Generate Scripts;
Run script file on empty target DB (now DB has correct PKs/FKs/constraints, but no data);
Disable all constraints;
Import data (Right-click target DB > Tasks > Import Data);
Re-enable constraints.
Hope this helps!
Now you can use the SQL Server Management Studio to do this.
Connect to the SQL Azure database.
Right click the database in Object Explorer.
Choose the option "Tasks" / "Deploy Database to SQL Azure".
In the step named "Deployment Settings", select your local database connection.
"Next" / "Next" / "Finish"...
Use the Import/Export service in SQL Azure to create a .bacpac file.
Then take a look at this method in another Stack Overflow article.
Azure SQL Database Bacpac Local Restore
If anyone has a problem to import a Bacpac of a DB that uses Azure SQL Sync, Sandrino Di Mattia developed a great simple application to solve this.
Export a Bacpac of your DB
Dowload Di Mattia's binary
With this console app repair the downloaded Bacpac
Lauch SSMS
Right Click on "Databases" and select "Import Data-tier Application"
Select the repaired Bacpac.
If anyone wants a free and effective option (and don't mind doing it manually) to backup database to Local then use schema and data comparison functionality built into the latest version Microsoft Visual Studio 2015 Community Edition (Free) or Professional / Premium / Ultimate edition. It works like a charm!
I have BizPark account with Azure and there is no way to backup database directly without paying. I found this option in VS works.
Answer is taken from https://stackoverflow.com/a/685073/6796187
Hi I'm using the SQLAzureMW tool for SQLAzure DB migration and management. Very useful one. It was downloaded from codeplex, but currently it's not available the codeplex will be going to shutdown, the same application tool is now available in GttHub. This below link is explain how to use this tool and also available the Application for download.
https://github.com/twright-msft/azure-content/blob/master/articles/sql-database/sql-database-migration-wizard.md
I need to copy an entire database from a SQL Server 2005 on my server over to my local SQL Express in order to run my application for a presentation. What is the fastest/easiest way to get this done?
EDIT: I have very limited access to my server so I don't think I can access the backup file that I could create so that is out.
If the database is not too big, you could use the Database Publishing Wizard.
This is a free tool from Microsoft which creates a complete SQL script of a database for you (not only the tables and stuff, but all data as well).
You can install the tool on your machine, connect to a remote server and let the tool create the script directly on your machine.
You can download the Database Publishing Wizard here.
Apparently the link above doesn't work anymore in 2019.
That's probably because in newer versions of SQL Server Management Studio, the functionality of the Database Publishing Wizard is included out-of-the-box, so there's no need to install it separately.
It's now called the Generate and Publish Scripts Wizard, but it does exactly the same.
You can right click the database -> Tasks -> Generate scripts. Here you can select one, multiple, or all objects. Then in the 'Set Scripting Options' step of the wizard, click Advanced. In here set the property 'Types of Data to script; to Schema and Data.
Having done these steps, make sure you publish to a file. Because only file can handle large amounts of data.
Now you should have all your objects, tables, and data scripted. Now start running the scripts and viola!
Back up the database on the server and then restore it locally in SQL Express.
EDIT: If this is part of your work, surely you can get someone in networks to get you a backup..?
If you can login to both servers (the Express and the 05 Server) using SQL Server Management Studio then you can do a DB Restore from one database to the other. No need for backup files at all.
You can use SSIS's database copy wizard, but it's not quick at all. Is there a DBA there that you can ask for the backup file? That will probably turn out to be the fastest way.
Depending which versions of SQL Server you are using, you might get some mileage out of the SQL Server Database Publishing Wizard from Microsoft. I've had mixed results with this tool in complex environments, but for most simple database scenarios it is a great tool.
An example of a scenario where I ran into difficulties was a DB with multiple distinct schemas, each with their own owner and extreme separation between the tables (don't ask...). As I said, though, most other scenarios have been fine.
Save your database as a sql script
EMS SQL Manager (for example) allows you to backup your database as a script in a .sql file. It is then possible to run this file against any other SQL server. Just update the first line of the script (CREATE DATABASE ....).
You can even fully parameter the script in order to include data from complete or filtered tables.