I'm in the process of migration from dedicated servers to Azure. In my existing SQL Server, I have a few jobs that move data from live database to archives.
From what I have read so far, in Azure you cannot use cross database scritps. The other options I have seen include Azure SQL Data Sync, Azure Factory and maybe SSIS. I have to note that there's some logic on what data is archived and I need the ability to specify this in the query.
Has anyone some experience and what would you recommend?
Thanx
You can use the copy feature inside of data factory to do this now directly in Azure.
Azure Data Factory
Related
I want to move all data from one Azure SQL Server to different Azure SQL Server which more than 90 days old, and after moving need to delete moved data from first Azure SQL Server.
I want to run these steps on daily basis.
I am new to Azure and able to do same with Azure Data Factory. Can you please suggest any other best suited approach?
You are already using the best approach.
Azure Data Factory is an easy to use when it comes to extract and copy the data between the services. It also provide scheduling the triggers, i.e., triggering the copy pipeline after specific interval of time or any event. Refer Create a trigger that runs a pipeline on a schedule.
If the volume of data is large, you can re-configure the Integration Runtime (IR) resources (Compute type and Core count) to overcome the performance issue, if required. Refer below image.
We have two Azure PaaS SQL Servers, one in our production resource group and another in our QA resource group.
I'm looking for the best way to 'copy' the data from an Azure SQL DB in the production resource group to an Azure SQL DB in the QA resource group. As part of this process the data must undergo some simple transformations to anonymise the data before it lands in the QA DB.
We have deletion locks in place on the resource groups so deleting and recreating databases is an option I'd rather avoid.
The 'copying' can take place on a weekly schedule and it would be acceptable to drop any existing data in the target Azure SQL DB each time the process is run.
What are the most suitable options in Azure to accomplish this?
Go with Azure Data Factory. You can use it for pretty much anything that SSIS was used for onsite, and more.
My project is to nightly upload employee next-of-kin details somewhere offsite in case of emergency. Security needs to be locked down to just three named users.
The data is already available in a VM SQL Server 2014 view.
My first bash, was to create a SQL Job to extract to CSV via BCP, then (step2) to upload to Azure file share via AZCopy.
I thought I'd nailed my first azure project ... but sadly this uses a shared access signature (appended to the URL) and not Azure AD, so I don't think this will do? (not sure)
Any ideas please?
If you don't need the data structured at the other end of your export then a flat-file as you suggested would work well and keep the complexity down. Shared signatures will work well, you just have to renew the signature when it expires. You'd have the same issue using Azure AD as well as the authentication token would just expire.
You could directly out of SQL Management Studio backup the full database to Azure Storage but this sounds like overkill for your requirements.
Another way to do it is by using Azure Data Factory, however, this has additional costs of executing the data pipeline to move your data and the complexity for a simple task.
Personally, the simplest way would be to export the data to a CSV on the file system, then have a scheduled task using PowerShell and put it up in your blob container in Azure Storage.
I'm aware of the various options in place for migrating a single database up to Azure. My problem is that these all only seem to cater for a single database at a time. However, I have a db per tenant model with over 2000 databases to migrate and not a lot of time to play with.
Can anyone point me in towards the best (ie fastest) way of doint this?
In the end we accomplished this with Powershell and the Azure API. Essentially batch creating bacpacs on the source server, uploading them to blob storage then importing them into Azure SQL server pools.
If I was facing the same challenge now I'd take a look at the Azure Database Migration Service - https://azure.microsoft.com/en-gb/services/database-migration/
I am also facing this problem and am going down the route of using the Visual Studio data compare tool.
All my tenant databases have the same schema so I made an empty template database in Azure, and just use the CREATE AS COPY command to make a new one each time ready for receiving the migration.
Then I ask Visual Studio to compare the empty database with the live database and automatically insert the data for me.
Seems to be working well so far, there's very little manual steps needed and it doesn't involve using the Azure Portal, or blob storage or creating databases outside of the elastic pool which is great. But the overall time will be slow to migrate data for all the databases.
I'm relatively new to Azure and am having trouble finding what options are out there for connecting to an existing SQL database to push data into it.
The situation is that we have an external client who needs to connect to our Azure SQL database to push data into it, on an on-going basis. We can't give them permission to get into our database, so we're looking at what we can do allow data in. At this point the best option seems to be to create a web service deployed in Azure that will validate the data and then push it into our database.
The question I have is, are there other options to do this in an easier way? Are there Azure services or processes that can be set up to automatically process a file and pull the data into a database? Any other go-between options when each side has their own database and for security reasons can't just open up access to it?
Azure Data Factory works great for basic ETL. If neither party can grant direct access, you can use an intermediate repository like Blob Storage to drop csv/xml/json files for ingestion. If they'll grant you access to pull, you can setup a linked service that more or less functions the same as a linked server in MSSQL. As of the last release ADF now supports Azure hosted SSIS packages too.
I would do this via SSIS using SQL Studio Managemenet Studio (if it's a one time operation). If you plan to do this repeatedly, you could schedule the SSIS job to execute on schedule. SSIS will do bulk inserts using small batches so you shouldn't have transaction log issues and it should be efficient (because of bulk inserting). Before you do this insert though, you will probably want to consider your performance tier so you don't get major throttling by Azure and possible timeouts.