This year We moved from hosted servers to Azure VM's, we run two production servers (SQL and IIS). A vital component of our business is bulk transfer of data file. We take customers data from our SQL Server and then write it out to a file (XLS, CSV, XML, PDF, Word, etc.) and then either email these files to customers or in most cases, push them into their FTP server. We also have a few import procedures where we retrieve data files. All of this is currently done with SSIS packages.
We're examining a move to Azure Data Factory as a replacement for SSIS so that we can possibly move to either Azure SQL (if we can work out Broker Services limitations) or an Azure SQL Managed Instance.
I've done some preliminary work with ADF but I saw a couple of posts about lack of FTP support. Is it possible to create/deliver files to FTP and retrieve/consume files from FTP using ADF? Also, almost all of these jobs are automated and we use SQL Agent to run the packages. What is the Azure equivalent for scheduling these jobs to run?
There is automation in ADF but the scheduler is per pipeline. Azure Automation is more powerful and can automate more than one pipeline (Azure Data Factory v2), if needed.
Automation with Azure Data Factory (ADF)
You can receive files from FTP into an Azure Data Factory pipeline: Copy data from FTP server by using Azure Data Factory The idea is that you receive a file via FTP to submit to a particular pipeline activity, and that activity pushes data to an Azure data source. It might be possible to reverse the flow, and send data out.
The Azure SQL Database Managed Instance is the most on-premise like database (PaaS) service but SQL Server deployed on an Azure VM still has more functionality.
Related
I typically use pyodbc when running jupyter notebooks from my machine, but this does not work on Azure ML. My assumption is that this is being caused by Azure ML not knowing if I'm on my company's network as I typically need a VPN to the server if I'm not in office. The only solutions I can find online involve copying the data over on Azure Data Factory however I need to avoid this if possible as there are many tables I will need to experiment with, but nothing is intended to be long term and I'm unsure what I will even end up using.
Ideally there is a way to make pyodbc work but any other suggestions are welcome. I have researched integration runtimes but was unsure if that would solve my problem here.
The only solutions I can find online involve copying the data over on
Azure Data Factory however I need to avoid this if possible as there
are many tables I will need to experiment with, but nothing is
intended to be long term and I’m unsure what I will even end up using.
Ideally there is a way to make pyodbc work but any other suggestions
Unfortunately, the on-Prem SQL Server is not supported as a Data Source in Azure ML.
Only the Data sources available below are supported:-
Approach1)
You can copy your data from the on-premises SQL database to Azure SQL via copy tool in Azure Data factory and connect to Azure SQL via Azure Machine learning by directly connecting to it via Datasource like below:-
You can also use Self-hosted integration run time to connect to your SQL server on-prem in your data factory:-
Click on Option 2 to download the Integration runtime and set it in your local machine with the Registration keys mentioned above:-
Approach2)
If there’s a large data You can automate your entire copy process from the on-prem SQL server to Azure SQL by using the Azure DevOps pipeline.
References:-
https://learn.microsoft.com/en-us/answers/questions/775844/unable-to-connect-sql-server-to-azure-ml-pipeline By Ramr-msft
How To: Azure Data Factory CI/CD with Azure DevOps pipelines — The YAML WAY! | by Raghavendra Bharadwaj | Servian
I am not able to copy data from ADLS gen2 to SQL Server (its not Azure SQL) using ADF.
What I have done is like this:
Created Data Set: Adls gen2 dataset Src
SQL Server DataSet tgt
But it doesn't allow me to choose tgt as my sink, though it lists down to choose the sink if the data set is either from (Azure SQL or Data Lake).
You will have to create an Integration Runtime and configure the same in your SQL Server Linked Service in ADF.
SQL Server is supported as sink, you can find the details here
As SQL Server is a different compute environment than Azure, you will have to create IR (Integration Runtime) so that Azure and SQL Server can communicate with each other.
Integration Runtime
If you want create on-premise SQL Server as dataset, you must install the Self-hosted integration manually:
A self-hosted integration runtime can run copy activities between a
cloud data store and a data store in a private network. It also can
dispatch transform activities against compute resources in an
on-premises network or an Azure virtual network. The installation of
a self-hosted integration runtime needs an on-premises machine or a
virtual machine inside a private network.
If you're using Data Flow, Data Flow doesn't support self-hosted integration so that we can't use SQL Server as connector:
You must use Copy active instead.
HTH.
We are developing and SSIS service to import some data in Excel and CSV files in Azure. For uploading the files we have chosen Azure File Storage and we are running the SSIS packages on a VM. For picking up the files from file storage, we have mapped the File Storage as mapped network drive on the VM. This works file when we manually trigger the SSIS jobs. However, this fails when running as SQL Server Agent job. As far as I understand, the mapped drives are per user and they do not work for service account used for SQL Server Agent. Is there a way by which we can access the file storage in SSIS packages as SQL Agent Jobs?
I found this page but this is for basic windows network file sharing. Does not work for us as we also need to use the Shared Access Key for Azure File Storage.
I solved the problem using this solution.
Add credential via cmd not via the GUI
We're in the final phase of migrating our SQL Server on VM to Azure SQL Server. What is the most time-efficient approach to getting our existing SQL Server Agent Jobs out to Azure? I've noticed that inside of SSMS, when connected to an Azure SQL Server, the Agent does not exist - we were aware of this before migrating, but I have yet to have found whether there is an Azure service that replaces the SQL Server Agent Jobs functionality.
There are a number of options.
If you need timed execution of some data momement, Azure Data Factory can be a good replacement.
There is also an Azure Scheduler, that is often mentioned as an alternative. However, I believe that would work better on API's than directly on a database.
I think the best option is Azure automation and create a runbook that connects to your database and performs some action.
An interesting blog that details these options in more detail can be foond at: https://blogs.technet.microsoft.com/uktechnet/2016/02/05/is-sql-server-agent-missing-from-azure-sql-database/
We have two Sql 2008 R2 database. First and main one is on-premise which is used by our stock management and accounting systems.
The second is for our web site and it is on a 3th party hosting firm and that database is updated manually by a standard XML file operation
(creating XML from on-premise database, sending it to web server, reading from XML file & insert/update/delete web database)
we need to get rid of that manual XML operation and sync that two databases automatically, but problem is our hosting firm does not allow "Replication" or "Linked Server" facilities on their "Sql Server" system. That's way we can not sync databases with these "Sql Server" facilities.
I am trying find out sync alternatives for that scenario without changing the hosting system.
What options can we use ?
Is Microsoft Sync Framework can be used for this scenario? ( I'm not sure if it is just for the Microsoft Azure Sql system)
Thanks…
if you're fine coding, then Sync Framework can do this.
Sync Framework can sync SQL Compact, Express, Server, Azure or LocalDB
But bear in mind that Sync Framework will need to create some Sync-related objects on your databases (triggers, stored procedures, tables, etc...)
It sounds like you need a hosting provider that provides these services.
It is typical for a hosting provider to provide an on-demand or always VPN connection to the production servers from the client location to the servers at the hosting location. Such a service is often needed for support of online systems. If you had such a connection then having SQL Server do syncs would be no problem.
Any major (Rack Space, Peer1, etc) provider would be able to set up such a system.
It probably won't be as inexpensive as your current provider.
Well if changing a provider is not an option, at list in the short term and you need something right now, you can always automate parts or all your current process.
For that you can at least use:
PowerShell (or even plain old cmd.exe) script(batch) and bcp.exe to export and import your files on both ends and PowerShell and ftp to transfer your files from one server to another.
SQLServer Integration Services on premises to export and send files via ftp to your hosting location. And SSIS to grab and import those files at your hosting location.