I've currently got an ETL process that dynamically builds and executes sql jobs based on job steps that are saved in my database. Included in these jobs are steps to call SSIS packages that move data from one server to another and/or call stored procs on target servers to do further processing. I'm looking at what it would take to migrate our process from SQL Server to a Azure Managed Instance. One of the specific things I'm looking at is the feasibility of replacing the steps that call the SSIS packages with steps that execute Azure Data Factory pipelines or other ADF actions that accomplish the same results. So far I have not run across any examples of this. Anyone have any experience with accessing Data Factory functionality with SQL Agent jobs?
You can run powershell scripts via SQL agent as mentioned in below MSFT docs:
https://learn.microsoft.com/en-us/sql/powershell/run-windows-powershell-steps-in-sql-server-agent?view=sql-server-ver16
And via powershell and ADF REST APIS, you can trigger the ADF pipelines
Related
I want to move all data from one Azure SQL Server to different Azure SQL Server which more than 90 days old, and after moving need to delete moved data from first Azure SQL Server.
I want to run these steps on daily basis.
I am new to Azure and able to do same with Azure Data Factory. Can you please suggest any other best suited approach?
You are already using the best approach.
Azure Data Factory is an easy to use when it comes to extract and copy the data between the services. It also provide scheduling the triggers, i.e., triggering the copy pipeline after specific interval of time or any event. Refer Create a trigger that runs a pipeline on a schedule.
If the volume of data is large, you can re-configure the Integration Runtime (IR) resources (Compute type and Core count) to overcome the performance issue, if required. Refer below image.
I have two SQL server databases running on Azure - one Prod and one Test. I want every night, the Test server will automatically sync its data and structure (include any TABLE, Stored Proc changes ) with Prod - so Test is a mirror of Prod that can be used for development / QA.
I am wondering how to achieve this on Azure? Please note I would like this to happen automatically and on a schedule. Thanks everybody.
Azure the moment, we can use Azure data sync service to sync data between database. But we cannot use the service to sync Stored Procedures. For more details, please refer to here and here. So if we want to sync Stored Procedures, SQL Server Management Studio Generate Scripts Wizard is an easy means of producing a script that will copy all of your Stored Procedures to another database.
Besides, regarding how to create a schedule to start azure SQL data sync, please refer to the blog
My current model looks like this:
Gather disparate data sources and import into SQL Server.
Process and transform data using SSIS packages.
final step in the SSIS package uploads data to the data warehouse.
BI tools pull data from the data warehouse for end users.
Is this a logical work flow? I initially was going to use data factory and the Azure SSIS integration runtime to process data. However I didn't understand why these steps were needed, as it would seem simpler in my situation just to build my SSIS packages on premises and upload the processed data to my data warehouse. What benefits would I gain from using data factory and the integration runtime? My main concern is that my current model will make automation difficult but I'm not entirely sure. Any help is appreciated.
Your possible paths here would be SSIS on prem, SSIS on VM in Cloud, SSIS in ADF or natively build the pipelines in ADF.
ADF is an Azure Cloud PaaS managed service for data movement and data integration orchestration. To reach back into on-prem data sources, you need to use an Integration Runtime gateway on the source side. So, if you are looking to move to a Cloud-first architecture or migrating into Azure, ADF is a good solution (use V2).
If you are remaining all on-prem SSIS on-prem is the best scenario.
If this is hybrid, where you will continue to have some data on prem and load Azure Data Warehouse in the Cloud, then you can still use SSIS on prem with connectors into ADW as the target. Or if you have to eliminate the local server concept, you can run that SSIS in a VM in Azure.
If you want to eliminate both the datacenter server and the need to patch, maintain, etc. the SSIS server, then use SSIS in ADF, which provides SSIS as a Service. In that case, you can still move data in a hybrid manner.
It really is going to depend on factors such as are you comfortable more in Visual Studio to develop SSIS jobs or do you want to build the pipelines in JSON in ADF? Do you have a plan or a need to move to Cloud? Do you want to move to a Cloud-Managed service (i.e. ADF V2)?
I hope that helps!!
I'm relatively new to Azure and am having trouble finding what options are out there for connecting to an existing SQL database to push data into it.
The situation is that we have an external client who needs to connect to our Azure SQL database to push data into it, on an on-going basis. We can't give them permission to get into our database, so we're looking at what we can do allow data in. At this point the best option seems to be to create a web service deployed in Azure that will validate the data and then push it into our database.
The question I have is, are there other options to do this in an easier way? Are there Azure services or processes that can be set up to automatically process a file and pull the data into a database? Any other go-between options when each side has their own database and for security reasons can't just open up access to it?
Azure Data Factory works great for basic ETL. If neither party can grant direct access, you can use an intermediate repository like Blob Storage to drop csv/xml/json files for ingestion. If they'll grant you access to pull, you can setup a linked service that more or less functions the same as a linked server in MSSQL. As of the last release ADF now supports Azure hosted SSIS packages too.
I would do this via SSIS using SQL Studio Managemenet Studio (if it's a one time operation). If you plan to do this repeatedly, you could schedule the SSIS job to execute on schedule. SSIS will do bulk inserts using small batches so you shouldn't have transaction log issues and it should be efficient (because of bulk inserting). Before you do this insert though, you will probably want to consider your performance tier so you don't get major throttling by Azure and possible timeouts.
Currently in our organisation we aren't using reporting services in Sql Server anywhere. I have a task where i need to fetch data using a query and dump it to some local drive on daily basis. All the dump task currently are done using bcp utility.
I want to know if this task can be done using Sql Server Reporting Services.
PS: i have read a lot about reporting services but i am not clear
1. if we can dump file on daily basis like a job does.
2. if we can add parametrised query to fetch data and pass parameters in configurable way.
please suggest if using reporting services would be beneficial in my case?
SQL Server Reporting Services is used to create and manage reports. It does not do administrative tasks, like execute jobs or scripts. So to answer your questions:
No, you cannot dump files on a daily basis, that is what SQL Server Agent does.
Yes, SSRS reports can accept parameters, but only to dynamically display data. There is nothing here in the way of executing BCP.