guys.
I know it is a general question. Let me give you my scenario:
I have a client who sends a bunch of Excel files to me and I use my on-premise SSIS package to export it to a database located on Azure. My SSIS package does call stored procedures stored on the Azure SQL Server to manipulate the data.
I want to move the whole process to the cloud and I want to know what is the best way and how can we achieve it. I was thinking maybe I can use blob storage in a container and by providing a cloud folder located on Azure and let my client through the files there. Then my an app (service) such as Data Factory can detect those files and run my SSIS package that is deployed on Azure "Somehow".
Any ideas or sample code would be great.
Thanks!
You can try below manual approach -
1. Copy all csv files to ADLS (Azure Data Lake Storage) (For Automation - Copy activity, for loop and lookup activity, you may use)
2. For any transformation of data, use USQL jobs (ADLA) whose output is also stored in ADLS. You can save USQL scripts in blob storage (for ADF AUTOMATOMATION).
3. To transfer the data from ADLS files to Azure sql database, use copy activity of azure data factory and use sql as sink and csv as source format.
Related
I need to migrate .SQB files to Snowflake.
I have a data relay where MSSQL Server database files are saved in .SQB format (Redgate) and available via sSTP with full backups every week and hourly backups in between.
Our data warehouse is Snowflake and the rest of our data from other sources. I'm looking for the simplest, most cost effective solution to get my data to Snowflake.
My current ETL process is as follows.
AWS EC2 instance (Windows) that downloads the files, applies
Redgate's SQL Backup Converter
(https://documentation.red-gate.com/sbu7/tools-and-utilities/sql-backup-file-converter)
to convert the files to .BAK. This tool requires a license
Restore MS SQL database on the same AWS EC2 Instance
Migrate MS SQL database to Snowflake via Fivetran
Is there a simpler / better solution? I'd love to eliminate the need for the intermediate EC2 if possible.
The .SQB files come from an external vendor and there is no way to have them change the file format or delivery method.
This isn't a full solution to your problem, but it might help to know that you're okay to use the SQL Backup file converter wherever you need to, free of any licensing restrictions. This is true for all of SQL Backup's desktop and command-line tools. Licensing only gets involved when dealing with the Server Components, but once a .SQB file has been created you're free to use SQBConverter.exe to convert it to a .BAK file wherever you need to.
My advice would be to either install SQL Backup on whichever machine you want to use the tooling on, or just copy all the files from an existing installation. Both should work fine, so pick whichever is easiest for you.
(FYI: I'm a current Redgate software engineer and I used to work on SQL Backup until fairly recently.)
You can
Step 1: Export Data from SQL Server Using SQL Server Management Studio.
Step 2: Upload the CSV File to an Amazon S3 Bucket.
Step 3: Upload Data to Snowflake From S3 using COPY INTO command.
You can use your own AWS S3 bucket for this and then create a External Stage pointing to the S3 bucket or You can upload the files into internal Snowflake Stage.
Copy Into from External Stage -
https://docs.snowflake.com/en/sql-reference/sql/copy-into-table.html#loading-files-from-a-named-external-stage
Copy Into from an Internal Stage -
https://docs.snowflake.com/en/sql-reference/sql/copy-into-table.html#loading-files-from-an-internal-stage
Creating External Stage-
https://docs.snowflake.com/en/sql-reference/sql/create-stage.html
I have one Storage Account on Azure where I get lots of json files from somewhere. I am running an exe on my computer constantly to get the Json File and read the data from json and convert the data into table and upload it on an SQL Server database on azure - A synchronization process
I have not worked with WebJob or FunctionApp
I believe this can be done by FunctionApp triggering on blob. I am not sure if I will be able to do the complete process as I do locally on FunctionApp.
Is this the right decision for this purpose or would you suggest WebJob for this?
If you prefer to use azure web jobs, this should be as simple as just uploading a .zip file of your project. You can follow this article to create the webjob, and the webjob should work as what you do locally.
If you prefer to use azure function, then you can use blob trigger do the same thing, but you may make some changes as per your need.
Currently I have a web app that imports Excel files, parses them (using openXML) then imports the data into SQL Server. The user does not need to predefine the Excel template as long as he sends the file in the same order of the database table he wants to insert it, if a value is not compatible with a column type, the system generates a log error file which basically says: "Excel Address -> A1, Value -> XXXXX (string), Expected -> (Date yyyy/mm/dd).
On parsing the file I check if the formats are compatible with the db table
I want to migrate this service to the cloud, specifically azure since I use Visual Studio to develop and the integrations facilitate the job. However I'm a little lost on the multiple services and if its possible to even do this kind of work.
I would like to store the files in Azure Blob storage then send to Data Lake(?) to import the data to a SQL Server database, is this possible? Are these the services I should use?
The simplest (and probably cheapest) solution I can think of given these constraints would be to upload to an Azure Function. Your Azure Function can then perform the parsing and insert records into the SQL database. It can also return an HTTP error code and message if the parsing cannot complete due to data errors.
The downside of Azure Functions is runtime constraints. If you are pushing so much data that you run into those constraints, you could spin up an Azure WebApp to do the same thing.
Can't you just create your tables in Azure SQL and then change your current web application to connect to Azure SQL instead of your on-premise SQL Server?
Azure SQL is basically a SQL Server in the cloud with a few differences (nowadays the on-premise versions also contains most of the Azure features).
If you are lucky you would just need to change your connection string:)
My current model looks like this:
Gather disparate data sources and import into SQL Server.
Process and transform data using SSIS packages.
final step in the SSIS package uploads data to the data warehouse.
BI tools pull data from the data warehouse for end users.
Is this a logical work flow? I initially was going to use data factory and the Azure SSIS integration runtime to process data. However I didn't understand why these steps were needed, as it would seem simpler in my situation just to build my SSIS packages on premises and upload the processed data to my data warehouse. What benefits would I gain from using data factory and the integration runtime? My main concern is that my current model will make automation difficult but I'm not entirely sure. Any help is appreciated.
Your possible paths here would be SSIS on prem, SSIS on VM in Cloud, SSIS in ADF or natively build the pipelines in ADF.
ADF is an Azure Cloud PaaS managed service for data movement and data integration orchestration. To reach back into on-prem data sources, you need to use an Integration Runtime gateway on the source side. So, if you are looking to move to a Cloud-first architecture or migrating into Azure, ADF is a good solution (use V2).
If you are remaining all on-prem SSIS on-prem is the best scenario.
If this is hybrid, where you will continue to have some data on prem and load Azure Data Warehouse in the Cloud, then you can still use SSIS on prem with connectors into ADW as the target. Or if you have to eliminate the local server concept, you can run that SSIS in a VM in Azure.
If you want to eliminate both the datacenter server and the need to patch, maintain, etc. the SSIS server, then use SSIS in ADF, which provides SSIS as a Service. In that case, you can still move data in a hybrid manner.
It really is going to depend on factors such as are you comfortable more in Visual Studio to develop SSIS jobs or do you want to build the pipelines in JSON in ADF? Do you have a plan or a need to move to Cloud? Do you want to move to a Cloud-Managed service (i.e. ADF V2)?
I hope that helps!!
I'm in the process of migration from dedicated servers to Azure. In my existing SQL Server, I have a few jobs that move data from live database to archives.
From what I have read so far, in Azure you cannot use cross database scritps. The other options I have seen include Azure SQL Data Sync, Azure Factory and maybe SSIS. I have to note that there's some logic on what data is archived and I need the ability to specify this in the query.
Has anyone some experience and what would you recommend?
Thanx
You can use the copy feature inside of data factory to do this now directly in Azure.
Azure Data Factory