I am working on Azure Logic App based solution to export the data from Database to CSV file.
So far, I am able to do the following:
Use SQL database connector and execute the stored procedure to retrieve data
Create a CSV file
Use FTP connector to upload the file to an FTP server
With this approach, the issue is going to be data size. I am trying to figure out a way(s) to handle large data export. Here are the different approaches I can think of:
Option 1: Use pagination in SP, and iteratively collect the data.
Question 1.1: In each loop in the Logic App, multiple CSV files will be generated. How to combine the data from each iteration into a single CSV file?
Question 1.2: I am thinking of using a variable array to collect the data from each iteration, and then create CSV file from the variable array. Will I run into any issue if the size of the variable array becomes too large?
Option 2: Somewhere I read that you can overcome the data size issue with chunking.
Question: I am not even sure if the database connector supports chunking and will I able export the data into a single CSV file?
Option 3: Create multiple CSV files and then merge them into one CSV file?
Question: Is that possible to do it in Logic apps? Or will I need to implement a Logic function to handle the merging of CSV files?
Its hard to say different options without knowing the amount of data you are talking about.
1- Yes you can use pagination, i would recommend using a SP in your sql server to return the data and also the page size and number: https://social.technet.microsoft.com/wiki/contents/articles/40060.sql-pagination-for-bulk-data-transfer-with-logic-apps.aspx
1.1 By default for-each in LogicApps is executed in parallel, check the For each settings and change the concurrency control: https://learn.microsoft.com/sv-se/azure/logic-apps/logic-apps-control-flow-loops#foreach-loop-sequential
The data returned from the SP can be converted to csv by using "create csv table"
https://learn.microsoft.com/sv-se/azure/logic-apps/logic-apps-perform-data-operations#create-csv-table-action
Related
I have a python script that fetches data twice a day from a server of mine. The script returns around 40 JSON files containing various data. The files aren't particularly big and the combined size of all the files is around 250KB.
Alongside my script I am developing a dashboard in React that renders the data from each file into a table, allowing me a visual representation of the data.
I have been looking at what would be the best way to store these files, something that allows me to upload and fetch them twice a day.
Someone mentioned to me about using MongoDB to store the files, but after some research I feel like Mongo is better at storing the contents of the file rather than the file itself. I tried to develop a solution but I couldn't figure out how it could be done when each object is stored as a document with no clear way (to me) which document came from which file.
Other options I have considered are:
Storing the files on the server that is hosting my React project and rendering them locally as I am doing now during development
Storing the files using a provider such as AWS/Firebase
Storing them in a different database (I see SQL now support the storing of JSON files)
Are there any other solutions that you think would work best for this scenario? If so, why?
Hello,
Check about use of FTP server.
We have clients that send us data every 10 min via FTP that is inside XML files, then I have NodeJS back-end which read these files.
You can use it for your scenario with JSON files.
My basic workflow is this: I check an FTP server for a specific file. If the file exist, I pick up the file and sends it to a Blob Storage. My problem is this: I want to filter the file content, eg. remove first and last row since they dont contain any real data before I send it to the blob. The first row consist of a time stamp and the last row contains a "row count". The file contains comma separated fields. How do I accomplish this? Is it even possible?
Thanks
Ausgar
There is no simple solution for this problem. You can try converting csv to json, delete unnecessary data from it, and create a blob based on that json, but this is sounds harder than it should be.
Consider using Azure functions:
Azure Functions allows you to run small pieces of code (called
"functions") without worrying about application infrastructure. With
Azure Functions, the cloud infrastructure provides all the up-to-date
servers you need to keep your application running at scale.
It will be much easier to do such file manipulation there.
I am using Oracle ODI 11.1.1.7.
I have 6, pipe delimited files. Each file has different number of columns. Number of columns is fixed in each file. So I know the format. I want to load all these files in a single table in database.
I can create ODI process with steps in sequential order and call interfaces created for these files to accomplish the task.
Is there any better way to do this? Like creating only one interface which can work with all these files. Something i can do through a loop?
Thanks in advance.
Unfortunately as these files have a different structure (number of columns), you will need have a different source datastore for each of them so you will need different interfaces.
If the structure were the same, you could use only one datastore and one interface. You would need to use a variable as the filename in the datastore definition and create a loop in a package that would change the value of the variable and execute the interface loading the file.
I want to export all my queries as individual files for purposes of putting them into mercurial source control, but I don't know how to export the individual queries as individual files without having to open each one, then save to the folder, then add into the project, or some equally convoluted process.
I wouldn't mind having to add each one individually, but how do I get them out of the database as individual files without opening them all and doing each one save as? Ostensibly I would like them named with the name they have in the database right now.
I could easily dump the whole lot into one long file using database tasks, but that's not really super helpful is it?
I have SSMS 2k5 and 2k8 (and VS 2k5, 2k8, 2010 to boot) to work with, any thoughts?
Right click on the database. Select Generate Script. On the last page. Script To file you can choose single file or file per object
When you script a database in SSMS you have the option of one file per objects.
SMO is useful with a small app to iterate through
Third party tools like Red Gate SQL Compare (there are other free tools) can script too
I would write a small C# program which extracts your database object via SMO and stores them in your filesystem the way you want.
It is rather easy to write stored procedures which fetches the definition into the result as text. sp_helptext could be used as start.
Than you can use PowerShell to write the Output to the file system.
It sounds as if this would fit rather good into the Really Simple Data Dictionary codeplex project. link text
Working on a school project, the program is supposed to read from a text file that has a record about a song in every line, fields separated by ";".
Anyways I have no knowledge of databases, and I just want the quickest way to create a database from that text file, and also i will need to change some of the fields of the records once in a while from the program... Also the program needs to search through the database based on certain fields.
Anyways so far all our projects didn't keep a database, so when we closed the program, every info was gone, now i actually need to keep some info for the next time the program runs. What's the fastest way to accomplish this?
Also I wanna be able to keep some info about the software, like the path of the original text file for weekly updates. Where can i save info like that?
EDIT: it doesn't have to an actual database, as long as i can search and edit it efficiently.
If you can use SQL database, I'd suggest simple file-based database SQLite
With SQLite, you can query, insert and update records by executing regular SQL statements.
Here you will find introduction to C++ interface It's easy to embed SQLite support in an application because SQLite comes as a library, meaning a bunch of header files and 1-2 binary archive with library.
Your comma-delimited textfile is aleady a database. You can add records, delete records, and modify records using the standard textfile routines provided by the standard C++ libraries.
Alternatively, you can import your textfile into SQL Server using BULK INSERT.
Finally, you can access your CSV (comma-delimited text) file using SQL queries. You need to find the correct connection string. See http://www.connectionstrings.com/textfile.