Is it possible to import CSV data files directly to popSQL to work on and make an analysis on it, If yes, How can I do it ?
Related
I am working on Azure Logic App based solution to export the data from Database to CSV file.
So far, I am able to do the following:
Use SQL database connector and execute the stored procedure to retrieve data
Create a CSV file
Use FTP connector to upload the file to an FTP server
With this approach, the issue is going to be data size. I am trying to figure out a way(s) to handle large data export. Here are the different approaches I can think of:
Option 1: Use pagination in SP, and iteratively collect the data.
Question 1.1: In each loop in the Logic App, multiple CSV files will be generated. How to combine the data from each iteration into a single CSV file?
Question 1.2: I am thinking of using a variable array to collect the data from each iteration, and then create CSV file from the variable array. Will I run into any issue if the size of the variable array becomes too large?
Option 2: Somewhere I read that you can overcome the data size issue with chunking.
Question: I am not even sure if the database connector supports chunking and will I able export the data into a single CSV file?
Option 3: Create multiple CSV files and then merge them into one CSV file?
Question: Is that possible to do it in Logic apps? Or will I need to implement a Logic function to handle the merging of CSV files?
Its hard to say different options without knowing the amount of data you are talking about.
1- Yes you can use pagination, i would recommend using a SP in your sql server to return the data and also the page size and number: https://social.technet.microsoft.com/wiki/contents/articles/40060.sql-pagination-for-bulk-data-transfer-with-logic-apps.aspx
1.1 By default for-each in LogicApps is executed in parallel, check the For each settings and change the concurrency control: https://learn.microsoft.com/sv-se/azure/logic-apps/logic-apps-control-flow-loops#foreach-loop-sequential
The data returned from the SP can be converted to csv by using "create csv table"
https://learn.microsoft.com/sv-se/azure/logic-apps/logic-apps-perform-data-operations#create-csv-table-action
I am currently working on a request, through which I am trying to export an excel sheet from a div based structure. How can we add styles, so that that the div structure will fit the Excel table structure?
From another question
Depends if you want a CSV file or an XLS file. An XLS file can include formatting information for the cells, as well as row/column locking, protections and other features that are impossible in a CSV file. Also, keep in mind that Excel does not correctly support UTF-8 encoded content when opening CSV files.
This means you have to know the format XLS files are made of if you want to style them. Or use the standard CSV extension, without styling.
I got data file with .LOD extension from which I have to import data in the PostgreSQL database. I never worked with .LOD files. Can someone please help me with commands or steps that I need to follow to import data in my database?
If your text data are well formatted try using the COPY function http://www.postgresql.org/docs/9.4/static/sql-copy.html or you can try http://pgloader.io/ too. If the datas are not well formatted you'll have to write a script to do the load.
I have a Lightspeed POS 3.x Backup file. I would like to abstract the database out of it. Is this at all possible? The file extension of the backup file is .lightspeedBackup.
I believe you should be able to just rename it as a .zip and extract it. Once you do you should have several directories and files with all your data in it.
Not sure why they try to hide that or are using their own extension.
Background:
I've written a backup tool for the Mercurial/Git hoster Bitbucket, which asks for a username, gets a list of the user's repositories via the Bitbucket API, and clones each of them to the local machine.
Each Bitbucket repository has an issue tracker, and now I've got a new feature request: backup the issues as well.
I can actually get the issues in JSON format from the Bitbucket API.
Here is an example: https://api.bitbucket.org/1.0/repositories/christianspecht/bitbucket-backup/issues/
But how to backup this to the local machine?
I could just save the JSON in a text file, but I don't think you can do anything useful with a list of issues in JSON format.
Is there such a thing like a standard file format for bugs/issues?
In a perfect world, maybe anything that the most widespread issue tracking systems are able to import?
I suspect the answer is No, there's no standard. Here are some options:
Bugzilla:
Bugzilla can import/export both CSV and XML:
Here's this search: https://bugzilla.mozilla.org/buglist.cgi?quicksearch=css3+text exported to CSV and XML and here's this bug: https://bugzilla.mozilla.org/show_bug.cgi?id=156881 exported to XML.
JIRA:
JIRA can import from Pivotal Tracker, Bugzilla, FogBugz, Mantis, Trac and more, apparently, and can export as XML and CSV.
Here's this bug: https://jira.atlassian.com/browse/WBS-49 exported as XML and this search: https://jira.atlassian.com/browse/GHS-5597?jql=text~%22greenhopper%20api%20plugin%22 exported as XML - you can also export these as .xls files.
Roundup
The roundup-admin CLI has a sub command export which can be used to create a bunch of .csv files together with sub directories containing the messages and file attachments. See http://roundup.sourceforge.net/docs/admin_guide.html#migrating-backends for how this can be used to migrate the issue data to another database.
TRAC
Trac can export to CSV & TSV. Here's this search: http://trac.edgewall.org/report/2 exported to CSV and this bug: http://trac.edgewall.org/ticket/886 exported to CSV and TSV.