I am attempting to create a temp table to store the values of an xlsx file on my azure blob storage, I have followed numerous Microsoft articles now and I am under the impression that I should be using SELECT * FROM OPENROWSET(), this seems to be working or at least selecting something.
Here is my code:
SELECT * INTO ##TempTest FROM OPENROWSET(BULK 'test.xlsx',
DATA_SOURCE = 'DevStoreAccount', SINGLE_CLOB) AS a;
SELECT * FROM ##TempTest
This all runs fine, but the output is not what I am expecting, surely this should return all my columns / rows from the excel file? Or am I mistaken?
The above code returns the following:
What exactly is it returning and should I be doing something different? Any help would really be appreciated.
I'm trying this route as the columns in the excel file could change at any time, so I need to dynamically create my tables.
I'd recommend checking this thread, although the post is old, it is still relevant to your question.
The approach taken for the similar scenario:
1- Create and update Excel file using Open XML SDK
2- Upload Excel Template in Azure BLOB
3- Download Excel template in azure web role local storage
4- Read and update excel file from azure web role local storage
5- Upload updated excel in Azure BLOB.
II
You could also use another similar concept as mentioned here
Downloading excel file as Stream from BLOB
Creating Excel document using Open XML SDK
After edit saving doc to Stream
Uploading back the Stream to BLOB
Related
I have created a data flow within Azure synapse to:
take data from a dedicated SQL pool
perform some transformations
send the resulting output to parquet files
I am then creating a View based on the resulting parquet file using OPENROWSET to allow PowerBI to use the data via the built-in serverless SQL pool
My issue is that whatever the file name I enter on the integration record, the parquet files always look like part-00000-2a6168ba-6442-46d2-99e4-1f92bdbd7d86-c000.snappy.parquet - or similar
Is there a way to have a fixed filename which is updated each time the pipeline is run, or alternatively is there a way to update the parquet file to which the View refers each time the pipeline is run, in an automated way.
Fairly new to this kind of integration, so if there is a better way to acheive this whole thing then please let me know
Azure Synapse Data Flows - parquet file names not working
I repro'd the same and got the file name as in below image.
In order to have the fixed name for sink file name,
Set Sink settings as follows
File name Option: Output to single file
Output to single file: tgtfile (give the file name)
In optimize, Select single partition.
Filename is as per the settings
We are migrating our data from on-premise to the cloud. One requirement is to store the attachments (.pdf, .txt, .png etc) data from a SQL Server table in Azure blob containers in a particular format.
The table structure looks like this:
TABLE
(
Id int,
Name nvarchar,
EntityId int,
RecordId int,
BlobData image
)
The files should be saved in following way in the container.
Folder structure : Container//Entity ID//Record ID//Files
For example:
BlobContainer//10//1//xyz.pdf, png.png
Here I consider both files xyz and abc to belong to same entity (10) and record (1).
I tried Azure Data Factory, but I am unable to find a proper solution as I am facing a source sink mismatch issue when trying to use "Copy Data" activity in a pipeline.
Has anyone faced the same problem or do you have any other solution to complete this task?
We are writing a new application, and while testing, we will need a bunch of dummy data. I've added that data by using MS Access to dump excel files into the relevant tables into the Postgres database.
What should I do now to generate an Insert statements from the PGAdmin4 Tool similar to what SQL Studio allow us to generate an Insert statements for SQL Server? There are no options available to me. I can't use the closest one, which is to export and import the data via CSV.
I understand that you cannot import the CSV file into the actual DB as this needs to be done through ASP.NET core EF. Perhaps, you can probably create a test schema and import the CSV file into the test schema. Once you have the data imported into the test schema, you can use that to generate SQL statements using the steps below:
Right click on target table and select "Backup".
Select a file path to store the backup. You can save the file name as data.backup
Choose "Plain" as Format.
Open the tab "Options" check "Use Column Inserts".
Click the Backup-button.
Once the file gets generated you can open with Notepad++ or VSCode to get the SQL insert statements
You can use the statements generated and delete the test schema created
Here is a resource that might help you in loading data from Excel file into PostgresSQL if you still need to take this path Transfer Data from Excel to PostgreSQL
I have a table that stores users FileData as such Data Type: varbinary(MAX) FILESTREAM null
In my web application, the user selects multiple file Ids and eventually wants a zip file of those selected FileIds
Currently my solution is to bring the FileData into C# and call some C# function/library that zips the file and returns that to the user. The problem with this is that the user could potentially select a ton of files causing a lot of temporary data to exist in C#.
Is there a way that I can zip these files in SQL Server and then return the zipped result to C# without having to bring the selected FileDatas into C# memory?
You can certainly do this through a stored procedure, which would write the files and zip, but you would be writing SQL which writes files to disk and executes windows system commands. You can read up on xp_cmdshell. I would advise against this personally
You are still going to have a large zip file blob coming back to your server in that model. Couldn't your users still overload your system? You would get around this using streaming which could be done with your zipping.
Are you using the most recent ZipArchive? It provides streaming access both in and out if used properly. See here for an example writing without bumping into memory Basically you will write your code to use an output stream so that data doesnt build up in memory ...new ZipArchive(myOutPutStream, ZipArchiveMode.Update, true or false)
I am using snowflake cloud datawarehouse, which is like teradata that hosts data. I am able run queries and get results on the web UI itself. But I am unclear how can one export the results to a local PC so that we can report based on the data.
Thanks in advance
You have 2 options which both use sfsql which is based on henplus. The first option is to export the result of your query to a S3 staging file as shown below:
CREATE STAGE my_stage URL='s3://loading/files/' CREDENTIALS=(AWS_KEY_ID=‘****' AWS_SECRET_KEY=‘****’);
COPY INTO #my_stage/dump
FROM (select * from orderstiny limit 5) file_format=(format_name=‘csv' compression=‘gzip'');
The other option is to capture the sql result into a file.
test.sql:
set-property column-delimiter ",";
set-property sql-result-showheader off;
set-property sql-result-showfooter off;
select current_date() from dual;
$ ./sfsql < test.sql > result.txt
For more details and help, login to your snowflake account and access the online documentation or post your question to Snowflake support via the Snowflake support portal which is accessible through the Snowflake help section. Help -> Support Portal.
Hope this helps.
You can use a COPY command to export a table (or query results) into a file on S3 (using "stage" locations), and then a GET command to save it onto your local filesystem. You can only do it from the "sfsql" Snowflake command line tool (not from web UI).
Search the documentation for "unloading", you'll find more info there.
You can directly download the data from Snowflakes to Local Filesystem without staging to S3 or redirecting via unix pipe
Use COPY INTO to load table data to table staging
https://docs.snowflake.net/manuals/sql-reference/sql/copy-into-location.html
snowsql$> copy into #%test_table/result/data_ from test_table
file_format = (TYPE ='[FILE_TYPE]' compression='[COMPRESSION_TYPE]');
Use GET command to download data from table staging to Local FS
https://docs.snowflake.net/manuals/sql-reference/sql/get.html
snowsql$> get #%test_table/result/data_ file:///tmp/;