I have created a data flow within Azure synapse to:
take data from a dedicated SQL pool
perform some transformations
send the resulting output to parquet files
I am then creating a View based on the resulting parquet file using OPENROWSET to allow PowerBI to use the data via the built-in serverless SQL pool
My issue is that whatever the file name I enter on the integration record, the parquet files always look like part-00000-2a6168ba-6442-46d2-99e4-1f92bdbd7d86-c000.snappy.parquet - or similar
Is there a way to have a fixed filename which is updated each time the pipeline is run, or alternatively is there a way to update the parquet file to which the View refers each time the pipeline is run, in an automated way.
Fairly new to this kind of integration, so if there is a better way to acheive this whole thing then please let me know
Azure Synapse Data Flows - parquet file names not working
I repro'd the same and got the file name as in below image.
In order to have the fixed name for sink file name,
Set Sink settings as follows
File name Option: Output to single file
Output to single file: tgtfile (give the file name)
In optimize, Select single partition.
Filename is as per the settings
Related
I have created CSV files into UNIX server using Informatica resides in. I want to load those CSV files directly from UNIX box to snowflake using snowsql, can someone help me how to do that?
Log into SnowSQL:
https://docs.snowflake.com/en/user-guide/getting-started-tutorial-log-in.html
Create a Database, Table and Virtual Warehouse, if not done so already:
https://docs.snowflake.com/en/user-guide/getting-started-tutorial-create-objects.html
Stage the CSV files, using PUT:
https://docs.snowflake.com/en/user-guide/getting-started-tutorial-stage-data-files.html
Copy the files into the target table using COPY INTO:
https://docs.snowflake.com/en/user-guide/getting-started-tutorial-copy-into.html
I am attempting to create a temp table to store the values of an xlsx file on my azure blob storage, I have followed numerous Microsoft articles now and I am under the impression that I should be using SELECT * FROM OPENROWSET(), this seems to be working or at least selecting something.
Here is my code:
SELECT * INTO ##TempTest FROM OPENROWSET(BULK 'test.xlsx',
DATA_SOURCE = 'DevStoreAccount', SINGLE_CLOB) AS a;
SELECT * FROM ##TempTest
This all runs fine, but the output is not what I am expecting, surely this should return all my columns / rows from the excel file? Or am I mistaken?
The above code returns the following:
What exactly is it returning and should I be doing something different? Any help would really be appreciated.
I'm trying this route as the columns in the excel file could change at any time, so I need to dynamically create my tables.
I'd recommend checking this thread, although the post is old, it is still relevant to your question.
The approach taken for the similar scenario:
1- Create and update Excel file using Open XML SDK
2- Upload Excel Template in Azure BLOB
3- Download Excel template in azure web role local storage
4- Read and update excel file from azure web role local storage
5- Upload updated excel in Azure BLOB.
II
You could also use another similar concept as mentioned here
Downloading excel file as Stream from BLOB
Creating Excel document using Open XML SDK
After edit saving doc to Stream
Uploading back the Stream to BLOB
I have a file Stream Sample database , I have added records into table.
When I use file.PathName() my sample project in c# SqlFileStream class recognize this address and retrieve my file but did not show in windows file Explorer?
What is this address? Is it fake address?This class may look at FileGroup path for finding real address?if not how this class find path?
\ComputerName\SQL2016\v02-A60EC2F8-2B24-11DF-9CC3-AF2E56D89593\FileStreamTestDB\dbo\BLOB_Table\FileData\00953530-2F65-4AC9-81E9-0281EFB89592\VolumeHint-HarddiskVolume3
Data in a FILESTREAM column are stored inside of the database. You can see the internal files stored in the database by browsing the local file system FILESTREAM filegroup directory but that path is not exposed for remote access and shouldn't be used at all. You'll need to use SqlFileStream to get a handle for access to FILESTREAM data via the Win32 API.
If you want to access files stored in the database via Windows Explorer or any other application, consider using Filetable instead. A FileTable leverages FILESTEAM internally but exposes the files stored in the table via a UNC path for non-transactional access. That allows files to be added/change/deleted via the share just regular files or with T-SQL INSERT/UPDATE/DELETE statements. In both cases, the changes are stored in the database FileTable and reflected in the FileTable directory share too.
I have a table that stores users FileData as such Data Type: varbinary(MAX) FILESTREAM null
In my web application, the user selects multiple file Ids and eventually wants a zip file of those selected FileIds
Currently my solution is to bring the FileData into C# and call some C# function/library that zips the file and returns that to the user. The problem with this is that the user could potentially select a ton of files causing a lot of temporary data to exist in C#.
Is there a way that I can zip these files in SQL Server and then return the zipped result to C# without having to bring the selected FileDatas into C# memory?
You can certainly do this through a stored procedure, which would write the files and zip, but you would be writing SQL which writes files to disk and executes windows system commands. You can read up on xp_cmdshell. I would advise against this personally
You are still going to have a large zip file blob coming back to your server in that model. Couldn't your users still overload your system? You would get around this using streaming which could be done with your zipping.
Are you using the most recent ZipArchive? It provides streaming access both in and out if used properly. See here for an example writing without bumping into memory Basically you will write your code to use an output stream so that data doesnt build up in memory ...new ZipArchive(myOutPutStream, ZipArchiveMode.Update, true or false)
Synopsis
I am needing to bridge a gap between a CSV file and an Access 2007 database linked to a SharePoint list. The end goal is to fully automate this process: Remedy > CSV > Access > SharePoint
Problem Details
Remedy ---> CSV
I am using a macro in BMC Remedy to run a report and export data to a CSV file. Each time the macro runs, it is set up to overwrite the previous version of the CSV file.
CSV --x Access
While I can import the CSV table (as a text file) to Access, the program won't let me update the database. Creating macros, relationships or queries is impossible since the CSV file is overwritten each time the Remedy macro runs. When I attempt to copy the CSV table to the linked database (using export feature in Access) I get the following error:
You can't delete the table 'tablename'; it is participating in one or more relationships.
Access --> SharePoint
The Access database I am wanting to update is linked to a SharePoint list so that any edits made (and saved) in Access update SharePoint.
Work-Around
I can copy & paste the data from the CSV to the Access database, but am wanting a better solution that doesn't require any maintenance.
I have tried creating a macro, but when I use RunCommand > SelectAll I get the following error:
The command or action 'SelectAll' isn't available now.
Is it possible to do this with a macro or do I need a VB script?