I have a table that stores users FileData as such Data Type: varbinary(MAX) FILESTREAM null
In my web application, the user selects multiple file Ids and eventually wants a zip file of those selected FileIds
Currently my solution is to bring the FileData into C# and call some C# function/library that zips the file and returns that to the user. The problem with this is that the user could potentially select a ton of files causing a lot of temporary data to exist in C#.
Is there a way that I can zip these files in SQL Server and then return the zipped result to C# without having to bring the selected FileDatas into C# memory?
You can certainly do this through a stored procedure, which would write the files and zip, but you would be writing SQL which writes files to disk and executes windows system commands. You can read up on xp_cmdshell. I would advise against this personally
You are still going to have a large zip file blob coming back to your server in that model. Couldn't your users still overload your system? You would get around this using streaming which could be done with your zipping.
Are you using the most recent ZipArchive? It provides streaming access both in and out if used properly. See here for an example writing without bumping into memory Basically you will write your code to use an output stream so that data doesnt build up in memory ...new ZipArchive(myOutPutStream, ZipArchiveMode.Update, true or false)
Related
I have created a data flow within Azure synapse to:
take data from a dedicated SQL pool
perform some transformations
send the resulting output to parquet files
I am then creating a View based on the resulting parquet file using OPENROWSET to allow PowerBI to use the data via the built-in serverless SQL pool
My issue is that whatever the file name I enter on the integration record, the parquet files always look like part-00000-2a6168ba-6442-46d2-99e4-1f92bdbd7d86-c000.snappy.parquet - or similar
Is there a way to have a fixed filename which is updated each time the pipeline is run, or alternatively is there a way to update the parquet file to which the View refers each time the pipeline is run, in an automated way.
Fairly new to this kind of integration, so if there is a better way to acheive this whole thing then please let me know
Azure Synapse Data Flows - parquet file names not working
I repro'd the same and got the file name as in below image.
In order to have the fixed name for sink file name,
Set Sink settings as follows
File name Option: Output to single file
Output to single file: tgtfile (give the file name)
In optimize, Select single partition.
Filename is as per the settings
Here's the situation, we have a store procedure that generates .txt files in a directory inside of a database. I recently migrated from Oracle DB SE on Linux to an RDS oracle database instance.
I know I'm not able to access files remotely (no scp, or sftp) to copy the files that it generates, so instead of I decided to use DBMS_FILE_TRANSFER.PUT_FILE using a DB link to another database.
The link is working fine, but whenever I try to copy the file I get the following error:
ERROR at line 1:
ORA-19505: failed to identify file "/u01/test.txt"
ORA-27046: file size is not a multiple of logical block size
Additional information: 1
Is there any alternatives to do this without using an EC2 instance?
File size must be a multiple of 512 (or whatever you've set the logical block size to)
Ensure when the file is made that the size is exactly a multiple of 512, pad with zeros or similar to ensure this
See https://asktom.oracle.com/pls/apex/asktom.search?tag=file-transfer
I have a file Stream Sample database , I have added records into table.
When I use file.PathName() my sample project in c# SqlFileStream class recognize this address and retrieve my file but did not show in windows file Explorer?
What is this address? Is it fake address?This class may look at FileGroup path for finding real address?if not how this class find path?
\ComputerName\SQL2016\v02-A60EC2F8-2B24-11DF-9CC3-AF2E56D89593\FileStreamTestDB\dbo\BLOB_Table\FileData\00953530-2F65-4AC9-81E9-0281EFB89592\VolumeHint-HarddiskVolume3
Data in a FILESTREAM column are stored inside of the database. You can see the internal files stored in the database by browsing the local file system FILESTREAM filegroup directory but that path is not exposed for remote access and shouldn't be used at all. You'll need to use SqlFileStream to get a handle for access to FILESTREAM data via the Win32 API.
If you want to access files stored in the database via Windows Explorer or any other application, consider using Filetable instead. A FileTable leverages FILESTEAM internally but exposes the files stored in the table via a UNC path for non-transactional access. That allows files to be added/change/deleted via the share just regular files or with T-SQL INSERT/UPDATE/DELETE statements. In both cases, the changes are stored in the database FileTable and reflected in the FileTable directory share too.
I want to download CSV file from FTP location and update that data into tables using stored procedure. I am not sure how to do that or whether stored procedure is right approach. I have gone through many posts but most of the post talk about pushing data to FTP location.
Any help much appreciated.
Thank you.
For this requirement you may have to write some Shell Script to connect required FTP Server and download to local system and use to sql loader function supported by the respective database.
If you using some programming language you can write program to download file from ftp location and read CSV file and insert as batch.
I wish to write a query that inserts a file that resides on the client (C# web server) into a column in the database server (SQL Server), something like INSERT … SELECT * FROM OPENROWSET(BULK…), but without having to save the file on the server machine first.
Is this even possible in SQL?
Although your context is unstated, I'm assuming that you're intending to run this from SSMS rather than from OSQL, a PowerShell script, or through some other means.
The file doesn't need to reside on the physical box running SQL Server, but SQL Server does need access to it. The typical approach, I believe, would be for an application server to copy the file to a shared repository and then pass it off to SQL Server through a UNC reference. The syntax to do so is relatively trivial and can be found in Importing Bulk Data by Using BULK INSERT or OPENROWSET(BULK...).
If instead you're interested in providing a mechanism for the SQL Server to save a file from some type of stream operation where the client is directly transmitting a file and there is no shared repository, I'm not aware of a way to do that. Even if you use an SQL FILESTREAM object you still need an accessible NTFS location to stream from. See Saving and Retrieving File Using FileStream SQL Server 2008.
At some point, the server will have to have a hand on the file. That does not mean that the server has to keep the file, but the file has to get to the server in order to be read and inserted into the db. Typically, this is achieved with a form and a file-type input. On the server, you can use the uploaded file to create your query, then delete it.
That said, storing files in a database is a debatable practice. Depending on the type and size of files you're storing, your database can quickly balloon in size. For starters, this makes backups slower and more prone to failure, along with a laundry list of other potential pitfalls. Check out this question on SO: Storing Images in DB - Yea or Nay? As you can see from the answers, there are a number of considerations to be made, but a good rule of thumb is to not do this unless you have compelling reasons to do so.
In SQL Server BLOB Data in .NET: Tutorial, Mohammad Elsheimy explains how it can be done:
using(SqlConnection con = new SqlConnection(conStr) )
using(SqlCommand command = new SqlCommand("INSERT INTO MyFiles VALUES (#Filename, #Data)", con) )
{
command.Parameters.AddWithValue("#Filename", Path.GetFileName(filename));
command.Parameters.AddWithValue("#Data", File.ReadAllBytes(filename));
connection.Open();
}
Basically, this way the file is read on the client and sent to the database server without a need for a temporary file on the server machine.