I have a standard WinForms application that connects to a SQL Server. The application allows users to upload documents which are currently stored in the database, in a table using an image column.
I need to change this approach so the documents are stored as files and a link to the file is stored in the database table.
Using the current approach - when the user uploads a document they are shielded from how this is stored, as they have a connection to the database they do not need to know anything about where the files are stored, no special directory permissions etc are required. If I set up a network share for the documents I want to avoid any IT issues such as the users having to have access to this directory to upload to or access existing documents.
What are the options available to do this? I thought of having a temporary database where the documents are uploaded to in the same way as the current approach and then a process running on the server to save these to the file store. This database could then be deleted and recreated to reclaim any space. Are there any better approaches?
ADDITIONAL INFO: There is no web server element to my application so I do not think a WCF service is possible
Is there a reason why you want to get the files out of the database in the first place?
How about still saving them in SQL Server, but using a FILESTREAM column instead of IMAGE?
Quote from the link:
FILESTREAM enables SQL Server-based applications to store unstructured
data, such as documents and images, on the file system. Applications
can leverage the rich streaming APIs and performance of the file
system and at the same time maintain transactional consistency between
the unstructured data and corresponding structured data.
FILESTREAM integrates the SQL Server Database Engine with an NTFS file
system by storing varbinary(max) binary large object (BLOB) data as
files on the file system. Transact-SQL statements can insert, update,
query, search, and back up FILESTREAM data. Win32 file system
interfaces provide streaming access to the data.
FILESTREAM uses the NT system cache for caching file data. This helps
reduce any effect that FILESTREAM data might have on Database Engine
performance. The SQL Server buffer pool is not used; therefore, this
memory is available for query processing.
So you would get the best out of both worlds:
The files would be stored as files on the hard disk (probabl faster compared to storing them in the database), but you don't have to care about file shares, permissions etc.
Note that you need at least SQL Server 2008 to use FILESTREAM.
I can tell you how I implemented this task. I wrote a WCF service which is used to send archived files. So, if I were you, I would create such a service which should be able to save files and send them back. This is easy and you also must be sure that the user under which context the WCF service works has permission to read write files.
You could just have your application pass the object to a procedure (CLR maybe) in the database which then writes the data out to the location of your choosing without storing the file contents. That way you still have a layer of abstraction between the file store and the application but you don't need to have a process which cleans up after you.
Alternatively a WCF/web service could be created which the application connects to. A web method could be used to accept the file contents and write them to the correct place, it could return the path to the file or some file identifier.
Related
I store large files (50-500MB) in database. Once loaded by the application, it doesn't need the whole file in memory. How do i fetch table row (or specifically the installer from the row) directly into the memory while avoiding loading the entire file into ram (So a sort of a buffered download into file)?
I haven't found a solution that avoid loading the file so far. Instead i forward requests to flask server that loads the entire file, and then allows the application instance to download it into a file. However this doesn't seem like a very good solution.
You are probably looking for FILESTREAM (SQL Server):
FILESTREAM enables SQL Server-based applications to store unstructured data, such as documents and images, on the file system. Applications can leverage the rich streaming APIs and performance of the file system and at the same time maintain transactional consistency between the unstructured data and corresponding structured data.
It is interesting because on SQL Server (for Windows) it can stream file data to Windows clients without having to load their entirety into the memory of the SQL Server:
The Win32 streaming support works in the context of a SQL Server transaction. Within a transaction, you can use FILESTREAM functions to obtain a logical UNC file system path of a file. You then use the OpenSqlFilestream API to obtain a file handle. This handle can then be used by Win32 file streaming interfaces, such as ReadFile() and WriteFile(), to access and update the file by way of the file system.
Do note that at this time it is not supported on SQL Server 2017 for Linux.
I am writing an asp.net web application that stores APPLICANTS data in a SQL Server database.
Applicant might post name, address, telephone and a file.
The file might be of any extension including .docx for resume, 'jpg, .pdf for photos.
or even an Excel file.
Is it possible to store all these file extension on my database?
Or will that be lengthy?
Please help
Good question! Personally I would use FILESTREAM in your case and here's why
In SQL Server, BLOBs can be standard varbinary(max) data that stores
the data in tables, or FILESTREAM varbinary(max) objects that store
the data in the file system. The size and use of the data determines
whether you should use database storage or file system storage. If the
following conditions are true, you should consider using FILESTREAM:
Objects that are being stored are, on average, larger than 1 MB.
Fast read access is important.
You are developing applications that use a middle tier for application logic.
For smaller objects, storing varbinary(max) BLOBs
in the database often provides better streaming performance.
You can read up on FILESTREAM here.
Also consider using it in conjunction with FILETABLE.
Finally, here's a .net C# example on how to read from FILESTREAM column.
Please note, FILESTREAM is available in SQL Server starting from 2008 version.
Hope it helps!
What would be the best way to insert metadata into a database that need to be logicaly connected files that are stored locally on a web-server?
In general, databases control their own storage. The proper procedure is to load data into tables in the database. This is important, because databases manage storage and memory. In a typical configuration, you don't want to be accessing files being updated by another application. And, you typically don't want to be storing database data over the network.
The general answer to the question is that you want to load data into the database.
That said, many database engines allow you to remotely access data in other databases or through a technology such as ODBC. You can get drivers for flat files, even those stored remotely on the network. However, this is not an optimal set up for querying. Alternatively, databases can be used to manage metadata for remote files, such as image files stored on disk. The purpose is to allow searches through the metadata which, in essence, retrieve file names that are then resolved (either on the client side or server side, depending on the architecture).
You should, perhaps, ask another question with a lot more detail about what you are trying to accomplish and about which database you are using.
I am using T-SQL and Microsoft Management Studio 2008 R2. I want to create a database in which I can store video files.
After google search and some reading I have learned that there is a option to use "File Stream Enable Database". It was said that this kind of database should be used only when your files are larger then 2MB. I want to store video files, so I think this is suitable for my goals.
Please, give me more information about the main difference in using BLOB and FileStream Enable database or just to store the files in a given directory and to save only the url in the database table column?
Thanks in advance.
Filestream was an interesting change when it came in for me; the bit that suprised me was Full Text Search was taken out of the operating system because it caused issues; but file stream put it back because Blobs caused issues.
Using Filestream is basically transparent to your application and it even backs the files up as if they were in the database - and thats the big benefit or cost over the save in database v save pointer in database.
You can insert files the same way as you did before and you can read them back in SQL in exactly the same way. The difference and benefit is that that SQL can take advantage of Windows system cache for reading and files saving its own resources to make other queries run quicker.
Please, give me more information about the main difference in using BLOB and FileStream Enable
database
The feature you call for is "FileStream" not "FileStream enable".
Some blogs are also around, like http://blogs.msdn.com/b/rdoherty/archive/2007/10/12/getting-traction-with-sql-server-2008-filestream.aspx
At kleast try reading the documentation before running around and have other people do your basic groundwork.
What I mean is being able to access it through Windows Explorer or other programs. I believe the answer is that it isn't possible. But I really want to know why it's not allowed. It seems that the files could be made available read-only through the network share.
You can't access the Filestream share directly and explore around. Any open to a Filestream file needs to be done using the path retrieved from SQL Server and by using NtCreateFile (or a wrapper) with the appropriate transaction context passed in through the EABuffer.
It is possible to create a new share and point it to the physical location of the files, however this is pretty pointless as there's no supported way to resolve a SQL Filestream row to a physical file location (the RsFx filter driver handles these conversions internally), the file location may change at any time due to concurrent updates / partition changes, and you'll need to relax security on the folder to an unacceptable level. It can also cause corruptions in the database if you move or delete files without the knowledge of SQL Server. Any locks held on physical files will interfere with deletes as mentioned in dportas' comment.
I agree it would be great to be able to browse a namespace of the Filestream files through explorer and open files directly through applications without requiring an application rewrite.
Yes it is possible. The point of filestream however is that you get that access via the filestream API rather than direct through the filesystem. Bear in mind that the file name could change without warning - for example updates may cause a new filestream file to be created. Possibly if you are holding file system locks (even shared locks) on a file that is needed by SQL Server then that may cause a contention problem. So if you access the data direct through the file system the results will be unsupported and may be unreliable - but then again it might work :-)
Yes it is possible if you are also using FileTables (I am using Sql Express 2017). When in Sql Server Configuration Manager, right click on your server instance, select Properties, and then go to the FILESTREAM tab. Check the "Allow remote clients access to FILESTREAM data". You may have to stop/start your instance. Now you can browse to the share, which is named according to your instance (in my case SqlExpress). In my database (SimioPortal) I had created a file (BlobStore) where I stored my files.
So, at the command prompt I can now type: dir \localhost\sqlexpress\SimioPortal\blobstore and see a list of my files. You can do a similar thing in File Explorer.