Related
Well we all know how many arguments and lives have been lost with the discussion of using databases for file storage (images specifically). I'm in a bit of a pickle when it comes to deciding on how to proceed with my project.
I have a website that allows admins to upload employee pictures. For now, these pictures are stored in BLOB in my MySQL database. Also, I have a windows application that runs alongside the website. This application enables employees to punch in and have their pictures appear when they've successfully done so. The picture is retrieved via a mysql query within the application (from a non-local remote location) that converts the image content to a readable image that's being outputted in a picture box, confirming the identity of the employee.
In my eyes, it is much much easier to have the images stored in the database and have them retrieved via a simple query. I found this a lot easier than storing image paths in the database and having to deal with the application downloading the images. I also don't have to deal with collisions, folder organization and security and paths being re-written for x,y reasons, etc etc.
The images stored in the DB are a mere 20 kb after being cropped to a certain size. My question is, is it still worth embedding the database with image paths or should they simply be stored as they are right now? If storing images in the database is still ill-advised in this case, is there a formal way to store image paths?
Any help on this would be greatly appreciated. If this question doesn't belong here, I'll be happy to move it.
If the images are user data, rather than part of your application's code or theme, then storing the images in the database is a good idea, because…
Backups are easier to manage if all you have to back up is the database. On the other hand, if you store some application data in the database and some in the filesystem, then you'll have to coordinate the backup schedules of your database and your filesystem to ensure that the two are consistent.
If you have a database administrator at your disposal, then great! Your backups should already be taken care of. If not, then database backups may be slightly tricky to set up, but once you do have a backup system, it can be better than filesystem backups. For example, many database systems have support for streaming replication.
If your application is load-balanced and served by a pool of multiple webservers, then you'll either have to replicate the data to all of the machines, or share them among your servers using a network filesystem.
Of course, having the images on a filesystem also has its advantages, namely in performance and simplicity, since most webservers are built to serve static files. A hybrid approach could give you the best of both worlds:
The images stored in the database would be the authoritative data.
Your application can have a feature to extract them as files in their local filesystem as a kind of cache. That cache can be rebuilt at any time, since it is not authoritative.
The webserver can then serve the files directly from the filesystem.
There are several reasons why I think storing images in a database is a bad idea:
1) The server will have timestamp info associated with files that the database won't keep track of. if you ever need to this for forensics reasons, the DB solution will likely be limited in this regard. Feel free to save info about images uploaded regarding IP info, timestamp, etc. in the DB though too.
2) If you ever want these files used by, say, another system/service you'll have to constantly reference the database and interact with it, when you could far more easily just target a specific folder.
3) Any time an image needs to be retrieved, you have to open a connection to the database just to generate it. This may add extra code and steps to things that could be easier to implement by pointing to a folder.
To avoid naming collisions, if I were on a Linux box, I'd use something like a Unix timestamp as a prefix to the filename when it's saved, or simply use that (+ maybe a short random #) as the image ID altogether. So instead of 'jane-image.jpg', it'd be '1407369600_img3547.jpg'. Then, just put a reference to that in the DB and viola, that's a random enough ID where there should never be a collision, unless time starts flowing backwards. Whatever the Windows timestamp equivalent is would be used, obviously.
NOTE: What you're doing now isn't bad and from what it sounds like it may work best for you...but generally speaking I try not to put everything in the hands of a database, just because I can. But that's me :)
I searched for this but only found ways to save the image as a blob to the database.
What I would like to do is to save the image location in the database and then retrieve the file automatically from the location instead of saving it as a blob into the database.
Does this make any sense at all? Or is it better and faster and cheaper to just save the image files into the database as a blob?
Depending on the locale i might have to get a different picture.
Thanks for your help!
I have experience with both (saving an image as a blob in a RDBMS) and only storing the link to it in a filesystem/url manner. What I have come to realize is the first approach is plain no scalable.
Here is a rather biased list of things about each approach.
Approach 1. Saving images as blobs:
Cons:
When the number of images increase, so does your database size and
you are limited to the filesystem your RDBMS engine runs on.
When you want to retrieve a large number of these blobs, and if they
are big in size, you waste IO/bandwdith and put a strain on your
RDBMS engine. You ideally want it to have short queries that execute
fast and move a little amount of bytes around. You just can't get
that if you save the data as a BLOB in your relational database.
While some might argue that for repeatable queries caching will
help, I will argue that I if those huge chunks of data weren't there
in the first place, I wouldn't have to put them in cache.
There is no reliable way for a db admin/ content manager to easily
retrieve the contents a blob that is in db, for example, to verify
if an image is broken. He would have to connect to the db and
extract the BLOB bytes in some format and then view it. Or
alternatively you can build some page to do that for him but that
would be a badly put together gimmicks in my honest opinion.
Pros:
You don't have to rely on file systems being available or external
systems on which you host your images to be available. You would
probably write a bit less code and you will have more control over
your code since all the stuff you want is in your RDBMS.
Approach 2. Saving images as a link to a filesystem/urls
Pros:
Greatly alleviates performance strain on your RDBMS engine.
If you store the images as links, a system admin/ content manager
can easily check them by just copying the link in a browser and
verifying it renders properly.
If you don't use an external image hosting service but rather an
internal, you still retain a great amount of control while having
the possibility in future to add more image hosting servers/
filesystems.
If you have a large amount of pictures being retrieved and they are
not hosted by you, you can distribute a lot of network load thus
making load times snappier.
Cons:
Things will be a bit decentralized adding some complexity to your application. If you are using an external hosting service, it might be down and you can have no control over it.
In conclusion, I wholeheartedly recommend using the second approach.
In general I agree with #baba's answer.
However it really depends on the number of and sizes of the images. If all the images are small thumbnails then I would store them in the database, so that everything is in one place.
It's also possible to do both...as long as the storage space is available for both the database and filesystem. This gives you the best of both worlds and a built-in backup.
This question already has answers here:
Closed 13 years ago.
Possible Duplicates:
Which is more secure: filesystem or database?
User images - database vs. filesystem storage
store image in database or in a system file ?
I can't decide which one I should follow. Can you guys give some opinions? Should I store my images in the file-system or DB? (I would like to prevent others from stealing my images)
When you answer this question, please include comparisons of the security, performances etc.
Thanks.
Exact Duplicate: User Images: Database or filesystem storage?
Exact Duplicate: Storing images in database: Yea or nay?
Exact Duplicate: Should I store my images in the database or folders?
Exact Duplicate: Would you store binary data in database or folders?
Exact Duplicate: Store pictures as files or or the database for a web app?
Exact Duplicate: Storing a small number of images: blob or fs?
Exact Duplicate: store image in filesystem or database?
Moving your images into a database and writing the code to extract the image may be more hassle than it's worth. It will all go back to the business requirements surrounding the need to protect the images, or the requirement for performance.
I'd suggest sticking to the tried and true system of storing the filepath or directory location in the DB, and keeping the files on disk. Here's why:
A filesystem is easier to maintain. Some thought has to be put into the structure and organization of the images. i.e. a directory for each customer, and a subdirectory for each [Attribute-X] and another subfolder for each [Attribute-Y]. Keeping too many images in one directory will end up slowing down the file access (i.e. hundreds of thousands)
If the idea of storing in a DB is a counter-measure against filesystem loss, (i.e. a disk goes down, or a directory is deleted by accident), then I'd counter with the suggestions that when you use source control, it's no problem to retrieve any lost/missing/delete files.
If you ever need to scale and move to a content distribution scenario, you'd have to move out back to the filesystem or perform a big extract to push out to the providers.
It also goes with the saying: "keep structured data in a database". Microsoft has an article on Managing Unstructured Data.
If security is an issue to be addressed, the filesystem has a whole structure with ACLs. Reinventing the wheel on security may be out of scope in the business requirements.
A large amount of discussion for this topic is also found at:
Question 3748
Question 561447
Having your images stored as varbinary or a blob of some kind (depending on your platform), I'd suggest it's more hassle than it's worth. The effort that you'll need to extend means more code that you'll have to maintain, unit test, and defend against defects.
If your environment can support SQL Server 2008, you can have the best of both worlds with their new FileStream datatype in SQL 2008.
An MSDN article is touting the FileStream datatype in SQL 2008 as high performance.
SQL Skills has a great article with some SQL 2008 Filestream performance measurements.
Here is an article addressing varbinary vs. FileStream and performance of both datatypes.
If you are a SQL Mag subscriber, you can see a great article at SQL Mag on SQL 2008 FileStream.
Microsoft Research article:To Blob or Not To Blob
I'd love to see studies in real-world scenarios with large user bases like Flickr or Facebook.
Again, it all goes back to your business requirements. Good luck!
It doesn't matter where you store them in terms of preventing "theft". If you deliver the bytestream to a browser, it can be intercepted and saved by anyone. There's no way to prevent that (I'm assuming you're talking about delivering the images to a browser).
If you're just talking about securing images on the machine itself, it also doesn't matter. The operating system can be locked down as securely as the database, preventing anyone from getting at the images.
In terms of performance (when presenting images to a browser), I personally think it'll be faster serving from a filesystem. You have to present the images in separate HTTP transactions anyway, which would almost certainly require multiple trips to the database. I suspect (although I have no hard data) that it would be faster to store the image URLs in the database which point to static images on the file system - then the act of getting an image is a simple file open by the web server rather than running code to access the database.
You're probably going to have to get a whole ton of "but the filesystem is a DB" answers. This isn't one of them.
The filesystem option depends on many factors, for example, does the server have write premissisons to the directory? (And yes, I have seen servers where apache couldn't write to DocumentRoot.)
If you want 100% cross-compatibility across platforms, then the Database option is the best way to go. It'll also let you store image-related metadata such as a user ID, the date uploaded, and even alternate versions of the same image (such as cached thumbnails).
On the down side, you need to write custom code to read images from the DB and serve them to the user, while any standard web server would just let you send the images as they are.
When it comes to the bottom line, though, you should just choose the option that fits your project, and your server configuration.
Store them in FileSystem, store the file path in the DB.
Of course you can make this scalable and distributed, you just need to keep the images dirs synched between them (for JackM). Or use a shared storage connected to multiple web frontend servers.
Anyway, the stealing part was covered in your other question and is basically impossible. People that can access the images will always be able (with more or less work) to save them locally ... even if it means "print-screen" and paste into photoshop and saving.
It depends on how many images you expect to handle, and what you have to do with them. I have an application that needs to temporarily store between 100K and several million images a day. I write them in 2gb contiguous blocks to the filesystem, storing the image identifier, filename, beginning position and length in a database.
For retrieval I just keep the indices in memory, the files open read only and seek back and forth to fetch them. I could find no faster way to do it. It is far faster than finding and loading an individual file. Windows can get pretty slow once you've dumped that many individual files into the filesystem.
I suppose it could contribute to security, since the images would be somewhat difficult to retrieve without the index data.
For scalability, it would not take long to put a web service in front of it and distribute it across several machines.
For a web application I look after, we store the images in the database, but make sure they're well cached in the filesystem.
A request from one of the web server frontends for an image requires a quick memcache
check to see if the image has changed in the database and, if not, serves it from the filesystem. If it has changed it fetches it from the central database and puts a copy in the filesystem.
This gives most of the advantages of storing them in the filesystem while keeping some
of the advantages of database - we only have one location for all the data which makes
backups easier and means we can scale across quite a few machines without issue. It
also doesn't put excessive load on the database.
If you want your application to be scalable, do not use a file system on the actual web servers. You can store the location of files in a persistent datastore such as a database or a NoSQL solution.
For an AWS solution to this for example you should:
Store the images on S3
Save the S3 key to the database
Serve yourimages on S3 through cloudfront (Amazon CDN)
Saving your files to the DB will provide a some security in terms that another user would need access to the DB in order to retrieve the files, but, as far as efficiency goes, a sql query for every image loaded, leaving all the load to the server side. Do yourself a favor and find a way to protect your images inside the filesystem, they are many.
The biggest out-of-the-box advantage of a database is that it can be accessed from anywhere on the network, which is essential if you have more than one server.
If you want to access a filesystem from other machines you need to set up some kind of sharing scheme, which could add complexity and could have synchronization issues.
If you do go with storing images in the database, you can still use local (unshared) disks as caches to reduce the strain on the DB. You'd just need to query the DB for a timestamp or something to see if the file is still up-to-date (if you allow files that can change).
If the issue is scalability you'll take a massive loss by moving things into the database. You can round-robin webservers via DNS but adding the overhead of both a CGI process and a database lookup to each image is madness. It also makes your database that much harder to maintain and your images that much harder to process.
As to the other part of your question, you can secure access to a file as easily as a database record, but at the end of the day as long as there is an URL that returns a file you have limited options to prevent that URL being used (at least without making cookies and/or javascript compulsory).
Store files in a file server, and store primitive data in a database. While file servers (especially HTTP-based) scale well, database servers do not. Don't mix them together.
If you need to edit, manage, or otherwise maintain the images, you should store it outside the database.
Also, the filesystem has many security features that a database does not.
The database is good for storing pointers (file paths) to the actual data.
When you save your images (supose you have lots of them) do you store then as blobs in your Database, or as files? Why?
Duplicate of: Storing Images in DB - Yea or Nay?
I usually go with storing them as files, and store the path in the database. To me, it's a much easier and more natural approach than pushing them into the database as blobs.
One argument for storing them in the database: much easier to do full backups, but that depends on your needs. If you need to be able to easily take full snapshots of your database (including the images), then storing them as blobs in the database is probably the way to go. Otherwise you have to pair your database backup with a file backup, and somehow try to associate the two, so that if you have to do a restore, you know which pair to restore.
It depends on the size of the image.
Microsoft Research has an interesting document on the subject
I've tried to use the db (SQL Server and MySQL) to store medium (< 5mb) files, and what I got was tons of trouble.
1) Some DBs (SQL Server Express) have size limits;
2) Some DBs (MySQL) become mortally slow;
3) When you have to display a list of object, if you inadvertedly do SELECT * FROM table, tons of data will try to go up and down from the db, resulting in a deadly slow response or memory fail;
4) Some frontends (ruby ActiveRecord) have very big troubles handling blobs.
Just use files. Don't store them all in the same directory, use some technique to put them on several dirs (for instance, you could use last two chars of a GUID or last two digits of an int id) and then store the path on db.
The performance hit of a database server is a moot issue. If you need the performance benefits of a file system, you simply cache it there on the first request. Subsequent requests can then be served directly from the file system by a direct link (which, in case of a web app, you could rewrite the HTML with before flushing the output buffer).
This provides the best of both worlds:
The authoritative store is the
database, keeping transactional and
referential integrity
You can deploy all user data by
simply deploying the database
Emptying this cache (e.g. by adding a
web server) would only cause a
temporary performance hit while it is
refilled automatically.
There is no need to constantly hammer the database for things that won't change all the time, but the important thing is that the user data is all there and not scattered around different places, making multi-server operation and deployment a total mess.
I'm always advocating the "database as the user data store, unless" approach, because it is better architecturally, and not necessarily slower with effective caching.
Having said that, a good reason to use the file system as the authoritative store would be when you really need to use external independent tools for accessing it, e.g. SFTP and whatnot.
Given that you might want to save an image along with a name, brief description, created date, created by, etc., you might find it better to save in a database. That way, everything is together. If you saved this same info and stored the image as a file, you would have to retrieve the whole "image object" from two places...and down the road, you might find yourself having syncing issues (some images not being found). Hopefully this makes sense.
By saving you mean to use them to show in a webpage or something like that?
If it's the case, the better option will be to use files, if you use a database it will be constantly hammered with the request for photos. And it's a situation that doesn't scale too well.
The question is, does your application handle BLOBS or other files like other application data? Do your users upload images alongside other data? If so, then you ought to store the BLOBs in the database. It makes it easier to back up the database and, in the event of a problem, to recover to a transactionally consistent state.
But if you mean images which are part of the application infratstructure rather than user data then probably the answer is, No.
If I'm running on one web server and will only ever be running on one web server, I store them as files. If I'm running across multiple webheads, I put the reference instance of the image in a database BLOB and cache it as a file on the webheads.
Blobs can be heavy on the db/scripts, why not just store paths. The only reason we've ever used blobs is if it needs to be merge replicated or super tight security for assets (as in cant pull image unless logged in or something)
I would suggest to go for File systems. First, let's discuss why not Blob? So to answer that, we need to think what advantages DB provides us over File system?
Mutability: We can modify the data once stored. Not Applicable in case of images. Images are just a series of 1s and 0s. Whenever we changes an image, it wouldn't be a matter of few 1s and 0s altered and hence, modifying the same image content doesn't make sense. It's better to delete the old one, and store new.
Indexing: We can create indexes for faster searching. But it doesn't apply on images as images are just 1s and 0s and we can't index that.
Then why File systems?
Faster access: If we are storing images in Blob inside our DB, then a query to fetch the complete record (select *) will result in a very poor performance of the query as a lots and lots of data will be going to and from the DB. Instead if we just store the URL of images in DB and store images in a distributed file system (DFS), it will be much faster.
Size limit: If DBs are storing images, a lot and lot of images then it might face performance issues and also, reach its memory limit (few DBs do have it).
Using file System is better as the basic feature you would be provided with while storing images as a blob would be
1. mutability which is not needed for an image as we won't be changing the binary data of images, we will be removing images as whole only
2. Indexed searching :which is not needed for image as the content of images can't be indexed and indexed searching searches the content of the BLOB.
Using file system is beneficial here because
1. its cheaper
2. Using CDN for fast access
hence one way forward could be to store the images as a file and provide its path in database
I have been hired to help write an application that manages certain information for the end user. It is intended to manage a few megabytes of information, but also manage scanned images in full resolution. Should this project use a database, and why or why not?
Any question "Should I use a certain tool?" comes down to asking exactly what you want to do. You should ask yourself - "Do I want to write my own storage for this data?"
Most web based applications are written against a database because most databases support many "free" features - you can have multiple webservers. You can use standard tools to edit, verify and backup your data. You can have a robust storage solution with transactions.
The database won't help you much in dealing with the image data itself, but anything that manages a bunch of images is going to have meta-data about the images that you'll be dealing with. Depending on the meta-data and what you want to do with it, a database can be quite helpful indeed with that.
And just because the database doesn't help you much with the image data, that doesn't mean you can't store the images in the database. You would store them in a BLOB column of a SQL database.
If the amount of data is small, or installed on many client machines, you might not want the overhead of a database.
Is it intended to be installed on many users machines? Adding the overhead of ensuring you can run whatever database engine you choose on a client installed app is not optimal. Since the amount of data is small, I think XML would be adequate here. You could Base64 encode the images and store them as CDATA.
Will the application be run on a server? If you have concurrent users, then databases have concepts for handling these scenarios (transactions), and that can be helpful. And the scanned image data would be appropriate for a BLOB.
You shouldn't store images in the database, as is the general consensus here.
The file system is just much better at storing images than your database is.
You should use a database to store meta information about those images, such as a title, description, etc, and just store a URL or path to the images.
When it comes to storing images in a database I try to avoid it. In your case from what I can gather of your question there is a possibilty for a subsantial number of fairly large images, so I would probably strong oppose it.
If this is a web application I would use a database for quick searching and indexing of images using keywords and other parameters. Then have a column pointing to the location of the image in a filesystem if possible with some kind of folder structure to help further decrease the image load time.
If you need greater security due to the directory being available (network share) and the application is local then you should probably bite the bullet and store the images in the database.
My gut reaction is "why not?" A database is going to provide a framework for storing information, with all of the input/output/optimization functions provided in a documented format. You can go with a server-side solution, or a local database such as SQLite or the local version of SQL Server. Either way you have a robust, documented data management framework.
This post should give you most of the opinions you need about storing images in the database. Do you also mean 'should I use a database for the other information?' or are you just asking about the images?
A database is meant to manage large volumes of data, and are supposed to give you fast access to read and write that data in spite of the size. Put simply, they manage scale for data - scale that you don't want to deal with. If you have only a few users (hundreds?), you could just as easily manage the data on disk (say XML?) and keep the data in memory. The images should clearly not go in to the database so the question is how much data, or for how many users are you maintaining this database instance?
If you want to have a structured way to store and retrieve information, a database is most definitely the way to go. It makes your application flexible and more powerful, and lets you focus on the actual application rather than incidentals like trying to write your own storage system.
For individual applications, SQLite is great. It fits right in an app as a file; no need for a whole DRBMS juggernaut.
There are a lot of factors to this. But, being a database weenie, I would err on the side of having a database. It just makes life easier when things changes. and things will change.
Depending on the images, you might store them on the file system or actually blob them and put them in the database (Not supported in all DBMS's). If the files are very small, then I would blob them. If they are big, then I would keep them on he file system and manage them yourself.
There are so many free or cheap DBMS's out there that there really is no excuse not to use one. I'm a SQL Server guy, but f your application is that simple, then the free version of mysql should do the job. In fact, it has some pretty cool stuff in there.
Our CMS stores all of the check images we process. It uses a database for metadata and lets the file system handle the scanned images.
A simple database like SQLite sounds appropriate - it will let you store file metadata in a consistent, transactional way. Then store the path to each image in the database and let the file system do what it does best - manage files.
SQL Server 2008 has a new data type built for in-database files, but before that BLOB was the way to store files inside the database. On a small scale that would work too.