I am downloading class objects that also contain images to the flutter app and store them locally. Simple data such as strings and numbers are stored in a hive database.
However, what is the best way to store the images? I found some comments that storing them directly in hive is not a good idea and conversion to base64 and Uint8List seem to emphasize this (actually is Uint8List needed or could I directly store base64 to hive - as appears to be possible with Mongo?).
The most obvious choice for me would then to store the images as a file and only keep the path/filename as a reference in hive.
Or are there better ways of doing?
I have looked-up quite a few questions and answers, but they dealt about either about how to store them in sqlite, hive etc. or about temporarily cached images or about static images that come with the app.
I'm wondering how could I go about storing images to a database. Is there a way to store the actual image to a sql or nosql database and then just query the image or would I need to only send the path of a image and then store all images in a folder and direct the image tag to that image. Is there any technologies that will allow me to put the actual image in the cloud and make a get call to it?
Some general direction: If you are already using a SQL database, don't want to change the database architecture, and everything else about that database is fine - then you may want to use a few option (remember this is if you are not used a NoSQL DB):
a table of paths to the location of the image on your server
a table of URLs to static URLs of where the images are being stored
If you are using a NoSQL then storing compressed images in your database may be fine. Some thing to keep in mind:
when storing images directly in your database, you can start to add a lot of load on your engine
images are large (> 1 MB average) and will easily start to occupy space fast
There are many other practices to keep in mind when choosing. I personally would use some cloud service to reference a static URL that links to my images. This creates a bit more portability and reduces the total size of the table greatly. Check out this discussion for some more details.
As a part of the registration prgress of my app, the user can select a profile picture. My first idea was to store the picture in my database, but I read this isn't a good idea. I think the other solution is to upload the image to the filesystem and store the path in the db. Is this the right way to do this and is it save to store an unknown file in my system?
You have to check Always what files are you accepting with your upload function.
As for images check this post
It is good idea to store pics in database specially when there is some related data with it..
storing pics in filesystem can cause concurrency and other issues which you would have handle yourself.
Storing images and other binary assets in the database can be a pain - you can't easily view them from the SQL command line, the process of encoding and decoding the image can be a performance problem, and there are relatively few database operations you can do on the binary blob.
However, there are also a number of benefits - the image is backed up along with the rest of your database, and in a load balanced server farm, you don't have to worry about replicating the images across multiple web server file systems.
To get the best of both worlds, I'd recommend storing the images in the database, but caching them on the web server's filesystem. That way, if you add more servers to your farm, the cache will be built up as the image is requested. By caching the image, you don't pay the "decoding" performance penalty on each request.
You should NEVER trust anything your users do - you should at the very least restrict the file types users can upload to images (.png, .jpg, etc.). If you're running a large, public website, consider running uploaded files through a virus scanner - there aren't any vulnerabilities in image file types right now that I'm aware of, but there were some a few years ago, and they may return.
I searched for this but only found ways to save the image as a blob to the database.
What I would like to do is to save the image location in the database and then retrieve the file automatically from the location instead of saving it as a blob into the database.
Does this make any sense at all? Or is it better and faster and cheaper to just save the image files into the database as a blob?
Depending on the locale i might have to get a different picture.
Thanks for your help!
I have experience with both (saving an image as a blob in a RDBMS) and only storing the link to it in a filesystem/url manner. What I have come to realize is the first approach is plain no scalable.
Here is a rather biased list of things about each approach.
Approach 1. Saving images as blobs:
Cons:
When the number of images increase, so does your database size and
you are limited to the filesystem your RDBMS engine runs on.
When you want to retrieve a large number of these blobs, and if they
are big in size, you waste IO/bandwdith and put a strain on your
RDBMS engine. You ideally want it to have short queries that execute
fast and move a little amount of bytes around. You just can't get
that if you save the data as a BLOB in your relational database.
While some might argue that for repeatable queries caching will
help, I will argue that I if those huge chunks of data weren't there
in the first place, I wouldn't have to put them in cache.
There is no reliable way for a db admin/ content manager to easily
retrieve the contents a blob that is in db, for example, to verify
if an image is broken. He would have to connect to the db and
extract the BLOB bytes in some format and then view it. Or
alternatively you can build some page to do that for him but that
would be a badly put together gimmicks in my honest opinion.
Pros:
You don't have to rely on file systems being available or external
systems on which you host your images to be available. You would
probably write a bit less code and you will have more control over
your code since all the stuff you want is in your RDBMS.
Approach 2. Saving images as a link to a filesystem/urls
Pros:
Greatly alleviates performance strain on your RDBMS engine.
If you store the images as links, a system admin/ content manager
can easily check them by just copying the link in a browser and
verifying it renders properly.
If you don't use an external image hosting service but rather an
internal, you still retain a great amount of control while having
the possibility in future to add more image hosting servers/
filesystems.
If you have a large amount of pictures being retrieved and they are
not hosted by you, you can distribute a lot of network load thus
making load times snappier.
Cons:
Things will be a bit decentralized adding some complexity to your application. If you are using an external hosting service, it might be down and you can have no control over it.
In conclusion, I wholeheartedly recommend using the second approach.
In general I agree with #baba's answer.
However it really depends on the number of and sizes of the images. If all the images are small thumbnails then I would store them in the database, so that everything is in one place.
It's also possible to do both...as long as the storage space is available for both the database and filesystem. This gives you the best of both worlds and a built-in backup.
This question already has answers here:
Closed 13 years ago.
Possible Duplicates:
Which is more secure: filesystem or database?
User images - database vs. filesystem storage
store image in database or in a system file ?
I can't decide which one I should follow. Can you guys give some opinions? Should I store my images in the file-system or DB? (I would like to prevent others from stealing my images)
When you answer this question, please include comparisons of the security, performances etc.
Thanks.
Exact Duplicate: User Images: Database or filesystem storage?
Exact Duplicate: Storing images in database: Yea or nay?
Exact Duplicate: Should I store my images in the database or folders?
Exact Duplicate: Would you store binary data in database or folders?
Exact Duplicate: Store pictures as files or or the database for a web app?
Exact Duplicate: Storing a small number of images: blob or fs?
Exact Duplicate: store image in filesystem or database?
Moving your images into a database and writing the code to extract the image may be more hassle than it's worth. It will all go back to the business requirements surrounding the need to protect the images, or the requirement for performance.
I'd suggest sticking to the tried and true system of storing the filepath or directory location in the DB, and keeping the files on disk. Here's why:
A filesystem is easier to maintain. Some thought has to be put into the structure and organization of the images. i.e. a directory for each customer, and a subdirectory for each [Attribute-X] and another subfolder for each [Attribute-Y]. Keeping too many images in one directory will end up slowing down the file access (i.e. hundreds of thousands)
If the idea of storing in a DB is a counter-measure against filesystem loss, (i.e. a disk goes down, or a directory is deleted by accident), then I'd counter with the suggestions that when you use source control, it's no problem to retrieve any lost/missing/delete files.
If you ever need to scale and move to a content distribution scenario, you'd have to move out back to the filesystem or perform a big extract to push out to the providers.
It also goes with the saying: "keep structured data in a database". Microsoft has an article on Managing Unstructured Data.
If security is an issue to be addressed, the filesystem has a whole structure with ACLs. Reinventing the wheel on security may be out of scope in the business requirements.
A large amount of discussion for this topic is also found at:
Question 3748
Question 561447
Having your images stored as varbinary or a blob of some kind (depending on your platform), I'd suggest it's more hassle than it's worth. The effort that you'll need to extend means more code that you'll have to maintain, unit test, and defend against defects.
If your environment can support SQL Server 2008, you can have the best of both worlds with their new FileStream datatype in SQL 2008.
An MSDN article is touting the FileStream datatype in SQL 2008 as high performance.
SQL Skills has a great article with some SQL 2008 Filestream performance measurements.
Here is an article addressing varbinary vs. FileStream and performance of both datatypes.
If you are a SQL Mag subscriber, you can see a great article at SQL Mag on SQL 2008 FileStream.
Microsoft Research article:To Blob or Not To Blob
I'd love to see studies in real-world scenarios with large user bases like Flickr or Facebook.
Again, it all goes back to your business requirements. Good luck!
It doesn't matter where you store them in terms of preventing "theft". If you deliver the bytestream to a browser, it can be intercepted and saved by anyone. There's no way to prevent that (I'm assuming you're talking about delivering the images to a browser).
If you're just talking about securing images on the machine itself, it also doesn't matter. The operating system can be locked down as securely as the database, preventing anyone from getting at the images.
In terms of performance (when presenting images to a browser), I personally think it'll be faster serving from a filesystem. You have to present the images in separate HTTP transactions anyway, which would almost certainly require multiple trips to the database. I suspect (although I have no hard data) that it would be faster to store the image URLs in the database which point to static images on the file system - then the act of getting an image is a simple file open by the web server rather than running code to access the database.
You're probably going to have to get a whole ton of "but the filesystem is a DB" answers. This isn't one of them.
The filesystem option depends on many factors, for example, does the server have write premissisons to the directory? (And yes, I have seen servers where apache couldn't write to DocumentRoot.)
If you want 100% cross-compatibility across platforms, then the Database option is the best way to go. It'll also let you store image-related metadata such as a user ID, the date uploaded, and even alternate versions of the same image (such as cached thumbnails).
On the down side, you need to write custom code to read images from the DB and serve them to the user, while any standard web server would just let you send the images as they are.
When it comes to the bottom line, though, you should just choose the option that fits your project, and your server configuration.
Store them in FileSystem, store the file path in the DB.
Of course you can make this scalable and distributed, you just need to keep the images dirs synched between them (for JackM). Or use a shared storage connected to multiple web frontend servers.
Anyway, the stealing part was covered in your other question and is basically impossible. People that can access the images will always be able (with more or less work) to save them locally ... even if it means "print-screen" and paste into photoshop and saving.
It depends on how many images you expect to handle, and what you have to do with them. I have an application that needs to temporarily store between 100K and several million images a day. I write them in 2gb contiguous blocks to the filesystem, storing the image identifier, filename, beginning position and length in a database.
For retrieval I just keep the indices in memory, the files open read only and seek back and forth to fetch them. I could find no faster way to do it. It is far faster than finding and loading an individual file. Windows can get pretty slow once you've dumped that many individual files into the filesystem.
I suppose it could contribute to security, since the images would be somewhat difficult to retrieve without the index data.
For scalability, it would not take long to put a web service in front of it and distribute it across several machines.
For a web application I look after, we store the images in the database, but make sure they're well cached in the filesystem.
A request from one of the web server frontends for an image requires a quick memcache
check to see if the image has changed in the database and, if not, serves it from the filesystem. If it has changed it fetches it from the central database and puts a copy in the filesystem.
This gives most of the advantages of storing them in the filesystem while keeping some
of the advantages of database - we only have one location for all the data which makes
backups easier and means we can scale across quite a few machines without issue. It
also doesn't put excessive load on the database.
If you want your application to be scalable, do not use a file system on the actual web servers. You can store the location of files in a persistent datastore such as a database or a NoSQL solution.
For an AWS solution to this for example you should:
Store the images on S3
Save the S3 key to the database
Serve yourimages on S3 through cloudfront (Amazon CDN)
Saving your files to the DB will provide a some security in terms that another user would need access to the DB in order to retrieve the files, but, as far as efficiency goes, a sql query for every image loaded, leaving all the load to the server side. Do yourself a favor and find a way to protect your images inside the filesystem, they are many.
The biggest out-of-the-box advantage of a database is that it can be accessed from anywhere on the network, which is essential if you have more than one server.
If you want to access a filesystem from other machines you need to set up some kind of sharing scheme, which could add complexity and could have synchronization issues.
If you do go with storing images in the database, you can still use local (unshared) disks as caches to reduce the strain on the DB. You'd just need to query the DB for a timestamp or something to see if the file is still up-to-date (if you allow files that can change).
If the issue is scalability you'll take a massive loss by moving things into the database. You can round-robin webservers via DNS but adding the overhead of both a CGI process and a database lookup to each image is madness. It also makes your database that much harder to maintain and your images that much harder to process.
As to the other part of your question, you can secure access to a file as easily as a database record, but at the end of the day as long as there is an URL that returns a file you have limited options to prevent that URL being used (at least without making cookies and/or javascript compulsory).
Store files in a file server, and store primitive data in a database. While file servers (especially HTTP-based) scale well, database servers do not. Don't mix them together.
If you need to edit, manage, or otherwise maintain the images, you should store it outside the database.
Also, the filesystem has many security features that a database does not.
The database is good for storing pointers (file paths) to the actual data.