i'm working on a new project and looking into a good approach/ Best practise for file storage and how to efficiently map the files to several resources in a relational data model.
Note: All files are uploaded on a filesystem (not a database)
Option 1
Creating a Files table to store meta data of each file and create a relation with each resource (e.g. user_files, product_files, ... )
Questions: Will all files be stored in a single table? If I want to fetch the user avatar I have to search in a table that also contains product images or pdf documents?
Option 2
Store the needed file meta data with each resource.
Downside: each resource can only have 1 image. (good for user avatar?)
Option 3
Files are created using a hash of the resource (e.g. user_1243_avatar)
No relation will be stored and the url will be build when the resource is fetched.
Are there other options to consider?
all input is welcome.
It's good that you store files in the filesystem. I used option 1 in a recent project and would recommend it.
If I want to fetch the user avatar I have to search in a table that also contains product images or pdf documents?
You make it sound as if you'll have to do a table scan. In my DB, a user's avatar was indicated by an avatar_file_id in the users table, so I could join from users to files directly using an index.
Your user_files sounds too generic. Users could have different files for different purposes, are you just going to lump them all together? Would you create a table person_to_person to lump together friends, marriages, manager/employee, parent/child, etc? I wouldn't.
Create different tables for different purposes, e.g. a one-to-one table user_avatar (or denormalize it into your users table). Some more examples might be product_images, product_specs, product_helpfiles, etc.
When we use the export records function in Filemaker we can see header information in the file (i.e., using the merge format) but the table information is missing. Is there a way of keeping the table info in the exported file?
So for example we have a table named 'T3' but when we export fields from this table the resulting .csv file reads:
__Delirium_DRStotal_score
instead of
T3::__Delirium_DRStotal_score
Any help much appreciated,
Many thanks
Steve
FileMaker can export related fields too and theese preserve their table name on export. At least this happens when you export into XML. So if you self-join the table to itself by a unique ID and export (identical) data from the related copy, it should give you the names you want. (Almost, because you'll have to name that other table differently.)
When user upload image I resize that image in 3 different sizes. I can't decide what is better option:
1) Save image path in one column in database like: User9876/ImageName and extension in another column. When I get user profile data from database I have business object with three properties for different image size. And in code I fill this properties by adding extension to image path from database like:
User9876/ImageName_Original.jpg
User9876/ImageName_Small.jpg
User9876/ImageName_Smallest.jpg
2) Or it is better to have three columns in database for each size?
Couple of recommendations:
Don't store the extension separately, unless you plan on querying for particular extensions. Just adds more complexity for no real gain.
I would store the image size code as an id in your images table. This way you can update/edit image codes without having to run an update statement on your entire table.
create table image (
id int(11) unsigned not null auto_increment,
path varchar(255) not null,
size_code_id smallint(3) not null,
PRIMARY KEY (id)
)Engine=MyISAM Default Charset=utf8;
Example row:
1 | user9876/imagename_og.jpg | 1
2 | user9876/imagename_me.jpg | 2
3 | user9876/imagename_sm.jpg | 3
This design will allow you to query for the 'small' image without having to parse the filename, yet you can still make the distinction of a small image from medium and original images on the filesystem.
I think the answer to your question can be found by asking yourself another question: what is going to happen if you later choose to have not three, but four different image sizes? Clearly, the solution which does not require you to reorganize your entire database is better.
One of my apps is a Document Management System in which the documents are stored as blob fields into a db. This is not a language specific question, anyway I put Delphi in tags since this is the community to which i tipically ask questions (and many people that uses Delphi faces these problems).
One feature I need to add is to programmatically add some data to the document. I make a simple example just to get the idea. One field is the date in which the document has been created. For this the user will type "a tag" for example <DOCUMENT_DATE> and the date will be automatically substituted when the docuemnt is extracted from db.
So I have 2 main concerns. ONe is what to use as "tag". The simplest thing is to use a text tag, so simply typing into the docuemnt and then do Search & Replace text (using for example MS Word ActiveX). I already do this for other purposes. AN alternative could be using bookmarks or another technique.
The other question is strictly related with the previous.
How do I store it? My first idea is to store the document in DB with the "tags", so when it is "checked out" the user sees the tags, while when the user opens it (in readonly mode) he sees the subsituted text. (so in first case he sees and in the second "12 october 2011").
In this way I store the file once, but every time it is opened there is an overhead in processing it and doing the Search Replace thing, that can be also relatively slow. So this is why I asked for other techniques. Like serach replace for bookmark. The fastest the better.
The alternative is to store the document twice: once with the "tags" and the other with the "substituted veresion". This will be good for performance: no Searh & Replace but simply when the document is openeed in "checkout" mode I will open the one with the tags, while whne I open it in readonly mode I will open the subsituted one.
This of course takes more storage, for every document version (revision1, revision2, ...) I need to store 2 files.
I feel double storage is the best, because it won't affect perfomance at all, I mean it will be as fast as now, just the checkin process will be slower since I need to save 2 files and not one. Moreover by not enabling this auto substitution feature on all documents by default I won't have double db size.
But anyway I would like to hear some comments, since it is a quite crucial decision.
It really does not make sense to store identical data twice.
in fact it is a really bad idea, mainly from a consistancy point of view.
The way you do this is to store stuff in different tables and create links between the tables.
This is a process called normalization.
Here's an example loosely inspired by your post using MySQL:
TABLE document
--------------
id UNSIGNED INTEGER AUTO_INCREMENT PRIMARY KEY,
data BLOB
TABLE tag
------------
id UNSIGNED INTEGER AUTO_INCREMENT PRIMARY KEY,
tag VARCHAR(20)
TABLE tag_link
-------------------
tag_id UNSIGNED INTEGER,
reference_nr UNSIGNED INTEGER,
PRIMARY KEY (tag_id, reference_nr)
FOREIGN KEY (tag_id) REFERENCES tag(id) ON DELETE CASCADE ON UPDATE CASCADE,
FOREIGN KEY (reference_nr) REFERENCES post(reference_nr) ON DELETE CASCADE ON UPDATE CASCADE,
TABLE post
----------------
reference_nr UNSIGNED INTEGER NOT NULL,
revision UNSIGNED INTEGER NOT NULL DEFAULT 1,
document_id UNSIGNED INTEGER,
title VARCHAR(255),
creation_date TIMESTAMP,
other_fields .....
PRIMARY KEY (reference_nr, revision),
FOREIGN KEY (document_id) REFERENCES document(id) ON DELETE SET NULL ON CASCADE UPDATE
Now you can add tags to a post, all revisions of a post share the same tags.
Revisions of a post can link to the same document, or to different documents no need to duplicate data.
If you want to get all the lastest revisions of documents with certain tags, you use the following query:
SELECT p.title, d.data, GROUP_CONCAT(t.tag) AS tags
FROM post p
LEFT JOIN d.data ON (p.document_id = d.id)
INNER JOIN taglink tl ON (tl.reference_nr = p.reference_nr)
INNER JOIN tags t ON (tl.tag_id = t.id)
WHERE t.tag IN ('test','test2')
GROUP BY p.reference_nr /*only works in MySQL because other db's do not support ANSI SQL 2003*/
HAVING p.revision = MAX(p.revision)
ORDER BY p.creation_date DESC
I see two other possibilities worth considering.
1. Use RTF
If your document templates are Word documents, I'd rather store them as RTF.
RTF is just plain ASCII, and even if it is a proprietary format, it is well documented, and can be easily parsed. Word is able to save its content and read it as RTF. If you have pictures within, it can grow, but you can zip it before storing as BLOB in your database (and you may embed EMF pictures).
Then you can process those RTF content very fast in your code, changing all <DOCUMENT_DATE> using the latest version of the date field value.
I use this technique in several applications, and it gives very good results. See for instance how our SynProject tool generates Word documents from plain text, replacing tags, setting bookmarks or indexes on the fly. With RTF, you can do much more than just replacing a tag, but create a whole document easily.
For end-user input, you can use a basic TRichEdit or a more advanced (but not free) TRichView instead of Word.
You may consider using HTML instead of RTF, but it is much less printing-friendly.
2. Use a report engine
Another possibility could be to use a code-based report engine, then create PDF files.
Our Open Source units can be used from a simple reporting class to create easily the file content, preview it on screen and/or print/export as PDF. It is much easier than RTF to work with, but the layout has to be set in your code, or with text-based / wiki-like templates to be stored in your DB.
I am setting up full text search on an existing database. We have Document table with the following schema:
ID int Not Null,
Data varbinary(max) Not Null
MimeType varchar Not Null
I want to use full text search on the data column, using the mimetype to specify the document type.
I was hoping it would be possible to register new types into whatever tables are used when you use the view sys.fulltext_document_types. Is this possible?
No it isn't possible. IFilters are mapped to a document extension. A new column has to be added and populated.