I've read the many discussions about Databases vs. file systems for storing files. Most of these discussions talk about images and media files. My question is:
1) Do the same arguments apply to storing .doc, .pdf, .xls, .txt? Are there anything special about document files I should be aware of?
2) If I store in a database as binary, will there be endian issues if my host swaps machines? e.g., I insert into the database on a big-endian machine, it gets ported to an little-endian machine, then I try to extract (e.g., write to file, send it to my desktop, then try to open).
Thanks for any guidance!
1) Yes, pretty much the same arguments apply to storing PDFs and whatnot... anything that's compressed also comes to mind.
Every file format that's non-text has to deal with the question of endianness if it wants to be portable across hosts of different endianness. They mostly do it by defining what the endianness of all binary fields within the file that are longer than one byte should be. Software that writes and reads the format than has to take special care to byte-swap iff it's running on a platform of the opposite endianness. Images are no different than other binary file formats. The choice is arbitrary, but big endian (network byte order) is a popular choice especially with network software because of the ubiquity of macros in C that deal with this almost automatically.
Another way of defining binary file formats so that they are endian-portable is to support either endianness for binary fields, and include a marker in the header to say which one was used. On opening the file, readers consult the marker. That way the file can be read back slightly more efficiently on the same host where it was written or other hosts with the same endianness (which is the common case) while hosts of the opposite endianness need to expend a little bit more effort.
As for the database, assuming you are using a field type like a blob, you'll get back the very same bytestream when you read as whatever you wrote, so you don't have to worry about the endianness of the database client or server.
2) That depends on the database. The database might use an underlying on-disk format that is compatible with any endianness, by defining its on-disk format as described above.
Databases aren't often aiming for portability of their underlying file formats though, considering (correctly) that moving the underlying data files to a database host of different endianness is rare. According to this answer, for example, MySQL's MyISAM is not endian-portable.
I don't think you need to worry about this too much though. If the database server is ever switched to a host of different endianness, ensuring that the data remains readable is an important step of the process and the DBA handling the task (perhaps yourself?) won't forget to do it, because if they do forget, then nothing will work (that is, the breakage won't be limited to binary BLOBs!)
Related
File systems provide a mechanism for categorizing (and thus navigating) data on a disk. This makes sense to me. If I want to find some "group" of data, I don't want to have to remember byte offsets myself. I would rather have some look up system that I can dynamically navigate.
However, I don't understand why different file systems must exist. For example, why NTFS, FAT16/32, EXT?
Why should different operating systems (Linux, Windows, etc.) rely on different methods for organizing data on disk?
I think a more appropriate question (and the question you'd like answered) is "Why do multiple file systems exist?". The answer depends on the particular file system, but in many cases it comes down to one (or a mix of) of three reasons:
addressing some type of issue in existing file systems, or
a split due to difference in opinion, or
corporate interests.
The FAT family
The original FAT file system was introduced in the late 1970s. In many ways, FAT is great: it has a low memory footprint, and simple design. IIRC, it's still used in embedded systems to this date.
The FAT family of file systems comprises of the original 8-bit FAT, FAT12, FAT16, and FAT32. (There are several other versions, but they're not relevant to this answer.) There were several feature-differences between each version of the FAT file systems, some of which demonstrate the motivation for creating a new version. For example, in moving from 8-bit FAT to FAT12:
the maximum filename length increased from 9 characters to 11 or 255 characters by switching from 6.3 filename encoding to 8.3 filename encoding or LFN extensions, respectively.
support for subdirectories was added.
file size granularity decreased from 128 bytes to 1 byte.
None of these features individually were likely the motivation for the creation of FAT12, but together these features are a clear win over 8-bit FAT. Refer to the FAT Wikipedia page for a more complete list of differences.
NTFS
Before discussing NTFS, we should look at its predecessor: HPFS. The simple design of FAT turned out to be a problem: it constrained what features FAT could offer, and how it performed. HPFS was created to address the shortcomings of FAT. For example, HPFS provide several features FAT could not:
Support for mixed case file names, in different code pages
More efficient use of disk space (files are not stored using multiple-sector clusters but on a per-sector basis)
An internal architecture that keeps related items close to each other on the disk volume
Separate datestamps for last modification, last access, and creation (as opposed to last-modification-only datestamp in then-times implementations of FAT)
Root directory located at the midpoint, rather than at the beginning of the disk, for faster average access
That should be compelling enough to demonstrate why HPFS was created, but how does NTFS fit into the picture? HPFS was a joint project by Microsoft and IBM. Due to several differences in opinion, they separated, and Microsoft created NTFS. This is another reason new file systems are created: difference in opinion. There's nothing inherently wrong with this, but it does have the side effect of occasionally fragmenting projects.
The extended family
As with NTFS, we need to examine the predecessor of ext to understand why it was created. The predecessor of ext is the MINIX file system. MINIX was created for teaching purposes, so it was simple and elided several complex features the UNIX file system offered. The first file system supported by Linux was the MINIX filesystem. The simplicity of the MINIX file system soon became an issue:
MINIX restricted filename lengths to 14 characters (30 in later versions), it limited partitions to 64 megabytes, and the file system was designed for teaching purposes, not performance.
And thus, the extended file system (ie. ext) was created to address the shortcomings of the MINIX file system.
In a similar vain, ext2 was created to address the shortcomings of ext, and so on. For example, ext2 added three separate timestamps (atime, ctime, and mtime), ext3 adding journaling, and ext4 extended storage limits. These were all breaking changes which required a "new" file system. They weren't the only changes between versions, but these changes demonstrate why creating another file system was necessary.
Why do different operating systems use different file systems?
Several file systems are widely used today. Apple File System (APFS) on Apple devices, NTFS on Windows devices, and several different file systems on Linux. Why do different operating systems use different file systems? For Linux, the reason is obvious: Linux needed an open source file system. That's why it initially used the MINIX file system.
For Windows and Apple devices, the difference is more, shall we say, political. Microsoft created NTFS to address the issues it thought were important, and Apple created APFS to address the issues it thought were important. Commercial OS vendors also create their own file systems for product differentiation.
Why does Linux use several different file systems?
We can kinda see why different OSs use different file systems, but several file systems are actively in use on Linux alone, eg. ext4, Btrfs, ZFS, XFS, and F2FS. What gives?
Linux is a different environment. The Linux kernel source is openly available, and can be modified, booted, and tested by any user. So, if one file system does not support the features you want, or offer the performance you need, you can create a new file system (which is, of course, easier said than done). For example,
Btrfs addressed (among other things) the lack of snapshots on ext3/4.
ZFS was created for the Solaris operating system, but later ported to Linux. (ZFS also has a very rich set of features.)
XFS was created to improve performance by using different underlying data structures (ie. B-trees).
F2FS was created to address performance on solid state media. SSDs offer lower latency, and greater throughput compared to spinning disks. It turns out simply using a faster disk does not necessary equate to better file system performance.
different OS uses different FS Because each of them has a different philosophy and different goals.
For example windows use ntfs because they want secure and smart FS (without have philosophy like fast or small)
Ubuntu (with most modern distributions) use ext4 (And also supports others) Mostly because its simple and speed.
I don't think it's something technical, it's just different companies worked on the same thing at the same time plus the closed source nature of some OSs like windows and mac which make it hard for other companies to replicate the full functionality and illegal to reverse engineer it, it's like why different OSs in the first place.
I've been searching for hours and my google-fu has failed me so thought I'd just ask. Is there an easy and efficient way of breaking data into small chunks in C?
For example. If I collect a bunch of info from somewhere; database, file, user input, whatever. Then maybe use a serialization library or something to create a single large object in memory. I have the pointer to this object. Let's say somehow this object ends up being like... 500 kb or something. If your goal was to break this down into 128 byte sections. What would you do? I would like a kind of general answer, whether you wanted to send these chunks over network, store them in a bunch of little files, or pass them through a looped process or something. If there is not a simple process for all, but if there does exist some for specific use cases, that'd be cool to know too.
What has brought this question about: I've been learning network sockets and protocols. I often see discussion about packet fragmentation and the like. Lots of talk about chunking things and sending them in smaller parts. But I can never seem to find what they use to do this before they move on to how they send it over the network, which seems like the easy part... So I started wondering how large data would manually be broken up into chunks to send small bits over the socket at a time. And here we are.
Thanks for any help!
Is there an easy and efficient way of breaking data into small chunks in C?
Data is practically a consecutive sequence of bytes.
You could use memmove to copy or move it and slice it in smaller chunks (e.g. of 1024 bytes each). For non-overlapping data, consider memcpy. In practice, a byte is often a char (perhaps an unsigned char or a signed char) but see also the standard uint8_t type and related types. In practice, you can cast void* from or to char* on Von Neumann architectures (like x86 or RISC-V).
Beware of undefined behavior.
In practice I would recommend organizing data at a higher level.
If your operating system is Linux or Windows or MacOSX or Android, you could consider using a database library such as sqlite (or indexed files à la Tokyo Cabinet). It is open source software, and doing such slicing at the disk level.
If you have no operating system and your C code is freestanding (read the C11 standard n1570 for the terminology) things are becoming different. For example, a typical computer mouse contains a micro-controller whose code is mostly in C. Look into Arduino for inspiration (and also RaspBerryPi). You'll have to then handle data at the bit level.
But I can never seem to find what they use to do this before they move on to how they send it over the network, which seems like the easy part...
You'll find lots of open source network code.
The Linux kernel has some. FreeRTOS has some. FreeBSD has some. Xorg has some. Contiki has some. OSdev links to more resources (notably on github or gitlab). You could download such source code and study it.
You'll find many HTTP (libonion, libcurl, etc...) or SMTP (postfix, vmime, etc...) related networking open source programs on Linux etc... And other network programs (PostGreSQL, etc...). Study their source code
This question has been asked with varying degrees of success in the past...
Are there tools, or C/C++ unix functions to call that would enable me to retrieve the location on disk of a file? Not some virtual address of the file, but the disk/sector/block the file resides in?
The goal here is to enable overwriting of the actual bits that exist on disk. I would probably need a way to bypass the kernel's superimposition of addresses. I am willing to consider an x86 asm based solution...
However, I feel there are tools that do this quite well already.
Thanks for any input on this.
Removing files securely is only possible under very specific circumstances:
There are no uncontrolled layers of indirection between the OS and the actual storage medium.
On modern systems that can no longer be assumed. SSD drives with firmware wear-leveling code do not work like this; they may move or copy data at will with no logging or possibility of outside control. Even magnetic disk drives will routinely leave existing data in sectors that have been remapped after a failure. Hybrid drives do both...
The ATA specification does support a SECURE ERASE command which erases a whole drive, but I do not know how thorough the existing implementations are.
The filesystem driver has a stable and unique mapping of files to physical blocks at all times.
I believe that ext2fs does have this feature. I also think that ext3fs and ext4fs also work like this in the default journaling mode, but not when mounted with the data=journal option which allows for file data to be stored in the journal, rather than just metadata.
On the other hand reiserfs definitely works differently, since it stores small amounts of data along with the metadata, unless mounted with the notail option.
If these two conditions are met, then a program such as shred may be able to securely remove the content of a file by overwriting its content multiple times.
This method still does not take into account:
Backups
Virtualized storage
Left over data in the swap space
...
Bottom line:
You can no longer assume that secure deletion is possible. Better assume that it is impossible and use encryption; you should probably be using it anyway if you are handling sensitive data.
There is a reason that protocols regarding sensitive data mandate the physical destruction of the storage medium. There are companies that actually demagnetize their hard disk drives and then shred them before incinerating the remains...
Is there any book or tutorial that can learn me how to read binary files with a complex structure. I did a lot of attempts to make a program that has to read a complex file format and save it in a struct. But it always failed because of heap overruns etc. that made the program crash.
Probably your best bet is to look for information on binary network protocols rather than file formats. The main issues (byte order, structure packing, serializing and unserializing pointers, ...) are the same but networking people tend to be more aware of the issues and more explicit in how they are handled. Reading and writing a blob of binary to or from a wire really isn't much different than dealing with binary blobs on disk.
You could also find a lot of existing examples in open source graphics packages (such as netpbm or The Gimp). An open source office package (such as LibreOffice) would also give you lots of example code that deals with complex and convoluted binary formats.
There might even be something of use for you in Google's Protocol Buffers or old-school ONC RPC and XDR.
I don't know any books or manuals on such things but maybe a bunch of real life working examples will be more useful to you than a HOWTO guide.
One of the best tools to debug memory access problems is valgrind. I'd give that a try next time. As for books, you'd need to be more specific about what formats you want to parse. There are lots of formats and many of them are radically different from each other.
Check out Flavor. It allows you to specify the format using C-like structure and will auto-generate the parser for the data in C++ or Java.
Most games come with their resources (models, textures, etc.) packed into special files (like .pk3 files in Quake 3 for example). Apparently, those files somehow get "mounted" and are used as if they were separate file systems.
I'd like to know how this is achieved. The only strategy I came up with so far is placing offset-size information in the file's header, then memory-mapping the file and accessing the resources as if they were independent write-protected chunks of memory.
I'd like to know if my strategy is viable and if there are better alternatives.
Thanks!
Your strategy is quite reasonable; in fact, it's an exact analogue of what a filesystem driver does with a raw block device. What problems are you having with that implementation?
Your approach sounds reasonable. Basically you'll have an ISAM file with (optional) metadata. You can split the file into sections ("directories") based on criteria (content type, frequency of use, locality of use, etc) and have a separate index/table of contents for each section. If you allow a section as a content type, then you can nest them and handle them in a consistent fashion.
If your requirements are fairly basic, you could consider simply using a zip/tar file as a container. Looking at the code for these would probably be a good place to start, regardless.
I can't say about exact format of Quake3, but there are several approaches to do what you need:
archive files (ZIP, Tar, etc)
compound storage of different kind. On Windows there's Microsoft Structured Storage
embedded single-file database
virtual file system, which implements an actual file system, but stored not on the disk partition, but somewhere else (in resources, in memory etc).
Each of those approaches has it's own strengths and weaknesses.
Our SolFS product is an example of virtual file system mentioned above. It was designed for tasks similar to yours.
Archives and some compound file implementations usually have linear sequential structure - the files are written one by one with some directory at the beginning or at the end of file.
Some compound file implementations, databases and virtual file system have page-based structure ("page" is similar to sector or cluster in FAT or NTFS) where files can be scattered across the storage.