I'm trying to wrap my head around the EXT2 file system, but I can't find a single place that shows me the EXT2 file system in detail.
I finally drew up a diagram myself. So I got that far. Now I'm trying to figure out the following (I've found some info already):
Number of bytes per sector: 0.5kB - 4kB
Number of bytes per block: 4kB - 64kB
Number of sectors per block: 1 - 128
Number of blocks per block group: ?
Number of block groups per partition: ?
It's crazy to me that I can't find a single place that has this information.
EDIT: Also just found this, which means my bytes per block number is probably wrong:
#define EXT2_MIN_BLOCK_SIZE 1024
#define EXT2_MAX_BLOCK_SIZE 4096
I usually find my information about ext2 at the osdev wiki which in turn links here.
The number of bytes per block is 1024<<n where n is given in the superblock and is a 32 bit integer. So in theory, a block could be anywhere between 1024 and ... lots of bytes. Normally, block sizes of 1, 2, 4 or 8 kB are used, though depending on several factors such as partition size and expected average file size.
Each block group contains a single block bitmap of free blocks. This gives the constraint 8*block size to the number of blocks per block group. The same is true for inodes per block group. The actual values are found in the superblock.
This in turn gives a lower bound to the number of block groups needed to fill the partition.
Related
I have two zpools on ZFD:
the zpool on top is 8 disks 2 TB each in raidZ3
the zpool on bottom is 4 disks 4 TB each in raidZ3
The data is EXACTLY the same. I even ran diff -qr /top/zpool/ /bottom/zpool/ to confirm.
Why is it that ALLOC and CAP fields differ, if data is exactly duplicated?
Dealing with ZFS space could be hard.
To be sure that The data is EXACTLY the same, give a try to zfs list -o space.
About the difference between ALLOC and CAP, docs.oracle.com says:
ALLOC: The amount of physical space allocated to all datasets and internal metadata. Note that this amount differs from the amount of disk space as reported at the file system level.
CAP (CAPACITY): The amount of disk space used, expressed as a percentage of the total disk space.
And for a more detailed answer you need to consider the block size and the average size of the data stored. A whole explanation can by found in Matt Ahrens explanation on Delphix blog.
Info on minecraft's region files
Minecraft's region files are stored in 3 sections, the first two giving information about where the chunks are stored, and information about the chunks themselves. In the final section, chunks are given as a 4-byte number length, the type of compression it uses, (almost always is zlib, RFC1950)
Here's more (probably better) information: https://minecraft.gamepedia.com/Region_file_format
The problem
I have a program that successfully loads chunk data. However, I'm not able to find how big the chunks will be when decompressed, and so I just use a maximum amount it could take when allocating space.
In the player data files, they do give the size that it takes when decompressed, and (I think) it uses the same type of compression.
The end of a player.dat file giving the size of the decompressed data (in little-endian):
This is the start of the chunk data, first 4 bytes giving how many bytes is in the following compressed data:
Mystery data
However, if I look where the compressed data specifically "ends", there's still a lot of data after it. This data doesn't seem to have a use, but if I try to decompress any of it with the rest of the chunk, I get an error.
Highlighted chunk data, and unhighlighted mystery data:
Missing decompressed size (header?)
And there's no decompressed size (or header? I could be wrong here) given.
The final size of this example chunks is 32,562 bytes, and this number (or any close neighbours) is nowhere to be found within the chunk data or mystery data. (Checked both big-endian, and little-endian)
Decompressed data terminating at index 32562, (Visual Studio locals watch):
Final Questions
Is there something I'm missing? Is this compression actually different from the player data compression? What's the mystery data? And am I stuck loading in 1<<20 bytes every time I want to load a chunk from a region file?
Thank you for any answers or suggestions
Files used
Isolated chunk data: https://drive.google.com/file/d/1n3Ix8V8DAgR9v0rkUCXMUuW4LJjT1L8B/view?usp=sharing
Full region data: https://drive.google.com/file/d/15aVdyyKazySaw9ZpXATR4dyvhVtrL6ZW/view?usp=sharing
(Not linking player data for possible security reasons)
In the region data, the chunk data starts at index 1208320 (or 0x127000)
The format information you linked is quite helpful. Have you read it?
In there it says: "The remainder of the file consists of data for up to 1024 chunks, interspersed with unused space." Furthermore, "Minecraft always pads the last chunk's data to be a multiple-of-4096B in length" (Italics mine.) Everything is in multiples of 4K, so the end of every chunk is padded to the next 4K boundary.
So your "mystery" data is not a mystery at all, as it is entirely expected per the format documentation. That data is simply junk to be ignored.
Note that, from the documentation, that the data "length" in the first three bytes of the chunk is actually one more than the number of bytes of data in the chunk (following the five-byte header).
Also from the documentation, there is indeed no uncompressed size provided in the format.
zlib was designed for streaming data, where you don't know ahead of time how much there will be. You can use inflate() to decompress into whatever buffer size you like. If there's not enough room to finish, you can either do something with that data and then repeat into the same buffer, or you can grow the buffer with realloc() in C, or the equivalent for whatever language you're using. (Not noted in the question or tags.)
I am implementing a FAT file system in C. I am following the specs published by microsoft (http://read.pudn.com/downloads77/ebook/294884/FAT32%20Spec%20%28SDA%20Contribution%29.pdf)
But I don't understand how to compute the FAT size field of boot sector. In the specification document appear the following code on page 14.
RootDirSectors = ((BPB_RootEntCnt * 32) + (BPB_BytsPerSec – 1)) / BPB_BytsPerSec;
TmpVal1 = DskSize – (BPB_ResvdSecCnt + RootDirSectors);
TmpVal2 = (256 * BPB_SecPerClus) + BPB_NumFATs;
If(FATType == FAT32)
TmpVal2 = TmpVal2 / 2;
FATSz = (TMPVal1 + (TmpVal2 – 1)) / TmpVal2;
If(FATType == FAT32) {
BPB_FATSz16 = 0;
BPB_FATSz32 = FATSz;
} else {
BPB_FATSz16 = LOWORD(FATSz);
/* there is no BPB_FATSz32 in a FAT16 BPB */
}
From this code I don't understand
What is TmpVal2?
Why number 256 is used?
Why if it is FAT32 it divide by 2?
I am not sure why the constant of 256 was chosen however here are some thoughts on your other questions.
There is a note below the source code segment which states that the math is an approximation.
NOTE: The above math does not work perfectly. It will occasionally set
a FATSz that is up to 2 sectors too large for FAT16, and occasionally
up to 8 sectors too large for FAT32. It will never compute a FATSz
value that is too small, however. Because it is OK to have a FATSz
that is too large, at the expense of wasting a few sectors, the fact
that this computation is surprisingly simple more than makes up for it
being off in a safe way in some cases.
The way I read the code is that the calculation is for a FAT16 size and then there is an adjustment to the calculation if the target is actually FAT32.
The value of the variable TmpVal2 looks to be a unit size in that the amount of space calculated for the value of TmpVal1 is then divided by the unit size value of TmpVal2 in order to determine the number of units of disk space. However in the case of FAT32 the unit size is smaller than in FAT16 so there needs to be an adjustment.
It appears that FAT16 used a specific size for the File Allocation Table and as the hard disk space available for a volume was increased with improvements in disk technology, the cluster size was based on the volume size. So with a smaller volume size the cluster size, the number of disk sectors in an allocation unit, was smaller than the cluster size for a large volume size. See FAT16 vs. FAT32 in Microsoft TechNet as well as the tables in the source code on page 13 of the document you reference.
With FAT32, a standard cluster size of 4K was used and the File Allocation Table storage was changed from a fixed size to a variable size and was no longer at a fixed location on the disk.
This artice, File systems (FAT, FAT8, FAT16, FAT32, and NTFS) explained, goes into some details about the differences between these various file system versions.
The Wikipedia article, File Allocation Table, has quite a bit of technical information with links to other articles.
You may also find the following stackoverflow articles of interest.
Converting the cluster number stored in FAT table (of FAT12 filesystem) for reading from a floppy disk
Why did Windows use the FAT structure instead of a conventional linked list with a next pointer for each data block of a file?
I've been given 2k bytes to make a ultra minimalistic file system and I thought about making a stripped out version of FAT16.
My only problem is understanding how do I store the FAT in the volume. Let's say I use 2 bytes per block hence I'd have 1024 blocks. I need a table with 1024 rows and in each row I'll save the next block of a file.
As each of this block can address other 1023 blocks, I fail to see how this table would not use my entire 2k space. I do not understand how to save this table into my hard drive and use only a few bytes rather than just using 1024 block for writing a 1024 row table.
Given that you are allowed to implement a flat filesystem and have such a small space to work with, I would look at something like the Apple DOS 3.3 filesystem rather than a hierarchical filesystem like FAT16. Even the flat filesystem predecessor of FAT16, FAT12, is overly complex for your purposes.
I suggest that you divide your 2 kiB volume up into 256 byte "tracks" with 16 byte "sectors," to use the Apple DOS 3.3 nomenclature. Call them what you like in your own implementation. It just helps you to map the concepts if you reuse the same terms here at the design stage.
You don't need a DOS boot image, and you don't have the seek time of a moving disk drive head to be concerned about, so instead of setting aside tracks 0-2 and putting the VTOC track in the middle of the disk, let's put our VTOC on track 0. The VTOC which contains the free sector bitmap, the location of the first catalog sector, and other things.
If we reserve the entirety of track 0 for the VTOC, we would have 112 of our 16-byte sectors left. Those will pack up into only 14 bytes for the bitmap, which suggests that we really don't need the entirety of track 0 for this.
Let's set aside the first two sectors of track 0 instead, and include track 0 in the free sector bitmap. That causes a certain amount of redundancy, in that we will always have the first two sectors mapped as "used," but it makes the implementation simpler, since there are now no special cases.
Let's split Apple DOS 3.3's VTOC concept into two parts: the Volume Label Sector (VLS) and the volume free sector bitmap (VFSB).
We'll put the VLS on track 0 sector 0.
Let's set aside the first 2-4 bytes of the VLS for a magic number to identify this volume file as belonging to your filesystem. Without this, the only identifying characteristic of your volume files is that they are 2 kiB in size, which means your code could be induced to trash an innocent file that happened to be the same size. You want more insurance against data destruction than that.
The VLS should also name this volume. Apple DOS 3.3 just used a volume number, but maybe we want to use several bytes for an ASCII name instead.
The VLS also needs to point to the first catalog sector. We need at least 2 bytes for this. We have 128 tracks, which means we need at least 7 bits. Let's use two bytes: track and sector. This is where you get into the nitty-gritty of design choices. We can now consider moving to 4 kiB volume sizes by defining 256 tracks. Or, maybe at this point we decide that 16-byte sectors are too small, and increase them so we can move beyond 4 kiB later. Let's stick with 16 byte sectors for now, though.
We only need one sector for the VFSB: the 2 kiB volume ÷ 16 bytes per sector = 128 sectors ÷ 8 bits per byte = 16 bytes. But, with the above thoughts in mind, we might consider setting aside a byte in the VLS for the number of VFSB sectors following the VL, to allow for larger volumes.
The Apple DOS 3.3 catalog sector idea should translate pretty much directly over into this new filesystem, except that with only 16 bytes per sector to play with, we can't describe 7 files per sector. We need 2 bytes for the pointer to the next catalog sector, leaving 14 bytes. Each file should have a byte for flags: deleted, read-only, etc. That means we can have either a 13-byte file name for 1 file per catalog sector, or two 6-byte file names for 2 files per catalog sector. We could do 7 single-letter file names, but that's lame. If we go with your 3-character file name idea, that's 3 files per catalog sector after accounting for the flag byte per file, leaving 2 extra bytes to define. I'd go with 1 or 2 files per sector, though.
That's pretty much what you need. The rest is implementation and expansion.
One other idea for expansion: what if we want to use this as a bootable disk medium? Such things usually do need a boot loader, so do we need to move the VLS and VFSB sectors down 1, to leave track 0 sector 0 aside for a boot image? Or, maybe the VLS contains a pointer to the first catalog sector that describes the file containing the boot image instead.
I am a unclear about file system implementation. Specifically (Operating Systems - Tannenbaum (Edition 3), Page 275) states "The first word of each block is used as a pointer to the next one. The rest of block is data".
Can anyone please explain to me the hierarchy of the division here? Like, each disk partition contains blocks, blocks contain words? and so on...
I don't have the book in front of me, but I'm suspect that quoted sentence isn't really talking about files, directories, or other file system structures. (Note that a partition isn't a file system concept, generally). I think your quoted sentence is really just pointing out something about how the data structures stored in disk blocks are chained together. It means just what it says. Each block (usually 4k, but maybe just 512B) looks very roughly like this:
+------------------+------------- . . . . --------------+
| next blk pointer | another 4k - 4 or 8 bytes of stuff |
+------------------+------------- . . . . --------------+
The stuff after the next block pointer depends on what's stored in this particular block. From just the sentence given, I can't tell how the code figures that out.
With regard to file system structures:
A disk is an array of sectors, almost always 512B in size. Internally, disks are built of platters, which are the spinning disk-shaped things covered in rust, and each platter is divided up into many concentric tracks. However, these details are entirely hidden from the operating system by the ATA or SCSI disk interface hardware.
The operating system divides the array of sectors up into partitions. Partitions are contiguous ranges of sectors, and partitions don't overlap. (In fact this is allowed on some operating systems, but it's just confusing to think about.)
So, a partition is also an array of sectors.
So far, the file system isn't really in the picture yet. Most file systems are built within a partition. The file system usually has the following concepts. (The names I'm using are those from the unix tradition, but other operating systems will have similar ideas.)
At some fixed location on the partition is the superblock. The superblock is the root of all the file system data structures, and contains enough information to point to all the other entities. (In fact, there are usually multiple superblocks scattered across the partition as a simple form of fault tolerance.)
The fundamental concept of the file system is the inode, said "eye-node". Inodes represent the various types of objects that make up the file system, the most important being plain files and directories. An inode might be it's own block, but some file system pack multiple inodes into a single block. Inodes can point to a set of data blocks that make up the actual contents of the file or directory. How the data blocks for a file is organized and indexed on disk is one of the key tasks of a file system. For a directory, the data blocks hold information about files and subdirectories contained within the directory, and for a plain file, the data blocks hold the contents of the file.
Data blocks are the bulk of the blocks on the partition. Some are allocated to various inodes (ie, to directories and files), while others are free. Another key file system task is allocating free data blocks as data is written to files, and freeing data blocks from files when they are truncated or deleted.
There are many many variations on all of these concepts, and I'm sure there are file systems where what I've said above doesn't line up with reality very well. However, with the above, you should be in a position to reason about how file systems do their job, and understand, at least a bit, the differences you run across in any specific file system.
I don't know the context of this sentence, but it appears to be describing a linked list of blocks. Generally speaking, a "block" is a small number of bytes (usually a power of two). It might be 4096 bytes, it might be 512 bytes, it depends. Hard drives are designed to retrieve data a block at a time; if you want to get the 1234567th byte, you'll have to get the entire block it's in. A "word" is much smaller and refers to a single number. It may be as low as 2 bytes (16-bit) or as high as 8 bytes (64-bit); again, it depends on the filesystem.
Of course, blocks and words isn't all there is to filesystems. Filesystems typically implement a B-tree of some sort to make lookups fast (it won't have to search the whole filesystem to find a file, just walk down the tree). In a filesystem B-tree, each node is stored in a block. Many filesystems use a variant of the B-tree called a B+-tree, which connects the leaves together with links to make traversal faster. The structure described here might be describing the leaves of a B+-tree, or it might be describing a chain of blocks used to store a single large file.
In summary, a disk is like a giant array of bytes which can be broken down into words, which are usually 2-8 bytes, and blocks, which are usually 512-4096 bytes. There are other ways to break it down, such as heads, cylinders, sectors, etc.. On top of these primitives, higher-level index structures are implemented. By understanding the constraints a filesystem developer needs to satisfy (emulate a tree of files efficiently by storing/retrieving blocks at a time), filesystem design should be quite intuitive.
Tracks >> Blocks >> Sectors >> Words >> Bytes >> Nibbles >> Bits
Tracks are concentric rings from inside to the outside of the disk platter.
Each track is divided into slices called sectors.
A block is a group of sectors (1, 2, 4, 8, 16, etc). The bigger the drive, the more sectors that a block will hold.
A word is the number of bits a CPU can handle at once (16-bit, 32-bit, 64-bit, etc), and in your example, stores the address (or perhaps offset) of the next block.
Bytes contain nibbles and bits. 1 Byte = 2 Nibbles; 1 Nibble = 4 Bits.