How does a file-system block gets translated to lba? - filesystems

I understand a file-system can choose the size of blocks it uses on the disk.
On the other hand i understand that the disk is divided into LBA's.
The LBA is an address of a sector on the disk.
So whats the connection between the block used by the file system and the disk sectors (lba)?
Is there some kind of translation from a fs block and lba?
Is it different from fs to fs?
where can i read more about this?
thanks

Yes. File system usually sees a a continuous logical space without knowledge of the spindles underneath, thus it doesn't know disk LBA either. The translation work is usually done in a layer called volume, which is to hide the disk detail and present the file system a logically continuous space. For example, in Linux there's LVM (Logical Volume Manager) playing such roles.
The volume exposed to fs might not be disks. It could be constructed upon other volumes, thus sometimes come up with a very large disk.
The volume could also provide the functionality of RAID, which put several disks together that could relieve you from disk failure in some extent at the expense of performance and space efficiency.
Some file systems can manage disks directly and operate on raw disks, thus no layer of volume. As far as I know, NETAPP's WAFL is doing in that way.

Related

Non-block file systems

I was wandering if there is a filesystem that is non-block based.
Every system I know stores or retrieves chunks of fixed size (the blocks, or clusters).
Is there a filesystem that will allow to write or read a single byte, in the context of external storage, as hard disk, flash memory etc?
I googled around, but I could't find anything conclusive.
Thanks.
There's a good reason you won't find it: none of these underlying storage technologies support byte addressing. Flash drives and hard drives connected with SATA or SCSI are all divided into either 512-byte sectors or 4 KiB sectors. Even if you need only one byte from a sector, you have to read the whole thing.
Part of what a filesystem does is provide an abstraction on top of these different storage technologies.

linux kernel: how can I copy files before panic?

I have a file on tmpfs partition, which is updated alot.
I want to copy it to other partition (flash partition) before crash/reboot.
It is not an option to keep this file in the first place on the flash partition,
because this flash has limited read/write life-cycle and I'm trying to avoid excessive read/writes to it.
too many writes will damage the flash, that is why the file is on tmpfs.
regrading reboot - I can modify the reboot process to copy before reboot - is there more neat way?
regrading crash - I don't know any way to do so. any ideas?
I know that that I should not mess with files from kernel space.
Thanks
Only a Kernel panic occurs its possible that in-core data structures are already corrupted and unreliable. Ideally, your kernel is not expected to panic, if the version you are using is a stable and tested release. I would recommend capturing a vmcore using crash tool and working with the vendor on the root cause of the kernel panic.
However, if you are referring to an abrupt system shutdown dew to a power failure, etc which could possibly cause loss of the data / file stored in the memory, you could write a cron-job to sync the file to disk on intervals and tune the kernel on how frequently the dirty page get synced. Having said that, if the file you are writing to is quite important, why design it to be kept in the memory in the first place.
You should be syncing this file back to the disk once in every few seconds or in regular intervals. In this way you will not loose the complete data.
As the numbers of read/writes are heavy on the tmpfs file, it may be worth considering using a SSD for this purpose. Read about how file system transaction logs are configured to be stored on SSD drives.
Write a cron-job for syncing the tmpfs file to the SSD or disk in frequent intervals or when ever there are updates. You may want to consider changing some kernel tunables (such as, vm.dirty_expire_centisecs=0, vm.dirty_background_ratio=0) such that any dirty pages would get synced to the disk immediately. A word of caution, doing this would cause higher CPU% and I/O loads, as pages would synced to the disk frequently, although the data loss would be kept to minimal.

How Database handles the Disk Block while it is control by OS

I am not clear how database system control the disk blocks. I have read that the data which are related, put them in the same or nearby block. How it is possible for a database system(an application software). What I know is disk block is maintained by the OS so whatever algorithm is implemented in the OS it would allocate the disk block according to that.
If I am wrong, please correct me. I am novice in that.

reading and writing hard disk directly in win32api like the biosdisk or absread in ms-dos

I had been playing with the disk drives 12bit FAT (FAT12) and 16bit FAT(FAT16) in C language (Turbo C) which runs under the 16 bit OS MS-DOS.
I was able to manipulate sectors directly.
FAT32 was little complicated because the sectors are stored like linked list unlike other FAT lower than FAT32.
I want to read write hard disks, USB Disks directly using 32 bit C language (win32 api).
I saw some code and it were using /device/ to access a disk where as in biosdisk the disks were numbered from 0 onwards i think. i was manipulating like heads, sectors, cylinders ...
Please advice on how to read write hard disks directly sector by sector or how to read write hdd in low level.
do i have to go for assembly language?
EDIT
one scenario why i need to directly manipulate the hard disk is i want to write a file maintaining my own FAT even hiding it from the FileSystem but marking those sectors as used. So it is just hiding a file from the other users, the operating system and even me except the program i write which can only access those files. this is just one point and the others would be just playing around. :)
If you use WinAPI, then you open raw disk device using CreateFile() API (see Physical Disks and Volumes section there) and then use ReadFile() and WriteFile() methods to read and write data.
Note, however, that recent versions of Windows (Vista, Windows 7) restrict your access even if you are an administrator. Our RawDisk product lets you bypass these restrictions. Free non-commercial licenses are available for RawDisk.

Accessing same resource across restarts in Windows

I will write some thing in a file/memory just before system shutdown or a service shutdown. In the next restart of system, Is it possible to access same file or same memory on the disk, before filesystem loads? Actual requirement is like this, we have a driver that sits between volume level drivers and filesystem driver...in that part of the driver code, I want to access some memory or file.
Thanks & Regards,
calvin
The logical thing here is to read/write this into the registry if it is not too big. Is there a reason you do not want to use the registry?
If you need to access large data and you are writing a volume or device filter and cannot rely on ZwOpen/Read/Write/Close functions in the kernel an approach would be to create the file in user mode, get its device name and cluster chain and store them in the registry. On the next boot, you can get the device and clusters from registry, and do direct I/O on them.
Since you want to access this before the filesystem loads, my first thought is to allocate and use a block of storage space on the hard drive outside of the filesystem. You can create a hidden mini-partition on the drive and use low-level I/O commands to read and write your data.
This is a common task in the world of embedded systems, and we often implement it by adding some sort of non-volatile memory device into the system (flash, battery-backed DRAM, etc) and reading and writing to that device. Since you likely don't have the same level of control over the available hardware as embedded developers do, the closest analogue I can think of would be to reserve a chunk of space on a physical disk that you can read from without having to mount as a filesystem. A dedicated mini-partition might work the best because if you know the size of it, you can treat it as one big raw-access buffer and can avoid having to hassle with filenames, filesystems, etc.

Resources