I was wandering if there is a filesystem that is non-block based.
Every system I know stores or retrieves chunks of fixed size (the blocks, or clusters).
Is there a filesystem that will allow to write or read a single byte, in the context of external storage, as hard disk, flash memory etc?
I googled around, but I could't find anything conclusive.
Thanks.
There's a good reason you won't find it: none of these underlying storage technologies support byte addressing. Flash drives and hard drives connected with SATA or SCSI are all divided into either 512-byte sectors or 4 KiB sectors. Even if you need only one byte from a sector, you have to read the whole thing.
Part of what a filesystem does is provide an abstraction on top of these different storage technologies.
Related
I have bought an I2C EEPROM. I want to store sensor and voltage data. I'm assuming that value can be bigger than one byte, and there can be a lot of data. Is it worth is such case to implement a filesystem with small file allocation table? It would make me easier to peek trought EEPROM for example.
I see two causes for a FAT on EEPROM
If there is a requirement for the flexibility of having different files. Such as
for data logging or configurations. It allows multiple such configuration/log files, to be independent and easily added in the future. This can be a very successful building block for future projects.
For ease of access by other devices or libraries. Typically only an option if the memory device is directly accessible by other interface. Where as in this case it is an EEPROM. If your device was directly USB capable, such as a ATmega32u4 (leo) then you can use LUFA tools to have the USB show up as MASS storage. Making FAT an ideal solution. Or possibly if the device has an Ethernet Shield.
with all being said and if this case simply a datalogger, then the KISS (Keep It Simple Solution) may be a good way to go. So that one can keep focus on the original subject for collecting the data itself.
It is worth noting that SdCards can be easily added for cheap either of the well established Sd library (IDE stock) or SdFat Library (GitHub more features) adding an almost infinite capacity of logging of FAT32. The only trade off is they consume a fair chunk of code space.
I think mpflaga is on the right path.
Some options you should consider include:
Is the device/microcontroller that is writing the data going to be the same as the one that is reading the data?
How many records are you hoping to fit into your storage device?
How robust/recoverable do you want your storage format to be to events such as reboots/power outages/etc?
My opinion regarding these points is that:
It is going to be the same device reading and writing, so you can probably get away with a very specific/custom format rather than a full blown file system.
You probably want to extract as many bytes as possible for use as storage, so a format well-designed for your application will probably help.
This is tricky. You could use self-describing structures, such as a TLV, which would pack your bytes tightly but be harder to search; OR you could use a fixed-length structure, which wastes a lot of bytes but allows easy access. Also, you could just assume the storage will always remain valid, but what happens if power is removed half-way through a write!
Overall, my recommendation would be:
Use an existing library
Use an application-specific format first, but ensure you abstract the storage of the data from the data itself.
If you find you need a filesystem, rewrite the storage layer to use a filesystem.
Having a small standard file system, like FAT16, is worth implementing because you can map this file system over the USB or Network to other devices/computers.
Standardization in your design is a big compliance advantage.
You can find ready sources/libraries or, if it's FAT16 and because it is really simple and well described/documented, try implementing yourself.
I understand a file-system can choose the size of blocks it uses on the disk.
On the other hand i understand that the disk is divided into LBA's.
The LBA is an address of a sector on the disk.
So whats the connection between the block used by the file system and the disk sectors (lba)?
Is there some kind of translation from a fs block and lba?
Is it different from fs to fs?
where can i read more about this?
thanks
Yes. File system usually sees a a continuous logical space without knowledge of the spindles underneath, thus it doesn't know disk LBA either. The translation work is usually done in a layer called volume, which is to hide the disk detail and present the file system a logically continuous space. For example, in Linux there's LVM (Logical Volume Manager) playing such roles.
The volume exposed to fs might not be disks. It could be constructed upon other volumes, thus sometimes come up with a very large disk.
The volume could also provide the functionality of RAID, which put several disks together that could relieve you from disk failure in some extent at the expense of performance and space efficiency.
Some file systems can manage disks directly and operate on raw disks, thus no layer of volume. As far as I know, NETAPP's WAFL is doing in that way.
I had been playing with the disk drives 12bit FAT (FAT12) and 16bit FAT(FAT16) in C language (Turbo C) which runs under the 16 bit OS MS-DOS.
I was able to manipulate sectors directly.
FAT32 was little complicated because the sectors are stored like linked list unlike other FAT lower than FAT32.
I want to read write hard disks, USB Disks directly using 32 bit C language (win32 api).
I saw some code and it were using /device/ to access a disk where as in biosdisk the disks were numbered from 0 onwards i think. i was manipulating like heads, sectors, cylinders ...
Please advice on how to read write hard disks directly sector by sector or how to read write hdd in low level.
do i have to go for assembly language?
EDIT
one scenario why i need to directly manipulate the hard disk is i want to write a file maintaining my own FAT even hiding it from the FileSystem but marking those sectors as used. So it is just hiding a file from the other users, the operating system and even me except the program i write which can only access those files. this is just one point and the others would be just playing around. :)
If you use WinAPI, then you open raw disk device using CreateFile() API (see Physical Disks and Volumes section there) and then use ReadFile() and WriteFile() methods to read and write data.
Note, however, that recent versions of Windows (Vista, Windows 7) restrict your access even if you are an administrator. Our RawDisk product lets you bypass these restrictions. Free non-commercial licenses are available for RawDisk.
I need to save very large amounts of data (>500GB) which is being streamed (800Mb/s) from another device connected to my PC. The speed rules out use of a database e.g. MySQl/ISAM and I am looking for a fast, light library which sits on top of the 'C' stdio file lib (i.e. fopen/fclose/fwrite) which will allow me to write/read a very large file (up to available disk-space).
Behind-the-scenes, the large file can be broken up into smaller files e.g. 1GB and I want the API to take care of these details.
The data arrives at the PC in a compressed binary format and no further processing is needed before writing it to the hard-disk.
The library should be work for Windows and Linux.
if you need random access into the data, take a look at memory mapped files.
It lets you map a file (or a section of a file) into memeory transparently, without having to explicitly allocate memeory and read data. It works on windows/Linux (there is a boost lib that wraps the differences).
On Windows you can handle files >>4gb on a 32bit os by using multiple windows into the file.
edit: Sorry 800Mb/s !! I don't know any disks that can cope with that. You migth be lookign at a raid array of SSD drives.
There used to be image capture cards that used an attached drive as a simple series of bytes with no filesystem to get very high speed sustained writes. I don't know if you are going to need somethign like that.
For ultimate speed, I suggest you go highly platform specific.
The objective is to get as close as you can to connecting the input device directly to hard drive. One method is to write a driver for the input device that writes directly to the hard drive.
The generic algorithm is to use either a very large circular byte buffer or use multiple buffers. You need extra space to compensate for the speed difference between the input device and the output device; provided the input device is non-stop.
If you can pause the input device, the issue becomes easier.
I will write some thing in a file/memory just before system shutdown or a service shutdown. In the next restart of system, Is it possible to access same file or same memory on the disk, before filesystem loads? Actual requirement is like this, we have a driver that sits between volume level drivers and filesystem driver...in that part of the driver code, I want to access some memory or file.
Thanks & Regards,
calvin
The logical thing here is to read/write this into the registry if it is not too big. Is there a reason you do not want to use the registry?
If you need to access large data and you are writing a volume or device filter and cannot rely on ZwOpen/Read/Write/Close functions in the kernel an approach would be to create the file in user mode, get its device name and cluster chain and store them in the registry. On the next boot, you can get the device and clusters from registry, and do direct I/O on them.
Since you want to access this before the filesystem loads, my first thought is to allocate and use a block of storage space on the hard drive outside of the filesystem. You can create a hidden mini-partition on the drive and use low-level I/O commands to read and write your data.
This is a common task in the world of embedded systems, and we often implement it by adding some sort of non-volatile memory device into the system (flash, battery-backed DRAM, etc) and reading and writing to that device. Since you likely don't have the same level of control over the available hardware as embedded developers do, the closest analogue I can think of would be to reserve a chunk of space on a physical disk that you can read from without having to mount as a filesystem. A dedicated mini-partition might work the best because if you know the size of it, you can treat it as one big raw-access buffer and can avoid having to hassle with filenames, filesystems, etc.