After reading about File vs. Block storage at File vs. Block storage on 18.15m , looks like both both block based storages(SAN) and files
based storage devices(NAS) stores the internally as block .
But at couple of other resources like https://stonefly.com/resources/what-is-file-level-storage-vs-block-level-storage and https://www.networkworld.com/article/3256312/what-is-a-san-and-how-does-it-differ-from-nas.html I see below claim
A SAN stores data at the block level, while NAS accesses data as
files.
It stores files and folders and the visibility is the same to the
clients accessing and to the system which stores it.
From this looks like File based storages stores the data as file(not blocks) in sequential fashion. Not sure which one is true ?
It's a matter of terminology only. On their back-ends, both types of systems store data in blocks and the difference between them is in front-end protocols they provide to a client. Block storage systems allow access via block-level protocol - in the most cases it's SCSI (SCSI over FC or SCSI over TCP/IP - iSCSI). Respectively, front-end of a file storage operates on file-level protocols: NFS, SMB/CIFS. It happens, a storage device can work in both modes: block and file.
Related
From Wikipedia
Disk formatting
Disk formatting is the process of preparing a data storage device such as a hard disk drive, solid-state drive, floppy disk or USB flash drive for initial use.
Mount (computing)
Mounting is a process by which the operating system makes files and directories on a storage device (such as hard drive, CD-ROM, or network share) available for user to access via the computer's file system.
I don't understand how these two sentences are different
You format a device. You mount a filesystem.
Mounting a Filesystem
Typically, to make a device ready to be written to by the system, you'd have to do the following:
Physically connect the device to the system
(optional but recommended) Create a partition on your device
Create a filesystem on the device/partition
Mount the filesystem on the device/partition to a directory on your system's filesystem, so that you can access the filesystem of the device/partition from your system.
Formatting a Device
Therefore, every filesystem is backed by some block storage device. The storage medium of a block storage device (e.g. a hard disk) is divided into many sections, called sectors. Each sector has a unique address. Whenever you write to, or read from, the device, it is done by writing/reading whole sectors. Therefore, the sector is a smallest addressable physical unit for a block storage device.
When you format a device, you are adding 'marks' onto the storage medium to mark the location of these sectors.
Source of Confusion
Formatting has nothing to do with the filesystem, or mounting. The confusion often arise from Windows users, because on Windows, the term 'formatting' also includes the creation of the filesystem. On Windows, when you format a disk, you are also creating a filesystem on it (and Windows also automatically mount devices you connect).
To distinguish between Window's version of 'formatting', and actual formatting, people have used the term low-level formatting for actual formatting, and high-level formatting for creating the filesystem.
Formatting a disk or partition usually means that you're actually creating or have already created a volume, and it is the volume that you are formatting. Formatting a volume is to write an empty filesystem with the initialised filesystem structures like the MFT to the volume.
Mount has the notion of putting something on top of the other. But what is considered to be on top and what is considered to be at the bottom varies.
For mount points, the mount directory and the existing directory tree is considered to be on the bottom, and you mount the volume on top of / to the directory. This is despite the mount points actually being logically higher than the volume in the driver stack and the process of interacting with files. You sometimes see people say mounting the filesystem to the mount point, like the Linux df command. This makes sense because the file system is indeed logically between the volume and the mount point.
For mounting filesystems to volumes, the filesystem is on top and the volume is below. You mount the filesystem onto the volume. It is actually logically on top of the volume because on Windows it is consulted by the I/O Manager before the volume and it is up to the file system to interact with the volume stack.
On Windows, the volume is mounted to its mount points (C:\ if the symbolic link is \DosDevices\C:) and then you mount the file system to the volume by formatting the volume. Formatting the volume with a file system causes the file system driver to mount the volume. The file system will then mount itself to the volume every time the volume is detected 'arrives' on the system so long as it has been formatted with that file system because the file system driver detects its filesystem on the volume.
Formatting is usually done ONCE during the lifetime of the hard drive.
The hard drive is written to, wiping away anything that was on it previously, so that it can be accepted as a filesystem.
Mounting usually happens every time the system has been restarted. It allows the operating system to access the hard drive as a filesystem. Mounting a drive does NOT alter the hard drive, although once a filesystem has been mounted it can be modified (unless it was mounted read-only) by typical filesystem operations like creating a directory/folder, creating files, modifying files, etc ....
Mounting tells the operating system which filesystem type (ext3, ext4, HFS+, NTFS, ZFS ...) the hard drive is allowing the OS kernel to map the virtual filesystem functions to the real filesystem functions.
I am developing a system which copies and writes files on NTFS inside a virtual machine. At any time the VM can poweroff (direct shutdown). The poweroff is controlled from the outside so I do not have any way to detect it. Due to that files and complete directories which are being written to get lost. Is there any way to prevent that or do I have to develop my own file system? I have to store the files on the local disk and cannot send files via network.
There always exists a [short] period between when your data is written (sent to the API) and when this data is written to the physical hardware. If the system crashes in the middle, the data will be lost.
There is a setting in Windows to disable system write cache for certain disks. This setting can help you ensure that the data is at least sent to the host's hardware. Probably that's the answer you've been looking for.
Writing your own filesystem won't help much because it's mainly the write cache that causes the data to be lost. There can exist a filesystem-level cache as well, though, and I don't know if the write cache setting I mentioned above also affects internal filesystem cache.
If you write data to a file opened with "write through" enabled, the method only returns after the data is physically written to the disk so you can be sure it got written. You normally do that by passing in a WRITE_THROUGH flag when you open the file.
I understand a file-system can choose the size of blocks it uses on the disk.
On the other hand i understand that the disk is divided into LBA's.
The LBA is an address of a sector on the disk.
So whats the connection between the block used by the file system and the disk sectors (lba)?
Is there some kind of translation from a fs block and lba?
Is it different from fs to fs?
where can i read more about this?
thanks
Yes. File system usually sees a a continuous logical space without knowledge of the spindles underneath, thus it doesn't know disk LBA either. The translation work is usually done in a layer called volume, which is to hide the disk detail and present the file system a logically continuous space. For example, in Linux there's LVM (Logical Volume Manager) playing such roles.
The volume exposed to fs might not be disks. It could be constructed upon other volumes, thus sometimes come up with a very large disk.
The volume could also provide the functionality of RAID, which put several disks together that could relieve you from disk failure in some extent at the expense of performance and space efficiency.
Some file systems can manage disks directly and operate on raw disks, thus no layer of volume. As far as I know, NETAPP's WAFL is doing in that way.
I need to save very large amounts of data (>500GB) which is being streamed (800Mb/s) from another device connected to my PC. The speed rules out use of a database e.g. MySQl/ISAM and I am looking for a fast, light library which sits on top of the 'C' stdio file lib (i.e. fopen/fclose/fwrite) which will allow me to write/read a very large file (up to available disk-space).
Behind-the-scenes, the large file can be broken up into smaller files e.g. 1GB and I want the API to take care of these details.
The data arrives at the PC in a compressed binary format and no further processing is needed before writing it to the hard-disk.
The library should be work for Windows and Linux.
if you need random access into the data, take a look at memory mapped files.
It lets you map a file (or a section of a file) into memeory transparently, without having to explicitly allocate memeory and read data. It works on windows/Linux (there is a boost lib that wraps the differences).
On Windows you can handle files >>4gb on a 32bit os by using multiple windows into the file.
edit: Sorry 800Mb/s !! I don't know any disks that can cope with that. You migth be lookign at a raid array of SSD drives.
There used to be image capture cards that used an attached drive as a simple series of bytes with no filesystem to get very high speed sustained writes. I don't know if you are going to need somethign like that.
For ultimate speed, I suggest you go highly platform specific.
The objective is to get as close as you can to connecting the input device directly to hard drive. One method is to write a driver for the input device that writes directly to the hard drive.
The generic algorithm is to use either a very large circular byte buffer or use multiple buffers. You need extra space to compensate for the speed difference between the input device and the output device; provided the input device is non-stop.
If you can pause the input device, the issue becomes easier.
I will write some thing in a file/memory just before system shutdown or a service shutdown. In the next restart of system, Is it possible to access same file or same memory on the disk, before filesystem loads? Actual requirement is like this, we have a driver that sits between volume level drivers and filesystem driver...in that part of the driver code, I want to access some memory or file.
Thanks & Regards,
calvin
The logical thing here is to read/write this into the registry if it is not too big. Is there a reason you do not want to use the registry?
If you need to access large data and you are writing a volume or device filter and cannot rely on ZwOpen/Read/Write/Close functions in the kernel an approach would be to create the file in user mode, get its device name and cluster chain and store them in the registry. On the next boot, you can get the device and clusters from registry, and do direct I/O on them.
Since you want to access this before the filesystem loads, my first thought is to allocate and use a block of storage space on the hard drive outside of the filesystem. You can create a hidden mini-partition on the drive and use low-level I/O commands to read and write your data.
This is a common task in the world of embedded systems, and we often implement it by adding some sort of non-volatile memory device into the system (flash, battery-backed DRAM, etc) and reading and writing to that device. Since you likely don't have the same level of control over the available hardware as embedded developers do, the closest analogue I can think of would be to reserve a chunk of space on a physical disk that you can read from without having to mount as a filesystem. A dedicated mini-partition might work the best because if you know the size of it, you can treat it as one big raw-access buffer and can avoid having to hassle with filenames, filesystems, etc.