i know the cluster size of a USB flash drive can be changed , can we change the sector size too ??
Sector size isn't a configurable parameter in ATA/SATA/SCSI/etc devices and, from my experience, USB flash drives implement one of these protocols. The sector size is reported by the device itself but, even if you could set it to something other than 512, you would likely run into a latent bug somewhere in a driver or file system package that assumed a sector size of 512.
There are real reasons for using a sector size like 512, for example, addressing of larger sectors can be done more quickly and efficiently (not just in time but in size/space as well). Throughput to these devices is also better with something like 512. Consider that, if you could set sector size to something like 16-bytes, you might have less wasted space with 16-byte sectors compared to having a number of half-full 512-byte sectors, but your throughput to the device would probably be worse. In fact, writing a single 16-byte sector would only be slightly faster than writing a single 512-byte sector. On the other hand, writing 32 16-byte sectors (to write a total of 512 bytes) would likely take longer than writing a single 512 byte sector simply due to the overhead associated with transferring multiple sectors.
I would suggest you buy a larger USB flash drive if you are worried about wasting space with 512-byte sectors.
Related
Can get size of disk sector via the Linux API/ABI? It's about the quantum of I/O disk, normally it's equal 512 bytes, but others values can be too (usually multiple 512 bytes).
Also it should not confuse to size of logical block or to size of sector of a file system.
A block device is reflected as file in a file system of an UNIX (/dev/sda, /dev/sr etc.) It means, can open that file and make some manipulations to its content like with content of the corresponded block device.
So specifically the work to a true block device similar the work to a virtual hard disk (the .vhd format for instance).
But i don't know how to get size of sector in general case.
At moment i've single solution: get the maximal CHS address and size of hard drive, both action via BIOS. But i think, it's bad idea, because portability lost
I have been asked to help out on an embedded firmware project where they are trying to mount a file system on an SPI flash chip (Cypress S25FL512S) with a 256KB (Kilo Byte) Erase sector size.
My past experience with file systems is that the file system has a block size upto 4Kbytes which is mapped onto erase sectors of 512bytes to 4Kbytes
The embedded controller is a small NXP device running at 180MHz with 512KBytes of RAM so I cannot even cache an erase sector. I note that the chip family does have pin compatible devices with smaller erase sectors.
My general question is how do you mount a file system with a block/cluster size that is smaller than the flash erase sector size? I've not been able to find any articles addressing this.
You can't do this in any sensible way. Your specification needs to be modified.
Possible solutions are:
Pick a flash/eeprom circuit with smaller erase size.
Pick a flash/eeprom with more memory and multiple segments, so that you can back-up the data in one segment while programming another.
Add a second flash circuit which mirrors the first one, erase one at a time and overwrite with contents of the other.
Pick a MCU with more RAM.
Backup the flash inside MCU flash (very slow and likely defeats the purpose of having external flash to begin with).
I have a project where I have to update the data on disk very frequently in case of power loss. When overwriting exactly 512b (1 sector of my drive) in the file with data of equal length does the file-system mark the sectors that have been changed and update them on disk when ready to flush? Or does it write the whole file every time it flushes a change? I am mainly concerned with ext4 but I am curious if it is the same with every file-system.
If the standard is not to track changes but to overwrite the entire file is there a way to change this? Some write options?
In general with Linux files are cached in the page cache, and whether or not a page is dirty is tracked at the page level. On Intel x86 platforms, the page size is 4k, so if you dirty a 4k page, it is the 4k page which gets written back.
If you want to only overwrite a single 512 byte sector, and you have a HDD that has 512 byte sectors, you can open the file with the O_DIRECT flag and if you issue a 512 byte write, on a file offset which is a multiple of 512 bytes, and where the memory buffer from where you source the write is also 512 byte aligned, then you can bypass the page cache, and the write will go directly to the disk (hence O_DIRECT).
Note that a number of modern disks are really using a 4k physical sector, but they are emulating 512 byte sectors for backwards compatibility reasons. These disks are sometimes called 512e sectors (e for emulated). On these drives, if you do a 512 byte sector write, the disk will do a read-modify-write cycle, since the drive internally can only write 4k at a time. This will be visible to you as a performance hit, but from a functional perspective, it will otherwise look the same as a traditional, old-fashioned 512 sectored HDD.
I have ported FATFS for Free RTOS on STM32F103 SPI Flash of 32 Mbit. In a demo Application I have successfully created a file, written a file, and read back from the file. My requirement is like I have to store multiple files (images) in SPI flash and read it back when required.
I have the following conditions/queries.
I have set the Sector Size to 512 bytes, and block erase size for SPI flash is 4K. As in the SPI Flash, block needs to be erased before written. Do I need to keep track on whether a particular block is erased or not or its the file System who is managing this?
How can I verify that the sector, which I am writing in erased or not? What I am currently doing is, Erase the Complete Block for the sector, which I am going to write?
How can I make sure, The Block for SPI flash I am going to erase will not affect any Sector containing useful data?
Thanking in an Anticipation,
Regards,
AK
The simplest solution is to define the "cluster" size to 4K, the same as the page size of your flash. That mean each file, even if only 1 byte, takes 4K, which is 8 consecutive sectors of 512 bytes each.
As soon as you need to reserve one more cluster, when the file grow above 4096 bytes, you pick a free cluster, chain it to the FAT, and write the next byte.
For performance reason and to increase the durability of the flash, you should avoid to erase of flash sector when not needed. It is many order of magnitude faster to read then erasing. So, as you select a free cluster, you can start a loop to read each of the 8 sectors. As soon as you find even a single byte not equal to 0xFF, then you abort the loop and call the flash erase for that sector.
A further optimization is possible if the flash controller is able to perform the blank test directly. Such test could be done in a few microsecond while reading 8 sectors and looping to check each of the 4096 bytes is probably slower.
I have understood why memory should be aligned to 4 byte and 8 byte based on data width of the bus. But following statement confuses me
"IoDrive requires that all I/O performed on a device using O_DIRECT must be 512-byte
alligned and a multiple of 512 bytes in size."
What is the need for aligning address to 512 bytes.
Blanket statements blaming DMA for large buffer alignment restrictions are wrong.
Hardware DMA transfers are usually aligned on 4 or 8 byte boundaries since the PCI bus can physically transfer 32 or 64bits at a time. Beyond this basic alignment, hardware DMA transfers are designed to work with any address provided.
However, the hardware deals with physical addresses, while the OS deals with virtual memory addresses (which is a protected mode construct in the x86 cpu). This means that a contiguous buffer in process space may not be contiguous in physical ram. Unless care is taken to create physically contiguous buffers, the DMA transfer needs to be broken up at VM page boundaries (typically 4K, possibly 2M).
As for buffers needing to be aligned to disk sector size, this is completely untrue; the DMA hardware is completely oblivious to the physical sector size on a hard drive.
Under Linux 2.4 O_DIRECT required 4K alignment, under 2.6 it's been relaxed to 512B. In either case, it was probably a design decision to prevent single sector updates from crossing VM page boundaries and therefor requiring split DMA transfers. (An arbitrary 512B buffer has a 1/4 chance of crossing a 4K page).
So, while the OS is to blame rather than the hardware, we can see why page aligned buffers are more efficient.
Edit: Of course, if we're writing large buffers anyways (100KB), then the number of VM page boundaries crossed will be practically the same whether we've aligned to 512B or not.
So the main case being optimized by 512B alignment is single sector transfers.
Usually large alignment requirements like that are due to underlying DMA hardware. Large block transfers can sometimes be made much faster by requiring much stronger alignment restrictions than what you have here.
On several ARM processors, the first level translation table has to be aligned on a 16 KB boundary!
If you don't know what you're doing, don't use O_DIRECT.
O_DIRECT means "direct device access". This means it bypasses all OS caches, hitting the disk (or possibly RAID controller, etc) directly. Disk accesses are on a per-sector basis.
EDIT: The alignment requirement is for the IO offset/size; it's not usually a memory-alignment requirement.
EDIT: If you're looking at this page (it appears to be the only hit), it also says that the memory must be page-aligned.