I would like to know which entry under /dev a file is in. For example, if /dev/sdc1 is mounted under /media/disk, and I ask for /media/disk/foo.txt, I would like to get /dev/sdc as response.
Using stat system call on that file I will get its partition major and minor numbers (8 and 33, for sdc1). Now I need to get the "root" device (sdc) or its major/minor from that. Is there any syscall or library function I could use to link a partition to its main device? Or even better, to get that device directly from the file?
brw-rw---- 1 root floppy 8, 32 2011-04-01 20:00 /dev/sdc
brw-rw---- 1 root floppy 8, 33 2011-04-01 20:00 /dev/sdc1
Thanks in advance!
The quick and dirty version: df $file | awk 'NR == 2 {print $1}'.
Programmatically... well, there's a reason I started with the quick and dirty version. There's no portable way to programmatically get the list of mounted filesystems. (getmntent() gets fstab entries, which is not the same thing.) Moreover, you can't even parse the output of mount(8) reliably; on different Unixes, the mountpoint may be the first or the last item. The most portable way to do this ends up being... parsing df output (And even that is iffy, as you noticed with the partition number.). So you're right back to the quick and dirty shell solution anyway, unless you want to traverse /dev and look for block devices with matching major(st_rdev) (major() being from sys/types.h).
If you restrict this to Linux, you can use /proc/mounts to get the list of mounted filesystems. Other specific Unixes can similarly be optimized: for example, on OS X and I think FreeBSD, you can use sysctl() on the vfs tree to get mountpoints. At worst you can find and use the appropriate header file to decipher whatever the mount table file is (and yes, even that varies: on Solaris it's /etc/mnttab, on many other systems it's /etc/mtab, some systems put it in /var/run instead of /etc, and on many Linuxes it's either nonexistent or a symlink to /proc/mounts). And its format is different on pretty much every Unix-like OS.
The information you want exists in sysfs which exposes the linux device tree. This models the relationships between the devices on the system and since you are trying to determine a parent disk device from a partition, this is the place to look. I don't know if there are any hard and fast rules you can rely on to stop your code breaking with future versions of the kernel, but the kernel developers do try to maintain sysfs as a stable interface.
If you look at /sys/dev/block/<major>:<minor>, you'll see it is a symlink with the tail components being block/<disk-device-name>/<partition-device-name>. If you were to perform a readlink(2) system call on that, you could parse the link destination to get the disk device name. In shell (since it's easier to express this way, but doing it in C will be pretty easy):
$ echo $(basename $(dirname $(readlink /sys/dev/block/8:33)))
sdc
Alternatively, you could take advantage of the nesting of partition directories in the disk directories (again in shell, but from C, its an open(2), read(2), and close(2)):
$ cat /sys/dev/block/8:33/../dev
8:32
That assumes your starting major:minor is actually for a partition, not some other sort of non-nested device.
What you looking for is impossible - there is no 1:1 connection between a block device file and the partition it is describing.
Consider:
You can create multiple block device files with different names (but the same major and minor numbers) and they are indistinguishable (N:1)
You can use a block device file as an argument to mount to mount a partition and then delete the block device file leaving the partition mounted. (0:1)
So there is no way to do what you want except in a few specific and narrow cases.
Major number will tell you which device it is: 3 - IDE on 1st controller, 22 - IDE on 2nd controller and 8 for SCSI.
Minor number will tell you partition number and - for IDE devices - if it's primary or secondary drive. This calculation is different for IDE and SCSI.
For IDE it is: x*64 + p, x is drive number on the controller (0 or 1) and p is partition
For SCSI it is: y*16 + p, where y is drive number and p is partition
Not a syscall, but:
df -h /path/to/my/file
From https://unix.stackexchange.com/questions/128471/determine-what-device-a-directory-is-located-on
So you could look at df's source code and see what it does.
I realize this post is old, but this question was the 2nd result in my search and no one has mentioned df -h
Related
i got the problem that a device isnt booting up into linux.
It just holds on "Starting kernel ...".
To get a better grip on what goes wrong i thought it would be nice to get access to the logs from linux.
I can access the userland from uboot via "ls":
Zynq> ls mmc 0:2
ostree/deploy/poky/deploy/9d325972b955e6584d3fad0a7ff1bf1a8.0/etc
<DIR> 2048 .
<DIR> 1024 ..
<DIR> 1024 modprobe.d
0 motd
<DIR> 1024 xdg
<DIR> 1024 logrotate.d
58 rpcbind.conf
1633 inputrc
828 mke2fs.conf
15 timestamp
10929 login.defs
324 issue
<DIR> 1024 sudoers.d
etc ...
Now im looking for a way to copy files from the userland to another device(remote-pc).
I learned about "tftpput" which is available in uboot.
My problem is that "tftpput" expects a save address and size. But i dont know how to get those information.
tftpput - TFTP put command, for uploading files to a server
Usage:
tftpput Address Size [[hostIPaddr:]filename]
I was not able to find a good documentation on "tftpput". Maybe someone has a link for me or provide me a small "how to" about this?
Thanks in advance
To answer the specific question, you need a tftp server on another machine. Then when you use 'load' to bring a file into memory you will now have that address, $filesize will now be set for you (for the size parameter) and the machine you setup a tftp server on is the final part of the command.
That said, if you only see "Starting kernel" and nothing else, it is quite likely that the linux kernel isn't getting to the point where the rootfs is mounted, userland runs and you're going to see log files. Without more information it's hard to say what you need to do here, but your bootargs are the first place to make sure are correct.
To analyze why the kernel is not booting you could enable the early console.
For ARM 64bit systems the early console is enabled via the kernel command line parameters. U-Boot takes these from the environment variable bootargs.
The arguments for earlycon depend on your board, e.g. for the Odroid C2:
setenv bootargs earlycon=meson,0xc81004c0
For an early console on 32bit ARM system you will have to compile the kernel with appropriate configuration options, e.g. for the Banana Pi:
CONFIG_DEBUG_LL=y
CONFIG_DEBUG_SUNXI_UART0=y
CONFIG_EARLY_PRINTK=y
lets assume that file.txt has 16bytes of size (it is 10 in hex)
First it is necessary load the file into the memory
fatload mmc 1:1 0x40400000 file.txt
Then you can send it to tftp server:
tftpput 0x40400000 10 192.168.7.1:filetxt
I created a ~2MiB file.
dd if=/dev/urandom of=file.bin bs=2M count=1
Then I copied that file a large number of times and generated a checksum for each (identical) copy.
for i in `seq 50000`;
do
name="file.${i}.bin"
cp file.bin "${name}"
sha512sum "${name}" > "${name}.sha512"
done
I then verified all of those checksummed files with a validation script to run sha512sum against each file.
for file in `find . -regex ".*\.sha512"`
do
sha512sum --check --quiet "${file}" || (
cat "${file}" && sha512sum "${file%.sha512}"
)
done
I just created these files, and when I validate them moments later, I see intermittent failures and inconsistencies in the data (console text truncated for readability)
will:/mnt/usb $ for file in `find ...
file.5602.bin: FAILED
sha512sum: WARNING: 1 computed checksum did NOT match
91fc201a3812e93ef3d4890 ... file.5602.bin
b176e8e3ea63a223130f3a0 ... ./file.5602.bin
The checksum files are all identical since the source files are all identical
The problem seems to be that my computer is, seemingly at random, generating the wrong checksum for some of my files when I go to validate. A different file fails the checksum every time, and files that previously failed will pass.
will:/mnt/usb $ for file in `find ...
sha512sum: WARNING: 1 computed checksum did NOT match
91fc201a3812e93ef3d4890 ... file.3248.bin
442a1d8805ed134c9ab5252 ... ./file.3248.bin
Keep in mind that all of these files are identical.
I see the same behavior with SATA SSD and HDD, and USB devices, with md5 and sha512, with xfs, btrfs, ext4, and vfat. I tried live booting to another OS. I see this same stranger behavior regardless. I also see rsync --checksum for these files thinks checksums are wrong and re-copies these files even though they have not changed.
What could explain this behavior? Since it's happening on multiple devices with all the scenarios I described, I doubt this is bit rot. My kernel logs show no obvious errors. I would assume this is a hardware issue based on my troubleshooting, but how can this be diagnosed? Is it the CPU, the motherboard, the RAM?
What could explain this behavior? How can this be diagnosed?
From what I've read, a number of issues could explain this behavior. Bad disk(s), bad PSU (power supply), bad RAM, filesystem issues.
I tried the following to determine what was happening. I repeated the experiment with different...
Disks
Types of disks (SDD vs HDD)
External drives (3.5 and 2.5 enclosures)
Flash drives (USB 2 and 3 on various ports)
Filesystems (ext4, vfat (fat32), xfs, btrfs)
Different PSU
Different OS (live boot)
Nothing seemed to resolve this.
Finally, I gave memtest86+ v5.0.1 a try via an Ubuntu live USB.
voila. It found bad memory. Through process of elimination I determined one of my memory sticks was bad, and then tested the other over night to ensure it was in good shape. I re-ran my experiment again and I am seeing consistent checksums on all my files.
What a subtle bug. I only noticed this bad behavior by accident. If I hadn't been messing around with file checksums, I do not think I would have found this bad RAM.
This makes me want to regularly schedule a routine in which I verify and test my RAM. A consequence of this bad memory stick is that some of my test data did end up corrupt, but more often than not, the checksum verifications were just interimmitent failures.
In one sample data pool, all the checksums start with cb2848ca0e1ff27202a309408ec76..., because all ~50,000 files are identical.
Though, there are two files that are corrupt, but this is not bit rot or file integrity damage.
What seems most likely is that these files were created with corruption because cp encountered bad RAM when I created these files. Those files consistently return bad checksums of 58fe24f0e00229e8399dc6668b9... and bd85b51065ce5ec31ad7ebf3..., while the other 49,998 files return the same checksum.
This has been a fun extremely frustrating experiment in debugging.
I have an application which needs to read information from a hard disk, stuff like serial model etc.
Now of course it matters if the drive is a SAS, SATA or FC drive.
Is there a reliable way that I can identify which protocol a connected drive uses? Either via an OS command or checking some logs or inquiring the device?
I don't want to use sysfs structure. I want to know how the OS know if it's an ATA, SCSI or whatever type of disk.
As you have mentioned in comments to user3588161's answer, you are having SATA and SAS disk attached to the same SAS controller, so I'd suggest to use the smartctl command!
The smartctl command act as a control and monitor Utility for SMART disks under Linux and Unix like operating systems. Type the following command to get information about /dev/sda (SATA disk):
# smartctl -d ata -a -i /dev/sda
For SAS disk use one of the following syntax:
# smartctl -d scsi --all /dev/sgX
# smartctl -d scsi --all /dev/sg1
# smartctl -d scsi --all /dev/sg1 -H
I guess all of the information is somehow related to this location :-
/sys/class/scsi_device/?:?:?:?/device/model
I suggest you try doing this too to check what output does it render.
cat /sys/class/scsi_device/0\:0\:0\:0/device/{model,vendor}
(The backslashes next to zeros are for escaping special char :.)
Also, I'd like to suggest you to visit these two links in order for more information or detail like sample output,etc :-
Find Out Hard Disk Specs
To Check Disk behind Adaptec RAID Controllers
Checking boot information, it seems the disk type is set in kernel ahci calls. You can check (as root) with dmesg | grep ahci (on sysvinit systems) or with journalctl -k -b -0 -l --no-pager | grep ahci (with systemd). The relevant query/setting looks to be:
kernel: ahci 0000:00:12.0: version 3.0
kernel: ahci 0000:00:12.0: controller can't do 64bit DMA, forcing 32bit
kernel: ahci 0000:00:12.0: AHCI 0001.0100 32 slots 4 ports 3 Gbps 0xf impl SATA mode
kernel: ahci 0000:00:12.0: flags: ncq sntf ilck pm led clo pmp pio slum part ccc
The third line holds the controller/type information you are looking for. This seems to be where the information comes from, but from your questions standpoint, it isn't a viable solution.
The question becomes where does this information get recorded or stored within /dev /proc or /sys. I have looked and cannot find a one-to-one correlation between this initial determination of disk type on boot and any flag stored. This information may well be part of the coded data, for example, /sys/class/scsi_disk/0:0:0:0/device or similar location. Hopefully this information may allow you or others to help pinpoint if, and if so, where this information is captured and available on a running system.
Answer rewritten in view of clarification: libATA is what you want. It's what hdparm calls and it reports the transport too. It's hard to find up to date docs on it though. See http://docs.huihoo.com/linux/kernel/2.6.26/libata/index.html for example.
I have not used libATA (directly) myself, so I can't be more specific as to the API calls needed. Since not many people need to write something like hdparm themselves, your best bet is to consult its sources to see what exactly it calls.
hdparm can report stuff like:
[root#alarmpi ~]# hdparm -I /dev/sdb
/dev/sdb:
ATA device, with non-removable media
Model Number: TOSHIBA DT01ACA200
Serial Number: Z36GKMKGS
Firmware Revision: MX4OABB0
Transport: Serial, ATA8-AST, SATA 1.0a, SATA II Extensions, SATA Rev 2.5, SATA Rev 2.6, SATA Rev 3.0; Revision: ATA8-AST T13 Project D1697 Revision 0b
If your actual problem is that only sdparm works on your system for SCSI drives (can happen) then it seems the problem is reduced to figuring out which of hdparm or sdparm to call isn't it? You could use udevinfo for that. See https://chromium.googlesource.com/chromiumos/third_party/laptop-mode-tools/+/775acea9e819bdee90cca8d2363827c13967a14b/laptop-mode-tools_1.52/usr/share/laptop-mode-tools/modules/hdparm for example.
On a typical OS how many files can i have opened at once using standard C disc IO?
I tried to read some constant that should tell it, but on Windows XP 32 bit that was a measly 20 or something. It seemed to work fine with over 30 though, but i haven't tested it extensively.
I need about 400 files opened at once at max, so if most modern OS's support that, it would be awesome. It doesn't need to support XP but should support Linux, Win7 and recent versions of Windows server.
The alternative is to write my own mini file system which i want to avoid if possible.
On Linux, this is dependent on the amount of available file descriptors.
You can use ulimit -n to set / show the number of available FD's per shell.
See these instructions to how to check (or change) the value of available total FD:s in Linux.
This IBM support article suggests that on Windows the number is 512, and you can change it in the registry (as instructed in the article)
As open() returns the fd as int - size of int limits also the upper limit.
(irrelevant as INT_MAX is a lot)
A process can query the limit using the getrlimit system-call.
#include<sys/resource.h>
struct rlimit rlim;
getrlimit(RLIMIT_NOFILE, &rlim);
printf("Max number of open files: %d\n", rlim.rlim_cur-1);
FYI, as root, you have first to modify the 'nofile' item in /etc/security/limits.conf . For example:
* hard nofile 10240
* soft nofile 10240
(changes in limits.conf typically take effect when the user logs in)
Then, users can use the ulimit -n bash command. I've tested this with up to 10,240 files on Fedora 11.
ulimit -n <max_number_of_files>
Lastly, all this is limited by the kernel limit, given by: (I guess you could echo a value into this to go even higher... at your own risk)
cat /proc/sys/fs/file-max
Also, see http://www.karakas-online.de/forum/viewtopic.php?t=9834
I'd like to create a sparse file such that all-zero blocks don't take up actual disk space until I write data to them. Is it possible?
There seems to be some confusion as to whether the default Mac OS X filesystem (HFS+) supports holes in files. The following program demonstrates that this is not the case.
#include <stdio.h>
#include <string.h>
#include <fcntl.h>
#include <unistd.h>
void create_file_with_hole(void)
{
int fd = open("file.hole", O_WRONLY|O_TRUNC|O_CREAT, 0600);
write(fd, "Hello", 5);
lseek(fd, 99988, SEEK_CUR); // Make a hole
write(fd, "Goodbye", 7);
close(fd);
}
void create_file_without_hole(void)
{
int fd = open("file.nohole", O_WRONLY|O_TRUNC|O_CREAT, 0600);
write(fd, "Hello", 5);
char buf[99988];
memset(buf, 'a', 99988);
write(fd, buf, 99988); // Write lots of bytes
write(fd, "Goodbye", 7);
close(fd);
}
int main()
{
create_file_with_hole();
create_file_without_hole();
return 0;
}
The program creates two files, each 100,000 bytes in length, one of which has a hole of 99,988 bytes.
On Mac OS X 10.5 on an HFS+ partition, both files take up the same number of disk blocks (200):
$ ls -ls
total 400
200 -rw------- 1 user staff 100000 Oct 10 13:48 file.hole
200 -rw------- 1 user staff 100000 Oct 10 13:48 file.nohole
Whereas on CentOS 5, the file without holes consumes 88 more disk blocks than the other:
$ ls -ls
total 136
24 -rw------- 1 user nobody 100000 Oct 10 13:46 file.hole
112 -rw------- 1 user nobody 100000 Oct 10 13:46 file.nohole
As in other Unixes, it's a feature of the filesystem. Either the filesystem supports it for ALL files or it doesn't. Unlike Win32, you don't have to do anything special to make it happen. Also unlike Win32, there is no performance penalty for using a sparse file.
On MacOS, the default filesystem is HFS+ which does not support sparse files.
Update: MacOS used to support UFS volumes with sparse file support, but that has been removed. None of the currently supported filesystems feature sparse file support.
This thread becomes a comprehensive source of info about the sparse files. Here is the missing part for Win32:
Decent article with examples
Tool that estimates if it makes sense to make file as sparse
Regards
hdiutil can handle sparse images and files but unfortunately the framework it links against is private.
You could try defining external symbols as defined by the DiskImages framework below but this is most likely not acceptable for production code, plus since the framework is private you'd have to reverse engineer its use cases.
cristi:~ diciu$ otool -L /usr/bin/hdiutil
/usr/bin/hdiutil:
/System/Library/PrivateFrameworks/DiskImages.framework/Versions/A/DiskImages (compatibility version 1.0.8, current version 194.0.0)
[..]
cristi:~ diciu$ nm /System/Library/PrivateFrameworks/DiskImages.framework/Versions/A/DiskImages | awk -F' ' '{print $3}' | c++filt | grep -i sparse
[..]
CSparseFile::sector2Band(long long)
CSparseFile::addIndexNode()
CSparseFile::readIndexNode(long long, SparseFileIndexNode*)
CSparseFile::readHeaderNode(CBackingStore*, SparseFileHeaderNode*, unsigned long)
[... cut for brevity]
Later Edit
You could use hdiutil as an external process and have it create an sparse disk image for you. From the C process you would then create a file in the (mounted) sparse disk image.
If you seek (fseek, ftruncate, ...) to past the end, the file size will be increased without allocating blocks until you write to the holes. But there's no way to create a magic file that automatically converts blocks of zeroes to holes. You have to do it yourself.
This may be helpful to look at (the OpenBSD cp command inserts holes instead of writing zeroes).
patch
If you want portability, the last resort is to write your own access function so that you manage an index and a set of blocks.
In essence you manage a single file as the OS manages the disk keeping the chain of the blocks that are part of the file, the bitmap of allocated/free blocks etc.
Of course this will lead to a non optimized and slower access, I would reccomend this apprach only if the requirement to save space is absolutely critical and you have enough time to write a robust set of access functions.
And even in that case, I would first investigate if your problem is in need of a different solution. Probably you should store your data differently?
It looks like OS X supports sparse files on UDF volumes. I tried titaniumdecoy's test program on OS X 10.9 and it did generate a sparse file on a UDF disk image. Also, not that UFS is no longer supported in OS X, so if you need sparse files, UDF is the only natively supported file system that supports them.
I also tried the program on SMB shares. When the server is Ubuntu (ext4 filesystem) the program creates a sparse file, but 'ls -ls' through SMB doesn't show that. If you do 'ls -ls' on the Ubuntu host itself it does show the file is sparse. When the server is Windows XP (NTFS filesystem) the program does not generate a sparse file.