For those reaching here; Unfortunately I could not recover the data, after various tries and reproducing the problem it was too costy to keep trying, so we just used a past backup to recreate the information needed
A human error broke an 150G UFS filesystem (Solaris).
When trying to do a backup of the filesytem (c0t0d0s3) the ufsdump(1M) hasn't been correctly used.
I will explain the background that led to this ...
The admin used:
# ufsdump 0f /dev/dsk/c0t0d0s3 > output_1
root#ats-br000432 # ufsdump 0f /dev/dsk/c0t0d0s3 > output_1
Usage: ufsdump [0123456789fustdWwnNDCcbavloS [argument]] filesystem
This is a bad usage, so it created a file called output_1 with 0 bytes:
# ls -la output_1
-rw-r--r-- 1 root root 0 abr 12 14:12 output_1
Then, the syntax used was:
# ufsdump 0f /dev/rdsk/c0t0d0s3 output_1
Which wrote that 0 bytes file output_1 to /dev/rdsk/c0t0d0s3 - which was the partition slice
Now, interestingly, due to being a 0 bytes file, we thought that this would cause no harm to the filesystem, but it did.
When trying to ls in the mountpoint, the partition claimed there was an I/O error, when umounting and mounting again, the filesystem showed no contents, but the disk space was still showing as used just like it was previously.
I assume, at some point, the filesystem 'header' was affected, right? Or was it the slice information?
A small fsck try brings up this:
** /dev/rdsk/c0t0d0s3
** Last Mounted on /database
** Phase 1 - Check Blocks and Sizes
INCORRECT DISK BLOCK COUNT I=11 (400 should be 208)
CORRECT?
Disk block count / I=11
this seems that the command broke filesystem information regarding its own contents, right?
When we tried to fsck -y -F ufs /dev/dsk.. various files have been recovered, but not the dbf files we are after (which are GB sized)
What can be done now? Should I try every superblock information from newfs -N ?
EDIT: new information regarding partition
newfs output showing superblock information
# newfs -N /dev/rdsk/c0t0d0s3
Warning: 2826 sector(s) in last cylinder unallocated
/dev/rdsk/c0t0d0s3: 265104630 sectors in 43149 cylinders of 48 tracks, 128 sectors
129445,6MB in 2697 cyl groups (16 c/g, 48,00MB/g, 5824 i/g)
super-block backups (for fsck -F ufs -o b=#) at:
98464, 196896, 295328, 393760, 492192, 590624, 689056, 787488, 885920,
Initializing cylinder groups:
.....................................................
super-block backups for last 10 cylinder groups at:
264150944, 264241184, 264339616, 264438048, 264536480, 264634912, 264733344,
264831776, 264930208, 265028640
Related
about a week ago, I deleted all files from my hard drive using rm -rf * and rm -rf .*, thinking that all important files would be protected, but it seems like I was wrong.
After reboot, it wasn't able to mount the drive anymore.. I tried a couple more things to make it work (fill the disk with zeros, with the dd command), but I fear I only made things worse.
Now, lsblk command returns:
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 3,9G 0 disk
sdb 8:16 0 465,8G 0 disk
├─sdb1 8:17 0 448,6G 0 part /
└─sdb2 8:18 0 17,2G 0 part [SWAP]
The disk is supposed to be 1Tb, so there's a problem here. All of the partition or file system tools I tried get stuck when scanning the drive (I guess), including sfdisk, wipefs, fsck, and the gui KDE disk manager.
I'm under Manjaro, KDE, but that shouldn't be relevant...
Thanks!
I accidentally removed a wrong folder using rm -rf, every tool I've tried tell me either I have no hard disk or that the filesystem is not found.
When I type df I get :
Filesystem Size Used Avail Use% Mounted on
/dev/simfs 25G 7.6G 18G 31% /
none 256M 12K 256M 1% /dev
But if I type debugfs -w /dev/simfs I get the error :
/dev/simfs: No such file or directory while opening filesystem
I'm using a VPS hosted by OVH, what's wrong with that filesystem ?
I have the same scenario/issue. To take the conversation a step further, I've also tried
tune2fs -l /dev/simfs
which outputs
tune2fs 1.42.9 (4-Feb-2014)
tune2fs: No such file or directory while trying to open /dev/simfs
Couldn't find valid filesystem superblock.
You'll get the same invalid superblock error if you try to scan an entire disk eg /dev/sda
My Posix C program needs to grow a file to X bytes large - typically 128MB or 256MB.
The current approach is to initialise a memory buffer of 16MB and repeatedly write the buffer into the opened file using fwrite.
Is there a more efficient approach?
You can use the ftruncate system call.
You can quickly fill a file with all zeros by seeking to an offset and writing a byte there. The contents of the file before that offset will be filled with zeros if the file was not that large already.
On Linux this will create a sparse file. The file will appear be of size 256MB but it will actually use very little space on disk.
Use ftruncate , despite its name it can be used to extend files too. It might be slightly less portable compared to your current method as posix does not require ftruncate to be able to extend a file (but XSI does)
Note about sparse files - which all of the answers create for you
code:
// ft.c
int main()
{
FILE*fp=fopen("sparse", "w");
ftruncate(fileno(fp), 2147483647UL);
fclose(fp);
return 0;
}
show the disk before and after running ./ft:
me#mycomputer ~> gcc ft.c -o ft
me#mycomputer ~> du -sh .
368M .
me#mycomputer ~> df -h .
Filesystem Size Used Available Capacity Mounted on
/ 98G 8.3G 89G 9% /
me#mycomputer ~> ./ft
me#mycomputer ~> df -h .
Filesystem Size Used Available Capacity Mounted on
/ 98G 8.3G 89G 9% /
me#mycomputer ~> du -sh .
368M .
me#mycomputer ~> ls -l sparse
-rw-r--r-- 1 me other 2147483647 Dec 10 12:55 sparse
Now we back up sparse:
me#mycomputer ~> tar cvf /tmp/my.tar.tar ./sparse
a ./sparse 2097152K
Next we restore sparse from the backup:
me#mycomputer ~> tar xf /tmp/my.tar.tar .
me#mycomputer ~> du -sh .
2.4G .
me#mycomputer ~> df -h .
Filesystem Size Used Available Capacity Mounted on
/ 98G 10G 87G 11% /
Voila - we gained 2GB. Sparse files are a great way to create a sort of future negative payback - a possible disk full error. The above example is on the system disk - once the system disk is 100% full the system has all kinds of nasty problems.
This is because restoring files (months later maybe) inflates sparse files on disk to their "real" size. And you forgot about them in the meantime.
Seriously consider not creating these things in the first place, there usually are no requirements to do it.
I have partition structure like :
$ df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda6 51606140 16939248 34142692 34% /
/dev/sda5 495844 72969 397275 16% /boot
/dev/sda7 113022648 57515608 49765728 50% /home
/dev/sda8 113022648 57515608 49765728 4% /mnt
while parsing directories content using readdir() - how to find out which file resides on what device?
readdir() invoked from root directory and parses the file name and prints its size.
like from device : /dev/sda6 and list the filenames under that partition.
When it reads contents from /home - it should display reading content from /dev/sda7 and list filenames
Please let me know,if you need more details/info
you can just do
df <file_name>
that will give you the device and partition for the particuar file
There is a st_dev member in struct stat, it should uniquely identify one partition.
Example in bash:
stat ~/.vimrc
File: `/home2//leonard/.vimrc' -> `local-priv/vimrc'
Size: 16 Blocks: 0 IO Block: 4096 symbolic link
Device: 802h/2050d Inode: 6818899 Links: 1
Access: (0777/lrwxrwxrwx) Uid: ( 1024/ leonard) Gid: ( 1024/ leonard)
Access: 2012-06-22 16:36:45.341371003 +0300
Modify: 2012-06-22 16:36:45.341371003 +0300
Change: 2012-06-22 16:36:45.341371003 +0300
The stat utility does no additional magic. Here is strace -vvv output:
lstat64("/home2//leonard/.vimrc", {st_dev=makedev(8, 2), st_ino=6818899, st_mode=S_IFLNK|0777, st_nlink=1, st_uid=1024, st_gid=1024, st_blksize=4096, st_blocks=0, st_size=16, st_atime=2012/06/22-16:36:45, st_mtime=2012/06/22-16:36:45, st_ctime=2012/06/22-16:36:45}) = 0
0x0802 is major 8(sd) partition 2, so /dev/sda2
In order to map this to actual partitions you can iterate /proc/mounts and stat all the devices (first column). The contents of /proc/mounts is just like the output of mount(1) except it comes directly from the kernel. Some distros symlink /etc/mtab to /proc/mounts.
Or you can parse /proc/partitions:
$ cat /proc/partitions
major minor #blocks name
8 0 976762584 sda
8 1 3998720 sda1
8 2 972762112 sda2
Of course /dev/sda might not actually exist, the device could be using a long udev name like /dev/disk/by-uuid/c4181217-a753-4cf3-b61d-190ee3981a3f. Major/Minor numbers should be a reliable unique identifier of a partition.
I have a directory I’m archiving:
$ du -sh oldcode
1400848
$ tar cf oldcode.tar oldcode
So the directory is 1.4gb. The file is significantly smaller, though:
$ ls -l oldcode.tar
-rw-r--r-- 1 ieure ieure 940339200 2002-01-30 10:33 oldcode.tar
Only 897mb. It’s not compressed in any way:
$ file oldcode.tar
oldcode.tar: POSIX tar archive
Why is the tar file smaller than its contents?
You get a difference because of the way the filesystem works.
In a nutshell your disk is made out of clusters. Each cluster has a fixed size of - let's say - 4 kilobytes. If you store a 1kb file in such a cluster 3kb will be unused. The exact details vary with the kind of file-system that you use, but most file-systems work that way.
3kb wasted space is not much for a single file, but if you have lots of very small files the waste can become a significant part of the disk usage.
Inside the tar-archive the files are not stored in clusters but one after another. That's where the difference comes from.
Having no knowledge of what tar you're using or what sort of Unix system you're using, here's my guess: oldcode contains numerous smaller files, which when by themselves use disk space inefficiently, since disk space is allocated by some sort of block, rather than byte by byte. In the tar file, they're concatenated, and make maximum use of the disk space they're assigned.
This has something to do with the blocksize of your filesystem. man 1 du on MacOSX 10.5.6 states:
The du utility displays the file system block usage for each file argument and for each directory in the file hierarchy rooted in each directory argument. If no file is specified, the block usage of the hierarchy rooted in the current directory is displayed.
[mirko#borg foo]$ ls -la
total 0
drwxr-xr-x 2 mirko wheel 68 Jan 30 21:20 .
drwxrwxrwt 10 root wheel 340 Jan 30 21:16 ..
[mirko#borg foo]$ du -sh
0B .
[mirko#borg foo]$ touch foo
[mirko#borg foo]$ ls -la
total 0
drwxr-xr-x 3 mirko wheel 102 Jan 30 21:20 .
drwxrwxrwt 10 root wheel 340 Jan 30 21:16 ..
-rw-r--r-- 1 mirko wheel 0 Jan 30 21:20 foo
[mirko#borg foo]$ du -sh
0B .
[mirko#borg foo]$ echo 1 > foo
[mirko#borg foo]$ ls -la
total 8
drwxr-xr-x 3 mirko wheel 102 Jan 30 21:20 .
drwxrwxrwt 10 root wheel 340 Jan 30 21:16 ..
-rw-r--r-- 1 mirko wheel 2 Jan 30 21:20 foo
[mirko#borg foo]$ du -sh
4.0K .
As you see even a file of 2 bytes takes a whole block of 4kb. There are some filesystems which avoid this waste of space by block suballocation.
There are 2 possibilities.
Small files
Most likely, it isn't smaller than its contents. As Nils Pipenbrinck wrote, du displays the amount of space the filesystem allocates, which since files are stored in filesystem blocks is more than the logical size of the file.
To view the logical size of the file, use du --apparent-size. In this case, the result should be smaller than the tar file.
Sparse files
Tar files can store sparse files. If the tarball was created using --sparse, the holes in the sparse files will be recorded, so the tarball could be smaller than the logical size of the files.
If the sparseness information in your extracted copy was somehow lost (e.g. if you extracted the tarball onto a filesystem that doesn't support sparse files, or if it was zipped and then unzipped, etc.), then df will report the expanded size.
du counts disk blocks, not file size duder.