NAND flash device has Block size 16384, page size 512, OOB size 16 bytes.
A partition dump (cleaned from OOB data) is 13548080 bytes in size, so its not multiple of 512. Since all writes must be 512 byte aligned, and the blob size must be a multiple of 512 bytes, I should to add 'n' bytes at the end of the binary (fill with 'FF').
13548080 bytes is incomplete 26462 pages. A complete 26462 pages is 26462 x 512 = 13548544 bytes. The difference is 464 bytes that I need to add.
I tried two step way: first, I created 464 bytes padded.bin file dd if=/dev/zero bs=1 count=464 | tr '\000' '\377' >padded.bin, then appended the original file to padded file:dd if=padded.bin bs=1 count=464 >>original.bin
Perhaps there is another way to append n bytes at the end of file use shell command?
Since the number of pages is known, you can use the count= operand of dd; the desired 'FF' bytes can be provided by tr.
(cat inputfile; tr </dev/zero \\0 \\377) | dd count=26462 iflag=fullblock >outputfile
Related
So to pull super block in file system (i.e. if my sda storage is ext2 formatted) is easy. I just need to skip 1024 bytes to get the super block fro sda storage
lseek(fd, 1024, SEEK_SET);
read(fd, &super_block, sizeof(super_block));
and to pull the group descriptor is also super easy (only if I understood correctly from looking at code)
lseek(fd, 1024 + [block_size_ext_1024_bytes]=1024, SEEK_SET);
read(fd, &block_group, sizeof(block_group));
or
lseek(fd, 1024 + 1024, SEEK_SET);
read(fd, &block_group, sizeof(block_group));
1024=Base offset
But I am not feeling at confort because the real challege I found is to pull inode is only I have file name. I know file names are stored in directory struct so first challege is to extract directory struct from there and in directory struct I can get the inode number. and from Inode number I can extract inode struct. but I do not know how to extract directory struct in ext2 formatted image. Can anyone please telll me this? thanks
Yes pulling super block is just a matter of skipping Base_Offset=1024 bytes in ext2 and then reading it like so
lseek(fd, BASE_OFFSET + block_size, SEEK_SET);
//BASE_OFFSET for EXT2 == 1024
read(fd, &super_block, sizeof(super_block));
block_size = 1024 << super_block.s_log_block_size;
printf("Block size is [%d]\n",super_block.s_log_block_size);
The size of a super-block is given by s_log_block_size. This value expresses the size of a block as a power of 2, using 1024(specifically for ext2) bytes as the unit. Thus, 0 denotes 1024-byte blocks, 1 denotes 2048-byte blocks, and so on. To calculate the size in bytes of a block:
unsigned int block_size = 1024 << super.s_log_block_size; /* block
super.s_log_block_size always 0 if need to hardcode 1024 and super.s_log_block_size is multiple of 2 so if 1024 block size super.s_log_block_size is should be 0
Then I can extract group descriptor. So for my image there is only one group descrptor. I dont know how many descriptor will I have if I have 1TB of storage as I do have this and file system is ext4. May be someone will tell me this.
Like this to extract group descriptor by further moving forward 1024 bytes
lseek(fd, BASE_OFFSET + block_size, SEEK_SET);
read(fd, &block_group, sizeof(block_group));
I think this gives the idea of finding out how many group desciptors are there in storage in ext2
unsigned int group_count = 1 + (super_block.s_blocks_count-1) / super_block.s_blocks_per_group;
so for example On my device image it has 128 blocks so first block always Boot info, second block contains super block, the third block contains first group descriptor -- still like to know what would be the offset of my second group descriptor if I had more space on my storage. Please someone shed light on this
Moving on, to extract specific inode the formula is this to seek the offset of specific inode
lseek(fd, BLOCK_OFFSET(block_group->bg_inode_table)+(inode_no-1)*sizeof(struct ext2_inode),
SEEK_SET);
bg_inode_table can be used to extract inode
The group descriptor tells us the location of the block/[inode bitmaps] and of the inode table (described later) through the bg_block_bitmap, bg_inode_bitmap and bg_inode_table fields.
Now to extract root inode=(should be ino_num=2) for example I just need to do
lseek(fd, BLOCK_OFFSET(block_group->bg_inode_table)+(2-1)*sizeof(struct ext2_inode),
SEEK_SET);
The block number of the first block of the inode table is stored in the bg_inode_table field of the group descriptor.
so inode table came to help in find specific inode
To extract the directory struct I just need to use inode.i_block[0] array. filled in last step
each i_block element is number that can be used in this way. basically a pointer points to actual blocks containing content of file with inode
lseek(...BASE_OFFSET+(i_block[x]-1)*block_size...)
block_size always 1024 for ext2
This way I can read the block at whose base contain directory struct in ext2 file system
read
void *block;
read(fd, block, block_size);
and the above line give me first directory mapped to specific inode
I can simple do a loop to get all entries
http://www.science.smith.edu/~nhowe/teaching/csc262/oldlabs/ext2.html
I want to use mmap() to read a file with fixed length (eg. 64MB), but there also some files < 64MB.
I mmap this files (<64MB, eg.30MB) with length = 64MB, when read file data beyond file size(30MB - 64MB), the program got a bus-error.
I want mmap these files with fixed length, and read 0x00 when pointer beyond file size. How to do that?
One method I can think is ftruncate file first, and ftruncate back to ori size, but I don't think this method is perfect.
This is one of the few reasonable use cases for MAP_FIXED, to remap part of an existing mapping to use a new backing file.
A simple solution here is to unconditionally mmap 64 MB of anonymous memory (or explicitly mmap /dev/zero), without MAP_FIXED and store the resulting pointer.
Next, mmap 64 MB or your actual file size (whichever is less) of your actual file, passing in the result of the anonymous/zero mmap and passing the MAP_FIXED flag. The pages corresponding to your file will no longer be anonymous/zero mapped, and instead will be backed by your file's data; the remaining pages will be backed by the anonymous/zero pages.
When you're done, a single munmap call will unmap all 64 MB at once (you don't need to separately unmap the real file pages and the zero backed pages).
Extremely simple example (no error checking, please add it yourself):
// Reserve 64 MB of contiguous addresses; anonymous mappings are always zero backed
void *mapping = mmap(NULL, 64 * 1024 * 1024, PROT_READ, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
// Open file and check size
struct stat sb;
int fd = open(myfilename, O_RDONLY);
fstat(fd, &sb);
// Use smaller of file size or 64 MB
size_t filemapsize = sb.st_size > 64 * 1024 * 1024 ? 64 * 1024 * 1024 : sb.st_size;
// Remap up to 64 MB of pages, replacing some or all of original anonymous pages
mapping = mmap(mapping, filemapsize, PROT_READ, MAP_SHARED | MAP_FIXED, fd, 0);
close(fd);
// ... do stuff with mapping ...
munmap(mapping, 64 * 1024 * 1024);
I have a client server program in which i read the file from the server to the client in chunks, over TCP. However, not all chunks being sent are of the size I have given. Some of them are less than the requested size or even 1.
This is my server side code for reading from the file and writing to socket
rd=read(filed, buffer,128);
write(socketd,buffer,128);
This true for only non text files. On using a txt file, it works perfectly, but fails for pdfs, ppts, images. Is there any way to read these non text files in chunks?
Sample of size in bytes being read in each iteration for a pdf file
128
128
128
128
116
128
128
36
You should only write the amount that was actually read from the file. If the size of the file isn't a multiple of 128, then the last read can't return 128 bytes, it just returns what's left in the file. Use rd as the amount to write to the socket.
while ((rd = read(filed, buffer, 128)) > 0) {
write(socketd, buffer, rd);
}
Issue resolved. I was using strlen(buffer) which gave those numbers. The integer returned by read() was 128, as desired.
When we use read() function with file descriptor(not socket), it has byte-offset like this.
read(file descriptor, message1, 20);
read(file descriptor, message2, 40);
After number1 command, the file descriptor's byte offset is 20 as it says.
Then, number 2 command read the file at the byte offset 20, and the file descriptor's final offset is 60 now.
So, now my question is about socket. Let me explain you situations.
On server-client model, client sends server 100 bytes(lets assum it's string data). But, unfortunately, server receive data sequentially 10, 20, 70(not 100 bytes at once.)
So, because of this situation, server source code is like this.
51 while(recv_len<100)
52 {
53 recv_count=read(cli_sock, message[recv_len], 1024);
54
55 if(recv_count==-1)
56 error_handling("read() error");
57
58 recv_len+=recv_count;
59 }
So client writes 10 bytes data one of 100 bytes string to server, at first.
Anyway, server execute read() function, and reads 1024 bytes. So, the server takes 10 bytes. But, its byte-offset is 1024 now.
So, after client's second data(20 bytes data), server must read at byte offset 1024. Then, how can server reads 20 bytes which is sent from client? I mean originally, server must read 20 bytes at byte offset 10(not 1024), because 10 bytes data had sent before 20bytes. But, server is at byte offset 1024. How this can be possible?
This
recv_count=read(cli_sock, message[recv_len], 1024);
will not do what you think.
It should be :
recv_count=read(cli_sock, message + recv_len, 1024 - recv_len);
or
recv_count=read(cli_sock, &message[recv_len], 1024 - recv_len);
read the man page and if don't understand something, don't hesitate to ask for precision (even thought that blue112 already explain it all).
You've got it wrong. The third parameter of a read() call isn't an offset, it represents the number of bytes you want to read, at max.
The actual number read is returned by the read() call.
Look here : http://man7.org/linux/man-pages/man2/read.2.html
Also, you should pass a pointer to read() calls.
So message[recv_len] isn't gonna do. You're looking for message + recv_len
I am having a problem understanding how to find Block Group Descriptor table. In literature (D.Poirier: "The 2nd extended filesystem") is stated that block group descriptor is located in block right after superblock.
Now, when I look at first disk, with block size of 1024 bytes, structure is like this:
MBR, 0-512 bytes
Superblock, 1536-2560 bytes
BG Descriptor, 2560 - ... bytes
And this structure is fine, because superblock starts with 3rd sector and BGD follows right after. However, when I look at second disk with block size of 4096 bytes, structure is like this:
MBR, 0-512 bytes
Superblock, 1536-2560 bytes
BG Descriptor, 4608 - ... bytes
In this case, BGD is located 3072(?) bytes away from superblock. Could someone enlight me and tell me how exactly is BGD position determined, because I'm writing a program that reads and analyses ext structure, and I can't write a generic program that knows how to find BGD.
the BGD starts offset can vary depending on the block size (1k, 2k, 4k).
In a partition, the first 1024 bytes are reserved, then followed with 1024 bytes of SUPER BLOCK. Depending on the block size, the BGD starts from:
BLK=1K, the BGD starts at partition offset 2048 (1024 reserved + 1024 super block).
BLK=2K, the BGD starts at partition offset 2048 (1024 reserved + 1024 super block).
BLK=4K, the BGD starts at partition offset 4096, which is 1 block from start, that is the result you see 3072 bytes apart from the super block.