I am working on a final project for school that involves making a program that reads and writes to files on persistent memory. I don't have access to an actual persistent memory device so I followed the tutorial at the following link to emulate it:
https://kb.pmem.io/howto/100000012-How-To-Emulate-Persistent-Memory-using-the-Linux-memmapKernel-Option/
user#node:~/test2$ sudo fdisk -l /dev/pmem0
Disk /dev/pmem0: 8 GiB, 8589934592 bytes, 16777216 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 4096 bytes I/O size (minimum/optimal): 4096 bytes / 4096 bytes
According to the above output, I have an emulated persistent memory partition on my dram hard drive named pmem0. I used the tutorial at the following link to get some sample code on how to read and write from files on persistent memory:
https://www.intel.com/content/www/us/en/developer/articles/code-sample/create-a-persistent-memory-hello-world-program-using-libpmemobj-with-c.html
I compile and run the code and the output of the program says it is writing a file to persistent memory. I wanted to check if the file is being written to the correct partition so I used the command:
df -P fileName | tail -1 | cut -d' ' -f 1 (which I got from another stackoverflow post) and the output is /dev/sda1 when I believe it should be /dev/pmem0 if it is actually using the persistent memory partition.
I'm looking for tips on how to ensure I am actually mapping my code to the persistent memory partition I created.
The code from the second link needs to have the pmdk library setup to run properly and I needed to modify my linux kernel boot parameters to setup the partition so I'm not sure if I can supply a much better minimal example.
Edit: I believe the exact lines of code I need to modify would be one of:
pop = pmemobj_create(path, LAYOUT, PMEMOBJ_MIN_POOL, 0666);
PMEMoid root = pmemobj_root(pop, sizeof (struct my_root));
pmemobj_persist(pop, &rootp->len, sizeof (rootp->len));`
This creates the persistent memory pool but the tutorial doesn't seem to mention how to actually map to a persistent memory device.
why does the string get written to the hard drive in sda1, which I believe is regular memory?
The referred C program calls pmemobj_create() with the path you specify as the second program argument, in your case fileName. If the current working directory is on the hard drive, the created memory pool file fileName is of course also residing on the hard drive. If you want a file to be created on your /dev/pmem0 disk, you have to mount that file system and use a path to a file thereon.
Related
I know that file system use clusters (n x sectors (512 B) usualy 4KB in size) for storing files. If I have file of size 5 KB then it use two cluster to store and remaining space is called slack space. My question is related to situation where user read file from disk, modify (add few characters) and save this file again. What will happened, will OS (overwrite) write file from location from it started to read file, or file will be writen in new cluster completely, and address of file starting cluster will be erased and replaced with new cluster address.
new part:
I just read in a book "Information technologie:An Introduction for Today’s Digital World" that if file use 2 bloks (clusters) and second file use 4 consecutive blocks after first file. First file is edited and modified, his file size increased to total of 3 blocks. This file will be writen after second file and previously occupied 2 blocks are free. But still don t know what will happend if I for example increase file with one character and file is still smaller then total of 2 blocks. Will this data be added on existing file, to existing first two blocks, or it will be stored on new disk physical location (new 2 blocks)?
When user store file it will occupy some space on disk (cluster = combine several sectors = 4 KB since sector is usually 512 bytes). If file take 3KB then 1KB stay unused in this cluster. Now what will happened if I increase little file adding some data to this file. Answer now depend of procedure that user use to modify file.
1. If I manualy add data to file (using echo "some text" >> filename) this data will add this data in existing cluster since there is 1KB of space availabile. If file site increase it will take another free sectors (file use "extents" to address all this sectors)
2. If i use text editor it will copy file on other location on disc (because of multiuser and situation when two users access same file in a same time). Previous location will be "free" (content in sector stay but File system don t have reference to that) and replace with new location on disk.
Since majority of users use some editor for editing file then scenario 2 is most common.
I was trying to get number of blocks allocated to a file using C. I used the stat struct with its variable called st_blocks. However this is returning different number of blocks as compared to ls -s. Can anybody explain the reason for this and if there is a way to correct this?
There is no discrepancy; just a misunderstanding. There are two separate "block sizes" here. Use ls -s --block-size=512 to use 512 byte block size for ls, too.
The ls -s command lists the size allocated to the file in user-specified units ("blocks"), the size of which you can specify using the --block-size option.
The st_blocks field in struct stat is in units of 512 bytes.
You see a discrepancy, because the two "block sizes" are not the same. They just happen to be called the same name.
Here is an example that you can examine this effect. This works on all POSIXy/Unixy file systems (that support sparse file), but not on FAT/VFAT etc.
First, let's create a file that but is one megabyte long, but has a hole at the beginning (they read zeros, but are not actually stored on disk), with a single byte at end (I'll use 'X').
We do this by using dd to skip the first 1048575 bytes of the file (creating a "hole", and thus a sparse file on filesystems that support such):
printf 'X' | dd bs=1 seek=1048575 of=sparse-file count=1
We can use the stat utility to examine the file. Format specifier %s provides the logical size of the file (1048576), %b the number of blocks (st_blocks):
stat -c 'st_size=%s st_blocks=%b' sparse-file
On my system, I get st_size=1048576 st_blocks=8, because the actual filesystem block size is 4096 bytes (= 8×512), and this sparse file needs only one filesystem block.
However, using ls -s sparse-file I get 4 sparse-file, because the default ls block size is 1024 bytes. If I run
ls --block-size=512 -s sparse-file
then I see 8 sparse-file, as I'd expect.
"Blocks" here are not real filesystem blocks. They're convenient chunks for display.
st_blocks is using probably 512 byte blocks. See the POSIX spec.
st_blksize is the preferred block size for this file, but not necessarily the actual block size.
BSD ls -s always uses 512 byte "blocks". OS X, for example, uses BSD ls by default.
$ /bin/ls -s index.html
560 index.html
GNU ls appears to use 1K blocks unless overriden with --block-size.
$ /opt/local/bin/gls -s index.html
280 index.html
printf("%lld / %d\n", buf.st_blocks, buf.st_blksize); produces 560 / 4096. The 560 "blocks" are in 512 byte chunks, but the real filesystem blocks are 4k.
The file contains 284938 bytes of data...
$ ls -l index.html
-rw-r--r-- 1 schwern staff 284938 Aug 11 2016 index.html
...but we can see it uses 280K on disk or 70 bytes.
Note that OS X further confuses the issue by using 1000 bytes for a "kilobyte" instead of the correct 1024 bytes, that's why it says 287 KB for 70 4096 KB blocks (ie. 286720 bytes) instead of 280 KB. This was done because hard drive manufacturers started using 1000 byte "kilobytes" in order to inflate their size, and Apple got tired of customers complaining about "lost" disk space.
The 4K block size can be seen by making a tiny file.
Let me explain clearly.
The following is my requirement:
Let's say there is a command which has an option specified as '-f' that takes a filename as argument.
Now I have 5 files and I want to create a new file merging those 5 files and give the new filename as argument for the above command.
But there is a difference between
reading a single file and
merging all files & reading the merged file.
There is more IO (read from 5 files + write to the merged file + any IO our command does with the given file) generated in the second case than IO (any IO our command does with the given file) generated in the first case.
Can we reduce this unwanted IO?
In the end, I really don't want the merged file at all. I only create this merged file just to let the command read the merged files content.
And to say, I also don't want this implementation. The file sizes are not so big and it is okay to have that extra negligible IO. But, I am just curious to know if this can be done.
So in order to implement this, I have following understanding/questions:
Generally what all the commands (that takes the filename argument) does is it reads the file.
In our case, the filename(filepath) is not ready, it's just an virtual/imaginary filename that exists (as the mergation of all files).
So, can we create such virtual filename?
What is a filename? It's an indirect inode entry for a storage location.
In our case, the individual files have different inode entries and all inode entries have different storage locations. And our virtual/imaginary file has in fact no inode and even if we could create an imaginary inode, that can only point to a storage in memory (as there is no reference to the storage location of another file from a storage location of one file in disk)
But, let's say using advanced programming, we are able to create an imaginary filepath with imaginary inode, that points to a storage in memory.
Now, when we give that imaginary filename as argument and when the command tries to open that imaginary file, it finds that it's inode entry is referring to a storage in memory. But the actual content is there in disk and not in the memory. So, the data is not loaded into memory yet, unless we read it explicitly. Hence, again we would need to read the data first.
Simply saying, as there is no continuity or references at storage in disk to the next file data, the merged data needs to be loaded to memory first.
So, with my deduction, it seems we would at least need to put the data in memory. However, as the command itself would need the file to be read (if not the whole file, at least a part of it until the commands's operation is done - let it be parsing or whatever). So, using this method, we could save some significant IO, if it's really a big file.
So, how can we create that virtual file?
My first answer is to write the merged file to tmpfs and refer to that file. But is it the only option or can we actually point to a storage location in memory, other than tmpfs? tmpfs is not option because, my script can be run from any server and we need to have a solution that work from all servers. If I mention to create merged file at /dev/shm in my script, it may fail in the server where it doesn't have /dev/shm. So I should be able to load to memory directly. But I think normal user will not have access to memory and so, it seems can not be done without shm.
Please let me know your comments and also kindly correct me if my understanding anywhere is wrong. Even if it is complicated for my level, kindly post your answer. At least, I might understand it after few months.
Create a fifo (named pipe) and provide its name as an argument to your program. The process that combines the five input files writes to this fifo
mkfifo wtf
cat file1 file2 file3 file4 file5 > wtf # this will block...
[from another terminal] cp wtf omg
Here I used cp as your program, and cat as the program combining the five files. You will see that omg will contain the output of your program (here: cp) and that the first terminal will unblock after the program is done.
Your program (here:cp) is not even aware that its 1st argument wtf refers to a fifo; it just opens it and reads from it like it would do with an ordinary file. (this will fail if the program attempts to seek in the file; seek() is not implemented for pipes and fifos)
I am looking for the best optimized way we can use to transfer the large log files from local path to NFS path.
Here the log files will keep on changing dynamically with time.
What i am currently using is a java utility which will read the file from local path and will transfer it to NFS path. But this seems to be consuming high time.
We cant use copy commands, as the log file are getting appended with more new logs. So this will not work.
What i am looking for is .. Is there any way other than using a java utility which will transfer the details of log file from local path to NFS path.
Thanks in Advance !!
If your network speed is higher than log growing speed, you can just cp src dst.
If log grows too fast and you can't push that much data, but you only want to take current snapshot, I see three options:
Like you do now, read whole file into memory, as you do now, and then copy it to destination. With large log files it may result in very large memory footprint. Requires special utility or tmpfs.
Make a local copy of file, then move this copy to destination. Quite obvious. Requires you to have enough free space and increases storage device pressure. If temporary file is in tmpfs, this is exactly the same as first method, but doesn't requires special tools (still needs memory and large enough tmpfs).
Take current file size and copy only that amount of data, ignoring anything that will be appended during copying.
E.g.:
dd if=src.file of=/remote/dst.file bs=1 count=`stat -c '%s' src.file`
stat takes current file size, and then this dd is instructed to copy only that amount of bytes.
Due to low bs, for better performance you may want to combine it with another dd:
dd status=none bs=1 count=`stat -c '%s' src.file` | dd bs=1M of=/remote/dst.file
I am trying to automatically determine the size of a ext2 ramdisk filesystem that a directory will make. What I am currently doing is:
BLOCK_COUNT=`du $RAMDISK_FS_DIR| tail -1 |awk '{print $1}'
dd if=/dev/zero of=ramdisk.img bs=1024 count=$BLOCK_COUNT
mke2fs -F ramdisk.img -L "ramdisk" -b 1024 -m 0
tune2fs ramdisk.img -i 0
But when I mount and cp -r $RAMDISK_FS_DIR into ramdisk.img i get messages like these
cp: cannot create directory `ramdisk/var/www': No space left on device
I determined that in my this specific case increasing BLOCK_COUNT by 48 blocks is exactly how much I need for the operation to succeed. I need a way to find this number for arbitrary directory sizes.
Note that my host filesystem is ext4.
I don't think there's a good way to do this in general. In general, superblocks, inode tables, block bitmaps, and various other structures in the filesystem will vary in size depending on exactly how big the filesystem is and what's in it. Even the space occupied by files (as computed by du) may not be the same on the source filesystem as on the destination filesystem.
I wonder why you are trying to make a filesystem with a size exactly equal to its contents. Since this filesystem will always be full after it is built, you can't add anything to it (unless you delete stuff first), which makes me think it might be intended to be read-only. Buy if it's read-only, why don't you use a filesystem type like cramfs or squashfs that is specifically meant for this kind of application?