I have a rather odd program where I need to load a file into memory, close that file handle, and then use the file image in memory like a file (where i use ReadFile, WriteFile with a HANDLE)... so basically I'm looking at doing the inverse of CreateMapFile... is this possible within the Windows API?
Thanks!
When you call CreateFile, you can use FILE_ATTRIBUTE_TEMPORARY. This attempts to hold the data for the file in RAM if possible, but it does not guarantee it -- the data could be written out to disk if memory gets low enough.
Related
I have written my own cp function which copies only half of the mp4 file from one partition to my own additionally created partition. This additionally created partition is half of the size of mp4 file and i am playing mp4 file from this partition.
After this half mp4 file is played, i want to remove the blocks of this part of a file so that I can copy the rest of the file to this partition and play it.
I am using low level i/o functions open,read,write,close in C file handling to write cp function.
This functionality is going to be dependant on the OS you are using. I am not aware of a universal way to do this. On windows, you can use the function _chsize() in unix based systems you want ftruncate(). you can use each of these functions to set the size of the file to 0, effectively getting rid of your allocated blocks.
I would use mmap(2) for that purpose and get rid of the scratch partition. On most systems, this is way faster since disk I/O is handled in the fashion that is just like swap file, which is very highly optimized and accerlerated.
When start playing, map the first portion of the file to a memory address and start working from that. Then when the end of the current part is coming map the next part in.
I am using an embedded system running linux.I use a ramdisk filesystem on the embedded target.
My application captures real-time data and does Standard C "fwrite" to a file in this ramdisk fs.As there is limited amount of memory , I would like to set a max size for the file and cause fwrite to wrap around like a circular buffer. Is there a way to do this in manner that is transparent to the application ?
I would prefer the application to remain unchanged when I migrate to a filesystem on a storage device (eSATA) having much larger capacity.
There's no built in method of achieving this.
The best option is probably to write a small wrapper function that takes care of the file write while maintaining a count of the number of bytes written.
Once it has reached the maximum size that you set it should call rewind() (or fseek() etc.) to go back to the start of the file.
It might be easier to use mmap() to memory map your file and then treat it like a circular buffer. But again you would need to implement the wrapping yourself.
There's an AudioFileOpenURL function which opens an file. With AudioFileReadPackets that file is accessed to read packets. But one thing that stucks in my brain is: Does AudioFileOpenURL actually load the whole monster into memory? Or is that a lightweight operation?
So is it possible to read data from a file, only a specific portion, without having the whole terabytes of stuff in memory?
Does AudioFileOpenURL actually load the whole monster into memory?
No, it just gets you a file pointer.
Or is that a lightweight operation?
Yep, fairly lightweight. Just requires a filesystem lookup.
So is it possible to read data from a file, only a specific portion, without having the whole terabytes of stuff in memory?
Yes, you can use fseek to go a certain point in the file, then fread to read it into a buffer (or AudioFileReadBytes).
No, it doesn't load the entire file into memory. "Opens a file" returns a handle to you allowing you to read from or write to a file.
I don't know about objective-c, but with most languages you open the file, and that just gives you the ability to THEN access the contents with a READ operation. In your case, you can perform a SEEK to move the file pointer to the desired location, then read the number of bytes you need.
AudioFileOpenURL will open(2) the file and read the necessary info (4096 bytes) to determine the audio type.
open(2) won't load the whole file into RAM.
(AudioFileOpenURL is a C API, not Objective-C.)
I want to copy a large a ram-based file (located at /dev/shm direcotry) to local disk, is there some way for an efficient copy instead of read char one by one or create another piece memory? I can use only C language here. Is there anyway that I can put the memory file directly to disk? Thanks!
I would mmap() the files and do memcpy() between them.
Thanks you guys for the help! I made it by mmap the ram-based file and write the entire block directly to the destination. memcopy was not used because I am actually writing to a parallel file system (pvfs), which does not support mmap operation.
/dev/shm is shared memory, so one way to copy it would be to open it as shared memory, but frankly I don't think you will gain anything.
when writing your memory file to disk, the bottleneck will be the disk.
just be sure to write data in big chunks, and you should be fine.
You can just copy it like any other file:
cp /dev/shm/tmp ~/tmp
So, a quick, simple way is to issue a cp command via system().
You could try to see if the splice system call works for this. I'm not sure if it will since it has some restrictions about the types of files that it can work with, but if it did work you would call it repeatedly with memory page sized (or some multiple page size) requests repeatedly until it finished, and the kernel would handle it very efficently.
If this doesn't work you'll need to do either mmap or do plain old read/write.
Reading and Writing in memory page sized chunks makes things much more efficient. It can be even more efficient if your buffers are memory page size aligned since it opens up the oppurtunity for the kernel to just move the data to/from your process's memory via memory managment trickery rather than actually copying the data around.
The only thing you can do is read() in page size aligned chunks. I'm assuming you need to guarantee the data as written, which is going to mean bypassing buffers via posix_fadvise() or using O_DIRECT (I typically use posix_fadvise(), but O_DIRECT is appropriate here).
In this case, the speed of the media being written to alone dictates how quickly this will happen.
If you don't need to bypass buffers, the operation will complete faster, but there's no guarantee that the data will actually be written in the event of a reboot / power outage / etc. Since the source of the data is in shared memory, I'm (again) guessing you want the write to be guaranteed.
The only thing you can optimize is how long it takes read() to get data from shared memory into your own address space, which page size aligned chunks will improve.
I want are the steps that an application takes inorder to open the file and allow user to read. File is nothing more than sequence of bits on the disk. What steps does it take to show show the contents of the file?
I want to programatically do this in C. I don't want to begin with complex formats like word/pdf but something simpler. So, which format is best?
If you want to investigate this, start with plain ASCII text. It's just one byte per character, very straightforward, and you can open it in Notepad or any one of its much more capable replacements.
As for what actually happens when a program reads a file... basically it involves making a system call to open the file, which gives you a file handle (just a number that the operating system maps to a record in the filesystem). You then make a system call to read some data from the file, and the OS fetches it from the disk and copies it into some region of RAM that you specify (that would be a character/byte array in your program). Repeat reading as necessary. And when you're done, you issue yet another system call to close the file, which simply tells the OS that you're done with it. So the sequence, in C-like pseudocode, is
int f = fopen(...);
while (...) {
byte foo[BLOCK_SIZE];
fread(f, foo, BLOCK_SIZE);
do something with foo
}
fclose(f);
If you're interested in what the OS actually does behind the scenes to get data from the disk to RAM, well... that's a whole other can of worms ;-)
Start with plain text files