I am using open_memstream in a library of mine, but I would like to port this library to MSVC. It seems there are no equivalent function available, but is there something similar enough?
What open_memstream does is it takes a char** destination and size and returns a FILE* which you many write to, the data is stored in a dynamically allocated buffer (accessible from the char** argument). When closing the FILE the char** contains the data that was written to the stream. This makes an easy way to construct large and complex string streams.
While it is possible to both read and seek from the memstream I only write to it.
Is there a way to open a similar memory FILE stream in MSVC? Also, this is pure C, no C++.
A similar function on Windows would be CreateStreamOnHGlobal(). That however works with the IStream COM interface, it isn't a drop-in replacement for FILE. You might want to take a peek at the Cygwin source code to see what they did.
https://github.com/Snaipe/fmem is a wrapper for different platform/version specific equivalents of open_memstream
It tries in sequence the following implementations:
open_memstream.
fopencookie, with growing dynamic buffer.
funopen, with growing dynamic buffer.
WinAPI temporary memory-backed file.
When no other mean is available, fmem falls back to tmpfile()
Related
I'm wondering if the fopen command is smart enough to stop reading a file if it's to large and then wait for some proceeding read command to continue reading.
For that matter, how large is _iobuf?
fopen(...) doesn't do any size checks; it just returns a file pointer. Are you thinking of fread(...), by any chance?
You can always find the size of the file that you are going to read by using stat(...) system call.
Per the C standard, setbuf()/setvbuf() shall be called after fopen() and before anything else. These set the buffering mode and buffer size for the freshly opened file. This implies that at least at the C level fopen() reads nothing from the file it has opened.
The underlying file system implementation in the OS, however, may read ahead into a file cache, but this is clearly not defined in the language standard. You need to find this out in the OS documentation or by experimentation.
_iobuf is not defined in the C standard. While I may "guess" what it is, it's unlikely something you need to concern yourself with (it would not contain a fixed-size C file buffer anyway and rather contain a pair of values: a pointer to a buffer and the buffer size).
If you look at here, you'll see that #define BUFSIZ 1024
But it depends on your fopen lib implementation. AFAIK, there is no standard definition.
I need a cross platform way of treating memory buffer as FILE*. I have seen other questions which point out that there is no portable way to do this (fmemopen in linux is what I need but it fails on Windows platform).
I have tried using the setvbuf and it seems to work. Can anyone please point out the exact problem of using setvbuf function?
Also , I have seen the C standard draft WG14/N1256 and 7.19.5.6 says:
the contents of array at any time are indeterminate.
I don't understand if I use my own buffer how can its contents be indeterminate?
EDIT: Thanks for all the answers. Not using this method anymore.
No really, there's no portable way to do this.
Using setvbuf may appear to work but you're really invoking undefined behavior, and it will fail in unexpected ways at unexpected times. The GNU C library does have fmemopen(3) as an extension, as you mentioned, but it's not portable to non-GNU systems.
If you're using some library that requires a FILE* pointer and you only have the required data in memory, you'll just have to write it out to a temporary file and pass in a handle to that file. Ideally, your library should provide an alternative function that takes a memory pointer instead of a file pointer, but if not, you're out of luck (and you should complain to the library writer about that deficiency).
Function setvbuf() is used to tell the FILE the memory to be used as buffer, but it does not specify how this memory will be used: that's up to the implementation.
Thus, the contents of the buffer are indeterminate at any time, and if it happens to work for you, it is just by chance.
It depends on what you want to do with the buffer/FILE*. You can certainly perform simple operations and get away with them, but you cannot guarantee that all of the FILE* operations will perform as expected on your memory buffer.
Sorry, there is simply no cross-platform one-liner to get full FILE* characteristics, I've tried myself many times haha
what you can try:
#define-wrapped OS-specific logic
Look further into the interface you are trying to interact with. At some point it just plays with a buffer anyway. Then splice in your buffer. This is what I did.
Your technique + faith.
I am working with some legacy code which uses something like this:
void store_data(FILE *file);
However, I don't want to store data on the disk, I want to store it in memory (char *buf). I could edit all of the code, but the code jumps all over the place and fwrite gets called on the file all over the place. So is there an easier way, for example that I can map a FILE* object to an (auto-growing) buffer? I do not know the total size of the data before-hand.
The solution has to be portable.
There is no way to do this using only the facilities provided in the C standard. The closest you can come is
FILE *scratch = tmpfile();
...
store_data(scratch);
...
/* after you're completely done calling the legacy code */
rewind(scratch);
buf = read_into_memory_buffer(scratch);
fclose(scratch);
This does hit the disk, at least potentially, but I'd say it's your best bet if you need wide portability and can't modify the "legacy code".
In POSIX.1-2008, there is open_memstream, which does exactly what you want; however, this revision of POSIX is not yet widely adopted. GNU libc (used on Linux and a few others) has it, but it's not available on OSX or the *BSDs as far as I know, and certainly not on Windows.
You might want to look at fmemopen and open_memstream. They do something in the direction of what you want.
From the man page:
The open_memstream() function opens a stream for writing to a buffer.
The buffer is dynamically allocated (as with malloc(3)), and automati‐
cally grows as required. After closing the stream, the caller should
free(3) this buffer.
I don't know if it's a good idea, but it's an idea.
You can "redefine" fwrite using a macro.
#define fwrite(a, b, c) your_memory_write_function(a, b, c)
Then implement memory_write_function to write data to your auto growing buffer instead of a file.
You will need to call store_data with a pointer to something else though (not a pointer to FILE). But that's possible with C so you will have no issues there.
On what platform are you running? Can't you use tmpfs? If you open a file on tmpfs, is it not, from the point of view of the kernel, the same as a regular file, but written to memory?
You may want to look into fmemopen(). If that's not available to you, then you could possibly use a named shared memory segment along with fdopen() to convert the file descriptor returned by shm_open() to a FILE*.
In C, when opening a file with
FILE *fin;
fin=fopen("file.bin","rb");
I only have a pointer to a structure of FILE. Where is the actual FILE struct allocated on Windows machine? And does it contain all the necessary information for accessing the file?
My aim is to dump the whole data segment to disk and then to reload the dumped file back to the beginning of the data segment. The code that reloads the dumped file is placed in a separate function. This way, the fin pointer is local and is on the stack, thus is not being overwritten on reload. But the FILE struct itself is not local. I take care not to overwrite the memory region of size sizeof(FILE) that starts at the address fin.
The
fread(DataSegStart,1,szTillFin,fin);
fread(dummy,1,sizeof(FILE),fin);
fread(DataSegAfterFin,1,szFinTillEnd,fin);
operations completes successfully, but I get an assertion failure on
fclose(fin)
Do I overwrite some other necessary file data other than in the FILE struct?
The actual instance of the FILE structure exists within the standard library. Typically the standard library allocates some number of FILE structures, which may or may not be a fixed number of them. When you call fopen(), it returns a pointer to one of those structures.
The data within the FILE structure likely contains pointers to other things such as buffers. You're unlikely to be able to save and restore those structures to disk without some really deep integration with your standard library implementation.
You may be interested in something like CryoPID which does process save and restore at a different level.
It seems like you're trying to do something dangerous, unlikely to work.
fopen allocates a FILE structure and initializes it. fclose releases it. How it allocates it and what it puts in it is implementation dependent. It could contain a pointer to another piece of memory, which is also allocated somewhere (since it's buffered I/O, I guess it does allocate a buffer somewhere).
Writing code that relies on the internals of fopen is dangerous, most likely won't work, and surely won't be stable and portable.
Well, you have a pointer to a FILE object, so technically you know where it is but you should be aware that FILE is deliberately an opaque type. You shouldn't need to know what it contains, you just need to know that you can pass it to functions that know about it to perform certain actions. Additionally, FILE may not be a complete type so sizeof(FILE) might not be correct and, additionally, the object might contain pointers to other structures. Simply avoiding overwriting the FILE object is not likely to be sufficient for you to avoid corrupting the program by writing over most of its memory.
FILE is defined in stdio.h. It contains all the information about the file but, looking at the code you show, I think you don't understand its purpose. It is created and run through the operating system with the C library which fills FILE with information about the file but it is not contained in the file itself.
I am facing a problem where the C library exposes only an interface that writes the data to a C FILE structure. This means all the data get saved to disk. I want this library to save the data to memory instead.
Is there a way to emulate FILE structure in my code and make it save the data in memory instead? I imagine FILE has various pointers to functions and I could write my own functions and make them allocate memory and on fwrite write the data to memory.
Can this be done?
Thanks, Boda Cydo.
A portable solution would be to use fmemopen or open_memstream if these functions are available (they've been part of GNU libc for a long time and they were added to POSIX in 2008) and simply write to a temporary file then read back into memory if they're not.
You can have a look at fmemopen.
On a UNIX system, you can use fdopen(3) to get a FILE* that uses a stream you previously opened. Then all you have to do is write some code to read from the other side of that pipe and shovel it into memory.