mmap() is used to create memory region that is backed by file system. However, I want the reverse: a file that is backed by memory. Is that possible? I have a legacy static library (meaning it's not possible to change it) can only open a local file. I can't change it to use a redirected fd, or a stdin (because pipe does not support seek) I want the file content to be streamed from a Windows share (CIFS/SMB). Is it possible to create a virtual file on local file system with fake size and when the legacy static library access any part of the file(seek or read, no write), we handle it by doing a fetch from CIFS/SMB and return to the legacy library (just like handling a page fault)? Then legacy library would not notice any difference...
Mounting the CIFS/SMB share is not possible due to permission issue.
Assume the environment is POSIX, however, OS specific advice is welcomed as well.
Probably you are looking for shm_open. shm_overview(7) man page is good place to start searching.
Related
I'd like to create shared library in C for linux, some abstract implementation of database management. Shared library whould be responsible for read file containing database and write differences into it. But I have no idea how to handle multiprocessing problems of file handling for this case eg.: App1 try to write differences into database file and App2 has currently opened file with database to read it. In the case of this example I'd like to inform app1 that file is currently open and delay write sequence until App2 will finish database file read.
I was thinking of using some mutual exclusion mechanisms or by using global enum variable to manage current file status, but after I read some of posts I understood that every application that uses shared library create it's own copy in memory and they don't share any memory section during work.
Shared library whould be responsible for read file containing database and write differences into it.
That is possible but quit complicated solution.
While you would need to make sure that multiple processes do not interfere with each other. It's possible to do so with file logs (see man flock), and record locking in man fcntl but you must insure that multiple processes update disjoint file chunks "in place" (without resizing the file).
This is also prone to deadlocks -- if one of the processes takes a lock on a region and then goes into infinite loop, other processes may get stuck as well.
A much simpler solution involves client-server, where the server implements all writes and clients send it read and modify requests.
P.S. There are existing libraries implementing either approach. You will likely save yourself several months of development time using existing solutions.
Are temporary files created with mkstemp synced to disk?
Here is what I have:
Program creates temporary file using mkstemp and sends fd to another program.
This temporary file is mmap-ped by both programs and used heavily (up to 400 MB/sec of writes and 400 MB/sec of reads; up to 60 reads and writes per second).
I can't use memfd_create (may not be supported on target devices).
Lets also assume (and this is almost true) that I can't create this file on tmpfs (like in /tmp).
What I need is guarantee that such file will not stress hard disk. I can't allow it to be written to disk even if this only happens once every 5 seconds. If I can't get such guarantee, I will look for another way.
Additional info (not important):
I am writing wayland compositor for Android devices. Currently temporary files (wayland surfaces actually) are created on tmpfs. And everything works fine as long as SELinux is not enabled. But if I enable SELinux, it prevents fd's from being transferred from client to compositor. Only solution I currently know is to create temporary files in app's home dir. But if such way is dangerous, I will find another.
Are temporary files created with mkstemp synced to disk?
The mkstemp function does not impart any special properties to files it opens that would prevent them from being synced to disk. The filesystem on which they are created might have such a property, but that's independent of file creation. In particular, files created via mkstemp() will persist indefinitely if not removed.
What I need is guarantee that such file will not stress hard disk. I can't allow it to be written to disk even if this only happens once every 5 seconds. If I can't get such guarantee, I will look for another way.
As far as I am aware, even tmpfs filesystems do not guarantee that their contents will remain locked in memory, as opposed to being paged out. They are backed by virtual memory. But if the actual file is comparatively small and all its pages are hot, then they are likely to remain in memory only.
With regard to the larger problem,
everything works fine as long as SELinux is not enabled. But if I
enable SELinux, it prevents fd's from being transferred from client to
compositor. Only solution I currently know is to create temporary
files in app's home dir.
By default, newly-created files inherit the SELinux type of their parent directory. Your Wayland clients presumably do not have sufficient privilege to modify the SELinux labels of the files they create, but you should be able to administratively create a directory wherever you like with a label conducive to your needs. For example, you could cause a subdirectory of /dev/shm to be created for the purpose (at every boot), and chconned to have an appropriate label. If the clients create their temp files there then they should inherit the SELinux type you choose.
I would like to open an anonymous file that is what would be the result under linux of opening a file an unlink it or using memfd_create, but none of these seem to be available under windows (you could make delete a file work, but it's name doesn't seem to be removed until the file is closed). Getting a file descriptor that isn't backed with something visible in the file system.
Is there a way to achieve this under windows? Preferably I'd like it to never appear in the file system.
The reason why I want this is because I need a FILE* to be sent as an argument to a function that expects that (and I don't want it to clobber the file system). Changing the libraries does not look like a feasible option (besides the libraries has to work on other OSes as well - so they can't rely on windows specific abstractions anyway).
The most reasonably close to memfd_create you have in Windows are Memory-Mapped files. MSDN article about it here: http://go.microsoft.com/fwlink/?linkid=180801
But basically, the CreateFileMapping/OpenFileMapping API calls.
This does not use the physical disk (unless it needs the memory paged on disk) for it, but as far as I know, neither does memfd_create
No, there is no such thing as an anonymous file in Windows.
(Of course, that does not necessarily mean that you cannot have a FILE * that does what you need; for example, I like Ross's suggestion of using a named pipe.)
I want to read a file from a driver for 3rd party application
(simple C absed DLL runnning in user space but under control of the 3rd party application)
This file will be written to by an separate C# application.
What shall I use so that I do not face any problems?
What is advantage of using _sopen_s over fopen, I understand the former is more secure but what is the feature of 'sharing' it supports?
I did Google it out number of tims but could not find it.
_sopen_s is a secure version of open() with sharing. It uses unbuffered I/O. It works with file handles (int). This is Microsoft specific. open() is cross platform. There's also sopen() which is the shared/access version.
fopen uses buffering and no file sharing. Works with FILE* structures.
File sharing means that you allow other processes to access the file (or not). E.g. when read sharing is denied, another process will not be open the file for reading.
All are legitimate to use. The unbuffered I/O versions work faster if you read the file in large chunks.
Suppose I have an open file. How can I detect when the file is changed by another program in the background. Some text editors can detect and update the open file if it is changed by another process.
I'm specifically asking for this with C under Linux(this seems to be OS dependent).
If you don't want to poll the file using stat, and don't mind being Linux-specific, then you can use the inotify API. Your kernel needs to be 2.6.13 or newer and glibc 2.4 or newer (which they will be if you're targeting anything from the past 2 or 3 years). The API basically gives you a file descriptor that you can poll or select, and read to get information about modified files. If your application is interactive, like an editor, then it will typically have some sort of event loop that calls select or poll, and can watch your inotify file descriptor for events.
Using inotify is generally preferable stat, because you get notifications immediately and you don't waste time and disk I/O polling when the file isn't changing. The downside is that might not work over NFS or other networked file systems, and it's not portable.
This page at IBM Developerworks gives some example C code, and the man page is the definitive reference.
use stat function. Example in the page.
Text editors I've seen on Windows and Linux have done it the same way: they don't check to see whether the file has actually changed, they just looking at the file's stat mtime.