Is there a way in Linux, that's reasonably portable (mainly to OSX/BSDs) to wait for a process to open a file? I know I can use inotify for this, but OSX doesn't support it. One caveat is that I won't actually know the PID of the process, so it's exec()-ed via /bin/sh like popen, so just monitoring it's /proc//fd directory won't work either.
There are several available libraries or command-line tools suitable for this purpose with backends that span Linux and MacOS.
Consider fswatch, or one of its competitors (if we got down to debating individual libraries, this would be in explicitly-off-topic territory).
Related
I want to write software that will detect all used/created/modified/deleted files during the execution of a process (and its child processes). The process has not yet run - the user provides a command line which will later be subprocessed via bash, so we can do things before and after execution, and control the environment the command is run in.
I have thought of four methods so far that could be useful:
Parse the command line to identify files and directories mentioned. Assume all files explicitly mentioned are used. Check directories before/after for created/deleted files. MD5 existing files before/after to see any are modified. This works on all operating systems and environments, but obviously has serious limitations (doesnt work when command is "./script.sh")
Run the process via another process like strace (dtruss for OSX, and there are equivalent windows programs), which listens for system calls. Parse output file to find files used/modified/deleted/created. Pros that its more sensitive than MD5 method and can deal with script.sh. Cons that its very OS specific (dtruss requires root privileges even if the process being run does not - outputs of tools all different). Could also create huge log files if there are a lot of read/write operations, and will certainly slow things down.
Integrate something similar to the above into the kernel. Obviously still OS specific, but at least now we are calling the shots, creating common output format for all OS's. Wouldn't create huge log files, and could even stop hooking syscalls to, say, read() after process has requested the first read() to the file. I think this is what the tool inotify is doing, but im not familiar with it at all, nor kernel programming!
Run the process using the LD_PRELOAD trick (called DYLD_INSERT_LIBRARIES on OSX, not sure if it exists in Windows) which basically overwrites any call to open() by the process with our own version of open() which logs what we're opening. Same for write, read, etc. It's very simple to do, and very performant since you're essentially teaching the process to log itself. The downside is that it only works for dynamically-linked process, and i have no idea of the prevalence of dynamic/statically linked programs. I dont even know if it is possible before execution to tell if a process is dynamically or statically linked (with the intention of using this method by default, but falling back to a less-performant method if its not possible).
I need help choosing the optimal path to go down. I have already implemented the first method because it was simple and gave me a way to work on the logging backend (http://ac.gt/log) but really i need to upgrade to one of the other methods. Your advice would be invaluable :)
Take a look to the source code of "strace" (and its -f to trace children). It does basically what you are trying to do. It captures all the system calls of the process (or its childs) so you can grep for operations like "open", etc.
The following link provides some examples of implementing your own strace by using the ptrace system call:
https://blog.nelhage.com/2010/08/write-yourself-an-strace-in-70-lines-of-code/
My goal is to determine when executing a command, precisely which files it reads and writes. On Linux I can do this using ptrace (with work, akin to what strace does) and on FreeBSD and MacOS I can do this with the ktrace system command. What would you use to obtain this information on Windows?
My research so far suggests that I either use the debugger interface (similar to ptrace in many ways) or perhaps ETW. A third alternative is to interpose a DLL to intercept system calls as they are made. Unfortunately, I don't have the experience to guess as to how challenging each of these approaches will be.
Any suggestions?
Unfortunately it seems there is no easy way to intercept file level operations on Windows.
Here are some hints:
you could try to use FileMon from Sysinternals if it is enough for your needs, or try to look at the source of the tool
you could make use of commercial software like Detours - beware, I never used that myself and I'm not sure it really meets your needs
If you want a better understanding and are not frightened at doing it by hand, the Windows way of intercepting file I/O is using a File System Filter Driver. In fact, there is a FilterManager embedded in Windows system that can forward all file system calls to minifilters.
To build it, the interface with the system is provided by the FilterManager, and you have just (...) to code and install the minifilter that does the actual filtering - beware again never tested that ...
As you suggested, this is a fairly simple task to solve with API hooking with DLL injection.
This is a pretty good article about the application: API hooking revealed
I believe you can find more recent articles about the issue.
However, you probably need to use C++ to implement such a utility. By the way, programs can disable DLL injection. For example, I weren't able to use this approach on the trial version of Photoshop.
So, you may want to check if you can inject DLL files in the process you want with an existing solution before you start writing your own.
Please, take a look to the article CDirectoryChangeWatcher - ReadDirectoryChangesW all wrapped up.
It is a very old, but running, way to watch directory changes.
Microsoft owns a bunch of tools called Sysinternals. There is a program called Process Monitor that will show you all the file accesses for a particular process. This is very likely what you want.
Check this particular Stack Overflow question out for your question... This might help you:
Is there something like the Linux ptrace syscall in Windows?
Also, if you are running lower versions like Windows XP then you should check out Process Monitor.
Also, I would like you to check this out...
Monitoring certain system calls done by a process in Windows
So if you want to parse command line options when starting the program you use getopt(). But how are you doing this if the program is already running in the background? I couldn't find info. let's say for example that you have a server running, but you want to change something in the way it works. How to do it? I want to do this in Linux.
There is no platform-independent way of doing this; the C programming language doesn't specify (or require the existance of) a mechanism to talk to a running program.
You're going to have to look for either platform-specific code, or some existing library which abstracts the platforms into something portable of its own.
In Linux, a Unix domain socket is one way of implementing this. Another is shared memory.
If you're on Un*x you have many options.
A FIFO pipe looks reasonable and easy to implement :)
There are a couple of ways you can do this, but they all have a common theme - Interprocess communication.
My preferred way to do this is via some sort of sockets (typically these days I use ZMQ for these purposes, but if you're starting out, read up on sockets in general before you get caught up using ZMQ). Depending whether you're on Windows or some sort of Unix will dictate what sort of sockets you have available to you.
There are other ways to do this also - such as shared memory, but sockets would be your best bet especially since you mentioned "server". I suggest you study the "client server model".
A simplest solution with server I have used - is to make file and ask server to read file once a 10 second. Put command there. That is cross platform).
Second more or less cross platform solution is to use some standard library for concurrency (pthread, for example), or to use new standard of C++ (thread and mutex libs). Make thread that will wait for some commands, while other will execute something.
You could use a configuration file and have the program listen to changes to that file.
If you are programming in Linux you can use inotify (#include <linux/inotify.h>).
In MacOS/iOS use FSEvents.
In Windows use FindFirstChangeNotification.
Is there a way to write a C code that allow us to determine if a previous instance of an application is already running? I need to check this in a portable way for Linux and Windows, both using the last version of GCC avaiable.
Any examples of portable code would be of enormous help. I see two options now:
Check process list. Here linux has good tools, but I don't think the same functions apply to windows. Maybe some gnu libraries for both SO? What libraries, or functions?
Save and lock a file. Now, how to do that in a way that both systems can understand? One problem is where to save the file? Path trees are different from each systems. Also, if a relative path is chosen, two applications can still run with different locked files in different directories.
Thanks!
Beco.
PS. The SO have different requisites, so if you know one and not another, please answer. After all, if there is no portable "single" way, I still may be able to use #ifdef and the codes proposed as answer.
C language (not c++), console application, gcc, linux and windows
Unfortunately, if you limit yourself to C, you may have difficulty doing this portably. With C++, there's boost interprocess's named_mutex, but on C, you will have to either:
UNIXes (including Mac OS): Open and flock a file somewhere. Traditionally you will also write your current PID into this file. NOTE: This may not be safe on NFS; but your options are extremely limited there anyway. On Linux you can use a /dev/shm path if you want to ensure it's local and safe to lock.
Windows: Open and lock a named mutex
for windows, a mutex works well.
http://msdn.microsoft.com/en-us/library/ms682411(v=vs.85).aspx
the article also mentions an alternative to a mutex....
To limit your application to one instance per user, create a locked file in the user's profile directory.
The sort of canonical method in Unixland is to have the process write its own PID to a file in a known location. If this file exists, then the program can check its own pid (available by system call) with the one in that file, and if it's unfamiliar you know that another process is running.
C does not give in-built facilities to check if an application is already running, so, making it cross platform is difficult/impossible. However, on Linux, one can use IPC. And, on Windows (I'm not very experienced in this category), you may find this helpful.
I have written a small custom web server application in C running on Linux. When the application receives a request it calls fork() and handles the request in a separate process, which is chrooted into a specific directory containing the files I want to make available.
I want to port the application to Windows, but neither fork() nor chroot() are available on this platform, and there don't seem to be any direct equivalents. Can you point me to a simple (and preferably well written) example of code that will provide this functionality in Windows? My C isn't all that good, so the simpler the better.
First of all, the Windows equivalent of chroot is RUNAS which is documented here. If you need to do this from a program, then studying this C++ source code should help you understand how to use the Windows API. It is not precisely the same as chroot() but Windows folk use it to create something like a chroot jail by creating a user with extremely limited permissions and only giving that user read permission on the application folder, and write permission on one folder for data.
You probably don't want to exactly emulate fork() on Windows because it doesn't sound like you need to go that far. To understand the Windows API for creating processes and how it differs from fork(), check Mr. Peabody Explains fork(). The actual current source code for Cygwin's fork implementation shows you the current state of the art.
The Microsoft documentation for CreateProcess() and CreateThread() are the place to look for more info on the differences between them.
And finally, if you don't want to learn all the nitty-gritty platform details, just write portable programs that work on Windows and Unix, why not just use the Apache Portable Runtime library itself. Here are some docs on process creation with some sample code, in C, to create a new process.
There's no such thing as fork() on Windows. You need to call CreateProcess() - this will start a separate process (mostly equivalent to calling fork() and then immediately exec() for the spawned process) and pass the parameters to it somehow. Since you seem to have all the data to process in a dedicated directory you can make use of lpCurrentDirectory parameter of CreateProcess() - just pass the directory path you previously used with chroot() there.
The absolutely simplest way of doing it is using Cygwin, the free Unix emulation layer for Windows. Download it and install a complete development environment. (Choose in the installer.) If you are lucky, you will be able to compile your program as is, no changes at all.
Of course there are downsides and some might consider this "cheating" but you asked for the simplest solution.
Without using a compatibility framework (Interix, Cygwin, ...) you're looking at using the Windows paradigm for this sort of thing.
fork/vfork is a cheap operation on UNIXes, which is why it's used often compared to multi-threading. the Windows equivalent - CreateProcess() - is by comparison an expensive operation, and for this reason you should look at using threads instead, creating them with CreateThread(). There's a lot of example code out there for CreateThread().
In terms of chroot(), Windows doesn't have this concept. There's libraries out there that claim to emulate what you need. However it depends why you want to chroot in the first place.
Reading comments, if it's simply to stop people going up the tree with ../../../../(etc), chroot would do the job, but it's no substitue for parsing input in the first place and making sure it's sane: i.e., if too many parents are specified, lock the user into a known root directory. Apache almost certainly does this as I've never had to create a chroot() environment for Apache to work...
Using fork/chroot is simply not how things are done on Windows. If you are concerned about security in subprocesses, maybe some form of virtualization or sandboxing is what you want to use. Passing complex information to the subprocess can be done by some form of RPC-solution.
It sounds to me as if you have designed your application in the Unix way, and now you want to run in on Windows without having to change anything. In that case, you may want to consider using Cygwin, but I'm not sure if/how Cygwin emulates chroot.
Consider SUA ( aka Windows Services for Unix ). It has nearly everything you need to port applications.
man chroot(interix)