I know I can use a system call like execl("/bin/sh", "-c", some_string, 0) to interpret a "snippet" of shell code using a particular shell/interpreter. But in my case I have an arbitrary string in memory that represents some complete script which needs to be run. That is, the contents of this string/memory buffer could be:
#! /bin/bash
echo "Hello"
Or they might be:
#! /usr/bin/env python
print "Hello from Python"
I suppose in theory the string/buffer could even include a valid binary executable, though that's not a particular priority.
My question is: is there any way to have the system launch a subprocess directly from a buffer of memory I give it, without writing it to a temporary file? Or at least, a way to give the string to a shell and have it route it to the proper interpreter?
It seems that all the system calls I've found expect a path to an existing executable, rather than something low level which takes an executable itself. I do not want to parse the shebang or anything myself.
You haven't specified the operating system, but since #! is specific to Unix, I assume that's what you're talking about.
As far as I know, there's no system call that will load a program from a block of memory rather than a file. The lowest-level system call for loading a program is the execve() function, and it requires a pathname of the file to load from.
My question is: is there any way to have the system launch a
subprocess directly from a buffer of memory I give it, without writing
it to a temporary file? Or at least, a way to give the string to a
shell and have it route it to the proper interpreter?
It seems that all the system calls I've found expect a path to an
existing executable, rather than something low level which takes an
executable itself. I do not want to parse the shebang or anything
myself.
Simple answer: no.
Detailed answer:
execl and shebang convention are POSIXisms, so this answer will focus on POSIX systems. Whether the program you want to execute is a script utilizing the shebang convention or a binary executable, the exec-family functions are the way for a userspace program to cause a different program to run. Other interfaces such as system() and popen() are implemented on top of these.
The exec-family functions all expect to load a process image from a file. Moreover, on success they replace the contents of the process in which they are called, including all memory assigned to it, with the new image.
More generally, substantially all modern operating systems enforce process isolation, and one of the central pillars of process isolation is that no process can access another's memory.
Related
Let's say I have included a binary into my program during compilation so, I keep it in a variable something like
var myExec =[]byte{'s','o','m','e',' ','b','y','t','e','s'}
So my question is whether there is a way to execute this binary within my program without writing it back to the disc and calling exec or fork on it?
I am writing my app in Golang so the method I am seeking for is to do it using Go or C (using CGO).
Basically, I am seeking something like piping the bash script into bash just I don't know where can I pipe the bytes of a native executable to run it and writing it back to disk and then letting os to read it again seems a lot of extra work to be done
In C and assuming Linux, you can change the protection of a memory region by means of the mprotect() system call, so that it can be executed (i.e.: turn a data region into a code region). After that, you could execute that region of memory by jumping into it.
I want to use the above command in a c program in linux.
I have searched so far that there are system calls and exec calls that one may make in a code. Is there any other way using exec or system commands?
Using the system command isn't an ideal command for a multi-threaded server ,what do you suggest?
First make sure you have lp installed in this path. (Using which lp in the terminal).
You may want to understand the lp command. It's a classic unix command to send data to the "line printer", but it works with e.g. .pdf files too nowadays, depending on your printer system. However, it isn't necessarily installed. Sometimes, lpr may work better, too.
See also: http://en.wikipedia.org/wiki/Lp_%28Unix%29
The second part is about executing unix commands. system is the easiest (also the easiest to introduce a security issue into your program!), using fork and execve is one of a number of alternatives (have a look at man execve).
Yes, this code is ok. It will print the file named filename provided that the lp is found at /usr/bin and the filename file exists. You can add checks for that if you want your program to report if something went wrong, other than that it will do exactly what you expect.
Doing system("lp filename"); would work if you don't mind your program blocking after that system() call and until lp finishes.
You could also use popen(3) (instead of system(3)). But you always need to fork a process (both system and popen are calling fork(2)). BTW, if you have a CUPS server you might use some HTTP client protocol library like libcurl but that is probably inconvenient. Better popen or system an lp (or lpr) command.
BTW, printing is a relatively slow and complex operation, so the overhead of forking a process is negligible (I believe you could do that in a server; after all people usually don't print millions of pages). Some libraries might give you some API (e.g. QPrinter in Qt).
Notice that the lp (or lpr) command is not actually doing the printing, it is simply interacting with some print daemon (cupsd, lpd ...) and its spooling system. See e.g. CUPS. So running the lp or lpr command is reasonably fast (much faster than the printing itself), generally a few milliseconds (certainly compatible with a multi-threaded or server application).
Quite often, the command passed to popen or system is constructed (e.g. with snprintf(3) etc...), e.g.
char cmdbuf[128];
snprintf (cmdbuf, sizeof(cmdbuf), "lp %s", filename);
but beware of code injection (think about filename containing foo; rm -rf $HOME) and of buffer overflow
Of course, notice that library functions like system, popen, fopen are generally built above existing syscalls(2). Read Advanced Linux Programming
I am currently trying to check wether the copy of a file from a directory to another is done.
I would like to know if the target file is still being copied.
So I would like to get the number of file descriptors openned on this file.
I use C langage and don't really find a way to resolve that problem.
If you have control of it, I would recommend using the copy-move idiom on the program doing the copying:
cp file1 otherdir/.file1.tmp
mv otherdir/.file1.tmp otherdir/file1
The mv just changes some filesystem entries and is atomic and very fast compared to the copy.
If you're able to open the file for writing, there's a good chance that the OS has finished the copy and has released its lock on it. Different operating systems may behave differently for this, however.
Another approach is to open both the source and destination files for reading and compare their sizes. If they're of identical size, the copy has very likely finished. You can use fseek() and ftell() to determine the size of a file in C:
fseek(fp, 0L, SEEK_END);
sz = ftell(fp);
In linux, try the lsof command, which lists all of the open files on your system.
edit 1: The only C language feature that comes to mind is the fstat function. You might be able to use that with the struct's st_mtime (last modification time) field - once that value stops changing (for, say, a period of 10 seconds), then you could assume that file copy operation has stopped.
edit 2: also, on linux, you could traverse /proc/[pid]/fd to see which files are open. The files in there are symlinks, but C's readlink() function could tell you its path, so you could see whether it is still open. Using getpid(), you would know the process ID of your program (if you are doing a file copy from within your program) to know where to look in /proc.
I think your basic mistake is trying to synchronize a C program with a shell tool/external program that's not intended for synchronization. If you have some degree of control over the program/script doing the copying, you should modify it to perform advisory locking of some sort (preferably fcntl-based) on the target file. Then your other program can simply block on acquiring the lock.
If you don't have any control over the program performing the copy, the only solutions depend on non-portable hacks like lsof or Linux inotify API.
(This answer makes the big, big assumption that this will be running on Linux.)
The C source code of lsof, a tool that tells which programs currently have an open file descriptor to a specific file, is freely available. However, just to warn you, I couldn't make any sense out of it. There are references to reading kernel memory, so to me it's either voodoo or black magic.
That said, nothing prevents you from running lsof through your own program. Running third-party programs from your own program is normally something you try to avoid for several reasons, like security (if a rogue user changes lsof for a malicious program, it will run with your program's privileges, with potentially catastrophic consequences) but inspecting the lsof source code, I came to the conclusion that there's no public API to determine which program has which file open. If you're not afraid of people changing programs in /usr/sbin, you might consider this.
int isOpen(const char* file)
{
char* command;
// BE AWARE THAT THIS WILL NOT WORK IF THE FILE NAME CONTAINS A DOUBLE QUOTE
// OR IF IT CAN SOMEHOW BE ALTERED THROUGH SHELL EXPANSION
// you should either try to fix it yourself, or use a function of the `exec`
// family that won't trigger shell expansion.
// It would be an EXTREMELY BAD idea to call `lsof` without an absolute path
// since it could result in another program being run. If this is not where
// `lsof` resides on your system, change it to the appropriate absolute path.
asprintf(&command, "/usr/sbin/lsof \"%s\"", file);
int result = system(command);
free(command);
return result;
}
If you also need to know which program has your file open (presumably cp?), you can use popen to read the output of lsof in a similar fashion. popen descriptors behave like fopen descriptors, so all you need to do is fread them and see if you can find your program's name. On my machine, lsof output looks like this:
$ lsof document.pdf
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
SomeApp 873 felix txt REG 14,3 303260 5165763 document.pdf
As poundifdef mentioned, the fstat() function can give you the current modification time. But fstat also gives you the size of the file.
Back in the dim dark ages of C when I was monitoring files being copied by various programs I had no control over I always:
Waited until the target file size was >= the source size, and
Waited until the target modification time was at least N seconds older than the current time. N being a number such a 5, and set larger if experience showed that was necessary. Yes 5 seconds seems extreme, but it is safe.
If you don't know what the target file is then the only real choice you have is #2, but user a larger N to allow for the worse case network and local CPU delays, with a healthy safety factor.
using boost libs will solve the issue
boost::filesystem::fstream fileStream(filePath, std::ios_base::in | std::ios_base::binary);
if(fileStream.is_open())
//not getting copied
else
//Wait, the file is getting copied
I'm designing a program I plan to implement in C and I have a question about the best way (in terms of performance) to call external programs. The user is going to provide my program with a filename, and then my program is going to run another program with that file as input. My program is then going to process the output of the other program.
My typical approach would be to redirect the other program's output to a file and then have my program read that file when it's done. However, I understand I/O operations are quite expensive and I would like to make this program as efficient as possible.
I did a little bit of looking and I found the popen command for running system commands and grabbing the output. How does the performance of this approach compare to the performance of the approach I just described? Does popen simply write the external program's output to a temporary file, or does it keep the program output in memory?
Alternatively, is there another way to do this that will give better performance?
On Unix systems, popen will pass data through an in-memory pipe. Assuming the data isn't swapped out, it won't hit disk. This should give you just about as good performance as you can get without modifying the program being invoked.
popen does pretty much what you are asking for: it does the pipe-fork-exec idiom and gives you a file pointer that you can read and write from.
However, there is a limitation on the size of the pipe buffer (~4K iirc), and if you arent reading quickly enough, the other process could block.
Do you have access to shared memory as a mount point? [on linux systems there is a /dev/shm mountpoint]
1) popen keep the program output in memory. It actually uses pipes to transfer data between the processes.
2) popen looks IMHO as the best option for performance.
It also have an advantage over files of reducing latency. I.e. your program will be able to get the other program output on the fly, while it is produced. If this output is large, then you don't have to wait until the other program is finished to start processing its output.
The problem with having your subcommand redirect to a file is that it's potentially insecure while popen communication can't be intercepted by another process. Plus you need to make sure the filename is unique if you're running several instances of your master program (and thus of your subcommand). The popen solution doesn't suffer from this.
The performance of popen is just fine as long as your don't read/write one byte chunks. Always read/write multiples of 512 (like 4096). But that does apply to file operations as well. popen connects your process and the child process through pipes, so if you don't read then the pipe fills up and the child can't write and vice versa. So all the exchanged data is in memory, but it's only small amounts.
(Assuming Unix or Linux)
Writing to the temp file may be slow if the file is on a slow disk. It also means the entire output will have to fit on the disk.
popen connects to the other program using a pipe, which means that output will be sent to your program incrementally. As it is generated, it is copied to your program chunk-by-chunk.
I am trying to get the CPU usage and details of all the running processes on embedded Linux while a C test is running. Is there a top equivalent in C that I could use ?
I'm not sure what aspect of top confuses you, but just take a look at the source code for top.
I think you need to parse the /proc file system. The kernel has a special file system which is normally mounted at /proc with a unique directory for each process (eg. /proc/30 belongs to the process with PID=30). The files in these directories are all text files and contains lots of info on the running processes.
Instead of top, you should probably use ps with the right arguments and you can parse the output.
You'll have to call that with a system call or popen from your C code. You cannot get this information strictly inside your C code (at least I don't think you can).