I'm trying to write a shell that will eventually take advantage of concurrency. Right now I've got a working shell parser but I'm having trouble figuring out how to execute commands. I've looked a bit at exec (execvp etc.) and it looks promising but I have some questions.
Can exec handle file input/output redirection? Can I set up pipes using exec?
I'm also wondering about subshells. What should subshells return; the exit status of its last statement? Can subshells be a part of a pipe?
These might seem like really dumb questions but please bear with my inexperience.
Can exec handle file input/output redirection?
No, you do that with open() and dup() or dup2() (and close()).
Can I set up pipes using exec?
No, you do that with pipe(), dup() or dup2() and lots of close() calls.
I'm also wondering about subshells. What should subshells return, the exit status of its last statement?
That's the normal convention, yes.
Can subshells be a part of a pipe?
Yes. In a normal shell, you can write something like:
(cd /some/where; find . -name '*.png') | sed 's/xyz/prq/' > mapped.namelist
If you want to get scared, you could investigate posix_spawn() and its support functions. Search for 'spawn' at the POSIX 2008 site, and be prepared to be scared. I think it is actually easier to do the mapping work ad hoc than to codify it using posix_spawn() and its supporters.
The standard technique for a shell is to use fork-exec. In this model, to execute an application the shell uses fork to create a new process that is a copy of itself, and then uses one of the exec variants to replace its own code, data, etc. with the information specified by an executable file on disk.
The nice thing about this model is that the shell can do a little extra work with file descriptors (which are not thrown away by exec) before it changes out its address space. So to implement redirection, it changes out the file descriptors 0, 1, and 2 (stdin, stdout, and stderr, respectively) to point to another open file, instead of console I/O. You use dup2 to change out the meaning of one of these file descriptors, and then exec to start the new process.
Related
I've got a cmd line app in C under Linux that has to run another process, the problem is that the child process prints a lot in a comand line and the whole app gets messy.
Is it possible to disallow child process to print anything in cmd line from parent process? It would be very helpful to for example being able to define a command that allows or disallows printing by a child process.
There's the time-honoured tradition of just redirecting the output to the bit bucket(a), along the lines of:
system("runChild >/dev/null 2>&1");
Or, if you're doing it via fork/exec, simply redirect the file handles using dup2 between the fork and exec.
It won't stop a determined child from outputting to your standard output but it will have to be very tricky to do that.
(a) I'm not usually a big fan of that, just in case something goes wrong. I'd prefer to redirect it to a real file which can be examined later if need be (and deleted eventually if not).
Read Advanced Linux Programming then syscalls(2).
On recent Linux, every executable is in ELF format (except init or systemd; play with pstree(1) or proc(5)) is running in a process started by fork(2) (or clone(2)...) and execve(2).
You might use cleverly dup2(2) with open(2) to redirect STDOUT_FILENO to /dev/null (see null(4), stdout(3), fileno(3))
I've got a cmd line app in C under Linux that has to run another process, the problem is that the child process prints a lot in a comand line
I would instead provide a way to selectively redirect the child process' output. You could use program arguments or environment variables (see getenv(3) and/or environ(7)) to provide such an option to your user.
An example of such a command program starting and redirecting subprocesses and redirecting them is your GCC compiler (see gcc(1); it runs cc1 and as(1) and ld(1)...). Consider downloading and studying its source code.
Study also -for inspiration- the source code of some shell (e.g. sash), or write your own one.
Is there a way to write the error messages to a log file and also print them on the terminal screen?
I tried the following:
dup2(fileno(pFile), STDERR_FILENO); /* redirect stderr to file */
which redirects stderr to the file. However, that writes the error messages to the file but does not show them on screen.
Can this be done without reading and copying the contents of stderr to the file?
Note: I would like to not invoke the shell (system, popen). I did look the implementation of the tee command of coreutils. It copies the standard stream to files.
You have to copy the data "by hand" to send it to two places, though it can be made slightly more efficient with the linux-specific tee and splice system calls (note tee syscall not tee command, http://blog.superpat.com/2010/07/08/a-cup-of-tee-and-a-splice-of-cake/)
It sounds like you might know how to do it and were just hoping not to, but for others, the solution could be something like:
dup2 the original stderr (typically the terminal) to a new file descriptor to save it. Then make a pipe with pipe(). dup2 the write end of the pipe to fd 2. close original write end fd. Now your stderr is a pipe. Start up a thread, or process. In this thread, you copy data from the read end of your pipe into both the file and the original stderr which you saved. When the thread or process gets EOF reading the pipe, close it and exit.
The popen("tee") solution is the same except that it creates an extra shell process (and you have to properly quote the filename passed to the shell in case there are special chars in it... be sure to test weird filenames with greater than sign, spaces, and quote marks in them...). If using popen I think you may also have a (cosmetic?) problem that you can't pclose() because your stderr would stop working, so you will always leave an extra fd open and won't wait4() the child.
If you were using something like GLib, it has a g_spawn_async family of functions which could be used to spawn the tee command without a shell. Otherwise, you would have to do the fork/exec stuff by hand to avoid the shell; you could exec the tee command, or you could fork but NOT exec - just have the child code after fork do the stderr copying, don't rely on the tee command. Or, you could use a thread instead of a process.
If using fork, you may find the source code in gspawn.c helpful; in any complex program it's quite a nightmare that FD_CLOEXEC is not the default on unix for example and it's easy to have the child inherit descriptors that cause problems.
It is also pretty annoying to avoid zombie processes and handle all the errors and blah blah.
Anyway yeah it is more code than one might hope but between the tee source in coreutils and gspawn.c you might have most of it available to copy.
Use a pipe to the tee command.
pFile = popen("tee logfile", "w");
dup2(fileno(pFile), STDERR_FILENO);
There are several ways of redirecting the output of a child process:
using freopen(3)
using dup(3)
using popen(3)
...
What should one pick if all is wanted is to execute a child process and have it output saved in a given file, pretty much like the ls > files.txt works?
What is normally used by shells?
You can discover what your favorite shell uses by strace(1)ing your shell.
In one terminal:
echo $$
In another terminal:
strace -o /tmp/shell -f -p [PID from the first shell]
In the first terminal again:
ls > files.txt
In the second terminal, ^C your strace(1) command and then edit the /tmp/shell output file to see what system calls it made to do the redirection.
freopen(3) manipulates the C standard IO FILE* pointers. All this will be thrown away on the other side of the execve(2) call, because it is maintained in user memory. You could use this after the execve(2) call, but that would be awkward to use generically.
popen(3) opens a single unidirectional pipe(7). This is useful, but extremely limited -- you get either the standard output descriptor or the standard input descriptor. This would fail for something like ls | grep foo | sort where both input and output must be redirected. So this is a poor choice.
dup2(2) will manage file descriptors -- a kernel-implemented resource -- so it will persist across execve(2) calls and you can set up as many file descriptors as you need, which is nice for ls > /tmp/output 2> /tmp/error or handling both input and output: ls | sort | uniq.
There is another mechanism: pty(7) handling. The forkpty(3), openpty(3), functions can manage a new pseudo-terminal device created specifically to handle another program. The Advanced Programming in the Unix Environment, 2nd edition book has a very nice pty example program in its source code, though if you're having trouble understanding why this would be useful, take a look at the script(1) program -- it creates a new pseudo-terminal and uses it to record all input and output to and from programs and stores the transcript to a file for later playback or documentation. You can also use it to script actions in interactive programs, similar to expect(1).
I would expect to find dup2() used mainly.
Neither popen() nor freopen() is designed to handle redirections such as 3>&7. Up to a point, dup() could be used, but the 3>&7 example shows where dup() starts to creak; you'd have to ensure that file descriptors 4, 5, and 6 are open (and 7 is not) before it would handle what dup2() would do without fuss.
I have C program ( program.c ) that calls a shell script ( command.sh ).
command.sh returns an output (a password), i need to get this output in my program.c.
using system(command); , i can't get hold of the output.
is there any other way in C to solve my problem?
Not in pure C. You want POSIX. Specifically popen function.
The specification has a nice example, that you can just copy 1:1
http://pubs.opengroup.org/onlinepubs/009604499/functions/popen.html
Sounds like you're afraid to use libraries. Please try and use libraries, they're just as much part of Unix as shell tools.
In pure C (well ... ignoring the contents of the system() call) you can redirect the shell script output to a file, then read that file.
system("whatever > file");
handle = fopen("file", "r");
/* if ok use handle */
fclose(handle);
You can use popen() as suggested above, but note that in this case you have no control over the process you created (e.g. you will not be able to kill it), you also will not know its exit status.
I suggest using classical pipe/fork/exec combination. Then you will know the pid of your child process, so you will be able so send signals, also with pipe() you are able to redirect process standard output, so you can easily read it in your parent process. As example you can see my accepted answer to popen() alternative.
You should open a pipe using popen but this sounds tedious. A C program calling a shell script for a password.
Can a process 'foo' write to file descriptor 3, for example, in such a way that inside a bash shell one can do
foo 1>f1 2>f2 3>f3
and if so how would you write it (in C)?
You can start your command with:
./foo 2>/dev/null 3>file1 4>file2
Then if you ls -l /proc/_pid_of_foo_/fd you will see that file descriptors are created, and you can write to them via eg.:
write(3,"abc\n",4);
It would be less hacky perhaps if you checked the file descriptor first (with fcntl?).
The shell opens the file descriptors for your program before executing it. Simply use them like you would any other file descriptor, e.g. write(3, buf, len); etc. You may want to do error checking to make sure they were actually opened (attempting to dup them then closing the duplicate would be one easy check).
No.
The file descriptors are opened by the shell and the child process inherits them. It is not the child process which opens these command-line accessible file descriptors, it is the bash process.
There might be a way to convince bash to open additional file descriptors on behalf of the process. That wouldn't be portable to other shells, and I'm not sure if a mechanism even exists -- I am just speculating.
The point is that you can't do this from coding the child process in a special way. The shell would have to abide your desires.
Can a process 'foo' write to file descriptor 3, for example, in such a way that inside a bash shell one can do [...] and if so how would you write it (in C)?
I'm not sure what you are precisely after, but whatever it is, starting point going to be the man dup/man dup2 - this is how the shells make out of a random file descriptor a file descriptor with given number.
But obviously, the process foo has to know somehow that it can write to the file descriptor 3. POSIX only specifies 0, 1 and 2: shell ensures that whatever is started gets the file descriptors and libc in application's context also expects them to be the stdin/stdout/stderr. Starting from 3 and beyond - is up to application developer.