I'm writing a utility for running programs, and I need to capture unbuffered stdout and stderr from the programs. I need to:
Capture stdout and stderr to separate files.
Output needs to not be buffered (or be line buffered).
Without modifying the source of the program being run.
The problem is, when piping output to a file, the stdout stream becomes block buffered rather than line buffered. If the program crashes, the output never gets flushed, and is blank. So I need to capture stdout without buffering (or with line buffering).
I think this can be done with pty's but I'm having difficulty finding any examples that do exactly what I want (most ignore stderr). In fact, I'm not sure I've found any pty examples in C at all; most use a higher-level interface like Python's pty and subprocess modules.
Can anyone help (with code snippets or links)? Any help would be appreciated.
EDIT: I think I've solved it. The following two links were pretty helpful.
http://publib.boulder.ibm.com/infocenter/zos/v1r10/index.jsp?topic=/com.ibm.zos.r10.bpxbd00/posixopenpt.htm
http://www.gidforums.com/t-3369.html
My code is available as a repository:
https://bitbucket.org/elliottslaughter/pty
see man 7 pty
In particular:
Unix 98 pseudo-terminals
An unused Unix 98 pseudo-terminal master is opened by calling
posix_openpt(3). (This function opens the master clone device,
/dev/ptmx; see pts(4).) After performing any program-specific initializations,
changing the ownership and permissions of the slave device
using grantpt(3), and unlocking the slave using unlockpt(3)), the corresponding
slave device can be opened by passing the name returned by
ptsname(3) in a call to open(2).
And now that you know the names of the library functions such a code will need to call, you can do two useful things:
Look up their man pages
Google for example code. Since you know what keywords to use with the search engine I suspect you will have much more luck hunting down examples.
Related
I know already how to implement methods regarding usual freopen(), popen() or similar stdout/stdin/stderr -based redirecting mechanisms, but I wondered how should I apply the said mechanism to static (own) libraries in C? Say, I want to use a library to capture any program with printf() commands or so into a file (for instance) without letting it appear on the console - are there some things I need to acknowledge before applying simple fd dups and just calling the library in the main program? Even piping seems to be complex seeing as execing here is risky...
thanks in advance.
There's an old-timers' trick to force the entire process, regardless of what library the code comes from, to have one of the standard IO ports connected to a different filehandle. You simply close the filehandle in question, then open a new one. If you close(1), then open('some_file', 'w'), then ALL calls that would result in a write to stdout will go to some_file from that point forward.
This works because open() always uses the first file descriptor that isn't currently in use. Presuming that you haven't closed stdin (fd=0), the call to open will get a file descriptor of 1.
There are some caveats. FILE outputs that haven't flushed their buffers will have undefined behavior, but you probably won't be doing this in the middle of execution. Set it up as your process starts and you'll be golden.
I wrote a C program and in the program there are many printf() which output log information to stdout. Now I want to use multiple processes to run the program simultaneously with different arguments. And I want to redirect the output from stdout to a log file using >.
But multiple processes are running at the same time, their log information output overlap with each other, which can be confusing for future analysis.
one solution is: considering that different processes will exit at different time,modify the C program, so each log information is temporarily written into a temporal file. When the C program is about to exit. Read from the temporal file and write the content to stdout, this requires a lot of modification.
My idea is: I hope in the C program, all the printf() output can be buffered, the outputs put into stdout/redirection only when the process exits.
is it possible or not?
thanks!
This is not really possible, unless you are sure that the output is reasonably bounded (e.g. the total output is less than a few megabytes), otherwise use a logging mechanism which send to some central logger (like syslog).
On Linux and most Posix systems, the simplest way to do logging would be to use syslog(3) which is designed for logging (and is able to deal with different processes). I think this is the preferable approach.
With GNU libc, you could consider using open_memstream(3) -to write to memory, and here you need to be sure the total output is bounded- and use atexit(3) to have the memory stream written at the exit of the program into some file; you probably want to use some locking mechanism like flock(2) etc...
As commented by J.Holetzeck the simplest way is to redirect output into different files (perhaps using freopen(3), or simply in the invoking shell), and later merge these files.
I'm guessing you use Linux, or some Posix system. For Windows, I have no idea.
My program is controlling an external application on Linux, passing in input commands via a pipe to the external applications stdin, and reading output result via a pipe from the external applications stdout.
The problem is that writes to pipes are buffered by block, and not by line, and therefore delays occur before my app receives data output by the external application. The external application cannot be altered to add explicit fflush() calls.
When I set the external application to /bin/cat -n (it echoes back the input, with line numbers added), it works correctly, it seems, cat flushes after each line. The only way to force the external application to flush, is sending exit command to it; as it receives the command, it flushes, and all the answers appears on the stdout, just before exiting.
I'm pretty sure, that Unix pipes are appropiate solution for that kind of interprocess communication (pseudo server-client), but maybe I'm wrong.
(I've just copied some text from a similar question: Force another program's standard output to be unbuffered using Python)
Don't use a pipe. Use a pty instead. Pty's (pseudo-ttys) have the benefit of being line buffered if you want it, which provides you with simple framing for your data stream.
Using a PTY may be an overkill for the problem at hand (although it will work).
If the "target application" (the Delphi command-line utility) is dynamically linked, a possibly much simpler solution is to interpose (via LD_PRELOAD) a small library into the application. That library simply needs to implement isatty, and answer true (return 1) regardless of whether the output is going to a pipe or a terminal. You may wish to do that for all file descriptors, or just for STDOUT_FILENO.
Most UNIX implementations will call isatty to decide whether to do full buffering or line buffering for a given file descriptor.
Hmm, glibc doesn't. It calls __fxstat, and then only calls isatty if the status indicates that fd is going to a character device. So you'll need to interpose both __fxstat and isatty. More on library interposition here.
By default standard input and standard output are fully buffered unless they are connected to an interactive device in which cases they are line buffered [1]. Pipes are non-interactive devices. PTYs are interactive devices. "Fully buffered" means "use a chunk of memory of a certain size".
I'm sure you want line buffering. Therefore using a master/slave PTY instead of pipes should bring the controlled application into the right buffering mode automatically.
[1] see "stdin(3)" and "setbuf(3)" for details.
Why calling fflush suitably (on the write side) don't work for you?
You can use poll (or other syscalls like ppoll, pselect, select) to check availability of input on the read side.
If the external application is using <stdio.h> without calling fflush appropriately (perhaps by setbuf making it happen on newlines ....), data would remain inside its FILE* buffer without even being sent (with a write syscall) to the pipe!
An application can detect if its output is a terminal with e.g. isatty. But it should ensure that flushing happens...
As Michael Dillon suggested, using pty-s is probably the best. But it is hard (I forgot the gory details).
How can I capture another process's output using pure C? Can you provide sample code?
EDIT: let's assume Linux. I would be interested in "pretty portable" code. All I want to do is to execute a command, capture it's output and process it in some way.
There are several options, but it does somewhat depend on your platform. That said popen should work in most places, e.g.
#include <stdio.h>
FILE *stream;
stream = popen("acommand", "r");
/* use fread, fgets, etc. on stream */
pclose(stream);
Note that this has a very specific use, it creates the process by running the command acommand and attaches its standard out in a such as way as to make it accessible from your program through the stream FILE*.
If you need to connect to an existing process, or need to do richer operations, you may need to look into other facilities. Unix has various mechanisms for hooking up a processes stdout etc.
Under windows you can use the CreateProcess API to create a new process and hook up its standard output handle to what you want. Windows also supports popen.
There's no plain C way to do this that I know of though, so it's always going somewhat dependent on platform specific APis.
Based on your edits popen seems ideal, it is "pretty portable", I don't think there's a unix like OS without it, indeed it is part of the Single Unix Specification, and POSIX, and it lets you do exactly what you want, execute a process, grab its output and process it.
If you can use system pipes, simply pipe the other process's output to your C program, and in your C program, just read the standard input.
otherprocess | your_c_program
Which OS are you using? On *nix type OS if you are process is outputting to STDOUT or STDERR you can obviously use pipes
I've coded a program in C that sends messages to the stdout using printf and I'm having trouble redirecting the output to a file (running from bash).
I've tried:
./program argument >> program.out
./program argument > program.out
./program >> program.out argument
./program > program.out argument
In each case, the file program.out is created but it remains empty. After the execution ends the file size is 0.
If I omit the redirection when executing the program:
./program argument
Then, all messages sent to stdout using printf are shown in the terminal.
I have other C programs for which I've no problem redirecting the output this way.
Does it have to do with the program itself? with the argument passing?
Where should look for the problem?
Some details about the C program:
It does not read anything from stdin
It uses BSD Internet Domain sockets
It uses POSIX threads
It assigns a special handler function for SIGINT signal using sigaction
It sends lots of newlines to stdout (for those of you thinking I should flush)
Some code:
int main(int argc, char** argv)
{
printf("Execution started\n");
do
{
/* lots of printf here */
} while (1);
/* Code never reached */
pthread_exit(EXIT_SUCCESS);
}
Flushing after newlines only works when printing to a terminal, but not necessarily when printing to a file. A quick Google search revealed this page with further information: http://www.pixelbeat.org/programming/stdio_buffering/
See the section titled "Default Buffering modes".
You might have to add some calls to fflush(stdout), after all.
You could also set the buffer size and behavior using setvbuf.
Flushing the buffers is normally handled by the exit() function, which is usually called implicitly by a return from main(). You are ending your program by raising SIGINT, and apparently the default SIGINT handler does not flush the buffers.
Take a look at this article:
Applying Design Patterns to Simplify Signal Handling. The article is mostly C++, but there is a useful C example in the 2nd section, which shows how to use SIGINT to exit your program gracefully.
As for why the behavior of a terminal differs from a file,
take a look at Stevens' Advanced Programing in the UNIX Environment Section 5.4 on Buffering. He says that:
Most implementations default to the following types of buffering.
Standard error is always unbuffered.
All other streams are line buffered if they refer to a terminal device; otherwise, they are fully buffered.
The four platforms discussed in this book follow these conventions for standard I/O buffering: standard error is unbuffered, streams open to terminal devices are line buffered, and all other streams are fully buffered.
Has the program terminated by the time you check the contents of the redirected file? If it's still running, your output might still be buffered somewhere up the chain, so you don't see it in the file.
Apart from that, and the other answers provided so far, I think it's time to show a representative example of the problem code. There's too many esoteric possibilities.
EDIT
From the look of the sample code, if you've got a relatively small amount of printing happening, then you're getting caught in the output buffer. Flush after each write to be sure that it's gone to disk. Typically you can have up to a page size's worth of unwritten data lying around otherwise.
In the absence of a flush, the only time you can be sure you've got everything on disk is when the program exits. Even a thread terminating won't do it, since output buffers like that aren't per-thread, they're per-process.
Suggestions:
Redirect stderr to a file as well.
Try tail -f your output file(s).
Open a file and fprintf your logging (to help figure out what's going on).
Search for any manual closes/duplication/piping of std* FILE handles or 1-3 file descriptors.
Reduce complexity; cut out big chunks of functionality until printfs work. Then readd them until it breaks again. Continue until you identify the culprit code.
Just for the record, in Perl you would use:
use IO::Handle;
flush STDOUT;
autoflush STDOUT;