I want execute a Linux command in a C program and read (parse) stdout from this command in the program. The code below works but I don't know how to limit execution time of the command, in addition to the string and bytes read limits. Any ideas?
FILE *ps_pipe;
int bytes_read;
int nbytes = 100;
char *my_string=NULL;
char message[1024];
message=sprintf(message,"any command here");
ps_pipe = popen (message, "r");
my_string = (char *) malloc (nbytes + 1);
bytes_read = getdelim (&my_string, &nbytes, "delimiter_word", ps_pipe);
pclose(ps_pipe);
free(my_string);
You could do that with select(). Select can "wait" on one or more file descriptors for an event to happen (readable, writable, ...), with an optional time-out. Since it operates on file descriptors, you'll also need fileno(ps_pipe).
Keep in mind however that you won't be able to kill the forked process easily, because popen hides certain details of the child process. If you need such control, you'll need to use lower level functions fork(), pipe(), dup(), exec(), wait() and possibly kill().
Related
I'm just starting to learn C programming and I have some uncertainty about fork(), exec(), pipe(), etc.
I've developed this code, but when I execute it, the variable c remains empty, so I don't know if the child isn't writing to the pipe, or the parent isn't reading from it.
Could you help me please? This is the code:
int main() {
int pid=0;
int pipefd[2];
char* c=(char *)malloc(sizeof(char));
FILE *fp;
pipe(pipefd);
pid=fork();
if (pid==0){
close(pipefd[0]);
dup2(pipefd[1],1);
close(pipefd[1]);
execl("ls -l | cut -c28","ls -l | cut -c28", (char *) 0);
}
else{
close(pipefd[1]);
read(pipefd[0], c, 1);
char* path="/home/random";
char* txt=".txt";
char* root=malloc(strlen(path) + strlen(txt) + sizeof(char));
strcpy(root,path);
strcat(root,c);
strcat(root,txt);
close(pipefd[0]);
fp=fopen(root,"w+");
(...)
}
The problem is that the final root string its only "/home/random.txt" because there is nothing in the char c, and what I want is to open the file "/home/random(number stored in char c).txt".
execl executes a single command, and is not aware of shell concepts such as pipes. If you want to execute a shell command, you will have to execute a shell, as follows:
execl("/bin/sh","/bin/sh","-c","ls -l | cut -c28", (char*) 0);
Always check the return value of the system calls (like execve(2) and derived functions like execl(3)), and use the errno(3) to figure out what went wrong.
In your case the execl line fails.
Using strcpy/strcat seems a bit excessively complex. snprintf can turn those 3 lines into one.
snprintf( root, size_of_buf, "/home/random%s", c );
Additionally, check your error codes. As noted, execl is failing and you don't know it. fork, dup2, ...,can also fail, you want to know sooner rather than later.
Basically I want to do in C (and without buffering) the same as this bash-script:
#!/bin/sh
cat ./fifo_in | myprogram > ./fifo_out
In other words I want to exec "myprogram" and redirect its stdin and stdout to two pipes which have been created previously.
Another program is feeding data into fifo_in and reading out of fifo_out.
Of course it would be easy to just read from ./fifo_in, buffer it in the parent and write to myprogram's stdin (and reverse for stdout and ./fifo_out) but I think there is probably a way to let "myprogram" read/write directly from/to the fifos without buffering in the parent process.
Edit:
Eugen's answer seems to be the correct one, but I cannot get it to work.
I use this function on the C-side, which seems correct to me:
pid_t execpipes(const char *wd, const char *command, const char *pipename)
{
char pipename_in[FALK_NAMESIZE];
char pipename_out[FALK_NAMESIZE];
strcpy(pipename_in, FALKPATH);
strcat(pipename_in, "/");
strcat(pipename_in, FALK_FIFO_PATH);
strcat(pipename_in, "/");
strncat(pipename_in, pipename, FALK_NAMESIZE-2);
strcpy(pipename_out, pipename_in);
strcat(pipename_out, "R");
pid_t pid;
pid = fork();
if (pid < 0)
{ //Error occured
perror("fork");
exit(1);
}
if (pid == 0)
{
chdir(wd);
d("execpipes: pipename_in=\"%s\"\n", pipename_in);
d(" pipename_out=\"%s\"\n", pipename_out);
freopen(pipename_in,"r",stdin);
freopen(pipename_out,"w",stdout);
d("execpipes: command=\"%s\"\n", command);
execl("/bin/sh", "sh", "-c", command, (char *)NULL); // using execv is probably faster
// Should never get here
perror("execl");
exit(1);
}
return pid;
}
I read and write the pipes from a PHP-script (only relevant part posted):
$pipe_in = fopen($fp.$pipename, "w");
$DEBUG .= "Write to pipe_in\n";
$ret = fwrite($pipe_in, $in);
$pipe_out = fopen($fp.$pipename.'R', "r");
$DEBUG .= "Read from pipe_out\n";
$atext = fread($pipe_out, 200000); // Program hangs here
The program is started correctly, receives the input via $pipe_in correctly, processes the data correctly and (because it ran fine for many months) I assume it puts out the data correctly to stdout, but when I try to read from $pipe_out, it hangs. I know that the pipes themselves are set up correctly because if I don't open $pipe_out, the program does not get any input - which makes sense because there is no reader for $pipe_out and therefore the pipeline is not complete. So I can open $pipe_out, but I cannot read anything from it, which is quite strange.
Edit2:
Program works now, thanks guys - For some reason the first pipe has to be closed before you can read from the second pipe:
$pipe_in = fopen($fp.$pipename, "w");
$pipe_out = fopen($fp.$pipename.'R', "r");
$DEBUG .= "Write to pipe_in\n";
$ret = fwrite($pipe_in, $in);
fclose($pipe_in);
$DEBUG .= "Read from pipe_out\n";
$atext = fread($pipe_out, 200000);
fclose($pipe_out);
unlink($fp.$pipename);
unlink($fp.$pipename.'R');
I'd write a small wrapper for myprogram, that does
freopen("./fifo_in","r",stdin)
freopen("./fifo_out","w",stdout)
(Ofcourse not with constant paths!), then execve myprogram
Korn shell supports coprocesses, which I think effectively does what you ask: read from a pipe and write to a pipe (which can be stdout and stdin of a C process)
http://www.dartmouth.edu/~rc/classes/ksh/coprocesses.html
How about
myprogram < ./fifo_in > ./fifo_out
?
As for getting rid of the buffering: Since your program directly reads/writes the pipes, the buffering shouldn't hurt you.
An important point is that the process which writes fifo_in should flush properly so you don't have to wait. The same goes for your output: As soon as a "work unit" is complete, flush your stdout which will make the data available to whoever reads the output pipe.
But you can't do anything in myprogram to make the writer of fifo_in flush its buffers.
[EDIT] To do this from C (without the help of a shell), use code like this:
- Put the names of the two pipes into local variables on the stack
- Call `fork()`. If that returns '0', then open the two fifos with `freopen()` [like Eugen suggested][1]
- Call `execve` to launch the real exec.
That's (in a nutshell) what the shell is doing when it runs commands. Make sure the parent process (the one where fork() returns a PID != 0) handles the signal SIGCHLD
Perhaps you are looking of a named pipe? For example:
mkfifo fifo_in
As a test stub for my_program.c, to read fifo_in via the buffered stdin:
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
int main(void) {
char buf[80];
if (!freopen("./fifo_in", "r", stdin)) {
perror("freopen");
exit(EXIT_FAILURE);
}
while (!ferror(stdin)) {
while (fgets(buf, sizeof buf, stdin))
fputs(buf, stdout);
sleep(1);
}
return 0;
}
Then as a test for the writer, using the bash shell:
for x in {1..10}; do
echo $x
echo $x >> fifo_in
sleep 1
done
Notes:
I'd prefer to use unbuffered I/O.
The writer, at least on my machine, blocks until there is a reader.
The reader, in this sample, cannot tell when the writer is finished.
I have a server and client application, using IPC queues. The server is (for now) simply sending back the text received from the client. I would like the server to change the letters in the message from lower to upper case. I'm wondering how to achieve it. Do I have to create a pipe? I'm thinking about 'grabbing' the text from the received queue, executing the tr command on it and sending back to the client. But if using a pipe, from where do I get the file descriptors? I mean, int fds[2]; and pipe(fds); gives me a pipe, but it's not working on two char arrays like this:
int fds[2];
pipe(fds);
char a[100];
char b[100];
fds[0] = open(a,O_RDOLNY);
fds[1] = open(b,O_WRONLY);
How can I execute a tr command on a text held by a message queue?
I wouldn't fork a program for this:
p = str;
while (*p) {
*p = toupper(*p);
p++;
}
More seriously, you should probably use popen that automatically (and robustly) forks and uses a pipe to setup a FILE * for you.
FILE *cmd = popen("tr ... ", "r");
And then simple fgets from it (don't forget to pclose it). Sadly on Linux you can't write and read to a popened file at the same time (you can on FreeBSD).
EDIT
Since this is a homework question (and frankly because I don't think it's trivial to get it completely right at this time of night), here is what popen actually does:
Create a pipe
Fork a shell that will run the command
Return a FILE * (possibly via fdopen)
The last step is really optional as you could always read from the file descriptor directly.
int main(int argc, char *argv[])
{
FILE *fp = fopen("a.txt", "wt");
fprintf(fp, "AAAA");
// No flush. and No close
raise(SIGTERM);
exit(EXIT_SUCCESS);
}
result: No data has written to a.txt
I expected this is fine. Because the system will close the file handle and then the filesystem driver flushes the unflushed data in his Close handler. But it wasn't.
I tested this code on EXT4, ubuntu 11.10
Question:
I thought ALL filesystems must flush unflushed data at his close processing.
Posix doesn't have the rule?
P.S This code worked well (flushed well) on NTFS, Win7
int _tmain(int argc, _TCHAR* argv[])
{
HANDLE h = CreateFile(L"D:\\a.txt", GENERIC_READ|GENERIC_WRITE,
0, 0, OPEN_ALWAYS, 0, 0);
BYTE a[3];
memset(a, 'A', 3);
DWORD dw;
WriteFile(h, (PVOID)a, 3, &dw, 0);
TerminateProcess(GetCurrentProcess(), 1);
return 0;
}
Edit:
I tested it again with system call write. And it was flushed well.
int main(int argc, char** argv)
{
int fd = open("a.txt", O_CREAT|O_TRUNC|O_WRONLY);
char buf[3];
memset(buf, 'A', 3);
size_t result = write(fd, buf, 3);
raise(SIGTERM);
exit(EXIT_SUCCESS);
return 0;
}
This isn't anything to do with the filesystem, rather it's the behaviour of the C implementation you are using that determines when open streams are flushed or not.
Under POSIX, the default action action for the SIGTERM signal is:
Abnormal termination of the process. The process is terminated with all the consequences of _exit()...
_exit() is equivalent to _Exit() according to the C standard and the choice of whether to flush streams is not specified by the standard:
The _Exit() and _exit() functions shall not call functions registered with atexit() nor any registered signal handlers. Whether open streams are flushed or closed, or temporary files are removed is implementation-defined...
Assuming you are using glibc on Linux, from the documentation (emphasis mine):
When a process terminates for any reason—either because the program terminates, or as a result of a signal—the following things happen:
All open file descriptors in the process are closed. See Low-Level I/O. Note that streams are not flushed automatically when the process terminates; see I/O on Streams.
I'm not familiar with the Windows' WriteFile and TerminateProcess so I can't comment on what the documented behaviour is.
It doesn't have anything to do with file system drivers. The issue is that the CRT is buffering the file stream itself. You set the buffer size with setvbuf(), it uses a default if you don't use this function. There's no buffering in the application when you use WriteFile(), output is buffered in the operating system's file system cache. Immune from abrupt app aborts.
You'll have to call fflush() to achieve the same.
I have a C application with many worker threads. It is essential that these do not block so where the worker threads need to write to a file on disk, I have them write to a circular buffer in memory, and then have a dedicated thread for writing that buffer to disk.
The worker threads do not block any more. The dedicated thread can safely block while writing to disk without affecting the worker threads (it does not hold a lock while writing to disk). My memory buffer is tuned to be sufficiently large that the writer thread can keep up.
This all works great. My question is, how do I implement something similar for stdout?
I could macro printf() to write into a memory buffer, but I don't have control over all the code that might write to stdout (some of it is in third-party libraries).
Thoughts?
NickB
I like the idea of using freopen. You might also be able to redirect stdout to a pipe using dup and dup2, and then use read to grab data from the pipe.
Something like so:
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#define MAX_LEN 40
int main( int argc, char *argv[] ) {
char buffer[MAX_LEN+1] = {0};
int out_pipe[2];
int saved_stdout;
saved_stdout = dup(STDOUT_FILENO); /* save stdout for display later */
if( pipe(out_pipe) != 0 ) { /* make a pipe */
exit(1);
}
dup2(out_pipe[1], STDOUT_FILENO); /* redirect stdout to the pipe */
close(out_pipe[1]);
/* anything sent to printf should now go down the pipe */
printf("ceci n'est pas une pipe");
fflush(stdout);
read(out_pipe[0], buffer, MAX_LEN); /* read from pipe into buffer */
dup2(saved_stdout, STDOUT_FILENO); /* reconnect stdout for testing */
printf("read: %s\n", buffer);
return 0;
}
If you're working with the GNU libc, you might use memory streams string streams.
You can "redirect" stdout into file using freopen().
man freopen says:
The freopen() function opens the file
whose name is the string pointed to
by path and associates the stream
pointed to by stream with it. The
original stream (if it exists) is
closed. The mode argument is used
just as in the fopen() function.
The primary use of the freopen()
function is to change the file
associated with a standard text
stream (stderr, stdin, or stdout).
This file well could be a pipe - worker threads will write to that pipe and writer thread will listen.
Why don't you wrap your entire application in another? Basically, what you want is a smart cat that copies stdin to stdout, buffering as necessary. Then use standard stdin/stdout redirection. This can be done without modifying your current application at all.
~MSalters/# YourCurrentApp | bufcat
You can change how buffering works with setvbuf() or setbuf(). There's a description here: http://publications.gbdirect.co.uk/c_book/chapter9/input_and_output.html.
[Edit]
stdout really is a FILE*. If the existing code works with FILE*s, I don't see what prevents it from working with stdout.
One solution ( for both things your doing ) would be to use a gathering write via writev.
Each thread could for example sprintf into a iovec buffer and then pass the iovec pointers to the writer thread and have it simply call writev with stdout.
Here is an example of using writev from Advanced Unix Programming
Under Windows you would use WSAsend for similar functionality.
The method using the 4096 bigbuf will only sort of work. I've tried this code, and while it does successfully capture stdout into the buffer, it's unusable in a real world case. You have no way of knowing how long the captured output is, so no way of knowing when to terminate the string '\0'. If you try to use the buffer you get 4000 characters of garbage spit out if you had successfully captured 96 characters of stdout output.
In my application, I'm using a perl interpreter in the C program. I have no idea how much output is going to be spit out of what ever document is thrown at the C program, and hence the code above would never allow me to cleanly print that output out anywhere.