I want to spawn a child process and capture its stdout (and stderr) using overlapped I/O without using threads. Here's my current knowledge of all the stars that must align in order to achieve that, i.e. here's the recipe:
Set an inheritable handle as the stdout (and stderr) of the process when creating the process (set hStdOutput and hStdError fields of STARTUPINFO).
Tell the process to inherit any inheritable handles from its parent so that it will inherit said stdout handle (arg bInheritHandles of CreateProcess()).
The handle itself must be the writing end of an anonymous pipe. I will then capture the process' stdout by reading from the reading end of that pipe.
The pipe must be overlapped-I/O-enabled.
Since anonymous pipes don't support overlapped I/O, I must emulate them using a named pipe (which I create with FILE_FLAG_OVERLAPPED). This pipe will serve as the writing end of the anonymous pipe. I then open this pipe using CreateFile to get a handle to the reading end (this is more/less how anonymous pipes are implemented in Windows also).
The reading end of the pipe must not be inherited by the child process, so I am careful to not make it inheritable. (does anyone have a good explanation for why that is?)
After the process is created and the writing handle is thus inherited, I close said handle in the parent process. This is so that (thanks #o11c in the comments) the writing end of the pipe is left with only one handle open to it (the handle that the child holds), so that when the child exists, the pipe is closed and reading from it fails with a broken pipe error (otherwise we would never know when to stop reading).
Now that everything is set up I can start reading from the pipe: I create a completion port, perform an overlapped ReadFile() and then check the completion status.
And here's the problem: GetQueuedCompletionStatus() hangs until timeout and then returns WAIT_TIMEOUT instead of returning immediately with either some data or with ERROR_IO_PENDING so I can check again.
Below is the minimum amount of C code that reproduces the problem. Any help appreciated. Thanks!
#include <assert.h>
#include <stdio.h>
#include <string.h>
#include <windows.h>
PROCESS_INFORMATION pi;
STARTUPINFO si;
SECURITY_ATTRIBUTES sa;
OVERLAPPED o;
#define sz 1024
unsigned char buf[sz];
char* pipe_name = "\\\\.\\pipe\\t1";
int main(int argc, char **argv) {
sa.nLength = sizeof(sa);
sa.bInheritHandle = 1;
HANDLE stdout_r = CreateNamedPipe(pipe_name,
PIPE_ACCESS_INBOUND | FILE_FLAG_OVERLAPPED,
0,
1,
8192, 8192,
120 * 1000,
0
);
assert(stdout_r != INVALID_HANDLE_VALUE);
HANDLE stdout_w = CreateFile(pipe_name,
GENERIC_WRITE,
0,
&sa,
OPEN_EXISTING,
FILE_ATTRIBUTE_NORMAL | FILE_FLAG_OVERLAPPED,
0
);
assert(stdout_w != INVALID_HANDLE_VALUE);
si.hStdOutput = stdout_w;
si.hStdError = stdout_w;
si.dwFlags = STARTF_USESTDHANDLES;
assert(CreateProcess(0, "dir", 0, 0, 1, 0, 0, 0, &si, &pi) != 0);
assert(CloseHandle(stdout_w) != 0);
HANDLE iocp = CreateIoCompletionPort(INVALID_HANDLE_VALUE, 0, 0, 0);
assert(iocp != INVALID_HANDLE_VALUE);
if (ReadFile(stdout_r, buf, sz, 0, &o) == 0) {
if (GetLastError() == ERROR_IO_PENDING) {
DWORD n;
ULONG_PTR compkey;
LPOVERLAPPED po;
int ret = GetQueuedCompletionStatus(iocp, &n, &compkey, &po, 1000);
if (!ret) {
assert(GetLastError() != WAIT_TIMEOUT);
}
}
}
return 0;
}
NOTE: I built/tested this with mingw but VC should work too (in ANSI mode).
Related
I have written a program that implements a tiny shell to process commands from the user.
If the entered command is recognised as internal command, my program executes this command.
These commands are implemented as internal functions and their output is being processed by another internal function that is able to send the text to the console and / or to file for logging purposes.
If the entered command is not recognised, I try to execute the entered command as part of the windows command shell, e.g. : cmd dir would execute the dir command and the output gets printed on the console. This is done via CreateProcess. Until now I did not specify the members hStdError, hStdOutput and hStdInput of the STARTUPINFO parameter.
I tried to implement and adapt the example of Creating a Child Process with Redirected Input and Output.
I did not use their implementation of the child process, but tried to get the output of the dir command into my application:
#include "pch.h"
#include <windows.h>
#define BUFSIZE 512
HANDLE g_hChildStd_IN_Rd = NULL;
HANDLE g_hChildStd_IN_Wr = NULL;
HANDLE g_hChildStd_OUT_Rd = NULL;
HANDLE g_hChildStd_OUT_Wr = NULL;
PROCESS_INFORMATION piProcInfo;
STARTUPINFO siStartInfo;
void CreateChildProcess()
// Create a child process that uses the previously created pipes for STDIN and STDOUT.
{
TCHAR szCmdline[] = TEXT("cmd /c dir q:\\Sicherung\\Bilder /s");
BOOL bSuccess = FALSE;
// Set up members of the PROCESS_INFORMATION structure.
ZeroMemory(&piProcInfo, sizeof(PROCESS_INFORMATION));
// Set up members of the STARTUPINFO structure.
// This structure specifies the STDIN and STDOUT handles for redirection.
ZeroMemory(&siStartInfo, sizeof(STARTUPINFO));
siStartInfo.cb = sizeof(STARTUPINFO);
siStartInfo.hStdError = g_hChildStd_OUT_Wr;
siStartInfo.hStdOutput = g_hChildStd_OUT_Wr;
siStartInfo.hStdInput = g_hChildStd_IN_Rd;
siStartInfo.dwFlags |= STARTF_USESTDHANDLES;
// Create the child process.
bSuccess = CreateProcess(NULL,
szCmdline, // command line
NULL, // process security attributes
NULL, // primary thread security attributes
TRUE, // handles are inherited
0, // creation flags
NULL, // use parent's environment
NULL, // use parent's current directory
&siStartInfo, // STARTUPINFO pointer
&piProcInfo); // receives PROCESS_INFORMATION
// If an error occurs, exit the application.
if (!bSuccess)
return; // ErrorExit(("CreateProcess"));
else
{
// Close handles to the child process and its primary thread.
// Some applications might keep these handles to monitor the status
// of the child process, for example.
//CloseHandle(piProcInfo.hProcess);
//CloseHandle(piProcInfo.hThread);
}
}
void ReadFromPipe(void)
// Read output from the child process's pipe for STDOUT
// and write to the parent process's pipe for STDOUT.
// Stop when there is no more data.
{
DWORD dwRead, dwWritten;
CHAR chBuf[BUFSIZE];
BOOL bSuccess = FALSE;
HANDLE hParentStdOut = GetStdHandle(STD_OUTPUT_HANDLE);
for (;;)
{
DWORD objectstat = WAIT_TIMEOUT;
//do
//{
// objectstat = WaitForSingleObject(piProcInfo.hProcess, 0);
//} while (objectstat != WAIT_OBJECT_0);
memset(&chBuf[0], 0x00, BUFSIZE);
bSuccess = ReadFile(g_hChildStd_OUT_Rd, chBuf, BUFSIZE, &dwRead, NULL);
if (!bSuccess)
break;
bSuccess = WriteFile(hParentStdOut, chBuf,
dwRead, &dwWritten, NULL);
if (!bSuccess)
break;
if (dwRead == 0)
break;
}
}
int main()
{
SECURITY_ATTRIBUTES saAttr;
printf("\n->Start of parent execution.\n");
// Set the bInheritHandle flag so pipe handles are inherited.
saAttr.nLength = sizeof(SECURITY_ATTRIBUTES);
saAttr.bInheritHandle = TRUE;
saAttr.lpSecurityDescriptor = NULL;
// Create a pipe for the child process's STDOUT.
if (!CreatePipe(&g_hChildStd_OUT_Rd, &g_hChildStd_OUT_Wr, &saAttr, 0))
return -1;// ErrorExit("StdoutRd CreatePipe");
// Ensure the read handle to the pipe for STDOUT is not inherited.
if (!SetHandleInformation(g_hChildStd_OUT_Rd, HANDLE_FLAG_INHERIT, 0))
return -2;// ErrorExit(("Stdout SetHandleInformation"));
// Create a pipe for the child process's STDIN.
if (!CreatePipe(&g_hChildStd_IN_Rd, &g_hChildStd_IN_Wr, &saAttr, 0))
return -3 ;// ErrorExit(("Stdin CreatePipe"));
// Ensure the write handle to the pipe for STDIN is not inherited.
if (!SetHandleInformation(g_hChildStd_IN_Wr, HANDLE_FLAG_INHERIT, 0))
return -4;// ErrorExit(("Stdin SetHandleInformation"));
// Create the child process.
CreateChildProcess();
ReadFromPipe();
CloseHandle(piProcInfo.hProcess);
CloseHandle(piProcInfo.hThread);
return 0;
}
I know, that the problem has to be with ReadFile. I can not determine when all output of the dir command has been processed. Checking dwRead for 0 or for BUFSIZE does not work. dwReadnever becomes 0, and it can happen that it is less than BUFSIZE, because the dir command is not fast enough.
So, how am i supposed to end processing of the pipe data?
Ok, after i searched some different terms in google, I came up with this link to stackoverflow ;) :
How to read output from cmd.exe using CreateProcess() and CreatePipe()
Ian Boyd wrote there :
Once you've launched your child process: be sure to close those ends of the pipe you no longer need.
result = CreateProcess(...);
//CreateProcess demands that we close these two populated handles when we're done with them. We're done with them.
CloseHandle(pi.hProcess);
CloseHandle(pi.hThread);
/*
We've given the console app the writable end of the pipe during CreateProcess; we don't need it anymore.
We do keep the handle for the *readable* end of the pipe; as we still need to read from it.
The other reason to close the writable-end handle now is so that there's only one out-standing reference to the writeable end: held by the console app.
When the app closes, it will close the pipe, and ReadFile will return code 109 (The pipe has been ended).
That's how we'll know the console app is done. (no need to wait on process handles with buggy infinite waits)
*/
CloseHandle(g_hChildStd_OUT_Wr);
g_hChildStd_OUT_Wr = 0;
CloseHandle(g_hChildStd_IN_Rd);
g_hChildStd_OUT_Wr = 0;
The common problem with most solutions is that people try to wait on a process handle. There are many problems with this; the main one being that if you wait for the child the terminate, the child will never be able to terminate.
After closing the unneeded handles ReadFile works as expected.
I am wanting to pass an anonymous Pipe HANDLE to a Child Process. This answer seems to explain it well for C++, however I am wanting to do this in C.
Do I convert the handle into an integer? Or do I pass the memory address of the HANDLE to the child process, and then make another HANDLE pointing to that?
For example:
Parent:
BOOL bCreatePipe, bReadFile;
HANDLE hRead = NULL;
HANDLE hWrite = NULL;
SECURITY_ATTRIBUTES lpPipeAttributes;
lpPipeAttributes.nLength = sizeof(lpPipeAttributes);
lpPipeAttributes.lpSecurityDescriptor = NULL;
lpPipeAttributes.bInheritHandle = TRUE;
// Create pipe file descriptors for parent and child
bCreatePipe = CreatePipe(&hRead, &hWrite, &lpPipeAttributes, (DWORD)BUFFER_SIZE);
if (bCreatePipe == FALSE) {
printf("[-]Error creating IPC pipe : %d", GetLastError());
exit(-1);
}
// Create command line arguments for child process
snprintf(child_cmd, CMD_LINE_SIZE, "%d", &hWrite);
// Create child process to handle request
if ( !CreateProcess(
"C:\\Users\\Child.exe", // No module name (use command line)
child_cmd, // Command line
NULL, // Process handle not inheritable
NULL, // Thread handle not inheritable
TRUE, // Set handle inheritance to TRUE (for pipe)
0, // No creation flags
NULL, // Use parent's environment block
NULL, // Use parent's starting directory
&si, // Pointer to STARTUPINFO structure
&pi) // Pointer to PROCESS_INFORMATION structure
)
{
printf("[-]CreateProcess failed : %d\n", GetLastError());
exit(-1);
}
Child:
// Set variables to arguments passed by parent
HANDLE hWrite = atoi(argv[0]);
yes, this is ok pass HANDLE by value. in practice currently your code will be work ok. however need remember that HANDLE is 64-bit size on 64-bit system - so not fit in int which is 32-bit size (now user mode handle values in practice fit to 32bit). so need use say %I64x format to encode handle value and _atoi64 or _wcstoi64 to decode.
for example in parent:
WCHAR child_cmd[32];
swprintf(child_cmd, L"<%I64x>", (ULONG64)(ULONG_PTR)hWrite);
and in child:
HANDLE hWrite = 0;
if (PWSTR sz = wcschr(GetCommandLineW(), '<'))
{
hWrite = (HANDLE)(ULONG_PTR)_wcstoi64(sz + 1, &sz, 16);
if (*sz != '>')
{
hWrite = 0;
}
}
as separate note - use CreatePipe not the best choice - this api very bad design, say one handle only for write, another only for read, can not select asynchronous I/O, can not make one handle inherit and another not (as need in this case) - better use CreateNamedPipeW + CreateFileW for create pipe pair. or this way if you want no name on pipe (work from win7)
I have been attempting to use anonymous pipes to communicate with a spawned subprocess via their stdin and stdout, similar to this example. So far, so good - I communicate with it via WriteFile(), and I read data from it via PeekNamedPipe() and then ReadFile(). Life is good.
However, I'm encountering something distressing - when the subprocess closes unexpectedly, a parent process call to WriteFile() never seems to return any sort of failure state indicating that the pipe has closed, and my parent process happily continues chugging along writing data until I overflow the pipe's internal buffer and block forever.
My setup code looks like this:
// Set up pipes
SECURITY_ATTRIBUTES sec_attrs;
memset(&sec_attrs, 0, sizeof(SECURITY_ATTRIBUTES));
sec_attrs.nLength = sizeof(SECURITY_ATTRIBUTES);
sec_attrs.bInheritHandle = TRUE;
sec_attrs.lpSecurityDescriptor = NULL;
if (!CreatePipe(&midi_process_in_reader, &midi_process_in_writer, &sec_attrs, 0))
{
DEBUGOUT("Could not initialize midiproc stdin");
return false;
}
if (!SetHandleInformation(midi_process_in_writer, HANDLE_FLAG_INHERIT, 0))
{
DEBUGOUT("Could not disinherit midiproc stdin");
return false;
}
if (!CreatePipe(&midi_process_out_reader, &midi_process_out_writer, &sec_attrs, 0))
{
DEBUGOUT("Could not initialize midiproc stdout/stderr");
return false;
}
if (!SetHandleInformation(midi_process_out_reader, HANDLE_FLAG_INHERIT, 0))
{
DEBUGOUT("Could not disinherit midiproc stdin");
return false;
}
// Launch the subprocess
PROCESS_INFORMATION proc_info;
memset(&proc_info, 0, sizeof(proc_info));
STARTUPINFO startup_info;
memset(&startup_info, 0, sizeof(startup_info));
startup_info.cb = sizeof(startup_info);
startup_info.hStdInput = midi_process_in_reader;
startup_info.hStdOutput = midi_process_out_writer;
startup_info.dwFlags = STARTF_USESTDHANDLES;
BOOL ok = CreateProcess(TEXT(module), TEXT(cmdline), NULL, NULL, TRUE,
CREATE_NEW_CONSOLE, NULL, NULL, &startup_info, &proc_info);
The only difference that I can see between the example and my code is that I don't hook up stderr - I want to keep it around for fprintf debugging.
I am probably going to need to go with some sort of "heartbeat" message in the long run, because the subprocess could become a zombie and be completely alive and keep their handles open but never read off of stdin, which would wedge things in a different way. However, I want to make sure that I'm not missing something in my understanding of anonymous pipes on Windows.
You must close midi_process_in_reader and midi_process_out_writer after calling CreateProcess(). Your issue is not because the "inherited pipes don't close on subprocess exit", it is because you forgot to close the pipes in your own process.
I am trying to implement a client-server type of communication system using the poll function in C. The flow is as follows:
Main program forks a sub-process
Child process calls the exec function to execute some_binary
Parent and child send messages to each other alternately, each message that is sent depends on the last message received.
I tried to implement this using poll, but ran into problems because the child process buffers its output, causing my poll calls to timeout. Here's my code:
int main() {
char *buffer = (char *) malloc(1000);
int n;
pid_t pid; /* pid of child process */
int rpipe[2]; /* pipe used to read from child process */
int wpipe[2]; /* pipe used to write to child process */
pipe(rpipe);
pipe(wpipe);
pid = fork();
if (pid == (pid_t) 0)
{
/* child */
dup2(wpipe[0], STDIN_FILENO);
dup2(rpipe[1], STDOUT_FILENO);
close(wpipe[0]); close(rpipe[0]);
close(wpipe[1]); close(rpipe[1]);
if (execl("./server", "./server", (char *) NULL) == -1)
{
fprintf(stderr, "exec failed\n");
return EXIT_FAILURE;
}
return EXIT_SUCCESS;
}
else
{
/* parent */
/* close the other ends */
close(wpipe[0]);
close(rpipe[1]);
/*
poll to check if write is good to go
This poll succeeds, write goes through
*/
struct pollfd pfds[1];
pfds[0].fd = wpipe[1];
pfds[0].events = POLLIN | POLLOUT;
int pres = poll(pfds, (nfds_t) 1, 1000);
if (pres > 0)
{
if (pfds[0].revents & POLLOUT)
{
printf("Writing data...\n");
write(wpipe[1], "hello\n", 6);
}
}
/*
poll to check if there's something to read.
This poll times out because the child buffers its stdout stream.
*/
pfds[0].fd = rpipe[0];
pfds[0].events = POLLIN | POLLOUT;
pres = poll(pfds, (nfds_t) 1, 1000);
if (pres > 0)
{
if (pfds[0].revents & POLLIN)
{
printf("Reading data...\n");
int n = read(rpipe[0], buffer, 1000);
buffer[n] = '\0';
printf("child says:\n%s\n", buffer);
}
}
kill(pid, SIGTERM);
return EXIT_SUCCESS;
}
}
The server code is simply:
int main() {
char *buffer = (char *) malloc(1000);
while (scanf("%s", buffer) != EOF)
{
printf("I received %s\n", buffer);
}
return 0;
}
How do I prevent poll calls from timing out because of buffering?
EDIT:
I would like the program to work even when the execed binary is external, i.e., I have no control over the code - like a unix command, e.g., cat or ls.
There seem to be two problems in your code. "stdout" is by default buffered,
so the server should flush it explicitly:
printf("I received %s\n", buffer);
fflush(stdout);
And the main program should not register for POLLOUT when trying to read
(but you may want register for POLLERR):
pfds[0].fd = rpipe[0];
pfds[0].events = POLLIN | POLLERR;
With these modifications you get the expected output:
$ ./main
Writing data...
Reading data...
child says:
I received hello
Generally, you should also check the return value of poll(), and repeat the call if
necessary (e.g. in the case of an interrupted system call or timeout).
You need, as I answered in a related answer to a previous question by you, to implement an event loop; as it name implies, it is looping, so you should code in the parent process:
while (1) { // simplistic event loop!
int status=0;
if (waitpid(pid, &status, WNOHANG) == pid)
{ // clean up, child process has ended
handle_process_end(status);
break;
};
struct pollpfd pfd[2];
memset (&pfd, 0, sizeof(pfd)); // probably useless but dont harm
pfd[0].fd = rpipe[0];
pfd[0].events = POLL_IN;
pfd[1].fd = wpipe[1];
pfd[0].event = POLL_OUT;
#define DELAY 5000 /* 5 seconds */
if (poll(pfd, 2, DELAY)>0) {
if (pfd[0].revents & POLL_IN) {
/* read something from rpipe[0]; detect end of file;
you probably need to do some buffering, because you may
e.g. read some partial line chunk written by the child, and
you could only handle full lines. */
};
if (pfd[1].revents & POLL_OUT) {
/* write something on wpipe[1] */
};
}
fflush(NULL);
} /* end while(1) */
you cannot predict in which order the pipes are readable or writable, and this can happen many times. Of course, a lot of buffering (in the parent process) is involved, I leave the details to you.... You have no influence on the buffering in the child process (some programs detect that their output is or not a terminal with isatty).
What an event polling loop like above gives you is to avoid the deadlock situation where the child process is blocked because its stdout pipe is full, while the parent is blocked writing (to the child's stdin pipe) because the pipe is full: with an event loop, you read as soon as some data is polled readable on the input pipe (i.e. the stdout of the child process), and you write some data as soon as the output pipe is polled writable (i.e. is not full). You cannot predict in advance in which order these events "output of child is readable by parent" and "input of child is writable by parent" happen.
I recommend reading Advanced Linux Programming which has several chapters explaining these issues!
BTW my simplistic event loop is a bit wrong: if the child process terminated and some data remains in its stdout pipe, its reading is not done. You could move the waitpid test after the poll
Also, don't expect that a single write (from the child process) into a pipe would trigger one single read in the parent process. In other words, there is no notion of message length. However, POSIX knows about PIPE_MAX .... See its write documentation. Probably your buffer passed to read and write should be of PIPE_MAX size.
I repeat: you need to call poll inside your event loop and very probably poll will be called several times (because your loop will be repeated many times!), and will report readable or writable pipe ends in an unpredictable (and non-reproducible) order! A first run of your program could report "rpipe[0] readable", you read 324 bytes from it, you repeat the event loop, poll says you "wpipe[1] writable", you can write 10 bytes to it, you repeat the event loop, poll tells that "rpipe[0] readable", you read 110 bytes from it, you repeat the event loop, poll tells again "rpipe[0] readable", you read 4096 bytes from it, etc etc etc... A second run of the same program in the same environment would give different events, like: poll says that "wpipe[1] writable", you write 1000 bytes to it, you repeat the loop, poll says that "rpipe[0] readable, etc....
NB: your issue is not the buffering in the child ("client") program, which we assume you cannot change. So what matters is not the buffered data in it, but the genuine input and output (that is the only thing your parent process can observe; internal child buffering is irrelevant for the parent), i.e. the data that your child program has been able to really read(2) and write(2). And if going thru a pipe(7), such data will become poll(2)-able in the parent process (and your parent process can read or write some of it after POLL_IN or POLL_OUT in the updated revents field after poll). BTW, if you did code the child, don't forget to call fflush at appropriate places inside it.
Hi I'm creating more than one process with the CreateProcess
and I need to wait all of them to finish, to analyze the results.
And I cant WaitForSingleObject because I need all of the processes running at the same time.
Since each process has a handle at Process_Information (hProcess)
I tought it was ok to use WaitForMultipleObjects,but the parent process ends without waiting for the child.
Is it ok to use WaitForMultipleObjects or there is a better way?
This is how I'm creating the processes:
#define MAX_PROCESS 3
STARTUPINFO si[MAX_PROCESS];
PROCESS_INFORMATION pi[MAX_PROCESS];
WIN32_FIND_DATA fileData;
HANDLE find;
int j=0, t=0;
ZeroMemory(&si, sizeof(si));
for (t = 0; t < MAX_PROCESS; t++)
si[t].cb = sizeof(si[0]);
ZeroMemory(&pi, sizeof(pi));
while (FindNextFile(find, &fileData) != 0)
{
// Start the child process.
if (!CreateProcess(_T("C:\\Users\\Kumppler\\Documents\\Visual Studio 2010\\Projects\ \teste3\\Debug\\teste3.exe"), // No module name (use command line)
aux2, // Command line
NULL, // Process handle not inheritable
NULL, // Thread handle not inheritable
TRUE, // Set handle inheritance to FALSE
0, // No creation flags
NULL, // Use parent's environment block
NULL, // Use parent's starting directory
&si[j], // Pointer to STARTUPINFO structure
&pi[j]) // Pointer to PROCESS_INFORMATION structure
)
{
printf("CreateProcess failed (%d).\n", GetLastError());
return;
}
j++;
//find next file related
}
FindClose(find);
WaitForMultipleObjects(MAX_PROCESS, &pi[j].hProcess, FALSE, INFINITE);
//wait and analyze results
Btw I'm trying not to use threads.
WaitForMultipleObjects expects array of handles:
HANDLE hanldes[MAX_PROCESS];
for (int i = 0; i < MAX_PROCESS; ++i)
{
handles[i] = pi[i].hProcess;
}
WaitForMultipleObjects(MAX_PROCESS, handles, TRUE, INFINITE);
Also you should know that maximum array size of handles for WaitForMultipleObjects is limited to MAXIMUM_WAIT_OBJECTS (which is 64).
If you want to wait for all the HANDLEs set 'bWaitAll' (the third parameter) to 'TRUE'.