AllocConsole issue when manually mapping PE + fetching output - c

Question

I am working on a project that involves manually mapping and executing a PE inside of the project process. The main project whose code I am using as a basis for this can be found here: https://github.com/aaaddress1/RunPE-In-Memory/blob/master/RunPE-In-Memory/RunPEinMemory/RunPEinMemory.cpp
The project above compiles and runs fine. During the mapping of the PE that this project is used to run, it will hook the API's related to processing commandline arguments, allowing the user to manually specify arguments to the mapped PE rather than the mapped PE trying to use the arguments provided to RunPEinMemory.exe (since this is all happening inside the same process).
The project I am working towards differs from this base project in that:
I (have successfully already) hooked API's like ExitProcess and exit() and redirect them to ExitThread to prevent the mapped PE ending the RunPEinMemory.exe process
I need to eventually send the output from whatever the mapped PE is elsewhere. To do this, I am redirecting stdout/stderr to anonymous pipes that I later read from in RunPEinMemory prior to running the mapped PE.
I am struggling on the second account.
My project needs to be compiled as a GUI app (subsystem=windows) rather than a Console app (subsystem=console).
The issue arises in that in order to manipulate the stdin/stdout/stderr handles, a console needs to be attached to the process. There are numerous StackOverflow posts about this topic that cover using AllocConsole and then reopening the std handles in order to redirect output to the new console (or elsewhere). A prominent post on this matter can be seen here: Redirecting stdout in win32 does not redirect stdout
I have implemented this code and can successfully manually map/run powershell.exe (passing 'gci' as an argument for example). The stdout/stderr is redirected to anonymous pipes and then read, after which is it written out to a file (for testing purposes currently). This successful test is done with my project.exe compiled for subsystem=windows as intended.
This falls apart when I try to do the same thing with cmd.exe (passing '/c dir' as arguments). I this case, the output fails to make it to stdout or stderr, and there are no bytes to read from the anonymous pipes.
Now when I instead compile this project as a Console program (subsystem=console) and I remove the AllocConsole call (so using the console that Windows allocates for me), the program succeeds. I notice that the cmd.exe output goes to stderr, rather than stdout. But regardless, I am able to successfully redirect that output to one of the anonymous pipes, read it, and then write it out to file.
This leads me to believe that I am missing something when it comes to calling AllocConsole, some follow-on step to fully set up the environment so that certain programs can successfully send output to stdout/stderr. This post mentions a bit about cmd.exe using win32Api's to write to stdout/stderr (as opposed to the other ways to do so as mentioned in a previous link), so I'm wondering if I'm not successfully setting something up when manually allocating a console: https://stackoverflow.com/a/66689266/18776214
The relevant code is as such:
//Allocate console. This line is the one that gets commented out between console/windows test
BOOL suc = AllocConsole();
//Reopen streams after allocating console + disable buffering
FILE* fout;
FILE* ferr;
freopen_s(&fout, "CONOUT$", "r+", stdout);
freopen_s(&ferr, "CONOUT$", "r+", stderr);
setvbuf(stdout, NULL, _IONBF, 0);
setvbuf(stderr, NULL, _IONBF, 0);
//Open anonymous pipes for stdout and stderr
HANDLE hreadout;
HANDLE hwriteout;
HANDLE hreaderr;
HANDLE hwriteerr;
SECURITY_ATTRIBUTES sao = { sizeof(sao),NULL,TRUE };
SECURITY_ATTRIBUTES sae = { sizeof(sae),NULL,TRUE };
CreatePipe(&hreadout, &hwriteout, &sao, 0);
CreatePipe(&hreaderr, &hwriteerr, &sae, 0);
printf("CreatePipe last error: %d\n", GetLastError());
printf("hreadout is: %p\n", hreadout);
printf("hwriteout is: %p\n", hwriteout);
printf("hreaderr is: %p\n", hreaderr);
printf("hwriterr is: %p\n", hwriteerr);
//Set std_output_handle and std_error_handle to the write-ends of anonymous pipes
SetStdHandle(STD_OUTPUT_HANDLE, hwriteout);
SetStdHandle(STD_ERROR_HANDLE, hwriteerr);
//Convert write-ends of anonymous pipes to file descriptors and use _dup2 to set stdout/stderr to anonymous pipes
int fo = _open_osfhandle((intptr_t)(hwriteout), _O_TEXT);
int fe = _open_osfhandle((intptr_t)(hwriteerr), _O_TEXT);
int res = _dup2(fo, _fileno(fout)); //_fileno(fout)
int res2 = _dup2(fe, _fileno(ferr)); //_fileno(ferr)
printf("fo is: %d\n", fo);
printf("fe is: %d\n", fe);
printf("res is: %d\n", res);
printf("res1 is: %d\n", res2);
//Execute manually mapped PE now that stdout/stderr have been redirected
Sleep(2000);
HANDLE hThread = CreateThread(0, 0, (LPTHREAD_START_ROUTINE)retAddr, 0, 0, 0);
WaitForSingleObject(hThread, 5000);
Sleep(2000);
//Reopen streams again to set stdout/stderr back to console
freopen_s(&fout, "CONOUT$", "r+", stdout);
freopen_s(&ferr, "CONOUT$", "r+", stderr);
//check how much data there is to be read from pipe + allocate buffer
DWORD cbBytesAvailOut;
PeekNamedPipe(hreadout, NULL, NULL, NULL, &cbBytesAvailOut, NULL);
printf("PeekNamedPipe last error: %d\n", GetLastError());
printf("stdout bytes avail is: %d\n", cbBytesAvailOut);
DWORD cbBytesAvailErr;
PeekNamedPipe(hreaderr, NULL, NULL, NULL, &cbBytesAvailErr, NULL);
printf("PeekNamedPipe last error: %d\n", GetLastError());
printf("stderr bytes avail is: %d\n", cbBytesAvailErr);
//Allocate buffer based on number of bytes available to read
wchar_t* pipeBuf = calloc(cbBytesAvailErr + 2, sizeof(wchar_t));
char* convertBuf;
Sleep(2000);
//Currently only reading from a single pipe, edit this block as needed to get data when it exists
//Read from pipe
DWORD bytesRead;
printf("right before readfile!\n");
BOOL fSuccess = ReadFile(hreaderr, pipeBuf, (cbBytesAvailErr + 2) * sizeof(wchar_t), &bytesRead, NULL);
printf("fSuccess is: %d\n", fSuccess);
printf("ReadFile last error: %d\n", GetLastError());
printf("wide string: %ls\n", pipeBuf);
printf("normal string: %s\n", pipeBuf);
printf("bytesread is: %d\n", bytesRead);
//Write buffer out to disk
Sleep(2000);
char* filename = "C:\\Users\\User\\Inline-Execute-PE-main\\Inline-Execute-PE\\x64\\Release\\outfile.txt";
DWORD byteswritten;
HANDLE hFileOut = CreateFileA(filename, GENERIC_WRITE, 0, NULL, CREATE_ALWAYS, FILE_ATTRIBUTE_NORMAL, NULL);
WriteFile(hFileOut, pipeBuf, bytesRead, &byteswritten, NULL);
So the two scenarios are this:
Compile as console app with the above code with the first line (AllocConsole()) commented out, everything else the same, manually mapping cmd.exe with commandline args "cmd.exe /c dir" -> works, cmd.exe output is sent to stderr pipe and can be read
Compile as windows app with the above code, using AllocConsole(), manually mapping cmd.exe with commandline args "cmd.exe /c dir" -> fails, cmd.exe output is not captured
Again using powershell.exe works just fine. I'm thinking because maybe it uses a different means to write to stdout/stderr than cmd.exe does.
Does anyone have any ideas?
Edit: As an update, I tried putting the AllocConsole() call BEFORE the PE is mapped into memory by the RunPEinMemory project. This results in the output from cmd.exe displaying to the outputted console, however it isn't being redirected to the pipe still. That is progress at least, but it seems like the stdout/stderr from cmd.exe still isn't linked or connected to the overall process for some reason

Answer 1

I was finally able to resolve this problem.
The issue appears to be WHEN I was calling AllocConsole and redirecting the stdout/stderr.
Previously the order was:
Manually map PE
Fix IAT of PE
AllocConsole
Redirect stdout/stderr using freopen_s and SetStdHandle
Transform Win32 handles to C std handles using open_osfhandle and _dup2
CreateThread to run PE
The working order was:
AllocConsole
Redirect stdout/stderr using freopen_s and SetStdHandle
Manually map PE
Fix IAT of PE
Transform Win32 handles to C std handles using open_osfhandle and _dup2
CreateThread to run PE
So it looks like when the IAT of the mapped PE gets fixed/setup the console needs to already be there and properly set up first.