Need Suggestion while handle huge pipe data - c

Question

I'm practicing C code with pipe system call, it works well with small chunks of data. but as the data goes beyond the pipe capacity, dead lock occurs.
My test system is Debian Sid, but i believe it share the common ground with other Linux distributions. This piece of code works well while the input file '/tmp/a.out' is small enough to fit within the pipe, but blocked as the file is up to 1M.
#include <sys/errno.h>
#include <unistd.h>
#include <fcntl.h>
#include <stdlib.h>
#include <sys/wait.h>
#include <stdio.h>
#define CHUNK 2048
int main() {
int fd=open("/tmp/a.out",O_RDONLY);
int pin[2];
int pout[2];
int nread;
char buff[CHUNK];
pipe(pin);
pipe(pout);
int rc;
pid_t pid=fork();
if (pid == 0) {
close(pin[1]);
dup2(pin[0],STDIN_FILENO);
close(pout[0]);
dup2(pout[1],STDOUT_FILENO);
execlp("cat","cat",(char *)0);
} else if (pid > 0) {
close(pin[0]);
close(pout[1]);
/* I think dead lock occurs here, but i can't figure out a way to avoid it */
while ( (nread=read(fd,buff,CHUNK)) > 0) write(pin[1],buff,nread);
close(pin[1]);
while ( (nread=read(pout[0],buff,CHUNK)) >0) write(STDOUT_FILENO,buff,nread);
waitpid(pid,&rc,0);
exit(rc);
} else {
perror("fork");
exit(errno);
}
}
Any suggestions? I know Python's subprocess class have something like subprocess.communicate() to avoid this kind of dead lock, but i don't know how to deal with it in C.
Many thanks.

Answer 1

The first process pipes into cat and cat pipes back into the first process. Hence, for cat to not block on piping back, the first process must also drain that pipe. E.g.:
fcntl(pout[0], F_SETFL, fcntl(pout[0], F_GETFL) | O_NONBLOCK);
while((nread=read(fd, buff, CHUNK)) > 0) {
write(pin[1], buff, nread); // TODO: check errors and partial writes here.
while((nread=read(pout[0],buff,CHUNK)) > 0) // pout[0] must be set into non-blocking mode.
write(STDOUT_FILENO, buff, nread);
}
A more robust way is to set both pin[1] and pout[0] into non-blocking mode, use select to determine whether pin[1] is ready for write and pout[0] for read and then do write/read correspondingly and handle partial reads and writes.

Answer 2

From your suggestions at least I have 2 ways to solve this problem
1. Setting 'NON-BLOCK' mode by 'fcntl' or 'select/poll/epoll'
Use concurrency such as 'pthread' for stdin pipe
piece of code attached.
struct data {
int from_fd;
int to_fd;
};
and code for pipes should look like
pthread_t t;
struct data d;
d.from_fd=fd;
d.to_fd=pin[1];
pthread_create(&t,NULL,&fd_to_pipe,(void*) &d);
while ( (nread=read(pout[0],buff,CHUNK)) >0) write(STDOUT_FILENO,buff,nread);
waitpid(pid,&rc,0);
pthread_join(t,NULL);
Thank you !