I'm trying to output ordered set of lines created by multiple processes.
I found that printf() and fprintf() is not suitable for such task. Right now I'm using this set of commands:
sprintf(buff,"%d: some string", (*counter)++); //build string (write can't do that)
write(file, buff, strlen(buff));
memset(buff, 0, strlen(buff)); //clear buffer before next writing
File opening and starting processes is shown below:
int file;
int main(){
pid_t busPID, riderPID;
file = open("outputFile.txt", O_WRONLY | O_CREAT | O_APPEND | O_TRUNC, 0666);
if((busPID = fork()) == 0){
bus();
else{
if((riderPID = fork()) == 0){
for(int i = 0; i < 10; i++){
rider();
}
}else{
pid_t waitPID;
while ((waitPid = wait(&status)) > 0); //wait for all riders to finish
}
}
waitpid(busPID, NULL, 0);
return 0;
}
Here are functions, which prints output:
void bus() {
char buff[50];
//writing
do {
//writing
if(*onStop > 0) {
//writing
sem_post(semRider);
sem_wait(semBus);
//writing
*onStop = 0; //internal value, irrelevant for answer
}
//writing
usleep(rand()%busSleep);
//writing
sem_post(semRider);
sem_wait(semBus);
departuredAkt += temp; //internal value, irrelevant for answer
} while(departuredAkt < departuredTotal);
//writing
exit(EXIT_SUCCESS); //exit this process
}
void rider() {
char buff[50];
//writing
int pos = ++(*onStop);
//writing
sem_wait(semRider);
//writing
sem_post(semBus);
sem_wait(semRider);
//writing
sem_post(semBus);
exit(EXIT_SUCCESS);
}
There is only 1 process using bus() function and N processes using rider()function (specified by argument). desired output is:
1: bus string
2: bus string
3: bus string
4: rider 1 string
5: rider 1 string
.
.
.
25: rider N string
26: bus string
My current output looks like this:
1: bus string
2: bus string
3: bus string
4: rider 1 string
6: bus string //here is the problem
5: rider 1 string
Question is, how can I achieve printing lines in correct order?
First of all, side note: never use sprintf, this function is completely unsecure. Use snprintf. Example:
snprintf (buff, sizeof (buff), "%d: some string", (*counter)++);
Second: you missed information we need to understand your question. I mean the following information:
How exactly you opened file?
How you started you processes?
Are this processes really processes or they are threads?
Did you open file in one process and somehow shared this opened file with other processes or did you opened the file in each process separately?
This details are critical for understanding your question.
Next time you will writing some question, please, provide FULL EXAMPLE. I. e. minimal working example we can compile and run. It should include all relevant details, i. e. starting processes, opening files, etc. And, of course, you should strip all unnecessary details.
Okey, there is two different notion in POSIX: "file descripTOR" and "file descripTION". Websearch for them. Type "man 2 open" in UNIX shell and read carefully, this manual page talks about distinction.
Exact details about how you started your processes and how you opened your file causes (or not causes) sharing of file description between processes and thus affects behavior of "write".
I wrote big text about file descriptors and descriptions. I put it here: https://zerobin.net/?eb2d99ee02f36b92#hQY7vTMCD9ekThAod+bmjlJgnlBxyDSXCYcgmjVSu2w= , because it is not much relevant to this question, but still will be useful for education.
Okey, what to do?
Well, if you for whatever reason cannot share ONE file DESCRIPTION, then simply open file with O_APPEND. :) You don't need to open the file every time you write to it. Simply open it with O_APPEND one time in each process and all will be OK.
Related
I am trying to understand how the seek pointer mechanism works when a parent process opens a new file and then creating a new child process using fork().
Assume I have the following code :
int main(){
int fd = open("myFile.txt", O_RDWR | O_CREAT)
if(fork() == 0){
write(fd,"stack overflow", 16);
close(fd);
}
else{
wait(NULL);
char buff[17];
read(fd, &buff, 16);
printf("%s", buff);
}
}
I get nothing printing to stdout, but I don't really understand why would it behave that way. Wouldn't it be smarter if there were a different seek pointer for read and write so running this code will result in printing "stack overflow" to stdout?
Apparently not, Would like to get an explanation
Both processes are using the same file description (i.e. the same open file).
They are accessing the same file pointer because it's part of the same file description. When one of them writes some characters, the file pointer advances for all of them, because it's the same file pointer.
You can get the file pointer by calling lseek(fd, 0, SEEK_CUR). If you print the file pointer before starting the child program, and after waiting for it, you should see that it has changed.
I am writing some code that has to communicate with gdb mi. So I forked my programm, put two pipes in place and started gdb mi in the child, so that I can communicate with gdb mi from the parent.
gdb always returns "(gdb) \n" when it is finished, so I look for that and write my next command. This is a minimum example of my code:
int main(){
printf("Starting Test\n");
int fromGDB[2], toGDB[2], nbytes;
pid_t childpid;
char readbuffer[80] = "";
pipe(fromGDB);
pipe(toGDB);
if((childpid = fork())==-1)
{
perror("fork");
exit(1);
}
if(childpid == 0){
close(toGDB[1]);
close(fromGDB[0]);
int backup = dup(1); //store stdout
if (dup2(fromGDB[1],1) < 0){puts("hat nicht geklappt");}
int backupStdin = dup(0); //store stdin
if (dup2(toGDB[0],0) < 0){puts("hat nicht geklappt");}
system("gdb -q --interpreter=mi2"); // start gdb mi
dup2(backup,1); // restore stdout
puts("child fertig");
exit(0);
}else{
close(toGDB[0]);
close(fromGDB[1]);
char* writeCommand = "";
int commandCounter = 0;
while (commandCounter <3){
nbytes = read(fromGDB[0],readbuffer,sizeof(readbuffer));
printf("parent recived: %s", readbuffer);
if (strncmp(readbuffer+strlen(readbuffer)-strlen("(gdb) \n"),"(gdb) \n", strlen("(gdb) \n")) == 0){
switch (commandCounter){
case 0: writeCommand = "-file-exec-and-symbols /home/dev/spielwiese/build/main\n"; break;
case 1: writeCommand = "-gdb-exit\n"; break;
default: writeCommand = "you should never reach here\n"; break;
}
write(toGDB[1],writeCommand,strlen(writeCommand)+1);
printf("wrote: %s", writeCommand);
commandCounter++;
}else if(strncmp(readbuffer,"^exit", sizeof("^exit")-1) == 0){
break;
}
memset(readbuffer,'\0',strlen(readbuffer)); //clear the readbuffer
}
puts("parent fertig");
sleep(5);
}
return 0;
}
If I call the same commands by hand, thats the output I get (-> means input from me)
-> gdb -q --interpreter=mi2
=thread-group-added,id="i1"
(gdb)
-> -file-exec-and-symbols /home/dev/spielwiese/build/main
^done
(gdb)
-> -gdb-exit
^exit
But if I run my code, which should be essentialy the same, I get this output:
Starting Test
parent recived: =thread-group-added,id="i1"
parent recived: (gdb)
wrote: -file-exec-and-symbols /home/dev/spielwiese/build/main
parent recived: ^done
(gdb)
wrote: -gdb-exit
parent recived: &"\n"
parent recived: ^done
(gdb)
wrote: you should never reach here
parent fertig
According to the gdb mi manual, an & preceeds a log entry, but this log entry is empty, exept for the newline. Also, I don't know why there should be a log entry for exiting, or why it fails to exit, but doesen't produce an error.
Also, if you know any better sources for gdb mi than this: https://sourceware.org/gdb/current/onlinedocs/gdb/GDB_002fMI.html#GDB_002fMI , please let me know.
There are several issues with your example code, many of which have already been summarized in comments:
read() and write() transfer only the bytes specified, or a leading subsequence of those. They do not append null bytes to the transfer.
read() and write() are not guaranteed to transfer the full number of bytes requested on any given call.
your buffer might not be long enough to accommodate some responses from the child all in a single read().
According to the gdb mi manual, an & preceeds a log entry, but this log entry is empty, exept for the newline. Also, I don't know why there should be a log entry for exiting, or why it fails to exit, but doesen't produce an error.
None of the above explain the log entry or other behavior differences, but this probably does:
write(toGDB[1],writeCommand,strlen(writeCommand)+1);
Supposing that the full number of bytes requested is transferred, you are writing not only the command, but also a string terminator. The terminator is not part of the MI protocol, so your program is not behaving the same as your interactive session. Moreover, the particular error -- an extra null byte -- is one that is especially likely to produce mysterious output. I can only speculate about the specifics, but I could believe that gdb is treating the extra null byte immediately following a newline as if it terminated a zero-byte command. If so, then you are effectively issuing three commands to gdb, not two, and the gdb log output is about the null byte / empty command.
You would probably find stream I/O to be more convenient for your purposes than raw I/O. That would insulate you from many of the idiosyncrasies of raw I/O, and it would make fgets() available to you for input, which would be to your advantage. The fdopen() function can wrap streams around your file descriptors:
#define BUFFER_SIZE 1024
// Parent
FILE *fFromGDB = fdopen(fromGDB[0], "r");
FILE *fToGDB = fdopen(fToGDB[1], "w");
if (!fFromGDB || ! fToGDB) // ...
// You probably want fToGDB to be line-buffered, not block buffered:
setvbuf(fToGDB, NULL, _IOLBF, BUFFER_SIZE);
Then use fputs(), fgets(), fprintf(), etc. to interact with the child.
Also, if you know any better sources for gdb mi than this: https://sourceware.org/gdb/current/onlinedocs/gdb/GDB_002fMI.html#GDB_002fMI , please let me know.
Requests for external resources are off-topic here.
In any event, you are referencing the appropriate manual, and short of analyzing the source code, that is the ultimate source of knowledge on the matter.
(correct me if im wrong on my terms) So i need to read from a file descriptor, but the read method takes in a int for byte size to read that much OR i can use O_NONBLOCK, but i still have to setup up a buffer size of an unknown size. making it difficult. heres what i have so far
this is my method that handles all the polling and mkfifo. and N is already predefined in main
struct pollfd pfd[N];
int i;
for(i = 0; i < N; i++)
{
char fileName[32];
snprintf (fileName, sizeof(fileName), "%d_%di", pid, i);
mkfifo(fileName, 0666);
pfd[i].fd = open(fileName, O_RDONLY | O_NDELAY);
pfd[i].events = POLLIN;
pfd[i].revents = 0;
snprintf (fileName, sizeof(fileName), "%d_%do", pid, i);
mkfifo(fileName, 0666);
i++;
pfd[i].fd = open(fileName, O_WRONLY | O_NDELAY);
pfd[i].events = POLLOUT;
pfd[i].revents = 0;
i--;
}
while(1)
{
int len, n;
n = poll(pfd, N, 2000);
if( n < 0 )
{
printf("ERROR on poll");
continue;
}
if(n == 0)
{
printf("waiting....\n");
continue;
}
for(i = 0; i < N; i++)
{
char buff[1024]; <---i dont want to do this
if (pfd[i].revents & POLLIN)
{
printf("Processing input....\n");
read(pfd[i].fd, buff, O_NONBLOCK);
readBattlefield(buff);
print_battleField_stats();
pfd[i].fd = 0;
}
}
}
i also read somewhere that once read() reads all the data coming, it empties the pipe, meaning i can use the same again for another incoming data. but it doesnt empty the pipe because i cant use the same pipe again. I asked my professor but all he says was to use something like scanf, but how do use scanf if scanf takes a FILE stream, and the poll.fd is an int? essentially my ultimate question is, how to read the incoming data through the file descriptor using scan or of other sort? using scan will help me more with handling the data.
EDIT:
in another terminal i have to put cat file > (named_file)
and my main program will read the input data. heres what the input data looks like
3 3
1 2 0
0 2 0
3 0 0
first 2 numbers are grid information and player number, and after that is grid, but this a simplified version, ill be dealing with sizes over 100's of players and grids of over 1000's
char buff[1024]; <---i dont want to do this
What would you like to do then? This is how it works. This is not how it works:
read(pfd[i].fd, buff, O_NONBLOCK);
This will compile because O_NONBLOCK is an integer #define, but it is absolutely and unequivocally incorrect. The third argument to read() is a number of bytes to read. Not a flag. Period. It may be zero, but what you've done here is pass an arbitrary number -- whatever the value of O_NONBLOCK is, which could easily be more than 1024, the size of your buffer. This does not set the read non-block. recv() is similar to read() and does take such flags as a forth argument, but you can't use that with a file descriptor. If you want to set non-block on a file descriptor, you must do it with open() or fcntl().
how to read the incoming data through the file descriptor using scan or of other sort?
You can create a FILE* stream from an open descriptor with fdopen().
i also read somewhere that once read() reads all the data coming, it empties the pipe, meaning i can use the same again for another incoming data. but it doesnt empty the pipe because i cant use the same pipe again.
Once you reach EOF (because the writer closed the connection), read() will return 0, and continue to return 0 immediately until someone opens the pipe again.
If you set the descriptor non-block, read() will always return immediately; if there is someone connected and nothing to read, it will return -1 but errno will == EAGAIN. See man 2 read.
man fifo is definitely something you should read; if there's anything you aren't sure about, ask a specific question based on that.
And don't forget: Fix that read() call. It's wrong. W R O N G. Your prof/TA/whoever will not miss that.
So I have a mytee program (with much much less functionality). Trying to learn how to work with pipes / children / etc
(1) I do pipe
(2) Create the file(s)
(3) fork
(4) the parent does scanf to get the text
(5) sends the text to the pipe
(6) child receives it and writes it to files
-> #4 should be a loop until the user writes '.'
-> #6 should continue writing new lines, but somewhere there is a breakdown.
Some of the things that I think it might be:
1. Something is wrong with my permissions (but O_APPEND is there, and not sure what else I would need)
2. there may be a problem in parent do while loop, where it should send the msg to the pipe (fd[1])
3. #6 where I strongly think my problem lies. After the initial write it doesn, continue writing. I am not sure if I need to somehow keep track of the size of bytes already written, but if that was the case I would expect the last message to be there not the first.
I'm pretty much at a loss right now
I run it using
./mytee test1
Code:
ret = pipe (fd);
if (ret == -1)
{
perror ("pipe");
return 1;
}
for (i=0;i<argc-1;i++) {
if ((filefd[i] = open(argv[i+1], O_CREAT|O_TRUNC|O_WRONLY|O_APPEND, 0644)) < 0) {
perror(argv[i]); /* open failed */
return 1;
}
}
pid = fork();
if (pid==0) /* child */
{
int read_data;
do {
read_data = read(fd[0], buffer, sizeof(buffer));
for(i=0;i<argc;i++) {
write(filefd[i], buffer, read_data);
}
} while (read_data > 1);
for (i=0; i<argc; i++)
close(filefd[i]);
return 0;
}
else { /* parent */
char msg[20];
do{
scanf("%s",msg);
write(fd[1],msg,sizeof(msg));
}while (strcmp(msg,".")!=0);
while ((pid = wait(&status)) != -1)
fprintf(stderr, "process %d exits with %d\n", pid, WEXITSTATUS(status));
return 0;
}
Adding Output:
$ ./a.out test1
qwe
asd
zxc
.
^C
It doesn't exit properly. I think the child is stuck in the loop
And the contents of test1:
qwe
Working through this with the OP, reportedly the problem was unconditionally writing all 20 bytes of msg instead of just the NUL-terminated string contained within it. Suggested minimal fix: change
scanf("%s",msg);
write(fd[1],msg,sizeof(msg));
to
scanf("%19s",msg);
write(fd[1],msg,strlen(msg));
I see a couple of issues, which could potentially cause that behaviour.
Firstly, your loop condition doesn't look right. Currently it will terminate if a single byte is read. Change it to this:
while (read_data > 0);
The other issue I see is that you're writing to more files than you opened. Make sure you loop to argc-1, not argc:
for (i=0; i<argc-1; i++)
I can't think of any way to implement pipelining in c that would actually work. That's why I've decided to write in here. I have to say, that I understand how do pipe/fork/mkfifo work. I've seen plenty examples of implementing 2-3 pipelines. It's easy. My problem starts, when I've got to implement shell, and pipelines count is unknown.
What I've got now:
eg.
ls -al | tr a-z A-Z | tr A-Z a-z | tr a-z A-Z
I transform such line into something like that:
array[0] = {"ls", "-al", NULL"}
array[1] = {"tr", "a-z", "A-Z", NULL"}
array[2] = {"tr", "A-Z", "a-z", NULL"}
array[3] = {"tr", "a-z", "A-Z", NULL"}
So I can use
execvp(array[0],array)
later on.
Untli now, I believe everything is OK. Problem starts, when I'm trying to redirect those functions input/output to eachother.
Here's how I'm doing that:
mkfifo("queue", 0777);
for (i = 0; i<= pipelines_count; i++) // eg. if there's 3 pipelines, there's 4 functions to execvp
{
int b = fork();
if (b == 0) // child
{
int c = fork();
if (c == 0)
// baby (younger than child)
// I use c process, to unblock desc_read and desc_writ for b process only
// nothing executes in here
{
if (i == 0) // 1st pipeline
{
int desc_read = open("queue", O_RDONLY);
// dup2 here, so after closing there's still something that can read from
// from desc_read
dup2(desc_read, 0);
close(desc_read);
}
if (i == pipelines_count) // last pipeline
{
int desc_write = open("queue", O_WRONLY);
dup2(desc_write, 0);
close(desc_write);
}
if (i > 0 && i < pipelines_count) // pipeline somewhere inside
{
int desc_read = open("queue", O_RDONLY);
int desc_write = open("queue", O_WRONLY);
dup2(desc_write, 1);
dup2(desc_read, 0);
close(desc_write);
close(desc_read);
}
exit(0); // closing every connection between process c and pipeline
}
else
// b process here
// in b process, i execvp commands
{
if (i == 0) // 1st pipeline (changing stdout only)
{
int desc_write = open("queue", O_WRONLY);
dup2(desc_write, 1); // changing stdout -> pdesc[1]
close(desc_write);
}
if (i == pipelines_count) // last pipeline (changing stdin only)
{
int desc_read = open("queue", O_RDONLY);
dup2(desc_read, 0); // changing stdin -> pdesc[0]
close(desc_read);
}
if (i > 0 && i < pipelines_count) // pipeline somewhere inside
{
int desc_write = open("queue", O_WRONLY);
dup2(desc_write, 1); // changing stdout -> pdesc[1]
int desc_read = open("queue", O_RDONLY);
dup2(desc_read, 0); // changing stdin -> pdesc[0]
close(desc_write);
close(desc_read);
}
wait(NULL); // it wait's until, process c is death
execvp(array[0],array);
}
}
else // parent (waits for 1 sub command to be finished)
{
wait(NULL);
}
}
Thanks.
Patryk, why are you using a fifo, and moreover the same fifo for each stage of the pipeline?
It seems to me that you need a pipe between each stage. So the flow would be something like:
Shell ls tr tr
----- ---- ---- ----
pipe(fds);
fork();
close(fds[0]); close(fds[1]);
dup2(fds[0],0);
pipe(fds);
fork();
close(fds[0]); close(fds[1]);
dup2(fds[1],1); dup2(fds[0],0);
exex(...); pipe(fds);
fork();
close(fds[0]); etc
dup2(fds[1],1);
exex(...);
The sequence that runs in each forked shell (close, dup2, pipe etc) would seem like a function (taking the name and parameters of the desired process). Note that up until the exec call in each, a forked copy of the shell is running.
Edit:
Patryk:
Also, is my thinking correct? Shall it work like that? (pseudocode):
start_fork(ls) -> end_fork(ls) -> start_fork(tr) -> end_fork(tr) ->
start_fork(tr) -> end_fork(tr)
I'm not sure what you mean by start_fork and end_fork. Are you implying that ls runs to completion before tr starts? This isn't really what is meant by the diagram above. Your shell will not wait for ls to complete before starting tr. It starts all of the processes in the pipe in sequence, setting up stdin and stdout for each one so that the processes are linked together, stdout of ls to stdin of tr; stdout of tr to stdin of the next tr. That is what the dup2 calls are doing.
The order in which the processes run is determined by the operating system (the scheduler), but clearly if tr runs and reads from an empty stdin it has to wait (to block) until the preceding process writes something to the pipe. It is quite possible that ls might run to completion before tr even reads from its stdin, but it is equally possible that it wont. For example if the first command in the chain was something that ran continually and produced output along the way, the second in the pipeline will get scheduled from time to time to prcess whatever the first sends along the pipe.
Hope that clarifies things a little :-)
It might be worth using libpipeline. It takes care of all the effort on your part and you can even include functions in your pipeline.
The problem is you're trying to do everything at once. Break it into smaller steps instead.
1) Parse your input to get ls -al | out of it.
1a) From this you know you need to create a pipe, move it to stdout, and start ls -al. Then move the pipe to stdin. There's more coming of course, but you don't worry about it in code yet.
2) Parse the next segment to get tr a-z A-Z |. Go back to step 1a as long as your next-to-spawn command's output is being piped somewhere.
Implementing pipelining in C. What would be the best way to do that?
This question is a bit old, but here's an answer that was never provided. Use libpipeline. libpipeline is a pipeline manipulation library. The use case is one of the man page maintainers who had to frequently use a command like the following (and work around associated OS bugs):
zsoelim < input-file | tbl | nroff -mandoc -Tutf8
Here's the libpipeline way:
pipeline *p;
int status;
p = pipeline_new ();
pipeline_want_infile (p, "input-file");
pipeline_command_args (p, "zsoelim", NULL);
pipeline_command_args (p, "tbl", NULL);
pipeline_command_args (p, "nroff", "-mandoc", "-Tutf8", NULL);
status = pipeline_run (p);
The libpipeline homepage has more examples. The library is also included in many distros, including Arch, Debian, Fedora, Linux from Scratch and Ubuntu.