I want to show a progress bar in my program so I add this
#include <stdio.h>
#include <unistd.h>
int main() {
puts("begin");
for (int i = 0; i < 100; i++) {
printf("%d%% finished.\r", i);
fflush(stdout);
usleep(100000);
}
puts("end");
}
when it outputs to stdout it shows nicely a line indicating the current progress.
However if I direct its output to a file, then all the lines with printf("%d%% finished.\r", i); are missing.
Is it possible to keep all the lines in a file?
You can't really use formatting like that when output is redirected to a file, because output to the file only writes forward, it never writes backwards.
If you're on a POSIX system (like Linux or macOS) then you can use isatty to detect if your writing to an actual terminal or not, and change your output formatting accordingly.
It is unlikely that the lines be missing in your output file, but it is possible that copying the file to the terminal occurs so quickly that each line overwrites the previous one without a chance for you to see it. Unlike your program, cat will not pause between the lines. Only the last line will be visible at the end, if it is at least as long as the previous ones.
Dump the file in hexadecimal with od -x to check the actual file contents.
Related
When I start a process under Linux, I can pass multiple input and output redirections. For example, if I'd like to have 14 output files, I'll do that:
command >f1 3>f2 4>f3 5>f4 6>f5 7>f6 8>f7 9>f8 10>f9 11>f10 12>f11 13>f12 14>f13 15>f14
Can my command know exactly how many such files were passed on the command line?
I know I can always add a command line option such as --count 14. I hope we don't have to.
There is no way to distinguish between redirections established on the command line and redirections which were already present in the execution environment.
For example, when utility util runs in the script
exec 4<file4
# ...
util 3<file3
it will see both fd 3 and fd 4 open for reading; it can't tell that only one was opened on the command line. If you care about that, which you probably should because there can be a number of these.
That aside, you can certainly figure out which fds are currently open, for example by cycling through all of them or examining the /proc/self/fd pseudo-directory. See this question for some example solutions to finding all the open file descriptors under Linux.
Can my command know exactly how many such files were passed on the command line?
Yes, if you order them sequentially. You can open() a dummy file (like /dev/null) and then subtract 3 from the fd returned by open to get the number of extra open files (other than stdin, stdout, and stderr) in the process. You can loop through them by starting from 3 and looping until you reach the dummy fd.
Example:
int dummy_fd = open("/dev/null", O_RDONLY);
printf("number of files open: %d\n", dummy_fd - 3);
for(int fd_ctr = 3; fd_ctr < fd; fd_ctr++)
/* ... */;
close(dummy_fd);
I know I can always add a command line option such as --count 14.
Good idea. While you're at it, how about you just pass the file names as command-line arguments, too?
Your process never sees any of those redirections. What happens is that the shell will connect up all the files to the equivalent file handles (opening them as necessary) and then run your code.
All your code can do is use the file handles that have been opened for you. Any tricks you use to find the maximum open handle inside your code will not work if you have gaps, like:
command >f1 5>f5
I suspect you'd be better off just using command line options to do this, something like:
command --output=file1 --output=file2 --output==file3
Then you have full control over how you handle those arguments in your code. For example, you can create a map of the files as you open each one.
It will also allow you to use different handles for other purposes, such as data files you need to access (I imagine you'd be rather miffed if you overwrote them with output data unrelated to the expected content simply because they were a file handle that was open).
On linux, you can just inspect the files in /proc/self/fd/. Each filename will correspond to an open fd.
Such example program:
#include <dirent.h>
#include <assert.h>
#include <errno.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <stdint.h>
int main() {
printf("Open fds:");
DIR *dir = opendir("/proc/self/fd/");
assert(dir != NULL);
struct dirent *d;
while (d = readdir(dir), d != NULL) {
if (d->d_type != DT_LNK) continue;
errno = 0;
const int i = atoi(d->d_name);
assert(errno == 0);
printf(" %d", i);
}
printf("\n");
}
When executes does this:
$ ./a.out 5>a 6>b 100>c
Open fds: 0 1 2 3 5 6 100
I have a simple C program with the read function and I don't understand the output.
//code1.c
#include <unistd.h>
#include <stdio.h>
#include <fcntl.h>
int main()
{
int r;
char c; // In C, char values are stored in 1 byte
r = read ( 0, &c, 1);
// DOC:
//ssize_t read (int filedes, void *buffer, size_t size)
//The read function reads up to size bytes from the file with descriptor filedes, storing the results in the buffer.
//The return value is the number of bytes actually read.
// Here:
// filedes is 0, which is stdin from <stdio.h>
// *buffer is &c : address in memory of char c
// size is 1 meaning it will read only 1 byte
printf ("r = %d\n", r);
return 0;
}
And here is a screenshot of the result:
I ran this program 2 times as showed above and typed "a" for the first try and "aecho hi" for the second try.
How I try to explain the results:
When read is called it sees that stdin is closed and opens it (from my point of view, why? It should just read it. I don't know why it opens it).
I type "aecho hi" in the bash and press enter.
read has priority to process stdin and reads the first byte of "aecho hi" : "a".
I get the confirmation that read has processed 1 byte with the printf.
a.out has finished and is terminated.
Somehow the remaining data in stdin is processed in bash (the father of my program) and goes to stdout which executes it and for some reason the first byte has been deleted by read.
This is all hypothetical and very blurry. Any help understanding what is happening would be very welcome.
When you type at your terminal emulator, it writes your keystrokes to a "file", in this case an in-memory buffer that, thanks to the file system, looks just like any other file that might be on disk.
Every process inherits 3 open file handles from its parent. We are interested in one of them here, standard input. The program executed by the terminal emulator (here, bash), is given as its standard input the in-memory buffer described in the first paragraph.
a.out, when run by bash, also receives this same file as its standard input. Keep this in mind: bash and a.out are reading from the same, already-opened file.
After you run a.out, its read blocks, because its standard input is empty. When you type aecho hi<enter>, the terminal writes these characters to the buffer (<enter> becoming a single linefeed character). a.out only requests one character, so it gets a and leaves the rest of the characters in the file. (Or more precisely, the file pointer is still pointing at the e after a is read.)
After a.out completes, bash tries to read from the same file. Normally, the file is empty (i.e., the file pointer is at the end of the file), so bash blocks waiting for another command. In this case, though, there is input available already: echo hi\n. bash reads this now the same as if you had typed it after a.out completed.
Check this. As alk suggests stdin and stdout are already open with the program. Now you have to understand, once you type:
aecho hi
and hit return the stdin buffer is filled with all those letters (and space) - and will continue to be as long as you don't flush it. When the program exits, the stdin buffer is still full, and your terminal automatically handles a write into stdin by echoing it to stdout - this is what you're seeing at the end - your shell reading stdin.
Now as you point out, your code "presses return" for you so to speak - in the first execution adding an empty shell line, and in the second executing echo hi. But you must remember, you pressed return, so "\n" is in the buffer! To be explicit, you in fact typed:
aecho hi\n
Once your program exits the shell reads the remaining characters in the buffer, including the return, and that's what you see!
I wrote this code in C:
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <time.h>
void random_seed(){
struct timeval tim;
gettimeofday(&tim, NULL);
double t1=tim.tv_sec+(tim.tv_usec/1000000.0);
srand (t1);
}
void main(){
FILE *f;
int i;
int size=100;
char *buf=(char*)malloc(size);
f = fopen("output.txt", "a");
setvbuf (f, buf, _IOFBF, size);
random_seed();
for(i=0; i<200; i++){
fprintf(f, "[ xx - %d - 012345678901234567890123456789 - %d]\n", rand()%10, getpid());
fflush(f);
}
fclose(f);
free(buf);
}
This code opens in append mode a file and attaches 200 times a string.
I set the buf of size 100 that can contains the full string.
Then I created multi processes running this code by using this bash script:
#!/bin/bash
gcc source.c
rm output.txt
for i in `seq 1 100`;
do
./a.out &
done
I expected that in the output the strings are never mixed up, as I read that when opening a file with O_APPEND flag the file offset will be set to the end of the file prior to each write and i'm using a fully buffered stream, but i got the first line of each process is mixed as this:
[ xx - [ xx - 7 - 012345678901234567890123456789 - 22545]
and some lines later
2 - 012345678901234567890123456789 - 22589]
It looks like the write is interrupted for calling the rand function.
So...why appear these lines?
Is the only way to prevent this the use file locks...even if i'm using only the append mode?
Thanks in advance!
You will need to implement some form of concurrency control yourself, POSIX makes no guarantees with respect to concurrent writes from multiple processes. You get some guarantees for pipes, but not for regular files written to from different processes.
Quoting POSIX write():
This volume of POSIX.1-2008 does not specify behavior of concurrent writes to a file from multiple processes. Applications should use some form of concurrency control.
(At the end of the Rationale section.)
You open the file in the fully buffered mode. That means that every line of the output first goes into the buffer and when the buffer overflows it gets flushed to the file regardless whether it contains incomplete lines. That causes chunks of output from different processes writing into the same file concurrently to be interleaved.
An easy fix would be to open the file in line buffered mode _IOLBF, so that the buffer gets flushed on each complete line. Just make sure that the buffer size is at least as big as your longest line, otherwise it will end up writing incomplete lines. The buffer is normally flushed with a single write() system call, so that lines from different processes won't interleave each other.
There is no guarantee that write() system call is atomic for different filesystems though, but it normally works as expected because write() normally locks the file descriptor in the kernel with a mutex before proceeding.
I am writing a C program on unix which should redirect it's output to the file, and write to it some text every second in infinite loop:
#include <fcntl.h>
#include <stdio.h>
#include <unistd.h>
int main(void) {
int outDes = open("./output.txt", O_APPEND | O_WRONLY);
dup2(outDes, 1);
while(1) {
printf("output text\n");
sleep(1);
}
}
But it writes nothing to the output file. I tried to change the 'while' loop for 'for' with 10 loops, and I found that it writes all 10 lines to the file at once after the series ends. It is not very good for me, while I need to have an infinite loop.
When I'm not redirecting output, it is all ok, and new line appears every second on terminal.
I also tried to put one
printf("text\n");
before redirecting output to the file. Then the program wrote the lines to the file in real time, which is good, but wrote there the first (non redirected) line too. I don't want this first line in my output file, I don't understand how it could be written into file when output was not redirected yet (maybe redirect remained there since last run?), and how it could cause that the lines are suddenly written in real time.
Can anyone explain me how does it work?
You are not checking the return value of open() and dup2(). If either open() or dup2() failed, it won't write anything in output.txt.
if (outDes < -1) {
perror("open");
return 1;
}
if (dup2(outDes, 1) == -1) {
perror("dup2");
return 1;
}
stdio streams are buffered, and the writes happen in memory before being done on the real file description.
Try adding a fflush(stdout) after printf().
You're running afoul of a poorly documented DWIMmy feature in many Unix C libraries. The first time you write to stdout or stderr, the library probes the underlying file descriptor (with isatty(3)). If it's a (pseudo-)terminal, the library puts the FILE in "line buffered" mode, meaning that it'll buffer input until a newline is written and then flush it all to the OS. But if the file descriptor is not a terminal, it puts the FILE in "fully buffered" mode, where it'll buffer something like BUFSIZ bytes of output before flushing them, and pays no attention to line breaks.
This is normally the behavior you want, but if you don't want it (as in this case), you can change it with setvbuf(3). This function (although not the behavior I described above) is ISO standard C. Here's how to use it in your case.
#include <stdio.h>
#include <unistd.h>
int
main(void)
{
if (freopen("output.txt", "a", stdout)) {
perror("freopen");
return 1;
}
if (setvbuf(stdout, 0, _IOLBF, 0)) {
perror("setvbuf");
return 1;
}
for (;;) {
puts("output text");
sleep(1);
}
/* not reached */
}
In most modern shells, you can hit the up and down arrows and it will put, at the prompt, previous commands that you have executed. My question is, how does this work?!
It seems to me that the shell is somehow manipulating stdout to overwrite what it has already written?
I notice that programs like wget do this as well. Does anybody have any idea how they do it?
It's not manipulating stdout -- it's overwriting the characters which have already been displayed by the terminal.
Try this:
#include <stdio.h>
#include <unistd.h>
static char bar[] = "======================================="
"======================================>";
int main() {
int i;
for (i = 77; i >= 0; i--) {
printf("[%s]\r", &bar[i]);
fflush(stdout);
sleep(1);
}
printf("\n");
return 0;
}
That's pretty close to wget's output, right? \r is a carriage-return, which the terminal interprets as "move the cursor back to the start of the current line".
Your shell, if it's bash, uses the GNU Readline library, which provides much more general functionality, including detecting terminal types, history management, programmable key bindings, etc.
One more thing -- when in doubt, the source for your wget, your shell, etc. are all available.
To overwrite the current standard output line (or parts of it) use \r (or \b.) The special character \r (carriage return) will return the caret to the beginning of the line, allowing you to overwrite it. The special character \b will bring the caret back one position only, allowing you to overwrite the last character, e.g.
#include <stdio.h>
#include <unistd.h>
int i;
const char progress[] = "|/-\\";
for (i = 0; i < 100; i += 10) {
printf("Processing: %3d%%\r",i); /* \r returns the caret to the line start */
fflush(stdout);
sleep(1);
}
printf("\n"); /* goes to the next line */
fflush(stdout);
printf("Processing: ");
for (i = 0; i < 100; i += 10) {
printf("%c\b", progress[(i/10)%sizeof(progress)]); /* \b goes one back */
fflush(stdout);
sleep(1);
}
printf("\n"); /* goes to the next line */
fflush(stdout);
Use fflush(stdout); because standard output is usually buffered and the information may not otherwise be immediately printed on the output or terminal
In addition to \r and \b, take a look at ncurses for some advanced control over what's on the console screen. (Including columns, moving around arbitrarily, etc).
A program running in a text terminal / console can manipulate the text displayed in its console in various ways (make text bold, move cursor, clear screen etc.). This is accomplished by printing special character sequences, called "escape sequences" (because they usually start with Escape, ASCII 27).
If stdout goes to a terminal which understands these escape sequences, the display of the terminal will change accordingly.
If you redirect stdout to a file, the escape sequences will appear in the file (which is usually not what you want).
There is no complete standard for escape sequences, but most terminals use the sequences introduced by VT100, with many extensions. This is what most terminals under Unix/Linux (xterm, rxvt, konsole) and others like PuTTY understand.
In practice, you would not directly hardcode escape sequences into your software (though you could), but use a library to print them, such as ncurses or GNU readline mentioned above. This allows compatibility with different terminal types.
It's done with the readline library... I'm not sure how it works behind the scenes but I don't think it has anything to do with stdout or streams. I suspect readline uses some sort of cryptic (to me, at least) terminal commands - that is, it cooperates with the terminal program that actually displays your shell session. I don't know that you can get readline-like behavior just by printing output.
(Think about this: stdout can be redirected to a file, but the up/down-arrow keys trick doesn't work on files.)
You can use carriage return to simulate this.
#include <stdio.h>
int main(int argc, char* argv[])
{
while(1)
{
printf("***********");
fflush(stdout);
sleep(1);
printf("\r");
printf("...........");
sleep(1);
}
return 0;
}
The program does this by printing special characters that the terminal interprets in a special way. The most simple version of this is (on most linux/unix terminals) to print '\r' (carriage return) to the normal stdout which resets the cursor position to the first character in the current line. So the thing you write next will overwrite the line you wrote previously. This can be used for simple progress indicators, for example.
int i = 0;
while (something) {
i++;
printf("\rprocessing line %i...", i);
...
}
But there are more complicated escape characters sequences that are interpreted in various ways. All kinds of things can be done with this, like positioning the cursor at a specific position on the screen or setting the text color. If or how these character sequences are interpreted depends on your terminal, but a common class supported by most terminals are ansi escape sequences. So if you want red text, try:
printf("Text in \033[1;31mred\033[0m\n");
The simplest way is to print to stdout the carriage return character ('\r').
The cursor will be moved to the start of the line, allowing you to overwrite its contents.