I have a program that regularly writes to stdout. Something like this:
fprintf(stdout, ...);
fprintf(stdout, ...);
fprintf(stdout, ...);
This makes the program easy to read but I'm curious to know how efficient is it compared to concatenating strings to some char[] and then calling a single fprintf(stdout...) on that char[]. By efficiency, I'm referring to processing efficiency.
The whole of stdio.h is notoriously slow, as are writes to the screen or files in general. What makes stdio.h particularly bad, is that it's a cumbersome wrapper around the underlying OS API. printf/scanf-like functions have an horrible interface forcing them to deal with both format string parsing and variable argument lists, before they can even pass along the data to the function doing the actual work.
Minimizing those fprintf calls into a single one will almost definitely improve performance. But then that depends on how you "concatenate strings", if it is done with sprintf, then you have only moved all the calling/parsing overhead from one icky stdio.h function to another.
The only reason you would ever use stdio.h is if you need to create very portable console and file I/O code. Otherwise, you'd call the OS API directly.
That being said, you should only manually optimize code when there is a need for it. If the program runs "fast enough" without any known bottlenecks, then leave it be and strive to write as readable code as possible.
There are 3 bottlenecks i know of that can cause slow performance when you call fprintf(stdout,....):
Format parsing
Buffering
Your Terminal or other stdout device
To avoid the format parsing, you could write using fwrite(), but then you have to create the output string in a other way and if this is faster is questionable.
Normally, stdout uses a line buffer, this means that the data has to be checked for \n characters and, assuming you running on a OS, for every line a syscall is used. Syscalls are relatively slow compared to normal function calls. Setting the buffer to full buffering with setvbuf and
_IOFBF is probably the fastest buffer method. Use BUFSIZ or try different buffer sizes and benchmark them to find the best value.
When your terminal is slow, then there is nothing you can do about in your program. You could write to a file which can be faster or use a faster terminal. AFAIK Alacritty is probably the fastest terminal on Linux.
Related
We have a daemon that contains a lot of print messages. Since we are working on an embedded device with a weak CPU and other constraint hardware, we want to minimize any kinds of costs (IO, CPU, etc..) of printf messages in our final version. (Users don't have a console)
My teammate and I have a disagreement. He thinks we can just redirect everything to /dev/null. It won't cost any IO so affections will be minimal. But I think it will still cost CPU and we better define a macro for printf so we can rewrite "printf" (maybe just return).
So I need some opinions about who is right. Will Linux be smart enough to optimize printf? I really doubt it.
Pretty much.
When you redirect the stdout of the program to /dev/null, any call to printf(3) will still evaluate all the arguments, and the string formatting process will still take place before calling write(2), which writes the full formatted string to the standard output of the process. It's at the kernel level that the data isn't written to disk, but discarded by the handler associated with the special device /dev/null.
So at the very best, you won't bypass or evade the overhead of evaluating the arguments and passing them to printf, the string formatting job behind printf, and at least one system call to actually write the data, just by redirecting stdout to /dev/null. Well, that's a true difference on Linux. The implementation just returns the number of bytes you wanted to write (specified by the 3rd argument of your call to write(2)) and ignores everything else (see this answer). Depending on the amount of data you're writing, and the speed of the target device (disk or terminal), the difference in performance may vary a lot. On embedded systems, generally speaking, cutting off the disk write by redirecting to /dev/null can save quite some system resources for a non-trivial amount of written data.
Although in theory, the program could detect /dev/null and perform some optimizations within the restrictions of standards they comply to (ISO C and POSIX), based on general understanding of common implementations, they practically don't (i.e. I am unaware of any Unix or Linux system doing so).
The POSIX standard mandates writing to the standard output for any call to printf(3), so it's not standard-conforming to suppress the call to write(2) depending on the associated file descriptors. For more details about POSIX requirements, you can read Damon's answer. Oh, and a quick note: All Linux distros are practically POSIX-compliant, despite not being certified to be so.
Be aware that if you replace printf completely, some side effects may go wrong, for example printf("%d%n", a++, &b). If you really need to suppress the output depending on the program execution environment, consider setting a global flag and wrap up printf to check the flag before printing — it isn't going to slow down the program to an extent where the performance loss is visible, as a single condition check is much faster than calling printf and doing all the string formatting.
The printf function will write to stdout. It is not conforming to optimize for /dev/null.
Therefore, you will have the overhead of parsing the format string and evaluating any necessary arguments, and you will have at least one syscall, plus you will copy a buffer to kernel address space (which, compared to the cost of the syscall is neglegible).
This answer is based on the specific documentation of POSIX.
System Interfaces
dprintf, fprintf, printf, snprintf, sprintf - print formatted output
The fprintf() function shall place output on the named output stream. The printf() function shall place output on the standard output stream stdout. The sprintf() function shall place output followed by the null byte, '\0', in consecutive bytes starting at *s; it is the user's responsibility to ensure that enough space is available.
Base Definitions
shall
For an implementation that conforms to POSIX.1-2017, describes a feature or behavior that is mandatory. An application can rely on the existence of the feature or behavior.
The printf function writes to stdout. If the file descriptor connected to stdout is redirected to /dev/null then no output will be written anywhere (but it will still be written), but the call to printf itself and the formatting it does will still happen.
Write your own that wraps printf() using the printf() source as a guideline, and returning immediately if a noprint flag is set. The downside of this is when actually printing it will consume more resources because of having to parse the format string twice. But it uses negligible resources when not printing. Can't simply replace printf() because the underlying calls inside printf() can change with a newer version of the stdio library.
void printf2(const char *formatstring, ...);
Generally speaking, an implementation is permitted to perform such optimisations if they do not affect the observable (functional) outputs of the program. In the case of printf(), that would mean that if the program doesn't use the return value, and if there are no %n conversions, then the implementation would be allowed to do nothing.
In practice, I'm not aware of any implementation on Linux that currently (early 2019) performs such an optimisation - the compilers and libraries I'm familiar with will format the output and write the result to the null device, relying on the kernel' to ignore it.
You may want to write a forwarding function of your own if you really need to save the cost of formatting when the output is not used - you'll want to it to return void, and you should check the format string for %n. (You could use snprintf with a NULL and 0 buffer if you need those side-effects, but the savings are unlikely to repay the effort invested).
in C writing 0; does execute and nothing, which is similar to ;.
means you can write a macro like
#if DEBUG
#define devlognix(frmt,...) fprintf(stderr,(frmt).UTF8String,##__VA_ARGS__)
#else
#define nadanix 0
#define devlognix(frmt,...) nadanix
#endif
#define XYZKitLogError(frmt, ...) devlognix(frmt)
where XYZKitLogError would be your Log command.
or even
#define nadanix ;
which will kick out all log calls at compile time and replace with 0; or ; so it gets parsed out.
you will get Unused variable warnings, but it does what you want and this side effect can even be helpful because it tells you about computation that is not needed in your release.
.UTF8String is an Objective-C method converting NSStrings to const char* - you don't need.
After going through most of the book, "The C Programming Language," I think I have a decent grasp on programming in C. One common C idiom presented in that book is reading a file a single byte at a time, using functions like getchar() and fgetc(). So far, I've been using these functions to do all IO in my C programs.
My question is, is this an efficient way of reading a file? Does a call to get a single byte require a lot of overhead that can be minimized if I read multiple bytes into a buffer at a time, for instance by using the read() system call on Unix systems? Or do the operating system and C library handle a buffer behind the scenes to make it more efficient? Also, does this work the same way for writing to files a single byte at a time?
I would like to know how this generally works in C, but if it is implementation or OS specific, I would like to know how it works in GCC on common Unix-like systems (like macOS and linux).
Using getchar() etc is efficient because the standard I/O library uses buffering to read many bytes at once (saving them in a buffer) and doles them out one at a time when you call getchar().
Using read() to read a single byte at a time is much slower, typically, because it makes a full system call each time. It still isn't catastrophically slow, but it is nowhere near as fast as reading 512, or 4096, bytes into a buffer.
Those are broad, sweeping statements. There are many caveats that could be added, but they are a reasonable general outline of the performance of getchar(), etc.
I'm writing a fairly basic program for personal use but I really want to make sure I use good practices, especially if I decide to make it more robust later on or something.
For all intents and purposes, the program accepts some input files as arguments, opens them using fopen() read from the files, does stuff with that information, and then saves the output as a few different files in a subfolder. eg, if the program is in ~/program then the output files are saved in ~/program/csv/
I just output directly to the files, for example output = fopen("./csv/output.csv", "w");, print to it with fprintf(output,"%f,%f", data1, data2); in a loop, and then close with fclose(output); and I just feel like that is bad practice.
Should I be saving it in a temp directory wile it's being written to and then moving it when it's finished? Should I be using more advanced file i/o libraries? Am I just completely overthinking this?
Best practices in my eyes:
Check every call to fopen, printf, puts, fprintf, fclose etc. for errors
use getchar if you must, fread if you can
use putchar if you must, fwrite if you can
avoid arbitrary limits on input line length (might require malloc/realloc)
know when you need to open output files in binary mode
use Standard C, forget conio.h :-)
newlines belong at the end of a line, not at the beginning of some text, i.e. it is printf ("hello, world\n"); and not "\nHello, world" like those mislead by the Mighty William H. often write to cope with the sillyness of their command shell. Outputting newlines first breaks line buffered I/O.
if you need more than 7bit ASCII, chose Unicode (the most common encoding is UTF-8 which is ASCII compatible). It's the last encoding you'll ever need to learn. Stay away from codepages and ISO-8859-*.
Am I just completely overthinking this?
You are. If the task's simple, don't make a complicated solution on purpose just because it feels "more professional". While you're a beginner, focus on code readability, it will facilitate your and others' lives.
It's fine. I/O is fully buffered by default with stdio file functions, so you won't be writing to the file with every single call of fprintf. In fact, in many cases, nothing will be written to it until you call fclose.
It's good practice to check the return of fopen, to close your files when finished, etc. Let the OS and the compiler do their job in making the rest efficient, for simple programs like this.
If no other program is checking for the presence of ~/program/csv/output.csv for further processing, then what you're doing is just fine.
Otherwise you can consider writing to a FILE * obtained by a call to tmpfile in stdio.h or some similar library call, and when finished copy the file to the final destination. You could also put down a lock file output.csv.lck and remove that when you're done, but that depends on you being able to modify the other program's behaviour.
You can make your own cat, cp, mv programs for practice.
I'm writing a little parser and I would like to know the advantages and disadvantages of the different ways to load the data to be parsed. The two ways that I thought of are:
Load the file's contents into a string then parse the string (access the character at an array position)
Parse as reading the file stream (fgetc)
The former will allow me to have two functions: one for parse_from_file and parse_from_string, however I believe this mode will take up more memory. The latter will not have that disadvantage of using more memory.
Does anyone have any advice on the matter?
Reading the entire file in or memory mapping it will be faster, but may cause issues if you want your language to be able to #include other files as these would be memory mapped or read into memory as well.
The stdio functions would work well because they usually try to buffer up data for you, but they are also general purpose so they also try to look out for usage patterns which differ from reading a file from start to finish, but that shouldn't be too much overhead.
A good balance is to have a large circular buffer (x * 2 * 4096 is a good size) which you load with file data and then have your tokenizer read from. Whenever a block's worth of data has been passed to your tokenizer (and you know that it is not going to be pushed back) you can refill that block with new data from the file and update some buffer location info.
Another thing to consider is if there is any chance that the tokenizer would ever need to be able to be used to read from a pipe or from a person typing directly in some text. In these cases your reads may return less data than you asked for without it being at the end of the file, and the buffering method I mentioned above gets more complicated. The stdio buffering is good for this as it can easily be switched to/from line or block buffering (or no buffering).
Using gnu fast lex (flex, but not the Adobe Flash thing) or similar can greatly ease the trouble with all of this. You should look into using it to generate the C code for your tokenizer (lexical analysis).
Whatever you do you should try to make it so that your code can easily be changed to use a different form of next character peek and consume functions so that if you change your mind you won't have to start over.
Consider using lex (and perhaps yacc, if the language of your grammar matches its capabilities). Lex will handle all the fiddly details of lexical analysis for you and produce efficient code. You can probably beat its memory footprint by a few bytes, but how much effort do you want to expend into that?
The most efficient on a POSIX system would probably neither of the two (or a variant of the first if you like): just map the file read-only with mmap, and parse it then. Modern systems are quite efficient with that in that they prefetch data when they detect a streaming access etc., multiple instances of your program that parse the same file will get the same physical pages of memory etc. And the interfaces are relatively simple to handle, I think.
I am looking for various ways of reading/writing data from stdin/stdout. Currently I know about scanf/printf, getchar/putchar and gets/puts. Are there any other ways of doing this? Also I am interesting in knowing that which one is most efficient in terms of Memory and Space.
Thanks in Advance
fgets()
fputs()
read()
write()
And others, details can be found here: http://www.cplusplus.com/reference/clibrary/cstdio/
As per your time question take a look at this: http://en.wikipedia.org/wiki/I/O_bound
Stdio is designed to be fairly efficient no matter which way you prefer to read data. If you need to do character-by-character reads and writes, they usually expand to macros which just access the buffer except when it's full/empty. For line-by-line text io, use puts/fputs and fgets. (But NEVER use gets because there's no way to control how many bytes it will read!) The printf family (e.g. fprintf) is of course extremely useful for text because it allows you to skip constructing a temporary buffer in memory before writing (and thus lets you avoid thinking about all the memory allocation, overflow, etc. issues). fscanf tends to be much less useful, but mostly because it's difficult to use. If you study the documentation for fscanf well and learn how to use %[, %n, and the numeric specifiers, it can be very powerful!
For large blocks of text (e.g. loading a whole file into memory) or binary data, you can also use the fread and fwrite functions. You should always pass 1 for the size argument and the number of bytes to read/write for the count argument; otherwise it's impossible to tell from the return value how much was successfully read or written.
If you're on a reasonably POSIX-like system (pretty much anything) you can also use the lower-level io functions open, read, write, etc. These are NOT part of the C standard but part of POSIX, and non-POSIX systems usually provide the same functions but possibly with slightly-different behavior (for example, file descriptors may not be numbered sequentially 0,1,2,... like POSIX would require).
If you're looking for immediate-mode type stuff don't forget about Curses (more applicable on the *NIX side but also available on Windows)