#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include<stdio.h>
int main(){
int fd;
char bf[4096];
int buf_size=4096;
fd = open("/proc/18022/cmdline", O_RDONLY);
int bytes = read(fd, bf, buf_size-1);
printf("%d %s\n\n",bytes,bf);
close(fd);
}
Above code is always reading only 3072 bytes while cmdline have more characters than 3072.
If i copy content of cmdline to gedit and then run the above code on this newly created file then it is reading all bytes of the file.
I googled it and found that it reads bytes up to SSIZE_MAX but my doubt is why it is reading all the bytes in second case.
You should not rely on reading a whole file from the first try, even if you know you've allocated enough space for the read. Instead you should read in chunks and process the bytes chunk-by-chunk:
char buff[4096];
while((cnt = read(fd, bf, buf_size-1)) > 0) {
// process the bytes just read, or append them to
// a larger buffer
}
Quoting from the man page for read():
It is not an error if this number is smaller than the number of bytes requested; this may happen for example because fewer bytes are actually available right now (maybe because we were close to end-of-file, or because we are reading from a pipe, or from a terminal), or because read() was interrupted by a signal.
For /proc files, we can see here that:
The most distinctive thing about files in this directory is the fact that all of them have a file size of 0, with the exception of kcore, mtrr and self.
and
You might wonder how you can see details of a process that has a file size of 0. It makes more sense if you think of it as a window into the kernel. The file doesn't actually contain any data; it just acts as a pointer to where the actual process information resides.
which means that the contents of those pseudo-files are sent by the kernel, in batches as large as the kernel wants. This looks very similar to a pipe, where a produces writes data and a consumer reads it, each of them operating at different speeds.
Related
As per the manual of man 2 pipe, which says that[emphasis mine]:
pipe() creates a pipe, a unidirectional data channel that can be used for interprocess communication. The array pipefd is used to return two file descriptors referring to the ends of the pipe. pipefd[0] refers to the read end of the pipe. pipefd[1] refers to the write end of the pipe. Data written to the write end of the pipe is buffered by the kernel until it is read from the read end of the pipe. For further details, see pipe(7)
But the quotation above and man 7 pipe do not mention which type of buffering is used by the two file descriptors returned by pipe?
And as per the document, which says there are three types of buffering[emphasis mine]:
Standard I/O Library Buffering
The stdio library buffers data with the goal of minimizing the number of calls to the read() and write() system calls. There are three different types of buffering used:
Fully (block) buffered. As characters are written to the stream, they are buffered up to the point where the buffer is full. At this stage, the data is written to the file referenced by the stream. Similarly, reads will result in a whole buffer of data being read if possible.
Line buffered. As characters are written to a stream, they are buffered up until the point where a newline character is written. At this point the line of data including the newline character is written to the file referenced by the stream. Similarly for reading, characters are read up to the point where a newline character is found.
Unbuffered. When an output stream is unbuffered, any data that is written to the stream is immediately written to the file to which the stream is associated.
The ANSI C standard dictates that standard input and output should be fully buffered while standard error should be unbuffered. Typically, standard input and output are set so that they are line buffered for terminal devices and fully buffered otherwise.
And I did a simple test on Ubuntu16.04, here is the code snippet:
#include <sys/wait.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <string.h>
#include <string>
#include <thread>
#include <array>
#include <iostream>
int
main(int argc, char *argv[])
{
int pipefd[2];
pid_t cpid;
std::array<char, 1024> buf;
if (pipe(pipefd) == -1) {
perror("pipe");
exit(EXIT_FAILURE);
}
cpid = fork();
if (cpid == -1) {
perror("fork");
exit(EXIT_FAILURE);
}
if (cpid == 0) { /* Child reads from pipe */
close(pipefd[1]); /* Close unused write end */
int size;
while ((size = read(pipefd[0], buf.data(), buf.size())) > 0)
std::cout << size << std::endl;
write(STDOUT_FILENO, "\n", 1);
close(pipefd[0]);
_exit(EXIT_SUCCESS);
} else { /* Parent writes argv[1] to pipe */
close(pipefd[0]); /* Close unused read end */
std::string str{"hello world"};
for(int i=0; i<3; i++)
{
write(pipefd[1], str.c_str(), str.size());
std::this_thread::sleep_for(std::chrono::seconds(3));
}
close(pipefd[1]); /* Reader will see EOF */
wait(NULL); /* Wait for child */
exit(EXIT_SUCCESS);
}
}
Here is the output of the aforementioned code snippet:
11
//about three seconds later
11
//about three seconds later
You see that what the write end writes to the pipe does not contain a \n, whereas the read end could read out a full string every three seconds. So I think that the two file descriptors returned by pipe are neither block buffered nor line buffered. Two choices are stripped out, then there is only unbuffered buffering.
But the manual of man 7 pipe also says that:
PIPE_BUF
POSIX.1-2001 says that write(2)s of less than PIPE_BUF bytes must be atomic: the output data is written to the pipe as a contiguous sequence. Writes of more
than PIPE_BUF bytes may be nonatomic: the kernel may interleave the data with data written by other processes. POSIX.1-2001 requires PIPE_BUF to be at least
512 bytes. (On Linux, PIPE_BUF is 4096 bytes.)
As per the quotation above, there are buffers for the two file descriptors indeed.
So I am really confused which type of buffering is used by the two file descriptors returned by pipe. And since there is buffer provided to each of the file descriptor, how could I receive the string on time(i.e. one string every three seconds other than three strings together)?
Could anybody shed some light on this matter?
There is buffering in two different places, and it's important not to confuse them.
The document you quoted is about "Standard I/O Library Buffering", which refers to the standard C library (libc). It applies to libc functions like fprintf and fwrite. This buffering happens in userspace, i.e. in the process of your program. Under the hood, when this buffer is flushed, libc invokes write to send the data to the underlying file descriptor.
However, pipe is a direct¹ system call to the kernel, which has nothing to do with the buffering in libc. You can tell the difference because it works with file descriptors, not with FILE*s. The read and write functions you're using in your code are also system calls.
The pipe is still buffered in kernel space, but it makes no sense there to talk about "line buffered" or "block buffered" because there is no automatic flushing going on. If the buffer is full, any write call simply blocks until there is space again. The only way that the buffer in the kernel gets drained is through read calls.
¹ Through a thin wrapper which is also in the C library, but that's beside the point.
I'm having some trouble comprehending exactly how read() works. For example, given the following program, with the file infile containing the string "abcdefghijklmnop":
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <fcntl.h>
int main() {
int fd;
char buf[5] = "WXYZ";
fd = open("infile", O_RDONLY);
read(fd, buf, 2);
read(fd, buf+2, 2);
close(fd);
printf("%c%c%c%c\n", buf[0], buf[1], buf[2], buf[3]);
return 0;
}
Looking at the read system call function:
ssize_t read(int fildes, void *buf, size_t nbyte);
I get that *buf is the buffer that holds the bytes read, and nbyte is the number of bytes that's being read. So after the first read(), only the first 2 characters of infile are read ("a" and "b"). Why isn't the output just "abcd"? Why are there other possibilities such as "aXbZ" or "abcZ"?
The manual page for the version of read I have says:
read() attempts to read nbyte bytes of data from the object referenced by the descriptor fildes into the buffer pointed to by buf.
and:
Upon successful completion, read(), readv(), and pread() return the number of bytes actually read and placed in the buffer. The system guarantees to read the number of bytes requested if the descriptor references a normal file that has that many bytes left before the end-of-file, but in no other case.
Thus, in the case you describe, with “the file infile containing the string "abcdefghijklmnop"”, the two read calls are guarantee to put “ab” and “cd” into buf, so the program will print “abcd” and a new-line character. (I would not take that guarantee literally. Certainly the system can guarantee that it will not allow unrelated interrupts to prevent the read from completely reading the requested data, but it could not guarantee there is no hardware failure, such as a disk drive failing before the read is completed.)
In other situations, when read is reading from a source other than a normal file, each of the two read calls may read 0, 1, or 2 bytes. Thus, the possible buffer contents are:
Bytes read in first read
Bytes read in second read
Buffer contents
0
0
WXYZ
0
1
WXaZ
0
2
WXab
1
0
aXYZ
1
1
aXbZ
1
2
aXbc
2
0
abYZ
2
1
abcZ
2
2
abcd
I am trying to speed up my C program to spit out data faster.
Currently I am using printf() to give some data to the outside world. It is a continuous stream of data, therefore I am unable to use return(data).
How can I use write() or fwrite() to give the data out to the console instead of file?
Overall my setup consist of program written in C and its output goes to the python script, where the data is processed further. I form a pipe:
./program_in_c | script_in_python
This gives additional benefit on Raspberry Pi by using more of processor's cores.
#include <unistd.h>
ssize_t write(int fd, const void *buf, size_t count);
write() writes up to count bytes from the buffer starting at buf to
the file referred to by the file descriptor fd.
the standard output file descriptor is: 1 in linux at least!
concern using flush the stdoutput buffer as well, before calling to write system call to ensure that all previous garabge was cleaned
fflush(stdout); // Will now print everything in the stdout buffer
write(1, buf, count);
using fwrite:
size_t fwrite(const void *ptr, size_t size, size_t nmemb, FILE *stream);
The function fwrite() writes nmemb items of data, each size bytes
long, to the stream pointed to by stream, obtaining them from the
location given by ptr.
fflush(stdout);
int buf[8];
fwrite(buf, sizeof(int), sizeof(buf), stdout);
Please refare to man pages for further reading, in the links below:
fwrite
write
Well, there's little or no win in trying to overcome the already used buffering system of the stdio.h package. If you try to use fwrite() with larger buffers, you'll probably win no more time, and use more memory than is necessary, as stdio.h selects the best buffer size appropiate to the filesystem where the data is to be written.
A simple program like the following will show that speed is of no concern, as stdio is already buffering output.
#include <stdio.h>
int
main()
{
int c;
while((c = getchar()) >= 0)
putchar(c);
}
If you try the above and below programs:
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
int
main()
{
char buffer[512];
int n;
while((n = read(0, buffer, sizeof buffer)) > 0)
write(1, buffer, n);
if (n < 0) {
perror("read");
return EXIT_FAILURE;
}
return EXIT_SUCCESS;
}
You will see that there's no significative difference or, even, the first program will be faster, despite it is doing I/O on a per character basis. (as B. Kernighan & Dennis Ritchie wrote it in her first edition of "The C programming language") Most probably the first program will win.
The calls to read() and write() involve a system call each, with a buffer size decided by you. The individual getchar() and putchar() calls don't. They just store the received chars in a memory buffer, as you print them, whose size has been decided by the stdio.h library implementation, based on the filesystem, and it flushes the buffer, once it is full of data. If you grow the buffer size in the second program, you'll see that you get better results increasing it up to a point, but after that you'll see no more increment in speed. The number of calls made to the library is insignificant with respect to the time involved in doing the actual I/O, and selecting a very large buffer, will eat much memory from your system (and a Raspberry Pi memory is limited in this sense, to 1Gb or ram) If you end making swap due to a so large buffer, you'll lose the battle completely.
Most filesystems have a preferred buffer size, because the kernel does write ahead (the kernel reads more than what you asked for, on sequential reads, in prevision that you'll continue reading more after you consumed the data) and this affects the optimum buffer size. For that, the stat(2) system call tells you what is the optimum buffer size, and stdio uses that when it selects the actual buffer size.
Don't think you are going to get better (or much better) than the program listed first above. Even if you use large enough buffers.
What is not correct (or valid) is to intermix calls that do buffering (like all the stdio package's) with basic system calls (like read(2) or write(2) ---as I've seen recommending you to use fflush(3) after write(2), which is totally incoherent--- that do not buffer the data) there's no earn (and probably you'll get your output incorrectly ordered, if you do part of the calls using printf(3) and part using write(2) (this happens more in pipelines like you plan to do, because the buffers are not line oriented ---another characteristic of buffered output in stdio---)
Finally, I recomend you to read "The Unix programming environment" by Dennis Ritchie and Rob Pike. It will teach you a lot of unix, but one very good thing is that it will teach you to use perfectly the stdio package and the unix filesystem calls for reading and writing. With a little of luck you'll find it in .pdf on internet.
The next program shows you the effect of buffering:
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
int
main()
{
int i;
char *sep = "";
for (i = 0; i < 10; i++) {
printf("%s%d", sep, i);
sep = ", ";
sleep(1);
}
printf("\n");
}
One would assume you are going to see (on the terminal) the program, writing the numbers 0 to 9, separated by , and paced on one second intervals.
But due to the buffering, what you observe is quite different, you'll see how your program waits for 10 seconds without writing anything at all on the terminal, and at the end, writes everything in one shot, including the final line end, when the program terminates, and the shell shows you the prompt again.
If you change the program to this:
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
int
main()
{
int i;
char *sep = "";
for (i = 0; i < 10; i++) {
printf("%s%d", sep, i);
fflush(stdout);
sep = ", ";
sleep(1);
}
printf("\n");
}
You'll see the expected output, because you have told stdio to flush the buffer at each loop pass. In both programs you did 10 calls to printf(3), but there was only one write(2) at the end to write the full buffer. In the second version you forced stdio to do one such write(2) after each printf, and that showed the data out as the program passed through the loop.
Be careful, because another characteristic of stdio can be confounding you, as printf(3), when you print to a terminal device, flushes the output at each \n, but when you run it through a pipe, it does it only when the buffer fills up completely. This saves system calls (in FreeBSD, for example, the buffer size selected by stdio is around 32kb, large enough to force two blocks to write(2) and optimum (you'll not get better going above that size)
The console output in C works almost the same way as a file. Once you have included stdio.h, you can write on the console output, named stdout (for "standard output"). In the end, the following statement:
printf("hello world!\n");
is the same as:
char str[] = "hello world\n";
fwrite(str, sizeof(char), sizeof(str) - 1, stdout);
fflush(stdout);
I am working with huge files. (>>>2gb). the question I have is it safe to use fileno() on the file descriptor if the file is larger than sizeof(int) is?
Here a quick code snippet:
#define _FILE_OFFSET_BITS 64
#include <stdio.h>
#include <inttypes.h>
int readstuff(FILE *fp,uint64_t seekpoint, uint64_t seekwidth) {
int j;
char buf[seekwidth];
if (pread(fileno(fp),buf,seekwidth,seekpoint)!=-1) {
/* do stuf */
return 1;
}
else {
return 2;
}
}
int main() {
FILE *FP;
FP=fopen("/testfile","r");
readstuff(FP,0,10000);
}
The file descriptor returned by fileno() is an int, regardless of the size of the file that it is used to open.
Yes, you can. The value of a file descriptor is unrelated to the size of that file.
fileno(3) gives a file descriptor -which is a fixed small integer once you fopen-ed a stdio FILE stream. A process usually has only dozens of file descriptors, and occasionally (for some servers, see C10K problem) a few dozen of thousands of them.
The kernel is allocating file descriptors (with e.g. open(2) etc...), and gives "small" (generally contiguous) integers
Typically, a file descriptor is a non-negative integer often less than 100, and generally less than 100000. Of course the file descriptor is unrelated to the file size.
Try ls /proc/self/fd/ to list the file descriptors of the process running that ls command and ls /proc/1234/fd/ to list the file descriptors of process of pid 1234.
The command cat /proc/sys/fs/file-max gives the total cumulated maximum number of file descriptors on your system (on mine, it is 1629935 now), and each process has a fraction of that.
You may limit the number of file descriptors a process (and its children) can have with setrlimit(2) using RLIMIT_NOFILE. The bash builtin ulimit calls that syscall (on my system, the descriptors limit is 1024 by default).
Read Advanced Linux Programming.
I wrote this code in C:
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <time.h>
void random_seed(){
struct timeval tim;
gettimeofday(&tim, NULL);
double t1=tim.tv_sec+(tim.tv_usec/1000000.0);
srand (t1);
}
void main(){
FILE *f;
int i;
int size=100;
char *buf=(char*)malloc(size);
f = fopen("output.txt", "a");
setvbuf (f, buf, _IOFBF, size);
random_seed();
for(i=0; i<200; i++){
fprintf(f, "[ xx - %d - 012345678901234567890123456789 - %d]\n", rand()%10, getpid());
fflush(f);
}
fclose(f);
free(buf);
}
This code opens in append mode a file and attaches 200 times a string.
I set the buf of size 100 that can contains the full string.
Then I created multi processes running this code by using this bash script:
#!/bin/bash
gcc source.c
rm output.txt
for i in `seq 1 100`;
do
./a.out &
done
I expected that in the output the strings are never mixed up, as I read that when opening a file with O_APPEND flag the file offset will be set to the end of the file prior to each write and i'm using a fully buffered stream, but i got the first line of each process is mixed as this:
[ xx - [ xx - 7 - 012345678901234567890123456789 - 22545]
and some lines later
2 - 012345678901234567890123456789 - 22589]
It looks like the write is interrupted for calling the rand function.
So...why appear these lines?
Is the only way to prevent this the use file locks...even if i'm using only the append mode?
Thanks in advance!
You will need to implement some form of concurrency control yourself, POSIX makes no guarantees with respect to concurrent writes from multiple processes. You get some guarantees for pipes, but not for regular files written to from different processes.
Quoting POSIX write():
This volume of POSIX.1-2008 does not specify behavior of concurrent writes to a file from multiple processes. Applications should use some form of concurrency control.
(At the end of the Rationale section.)
You open the file in the fully buffered mode. That means that every line of the output first goes into the buffer and when the buffer overflows it gets flushed to the file regardless whether it contains incomplete lines. That causes chunks of output from different processes writing into the same file concurrently to be interleaved.
An easy fix would be to open the file in line buffered mode _IOLBF, so that the buffer gets flushed on each complete line. Just make sure that the buffer size is at least as big as your longest line, otherwise it will end up writing incomplete lines. The buffer is normally flushed with a single write() system call, so that lines from different processes won't interleave each other.
There is no guarantee that write() system call is atomic for different filesystems though, but it normally works as expected because write() normally locks the file descriptor in the kernel with a mutex before proceeding.