Read from a file which is being actively written to - c

Question

I have two programs: one (program1) writes to a file continuously, and I want the other program (program2) to read the file continuously. What is happening is that my second program is only reading up to the data point which was written when the second code was executed and stops instead of continuously reading.
Is there any way to achieve the thing?
Basically, I want the output of program1 to be used as the input of program2. Is there any way to read and write in RAM instead of file as disk read costs more time.
Code 2:
#include <stdio.h>
int main(){
FILE *fptr;
fptr = fopen("gbbct1.seq","r");
char c;
c = fgetc(fptr);
while (c != EOF){
printf("%c", c);
c = fgetc(fptr);
}
}
I am looking for platform independent approach. If that's not possible, I would like to know for Linux platform. I don't need to preserve the data once read. I don't want to block program1.

Answer 1

The most basic version of your code needs to reset the file stream status when it encounters EOF and then sleep for a while. For example, assuming POSIX and using only the simplest (most ubiquitous) functions:
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
int main(void)
{
const char filename[] = "gbbct1.seq";
FILE *fptr = fopen(filename, "r");
if (fptr == 0)
{
fprintf(stderr, "failed to open file '%s' for reading\b", filename);
exit(EXIT_FAILURE);
}
while (1)
{
int c;
while ((c = fgetc(fptr)) != EOF)
fputc(c, stdout);
clearerr(fptr);
sleep(1);
}
/*NOTREACHED*/
return EXIT_FAILURE;
}
The sleep() function sleeps for an integral number of seconds; if you want sub-second sleeps, you can consider usleep(),
nanosleep(), timer_create() and relatives, etc.
I have a program I call dribbler (because it dribbles data to its output):
Usage: dribbler [-hlntV][-s nap.time][-r std.dev][-f outfile][-i infile][-m message][-o openstr][-F format]
-V Print version information and exit
-f outfile Write to named file (dribbler.out)
-h Print this help message and exit
-i infile Read lines from input file
-l Loop back to start of input file on EOF
-m message Write message on each line of output
-n Number lines read from input file
-o openstr Flags passed to fopen() (a+)
-s nap.time Sleep for given interval between writes (1.000 second)
-r std.dev Randomize the time (Gaussian around nap.time with std.dev)
-t Write to standard output instead of file
-F format Printf format to use instead of %zu
I used:
$ dribbler -s 3 -r 1.3 -f gbbct1.seq &
[1] 81129
$
to write to the control file that program2 is coded to read. I then ran program2 on it, and it produced the outputs as it proceeded.
It's hard to show the time sequence on SO. I have another program (the story of my life) called tstamp which reads lines of input and prints them with a timestamp prefixed to the line:
Usage: tstamp [-hV][-f num][-F format]
-f num Number of fractional digits (0, 3, 6, 9)
-h Print this help message and exit
-F fmt Time format (strftime(3)) to use
-V Print version information and exit
I tried modifying program2.c to set line buffered mode on my Mac (macOS Sierra 10.12.5, GCC 7.1.0), by adding the line below before the while loop in program2.c, but it was effectively ignored, somewhat to my surprise and chagrin:
setvbuf(fptr, 0, _IOLBF, 0);
So, I rewrote the while loop as:
while ((c = fgetc(fptr)) != EOF)
{
fputc(c, stdout);
if (c == '\n')
fflush(stdout);
}
Then I was able to run dribbler in the background, and program2 | tstamp -f 3 to get output like:
$ program2 | tstamp -f 3
2017-06-03 23:52:44.836: 0: message written to file
2017-06-03 23:52:44.836: 1: message written to file
2017-06-03 23:52:44.836: 2: message written to file
2017-06-03 23:52:44.836: 3: message written to file
[…more similar lines with the same time stamp…]
2017-06-03 23:52:44.836: 22: message written to file
2017-06-03 23:52:44.836: 23: message written to file
2017-06-03 23:52:44.836: 24: message written to file
2017-06-03 23:52:44.836: 25: message written to file
2017-06-03 23:52:50.859: 26: message written to file
2017-06-03 23:52:54.866: 27: message written to file
2017-06-03 23:52:58.880: 28: message written to file
2017-06-03 23:53:02.888: 29: message written to file
2017-06-03 23:53:05.902: 30: message written to file
2017-06-03 23:53:07.907: 31: message written to file
2017-06-03 23:53:09.913: 32: message written to file
2017-06-03 23:53:12.925: 33: message written to file
2017-06-03 23:53:14.935: 34: message written to file
2017-06-03 23:53:15.938: 35: message written to file
2017-06-03 23:53:19.954: 36: message written to file
2017-06-03 23:53:21.964: 37: message written to file
2017-06-03 23:53:23.972: 38: message written to file
^C
$ kill %1
[1]+ Terminated: 15 dribbler -s 3 -r 1.3 -f gbbct1.seq
$
You can see that I'd had dribbler running for a while when I started program2 (it got modified and recompiled — part of my chagrin), so there was quite a lot of date to read immediately (hence the multiple lines with the timestamp 2017-06-03 23:52:44.836:), but then it was waiting on dribbler to write more, and as you can see, it sometimes waited nearly 6 seconds between lines, and other times about 1 second, and various intervals in between. The gaps are made more uniform by program2 sleeping for a second at a time. (Yes, I wrote these tools to help answer questions on SO — but dribbler and tstamp both pre-date this question by months.)

Answer 2

I have two programs: one (program1) writes to a file continuously, and I want the other program (program2) to read the file continuously.
What happens then is platform-specific. BTW, the mere ability to run several programs at once (in several processes) is provided by the operating system (and is not defined in, since outside of, the C11 standard). Read Operating Systems : Three Easy Pieces (freely downloadable chapters).
IIRC, on Windows (that I don't know and never used) that is not allowed to happen (one of the programs would be blocked or would have its file open fail).
However, if on Linux with a native local file system (such as Ext4), you could consider using inotify(7) facilities (which won't work with remote file systems à la NFS, and probably won't work on FAT filesystems like e.g. some USB key; but you need to check).
Basically, I want the output of program1 to be used as the input of program2. Is there any way to read and write in RAM instead of file as disk read costs more time.
(I am supposing you wrote both program1 and program2, or at least have their source code and can modify it)
BTW, application programs don't read directly from RAM; they work in virtual memory and each process has its own virtual address space. See this answer.
You surely want to have some inter-process communications, which are provided by your operating system.
On Linux there are many ways of doing that (you should read Advanced Linux Programming whose chapters are freely downloadable). I suggest considering some fifo(7), or some pipe(7) (if both running programs can be started from a common process), or some unix(7) sockets.
You surely need to multiplex for I/O (in both processes) e.g. by having some event loop around a poll(2).
Windows also has inter-process communication facilities. But I don't know them.
(I strongly recommend to spend a few days reading, notably Advanced Linux Programming or some other similar book, before writing a single line of code. You lack an overall picture on OSes and on Linux)
I would recommend using a pipe(7), or else some named fifo(7), or else some unix(7) socket. You could then write code portable on all POSIX systems. I don't recommend to use a file and inotify(7) (which is complex and Linux specific). See also popen(3).
You might find some framework libraries (e.g. Glib from GTK, QtCore, POCO, libevent, 0mq) to help you write portable code able to run on many platforms.