awk read data from file with getline as it's being written - file

Question

I have a script that is running two commands. The first command is writing data to a temp file. The second command is piping to awk while the first command is running in the background. awk, in the second command, needs to read the data from the temp file, but it's parsing its own data faster than data is getting written to the temp file.
Here's an example:
#!/bin/bash
command1 > /tmp/data.txt &
# command1 takes several minutes to run, so start command 2 while it runs in the background
command2 | awk '
/SEARCH/ {
#Matched input so pull next line from temp file
getline temp_line < "/tmp/data.txt"
}
'
This works, unless awk parses the data from command2 so fast that command1 can't keep up with it. I.e. awk is getting an EOF from /tmp/data.txt before command1 has finished writing to it.
I've also tried wrapping some checks around getline, like:
while ((getline temp_line < "/tmp/data.txt") < 0) {
system("sleep 1") # let command1 write more to the temp file
}
# Keep processing now that we have read the next line
But I think once it hits an EOF in the temp file, it stops trying to read from it. Or something like that.
The overall script works as long as command1 writes to the temp file faster than awk tries to read from it. If I put a sleep 10 command between the two commands, then the temp file builds enough buffer and the script produces the output I need. But I may be parsing files much larger than what I've tested on, or the commands might run at different speeds on different systems, etc, so I'd like a safety mechanism to wait for the file until data has been written to it.
Any ideas how I can do this?

Answer 1

I think you'd need to close the file between iterations and read it from the start again back to where you had read it before, something like this (untested);
sleepTime = 0
while ((getline temp_line < "/tmp/data.txt") <= 0) {
close("/tmp/data.txt")
system("sleep " ++sleepTime) # let command1 write more to the temp file
numLines = 0
while (++numLines < prevLines) {
if ( (getline temp_line < "/tmp/data.txt") <= 0 ) {
print "Aaargghhh, my file is gone!" | "cat>&2"
exit
}
}
}
++prevLines
Note that I built in a variable "sleepTime" to have your command sleep longer each time through the loop so if it's taking your tmp file a long time to fill up your 2nd command waits longer for it each iteration. Use that or not as you like.
Using getline in nested loops with system() commands all seems a tad clumsy and error-prone though - I can't help thinking there's probably a better approach but I don't know what off the top of my head.