How can I read/store an array in bash in parallel? - arrays

Question

I have already read posts like How can I store the “find” command results as an array in Bash or Creating an array from a text file in Bash or Store output of command into the array
Now my issue is the following: How to do this in parallel?
Background:
I have a script for processing a large git repository with a lot of submodules and perform certain actions within these. Sometimes there are some tasks that take a while so meanwhile I want to give some user feedback to indicate that something is still happening and the code isn't just stuck ^^
I have a function
function ShowSpinner()
{
pid=$!
while [ -d /proc/$pid ]
do
for x in '-' '/' '|' '\\'
do
echo -ne ${x}" \r"
sleep 0.1
done
done
}
for displaying a little spinner while doing long tasks. And so far currently I use this e.g. like
while IFS= read -r line
do
# Some further processing of the output lines here
done <<< $(git pull 2>&1) & ShowSpinner
which works fine and always displays the spinner until the task is finished.
In particular I use this also for finding submodules in a git repository like
function FindSubmodules()
{
# find all .git FILES and write the result to the temporary file .submodules
find -name ".git" -type f > .submodules & ShowSpinner
# read in the temporary file
SUBMODULES=$(cat .submodules)
# and delete the temporary file
rm .submodules
}
later I iterate the submodules using e.g.
function DoSomethingWith()
{
for submodule in ${SUBMODULES}
do
echo $submodule
done
}
FindSubmodules
DoSomethingWith
Of course I do more stuff in there, this is only a short example.
This works find, but what I don't like here is that this file .submodules is created (and if only temporary). I would prefer to directly store the result in an array and then iterate that one directly.
So after reading mentioned posts I tried to use something like simply
IFS=$'\n'
SUBMODULES=( $(find -name ".git" -type f)) & ShowSpinner
or from the links also
readarray SUBMODULES < <(find -name ".git" -type f) & ShowSpinner
or
readarray -t SUBMODULES "$(find -name ".git" -type f)" & ShowSpinner
and then iterate like
for submodule in ${SUBMODULES [#]}
do
echo $submodule
done
For all three options the result is basically the same: The spinner works fine but all that I get using this is one single entry with the last char of the ShowSpinner instead of the results of find. Without the & ShowSpinner it works fine but of course doesn't show any feedback of a long tasks.
What am I doing wrong? How can I get the readarray to work in parallel with the ShowSpinner function?
Update as suggested I have put it to a function (actually I already had functions just didn't put the spinner behind the entire function so far)
function FindSubmodules()
{
echo ""
echo ${BOLD}"Scanning for Submodules ... "${NORMAL}
SUBMODULES=($(find -name ".git" -type f))
for submodule in "${SUBMODULES[#]}"
do
echo $submodule
done
}
function CheckAllReposForChanges()
{
# Check Submodules first
for submodule in "${SUBMODULES[#]}"
do
# remove prefixed '.'
local removedPrefix=${submodule#.}
# remove suffix '.git'
local removedSuffix=${removedPrefix%.git}
echo "${BASEPATH}${removedSuffix}"
done
# Check the main repo itself
echo "${BASEPATH}"
echo ""
}
FindSubmodules & ShowSpinner
CheckAllReposForChanges
the CheckRepoForChanges function itself works just fine.
What I get now is the spinner and then the correct output from the first FindSubmodules like e.g.
./SomeFolder/.git
./SomeOtherFolder/.git
./SomeThirdFolder/.git
etc
However when it comes to the CheckAllReposForChanges (again the echo is just an example for debugging) I don't get any output except the main repository path. It seems like now SUBMODULES is empty since it is being filled in the background. It worked with the solution I used originally.

Answer 1

Of course the array is empty; you have backgrounded the function which will eventually populate it (and anyway, there is no way really for the background process to populate a variable in its parent once it finishes).
Run both functions in the same background process and they will be able to communicate properly.
{ FindSubmodules
CheckAllReposForChanges
} & ShowSpinner

Answer 2

Maybe I'm misreading the question but it seems (to me) the requirement is to pass data 'up' from a backgrounded/child process to the calling/parent process.
A backgrounded script/function call spawns a new, asynchronous OS-level process; there is no easy way to pass data 'up' from the child process to the parent process.
While it may be possible to build some sort of inter-process shared memory structure to share data between parent and child processes, it's a bit easier if we can use some sort of intermediate storage (eg, fifo, file, database table, queuing system, etc) that the various processes can 'share'.
One idea:
parent process creates one or more temp directories (eg, one for each distinct array to be populated)
each child process writes data to a file (filename = ${BASHPID}) in a particular temp directory in a format that can be easily parsed (and loaded into an array) by the parent
the parent calls the child process, waits for the child process to complete, and then ...
the parent process reads the contents of all files in the temporary directory(s) and loads the appropriate array(s)
For sake of an example I'll assume we just need to populate a single array; I'm also going to use the same temp directory for capturing/storing modules for the functions regardless of whether each function is run in the background or foreground:
unset submodules # delete any variable with this name
submodules=() # init array
outdir=$(mktemp -d) # create temp directory
FindSubmodules()
{
... snip ...
echo "$submodule" >> "${outdir}/${BASHPID}" # write module to temp file
... snip ...
}
CheckAllReposForChanges()
{
... snip ...
echo "$submodule" >> "${outdir}/${BASHPID}" # write module to temp file
... snip ...
}
FindSubmodules & ShowSpinner
CheckAllReposForChanges
# now pull modules from temp file(s) into array; NOTE: assumes each temp file contains a single module name on each line, and no blank lines, otherwise OP can add some logic to address a different format
while read -r modname
do
submodules+=(${modname})
done < <(cat "${outdir}"/[0-9]*)
# remove temp directory and file(s)
'rm' -rf ${outdir}

Answer 3

If you can write your parallelism using GNU Parallel, you can use parset:
dostuff() {
# Do real stuff here
sleep 10;
}
export -f dostuff
function ShowSpinner()
{
while [ -d /proc/$pid ]
do
for x in '-' '/' '|' '\\'
do
echo -ne ${x}" \r"
sleep 0.1
done
done
}
sleep 1000000 &
pid=$!
ShowSpinner &
parset myout dostuff < <(find -name ".git" -type f)
kill $pid
echo
Or (if you are willing to change ShowSpinner:
dostuff() {
# Do real stuff here
sleep 10;
}
export -f dostuff
function ShowSpinner()
{
while true; do
for x in '-' '/' '|' '\\'
do
echo -ne ${x}" \r"
sleep 0.1
done
done
}
ShowSpinner &
pid=$!
parset myout dostuff < <(find -name ".git" -type f)
kill $pid
echo