Parameter expansion for find command - arrays

Question

Consider the code (the variable $i is there because it was in a loop, adding several conditions to the pattern, e.g. *.a and *.b, ... but to illustrate this problem only one wildcard pattern is enough):
#!/bin/bash
i="a"
PATTERN="-name bar -or -name *.$i"
find . \( $PATTERN \)
If ran on a folder containing files bar and foo.a, it works, outputting:
./foo.a
./bar
But if you now add a new file to the folder, namely zoo.a, then it no longer works:
find: paths must precede expression: zoo.a
Presumably, because the wildcard in *.$i gets expanded by the shell to foo.a zoo.a, which leads to an invalid find command pattern. So one attempt at a fix is to put quotes around the wildcard pattern. Except it does not work:
with single quotes -- PATTERN="-name bar -or -name '*.$i'" the find command outputs only bar. Escaping the single quotes (\') yields the same result.
idem with double quotes: PATTERN="-name bar -or -name \"*.$i\"" -- only bar is returned.
in the find command, if $PATTERN is replaced with "$PATTERN", out comes an error (for single quotes same error, but with single quotes around the wildcard pattern):
find: unknown predicate -name bar -or -name "*.a"'
Of course, replacing $PATTERN with '$PATTERN' also does not work... (no expansion whatsoever takes place).
The only way I could get it to work was to use... eval!
FINDSTR="find . \( $PATTERN \)"
eval $FINDSTR
This works properly:
./zoo.a
./foo.a
./bar
Now after a lot of googling, I saw it mentioned several times that to do this kind of thing, one should use arrays. But this doesn't work:
i="a"
PATTERN=( -name bar -or -name '*.$i' )
find . \( "${PATTERN[#]}" \)
# result: ./bar
In the find line the array has to be enclosed in double quotes, because we want it to be expanded. But single quotes around the wildcard expression don't work, and neither does not quotes at all:
i="a"
PATTERN=( -name bar -or -name *.$i )
find . \( "${PATTERN[#]}" \)
# result: find: paths must precede expression: zoo.a
BUT DOUBLE QUOTES DO WORK!!
i="a"
PATTERN=( -name bar -or -name "*.$i" )
find . \( "${PATTERN[#]}" \)
# result:
# ./zoo.a
# ./foo.a
# ./bar
So I guess my question are actually two questions:
a) in this last example using arrays, why are double quotes required around the *.$i?
b) using an array in this way is supposed to expand «to all elements individually quoted». How would do this with a variable (cf my first attempt)? After getting this to function, I went back and tried using a variable again, with blackslashed single quotes, or \\', but nothing worked (I just got bar). What would I have to do to emulate "by hand" as it were, the quoting done when using arrays?
Thank you in advance for your help.

Answer 1

Required reading:
BashFAQ — I'm trying to put a command in a variable, but the complex cases always fail!
a) in this last example using arrays, why are double quotes required around the *.$i?
You need to use some form of quoting to prevent the shell from performing glob expansion on *. Variables are not expanded in single quotes so '*.$i' doesn't work. It does inhibit glob expansion but it also stops variable expansion. "*.$i" inhibits glob expansion but allows variable expansion, which is perfect.
To really delve into the details, there are two things you need to do here:
Escape or quote * to prevent glob expansion.
Treat $i as a variable expansion, but quote it to prevent word splitting and glob expansion.
Any form of quoting will do for item 1: \*, "*", '*', and $'*' are all acceptable ways to ensure it's treated as a literal asterisk.
For item 2, double quoting is the only answer. A bare $i is subject to word splitting and globbing -- if you have i='foo bar' or i='foo*' the whitespace and globs will cause problems. \$i and '$i' both treat the dollar sign literally, so they're out.
"$i" is the only quoting that does everything right. It's why common shell advice is to always double quote variable expansions.
The end result is, any of the following would work:
"*.$i"
\*."$i"
'*'."$i"
"*"."$i"
'*.'"$i"
Clearly, the first is the simplest.
b) using an array in this way is supposed to expand «to all elements individually quoted». How would do this with a variable (cf my first attempt)? After getting this to function, I went back and tried using a variable again, with blackslashed single quotes, or \\', but nothing worked (I just got bar). What would I have to do to emulate "by hand" as it were, the quoting done when using arrays?
You'd have to cobble together something with eval, but that's dangerous. Fundamentally, arrays are more powerful than simple string variables. There's no magic combination of quotes and backslashes that will let you do what an array can do. Arrays are the right tool for the job.
Could you explain in a little more detail, why ... PATTERN="-name bar -or -name \"*.$i\"" does not work? The quoted double quotes should, when the find command is actually ran, expand the $i but not the glob.
Sure. Let's say we write:
i=a
PATTERN="-name bar -or -name \"*.$i\""
find . \( $PATTERN \)
After the first two line runs, what is the value of $PATTERN? Let's check:
$ i=a
$ PATTERN="-name bar -or -name \"*.$i\""
$ printf '%s\n' "$PATTERN"
-name bar -or -name "*.a"
You'll notice that $i has already been replaced with a, and the backslashes have been removed.
Now let's see how exactly the find command is parsed. In the last line $PATTERN is unquoted because we want all the words to be split apart, right? If you write a bare variable name Bash ends up performing an implied split+glob operation. It performs word splitting and glob expansion. What does that mean, exactly?
Let's take a look at how Bash performs command-line expansion. In the Bash man page under the "Expansion" section we can see the order of operations:
Brace expansion
Tilde expansion, parameter and variable expansion, arithmetic expansion, command substitution, and process substitution
Word splitting
Pathname (AKA glob) expansion
Quote removal
Let's run through these operations by hand and see how find . \( $PATTERN \) is parsed. The end result will be a list of strings, so I'll use a JSON-like syntax to show each stage. We'll start with a list containing a single string:
['find . \( $PATTERN \)']
As a preliminary step, the command-line as a whole is subject to word splitting.
['find', '.', '\(', '$PATTERN', '\)']
Brace expansion -- No change.
Variable expansion
['find', '.', '\(', '-name bar -or -name "*.a"', '\)']
$PATTERN is replaced. For the moment it is all a single string, whitespace and all.
Word splitting
['find', '.', '\(', '-name', 'bar', '-or', '-name', '"*.a"', '\)']
The shell scans the results of variable expansion that did not occur within double quotes for word splitting. $PATTERN was unquoted, so it's expanded. Now it is a bunch of individual words. So far so good.
Glob expansion
['find', '.', '\(', '-name', 'bar', '-or', '-name', '"*.a"', '\)']
Bash scans the results of word splitting for globs. Not the entire command-line, just the tokens -name, bar, -or, -name, and "*.a".
It looks like nothing happened, yes? Not so fast! Looks can be deceiving. Bash actually performed glob expansion. It just happened that the glob didn't match anything. But it could...†
Quote removal
['find', '.', '(', '-name', 'bar', '-or', '-name', '"*.a"', ')']
The backslashes are gone. But the double quotes are still there.
After the preceding expansions, all unquoted occurrences of the characters \, ', and " that did not result from one of the above expansions are removed.
And that's the end result. The double quotes are still there, so instead of searching for files named *.a it searches for ones named "*.a" with literal double quotes characters in their name. That search is bound to fail.
Adding a pair of escaped quotes \" didn't at all do what we wanted. The quotes didn't disappear like they were supposed to and broke the search. Not only that, but they also didn't inhibit globbing like they should have.
TL;DR — Quotes inside a variable aren't parsed the same way as quotes outside a variable.
† The first four tokens have no special characters. But the last one, "*.a", does. That asterisk is a wildcard. If you read the "pathname expansion" section of the man page carefully you'll see that there's no mention of quotes being ignored. The double quotes do not protect the asterisk.
Hang on! What? I thought quotes inhibit glob expansion!
They do—normally. If you write quotes out by hand they do indeed stop glob expansion. But if you put them inside an unquoted variable, they don't.
$ touch 'foobar' '"foobar"'
$ ls
foobar "foobar"
$ ls foo*
foobar
$ ls "foo*"
ls: foo*: No such file or directory
$ var="\"foo*\""
$ echo "$var"
"foo*"
$ ls $var
"foobar"
Read that over carefully. If we create a file named "foobar"—that is, it has literal double quotes in its filename—then ls $var prints "foobar". The glob is expanded and matches the (admittedly contrived) filename!
Why didn't the quotes help? Well, the explanation is subtle, and tricky. The man page says:
After word splitting ... bash scans each word for the characters *, ?, and [.
Any time Bash performs word splitting it also expands globs. Remember how I said unquoted variables are subject to an implied split+glob operator? This is what I meant. Splitting and globbing go hand in hand.
If you write ls "foo*" the quotes prevent foo* from being subject to splitting and globbing. However if you write ls $var then $var is expanded, split, and globbed. It wasn't surrounded by double quotes. It doesn't matter that it contains double quotes. By the time those double quotes show up it's too late. Word splitting has already been performed, and so globbing is done as well.