How do I use session variables in Netezza nzsql? - netezza

How do I create and use session variables in Netezza nzsql?
How can I use session variables as part of strings?
Can I concatenate session variables with strings?
Can I embed session variables in strings?
How can I use them as part of table names or column names?

Basic Variable Usage
The documentation for session variables in Netezza nzsql is somewhat lacking.
It states that in order to set a variable within a script or at the nzsql prompt, you use \set.
\set var value
You can also specify the variable at the command line. This is useful for passing variables into scripts.
nzsql -v var=value
Within your session or script, you can access the value of that variable as :var
DB.TST(LLAMA)=> \set foo example_table
DB.TST(LLAMA)=> \d :foo
Table "EXAMPLE_TABLE"
Attribute | Type | Modifier | Default Value
----------------+-----------------------+----------+---------------
EXAMPLE_COLUMN | CHARACTER VARYING(16) | |
Distributed on hash: "EXAMPLE_COLUMN"
DB.TST(LLAMA)=> SELECT * FROM :foo;
EXAMPLE_COLUMN
----------------
Hello World
Advanced Variable Usage
The \set command also has undocumented capabilities that add greater flexibility.
In reality, \set takes all values passed to it and concatenates them together.
DB.TST(LLAMA)=> \set foo bar baz qux
DB.TST(LLAMA)=> \echo :foo
barbazqux
The command also supports quoting in a manner similar to shell scripts which allows you to include whitespace within your variables.
Be careful though, quoted and unquoted strings will still be concatenated with each other.
DB.TST(LLAMA)=> \set foo 'bar baz qux'
DB.TST(LLAMA)=> \echo :foo
bar baz qux
DB.TST(LLAMA)=> \set foo 'bar baz' qux
DB.TST(LLAMA)=> \echo :foo
bar bazqux
Double quoting also works to preserves whitespace. However, the double quotes will still be preserved in the variable.
DB.TST(LLAMA)=> \set foo "bar baz qux"
DB.TST(LLAMA)=> \echo :foo
"bar baz qux"
DB.TST(LLAMA)=> \set foo "bar baz" qux
DB.TST(LLAMA)=> \echo :foo
"bar baz"qux
Of course, the different types of quoting can be mixed:
DB.TST(LLAMA)=> \set foo "Hello World" 'Goodbye World'
DB.TST(LLAMA)=> \echo :foo
"Hello World"Goodbye World
Single Quotes
Properly embedding single quotes in a \set command can be difficult.
Incidentally, because double quotes are always preserved, they rarely present a problem.
Single quotes within an unquoted word will be preserved.
DB.TST(LLAMA)=> \set foo bar'baz'qux
DB.TST(LLAMA)=> \echo :foo
bar'baz'qux
Single quotes within a quoted word may result in issues.
DB.TST(LLAMA)=> \set foo 'bar'baz'qux'
DB.TST(LLAMA)=> \echo :foo
barbaz'qux'
Single quotes within a doubly quoted value are preserved.
DB.TST(LLAMA)=> \set foo "This'll work fine!"
DB.TST(LLAMA)=> \echo :foo
"This'll work fine!"
Single quotes on their own need to be quoted and escaped.
DB.TST(LLAMA)=> \set foo '
parse error at end of line
DB.TST(LLAMA)=> \set foo \'
Invalid command \'. Try \? for help.
DB.TST(LLAMA)=> \set foo '\''
DB.TST(LLAMA)=> \echo :foo
'
When in doubt: single quote the phrase and escape all remaining single quotes with backslashes.
DB.TST(LLAMA)=> \set foo '\'bar\'baz\'qux\''
DB.TST(LLAMA)=> \echo :foo
'bar'baz'qux'
Variables In Identifiers
Occasionally you will need to use a variable as part of an identifier (i.e. a column or table name).
Consider the following example table and variable:
DB.TST(LLAMA)=> SELECT * FROM example_table;
BAR_COLUMN | QUX_COLUMN
-------------+-------------
This is bar | This is qux
(1 row)
DB.TST(LLAMA)=> \set foo bar
In this example, you want to select bar_column using your variable :foo (which contains bar) and the text _column.
The following will not work:
DB.TST(LLAMA)=> SELECT :foo_column FROM example_table;
foo_column:
ERROR: 'SELECT FROM example_table;'
error ^ found "FROM" (at char 9) expecting an identifier found a keyword
The above example fails because nzsql cannot determine where the variable name ends (:foo) and the remaining column (_column) name begins.
To fix this, you need to make a new variable with \set by concatenating the value of :foo and the rest of the column name:
DB.TST(LLAMA)=> \set fixed_foo :foo _column
DB.TST(LLAMA)=> \echo :fixed_foo
bar_column
DB.TST(LLAMA)=> SELECT :fixed_foo FROM example_table;
BAR_COLUMN
-------------
This is bar
(1 row)
If the variable contains the end of the identifier you wish to use, no intermediate variables need to be created.
In that specific case, nzsql will properly expand the variable (e.g. column_:foo -> column_bar).
Stringification
Sometimes you will need to use the contents of a variable as a string.
Consider the following example table and variable:
DB.TST(LLAMA)=> SELECT * FROM example_table;
EXAMPLE_COLUMN
----------------
Hello World
Whatever
Something
(3 rows)
DB.TST(LLAMA)=> \set foo Something
If you simply quote the variable within the statement then it will be treated as literal text.
DB.TST(LLAMA)=> SELECT * FROM example_table WHERE example_column = 'Something';
EXAMPLE_COLUMN
----------------
Something
(1 row)
DB.TST(LLAMA)=> SELECT * FROM example_table WHERE example_column = ':foo';
EXAMPLE_COLUMN
----------------
(0 rows)
DB.TST(LLAMA)=> \p
SELECT * FROM example_table WHERE example_column = ':foo';
If you leave the variable unquoted then it will be used as an identifier.
DB.TST(LLAMA)=> SELECT * FROM example_table WHERE example_column = :foo;
ERROR: Attribute 'SOMETHING' not found
To fix this, you need to use \set and your knowledge of quoting to create a usable variable.
You can accomplish this by making a new variable by combining a single quote (properly escaped!), the variable's contents, and another single quote.
DB.TST(LLAMA)=> \set quoted_foo '\'' :foo '\''
DB.TST(LLAMA)=> \echo :quoted_foo
'Something'
DB.TST(LLAMA)=> SELECT * FROM example_table WHERE example_column = :quoted_foo;
EXAMPLE_COLUMN
----------------
Something
(1 row)
If your variable needs to be used inside of a string, it may be easier to stringify your variable and use regular string concatenation.
DB.TST(LLAMA)=> SELECT * FROM example_table WHERE example_column LIKE '%ello%';
EXAMPLE_COLUMN
----------------
Hello World
(1 row)
DB.TST(LLAMA)=> \set foo ello
DB.TST(LLAMA)=> \set quoted_foo '\'' :foo '\''
DB.TST(LLAMA)=> \echo :quoted_foo
'ello'
DB.TST(LLAMA)=> SELECT * FROM example_table WHERE example_column LIKE '%' || :quoted_foo || '%';
EXAMPLE_COLUMN
----------------
Hello World
(1 row)

Related

Parsing array in Bash

I want to "split" an array (in Bash) to a sub-array while retaining all the array-like properties. Apparently, what I have tried has reduced the array to a string.
myscript.sh
#!/bin/bash
A=('foo' 'bar' 'bat' 'baz')
B=${A[#]:0:2} # Get first half of array
for i in ${!B[#]}; do
echo "B[$i]: ${B[$i]}" # result: B[0]: foo bar
done
The result of this code is:
B[0]: foo bar
But the result I seek is:
B[0]: foo
B[1]: bar
What can I do to retain the array properties in B that will allow me to properly loop thru its elements?
A=('foo 123' 'bar 123' 'bat 123' 'baz 123')
B=("${A[#]:0:2}")
declare -p B
Output:
declare -a B='([0]="foo 123" [1]="bar 123")'

Grouping words together in array literal `%w`

Is there any way to combine more than one word together in %w(foo bar) notation to get the resulting array as follows?
%w(foo bar foo or bar foo and bar) # => ["foo", "bar", "foo or bar", "foo and bar"]
Yes, there is a way. You can excape a whitespace with \:
%w(foo bar foo\ or\ bar foo\ and\ bar)
#=> ["foo", "bar", "foo or bar", "foo and bar"]
But I am not sure if this improves readability...
It's possible by using \ as the space character:
%w(foo bar foo\ or\ bar foo\ and\ bar) => ["foo", "bar", "foo or bar", "foo and bar"]
Not using %w notation, but with a similar spirit, using | instead of space for a delimiter:
"foo|bar|foo or bar|foo and bar".split("|")

Surprising array expansion behaviour

I've been surprised with the line marked (!!) in the following example:
log1 () { echo $#; }
log2 () { echo "$#"; }
X=(a b)
IFS='|'
echo ${X[#]} # prints a b
echo "${X[#]}" # prints a b
echo ${X[*]} # prints a b
echo "${X[*]}" # prints a|b
echo "---"
log1 ${X[#]} # prints a b
log1 "${X[#]}" # prints a b
log1 ${X[*]} # prints a b
log1 "${X[*]}" # prints a b (!!)
echo "---"
log2 ${X[#]} # prints a b
log2 "${X[#]}" # prints a b
log2 ${X[*]} # prints a b
log2 "${X[*]}" # prints a|b
Here is my understanding of the behavior:
${X[*]} and ${X[#]} both expand to a b
"${X[*]}" expands to "a|b"
"${X[#]}" expands to "a" "b"
$* and $# have the same behavior as ${X[*]} and ${X[#]}, except for their content being the parameters of the program or function
This seems to be confirmed by the bash manual.
In the line log1 "${X[*]}", I therefore expect the quoted expression to expand to "a|b", then to be passed to the log1 function. The function has a single string parameter which it displays. Why does something else happen?
It'd be cool if your answers were backed by manual/standard references!
IFS is used not just to join the elements of ${X[*]}, but also to split the unquoted expansion $#. For log1 "${X[*]}", the following happens:
"${X[*]}" expands to a|b as expected, so $1 is set to a|b inside log1.
When $# (unquoted) is expanded, the resulting string is a|b.
The unquoted expansion undergoes word-splitting with | as the delimiter (due to the global value of IFS), so that echo receives two arguments, a and b.
That's because $IFS is set to |:
(X='a|b' ; IFS='|' ; echo $X)
Output:
a b
man bash says:
IFS The Internal Field Separator that is used for word splitting after expansion ...
In the POSIX spec section on [Special Parameters[(http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_05_02) we find.
#
Expands to the positional parameters, starting from one. When the expansion occurs within double-quotes, and where field splitting (see Field Splitting) is performed, each positional parameter shall expand as a separate field, with the provision that the expansion of the first parameter shall still be joined with the beginning part of the original word (assuming that the expanded parameter was embedded within a word), and the expansion of the last parameter shall still be joined with the last part of the original word. If there are no positional parameters, the expansion of '#' shall generate zero fields, even when '#' is double-quoted.
*
Expands to the positional parameters, starting from one. When the expansion occurs within a double-quoted string (see Double-Quotes), it shall expand to a single field with the value of each parameter separated by the first character of the IFS variable, or by a if IFS is unset. If IFS is set to a null string, this is not equivalent to unsetting it; its first character does not exist, so the parameter values are concatenated.
So starting with the quoted variants (they are simpler):
We see that the * expansion "expand[s] to a single field with the value of each parameter separated by the first character of the IFS variable". This is why you get a|b from echo "${X[*]" and log2 "${X[*]}".
We also see that the # expansion expands such that "each positional parameter shall expand as a separate field". This is why you get a b from echo "${X[#]}" and log2 "${X[#]}".
Did you see that note about field splitting in the spec text? "where field splitting (see Field Splitting) is performed"? That's the key to the mystery here.
Outside of quotes the behavior of the expansions is the same. The difference is what happens after that. Specifically, field/word splitting.
The simplest way to show the problem is to run your code with set -x enabled.
Which gets you this:
+ X=(a b)
+ IFS='|'
+ echo a b
a b
+ echo a b
a b
+ echo a b
a b
+ echo 'a|b'
a|b
+ echo ---
---
+ log1 a b
+ echo a b
a b
+ log1 a b
+ echo a b
a b
+ log1 a b
+ echo a b
a b
+ log1 'a|b'
+ echo a b
a b
+ echo ---
---
+ log2 a b
+ echo a b
a b
+ log2 a b
+ echo a b
a b
+ log2 a b
+ echo a b
a b
+ log2 'a|b'
+ echo 'a|b'
a|b
The thing to notice here is that by the time log1 is called in all but the final case the | is already gone.
The reason it is already gone is because without quotes the results of the variable expansion (in this case the * expansion) are field/word split. And since IFS is used both to combine the fields being expanded and then to split them again the | gets swallowed by field splitting.
And to finish the explanation (for the case actually in question), the reason this fails for log1 even with the quoted version of the expansion in the call (i.e. log1 "${X[*]}" which expands to log1 "a|b" correctly) is because log1 itself does not use a quoted expansion of # so the expansion of # in the function is itself word-split (as can be seen by echo a b in that log1 case as well as all the other log1 cases).

Iterating arrays with elements containing spaces

Given an array of two elements,
array=("foo bar" baz)
Using a for..in loop with the "normal" syntax yields three iterations (elements of the array are expanded).
for element in ${array[#]}
do echo $element
done
Output:
foo
bar
baz
While a for..in loop with the "index" syntax works as intended (elements are not expanded).
for i in ${!array[#]}
do echo ${array[i]}
done
Output:
foo bar
baz
Is there any way to use the first syntax construct as I intend (i.e. to get the same results as those I get using the second construct)?
GNU bash, version 4.1.2(1)-release
Quotes make the difference, so you need to update your code to the following:
for element in "${array[#]}"
do
echo $element
done
Note
for element in "${array[#]}"
^ ^
instead of
for element in ${array[#]}
Test
$ array=("foo bar" baz)
$ for element in "${array[#]}"; do echo $element; done
foo bar
baz
Did you try:
for element in "${array[#]}"; do
echo "${element}"
done
The manual would tell:
IFS is a list of characters that separate fields; used when the shell
splits words as part of expansion.
Doing:
IFS=$''
for element in ${array[#]}; do
echo "${element}"
done
would also give the expected result.

How to pass an array argument to the Bash script

It is surprising me that I do not find the answer after 1 hour search for this.
I would like to pass an array to my script like this:
test.sh argument1 array argument2
I DO NOT want to put this in another bash script like following:
array=(a b c)
for i in "${array[#]}"
do
test.sh argument1 $i argument2
done
Bash arrays are not "first class values" -- you can't pass them around like one "thing".
Assuming test.sh is a bash script, I would do
#!/bin/bash
arg1=$1; shift
array=( "$#" )
last_idx=$(( ${#array[#]} - 1 ))
arg2=${array[$last_idx]}
unset array[$last_idx]
echo "arg1=$arg1"
echo "arg2=$arg2"
echo "array contains:"
printf "%s\n" "${array[#]}"
And invoke it like
test.sh argument1 "${array[#]}" argument2
Have your script arrArg.sh like this:
#!/bin/bash
arg1="$1"
arg2=("${!2}")
arg3="$3"
arg4=("${!4}")
echo "arg1=$arg1"
echo "arg2 array=${arg2[#]}"
echo "arg2 #elem=${#arg2[#]}"
echo "arg3=$arg3"
echo "arg4 array=${arg4[#]}"
echo "arg4 #elem=${#arg4[#]}"
Now setup your arrays like this in a shell:
arr=(ab 'x y' 123)
arr2=(a1 'a a' bb cc 'it is one')
And pass arguments like this:
. ./arrArg.sh "foo" "arr[#]" "bar" "arr2[#]"
Above script will print:
arg1=foo
arg2 array=ab x y 123
arg2 #elem=3
arg3=bar
arg4 array=a1 a a bb cc it is one
arg4 #elem=5
Note: It might appear weird that I am executing script using . ./script syntax. Note that this is for executing commands of the script in the current shell environment.
Q. Why current shell environment and why not a sub shell?
A. Because bash doesn't export array variables to child processes as documented here by bash author himself
If values have spaces (and as a general rule) I vote for glenn jackman's answer, but I'd simplify it by passing the array as the last argument. After all, it seems that you can't have multiple array arguments, unless you do some complicated logic to detect their boundaries.
So I'd do:
ARRAY=("the first" "the second" "the third")
test.sh argument1 argument2 "${ARRAY[#]}"
which is the same as:
test.sh argument1 argument2 "the first" "the second" "the third"
And in test.sh do:
ARG1="$1"; shift
ARG2="$1"; shift
ARRAY=("$#")
If values have no spaces (i.e. they are urls, identifiers, numbers, etc) here's a simpler alternative. This way you can actually have multiple array arguments and it's easy to mix them with normal arguments:
ARRAY1=(one two three)
ARRAY2=(four five)
test.sh argument1 "${ARRAY1[*]}" argument3 "${ARRAY2[*]}"
which is the same as:
test.sh argument1 "one two three" argument3 "four five"
And in test.sh you do:
ARG1="$1"
ARRAY1=($2) # Note we don't use quotes now
ARG3="$3"
ARRAY2=($4)
I hope this helps. I wrote this to help (you and me) understand how arrays work, and how * an # work with arrays.
Passing array as argument to script by using alternate separator
And passing array as argument to remote script by using alternate separator
Practical sample:
Choosing bell character: ascii 0x07 used by C sequence: \a.
Little preparation: Having a function to prepare variables:
mergeArray() {
local IFS=$'\a'
local -n src=$1 tgt=$2
tgt=${src[*]}
}
Then the script could look like:
#!/bin/bash
arg1="$1"
IFS=$'\a' read -ra arg2 <<<"$2"
arg3="$3"
declare -p arg{1,2,3}
In practice:
var1=Hello var2=(foo bar baz) var3=world.
mergeArray var2 mvar2
bash script "$var1" "$mvar2" "$var3"
must output:
declare -- arg1="Hello"
declare -a arg2=([0]="foo" [1]="bar" [2]="baz")
declare -- arg3="world."
Explanations
by using local IFS= I ensure IFS varialbe won't be modified in environment
local -n is a nameref for binding variables.
${src[*]} will merge all elements of array to one string by using 1st character of $IFS
Of course, I use \a for this, this could replaced (on both side) by any other single byte non null character (ascii) from \1 to \377, while choosed character is not used in your array.
Installation and remote execution
The function mergeArray could be installed into your .bashrc file or into a separated file to be sourced before you run your script.
Once done, the script himself could even be located remotely. then run by same command:
ssh user#remote /path/to/script "$var1" "$mvar2" "$var3"
But for more robust remote execution I prefer:
myArray=("foo bar baz" 'alice bob charlie' 'strawberry raspberry')
mergeArray myArray mergedArray
var1="Hello world!"
ssh user#remote /bin/bash -s <<eoRemote
/path/to/script "$var1" "$mergedArray" "This seem nice, isnt't it?"
eoRemote
From my remote host, I will read:
declare -- arg1="Hello world!"
declare -a arg2=([0]="foo bar baz" [1]="alice bob charlie" [2]="strawberry raspberry")
declare -- arg3="This seem nice, isnt't it?"
( Notice the mixing of single quotes and double quotes!! ;-)
Passing Associative arrays as argument to remote bash via ssh
Things become a little stronger!
mergeArray () {
local -n _srce=${1?Source var missing.} _trgt=${2?Target var missing.}
local _tvar
IFS=\ read _ _tvar _ <<< "${_srce#A}"
case ${_tvar#-} in
*a*) local IFS=$'\a'; _trgt=${_srce[*]} ;;
*A*) _trgt=''
for _tvar in "${!_srce[#]}" ;do
printf -v _tvar '%s\a%s\a' "$_tvar" "${_srce[$_tvar]}"
_trgt+="$_tvar"
done
_trgt=${_trgt%$'\a'} ;;
* ) printf >&2 '%s ERROR: Variable "%s" is not an array.\n' \
$FUNCNAME "$1"
return 1 ;;
esac
}
Then parsing args in script will become:
#!/bin/bash
arg1=${1}
IFS=$'\a' read -ra arg2 <<<"$2"
IFS=$'\a' read -ra _tmpvar <<<"$3"
printf -v _tmpvar '[%s]="%s" ' "${_tmpvar[#]//\"/\\\"}"
declare -A arg3="($_tmpvar)"
arg4=$4
declare -p arg{1,2,3,4}
In action:
var1="Hello world!"
myArray=("foo bar baz" 'alice bob charlie' 'strawberry raspberry')
declare -A myAArray='([Full name]="John Doo" [Birth date]="1970/01/02 12:34:56"
[Status]="Maried" [Title]="Chief")'
mergeArray myArray mergedArray
mergeArray myAArray mergedAArray
ssh user#remote /bin/bash -s <<eoRemote
/path/to/script "$var1" "$mergedArray" "$mergedAArray" "Still seem nice, isn't it?"
eoRemote
declare -- arg1="Hello world!"
declare -a arg2=([0]="foo bar baz" [1]="alice bob charlie" [2]="strawberry raspberry")
declare -A arg3=(["Birth date"]="1970/01/02 12:34:56" [Title]="Chief" [Status]="Maried" ["Full name"]="John Doo" )
declare -- arg4="Sill seem nice, isn't it?"
Passing complex variables as arguments over ssh
And for holding double-quotes in addition to already supported single-guotes, spaces and others,
remplace $varNAme by ${varName//\"/\\\"}:
var1="Hello world!"
myArray=("foo bar baz" 'alice bob charlie' 'strawberry raspberry')
declare -A myAArray='([Full name]="John Doo" [Birth date]="1970/01/02 12:34:56"
[Status]="Maried")'
myAArray[Title]="Chief's pain sufferer"
myAArray[datas]='{ "height": "5.5 feet", "weight":"142 pounds",'
myAArray[datas]+=$' "phrase": "Let\'s go!" }'
mergeArray myArray mergedArray
mergeArray myAArray mergedAArray
ssh user#remote /bin/bash -s <<eoRemote
/path/to/script "${var1//\"/\\\"}" "${mergedArray//\"/\\\"}" \
"${mergedAArray//\"/\\\"}" "This still seem nice, isn't it?"
eoRemote
declare -- arg1="Hello world!"
declare -a arg2=([0]="foo bar baz" [1]="alice bob charlie" [2]="strawberry raspberry")
declare -A arg3=([Title]="Chief's pain sufferer" [Status]="Maried" ["Birth date"]="1970/01/02 12:34:56" ["Full name"]="John Doo" [datas]="{ \"height\": \"5.5 feet\", \"weight\":\"142 pounds\", \"phrase\": \"Let's go!\" }" )
declare -- arg4="This still seem nice, isn't it?"
Or after some foldering:
declare -- arg1="Hello world!"
declare -a arg2=([0]="foo bar baz" [1]="alice bob charlie"
[2]="strawberry raspberry" )
declare -A arg3=([Title]="Chief's pain sufferer" [Status]="Maried"
["Birth date"]="1970/01/02 12:34:56" ["Full name"]="John Doo"
[datas]="{ \"height\": \"5.5 feet\", \"weight\":\"142 pounds\",
\"phrase\": \"Let's go!\" }" )
declare -- arg4="This still seem nice, isn't it?"
( By adding: jq <<<${arg3[datas]} at end of my script, I see:
{
"height": "5.5 feet",
"weight": "142 pounds",
"phrase": "Let's go!"
}
:-)
if the length of your array is not too long, you can convert the array to a string which joining with ','
file1.sh
s=${1}
arr=(`echo ${s} | tr ',' ' '`) # then we can convert str to arr
file2.sh
a="1,2,3,4"
sh file1.sh ${a}
You can write your array to a file, then source the file in your script.
e.g.:
array.sh
array=(a b c)
test.sh
source $2
...
Run the test.sh script:
./test.sh argument1 array.sh argument3
The best solution that I'm found here
f() {
declare -a argAry1=("${!1}")
echo "${argAry1[#]}"
# ...
}
arr1=(
"a"
"b"
)
f arr1[#] arr2[#] arg2
If this is your command:
test.sh argument1 ${array[*]} argument2
You can read the array into test.sh like this:
arg1=$1
arg2=${2[*]}
arg3=$3
It will complain at you ("bad substitution"), but will work.

Resources