I declare an associative array:
$ declare -A dikv
Initialize it with some key/value pairs:
$ dikv=( ["k1"]="v1" ["k2"]="v2" ["k3"]="v3" ["k4"]="v4")
Then I can save the contents to a file:
$ declare -p dikv > /tmp/dikv.saved
This is the content of /tmp/dikv.saved:
$ cat /tmp/dikv.saved
declare -A dikv=([k4]="v4" [k1]="v1" [k2]="v2" [k3]="v3" )
Now, in a new shell environment, I can load the saved associative array:
$ source /tmp/dikv.saved
And the content is properly accesible:
$ echo "${!dikv[#]}"
k4 k1 k2 k3
$ echo "${dikv[#]}"
v4 v1 v2 v3
This works as expected, nice.
Now I want to do exactly the same but using a bash function:
#! /bin/bash
declare -A dikv
backup_dictionary()
{
local -n dict_ref=$1
FILE=$2
echo "${!dict_ref[#]}"
echo "${dict_ref[#]}"
declare -p dict_ref > $FILE
}
dikv=( ["k1"]="v1" ["k2"]="v2" ["k3"]="v3" ["k4"]="v4")
backup_dictionary dikv /tmp/dikv.saved
As you can see, I pass the associative array to the function using local -n. When I run this code, the echo's inside the function print the content of the associative array properly. So, as far as I understand, the associative array has been properly passed as argument.
However, this statement is not working as expected:
$ declare -p dict_ref > $FILE
This is the content of $FILE:
$ cat /tmp/dikv.saved
declare -n dict_ref="dikv"
I expected to see something like this:
dikv=( ["k1"]="v1" ["k2"]="v2" ["k3"]="v3" ["k4"]="v4")
As it happens when not using a bash funcion. Can you explain what happens here? And what should be the proper way to fix this? Thanks!
Your dict_ref stores the string dikv. Hence if you do a declare -p dict_ref, I would expect to see as output something like
declare -n dict_ref=dikv
Note that dereferencing (due to the -n declaration) occurs during parameter expansion, i.e. when you do a $dict_ref.
You could do a
declare -p "$1" >$FILE
This question already has answers here:
Reading output of a command into an array in Bash
(4 answers)
Closed 1 year ago.
I'm new to bash scripting and trying to learn.
I have a variable $temp, that contains what I get from an API call, which is a long list of SHA1 hashes. I want to take what is in $temp and create an array that I can loop through and compare each entry to a string I have defined. Somehow, I can not seem to get it to work. The array I create always only contains only one element. It is like the rest of the string in $temp is being ignored.
I have tried multiple approaches, and can't get anything working the way I want it to.
Here's the code snippet:
temp=$(curl --silent --request GET "https://api.pwnedpasswords.com/range/$shortendHash")
echo "Temp is: $temp"
declare -A myArr
read myArr <<<$temp
echo ${myArr[#]}
echo ${myArr[0]}
echo ${myArr[1]}
When I run this, I get a full list of hashes, when I echo $temp (line 2), something like this:
"Temp is: 002C3CADF9FC86069F09961B10E0EDEFC45:1
003BE35BD9466D19258E79C8665AFB072F6:1
0075B2ADF782F9677BDDF8CC5778900D986:8
00810352C02B145FF304FCBD5BEF4F7AA9F:1
013720F40B99D6BCF6F08BD7271523EBB49:1
01C894A55EBCB6048A745B37EC2D989F102:1
030B46D53696C8889910EE0F9EB42CEAE97:4
03126B56153F0564F4A6ED6E734604C67E8:2
043196099707FCB0C01F8C2111086A92A0B:1
0452CABBEF8438F358F0BD8089108D0910E:5
0490255B206A2C0377EBD080723BF72DDAE:2
05AD1141C41237E061460DB5CA5412C1A48:4
05C1D058E4439F8F005EFB32E7553E6AA5B:1
067AF515CC712AC4ACA012311979DBC7C9A:2 ... and so on.."
But the
echo ${myArr[#]}
echo ${myArr[0]}
echo ${myArr[1]}
returns (first value is ${myArr[#]}, second is ${myArr[0]}, and last one is an empty line, indicating that there is no value at array position 1):
002C3CADF9FC86069F09961B10E0EDEFC45:1
002C3CADF9FC86069F09961B10E0EDEFC45:1
I hope someone can help with this.
Thanks a lot in advance!
declare -A myArr # creates an associative array (hash)
declare -a myArr # creates an array
Neither is necessary here. read can create the array for you. See help declare and help read.
temp=$(curl --silent --request GET "https://api.pwnedpasswords.com/range/$shortendHash")
# -a: assign the words read to sequential indices of the array
read -a myArr <<<"$temp"
declare -p myArr
Output:
declare -a myArr=([0]="002C3CADF9FC86069F09961B10E0EDEFC45:1" [1]="003BE35BD9466D19258E79C8665AFB072F6:1" [2]="0075B2ADF782F9677BDDF8CC5778900D986:8" [3]="00810352C02B145FF304FCBD5BEF4F7AA9F:1" [4]="013720F40B99D6BCF6F08BD7271523EBB49:1" [5]="01C894A55EBCB6048A745B37EC2D989F102:1" [6]="030B46D53696C8889910EE0F9EB42CEAE97:4" [7]="03126B56153F0564F4A6ED6E734604C67E8:2" [8]="043196099707FCB0C01F8C2111086A92A0B:1" [9]="0452CABBEF8438F358F0BD8089108D0910E:5" [10]="0490255B206A2C0377EBD080723BF72DDAE:2" [11]="05AD1141C41237E061460DB5CA5412C1A48:4" [12]="05C1D058E4439F8F005EFB32E7553E6AA5B:1" [13]="067AF515CC712AC4ACA012311979DBC7C9A:2")
Assuming ...
$(curl --silent --request GET "https://api.pwnedpasswords.com/range/$shortendHash")
Returns ...
002C3CADF9FC86069F09961B10E0EDEFC45:1
003BE35BD9466D19258E79C8665AFB072F6:1
0075B2ADF782F9677BDDF8CC5778900D986:8
00810352C02B145FF304FCBD5BEF4F7AA9F:1
013720F40B99D6BCF6F08BD7271523EBB49:1
01C894A55EBCB6048A745B37EC2D989F102:1
030B46D53696C8889910EE0F9EB42CEAE97:4
03126B56153F0564F4A6ED6E734604C67E8:2
043196099707FCB0C01F8C2111086A92A0B:1
0452CABBEF8438F358F0BD8089108D0910E:5
0490255B206A2C0377EBD080723BF72DDAE:2
05AD1141C41237E061460DB5CA5412C1A48:4
05C1D058E4439F8F005EFB32E7553E6AA5B:1
067AF515CC712AC4ACA012311979DBC7C9A:2
Then assign this to an array by wrapping in a set of parens:
# unset current variable
$ unset myArr
# no need to issue a `declare`; the `myArr=( ... )` will automatically create `myArr` as a normal integer-indexed array
$ myArr=( $(curl --silent --request GET "https://api.pwnedpasswords.com/range/$shortendHash") )
# verify array contents:
$ typeset myArr
declare -a myarr=([0]="002C3CADF9FC86069F09961B10E0EDEFC45:1" [1]="003BE35BD9466D19258E79C8665AFB072F6:1" [2]="0075B2ADF782F9677BDDF8CC5778900D986:8" [3]="00810352C02B145FF304FCBD5BEF4F7AA9F:1" [4]="013720F40B99D6BCF6F08BD7271523EBB49:1" [5]="01C894A55EBCB6048A745B37EC2D989F102:1" [6]="030B46D53696C8889910EE0F9EB42CEAE97:4" [7]="03126B56153F0564F4A6ED6E734604C67E8:2" [8]="043196099707FCB0C01F8C2111086A92A0B:1" [9]="0452CABBEF8438F358F0BD8089108D0910E:5" [10]="0490255B206A2C0377EBD080723BF72DDAE:2" [11]="05AD1141C41237E061460DB5CA5412C1A48:4" [12]="05C1D058E4439F8F005EFB32E7553E6AA5B:1" [13]="067AF515CC712AC4ACA012311979DBC7C9A:2")
Assuming OP wants to populate the array from the variable $temp, and assuming the white space in $temp consists of linefeeds and not spaces:
$ mapfile -t myArr <<< "${temp}"
$ typeset -p myArr
declare -a x=([0]="002C3CADF9FC86069F09961B10E0EDEFC45:1" [1]="003BE35BD9466D19258E79C8665AFB072F6:1" [2]="0075B2ADF782F9677BDDF8CC5778900D986:8" [3]="00810352C02B145FF304FCBD5BEF4F7AA9F:1" [4]="013720F40B99D6BCF6F08BD7271523EBB49:1" [5]="01C894A55EBCB6048A745B37EC2D989F102:1" [6]="030B46D53696C8889910EE0F9EB42CEAE97:4" [7]="03126B56153F0564F4A6ED6E734604C67E8:2" [8]="043196099707FCB0C01F8C2111086A92A0B:1" [9]="0452CABBEF8438F358F0BD8089108D0910E:5" [10]="0490255B206A2C0377EBD080723BF72DDAE:2" [11]="05AD1141C41237E061460DB5CA5412C1A48:4" [12]="05C1D058E4439F8F005EFB32E7553E6AA5B:1" [13]="067AF515CC712AC4ACA012311979DBC7C9A:2")
The file /tmp/file.csv contains the following:
name,age,gender
bob,21,m
jane,32,f
The CSV file will always have headers.. but might contain a different number of fields:
id,title,url,description
1,foo name,foo.io,a cool foo site
2,bar title,http://bar.io,a great bar site
3,baz heading,https://baz.io,some description
In either case, I want to convert my CSV data into an array of associative arrays..
What I need
So, I want a Bash 4.3 function that takes CSV as piped input and sends the array to stdout:
/tmp/file.csv:
name,age,gender
bob,21,m
jane,32,f
It needs to be used in my templating system, like this:
{{foo | csv_to_array | foo2}}
^ this is a fixed API, I must use that syntax .. foo2 must receive the array as standard input.
The csv_to_array func must do it's thing, so that afterwards I can do this:
$ declare -p row1; declare -p row2; declare -p new_array;
and it would give me this:
declare -A row1=([gender]="m" [name]="bob" [age]="21" )
declare -A row2=([gender]="f" [name]="jane" [age]="32" )
declare -a new_array=([0]="row1" [1]="row2")
..Once I have this array structure (an indexed array of associative array names), I have a shell-based templating system to access them, like so:
{{#new_array}}
Hi {{item.name}}, you are {{item.age}} years old.
{{/new_array}}
But I'm struggling to generate the arrays I need..
Things I tried:
I have already tried using this as a starting point to get the array structure I need:
while IFS=',' read -r -a my_array; do
echo ${my_array[0]} ${my_array[1]} ${my_array[2]}
done <<< $(cat /tmp/file.csv)
(from Shell: CSV to array)
..and also this:
cat /tmp/file.csv | while read line; do
line=( ${line//,/ } )
echo "0: ${line[0]}, 1: ${line[1]}, all: ${line[#]}"
done
(from https://www.reddit.com/r/commandline/comments/1kym4i/bash_create_array_from_one_line_in_csv/cbu9o2o/)
but I didn't really make any progress in getting what I want out the other end...
EDIT:
Accepted the 2nd answer, but I had to hack the library I am using to make either solution work..
I'll be happy to look at other answers, which do not export the declare commands as strings, to be run in the current env, but instead somehow hoist the resultant arrays of the declare commands to the current env (the current env is wherever the function is run from).
Example:
$ cat file.csv | csv_to_array
$ declare -p row2 # gives the data
So, to be clear, if the above ^ works in a terminal, it'll work in the library I'm using without the hacks I had to add (which involved grepping STDIN for ^declare -a and using source <(cat); eval $STDIN... in other functions)...
See my comments on the 2nd answer for more info.
The approach is straightforward:
Read the column headers into an array
Read the file line by line, in each line …
Create a new associative array and register its name in the array of array names
Read the fields and assign them according to the column headers
In the last step we cannot use read -a, mapfile, or things like these since they only create regular arrays with numbers as indices, but we want an associative array instead, so we have to create the array manually.
However, the implementation is a bit convoluted because of bash's quirks.
The following function parses stdin and creates arrays accordingly.
I took the liberty to rename your array new_array to rowNames.
#! /bin/bash
csvToArrays() {
IFS=, read -ra header
rowIndex=0
while IFS= read -r line; do
((rowIndex++))
rowName="row$rowIndex"
declare -Ag "$rowName"
IFS=, read -ra fields <<< "$line"
fieldIndex=0
for field in "${fields[#]}"; do
printf -v quotedFieldHeader %q "${header[fieldIndex++]}"
printf -v "$rowName[$quotedFieldHeader]" %s "$field"
done
rowNames+=("$rowName")
done
declare -p "${rowNames[#]}" rowNames
}
Calling the function in a pipe has no effect. Bash executes the commands in a pipe in a subshell, therefore you won't have access to the arrays created by someCommand | csvToArrays. Instead, call the function as either one of the following
csvToArrays < <(someCommand) # when input comes from a command, except "cat file"
csvToArrays < someFile # when input comes from a file
Bash scripts like these tend to be very slow. That's the reason why I didn't bother to extract printf -v quotedFieldHeader … from the inner loop even though it will do the same work over and over again.
I think the whole templating thing and everything related would be way easier to program and faster to execute in languages like python, perl, or something like that.
The following script:
csv_to_array() {
local -a values
local -a headers
local counter
IFS=, read -r -a headers
declare -a new_array=()
counter=1
while IFS=, read -r -a values; do
new_array+=( row$counter )
declare -A "row$counter=($(
paste -d '' <(
printf "[%s]=\n" "${headers[#]}"
) <(
printf "%q\n" "${values[#]}"
)
))"
(( counter++ ))
done
declare -p new_array ${!row*}
}
foo2() {
source <(cat)
declare -p new_array ${!row*} |
sed 's/^/foo2: /'
}
echo "==> TEST 1 <=="
cat <<EOF |
id,title,url,description
1,foo name,foo.io,a cool foo site
2,bar title,http://bar.io,a great bar site
3,baz heading,https://baz.io,some description
EOF
csv_to_array |
foo2
echo "==> TEST 2 <=="
cat <<EOF |
name,age,gender
bob,21,m
jane,32,f
EOF
csv_to_array |
foo2
will output:
==> TEST 1 <==
foo2: declare -a new_array=([0]="row1" [1]="row2" [2]="row3")
foo2: declare -A row1=([url]="foo.io" [description]="a cool foo site" [id]="1" [title]="foo name" )
foo2: declare -A row2=([url]="http://bar.io" [description]="a great bar site" [id]="2" [title]="bar title" )
foo2: declare -A row3=([url]="https://baz.io" [description]="some description" [id]="3" [title]="baz heading" )
==> TEST 2 <==
foo2: declare -a new_array=([0]="row1" [1]="row2")
foo2: declare -A row1=([gender]="m" [name]="bob" [age]="21" )
foo2: declare -A row2=([gender]="f" [name]="jane" [age]="32" )
The output comes from foo2 function.
The csv_to_array function first reads the headaers. Then for each read line it adds new element into new_array array and also creates a new associative array with the name row$index with elements created from joining the headers names with values read from the line. On the end the output from declare -p is outputted from the function.
The foo2 function sources the standard input, so the arrays come into scope for it. It outputs then those values again, prepending each line with foo2:.
What is the added value of declaring a bash array with the -a flag like below:
with flag
declare -a arr1
arr1[0]="array1-zero"
arr1[1]="array1-one"
echo ${arr1[0]}
echo ${arr1[1]}
...when the behaviour is the same if you omit the option, like below:
without -a option
declare arr2
arr2[0]="array2-zero"
arr2[1]="array2-one"
echo ${arr2[0]}
echo ${arr2[1]}
The same goes for associative arrays (option -A).
Omitting it renders the same result as providing it.
You can skip declare -a for numerically-indexed arrays, but you can't skip declare -A for associative arrays. To demonstrate the latter:
declare array
array[one]=1
array[two]=2
echo "${array[one]}"
...emits:
2
Why? Let's use declare -p array to see:
$ declare -p array
declare -a array='([0]="2")'
The words one and two were both coerced to the numeric value 0, so they overwrote the same key.
That said, declare -a array isn't a complete noop. Compare:
unset array # start from a blank slate
declare -a array
declare -p array
...emits:
declare -a array='()'
Whereas if we leave out the -a, we see that no type data is assigned, so our variable is assumed to just be a string until it's assigned to in a way that makes it clear that it's supposed to be an array:
unset array # start from a blank slate
declare array
declare -p array
...emits declare -- array="", unless there's as assignment:
unset array # start from a blank slate
declare array
array[0]="zero"
declare -p array
...which properly emits declare -a array='([0]="zero")', meaning that in the above code, only after the array[0]="zero" line does array actually become an array.
Given an array (in Bash), is there a command that prints the contents of the array according to indices?
Something like that: arr[0]=... , arr[1]=... ,...
I know that I can print it in a for loop, but I am looking for a command that does it.
Given an array with contiguous indices starting at 0:
$ arr=(one two three)
And non-contiguous indices:
$ declare -a arr2='([0]="one" [2]="two" [5]="three")'
You can print the values:
$ echo ${arr[*]} # same with arr2
one two three
Or, use a C style loop for arr:
$ for (( i=0;i<${#arr[#]};i++ )); do echo "arr[$i]=${arr[$i]}"; done
arr[0]=one
arr[1]=two
arr[2]=three
But that won't work for arr2.
So, you can expand the indices (contiguous or not) and print index, value like so:
$ for i in "${!arr2[#]}"; do echo "arr2[$i]=${arr2[$i]}"; done
arr2[0]=one
arr2[2]=two
arr2[5]=three
Or inspect it with declare -p:
$ declare -p arr
declare -a arr='([0]="one" [1]="two" [2]="three")'
Which also works if the array has non-contiguous indices (where the C loop would break):
$ declare -p arr2
declare -a arr2='([0]="one" [2]="two" [5]="three")'
Note: A common mistake is to use the sigil $ and thinking you are addressing that array or named value. It is the unadorned name that is used since the sigil will dereference and tell details of the name contained instead of that name:
$ k=arr
$ declare -p $k
declare -a arr='([0]="one" [1]="two" [2]="three")' # note this is 'arr' , not 'k'
$ declare -p k
declare -- k="arr"
Since declare with no arguments will print the entire Bash environment at that moment, you can also use utilities such as sed grep or awk against that output:
$ declare | grep 'arr'
arr=([0]="one" [1]="two" [2]="three")
k=arr