An awk command creates an array which I want to return to bash.
GIT=($(history | grep -c git | awk '{ TMP[4]=$1 } END { print TMP }'))
and getting an error attempt to use array TMP in a scalar context.
So I try inside an awk use already existed array
GIT=()
history | grep -c git | awk 'END { GIT[4]=$1 }'
echo "${GIT[#]}" # empty result
but the result GIT arrary is empty.
The example which I presented doesn't make a lot of sense eg. why 4 in an index? It's just an example. I need use array/map created by awk where the key is an integer. And I need these original indexes, because not all keys are in sequence. I have 1,2,4,7,8 and so on.
You can have awk emit shell commands and then the awk output can be sourced
GIT=()
source <(
history |
grep -c git |
awk -v shellVarname="GIT" '
{tmp[4] = $1}
END {
for (idx in tmp)
printf "%s[%d]=\"%s\"\n", shellVarname, idx, tmp[idx]
}
'
)
declare -p GIT # just to dump the array contents for review
I have been trying to split up a string and putting it into an Array in Bash on my Mac without success.
Here is my sample code:
#!/bin/bash
declare -a allDisks
allDisksString="`ls /dev/disk* | grep -e 'disk[0-9]s.*' | awk '{ print $NF }'`"
#allDisksString="/dev/disk0s1 /dev/disk1s1"
echo allDisksString is $allDisksString
IFS=' ' read -ra allDisks <<< "$allDisksString"
echo allDIsks is "$allDisks"
echo The second item in allDisks is "${allDisks[1]}"
for disk in "${allDisks[#]}"
do
printf "Loop $disk\n"
done
And below is the output:
allDisksString is /dev/disk0s1 /dev/disk0s2 /dev/disk0s3 /dev/disk0s4 /dev/disk1s1
allDIsks is /dev/disk0s1
The second item in allDisks is
Loop /dev/disk0s1
Interesting if I execute the following in the Mac Terminal:
ls /dev/disk* | grep -e 'disk[0-9]s.*' | awk '{ print $NF }'
I get the following output
/dev/disk0s1
/dev/disk0s2
/dev/disk0s3
/dev/disk0s4
/dev/disk1s1
So I have also tried setting IFS to IFS=$'\n' without any success.
So no luck in getting a list of my drives into an array.
Any ideas on what I am doing wrong?
You're making this much more complicated than it needs to be. You don't need to use ls, you can just use a wildcard to match the device names you want, and put that in an array assignment.
#!/bin/bash
declare -a allDisks
allDisks=(/dev/disk[0-9]s*)
echo allDIsks is "$allDisks"
echo The second item in allDisks is "${allDisks[1]}"
for disk in "${allDisks[#]}"
do
printf "Loop $disk\n"
done
read only reads one line.
Use an assignment instead. When assigning to an array, you need to use parentheses after the = sign:
#!/bin/bash
disks=( $(ls /dev/disk* | grep -e 'disk[0-9]s.*' | awk '{ print $NF }') )
echo ${disks[1]}
I'm tying to identify disks based on a list of UUID I get out of an api call.
The list of devices on the linux VM is following:
NAME="sda" SERIAL="NUTANIX_NFS_5_0_70509_a07b6add_60c9_4758_8b27_0afe77820dbd"
NAME="sdb" SERIAL="NUTANIX_NFS_18_0_625_ae748138_9bc0_4499_8068_d00a84a800b6"
NAME="sdc" SERIAL="NUTANIX_NFS_18_0_626_8a082956_a2be_42ca_a14b_7d21ed3d5a2d"
NAME="sdd" SERIAL="NUTANIX_NFS_18_0_627_6185353f_5c13_47ab_af58_b72d4d07106b"
NAME="sde" SERIAL="NUTANIX_NFS_18_0_628_fbeed366_6956_4de1_a217_487d75fd003c"
NAME="sdf" SERIAL="NUTANIX_NFS_18_0_629_019dcefb_0e1b_4086_96a9_982ae52fd88a"
NAME="sdg" SERIAL="NUTANIX_NFS_18_0_630_5d0b55e8_d0bd_4ee4_a2be_0bef88d94732"
NAME="sdh" SERIAL="NUTANIX_NFS_18_0_631_823beaed_1c2e_4203_ab5f_be294382d0c5"
NAME="sdi" SERIAL="NUTANIX_NFS_18_0_632_e4ee8620_5b53_4187_8f8c_7123ee8d3486"
NAME="sdj" SERIAL="NUTANIX_NFS_18_0_633_41551a1b_7dd9_4fd4_95bc_1fcdebb9dc85"
NAME="sdk" SERIAL="NUTANIX_NFS_18_0_634_12d517fc_e726_4512_b176_5d4075d0ecfe"
NAME="sdl" SERIAL="NUTANIX_NFS_18_0_635_d682eee4_cec0_4809_9efe_97aec7bf15d0"
NAME="sdm" SERIAL="NUTANIX_NFS_18_0_636_59ea679e_cbd3_4cf5_a974_67409b719391"
NAME="sdn" SERIAL="NUTANIX_NFS_18_0_637_5523c88e_310a_4fb6_b76f_48f624d1ce1f"
NAME="sdo" SERIAL="NUTANIX_NFS_18_0_638_0e189f7a_785d_46f1_bb56_8f422d421ecb"
NAME="sdp" SERIAL="NUTANIX_NFS_18_0_639_9143ba87_d742_4130_bbe0_2084e8ebad3f"
NAME="sdq" SERIAL="NUTANIX_NFS_18_0_640_3b823817_c2b6_49af_9a55_fc4706216501"
The list of drives I want to identify is in this format:
'ae748138_9bc0_4499_8068_d00a84a800b6'
'8a082956_a2be_42ca_a14b_7d21ed3d5a2d'
'6185353f_5c13_47ab_af58_b72d4d07106b'
'fbeed366_6956_4de1_a217_487d75fd003c'
'019dcefb_0e1b_4086_96a9_982ae52fd88a'
'5d0b55e8_d0bd_4ee4_a2be_0bef88d94732'
'823beaed_1c2e_4203_ab5f_be294382d0c5'
'e4ee8620_5b53_4187_8f8c_7123ee8d3486'
'41551a1b_7dd9_4fd4_95bc_1fcdebb9dc85'
'12d517fc_e726_4512_b176_5d4075d0ecfe'
'd682eee4_cec0_4809_9efe_97aec7bf15d0'
'59ea679e_cbd3_4cf5_a974_67409b719391'
'5523c88e_310a_4fb6_b76f_48f624d1ce1f'
'0e189f7a_785d_46f1_bb56_8f422d421ecb'
'9143ba87_d742_4130_bbe0_2084e8ebad3f'
'3b823817_c2b6_49af_9a55_fc4706216501'
I need to identify the logical name for only the UUIDs on that list in order to create a LVM volume group out of them. But I'm don't understand how to proceed from there. Anyone can propose an example ?
After removing the 's from you list of UUIDs (can be done with tr) you could use simply grep to extract the corresponding lines from your list of devices. To extract only the values of NAME="..." from these lines use cut.
grep -Ff <(tr -d \' < fileListOfUUIDs) fileListOfDevices | cut -d\" -f2
If the inputs you showed us are not stored in files but generated by commands you can use
commandListOfDevices | grep -Ff <(commandListOfUUIDs | tr -d \') | cut -d\" -f2
You are trying to extract field values from match results generated by a set of tokens. sed or awk can do the trick.
To extract the value of a field called NAME from one of the match results simply do this:
josh#linux ~ $ result='NAME="sda" FSTYPE="ext4" SERIAL="NUTANIX_...dbd"'
josh#linux ~ $ echo $result | sed -e 's/.*NAME="//; s/".*//;'
sda
To extract the value of a field called FSTYPE from the same match result:
josh#linux ~ $ echo $result | sed -e 's/.*FSTYPE="//; s/".*//;'
ext4
What if the order of the field changes? ..well the same code works.
josh#linux ~ $ result='FSTYPE="ext4" NAME="sda" SERIAL="NUTANIX_...dbd"'
josh#linux ~ $ echo $result | sed -e 's/.*NAME="//; s/".*//;'
sda
With this in mind, extraction of field values from all match results can then be processed in a loop. I have created a small script to automate the process and I am posting it here in the hope that it would be useful.
#!/bin/bash
#
# usage: doodle -F <file-containing-pattern(s)>
# doodle -P <pattern(s)>
#
if [ $# -ne 2 ]; then
echo 'invalid argument...'; exit;
else
pattern=
fi
if [ "$1" == "-F" ]; then
if [ ! -r $2 ]; then
echo "unable to read - $2"; exit
else
pattern=$(cat $2)
fi
elif [ "$1" == "-P" ]; then
pattern=$2
else
echo 'invalid argument...'; exit;
fi
if [ -z "$pattern" ]; then
echo 'invalid argument...'; exit;
fi
declare -a buffer
readarray buffer
if [ ${#buffer[*]} -gt 0 ]; then
for token in $pattern; do
for i in ${!buffer[*]}; do
match=$(echo ${buffer[i]} | sed -n "/$token/=");
if [ -n "$match" ]; then
##### Option 1 ##############################################################
# Uncomment the next line to print the same result that 'grep' would print.
#echo -e $(echo ${buffer[i]} | sed "s/$token/\\\\e[91m$token\\\\e[0m/");
##### Option 2 ##############################################################
# Uncomment the next line to print characters before the first [:space:]
#echo ${buffer[i]} | awk -F' ' '{ print $1 }';
##### Option 3 ##############################################################
# Uncomment the next line to email device-id to NSA
#TODO...
##### Option 4 ##############################################################
# Uncomment the next line to extract value of 'NAME' field
echo ${buffer[i]} | sed -e 's/.*NAME="//; s/".*//;'
##### Additional Option #####################################################
# Uncomment the next line (break command) to process one match per token
break;
fi
done
done
fi
unset buffer
To use this script, simply:
Copy the text in the above snippet into a file called doodle.
Open terminal and type:
josh#linux ~ $ chmod a+x doodle
To process tokens saved in a file called device-id.txt:
josh#linux ~ $ lsblk -nodeps -no name,serial -PS /dev/sd* | ./doodle -F device-ids.txt
To process tokens supplied as a single string:
josh#linux ~ $ lsblk -nodeps -no name,serial -PS /dev/sd* | ./doodle -P 'ae748138_9bc0_4499_8068_d00a84a800b6
8a082956_a2be_42ca_a14b_7d21ed3d5a2d
6185353f_5c13_47ab_af58_b72d4d07106b
...
3b823817_c2b6_49af_9a55_fc4706216501'
The script is very generic and you an can easily adapt it for use in related problems. You will find lots of tutorials on bash, sed and awk on the internet. There is also a comprehensive bash reference at gnu.org.
I parsed a json file with jq like this :
# cat test.json | jq '.logs' | jq '.[]' | jq '._id' | jq -s
It returns an array like this : [34,235,436,546,.....]
Using bash script i described an array :
# declare -a msgIds = ...
This array uses () instead of [] so when I pass the array given above to this array it won't work.
([324,32,45..]) this causes problem. If i remove the jq -s, an array forms with only 1 member in it.
Is there a way to solve this issue?
We can solve this problem by two ways. They are:
Input string:
// test.json
{
"keys": ["key1","key2","key3"]
}
Approach 1:
1) Use jq -r (output raw strings, not JSON texts) .
KEYS=$(jq -r '.keys' test.json)
echo $KEYS
# Output: [ "key1", "key2", "key3" ]
2) Use #sh (Converts input string to a series of space-separated strings). It removes square brackets[], comma(,) from the string.
KEYS=$(<test.json jq -r '.keys | #sh')
echo $KEYS
# Output: 'key1' 'key2' 'key3'
3) Using tr to remove single quotes from the string output. To delete specific characters use the -d option in tr.
KEYS=$((<test.json jq -r '.keys | #sh')| tr -d \')
echo $KEYS
# Output: key1 key2 key3
4) We can convert the comma-separated string to the array by placing our string output in a round bracket().
It also called compound Assignment, where we declare the array with a bunch of values.
ARRAYNAME=(value1 value2 .... valueN)
#!/bin/bash
KEYS=($((<test.json jq -r '.keys | #sh') | tr -d \'\"))
echo "Array size: " ${#KEYS[#]}
echo "Array elements: "${KEYS[#]}
# Output:
# Array size: 3
# Array elements: key1 key2 key3
Approach 2:
1) Use jq -r to get the string output, then use tr to delete characters like square brackets, double quotes and comma.
#!/bin/bash
KEYS=$(jq -r '.keys' test.json | tr -d '[],"')
echo $KEYS
# Output: key1 key2 key3
2) Then we can convert the comma-separated string to the array by placing our string output in a round bracket().
#!/bin/bash
KEYS=($(jq -r '.keys' test.json | tr -d '[]," '))
echo "Array size: " ${#KEYS[#]}
echo "Array elements: "${KEYS[#]}
# Output:
# Array size: 3
# Array elements: key1 key2 key3
To correctly parse values that have spaces, newlines (or any other arbitrary characters) just use jq's #sh filter and bash's declare -a. (No need for a while read loop or any other pre-processing)
// foo.json
{"data": ["A B", "C'D", ""]}
str=$(jq -r '.data | #sh' foo.json)
declare -a arr="($str)" # must be quoted like this
$ declare -p arr
declare -a arr=([0]="A B" [1]="C'D" [2]="")
The reason that this works correctly is that #sh will produce a space-separated list of shell-quoted words:
$ echo "$str"
'A B' 'C'\''D' ''
and this is exactly the format that declare expects for an array definition.
Use jq -r to output a string "raw", without JSON formatting, and use the #sh formatter to format your results as a string for shell consumption. Per the jq docs:
#sh:
The input is escaped suitable for use in a command-line for a POSIX shell. If the input is an array, the output will be a series of space-separated strings.
So can do e.g.
msgids=($(<test.json jq -r '.logs[]._id | #sh'))
and get the result you want.
From the jq FAQ (https://github.com/stedolan/jq/wiki/FAQ):
𝑸: How can a stream of JSON texts produced by jq be converted into a bash array of corresponding values?
A: One option would be to use mapfile (aka readarray), for example:
mapfile -t array <<< $(jq -c '.[]' input.json)
An alternative that might be indicative of what to do in other shells is to use read -r within a while loop. The following bash script populates an array, x, with JSON texts. The key points are the use of the -c option, and the use of the bash idiom while read -r value; do ... done < <(jq .......):
#!/bin/bash
x=()
while read -r value
do
x+=("$value")
done < <(jq -c '.[]' input.json)
++ To resolve this, we can use a very simple approach:
++ Since I am not aware of you input file, I am creating a file input.json with the following contents:
input.json:
{
"keys": ["key1","key2","key3"]
}
++ Use jq to get the value from the above file input.json:
Command: cat input.json | jq -r '.keys | #sh'
Output: 'key1' 'key2' 'key3'
Explanation: | #sh removes [ and "
++ To remove ' ' as well we use tr
command: cat input.json | jq -r '.keys | #sh' | tr -d \'
Explanation: use tr delete -d to remove '
++ To store this in a bash array we use () with `` and print it:
command:
KEYS=(`cat input.json | jq -r '.keys | #sh' | tr -d \'`)
To print all the entries of the array: echo "${KEYS[*]}"
The below script:
#!/bin/bash
otscurrent="
AAA,33854,4528,38382,12
BBB,83917,12296,96213,13
CCC,20399,5396,25795,21
DDD,27198,4884,32082,15
EEE,2472,981,3453,28
FFF,3207,851,4058,21
GGG,30621,4595,35216,13
HHH,8450,1504,9954,15
III,4963,2157,7120,30
JJJ,51,59,110,54
KKK,87,123,210,59
LLL,573,144,717,20
MMM,617,1841,2458,75
NNN,234,76,310,25
OOO,12433,1908,14341,13
PPP,10627,1428,12055,12
QQQ,510,514,1024,50
RRR,1361,687,2048,34
SSS,1,24,25,96
TTT,0,5,5,100
UUU,294,1606,1900,85
"
IFS="," array1=(${otscurrent})
echo ${array1[4]}
Prints:
$ ./test.sh
12
BBB
I'm trying to get it to just print 12... And I am not even sure how to make it just print row 5 column 4
The variable is an output of a sqlquery that has been parsed with several sed commands to change the formatting to csv.
otscurrent="$(sqlplus64 user/password#dbserverip/db as sysdba #query.sql |
sed '1,11d; /^-/d; s/[[:space:]]\{1,\}/,/g; $d' |
sed '$d'|sed '$d'|sed '$d' | sed '$d' |
sed 's/Used,MB/Used MB/g' |
sed 's/Free,MB/Free MB/g' |
sed 's/Total,MB/Total MB/g' |
sed 's/Pct.,Free/Pct. Free/g' |
sed '1b;/^Name/d' |
sed '/^$/d'
)"
Ultimately I would like to be able to call on a row and column and run statements on the values.
Initially i was piping that into :
awk -F "," 'NR>1{ if($5 < 10) { printf "%-30s%-10s%-10s%-10s%-10s\n", $1,$2,$3,$4,$5"%"; } else { echo "Nothing to do" } }')"
Which works but I couldn't run commands from if else ... or atleaste I didn't know how.
If you have bash 4.0 or newer, an associative array is an appropriate way to store data in this kind of form.
otscurrent=${otscurrent#$'\n'} # strip leading newline present in your sample data
declare -A data=( )
row=0
while IFS=, read -r -a line; do
for idx in "${!line[#]}"; do
data["$row,$idx"]=${line[$idx]}
done
(( row += 1 ))
done <<<"$otscurrent"
This lets you access each individual item:
echo "${data[0,0]}" # first field of first line
echo "${data[9,0]}" # first field of tenth line
echo "${data[9,1]}" # second field of tenth line
"I'm trying to get it to just print 12..."
The issue is that IFS="," splits on commas and there is no comma between 12 and BBB. If you want those to be separate elements, add a newline to IFS. Thus, replace:
IFS="," array1=(${otscurrent})
With:
IFS=$',\n' array1=(${otscurrent})
Output:
$ bash test.sh
12
All you need to print the value of the 4th column on the 5th row is:
$ awk -F, 'NR==5{print $4}' <<< "$otscurrent"
3453
and just remember that in awk row (record) and column (field) numbers start at 1, not 0. Some more examples:
$ awk -F, 'NR==1{print $5}' <<< "$otscurrent"
12
$ awk -F, 'NR==2{print $1}' <<< "$otscurrent"
BBB
$ awk -F, '$5 > 50' <<< "$otscurrent"
JJJ,51,59,110,54
KKK,87,123,210,59
MMM,617,1841,2458,75
SSS,1,24,25,96
TTT,0,5,5,100
UUU,294,1606,1900,85
If you'd like to avoid all of the complexity and simply parse your SQL output to produce what you want without 20 sed commands in between, post a new question showing the raw sqlplus output as the input and what you want finally output and someone will post a brief, clear, simple, efficient awk script to do it all at one time, or maybe 2 commands if you still want an intermediate CSV for some reason.