If you look at output of this awk test, you see that array in awk seems to be printed at some random pattern. It seems to be in same order for same number of input. Why does it do so?
echo "one two three four five six" | awk '{for (i=1;i<=NF;i++) a[i]=$i} END {for (j in a) print j,a[j]}'
4 four
5 five
6 six
1 one
2 two
3 three
echo "P04637 1A1U 1AIE 1C26 1DT7 1GZH 1H26 1HS5 1JSP 1KZY 1MA3 1OLG 1OLH 1PES 1PET 1SAE 1SAF 1SAK 1SAL 1TSR 1TUP 1UOL 1XQH 1YC5 1YCQ" | awk '{for (i=1;i<=NF;i++) a[i]=$i} END {for (j in a) print j,a[j]}'
17 1SAF
4 1C26
18 1SAK
5 1DT7
19 1SAL
6 1GZH
7 1H26
8 1HS5
9 1JSP
10 1KZY
20 1TSR
11 1MA3
21 1TUP
12 1OLG
22 1UOL
13 1OLH
23 1XQH
14 1PES
1 P04637
24 1YC5
15 1PET
2 1A1U
25 1YCQ
16 1SAE
3 1AIE
Why does it do so, is there rule for this?
From 8. Arrays in awk --> 8.5 Scanning All Elements of an Array in the GNU Awk user's guide when referring to the for (value in array) syntax:
The order in which elements of the array are accessed by this
statement is determined by the internal arrangement of the array
elements within awk and cannot be controlled or changed. This can lead
to problems if new elements are added to array by statements in the
loop body; it is not predictable whether or not the for loop will
reach them. Similarly, changing var inside the loop may produce
strange results. It is best to avoid such things.
So if you want to print the array in the order you store it, then you have to use the classical for loop:
for (j=1; j<=NF; j++) print j,a[j]
Example:
$ awk '{for (i=1;i<=NF;i++) a[i]=$i} END {for (j=1; j<=NF; j++) print j,a[j]}' <<< "P04637 1A1U 1AIE 1C26 1DT7 1GZH 1H26 1HS5 1JSP 1KZY 1MA3 1OLG 1OLH 1PES 1PET 1SAE 1SAF 1SAK 1SAL 1TSR 1TUP 1UOL 1XQH 1YC5 1YCQ"
1 P04637
2 1A1U
3 1AIE
4 1C26
5 1DT7
6 1GZH
7 1H26
8 1HS5
9 1JSP
10 1KZY
11 1MA3
12 1OLG
13 1OLH
14 1PES
15 1PET
16 1SAE
17 1SAF
18 1SAK
19 1SAL
20 1TSR
21 1TUP
22 1UOL
23 1XQH
24 1YC5
25 1YCQ
Awk uses hash tables to implement associative arrays. This is just an inherent property of this particular data structure. The location that a particular element is stored into the array depends on the hash of the value. Other factors to consider is the implementation of the hash table. If it is memory efficient, it will limit the range each key gets stored in using the modulus function or some other method. You also may get clashing hash values for different keys so chaining will occur, again affecting the order depending on which key was inserted first.
The construct (key in array) is perfectly fine when used appropriately to loop over every key but you cannot count on the order and you should not update array whilst in the loop as you may end up process array[key] multiple times by mistake.
There is a good decription of hash tables in the book Think Complexity.
The issue is the operator you use to get the array indices, not the fact that the array is stored in a hash table.
The in operator provides the array indices in a random(-looking) order (which IS by default related to the hash table but that's an implementation choice and can be modified).
A for loop that explicitly provides the array indices in a numerically increasing order also operates on the same hash table that the in operator on but that produces output in a specific order regardless.
It's just 2 different ways of getting the array indices, both of which work on a hash table.
man awk and look up the in operator.
If you want to control the output order using the in operator, you can do so with GNU awk (from release 4.0 on) by populating PROCINFO["sorted_in"]. See http://www.gnu.org/software/gawk/manual/gawk.html#Controlling-Array-Traversal for details.
Some common ways to access array indices:
To print array elements in an order you don't care about:
{a[$1]=$0} END{for (i in a) print i, a[i]}
To print array elements in numeric order of indices if the indices are numeric and contiguous starting at 1:
{a[++i]=$0} END{for (i=1;i in a;i++) print i, a[i]}
To print array elements in numeric order of indices if the indices are numeric but non-contiguous:
{a[$1]=$0; min=($1<min?$1:min); max=($1>max?$1:max)} END{for (i=min;i<=max;i++) if (i in a) print i, a[i]}
To print array elements in the order they were seen in the input:
{a[$1]=$0; b[++max]=$1} END{for (i=1;i <= max;i++) print b[i], a[b[i]]}
To print array elements in a specific order of indices using gawk 4.0+:
BEGIN{PROCINFO["sorted_in"]=whatever} {a[$1]=$0} END{for (i in a) print i, a[i]}
For anything else, write your own code and/or see gawk asort() and asorti().
If you are using gawk or mawk, you can also set an env variable WHINY_USERS, which will sort indices before iterating.
Example:
echo "one two three four five six" | WHINY_USERS=true awk '{for (i=1;i<=NF;i++) a[i]=$i} END {for (j in a) print j,a[j]}'
1 one
2 two
3 three
4 four
5 five
6 six
From mawk's manual:
WHINY_USERS
This is an undocumented gawk feature. It tells mawk to sort array indices before it starts to iterate over the elements of an array.
Related
The syntax to delete an element from an array can be found here: Remove an element from a Bash array
Also, here is how to find the last element of an array: https://unix.stackexchange.com/questions/198787/is-there-a-way-of-reading-the-last-element-of-an-array-with-bash
But how can I mix them (if possible) together to remove the last element of the array ?
I tried this:
TABLE_COLUMNS=${TABLE_COLUMNS[#]/${TABLE_COLUMNS[-1]}}
But it throws:
bad array subscript
You can use unset to remove a specific element of an array given its position.
$ foo=(1 2 3 4 5)
$ printf "%s\n" "${foo[#]}"
1
2
3
4
5
$ unset 'foo[-1]'
$ printf "%s\n" "${foo[#]}"
1
2
3
4
Edit: This is useful for printing elements except the last without altering the array. See chepner's answer for a far more convenient solution to OP.
Substring expansions* could be used on arrays for extracting subarrays, like:
TABLE_COLUMNS=("${TABLE_COLUMNS[#]::${#TABLE_COLUMNS[#]}-1}")
* The syntax is:
${parameter:offset:length}
Both offset and length are arithmetic expressions, an empty offset implies 1. Used on array expansions (i.e. when parameter is an array name subscripted with * or #), the result is at most length elements starting from offset.
Could any one please let me know, how the hash function works in tcl arrays.
for {set i 1} {$i<21} {incr i} {
set a($i) $i
}
set a(30) $i
puts [lindex [split [array statistics a] \n] 0]
puts [array names a]
for this program i am getting the output as below.
21 entries in table, 16 buckets
17 18 1 20 19 2 3 4 5 6 7 8 9 10 11 30 12 13 14 15 16
i want to know, in which order the keys are printing here.
The array names command produces the keys of the array in “arbitrary” order. It's actually the current natural iteration order of the underlying hash table, which can change as the array grows, and is actually extremely difficult to explain other than by reference to the implementation and the history of the array. Because it is so difficult, we just tend to say “random”; it's not, but it might as well be (for purposes other than generating random numbers; it'd be a terrible PRNG). It's always worked like this. The other array subcommands will use the same order, especially array get and the very-rarely-used iterator subcommands.
If you want a particular order, you have to sort them yourself. The lsort command might be useful for that. Or you can use a dict; those maintain insertion order (but are technically values, and so you can't trace an element or upvar to it).
Here is an example what I need:
INPUT:
a 5
a 7
a 11
b 10
b 11
b 12
.
.
.
OUTPUT:
a 2
b 0
So on output should be hits lower than my threshold (in this case it is $2 < 10).
My code is:
awk 'OFS="\t" {v[$1]+=$2; n[$1]++} END {for (l in n) {print l, n[l]} }' input
and my output is
a 3
b 3
I am not sure where to put condition $2 < 10.
You can check the threshold condition with something like $2 < value, where value is an awk variable given with -v value=XX.
Also, you are using v[$1]+=$2: this sums, not counts the matching cases.
All together, I would use this:
awk -v t=10 '{list[$1]} $2<t {count[$1]++} END {for (i in list) print i, count[i]+0}' file
Note we need to use two arrays: one to keep track of the counters and another one the keep track of all possible values.
Explanation
-v t=10 provide threshold.
{list[$1]} keep track of all possible first fields appearing.
$2<t {count[$1]++} if the 2nd field is smaller than the threshold, increment the counter.
END {for (i in list) print i, count[i]+0} finally, loop through all the first fields and print the number of times they had a value lower than the threshold. The count[i]+0 trick makes it print 0 if the value is not set.
Test
$ awk -v t=10 '{list[$1]} $2<t {count[$1]++} END {for (i in list) print i, count[i]+0}' a
a 2
b 0
I'm trying to figure out the right way to pull this off with awk (not very familiar with awk) but I can't seem to get it.
Basically I have a text file with two columns. I want to sum up the second column and then divide each entry of the second column by the sum and increment a counter if the result is less than 0.25. In order to do this it seems like I have to loop through twice, once to get the sum and once to evaluate each entry with the sum. How can I pull this off with a one-liner?
Example Input:
0 5
1 5
2 10
3 5
Example Output:
3 (the sum is 25 and three of the entries result in a value less than 0.25 when divided by 25)
I got stuck trying to do this in bash and realizing I need to use awk to deal with decimals. I can loop through and get the sum and loop through and do a conditional check on each entry but I don't understand how to do both simultaneously.
Untested:
awk '
{ sum+=$2 ; row[NR]=$2 }
END{ for(i=1;i<=NR;i++) if (row[i]/sum < 0.25) {counter+=1}; print counter }
' file
Using awk
$ awk '{sum+=$2;a[NR]=$2}END{for (i in a) if (a[i]/sum<0.25) count++;print count}' file
Explanation
sum+=$2 get the summary on column 2 and save to sum
a[NR]=$2 record column 2 into array a (NR, the line number, as index)
i in a get index from array a one by one
if (a[i]/sum<0.25) count++ do the calculate and increase count with the condition (<0.25)
I am writing a program and trying to break up data, which is stored in an array, in order to make it run faster.
I am attempting to go about it this way:
data_to_analyze=(1 2 3 4 5 6 7 8 9 10)
#original array size
dataSize=(${#data_to_analyze[#]})
#half of that size
let samSmall="$dataSize/2"
#the other half
let samSmall2=("$dataSize - $samSmall -1")
#the first half
smallArray=("${data_to_analyze[#]:0:$samSmall}")
#the rest
smallArray2=("${data_to_analyze[#]:$samSmall:$samSmall2}")
#an array of names(which correspond to arrays)
combArray=(smallArray smallArray2)
sizeComb=(${#combArray[#]})
#for the length of the new array
for ((i=0; i<= $sizeComb ; i++)); do
#through first set of data and then loop back around for the second arrays data?
for sample_name in ${combArray[i]}; do
command
wait
command
wait
done
What I imagine this does is gives only the first array of data to the for loop at first. When the first array is done it should go through again with the second array set.
That leaves me with two questions. Is combArray really passing the two smaller arrays? And is there a better way?
You can make a string that looks like an array reference then use it to indirectly access the elements of the referenced array. It even works for elements that contain spaces!
combArray=(smallArray smallArray2)
for array in "${combArray[#]}"
do
indirect=$array[#] # make a string that sort of looks like an array reference
for element in "${!indirect}"
do
echo "Element: $element"
done
done
#!/bin/bash
data_to_analyze=(1 2 3 4 5 6 7 8 9 10)
dataSize=${#data_to_analyze[#]}
((samSmall=dataSize/2,samSmall2=dataSize-samSmall))
smallArray=("${data_to_analyze[#]:0:$samSmall}")
smallArray2=("${data_to_analyze[#]:$samSmall:$samSmall2}")
combArray=(smallArray smallArray2)
sizeComb=${#combArray[#]}
for ((i=0;i<$sizeComb;i++));do
eval 'a=("${'${combArray[i]}'[#]}")'
for sample_name in "${a[#]}";do
...
done
done
EDIT: removed the double quotes near ${combArray[i]} and replaced <= by < in for