There is this typical problem: given a list of values, check if they are present in an array.
In awk, the trick val in array does work pretty well. Hence, the typical idea is to store all the data in an array and then keep doing the check. For example, this will print all lines in which the first column value is present in the array:
awk 'BEGIN {<<initialize the array>>} $1 in array_var' file
However, it is initializing the array takes some time because val in array checks if the index val is in array, and what we normally have stored in array is a set of values.
This becomes more relevant when providing values from command line, where those are the elements that we want to include as indexes of an array. For example, in this basic example (based on a recent answer of mine, which triggered my curiosity):
$ cat file
hello 23
bye 45
adieu 99
$ awk -v values="hello adieu" 'BEGIN {split(values,v); for (i in v) names[v[i]]} $1 in names' file
hello 23
adieu 99
split(values,v) slices the variable values into an array v[1]="hello"; v[2]="adieu"
for (i in v) names[v[i]] initializes another array names[] with names["hello"] and names["adieu"] with empty value. This way, we are ready for
$1 in names that checks if the first column is any of the indexes in names[].
As you see, we slice into a temp variable v to later on initialize the final and useful variable names[].
Is there any faster way to initialize the indexes of an array instead of setting one up and then using its values as indexes of the definitive?
No, that is the fastest (due to hash lookup) and most robust (due to string comparison) way to do what you want.
This:
BEGIN{split(values,v); for (i in v) names[v[i]]}
happens once on startup and will take close to no time while this:
$1 in array_var
which happens once for every line of input (and so is the place that needs to have optimal performance) is a hash lookup and so the fastest way to compare a string value to a set of strings.
not an array solution but one trick is to use pattern matching. To eliminate partial matches wrap the search and array values with the delimiter. For your example,
$ awk -v values="hello adieu" 'FS values FS ~ FS $1 FS' file
hello 23
adieu 99
I'm trying to assign an array of three values to a variable if it hasn't been assigned yet with the line
: ${SEAFILE_MYSQL_DB_NAMES:=(ccnet-db seafile-db seahub-db)}
Unfortunately, echoing ${SEAFILE_MYSQL_DB_NAMES[#]} results in (ccnet-db seafile-db seahub-db) and ${SEAFILE_MYSQL_DB_NAMES[2]} prints nothing. It seems, the value has been interpreted as a string and not as an array. Is there any way I can make my script assign an array this way?
How about doing it in several stages? First declare the fallback array, then check if SEAFILE_MYSQL_DB_NAMES is set, and assign if needed.
DBS=(ccnet-db seafile-db seahub-db)
[[ -v SEAFILE_MYSQL_DB_NAMES ]] || read -ra SEAFILE_MYSQL_DB_NAMES <<< ${DBS[#]}
Based on this answer.
I'm a total Perl newb, but still cannot believe I cannot figure this out with all the info I've read through online, but, I've burned too much time and am suffering from block at this point. Hoping to learn something based on my real life example...
Ok, I think I have an array of arrays, created like this:
my #array1 = ();
my #array2 = ();
my $ctr1 = 0;
my $col;
[sql query]
while(($col)=$sth->fetchrow_array() ) {
$array1[$ctr1]=$col;
$ctr1++;
}
print STDERR "#array1";
##results in 10 rows, a mac address in each
##00:00:00:00:00:00 00:11:11:11:11:11 22:22:22:22:22:22 33:33:33:33:33:33 ...
Now I do another query. While looping through results, I am looking for those 10 mac addresses. When I find one, I add a row to array2 with the mac and the sequential number accumulated to the point, like this:
[sql query]
while(($col)=$sth->fetchrow_array() ) {
$ctr2++;
if( my ($matched) = grep $_ eq $col, #array1 ) {
push( #array2, ($col,$ctr2) );
}
}
print STDERR "#array2";
##results in 10 rows, a mac address and an integer in each
##00:00:00:00:00:00 2 00:11:11:11:11:11 24 22:22:22:22:22:22 69 33:33:33:33:33:33 82 ...
Now the easy part. I want to loop through array2, grabbing the mac address to use as part of a sql query. Therein lies the problem. I am so ignorant as to exactly what I am doing that even though I had it almost working, I can't get back to that point. Ignorance is definitely not bliss.
When I loop through array2, I am getting a host of errors, based on the different forms of the statement. The one I think is right is listed below along with the error message...
my $ctr3 = 0;
foreach $ctr3 (#array2) {
my $chkmac = $array2[$ctr3][0]; <--- gacks here with the error below - line 607
[SQL query]
[Thu May 30 14:05:09 2013] [error] Can't use string ("00:66:55:77:99:88") as an ARRAY ref while "strict refs" in use at /path/to/test.cgi line 607.\n
I believe the issue is that my array of arrays is not an array of arrays. If it were, it would work as coded, or so I think from the reading... That said, I cannot fathom what I am dealing with otherwise. This will be a head slapper I'm all but sure, but I am stumped.... Little help, please?
TIA
O
For an array of arrays you want to use an array reference, e.g.
push #array2, [$col, $ctr2];
When accessing an element within an array refernce, you'll want to use the -> operator. Also, when looping through an array, it's not necessary to index back into that same array. So the last part would look more like:
foreach $ctr3 (#array2) {
my $chkmac = $ctr3->[0];
....
When you do the foreach there, $ctrl3 won't have the index in it, it'll have the value. So you should just need to do $ctrl3->[0]. Note the -> which dereferences the array reference (#array2 is actually an array of array references).
EDIT: As AKHolland pointed out, #array2 actually isn't an array of array references, although that's what it should be. You also need to change:
push( #array2, ($col, $ctr2) );
To
push( #array2, [$col, $ctr2] );
This makes an array reference, rather than a list. A list in this context just collapses down into regular arguments to push, meaning you're pushing two separate strings into #array2.
You are correct that your array of arrays is not an array of arrays, since in Perl there is no such thing. So what do you have instead? There's two ways to see.
First, when you print #array2, you come up with a string composed of alternating MACs and counts, separated by spaces. Since the spaces sort-of-signify the division between array elements, we might surmise that what we've got is a single array of heterogeneous elements, such that element 0 is a MAC, element 1 is a count, element 2 is another MAC, and so on.
The other perspective is to look at how #array2 is constructed:
push( #array2, ($col,$ctr2) );
From the documentation for push, we find that push ARRAY LIST works by appending the elements of LIST to the end of ARRAY. This has the effect of flattening the list into the array such that its original identity as a list is lost. You can add all the parentheses you want, when Perl expects a list it flattens all of them away.
So how do you achieve the effect you want? The List-of-Lists documentation has a detailed treatment, but the short answer is that you make a list of array references. Array references are scalars and are therefore legal elements in an array. But they retain their identify as array references!
The anonymous array reference constructor is the square bracket []. In order to push an array reference containing the elements $col and $ctr2 onto the end of #array2, you simply do this:
push( #array2, [$col, $ctr2] );
The code you wrote for accessing a particular element of the array-reference-in-an-array now works. But since I've already written a bunch of paragraphs on the subject, let me finish by explaining what was wrong originally and how changing the push statements suddenly makes it work.
The expression $array2[$ctr3][0] is sometimes written as $array2[$ctr3]->[0] to clarify what it's actually doing. What it does is to take the value of $array2[$ctr3] and treat it as an array reference, taking its 0 element. If we take $ctr3 to be 0 (as it would be at the top of the loop) the value of $array2[$ctr3] is the first element, 00:00:00:00:00:00. When you then subsequently ask Perl to treat 00:00:00:00:00:00 as an array reference, Perl dies because it doesn't know how to treat a string as an array reference.
When instead the value of $array2[$ctr3] is an array reference because that is what you pushed onto #array2 when constructing it, Perl is able to do as you ask, dereferencing the array reference and looking at element 0 of the resulting array, whose value happens to be 00:00:00:00:00:00.
I have the following code:
commit_hashes_raw=(`git cherry origin/Server_Dev`)
echo ${#commit_hashes_raw[#]}
echo ${commit_hashes_raw[#]}
that is producing the following output:
2
+ 6f0de9d07538db5d6428acd083c4a4527751596b
The first line is supposed to be the size of the array and the second is the contents. The obvious problem here is the discrepancy between the two values though. I've run this on another data set and a 50 element array was being reporting by the middle line as 100 elements long.
Am I using the wrong method for finding the size or is there something funky with my array?
Your array has two elements - the + and then the hash. When setting an array from command output, the elements are whitespace-separated, not newline-separated.
You could do this:
commit_hashes_raw=($(git cherry origin/Server_Dev | awk '{print $NF}'))
Or, somewhat less efficiently without calling out of the shell:
commit_hashes_raw=()
while read plus hash; do
commit_hashes_raw+=("$hash")
done < <(git cherry origin/Server_Dev)
It looks to me like the + symbols are adding extra elements to your array. You could try and filter them out when creating the array. Something like:
commit_hashes_raw=(`git cherry origin/Server_Dev | cut -d' ' -f2`)
I am writing a program and trying to break up data, which is stored in an array, in order to make it run faster.
I am attempting to go about it this way:
data_to_analyze=(1 2 3 4 5 6 7 8 9 10)
#original array size
dataSize=(${#data_to_analyze[#]})
#half of that size
let samSmall="$dataSize/2"
#the other half
let samSmall2=("$dataSize - $samSmall -1")
#the first half
smallArray=("${data_to_analyze[#]:0:$samSmall}")
#the rest
smallArray2=("${data_to_analyze[#]:$samSmall:$samSmall2}")
#an array of names(which correspond to arrays)
combArray=(smallArray smallArray2)
sizeComb=(${#combArray[#]})
#for the length of the new array
for ((i=0; i<= $sizeComb ; i++)); do
#through first set of data and then loop back around for the second arrays data?
for sample_name in ${combArray[i]}; do
command
wait
command
wait
done
What I imagine this does is gives only the first array of data to the for loop at first. When the first array is done it should go through again with the second array set.
That leaves me with two questions. Is combArray really passing the two smaller arrays? And is there a better way?
You can make a string that looks like an array reference then use it to indirectly access the elements of the referenced array. It even works for elements that contain spaces!
combArray=(smallArray smallArray2)
for array in "${combArray[#]}"
do
indirect=$array[#] # make a string that sort of looks like an array reference
for element in "${!indirect}"
do
echo "Element: $element"
done
done
#!/bin/bash
data_to_analyze=(1 2 3 4 5 6 7 8 9 10)
dataSize=${#data_to_analyze[#]}
((samSmall=dataSize/2,samSmall2=dataSize-samSmall))
smallArray=("${data_to_analyze[#]:0:$samSmall}")
smallArray2=("${data_to_analyze[#]:$samSmall:$samSmall2}")
combArray=(smallArray smallArray2)
sizeComb=${#combArray[#]}
for ((i=0;i<$sizeComb;i++));do
eval 'a=("${'${combArray[i]}'[#]}")'
for sample_name in "${a[#]}";do
...
done
done
EDIT: removed the double quotes near ${combArray[i]} and replaced <= by < in for