How to pass bash parameter to awk script? - file

I have awk file:
#!/bin/awk -f
BEGIN {
}
{
filetime[$'$colnumber']++;
}
END {
for (i in filetime) {
print filetime[i],i;
}
}
And bash script:
#!/bin/bash
var1=$1
awk -f myawk.awk
When I run:
ls -la | ./countPar.sh 5
I receive error:
ls -la | ./countPar.sh 5
awk: myawk.awk:6: filetime[$'$colnumber']++;
awk: myawk.awk:6: ^ invalid char ''' in expression
Why? $colnumber must be replaced with 5, so awk should read 5th column of ls ouput.
Thanks.

You can pass variables to your awk script directly from the command line.
Change this line:
filetime[$'$colnumber']++;
To:
filetime[colnumber]++;
And run:
ls -al | awk -f ./myawk.awk -v colnumber=5
If you really want to use a bash wrapper:
#!/bin/bash
var1=$1
awk -f myawk.awk colnumber=$var1
(with the same change in your script as above.)
If you want to use environment variables use:
#!/bin/bash
export var1=$1
awk -f myawk.awk
and:
filetime[ENVIRON["var1"]]++;
(I really don't understand what the purpose of your awk script is though. The last part could be simplified to:
END { print filetime[colnumber],colnumber; }
and parsing the output of ls is generally a bad idea.)

The easiest way to do it:
#!/bin/bash
var=$1
awk -v colnumber="${var}" -f /your/script
But within your awk script, you don't need the $ in front of colnumber.
HTH

Passing 3 variable to script myscript.sh
var1 is the column number on which condition has set.
While var2 & var3 are input and temp file.
#!/bin/ksh
var1=$1
var2=$2
var3=$3
awk -v col="${var1}" -f awkscript.awk ${var2} > $var3
mv ${var3} ${var2}
execute it like below -
./myscript.sh 2 file.txt temp.txt

Related

Return an array/map from awk to bash

An awk command creates an array which I want to return to bash.
GIT=($(history | grep -c git | awk '{ TMP[4]=$1 } END { print TMP }'))
and getting an error attempt to use array TMP in a scalar context.
So I try inside an awk use already existed array
GIT=()
history | grep -c git | awk 'END { GIT[4]=$1 }'
echo "${GIT[#]}" # empty result
but the result GIT arrary is empty.
The example which I presented doesn't make a lot of sense eg. why 4 in an index? It's just an example. I need use array/map created by awk where the key is an integer. And I need these original indexes, because not all keys are in sequence. I have 1,2,4,7,8 and so on.
You can have awk emit shell commands and then the awk output can be sourced
GIT=()
source <(
history |
grep -c git |
awk -v shellVarname="GIT" '
{tmp[4] = $1}
END {
for (idx in tmp)
printf "%s[%d]=\"%s\"\n", shellVarname, idx, tmp[idx]
}
'
)
declare -p GIT # just to dump the array contents for review

Parse variables from string and add them to an array with Bash

In Bash, how can I get the strings between acolades (without the '_value' suffix) from for example
"\\*\\* ${host_name_value}.${host_domain_value} - ${host_ip_value}\\*\\*"
and put them into an array?
The result for the above example should be something like:
var_array=("host_name" "host_domain")
The string could also contain other stuff such as:
"${package_updates_count_value} ${package_updates_type_value} updates"
The result for the above example should be something like:
var_array=("package_updates_count" "package_updates_type")
All variables end with _value. There could 1 or more variables in the string.
Not sure what would be the most efficient way and how I'd best handle this. Regex? Sed?
input='\\*\\* ${host_name_value}.${host_domain_value} \\*\\*'
# would also work with cat input or the like.
myarray=($(echo "$input" | awk -F'$' \
'{for(i=1;i<=NF;i++) {match($i, /{([^}]*)_value}/, a); print a[1]}}'))
Split your line(s) on $. Check if a column contains { }. If it does, print what's after { and before _value}. (If not, it will print out the empty string, which bash array creation will ignore.)
If there are only two variables, this will work.
input='\\*\\* ${host_name_value}.${host_domain_value} \\*\\*'
first=$(echo $input | sed -r -e 's/[}].+//' -e 's/.+[{]//')
last=$(echo $input | sed -r -e 's/.+[{]//' -e 's/[}].+//')
output="var_array=(\"$first\" \"$last\")"
Maybe not very efficient and beautiful, but it works well.
Starting with a string variable:
$ str='\\*\\* ${host_name_value}.${host_domain_value} - ${host_ip_value}\\*\\*'
Use grep -o to print all matching words.
$ grep -o '\${\w*_value}' <<< "$str"
${host_name_value}
${host_domain_value}
${host_ip_value}
Then remove ${ and _value}.
$ grep -o '\${\w*_value}' <<< "$str" | sed 's/^\${//; s/_value}$//'
host_name
host_domain
host_ip
Finally, use readarray to safely read the results into an array.
$ readarray -t var_array < <(grep -o '\${\w*_value}' <<< "$str" | sed 's/^\${//; s/_value}$//')
$ declare -p var_array
declare -a var_array=([0]="host_name" [1]="host_domain" [2]="host_ip")

write a shell script with a loop

i have an awk command as below :
awk 'FNR==NR{a[$1$2]=$3;next} ($1$2 in a) {print$1,$2,a[$1$2],$3}' ""each line of file 1"" >./awkfile/12as_132l.txt
and f1.txt content is :
1sas.txt 12ds.txt
13sa.txt 21sa.txt
i want that my script read each line of fil1.txt and put the contents in this awk command...instead of """each line of file1"".. and execute the command like below:
awk 'FNR==NR{a[$1$2]=$3;next} ($1$2 in a) {print$1,$2,a[$1$2],$3}' 1sas.txt 12ds.txt >./awkfile/12as_132l.txt
awk 'FNR==NR{a[$1$2]=$3;next} ($1$2 in a) {print$1,$2,a[$1$2],$3}' 13sa.txt 21sa.txt >./awkfile/12as_132l.txt
i need a loop but the problem is that's a little strange for me .
Here is Snippet, which reads 2 fields line by line from f1.txt, put it into variable, and used in awk
#!/usr/bin/env bash
while read -r col1file col2file; do
# your command goes here
# this awk does nothing, replace it with your command
awk '{ }' "$col1file" "$col2file" > someoutfile
done < "f1.txt"
Test Results:
akshay#db-3325:/tmp$ cat f1.txt
1sas.txt 12ds.txt
13sa.txt 21sa.txt
akshay#db-3325:/tmp$ cat test.sh
while read -r col1file col2file; do
echo "col1 : $col1file"
echo "col2 : $col2file"
# your command goes here
# this awk does nothing, replace it with your command
# awk '{ }' "$col1file" "$col2file" > someoutfile
done < "f1.txt"
akshay#db-3325:/tmp$ bash test.sh
col1 : 1sas.txt
col2 : 12ds.txt
col1 : 13sa.txt
col2 : 21sa.txt
change the IFS to IFS=' ' and use loop .

Using array inside awk in shell script

I am very new to Unix shell script and trying to get some knowledge in shell scripting. Please check my requirement and my approach.
I have a input file having data
ABC = A:3 E:3 PS:6
PQR = B:5 S:5 AS:2 N:2
I am trying to parse the data and get the result as
ABC
A=3
E=3
PS=6
PQR
B=5
S=5
AS=2
N=2
The values can be added horizontally and vertically so I am trying to use an array. I am trying something like this:
myarr=(main.conf | awk -F"=" 'NR!=1 {print $1}'))
echo ${myarr[1]}
# Or loop through every element in the array
for i in "${myarr[#]}"
do
:
echo $i
done
or
awk -F"=" 'NR!=1 {
print $1"\n"
STR=$2
IFS=':' read -r -a array <<< "$STR"
for i in "${!array[#]}"
do
echo "$i=>${array[i]}"
done
}' main.conf
But when I add this code to a .sh file and try to run it, I get syntax errors as
$ awk -F"=" 'NR!=1 {
> print $1"\n"
> STR=$2
> FS= read -r -a array <<< "$STR"
> for i in "${!array[#]}"
> do
> echo "$i=>${array[i]}"
> done
>
> }' main.conf
awk: cmd. line:4: FS= read -r -a array <<< "$STR"
awk: cmd. line:4: ^ syntax error
awk: cmd. line:5: for i in "${!array[#]}"
awk: cmd. line:5: ^ syntax error
awk: cmd. line:8: done
awk: cmd. line:8: ^ syntax error
How can I complete the above expectations?
This is the awk code to do what you want:
$ cat tst.awk
BEGIN { FS="[ =:]+"; OFS="=" }
{
print $1
for (i=2;i<NF;i+=2) {
print $i, $(i+1)
}
print ""
}
and this is the shell script (yes, all a shell script does to manipulate text is call awk):
$ awk -f tst.awk file
ABC
A=3
E=3
PS=6
PQR
B=5
S=5
AS=2
N=2
A UNIX shell is an environment from which to call UNIX tools (find, sort, sed, grep, awk, tr, cut, etc.). It has its own language for manipulating (e.g. creating/destroying) files and processes and sequencing calls to tools but it is NOT intended to be used to manipulate text. The guys who invented shell also invented awk for shell to call to manipulate text.
Read https://unix.stackexchange.com/questions/169716/why-is-using-a-shell-loop-to-process-text-considered-bad-practice and the book Effective Awk Programming, 4th Edition, by Arnold Robbins.
First off, a command that does what you want:
$ sed 's/ = /\n/;y/: /=\n/' main.conf
ABC
A=3
E=3
PS=6
PQR
B=5
S=5
AS=2
N=2
This replaces, on each line, the first (and only) occurrence of = with a newline (the s command), then turns all : into = and all spaces into newlines (the y command). Notice that
this works only because there is a space at the end of the first line (otherwise it would be a bit more involved to get the empty line between the blocks) and
this works only with GNU sed because it substitutes newlines; see this fantastic answer for all the details and how to get it to work with BSD sed.
As for what you tried, there is almost too much wrong with it to try and fix it piece by piece: from the wild mixing of awk and Bash to syntax errors all over the place. I recommend you read good tutorials for both, for example:
The BashGuide
Effective AWK Programming
A Bash solution
Here is a way to solve the same in Bash; I didn't use any arrays.
#!/bin/bash
# Read line by line into the 'line' variable. Setting 'IFS' to the empty string
# preserves leading and trailing whitespace; '-r' prevents interpretation of
# backslash escapes
while IFS= read -r line; do
# Three parameter expansions:
# Replace ' = ' by newline (escape backslash)
line="${line/ = /\\n}"
# Replace ':' by '='
line="${line//:/=}"
# Replace spaces by newlines (escape backslash)
line="${line// /\\n}"
# Print the modified input line; '%b' expands backslash escapes
printf "%b" "$line"
done < "$1"
Output:
$ ./SO.sh main.conf
ABC
A=3
E=3
PS=6
PQR
B=5
S=5
AS=2
N=2

pipe awk output into c program

Hi I have written a c program that takes 3 integers as input:
./myprogram 1 2 3
and I am aiming to pipe data from a csv file into the input of the c program. I grab each line from the c program using:
for i in $(seq 1 `wc -l "test.csv" | awk '{print $1}'`); do sed -n $i'p' "test.csv"; done;
and then would like to pipe the output of this into my c program. I have tried doing:
for i in $(seq 1 `wc -l "test.csv" | awk '{print $1}'`); do sed -n $i'p' "test.csv"; done; | ./myprogram
however I get:
Line
bash: syntax error near unexpected token `|'
how do I pipe the output into my c program?
Thanks
It helps when you really try to understand error messages the shell gives you:
Line
bash: syntax error near unexpected token `|'
If you think about it, when you chain commands together in a pipeline, there is never a ; before a |, for example:
ls | wc -l
# and not: ls; | wc -l
Whatever comes after a ; is like an independent new command, as if you typed it on a completely new, clear command line. If you type | hello on a clear command line, you'll get the exact same error, because that's the exact same situation as ; | ... in your script, for example:
$ | hello
-bash: syntax error near unexpected token `|'
Others already answered this, but I also wanted to urge you to make other improvements in your script:
Always use $() instead of backticks, for example:
for i in $(seq 1 $(wc -l "test.csv" | awk '{print $1}')); ...
You didn't need the awk there, this would work just as well:
for i in $(seq 1 $(wc -l "test.csv")); ...
You could reduce your entire script to simply this, for the same effect:
./myprogram < test.csv
In the shell, it doesn't like an explicit line termination followed by a pipe (|). The pipe already delimits the commands. So you want:
for i in $(seq 1 `wc -l "test.csv" | awk '{print $1}'`); do sed -n $i'p' "test.csv"; done | ./myprogram

Resources