i have an awk command as below :
awk 'FNR==NR{a[$1$2]=$3;next} ($1$2 in a) {print$1,$2,a[$1$2],$3}' ""each line of file 1"" >./awkfile/12as_132l.txt
and f1.txt content is :
1sas.txt 12ds.txt
13sa.txt 21sa.txt
i want that my script read each line of fil1.txt and put the contents in this awk command...instead of """each line of file1"".. and execute the command like below:
awk 'FNR==NR{a[$1$2]=$3;next} ($1$2 in a) {print$1,$2,a[$1$2],$3}' 1sas.txt 12ds.txt >./awkfile/12as_132l.txt
awk 'FNR==NR{a[$1$2]=$3;next} ($1$2 in a) {print$1,$2,a[$1$2],$3}' 13sa.txt 21sa.txt >./awkfile/12as_132l.txt
i need a loop but the problem is that's a little strange for me .
Here is Snippet, which reads 2 fields line by line from f1.txt, put it into variable, and used in awk
#!/usr/bin/env bash
while read -r col1file col2file; do
# your command goes here
# this awk does nothing, replace it with your command
awk '{ }' "$col1file" "$col2file" > someoutfile
done < "f1.txt"
Test Results:
akshay#db-3325:/tmp$ cat f1.txt
1sas.txt 12ds.txt
13sa.txt 21sa.txt
akshay#db-3325:/tmp$ cat test.sh
while read -r col1file col2file; do
echo "col1 : $col1file"
echo "col2 : $col2file"
# your command goes here
# this awk does nothing, replace it with your command
# awk '{ }' "$col1file" "$col2file" > someoutfile
done < "f1.txt"
akshay#db-3325:/tmp$ bash test.sh
col1 : 1sas.txt
col2 : 12ds.txt
col1 : 13sa.txt
col2 : 21sa.txt
change the IFS to IFS=' ' and use loop .
Related
I have been trying to split up a string and putting it into an Array in Bash on my Mac without success.
Here is my sample code:
#!/bin/bash
declare -a allDisks
allDisksString="`ls /dev/disk* | grep -e 'disk[0-9]s.*' | awk '{ print $NF }'`"
#allDisksString="/dev/disk0s1 /dev/disk1s1"
echo allDisksString is $allDisksString
IFS=' ' read -ra allDisks <<< "$allDisksString"
echo allDIsks is "$allDisks"
echo The second item in allDisks is "${allDisks[1]}"
for disk in "${allDisks[#]}"
do
printf "Loop $disk\n"
done
And below is the output:
allDisksString is /dev/disk0s1 /dev/disk0s2 /dev/disk0s3 /dev/disk0s4 /dev/disk1s1
allDIsks is /dev/disk0s1
The second item in allDisks is
Loop /dev/disk0s1
Interesting if I execute the following in the Mac Terminal:
ls /dev/disk* | grep -e 'disk[0-9]s.*' | awk '{ print $NF }'
I get the following output
/dev/disk0s1
/dev/disk0s2
/dev/disk0s3
/dev/disk0s4
/dev/disk1s1
So I have also tried setting IFS to IFS=$'\n' without any success.
So no luck in getting a list of my drives into an array.
Any ideas on what I am doing wrong?
You're making this much more complicated than it needs to be. You don't need to use ls, you can just use a wildcard to match the device names you want, and put that in an array assignment.
#!/bin/bash
declare -a allDisks
allDisks=(/dev/disk[0-9]s*)
echo allDIsks is "$allDisks"
echo The second item in allDisks is "${allDisks[1]}"
for disk in "${allDisks[#]}"
do
printf "Loop $disk\n"
done
read only reads one line.
Use an assignment instead. When assigning to an array, you need to use parentheses after the = sign:
#!/bin/bash
disks=( $(ls /dev/disk* | grep -e 'disk[0-9]s.*' | awk '{ print $NF }') )
echo ${disks[1]}
Can anyone explain why the following doesn't work please?
list
the letter is d
the number is 4
the number is 2
the letter is g
script.sh
#!/bin/bash
cat "$1" | grep letter | array=($(awk '{print $4}'))
for i in "${array[#]}"
do
:
echo $i
done
If I run this bash script.sh list I expect the array to print d and g, but it doesn't. I think its because of how I am trying to set the array.
I think its because of how I am trying to set the array.
Each command in a pipeline | is run in a subshell - as a separate process. The parent process does not "see" variable changes from a child process.
Just:
array=($(grep letter "$1" | awk '{print $4}'))
or
array=($(awk '/letter/{print $4}' "$1"))
Run variable assignment in the parent shell.
You should assign the complete row of piped commands to a variable.
array=($(cat "$1" | grep letter | awk '{print $4}'))
The cat and grep command can be combined with awk, but why do you want an array?
I think you want the process each element in one loop, so first remove the double quotes:
for i in ${array[#]}
do
:
echo $i
done
Next, try to do this without an array
while read -r i; do
:
echo $i
done < <(awk '/letter/ {print $4}' "$1")
How can I have a list that tells me when a word (SecondWord in my example) is missing when another word (FirstWord in ma example) has been write.
File1.txt:
Hello FirstWord ... SecondWord
...
... FirstWord ...
FirstWord ... SecondWord
FirstWord ... ...
Hello ... SecondWord
File2.txt:
Hello FirstWord ... SecondWord
FirstWord ...
I would like a file at the end with line and filename :
3 : File1
5 : File1
2 : File2
I think what I try to do is overkill :/
filename='test.txt'
lineArray[0]=0
filesAwaitable[0]=$lineArray
grep -r -n 'FirstWord ' Sources/ --include=\*.txt | awk -F: '{print $2":"$1}' > $filename
while read line;
do
IFS=:
ary=($line)
for key in "${!ary[#]}";
do
if [[ $filesAwaitable["$key"] != *"${ary[$key]}"* ]]; then
filesAwaitable["$key"] = "${ary[$key]}"
echo "${ary[$key]}";
fi
#grep -n -r 'SecondWord' line | awk -F: '{print $2":"$1}'
done
done < $filename
Thx for your help
In case a Perl solution is okay, here is a straightforward one with the same regular expression as for GNU grep:
perl -ne 'print "$. : $ARGV\n" if /FirstWord((?!SecondWord).)*$/; close ARGV if eof' File?.txt
How can I use this command in all Source/ folder for all .txt files?
You can use find for this:
find Source -name \*.txt|xargs perl …
(without the File?.txt at the end of the perlcommand).
In my example I want to know when SecondWord is missing when FirstWord has been written before.
If you have a grep which understands perl regular expressions, you can directly specify a pattern for FirstWord not followed by SecondWord anywhere until the end of line:
grep -nP 'FirstWord((?!SecondWord).)*$' File?.txt
This yields for the given examples:
File1.txt:3:... FirstWord ...
File1.txt:5:FirstWord ... ...
File2.txt:2:FirstWord ...
To rearrange that to the desired output format, you can pipe it through
awk -F: '{print $2,":",$1}'
If the shell is bash, the problem can be solved easily with the extended pattern matching operator !(pattern), which matches anything except the given pattern:
shopt -s extglob # enable extended pattern matching features
for file in "$#"
do lineno=0
while read line
do ((++lineno))
case "$line" in *FirstWord!(*SecondWord)) echo $lineno : $file
esac
done <$file
done
I am very new to Unix shell script and trying to get some knowledge in shell scripting. Please check my requirement and my approach.
I have a input file having data
ABC = A:3 E:3 PS:6
PQR = B:5 S:5 AS:2 N:2
I am trying to parse the data and get the result as
ABC
A=3
E=3
PS=6
PQR
B=5
S=5
AS=2
N=2
The values can be added horizontally and vertically so I am trying to use an array. I am trying something like this:
myarr=(main.conf | awk -F"=" 'NR!=1 {print $1}'))
echo ${myarr[1]}
# Or loop through every element in the array
for i in "${myarr[#]}"
do
:
echo $i
done
or
awk -F"=" 'NR!=1 {
print $1"\n"
STR=$2
IFS=':' read -r -a array <<< "$STR"
for i in "${!array[#]}"
do
echo "$i=>${array[i]}"
done
}' main.conf
But when I add this code to a .sh file and try to run it, I get syntax errors as
$ awk -F"=" 'NR!=1 {
> print $1"\n"
> STR=$2
> FS= read -r -a array <<< "$STR"
> for i in "${!array[#]}"
> do
> echo "$i=>${array[i]}"
> done
>
> }' main.conf
awk: cmd. line:4: FS= read -r -a array <<< "$STR"
awk: cmd. line:4: ^ syntax error
awk: cmd. line:5: for i in "${!array[#]}"
awk: cmd. line:5: ^ syntax error
awk: cmd. line:8: done
awk: cmd. line:8: ^ syntax error
How can I complete the above expectations?
This is the awk code to do what you want:
$ cat tst.awk
BEGIN { FS="[ =:]+"; OFS="=" }
{
print $1
for (i=2;i<NF;i+=2) {
print $i, $(i+1)
}
print ""
}
and this is the shell script (yes, all a shell script does to manipulate text is call awk):
$ awk -f tst.awk file
ABC
A=3
E=3
PS=6
PQR
B=5
S=5
AS=2
N=2
A UNIX shell is an environment from which to call UNIX tools (find, sort, sed, grep, awk, tr, cut, etc.). It has its own language for manipulating (e.g. creating/destroying) files and processes and sequencing calls to tools but it is NOT intended to be used to manipulate text. The guys who invented shell also invented awk for shell to call to manipulate text.
Read https://unix.stackexchange.com/questions/169716/why-is-using-a-shell-loop-to-process-text-considered-bad-practice and the book Effective Awk Programming, 4th Edition, by Arnold Robbins.
First off, a command that does what you want:
$ sed 's/ = /\n/;y/: /=\n/' main.conf
ABC
A=3
E=3
PS=6
PQR
B=5
S=5
AS=2
N=2
This replaces, on each line, the first (and only) occurrence of = with a newline (the s command), then turns all : into = and all spaces into newlines (the y command). Notice that
this works only because there is a space at the end of the first line (otherwise it would be a bit more involved to get the empty line between the blocks) and
this works only with GNU sed because it substitutes newlines; see this fantastic answer for all the details and how to get it to work with BSD sed.
As for what you tried, there is almost too much wrong with it to try and fix it piece by piece: from the wild mixing of awk and Bash to syntax errors all over the place. I recommend you read good tutorials for both, for example:
The BashGuide
Effective AWK Programming
A Bash solution
Here is a way to solve the same in Bash; I didn't use any arrays.
#!/bin/bash
# Read line by line into the 'line' variable. Setting 'IFS' to the empty string
# preserves leading and trailing whitespace; '-r' prevents interpretation of
# backslash escapes
while IFS= read -r line; do
# Three parameter expansions:
# Replace ' = ' by newline (escape backslash)
line="${line/ = /\\n}"
# Replace ':' by '='
line="${line//:/=}"
# Replace spaces by newlines (escape backslash)
line="${line// /\\n}"
# Print the modified input line; '%b' expands backslash escapes
printf "%b" "$line"
done < "$1"
Output:
$ ./SO.sh main.conf
ABC
A=3
E=3
PS=6
PQR
B=5
S=5
AS=2
N=2
I have awk file:
#!/bin/awk -f
BEGIN {
}
{
filetime[$'$colnumber']++;
}
END {
for (i in filetime) {
print filetime[i],i;
}
}
And bash script:
#!/bin/bash
var1=$1
awk -f myawk.awk
When I run:
ls -la | ./countPar.sh 5
I receive error:
ls -la | ./countPar.sh 5
awk: myawk.awk:6: filetime[$'$colnumber']++;
awk: myawk.awk:6: ^ invalid char ''' in expression
Why? $colnumber must be replaced with 5, so awk should read 5th column of ls ouput.
Thanks.
You can pass variables to your awk script directly from the command line.
Change this line:
filetime[$'$colnumber']++;
To:
filetime[colnumber]++;
And run:
ls -al | awk -f ./myawk.awk -v colnumber=5
If you really want to use a bash wrapper:
#!/bin/bash
var1=$1
awk -f myawk.awk colnumber=$var1
(with the same change in your script as above.)
If you want to use environment variables use:
#!/bin/bash
export var1=$1
awk -f myawk.awk
and:
filetime[ENVIRON["var1"]]++;
(I really don't understand what the purpose of your awk script is though. The last part could be simplified to:
END { print filetime[colnumber],colnumber; }
and parsing the output of ls is generally a bad idea.)
The easiest way to do it:
#!/bin/bash
var=$1
awk -v colnumber="${var}" -f /your/script
But within your awk script, you don't need the $ in front of colnumber.
HTH
Passing 3 variable to script myscript.sh
var1 is the column number on which condition has set.
While var2 & var3 are input and temp file.
#!/bin/ksh
var1=$1
var2=$2
var3=$3
awk -v col="${var1}" -f awkscript.awk ${var2} > $var3
mv ${var3} ${var2}
execute it like below -
./myscript.sh 2 file.txt temp.txt