I have two commands:
cat BIG_DATAfinal.txt | grep "STATUS" | awk '{print $5}'
cat BIG_DATAfinal.txt | grep "start" | awk '{print $3}' | sed 's/time;//g'
I want concat this two command in a file.
example:
STATUS REPORT FOR JOB: CargaDestino
Generated: 2016-06-17 10:52:14
Job start time;2015-03-30 13:11:45
Job end time;2016-06-17 10:52:14
Job elapsed time;10677:40:29
Job status;99 (Not running)
The expected result would be:
CargaDestino;2015-03-30
Thanks a lot!
For example:
a=`cat BIG_DATAfinal.txt | grep "STATUS" | awk '{print $5}'`
b=`cat BIG_DATAfinal.txt | grep "start" | awk '{print $3}' | sed 's/time;//g'`
echo "$a;$b"
You can do this with a single awk program:
awk -v OFS=";" '
/STATUS/ {status=$5}
/start/ {split($3, a, /;/); start=a[2]}
status && start {print status, start; status=start=""}
' BIG_DATAfinal.txt
If the output looks like ;2015-03-30o, then your file has \r\n line endings, and you should do this:
sed 's/\r$//' BIG_DATAfinal.txt | awk -v OFS=";" '
/STATUS/ {status=$5}
/start/ {split($3, a, /;/); start=a[2]}
status && start {print status, start; status=start=""}
'
Related
Struggling with this...
for i in `cat services.txt`
do
if ! grep -q $i; then
echo " $i Is NOT Running"
else
echo " Checking $i on `hostname`..."
ps aux | grep -i $i | awk '{print $1, $11}'| cut -d' ' -f1-2| sort
echo -e " "
sleep 4
fi
done
The block just hangs - Ive not been able to capture the success/failure of grep
If a string in services.txt is NOT found ... the script hangs... Id like for grep to skip it if not found
services.txt contain just single words
thanks!
The reason your script hangs is beacuse the command grep -q $i is waiting for an input. You can try running that command separately in a shell and verify that it prompts for an input.
Changing your command to ps aux | grep -i $i in the if statement should fix your issue.
NOTE: ps aux | grep -i $i lists grep also as one of the process. Make sure you exclude that process by piping it to another grep ps aux | grep -i $i | grep -v 'grep'
here is the working code
checkservices() {
cat >$HOME/services.txt <<EOF
ganglia
hbase
hdfs
hive
hue
livy
mapred
test-missing-service
mysql
oozie
presto
spark
yarn
zeppelin
EOF
for i in `cat $HOME/services.txt`
do
if `ps -ef | grep ^$i | grep -v grep >/dev/null`
then
i=$(echo "$i" | awk '{print toupper($0)}')
echo "$i -- is running on `hostname`"
echo ""
sleep 2
else
i=$(echo "$i" | awk '{print tolower($0)}')
echo "$i -- IS NOT running on `hostname` error"
echo ""
fi
done
}
I have a command as follows, which takes lines from the Allergens file, based on lines from the IDs file.
awk '
FNR==NR{
a[$1]
next
}
/^Query/ || $2 in a
' IDs C100_Allergens | grep -B 1 'Hit: ' | grep -v '^--' > C100_Allergens_matches.txt
However, I have numerous sample_Allergens files, and want to run it in a loop as such, where list is a file with different sample names:
for i in `cat list`
do
awk '
FNR==NR{
a[$1]
next
}
/^Query/ || $2 in a
' IDs "$i"_Allergens | grep -B 1 'Hit: ' | grep -v '^--' > "$i"_Allergens_matches.txt
done
I tried this loop, including using the variable flag for awk, i.e. -v i="$i":
for i in `cat list`
do
awk -v i="$i" '
FNR==NR{
a[$1]
next
}
/^Query/ || $2 in a
' IDs "$i"_Allergens | grep -B 1 'Hit: ' | grep -v '^--' > "$i"_Allergens_matches.txt
done
I only keep getting empty files. Thanks in advance for your help!
I wrote two function in a script
for TYPE in $LIST_TYPE
do
egrep -v 'dc02|vh|dc03kh00|r0' $REPORT_37 | grep CP_ASK | grep $DC |awk -F, -v type=$TYPE '$NF == type { print $1 }' > ./$DC_'type_'$TYPE
EMPTY_TYPE_$TYPE=$(cat $DC'_type_'$TYPE | wc -l)
done
the second will use in an other for EMPTY_TYPE_$TYPE
so I did
for TYPE in $LIST_TYPE
do
eval echo \$EMPTY_TYPE_$TYPE
egrep -v 'dc02|vh|dc03kh00|r[0-2]' $REPORT_33 | grep NOT | grep $DC |awk -F, -v days=$DAYS -v version=$VERSION -v type=$TYPE '{if (($(NF-15) > days || $(NF-02)!= version) && ($(NF-20) == type)) print $1}' | eval tail -n \ EMPTY_TYPE_$TYPE > ./$DC'_TO_WORK'
done
the problem is that I get error
awk -F, -v days=180 -v version=6.7 -v type=E '{if (($(NF-15) > days || $(NF-02)!= version) && ($(NF-20) == type)) print $1}' + eval tail -n '$EMPTY_TYPE_E'
when I run only the command (not in bash script)
egrep -v 'dc02|vh|dc03kh00|r[0-2]' report_33.20170720.075339 | grep NOT | grep $DC |awk -F, -v days=$DAYS -v version=$VERSION '{if (($(NF-15) > days || $(NF-02)!= version) && ($(NF-20) == "A")) print $1}' | eval tail -n \$EMPTY_TYPE_$TYPE
it works.
in the first function
EMPTY_TYPE_$TYPE=$(cat $DC'type'$TYPE | wc -l) should be
eval EMPTY_TYPE_$TYPE=$(cat $DC'type'$TYPE | wc -l)
that was the issue
I'm trying to understand what I'm doing wrong here, but can't seem to determine the cause. I would like to create a set of arrays from an output for a for loop in bash. Below is the code I have so far:
for i in `onedatastore list | grep pure02 | awk '{print $1}'`;
do
arr${i}=($(onedatastore show ${i} | sed 's/[A-Z]://' | cut -f2 -d\:)) ;
echo "Output of arr${i}: ${arr${i}[#]}" ;
done
The output for the condition is as such:
107
108
109
What I want to do is based on these unique IDs is create arrays:
arr107
arr108
arr109
The arrays will have data like such in each:
[oneadmin#opennebula/]$ arr107=($(onedatastore show 107 | sed 's/[A-Z]://' | cut -f2 -d\:))
[oneadmin#opennebula/]$ echo ${arr107[#]}
DATASTORE 107 INFORMATION 107 pure02_vm_datastore_1 oneadmin oneadmin 0 IMAGE vcenter vcenter /var/lib/one//datastores/107 FILE READY DATASTORE CAPACITY 60T 21.9T 38.1T - PERMISSIONS um- u-- --- DATASTORE TEMPLATE CLONE_TARGET="NONE" DISK_TYPE="FILE" DS_MAD="vcenter" LN_TARGET="NONE" RESTRICTED_DIRS="/" SAFE_DIRS="/var/tmp" TM_MAD="vcenter" VCENTER_CLUSTER="CLUSTER01" IMAGES
When I try this in the script section though I get output errors as such:
./test.sh: line 6: syntax error near unexpected token `$(onedatastore show ${i} | sed 's/[A-Z]://' | cut -f2 -d\:)'
I can't seem to figure out the syntax to use on this scenario.
In the end what I want to do is be able to compare different datastores and based on which on has more free space, deploy VMs to it.
Hope someone can help. Thanks
You can use the eval (potentially unsafe) and declare (safer) commands:
for i in $(onedatastore list | grep pure02 | awk '{print $1}');
do
declare "arr$i=($(onedatastore show ${i} | sed 's/[A-Z]://' | cut -f2 -d\:))"
eval echo 'Output of arr$i: ${arr'"$i"'[#]}'
done
readarray or mapfile, added in bash 4.0, will read directly into an array:
while IFS= read -r i <&3; do
readarray -t "arr$i" < <(onedatastore show "$i" | sed 's/[A-Z]://' | cut -f2 -d:)
done 3< <(onedatastore list | awk '/pure02/ {print $1}')
Better, back through bash 3.x, one can use read -a to read to an array:
shopt -s pipefail # cause pipelines to fail if any element does
while IFS= read -r i <&3; do
IFS=$'\n' read -r -d '' -a "arr$i" \
< <(onedatastore show "$i" | sed 's/[A-Z]://' | cut -f2 -d: && printf '\0')
done 3< <(onedatastore list | awk '/pure02/ {print $1}')
Alternately, one can use namevars to create an alias for an array with an arbitrarily-named array in bash 4.3:
while IFS= read -r i <&3; do
declare -a "arr$i"
declare -n arr="arr$i"
# this is buggy: expands globs, string-splits on all characters in IFS, etc
# ...but, well, it's what the OP is asking for...
arr=( $(onedatastore show "$i" | sed 's/[A-Z]://' | cut -f2 -d:) )
done 3< <(onedatastore list | awk '/pure02/ {print $1}')
I'm having some trouble getting around an issue related to performing a sed substitution with array's being passed into it as variables. I'm nearly certain it has something to do with the way I'm passing the variables, but I've been scouring for hours for a solution to no avail.
The arrays are initialized in the first six lines of code before attempting a sed substitution. The full script is below:
#!/bin/bash
scalarLineNums=( $(grep -nr 'MCSTEP\|NCSTEP\|ISAVE\|DCGRAX\|DCGRAY\|DCGRAZ\|DCSTEC\|DCTIME\|ICOUTF\|ICOUTI\|D1PEKS\|D1PEFR\|D1PEPF\|MBCON\|NBCON\|D1BNVX\|D1BNVY\|D1BNVZ\|I1BNVX\|I1BNVY\|I1BNVZ\|D1BNFX\|D1BNFY\|D1BNFZ\|D1BNAX\|D1BNAY\|D1BNAZ' Layer_Rough.Y3D | cut -d ":" -f 1) )
vectorLineNums=( $(sed -n -e '/D1BNVX\|D1BNVY\|D1BNVZ\|I1BNVX\|I1BNVY\|I1BNVZ\|D1BNFX\|D1BNFY\|D1BNFZ\|D1BNAX\|D1BNAY\|D1BNAZ/{n;=;p;}' Layer_Rough.Y3D | sed -n 1~2p) )
scalarValuesOriginal=( $(grep -nr 'MCSTEP\|NCSTEP\|ISAVE\|DCGRAX\|DCGRAY\|DCGRAZ\|DCSTEC\|DCTIME\|ICOUTF\|ICOUTI\|D1PEKS\|D1PEFR\|D1PEPF\|MBCON\|NBCON\|D1BNVX\|D1BNVY\|D1BNVZ\|I1BNVX\|I1BNVY\|I1BNVZ\|D1BNFX\|D1BNFY\|D1BNFZ\|D1BNAX\|D1BNAY\|D1BNAZ' Layer_Rough.Y3D | awk -F ' +' '{print $2}') )
vectorValuesOriginal=( $(sed -n -e '/D1BNVX\|D1BNVY\|D1BNVZ\|I1BNVX\|I1BNVY\|I1BNVZ\|D1BNFX\|D1BNFY\|D1BNFZ\|D1BNAX\|D1BNAY\|D1BNAZ/{n;=;p;}' Layer_Rough.Y3D | sed -n 2~2p) )
scalarValuesNew=( $(grep -nr 'MCSTEP\|NCSTEP\|ISAVE\|DCGRAX\|DCGRAY\|DCGRAZ\|DCSTEC\|DCTIME\|ICOUTF\|ICOUTI\|D1PEKS\|D1PEFR\|D1PEPF\|MBCON\|NBCON\|D1BNVX\|D1BNVY\|D1BNVZ\|I1BNVX\|I1BNVY\|I1BNVZ\|D1BNFX\|D1BNFY\|D1BNFZ\|D1BNAX\|D1BNAY\|D1BNAZ' new-variable-list.txt | awk -F ' +' '{print $2}') )
vectorValuesNew=( $(sed -n -e '/D1BNVX\|D1BNVY\|D1BNVZ\|I1BNVX\|I1BNVY\|I1BNVZ\|D1BNFX\|D1BNFY\|D1BNFZ\|D1BNAX\|D1BNAY\|D1BNAZ/{n;=;p;}' new-variable-list.txt | sed -n 2~2p) )
i=0
for linenumber in "${scalarLineNums[#]}"
do
sed -i "${linenumber}s/${scalarValuesOriginal[$i]}/${scalarValuesNew[$i]}/" Layer_Rough.Y3D
i=$((i+1))
done
The error I receive when trying to run the script is
sed: -e expression #1, char 2: unknown command: `
'
The for loop is an attempt to perform a substitution on a per line basis, i.e., sed 'line#s/oldvalue/newvalue/'. A few of the elements contain '+' and '-' characters as some of the values are stored in scientific notation, but do not contain any slashes or whitespace.