I have following piece of code which is working fine if executed standalone -
I am facing a weird error, code is executed fine when executed standalone but throws error when embedded with another piece of code described below in the post
date=$1
set -A max_month 0 31 28 31 30 31 30 31 31 30 31 30 31
eval $(echo $date|sed 's!\(....\)\(..\)\(..\)!year=\1;month=\2;day=\3!')
(( year4=year%4 ))
(( year100=year%100 ))
(( year400=year%400 ))
if [ \( $year4 -eq 0 -a \
$year100 -ne 0 \) -o \
$year400 -eq 0 ]
then
set -A max_month 0 31 29 31 30 31 30 31 31 30 31 30 31
fi
day=$((day+1))
echo $day ${max_month[$month]}
if [ $day -gt ${max_month[$month]} ]
then
day=1
month=$((month+1))
if [ $month -gt 12 ]
then
year=$((year+1))
month=1
fi
fi
new_date=$(printf "%4.4d%2.2d%2.2d" $year $month $day)
echo $new_date
When I try to embed it into following code highlighted in red it throws error, obviously I replaced date=$1 to julian_date_14 -
#!/bin/bash
cd
unset project_env
cd /wload/baot/home/baotasa0/UKRB_UKBE/sandboxes/EXTRACTS/UK/RB/UKBA/ukrb_ukba_pbe_acq
. ab* . >> project_setup.log 2>&1
echo unset the environment is doNe
cd /wload/baot/app/data_abinitio/serial/uk_cust
param1=$1
param2=$2
param3=$3
email=$4
header_date_14=$(m_dump /wload/baot/app/data_abinitio/serial/uk_cust/ukrb_ukba_acnt_bde27_src.dml $param1 | head -35)
hdr_dt_14=$(echo "$header_date_14" | awk '$1=="bdfo_run_date" {print $2}')
julian_date_14=$(m_eval '(date("YYYYMMDD"))( unsigned integer(2)) '$hdr_dt_14'') 2>&1
header_date_15=$(m_dump /wload/baot/app/data_abinitio/serial/uk_cust/ukrb_ukba_acnt_bde27_src.dml $param2 | head -35)
hdr_dt_15=$(echo "$header_date_15" | awk '$1=="bdfo_run_date" {print $2}')
julian_date_15=$(m_eval '(date("YYYYMMDD"))( unsigned integer(2)) '$hdr_dt_15'')
header_date_16=$(m_dump /wload/baot/app/data_abinitio/serial/uk_cust/ukrb_ukba_acnt_bde27_src.dml $param3 | head -35)
hdr_dt_16=$(echo "$header_date_16" | awk '$1=="bdfo_run_date" {print $2}')
julian_date_16=$(m_eval '(date("YYYYMMDD"))( unsigned integer(2)) '$hdr_dt_16'')
echo $julian_date_16
if [ "$julian_date_14" = "$julian_date_15" -a "$julian_date_15" = "$julian_date_16" ]
then
echo all are same
else
echo check the file date please
fi
DATE=`echo $julian_date_14 | cut -c8-9`
Date_minus_1=`expr $DATE - 1`
DATE_1=`echo $julian_date_14 | cut -c2-7`
DATE_FINAL="$DATE_1$Date_minus_1"
echo $DATE_FINAL
Error is below -
./auto1.sh: line 70: set: -A: invalid option
set: usage: set [-abefhkmnptuvxBCHP] [-o option-name] [--] [arg ...]
Any help will be greatly appreciated.
Thank you in advance!
Related
Let's say we have a shell variable $x containing a space separated list of numbers from 1 to 30:
$ x=$(for i in {1..30}; do echo -n "$i "; done)
$ echo $x
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30
We can print the first three input record fields with AWK like this:
$ echo $x | awk '{print $1 " " $2 " " $3}'
1 2 3
How can we print all the fields starting from the Nth field with AWK? E.g.
4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30
EDIT: I can use cut, sed etc. to do the same but in this case I'd like to know how to do this with AWK.
Converting my comment to answer so that solution is easy to find for future visitors.
You may use this awk:
awk '{for (i=3; i<=NF; ++i) printf "%s", $i (i<NF?OFS:ORS)}' file
or pass start position as argument:
awk -v n=3 '{for (i=n; i<=NF; ++i) printf "%s", $i (i<NF?OFS:ORS)}' file
Version 4: Shortest is probably using sub to cut off the first three fields and their separators:
$ echo $x | awk 'sub(/^ *([^ ]+ +){3}/,"")'
Output:
4 5 6 7 8 9 ...
This will, however, preserve all space after $4:
$ echo "1 2 3 4 5" | awk 'sub(/^ *([^ ]+ +){3}/,"")'
4 5
so if you wanted the space squeezed, you'd need to, for example:
$ echo "1 2 3 4 5" | awk 'sub(/^ *([^ ]+ +){3}/,"") && $1=$1'
4 5
with the exception that if there are only 4 fields and the 4th field happens to be a 0:
$ echo "1 2 3 0" | awk 'sub(/^ *([^ ]+ +){3}/,"")&&$1=$1'
$ [no output]
in which case you'd need to:
$ echo "1 2 3 0" | awk 'sub(/^ *([^ ]+ +){3}/,"") && ($1=$1) || 1'
0
Version 1: cut is better suited for the job:
$ cut -d\ -f 4- <<<$x
Version 2: Using awk you could:
$ echo -n $x | awk -v RS=\ -v ORS=\ 'NR>=4;END{printf "\n"}'
Version 3: If you want to preserve those varying amounts of space, using GNU awk you could use split's fourth parameter seps:
$ echo "1 2 3 4 5 6 7" |
gawk '{
n=split($0,a,FS,seps) # actual separators goes to seps
for(i=4;i<=n;i++) # loop from 4th
printf "%s%s",a[i],(i==n?RS:seps[i]) # get fields from arrays
}'
Adding one more approach to add all value into a variable and once all fields values are done with reading just print the value of variable. Change the value of n= as per from which field onwards you want to get the data.
echo "$x" |
awk -v n=3 '{val="";for(i=n; i<=NF; i++){val=(val?val OFS:"")$i};print val}'
With GNU awk, you can use the join function which has been a built-in include since gawk 4.1:
x=$(seq 30 | tr '\n' ' ')
echo "$x" | gawk '#include "join"
{split($0, arr)
print join(arr, 4, length(arr), "|")}
'
4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30
(Shown here with a '|' instead of a ' ' for clarity...)
Alternative way of including join:
echo "$x" | gawk -i join '{split($0, arr); print join(arr, 4, length(arr), "|")}'
Using gnu awk and gensub:
echo $x | awk '{ print gensub(/^([[:digit:]]+[[:space:]]){3}(.*$)/,"\\2",$0)}'
Using gensub, split the string into two sections based on regular expressions and print the second section only.
I doing a project for a Unix/Linux class and have a few Arrays in another file, like this
All the arrays should store is a 1 or 0
T1=( 1 1 1 1 1 )
T2=( 1 1 1 1 1 )
T3=( 1 1 1 1 1 )
T4=( 1 1 1 1 1 )
T5=( 1 1 1 1 1 )
However i'm having difficulty editing the arrays to 0 and making changes stick
#!/bin/bash
i=0
for line in `cat JeopardyOut`
do
let i=i+1
#Creating one big Array with all the values from the arrays file
Array[i]=$line
done
cat JeopardyOut
#Parsing the giant Array into the main 5 arrays I'm using
createArray()
{
y=0
for y in 0 1 2 3 4
do
for i in 2 3 4 5 6
do
#echo "${Array[i]}"
T1[$y]=${Array[i]}
done
for i in 9 10 11 12 13
do
#echo "${Array[i]}"
T2[$y]=${Array[i]}
done
for i in 16 17 18 19 20
do
#echo "${Array[i]}"
T3[$y]=${Array[i]}
done
for i in 23 24 25 26 27
do
#echo "${Array[i]}"
T4[$y]=${Array[i]}
done
for i in 30 31 32 33 34
do
#echo "${Array[i]}"
T5[$y]=${Array[i]}
done
done
}
createArray
ArrayNum=$1
Index=$2
There's likely way better ways to do this, However this is what ended up working for me.
#Changing the necessary indexes, this will be used by a completely
#different script
ChangeArray()
{
if [[ $ArrayNum == "1" ]]; then
T1[ $Index ]=0
elif [[ $ArrayNum == "2" ]]; then
T2[ $Index ]=0
elif [[ $ArrayNum == "3" ]]; then
T3[ $Index ]=0
elif [[ $ArrayNum == "4" ]]; then
T4[ $Index ]=0
elif [[ $ArrayNum == "5" ]]; then
T5[ $Index ]=0
else
echo "Invalid Parameters"
fi
}
if [[ $ArrayNum -ne "" || $Index -ne "" ]]; then
if [[ $ArrayNum == "5" && $Index == "5"]]; then
reset
else
ChangeArray
fi
fi
# And the part that's likely at fault for my issue but don't know how I
# should fix it
echo "T1=( ${T1[*]} )" > JeopardyOut
echo "T2=( ${T2[*]} )" >> JeopardyOut
echo "T3=( ${T3[*]} )" >> JeopardyOut
echo "T4=( ${T4[*]} )" >> JeopardyOut
echo "T5=( ${T5[*]} )" >> JeopardyOut
cat JeopardyOut
Something is wrong with the way I am trying to edit the Arrays...
While I can get any index of any of the arrays to 0, I do not know why the 1s I change to 0 turn back into 1 when I rerun the script.
PS. This is a basis class for Linux programming in Sierra, I don't really understand a lot of the bash script other than what I've learned through trial and error.
Maybe this example can help you
#!/bin/bash
while read -r line; do
#Creating one big Array with all the values from the arrays file
Array+=(`grep -oP '\(\K(.+?)(?=\))' <<< "$line"`)
done < JeopardyOut
echo "Giant Array value: ${Array[#]}"
# Or you can clone the arrays in the file
i=0
while read -r line; do
((i++))
eval "T$i=(`grep -oP '\(\K(.+?)(?=\))' <<< "$line"`)"
done < JeopardyOut
echo "This is the contents of array T1: ${T1[#]}"
echo "This is the contents of array T2: ${T2[#]}"
echo "This is the contents of array T3: ${T3[#]}"
echo "This is the contents of array T4: ${T4[#]}"
echo "This is the contents of array T5: ${T5[#]}"
# This can help you to make a new file
# I change some value to our arrays
T1[2]=0
T2[1]=true
# Now let's go to make a new arrays file
echo "T1=(${T1[#]})" > JeopardyOut2
echo "T2=(${T2[#]})" >> JeopardyOut2
echo "T3=(${T3[#]})" >> JeopardyOut2
echo "T4=(${T4[#]})" >> JeopardyOut2
echo "T5=(${T5[#]})" >> JeopardyOut2
echo "This is the content of JeopardyOut2:"
cat JeopardyOut2
Output
$ bash example
Giant Array value: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
This is the contents of array T1: 1 1 1 1 1
This is the contents of array T2: 1 1 1 1 1
This is the contents of array T3: 1 1 1 1 1
This is the contents of array T4: 1 1 1 1 1
This is the contents of array T5: 1 1 1 1 1
This is the content of JeopardyOut2:
T1=(1 1 0 1 1)
T2=(1 true 1 1 1)
T3=(1 1 1 1 1)
T4=(1 1 1 1 1)
T5=(1 1 1 1 1)
$
Let's say I have an array with some numbers, ordered from the lowest to the highest numeral:
root#blubb:~# min=1
root#blubb:~# echo $array
1 2 3 6 16 26 27
I want a bash script to always try the minimum number defined first (in this case min=1) and if that is not possible add 1 and try it again, until it finally works (in this case it would take 4)
I tried a lot with shuf and while-/for loops but could not get it to run properly.
min=1
array="1 2 3 6 16 26 27"
found=0
res=$min
for elem in $array; do
if [[ $elem = $res ]]; then
found=1
continue
fi
if [[ $found = 1 ]]; then
((res+=1))
if ((elem>res)); then
break
fi
fi
done
echo $res
Or with a function
function min_array {
local min array res found
min=$1
shift
array=$*
found=0
res=$min
for elem in $array; do
if [[ $elem = $res ]]; then
found=1
continue
fi
if [[ $found = 1 ]]; then
((res+=1))
if ((elem>res)); then
break
fi
fi
done
echo $res
}
$ min_array 1 1 2 3 6 16 26
4
$ min_array 6 1 2 3 6 16 26
7
$ min_array 8 1 2 3 6 16 26
8
EDIT one case was missing, another version
function min_array {
local min array res found
min=$1
shift
array=$*
found=0
res=$min
for elem in $array; do
if [[ $elem = $res ]]; then
found=1
((res+=1))
continue
fi
if [[ $found = 1 ]]; then
if ((elem>res)); then
break
else
((res+=1))
fi
fi
done
echo $res
}
Try this.
array=(7 5 3 1)
min()
{
local min=$1; shift
local n
for n in "$#"; do
if ((n<min)); then
min=$n
fi
done
echo "$min"
}
min "${array[#]}"
I've written a shell script to get the PIDs of specific process names (e.g. pgrep python, pgrep java) and then use top to get the current CPU and Memory usage of those PIDs.
I am using top with the '-p' option to give it a list of comma-separated PID values. When using it in this mode, you can only query 20 PIDs at once, so I've had to come up with a way of handling scenarios where I have more than 20 PIDs to query. I'm splitting up the list of PIDs passed to the function below and "despatching" multiple top commands to query the resources:
# $1 = List of PIDs to query
jobID=0
for pid in $1; do
if [ -z $pidsToQuery ]; then
pidsToQuery="$pid"
else
pidsToQuery="$pidsToQuery,$pid"
fi
pidsProcessed=$(($pidsProcessed+1))
if [ $(($pidsProcessed%20)) -eq 0 ]; then
debugLog "DESPATCHED QUERY ($jobID): top -bn 1 -p $pidsToQuery | grep \"^ \" | awk '{print \$9,\$10}' | grep -o '.*[0-9].*' | sed ':a;N;\$!ba;s/\n/ /g'"
resourceUsage[$jobID]=`top -bn 1 -p "$pidsToQuery" | grep "^ " | awk '{print $9,$10}' | grep -o '.*[0-9].*' | sed ':a;N;$!ba;s/\n/ /g'`
jobID=$(($jobID+1))
pidsToQuery=""
fi
done
resourceUsage[$jobID]=`top -bn 1 -p "$pidsToQuery" | grep "^ " | awk '{print $9,$10}' | grep -o '.*[0-9].*' | sed ':a;N;$!ba;s/\n/ /g'`
The top command will return the CPU and Memory usage for each PID in the format (CPU, MEM, CPU, MEM etc)...:
13 31.5 23 22.4 55 10.1
The problem is with the resourceUsage array. Say, I have 25 PIDs I want to process, the code above will place the results of the first 20 PIDs in to $resourceUsage[0] and the last 5 in to $resourceUsage[1]. I have tested this out and I can see that each array element has the list of values returned from top.
The next bit is where I'm having difficulty. Any time I've ever wanted to print out or use an entire array's set of values, I use ${resourceUsage[#]}. Whenever I use that command in the context of this script, I only get element 0's data. I've separated out this functionality in to a script below, to try and debug. I'm seeing the same issue here too (data output to debug.log in same dir as script):
#!/bin/bash
pidList="1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25"
function quickTest() {
for ((i=0; i<=1; i++)); do
resourceUsage[$i]=`echo "$i"`
done
echo "${resourceUsage[0]}"
echo "${resourceUsage[1]}"
echo "${resourceUsage[#]}"
}
function debugLog() {
debugLogging=1
if [ $debugLogging -eq 1 ]; then
currentTime=$(getCurrentTime 1)
echo "$currentTime - $1" >> debug.log
fi
}
function getCurrentTime() {
if [ $1 -eq 0 ]; then
echo `date +%s`
elif [ $1 -eq 1 ]; then
echo `date`
fi
}
jobID=0
for pid in $pidList; do
if [ -z $pidsToQuery ]; then
pidsToQuery="$pid"
else
pidsToQuery="$pidsToQuery,$pid"
fi
pidsProcessed=$(($pidsProcessed+1))
if [ $(($pidsProcessed%20)) -eq 0 ]; then
debugLog "DESPATCHED QUERY ($jobID): top -bn 1 -p $pidsToQuery | grep \"^ \" | awk '{print \$9,\$10}' | grep -o '.*[0-9].*' | sed ':a;N;\$!ba;s/\n/ /g'"
resourceUsage[$jobID]=`echo "10 10.5 11 11.5 12 12.5 13 13.5"`
debugLog "Resource Usage [$jobID]: ${resourceUsage[$jobID]}"
jobID=$(($jobID+1))
pidsToQuery=""
fi
done
#echo "Dispatched job: $pidsToQuery"
debugLog "DESPATCHED QUERY ($jobID): top -bn 1 -p $pidsToQuery | grep \"^ \" | awk '{print \$9,\$10}' | grep -o '.*[0-9].*' | sed ':a;N;\$!ba;s/\n/ /g'"
resourceUsage[$jobID]=`echo "14 14.5 15 15.5"`
debugLog "Resource Usage [$jobID]: ${resourceUsage[$jobID]}"
memUsageInt=0
memUsageDec=0
cpuUsage=0
i=1
debugLog "Row 0: ${resourceUsage[0]}"
debugLog "Row 1: ${resourceUsage[1]}"
debugLog "All resource usage results: ${resourceUsage[#]}"
for val in ${resourceUsage[#]}; do
resourceType=$(($i%2))
if [ $resourceType -eq 0 ]; then
debugLog "MEM RAW: $val"
memUsageInt=$(($memUsageInt+$(echo $val | cut -d '.' -f 1)))
memUsageDec=$(($memUsageDec+$(echo $val | cut -d '.' -f 2)))
debugLog " MEM INT: $memUsageInt"
debugLog " MEM DEC: $memUsageDec"
elif [ $resourceType -ne 0 ]; then
debugLog "CPU RAW: $val"
cpuUsage=$(($cpuUsage+$val))
debugLog "CPU TOT: $cpuUsage"
fi
i=$(($i+1))
done
debugLog "$MEM DEC FINAL: $memUsageDec (pre)"
memUsageDec=$(($memUsageDec/10))
debugLog "$MEM DEC FINAL: $memUsageDec (post)"
memUsage=$(($memUsageDec+$memUsageInt))
debugLog "MEM USAGE: $memUsage"
debugLog "CPU USAGE: $cpuUsage"
debugLog "MEM USAGE: $memUsage"
debugLog "PROCESSED VALS: $cpuUsage,$memUsage"
echo "$cpuUsage,$memUsage"
I'm really stuck here as I've printed out entire arrays before in Bash Shell with no problem. I've even repeated this in the shell console with a few lines and it works fine there:
listOfValues[0]="1 2 3 4"
listOfValues[1]="5 6 7 8"
echo "${listOfValues[#]}"
Am I missing something totally obvious? Any help would be greatly appreciated!
Thanks in advance! :)
Welcome to StackOverflow, and thanks for providing a test case! The bash tag wiki has additional suggestions for creating small, simplified test cases. Here's a minimal version that shows your problem:
log() {
echo "$1"
}
array=(foo bar)
log "Values: ${array[#]}"
Expected: Values: foo bar. Actual: Values: foo.
This happens because ${array[#]} is magic in quotes, and turns into multiple arguments. The same is true for $#, and for brevity, let's consider that:
Let's say $1 is foo and $2 is bar.
The single parameter "$#" (in quotes) is equivalent to the two arguments "foo" "bar".
"Values: $#" is equivalent to the two parameters "Values: foo" "bar"
Since your log statement ignores all arguments after the first one, none of them show up. echo does not ignore them, and instead prints all arguments space separated, which is why it appeared to work interactively.
This is as opposed to ${array[*]} and $*, which are exactly like $# except not magic in quotes, and does not turn into multiple arguments.
"$*" is equivalent to "foo bar"
"Values: $*" is equivalent to "Values: foo bar"
In other words: If you want to join the elements in an array into a single string, Use *. If you want to add all the elements in an array as separate strings, use #.
Here is a fixed version of the test case:
log() {
echo "$1"
}
array=(foo bar)
log "Values: ${array[*]}"
Which outputs Values: foo bar
I would use ps, not top, to get the desired information. Regardless, you probably want to put the data for each process in a separate element of the array, not one batch of 20 per element. You can do this using a while loop and a process substitution. I use a few array techniques to simplify the process ID handling.
pid_array=(1 2 3 4 5 6 7 8 9 ... )
while (( ${#pid_array[#]} > 0 )); do
printf -v pidsToQuery "%s," "${pid_array[#]:0:20}"
pid_array=( "${pid_array[#]:20}" )
while read cpu mem; do
resourceUsage+=( "$cpu $mem" )
done < <( top -bn -1 -p "${pidsToQuery%,}" ... )
done
I have two sets of arrays with a variable number of elements, for instance:
chain=(B C)
hresname=(BMA MAN NAG NDG)
I am parsing a number of files that may contain elements from the array chain at a given position and elements of the array hresname at a different position (position is always fixed in both cases). This is a sample of the data:
ATOM 5792 CB MET D 213 49.385 -5.683 125.489 1.00142.66 C
ATOM 5793 CG MET D 213 50.834 -5.674 125.990 1.00154.50 C
ATOM 5794 SD MET D 213 51.530 -7.337 126.277 1.00164.73 S
ATOM 5795 CE MET D 213 52.854 -7.386 125.068 1.00169.73 C
HETATM 5797 C1 NAG B 323 70.090 50.934 125.869 1.00 86.35 C
HETATM 5798 C2 NAG B 323 69.687 52.074 126.879 1.00 95.95 C
HETATM 5799 C3 NAG B 323 68.377 52.740 126.390 1.00 87.65 C
HETATM 5800 C4 NAG B 323 68.598 53.314 125.014 1.00 83.97 C
First I need to copy lines starting with ATOM whose 5th column matches each of the elements of the array chain to separate files:
while read pdb ; do
for c in "${chain[#]}" ; do
#if [ ${#chain[#]} -eq 1 ] && \
if [ $(echo "$pdb" | cut -c1-4) == "ATOM" ] && \
[ $(echo "$pdb" | cut -c22-23) == "${chain[$c]}" ]; then
echo "$pdb" >> ../../properpdb/${pdbid}_${chain[$c]}.pdb
fi
done
done < ${pdbid}.pdb
This works well (slow but sure). Both the commented and the uncommented versions work.
Next I want to copy lines that start with HETATM and whose 4th column matches elements of the hresname but only if those lines also match an element form the chain array at cloumn number 5th:
while read pdb ; do
for c in "${chain[#]}" ; do
for h in "${hresname[#]}" ; do
if [ ${#chain[#]} -eq 1 ] && \
[ $(echo "$pdb" | cut -c1-6) == "HETATM" ] && \
[ $(echo "$pdb" | cut -c22-23) == "${chain[$c]}" ] \
[ $(echo "$pdb" | cut -c18-20) == "${hresname[$h]}" ] ; then
echo "$pdb" >> ../../properpdb/${pdbid}_${chain[$c]}.pdb
fi
done
done
done < ${pdbid}.pdb
However, this does not work. I repeatedly receive an error:
line 66: [: too many arguments
Line 66 is:
[ $(echo "$pdb" | cut -c22-23) == "${chain[$c]}" ] \
Which puzzles me because the error happens even if I restrict the loop to chain arrays containing a single element.
According to other StackOverflow questions, it should be perfectly possible to do this in bash. Any idea what the problem could be?
You for get to add &&, change this line:
[ $(echo "$pdb" | cut -c22-23) == "${chain[$c]}" ] \
To
[ $(echo "$pdb" | cut -c22-23) == "${chain[$c]}" ] &&\
Update:
You have too many errors in the script, I fixed it and it's now working. I suggest you read the for loop syntax in the bash manual first.
chain=(B C)
hresname=(BMA MAN NAG NDG)
while read pdb ; do
for c in ${chain[#]} ; do
for h in ${hresname[#]} ; do
if [ $(echo "$pdb" | cut -c1-6) == "HETATM" ] && \
[ $(echo "$pdb" | cut -c22-23) == "$c" ] && \
[ $(echo "$pdb" | cut -c18-20) == "$h" ] ; then
echo "$pdb" >> ../../properpdb/${pdbid}_${chain[$c]}.pdb
fi
done
done
done