Array job gives error in bash - arrays

As I want to run several simulations with different values in R, I have been recommended to use a job array in bash.
1) I generated the combination of parameters and saved it in a txt file, called parameters.txt.
2) I want now to use each combination of parameters into R. Each combination is represented by a line of 3 numbers (the 3 parameters) in parameters.txt.
When I run my script, an error message appears :
head: parameters.txt: invalid number of lines
head: parameters.txt: invalid number of lines
head: parameters.txt: invalid number of lines
Job array item : rx=, ry=, rz=
Here is my script:
# Sweeping parameters.txt
N=${SLURM_ARRAY_TASK_ID}
rx=`head -n ${N} parameters.txt | tail -n 1 | cut -d' ' -f1`
ry=`head -n ${N} parameters.txt | tail -n 1 | cut -d' ' -f2`
rz=`head -n ${N} parameters.txt | tail -n 1 | cut -d' ' -f3`
# Display
echo "Job array item $N: rx=$rx, ry=$ry, rz=$rz"
echo "---------------------------------"
# Run
R CMD BATCH ex.R $rx $ry $rz

Seems SLURM_ARRAY_TASK_ID is None (not set) and as a result N is None here:
N=${SLURM_ARRAY_TASK_ID}
Then bash translates it as
rx=`head -n parameters.txt ...
You can wrap with if statement as follows:
N=${SLURM_ARRAY_TASK_ID}
if [ -n "${N}" ]; then
rx=`head -n ${N} parameters.txt | tail -n 1 | cut -d' ' -f1`
ry=`head -n ${N} parameters.txt | tail -n 1 | cut -d' ' -f2`
rz=`head -n ${N} parameters.txt | tail -n 1 | cut -d' ' -f3`
# Display
echo "Job array item $N: rx=$rx, ry=$ry, rz=$rz"
echo "---------------------------------"
# Run
R CMD BATCH ex.R $rx $ry $rz
else
echo "SLURM_ARRAY_TASK_ID / N is None"
fi

Related

Weird array output in shell script

The content extracted from the file is separated and stored in an array, and print the content using loop. Only printing the last element is weird. I'll show you my code.
How can I resolve this problem?
[config.json]
{
"id": "hello",
"passwd": "1234",
"languageList": ["ko", "en"]
}
[test.sh]
# BEFORE_CONFIG and AFTER_CONFIG have same code
BEFORE_CONFIG=~/workspace/env/config.json
AFTER_CONFIG=~/workspace/config/config.json
BEF_LANG=$(grep "\[" ${BEFORE_CONFIG} | tr -d '\[' | tr -d '\]' | tr -d '"' | tr -d ' ' | cut -d ":" -f2)
AF_LANG=$(grep "\[" ${AFTER_CONFIG} | tr -d '\[' | tr -d '\]' | tr -d '"' | tr -d ' ' | cut -d ":" -f2)
echo "before lang :: ${BEF_LANG}"
echo "after lang :: ${AF_LANG}"
IFS=',' read -r -a AF_LANG_LIST <<< "$AF_LANG"
echo "after lang list print >> ${AF_LANG_LIST[#]}"
echo "list length >> ${#AF_LANG_LIST[#]}"
for element in ${AF_LANG_LIST[#]}
do
echo "${element}"
echo "This language !!! ${element} !!! print !!!!"
done
[result]
$ source tesh.sh
before lang :: ko,en
after lang :: ko,en
after lang list >> ko en
list length >> 2
ko
This language !!! ko !!! print !!!!
en
!!! print !!!!!! en # expect result → This language !!! en !!! print !!!!
You can use jq to parse the json correctly, and extract languageList do:
cat ~/workspace/env/config.json|jq .languageList[] -r|xargs
which will output:
ko en
which you can later use in your script
Trying to parse JSON with tr & cut is prone to so many errors.
Here is the example for your script:
#!/bin/bash
# BEFORE_CONFIG and AFTER_CONFIG have same code
BEFORE_CONFIG=~/workspace/env/config.json
AFTER_CONFIG=~/workspace/config/config.json
BEF_LANG=$(grep "\[" ${BEFORE_CONFIG} | tr -d '\[' | tr -d '\]' | tr -d '"' | tr -d ' ' | cut -d ":" -f2)
#AF_LANG=$(grep "\[" ${AFTER_CONFIG} | tr -d '\[' | tr -d '\]' | tr -d '"' | tr -d ' ' | cut -d ":" -f2)
AF_LANG=$(cat ${BEFORE_CONFIG}|jq .languageList[] -r|xargs)
echo "before lang :: ${BEF_LANG}"
echo "after lang :: ${AF_LANG}"
# you do not need this
#IFS=',' read -r -a AF_LANG_LIST <<< "$AF_LANG"
#echo "after lang list print >> ${AF_LANG_LIST[#]}"
#echo "list length >> ${#AF_LANG_LIST[#]}"
for element in ${AF_LANG[#]}
do
echo "${element}"
echo "This language !!! ${element} !!! print !!!!"
done

Convert field names to lower case using miller

I would like to use miller (mlr) to convert column names to lower case. The closest I get is using the rename verb with a regular expression. \L should change the case, but instead the the column names are getting prefixed by "\L".
I'm using macOS Catalina and miller 5.10.0
echo -e 'A,B,C\n1,2,3' | mlr --csv --opprint rename -r '(.*),\L\1'
prints
\LA \LB \LC
1 2 3
But I would like it to print
a b c
1 2 3
Two examples ways:
echo -e 'A,B,C\n1,2,3' | mlr --csv put '
map inrec = $*;
$* = {};
for (oldkey, value in inrec) {
newkey = tolower(oldkey);
$[newkey] = value;
}
'
or
echo -e 'A,B,C\n1,2,3' | mlr --csv -N put -S 'if (NR == 1) {for (k in $*) {$[k] = tolower($[k])}}'
Sometimes, standard tools are easier to use:
echo -e 'A,B,C\n1,2,3' | awk 'NR == 1 {print tolower($0); next} 1'
UPDATE
with Miller:
echo -e 'A,B,C\n1,2,3' |
mlr --csv -N put 'NR == 1 {for (k,v in $*) {$[k] = tolower(v)}}'

Submitting a job on PBS with while loop, changing output file names

I have a (let's call it original) script to parse a file (~1000 lines) line by line, generate arguments to execute a C++ program.
#!/bin/bash
i = 0
while IFS='' read -r line || [[ -n "$line" ]]; do
a="$line" | cut -c1-2
b="$line" | cut -c3-4
c="$line" | cut -c5-6
d="$line" | cut -c7-8
e="$line" | cut -c9-10
f="$line" | cut -c11-12
g="$line" | cut -c13-14
h="$line" | cut -c15-16
i="$line" | cut -c17-18
j="$line" | cut -c19-20
k="$line" | cut -c21-22
l="$line" | cut -c23-24
m="$line" | cut -c25-26
n="$line" | cut -c27-28
o="$line" | cut -c29-30
p="$line" | cut -c31-32
./a.out "$a" "$b" "$c" "$d" "$e" "$f" "$g" "$h" "$i" "$j" "$k" "$l" "$m" "$n" "$o" "$p" > $(echo some-folder/output_${i}.txt)
done < test_10.txt
I want to schedule this job in a batch, so that each run is queued and ran on separate cores.
I checked the PBS and qsub writing styles. I could write a PBS file (simple one, without all options for now. Lets call it callPBS.PBS):
#!/bin/bash
cd $PBS_O_WORKDIR
qsub ./a.out -F "my arguments"
exit 0
I can call this file instead of ./a.out -------- in original script. BUT how do I pass "my arguments"? Problem is they are not fixed.
Secondly I know qsub takes -o as an option for output file. But I want my output file name to be changed. I can pass that as an argument again, but how?
Can I do this in my original script:
callPBS.pbs > $(echo some-folder/output_${i}.txt)
I am sorry if I am missing something here. I am trying to use all that I know!

shell script array won't populate from for loop

Can anyone tell me why this array creation: cccr[$string_1]=$string_2 #doesn't work?
#!/bin/bash
firstline='[Event "Marchand Open"][Site "Rochester NY"][Date "2005.03.19"][Round "1"][White "Smith, Igor"][Black "Jones, Matt"][Result "1-0"][ECO "C01"][WhiteElo "2409"][BlackElo "1911"]'
unset cccr
declare -A cccr
(IFS='['; for word in $firstline; do
string_1=$(echo $word | cut -f1 -d'"' | tr -d ' ')
string_2=$( echo $word | cut -f2 -d'"' )
if [ ! -z $string_1 ]; then # If $string_1 is not empty
cccr[$string_1]=$string_2 # why doesn't this line work?
fi
done)
echo ${cccr[Event]} # echos null string
It happens because the value of string_1 is empty at the first iteration.
Example :
#!/bin/bash
firstline='[Event "Marchand Open"][Site "Rochester NY"][Date "2005.03.19"][Round "1"][White "Smith, Igor"][Black "Jones, Matt"][Result "1-0"][ECO "C01"][WhiteElo "2409"][BlackElo "1911"]'
unset cccr
declare -A cccr
(IFS='['; for word in $firstline; do
string_1=$( echo $word | cut -f1 -d'"' )
string_2=$( echo $word | cut -f2 -d'"' )
echo "$string_1 - $string_2"
#cccr[$string_1]=$string_2
done)
Output :
- # Problem !
Event - Marchand Open
Site - Rochester NY
...
You have to modify your script to prevent the value of being empty.
A very simple workaround is to check the value of string_1 before using it.
Example :
# ...
string_1=$( echo $word | cut -f1 -d'"' )
string_2=$( echo $word | cut -f2 -d'"' )
if [ ! -z $string_1 ]; then # If $string_1 is not empty
echo "$string_1 - $string_2"
cccr[$string_1]=$string_2
fi
# ...
From the man page of [
-z STRING
the length of STRING is zero
Output :
Event - Marchand Open
Site - Rochester NY
# ... No problem
EDIT
BTW, if look at the value of string_1, you will see that the value is Event' ' and not Event (there's a whitespace at the end of Event)
So cccr[Event] does not exist, but cccr[Event ] exists.
To fix that, you can delete the whitespaces in string_1 :
string_1=$(echo $word | cut -f1 -d'"' | tr -d ' ') # tr -d ' ' deletes all the whitespaces
EDIT 2
I forgot to tell you that it's normal if it does not work. Indeed, the loop is executed in a subshell environment. So the array is filled in the subshell, but not in the current shell.
From the man page of bash :
(list) list is executed in a subshell environment (see COMMAND EXECUTION ENVIRONMENT below). Variable
assignments and builtin commands that affect the shell's environment do not remain in effect
after the command completes. The return status is the exit status of list.
So there are 2 solutions :
1. Don't run the loop in a subshell (remove the parentheses).
# ...
OLDIFS=$IFS
IFS='['
for word in $firstline; do
string_1=$(echo $word | cut -f1 -d'"' | tr -d ' ')
string_2=$(echo $word | cut -f2 -d'"')
if [ ! -z $string_1 ]; then
cccr[$string_1]=$string_2
fi
done
IFS=$OLDIFS
echo "Event = ${cccr[Event]}"
echo "Site = ${cccr[Site]}"
Output :
Event = Marchand Open
Site = Rochester NY
2. Use your array in the subshell.
# ...
(IFS='['
for word in $firstline; do
string_1=$(echo $word | cut -f1 -d'"' | tr -d ' ')
string_2=$(echo $word | cut -f2 -d'"')
if [ ! -z $string_1 ]; then # If $string_1 is not empty
cccr[$string_1]=$string_2
fi
done
echo "Event = ${cccr[Event]}"
echo "Site = ${cccr[Site]}"
)
Output :
Event = Marchand Open
Site = Rochester NY

store the values from command into an array bash

svn mergeinfo --show-revs eligible http://svn.test.com/INT_1.0.0/ http://svn.test.com/DEV/ | cut -d"r" -f2 | cut -d" " -f1
6097
6099
when i put this in a script, i get only last value but not all:
#!/usr/bin/bash
src_url="http://svn.test.com/INT_1.0.0/"
target_url="http://svn.test.com/DEV/"
eligible_revs=(`svn mergeinfo --show-revs eligible $src_url $target_url | cut -d"r" -f2 | cut -d" " -f1`)
echo ${eligible_revs[#]}
output:
6099
If you are running Cygwin the line endings can mess it up
$ foo=(`printf 'bar\r\nbaz'`)
$ echo ${foo[*]}
baz

Resources