Lets say I have the following file named "test"
which contains:
[a1b2c3]=test1.xyz
[d4e5f6]=test2.xyz
[g7h8i9]=test3.xyz
...
Which is about 30 lines long. I would want to assign
a1b2c3
d4e5f6
g7h8i9
into an array named array how would I do it?
with bash 3.1 this would be my attempt
declare -a array
while read -r line;
do
array+=("$(echo "$line")")
done < test
But it seems that I need to upgrade my bash to 4 or 5 for declarative arrays. So the question would be, how do I separate specifically get the values inside the brackets per line to be assigned as different elements in an array array
This array values would be used in a curl script which should produce a file with the name of the values associated with it.
for i in "${array[#]}"
do
curl -X GET "https://api.cloudflare.com/client/v4/zones/$i/dns_records?" \
-H "X-Auth-Email: $X-Auth-Email" \
-H "X-Auth-Key: X-Auth-Key" \
-H "Content-Type: application/json" \
-o "${array[#]}.txt"
done
so the expectation is that curl -X GET "https://api.cloudflare.com/client/v4/zones/a1b2c3/dns_records? would be run and an output file named test1.xyz.txt would be produced containing the dns records of the said link. Which goes for the next line and test2.xyz.txt and so on and so forth.
Assuming the string inside the brackets is composed of alphanumeric
characters, would you please try:
#!/bin/bash
while IFS== read -r name val; do
name=${name#[} # remove left square bracket
name=${name%]} # remove right square bracket
printf -v "var_$name" %s "$val" # assign e.g. var_a1b2c3 to test1.xyz
done < test
for var in "${!var_#}"; do
name=${var#var_} # retrieve e.g. a1b2c3 by removing var_
curl -X GET "https://api.cloudflare.com/client/v4/zones/$name/dns_records?" \
-H "X-Auth-Email: $X-Auth-Email" \
-H "X-Auth-Key: X-Auth-Key" \
-H "Content-Type: application/json" \
-o "${!var}.txt"
done
As an indirect variable assignment, printf -v varname value works.
As an indirect variable referencing, "${!var_#}" is expanded
to a list of variable names which starts with var_.
${!var} refers to the value whose variable name is expressed by "$var".
Tested with bash-3.2.57. Hope it will be interoperable with bash-3.1.
I have been populating an array using:
AWS_STS_CREDS=( $(aws sts ...) )
This raises shellcheck error SC2207
Prefer mapfile or read -a to split command output
But the recommendation does not work as expected.
IFS=" " read -r -a AWS_STS_CREDS <<< "$(
aws sts assume-role \
--role-arn ${AWS_ROLE_ARN} --role-session-name ${AWS_SESSION_NAME} \
--query '[Credentials.AccessKeyId,Credentials.SecretAccessKey,Credentials.SessionToken]' \
--output text
)"
echo "Array contains ${#AWS_STS_CREDS[#]} elements"
#> Array contains 1 elements
echo "${AWS_STS_CREDS[0]}"
#> ASIAV2R3U... 4gXdep/GN... FwoGZXI...
I have also tried removing the quotes around the subcommand.
At first it appears as though the set IFS is having no effect but the below works:
IFS=" " read -r -a AWS_STS_CREDS <<< "$(echo '1 2 3')"
I am overlooking something but I'm having trouble identifying the problem and would like to understand the behaviour.
As per #Cyrus's suggestion, piping the subcommand output to cat -A clearly shows it is tab delimited. Indicated by ^I
echo "${AWS_STS_CREDS[0]}"
#> ASIAV2R3U...^I4gXdep/GN...^IFwoGZXI...
Amending the script as follows works as expected:
IFS=$'\t' read -r -a AWS_STS_CREDS <<< "$(
aws sts assume-role \
--role-arn ${AWS_ROLE_ARN} --role-session-name ${AWS_SESSION_NAME} \
--query '[Credentials.AccessKeyId,Credentials.SecretAccessKey,Credentials.SessionToken]' \
--output text
)"
echo "Array contains ${#AWS_STS_CREDS[#]} elements"
#> Array contains 3 elements
echo "${AWS_STS_CREDS[0]}"
#> ASIAV2R3U...
Hello Stackoverflow community,
Please forgive me for my naïveté, but I have a basic loop script running a more complex script that is looking to a text file for the inputs that I have as an array. I have it working, I guess, but I know this can be ran better and more automated.
Here is the text file my script looks to;
2014;14204;
2015;15042;
2015;15062;
...
end;
Here is the bash script I run as a loop;
{ while IFS=';' read YEAR1 PROJ1 YEAR2 PROJ2 YEAR3 PROJ3 fake
do
{ echo "$YEAR1" | egrep '^#|^ *$' ; } > /dev/null && continue
$local/script.sh \
--forinput1 $YEAR1/$PROJ1 \
--forinput2 $YEAR2/$PROJ2 \
--forinput3 $YEAR3/$PROJ3 \
done
} < textFile.txt
I have done some research myself and have found somethings I thought would work but haven't been able to properly implement into this. If you could some me some pointers I would appreciate it.
Edited:
I do apologize, so the script does recognize the text file as such:
YEAR1;PROJ1;
YEAR2;PROJ2;
YEAR3;PROJ3;
Using the ";" as its separator. It does run in a loop until the last variable it done. However, for it function I need to add to the text file extra lines
YEAR4;PROJ4;
YEAR5;PROJ5;
end;
then add the script
{ while IFS=';' read YEAR1 PROJ1 YEAR2 PROJ2 YEAR3 PROJ3 YEAR4 PROJ4 YEAR5 PROJ5 fake
do
{ echo "$YEAR1" | egrep '^#|^ *$' ; } > /dev/null && continue
$local/script.sh \
--forinput1 $YEAR1/$PROJ1 \
--forinput2 $YEAR2/$PROJ2 \
--forinput3 $YEAR3/$PROJ3 \
--forinput4 $YEAR4/$PROJ4 \
--forinput5 $YEAR5/$PROJ5 \
done
} < textFile.txt
What I am hoping to accomplish is adding the variables in the array but not having to add the extra syntax into the script
This is broken but I guess what I am looking along the lines of
{ while IFS=';' read -a YEAR PROJ < textFile.txt
for ((i = 0; i < "${#YEAR[#]};${#PROJ[#]}"; i++)); do
{ echo "$YEAR[$i]" | egrep '^#|^ *$' ; } > /dev/null && continue
$local/script.sh \
--forinput[$i] ${YEAR[$i]}/${PROJ[$i]} \
done
}
Your code suggests that each inputline has 6 fields, what might lead to code like
sed -nr 's#([^;]*);([^;]*);([^;]*);([^;]*);([^;]*);([^;]*);#$local/script.sh --forinput \1/\2 --forinput \3/\4 --forinput \5/\6#p' textFile.txt
# or shorter
f='([^;]*;)'
sed -nr 's#'$f$f$f$f$f$f'#$local/script.sh --forinput \1/\2 --forinput \3/\4 --forinput \5/\6#p' textFile.txt
When you have to combine 3 lines for the input, you should not try to be smart.
# Don't do this
cat textFile.txt | paste -d'X' - - - | tr -d 'X'
# Trying to make
sed -nr 's#'$f$f$f$f$f$f'#$local/script.sh --forinput \1/\2 --forinput \3/\4 --forinput \5/\6#p' <(cat textFile.txt | paste -d'X' - - - | tr -d 'X')
After proudly presenting the code, you see you will have to make the code even worse, when the second line is a comment (starts with '#'). You need to replace cat textFile.txt with grep -Ev '^#|^ *$' testFile.txt.
When you need to relate different lines, take a look at awk.
The solution without checks is
awk -F';' '{line++}
{param=param " --forinput " $1 "/" $2}
line==3 {print "$local/script.sh" param ; param=""; line=0}
' textFile.txt
You can add all kind of checks.
I'm going to assume your input file contains groups of 3 lines each with 2 fields, not lines of 6 fields.
$ cat file
y1a;p1a;
y2a;p2a;
y3a;p3a;
y1b;p1b;
y2b;p2b;
y3b;p3b;
Then, you can put multiple read commands as the while "condition":
while
IFS=';' read year1 proj1 x
IFS=';' read year2 proj2 x
IFS=';' read year3 proj3 x
do
echo script \
--forinput1 "$year1/$proj1" \
--forinput2 "$year2/$proj2" \
--forinput3 "$year3/$proj3"
done < file
script --forinput1 y1a/p1a --forinput2 y2a/p2a --forinput3 y3a/p3a
script --forinput1 y1b/p1b --forinput2 y2b/p2b --forinput3 y3b/p3b
However, this does not handle comments and blank lines. This version executes the script after the 3rd non-(comment/blank) line
$ cat file
# set 1
y1a;p1a;
y2a;p2a;
y3a;p3a;
# set 2
y1b;p1b;
y2b;p2b;
y3b;p3b;
and
n=0
args=()
while IFS=';' read year project x; do
{ [[ $year == *([[:blank:]])"#"* ]] || [[ $year == *([[:blank:]]) ]]; } && continue
((n++))
args+=( "--forinput$n" "$year/$project" )
if (( n == 3 )); then
echo script "${args[#]}"
args=()
n=0
fi
done < file
script --forinput1 y1a/p1a --forinput2 y2a/p2a --forinput3 y3a/p3a
script --forinput1 y1b/p1b --forinput2 y2b/p2b --forinput3 y3b/p3b
Another approach to handle arbitrary records per group:
$ cat file
# set 1
y1a;p1a;
y2a;p2a;
y3a;p3a;
y4a;p4a;
y5a;p5a;
end;
# set 2
y1b;p1b;
y2b;p2b;
y3b;p3b;
end;
$ grep -Pv '^\s*(#|$)' file | awk -F"\n" -v RS="\nend;\n" -v OFS=, '{
cmd = "script.sh"
for (i=1; i<=NF; i++) {
n = split($i, a, /;/)
cmd = sprintf( "%s --forinput%d \"%s/%s\"", cmd, i, a[1], a[2])
}
print cmd
}'
script.sh --forinput1 "y1a/p1a" --forinput2 "y2a/p2a" --forinput3 "y3a/p3a" --forinput4 "y4a/p4a" --forinput5 "y5a/p5a"
script.sh --forinput1 "y1b/p1b" --forinput2 "y2b/p2b" --forinput3 "y3b/p3b"
That uses grep to filter out the comments and blank lines. Then awk comes it to format the commands:
RS is the record separator, using the end; line
-F"\n" sets the field separator to a newline
then we iterate over the number of fields (NF) to construct the command you want to run, and print it out.
To actually execute it, pipe the awk output to | sh
I am not good at unix.
I have a csv file which I is having multiple columns. Out of which, one column is containing new line and ^M chars. I need to replace all of them between two " (which is a single cell value) by ~~ so that I can treat the cell value as single field. Here is the sample file :
"id","notes"
"N001","this is^M
test.
Again test
"
"N002","this is perfect"
"N00345","this is
having ^M
problem"
I need this file like :
"id","notes"
"N001","this is~~test.~~~~Again test~~~~"
"N002","this is perfect"
"N00345","this is~~~~having ~~problem"
So that the whole cell value can be read as a single field value.
I need to add one more case in this requirement where data within a cell contains " (double quotes). Here in this case we can identify ending " when it is followed by comma. Here are updated case data :
"id","notes"
"N001","this is^M
test. "Again test."
Again test
"
"N002","this is perfect"
"N00345","this is
having ^M
problem as it contains "
test"
We can keep " or remove it. The expected output is :
"id","notes"
"N001","this is~~test. "Again test."~~~~Again test~~~~"
"N002","this is perfect"
"N00345","this is ~~~~having ~~problem as it contains "~~test"
Try using sed
sed -i -e 's/^M//g' -e '/"$/!{:a N; s/\n/~~/; /"$/b; ba}' file
Note : To enter ^M, type Ctrl+V followed by Ctrl+M
File content after running command
"id","notes"
"N001","this is~~test.~~~~Again test~~~~"
"N002","this is perfect"
"N00345","this is~~~~having ~~problem"
Or using dos2unix followed by sed
dos2unix file
sed -i '/"$/!{:a N; s/\n/~~/; /"$/b; ba}' file
Short Description
Idea here is to remove newline character in each line not ending with "
sed -i ' # -i specifies in-place relace i.e. modifies file itself
/"$/!{ # if a line doesn't contain end pattern, " at the end of a line, then do following
:a # label 'a' for branching/looping
N; # append the next line of input into the pattern space
s/\n/~~/; # replace newline character '\n' with '~~' i.e. suppress new lines
/"$/b; # if a line contains end pattern then branch out i.e. break the loop
ba # branch to label 'a' i.e. this will create loop around label 'a'
}
' file # input file name
Refer to man sed for further details
EDIT
Sometimes data in the cell itself contains " within it.
Using sed
sed -i ':a N; s/\n/~~/; $s/"~~"/"\n"/g; ba' file
File content after running command for updated case data
"id","notes"
"N001","this is~~test. "Again test."~~~~Again test~~~~"
"N002","this is perfect"
"N00345","this is~~~~having ~~problem as it contains "~~test"
Using perl one-liner
perl -0777 -i -pe 's/\n/~~/g; s/"~~("|$)/"\n$1/g;' file
You can do this using sed command
To replace '^M' alone
sed -i 's|^M|~~|g' file_name
Edit:
Thanks for giving comment.
Adding a statement to replace '^M and new line'
To replace '^M and new line'**
sed -i ':a;N;$!ba;s|^M\n|~~|g' file_name
To get '^M' in console you should press Cntrl+v+m together
Use tr.
$ tr '<Ctrl>+m' '~'
sed 's/\^M/~~/;t nextline
b
: nextline
N
s/\n/~~/
s/^[^"]*\("[^"]*"\}\{1,\}[^"]*$
t
b nextline
"
not just change the ^M but also the new line between quote.
^M is obtain in unix session with a CTRL+V followed by a CTRL+M on keyboard
I have 2 files like below:
file1:
a1,b1,c1,d1,e1,f1,g1,h1
a2,b2,c2,d2,e2,f2,g2,h2
a3,b3,c3,d3,e3,f3,g3,h3
a4,b4,c4,d4,e4,f4,g4,h4
file2:
x1,y1,z1
x2,y2,z2
x3,y3,z3
x4,y4,z4
I want to read simultaneously from both and output the variables in a pattern like below:
a1,b1,c1,d1,x1,e1,f1,y1,g1,z1,h1
a2,b2,c2,d2,x2,e2,f2,y2,g2,z2,h2
a3,b3,c3,d3,x3,e3,f3,y3,g3,z3,h3
a4,b4,c4,d4,x4,e4,f4,y4,g4,z4,h4
Good news - I've managed to achieve it !!
Bad news - Too many arrays and while loops (too much computation!). I am looking for something simpler as the script will have to read through much data (4k lines and 1M words).
Limitation - BASH shell (probably not a limitation!)
This is what I've done
exec 5<file1 # Open file into FD 5
exec 6<file2 # Open file into FD 6
while IFS=$"," read -r line1 <&5
IFS=$"," read -r line2 <&6
do
array1=( `echo $line1` )
array2=( `echo $line2` )
array3=("${array1[#]}","${array2[#]}")
echo ${array3[#]} >> tmpline
done
while IFS="," read var1 var2 var3 var4 var5 var6 var7 var8 var9 var10 var11
do
echo -e "$var1,$var2,$var3,$var4,$var9,$var5,$var6,$var10,$var8,$var11,$var9" >> tcomb
done < tmpline
exec 5<&- # Close FD 5
exec 6<&- # Close FD 6
Thanks in advance -- I'm waiting patiently :) !!
Try this:
exec 5<file1 # Open file into FD 5
exec 6<file2 # Open file into FD 6
while IFS=, read -a t <&5 &&
IFS=, read -a u <&6
do
echo -n "${t[0]},${t[1]},${t[2]},${t[3]},${u[0]},${t[4]},"
echo "${t[5]},${u[1]},${t[6]},${u[2]},${t[7]}"
done >| tcomb
exec 5<&- # Close FD 5
exec 6<&- # Close FD 6
You can use paste to combine the lines of the files. Then, you have to reorder the columns, I used Perl for that:
paste file1 file2 -d, | \
perl -F, -ane 'chomp $F[-1]; $"=","; print "#F[0..3,8,4,5,9,6,10,7]\n"'
If you allow yourself to read the files more than once, and using bash process substitution:
paste -d , <(cut -d , -f 1-4 file1) \
<(cut -d , -f 1 file2) \
<(cut -d , -f 5-6 file1) \
<(cut -d , -f 2 file2) \
<(cut -d , -f 7 file1) \
<(cut -d , -f 3 file2) \
<(cut -d , -f 8 file1)