Curl, Adding hostname from array - arrays

I am trying to write a script that parses the output of a URL. Everything works great if I hard code a URL into the curl command, however when I try to add the URL from an array I am getting a syntax error.
Here is the script
#!/usr/bin/bash
hosts=(
room-f3185-dsp
room-c5180-dsp
room-f3187-dsp
room-f3183-dsp
)
for i in "${hosts[#]}"
url1=$(curl -d "action=login&userPassword=PASSWORD" -H "Content-Type: application/x-www-form-urlencoded" -X POST http://$i)
url2=$(echo $url | sed 's/,/\n/g')
url3=$(echo $url2 | sed 's/:/ /g' | grep macAddress ./mactest | awk '{print $2,$3,$4,$5,$6,$7,$8}')
url4=$(echo $url3 | sed 's/ /:/g' | tr -d '"')
mac=$(echo $url4 | sed 's/.$//' | sed 's/.$//' |sed 's/.$//')
echo "------------------------------------------------" >> dspmac
echo $i >> dspmac
echo "Mac Address = "$mac >> dspmac
done
Here is the error I am getting,
line 10: syntax error near unexpected token `url1=$(curl -d "action=login&userPassword=PASSWORD" -H "Content-Type: application/x-www-form-urlencoded" -X POST $i)'
Also in the url1 line when I put in HTTP:// it is highlighted as a link, but the $i is just showing as standard text
Thank in advance
If I replace the $i in the script with the full URL (http://room-f3185-dsp) it works great. I have also tried putting the full url as the host in the array, but still get the same error.
I also tried various quotes in different places.

How about adding do:
for i in "${hosts[#]}"
do
...
done

Related

get each line of a command in an array bash raspbian

I'm trying to improve my scripting skills.
I know there's online converters that are doing it but i'd like to make a script that compare in a certain folder (/opt/Citrix/ICAClient/keystore/cacerts/), if the .crt are already in the .pem format and if not convert them.
So i know this clearly isn't the best way to do so... But i've started like this:
`
#!/bin/bash
#remove both files if they already exists
rm crt.txt
rm pem.txt
#certificate selection in .crt format
Certificate_crt=$(sudo ls /opt/Citrix/ICAClient/keystore/cacerts/ | grep .crt | sed 's/\(.*\)\..*/\1/')
#certificate selection in .pem format
Certificate_pem=$(sudo ls /opt/Citrix/ICAClient/keystore/cacerts/ | grep .pem | sed 's/\(.*\)\..*/\1/')
#sending results in text files
echo "$Certificate_crt" >> crt.txt
echo "$Certificate_pem" >> pem.txt
#recovery of certificate names in .crt not having a .pem equivalent
Certificate_crt_WO_pem=$(cat crt.txt | grep -v "$Certificate_pem" | tr " " "\n")
#echo "$Certificate_crt_WO_pem"
#initialisation
i=0
#Print the split string
for i in "${Certificate_crt_WO_pem[#]}"
do
name_certificate=$(echo $i | tr " " "\n")
echo "${name_certificate[#]}"
echo "$i"
i=i+1
done
`
The thing is that when my "for" is launched, it stores all the result of "Certificate_crt_WO_pem" in the array $name_certificate[0] and then stop it self.
What i want is to store, line by line, the result of "cat crt.txt | grep -v "$Certificate_pem" | tr " " "\n"" into the array name_certificate.
This array will be use to launch something like this " openssl -in $name_certificate[$i].crt -out $name_certificate[$i].pem PEM" (in a for loop in the will to convert every namefile.crt in every namefile.pem).
If someone can help me i'll be more than gratefull... (And yes i've already tried to search on the net, had followed some online courses but none of them was saying the same thing about the bash's arrays, so i'm a bit lost...)
What i want is to store, line by line, the result of
Then do that.
var=$(something)
# ^^ - normal variable assignment
var=( $(something) )
# ^^ ^ - array assignment
# the unquoted result of expansions is split on IFS
# default on tabs, newlines and spaces
so I guess you want:
Certificate_crt_WO_pem=($(grep -v "$Certificate_pem" crt.txt))
Doing cat file | grep is a useless use of cat. Use grep .. file or < file grep ....
Remember do not parse ls output. Don't ls | something. Prefer globulation or find instead.
Read how to read a stream line by line in bashfaq. Read how to use arrays in bashfaq.
Note that grep parses a regex, so grep .pem matches any character followed by pem. I guess you wanted grep '\.pem'. I do not think grep -v "$Certificate_pem" does what you think it does - I guess you meant to use comm or join to filter elements from one newline separated separated that are present in the other list.
This array will be use to launch something like this " openssl -in $name_certificate[$i].crt -out $name_certificate[$i].pem PEM"
The best would be not do it, rather parse the data as they come, not store them in variables.
# get all .crt fiels
find /opt/Citrix/ICAClient/keystore/cacerts/ -maxdepth 1 -mindepth 1 -type f -name '*.crt' |
# remove extension
sed 's/\.crt$//' |
# filter out files where .pem does exists already
while IFS= read -r file; do
if [[ -r "$file".pem ]]; then continue; fi
printf "%s\n" "$file"
done |
# execute openssl for the rest
xargs -d'\n' -i{} openssl -in {}.crt -out {}.pem PEM
but if you want, rather use mapfile to store a string with a newline separated list into an array (be sure to understand how to store variables in a pipeline).
mapfile -t Certificate_crt_WO_pem < <(something)

How can I edit my code so that I can account for output that goes against my sed command

I am writing a code to put the species named matched from a remote NCBI BLAST database, and the file the matched name came from. I want to make my code more robust so that it can deal with files that do not get a match and that go against my current sed command
#!/bin/bash
for i in ./split.contigs.Parsed/*.csv ; do
sciname=$(head -1 $i | sed -E "s/([A-Z][a-z]+ [a-z]+) .+/\1/")
contigname=$(echo $i | sed -E "s/.fa.csv//" | sed -E
"s/\.\/split.contigs.Parsed\///")
echo "$sciname,$contigname"
done
Expected
Drosophila melanogaster,contig_66:1.0-213512.0_pilon
Drosophila melanogaster,contig_67:1.0-138917.0_pilon
Drosophila sechellia,contig_67:139347.0-186625.0_pilon
Drosophila melanogaster,contig_68:3768.0-4712.0_pilon
Actual
Drosophila ananassae,contig_393:1.0-13214.0_pilon
,contig_393:13217.0-13563.0_pilon
Drosophila sp. pallidosa-like-Wau w,contig_393:14835.0-18553.0_pilon
Apteryx australis,contig_393:19541.0-21771.0_pilon
,contig_393:21780.0-22772.0_pilon
Drosophila sp. pallidosa-like-Wau w,contig_393:22776.0-31442.0_pilon
Drosophila melanogaster,contig_394:1.0-89663.0_pilon
Simply skip the loop if $sciname is null. Put this one line after defining $sciname:
[[ -z $sciname ]] && continue

using command variable in column output not working

I'm using a script that uses curl to obtain specific array values from a configuration. I'd like to place the output into columns separating values (values are unknown to script). Here's my code:
# get overlay networks and their details
get_overlay=`curl -H "X-Person-Token: $auth_token" -H "X-Person-Email: $auth_email" -k "$api_host/api/v1/networks"`
# array of overlay names with uuid
overlay_name=`echo $get_overlay | jq '.[] | .name'`
overlay_uuid=`echo $get_overlay | jq '.[] | .uuid'`
echo ""
echo -e "Overlay UUID\n$oname $ouuid" | column -t
exit 0
Here's the ouput:
Overlay UUID
"TESTOVERLAY"
"Auto_API_Overlay"
"ANOTHEROVERLAYTEST" "ea178905-6ab0-4154-ab05-412dc4b39151"
"e5be9dbe-b0fc-4e30-aaf5-ac4bdcd863a7"
"850ebf6b-3651-4cf1-aae1-5a6c03fad61b"
What I was expecting was:
Overlay UUID
"TESTOVERLAY" "ea178905-6ab0-4154-ab05-412dc4b39151"
"Auto_API_Overlay" "e5be9dbe-b0fc-4e30-aaf5-ac4bdcd863a7"
"ANOTHEROVERLAYTEST" "850ebf6b-3651-4cf1-aae1-5a6c03fad61b"
I'm an absolute beginner at this, any insight is very much appreciated.
Thanks!
I would suggest using paste to combine your two variables line by line:
paste <(printf 'Overlay\n%s\n' "$name") <(printf 'UUID\n%s\n' "$uuid") | column -t
Two process substitutions are used to pass the contents of each variable along with their titles.

Output of wget (to CouchDB) as array used in for-loop

I try to use the output of wget to build an array I can use for a for loop:
RESULT="`wget -qO- http://myCouchDBServer.com:5984/_all_dbs`"
echo $RESULT
What I get is something like:
["_replicator","_users","mydb1","mydb2","mydb3","mydb4","mydb5"]
At the moment I loop through an array i wrote manually:
databases=(mydb1 mydb2 mydb2)
for var in "${databases[#]}"
do
# echo "${var}"
# do something on $var
filename="${var}_$(date +%Y_%m_%d_%H_%M_%S).json"
echo $filename
bash couchdb-backup.sh -b -H myHostIP_Adress -d ${var} -f $filename -u myUsername -p myPassword
done
Now I would like to use the response from wget and build an array from this response which I can use for my for-loop.
I tried
RESULT2= echo "$RESULT" | sed 's/\(\[\|\]\)//g'
to get rid of the square brackets and tried to build an array
but there is no output and therefore also no loop running...
STR_ARRAY=(`echo $RESULT2 | tr "," "\n"`)
for x in "${STR_ARRAY[#]}"
do
echo "> [$x]"
done
Any idea how to get a step further?
PS:
(my couchdb is only reachable through intranet but I found an url for testing: http://example.iriscouch.com/_all_dbs )
Since couch is spitting out JSON, you'll want to process it with a JSON parser. For example, jq:
echo '["_replicator","_users","mydb1","mydb2","mydb3","mydb4","mydb5"]' | jq -r '.[]|.'
_replicator
_users
mydb1
mydb2
mydb3
mydb4
mydb5
So, you could do this:
mapfile -t all_dbs < <(
wget -qO- http://myCouchDBServer.com:5984/_all_dbs | jq -r '.[]|.'
)
for db in "${all_dbs[#]}"; do ...; done

How to grep the output of grep?

After using bash -x to debug this script, it appears that a problem occurs near $InvalidText, where grep treats the output of InvalidText as separate file names.
The only solution I can think of is to direct the output of InvalidText to a file, then grep that file.. are there any other alternatives?
# find IP address of Invalid user and give the fake name they tried. Loop for all IP addresses
InvalidText=$(grep "Invalid user" syslog.log)
IPInvalidUser=$(grep "Invalid user" syslog.log | awk '{print $NF}' | uniq)
InvalidUserarray=( $IPInvalidUser )
for FAKEHOST in "${InvalidUserarray[#]}"
do
fakeinfo=$(grep ${FAKEHOST} $InvalidText | cut -d: -f5 | awk '{print $3}')
echo -e "Invalid user attempts:\n"
echo -e "$FAKEHOST $fakeinfo"
done

Resources