I am trying to make a array/list from a bash output and then I want to for loop it. I keep on getting Syntax error: "(" unexpected (expecting "done"). If I had to put it in python term, I want to break the string up by \n and then for loop it.
IFS=$'\n'
DELETE = ($($MEGACOPY --dryrun --reload --download --local $LOCALDIR --remote $REMOTEDIR | sed 's|F '$LOCALDIR'|'$REMOTEDIR'|g'))
unset IFS
# And remove it
for i in $DELETE; do
$MEGARM $i
done
First, shell is not python. Spaces around equal signs don't work:
DELETE = ($($MEGACOPY --dryrun --reload --download --local $LOCALDIR --remote $REMOTEDIR | sed 's|F '$LOCALDIR'|'$REMOTEDIR'|g'))
When the shell sees the above, it interprets DELETE as a program name and = as its first argument. The error that you see is because the shell was unable to parse the second argument.
Replace the above with:
DELETE=($("$MEGACOPY" --dryrun --reload --download --local "$LOCALDIR" --remote "$REMOTEDIR" | sed 's|F '"$LOCALDIR"'|'"$REMOTEDIR"'|g'))
Second, regarding the for loop, DELETE is an array and arrays have special syntax:
for i in "${DELETE[#]}"; do
"$MEGARM" "$i"
done
Notes:
Unless you want word splitting and pathname expansion, all shell variables should be inside double-quotes.
It is best practices to use lower or mixed case for variable names. The system uses all upper case variables for its name and you don't want to accidentally overwrite one of them.
Related
How to fix the following code so that it can store the result of sed, which will replace the _
with -?
My code:
names=()
for entry_ in $foo
do
names+=($entry_ | sed -e "s/_/-/g")
done
echo names
You don't need sed for this, you can use bash's built-in parameter expansion + substitution capability to replace all _ characters with -: ${var//_/-}. You can even use it to do this for the entire list of elements in a single operation, but how you do it depends on what the source variable, foo, actually is.
If foo is an array (the much better way to do things), you can combine [#] ("get me all elements of the array") with the substitution:
names=( "${foo[#]//_/-}" )
If foo is a plain string, and you need to use word splitting to break it into elements for the array, you can do essentially the same thing without the [#] ('cause it's not an array) or the double-quotes (which prevent word splitting):
names=( ${foo//_/-} )
Note: I recommend avoiding word splitting if possible -- it often does something close to what you want, but almost never exactly what you want.
P.s. I third the recommendation of shellcheck. Among other things, it'll flag anything involving word splitting as a probable mistake.
This should be enough to get you there.
names=()
names+=$(echo "hello_world" | sed -e "s/_/-/g")
echo $names
Note that you need $ before echoing your variable.
Also. Look into installing shellcheck for your code editor and it will help you catch sneaky bugs and build better shell programming practices.
While I've handled this task in other languages easily, I'm at a loss for which commands to use when Shell Scripting (CentOS/BASH)
I have some regex that provides many matches in a file I've read to a variable, and would like to take the regex matches to an array to loop over and process each entry.
Regex I typically use https://regexr.com/ to form my capture groups, and throw that to JS/Python/Go to get an array and loop - but in Shell Scripting, not sure what I can use.
So far I've played with "sed" to find all matches and replace, but don't know if it's capable of returning an array to loop from matches.
Take regex, run on file, get array back. I would love some help with Shell Scripting for this task.
EDIT:
Based on comments, put this together (not working via shellcheck.net):
#!/bin/sh
examplefile="
asset('1a/1b/1c.ext')
asset('2a/2b/2c.ext')
asset('3a/3b/3c.ext')
"
examplearr=($(sed 'asset\((.*)\)' $examplefile))
for el in ${!examplearr[*]}
do
echo "${examplearr[$el]}"
done
This works in bash on a mac:
#!/bin/sh
examplefile="
asset('1a/1b/1c.ext')
asset('2a/2b/2c.ext')
asset('3a/3b/3c.ext')
"
examplearr=(`echo "$examplefile" | sed -e '/.*/s/asset(\(.*\))/\1/'`)
for el in ${examplearr[*]}; do
echo "$el"
done
output:
'1a/1b/1c.ext'
'2a/2b/2c.ext'
'3a/3b/3c.ext'
Note the wrapping of $examplefile in quotes, and the use of sed to replace the entire line with the match. If there will be other content in the file, either on the same lines as the "asset" string or in other lines with no assets at all you can refine it like this:
#!/bin/sh
examplefile="
fooasset('1a/1b/1c.ext')
asset('2a/2b/2c.ext')bar
foobar
fooasset('3a/3b/3c.ext')bar
"
examplearr=(`echo "$examplefile" | grep asset | sed -e '/.*/s/^.*asset(\(.*\)).*$/\1/'`)
for el in ${examplearr[*]}; do
echo "$el"
done
and achieve the same result.
There are several ways to do this. I'd do with GNU grep with perl-compatible regex (ah, delightful line noise):
mapfile -t examplearr < <(grep -oP '(?<=[(]).*?(?=[)])' <<<"$examplefile")
for i in "${!examplearr[#]}"; do printf "%d\t%s\n" $i "${examplearr[i]}"; done
0 '1a/1b/1c.ext'
1 '2a/2b/2c.ext'
2 '3a/3b/3c.ext'
This uses the bash mapfile command to read lines from stdin and assign them to an array.
The bits you're missing from the sed command:
$examplefile is text, not a filename, so you have to send to to sed's stdin
sed's a funny little language with 1-character commands: you've given it the "a" command, which is inappropriate in this case.
you only want to output the captured parts of the matches, not every line, so you need the -n option, and you need to print somewhere: the p flag in s///p means "print the [line] if a substitution was made".
sed -n 's/asset\(([^)]*)\)/\1/p' <<<"$examplefile"
# or
echo "$examplefile" | sed -n 's/asset\(([^)]*)\)/\1/p'
Note that this returns values like ('1a/1b/1c.ext') -- with the parentheses. If you don't want them, add the -r or -E option to sed: among other things, that flips the meaning of ( and \(
In bash I have an array with path names, and I would like to replace each of them with different ones using sed, like so:
sed 's#^(.*?)master_repo(.*?)#\1"${SOME_REPO_NAME}"\2#g' <<< ${FULL_TGT_DIRS[${i}]}
A sample path name which is an element of the array would be:
/Volumes/munki/master_repo/pkgs/apps
I would like to replace the path name "master_repo" with for example "somedir", which is stored in $SOME_REPO_NAME, so I get:
/Volumes/munki/somedir/pkgs/apps
Or with built in string substitution:
for i in ${FULL_TGT_DIRS[#]}
do
FULL_TGT_DIRS[$i]=${FULL_TGT_DIRS[$i]/master_repo/$SOME_REPO_NAME}
#sed 's#^(.*?)master_repo(.*?)#\1"${SOME_REPO_NAME}"\2#g' <<< ${FULL_TGT_DIRS[${i}]}
done
I always get the following error when running my script:
> /usr/local/bin/repomgr: line 135:
> /Volumes/munki/master_repo/pkgs/apps: syntax error: operand expected
> (error token is "/Volumes/munki/master_repo/pkgs/apps")
I've tried using different separaters and sed options, as well as shuffling through different quote constellations. I don't write bash scripts on a daily basis so perhaps I'm missing something?
BTW, I run this on a Mac and therefore only have bash 3.2 at my disposal.
There's no need to use sed for this, bash has built-in string replacement in its parameter expansion.
var=/Volumes/munki/master_repo/pkgs/apps
$SOME_REPO_NAME=somedir
newvar=${var/master_repo/$SOME_REPO_NAME}
In a for-in loop, the variable gets set to the array elements, not the array indexes, so you shouldn't be using FULL_TGT_DIRS[$i] -- $i contains the pathname. So the loop should be:
for file in ${FULL_TGT_DIRS[#]}
do
file=${file/master_repo/$SOME_REPO_NAME}
# Do something with $file here
done
If you need to modify the array in place, you need a different loop for the indexes:
for ((i = 0; i < ${#FULL_TGT_DIRS[#]}; i++))
do
FULL_TGT_DIRS[$i]=${FULL_TGT_DIRS[$i]/master_repo/"$SOME_REPO_NAME"}
done
You can even go a step further using bashes own replacement:
for file in "${FULL_TGT_DIRS[#]/master_repo/somedir}"
do
...work on file variable here...
done
Bash script to create multiple arrays from csv with unknown columns.
I am trying to write a script to compare two csv files with similar columns. I need it to locate the matching column from the other csv and compare any differences. The kicker is I would like the script to be dynamic to allow any number of columns to be entered and it still be able to function. I thought I had a good plan to solve this but turns out I'm running into syntax errors. Here is a sample of a csv I need to compare.
IP address, Notes, Nmap-SSH, Nmap-SMTP, Nmap-HTTP, Nmap-HTTPS,
10.0.0.1, , open, closed, open, open,
10.0.0.2, , closed, open, closed, closed,
When I read the csv file I was planning to look for "IF column == open; then; populate this column's array with the IP address" This would have given me 4 lists in this scenario with the IPs that were listening on said port. I could then compare that to my security device configuration to make sure it was configured properly. Finally to the meat, here is what I thought would accomplish creating the arrays for me to search later. However I ran into a snag when I tried to use a variable inside an array name. Can my syntax be corrected or is there just a better way to do this sort of thing?
#!/bin/bash
#
#
# This script compares config_cleaned_<ip>.txt output against ext_web_env.csv and outputs the differences
#
#
# Read from ext_web_env.csv file and create Array
#
FILENAME=./tmp/ext_web_env.csv
#
index=0
#
while read line
do
# How many columns are in the .csv?
varEnvCol=$(echo $line | awk -F, '{print NF}')
echo "columns = $varEnvCol"
# While loop to create array for each column
while [ $varEnvCol != 2 ]
do
# Checks to see if port is open; if so then add IP address to array
varPortCon=$(echo $line | awk -F, -v i=$varEnvCol '{print $i}')
if [ $varPortCon = "open" ]
then
arr$varEnvCol[$index]="$(echo $line | awk -F, '{print $1}')"
# I get this error message "line29 : arr8[194]=10.0.0.194: command not found"
fi
echo "arrEnv$varEnvCol is: ${arr$varEnvCol[#]}"
# Another error but not as important since I am using this to debug "line31: arr$varEnvCol is: ${arr$varEnvCol[#]}: bad substitution"
varEnvCol=$(($varEnvCol - 1))
done
index=$(($index + 1 ))
done < $FILENAME
UPDATE
I also tried using the eval command since all the data will be populated by other scripts.
but am getting this error message:
./compare.sh: line 41: arr8[83]=10.0.0.83: command not found
Here is my new code for this example:
if [[ $varPortCon = *'open'* ]]
then
eval arr\$varEnvCol[$index]=$(echo $line | awk -F, '{print $1}')
fi
arr$varEnvCol[$index]="$(...)"
doesn't work the way you expect it to - you cannot assign to shell variables indirectly - via an expression that expands to the variable name - this way.
Your attempted workaround with eval is also flawed - see below.
tl;dr
If you use bash 4.3 or above:
declare -n targetArray="arr$varEnvCol"
targetArray[index]=$(echo $line | awk -F, '{print $1}')
bash 4.2 or earlier:
declare "arr$varEnvCol"[index]="$(echo $line | awk -F, '{print $1}')"
Caveat: This will work in your particular situation, but may fail subtly in others; read on for details, including a more robust, but cumbersome alternative based on read.
The eval-based solution mentioned by #shellter in a since-deleted comment is problematic not only for security reasons (as they mentioned), but also because it can get quite tricky with respect to quoting; for completeness, here's the eval-based solution:
eval "arr$varEnvCol[index]"='$(echo $line | awk -F, '\''{print $1}'\'')'
See below for an explanation.
Assign to a bash array variable indirectly:
bash 4.3+: use declare -n to effectively create an alias ('nameref') of another variable
This is by far the best option, if available:
declare -n targetArray="arr$varEnvCol"
targetArray[index]=$(echo $line | awk -F, '{print $1}')
declare -n effectively allows you to refer to a variable by another name (whether that variable is an array or not), and the name to create an alias for can be the result of an expression (an expanded string), as demonstrated.
bash 4.2-: there are several options, each with tradeoffs
NOTE: With non-array variables, the best approach is to use printf -v. Since this question is about array variables, this approach is not discussed further.
[most robust, but cumbersome]: use read:
IFS=$'\n' read -r -d '' "arr$varEnvCol"[index] <<<"$(echo $line | awk -F, '{print $1}')"
IFS=$'\n' ensures that that leading and trailing whitespace in each input line is left intact.
-r prevents interpretation of \ chars. in the input.
-d '' ensures that ALL input is captured, even multi-line.
Note, however, that any trailing \n chars. are stripped.
If you're only interested in the first line of input, omit -d ''
"arr$varEnvCol"[index] expands to the variable - array element, in this case - to assign to; note that referring to variable index inside an array subscript does NOT need the $ prefix, because subscripts are evaluated in arithmetic context, where the prefix is optional.
<<< - a so-called here-string - sends its argument to stdin, where read takes its input from.
[simplest, but may break]: use declare:
declare "arr$varEnvCol"[index]="$(echo $line | awk -F, '{print $1}')"
(This is slightly counter-intuitive, in that declare is meant to declare, not modify a variable, but it works in bash 3.x and 4.x, with the constraints noted below.)
Works fine OUTSIDE a FUNCTION - whether the array was explicitly declared with declare or not.
Caveat: INSIDE a function, only works with LOCAL variables - you cannot reference shell-global variables (variables declared outside the function) from inside a function that way. Attempting to do so invariably creates a LOCAL variable ECLIPSING the shell-global variable.
[insecure and tricky]: use eval:
eval "arr$varEnvCol[index]"='$(echo $line | awk -F, '\''{print $1}'\'')'
CAVEAT: Only use eval if you fully control the contents of the string being evaluated; eval will execute any command contained in a string, with potentially unwanted results.
Understanding what variable references/command substitutions get expanded when is nontrivial - the safest approach is to delay expansion so that they happen when eval executes rather than immediate expansion that happens when arguments are passed to eval.
For a variable assignment statement to succeed, the RHS (right-hand side) must eventually evaluate to a single token - either unquoted without whitespace or quoted (optionally with whitespace).
The above example uses single quotes to delay expansion; thus, the string passed mustn't contain single quotes directly and thus is broken into multiple parts with literal ' chars. spliced in as \'.
Also note that the LHS (left-hand side) of the assignment statement passed to eval must be a double-quoted string - using an unquoted string with selective quoting of $ won't work, curiously:
OK: eval "arr$varEnvCol[index]"=...
FAILS: eval arr\$varEnvCol[index]=...
I am processing some folders that each represent a page of a book. E.g. "Iliad-001" would be Book=Iliad, Page=001.
I want to iterate through all of the folders, create an array for each book and add an entry to that array for each page that is found, so that I can echo ${Iliad[#]} at the end of my script and it will give me a nice list of all the pages it found.
The catch I'm having is adding values to an array with a dynamic name. Here's the code that I think is intuitive (but clearly not right):
for j in */; do
vol_name=$(basename "$j" | sed 's/\(.*\)-[0-9]*/\1/')
page_name=$(basename "$j" | sed 's/.*-\([0-9]*\)/\1/')
$vol_name+=( "$page_name" )
done
This returns:
syntax error near unexpected token `"$page_name"'
If I change the variable assignment to this $vol_name+="( "$page_name" )" I get a little closer:
Iliad+=( 001 ): command not found
I was able to make it work using eval.
BTW, you do not need to run sed.
#! /bin/bash
for j in */; do
j=$(basename "$j")
vol_name=${j%-*}
page_name=${j#*-}
eval "$vol_name+=('$page_name')"
done
echo ${Iliad[#]}
try this
declare $vol_name+=( "$page_name" )