I started now with the bash script and I'm trying to make a for loop that uses an array where there are the arguments passed to the script (2 or more files) trying to "hang" with the ">>" command, the first ones "n-1" files passed as argument to the last file (n) , writing the input files in the order from right to left
for example :
myscript.sh file1 file2 file3 file4
file4 will contain in sequence file3 file2 file1
arr=($#)
j=$#-1
for i in { j-1..0..1 }
do
cat arr[i] >> $[j]
done
I tried to do it this way but it doesn't work, can someone help me?
Try:
#!/bin/bash
n=$#
for i in $(seq $((n - 1)) -1 1) ; do
cat ${#:$i:1} >> ${#:$n:1}
done
Explanation:
seq $((n - 1)) -1 1 generates the numbers from $n - 1 to 1 in reverse order.
${#:$i:1} gives the element $i of the array of the arguments of the script.
Or, without seq:
#!/bin/bash
n=$#
for ((i = n - 1; i; i--)) ; do
echo "cat ${#:$i:1} >> ${#:$n:1}"
done
Or even shorter, with a while loop instead of a for loop, and maybe off topic for this reason:
#!/bin/bash
i=${#}
while ((--i)) ; do
echo "cat ${#:$i:1} >> ${#:$#:1}"
done
Not sure what about "hang", but this concatenates the first files to the last filename
cat "${#:0:$#}" > "${#: -1:1}"
this is slicing the the arguments.
You could print all arguments except the last one on separate lines, reverse the files order and then cat all of them into the file that is the last argument:
printf "%s\n" "${#:1:$#-1}" | tac | xargs cat > "${#: -1}"
The same using zero as the stream separator:
printf "%s\0" "${#:1:$#-1}" | tac -s '' | xargs -0 cat > "${#: -1}"
It would be very advisable to protect against the count of arguments less then 2, because when $# = 0 then ${#: -1} will be $0. Then $0 could be your shell with ex full path like /bin/bash, so > could overwrite your shell executable file!
So do:
if (($# >= 2)); then printf "%s\0" "${#:1:$#-1}" | tac -s '' | xargs -0 cat > "${#: -1}"; fi
Related
This question already has answers here:
Using awk with variables
(3 answers)
How do I use shell variables in an awk script?
(7 answers)
Closed 1 year ago.
I am attempting to develop a nested for loop which will run an awk command with two sets of integers over a .txt file. The loop(s) should do the following:
For lines that are equal to or larger than a given length i and a given percentage j, print column 2
Count the number of unique lines
Output count to file
I've created this:
lens=(1000 2000 3000 5000)
percent=(80 90)
for i in ${lens[#]}
do
for j in ${percent[#]}
do
echo "Length is $i and percent is $j"
echo "Where length =>$i and % ID is >=$j, number of matches is: " >> output.txt
awk '{if ($4>=$i && $3>=$j) print $2}' input.txt | uniq | wc -l >> output.txt
awk '{if ($4>=1000 && $3>=80) print $2}' input.txt | uniq | wc -l >> output.txt
done
done
For some reason, using the variables i and j cause my outputs to always be 0, (even when they shouldn't be) - for example, the output from the second awk command returns the correct value, even though ostensibly the two lines should be equivalent during the first iteration of the loop. See the beginning of the output file:
Where length =>1000 and % ID is >=80, number of matches is:
0
775
The sense-check echo "Length is $i and percent is $j" prints normal output: Length is 1000 and percent is 80. Same for the second echo "Where length>=$i..." so I'm really stumped. Why should the presence of the array variables be causing problems in awk?
EDIT: Well, as usual, the answer was painfully simple and came down to a few ''. The proper code is below; note the shell variables $i and $j have been surrounded with '':
for i in ${lens[#]}
do
for j in ${percent[#]}
do
echo "Length is $i and percent is $j"
echo "Where length =>$i and % ID is >=$j, number of matches is: " >> output.txt
awk '{if ($4>=$'i' && $3>=$'j') print $2}' input.txt | uniq | wc -l >> output.txt
done
done
Why isn't this bash array populating? I believe I've done them like this in the past. Echoing ${#XECOMMAND[#]} shows no data..
DIR=$1
TEMPFILE=/tmp/dir.tmp
ls -l $DIR | tail -n +2 | sed 's/\s\+/ /g' | cut -d" " -f5,9 > $TEMPFILE
i=0
cat $TEMPFILE | while read line ;do
if [[ $(echo $line | cut -d" " -f1) == 0 ]]; then
XECOMMAND[$i]="$(echo "$line" | cut -d" " -f2)"
(( i++ ))
fi
done
When you run the while loop like
somecommand | while read ...
then the while loop is executed in sub-shell, i.e. a different process than the main script. Thus, all variable assignments that happen in the loop, will not be reflected in the main process. The workaround is to use input redirection and/or command substitution, so that the loop executes in the current process. For example if you want to read from a file you do
while read ....
do
# do stuff
done < "$filename"
or if you wan't the output of a process you can do
while read ....
do
# do stuff
done < <(some command)
Finally, in bash 4.2 and above, you can set shopt -s lastpipe, which causes the last command in the pipeline to be executed in the current process.
I think you're trying to construct an array consisting of the names of all zero-length files and directories in $DIR. If so, you can do it like this:
mapfile -t ZERO_LENGTH < <(find "$DIR" -maxdepth 1 -size 0)
(Add -type f to the find command if you're only interested in regular files.)
This sort of solution is almost always better than trying to parse ls output.
The use of process substitution (< <(...)) rather than piping (... |) is important, because it means that the shell variable will be set in the current shell, not in an ephimeral subshell.
Can anyone explain why the following doesn't work please?
list
the letter is d
the number is 4
the number is 2
the letter is g
script.sh
#!/bin/bash
cat "$1" | grep letter | array=($(awk '{print $4}'))
for i in "${array[#]}"
do
:
echo $i
done
If I run this bash script.sh list I expect the array to print d and g, but it doesn't. I think its because of how I am trying to set the array.
I think its because of how I am trying to set the array.
Each command in a pipeline | is run in a subshell - as a separate process. The parent process does not "see" variable changes from a child process.
Just:
array=($(grep letter "$1" | awk '{print $4}'))
or
array=($(awk '/letter/{print $4}' "$1"))
Run variable assignment in the parent shell.
You should assign the complete row of piped commands to a variable.
array=($(cat "$1" | grep letter | awk '{print $4}'))
The cat and grep command can be combined with awk, but why do you want an array?
I think you want the process each element in one loop, so first remove the double quotes:
for i in ${array[#]}
do
:
echo $i
done
Next, try to do this without an array
while read -r i; do
:
echo $i
done < <(awk '/letter/ {print $4}' "$1")
I'm trying to write a small script that will take the 4th columns of a file and store it in an array then do a little comparison. If the element in the array is greater than 0 and less than 500 I have to increment the counter. However when I run the script the counter always shows 0. Here's my script
#!/bin/bash
mapfile -t my_array < <(cat file1.txt | awk '{ print $4 }' > test.txt)
COUNTER=0
for i in ${my_array[#]}; do
if [["${my_array[$i]}" -gt 0 -a "${my_array[$i]}" -lt 500 ]]
then
COUNTER=$((COUNTER + 1))
fi
printf "%s\t%s\n" "%i" "${my_array[$i]}"//just to test if the mapfile command is working
done
echo $COUNTER
output:
./script1.bash
0
#!/bin/bash
mapfile -t my_array < <(awk '{ print $4 }' file1.txt | tee test.txt)
COUNTER=0
for idx in "${!my_array[#]}"; do
value=${my_array[$idx]}
if (( value > 0 )) && (( value < 500 )); then
COUNTER=$((COUNTER + 1))
fi
printf "%s\t%s\n" "$idx" "$value"
done
echo "$COUNTER"
The use of cat here is needless: It added nothing but inefficiency (requiring an extra process to be started, and forcing awk to read from a pipe rather than direct from a file).
mapfile had nothing to read because the output of awk was redirected to test.txt. If you want it to go to both a file and stdout, then you need to use tee.
-a is not valid in [[ ]]; use && instead there. However, since you're doing only arithmetic, (( )) is more appropriate. Incidentally, -a is officially marked obsolescent even for [ ] and test; see the current POSIX standard.
${my_array[#]} iterates over values. If you want to iterate over indexes, you need ${!my_array[#]} instead.
Whitespace is mandatory in separating command names. [["$foo" is a different command from [[, unless $foo is empty or starts with a character in $IFS.
If you redirect the output to a file: > test.txt then there is no output in "standard output" because it is consumed by the file. So, first, you need to remove that redirection. You may use:
mapfile -t my_array < <(cat file1.txt | awk '{ print $4 }' )
But since awk could perfectly well read a file, this is better:
mapfile -t my_array < <(awk '{ print $4 }' file1.txt)
And since you are using awk, it could do the comparison to 0 and 500 and output the whole count.
counter=$(awk '{if($4>0 && $4<500){c++}}END{print c}' file1.txt)
echo "$counter"
Simpler, faster.
That will also avoid some simple mistakes in your script, like missing an space in the […] construct:
if [[ "${my … # NOT "if [["${my …"
And some missing quotes:
for i in "${my_array[#]}" # NOT for i in ${my_array[#]}
In general, it is a good idea to check your script with ShellCheck.net to remove some simple mistakes.
First let me say I followed questions on stackoverflow.com that relate to my question and it seems the rules are not applying. Let me show you.
The following script:
#!/bin/bash
OUTPUT_DIR=/share/es-ops/Build_Farm_Reports/WorkSpace_Reports
TODAY=`date +"%m-%d-%y"`
HOSTNAME=`hostname`
WORKSPACES=( "bob" "mel" "sideshow-ws2" )
if ! [ -f $OUTPUT_DIR/$HOSTNAME.csv ] && [ $HOSTNAME == "sideshow" ]; then
echo "$TODAY","$HOSTNAME" > $OUTPUT_DIR/$HOSTNAME.csv
echo "${WORKSPACES[0]}," >> $OUTPUT_DIR/$HOSTNAME.csv
sed -i "/^'"${WORKSPACES[0]}"'/$/'"${WORKSPACES[1]}"'/" $OUTPUT_DIR/$HOSTNAME.csv
sed -i "/^'"${WORKSPACES[1]}"'/$/${WORKSPACES[2]}"'/" $OUTPUT_DIR/$HOSTNAME.csv
fi
I want the output to look like:
09-20-14,sideshow
bob,mel,sideshow-ws2
the sed statements are supposed to append successive array elements to preceding ones on the same line. Now I know there's a simpler way to do this like:
echo "${WORKSPACES[0]},${WORKSPACES[1]},${WORKSPACES[2]}" >> $OUTPUT_DIR/$HOSTNAME.csv
But let's say I had 30 elements in the array and I wanted to appended them one after the other on the same line? Can you show me how to loop through the elements in an array and append them one after the other on the same line?
Also let's say I had the output of a command like:
df -m /export/ws/$ws | awk '{if (NR!=1) {print $3}}'
and I wanted to append that to the end of the same line.
But when I run it I get:
+ OUTPUT_DIR=/share/es-ops/Build_Farm_Reports/WorkSpace_Reports
++ date +%m-%d-%y
+ TODAY=09-20-14
++ hostname
+ HOSTNAME=sideshow
+ WORKSPACES=("bob" "mel" "sideshow-ws2")
+ '[' -f /share/es-ops/Build_Farm_Reports/WorkSpace_Reports/sideshow.csv ']'
And the file right now looks like:
09-20-14,sideshow
bob,
I am happy to report that user syme solved this (see below) but then I realized I need the date in the first column:
09-7-14,bob,mel,sideshow-ws2
Can I do this using syme's for loop?
Okay user syme solved this too he said "Just add $TODAY to the for loop" like this:
for v in "$TODAY" "${WORKSPACES[#]}"
Okay now the output looks like this I changed the elements in the array btw:
sideshow
09-20-14,bob_avail,bob_used,mel_avail,mel_used,sideshow-ws2_avail,sideshow-ws2_used
Now below that the next line will be populated by a , in the first column skipping the date and then:
df -m /export/ws/$v | awk '{if (NR!=1) {print $3}}
which equals the value of available space on bob in the first iteration
and then:
df -m /export/ws/$v | awk '{if (NR!=1) {print $2}}
which equals the value of used space on bob in the 2nd iteration
and then we just move on to the next value in ${WORKSPACE[#]}
which will be mel and do the available and used as we did with bob or $v above.
I know you geniuses on here will make child's play out of this.
I solved my own last question on this thread:
WORKSPACES2=( "bob" "mel" "sideshow-ws2" )
separator="," # defined empty for the first value
for v in "${WORKSPACES2[#]}"
do
available=`df -m /export/ws/$v | awk '{if (NR!=1) {print $3}}'`
used=`df -m /export/ws/$v | awk '{if (NR!=1) {print $2}}'`
echo -n "$separator$available$separator$used" >> $OUTPUT_DIR/$HOSTNAME.csv # append, concatenated, the separator and the value to the file
done
produces:
sideshow
09-20-14,bob_avail,bob_used,mel_avail,mel_used,sideshow-ws2_avail,sideshow-ws2_used
,470400,1032124,661826,1032124,43443,1032108
echo -n permits to print text without the linebreak.
To loop over the values of the array, you can use a for-loop:
echo "$TODAY,$HOSTNAME" > $OUTPUT_DIR/$HOSTNAME.csv # with a linebreak
separator="" # defined empty for the first value
for v in "${WORKSPACES[#]}"
do
echo -n "$separator$v" >> $OUTPUT_DIR/$HOSTNAME.csv # append, concatenated, the separator and the value to the file
separator="," # comma for the next values
done
echo >> $OUTPUT_DIR/$HOSTNAME.csv # add a linebreak (if you want it)