Strange scope issues in bash while loop [duplicate] - arrays

I can get this to work in ksh but not in bash which is really driving me nuts.
Hopefully it is something obvious that I'm overlooking.
I need to run an external command where each line of the output will be stored at an array index.
This simplified example looks like it is setting the array in the loop correctly however after the loop has completed those array assignments are gone? It's as though the loop is treated completely as an external shell?
junk.txt
this is a
test to see
if this works ok
testa.sh
#!/bin/bash
declare -i i=0
declare -a array
echo "Simple Test:"
array[0]="hello"
echo "array[0] = ${array[0]}"
echo -e "\nLoop through junk.txt:"
cat junk.txt | while read line
do
array[i]="$line"
echo "array[$i] = ${array[i]}"
let i++
done
echo -e "\nResults:"
echo " array[0] = ${array[0]}"
echo " Total in array = ${#array[*]}"
echo "The whole array:"
echo ${array[#]}
Output
Simple Test:
array[0] = hello
Loop through junk.txt:
array[0] = this is a
array[1] = test to see
array[2] = if this works ok
Results:
array[0] = hello
Total in array = 1
The whole array:
hello
So while in the loop, we assign array[i] and the echo verifies it.
But after the loop I'm back at array[0] containing "hello" with no other elements.
Same results across bash 3, 4 and different platforms.

Because your while loop is in a pipeline, all variable assignments in the loop body are local to the subshell in which the loop is executed. (I believe ksh does not run the command in a subshell, which is why you have the problem in bash.) Do this instead:
while read line
do
array[i]="$line"
echo "array[$i] = ${array[i]}"
let i++
done < junk.txt
Rarely, if ever, do you want to use cat to pipe a single file to another command; use input redirection instead.
UPDATE: since you need to run from a command and not a file, another option (if available) is process substitution:
while read line; do
...
done < <( command args ... )
If process substitution is not available, you'll need to output to a temporary file and redirect input from that file.
If you are using bash 4.2 or later, you can execute these two commands before your loop, and the original pipe-into-the-loop will work, since the while loop is the last command in the pipeline.
set +m # Turn off job control; it's probably already off in a non-interactive script
shopt -s lastpipe
cat junk.txt | while read line; do ...; done
UPDATE 2: Here is a loop-less solution based on user1596414's comment
array[0]=hello
IFS=$'\n' array+=( $(command) )
The output of your command is split into words based solely on newlines (so that each line is a separate word), and appends the resulting line-per-slot array to the original. This is very nice if you are only using the loop to build the array. It can also probably be modified to accomodate a small amount of per-line processing, vaguely similar to a Python list comprehension.

Related

Fill Two Arrays on the fly From stdin File in Bash

I wrote a bash script that reads a file from stdin $1, and needs to read that file line by line within a loop, and based on a condition statement in each iteration, each line tested from the file will feed into one of two new arrays lets say named GOOD array and BAD array. Lastly, I'll display the total elements of each array.
#!/bin/bash
for x in $(cat $1); do
#testing something on x
if [ $? -eq 0 ]; then
#add the current value of x into array called GOOD
else
#add the current value of x into array called BAD
fi
done
echo "Total GOOD elements: ${#GOOD[#]}"
echo "Total BAD elements: ${#BAD[#]}"
What changes should i make to accomplish it?
#!/usr/bin/env bash
# here, we're checking the number of lines more than 5 characters long
# replace with your real test
testMyLine() { (( ${#1} > 5 )); }
good=( ); bad=( )
while IFS= read -r line; do
if testMyLine "$line"; then
good+=( "$line" )
else
bad+=( "$line" )
fi
done <"$1"
echo "Read ${#good[#]} good and ${#bad[#]} bad lines"
Note:
We're using a while read loop to iterate over file contents. This doesn't need to read more than one line into memory at a time (so it won't run out of RAM even with really big files), and it doesn't have unwanted side effects like changing a line containing * to a list of files in the current directory.
We aren't using $?. if foo; then is a much better way to branch on the exit status of foo than foo; if [ $? = 0 ]; then -- in particular, this avoids depending on the value of $? not being changed between when you assign it and when you need it; and it marks foo as "checked", to avoid exiting via set -e or triggering an ERR trap when your boolean returns false.
The use of lower-case variable names is intentional. All-uppercase names are used for shell-builtin variables and names with special meaning to the operating system -- and since defining a regular shell variable overwrites any environment variable with the same name, this convention applies to both types. See http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap08.html

How to get user input as number and echo the stored array value of that number in bash scripting

I have wrote a script that throws the output of running node processes with the cwd of that process and I store the value in an array using for loop and do echo that array.
How can I able to get the user enter the index of array regarding the output that the script throws and show the output against that input generated by user
Example Myscript
array=$(netstat -nlp | grep node)
for i in ${array[*]}
do
echo $i
done
output is something like that
1056
2064
3024
I want something more advance. I want to take input from user like
Enter the regarding index from above list = 1
And lets suppose user enter 1
Then next output should be
Your selected value is 2064
Is it possible in bash
First, you're not actually using an array, you are storing a plain string in the variable "array". The string contains words separated by whitespace, so when you supply the variable in the for statement, the unquoted value is subject to Word Splitting
You need to use the array syntax for setting the array:
array=( $(netstat -nlp | grep node) )
However, the unquoted command substitution still exposes you to Filename Expansion. The best way to store the lines of a command into an array is to use the mapfile command with a process substitution:
mapfile -t array < <(netstat -nlp | grep node)
And in the for loop, make sure you quote all the variables and use index #
for i in "${array[#]}"; do
echo "$i"
done
Notes:
arrays created with mapfile will start at index 0, so be careful of off-by-one errors
I don't know how variables are implemented in bash, but there is this oddity:
if you refer to the array without an index, you'll get the first element:
array=( "hello" "world" )
echo "$array" # ==> hello
If you refer to a plain variable with array syntax and index zero, you'll get the value:
var=1234
echo "${var[0]}" # ==> 1234

Shell Array Cleared for Unknown Reason [duplicate]

This question already has answers here:
A variable modified inside a while loop is not remembered
(8 answers)
Closed 6 years ago.
I have a pretty simple sh script where I make a system cat call, collect the results and parse some relevant information before storing the information in an array, which seems to work just fine. But as soon as I exit the for loop where I store the information, the array seems to clear itself. I'm wondering if I am accessing the array incorrectly outside of the for loop. Relevant portion of my script:
#!/bin/sh
declare -a QSPI_ARRAY=()
cat /proc/mtd | while read mtd_instance
do
# split result into individiual words
words=($mtd_instance)
for word in "${words[#]}"
do
# check for uboot
if [[ $word == *"uboot"* ]]
then
mtd_num=${words[0]}
index=${mtd_num//[!0-9]/} # strip everything except the integers
QSPI_ARRAY[$index]="uboot"
echo "QSPI_ARRAY[] at index $index: ${QSPI_ARRAY[$index]}"
elif [[ $word == *"fpga_a"* ]]
then
echo "found it: "$word""
mtd_num=${words[0]}
index=${mtd_num//[!0-9]/} # strip everything except the integers
QSPI_ARRAY[$index]="fpga_a"
echo "QSPI_ARRAY[] at index $index: ${QSPI_ARRAY[$index]}"
# other items are added to the array, all successfully
fi
done
echo "length of array: ${#QSPI_ARRAY[#]}"
echo "----------------------"
done
My output is great until I exit the for loop. While within the for loop, the array size increments and I can check that the item has been added. After the for loop is complete I check the array like so:
echo "RESULTING ARRAY:"
echo "length of array: ${#QSPI_ARRAY[#]}"
for qspi in "${QSPI_ARRAY}"
do
echo "qspi instance: $qspi"
done
Here are my results, echod to my display:
dev: size erasesize name
length of array: 0
-------------
mtd0: 00100000 00001000 "qspi-fsbl-uboot"
QSPI_ARRAY[] at index 0: uboot
length of array: 1
-------------
mtd1: 00500000 00001000 "qspi-fpga_a"
QSPI_ARRAY[] at index 1: fpga_a
length of array: 2
-------------
RESULTING ARRAY:
length of array: 0
qspi instance:
EDIT: After some debugging, it seems I have two different arrays here somehow. I initialized the array like so: QSPI_ARRAY=("a" "b" "c" "d" "e" "f" "g"), and after my for-loop for parsing the array it is still a, b, c, etc. How do I have two different arrays of the same name here?
This structure:
cat /proc/mtd | while read mtd_instance
do
...
done
Means that whatever comes between do and done cannot have any effects inside the shell environment that are still there after the done.
The fact that the while loop is on the right hand side of a pipe (|) means that it runs in a subshell. Once the loop exits, so does the subshell. And all of its variable settings.
If you want a while loop which makes changes that stick around, don't use a pipe. Input redirection doesn't create a subshell, and in this case, you can just read from the file directly:
while read mtd_instance
do
...
done </proc/mtd
If you had a more complicated command than a cat, you might need to use process substitution. Still using cat as an example, that looks like this:
while read mtd_instance
do
...
done < <(cat /proc/mtd)
In the specific case of your example code, I think you could simplify it somewhat, perhaps like this:
#!/usr/bin/env bash
QSPI_ARRAY=()
while read -a words; do␣
declare -i mtd_num=${words[0]//[!0-9]/}
for word in "${words[#]}"; do
for type in uboot fpga_a; do
if [[ $word == *$type* ]]; then
QSPI_ARRAY[mtd_num]=$type
break 2
fi
done
done
done </proc/mtd
Is this potentially what you are seeing:
http://mywiki.wooledge.org/BashFAQ/024

bash trouble assigning to an array index in a loop

I can get this to work in ksh but not in bash which is really driving me nuts.
Hopefully it is something obvious that I'm overlooking.
I need to run an external command where each line of the output will be stored at an array index.
This simplified example looks like it is setting the array in the loop correctly however after the loop has completed those array assignments are gone? It's as though the loop is treated completely as an external shell?
junk.txt
this is a
test to see
if this works ok
testa.sh
#!/bin/bash
declare -i i=0
declare -a array
echo "Simple Test:"
array[0]="hello"
echo "array[0] = ${array[0]}"
echo -e "\nLoop through junk.txt:"
cat junk.txt | while read line
do
array[i]="$line"
echo "array[$i] = ${array[i]}"
let i++
done
echo -e "\nResults:"
echo " array[0] = ${array[0]}"
echo " Total in array = ${#array[*]}"
echo "The whole array:"
echo ${array[#]}
Output
Simple Test:
array[0] = hello
Loop through junk.txt:
array[0] = this is a
array[1] = test to see
array[2] = if this works ok
Results:
array[0] = hello
Total in array = 1
The whole array:
hello
So while in the loop, we assign array[i] and the echo verifies it.
But after the loop I'm back at array[0] containing "hello" with no other elements.
Same results across bash 3, 4 and different platforms.
Because your while loop is in a pipeline, all variable assignments in the loop body are local to the subshell in which the loop is executed. (I believe ksh does not run the command in a subshell, which is why you have the problem in bash.) Do this instead:
while read line
do
array[i]="$line"
echo "array[$i] = ${array[i]}"
let i++
done < junk.txt
Rarely, if ever, do you want to use cat to pipe a single file to another command; use input redirection instead.
UPDATE: since you need to run from a command and not a file, another option (if available) is process substitution:
while read line; do
...
done < <( command args ... )
If process substitution is not available, you'll need to output to a temporary file and redirect input from that file.
If you are using bash 4.2 or later, you can execute these two commands before your loop, and the original pipe-into-the-loop will work, since the while loop is the last command in the pipeline.
set +m # Turn off job control; it's probably already off in a non-interactive script
shopt -s lastpipe
cat junk.txt | while read line; do ...; done
UPDATE 2: Here is a loop-less solution based on user1596414's comment
array[0]=hello
IFS=$'\n' array+=( $(command) )
The output of your command is split into words based solely on newlines (so that each line is a separate word), and appends the resulting line-per-slot array to the original. This is very nice if you are only using the loop to build the array. It can also probably be modified to accomodate a small amount of per-line processing, vaguely similar to a Python list comprehension.

how do I output the contents of a while read line loop to multiple arrays in bash?

I read the files of a directory and put each file name into an array (SEARCH)
Then I use a loop to go through each file name in the array (SEARCH) and open them up with a while read line loop and read each line into another array (filecount). My problem is its one huge array with 39 lines (each file has 13 lines) and I need it to be 3 seperate arrays, where
filecount1[line1] is the first line from the 1st file and so on. here is my code so far...
typeset -A files
for file in ${SEARCH[#]}; do
while read line; do
files["$file"]+="$line"
done < "$file"
done
So, Thanks Ivan for this example! However I'm not sure I follow how this puts it into a seperate array because with this example wouldnt all the arrays still be named "files"?
If you're just trying to store the file contents into an array:
declare -A contents
for file in "${!SEARCH[#]}"; do
contents["$file"]=$(< $file)
done
If you want to store the individual lines in a array, you can create a pseudo-multi-dimensional array:
declare -A contents
for file in "${!SEARCH[#]}"; do
NR=1
while read -r line; do
contents["$file,$NR"]=$line
(( NR++ ))
done < "$file"
done
for key in "${!contents[#]}"; do
printf "%s\t%s\n" "$key" "${contents["$key"]}"
done
line 6 is
$filecount[$linenum]}="$line"
Seems it is missing a {, right after the $.
Should be:
${filecount[$linenum]}="$line"
If the above is true, then it is trying to run the output as a command.
Line 6 is (after "fixing" it above):
${filecount[$linenum]}="$line"
However ${filecount[$linenum]} is a value and you can't have an assignment on a value.
Should be:
filecount[$linenum]="$line"
Now I'm confused, as in whether the { is actually missing, or } is the actual typo :S :P
btw, bash supports this syntax too
filecount=$((filecount++)) # no need for $ inside ((..)) and use of increment operator ++
This should work:
typeset -A files
for file in ${SEARCH[#]}; do # foreach file
while read line; do # read each line
files["$file"]+="$line" # and place it in a new array
done < "$file" # reading each line from the current file
done
a small test shows it works
# set up
mkdir -p /tmp/test && cd $_
echo "abc" > a
echo "foo" > b
echo "bar" > c
# read files into arrays
typeset -A files
for file in *; do
while read line; do
files["$file"]+="$line"
done < "$file"
done
# print arrays
for file in *; do
echo ${files["$file"]}
done
# same as:
echo ${files[a]} # prints: abc
echo ${files[b]} # prints: foo
echo ${files[c]} # prints: bar

Resources