I'm trying to read the lines of output from a subshell into an array, and I'm not willing to set IFS because it's global. I don't want one part of the script to affect the following parts, because that's poor practice and I refuse to do it. Reverting IFS after the command is not an option because it's too much trouble to keep the reversion in the right place after editing the script. How can I explain to bash that I want each array element to contain an entire line, without having to set any global variables that will destroy future commands?
Here's an example showing the unwanted stickiness of IFS:
lines=($(egrep "^-o" speccmds.cmd))
echo "${#lines[#]} lines without IFS"
IFS=$'\r\n' lines=($(egrep "^-o" speccmds.cmd))
echo "${#lines[#]} lines with IFS"
lines=($(egrep "^-o" speccmds.cmd))
echo "${#lines[#]} lines without IFS?"
The output is:
42 lines without IFS
6 lines with IFS
6 lines without IFS?
This question is probably based on a misconception.
IFS=foo read does not change IFS outside of the read operation itself.
Thus, this would have side effects, and should be avoided:
IFS=
declare -a array
while read -r; do
array+=( "$REPLY" )
done < <(your-subshell-here)
...but this is perfectly side-effect free:
declare -a array
while IFS= read -r; do
array+=( "$REPLY" )
done < <(your-subshell-here)
With bash 4.0 or newer, there's also the option of readarray or mapfile (synonyms for the same operation):
mapfile -t array < <(your-subshell-here)
In examples later added to your answer, you have code along the lines of:
lines=($(egrep "^-o" speccmds.cmd))
The better way to write this is:
mapfile -t lines < <(egrep "^-o" speccmds.cmd)
Are you trying to store the lines of the output in an array, or the words of each line?
lines
mapfile -t arrayname < <(your subshell)
This does not use IFS at all.
words
(your subshell) | while IFS=: read -ra words; do ...
The form var=value command args... puts the var variable into the environment of the command, and does not affect the current shell's environment.
Related
I have a text file with with the following lines of text:
https://i.imgur.com/9iLSS35.png
Delete page: https://imgur.com/delete/SbmZQDht9Xk2NbW
https://i.imgur.com/9iLSS336.png
Delete page: https://imgur.com/delete/SbmZQDht9Xk2NbW
https://i.imgur.com/9iLSS37.png
Delete page: https://imgur.com/delete/SbmZQDht9Xk2NbW
I need to read the odd lines and add to a array in bash. I have tried the following code but it generates a variable and not an array as I need.
A=$(sed -n 1~2p ./upimagen.txt)
How can I read the odd lines of the text file and add each line to an array and then access separately?
The result would be:
${A[0]}
https://i.imgur.com/9iLSS35.png
${A[1]}
https://i.imgur.com/9iLSS336.png
...
Array assignment requires an outer set of parentheses.
A=($(sed -n 1~2p ./upimagen.txt))
This will split on all whitespace, though, not just newlines. It will also expand any globs that use * or ?. It's better to use readarray plus process substitution to ensure that each line is preserved as is:
readarray -t A < <(sed -n 1~2p ./upimagen.txt)
Just a slight variation on John's answer using awk. Using either readarray or the synonymous mapfile reading from a process substitution is probably your best bet, e.g.
readarray -t A < <(awk 'FNR % 2 {print $1}' ./upimagen.txt)
While there is nothing wrong with the direct initialization using command substitution, e.g. A=( $(...) ), if your files are large, or if you only need part of the range of lines, or say you have a custom function you want to run on every X number of elements read, mapfile/readarray offers a wide set of features over straight command substitution.
Using awk :
readarray -t A < <(awk 'NR%2' ./upimagen.txt)
I thought setting IFS to $'\n' would help me in reading an entire file into an array, as in:
IFS=$'\n' read -r -a array < file
However, the above command only reads the first line of the file into the first element of the array, and nothing else.
Even this reads only the first line into the array:
string=$'one\ntwo\nthree'
IFS=$'\n' read -r -a array <<< "$string"
I came across other posts on this site that talk about either using mapfile -t or a read loop to read a file into an array.
Now my question is: when do I use IFS=$'\n' at all?
You are a bit confused as to what IFS is. IFS is the Internal Field Separator used by bash to perform word-splitting to split lines into words after expansion. The default value is [ \t\n] (space, tab, newline).
By reassigning IFS=$'\n', you are removing the ' \t' and telling bash to only split words on newline characters (your thinking is correct). That has the effect of allowing some line with spaces to be read into a single array element without quoting.
Where your implementation fails is in your read -r -a array < file. The -a causes words in the line to be assigned to sequential array indexes. However, you have told bash to only break on a newline (which is the whole line). Since you only call read once, only one array index is filled.
You can either do:
while IFS=$'\n' read -r line; do
array+=( $line )
done < "$filename"
(which you could do without changing IFS if you simply quoted "$line")
Or using IFS=$'\n', you could do
IFS=$'\n'
array=( $(<filename) )
or finally, you could use IFS and readarray:
readarray array <filename
Try them and let me know if you have questions.
Your second try almost works, but you have to tell read that it should not just read until newline (the default behaviour), but for example until the null string:
$ IFS=$'\n' read -a arr -d '' <<< $'a b c\nd e f\ng h i'
$ declare -p arr
declare -a arr='([0]="a b c" [1]="d e f" [2]="g h i")'
But as you pointed out, mapfile/readarray is the way to go if you have it (requires Bash 4.0 or newer):
$ mapfile -t arr <<< $'a b c\nd e f\ng h i'
$ declare -p arr
declare -a arr='([0]="a b c" [1]="d e f" [2]="g h i")'
The -t option removes the newlines from each element.
As for when you'd want to use IFS=$'\n':
As just shown, if you want to read a files into an array, one line per element, if your Bash is older than 4.0, and you don't want to use a loop
Some people promote using an IFS without a space to avoid unexpected side effects from word splitting; the proper approach in my opinion, though, is to understand word splitting and make sure to avoid it with proper quoting as desired.
I've seen IFS=$'\n' used in tab completion scripts, for example the one for cd in bash-completion: this script fiddles with paths and replaces colons with newlines, to then split them up using that IFS.
I'm using this code to load file into array in bash:
IFS=$'\n' read -d '' -r -a LINES < "$PAR1"
But unfortunately this code skips empty lines.
I tried the next code:
IFS=$'\n' read -r -a LINES < "$PAR1"
But this variant only loads one line.
How do I load file to array in bash, without skipping empty lines?
P.S. I check the number of loaded lines by the next command:
echo ${#LINES[#]}
You can use mapfile available in BASH 4+
mapfile -t lines < "$PAR1"
To avoid doing anything fancy, and stay compatible with all versions of bash in common use (as of this writing, Apple is shipping bash 3.2.x to avoid needing to comply with the GPLv3):
lines=( )
while IFS= read -r line; do
lines+=( "$line" )
done
See also BashFAQ #001.
I am able to read file into a regular array with a single statement:
local -a ary
readarray -t ary < $fileName
Not happening is reading a file into assoc. array.
I have control over file creation and so would like to do as simply as possible w/o loops if possible at all.
So file content can be following to be read in as:
keyname=valueInfo
But I am willing to replace = with another string if cuts down on code, especially in a single line code as above.
And ...
So would it be possible to read such a file into an assoc array using something like an until or from - i.e. read into an assoc array until it hits a word, or would I have to do this as part of loop?
This will allow me to keep a lot of similar values in same file, but read into separate arrays.
I looked at mapfile as well, but does same as readarray.
Finally ...
I am creating an options list - to select from - as below:
local -a arr=("${!1}")
select option in ${arr[*]}; do
echo ${option}
break
done
Works fine - however the list shown is not sorted. I would like to have it sorted if possible at all.
Hope it is ok to put all 3 questions into 1 as the questions are similar - all on arrays.
Thank you.
First thing, associative arrays are declared with -A not -a:
local -A ary
And if you want to declare a variable on global scope, use declare outside of a function:
declare -A ary
Or use -g if BASH_VERSION >= 4.2.
If your lines do have keyname=valueInfo, with readarray, you can process it like this:
readarray -t lines < "$fileName"
for line in "${lines[#]}"; do
key=${line%%=*}
value=${line#*=}
ary[$key]=$value ## Or simply ary[${line%%=*}]=${line#*=}
done
Using a while read loop can also be an option:
while IFS= read -r line; do
ary[${line%%=*}]=${line#*=}
done < "$fileName"
Or
while IFS== read -r key value; do
ary[$key]=$value
done < "$fileName"
This question already has answers here:
Creating an array from a text file in Bash
(7 answers)
Closed 7 years ago.
I am trying to read a file containing lines into a Bash array.
I have tried the following so far:
Attempt1
a=( $( cat /path/to/filename ) )
Attempt2
index=0
while read line ; do
MYARRAY[$index]="$line"
index=$(($index+1))
done < /path/to/filename
Both attempts only return a one element array containing the first line of the file. What am I doing wrong?
I am running bash 4.1.5
The readarray command (also spelled mapfile) was introduced in bash 4.0.
readarray -t a < /path/to/filename
Latest revision based on comment from BinaryZebra's comment
and tested here. The addition of command eval allows for the expression to be kept in the present execution environment while the expressions before are only held for the duration of the eval.
Use $IFS that has no spaces\tabs, just newlines/CR
$ IFS=$'\r\n' GLOBIGNORE='*' command eval 'XYZ=($(cat /etc/passwd))'
$ echo "${XYZ[5]}"
sync:x:5:0:sync:/sbin:/bin/sync
Also note that you may be setting the array just fine but reading it wrong - be sure to use both double-quotes "" and braces {} as in the example above
Edit:
Please note the many warnings about my answer in comments about possible glob expansion, specifically gniourf-gniourf's comments about my prior attempts to work around
With all those warnings in mind I'm still leaving this answer here (yes, bash 4 has been out for many years but I recall that some macs only 2/3 years old have pre-4 as default shell)
Other notes:
Can also follow drizzt's suggestion below and replace a forked subshell+cat with
$(</etc/passwd)
The other option I sometimes use is just set IFS into XIFS, then restore after. See also Sorpigal's answer which does not need to bother with this
The simplest way to read each line of a file into a bash array is this:
IFS=$'\n' read -d '' -r -a lines < /etc/passwd
Now just index in to the array lines to retrieve each line, e.g.
printf "line 1: %s\n" "${lines[0]}"
printf "line 5: %s\n" "${lines[4]}"
# all lines
echo "${lines[#]}"
One alternate way if file contains strings without spaces with 1string each line:
fileItemString=$(cat filename |tr "\n" " ")
fileItemArray=($fileItemString)
Check:
Print whole Array:
${fileItemArray[*]}
Length=${#fileItemArray[#]}
Your first attempt was close. Here is the simplistic approach using your idea.
file="somefileondisk"
lines=`cat $file`
for line in $lines; do
echo "$line"
done
#!/bin/bash
IFS=$'\n' read -d '' -r -a inlines < testinput
IFS=$'\n' read -d '' -r -a outlines < testoutput
counter=0
cat testinput | while read line;
do
echo "$((${inlines[$counter]}-${outlines[$counter]}))"
counter=$(($counter+1))
done
# OR Do like this
counter=0
readarray a < testinput
readarray b < testoutput
cat testinput | while read myline;
do
echo value is: $((${a[$counter]}-${b[$counter]}))
counter=$(($counter+1))
done