My configs.txt file is in the format
example1
example2
example3
I would like to read a text file configs.txt into an array and then loop through the array to perform a tar command to create a separate tarfile for each entry like example1.gz etc.
Not sure why you need an array
while read p; do
echo "$p";
# tar here
done < configs.txt
for p in $(cat configs.txt); do
echo "$p";
# tar here
done
Related
I've already read a lot of questions concerning reading and writing in ARRAY in bash. I could not find the solution to my issue.
Actually, I've got a file that contains the path of a lot of files.
cat MyFile
> ~/toto/file1.txt
> ~/toto/file2.txt
> ~/toto/file3.txt
> ~/toto/file4.txt
> ~/toto/file5.txt
I fill an array ARRAY to contain this list:
readarray ARRAY < MyFile.txt
or
while IFS= read -r line
do
printf 'TOTO %s\n' "$line"
ARRAY+=("${line}")
done <MyFile.txt
or
for line in $(cat ${MyFile.txt}) ;
do echo "==> $line";
ARRAY+=($line) ;
done
All those methods work well to fill the ARRAY,
echo "0: ${ARRAY[1]}"
echo "1: ${ARRAY[2]}"
> 0: ~/toto/file1.txt
> 1: ~/toto/file2.txt
This is awesome.
but my problem is that if I try to diff the content of the file it does not work, it looks like the it does not expand the content of the file
diff ${ARRAY[1]} ${ARRAY[2]}
diff: ~/toto/file1.txt: No such file or directory
diff: ~/toto/file2.txt: No such file or directory
but when a print the content:
echo diff ${ARRAY[1]} ${ARRAY[2]}
diff ~/toto/file1.txt ~/toto/file2.txt
and execute it I get the expected diff in the file
diff ~/toto/file1.txt ~/toto/file2.txt
3c3
< Param = {'AAA', 'BBB'}
---
> Param = {'AAA', 'CCC'}
whereas if I fill ARRAY manually this way:
ARRAY=(~/toto/file1.txt ~/toto/file2.txt)
diff works well.
Does anyone have an idea?
Thanks a lot
Regards,
Thomas
Tilde expansion does not happen when you use variable substitution from ${ARRAY[index]}.
Put the full path to the files in MyFile.txt and run your code again.
I need to read file line by line, and every line split by ",", and store to array.
File source_file.
usl-coop,/root
usl-dev,/bin
Script.
i=1
while read -r line; do
IFS="," read -ra para_$i <<< $line
echo ${para_$i[#]}
((i++))
done < source_file
Expected output.
para_1[0]=usl-coop
para_1[1]=/root
para_2[0]=usl-dev
para_2[1]=/bin
Script will out error about echo.
./sofimon.sh: line 21: ${para_$i[#]}: bad substitution
When I echo array one by one field, for example
echo para_1[0]
it shows, that variables are stored.
But I need use it with variable within, something like this.
${para_$i[1]}
Is possible to do this?
Thanks.
S.
There is a trick to simulate 2D arrays using associative arrays. It works nice and I think is the most flexible and extensible:
declare -A para
i=1
while IFS=, read -r -a line; do
for j in ${!line[#]}; do
para[$i,$j]="${line[$j]}"
done
((i++)) ||:
done < source_file
declare -p para
will output:
declare -A para=([1,0]="usl-coop" [1,1]="/root" [2,1]="/bin" [2,0]="usl-dev" )
Without modifying your script that much you could use indirect variable expansion. It's sometimes used in simpler scripts:
i=1
while IFS="," read -r -a para_$i; do
n="para_$i[#]"
echo "${!n}"
((i++)) ||:
done < source_file
declare -p ${!para_*}
or basically the same with a nameref a named reference to another variable (side note: see how [#] needs to be part of the variable in indirect expansion, but not in named reference):
i=1
while IFS="," read -r -a para_$i; do
declare -n n
n="para_$i"
echo "${n[#]}"
((i++)) ||:
done < source_file
declare -p ${!para_*}
both scripts above will output the same:
usl-coop /root
usl-dev /bin
declare -a para_1=([0]="usl-coop" [1]="/root")
declare -a para_2=([0]="usl-dev" [1]="/bin")
That said, I think you shouldn't read your file into memory at all. It's just a bad design. Shell and bash is build around passing your files with pipes, streams, fifos, redirections, process substitutions, etc. without ever saving/copying/storing the file. If you have a file to parse, you should stream it to another process, parse and save the result, without ever storing the whole input in memory. If you want some data to find inside a file, use grep or awk.
Here is a short awk script that do the task.
awk 'BEGIN{FS=",";of="para_%d[%d]=%s\n"}{printf(of, NR, 0, $1);printf(of, NR, 1, $2)}' input.txt
Provide the desired output.
Explanation:
BEGIN{
FS=","; # set field seperator to `,`
of="para_%d[%d]=%s\n" # define common printf output format
}
{ # for each input line
printf(of, NR, 0, $1); # output for current line, [0], left field
printf(of, NR, 1, $2) # output for current line, [1], right field
}
I use bash on Ubuntu and I have some files in a folder, some with space in their name, other non.
I would like an array with file's name.
Example : [foo.txt, I am a file.txt, bar.jpg, etc.]
My code :
for x in "$(ls -1 test/)"; do
fileList+=($x)
done
I get : [foo.txt, I, am, a, file.txt, bar.jpg, etc.]
If I put fileList+=("$x") I get one line array [foo.txt I am a file.txt bar.jpg etc.].
How can I do to get what I want?
Thank you.
Why not use shell globs? E.g.
for x in test/*; do
...
or
filelist=( test/* )
EDIT:
shopt -s nullglob
shopt -s dotglob
might be also wanted.
Try using read, like this:
ls | while read f ; do
echo "$f"
done
I have about 500 files with trailing NUL bytes, maybe produced with
truncate -s 8M <file>
How can I cut off the zeroes?
This perl script should do it:
for f in *; do
perl -e '$/=undef;$_=<>;s|\0+$||;print;' < $f > $f_fixed
done
This will keep all NULs within the file, remove any at the end, and save the result into <original filename>_fixed.
Script explanation: $/=undef tells perl to operate on the whole file rather than splitting it into lines; $_=<> loads the file; s|\0+|| removes any string of NULs at the end of the loaded file 'string'; and print outputs the result. The rest is standard Bash file redirection.
If the file is a "text" file and not a "binary" file, you can simply do
strings a.txt > b.txt
ref
Use tr:
cat $input_file | tr -d '\0' > $output_file
Note that $input_file and $output_file must be different
Following the suggestion of #Eevee, you can actually avoid truncating those files below 8M. Using the following condition in your loop and the fact that truncate will assume bytes as default if you don't append any suffix to the size parameter, this won't pad the files below 8M:
for file in $(ls -c1 directory); do
# ...
SIZE=$(stat -c%s $file)
LIMIT=$((8 * 1024 * 1024))
if [ "$SIZE" -lt "$LIMIT" ]; then
truncate -s $SIZE $file
else
truncate -s 8M $file
fi
# ...
done
Not really any Unix tool for this particular case. Here's a Python (3) script:
import sys
for fn in sys.argv[1:]:
with open(fn, 'rb') as f:
contents = f.read()
with open(fn, 'wb') as f:
f.write(contents.rstrip(b'\0'))
Run as:
python retruncate.py file1 file2 files* etc...
I read the files of a directory and put each file name into an array (SEARCH)
Then I use a loop to go through each file name in the array (SEARCH) and open them up with a while read line loop and read each line into another array (filecount). My problem is its one huge array with 39 lines (each file has 13 lines) and I need it to be 3 seperate arrays, where
filecount1[line1] is the first line from the 1st file and so on. here is my code so far...
typeset -A files
for file in ${SEARCH[#]}; do
while read line; do
files["$file"]+="$line"
done < "$file"
done
So, Thanks Ivan for this example! However I'm not sure I follow how this puts it into a seperate array because with this example wouldnt all the arrays still be named "files"?
If you're just trying to store the file contents into an array:
declare -A contents
for file in "${!SEARCH[#]}"; do
contents["$file"]=$(< $file)
done
If you want to store the individual lines in a array, you can create a pseudo-multi-dimensional array:
declare -A contents
for file in "${!SEARCH[#]}"; do
NR=1
while read -r line; do
contents["$file,$NR"]=$line
(( NR++ ))
done < "$file"
done
for key in "${!contents[#]}"; do
printf "%s\t%s\n" "$key" "${contents["$key"]}"
done
line 6 is
$filecount[$linenum]}="$line"
Seems it is missing a {, right after the $.
Should be:
${filecount[$linenum]}="$line"
If the above is true, then it is trying to run the output as a command.
Line 6 is (after "fixing" it above):
${filecount[$linenum]}="$line"
However ${filecount[$linenum]} is a value and you can't have an assignment on a value.
Should be:
filecount[$linenum]="$line"
Now I'm confused, as in whether the { is actually missing, or } is the actual typo :S :P
btw, bash supports this syntax too
filecount=$((filecount++)) # no need for $ inside ((..)) and use of increment operator ++
This should work:
typeset -A files
for file in ${SEARCH[#]}; do # foreach file
while read line; do # read each line
files["$file"]+="$line" # and place it in a new array
done < "$file" # reading each line from the current file
done
a small test shows it works
# set up
mkdir -p /tmp/test && cd $_
echo "abc" > a
echo "foo" > b
echo "bar" > c
# read files into arrays
typeset -A files
for file in *; do
while read line; do
files["$file"]+="$line"
done < "$file"
done
# print arrays
for file in *; do
echo ${files["$file"]}
done
# same as:
echo ${files[a]} # prints: abc
echo ${files[b]} # prints: foo
echo ${files[c]} # prints: bar