I use rsync to copy files according to pairwise defined sources/destinations, read from two config-files.
When transfering a single pair (one file), it works.
When using two or more pairs, the first transfer always fails with:
rsync: link_stat "MyFILEPATH\#015" failed: No such file or directory (2)
All following copies are successful.
Shuffeling the order of files, doesn't change the behavior.
It's always the first one.
So, I'd rule out corrupted/missing files as a cause.
This is my script:
#!/bin/bash
mapfile -t sources <"source-files.txt"
mapfile -t destinations <"destination-folders.txt"
for i in "${!sources[#]}"; do
rsync -av -- "${sources[i]}" "${destinations[i]}"
done
In the config-files, ...
sources-file lists one file-path ...
destination-file lists one folder-path ...
... per line without quotes or any whitespaces.
Related
I want to be able to store a directory's contents inside of an array. I know that you can use:
#!/bin/bash
declare -A array
for i in Directory; do
array[$i]=$i
done
to store directory contents in an associative array. But if there are subdirectories inside of a directory, I want to be able to store them and their contents inside of the same array. I also tried using:
declare -A arr1
find Directory -print0 | while read -d $'\0' file; do
arr1[$file]=$file
echo "${arr1[$file]}"
done
but this just runs into the problem where the array contents vanish once the while loop ends due to the subshell being discarded from the pipeline (not sure if I'm describing this correctly).
I even tried the following:
for i in $(find Directory/*); do
arr2[$i]="$i"
echo $i
done
but the output is a total disaster for files containing any spaces.
How can I store both a directory and all of its subdirectories (and their subdirectories if need be) inside of a single array?
So you know, you don't need associative arrays. A simpler way to add an element to a regular indexed array is:
array+=("$value")
Your find|while approach is on the right track. As you've surmised, you need to get rid of the pipeline. You can do that with process substitution:
while read -d $'\0' file; do
arr1+=("$file")
done < <(find Directory -print0)
Another way to do this without find is with globbing. If you just want the files under Directory it's as simple as:
array=(Directory/*)
If you want to recurse through all of its subdirectories as well, you can enable globstar and use **:
shopt -s globstar
array=(Directory/**)
The globbing methods are really nice because they automatically handle file names with whitespace and other special characters.
In a shell script, I need to apply the same shell code to all files that either have .F90 or .F as extension.
For the moment I use
for file in *.F90 ;do ...
# Code I need to run
for file in *.F; do ...
# Same block of code copy-pasted.
Is there a way to merge these two loops making an array of matching files then applying the action ?
I'm not sure I understand your question, because I don't see why you would need an array.
This would be legal syntax:
for file in *.F90 *.F ; do ...
To keep a list of the files impacted, you could do:
shopt -s nullglob
files="*.F90 *.F"
for file in ${files} ; do ...
Note: the nullglob line prevents lame (IMO) behavior should *.F90 or *.F not match any files.
I don't get the scenario of this given code. All I wanted is to compare the files that is given below. But, in this script nothings happen. I assume that this given code can executed wherever like in /root and it will run. Please check this out.
#!/bin/bash
for file in /var/files/sub/old/*
do
# Strip path from file name
file="${file##*/}"
# Strip everything after the first hyphen
prefix="${file%%-*}-"
# Strip everything before the second-to-last dot
suffix="$(echo $file | awk -F. '{ print "."$(NF-1)"."$NF }')"
# Create new file name from $prefix and $suffix, and any version number
new=$(echo "/var/files/new/${prefix}"*"${suffix}")
# If file exists in the 'new' folder:
if test -f "${new}"
then
# Do string comparison to see if new file is lexicographically "greater than" old
if [[ "${new##*/}" > "${file}" ]]
then
# If so, delete the old version.
rm /var/sub/files/old/"${file}"
else
# 'new' file is NOT newer, delete it instead.
rm "${new}"
fi
fi
done
# Move all new files into the old folder.
mv /var/files/new/* /var/files/sub/old/
Example files inside of each sub- directories ..
/var/files/sub/old/
firefox-24.5.0-1.el5_10.i386.rpm
firefox-24.5.0-1.el5_10.x86_64.rpm
google-1.6.0-openjdk-1.6.0.0-5.1.13.3.el5_10.x86_64.rpm
google-1.6.0-openjdk-demo-1.6.0.0-5.1.13.3.el5_10.x86_64.rpm
/var/files/new/
firefox-25.5.0-1.el5_10.i386.rpm
firefox-25.5.0-1.el5_10.x86_64.rpm
ie-1.6.0-openjdk-devel-1.6.0.0-5.1.13.3.el5_10.x86_64.rpm
ie-1.6.0-openjdk-javadoc-1.6.0.0-5.1.13.3.el5_10.x86_64.rpm
ie-1.6.0-openjdk-src-1.6.0.0-5.1.13.3.el5_10.x86_64.rpm
google-2.6.0-openjdk-demo-1.6.0.0-5.1.13.3.el5_10.x86_64.rpm
In this instance, I want to get the files that are the same. So the files that are the same in the given example are:
firefox-24.5.0-1.el5_10.i386.rpm
firefox-24.5.0-1.el5_10.x86_64.rpm
google-1.6.0-openjdk-demo-1.6.0.0-5.1.13.3.el5_10.x86_64.rpm
in the old/ directory and for the new/ directory the equivalents are:
firefox-25.5.0-1.el5_10.i386.rpm
firefox-25.5.0-1.el5_10.x86_64.rpm
google-2.6.0-openjdk-demo-1.6.0.0-5.1.13.3.el5_10.x86_64.rpm
The files have similarity for their first characters. It will display in the terminal. After that, there will be another comparing again of the files and the comparison is about which file is more updated one by the number after the name of the file like: firefox-24.5.0-1.el5_10.i386.rpm compared with firefox-25.5.0-1.el5_10.i386.rpm. So in that instance the firefox-24.5.0-1.el5_10.i386.rpm will be replaced by firefox-25.5.0-1.el5_10.i386.rpm because it has a greater value and more updated one and same as other files that are similar. And if the old one is removed and the new will take replacement of it.
So at this moment after the script has been executed the output will be like this.
/var/files/sub/old/
google-1.6.0-openjdk-1.6.0.0-5.1.13.3.el5_10.x86_64.rpm
firefox-25.5.0-1.el5_10.i386.rpm
firefox-25.5.0-1.el5_10.x86_64.rpm
ie-1.6.0-openjdk-devel-1.6.0.0-5.1.13.3.el5_10.x86_64.rpm
ie-1.6.0-openjdk-javadoc-1.6.0.0-5.1.13.3.el5_10.x86_64.rpm
ie-1.6.0-openjdk-src-1.6.0.0-5.1.13.3.el5_10.x86_64.rpm
google-2.6.0-openjdk-demo-1.6.0.0-5.1.13.3.el5_10.x86_64.rpm
/var/files/new/
<<empty all files here must to moved to other directory take as a replacement>>
Can anyone help me to make a script for this ? above is just an example. Let's assume that there are lots of files to considered as similar and need to removed and moved.
You can use rpm to get the name of the package without version or architecture strings:
rpm -qi -p /firefox-25.5.0-1.el5_10.i386.rpm
Gives:
Name : firefox
Version : 25.5.0
Release : 1.el5_10
Architecture: i386
....
So you can compare the Names to find related packages.
If the goal here is to have the newrpms directory have only the newest version of each RPM from a combination of sources then you most likely want to simply combine all the files in a single directory and then use the repomanage tool (from the yum-utils package, at least on CentOS) to have it inform you which of the RPMS are old and remove them.
Something like:
repomanage --old combined_rpms_directory | xargs -r rm
As to your initial script
for i in $(\ls -d ./new/*);
do
diff ${i} newrpms/;
rm ${i}
done
You generally don't want to "parse" the output from ls, especially when a glob will do what you want just as easily (for i in ./new/* in this case).
diff ${i} newrpms/ is attempting to diff a file and a directory (or two directories if your ls/glob happened to catch a directory) but in neither case will diff do what you want there. That being said what diff does doesn't really matter because, as Barmar said in his comment
your script is removing them without testing the result of diff
A bash script that does the checking. Here's how it works:
Traverse over each file in the old files directory. Get the prefix (package name with no version, architecture, etc), eg. firefox-; get the suffix (architecture.rpm), eg. .i386.rpm.
Attempt to match prefix and suffix with any version number within the new files directory, ie. firefox-*.i386.rpm. If there is a match, $new will contain the file name, eg. firefox-25.5.0-1.el5_10.i386.rpm; if no match, $new will equal the literal string firefox-*.i386.rpm which is not a file.
Check new files directory for existence of $new.
If it exists, check that $new is indeed newer than the old version. This is done by lexicographical string comparison, ie. firefox-24.5.0-1.el5_10.i386.rpm is less than firefox-25.5.0-1.el5_10.i386.rpm because it comes earlier in the alphabet. Conveniently, sane versioning schemes also happen to be alphabetical. NB: this may fail, for example, when comparing version 2 to version 10.
A new version of a file in the old files directory has been found! In this case, get rid of the old file with rm. If the file in the new directory is not newer, then delete it instead.
Done removing old versions. Old files directory has only files without newer versions.
Move all new files into old directory, leaving newest files in old directory, and new directory empty.
#!/bin/bash
for file in /var/files/sub/old/*
do
# Strip path from file name
file="${file##*/}"
# Strip everything after the first hyphen
prefix="${file%%-*}-"
# Strip everything before the second-to-last dot
suffix="$(echo $file | awk -F. '{ print "."$(NF-1)"."$NF }')"
# Create new file name from $prefix and $suffix, and any version number
new=$(echo "/var/files/new/${prefix}"*"${suffix}")
# If file exists in the 'new' folder:
if test -f "${new}"
then
# Do string comparison to see if new file is lexicographically "greater than" old
if [[ "${new##*/}" > "${file}" ]]
then
# If so, delete the old version.
rm /var/sub/files/old/"${file}"
else
# 'new' file is NOT newer, delete it instead.
rm "${new}"
fi
fi
done
# Move all new files into the old folder.
mv /var/files/new/* /var/files/sub/old/
I want to read a list of patterns that may (probably do) contain wildcards from a file
The patterns might look like this:
/vobs/of_app/unix/*
/vobs/of_app/bin/*
etc
My first attempt was to do this:
old_IFS=$IFS
IFS=$'\n'
array=($(cat $file))
This worked fine when the patterns did not match anything in the filesystem, but when they did match things in the filesystem, they got expanded so instead of containing the patterns, my array contained the directory listings of the specified directories. This was no good.
I next tried quoting like this
array=("$(cat $file)")
But this dumped the entire contents of the file into the 0th element of the array.
How can I prevent it from expanding the wildcards into directory listings while still putting each line of the file into a separate array element?
Bash 4 introduced readarray:
readarray -t array < "$file"
and you're done.
array=()
while read line; do
array+=("$line")
done < "$file"
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 9 years ago.
Improve this question
I have 5000 files in my hard-disk with name as ip_file_1,ip_file_2,....
I have a executable that can merge only 2 files. How can I write a script that takes all the file residing in the hardisk (whichs start with ip_file_*) and calls the function for merging all the files.
I have a 5000 files which are binaries that contain the logging information (time that each function call has taken). I have another executable that takes only two files and merges according to the timestamp and gives the merged output.
I execute with the format like the below,
./trace ip_file1 ip_file2 mergefile # I'm not using the trace tool. It's an example
I could use the executable to merge only two files. I thought of automating it to merge all the other files.
The merges has to be done in order (merged according to the timestamp). The logic to merge is already there. And the Output of the merge is sent to the file.
My question is not on how to merge the files. My question is how to automate and merge all the files instead of two files.
To avoid excessive number of parameters or length of parameters to a command line, you want to write your merge command so that it can take a previously merged output and merge another file. The description of merge in the original problem statement is quite scant, so I'll make the assumption that you can do this:
merge -o output_file input_file
Where output_file can be a previously merged file or a new file. If you can do that, then it would be simple to merge all of them by:
find drive_path -name "ip_file_*" -exec merge -o output_file {} \;
The order here is directory order in the file system. If a different order is needed, that will need to be specified.
ADDENDUM
If you need the files in timestamp order, then I would revamp this approach and create a merge command that accepts as an input a text file which lists all of the files to merge. Create this list of files using the information given in this post: https://superuser.com/questions/294161/unix-linux-find-and-sort-by-date-modified
Where your external merge tool is real_merge, and this tool writes merged output from two command-line arguments to stdout, the following recursive shell function will do the job:
merge_files() {
next=$1; shift
case $# in
0) cat "$next" ;;
1) real_merge "$next" "$1"
*) real_merge "$next" <(merge_files "$#")
esac
}
This approach is highly parallelized -- which means that it'll use as much CPU and disk IO as is available to it. Depending on your available resources, and your operating system's facility at managing those resources, this may or may not be a good thing.
The other approach is to use a temporary file:
swap() {
local var_curr=$1
local var_next=$2
local tmp
tmp="${!var_curr}"
printf -v "$var_curr" "${!var_next}"
printf -v "$var_next" "$tmp"
}
merge_files() {
local tempfile_curr=tempfile_A
local tempfile_next=tempfile_B
local tempfile_A="$(mktemp -t sort-wip-A.XXXXXX)"
local tempfile_B="$(mktemp -t sort-wip-B.XXXXXX)"
while (( $# )); do
if [[ -s ${!tempfile_curr} ]]; then
# we already populated our temporary file
real_merge "${!tempfile_curr}" "$1" "${!tempfile_next}"
swap tempfile_curr tempfile_next
elif (( $# >= 2 )); then
# only two arguments at all
real_merge "$1" "$2" "${!tempfile_curr}"
shift
else
# only one argument at all
cat "$1"
rm -f "$tempfile_A" "$tempfile_B"
return
fi
shift
done
# write output to stdout
cat "${!tempfile_curr}"
# ...and clean up.
rm -f "$tempfile_A" "$tempfile_B"
}
You can invoke it as: merge_files ip_file_* if the filenames' lexical sort order is accurate. (This will be true if their names are zero-padded, ie. ip_file_00001, but not true if they aren't padded). If not, you'll need to sort the stream of names first. If you're using bash and have GNU stat and sort available, this could be done as so:
declare -a filenames=()
while IFS='' read -r -d ' ' timestamp && IFS='' read -r -d '' filename; do
filenames+=( "$filename" )
done < <(stat --printf '%Y %n\0' ip_file_* | sort -n -z)
merge_files "${filenames[#]}"