To read each file and execute the func in loop [closed] - c

Question

Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 9 years ago.
Improve this question
I have 5000 files in my hard-disk with name as ip_file_1,ip_file_2,....
I have a executable that can merge only 2 files. How can I write a script that takes all the file residing in the hardisk (whichs start with ip_file_*) and calls the function for merging all the files.
I have a 5000 files which are binaries that contain the logging information (time that each function call has taken). I have another executable that takes only two files and merges according to the timestamp and gives the merged output.
I execute with the format like the below,
./trace ip_file1 ip_file2 mergefile # I'm not using the trace tool. It's an example
I could use the executable to merge only two files. I thought of automating it to merge all the other files.
The merges has to be done in order (merged according to the timestamp). The logic to merge is already there. And the Output of the merge is sent to the file.
My question is not on how to merge the files. My question is how to automate and merge all the files instead of two files.

Answer 1

To avoid excessive number of parameters or length of parameters to a command line, you want to write your merge command so that it can take a previously merged output and merge another file. The description of merge in the original problem statement is quite scant, so I'll make the assumption that you can do this:
merge -o output_file input_file
Where output_file can be a previously merged file or a new file. If you can do that, then it would be simple to merge all of them by:
find drive_path -name "ip_file_*" -exec merge -o output_file {} \;
The order here is directory order in the file system. If a different order is needed, that will need to be specified.
ADDENDUM
If you need the files in timestamp order, then I would revamp this approach and create a merge command that accepts as an input a text file which lists all of the files to merge. Create this list of files using the information given in this post: https://superuser.com/questions/294161/unix-linux-find-and-sort-by-date-modified

Answer 2

Where your external merge tool is real_merge, and this tool writes merged output from two command-line arguments to stdout, the following recursive shell function will do the job:
merge_files() {
next=$1; shift
case $# in
0) cat "$next" ;;
1) real_merge "$next" "$1"
*) real_merge "$next" <(merge_files "$#")
esac
}
This approach is highly parallelized -- which means that it'll use as much CPU and disk IO as is available to it. Depending on your available resources, and your operating system's facility at managing those resources, this may or may not be a good thing.
The other approach is to use a temporary file:
swap() {
local var_curr=$1
local var_next=$2
local tmp
tmp="${!var_curr}"
printf -v "$var_curr" "${!var_next}"
printf -v "$var_next" "$tmp"
}
merge_files() {
local tempfile_curr=tempfile_A
local tempfile_next=tempfile_B
local tempfile_A="$(mktemp -t sort-wip-A.XXXXXX)"
local tempfile_B="$(mktemp -t sort-wip-B.XXXXXX)"
while (( $# )); do
if [[ -s ${!tempfile_curr} ]]; then
# we already populated our temporary file
real_merge "${!tempfile_curr}" "$1" "${!tempfile_next}"
swap tempfile_curr tempfile_next
elif (( $# >= 2 )); then
# only two arguments at all
real_merge "$1" "$2" "${!tempfile_curr}"
shift
else
# only one argument at all
cat "$1"
rm -f "$tempfile_A" "$tempfile_B"
return
fi
shift
done
# write output to stdout
cat "${!tempfile_curr}"
# ...and clean up.
rm -f "$tempfile_A" "$tempfile_B"
}
You can invoke it as: merge_files ip_file_* if the filenames' lexical sort order is accurate. (This will be true if their names are zero-padded, ie. ip_file_00001, but not true if they aren't padded). If not, you'll need to sort the stream of names first. If you're using bash and have GNU stat and sort available, this could be done as so:
declare -a filenames=()
while IFS='' read -r -d ' ' timestamp && IFS='' read -r -d '' filename; do
filenames+=( "$filename" )
done < <(stat --printf '%Y %n\0' ip_file_* | sort -n -z)
merge_files "${filenames[#]}"