Why after the while loop I am only getting last row value? - loops

Question

This is the files I am reading,
#Log1
Time Src_id Des_id Address
0 34 56 x9870
2 36 58 x9872
4 38 60 x9874
6 40 62 x9876
8 42 64 x9878
#Log2
Time Src_id Des_id Address
1 35 57 x9871
3 37 59 x9873
5 39 61 x9875
7 41 63 x9877
9 43 65 x9879
This the code I wrote where I am reading line by line and then spliting it
#!usr/bin/perl
use warnings;
use strict;
my $log1_file = "log1.log";
my $log2_file = "log2.log";
open(IN1, "<$log1_file" ) or die "Could not open file $log1_file: $!";
open(IN2, "<$log2_file" ) or die "Could not open file $log2_file: $!";
my $i_d1;
my $i_d2;
my #fields1;
my #fields2;
while (my $line = <IN1>) {
#fields1 = split " ", $line;
}
while (my $line = <IN2>) {
#fields2 = split " ", $line;
}
print "#fields1\n";
print "#fields2\n";
close IN1;
close IN2;
Output I am getting
8 42 64 x9878
9 43 65 x9879
Output Desired
Time Src_id Des_id Address
0 34 56 x9870
2 36 58 x9872
4 38 60 x9874
6 40 62 x9876
8 42 64 x9878
9 43 65 x9879
Time Src_id Des_id Address
1 35 57 x9871
3 37 59 x9873
5 39 61 x9875
7 41 63 x9877
9 43 65 x9879
If I use push(#fields1 , split " ", $line); I am getting output like this,
Time Src_id Des_id Address 0 34 56 x9870 B 36 58 x9872 D 38 60 x9874 F 40 62 x9876 H 42 64 x9878
It should print whole array but printing just last row?
Also after this I need to compare both the "Times" part of both log & print in sequence way but don't know how to run both array simultaneously in while loop?
Please suggest in standard way without any modules because I need to run this in someone else server.

Answer 1

Following code demonstrates how to read and print log files
(OP does not specify why he splits lines into fields)
use strict;
use warnings;
use feature 'say';
my $fname1 = 'log1.txt';
my $fname2 = 'log2.txt';
my $div = "\t";
my $file1 = read_file($fname1);
my $file2 = read_file($fname2);
print_file($file1,$div);
print_file($file2,$div);
sub read_file {
my $fname = shift;
my #data;
open my $fh, '<', $fname
or die "Couldn't read $fname";
while( <$fh> ) {
chomp;
next if /^#Log/;
push #data, [split];
}
close $fh;
return \#data;
}
sub print_file {
my $data = shift;
my $div = shift;
say join($div,#{$_}) for #{$data};
}
Output
Time Src_id Des_id Address
0 34 56 x9870
2 36 58 x9872
4 38 60 x9874
6 40 62 x9876
8 42 64 x9878
Time Src_id Des_id Address
1 35 57 x9871
3 37 59 x9873
5 39 61 x9875
7 41 63 x9877
9 43 65 x9879
Let's assume that OP wants to merge two files into one with sorted lines on Time field
read files into %data hash with Time field as key
print header (#fields)
print hash values sorted on Time key
use strict;
use warnings;
use feature 'say';
my(#fields,%data);
my $fname1 = 'log1.txt';
my $fname2 = 'log2.txt';
read_data($fname1);
read_data($fname2);
say join("\t",#fields);
say join("\t",#{$data{$_}}) for sort { $a <=> $b } keys %data;
sub read_data {
my $fname = shift;
open my $fh, '<', $fname
or die "Couldn't open $fname";
while( <$fh> ) {
next if /^#Log/;
if( /^Time/ ) {
#fields = split;
} else {
my #line = split;
$data{$line[0]} = \#line;
}
}
close $fh;
}
Output
Time Src_id Des_id Address
0 34 56 x9870
1 35 57 x9871
2 36 58 x9872
3 37 59 x9873
4 38 60 x9874
5 39 61 x9875
6 40 62 x9876
7 41 63 x9877
8 42 64 x9878
9 43 65 x9879

Answer 2

Because #fields* gets overwritten during each loop. You need this:
while(my $line = <IN1>){
my #tmp = split(" ", $line);
push(#fields1, \#tmp);
}
foreach $item (#fields1){
print("#{$item}\n");
}
Then #fields1 contains references pointing to the splited array.
The final #fields1 looks like:
#fields1 = (
<ref> ----> ["0", "34", "56", "x9870"]
<ref> ----> ["2", "36", "58", "x9872"]
...
)
The print will print:
Time Src_id Des_id Address
0 34 56 x9870
2 36 58 x9872
4 38 60 x9874
6 40 62 x9876
8 42 64 x9878
And I guess it would be better if you do chomp($line).
But I'd like to simply do push(#fields1, $line). And split each array item when in comparison stage.
To compare the content of 2 files, I personally would use 2 while loops to read into 2 arrays just like what you have done. Then do the comparison in one for or foreach.

Answer 3

You can merge the log files using paste, and read the resulting merged file one line at a time. This is more elegant and saves RAM. Here is an example of a possible comparison of time1 and time2, writing STDOUT and STDERR into separate files. The example prints into STDOUT all the input fields if time1 < time2 and time1 < 4, otherwise prints a warning into STDERR:
cat > log1.log <<EOF
Time Src_id Des_id Address
0 34 56 x9870
2 36 58 x9872
4 38 60 x9874
6 40 62 x9876
8 42 64 x9878
EOF
cat > log2.log <<EOF
Time Src_id Des_id Address
1 35 57 x9871
3 37 59 x9873
5 39 61 x9875
7 41 63 x9877
9 43 65 x9879
EOF
# Paste files side by side, skip header, read data lines together, compare and print:
paste log1.log log2.log | \
tail -n +2 | \
perl -lane '
BEGIN {
for $file_num (1, 2) { push #col_names, map { "$_$file_num" } qw( time src_id des_id address ) }
}
my %val;
#val{ #col_names } = #F;
if ( $val{time1} < $val{time2} and $val{time1} < 4) {
print join "\t", #val{ #col_names};
} else {
warn "not found: #val{ qw( time1 time2 ) }";
}
' 1>out.tsv 2>out.log
Output:
% cat out.tsv
0 34 56 x9870 1 35 57 x9871
2 36 58 x9872 3 37 59 x9873
% cat out.log
not found: 4 5 at -e line 10, <> line 3.
not found: 6 7 at -e line 10, <> line 4.
not found: 8 9 at -e line 10, <> line 5.
The Perl one-liner uses these command line flags:
-e : Tells Perl to look for code in-line, instead of in a file.
-n : Loop over the input one line at a time, assigning it to $_ by default.
-l : Strip the input line separator ("\n" on *NIX by default) before executing the code in-line, and append it when printing.
-a : Split $_ into array #F on whitespace or on the regex specified in -F option.
SEE ALSO:
perldoc perlrun: how to execute the Perl interpreter: command line switches