Merging two file based on columns and sorting - c

Question

I have two files, FILE1 and FILE2, that have a different number of
columns and some columns in common. In both files the first column is
a row identifier. I want to merge the two files (FILE1 and FILE2)
without changing the order of the columns, and where there is a missing
value input the value '5'.
For example FILE1 (first column is the row ID, A1 is the first row, A2
the second, ...):
A1 1 2 5 1
A2 0 2 1 1
A3 1 0 2 2
The column names for FILE1 is (these are specified in another file),
Affy1
Affy3
Affy4
Affy5
which is to say that the value in row A1, column Affy1 is 1
and the value in row A3, column Affy5 is 2
v~~~~~ Affy3
A1 1 2 5 1
A2 0 2 1 1
A3 1 0 2 2
^~~~ Affy1
Similarly for FILE2
B1 1 2 0
B2 0 1 1
B3 5 1 1
And its column names,
Affy1
Affy2
Affy3
Meaning that
v~~~~~ Affy2
B1 1 2 0
B2 0 1 1
B3 5 1 1
^~~~ Affy1
I want to merge and sort columns based on the column names and put a
'5' for missing values. so the merged result would be as follows:
A1 1 5 2 5 1
A2 0 5 2 1 1
A3 1 5 0 2 2
B1 1 2 0 5 5
B2 0 1 1 5 5
B3 5 1 1 5 5
And the columns:
Affy1
Affy2
Affy3
Affy4
Affy5
Which is to say,
v~~~~~~~ Affy2
A1 1 5 2 5 1
A2 0 5 2 1 1
A3 1 5 0 2 2
B1 1 2 0 5 5
B2 0 1 1 5 5
B3 5 1 1 5 5
^~~~ Affy1
In reality I have over 700K columns and over 2K rows in each file. Thanks in advance!

Answer 1

The difficult part is ordering the headers when some of them appear only in one file. The best way I know is to build a directed graph using the Graph module and sort the elements topologically
Once that's done it's simply a matter of assigning the values from each file to the correct columns and filling the blanks with 5s
I've incorporated the headers as the first line of each data file, so this program works with this data
file1.txt
ID Affy1 Affy3 Affy4 Affy5
A1 1 2 5 1
A2 0 2 1 1
A3 1 0 2 2
file2.txt
ID Affy1 Affy2 Affy3
B1 1 2 0
B2 0 1 1
B3 5 1 1
And here's the code
consolidate_columns.pl
use strict;
use warnings 'all';
use Graph::Directed;
my #files = qw/ file1.txt file2.txt /;
# Make an array of two file handles
#
my #fh = map {
open my $fh, '<', $_ or die qq{Unable to open "$_" for input: $!};
$fh;
} #files;
# Make an array of two lists of header names
#
my #file_heads = map { [ split ' ', <$_> ] } #fh;
# Use a directed grapoh to sort all of the header names so thet they're
# still in the order that they were at the top of both files
#
my #ordered_headers = do {
my $g = Graph::Directed->new;
for my $f ( 0, 1 ) {
my $file_heads = $file_heads[$f];
$g->add_edge($file_heads->[$_], $file_heads->[$_+1]) for 0 .. $#$file_heads-1;
}
$g->topological_sort;
};
# Form a hash converting header names to column indexes for output
#
my %ordered_headers = map { $ordered_headers[$_] => $_ } 0 .. $#ordered_headers;
# Print the header and the reformed records from each file. Use the hash to
# convert the header names into column indexes
#
print "#ordered_headers\n";
for my $i ( 0 .. $#fh ) {
my $fh = $fh[$i];
my #file_heads = #{ $file_heads[$i] };
my #splice = map { $ordered_headers{$_} } #file_heads;
while ( <$fh> ) {
next unless /\S/;
my #columns;
#columns[#splice] = split;
$_ //= 5 for #columns[0 .. $#ordered_headers];
print "#columns\n";
}
}
output
ID Affy1 Affy2 Affy3 Affy4 Affy5
A1 1 5 2 5 1
A2 0 5 2 1 1
A3 1 5 0 2 2
B1 1 2 0 5 5
B2 0 1 1 5 5
B3 5 1 1 5 5

Answer 2

For the fun of it -- HTH
#!/usr/bin/perl
use warnings;
use strict;
use constant {A => 1, B => 2, BOTH =>3};
# I don't read data from file
my #columns = qw(Affy1 Affy2 Affy3 Affy4 Affy5);
my #locations = (BOTH, B, BOTH, A, A);
my #contentA = (["A1", 1, 2, 5, 1],
["A2", 0, 2, 1, 1],
["A3", 1, 0, 2, 2]);
my #contentB = (["B1", 1, 2, 0],
["B2", 0, 1, 1],
["B3", 5, 1, 1]);
#I assume both files have the same amount of lines
my #ares = ();
my #bres = ();
for(my $i = 0; $i < #contentA; ++$i){
# this uses a lot of memory whith huge amounts of data
# maybe you write this in two temp result files and cat them
# together at the end
# another alternative would be to iterate first over
# file A and then over file A
my #row_a = ();
my #row_b = ();
push #row_a, shift #{$contentA[$i]}; #id
push #row_b, shift #{$contentB[$i]}; #id
foreach my $loc (#locations){
if(A == $loc){
push #row_a, shift #{$contentA[$i]};
push #row_b, 5;
}
if(B == $loc){
push #row_a, 5;
push #row_b, shift #{$contentB[$i]};
}
if(BOTH == $loc){
push #row_a, shift #{$contentA[$i]};
push #row_b, shift #{$contentB[$i]};
}
}
push #ares, \#row_a;
push #bres, \#row_b;
}
foreach my $ar(#ares){
print join " ", #{$ar};
print "\n";
}
foreach my $br(#bres){
print join " ", #{$br};
print "\n";
}
print join("\n", #columns);
print "\n";