Efficient file comparison of two large text files - database

Question

In our use case, we get large snapshot text files (tsv, csv etc.) from our customer (size around 30GB) with millions of records. The data looks like this:
ItemId (unique), Title, Description, Price etc.
shoe-id1, "title1", "desc1", 10
book-id-2, "title2", "desc2", 5
Whenever, we get a snapshot from a customer, we need to compute a "delta":
Inserted - the records that were inserted (only present in latest file and not the previous one),
Updated - Same Id but different value in any of the other columns
Deleted (only present in previous file and not the latest one).
(The data may be out of order in subsequent files and is not really sorted on any column).
We need to be able to run this multiple times a day for different customers.
We currently store all our data from snapshot file 1 into SQL server (12 shards (partitioned by customerId), containing a billion rows in all) and compute diffs using multiple queries when snapshot file 2 is received. This is proving to be very inefficient (hours, deletes are particularly tricky). I was wondering if there are any faster solutions out there. I am open to any technology (e.g. hadoop, nosql database). The key is speed (minutes preferably).

Answer 1

Normally, the fastest way to tell if an id appears in a dataset is by hashing, so I would make a hash which uses the id as the key and an MD5 checksum or CRC of the remaining columns as the element stored at that key. That should relieve the memory pressure if your data has many columns. Why do I think that? Because you say you have GB of data for millions of records, so I deduce each record must be of the order of kilobytes - i.e. quite wide.
So, I can synthesise a hash of 13M old values in Perl and a hash of 15M new values and then find the added, changed and removed as below.
#!/usr/bin/perl
use strict;
use warnings;
# Set $verbose=1 for copious output
my $verbose=0;
my $million=1000000;
my $nOld=13*$million;
my $nNew=15*$million;
my %oldHash;
my %newHash;
my $key;
my $cksum;
my $i;
my $found;
print "Populating oldHash with $nOld entries\n";
for($i=1;$i<=$nOld;$i++){
$key=$i-1;
$cksum=int(rand(2));
$oldHash{$key}=$cksum;
}
print "Populating newHash with $nNew entries\n";
$key=$million;
for($i=1;$i<=$nNew;$i++){
$cksum=1;
$newHash{$key}=$cksum;
$key++;
}
print "Part 1: Finding new ids (present in newHash, not present in oldHash) ...\n";
$found=0;
for $key (keys %newHash) {
if(!defined($oldHash{$key})){
print "New id: $key, cksum=$newHash{rkey}\n" if $verbose;
$found++;
}
}
print "Total new: $found\n";
print "Part 2: Finding changed ids (present in both but cksum different) ...\n";
$found=0;
for $key (keys %oldHash) {
if(defined($newHash{$key}) && ($oldHash{$key}!=$newHash{$key})){
print "Changed id: $key, old cksum=$oldHash{$key}, new cksum=$newHash{$key}\n" if $verbose;
$found++;
}
}
print "Total changed: $found\n";
print "Part 3: Finding deleted ids (present in oldHash, but not present in newHash) ...\n";
$found=0;
for $key (keys %oldHash) {
if(!defined($newHash{$key})){
print "Deleted id: $key, cksum=$oldHash{$key}\n" if $verbose;
$found++;
}
}
print "Total deleted: $found\n";
That takes 53 seconds to run on my iMac.
./hashes
Populating oldHash with 13000000 entries
Populating newHash with 15000000 entries
Part 1: Finding new ids (present in newHash, not present in oldHash) ...
Total new: 3000000
Part 2: Finding changed ids (present in both but cksum different) ...
Total changed: 6000913
Part 3: Finding deleted ids (present in oldHash, but not present in newHash) ...
Total deleted: 1000000
For the purposes of testing, I made the keys in oldHash run from 0..12,999,999 and the keys in newHash run from 1,000,000..16,000,000 then I can tell easily if it worked since the new keys should be 13,000,000..16,000,000 and the deleted keys should be 0..999,999. I also made the checksums alternate between 0 and 1 so that 50% of the overlapping ids should appear different.
Having done it in a relatively simple way, I can now see that you only need the checksum part to find the altered ids, so you could do parts 1 and 3 without the checksum to save memory. You could also do part 2 one element at a time as you loaded the data so you wouldn't need to load all the old and all the new ids into memory up front. Instead, you would load the smaller of the old and the new datasets and then check for changes one id at a time as you read the other set of ids into memory which would make it less demanding of memory.
Finally, if the approach works, it could readily be re-done in C++ for example, to speed it up further and reduce further the memory demands.