Loading of large CSV file into bash associative array slow/stuck - arrays

Question

I have a very large CSV file (~10mil rows) with 2 numeric column representing ids. The requirement is: given the first id, return very fast the second id.
I need to get the CSV to behave like a map structure and it has to be in memory. I couldn't find a way to expose awk variables back to the shell so I thought of using bash associative arrays.
The problem is that loading the csv into an associative array gets very slow/stuck after ~8 mil rows. I've been trying to eliminate the causes of slowdown that I could think of: file reading/IO, associative arraylimitations. So, I have a couple of functions that read the file into an associative array, but all of them have the same slowness problem.
Here is the test data
loadSplittedFilesViaMultipleArrays -> assumes the original file was split into smaller files (1 mil rows) and uses a while read loop to build 4 associative arrays (max 3 mil records each)
loadSingleFileViaReadarray -> uses readarray to read the original file into a temp array and then goes through that to build the associative array
loadSingleFileViaWhileRead -> uses a while read loop to build the associative array
But I can't seem to figure it out. Maybe this way of doing it is completely wrong... Can anyone pitch in with some suggestions?

Answer 1

Bash is the wrong tool for an associative array of this size. Consider using a language more suited (Perl, Python, Ruby, PHP, js, etc etc)
For a Bash only environment you could use a sqlite3 sql database which is usually installed with Bash. (It is not POSIX however)
First you would create the database from your csv file. There are many ways to do this (Perl, Python, Ruby, GUI tools) but this is simple enough to do interactively in sqlite3 command line shell (exp.db must not exist at this point):
$ sqlite3 exp.db
SQLite version 3.19.3 2017-06-27 16:48:08
Enter ".help" for usage hints.
sqlite> create table mapping (id integer primary key, n integer);
sqlite> .separator ","
sqlite> .import /tmp/mapping.csv mapping
sqlite> .quit
Or, pipe in the sql statements:
#!/bin/bash
cd /tmp
[[ -f exp.db ]] && rm exp.db # must be a new db as written
echo 'create table mapping (id integer primary key, n integer);
.separator ","
.import mapping.csv mapping' | sqlite3 exp.db
(Note: as written, exp.db must not exist or you will get INSERT failed: UNIQUE constraint failed: mapping.id. You can write it so the database exp.db is updated rather than created by the csv file, but you would probably want to use a language like Python, Perl, Tcl, Ruby, etc to do that.)
In either case, that will create an indexed database mapping the first column onto the second. The import will take a little while (15-20 seconds with the 198 MB example) but it creates a new persistent database from the imported csv:
$ ls -l exp.db
-rw-r--r-- 1 dawg wheel 158105600 Nov 19 07:16 exp.db
Then you can quickly query that new database from Bash:
$ time sqlite3 exp.db 'select n from mapping where id=1350044575'
1347465036
real 0m0.004s
user 0m0.001s
sys 0m0.001s
That takes 4 milliseconds on my older iMac.
If you want to use Bash variables for your query you can concatenate or construct the query string as needed:
$ q=1350044575
$ sqlite3 exp.db 'select n from mapping where id='"$q"
1347465036
And since the db is persistent, you can just compare file times of the csv file to the db file to test whether you need to recreate it:
if [[ ! -f "$db_file" || "$csv_file" -nt "$db_file" ]]; then
[[ -f "$db_file" ]] && rm "$db_file"
echo "creating $db_file"
# create the db as above...
else
echo "reusing $db_file"
fi
# query the db...
More:
sqlite tutorial
sqlite home

Answer 2

Inspired by #HuStmpHrrr's comment, I thought about another, maybe simpler alternative.
You can use GNU Parallel to split the file up into 1MB (or other) sized chunks and then use all your CPU cores to search each of the resulting chunks in parallel:
parallel --pipepart -a mapping.csv --quote awk -F, -v k=1350044575 '$1==k{print $2;exit}'
1347465036
Takes under a second on my iMac and that was the very last record.

Answer 3

I made a little Perl-based TCP server that reads the CSV into a hash and then sits looping forever doing lookups for requests coming via TCP from clients. It is pretty self-explanatory:
#!/usr/bin/perl
use strict;
use warnings;
################################################################################
# Load hash from CSV at startup
################################################################################
open DATA, "mapping.csv";
my %hash;
while( <DATA> ) {
chomp $_;
my ($field1,$field2) = split /,/, $_;
if( $field1 ne '' ) {
$hash{$field1} = $field2;
}
}
close DATA;
print "Ready\n";
################################################################################
# Answer queries forever
################################################################################
use IO::Socket::INET;
# auto-flush on socket
$| = 1;
my $port=5000;
# creating a listening socket
my $socket = new IO::Socket::INET (
LocalHost => '127.0.0.1',
LocalPort => $port,
Proto => 'tcp',
Listen => 5,
Reuse => 1
);
die "cannot create socket $!\n" unless $socket;
while(1)
{
# waiting for a new client connection
my $client_socket = $socket->accept();
my $data = "";
$client_socket->recv($data, 1024);
my $key=$data;
chomp $key;
my $reply = "ERROR: Not found $key";
if (defined $hash{$key}){
$reply=$hash{$key};
}
print "DEBUG: Received $key: Replying $reply\n";
$client_socket->send($reply);
# notify client that response has been sent
shutdown($client_socket, 1);
}
So, you save the code above as go.pl and then make it executable with:
chmod +x go.pl
then start the server in the background with:
./go.pl &
Then, when you want to do a lookup as a client, you send your key to localhost:5000 using the standard socat utility like this:
socat - TCP:127.0.0.1:5000 <<< "1350772177"
1347092335
As a quick benchmark, it does 1,000 lookups in 8 seconds.
START=$SECONDS; tail -1000 *csv | awk -F, '{print $1}' |
while read a; do echo $a | socat - TCP:127.0.0.1:5000 ; echo; done; echo $START,$SECONDS
It could probably be speeded up by a slight change to handle multiple keys to lookup per request to reduce socket connection and teardown overhead.