Import variable time series data in SQL Server database - sql-server

Question

I've looked for a solution to this here on stack overflow and elsewhere, but can't find examples that deal with the volume I have to work with. If I have missed a solution to this that has been posted elsewhere, I'd be very grateful if someone could point me into the right direction.
I'm trying to import time series data from 45 different Excel worksheets (about 5 per Excel workbook). Each worksheet contains commodity price series, covering several years of daily prices for each commodity.
The raw Excel data has one row for each day for which prices might exist and one column for each commodity contract, which typically is a monthly future contract. The points for each contract hence are at least 30 each (but not much more) while the entire table has several thousand rows and 100+ columns.
I was able to build a SSIS package that reads the data and using unpivot, transforms the matrix into row based records with columns for:
Date, Price, Contract
The problem however is that in the unpivot transform I have to manually specify the destination column for each transformed input column. So 45 worksheets, each containing 100+ columns (some even several hundred) for contracts I'd be ending up 'hard-coding' those transforms manually for the next few days... On top, this is not as flexible/re-usable as I was hoping.
Example of the raw data attached (the entire Cocoa worksheet contains 9724 rows and 195 columns)
Here's how the unpivot for another single commodity is configured. The 'Destination Column' has to be filled in manually row-by-row.
I'm hoping that I have just missed the right steps in the unpivot configuration to make these columns dynamic. Ideally the SSIS solution can be re-used with equally formatted Excel workbooks later again. It's not required to run this on the Server as it's not a frequently recurring thing but rather once or twice per year max. So i can easily kick this off manually from within VS.
I'm trying to help an academic researcher who would otherwise spend a massive amount of time cleaning and analysing the data manually in Excel.

Answer 1

First you need to design and create the tables that are going to receive the data and experiment with some manual data entry to check the data model.
Make sure each spreadsheet has enough header information to know how to process the rows.
When that is done, I would save the sheets to text files with tab delimiters.
Next I would write a loading program in Perl. It reads the header rows first and determines the rules for inserting the rows into the database. Then each row gets converted into an insert into the database.
Here is an example from an invoice loading program I own (all rights):
if ($first) {
$obj->_hdr2keys(0); # convert spreadhseet header into a lookup
my $hdr = $obj->_copystruct($obj->{ar}[0]);
my #Hhdr = ('invoice header id');
my #Hcols = ('invhid');
my #Htypes = ('serial');
my #Dhdr = ('invoice detail id');
my #Dcols = ('invdid','invhid');
my #Dtypes = ('serial','integer');
for (my $col=0; $col <= $#{$hdr}; $col++) {
my $colname = lc($obj->_pomp($hdr->[$col]));
if ($colname eq 'invoicenumber') {
push #Hhdr, $hdr->[$col];
push #Hcols, $colname;
push #Htypes, 'char(32)';
}
elsif ($colname eq 'buysell') {
push #Hhdr, $hdr->[$col];
push #Hcols, $colname;
push #Htypes, 'boolean';
}
elsif ($colname eq 'suppliercustomer') {
push #Hhdr, $hdr->[$col];
push #Hcols, $colname;
push #Htypes, 'char(64)';
}
elsif ($colname eq 'date') {
push #Hhdr, 'Transaction Date';
push #Hcols, 'transactiondate';
push #Htypes, 'date';
}
elsif ($colname eq 'article') {
push #Dhdr, 'Article id';
push #Dcols, 'artid';
push #Dtypes, 'integer';
push #Dhdr, 'Article Description';
push #Dcols, 'description';
push #Dtypes, 'char(64)';
}
elsif ($colname eq 'qty') {
push #Dhdr, $hdr->[$col];
push #Dcols, $colname;
push #Dtypes, 'integer';
}
elsif ($colname eq 'priceexclbtw') {
push #Dhdr, $hdr->[$col];
push #Dcols, $colname;
push #Dtypes, 'double precision';
}
elsif ($colname eq 'btw') {
push #Dhdr, $hdr->[$col];
push #Dcols, $colname;
push #Dtypes, 'real';
}
}
$obj->_getset('INVHar',
['invoiceheader',
['PK','invhid'],
['__COLUMNS__'],
\#Hcols,
\#Htypes,
\#Hhdr
]
);
$obj->_getset('INVDar',
['invoicedetail',
['PK','invdid'],
['FK','invhid','invoiceheader','invhid'],
['FK','artid','article','artid'],
['__COLUMNS__'],
\#Dcols,
\#Dtypes,
\#Dhdr
]
);
}
$first = 0;
SALESROW: for (my $i=1; $i <= $#{$obj->{ar}}; $i++) {
my #Hrow = ('');
my #Drow = ('');
my $date = $obj->_selectar('', $i, 'Date');
$date =~ s/\-/\//g;
if ($date) {
$obj->_validCSV('date', $date)
or die "CSV format error date |$date| in file $file";
}
my $invtotal = ($obj->_selectar('', $i, 'Invoice Total incl. BTW'));
my $article = $obj->_selectar('', $i, 'Article');
$date or $article or next SALESROW;
if ($date) {
push #Hrow, $obj->_selectar('', $i, 'Invoice Number');
my $buysell = $obj->_selectar('', $i, 'Buy/Sell');
push #Hrow, ($buysell eq 'S') ? 1 : 0;
push #Hrow, $obj->_selectar('', $i, 'Supplier/Customer');
push #Hrow, $date;
push #{$obj->_getset('INVHar')}, \#Hrow;
$invhid++;
}
push #Drow, $invhid;
if ($article eq 'E0154') {
push #Drow, 1;
}
elsif ($article eq 'C0154') {
push #Drow, 2;
}
elsif ($article eq 'C0500') {
push #Drow, 3;
}
elsif ($article eq 'C2000') {
push #Drow, 4;
}
elsif ($article eq 'C5000') {
push #Drow, 5;
}
else {
die "unrecognised sales article $article\n"
. Dumper($obj->{ar}[$i]);
}
push #Drow, undef; # description is in article table
push #Drow, $obj->_selectar('', $i, 'Qty.');
push #Drow, $obj->_selectar('', $i, 'Price excl. BTW');
push #Drow, $obj->_selectar('', $i, 'BTW %');
push #{$obj->_getset('INVDar')}, \#Drow;
}
This creates header and detail records for invoices after the product table has already been loaded from another spreadsheet.
In the above example two array of arrays are created, INVHar and INVDar. When they are ready, the calling routine loads them into the database as follows. In this next code example the tables are created as well as the rows and also a metadb is updated for loading of future tables and managing foreign keys for existing tables. The array created in the previous snippet contains all information needed to create the table and insert the rows. There is also a simple routine _DBdatacnv that converts between the formats in the spreadsheet and the formats needed in the database. For example, the spreadsheet had currency symbols that need to be stripped before insertion.
sub _arr2db {
my ($obj) = #_;
my $ar = $obj->_copystruct($obj->_getset('ar'));
my $dbh = $obj->_getset('CDBh');
my $mdbh = $obj->_getset('MDBh');
my $table = shift #$ar;
$mdbh->{AutoCommit} = 0;
$dbh->{AutoCommit} = 0;
my #tables = $mdbh->selectrow_array(
"SELECT id FROM mtables
WHERE name = \'$table\'"
);
my $id = $tables[0] || '';
if ($id) {
$mdbh->do("DELETE FROM mcolumns where tblid=$id");
$mdbh->do("DELETE FROM mtables where id=$id");
}
# process constraints
my %constraint;
while ($#{$ar} >= 0
and $ar->[0][0] ne '__COLUMNS__') {
my $cts = shift #$ar;
my $type = shift #$cts;
if ($type eq 'PK') {
my $pk = shift #$cts;
$constraint{$pk} ||= '';
$constraint{$pk} .= ' PRIMARY KEY';
#$cts and die "unsupported compound key for $table";
}
elsif ($type eq 'FK') {
my ($col, $ft, $fk) = #$cts;
$ft && $fk or die "incomplete FK declaration in CSV for $table";
$constraint{$col} ||= '';
$constraint{$col} .=
sprintf( ' REFERENCES %s(%s)', $ft, $fk );
}
elsif ($type eq 'UNIQUE') {
while (my $uk = shift #$cts) {
$constraint{$uk} ||= '';
$constraint{$uk} .= ' UNIQUE';
}
}
elsif ($type eq 'NOT NULL') {
while (my $nk = shift #$cts) {
$constraint{$nk} ||= '';
$constraint{$nk} .= ' NOT NULL';
}
}
else {
die "unrecognised constraint |$type| for table $table";
}
}
shift #$ar;
unless ($mdbh->do("INSERT INTO mtables (name) values (\'$table\')")) {
warn $mdbh->errstr . ": mtables";
$mdbh->rollback;
die;
}
#tables = $mdbh->selectrow_array(
"SELECT id FROM mtables
WHERE name = \'$table\'"
);
$id = shift #tables;
$dbh->do("DROP TABLE IF EXISTS $table CASCADE")
or die $dbh->errstr;
my $create = "CREATE TABLE $table\n";
my $cols = shift #$ar;
my $types = shift #$ar;
my $desc = shift #$ar;
my $first = 1;
my $last = 0;
for (my $i=0; $i<=$#{$cols}; $i++) {
$last = 1;
if ($first) {
$first = 0;
$create .= "( "
}
else {
$create .= ",\n";
}
$create .= $cols->[$i]
. ' ' . $obj->_DBcnvtype($types->[$i]);
$constraint{$cols->[$i]}
and $create .= ' ' . $constraint{$cols->[$i]};
unless ($mdbh->do("INSERT INTO mcolumns (tblid,name,type,description)
values ($id,\'$cols->[$i]\',\'$types->[$i]\',\'$desc->[$i]\')"))
{
warn $mdbh->errstr;
$mdbh->rollback;
die;
}
}
$last and $create .= ')';
unless ($dbh->do($create)) {
warn $dbh->errstr;
$dbh->rollback;
die;
}
my $count = 0;
while (my $row = shift #$ar) {
$count++;
my $insert = "INSERT INTO $table (";
my $values = 'VALUES (';
my $first = 1;
for (my $i=0; $i<=$#{$cols}; $i++) {
my $colname = $cols->[$i];
unless (defined($constraint{$colname})
and $constraint{$colname} =~ /PRIMARY KEY/) {
if ($first) {
$first = 0;
}
else {
$insert .= ', ';
$values .= ', ';
}
$insert .= $colname;
my $val = $obj->_DBdatacnv('CSV', 'DB',
$types->[$i],$row->[$i]);
if ($val eq '%ABORT') {
$mdbh->rollback;
die;
}
$values .= $val;
}
}
$insert .= ')' . $values . ')';
unless ($dbh->do($insert)) {
warn $dbh->errstr;
warn $insert;
$mdbh->rollback;
die;
}
}
NOINSERT: $mdbh->commit;
$dbh->commit;
# warn "inserted $count rows into $table";
}
Update: ok I'll add the generic routine that converts from CSV to array ready for _arr2db above for all other cases I have for a system: The spreadsheets are first augmented with PK FK and other constraints followed by a header of column names for the database, a row of the database types (notional, actual are taken care of in _DBcnvdatatype) then a row of tags that go in the metadatabase and finally a token COLUMNS just before the rows of data to insert.
sub _csv2arr {
my ($obj, $csv ) = #_;
my $ar = [];
my $delim = $obj->_getset('csvdelim') || '\,';
my $basename = basename($csv);
$basename =~ s/\.txt$//;
$ar = [$basename];
open my $fh, $csv
or die "$!: $csv";
while (<$fh>) {
chomp;
my $sa = [];
#$sa = split /$delim/;
push #$ar, $sa;
}
close $fh;
$obj->{ar} = $ar;
}

Answer 2

I would do this as a series of nested loops:
Loop 1, iterate through all files in the folder. pass the file name to next loop
Loop 2 Open File, Iterate through sheets
Loop 3 in sheet X, Loop through the columns (A) > 1
Loop 4 - Loop through rows:
Read sheet X, Row B,
Get values from (Row B, Column 1) as Date, (Row 1, Column A) as Product. (Row B, Column A) as Price - write to destination.
End Loop 4
(optional, at the end of the column record some meta data about number of rows )
End of Loop 3
(optional, record some meta data about the number of columns in the sheet)
End of Loop 2
(optional, record some meta data about the number of sheets in file X)
End of Loop 1
(strongly suggested - record some meta data about the file X and number of sheets/rows/columns - you can test a sample later for your own confidence)
You may want to modify a copy of one of your files so that you can test for issues.
sheet empty
invalid data
missing header
text instead of price
This will give more confidence and will shorten the rework that is required when you discover new edge cases.
Final output table should be the denormalised data in 3 columns:
Date,
Product,
Price
EDIT
Here is a link that shows how you can dynamically loop through the columns of an Excel spreadsheet (and sheets) so that you can use this process to unpivot the data to a normalised form
Looping through Excel columns in SSIS