Database: Intra-row calculations / calculated rows - database

Question

I plan to design a database model for a Business Intelligence system that stores business figures for a set of locations and a set of years.
Some of these figures should be calculated from other figures for the same year and the same location. In the following text I'll call figures that are not being calculated "basic figures". To store the basic figures, a table design with these columns would make sense:
| year | location_id | goods_costs | marketing_costs | warehouse_costs | administrative_costs |
Using this table I could create a view that calculates all other necessary figures:
CREATE VIEW all_figures
SELECT *,
goods_costs + marketing_costs + warehouse_costs + administrative_costs
AS total_costs
FROM basic_figures
This would be great if I didn't run into the following problems:
Most databases (including MySQL which I'm planning to use [edit: but which I'm not bound to]) have some kind of colum count or row size limit. Since I have to store a lot of figures (and have to calculate even more), I'd exceed this limit.
It is not uncommon that new figures have to be added. (Adding a figure would require changes to the table design. And as such changes ususally perform poorly they would block any access to the table for quite a long time.)
I also have to store additional information for each figure, e.g. a description and a unit (all figures are decimal numbers, but some might be in US$/EUR whereas others might be in %). I'd have to make sure that the basic_figures table, the all_figures view and the table containing the figure information are all correctly updated if anything changes. (This is more a data normalization problem than a technical/implementation problem.)
~~
Therefore I considered to use this table design:
+---------+-------------+-------------+-------+
| year | location_id | figure_id | value |
+---------+-------------+-------------+-------+
| 2009 | 1 | goods_costs | 300 |
...
This entity-attribute-value-like design could be a first solution for these three issues. However, it would also have a new downside: Calculations get messy. Really messy.
To build a view similar to the one above, I'd have to use a query like this:
(SELECT * FROM basic_figures_eav)
UNION ALL
(SELECT a.year_id, a.location_id, "total_costs", a.value + b.value + c.value + d.value
FROM basic_figures_eav a
INNER JOIN basic_figures_eav b ON a.year_id = b.year_id AND a.location_id = b.location_id AND b.figure_id = "marketing_costs"
INNER JOIN basic_figures_eav c ON a.year_id = c.year_id AND a.location_id = c.location_id AND c.figure_id = "warehouse_costs"
INNER JOIN basic_figures_eav d ON a.year_id = d.year_id AND a.location_id = d.location_id AND d.figure_id = "administrative_costs"
WHERE a.figure_id = "goods_costs");
Isn't that a beauty? And notice that this is just the query for ONE figure. All other calculated figures (of whom there are many as I wrote above) would also have to UNIONed with this query.
~~
After this long explanation of my problems, I now conculde with my actual questions:
Which database design would you suggest? / Would you use one of the two designs above? (If yes, which and why? If no, why?)
Do you have a suggestion for a completely other approach? (Which I would very, very much appreciate!)
Should the database actually be the one that does the calculations after all? Does it make more sense to move the calculation to the application logic and simply store the results?
By the way: I already asked a similar question on the MySQL forums. However, since answers were a bit sparse and this is not just a MySQL issue after all, I completely rewrote my quesion and posted it here. (So this is not a cross-post.) Here's the link to the thread there: http://forums.mysql.com/read.php?125,560752,560752#msg-560752

Answer 1

The question is (at least somewhat) DBMS specific.
If you can consider other DBMS, you might want to look at PostgreSQL and it's hstore datatype which is essentially a key/value pair.
The downsize of that is, that you lose datatype checking with as everything is stored as a string in the map.
The design that you are aiming at is called "Entity Attribute Value". You might want to find other alternatives as well.
Edit, here is an example on how this could be used:
Table setup
CREATE TABLE basic_figures
(
year_id integer,
location_id integer,
figures hstore
);
insert into basic_figures (year_id, location_id, figures)
values
(1, 1, hstore ('marketing_costs => 200, goods_costs => 100, warehouse_costs => 400')),
(1, 2, hstore ('marketing_costs => 50, goods_costs => 75, warehouse_costs => 250')),
(1, 3, hstore ('adminstrative_costs => 100'));
Basic select
select year_id,
location_id,
to_number(figures -> 'marketing_costs', 'FM999999') as marketing_costs,
to_number(figures -> 'goods_costs', 'FM999999') as goods_costs,
to_number(figures -> 'warehouse_costs', 'FM999999') as warehouse_costs,
to_number(figures -> 'adminstrative_costs', 'FM999999') as adminstrative_costs
from basic_figures bf;
It's probably easier to create a view for that that hides the conversion of the hstore values. The downside of that is, that the view needs to be re-created each time a new cost type is added.
Getting the totals
To get the sum of all costs for each year_id/location_id you can use the following statement:
SELECT year_id,
location_id,
sum(to_number(value, '99999')) as total
FROM (
SELECT year_id,
location_id,
(each(figures)).key,
(each(figures)).value
FROM basic_figures
) AS data
GROUP BY year_id, location_id;
year_id | location_id | total
---------+-------------+-------
1 | 3 | 100
1 | 2 | 375
1 | 1 | 700
That could be joined to the query above, but it's probably faster and easier to use if you create a function that calculates the total for all keys in a single hstore column:
Function to sum the totals
create or replace function sum_hstore(figures hstore)
returns bigint
as
$body$
declare
result bigint;
figure_values text[];
begin
result := 0;
figure_values := avals(figures);
for i in 1..array_length(figure_values, 1) loop
result := result + to_number(figure_values[i], '999999');
end loop;
return result;
end;
$body$
language plpgsql;
That function can easily be used in the first select:
select bf.year_id,
bf.location_id,
to_number(bf.figures -> 'marketing_costs', '99999999') as marketing_costs,
to_number(bf.figures -> 'goods_costs', '99999999') as goods_costs,
to_number(bf.figures -> 'warehouse_costs', '99999999') as warehouse_costs,
to_number(bf.figures -> 'adminstrative_costs', '99999999') as adminstrative_costs,
sum_hstore(bf.figures) as total
from basic_figures bf;
Automatic view creation
The following PL/pgSQL block can be used to (re-)create a view that contains one column for each key in the figures column plus the totals based on the sum_hstore function above:
do
$body$
declare
create_sql text;
types record;
begin
create_sql := 'create or replace view extended_figures as select year_id, location_id ';
for types in SELECT distinct (each(figures)).key as type_name FROM basic_figures loop
create_sql := create_sql || ', to_number(figures -> '''||types.type_name||''', ''9999999'') as '||types.type_name;
end loop;
create_sql := create_sql ||', sum_hstore(figures) as total from basic_figures';
execute create_sql;
end;
$body$
language plpgsql;
After running that function you can simply do a:
select *
from extended_figures
and you'll get as many columns as there are different cost types.
Note that there is no error checking at all if the values in the hstore are actually numbers. That could potentially be done with a trigger.

Answer 2

This is a way to "denormalise" (pivot) an EAV table without needing pivot. Note the left JOIN and the coalesce, which causes non-existant rows to a appear as "zero cost".
NOTE: I had to replace the quoting of the string literals to single quotes.
CREATE TABLE basic_figures_eav
( year_id INTEGER
, location_id INTEGER
, figure_id varchar
, value INTEGER
);
INSERT INTO basic_figures_eav ( year_id , location_id , figure_id , value ) VALUES
(1,1,'goods_costs', 100)
, (1,1,'marketing_costs', 200)
, (1,1,'warehouse_costs', 400)
, (1,1,'administrative_costs', 800)
, (1,2,'goods_costs', 100)
, (1,2,'marketing_costs', 200)
, (1,2,'warehouse_costs', 400)
, (1,3,'administrative_costs', 800)
;
SELECT x.year_id, x.location_id
, COALESCE (a.value,0) AS goods_costs
, COALESCE (b.value,0) AS marketing_costs
, COALESCE (c.value,0) AS warehouse_costs
, COALESCE (d.value,0) AS administrative_costs
--
, COALESCE (a.value,0)
+ COALESCE (b.value,0)
+ COALESCE (c.value,0)
+ COALESCE (d.value,0)
AS total_costs
-- need this to get all the {year_id,location_id} combinations
-- that have at least one tuple in the EAV table
FROM (
SELECT DISTINCT year_id, location_id
FROM basic_figures_eav
-- WHERE <selection of wanted observations>
) AS x
LEFT JOIN basic_figures_eav a ON a.year_id = x.year_id AND a.location_id = x.location_id AND a.figure_id = 'goods_costs'
LEFT JOIN basic_figures_eav b ON b.year_id = x.year_id AND b.location_id = x.location_id AND b.figure_id = 'marketing_costs'
LEFT JOIN basic_figures_eav c ON c.year_id = x.year_id AND c.location_id = x.location_id AND c.figure_id = 'warehouse_costs'
LEFT JOIN basic_figures_eav d ON d.year_id = x.year_id AND d.location_id = x.location_id AND d.figure_id = 'administrative_costs'
;
Result:
CREATE TABLE
INSERT 0 8
year_id | location_id | goods_costs | marketing_costs | warehouse_costs | administrative_costs | total_costs
---------+-------------+-------------+-----------------+-----------------+----------------------+-------------
1 | 3 | 0 | 0 | 0 | 800 | 800
1 | 2 | 100 | 200 | 400 | 0 | 700
1 | 1 | 100 | 200 | 400 | 800 | 1500
(3 rows)

Answer 3

I just want to point out that the second half of your query is needlessly complicated. You can do:
(SELECT a.year_id, a.location_id, "total_costs",
sum(a.value)
FROM basic_figures_eav a
where a.figure_id in ('marketing_costs', 'warehouse_costs', 'administrative_costs',
'goods_costs')
)
Although this uses an aggregation, with a composite index on year_id, location_id, and figure_id, the performance should be similar.
As for the rest of your question, there is a problem with databases limiting the number of columns. I would suggest that you put your base data in a table, with an auto-incremented primary key. Then, create summary tables, linked by the same primary key.
In many environments, you can recreate the summary tables once per day or once per night. If you need real time information, you can use stored procedures/triggers to update the data. That is, when data is updated or inserted, then it can be modified in the summary tables.
Also, I tried to find out if calculated/computed columns in SQL Server count against the maximum number of columns in the table (1,024). I wasn't able to find anything definitive. This is easy enough to test, but I'm not near a database right now.