How to efficiently compare two subsets of rows of a large table? - database

Question

I use PostgreSQL 9.6 and my table schema is as follows: department, key, value1, value2, value3, ... Each department has hundreds of millions of unique keys, but the set of keys is more or less the same for all departments. It's possible that some keys don't exist for some departments, but such situations are rare.
I would like to prepare a report that for two departments points out differences in values for each key (comparison involves some logic based only on values for the key).
My first approach was to write an external tool in python that:
creates a server-side cursor for query: SELECT * FROM my_table WHERE department = 'ABC' ORDER BY key;
creates another server-side cursor for query: SELECT * FROM my_table WHERE department = 'XYZ' ORDER BY key;
iterates over both cursors, and compares the values.
It worked fine, but I thought it will be more efficient to perform the comparison inside a stored procedure in PostgreSQL. I wrote a stored procedure that takes two cursors as arguments, iterates over them and compares the values. Any differences are written into a temporary table. At the end, the external tool iterates just over the temporary table - there shouldn't be many rows there.
I thought that the latter approach would be more efficient, because it doesn't require transferring lots of data outside the database.
To my surprise, it turned out to be slower by almost 40%.
To isolate the problem I compared the performance of iterating a cursor inside a stored procedure, and in python:
FETCH cur_1 INTO row_1;
WHILE (row_1 IS NOT NULL) LOOP
rows = rows + 1;
FETCH FROM cur_1 INTO row_1;
END LOOP;
vs.
conn = psycopg2.connect(PG_URI)
cur = conn.cursor('test')
cur.execute(query)
cnt = 0
for row in cur:
cnt += 1
Query was the same in both cases. Again, the external tool was faster.
My hypothesis is that this is because the stored procedure fetches rows one-by-one (FETCH FROM curs_1 INTO row_1) while the application fetches rows in batches of 2000. But I couldn't find a way to FETCH a batch of rows from cursor inside a PgSQL procedure. Thus, I can't test the hypothesis.
So my question is it possible to speed up my stored procedure?
What is the best approach for problems like this?

Answer 1

Why can you not do a self-join rather than using cursors? Something like:
SELECT t1.key, t1.value1 - t2.value1 as diff1,
t1.value2 - t2.value2 as diff2, ...
FROM my_table t1 inner join my_table t2 on t1.key = t2.key
WHERE t1.department = 'XYZ' and t2.department = 'ABC'
UNION
SELECT t1.key, t1.value1 as diff1,
t1.value2 as diff2, ...
FROM my_table t1 WHERE NOT EXISTS (SELECT 1 FROM my_table t2 WHERE
t1.key = t2.key AND t2.dept = 'ABC') AND t1.dept = 'XYZ'
UNION
SELECT t1.key, t1.value1 as diff1,
t1.value2 as diff2, ...
FROM my_table t1 WHERE NOT EXISTS (SELECT 1 FROM my_table t2 WHERE
t1.key = t2.key AND t2.dept = 'XYZ') AND t1.dept = 'ABC';
The first part deals with all the common cases and the two unions pick up the missing values. I would have thought this would be much faster than a cursor approach.

Answer 2

This might be faster as it will only return those rows that are different in at least one of the values:
select *
from (
SELECT key, value1, value2, value3
FROM my_table
WHERE department = 'ABC'
) d1
full join (
SELECT key, value1, value2, value3
FROM my_table
WHERE department = 'XYZ'
) d2 using (key)
where d1 is distinct from d2;