sqlite diff of before and after transaction - c

Question

I'm now using the sqldiff tool to calculate the difference between my database before a transaction was executed and after it was executed. I'm doing this as follows:
Copy the entire sqlite file to a tmp location (old-database.sqlite)
Perform the transaction (for ex. INSERT, SCHEMA CHANGE, ...)
Than sqldiff --primary-key old-database.sqlite database.sqlite
This outputs the changes that the transaction has done. Although this works perfectly because my database isn't that large, a few megabytes maximum. I think this could be done more efficiently.
Is there any mechanism that I can use to get the same output without copying the database. Maybe something with the journal?
Kind regards,
Daan

Answer 1

Perhaps a little awkward to implement and perhaps at some stage (size wise) inefficient, but you could use a combination of TRIGGERS (4 per monitored table is the awkward part) and a before after snapshot of sqlite_master.
That is you could, per table to be monitored, have a copy of that table (structure wise) appended with two columns. 1 to indicate the action (inserted, before update, after update or deletion), the 2nd (optional) to record the timestamp.
Then for each table to be monitored have AFTER INSERT, BEFORE DELETE and AFTER and BEFORE UPDATE TRIGGERS to copy the add an entry to the table's log table, which would be emptied before the transaction (if wanted, if not then timestamp would probably be a required column).
For schema changes again you could have a copy of sqlite_master before and then compare it against sqlite_master after the transaction (you can't have TRIGGERS applied to system tables).
Example, consider the following example/demo:-
CREATE TABLE IF NOT EXISTS mytable (id INTEGER PRIMARY KEY, mycolumn TEXT);
CREATE TABLE IF NOT EXISTS mytablelog AS SELECT * FROM mytable WHERE 0 = 1;
ALTER TABLE mytablelog ADD COLUMN logtype INTEGER;
ALTER TABLE mytablelog ADD COLUMN timestamp INTEGER TEXT;
CREATE TABLE IF NOT EXISTS schema_log AS SELECT * FROM sqlite_master WHERE 0=1;
CREATE TABLE IF NOT EXISTS pretrans_schema AS SELECT * FROM sqlite_master WHERE 0=1;
ALTER TABLE schema_log ADD COLUMN logtype INTEGER;
ALTER TABLE schema_log ADD COLUMN timestamp INTEGER TEXT;
CREATE TRIGGER IF NOT EXISTS mytable_inserts AFTER INSERT ON mytable
BEGIN
INSERT INTO mytablelog SELECT *,0,strftime('%s','now') FROM mytable WHERE id = new.id;
END
;
CREATE TRIGGER IF NOT EXISTS mytable_deletions BEFORE DELETE ON mytable
BEGIN
INSERT INTO mytablelog SELECT *,1,strftime('%s','now') FROM mytable WHERE id = old.id;
END
;
CREATE TRIGGER IF NOT EXISTS mytable_preupdates BEFORE UPDATE ON mytable
BEGIN
INSERT INTO mytablelog SELECT *,2,strftime('%s','now') FROM mytable WHERE id = old.id;
END
;
CREATE TRIGGER IF NOT EXISTS mytable_postupdates AFTER UPDATE ON mytable
BEGIN
INSERT INTO mytablelog SELECT *,3,strftime('%s','now') FROM mytable WHERE id = new.id;
END
;
-- SELECT * FROM sqlite_master WHERE name LIKE 'sqlite_%';
/* BEFORE TRANSACTION PREPATION */
DELETE FROM mytablelog;
DELETE FROM schema_log;
DELETE FROM pretrans_schema;
INSERT INTO pretrans_schema SELECT * FROM sqlite_master;
/* DO SOMETHING AKA THE TRANSACTIONS */
CREATE TABLE IF NOT EXISTS newtable (id INTEGER PRIMARY KEY, acolumn TEXT);
INSERT INTO mytable (mycolumn) VALUES ('Mary')
-- ,('Fred'),('Jane'),('Anne'),('Alfred'),('George'),('Alan')
,('Susan'),('Betty'),('Catherine'),('John')
,(100),(200)
;
UPDATE mytable SET mycolumn = mycolumn||' has the detected letter.' WHERE mycolumn LIKE '%n%';
DELETE FROM mytable WHERE CAST(mycolumn AS INTEGER) > 0;
/* AFTER TRANSACTION */
SELECT rowid,'sm',* FROM sqlite_master UNION ALL SELECT rowid,'pt',* FROM pretrans_schema ORDER BY type,name; /* FOR DEMO/TESTING */
/* Get items added to the schema */
INSERT INTO schema_log SELECT *,4,strftime('%s','now') FROM sqlite_master WHERE name NOT IN (SELECT name FROM pretrans_schema);
/* Get items deleted from the schema */
INSERT INTO schema_log SELECT *,5,strftime('%s','now') FROM pretrans_schema WHERE name NOT IN (SELECT name FROM sqlite_master);
/* get original schema if schema updates */
INSERT INTO schema_log SELECT *,6,strftime('%s','now') FROM sqlite_master AS s
WHERE (sql <> (SELECT sql FROM pretrans_schema WHERE type = s.type AND name = s.name )) AND type IS NOT NULL /* AND NAME <> 'pretrans_schema' optional */
;
/* get current schema if schema updates */
INSERT INTO schema_log SELECT *,7,strftime('%s','now') FROM pretrans_schema AS s
WHERE (sql <> (SELECT sql FROM sqlite_master WHERE type = s.type AND name = s.name)) AND type IS NOT NULL /* AND NAME <> 'pretrans_schema' */
;
SELECT * FROM schema_log;
SELECT * FROM mytablelog;
Results
1 pre and post schema (part of)
Note the highlighted line, there is no pt entry, showing that the table has been added (no sm entry then it would have been deleted).
2 - The schema changes log (schema_log table)
4 indicates new item in the schema (5 deleted, 6 before update, 7 after update)
So the table newtable has been added as part of the transaction.
3 - The mytable transactions;
O's are insertions, 1's are deleted 2 and 3 (paired) pre and post update respectively.

Answer 2

Thank you for thinking along and especially #MikeT for the long and thourough answer. I have tried your answer and while it works it seemed not elegant to me and have a lot of overhead.
After a lot of search I have found that the solution is baked into SQLite but it must be enabled at compile time. It is called 'The Session Extension' and is exacly what I needed:
https://www.sqlite.org/sessionintro.html