Convert SQLite column data into rows in related tables - database

Question

I have a SQLite database that is currently just one table - it has been imported from a csv file. Two of the columns are semicolon separated lists of either categories or tags imported as TEXT fields. A typical row might look like this:
1 | Article Title | photography;my work | tips;lenses;gear | In this article I'll talk about...
How can I extract the category and tags columns, uniquely insert them into their own respective tables, and then create a relational table to tie them all together? So the end result would be something like:
Content
1 | Article Title | photography;my work | tips;lenses;gear | In this article I will talk about...
Categories
1 | photography
2 | my work
ContentCategories
1 | 1 | 1
2 | 2 | 1
This would effectively convert my one table database into a truly relational database.
I'm hoping this can be done both efficiently and quickly as there is a very large number of rows this solution would be used on.
This solution needs to be compatible with SQLite version 3.36 or later.

Answer 1

I believe that the following demonstrates how this can be done. However it is a 2 stage process and just for the categories. Similar two stage processes could be used for other columns.
Table/column names may differ.
/* Create Demo Environment */
DROP TABLE IF EXISTS contentcategories;
DROP TABLE IF EXISTS content;
DROP TABLE IF EXISTS category;
CREATE TABLE IF NOT EXISTS content (content_id INTEGER PRIMARY KEY,title TEXT, categories TEXT);
INSERT INTO content (title,categories) VALUES
('Article1','photography;my work;something;another;blah'),
('Article2','photography;thier work;not something;not another;not blah'),
('Article3','A;B;C;D;E;F;G;;');
CREATE TABLE IF NOT EXISTS category (category_id INTEGER PRIMARY KEY,category_name TEXT UNIQUE);
CREATE TABLE IF NOT EXISTS contentcategories (content_id_map,category_id_map, PRIMARY KEY (content_id_map,category_id_map));
/* Stage 1 populate the category table */
WITH
sep AS (SELECT ';'), /* The value separator */
justincase AS (SELECT 100), /* limiter for the number of iterations */
splt(value,rest) AS
(
SELECT
substr(categories,1,instr(categories,(SELECT * FROM sep))-1),
substr(categories,instr(categories,(SELECT * FROM sep))+1)||(SELECT * FROM sep)
FROM content
UNION ALL SELECT
substr(rest,1,instr(rest,(SELECT * FROM sep))-1),
substr(rest,instr(rest,(SELECT * FROM sep))+1)
FROM splt
WHERE length(rest) > 0
LIMIT (SELECT * FROM justincase) /* just in case limit iterations*/
)
INSERT OR IGNORE INTO category (category_name) SELECT value FROM splt WHERE length(value) > 0;
/* Show the resulktant Category table */
SELECT * FROM category;
/* Stage 2 populate the contentcategories mapping table */
WITH
sep AS (SELECT ';'), /* The value separator */
justincase AS (SELECT 100), /* limiter for the number of iterations */
splt(value,rest,contentid,categoryid) AS
(
SELECT
substr(categories,1,instr(categories,(SELECT * FROM sep))-1),
substr(categories,instr(categories,(SELECT * FROM sep))+1)||(SELECT * FROM sep),
content_id,
(SELECT category_id FROM category WHERE category_name = substr(categories,1,instr(categories,(SELECT * FROM sep))-1))
FROM content
UNION ALL SELECT
substr(rest,1,instr(rest,(SELECT * FROM sep))-1),
substr(rest,instr(rest,(SELECT * FROM sep))+1),
contentid,
(SELECT category_id FROM category WHERE category_name = substr(rest,1,instr(rest,(SELECT * FROM sep))-1))
FROM splt
WHERE length(rest) > 0
LIMIT (SELECT * FROM justincase) /* just in case limit iterations */
)
INSERT OR IGNORE INTO contentcategories SELECT contentid,categoryid FROM splt WHERE length(value) > 0;
/* Show the result of content joined via the mapping table with the category table */
SELECT content.*,category.*
FROM content
JOIN contentcategories ON content_id = content_id_map JOIN category ON category_id_map = category_id
;
/* Cleanup Demo Environment */
DROP TABLE IF EXISTS contentcategories;
DROP TABLE IF EXISTS content;
DROP TABLE IF EXISTS category;
So the content table has three rows each with a varying number of categories.
The first Stage uses recursion to split the values dropping the separators (the separator is coded as a CTE just the once so could be passed, like wise a value to limit the number of recursions can also be passed as it is a CTE).
The resulting CTE (splt) is then used for a SELECT INSERT to load the new category table with the extracted/split categories (OR IGNORE used to ignore any duplicates such as photography).
The second stage then splits the values again this time getting the id of the category from the new category table so that the mapping table contentcategories can be loaded.
After each stage a SELECT is used to show the result of the stage (these are included just to demonstrate).
So when the above is run then,
The **first result&& (after loading the category table) is:-
The second result is :-
i.e. everything is extracted via the joins as expected (not thoroughly checked though).
note that the erroneous ;; i.e. no value between the separators is discarded by WHERE length(value) > 0