Insert multiple rows after a check - sql-server

Question

I have a use case where I have 200 tables. I need to get the latest record from all the 200 tables store them in staging table. Now using each staging record need to check if it is already existing
in Final table and status column for that record is open or closed.
Initial table:(generic schema for all 200 tables)
ID, timestamp, name
Staging Table:
ID, timestamp, name
Final Table:
ID, timestamp, name, status, count
My approach:
Ordering by timestamp and limit 1 will give latest record in each table
Union all those latest record from 200 tables( 200 select statements with union)
staging table will now have 200 records
check each record if it is already existing in Final table, if existing and status="open" need
to increment the increment the count, if status="closed" or didn't find any match it should be
inserted as new record in Final table
came across TSQL "IF NOT EXISTS () BEGIN END ELSE BEGIN END" and while loop (not sure how use in this case)
All this process happens every 15 mins.
Any better approach or suggestions and how can I handle the last step of checking and inserting each row.
I am new to SQL.
More Info:
Those initial tables are in hive, where 200 different process trying to write simultaneously into tables, So table lock will happen for each write and remaining process should wait, so I had each table for each process. there will not be 200 records in staging every time, I gave the worst case. ideally it will be of range 0 to 10 at any given point, but it has to check all the 200 tables every 15 mins. this staging table from hive is brought into sql server and pushed to Final table to server other purpose

Answer 1

Although it sounds very strange that you have 200 Tables all with the same scheme, the following MERGE-Statement should achieve what you want.
WITH STAGING_DATA ([ID], [TIMESTAMP], [NAME])
as
(
SELECT TOP 1 [ID], [TIMESTAMP], [NAME] FROM <TABLE_1> ORDER BY [TIMESTAMP] DESC
UNION ALL
SELECT TOP 1 [ID], [TIMESTAMP], [NAME] FROM <TABLE_2> ORDER BY [TIMESTAMP] DESC
UNION ALL
...
UNION ALL
SELECT TOP 1 [ID], [TIMESTAMP], [NAME] FROM <TABLE_N> ORDER BY [TIMESTAMP] DESC
)
MERGE INTO <FINAL_TABLE> AS TARGET
USING (
SELECT [ID], [TIMESTAMP], [NAME] FROM STAGING_DATA
)
AS SOURCE ([ID], [TIMESTAMP], [NAME])
ON TARGET.ID = SOURCE.ID AND TARGET.STATUS = 'OPEN'
WHEN MATCHED THEN
UPDATE SET [COUNT] = ISNULL([COUNT], 0) + 1
WHEN NOT MATCHED BY TARGET THEN
INSERT ([ID], [TIMESTAMP], [NAME], [STATUS], [COUNT]) VALUES ([ID], [TIMESTAMP], [NAME], 'OPEN', 0)
The STAGING_DATA CTE is collecting all the data (the top 1 datatset from each table ordered by timestamp) and the merge statement takes care of merging the result into your final table. The merge statement also checks if a dataset with the same ID and the Status 'OPEN' already exists, in which case it just updates the according dataset in the final table by incrementing the counter by 1. Should the dataset not be found (or have another status than 'OPEN') we add a new dataset to the final table.
ORDER BY with UNION ALL Statements:
The ORDER BY does work with the UNION ALL as long as they are within the CTE. At least when I tested it on SQL Server 2012, 2017 and 2019 with the following setup:
WITH STAGING_DATA ([ID], [TIMESTAMP], [NAME])
as
(
SELECT TOP 1 [ID], [TIMESTAMP], [NAME]
FROM (VALUES
('1', '2021-01-01 00:00:00.000', 'Käser'),
('74', '2021-01-01 00:00:00.000', 'Valérie Maier'),
('2', '2021-01-01 00:00:00.000', 'Jäggi'),
('84', '2021-01-01 00:00:00.000', 'D'),
('83', '2021-01-01 00:00:00.000', 'Wyss')
) as DATA ([ID], [TIMESTAMP], [NAME])
ORDER BY [ID] ASC
UNION ALL
SELECT TOP 1 [ID], [TIMESTAMP], [NAME]
FROM (VALUES
('1', '2021-01-01 00:00:00.000', 'Käser'),
('74', '2021-01-01 00:00:00.000', 'Valérie Maier'),
('2', '2021-01-01 00:00:00.000', 'Jäggi'),
('84', '2021-01-01 00:00:00.000', 'D'),
('83', '2021-01-01 00:00:00.000', 'Wyss')
) as DATA ([ID], [TIMESTAMP], [NAME])
ORDER BY [ID] DESC
UNION ALL
SELECT TOP 2 [ID], [TIMESTAMP], [NAME]
FROM (VALUES
('1', '2021-01-01 00:00:00.000', 'Käser'),
('74', '2021-01-01 00:00:00.000', 'Valérie Maier'),
('2', '2021-01-01 00:00:00.000', 'Jäggi'),
('84', '2021-01-01 00:00:00.000', 'D'),
('83', '2021-01-01 00:00:00.000', 'Wyss')
) as DATA ([ID], [TIMESTAMP], [NAME])
ORDER BY [ID] ASC
UNION ALL
SELECT TOP 2 [ID], [TIMESTAMP], [NAME]
FROM (VALUES
('1', '2021-01-01 00:00:00.000', 'Käser'),
('74', '2021-01-01 00:00:00.000', 'Valérie Maier'),
('2', '2021-01-01 00:00:00.000', 'Jäggi'),
('84', '2021-01-01 00:00:00.000', 'D'),
('83', '2021-01-01 00:00:00.000', 'Wyss')
) as DATA ([ID], [TIMESTAMP], [NAME])
ORDER BY [ID] DESC
)
SELECT [ID], [TIMESTAMP], [NAME] FROM STAGING_DATA

Answer 2

Your approach to insert into the staging table would work logically, unfortunately in SQL Server you cannot UNION queries that contain an ORDER BY, so the following WILL NOT WORK
SELECT TOP(1) ID,[timestamp], [name] FROM dbo.TblA ORDER BY timestamp
UNION ALL
SELECT TOP(1) ID,[timestamp], [name] FROM dbo.TblB ORDER BY timestamp
UNION ALL
SELECT TOP(1) ID,[timestamp], [name] FROM dbo.TblC ORDER BY timestamp
If you want to do the UNION, you have to put the ORDER BY in a subquery and then do the UNION. It looks like this:
--INSERT INTO dbo.Staging (ID, [timestamp], [name])
SELECT q1.ID, q1.[timestamp], q1.[name] FROM
(SELECT TOP(1) ID, [timestamp], [name] FROM dbo.TblA ORDER BY [timestamp] DESC) AS q1
UNION ALL
SELECT q2.ID, q2.[timestamp], q2.[name] FROM
(SELECT TOP(1) ID, [timestamp], [name] FROM dbo.TblB ORDER BY [timestamp] DESC) AS q2
UNION ALL
SELECT q3.ID, q3.[timestamp], q3.[name] FROM
(SELECT TOP(1) ID, [timestamp], [name] FROM dbo.TblC ORDER BY [timestamp] DESC) AS q3
It is very ugly for sure and I don't know if you would be better off with 200 separate INSERT statements, but let's just stick with this approach for now. So you can stage those records now:
INSERT INTO dbo.Staging (ID, [timestamp], [name])
SELECT q1.ID, q1.[timestamp], q1.[name] FROM
(SELECT TOP(1) ID, [timestamp], [name] FROM dbo.TblA ORDER BY [timestamp] DESC) AS q1
UNION ALL
SELECT q2.ID, q2.[timestamp], q2.[name] FROM
(SELECT TOP(1) ID, [timestamp], [name] FROM dbo.TblB ORDER BY [timestamp] DESC) AS q2
UNION ALL
SELECT q3.ID, q3.[timestamp], q3.[name] FROM
(SELECT TOP(1) ID, [timestamp], [name] FROM dbo.TblC ORDER BY [timestamp] DESC) AS q3
I assume you TRUNCATE the staging table before each run so it will only contain the records you are about to load into the final table. I myself prefer to use a combination of INNER JOINs and LEFT OUTER JOINs to find what doesn't exist and what already exists (makes debugging and development easier in my opinion, but others may disagree), but there is the MERGE approach (I will not show that here).
So to load the final table you can do something like:
-- increment existing open records
-- the INNER JOIN guarantees an existing record that matches ID
UPDATE final SET final.[count] = final.[count] + 1
FROM dbo.Staging AS stage
INNER JOIN dbo.Final AS final ON final.ID = stage.ID AND final.[status] = 'open';
-- add closed records
-- same comment about the INNER JOIN
INSERT INTO dbo.Final(ID, [timestamp], [name], [status], [count])
SELECT final.ID, final.[timestamp], final.[name], 'open', 1
FROM dbo.Staging AS stage
INNER JOIN dbo.Final AS final ON final.ID = stage.ID AND final.[status] = 'closed'
-- no match, insert these records
-- the LEFT OUTER JOIN with the WHERE clause guarantees no matching record
INSERT INTO dbo.Final(ID, [timestamp], [name], [status], [count])
SELECT stage.ID, stage.[timestamp], stage.[name], 'open', 1
FROM dbo.Staging AS stage
LEFT OUTER JOIN dbo.Final AS final ON final.ID = stage.ID
WHERE final.ID IS NULL;
I just matched on the ID value, but you can modify what is considered a match easily in the ON clause.